Commit Graph

62 Commits

Author SHA1 Message Date
Kenneth Graunke
85d30846db nir: Print divergence status of SSA values if analysis was ever run.
After running divergence analysis, we include "div" or "con" for each
SSA def's divergence/convergence status:

	vec1 32 div ssa_35 = fddy ssa_34
	vec1 32 con ssa_36 = fddy ssa_6.x

We omit this before the first time divergence analysis has been run.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15445>
2022-03-21 22:46:34 -07:00
Timur Kristóf
4b99b528f5 nir: Introduce workgroup_index and ability to lower workgroup_id to it.
The workgroup_index is intended for situations when a 3 dimensional
workgroup_id is not available on the HW, but a 1 dimensional index is.
In this case, we can use lower the 3D ID to use this.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15103>
2022-03-08 17:36:31 +00:00
Samuel Pitoiset
74b932f8d3 nir: add nir_intrinsic_load_vrs_rates_amd
This intrinsic specific to RADV will be used to load VRS rates from
an user SGPR when RADV_FORCE_VRS is enabled by the application.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14713>
2022-02-16 08:11:12 +01:00
Jason Ekstrand
e8acc5a7ea nir: Add a new sample_pos_or_center system value
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14198>
2021-12-17 16:02:16 +00:00
Marek Olšák
2785141c16 nir: add nir_has_divergent_loop function
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13966>
2021-12-11 20:07:35 +00:00
Marek Olšák
2ab310b78b nir: handle more intrinsics in divergence analysis
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13966>
2021-12-11 20:07:35 +00:00
Jason Ekstrand
956199e870 nir: s/nir_var_mem_image/nir_var_image/g
We typically use nir_var_mem_* for stuff that has an explicit byte-based
memory layout.  Images are opaque.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13386>
2021-10-16 03:47:10 +00:00
Caio Marcelo de Oliveira Filho
de3705edb0 nir: Add nir_var_mem_image
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4743>
2021-10-15 14:58:55 +00:00
Bas Nieuwenhuizen
0d8bd8518d nir: Support ray launch size in divergence analysis.
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12592>
2021-09-21 01:53:39 +00:00
Bas Nieuwenhuizen
56b06c09b4 nir: Add AMD rt intrinsics.
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/12592>
2021-09-21 01:53:39 +00:00
Caio Marcelo de Oliveira Filho
27697d5eb8 nir/divergence_analysis: Handle Task/Mesh shaders
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10600>
2021-08-28 03:56:42 +00:00
Timur Kristóf
1bbea90f50 aco, nir, ac: Simplify sequence of getting initial NGG VS edge flags.
Instead of v_bfe + v_lshl_or for each vertex, get all 3 edge flags
at once of every vertex. This takes fewer VALU instructions than
previously.

Fossil DB results on Sienna Cichlid (with NGGC on):

Totals from 56917 (44.24% of 128647) affected shaders:
CodeSize: 161028288 -> 158751628 (-1.41%)
Instrs: 30917985 -> 30519571 (-1.29%)
Latency: 130617204 -> 129975532 (-0.49%); split: -0.50%, +0.01%
InvThroughput: 21280238 -> 20927401 (-1.66%)
Copies: 3011120 -> 3011125 (+0.00%); split: -0.00%, +0.00%

No Fossil DB changed with NGGC off.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11908>
2021-08-02 11:38:25 +00:00
Timur Kristóf
48e638ab29 nir: Add AMD specific intrinsics for NGG shader based culling.
The new intrinsics fall into the following categories:

1. New viewport intrinsics:
For missing components that we need.
RADV will emit new SGPR arguments which will contain the
viewport information for culling shaders. These are used to
compute the screen space coordinates for small primitive culling.

2. load_cull_xxx:
Load the culling settings in runtime.
These will be a new SGPR argument in RADV.

3. overwrite_xxx:
These are needed because system values such as vertex and
instance ID are not writeable, but we need to change them
after repacking shader invocations of VS and TES.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10525>
2021-07-13 23:56:33 +00:00
Timur Kristóf
43ce80a58f nir: Add AMD-specific byte and lane permute intrinsics.
These map directly to v_perm_b32 and v_permlane_b32.
Unfortunately there is no corresponding NIR opcode or
intrinsics, and it's too tedious to puzzle these things
together from the existing NIR instructions.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Tony Wasserka <tony.wasserka@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11072>
2021-06-09 16:48:51 +00:00
Caio Marcelo de Oliveira Filho
c8a7bd0dc8 nir: Rename WORK_GROUP (and similar) to WORKGROUP
Be consistent with other usages in Vulkan and SPIR-V, and the recently
added workgroup_size field.

Acked-by: Emma Anholt <emma@anholt.net>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Acked-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11190>
2021-06-07 22:34:42 +00:00
Caio Marcelo de Oliveira Filho
a71a780598 nir: Rename nir_intrinsic_load_local_group_size to nir_intrinsic_load_workgroup_size
Acked-by: Emma Anholt <emma@anholt.net>
Acked-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Acked-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11190>
2021-06-07 22:34:42 +00:00
Bas Nieuwenhuizen
2d6a6469b8 nir: Add bvh64_intersect_ray_amd intrinsic.
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10818>
2021-05-18 23:01:47 +02:00
Bas Nieuwenhuizen
aa82f91c38 nir: Add load_sbt_amd intrinsic.
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9767>
2021-05-18 18:29:36 +00:00
Timur Kristóf
0d6b6c850f nir: Add AMD specific intrinsics for merged shaders and NGG.
These intrinsics represent what the hardware can actually do.
Lowering our shaders to use these intrinsics will allow us to
deal with mapping the classic VS, TES, GS (and the future MS)
stages to the hardware capabilities using NIR, which makes our
backend compilers simpler.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10740>
2021-05-12 13:47:04 +00:00
Timur Kristóf
641707a807 nir: Allow load_primitive_id in VS in nir_divergence_analysis.
The lowered NIR code of NGG VS shaders uses this intrinsic
when the VS has to export the primitive ID.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10740>
2021-05-12 13:47:04 +00:00
Alyssa Rosenzweig
db2f6b87a3 nir/divergence_anlysis: Add intrinsics for Bifrost
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10022>
2021-05-07 18:20:30 +00:00
Iago Toral Quiroga
aebb47b7d1 compiler/nir: add a divergence analysis option for non-uniform workgroup id
The V3D hardware allows us to pack multiple workgroups together to avoid
wasting execution lanes in shader cores.

For example, if we dispatch 16 workgroups with a local size of 1 element, we
can pack all 16 workgroups in a single 16-wide dispatch where each lane
executes a different workgroup, instead of 16 1-wide dispatches.

When we do this, we don't have a uniform workgroup id any more.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10541>
2021-05-04 15:53:23 +00:00
Jason Ekstrand
1ba9c262fd nir: Add image atomic_fmin/fmax intrinsics
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8750>
2021-03-18 00:13:40 +00:00
Timur Kristóf
4c5c610f1d nir: Add AMD-specific Geometry Shader related intrinsics.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9201>
2021-03-17 12:42:23 +00:00
Timur Kristóf
38df949f98 nir: Add tessellation related AMD-specific intrinsics.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9201>
2021-03-17 12:42:23 +00:00
Timur Kristóf
eee3435757 nir: Add AMD-specific buffer load/store intrinsics.
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9201>
2021-03-17 12:42:23 +00:00
Eric Anholt
1e5ef4c60c nir: Add a nir_src_is_undef() helper, like nir_src_is_const().
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9345>
2021-03-03 00:51:44 +00:00
Arcady Goldmints-Orlov
8fb6cbdcb6 nir: store the results of divergence analysis on loops
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7726>
2021-02-01 08:11:48 +00:00
Arcady Goldmints-Orlov
019449dad7 nir: handle v3d intrinsics in divergence analysis
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7726>
2021-02-01 08:11:48 +00:00
Arcady Goldmints-Orlov
349e4f3c65 nir: add more intrinsics to divergence analysis
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7726>
2021-02-01 08:11:48 +00:00
Rhys Perry
30f40364f6 nir,spirv: allow non-uniform OpArrayLength
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7969>
2021-01-27 13:00:33 +00:00
Daniel Schürmann
08fbd5d454 nir/divergence_analysis: mark load_push_constant as uniform
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8439>
2021-01-12 14:46:13 +00:00
Rhys Perry
2d2decc905 nir: add sparse_residency_code_and
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7774>
2021-01-06 20:36:38 +00:00
Rhys Perry
4cbdf9ec4d nir,spirv: implement SpvOpImageSparseTexelsResident
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7774>
2021-01-06 20:36:38 +00:00
Rhys Perry
1fd8b46667 nir,spirv: add sparse image loads
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7774>
2021-01-06 20:36:38 +00:00
Rhys Perry
d1283083ea nir: improve divergence analysis for loads with non-uniform resources
If ACCESS_NON_UNIFORM is not specified, we can assume the resource is
uniform. This requires nir_lower_non_uniform_access to remove that flag.

A few Detroit: Become Human shaders use a index sourced from a fragment
input which is expected to be uniform.

shader-db (Navi):
Totals from 8 (0.01% of 127638) affected shaders:
SGPRs: 224 -> 384 (+71.43%)
VGPRs: 208 -> 112 (-46.15%)
CodeSize: 5360 -> 5344 (-0.30%); split: -1.49%, +1.19%
Instrs: 1036 -> 1028 (-0.77%); split: -1.93%, +1.16%
VMEM: 1320 -> 608 (-53.94%)
SMEM: 384 -> 336 (-12.50%); split: +14.58%, -27.08%
VClause: 24 -> 16 (-33.33%)
SClause: 48 -> 56 (+16.67%)
PreSGPRs: 124 -> 216 (+74.19%)
PreVGPRs: 168 -> 88 (-47.62%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5201>
2020-12-08 13:02:27 +00:00
Pierre-Eric Pelloux-Prayer
805b6b426e nir: update fallthrough comments
clang doesn't support /* fallthrough */ so switch to fallthrough
attribute.

Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7747>
2020-12-01 10:04:41 +01:00
Lionel Landwerlin
1c9488e0d1 nir: wire shading rate variables
v2: Fixup comment about bits in nir_intrinsics.py

v3: Use varying for primitive shading rate builtin (samuel)

v4: Reoder switch alphabetically
    Make divergence of frag_shading_rate an option

v5: Remove stage check for frag_shading_rate in divergence (Samuel)

v6: s/frag_shading_rate_per_subgroup/single_frag_shading_rate_per_subgroup/ (Jason)

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7795>
2020-12-01 08:20:38 +00:00
Jason Ekstrand
630e54a08b nir: Add a halt instruction type
Halt is like a return for the entire shader or exit() if you prefer to
think of it that way.  Once an invocation hits a halt, it's 100% dead.
Any writes to output variables which happened before the halt do,
however, still apply.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>
2020-11-25 05:37:09 +00:00
Erik Faye-Lund
33ccf0e9bc nir: drop unused alpha_ref_float
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7251>
2020-10-21 16:33:43 +00:00
Rhys Perry
37b6b0967c nir: allow divergence information to be updated when inserting instruction
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6558>
2020-10-13 12:47:21 +00:00
Rhys Perry
e1120f274f nir: move divergence analysis options to nir_shader_compiler_options
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6558>
2020-10-13 12:47:21 +00:00
Rhys Perry
1a912a550f nir: add last_invocation intrinsic
Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6558>
2020-10-13 12:47:20 +00:00
Jason Ekstrand
9750164c09 nir: Rename get_buffer_size to get_ssbo_size
This makes it explicit that this intrinsic is only for SSBOs.  For the
v3dv driver, we'll be adding a get_ubo_size intrinsic and we want to be
able to distinguish between the two.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6812>
2020-09-22 13:34:12 +00:00
Jason Ekstrand
ff124e3fe3 nir: Add a load_global_constant intrinsic
This has the same semantics as load_global except the memory it reads is
known to be constant so load_global_constant intrinsics can be CSEd
rather than relying on more complex copy-propagation.

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6379>
2020-09-01 20:50:03 +00:00
Karol Herbst
025bdbac3e nir: Add goto_if jump instruction
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/2401>
2020-08-14 20:35:36 +00:00
Neil Roberts
14dd65bb5b compiler: Add a system value for the line coord
The line coord is a coordinate along the axis perpendicular to the line.
It is in the range [0,1] between the two edges of the line. It is
available at least on Broadcom hardware.

Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5624>
2020-07-06 21:59:15 +00:00
Kenneth Graunke
6a5fb31fef nir: Fix divergence analysis for tessellation input/outputs
The load_per_vertex_{input,output} intrinsics simply mean that they're
reading an arrayed input/output, which have one element per invocation.
Most accesses to those use gl_InvocationID as the subscript.  However,
it's totally possible to read any element of the array.  For example,
an evaluation shader might read gl_in[2].gl_Position, or a control
shader might read output[0].

For threads processing a single patch, an input/output load is
convergent if and only if both sources (the per-vertex-array subscript
and the offset) are convergent.  For threads processing multiple
patches, we continued to mark them divergent.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Acked-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5613>
2020-06-24 03:25:10 +00:00
Daniel Schürmann
1f7d1541df nir: reset ssa-defs as non-divergent during divergence analysis instead of upfront
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4062>
2020-05-13 18:49:22 +00:00
Daniel Schürmann
1b881f3d8e nir: simplify phi handling in divergence analysis
This patch adds some control flow information to the
state to keep track whether a loop contains divergent
continue or break statements to not having to
recalculate this property for every phi.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4062>
2020-05-13 18:49:22 +00:00