Commit Graph

217 Commits

Author SHA1 Message Date
Georg Lehmann
e0bcab953d nir: add amd shared append/consume
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31075>
2024-09-19 16:21:47 +00:00
Alyssa Rosenzweig
4941d71846 nir/divergence_analysis: handle load_agx
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30981>
2024-09-02 23:27:14 +00:00
Ian Romanick
c160ed212e nir/divergence: resource_intel is less divergent than you thought
When the non_uniform flag is not set, the result is never divergent.

v2: Remove redundant assignment to is_divergent. Suggested by Caio.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30251>
2024-08-30 03:39:30 +00:00
Karol Herbst
fc88f04ba1 vtn, nir: handle OpImageQueryLevels on images
This is needed for cl_khr_mipmap_image, specifically the OpenCL C
function get_image_num_mip_levels.

Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30834>
2024-08-27 15:06:17 +00:00
Lionel Landwerlin
2158fe2ae2 nir/divergence: add missing load_constant_base_ptr
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Cc: mesa-stable
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30712>
2024-08-27 01:33:52 +00:00
Konstantin Seurer
ce24486ee4 nir: Introduce nir_debug_info_instr
Adds a new instruction type that stores metadata that might be useful
for debugging purposes. Passes must ignore these instructions when
making decisions.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18903>
2024-08-25 10:26:33 +00:00
Lionel Landwerlin
cf986dd589 nir: remove unused intel intrinsics
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Ivan Briano <ivan.briano@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30713>
2024-08-22 19:44:40 +00:00
Lionel Landwerlin
fbafa9cabd intel/nir: remove load_global_const_block_intel intrinsic
load_global_constant_uniform_block_intel is equivalent in terms of
loading, then for the predicate we just do a bcsel afterward in places
where that is required.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30659>
2024-08-16 11:12:39 +00:00
Job Noorman
e0bad1dd20 ir3: replace @load_uniform by new @load_const_ir3 intrinsic
Uniforms are a legacy thing and this intrinsic was only used to load
from const registers so the new naming is less confusing.

Signed-off-by: Job Noorman <jnoorman@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28341>
2024-08-15 12:07:27 +00:00
Alyssa Rosenzweig
f04ae930d9 nir,agx: add "active threads in subgroup" intrinsic
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30633>
2024-08-12 18:45:58 -04:00
Alyssa Rosenzweig
0566e9a51f nir/divergence_analysis: handle derivative intrinsics
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30565>
2024-08-08 15:26:07 +00:00
Alyssa Rosenzweig
340831dbcc nir/divergence_analysis: handle AGX stuff
bunch of vendor intrinsics, plus some standard intrinsics used in weird shader
stages.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30488>
2024-08-06 11:48:18 -04:00
Marek Olšák
b2d32ae246 nir: add nir_intrinsic_load_per_primitive_input, split from io_semantics flag
Instead of having 1 bit in nir_io_semantics indicating a per-primitive
FS input, add a dedicated intrinsic for it.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29895>
2024-07-23 16:13:16 +00:00
Georg Lehmann
2d3f536174 aco,nir: add dpp16_shift_amd intrinsic
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24650>
2024-07-17 15:04:38 +00:00
Marek Olšák
1b2cd628b8 nir: rename ordered_xfb_counter_add_gfx12_amd -> ordered_add_loop_gfx12_amd
because it can also be used by compute.

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30063>
2024-07-13 01:32:48 +00:00
Connor Abbott
ec37e65a2d ir3: Introduce elect_any_ir3
For preambles, we don't actually care which invocation we get, so we
don't have to enable helper invocations when the preamble uses "getone."
Introduce a new intrinsic with the right semantics and plumb it through.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29914>
2024-06-26 17:40:15 +00:00
Faith Ekstrand
7e3d157bee nak,nir: Drop r2ur_nv in favor of as_uniform
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29737>
2024-06-15 06:14:27 +00:00
Faith Ekstrand
b107240474 nir: Add some new _nv intrinsics
The ldc_nv and ldcx_nv intrinsics correspond to the index and bindless
forms of NVIDIA's LDC instruction, respectively.  ldc_nv is pretty much
load_ubo without some of the unnecessary constant bits while ldcx_nv
takes a 64-bit bindless handle instead of an index.  The other two give
us a little control over register allocation at the NIR level to ensure
that LDCX handles are placed in uniform registers.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29591>
2024-06-13 20:43:45 +00:00
Konstantin Seurer
a93f95c69c radv/rt: Remove load_rt_dynamic_callable_stack_base_amd
Reviewed-by: Friedrich Vock <friedrich.vock@gmx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28619>
2024-05-28 12:23:45 +00:00
Lionel Landwerlin
2be28ee58a nir: add a base offset for printf indexing
This will allow a driver to use a single table of printf strings
across all shaders.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Ivan Briano <ivan.briano@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25814>
2024-05-15 13:13:37 +00:00
Lionel Landwerlin
8d336f069e nir/divergence: add missing load_printf_buffer_address
Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Cc: mesa-stable
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Ivan Briano <ivan.briano@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25814>
2024-05-15 13:13:37 +00:00
Juan A. Suarez Romero
87cd11ecd2 nir,v3d: rename tlb_color_v3d intrinsic
As this is intended to be used also by VC4, change the suffix to
something more convenient, like tlb_color_brcm.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29119>
2024-05-13 10:44:17 +00:00
Marek Olšák
b06a71b3cd nir: add streamout intrinsics for AMD GFX12
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Acked-By: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28889>
2024-04-30 17:17:25 +00:00
Karol Herbst
d22f936019 nir: remove workgroup_id_zero_base
This removes the need for drivers to handle both versions. The base will
get added once in nir_lower_system_values when converting from deref to
intrinsic and will be replaced by a zero for users not supporting it.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26800>
2024-04-24 20:18:49 +00:00
Karol Herbst
3217838fef nir: remove global_invocation_id_zero_base
This removes the need for drivers to handle both versions. The base will
get added once in nir_lower_system_values when converting from deref to
intrinsic and will be replaced by a zero for users not supporting it.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26800>
2024-04-24 20:18:49 +00:00
Karol Herbst
3625a44dcc nir/divergence_analysis: handle load_base_global_invocation_id
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26800>
2024-04-24 20:18:48 +00:00
Marek Olšák
c1f750eed9 nir: add nir_intrinsic_optimization_barrier_sgpr_amd
for radeonsi

Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28606>
2024-04-13 16:45:08 +00:00
Connor Abbott
42ddbf4ebe nir/divergence_analysis: Fix load_view_index divergence in VS
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28573>
2024-04-08 23:20:08 +00:00
Connor Abbott
223e3727d5 nir/divergence_analysis: Add uniform_load_tears option
This "tear" is similar to the original concept of loads/stores tearing,
but across invocations in a wave instead of bytes. Qualcomm seems to
have this problem, at least for some GPUs. This fixes
spec@arb_shader_storage_buffer_object@execution@ssbo-atomiccompswap-int
on a630 once we start relying on divergence analysis for computing
reconvergence properties. For backends that have readFirstInvocation(),
it should be possible to fix the problem by inserting
readFirstInvocation() instead, but a5xx doesn't have it so we can't rely
on it in freedreno.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28573>
2024-04-08 23:20:08 +00:00
Connor Abbott
291e6d2940 nir/divergence_analysis: Add ir3-specific intrinsics
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28573>
2024-04-08 23:20:08 +00:00
Rhys Perry
543ca160a5 nir,aco: add test intrinsics
These don't really do anything. They're just a source and user of SSA
defs.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28301>
2024-04-08 18:38:39 +00:00
Timur Kristóf
411de8488c nir: Add two new AMD specific tess intrinsics.
These will be needed to implement some tessellation dynamic
states within the TCS as opposed to using an epilog.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28408>
2024-03-28 23:44:03 +00:00
Mike Blumenkrantz
2b2f74ab35 nir/divergence: add zink intrinsics
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28429>
2024-03-27 20:55:07 +00:00
Marek Olšák
1585a5cc6d nir,amd: add nir_intrinsic_load_debug_log_desc_amd and its use
for shader debugging

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27952>
2024-03-22 21:58:02 +00:00
Marek Olšák
6773595ed0 nir: rename AMD XFB intrinsics to *_gfx11_amd
to indicate it's only for gfx11.

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27952>
2024-03-22 21:58:02 +00:00
Marek Olšák
813f37a8ed nir: add nir_block::divergent to indicate a divergent entry condition
to be used by nir_opt_varyings

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28049>
2024-03-12 00:29:03 +00:00
Marek Olšák
867a0a7db9 nir/divergence_analysis: handle derefs of system values
needed by GLSL compiler optimizations that have unlowered sysvals

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28049>
2024-03-12 00:29:03 +00:00
Marek Olšák
eb670d6eaf nir/divergence_analysis: load_instance_id is convergent within a primitive
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28049>
2024-03-12 00:29:03 +00:00
Marek Olšák
310b13b7f0 nir/divergence_analysis: load_primitive_id is convergent within a primitive
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28049>
2024-03-12 00:29:03 +00:00
Marek Olšák
1621d4a0d3 nir/divergence_analysis: change function prototypes
for following commits

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28049>
2024-03-12 00:29:03 +00:00
Juan A. Suarez Romero
62e1dff256 v3d: add load_fep_w_v3d intrinsic
This intrinsic helps to read the W coordinate stored in the QPU register
when initializing the input data for the fragment shaders.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28072>
2024-03-11 12:42:49 +00:00
Georg Lehmann
1d8b2b159e nir/divergence_analysis: fix subgroup mask
These depend on the subgroup invocation id, so they are divergent.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Reviewed-by: Ian Romanick <ian.d.romanick@intel.com>

Fixes: df86c5ffb3 ("nir: add divergence analysis pass.")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27962>
2024-03-05 14:52:17 +00:00
Ian Romanick
75de4458a1 nir: Mark nir_intrinsic_load_global_block_intel as divergent
This is divergent because it specifically loads sequential values into
successive SIMD lanes.

No shader-db or fossil-db changes on any Intel platform.

Fixes: 9f44a26462 ("nir/divergence: handle load_global_block_intel")
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27044>
2024-02-27 08:36:42 -08:00
Timur Kristóf
cc1501628f nir: Clean up divergence analysis for TES patch input loads.
Just make the code a little bit easier to follow.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27680>
2024-02-26 14:53:23 +00:00
Timur Kristóf
870a2e4197 nir: Cleanup divergence analysis for mesh shaders.
1. Mesh shaders don't have inputs (only task payload),
so remove them from handling load_input.

2. Clarify in comments that loading any mesh shader
output is an NV_mesh_shader only feature.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27680>
2024-02-26 14:53:23 +00:00
Timur Kristóf
9553d67373 nir: Fix divergence analysis of load_patch_vertices_in.
load_patch_vertices_in can only occur in tessellation shaders,
and contains the number of vertices in an input patch.

* TCS: patch_vertices_in is equal to the input patch size
* TES: patch_vertices_in is equal to the TCS output patch size

The patch sizes may be set by a pipeline or dynamic states,
however in both cases it is definitely uniform within a subgroup.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27680>
2024-02-26 14:53:23 +00:00
Timur Kristóf
537c0029dd nir: Fix divergence of reductions.
By accident, the function would return without setting
the divergence information.

Cc: mesa-stable
Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27680>
2024-02-26 14:53:23 +00:00
Bas Nieuwenhuizen
c7b2ac3377 radv: Remove ray_launch_size_addr_amd system value.
Not used anymore, so clean it up.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27664>
2024-02-17 11:08:16 +00:00
Marek Olšák
e98bbcad17 nir: add vertex divergence into nir_divergence_analysis
This is a prerequisite for the new nir_opt_varyings pass.
It reuses the same divergent field in nir_def and nir_loop.

Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26918>
2024-02-02 16:45:52 -05:00
Georg Lehmann
1cb5bf7009 nir: add ballot_relaxed and as_uniform intrinsics
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27116>
2024-01-19 20:13:33 +00:00