Beaver.MLIR.Dialect.AMDGPU (beaver v0.4.8)

sourceA - Single, DotInTypes, vector of 16-bit float or bfloat16 type or 16-bit signless integer values of length 2 or vector of 8-bit signless integer or f8E4M3FN type or f8E5M2 type values of length 4 or vector of 4-bit signless integer values of length 8
sourceB - Single, DotInTypes, vector of 16-bit float or bfloat16 type or 16-bit signless integer values of length 2 or vector of 8-bit signless integer or f8E4M3FN type or f8E5M2 type values of length 4 or vector of 4-bit signless integer values of length 8
destC - Single, DotOutTypes, 32-bit float or 16-bit float or bfloat16 type or 32-bit signless integer

Results

destD - Single, DotOutTypes, 32-bit float or 16-bit float or bfloat16 type or 32-bit signless integer

Description

The amdgpu.dot op is an MLIR wrapper over the v_dot* family of intrinsics, which compute D = sum_i A[i] * B[i] + C.

Variants (source, dest, signedness, chipset -> intrinsic).

| A elem   | B elem   | destC | signedness | chipset                   | ROCDL op                     |
|----------|----------|-------|------------|---------------------------|------------------------------|
| f16      | f16      | f32   | n/a        | gfx906+                   | fdot2                        |
| f16      | f16      | f16   | n/a        | gfx11+                    | fdot2.f16.f16                |
| bf16     | bf16     | f32   | n/a        | gfx11+, gfx950+           | fdot2.f32.bf16               |
| bf16     | bf16     | bf16  | n/a        | gfx11+                    | fdot2.bf16.bf16              |
| i16      | i16      | i32   | s / u      | gfx906+, no gfx11+/gfx12+ | sdot2 / udot2                |
| i8       | i8       | i32   | s / u      | gfx906+                   | sdot4 / udot4                |
| i8       | i8       | i32   | mixed      | gfx11+                    | sudot4                       |
| i4       | i4       | i32   | s / u      | gfx906+                   | sdot8 / udot8                |
| i4       | i4       | i32   | mixed      | gfx11+                    | sudot8                       |
| fp8/bf8  | fp8/bf8  | f32   | n/a        | gfx11.7, gfx12+           | dot4.f32.{fp8,bf8}.{fp8,bf8} |

Example:

%r0 = amdgpu.dot %a * %b + %c : vector<4xi8>, vector<4xi8>, i32
%r1 = amdgpu.dot %a * %b + %c {unsignedA, unsignedB, clamp}
    : vector<8xi4>, vector<8xi4>, i32
%r2 = amdgpu.dot %a * %b + %c {unsignedB}
    : vector<4xi8>, vector<4xi8>, i32
%r3 = amdgpu.dot %a * %b + %c : vector<2xf16>, vector<2xf16>, f32
%r4 = amdgpu.dot %a * %b + %c : vector<2xf16>, vector<2xf16>, f16
%r5 = amdgpu.dot %a * %b + %c
    : vector<4xf8E4M3FN>, vector<4xf8E5M2>, f32

dpp()

Return op name amdgpu.dpp as a bitstring.

dpp(ssa)

amdgpu.dpp - AMDGPU DPP operation

This op has support for result type inference.

Attributes

kind - Single, AMDGPU_DPPPermAttr, The possible permutations for a DPP operation
permArgument - Optional, anonymous/composite constraint, 32-bit signless integer attribute or array attribute or unit attribute
row_mask - Single, I32Attr, 32-bit signless integer attribute
bank_mask - Single, I32Attr, 32-bit signless integer attribute
bound_ctrl - Single, BoolAttr, bool attribute

Operands

old - Single, AMDGPU_IntOrFloatOr1DVectorWidthLeq64, integer or float with element bitwidth <= 64 or fixed-length vector of integer or float with element bitwidth <= 64 values of ranks 1
src - Single, AMDGPU_IntOrFloatOr1DVectorWidthLeq64, integer or float with element bitwidth <= 64 or fixed-length vector of integer or float with element bitwidth <= 64 values of ranks 1

Results

result - Single, AnyType, any non-token type

Description

The amdgpu.dpp op performs a Data Parallel Primitives (DPP) lane permutation on a source value within a wavefront. Each lane reads its source data from another lane according to the permutation mode specified by kind. DPP operates at dword (32-bit) granularity: sub-32-bit types (e.g., f16, i16) are packed into an i32 during lowering, permuted, and extracted back.

Lanes are organized into rows of 16.
A Wave64 wavefront has 4 rows of 16 lanes each: row 0 = lanes 0-15, row 1 = lanes 16-31, row 2 = lanes 32-47, row 3 = lanes 48-63.
Similarly, a Wave32 wavefront has two rows of 16 lanes each, organized in the same fashion.
Each row is divided into 4 banks of 4 consecutive lanes: bank 0 = lanes 0-3, bank 1 = lanes 4-7, bank 2 = lanes 8-11, bank 3 = lanes 12-15 (lane numbers shown for row 0; add 16/32/48 for other rows).

The kind attribute selects the permutation. Some modes require a permArgument; others take no argument.

Quad permutation:

quad_perm([a, b, c, d]): Full permute within each group of 4 consecutive lanes (a quad). Each element is in [0, 3] and selects which lane within the quad to read from. Lane 4k+i reads from lane 4k+perm[i]. For example, quad_perm([1, 0, 3, 2]) swaps adjacent pairs within every quad.

Row shifts and rotates (operate within each 16-lane row independently):

row_shl(N): Shift left by N (1-15) within the row. Lane n reads from lane (n % 16) + N in the same row. Lanes where the source index exceeds 15 are out of bounds (see bound_ctrl).
row_shr(N): Shift right by N (1-15) within the row. Lane n reads from lane (n % 16) - N in the same row. Lanes where the source index is negative are out of bounds.
row_ror(N): Rotate right by N (1-15) within the row. Lane n reads from lane ((n % 16) - N) mod 16 in the same row. Always in bounds.

Wavefront shifts and rotates (not available on RDNA):

wave_shl: Shift left by 1. Lane n reads from lane n + 1. The last lane in the wavefront is out of bounds.
wave_shr: Shift right by 1. Lane n reads from lane n - 1. Lane 0 is out of bounds.
wave_rol: Rotate left by 1. Lane n reads from lane (n + 1) mod W, where W is the wavefront size.
wave_ror: Rotate right by 1. Lane n reads from lane (n - 1) mod W, where W is the wavefront size.

Row mirrors:

row_mirror: Reverse lanes within each 16-lane row. Lane n reads from lane 15 - (n % 16) within its row.
row_half_mirror: Reverse within each 8-lane half-row. Lane n reads from lane 7 - (n % 8) within its half-row.

Row broadcasts (not available on RDNA):

row_bcast_15: Lane 15 of each row broadcasts to all lanes of the next row. Lanes in row 0 are not affected (retain old).
row_bcast_31: Lane 31 broadcasts to all lanes in rows 2 and 3. Lanes in rows 0 and 1 are not affected (retain old).

Example:

// Swap adjacent pairs within each quad (lanes 0<->1, 2<->3, etc.)
%0 = amdgpu.dpp %old %src quad_perm( [1, 0, 3, 2] ) : i32

// Shift right by 1 lane within each 16-lane row.
// bound_ctrl=true -> lanes that would read past the row return 0.
// row_mask=0x5 (0b0101) -> only rows 0 and 2 apply the shift;
// rows 1 and 3 pass through %old unchanged.
%1 = amdgpu.dpp %old %src row_shr( 0x1 : i32 )
  { row_mask = 0x5 : i32, bound_ctrl = true } : f32

// Rotate left across the full wavefront by 1 lane
%2 = amdgpu.dpp %old %src wave_rol : i32

Operands:

$old: Fallback value. Lanes that are masked off by row_mask / bank_mask retain old. For lanes with an out-of-bounds source, behavior depends on bound_ctrl.
$src: Source value to be permuted across lanes.
$kind: A #amdgpu.dpp_perm enum selecting the permutation mode.
$permArgument: Mode-specific argument. Required for quad_perm (array of 4 integers in [0, 3]) and row_shl/row_shr/row_ror (integer in [1, 15]). Absent for all other modes.
$row_mask (default 0xf): 4-bit mask controlling which rows write results. Bit i enables row i (bit 0 = lanes 0-15, bit 1 = lanes 16-31, etc.). Disabled lanes retain old.
$bank_mask (default 0xf): 4-bit mask controlling which banks write results. Bit i enables bank i (bit 0 = lanes 0-3, 16-19, etc. across all rows). Disabled lanes retain old.
$bound_ctrl (default false): When false, out of bounds lanes retain old. When true, out-of-bounds lanes receive zero.

ds_async_barrier_arrive()

Return op name amdgpu.ds_async_barrier_arrive as a bitstring.

ds_async_barrier_arrive(ssa)

amdgpu.ds_async_barrier_arrive - Asynchronously arrive at an in-LDS barrier.

Operands

base - Single, anonymous/composite constraint, memref of State of an in-LDS barrier. values
indices - Variadic, Index, variadic of index

Description

Add a arrival to the LDS barrier at base[indices] to the sequence of pending asynchronous memory operations.

The indices must be non-negative and in-bounds for the corresponding dimensions of base.

This will add an "asynchronous memory operation" to the in-order list of pending asynchronous loads from global memory to LDS. When the queue of such operations issued before this operation is complete, the specified barrier will be arrived at, decrementing the pending count by 1 per lane that executes it and rolling over the phase if applicable.

This operation does not return the old barrier state.

Example:

amdgpu.ds_async_barrier_arrive %barrier[] : memref<!amdgpu.ds_barrier_state, #gpu.address_space<workgroup>>

This operation is only available on gfx1250+.

ds_barrier_arrive()

Return op name amdgpu.ds_barrier_arrive as a bitstring.

ds_barrier_arrive(ssa)

amdgpu.ds_barrier_arrive - Arrive at an in-LDS barrier and return old state.

This op has support for result type inference.

Operands

base - Single, anonymous/composite constraint, memref of State of an in-LDS barrier. values
indices - Variadic, Index, variadic of index
count - Single, I64, 64-bit signless integer

Results

out - Single, AMDGPU_DsBarrierStateType, State of an in-LDS barrier.

Description

Atomically arrive at the LDS barrier at base[indices] and decrement it by count, rolling over the phase if needed and returning the old barrier state.

The indices must be non-negative and in-bounds for the corresponding dimensions of base.

count is the number of participants that should be subtracted from the barrier's pending count per lane that executes the operation.

Example:

%old_state = amdgpu.ds_barrier_arrive %barrier[], %c1 : memref<!amdgpu.ds_barrier_state, #gpu.address_space<workgroup>>, i64 -> !amdgpu.ds_barrier_state

This operation is only available on gfx1250+.

ds_barrier_init()

Return op name amdgpu.ds_barrier_init as a bitstring.

ds_barrier_init(ssa)

amdgpu.ds_barrier_init - Initialize an in-LDS barrier.

Operands

base - Single, anonymous/composite constraint, memref of State of an in-LDS barrier. values
indices - Variadic, Index, variadic of index
participants - Single, I32, 32-bit signless integer

Description

Given the location !amdgpu.ds_barrier_state in LDS (as specified by base and indices), initialize the barrier structure so that the pending and init counts are equal to participants - 1, which will have its high bits masked off, and its phase is equal to 0.

The indices must be non-negative and in-bounds for the corresponding dimensions of base.

Note that we subtract 1 from participants when constructing the barrier state to provide clearer high-level semantics.

The subtraction means that, when the participantth arrival occurs, the phase will change. In practical terms, this means that you can use (for example) the number of subgroups or waves per workgroup as participants, instead of manually needing to remove one.

While the write of the initial state will be performed atomically, no synchronization between waves will be performed by this operation.

Example:

amdgpu.ds_barrier_init %barrier[], %c32 : memref<!amdgpu.ds_barrier_state, #gpu.address_space<workgroup>>, i32

This operation is only available on gfx1250+.

ds_barrier_poll_state()

Return op name amdgpu.ds_barrier_poll_state as a bitstring.

ds_barrier_poll_state(ssa)

amdgpu.ds_barrier_poll_state - Atomically read the state of an in-LDS barrier.

This op has support for result type inference.

Operands

base - Single, anonymous/composite constraint, memref of State of an in-LDS barrier. values
indices - Variadic, Index, variadic of index

Results

out - Single, AMDGPU_DsBarrierStateType, State of an in-LDS barrier.

Description

Atomically read and return the state of the barrier at base[indices...].

This will ultimately act like a memref.load, but this operation will ensure that appropriate atomic orderings and syncscopes are set. The indices must be non-negative and in-bounds for the corresponding dimensions of base.

Example:

%state = amdgpu.ds_barrier_poll_state %barrier[] : memref<!amdgpu.ds_barrier_state, #gpu.address_space<workgroup>> -> !amdgpu.ds_barrier_state

This operation is only available on gfx1250+.

ds_barrier_state_init_count()

Return op name amdgpu.ds_barrier_state_init_count as a bitstring.

ds_barrier_state_init_count(ssa)

amdgpu.ds_barrier_state_init_count - Extract the init count of a barrier state.

This op has support for result type inference.

Operands

state - Single, AMDGPU_DsBarrierStateType, State of an in-LDS barrier.

Results

res - Single, I32, 32-bit signless integer

Description

Extract the init count of the !amdgpu.ds_barrier_state state as a 32-bit value.

Example:

%init = amdgpu.ds_barrier_state_init_count %state : !amdgpu.ds_barrier_state -> i32

ds_barrier_state_pending_count()

Return op name amdgpu.ds_barrier_state_pending_count as a bitstring.

ds_barrier_state_pending_count(ssa)

amdgpu.ds_barrier_state_pending_count - Extract the pending count of a barrier state.

This op has support for result type inference.

Operands

state - Single, AMDGPU_DsBarrierStateType, State of an in-LDS barrier.

Results

res - Single, I32, 32-bit signless integer

Description

Extract the pending count of the !amdgpu.ds_barrier_state state as a 32-bit value.

Example:

%pending = amdgpu.ds_barrier_state_pending_count %state : !amdgpu.ds_barrier_state -> i32

ds_barrier_state_phase()

Return op name amdgpu.ds_barrier_state_phase as a bitstring.

ds_barrier_state_phase(ssa)

amdgpu.ds_barrier_state_phase - Extract the phase of a barrier state.

This op has support for result type inference.

Operands

state - Single, AMDGPU_DsBarrierStateType, State of an in-LDS barrier.

Results

res - Single, I32, 32-bit signless integer

Description

Extract the phase of the !amdgpu.ds_barrier_state state as a 32-bit value.

Example:

%phase = amdgpu.ds_barrier_state_phase %state : !amdgpu.ds_barrier_state -> i32

ds_barrier_state_phase_parity()

Return op name amdgpu.ds_barrier_state_phase_parity as a bitstring.

ds_barrier_state_phase_parity(ssa)

amdgpu.ds_barrier_state_phase_parity - Extract the phase parity of a barrier state.

This op has support for result type inference.

Operands

state - Single, AMDGPU_DsBarrierStateType, State of an in-LDS barrier.

Results

res - Single, I1, 1-bit signless integer

Description

Return the parity of the phase of the !amdgpu.ds_barrier_state state.

This is intended to simplify the case where the barrier is being used to repeatedly track completion of a task where the precise value of the phase won't mater, only that it changed since (or as a result of) the arrival.

Example:

%parity = amdgpu.ds_barrier_state_phase_parity %state : !amdgpu.ds_barrier_state -> i1

ext_packed_fp8()

Return op name amdgpu.ext_packed_fp8 as a bitstring.

ext_packed_fp8(ssa)

amdgpu.ext_packed_fp8 - Extend a fp8 value to a float or a vector of packed fp8 values to two floats

Attributes

index - Single, I32Attr, 32-bit signless integer attribute whose value is non-negative whose maximum value is 3

Operands

source - Single, anonymous/composite constraint, f8E5M2FNUZ type or f8E4M3FNUZ type or f8E5M2 type or f8E4M3FN type or vector of f8E5M2FNUZ type or f8E4M3FNUZ type or f8E5M2 type or f8E4M3FN type values of length 1/2/3/4

Results

res - Single, anonymous/composite constraint, 32-bit float or fixed-length vector of 32-bit float values of length 2

Description

Extend one or two 8-bit floats in source[index] to a 32-bit float or two floats and return them.

This rather unusual signature arises from the fact that AMD GPUs cannot easily work with sub 32-bit quantities, so the compiler intrinsics for extending 8-bit floats (which are, currently, the only way to work with this operation) take packed vectors of 4 such floats.

If the passed-in vector has fewer than four elements, or the input is scalar, the remaining values in the <4 x i8> will be filled with undefined values as needed.

fat_raw_buffer_cast()

Return op name amdgpu.fat_raw_buffer_cast as a bitstring.

fat_raw_buffer_cast(ssa)

amdgpu.fat_raw_buffer_cast - Create a raw buffer fat pointer that matches memref

This op has support for result type inference.

Attributes

boundsCheck - Single, BoolAttr, bool attribute
resetOffset - Optional, UnitAttr, unit attribute

Operands

source - Single, AnyMemRef, memref of any non-token type values
validBytes - Optional, I64, 64-bit signless integer
cacheSwizzleStride - Optional, anonymous/composite constraint, 14-bit signless integer

Results

result - Single, AnyMemRef, memref of any non-token type values

Description

Wraps the memory pointed to by source as a raw buffer fat pointer, or, in LLVM terms, a ptr addrspace(7), returning a memref that has the same sizes and layout but the #amdgpu.address_space<fat_raw_buffer> address space.

This memref can be used with standard memref operations like memref.load, memref.store, and memref.atomicrmw, which will be lowered to the relevant buffer intrinsics. (vector.masked_load/store will work once there's backend support for lowering them, and then this document will be updated)

If validBytes is given, it is the number of bytes that will be valid as an offset to out. If it is not provided, this will be inferred from the size of the memref during lowering. This size is max_{d = 0 upto rank(source)} (sizes[d] strides[d]) sizeof(element type).

The flags of the buffer descriptor will be set up to enable raw usage - for example, stride = 0, add_tid = 0, and so on. The boundsCheck property determines if bounds checking is enabled or not (on architectures where this can be controlled - that is, on RDNA chips).

If cacheSwizzleStride is provided, L1 cache swizzling will be enabled on architectures that support it. This swizzling, unlike the main swizzling mode (whose usage makes a buffer non-raw) does not affect index calculation, but does affect cache behavior. Mixing access between cache-swizzled raw buffers and other forms of memory access, like ordinary pointer loads or unswizzled buffer pointers can cause incorrect behavior and must be avoided.

This operation preserves the sizes, strides, and offset of the input memref - they'll be added in by memref.load later. However, if resetOffset is set, that offset will be added to the base pointer. If the value of the memref's offset is not uniform (independent of the lane/thread ID), this will lead to substantially decreased performance due to the need for a waterfall loop on the base address of the buffer resource.

gather_to_lds()

Return op name amdgpu.gather_to_lds as a bitstring.

gather_to_lds(ssa)

amdgpu.gather_to_lds - MLIR wrapper for CDNA Gather to LDS instructions

Attributes

transferType - Single, TypeAttr, any type attribute
async - Optional, UnitAttr, unit attribute

Operands

src - Single, AnyMemRef, memref of any non-token type values
srcIndices - Variadic, Index, variadic of index
dst - Single, AnyMemRef, memref of any non-token type values
dstIndices - Variadic, Index, variadic of index

Description

The amdgpu.gather_to_lds op is a wrapper around the global_load_lds instructions.

Operands:

$src: global memory (including fat buffer) memref to read from.
$srcIndices: indices into $src to read from for this thread. These indices must be non-negative and in-bounds when $src is not a fat raw buffer. Fat raw buffer sources permit out-of-bounds indices with raw buffer semantics.
$dst: LDS memory memref to write to.
$dstIndices: base indices into $dst to write to for the subgroup of this thread. These indices must be non-negative and in-bounds. The elements gathered by the subgroup will be written contiguously in order of lane ID starting at $dst[$dstIndices]. Byte-sized (ex. i8) or short-sized (ex. i16) types will be zero-padded/extended to 32 bits before being written. 96-bit types (ex. vector<3xf32>) will be zero-padded to 128 bits before being written. Only the offsets held by lane 0 are used.
$transferType: type of the data to be transferred by each thread. This is used to determine the size of the data to be transferred and the number of threads in the subgroup. The transfer type must be a scalar type or a vector type with a single element type.
If $async is set, the compiler will not attempt to infer the memory waits needed to ensure that the DMA operation has succeeded before a load that might access the stored-to LDS is performed. Instead, the rocdl.asyncmark and rocdl.wait.asyncmark N operations must be used to explicitly indicate the desired completion behavior. This enables more precise calculation of these waits at the cost of requiring user management of asynchrony.

The $dst, along with its indices, points to the memory location the subgroup of this thread will write to.

Note: only supported on gfx9 and gfx10.

global_load_async_to_lds()

Return op name amdgpu.global_load_async_to_lds as a bitstring.

global_load_async_to_lds(ssa)

amdgpu.global_load_async_to_lds - MLIR wrapper for async global load to lds instructions

Attributes

transferType - Single, TypeAttr, any type attribute

Operands

src - Single, AnyMemRef, memref of any non-token type values
srcIndices - Variadic, Index, variadic of index
dst - Single, AnyMemRef, memref of any non-token type values
dstIndices - Variadic, Index, variadic of index
mask - Optional, I1, 1-bit signless integer

Description

AMDGPU wrapper for global.load.async.to.lds instructions, which performs asynchronous load of data from global memory into LDS while bypassing VGPRs.

$src: global memory memref to read from (global addrspace only, no fat buffer).
$srcIndices: indices into $src for this thread's global read location. These indices must be non-negative and in-bounds.
$dst: LDS memref to write to (workgroup addrspace).
$dstIndices: indices into $dst for this thread's LDS write location. These indices must be non-negative and in-bounds when $mask is not provided. When $mask is provided, the destination indices are not guaranteed to be in-bounds because masked-off lanes may carry invalid destination indices.
$transferType: type of data to be transferred. Must be 8, 32, 64 or 128 bit scalar or vector type.
$mask: optional per-thread mask. When false, the thread's LDS write is masked off. The global read still occurs for all threads regardless of mask.

Note: only supported on gfx1250 and later.

Examples:

  amdgpu.global_load_async_to_lds %src[%i, %j], %dst[%k, %l]
    : f32, memref<128x64xf32, #gpu.address_space<global>>,
      memref<64x64xf32, #gpu.address_space<workgroup>>

  amdgpu.global_load_async_to_lds %src[%i, %j], %dst[%k, %l]
    : vector<4xf32>, memref<128x64xf32, #gpu.address_space<global>>,
      memref<64x64xf32, #gpu.address_space<workgroup>>

  amdgpu.global_load_async_to_lds %src[%i], %dst[%j]
    : i8, memref<512xi8, #gpu.address_space<global>>,
      memref<256xi8, #gpu.address_space<workgroup>>

global_prefetch()

Return op name amdgpu.global_prefetch as a bitstring.

global_prefetch(ssa)

amdgpu.global_prefetch - Prefetch data to caches.

Attributes

temporalHint - Single, AMDGPU_LoadTemporalHintAttr, AMDGPU-specific prefetch temporal hints for load instructions. RT - regular temporal for both near and far caches; NT - non-temporal for both near and far caches; HT - high-priority temporal for both near and far caches; LU - last-use; NT_RT - non-temporal for near cache(s) and regular for far caches; RT_NT - regular for near cache(s) and non-temporal for far caches; NT_HT - non-temporal for near cache(s) and high-priority temporal for far caches
cacheScope - Single, AMDGPU_CacheScopeAttr, AMDGPU-specific cache scopes. WGP - workgroup processor (CUs); SE - shader engine (GL2); DEV - device; SYS - system
speculative - Optional, UnitAttr, unit attribute

Operands

src - Single, AnyMemRef, memref of any non-token type values
indices - Variadic, I64, variadic of 64-bit signless integer

Description

Prefetches a cache line to high-level caches using the aligned address of the source memref and an offset provided by the indices of the element containing the cache line. This provides temporal hints (e.g., regular or high-priority). Note that out-of-bounds access is allowed in speculative mode. The provided memref must be in the global address space (#gpu.address_space<global> or 1).

This operation was introduced in gfx1250.

Example:

amdgpu.global_prefetch %src[%i, %j] RT SE speculative : memref<64x64xf16, #gpu.address_space<global>>

global_transpose_load()

Return op name amdgpu.global_transpose_load as a bitstring.

global_transpose_load(ssa)

amdgpu.global_transpose_load - MLIR wrapper for global memory transpose load instructions

Operands

src - Single, AnyMemRef, memref of any non-token type values
srcIndices - Variadic, Index, variadic of index

Results

result - Single, anonymous/composite constraint, fixed-length vector of 8-bit signless integer or f8E5M2FNUZ type or f8E4M3FNUZ type or f8E5M2 type or f8E4M3FN type or 16-bit float or bfloat16 type or 16-bit signless integer values of length 8 or fixed-length vector of 4-bit signless integer or f4E2M1FN type or 6-bit signless integer or f6E2M3FN type or f6E3M2FN type values of length 16

Description

The amdgpu.global_transpose_load op is a wrapper around the global_load_tr family of instructions introduced in gfx1200.

Each thread reads a column of a matrix stored in global memory and receives the corresponding row of the transposed matrix in its result register. The subgroup collectively performs a transpose of the tile.

This op is a direct wrapper around the ROCDL global.load.tr family intrinsics. Refer to the ISA manual for exact semantics.

Format example:

%0 = amdgpu.global_transpose_load %src[%i, %j]
       : memref<128x256xf16, #gpu.address_space<global>> -> vector<8xf16>

Operands:

$src: Global address space memref to read from.
$srcIndices: indices into $src for this thread. Indices must be non-negative and in-bounds for the corresponding dimension of $src, matching the constraints of memref.load.
$result: register this transpose load instruction writes to.

Valid (element bits, num elements) pairs:

(4, 16) -> global_load_tr4_b64 (gfx1250+)
(6, 16) -> global_load_tr6_b96 (gfx1250+)
(8, 8) -> global_load_tr_b64 (gfx1200+)
(16, 8) -> global_load_tr_b128 (gfx1200+)

Note: 8-bit and 16-bit element lowering requires gfx1200+.

  4-bit and 6-bit element lowering requires gfx1250+.

lds_barrier()

Return op name amdgpu.lds_barrier as a bitstring.

lds_barrier(ssa)

amdgpu.lds_barrier - Barrier that includes a wait for LDS memory operations.

Description

DEPRECATION NOTICE: Unless you need the inline-assembly-based workaround for gfx908/MI-100, you should represent this pattern with the equivalent

gpu.barrier memfence [#gpu.address_space<workgroup>]

instead.

amdgpu.lds_barrier is both a barrier (all workitems in a workgroup must reach the barrier before any of them may proceed past it) and a wait for all operations that affect the Local Data Store (LDS) issued from that workgroup to complete before the workgroup may continue. Since the LDS is per-workgroup memory, this barrier may be used, for example, to ensure all workitems have written data to LDS before any workitem attempts to read from it.

Note that lds_barrier does not force reads to or from global memory to complete before execution continues. Therefore, it should be used when operations on global memory can be issued far in advance of when their results are used (for example, by writing them to LDS).

WARNING: On architectures that do not support the BackOffBarrier feature, (those which will implement this barrier by emitting inline assembly), use of this operation will impede the usabiliity of memory watches (including breakpoints set on variables) when debugging.

make_dma_base()

Return op name amdgpu.make_dma_base as a bitstring.

make_dma_base(ssa)

amdgpu.make_dma_base - Pair of based addresses used when moving tiles between LDS and global memory.

Operands

global - Single, AnyMemRef, memref of any non-token type values
global_indices - Variadic, Index, variadic of index
lds - Single, AnyMemRef, memref of any non-token type values
lds_indices - Variadic, Index, variadic of index

Results

base - Single, AMDGPU_TDMBaseType, Pair of base addresses that move data between LDS and global storage.

Description

This operation creates a pair of addresses that will be used by tensor_load_to_lds and tensor_store_from_lds.

The global and LDS indices must be non-negative and in-bounds for the corresponding dimensions of their memrefs.

This operation creates a value corresponding to the tensor descriptor (D#) group 0 found in TensorLoadToLDSOp and TensorStoreFromLDSOp in the rocdl dialect.

For example:

  %base = amdgpu.make_dma_base %global[%idx0, %idx1], %lds[%idx2, %idx3] : memref<64x64xi32>, memref<64x64xi32, #gpu.address_space<workgroup>> -> !amdgpu.tdm_base<i32>
  %descriptor = amdgpu.make_dma_descriptor %base globalSize [2, 2] globalStride [2, 1] sharedSize [2, 2] : !amdgpu.tdm_base<i32> -> !amdgpu.tdm_descriptor
  amdgpu.tensor_load_to_lds %descriptor : !amdgpu.tdm_descriptor

  // pseudo-code
  %global_base = llvm.extractvalue %global_memref[1]
  %global_address = llvm.get_element_ptr ...

  %lds_base = llvm.extractvalue %lds_memref[1]
  %lds_address = llvm.get_element_ptr ...

  // Definition of %base
  %undef = llvm.mlir.undef : vector<4xi32>
  %v0 = llvm.insertelement %15, %undef[0] : vector<4xi32>
  %v1 = llvm.insertelement %lds_address, %v0[1] : vector<4xi32>
  %v2 = llvm.insertelement %global_address_low, %v1[2] : vector<4xi32>
  %base = llvm.insertelement %global_address_high, %v2[3] : vector<4xi32>

  rocdl.tensor.load.to.lds %base, %dgroup1, %dgroup2, %dgroup3 cachepolicy 0 : vector<4xi32>, vector<8xi32>

These tensor DMA operations were introduced in gfx1250.

make_dma_descriptor()

Return op name amdgpu.make_dma_descriptor as a bitstring.

make_dma_descriptor(ssa)

amdgpu.make_dma_descriptor - Make all descriptor groups needed by TensorLoadToLDS/TensorStoreFromLDS.

This op has support for result type inference.

Attributes

global_static_sizes - Single, DenseI64ArrayAttr, i64 dense array attribute
global_static_strides - Single, DenseI64ArrayAttr, i64 dense array attribute
shared_static_sizes - Single, DenseI64ArrayAttr, i64 dense array attribute

Operands

base - Single, AMDGPU_TDMBaseType, Pair of base addresses that move data between LDS and global storage.
global_dynamic_sizes - Variadic, Index, variadic of index
global_dynamic_strides - Variadic, Index, variadic of index
shared_dynamic_sizes - Variadic, Index, variadic of index
workgroup_mask - Optional, anonymous/composite constraint, fixed-length vector of 1-bit signless integer values of length 16
early_timeout - Optional, I1, 1-bit signless integer
pad_amount - Optional, I32, 32-bit signless integer
pad_interval - Optional, I32, 32-bit signless integer
atomic_barrier_address - Optional, anonymous/composite constraint, memref of State of an in-LDS barrier. values
atomic_barrier_indices - Variadic, Index, variadic of index
global_increment - Optional, Index, index
lds_increment - Optional, I32, 32-bit signless integer
iteration_count - Optional, Index, index

Results

desc - Single, AMDGPU_TDMDescriptorType, Descriptors used in tensor store/load operations.

Description

Make all descriptor groups needed by tensor memory operations.

The $base operand corresponds to the base pair addresses, one must be an address in LDS while the other must be a global memory location.

$global{static/dynamic}_sizes determine the size of the tensor. $global{static/dynamic}strides determine the strides of the tensor. $shared{static/dynamic}_sizes determines the size of the tile.

$workgroup_mask broadcast load to workgroups inside of a workgroup cluster (0 = do not broadcast result to workgroup, 1 = broadcast result to workgroup). Ignored for stores. An all zeros mask is interpreted as a non-broadcasted load.

$early_timeout return data to requesters as soon as cache supplies it.

Padding can be applied to the LDS address when copying from memory to LDS, but not when copying from LDS to memory. The values in the padded target addresses remain the same as before the operation was applied. $pad_interval must be a power of two contained in [2, 256]. $pad_amount must be a value contained in [1, 128].

If an atomic barrier is provided, it will be arrived at once after each load/store using this descriptor is completed. Its indices must be non-negative and in-bounds for the corresponding dimensions of the barrier memref.

2D and 3D tensors may be iterated over by setting $global_increment, $lds_increment, and $iteration_count. $global_increment determines how much to increment the starting global memory address per iteration in units of the $base's element type. $lds_increment determines how much to increment the starting LDS address per iteration in units of the $base's element type. $iterate_count determines how many times to iterate, it must be a value in the inclusive interval [1, 256].

 // Example of moving a two-dimensional tensor to LDS.
 %base = amdgpu.make_dma_base %global[0, 0], %lds[0, 0] : memref<64x64xi32>, memref<64x64xi32, #gpu.address_space<workgroup>> -> !amdgpu.tdm_base<i32>
 %descriptor = amdgpu.make_dma_descriptor %base globalSize [64, 64] globalStride [64, 1] sharedSize [64, 64] : !amdgpu.tdm_base<i32> -> !amdgpu.tdm_descriptor
 amdgpu.tensor_load_to_lds %descriptor : !amdgpu.tdm_descriptor

 // Example of moving a two dimension tensor to LDS where padding is applied after every integer.
 %base = amdgpu.make_dma_base %global[0, 0], %lds[0, 0] : memref<32x32xi32>, memref<64x64xi32, #gpu.address_space<workgroup>> -> !amdgpu.tdm_base<i32>
 %descriptor = amdgpu.make_dma_descriptor %base globalSize [32, 32] globalStride [32, 1] sharedSize [64, 64] padShared(%pad_amount every %pad_interval) : !amdgpu.tdm_base<i32> -> !amdgpu.tdm_descriptor
 amdgpu.tensor_load_to_lds %descriptor : !amdgpu.tdm_descriptor

make_gather_dma_base()

Return op name amdgpu.make_gather_dma_base as a bitstring.

make_gather_dma_base(ssa)

amdgpu.make_gather_dma_base - Pair of based addresses used when moving tiles between LDS and global memory.

Operands

global - Single, AnyMemRef, memref of any non-token type values
global_indices - Variadic, Index, variadic of index
lds - Single, AnyMemRef, memref of any non-token type values
lds_indices - Variadic, Index, variadic of index

Results

base - Single, AMDGPU_TDMGatherBaseType, Pair of base addresses that move data between LDS and global storage.

Description

This operation creates a pair of addresses that will be used by tensor_load_to_lds and tensor_store_from_lds.

The global and LDS indices must be non-negative and in-bounds for the corresponding dimensions of their memrefs.

This operation creates a value corresponding to the tensor descriptor (D#) group 0 found in TensorLoadToLDSOp and TensorStoreFromLDSOp in the rocdl dialect.

Unlike make_dma_base, this operation returns !amdgpu.tdm_gather_base<$element_type, $index_type> which is only compatible with make_gather_dma_descriptor. Using the descriptor returned by make_gather_dma_descriptor will set the tensor_load_to_lds and tensor_store_from_lds to gather mode.

  %base = amdgpu.make_gather_dma_base %global[%idx0, %idx1], %lds[%idx2, %idx3] : memref<64x64xi32>, memref<64x64xi32, #gpu.address_space<workgroup>> -> !amdgpu.tdm_gather_base<i32, i16>
  // %indices : i16
  %descriptor = amdgpu.make_gather_dma_descriptor %base[%indices] globalSize [2, 2] globalStride [2, 1] sharedSize [2, 2] : !amdgpu.tdm_gather_base<i32, i16>, i16 -> !amdgpu.tdm_descriptor
  amdgpu.tensor_load_to_lds %descriptor : !amdgpu.tdm_descriptor

make_gather_dma_descriptor()

Return op name amdgpu.make_gather_dma_descriptor as a bitstring.

make_gather_dma_descriptor(ssa)

amdgpu.make_gather_dma_descriptor - Make all descriptor groups needed by TensorLoadToLDS/TensorStoreFromLDS.

This op has support for result type inference.

Attributes

global_static_sizes - Single, DenseI64ArrayAttr, i64 dense array attribute
global_static_strides - Single, DenseI64ArrayAttr, i64 dense array attribute
shared_static_sizes - Single, DenseI64ArrayAttr, i64 dense array attribute

Operands

base - Single, AMDGPU_TDMGatherBaseType, Pair of base addresses that move data between LDS and global storage.
indices - Single, anonymous/composite constraint, vector of 32-bit signless integer values of at least length 1 of at most length 8 or vector of 16-bit signless integer values of at least length 1 of at most length 16
global_dynamic_sizes - Variadic, Index, variadic of index
global_dynamic_strides - Variadic, Index, variadic of index
shared_dynamic_sizes - Variadic, Index, variadic of index
workgroup_mask - Optional, anonymous/composite constraint, fixed-length vector of 1-bit signless integer values of length 16
early_timeout - Optional, I1, 1-bit signless integer
pad_amount - Optional, I32, 32-bit signless integer
pad_interval - Optional, I32, 32-bit signless integer
atomic_barrier_address - Optional, anonymous/composite constraint, memref of State of an in-LDS barrier. values
atomic_barrier_indices - Variadic, Index, variadic of index
global_increment - Optional, Index, index
lds_increment - Optional, I32, 32-bit signless integer
iteration_count - Optional, Index, index

Results

desc - Single, AMDGPU_TDMDescriptorType, Descriptors used in tensor store/load operations.

Description

Make all descriptor groups needed by tensor memory operations in gather mode.

If an atomic barrier is provided, its indices must be non-negative and in-bounds for the corresponding dimensions of the barrier memref.

memory_counter_wait()

Return op name amdgpu.memory_counter_wait as a bitstring.

memory_counter_wait(ssa)

amdgpu.memory_counter_wait - Wait for specified hardware counters

Attributes

load - Optional, I32Attr, 32-bit signless integer attribute
store - Optional, I32Attr, 32-bit signless integer attribute
ds - Optional, I32Attr, 32-bit signless integer attribute
exp - Optional, I32Attr, 32-bit signless integer attribute
tensor - Optional, I32Attr, 32-bit signless integer attribute

Description

Wait for the specified counters to be less-than or equal-to the provided values before continuing.

Counters can lower to different instructions on different architectires, including clamping to the some HW supported max value or combining multiple counters into one.

mfma()

Return op name amdgpu.mfma as a bitstring.

mfma(ssa)

amdgpu.mfma - MLIR wrapper for CDNA mfma instructions

This op has support for result type inference.

Attributes

m - Single, I32Attr, 32-bit signless integer attribute whose value is one of {4, 16, 32}
n - Single, I32Attr, 32-bit signless integer attribute whose value is one of {4, 16, 32}
k - Single, I32Attr, 32-bit signless integer attribute whose value is one of {1, 2, 4, 8, 16, 32, 64, 128}
blocks - Single, I32Attr, 32-bit signless integer attribute whose value is one of {1, 2, 4, 16}
cbsz - Single, I32Attr, 32-bit signless integer attribute
abid - Single, I32Attr, 32-bit signless integer attribute
blgp - Single, ROCDL_MFMAPermBAttr, permutations of the lanes storing B in an MFMA
reducePrecision - Optional, UnitAttr, unit attribute
negateA - Optional, UnitAttr, unit attribute
negateB - Optional, UnitAttr, unit attribute
negateC - Optional, UnitAttr, unit attribute

Operands

sourceA - Single, MFMAInTypes, 32-bit float or 64-bit float or 32-bit signless integer or 64-bit signless integer or vector of 32-bit float values of length 2 or vector of 16-bit float values of length 4/8 or vector of bfloat16 type values of length 2/4/8 or vector of 8-bit signless integer values of length 4/8/16 or vector of f8E5M2FNUZ type or f8E4M3FNUZ type values of length 8 or vector of f8E5M2 type or f8E4M3FN type values of length 8/32 or vector of f6E2M3FN type or f6E3M2FN type or f4E2M1FN type values of length 32
sourceB - Single, MFMAInTypes, 32-bit float or 64-bit float or 32-bit signless integer or 64-bit signless integer or vector of 32-bit float values of length 2 or vector of 16-bit float values of length 4/8 or vector of bfloat16 type values of length 2/4/8 or vector of 8-bit signless integer values of length 4/8/16 or vector of f8E5M2FNUZ type or f8E4M3FNUZ type values of length 8 or vector of f8E5M2 type or f8E4M3FN type values of length 8/32 or vector of f6E2M3FN type or f6E3M2FN type or f4E2M1FN type values of length 32
destC - Single, MFMAOutTypes, 64-bit float or vector of 32-bit float values of length 4/16/32 or vector of 32-bit signless integer values of length 4/16/32 or vector of 64-bit float values of length 4

Results

destD - Single, MFMAOutTypes, 64-bit float or vector of 32-bit float values of length 4/16/32 or vector of 32-bit signless integer values of length 4/16/32 or vector of 64-bit float values of length 4

Description

The amdgpu.mfma op is an MLIR wrapper around intrinsics for various mfma instructions in the CDNA architecture, which perform multiple outer products in order to allow fast matrix multiplication.

The wrapper will select an appropriate mfma instruction, if one is available, based on the provided m, k, n, and nBlks attributes, along with the types of the source and destination arguments.

For information on the layouts of the input and output matrices (which are stored in sourceA, sourceB, destC, and destD), see the CDNA ISA documentation.

The cbsz, abid, and blgp parameters control how the lanes of the wave are permuted when matrix data is being loaded: blgp can be any number of fixed permutations, cbsz specifies the log_2 of the number of chunks the lanes holding sourceA are split into, and abid selects one of those chunks.

Note, this wrapper allows specifying vector<4Kxi8> arguments to MFMA intrinsics that take an integer type of width 4K. For example, one can provide a vector<4xi8> as an argument to an MFMA instruction that logically takes 4 i8s but whose intrinsics are specified to take an i32. In these cases, the bytes in the vector will be concatenated in little-endian order (that is, v[0] will go to arg[7:0], v[1] to arg[15:8] and so on).

The negateA, negateB, and negateC flags are only supported for double-precision operations on gfx94x.

Example:

  %0 = amdgpu.mfma 16x16x16 %matA * %matB + %matC
    : vector<4xf16>, vector<4xf16>, vector<4xf32>

  %1 = amdgpu.mfma 32x32x1 %matD * %matE + %matF
    { abid = 1 : i32, cbsz = 1 : i32, blocks = 2 : i32 }
    blgp = bcast_second_32 : f32, f32, vector<32xf32>

packed_scaled_trunc()

Return op name amdgpu.packed_scaled_trunc as a bitstring.

packed_scaled_trunc(ssa)

amdgpu.packed_scaled_trunc - Round two floats into a packed vector of floats

Attributes

index - Single, I32Attr, 32-bit signless integer attribute whose value is non-negative whose maximum value is 7

Operands

source - Single, anonymous/composite constraint, vector of 32-bit float or 16-bit float or bfloat16 type values of length 1/2
scale - Single, F32, 32-bit float
existing - Optional, anonymous/composite constraint, fixed-length vector of f8E5M2 type or f8E4M3FN type values of length 4 or fixed-length vector of f4E2M1FN type values of length 8

Results

res - Single, anonymous/composite constraint, fixed-length vector of f8E5M2 type or f8E4M3FN type values of length 4 or fixed-length vector of f4E2M1FN type values of length 8

Description

Scale and round the inputs source (which is undefined if not specified) into the low or high word (bottom two or top two) elements of the returned vector, keeping the other two elements of existing unchanged if present (or undefined if it was not passed in).

The reason for this odd signature is that AMD GPUs cannot easily work with sub-registers, and so the conversion intrinsics take 32-bit wide packed vectors of float values.

packed_stoch_round_fp8()

Return op name amdgpu.packed_stoch_round_fp8 as a bitstring.

packed_stoch_round_fp8(ssa)

amdgpu.packed_stoch_round_fp8 - Round float stochiastically into a packed vector of 8-bit floats

Attributes

storeIndex - Single, I32Attr, 32-bit signless integer attribute whose value is non-negative whose maximum value is 3

Operands

source - Single, F32, 32-bit float
stochiasticParam - Single, I32, 32-bit signless integer
existing - Optional, anonymous/composite constraint, fixed-length vector of f8E4M3FNUZ type or f8E5M2FNUZ type or f8E4M3FN type or f8E5M2 type values of length 4

Results

res - Single, anonymous/composite constraint, fixed-length vector of f8E4M3FNUZ type or f8E5M2FNUZ type or f8E4M3FN type or f8E5M2 type values of length 4

Description

Round the input source, adding in stochiasticParam, and place it into the storeIndexth element of res.

If existing is passed in, elements of res other than the one at storeIndex are copied from existing.

The reason for this odd signature is that AMD GPUs cannot easily work with sub-registers, and so the conversion intrinsics (which are currently the only way to work with 8-bit float types) take packed vectors of 4 8-bit values.

packed_trunc_2xfp8()

Return op name amdgpu.packed_trunc_2xfp8 as a bitstring.

packed_trunc_2xfp8(ssa)

amdgpu.packed_trunc_2xfp8 - Round two floats into a packed vector of 8-bit floats

Attributes

wordIndex - Single, I32Attr, 32-bit signless integer attribute whose value is non-negative whose maximum value is 1

Operands

sourceA - Single, F32, 32-bit float
sourceB - Optional, F32, 32-bit float
existing - Optional, anonymous/composite constraint, fixed-length vector of f8E4M3FNUZ type or f8E5M2FNUZ type or f8E4M3FN type or f8E5M2 type values of length 4

Results

res - Single, anonymous/composite constraint, fixed-length vector of f8E4M3FNUZ type or f8E5M2FNUZ type or f8E4M3FN type or f8E5M2 type values of length 4

Description

Round the inputs sourceA and sourceB (which is undefined if not specified) into the low or high word (bottom two or top two) elements of the returned vector, keeping the other two elements of existing unchanged if present (or undefined if it was not passed in).

permlane_swap()

Return op name amdgpu.permlane_swap as a bitstring.

permlane_swap(ssa)

amdgpu.permlane_swap - AMDGPU permlane swap op

This op has support for result type inference.

Attributes

row_length - Single, I32Attr, 32-bit signless integer attribute
fetch_inactive - Single, BoolAttr, bool attribute
bound_ctrl - Single, BoolAttr, bool attribute

Operands

src - Single, AnyIntegerOrFloatOr1DVector, Integer or Float or fixed-length vector of Integer or Float values of ranks 1

Results

result - Single, AnyIntegerOrFloatOr1DVector, Integer or Float or fixed-length vector of Integer or Float values of ranks 1

Description

High-level wrapper on rocdl.permlane{16,32}.swap variants for permutations on rows of lanes in a subgroup.

Supports arbitrary int/float/vector types, which will be repacked to i32 and one or more rocdl.permlane_swap ops during lowering. Supported lane permutations:

Swap the data between odd and even rows of 16 lanes
Swap the data between the first 32 lanes and the last 32 lanes

Example:

%0 = amdgpu.permlane_swap %src 16 : f16
%1 = amdgpu.permlane_swap %src 32 { fetch_inactive = true, bound_ctrl = true } : f16

Operands:

$src: Vector register to permute across lanes of the subgroup.
$row_length: The length of a row to permute in number of lanes (valid values are 16 and 32).
$fetch_inactive: Optional. Used to dertermine behavior of a fetch from a disabled lane. fetch_inactive = false: If the source lane is disabled, use bound_ctrl to determine the source value. fetch_inactive = true: If the source lane is disabled, fetch the source value anyway (ignoring bound_ctrl).
$bound_ctrl: Optional. Used to determine what a thread should do if its source operand is from a disabled lane: use the value zero, or disable the write. bound_ctrl = false: Do not write when source is from a disabled lane bound_ctrl = true: Use zero as input if source is from a disabled lane

Note: Lowering is only supported on gfx950 and up.

permlane_var()

Return op name amdgpu.permlane_var as a bitstring.

permlane_var(ssa)

amdgpu.permlane_var - AMDGPU variable-selector permlane op (GFX12+)

This op has support for result type inference.

Attributes

cross - Single, BoolAttr, bool attribute
fetch_inactive - Single, BoolAttr, bool attribute
bound_ctrl - Single, BoolAttr, bool attribute

Operands

src - Single, AnyIntegerOrFloatOr1DVector, Integer or Float or fixed-length vector of Integer or Float values of ranks 1
selector - Single, I32, 32-bit signless integer

Results

result - Single, AnyIntegerOrFloatOr1DVector, Integer or Float or fixed-length vector of Integer or Float values of ranks 1

Description

High-level wrapper on rocdl.permlane16.var and rocdl.permlanex16.var for per-lane variable-selector permutations within a wave32 subgroup.

Supports arbitrary int/float/vector types, which will be repacked to i32 and one or more ROCDL intrinsic calls during lowering.

cross = false: intra-row permutation (each lane in a 16-lane row reads from a lane in the same row, selected by $selector). Maps to rocdl.permlane16.var.
cross = true: cross-row permutation (each lane reads from the opposite 16-lane row, selected by $selector). Maps to rocdl.permlanex16.var.

$selector is an i32 VGPR providing the per-lane source-lane index.

Example:

%0 = amdgpu.permlane_var %src, %sel { cross = false } : f16
%1 = amdgpu.permlane_var %src, %sel { cross = true } : f32

Note: Lowering is only supported on GFX12+.

raw_buffer_atomic_cmpswap()

Return op name amdgpu.raw_buffer_atomic_cmpswap as a bitstring.

raw_buffer_atomic_cmpswap(ssa)

amdgpu.raw_buffer_atomic_cmpswap - Raw Buffer Atomic compare-and-swap

This op has support for result type inference.

Attributes

boundsCheck - Single, BoolAttr, bool attribute
indexOffset - Optional, I32Attr, 32-bit signless integer attribute

Operands

src - Single, AnyType, any non-token type
cmp - Single, AnyType, any non-token type
memref - Single, AnyMemRef, memref of any non-token type values
indices - Variadic, I32, variadic of 32-bit signless integer
sgprOffset - Optional, I32, 32-bit signless integer

Results

value - Single, AnyType, any non-token type

Description

The amdgpu.raw_buffer_atomic_cmpswap op is a wrapper around the buffer-based atomic compare-and-swap min available on AMD GPUs.

The index into the buffer is computed as for memref.store with the addition of indexOffset (which is used to aid in emitting vectorized code) and, if present sgprOffset (which is added after bounds checks and includes any non-zero offset on the memref type).

All indexing components are given in terms of the memref's element size, not the byte lengths required by the intrinsic.

Out of bounds atomic operations are ignored in hardware.

See amdgpu.raw_buffer_load for a description of how the underlying instruction is constructed.

raw_buffer_atomic_fadd()

Return op name amdgpu.raw_buffer_atomic_fadd as a bitstring.

raw_buffer_atomic_fadd(ssa)

amdgpu.raw_buffer_atomic_fadd - Raw Buffer Floating-point Atomic Add (MI-* only)

This op has support for result type inference.

Attributes

boundsCheck - Single, BoolAttr, bool attribute
indexOffset - Optional, I32Attr, 32-bit signless integer attribute

Operands

value - Single, anonymous/composite constraint, 32-bit float or vector of 16-bit float or bfloat16 type values of length 2
memref - Single, AnyMemRef, memref of any non-token type values
indices - Variadic, I32, variadic of 32-bit signless integer
sgprOffset - Optional, I32, 32-bit signless integer

Results

oldValue - Single, anonymous/composite constraint, 32-bit float or vector of 16-bit float or bfloat16 type values of length 2

Description

The amdgpu.raw_buffer_atomic_fadd op is a wrapper around the buffer-based atomic floating point addition available on the MI-* series of AMD GPUs.

All indexing components are given in terms of the memref's element size, not the byte lengths required by the intrinsic.

The op returns the value observed before the atomic update.

See amdgpu.raw_buffer_load for a description of how the underlying instruction is constructed.

raw_buffer_atomic_fmax()

Return op name amdgpu.raw_buffer_atomic_fmax as a bitstring.

raw_buffer_atomic_fmax(ssa)

amdgpu.raw_buffer_atomic_fmax - Raw Buffer Floating-point Atomic Max (non-GFX9)

This op has support for result type inference.

Attributes

boundsCheck - Single, BoolAttr, bool attribute
indexOffset - Optional, I32Attr, 32-bit signless integer attribute

Operands

value - Single, anonymous/composite constraint, 32-bit float or 64-bit float
memref - Single, AnyMemRef, memref of any non-token type values
indices - Variadic, I32, variadic of 32-bit signless integer
sgprOffset - Optional, I32, 32-bit signless integer

Results

oldValue - Single, anonymous/composite constraint, 32-bit float or 64-bit float

Description

The amdgpu.raw_buffer_atomic_fmax op is a wrapper around the buffer-based atomic floating point max available on AMD GPUs (except GFX9).

All indexing components are given in terms of the memref's element size, not the byte lengths required by the intrinsic.

The op returns the value observed before the atomic update. Out of bounds atomic operations are ignored in hardware.

See amdgpu.raw_buffer_load for a description of how the underlying instruction is constructed.

raw_buffer_atomic_smax()

Return op name amdgpu.raw_buffer_atomic_smax as a bitstring.

raw_buffer_atomic_smax(ssa)

amdgpu.raw_buffer_atomic_smax - Raw Buffer Signed Integer Atomic Max

This op has support for result type inference.

Attributes

boundsCheck - Single, BoolAttr, bool attribute
indexOffset - Optional, I32Attr, 32-bit signless integer attribute

Operands

value - Single, I32, 32-bit signless integer
memref - Single, AnyMemRef, memref of any non-token type values
indices - Variadic, I32, variadic of 32-bit signless integer
sgprOffset - Optional, I32, 32-bit signless integer

Results

oldValue - Single, I32, 32-bit signless integer

Description

The amdgpu.raw_buffer_atomic_smax op is a wrapper around the buffer-based atomic signed integer max available on AMD GPUs.

All indexing components are given in terms of the memref's element size, not the byte lengths required by the intrinsic.

The op returns the value observed before the atomic update. Out of bounds atomic operations are ignored in hardware.

See amdgpu.raw_buffer_load for a description of how the underlying instruction is constructed.

raw_buffer_atomic_umin()

Return op name amdgpu.raw_buffer_atomic_umin as a bitstring.

raw_buffer_atomic_umin(ssa)

amdgpu.raw_buffer_atomic_umin - Raw Buffer Unsigned Integer Atomic Min

This op has support for result type inference.

Attributes

boundsCheck - Single, BoolAttr, bool attribute
indexOffset - Optional, I32Attr, 32-bit signless integer attribute

Operands

value - Single, I32, 32-bit signless integer
memref - Single, AnyMemRef, memref of any non-token type values
indices - Variadic, I32, variadic of 32-bit signless integer
sgprOffset - Optional, I32, 32-bit signless integer

Results

oldValue - Single, I32, 32-bit signless integer

Description

The amdgpu.raw_buffer_atomic_umin op is a wrapper around the buffer-based atomic signed integer min available on AMD GPUs.

All indexing components are given in terms of the memref's element size, not the byte lengths required by the intrinsic.

The op returns the value observed before the atomic update. Out of bounds atomic operations are ignored in hardware.

See amdgpu.raw_buffer_load for a description of how the underlying instruction is constructed.

raw_buffer_load()

Return op name amdgpu.raw_buffer_load as a bitstring.

raw_buffer_load(ssa)

amdgpu.raw_buffer_load - Raw Buffer load, exposing GCN features

Attributes

boundsCheck - Single, BoolAttr, bool attribute
indexOffset - Optional, I32Attr, 32-bit signless integer attribute

Operands

memref - Single, AnyMemRef, memref of any non-token type values
indices - Variadic, I32, variadic of 32-bit signless integer
sgprOffset - Optional, I32, 32-bit signless integer

Results

value - Single, AnyType, any non-token type

Description

The amdgpu.raw_buffer_load op is a wrapper around the buffer load intrinsics available on AMD GPUs, including extensions in newer GPUs.

The index into the buffer is computed as for memref.load with the additon of indexOffset and sgprOffset (which may or may not be considered in bounds checks and includes any offset present on the memref type if it's non-zero).

All indices and offsets are in units of the memref's data type and are converted to bytes during lowering.

When a load is out of bounds, the instruction returns zero. Partially-out of bounds have chipset-dependent behavior: whether reading 2 elements starting at index 7 of a memref<8xf32> returns the last element in the first vector component depends on the architecture.

The memref struct is converted into a buffer resource (a V#) and the arguments are translated to intrinsic arguments as follows:

The base address of the buffer is the base address of the memref
The stride is 0 to enable raw mode
The number of records is the size of the memref, in bytes In the case of dynamically-shaped memrefs, this is computed at runtime as max_d (size(d) stride(d)) sizeof(elementType(memref))
The offset enable bit is 1, the index enable bit is 0.
The thread ID addition bit is off
If boundsCheck is false and the target chipset is RDNA, OOB_SELECT is set to 2 to disable bounds checks, otherwise it is 3
The cache coherency bits are off

raw_buffer_store()

Return op name amdgpu.raw_buffer_store as a bitstring.

raw_buffer_store(ssa)

amdgpu.raw_buffer_store - Raw Buffer Store, exposing GCN features

Attributes

boundsCheck - Single, BoolAttr, bool attribute
indexOffset - Optional, I32Attr, 32-bit signless integer attribute

Operands

value - Single, AnyType, any non-token type
memref - Single, AnyMemRef, memref of any non-token type values
indices - Variadic, I32, variadic of 32-bit signless integer
sgprOffset - Optional, I32, 32-bit signless integer

Description

The amdgpu.raw_buffer_store op is a wrapper around the buffer store intrinsics available on AMD GPUs, including extensions in newer GPUs.

The store index is computed as in memref.store with the addition of indexOffset (which is included for uniformity with atomics and may be useful when writing vectorized code) and sgprOffset (which is added after bounds checks and implicitly includes the offset of the memref type if non-zero). All index components are in terms of the elements of the memref, not bytes, and are scaled up appropriately.

Out of bounds stores are ignored in hardware. Wthether a vector write that includes some in-bounds and soeme out-of-bounds components is partically completed is chipset-dependent.

See amdgpu.raw_buffer_load for a description of how the underlying instruction is constructed.

scaled_ext_packed()

Return op name amdgpu.scaled_ext_packed as a bitstring.

scaled_ext_packed(ssa)

amdgpu.scaled_ext_packed - Extend a vector of packed floating point values

Attributes

index - Single, I32Attr, 32-bit signless integer attribute whose value is non-negative whose maximum value is 7

Operands

source - Single, anonymous/composite constraint, vector of f8E5M2 type or f8E4M3FN type values of length 1/2/3/4 or vector of f4E2M1FN type values of length 1/2/3/4/5/6/7/8
scale - Single, F32, 32-bit float

Results

res - Single, anonymous/composite constraint, fixed-length vector of 32-bit float values of length 2 or fixed-length vector of 16-bit float values of length 2 or fixed-length vector of bfloat16 type values of length 2

Description

Extend and scale two packed floats in source[index] to two floats and return them.

If the passed-in vector has fewer than two elements, or the input is scalar, the remaining values in the <2 x i8> will be filled with undefined values as needed.

scaled_ext_packed_matrix()

Return op name amdgpu.scaled_ext_packed_matrix as a bitstring.

scaled_ext_packed_matrix(ssa)

amdgpu.scaled_ext_packed_matrix - Extend a wave-wide matrix of packed floating point values

Attributes

blockSize - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16, 32}
firstScaleLane - Single, I32Attr, 32-bit signless integer attribute whose value is one of {0, 16}
firstScaleByte - Single, I32Attr, 32-bit signless integer attribute whose minimum value is 0 whose maximum value is 3

Operands

source - Single, anonymous/composite constraint, vector<8xF4E2M1FN> of f4E2M1FN type values or vector<8xF8E4M3FN> of f8E4M3FN type values or vector<8xF8E5M2> of f8E5M2 type values or vector<16xF6E2M3FN> of f6E2M3FN type values or vector<16xF6E3M2FN> of f6E3M2FN type values
scale - Single, anonymous/composite constraint, vector<4xF8E8M0FNU> of f8E8M0FNU type values

Results

res - Single, anonymous/composite constraint, vector<8xF32> of 32-bit float values or vector<8xF16> of 16-bit float values or vector<8xBF16> of bfloat16 type values or vector<16xF32> of 32-bit float values or vector<16xF16> of 16-bit float values or vector<16xBF16> of bfloat16 type values

Description

Extend matrix of microfloats (8 or 16 elements per lane) using a set of scales that may be stored on other lanes.

The scales applied to the input microfloats are stored in bytes which come from the scales input provided in a half of the wave identified by firstScaleLane. The bytes used is selected by firstScaleByte and depends on the type of source. The 16 vectors in consecutive lanes starting from firstScaleLane (which we'll call the scale vectors) will be used by both halves of the wave (with lane L reading from L % 16'th scale vector).

When source is either F4E2M1FN, F6E2M3FN, or F6E3M2FN each half of the wave will use a different byte. The first one being firstScaleByte and the second one being firstScaleByte + 1. When the block size is 32, firstScaleByte can be either 0 or 2, selecting halves of the scale vectors. Lanes 0-15 will read from firstScaleByte and lanes 16-31 will read from firstScaleByte + 1.

For example:

// Input: 8-element vector of F8E4M3FN, converting to F32
// Lanes 0-15 read from byte 0, lanes 16-31 read from byte 1
%result = amdgpu.scaled_ext_packed_matrix %source scale(%scales)
  blockSize(32) firstScaleLane(0) firstScaleByte(0)
  : vector<8xf8E4M3FN>, vector<4xf8E8M0FNU> -> vector<8xf32>

// Input: 16-element vector of F6E2M3FN, converting to F16
// Lanes 0-15 read from byte 2, lanes 16-31 read from byte 3
%result = amdgpu.scaled_ext_packed_matrix %source scale(%scales)
  blockSize(32) firstScaleLane(16) firstScaleByte(2)
  : vector<16xf6E2M3FN>, vector<4xf8E8M0FNU> -> vector<16xf16>

When source is either F4E2M1FN, F6E2M3FN, or F6E3M2FN and the block size is 16, firstScaleByte can be 0 or 1. Lanes 0-15 read from the firstScaleByteth element of the scale vectors, while lanes 16-31 read from firstScaleByte + 2. For example:

// Input: 8-element vector of F8E5M2, converting to BF16
// Lanes 0-15 read from byte 0, lanes 16-31 read from byte 2 (0+2)
%result = amdgpu.scaled_ext_packed_matrix %source scale(%scales)
  blockSize(16) firstScaleLane(0) firstScaleByte(0)
  : vector<8xf8E5M2>, vector<4xf8E8M0FNU> -> vector<8xbf16>

// Input: 16-element vector of F6E3M2FN, converting to F32
// Lanes 0-15 read from byte 1, lanes 16-31 read from byte 3 (1+2)
%result = amdgpu.scaled_ext_packed_matrix %source scale(%scales)
  blockSize(16) firstScaleLane(16) firstScaleByte(1)
  : vector<16xf6E3M2FN>, vector<4xf8E8M0FNU> -> vector<16xf32>

Note: the layout for the scales generally mirrors how the WMMA instructions use for matrix scales. These selection operands allows one to choose portions of the matrix to convert.

When source is either F8E4M3FN or F8E5M2 and blockSize is 32, then the same byte will be used by both halves of the wave. In this case, firstScaleByte can be any value from 0 to 3.

When source is either F8E4M3FN or F8E5M2 and blockSize is 16, following combinations are allowed:

firstScaleLane(0), firstScaleByte(0)
firstScaleLane(16), firstScaleByte(2) all other combinations are reserved.

Available on gfx1250+.

scaled_mfma()

Return op name amdgpu.scaled_mfma as a bitstring.

scaled_mfma(ssa)

amdgpu.scaled_mfma - MLIR wrapper for CDNA scaled mfma instructions

This op has support for result type inference.

Attributes

m - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16, 32}
n - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16, 32}
k - Single, I32Attr, 32-bit signless integer attribute whose value is one of {64, 128}
scalesIdxA - Single, I32Attr, 32-bit signless integer attribute whose value is non-negative whose maximum value is 3
scalesIdxB - Single, I32Attr, 32-bit signless integer attribute whose value is non-negative whose maximum value is 3

Operands

sourceA - Single, ScaledMFMAInTypes, vector of f8E5M2 type or f8E4M3FN type values of length 32 or vector of f6E2M3FN type or f6E3M2FN type or f4E2M1FN type values of length 32
sourceB - Single, ScaledMFMAInTypes, vector of f8E5M2 type or f8E4M3FN type values of length 32 or vector of f6E2M3FN type or f6E3M2FN type or f4E2M1FN type values of length 32
destC - Single, ScaledMFMAOutTypes, vector of 32-bit float values of length 4/16
scalesA - Single, anonymous/composite constraint, f8E8M0FNU type or fixed-length vector of f8E8M0FNU type values of length 4
scalesB - Single, anonymous/composite constraint, f8E8M0FNU type or fixed-length vector of f8E8M0FNU type values of length 4

Results

destD - Single, ScaledMFMAOutTypes, vector of 32-bit float values of length 4/16

Description

The amdgpu.scaled_mfma op is an MLIR wrapper around intrinsics for various scaled versions of mfma instructions in the CDNA architecture, which perform multiple outer products in order to allow fast matrix multiplication.

Note, this wrapper allows specifying vector<4Kxi8> arguments to MFMA intrinsics that take an integer type of width 4K. For example, one can provide a vector<4xi8> as an argument to an MFMA instruction that logically takes 4 i8s but whose intrinsics are specified to take an i32. In these cases, the bytes in the vector will be concatenated in little-endian order (that is, v[0] will go to arg[7:0], v[1] to arg[15:8] and so on).

This wrapper takes inspiration from amdgpu.mfma, but has some key differences:

amdgpu.scaled_mfma operates on fp4 (f4E2M1FN), fp6 (f6E2M3FN and f6E3M2FN) and fp8 (f8E4M3FN and f8E5M2) types using either M=N=16, K=128 or M=N=32, K=64 as their tile size.
amdgpu.scaled_mfma does not support broadcasting. So, cbsz, abid, and blgp are omitted from this wrapper.
The negateA, negateB, and negateC flags in amdgpu.mfma are only supported for double-precision operations on gfx94x and so are not included here.

Example:

  %0 = amdgpu.scaled_mfma 32x32x64 (%arg0[0] * %arg1) * (%arg0[1] * %arg1) + %arg2
    : vector<4xf8E8M0FNU>, vector<32xf6E2M3FN>, f8E8M0FNU, vector<32xf6E2M3FN>, vector<16xf32>

scaled_wmma()

Return op name amdgpu.scaled_wmma as a bitstring.

scaled_wmma(ssa)

amdgpu.scaled_wmma - MLIR wrapper for scaled wmma instructions

This op has support for result type inference.

Attributes

m - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16, 32}
n - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16}
k - Single, I32Attr, 32-bit signless integer attribute whose value is one of {128}
a_first_scale_lane - Single, I32Attr, 32-bit signless integer attribute whose value is one of {0, 16}
b_first_scale_lane - Single, I32Attr, 32-bit signless integer attribute whose value is one of {0, 16}

Operands

sourceA - Single, ScaledWMMAInTypes, vector of f8E5M2 type or f8E4M3FN type values of length 64 or vector of f6E2M3FN type or f6E3M2FN type values of length 64 or vector of f4E2M1FN type values of length 64/128
sourceB - Single, ScaledWMMAInTypes, vector of f8E5M2 type or f8E4M3FN type values of length 64 or vector of f6E2M3FN type or f6E3M2FN type values of length 64 or vector of f4E2M1FN type values of length 64/128
destC - Single, ScaledWMMAOutTypes, vector of 32-bit float values of length 8/16
scaleA - Single, anonymous/composite constraint, vector of f8E8M0FNU type or f8E4M3FN type values of length 4/8
scaleB - Single, anonymous/composite constraint, vector of f8E8M0FNU type or f8E4M3FN type values of length 4/8

Results

destD - Single, ScaledWMMAOutTypes, vector of 32-bit float values of length 8/16

Description

The amdgpu.scaled_wmma op is an MLIR wrapper around intrinsics for scaled wmma instructions. These instructions perform matrix multiplication with per-block scaling of inputs, supporting fp4, fp6, and fp8 data formats.

The scale instructions support a block size of 16 or 32 and two tile sizes:

16x16x128 with mixed f8/f6/f4 formats (output: vector<8xf32>)
32x16x128 with f4 format only (output: vector<16xf32>)

Scale parameters (scaleA, scaleB) are small vectors of f8 scale values (either f8E8M0FNU, or f8E4M3FN) that are packed into i32/i64 values during lowering. Each lane can operate on 4 bytes (4 scale values), and the number of scales required for each matrix is determined by: num_scales_A = (M × K) / block_size num_scales_B = (N × K) / block_size

The index attributes (a_first_scale_lane, b_first_scale_lane) select which lane to start reading scale values from (0 or 16):

For block size 32, 32 lanes across a single wave are used for the scale values. If the number of scales (num_scales_A or num_scales_B) can fit into half of the available lanes (i.e., num_scales / scales_per_lane == 16 (num_lanes)), then then first_scale_lane can be either 0 or 16. If all lanes are required for storing the scale values (num_scales / scales_per_lane == 32 (num_lanes)), then the first_scale_lane must be 0.
For block size 16, the same rules apply as above except that there are 64 lanes across two waves that are used for the scale values. When num_scales / scales_per_lane == 32 (num lanes), then 16 lanes from each wave are used. first_scale_lane of 0 or 16 will decide which lanes are used for this. When num_scales / scales_per_lane == 64 (num_lanes), then first_scale_lane must be set to 0.

Example:

  // 16x16x128: fp8 inputs
  %0 = amdgpu.scaled_wmma 16x16x128 (%scaleVecA * %matA) * (%scaleVecB * %matB) + %matC
    {a_first_scale_lane = 0 : i32, b_first_scale_lane = 0 : i32}
    : vector<4xf8E8M0FNU>, vector<64xf8E4M3FN>,
    vector<4xf8E8M0FNU>, vector<64xf8E4M3FN>, vector<8xf32>

  // 32x16x128: fp4 inputs with different scale lanes
  %1 = amdgpu.scaled_wmma 32x16x128 (%scaleVecD * %matD) * (%scaleVecE * %matE) + %matF
    {a_first_scale_lane = 0 : i32, b_first_scale_lane = 16 : i32}
    : vector<8xf8E4M3FN>, vector<128xf4E2M1FN>,
    vector<8xf8E4M3FN>, vector<64xf4E2M1FN>, vector<16xf32>

sched_barrier()

Return op name amdgpu.sched_barrier as a bitstring.

sched_barrier(ssa)

amdgpu.sched_barrier - Barrier that limits the backend scheduler of instruction movement

Attributes

opts - Single, ROCDL_SchedGroupMaskAttr, instruction type mask for scheduling barriers

Description

amdgpu.sched_barrier serves as a barrier that could be configured to restrict movements of instructions through it as defined by the ROCDL scheduling group mask enum.

sparse_mfma()

Return op name amdgpu.sparse_mfma as a bitstring.

sparse_mfma(ssa)

amdgpu.sparse_mfma - MLIR wrapper for CDNA sparse mfma (smfmac) instructions

This op has support for result type inference.

Attributes

m - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16, 32}
n - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16, 32}
k - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16, 32, 64, 128}
cbsz - Single, I32Attr, 32-bit signless integer attribute
abid - Single, I32Attr, 32-bit signless integer attribute

Operands

sourceA - Single, SMFMACSparseInTypes, vector of 16-bit float values of length 4/8 or vector of bfloat16 type values of length 4/8 or vector of 8-bit signless integer values of length 8/16 or vector of f8E4M3FN type or f8E5M2 type values of length 8/16 or vector of f8E4M3FNUZ type or f8E5M2FNUZ type values of length 8/16
sourceB - Single, SMFMACDenseInTypes, vector of 16-bit float values of length 8/16 or vector of bfloat16 type values of length 8/16 or vector of 8-bit signless integer values of length 16/32 or vector of f8E4M3FN type or f8E5M2 type values of length 16/32 or vector of f8E4M3FNUZ type or f8E5M2FNUZ type values of length 16/32
destC - Single, SMFMACOutTypes, vector of 32-bit float values of length 4/16 or vector of 32-bit signless integer values of length 4/16
sparseIdx - Single, SMFMACIdxTypes, fixed-length vector of 8-bit signless integer values of length 4 or fixed-length vector of 16-bit signless integer values of length 2 or 32-bit signless integer

Results

destD - Single, SMFMACOutTypes, vector of 32-bit float values of length 4/16 or vector of 32-bit signless integer values of length 4/16

Description

The amdgpu.sparse_mfma op is an MLIR wrapper around intrinsics for various smfmac instructions in the AMDGPU architecture, which perform matrix multiply-accumulate operations using 2:4 structured sparsity on matrix A with dense matrices B, C, and D.

On gfx942, smfmac intrinsics support:

M=N=16, K=32 and M=N=32, K=16 for f16 and bf16 sources
M=N=16, K=64 and M=N=32, K=32 for i8 and fp8 sources

On gfx950, smfmac intrinsics additionally support:

M=N=16, K=64 and M=N=32, K=32 for f16 and bf16 sources
M=N=16, K=128 and M=N=32, K=64 for i8 and fp8 sources

The sparseIdx parameter contains packed 2-bit indices identifying which of every 4 dense-K positions are non-zero in the 2:4 sparse matrix A. The required sparseIdx type depends on the variant:

gfx942 16-bit ((m,k) in {(16,32), (32,16)}): 8 bits per lane, carried as vector<4xi8> (one 8-bit set per i8 element).
gfx942 8-bit ((m,k) in {(16,64), (32,32)}) and gfx950 16-bit ((m,k) in {(16,64), (32,32)}): 16 bits per lane, carried as vector<2xi16> (one 16-bit set per i16 element).
gfx950 8-bit ((m,k) in {(16,128), (32,64)}): 32 bits per lane (a full VGPR with no internal set structure), carried as i32.

The cbsz and abid parameters select which index set within the VGPR is used:

gfx942 16-bit: cbsz == 0 selects one of four 8-bit sets via abid[1:0] (range [0, 3]); cbsz != 0 selects the first set.
gfx942 8-bit and gfx950 16-bit: cbsz == 0 selects one of two 16-bit sets via abid[0] (range [0, 1]); cbsz != 0 selects the first set.
gfx950 8-bit: hardware ignores both cbsz and abid; both must be 0.

Example:

  %0 = amdgpu.sparse_mfma 16x16x32 %matA * %matB + %matC sparse(%idx : vector<4xi8>)
    : vector<4xf16>, vector<8xf16>, vector<4xf32>

  %1 = amdgpu.sparse_mfma 16x16x64 %matA * %matB + %matC sparse(%idx : vector<2xi16>)
    : vector<8xi8>, vector<16xi8>, vector<4xi32>

  %2 = amdgpu.sparse_mfma 16x16x64 %matA * %matB + %matC sparse(%idx : vector<2xi16>)
    { cbsz = 0 : i32, abid = 1 : i32 }
    : vector<8xf8E4M3FNUZ>, vector<16xf8E4M3FNUZ>, vector<4xf32>

  %3 = amdgpu.sparse_mfma 16x16x128 %matA * %matB + %matC sparse(%idx : i32)
    : vector<16xf8E4M3FN>, vector<32xf8E4M3FN>, vector<4xf32>

sparse_wmma()

Return op name amdgpu.sparse_wmma as a bitstring.

sparse_wmma(ssa)

amdgpu.sparse_wmma - MLIR wrapper for gfx12+ sparse wmma instructions

This op has support for result type inference.

Attributes

m - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16}
n - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16}
k - Single, I32Attr, 32-bit signless integer attribute whose value is one of {32, 64, 128}
unsignedA - Optional, UnitAttr, unit attribute
unsignedB - Optional, UnitAttr, unit attribute
reuseA - Optional, UnitAttr, unit attribute
reuseB - Optional, UnitAttr, unit attribute
clamp - Optional, UnitAttr, unit attribute
wave64 - Optional, UnitAttr, unit attribute

Operands

sourceA - Single, SWMMACSparseInTypes, vector of 16-bit float values of length 4/8/16 or vector of bfloat16 type values of length 4/8/16 or vector of 8-bit signless integer values of length 4/8/32 or vector of 4-bit signless integer values of length 8/16 or vector of f8E4M3FN type or f8E5M2 type values of length 4/8/16/32 or vector of f8E4M3FNUZ type or f8E5M2FNUZ type values of length 4/8/16/32
sourceB - Single, SWMMACDenseInTypes, vector of 16-bit float values of length 8/16/32 or vector of bfloat16 type values of length 8/16/32 or vector of 8-bit signless integer values of length 4/8/16/64 or vector of 4-bit signless integer values of length 8/16/32 or vector of f8E4M3FN type or f8E5M2 type values of length 4/8/16/64 or vector of f8E4M3FNUZ type or f8E5M2FNUZ type values of length 4/8/16/64
destC - Single, SWMMACOutTypes, vector of 32-bit float values of length 4/8/16 or vector of 16-bit float values of length 4/8 or vector of bfloat16 type values of length 4/8 or vector of 32-bit signless integer values of length 4/8
sparseIdx - Single, SWMMACIdxTypes, fixed-length vector of 8-bit signless integer values of length 4

Results

destD - Single, SWMMACOutTypes, vector of 32-bit float values of length 4/8/16 or vector of 16-bit float values of length 4/8 or vector of bfloat16 type values of length 4/8 or vector of 32-bit signless integer values of length 4/8

Description

The amdgpu.sparse_wmma op is an MLIR wrapper around intrinsics for various swmmac instructions in the AMDGPU architecture, which perform matrix multiply-accumulate operations using 2:4 structured sparsity on matrix A with dense matrices B, C, and D.

On gfx12, swmmac intrinsics support:

M=N=16, K=32 and M=N=32, K=16 for f16, bf16, i8 and i4 sources
M=N=16, K=64 for i4 sources

On gfx1250, swmmac intrinsics additionally support:

M=N=16, K=64 for f16 and bf16 sources
M=N=16, K=128 for f16, bf16 and i8 sources

The sparseIdx parameter contains packed indices identifying the positions of non-zero elements in the 2:4 sparse matrix A. For 16-bit source data, use vector<4xi8> (four 8-bit indices). For 8-bit source data, use vector<2xi16> (two 16-bit indices).

unsignedA and unsignedB flag that the int8 LLVM inputs are unsigned.

The clamp flag is used to saturate the output of type T to numeric_limits<T>::max() in case of overflow.

Example:

  %0 = amdgpu.sparse_wmma 16x16x32 %matA * %matB + %matC sparse(%idx : vector<4xi8>)
    : vector<4xf16>, vector<8xf16>, vector<4xf32>

  %1 = amdgpu.sparse_wmma 16x16x64 %matA * %matB + %matC sparse(%idx : vector<2xi16>)
    : vector<8xi8>, vector<16xi8>, vector<4xi32>

  %2 = amdgpu.sparse_wmma 16x16x64 %matA * %matB + %matC sparse(%idx : vector<2xi16>)
    { unsignedA = 0 : i1, unsignedB = 1 : i1, clamp = 0 : i1 }
    : vector<8xf8E4M3FNUZ>, vector<16xf8E4M3FNUZ>, vector<4xf32>

swizzle_bitmode()

Return op name amdgpu.swizzle_bitmode as a bitstring.

swizzle_bitmode(ssa)

amdgpu.swizzle_bitmode - AMDGPU ds_swizzle op, bitmode variant

This op has support for result type inference.

Attributes

and_mask - Single, I32Attr, 32-bit signless integer attribute
or_mask - Single, I32Attr, 32-bit signless integer attribute
xor_mask - Single, I32Attr, 32-bit signless integer attribute

Operands

src - Single, AnyIntegerOrFloatOr1DVector, Integer or Float or fixed-length vector of Integer or Float values of ranks 1

Results

result - Single, AnyIntegerOrFloatOr1DVector, Integer or Float or fixed-length vector of Integer or Float values of ranks 1

Description

High-level wrapper on bitmode rocdl.ds_swizzle op, masks are represented as separate fields so user won't need to do manual bitpacking.

Supports arbitrary int/float/vector types, which will be repacked to i32 and one or more rocdl.ds_swizzle ops during lowering.

tensor_load_to_lds()

Return op name amdgpu.tensor_load_to_lds as a bitstring.

tensor_load_to_lds(ssa)

amdgpu.tensor_load_to_lds - Load tensors from global memory to LDS.

Operands

desc - Single, AMDGPU_TDMDescriptorType, Descriptors used in tensor store/load operations.

Description

Load tensors of up to five dimensions from global memory to LDS.

This operation was introduced in gfx1250.

tensor_store_from_lds()

Return op name amdgpu.tensor_store_from_lds as a bitstring.

tensor_store_from_lds(ssa)

amdgpu.tensor_store_from_lds - Store tensors from LDS to global memory.

Operands

desc - Single, AMDGPU_TDMDescriptorType, Descriptors used in tensor store/load operations.

Description

Store tensors of up to five dimensions from LDS to global memory.

This operation was introduced in gfx1250.

transpose_load()

Return op name amdgpu.transpose_load as a bitstring.

transpose_load(ssa)

amdgpu.transpose_load - MLIR wrapper for CDNA transpose Load instructions

Operands

src - Single, AnyMemRef, memref of any non-token type values
srcIndices - Variadic, Index, variadic of index

Results

result - Single, anonymous/composite constraint, vector of any non-token type values

Description

The amdgpu.transpose_load op is a wrapper around the ds_read_tr instructions on gfx9 and the ds_load_tr family of instructions on gfx1250.

The transpose load op represents a subgroup load from LDS memory, where the subgroup of threads collectively reads a matrix from the source memref, with each thread reading a vector of the matrix, and gets a transposed matrix in as the result. That is, each thread reads a vector of the col-major matrix at different indices, and the thread's read result is a vector of the corresponding row of the transposed matrix.

This op is a direct wrapper around the ROCDL ds_read_tr family intrinsics on gfx950 and the ds_load_tr family of instructions on gfx1250. Please refer to the respective ISA documentation for more details about its exact semantics.

Format example:

%0 = amdgpu.transpose_load %src[%srcIndices] : memref<128x256xf16> -> vector<4xf16>

Operands:

$src: LDS memref to read from.
$srcIndices: indices into $src to read from for this thread. Indices must be non-negative and in-bounds for the corresponding dimension of $src, matching the constraints of memref.load.
$result: target register this transpose load instruction will write to.

Note: Lowering is only supported on gfx950 and gfx1250, with different permitted load types.

wmma()

Return op name amdgpu.wmma as a bitstring.

wmma(ssa)

amdgpu.wmma - MLIR wrapper for wmma instructions

This op has support for result type inference.

Attributes

m - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16}
n - Single, I32Attr, 32-bit signless integer attribute whose value is one of {16}
k - Single, I32Attr, 32-bit signless integer attribute whose value is one of {4, 16, 32, 64, 128}
subwordOffset - Single, I32Attr, 32-bit signless integer attribute whose value is one of {0, 1}
unsignedA - Optional, UnitAttr, unit attribute
unsignedB - Optional, UnitAttr, unit attribute
clamp - Optional, UnitAttr, unit attribute

Operands

sourceA - Single, WMMAInTypes, vector of 32-bit float values of length 2 or vector of 16-bit float or bfloat16 type values of length 4/8/16 or vector of 8-bit signless integer or 8-bit signed integer or 8-bit unsigned integer values of length 4/8/16/32 or vector of f8E4M3FN type or f8E5M2 type values of length 4/8/32/64 or vector of 4-bit signless integer or 4-bit signed integer or 4-bit unsigned integer values of length 4/8/16
sourceB - Single, WMMAInTypes, vector of 32-bit float values of length 2 or vector of 16-bit float or bfloat16 type values of length 4/8/16 or vector of 8-bit signless integer or 8-bit signed integer or 8-bit unsigned integer values of length 4/8/16/32 or vector of f8E4M3FN type or f8E5M2 type values of length 4/8/32/64 or vector of 4-bit signless integer or 4-bit signed integer or 4-bit unsigned integer values of length 4/8/16
destC - Single, WMMAOutTypes, vector of 32-bit float or 32-bit signless integer values of length 4/8 or vector of 16-bit float or bfloat16 type values of length 4/8/16

Results

destD - Single, WMMAOutTypes, vector of 32-bit float or 32-bit signless integer values of length 4/8 or vector of 16-bit float or bfloat16 type values of length 4/8/16

Description

The amdgpu.wmma op is an MLIR wrapper around intrinsics for various wmma instructions in the AMDGPU architecture, which perform matrix multiplication.

On gfx11/RDNA3, wmma intrinsics have M=N=K=16 dimensions.

On gfx12/RDNA4, wmma intrinsics have M=N=16 dimensions and support K=16 for all element types, and K=32 for i4 sources.

On gfx1250, wmma intrinsics have M=N=16 and K dimensions of 4, 32, 64, or 128, depending on the element types.

On gfx11/RDNA3, emitting f16->f16 (or bf16->bf16) wmma the output is a 16xf16 (or 16xbf16) vector containing only 8 valid values:

If subwordOffset is 0, then the output is stored at indices 0, 2, 4, ..., 14.
If subwordOffset is 1, then the output is stored at indices 1, 3, 5, ..., 15. On gfx12/RDNA4 and gfx1250, the result is instead returned as vector where all the values are valid and the subwordOffset must be 0, as it cannot be used.

unsignedA and unsignedB flag that the int8 LLVM inputs are unsigned.

The clamp flag is used to saturate the output of type T to numeric_limits<T>::max() in case of overflow.

The wave64attribute indicates whether an op is designed for 64 threads wavefont.

Example:

  %0 = amdgpu.wmma 16x16x16 %matA * %matB + %matC : vector<8xf16>, vector<8xf16>, vector<8xf16>

  %1 = amdgpu.wmma 16x16x64 %matD * %matE + %matF : vector<32xi8>, vector<8xf32>, vector<8xf32>

  %2 = amdgpu.wmma 16x16x128 %matG * %matH + %matI : vector<64xf4E2M1FN>, vector<64xf4E2M1FN>, vector<8xf32>

  %3 = amdgpu.wmma 16x16x4 %matJ * %matK + %matL : vector<2xf32>, vector<2xf32>, vector<8xf32>