ArchAttr
TT Arch
Syntax:
#tt.arch<
::mlir::tt::Arch # value
>
Enum cases:
- grayskull (
Grayskull
) - wormhole_b0 (
WormholeB0
) - blackhole (
Blackhole
)
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::Arch | an enum of type Arch |
ArgumentAllocationAttr
Argument allocation attribute in TT dialect
Syntax:
#tt.arg_alloc<
uint64_t, # address
uint64_t, # size
MemorySpace # memorySpace
>
Holds the metadata for the allocation of an function argument i.e. for graph inputs.
Parameters:
Parameter | C++ type | Description |
---|---|---|
address | uint64_t | |
size | uint64_t | |
memorySpace | MemorySpace |
ArgumentTypeAttr
Argument Type
Syntax:
#tt.argument_type<
::mlir::tt::ArgumentType # value
>
Enum cases:
- input (
Input
) - parameter (
Parameter
) - constant (
Constant
)
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::ArgumentType | an enum of type ArgumentType |
CPUDescAttr
TT cpu_desc attribute
Syntax:
#tt.cpu_desc<
CPURole, # role
StringAttr # target_triple
>
TT cpu_desc attribute
Parameters:
Parameter | C++ type | Description |
---|---|---|
role | CPURole | |
target_triple | StringAttr |
CPURoleAttr
TT CPU Role
Syntax:
#tt.cpu_role<
::mlir::tt::CPURole # value
>
Enum cases:
- host (
Host
) - device (
Device
)
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::CPURole | an enum of type CPURole |
ChipChannelAttr
TT chip_channel attribute
Syntax:
#tt.chip_channel<
unsigned, # deviceId0
::llvm::ArrayRef<int64_t>, # ethernetCoreCoord0
unsigned, # deviceId1
::llvm::ArrayRef<int64_t> # ethernetCoreCoord1
>
TT chip_channel attribute
Parameters:
Parameter | C++ type | Description |
---|---|---|
deviceId0 | unsigned | |
ethernetCoreCoord0 | ::llvm::ArrayRef<int64_t> | |
deviceId1 | unsigned | |
ethernetCoreCoord1 | ::llvm::ArrayRef<int64_t> |
ChipCoordAttr
TT chip_coord attribute
Syntax:
#tt.chip_coord<
unsigned, # rack
unsigned, # shelf
unsigned, # y
unsigned # x
>
TT chip_coord attribute
Parameters:
Parameter | C++ type | Description |
---|---|---|
rack | unsigned | |
shelf | unsigned | |
y | unsigned | |
x | unsigned |
ChipDescAttr
TT chip_desc attribute
Syntax:
#tt.chip_desc<
ArchAttr, # arch
::llvm::ArrayRef<int64_t>, # grid
unsigned, # l1Size
unsigned, # numDramChannels
unsigned, # dramChannelSize
unsigned, # nocL1AddressAlignBytes
unsigned, # pcieAddressAlignBytes
unsigned, # nocDRAMAddressAlignBytes
unsigned, # l1UnreservedBase
unsigned, # eriscL1UnreservedBase
unsigned, # dramUnreservedBase
unsigned, # dramUnreservedEnd
ChipPhysicalCoresAttr, # chipPhysicalCores
::llvm::ArrayRef<DataTypeAttr>, # supportedDataTypes
::llvm::ArrayRef<TileSizeAttr>, # supportedTileSizes
unsigned, # numCBs
unsigned, # numComputeThreads
unsigned # numDatamovementThreads
>
TT chip_desc attribute
Parameters:
Parameter | C++ type | Description |
---|---|---|
arch | ArchAttr | |
grid | ::llvm::ArrayRef<int64_t> | |
l1Size | unsigned | |
numDramChannels | unsigned | |
dramChannelSize | unsigned | |
nocL1AddressAlignBytes | unsigned | |
pcieAddressAlignBytes | unsigned | |
nocDRAMAddressAlignBytes | unsigned | |
l1UnreservedBase | unsigned | |
eriscL1UnreservedBase | unsigned | |
dramUnreservedBase | unsigned | |
dramUnreservedEnd | unsigned | |
chipPhysicalCores | ChipPhysicalCoresAttr | |
supportedDataTypes | ::llvm::ArrayRef<DataTypeAttr> | |
supportedTileSizes | ::llvm::ArrayRef<TileSizeAttr> | |
numCBs | unsigned | |
numComputeThreads | unsigned | |
numDatamovementThreads | unsigned |
ChipPhysicalCoresAttr
TT chip_physical_cores attribute
Syntax:
#tt.chip_physical_cores<
::llvm::ArrayRef<CoreCoordAttr>, # worker
::llvm::ArrayRef<CoreCoordAttr>, # dram
::llvm::ArrayRef<CoreCoordAttr>, # eth
::llvm::ArrayRef<CoreCoordAttr> # eth_inactive
>
TT chip_physical_cores attribute containing arrays of physical cores by core type in order of logical cores.
Parameters:
Parameter | C++ type | Description |
---|---|---|
worker | ::llvm::ArrayRef<CoreCoordAttr> | |
dram | ::llvm::ArrayRef<CoreCoordAttr> | |
eth | ::llvm::ArrayRef<CoreCoordAttr> | |
eth_inactive | ::llvm::ArrayRef<CoreCoordAttr> |
CoreCoordAttr
TT core_coord attribute
Syntax:
#tt.core_coord<
int64_t, # y
int64_t # x
>
TT core_coord attribute containing a single physical core coordinate.
Parameters:
Parameter | C++ type | Description |
---|---|---|
y | int64_t | |
x | int64_t |
DataTypeAttr
TT DataTypes
Syntax:
#tt.supportedDataTypes<
::mlir::tt::DataType # value
>
Enum cases:
- f32 (
Float32
) - f16 (
Float16
) - bf16 (
BFloat16
) - bfp_f8 (
BFP_Float8
) - bfp_bf8 (
BFP_BFloat8
) - bfp_f4 (
BFP_Float4
) - bfp_bf4 (
BFP_BFloat4
) - bfp_f2 (
BFP_Float2
) - bfp_bf2 (
BFP_BFloat2
) - u32 (
UInt32
) - u16 (
UInt16
) - u8 (
UInt8
) - si32 (
Int32
)
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::DataType | an enum of type DataType |
DeviceAttr
Device attribute in TT dialect.
Syntax:
#tt.device<
::mlir::tt::GridAttr, # workerGrid
AffineMap, # l1Map
AffineMap, # dramMap
::llvm::ArrayRef<int64_t>, # meshShape
::llvm::ArrayRef<unsigned> # chipIds
>
Describes the physical layout of a device in the system and is made up of a few components:
- A grid attribute that describes the device's compute grid shape. It not only describes the shape of the compute grid, but also carries an affine map that describes how the logical grid maps to the physical grid.
- Two affine maps that describe how a tensor layout's linear attribute maps to the L1 and DRAM memory spaces.
- A mesh shape that describes the virtual layout of the chips with respect to each other. Note that in a multi-chip system, this grid encapsulates the entire system's grid shape, e.g. 8x16 grid could be made up of a 1x2 mesh of chips side-by-side. The mesh attribute configures how the above grid/map attributes are created such that they implement this mesh topology.
- An array of chip ids that this device is made up of. This array's length must match the volume of the mesh shape and should be interpreted in row-major order.
Parameters:
Parameter | C++ type | Description |
---|---|---|
workerGrid | ::mlir::tt::GridAttr | TT grid attribute |
l1Map | AffineMap | |
dramMap | AffineMap | |
meshShape | ::llvm::ArrayRef<int64_t> | |
chipIds | ::llvm::ArrayRef<unsigned> |
GridAttr
TT grid attribute
Syntax:
#tt.grid<
::llvm::ArrayRef<int64_t>, # shape
AffineMap # mapping
>
TT grid attribute
Parameters:
Parameter | C++ type | Description |
---|---|---|
shape | ::llvm::ArrayRef<int64_t> | |
mapping | AffineMap |
IteratorTypeAttr
TT IteratorType
Syntax:
#tt.iterator_type<
::mlir::tt::IteratorType # value
>
Enum cases:
- parallel (
Parallel
) - reduction (
Reduction
)
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::IteratorType | an enum of type IteratorType |
MemorySpaceAttr
TT MemorySpace
Syntax:
#tt.memory_space<
::mlir::tt::MemorySpace # value
>
Enum cases:
- system (
System
) - mmio (
SystemMMIO
) - dram (
DeviceDRAM
) - l1 (
DeviceL1
)
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::MemorySpace | an enum of type MemorySpace |
MeshAttr
Mesh reference attribute in TT dialect.
Syntax:
#tt.mesh<
StringAttr, # name
::llvm::ArrayRef<int64_t> # shape
>
Describes a mesh config including name and shape.
Parameters:
Parameter | C++ type | Description |
---|---|---|
name | StringAttr | |
shape | ::llvm::ArrayRef<int64_t> |
MeshShardDirectionAttr
TT MeshShardDirection
Syntax:
#tt.shard_direction<
::mlir::tt::MeshShardDirection # value
>
Enum cases:
- full_to_shard (
FullToShard
) - shard_to_full (
ShardToFull
)
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::MeshShardDirection | an enum of type MeshShardDirection |
MeshShardTypeAttr
MeshShard shard_type attribute in TT dialect
Syntax:
#tt.shard_type<
::mlir::tt::MeshShardType # value
>
Define sharded tensor data of mesh_shard op.
- Identity: input and output tensors are pre-sharded (same data) and no sharding is required.
- Replicate: all of the devices has full tensor (same data).
- Maximal: one or part of the devcices has full tensor (same data).
- Devices: all or part of the devices has sharded (partial) tensor (different data).
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::MeshShardType | an enum of type MeshShardType |
MeshesAttr
TT system meshes attribute.
Syntax:
#tt.meshes<
::llvm::ArrayRef<MeshAttr> # meshes
>
TT system meshes attribute includes one or more mesh configs used for networks.
Parameters:
Parameter | C++ type | Description |
---|---|---|
meshes | ::llvm::ArrayRef<MeshAttr> |
MetalLayoutAttr
Tensor layout attribute
Syntax:
#tt.metal_layout<
AffineMap, # linear
OOBVal, # oob_val
GridAttr, # grid
MemRefType # memref
>
The tensor layout attribute captures how tensor data is sharded across a grid of devices, cores, and is laid out in memory.
Some high level goals
- Logical shapes: Keep the original tensor shape and rank intact and agnostic to underlying storage layout. Keeping the logical shapes not only makes some graph transformations vastly simpler, in particular convs, but it makes the lowered IR much easier to read and reason about. The original tensor shapes leave breadcrumbs that make it much easier to map back to the input representation.
- Flexible sharding: Enable flexibility in choosing grid shape, to get better parallelization and avoid resharding. This is particularly important in cases where tensor shapes are not clean powers of two and would otherwise force our hand in choosing non-optimal grid shapes.
- Logical-Physical Isomorphism: Encode this information with just a few attributes to enable derived conversions from logical to physical layout and back.
- Explicit: A single source of truth.
- Enable a direct way to query padded regions.
Please refer to the Tensor Layout Spec for more in depth documentation.
Examples:
tensor<8x300xf32,
#tt.metal_layout<(d0, d1) -> (d0, d1),
undef,
<1x2>,
memref<8x150xf32, #tt.memory_space<l1>>
>
>
tensor<8x96x32xf32,
#tt.metal_layout<(d0, d1, d2) -> (d0 * 96 + d1, d2),
undef,
<2x1>,
memref<384x32xf32, #tt.memory_space<l1>>
>
>
tensor<8x96x32xf32,
#tt.metal_layout<(d0, d1, d2) -> (d0 * 96 + d1, d1, d2),
undef,
<2x1x2>,
memref<384x96x16xf32, #tt.memory_space<l1>>
>
>
tensor<5x3x2x2x7x32x32xf32,
#tt.metal_layout<
(d0, d1, d2, d3, d4, d5, d6)
-> (d0 * 2688 + d1 * 896 + d2 * 448 + d3 * 224 + d4 * 32 + d5, d4, d5, d6),
undef,
<3x2x2x2>,
memref<4480x4x16x16xf32, #tt.memory_space<l1>>
>
>
Parameters:
Parameter | C++ type | Description |
---|---|---|
linear | AffineMap | An affine map that defines how the logical tensor dimensions map to a grid shape. |
oob_val | OOBVal | A tracked out of bounds value that fills padding space. |
grid | GridAttr | The grid shape that this tensor is divided onto. |
memref | MemRefType | A memref that describes the physical footprint allocation of the shard. It must also have a shape with rank equal to grid. |
OOBValAttr
TT OOBVal
Syntax:
#tt.oob_val<
::mlir::tt::OOBVal # value
>
Enum cases:
- undef (
Undef
) - zero (
Zero
) - one (
One
) - inf (
Inf
) - neginf (
NegInf
)
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::OOBVal | an enum of type OOBVal |
ReduceTypeAttr
TT Reduce Type
Syntax:
#tt.reduce_type<
::mlir::tt::ReduceType # value
>
Enum cases:
- sum (
Sum
) - mean (
Mean
) - max (
Max
) - min (
Min
) - std (
Std
) - var (
Var
)
Parameters:
Parameter | C++ type | Description |
---|---|---|
value | ::mlir::tt::ReduceType | an enum of type ReduceType |
ShardLayoutAttr
Shard layout attribute in TT dialect
Syntax:
#tt.shard<
::llvm::ArrayRef<int64_t>, # stride
uint32_t # buffers
>
Describes shard layout of a memref buffer.
- Stride: Stride of each dim in bytes.
- Buffers: Number of back buffers used for double buffering, I/O latency hiding, etc
The shard layout attribute is a description of how each shard of a memref is laid out in memory. Memref's with this layout type implicitly mean their data is distributed across a grid of cores.
Parameters:
Parameter | C++ type | Description |
---|---|---|
stride | ::llvm::ArrayRef<int64_t> | |
buffers | uint32_t |
StreamLayoutAttr
Stream layout attribute in TT dialect
Syntax:
#tt.stream<
AffineMap # affineMap
>
Describes a stream layout of a memref buffer.
- AffineMap: Provides affine map indexing into the associated data stream.
Only the stream_layout op should return memref's with this attribute. The stream layout attribute is necessary for two reasons:
- It provides a way to reblock the data stream into a different shape (via affine map). Usually this would be some subblock of the original backing memory to chunk the data into smaller pieces.
- The type itself is a signal to datamovement passes that the memref is a stream and should be treated as such.
Parameters:
Parameter | C++ type | Description |
---|---|---|
affineMap | AffineMap |
SystemDescAttr
TT system_desc attribute
Syntax:
#tt.system_desc<
::llvm::ArrayRef<CPUDescAttr>, # cpuDescs
::llvm::ArrayRef<ChipDescAttr>, # chipDescs
::llvm::ArrayRef<unsigned>, # chipDescIndices
::llvm::ArrayRef<ChipCapabilityAttr>, # chipCapabilities
::llvm::ArrayRef<ChipCoordAttr>, # chipCoords
::llvm::ArrayRef<ChipChannelAttr> # chipChannels
>
TT system_desc attribute
Parameters:
Parameter | C++ type | Description |
---|---|---|
cpuDescs | ::llvm::ArrayRef<CPUDescAttr> | |
chipDescs | ::llvm::ArrayRef<ChipDescAttr> | |
chipDescIndices | ::llvm::ArrayRef<unsigned> | |
chipCapabilities | ::llvm::ArrayRef<ChipCapabilityAttr> | |
chipCoords | ::llvm::ArrayRef<ChipCoordAttr> | |
chipChannels | ::llvm::ArrayRef<ChipChannelAttr> |
TensorMeshShardingAttr
Tensor mesh sharding attribute in TT dialect.
Syntax:
#tt.mesh_sharding<
StringAttr, # name
::llvm::ArrayRef<TensorMeshShardingAxisAttr> # tensor_mesh_sharding_axis
>
Describes a tensor's multi-device status.
-
Single device tensor has no TensorMeshShardingAttr. tensor<784x16384xf32>
-
Multi-device tensors have TensorMeshShardingAttr. (i) multi-device tensor without tensor mesh shard axis indicates all devices in "mesh" have full size tensors e.g., 784x16384 for tensor<784x16384xf32, #tt.mesh_sharding<"mesh">>
(ii) multi-device tensor with tensor mesh shard axis indicate all devices in "mesh" have sharded tensor defined by the TensorMeshShardingAxisAttr. e.g., 192x16384 for tensor<784x16384xf32, #tt.mesh_sharding<"mesh" : [ 4(1), 1]>>. Here, 4(1) indicates shard_shape(shard_dim), so 784 should be sharded by 4 at "mesh"'s second hardware dimension. 1 indicates no sharding, so 16384 is not being sharded.
Parameters:
Parameter | C++ type | Description |
---|---|---|
name | StringAttr | |
tensor_mesh_sharding_axis | ::llvm::ArrayRef<TensorMeshShardingAxisAttr> |
TensorMeshShardingAxisAttr
Tensor mesh sharding axis info attribute in TT dialect.
Syntax:
#tt.tensor_sharding<
int64_t, # shard_shape
::llvm::ArrayRef<int64_t> # axes
>
Details per tensor dimension sharding and axes info.
- shard_shape: shard shape at a tensor dimension.
- (optional) axes: mesh shard dimensions. Axes may be empty if it is not being sharded.
Parameters:
Parameter | C++ type | Description |
---|---|---|
shard_shape | int64_t | |
axes | ::llvm::ArrayRef<int64_t> |
TileSizeAttr
TT tile_size attribute
Syntax:
#tt.tile_size<
int64_t, # y
int64_t # x
>
TT tile_size attribute containing a supported Tensix tile shape.
Parameters:
Parameter | C++ type | Description |
---|---|---|
y | int64_t | |
x | int64_t |