Welcome to TT-NN documentation!

TTNN

• What is ttnn?
  • Key features of ttnn
• Getting Started
  • 1. Install and Build
  • 2. Explore our model demos
  • 3. TT-NN Tutorial: Multi-Head Attention (Simple)
  • 4. TT-NN Tutorial: Multi-Head Attention (Optimized)
  • Where to go from here
• Using ttnn
  • Basic Examples
    • 1. Converting from and to torch tensor
    • 2. Running an operation on the device
    • 3. Using __getitem__ to slice the tensor
    • 4. Enabling program cache
    • 5. Debugging intermediate tensors
    • 6. Tracing the graph of operations
    • 7. Using tt_lib operation in ttnn
    • 8. Enabling Logging
    • 9. Supported Python Operators
    • 10. Changing the string representation of the tensor
    • 11. Visualize using Web Browser
    • 12. Register pre- and/or post-operation hooks
    • 13. Query all operations
    • 14. Disable Fallbacks
• Tensor
  • Shape
  • Layout
  • Data Type
  • Storage
  • Tensor Sharding
  • Memory Config
  • APIs
    • Shape
      • Shape.rank
      • Shape.with_tile_padding()
• APIs
  • Device
    • ttnn.open_device
      • open_device()
    • ttnn.close_device
      • close_device()
    • ttnn.manage_device
      • manage_device()
    • ttnn.synchronize_device
      • synchronize_device()
  • Memory Config
    • ttnn.create_sharded_memory_config
      • create_sharded_memory_config()
  • Operations
    • Core
      • ttnn.as_tensor
        • as_tensor()
      • ttnn.from_torch
        • from_torch()
      • ttnn.to_torch
        • to_torch()
      • ttnn.to_device
        • to_device()
      • ttnn.from_device
        • from_device()
      • ttnn.to_layout
        • to_layout()
      • ttnn.dump_tensor
        • dump_tensor()
      • ttnn.load_tensor
        • load_tensor()
      • ttnn.deallocate
        • deallocate()
      • ttnn.reallocate
        • reallocate()
      • ttnn.to_memory_config
        • to_memory_config()
    • Tensor Creation
      • ttnn.arange
        • arange()
      • ttnn.empty
        • empty()
      • ttnn.zeros
        • zeros()
      • ttnn.zeros_like
        • zeros_like()
      • ttnn.ones
        • ones()
      • ttnn.ones_like
        • ones_like()
      • ttnn.full
        • full()
      • ttnn.full_like
        • full_like()
    • Matrix Multiplication
      • ttnn.matmul
        • matmul()
      • ttnn.linear
        • linear()
    • Pointwise Unary
      • ttnn.abs
        • abs()
      • ttnn.acos
        • acos()
      • ttnn.acosh
        • acosh()
      • ttnn.asin
        • asin()
      • ttnn.asinh
        • asinh()
      • ttnn.atan
        • atan()
      • ttnn.atan2
        • atan2()
      • ttnn.atanh
        • atanh()
      • ttnn.cbrt
        • cbrt()
      • ttnn.celu
        • celu()
      • ttnn.clip
        • clip()
      • ttnn.clone
        • clone()
      • ttnn.cos
        • cos()
      • ttnn.cosh
        • cosh()
      • ttnn.deg2rad
        • deg2rad()
      • ttnn.digamma
        • digamma()
      • ttnn.elu
        • elu()
      • ttnn.erf
        • erf()
      • ttnn.erfc
        • erfc()
      • ttnn.erfinv
        • erfinv()
      • ttnn.exp
        • exp()
      • ttnn.exp2
        • exp2()
      • ttnn.expm1
        • expm1()
      • ttnn.geglu
        • geglu()
      • ttnn.gelu
        • gelu()
      • ttnn.glu
        • glu()
      • ttnn.hardshrink
        • hardshrink()
      • ttnn.hardsigmoid
        • hardsigmoid()
      • ttnn.hardswish
        • hardswish()
      • ttnn.hardtanh
        • hardtanh()
      • ttnn.heaviside
        • heaviside()
      • ttnn.hypot
        • hypot()
      • ttnn.i0
        • i0()
      • ttnn.isfinite
        • isfinite()
      • ttnn.isinf
        • isinf()
      • ttnn.isnan
        • isnan()
      • ttnn.isneginf
        • isneginf()
      • ttnn.isposinf
        • isposinf()
      • ttnn.leaky_relu
        • leaky_relu()
      • ttnn.lerp
        • lerp()
      • ttnn.lgamma
        • lgamma()
      • ttnn.log
        • log()
      • ttnn.log10
        • log10()
      • ttnn.log1p
        • log1p()
      • ttnn.log2
        • log2()
      • ttnn.log_sigmoid
        • log_sigmoid()
      • ttnn.logical_not
      • ttnn.logit
        • logit()
      • ttnn.mish
        • mish()
      • ttnn.multigammaln
        • multigammaln()
      • ttnn.neg
        • neg()
      • ttnn.prelu
        • prelu()
      • ttnn.reglu
        • reglu()
      • ttnn.relu
        • relu()
      • ttnn.relu6
        • relu6()
      • ttnn.rsqrt
        • rsqrt()
      • ttnn.sigmoid
        • sigmoid()
      • ttnn.sigmoid_accurate
        • sigmoid_accurate()
      • ttnn.sign
        • sign()
      • ttnn.silu
        • silu()
      • ttnn.sin
        • sin()
      • ttnn.sinh
        • sinh()
      • ttnn.softmax
        • softmax()
      • ttnn.softplus
        • softplus()
      • ttnn.softshrink
        • softshrink()
      • ttnn.softsign
        • softsign()
      • ttnn.swish
        • swish()
      • ttnn.tan
        • tan()
      • ttnn.tanh
        • tanh()
      • ttnn.signbit
        • signbit()
      • ttnn.polygamma
        • polygamma()
      • ttnn.rad2deg
        • rad2deg()
      • ttnn.reciprocal
        • reciprocal()
      • ttnn.sqrt
        • sqrt()
      • ttnn.square
        • square()
      • ttnn.swiglu
        • swiglu()
      • ttnn.tril
        • tril()
      • ttnn.triu
        • triu()
      • ttnn.tanhshrink
        • tanhshrink()
      • ttnn.threshold
        • threshold()
    • Pointwise Binary
      • ttnn.add
        • add()
      • ttnn.multiply
        • multiply()
      • ttnn.subtract
        • subtract()
      • ttnn.pow
        • pow()
      • ttnn.ldexp
        • ldexp()
      • ttnn.logical_and
        • logical_and()
      • ttnn.logical_or
        • logical_or()
      • ttnn.logical_xor
        • logical_xor()
      • ttnn.logaddexp
        • logaddexp()
      • ttnn.logaddexp2
        • logaddexp2()
      • ttnn.xlogy
        • xlogy()
      • ttnn.squared_difference
        • squared_difference()
      • ttnn.gtz
        • gtz()
      • ttnn.ltz
        • ltz()
      • ttnn.gez
        • gez()
      • ttnn.lez
        • lez()
      • ttnn.nez
        • nez()
      • ttnn.eqz
        • eqz()
      • ttnn.gt
        • gt()
      • ttnn.ge
        • ge()
      • ttnn.lt
        • lt()
      • ttnn.le
        • le()
      • ttnn.eq
        • eq()
      • ttnn.ne
        • ne()
      • ttnn.isclose
        • isclose()
      • ttnn.polyval
        • polyval()
      • ttnn.nextafter
        • nextafter()
      • ttnn.maximum
        • maximum()
      • ttnn.minimum
        • minimum()
    • Pointwise Ternary
      • ttnn.addcdiv
        • addcdiv()
      • ttnn.addcmul
        • addcmul()
      • ttnn.mac
        • mac()
      • ttnn.where
        • where()
    • Losses
      • ttnn.l1_loss
        • l1_loss()
      • ttnn.mse_loss
        • mse_loss()
    • Reduction
      • ttnn.max
        • max()
      • ttnn.mean
        • mean()
      • ttnn.min
        • min()
      • ttnn.std
        • std()
      • ttnn.sum
        • sum()
      • ttnn.var
        • var()
      • ttnn.argmax
        • argmax()
      • ttnn.topk
    • Data Movement
      • ttnn.concat
        • concat()
      • ttnn.pad
        • pad()
      • ttnn.permute
        • permute()
      • ttnn.reshape
        • reshape()
      • ttnn.repeat
        • repeat()
      • ttnn.repeat_interleave
        • repeat_interleave()
    • Normalization
      • ttnn.group_norm
        • group_norm()
      • ttnn.layer_norm
        • layer_norm()
      • ttnn.rms_norm
        • rms_norm()
    • Transformer
      • ttnn.transformer.split_query_key_value_and_split_heads
        • split_query_key_value_and_split_heads()
      • ttnn.transformer.concatenate_heads
        • concatenate_heads()
      • ttnn.transformer.attention_softmax
        • attention_softmax()
      • ttnn.transformer.attention_softmax_
        • attention_softmax_()
      • ttnn.transformer.rotary_embedding
        • rotary_embedding()
    • Embedding
      • ttnn.embedding
        • embedding()
    • Pooling
      • ttnn.global_avg_pool2d
        • global_avg_pool2d()
      • ttnn.MaxPool2d
        • MaxPool2d
    • Vision
      • ttnn.upsample
        • upsample()
    • KV Cache
      • ttnn.kv_cache.fill_cache_for_user_
        • fill_cache_for_user_()
      • ttnn.kv_cache.update_cache_for_token_
        • update_cache_for_token_()
  • Model Conversion
    • ttnn.model_preprocessing.preprocess_model
      • preprocess_model()
    • ttnn.model_preprocessing.preprocess_model_parameters
      • preprocess_model_parameters()
  • Reports
    • ttnn.set_printoptions
      • set_printoptions()
  • Operation Hooks
    • ttnn.register_pre_operation_hook
      • register_pre_operation_hook()
    • ttnn.register_post_operation_hook
      • register_post_operation_hook()
• Tutorials
  • Tensor and Add Operation
    • Tensor and Add Operation
      • Creating a tensor
      • Host Storage: Borrowed vs Owned
      • Data Type
      • Layout
      • Device storage
      • Open the device
      • Initialize tensors a and b with random values using torch
      • Add tensor a and b
      • Inspect the output tensor of the add in ttnn
      • Convert to torch and inspect the attributes of the torch tensor
      • Close the device
  • Matmul Operation
    • Matrix Multiplication
      • Enable program cache
    • Configuration
      • Initialize tensors a and b with random values using torch
      • Matrix multiply tensor a and b
      • Inspect the layout of matrix multiplication output
      • Inspect the result of the matrix multiplication
      • Matrix multiply tensor a and b by using more performant config
      • Close the device
  • Multi-Head Attention
    • Multi-Head Attention
      • Enable program cache
      • Write Multi-Head Attention using ttnn
      • Configuration
      • Initialize activations and weights using torch
      • Convert activations and weights to ttnn
      • Run the first iteration of Multi-Head Attention
      • Run a subsequent iteration of Multi-Head Attention
      • Write optimized version of Multi-Head Attention
      • Pre-process the parameters of the optimized model
      • Run the first iteration of the optimized Multi-Head Attention
      • Run a subsequent iteration of the optimized Multi-Head Attention
      • Check that the output of the optimized version matches the output of the original implementation
      • Close the device
  • ttnn Tracer
    • Tracing ttnn operations and torch modules/functions
      • Trace torch functions
      • Trace torch functions and ttnn operations
      • Trace torch functions, torch modules and ttnn operations
      • Trace models written using ttnn
  • ttnn Profiling
    • Profiling ttnn operations
  • Resnet Basic Block
    • Resnet Block
      • Torch Module (from torchvision)
      • Create torch module and preprocess it to get ttnn parameters
      • Display the parameters of the module
      • Display the traced torch graph
      • Implement ttnn version of the module. Pass in the parameters into the constructor.
      • Run ttnn module and display the traced graph
  • Graphing Torch DiT_XL_2 With TTNN
    • Build a graph of a pytorch based model
      • Clone the library from https://github.com/facebookresearch/DiT.git
      • Download DiT-XL/2 Models
      • Sample from Pre-trained DiT Models and build the graph
      • Display the graph
• Onboarding New Functionality
• Converting torch Model to ttnn
  • Step 1 - Rewriting the Model
  • Step 2 - Switching to ttnn Operations
  • Step 3 - Optimizing the Model
  • More examples
• Adding New ttnn Operation
  • C++ Implementation
  • Add pybindings
  • Adding a unit test
  • Adding a sweep test
• Profiling ttnn Operations
  • Perf Report Headers
  • profile_this description
• Dependencies
  • TT-LIB
    • Overview
      • Operation Infrastructure
        • New Device Operation
        • New Device Operation with a member
        • New Device Operation with Optional Input Tensors
        • New Device Operation with Optional Output Tensors
        • New Host Operation
      • Profiler
      • Fast Dispatch
        • cpu()
        • Synchronize()
      • Program Caching
      • Logs
    • TT-LIB API through tt_lib
      • Primary Operations
        • layernorm()
        • add_layernorm()
        • softmax_in_place()
        • moreh_softmax()
        • moreh_softmax_backward()
        • moreh_softmin()
        • moreh_softmin_backward()
        • moreh_logsoftmax()
        • moreh_logsoftmax_backward()
        • scale_mask_softmax_in_place()
        • moreh_mean()
        • moreh_mean_backward()
        • moreh_groupnorm()
        • moreh_groupnorm_backward()
        • moreh_norm()
        • moreh_norm_backward()
      • Enums
        • BcastOpMath
        • BcastOpDim
        • ReduceOpMath
        • ReduceOpDim
      • Tensor elementwise operations
        • div()
        • div_no_nan()
        • add_unary()
        • sub_unary()
        • mul_unary()
        • div_unary()
        • gelu()
        • relu()
        • relu6()
        • relu_min()
        • relu_max()
        • exp()
        • recip()
        • sqrt()
        • log()
        • log2()
        • log10()
        • log1p()
        • tanh()
        • clip()
        • hardtanh()
        • deg2rad()
        • rad2deg()
        • cbrt()
        • hypot()
        • softplus()
        • mish()
        • polyval()
        • sign()
        • abs()
        • silu()
        • square()
        • neg()
        • add1()
        • mac()
        • sigmoid()
        • sigmoid_accurate()
        • hardsigmoid()
        • swish()
        • hardswish()
        • leaky_relu()
        • softsign()
        • softshrink()
        • hardshrink()
        • cos()
        • sin()
        • cosh()
        • sinh()
        • acos()
        • asin()
        • elu()
        • exp2()
        • tanhshrink()
        • heaviside()
        • right_shift()
        • left_shift()
        • atan()
        • atanh()
        • atan2()
        • logical_xor()
        • logical_xori()
        • logical_not_unary()
        • subalpha()
        • celu()
        • addalpha()
        • bias_gelu_unary()
        • isfinite()
        • isinf()
        • isposinf()
        • isneginf()
        • isnan()
        • logit()
        • lgamma()
        • logical_andi()
        • erfinv()
        • multigammaln()
        • assign()
        • isclose()
        • i0()
        • digamma()
        • tan()
        • logical_ori()
        • polygamma()
        • floor()
        • trunc()
        • round()
        • floor_div()
      • Tensor relational operations
        • gtz()
        • gez()
        • ltz()
        • lez()
        • eqz()
        • nez()
        • unary_ne()
        • unary_gt()
        • unary_lt()
      • Tensor ternary operations
        • where()
        • threshold()
      • Tensor matrix math operations
        • matmul()
        • bmm()
        • reshape()
        • transpose()
        • permute()
        • tilize()
        • untilize()
        • tilize_with_val_padding()
        • untilize_with_unpadding()
        • tilize_with_zero_padding()
        • pad()
        • unpad()
        • clone()
        • typecast()
        • copy()
      • Tensor creation operations
        • arange()
        • full()
        • ones()
        • ones_like()
        • zeros()
        • zeros_like()
        • full_like()
        • split_last_dim_two_chunks_tiled()
        • empty()
        • tril()
        • triu()
      • Broadcast and Reduce
        • bcast()
        • reduce()
        • global_min()
        • global_max()
        • global_sum()
        • global_mean()
        • rpow()
        • rsub()
        • rdiv()
    • Fallback Operations
      • full()
      • tensor_slice()
      • reshape()
      • chunk()
      • conv2d()
      • group_norm()
      • layer_norm()
      • pad()
      • interpolate()
      • repeat()
      • repeat_interleave()
      • concat()
      • silu()
      • softmax()
      • Conv2d
      • BatchNorm2d
      • GroupNorm
      • LayerNorm
      • MaxPool2d
      • AdaptiveAvgPool2d
      • ceil
      • floor
      • trunc
      • unary_fmod
      • binary_fmod
      • bitwise_not
      • unary_bitwise_or
      • unary_bitwise_and
      • unary_bitwise_xor
      • binary_bitwise_or
      • binary_bitwise_and
      • binary_bitwise_xor
      • unary_bitwise_left_shift
      • unary_bitwise_right_shift
      • binary_bitwise_left_shift
      • binary_bitwise_right_shift
      • torch_argmax
      • torch_argmin
    • Experimental Operations
      • Fused Operations from tt_lib Mini-Graph Library
        • Linear()
        • softmax()
        • Layernorm()
        • AddAndNorm()
      • Complex Operations
        • complex_add()
        • complex_sub()
        • complex_mul()
        • complex_div()
        • real()
        • imag()
        • complex_abs()
        • conj()
        • complex_recip()
        • polar()
      • Complex Operations (Type 2)
      • Other Operations
        • concat()
        • sum()
        • log_sigmoid()
        • expm1()
        • asinh()
        • acosh()
        • erf()
        • erfc()
        • rsqrt()
        • lerp()
        • signbit()
        • fill_rm()
        • fill_ones_rm()
        • conv()
        • layernorm()
        • groupnorm()
        • rmsnorm()
        • add_layernorm()
        • convert_conv_weight_tensor_to_tiled_layout()
        • xlogy()
        • prod()
        • tiled_prod()
        • addcmul()
        • addcdiv()
        • mean_hw()
        • var_hw()
        • logical_noti()
        • std_hw()
        • normalize_hw()
        • normalize_global()
        • glu()
        • geglu()
        • reglu()
        • swiglu()
        • embeddings()
        • nextafter()
        • lamb_optimizer()
        • repeat()
        • repeat_interleave()
        • pow()
        • identity()
        • argmax()
        • argmin()
      • Backward Operations
        • prod_bw()
        • addalpha_bw()
        • addcmul_bw()
        • addcdiv_bw()
        • conj_bw()
        • unary_mul_bw()
        • unary_add_bw()
        • unary_assign_bw()
        • binary_assign_bw()
        • unary_div_bw()
        • div_bw()
        • rdiv_bw()
        • sqrt_bw()
        • mul_bw()
        • max_bw()
        • min_bw()
        • add_bw()
        • tan_bw()
        • exp_bw()
        • exp2_bw()
        • expm1_bw()
        • unary_pow_bw()
        • embedding_bw()
        • where_bw()
        • tanh_bw()
        • fill_zero_bw()
        • fill_bw()
        • sub_bw()
        • unary_sub_bw()
        • log_bw()
        • rsub_bw()
        • abs_bw()
        • complex_abs_bw()
        • rsqrt_bw()
        • neg_bw()
        • lt_bw()
        • gt_bw()
        • relu_bw()
        • ne_bw()
        • clamp_bw()
        • clamp_min_bw()
        • clamp_max_bw()
        • binary_le_bw()
        • atan2_bw()
        • hypot_bw()
        • gelu_bw()
        • bias_gelu_bw()
        • bias_gelu_unary_bw()
        • squared_difference_bw()
        • lerp_bw()
        • ldexp_bw()
        • xlogy_bw()
        • logaddexp_bw()
        • logaddexp2_bw()
        • concat_bw()
        • hardsigmoid_bw()
        • i0_bw()
        • hardshrink_bw()
        • softshrink_bw()
        • hardswish_bw()
        • softplus_bw()
        • polygamma_bw()
        • atan_bw()
        • atanh_bw()
        • asin_bw()
        • asinh_bw()
        • cosh_bw()
        • cos_bw()
        • acosh_bw()
        • acos_bw()
        • erfinv_bw()
        • leaky_relu_bw()
        • elu_bw()
        • hardtanh_bw()
        • angle_bw()
        • sin_bw()
        • sinh_bw()
        • celu_bw()
        • binary_lt_bw()
        • subalpha_bw()
        • log10_bw()
        • log1p_bw()
        • binary_ne_bw()
        • erf_bw()
        • erfc_bw()
        • digamma_bw()
        • deg2rad_bw()
        • rad2deg_bw()
        • reciprocal_bw()
        • relu6_bw()
        • rpow_bw()
        • silu_bw()
        • selu_bw()
        • binary_ge_bw()
        • binary_eq_bw()
        • binary_gt_bw()
        • square_bw()
        • lgamma_bw()
        • trunc_bw()
        • frac_bw()
        • log_sigmoid_bw()
        • tanhshrink_bw()
        • threshold_bw()
        • unary_eq_bw()
        • logit_bw()
        • logiteps_bw()
        • softsign_bw()
        • sign_bw()
        • ceil_bw()
        • log2_bw()
        • ge_bw()
        • le_bw()
        • unary_fmod_bw()
        • unary_remainder_bw()
        • complex_recip_bw()
        • imag_bw()
        • real_bw()
        • complex_mul_bw()
        • complex_div_bw()
        • polar_bw()
        • complex_add_bw()
        • complex_sub_bw()
        • multigammaln_bw()
        • repeat_bw()
        • floor_bw()
        • round_bw()
        • unary_div_no_nan_bw()
      • Loss Functions
        • mseloss()
        • maeloss()
  • Tensor
    • Overview
    • Tensor Storage
    • Tensor API
      • Tensor
        • Tensor.__init__()
        • Tensor.buffer()
        • Tensor.device()
        • Tensor.get_dtype()
        • Tensor.get_layout()
        • Tensor.get_legacy_shape()
        • Tensor.pad()
        • Tensor.pad_to_tile()
        • Tensor.storage_type()
        • Tensor.to()
        • Tensor.unpad()
        • Tensor.unpad_from_tile()
    • MemoryConfig
      • MemoryConfig
        • MemoryConfig.__init__()
    • Examples of converting between PyTorch Tensor and TT Tensor
      • Converting a PyTorch Tensor to a TT Tensor
      • Converting a TT Tensor to a PyTorch Tensor
  • Examples of Tensor and TT-LIB Use
    • Run one OP from TT-LIB on TT Accelerator device
    • Run TT-LIB and PyTorch OPs
    • Tensors with odd size of last dim
• Building and Uplifting Demos
• Placeholder title

Models

• Getting Started
  • Prerequisites
  • Running an example model
  • Next steps
• Performance
  • Prerequisites
  • Running a perf file
  • Running all the perf files

Resources

• Support
  • Reporting bugs, feature proposals, or support requests
  • Troubleshooting and debugging tips
  • Community
• Contributing as a developer

Indices and tables

• Index

• Module Index

• Search Page