Welcome to TT-NN documentation!

TTNN

• What is ttnn?
  • Key features of ttnn
• Getting Started
  • Quick Start Guide
    • 1. Install and Build
    • 2. Explore our model demos
    • 3. TT-NN Tutorial: Multi-Head Attention (Simple)
    • 4. TT-NN Tutorial: Multi-Head Attention (Optimized)
    • Where to go from here
• Using ttnn
  • Basic Examples
    • 1. Converting from and to torch tensor
    • 2. Running an operation on the device
    • 3. Using __getitem__ to slice the tensor
    • 4. Enabling program cache
    • 5. Debugging intermediate tensors
    • 6. Tracing the graph of operations
    • 7. Using tt_lib operation in ttnn
    • 8. Enabling Logging
    • 9. Supported Python Operators
    • 10. Changing the string representation of the tensor
    • 11. Visualize using Web Browser
    • 12. Register pre- and/or post-operation hooks
    • 13. Query all operations
    • 14. Disable Fallbacks
• Tensor
  • Shape
  • Layout
  • Data Type
  • Storage
  • Tensor Sharding
  • Memory Config
  • APIs
    • Shape
      • Shape.rank
      • Shape.with_tile_padding()
• APIs
  • Device
    • ttnn.open_device
      • open_device()
    • ttnn.close_device
      • close_device()
    • ttnn.manage_device
      • manage_device()
    • ttnn.synchronize_device
      • synchronize_device()
  • Memory Config
    • ttnn.create_sharded_memory_config
      • create_sharded_memory_config()
  • Operations
    • Core
      • ttnn.as_tensor
        • as_tensor()
      • ttnn.from_torch
        • from_torch()
      • ttnn.to_torch
        • to_torch()
      • ttnn.to_device
        • to_device()
      • ttnn.from_device
        • from_device()
      • ttnn.to_layout
        • to_layout()
      • ttnn.dump_tensor
        • dump_tensor()
      • ttnn.load_tensor
        • load_tensor()
      • ttnn.deallocate
        • deallocate()
      • ttnn.reallocate
        • reallocate()
      • ttnn.to_memory_config
        • to_memory_config()
    • Tensor Creation
      • ttnn.arange
        • arange()
      • ttnn.empty
        • empty()
      • ttnn.zeros
        • zeros()
      • ttnn.zeros_like
        • zeros_like()
      • ttnn.ones
        • ones()
      • ttnn.ones_like
        • ones_like()
      • ttnn.full
        • full()
      • ttnn.full_like
        • full_like()
    • Matrix Multiplication
      • ttnn.matmul
        • matmul()
      • ttnn.linear
        • linear()
    • Pointwise Unary
      • ttnn.abs
        • abs()
      • ttnn.acos
        • acos()
      • ttnn.acosh
        • acosh()
      • ttnn.asin
        • asin()
      • ttnn.asinh
        • asinh()
      • ttnn.atan
        • atan()
      • ttnn.atan2
        • atan2()
      • ttnn.atanh
        • atanh()
      • ttnn.cbrt
        • cbrt()
      • ttnn.celu
        • celu()
      • ttnn.clip
        • clip()
      • ttnn.clone
        • clone()
      • ttnn.cos
        • cos()
      • ttnn.cosh
        • cosh()
      • ttnn.deg2rad
        • deg2rad()
      • ttnn.digamma
        • digamma()
      • ttnn.elu
        • elu()
      • ttnn.erf
        • erf()
      • ttnn.erfc
        • erfc()
      • ttnn.erfinv
        • erfinv()
      • ttnn.exp
        • exp()
      • ttnn.exp2
        • exp2()
      • ttnn.expm1
        • expm1()
      • ttnn.geglu
        • geglu()
      • ttnn.gelu
        • gelu()
      • ttnn.glu
        • glu()
      • ttnn.hardshrink
        • hardshrink()
      • ttnn.hardsigmoid
        • hardsigmoid()
      • ttnn.hardswish
        • hardswish()
      • ttnn.hardtanh
        • hardtanh()
      • ttnn.heaviside
        • heaviside()
      • ttnn.hypot
        • hypot()
      • ttnn.i0
        • i0()
      • ttnn.isfinite
        • isfinite()
      • ttnn.isinf
        • isinf()
      • ttnn.isnan
        • isnan()
      • ttnn.isneginf
        • isneginf()
      • ttnn.isposinf
        • isposinf()
      • ttnn.leaky_relu
        • leaky_relu()
      • ttnn.lerp
        • lerp()
      • ttnn.lgamma
        • lgamma()
      • ttnn.log
        • log()
      • ttnn.log10
        • log10()
      • ttnn.log1p
        • log1p()
      • ttnn.log2
        • log2()
      • ttnn.log_sigmoid
        • log_sigmoid()
      • ttnn.logical_not
      • ttnn.logit
        • logit()
      • ttnn.mish
        • mish()
      • ttnn.multigammaln
        • multigammaln()
      • ttnn.neg
        • neg()
      • ttnn.prelu
        • prelu()
      • ttnn.reglu
        • reglu()
      • ttnn.relu
        • relu()
      • ttnn.relu6
        • relu6()
      • ttnn.rsqrt
        • rsqrt()
      • ttnn.sigmoid
        • sigmoid()
      • ttnn.sigmoid_accurate
        • sigmoid_accurate()
      • ttnn.sign
        • sign()
      • ttnn.silu
        • silu()
      • ttnn.sin
        • sin()
      • ttnn.sinh
        • sinh()
      • ttnn.softmax
        • softmax()
      • ttnn.softplus
        • softplus()
      • ttnn.softshrink
        • softshrink()
      • ttnn.softsign
        • softsign()
      • ttnn.swish
        • swish()
      • ttnn.tan
        • tan()
      • ttnn.tanh
        • tanh()
      • ttnn.signbit
        • signbit()
      • ttnn.polygamma
        • polygamma()
      • ttnn.rad2deg
        • rad2deg()
      • ttnn.reciprocal
        • reciprocal()
      • ttnn.sqrt
        • sqrt()
      • ttnn.square
        • square()
      • ttnn.swiglu
        • swiglu()
      • ttnn.tril
        • tril()
      • ttnn.triu
        • triu()
      • ttnn.tanhshrink
        • tanhshrink()
      • ttnn.threshold
        • threshold()
    • Pointwise Binary
      • ttnn.add
        • add()
      • ttnn.multiply
        • multiply()
      • ttnn.subtract
        • subtract()
      • ttnn.pow
        • pow()
      • ttnn.ldexp
        • ldexp()
      • ttnn.logical_and
        • logical_and()
      • ttnn.logical_or
        • logical_or()
      • ttnn.logical_xor
        • logical_xor()
      • ttnn.logaddexp
        • logaddexp()
      • ttnn.logaddexp2
        • logaddexp2()
      • ttnn.xlogy
        • xlogy()
      • ttnn.squared_difference
        • squared_difference()
      • ttnn.gtz
        • gtz()
      • ttnn.ltz
        • ltz()
      • ttnn.gez
        • gez()
      • ttnn.lez
        • lez()
      • ttnn.nez
        • nez()
      • ttnn.eqz
        • eqz()
      • ttnn.gt
        • gt()
      • ttnn.ge
        • ge()
      • ttnn.lt
        • lt()
      • ttnn.le
        • le()
      • ttnn.eq
        • eq()
      • ttnn.ne
        • ne()
      • ttnn.isclose
        • isclose()
      • ttnn.polyval
        • polyval()
      • ttnn.nextafter
        • nextafter()
      • ttnn.maximum
        • maximum()
      • ttnn.minimum
        • minimum()
    • Pointwise Ternary
      • ttnn.addcdiv
        • addcdiv()
      • ttnn.addcmul
        • addcmul()
      • ttnn.mac
        • mac()
      • ttnn.where
        • where()
    • Losses
      • ttnn.l1_loss
        • l1_loss()
      • ttnn.mse_loss
        • mse_loss()
    • Reduction
      • ttnn.max
        • max()
      • ttnn.mean
        • mean()
      • ttnn.min
        • min()
      • ttnn.std
        • std()
      • ttnn.sum
        • sum()
      • ttnn.var
        • var()
      • ttnn.argmax
        • argmax()
      • ttnn.topk
        • topk()
    • Data Movement
      • ttnn.concat
        • concat()
      • ttnn.pad
        • pad()
      • ttnn.permute
        • permute()
      • ttnn.reshape
        • reshape()
      • ttnn.repeat
        • repeat()
      • ttnn.repeat_interleave
        • repeat_interleave()
    • Normalization
      • ttnn.group_norm
        • group_norm()
      • ttnn.layer_norm
        • layer_norm()
      • ttnn.rms_norm
        • rms_norm()
    • Transformer
      • ttnn.transformer.split_query_key_value_and_split_heads
        • split_query_key_value_and_split_heads()
      • ttnn.transformer.concatenate_heads
        • concatenate_heads()
      • ttnn.transformer.attention_softmax
        • attention_softmax()
      • ttnn.transformer.attention_softmax_
        • attention_softmax_()
      • ttnn.transformer.rotary_embedding
    • Embedding
      • ttnn.embedding
        • embedding()
    • Pooling
      • ttnn.global_avg_pool2d
        • global_avg_pool2d()
      • ttnn.MaxPool2d
    • Vision
      • ttnn.upsample
        • upsample()
    • KV Cache
      • ttnn.kv_cache.fill_cache_for_user_
        • fill_cache_for_user_()
      • ttnn.kv_cache.update_cache_for_token_
        • update_cache_for_token_()
  • Model Conversion
    • ttnn.model_preprocessing.preprocess_model
      • preprocess_model()
    • ttnn.model_preprocessing.preprocess_model_parameters
      • preprocess_model_parameters()
  • Reports
    • ttnn.set_printoptions
      • set_printoptions()
  • Operation Hooks
    • ttnn.register_pre_operation_hook
      • register_pre_operation_hook()
    • ttnn.register_post_operation_hook
      • register_post_operation_hook()
• Tutorials
  • Tensor and Add Operation
    • Tensor and Add Operation
      • Creating a tensor
      • Host Storage: Borrowed vs Owned
      • Data Type
      • Layout
      • Device storage
      • Open the device
      • Initialize tensors a and b with random values using torch
      • Add tensor a and b
      • Inspect the output tensor of the add in ttnn
      • Convert to torch and inspect the attributes of the torch tensor
      • Close the device
  • Matmul Operation
    • Matrix Multiplication
      • Enable program cache
    • Configuration
      • Initialize tensors a and b with random values using torch
      • Matrix multiply tensor a and b
      • Inspect the layout of matrix multiplication output
      • Inspect the result of the matrix multiplication
      • Matrix multiply tensor a and b by using more performant config
      • Close the device
  • Multi-Head Attention
    • Multi-Head Attention
      • Enable program cache
      • Write Multi-Head Attention using ttnn
      • Configuration
      • Initialize activations and weights using torch
      • Convert activations and weights to ttnn
      • Run the first iteration of Multi-Head Attention
      • Run a subsequent iteration of Multi-Head Attention
      • Write optimized version of Multi-Head Attention
      • Pre-process the parameters of the optimized model
      • Run the first iteration of the optimized Multi-Head Attention
      • Run a subsequent iteration of the optimized Multi-Head Attention
      • Check that the output of the optimized version matches the output of the original implementation
      • Close the device
  • ttnn Tracer
    • Tracing ttnn operations and torch modules/functions
      • Trace torch functions
      • Trace torch functions and ttnn operations
      • Trace torch functions, torch modules and ttnn operations
      • Trace models written using ttnn
  • ttnn Profiling
    • Profiling ttnn operations
  • Resnet Basic Block
    • Resnet Block
      • Torch Module (from torchvision)
      • Create torch module and preprocess it to get ttnn parameters
      • Display the parameters of the module
      • Display the traced torch graph
      • Implement ttnn version of the module. Pass in the parameters into the constructor.
      • Run ttnn module and display the traced graph
  • Graphing Torch DiT_XL_2 With TTNN
    • Build a graph of a pytorch based model
      • Clone the library from https://github.com/facebookresearch/DiT.git
      • Download DiT-XL/2 Models
      • Sample from Pre-trained DiT Models and build the graph
      • Display the graph
• Onboarding New Functionality
• Converting torch Model to ttnn
  • Step 1 - Rewriting the Model
  • Step 2 - Switching to ttnn Operations
  • Step 3 - Optimizing the Model
  • More examples
• Adding New ttnn Operation
  • C++ Implementation
  • Add pybindings
  • Adding a unit test
  • Adding a sweep test
• Profiling ttnn Operations
  • Perf Report Headers
  • profile_this description
• Dependencies
  • TT-LIB
    • Overview
      • Operation Infrastructure
        • New Device Operation
        • New Device Operation with a member
        • New Device Operation with Optional Input Tensors
        • New Device Operation with Optional Output Tensors
        • New Host Operation
      • Profiler
      • Fast Dispatch
      • Program Caching
      • Logs
    • TT-LIB API through tt_lib
      • Primary Operations
        • moreh_softmax()
        • moreh_softmax_backward()
        • moreh_softmin()
        • moreh_softmin_backward()
        • moreh_logsoftmax()
        • moreh_logsoftmax_backward()
        • moreh_groupnorm()
        • moreh_groupnorm_backward()
        • moreh_norm()
        • moreh_norm_backward()
      • Enums
      • Tensor elementwise operations
      • Tensor relational operations
      • Tensor ternary operations
      • Tensor matrix math operations
      • Tensor creation operations
      • Broadcast and Reduce
    • Fallback Operations
      • full()
      • tensor_slice()
      • reshape()
      • chunk()
      • conv2d()
      • group_norm()
      • layer_norm()
      • pad()
      • interpolate()
      • repeat()
      • repeat_interleave()
      • concat()
      • silu()
      • softmax()
      • Conv2d
      • BatchNorm2d
      • GroupNorm
      • LayerNorm
      • MaxPool2d
      • AdaptiveAvgPool2d
      • ceil
      • floor
      • trunc
      • unary_fmod
      • binary_fmod
      • bitwise_not
      • unary_bitwise_or
      • unary_bitwise_and
      • unary_bitwise_xor
      • binary_bitwise_or
      • binary_bitwise_and
      • binary_bitwise_xor
      • unary_bitwise_left_shift
      • unary_bitwise_right_shift
      • binary_bitwise_left_shift
      • binary_bitwise_right_shift
      • torch_argmax
      • torch_argmin
    • Experimental Operations
      • Fused Operations from tt_lib Mini-Graph Library
        • Linear()
        • softmax()
        • Layernorm()
        • AddAndNorm()
      • Complex Operations
      • Complex Operations (Type 2)
      • Other Operations
      • Backward Operations
      • Loss Functions
  • Tensor
    • Overview
    • Tensor Storage
    • Tensor API
    • MemoryConfig
    • Examples of converting between PyTorch Tensor and TT Tensor
      • Converting a PyTorch Tensor to a TT Tensor
      • Converting a TT Tensor to a PyTorch Tensor
  • Examples of Tensor and TT-LIB Use
    • Run one OP from TT-LIB on TT Accelerator device
    • Run TT-LIB and PyTorch OPs
    • Tensors with odd size of last dim
• Building and Uplifting Demos
• Placeholder title

Models

• Getting Started
  • Prerequisites
  • Running an example model
  • Next steps
• Performance
  • Prerequisites
  • Running a perf file
  • Running all the perf files

Resources

• Support
  • Reporting bugs, feature proposals, or support requests
  • Troubleshooting and debugging tips
  • Community
• Contributing as a developer

Indices and tables

• Index

• Module Index

• Search Page