TT-BUDA Installation

Overview

The TT-BUDA software stack can compile models from several different frameworks and execute them in many different ways on Tenstorrent hardware.

This user guide is intended to help you setup your system with the appropriate device drivers, firmware, system dependencies, and compiler / runtime software.

Note on terminology:

While TT-BUDA is the official Tenstorrent AI/ML compiler stack, PyBUDA is the Python interface for TT-BUDA. TT-BUDA is the core technology; however, PyBUDA allows users to access and utilize TT-BUDA’s features directly from Python. This includes directly importing model architectures and weights from PyTorch, TensorFlow, ONNX, and TFLite.

Prerequisites

OS Compatibility

Presently, Tenstorrent software is only supported on the Ubuntu 20.04 LTS (Focal Fossa) operating system.

Download

To access the PyBUDA software and associated files, please navigate to https://github.com/tenstorrent/tt-buda.

To install a release, please go to https://github.com/tenstorrent/tt-buda/releases. Once you have identified the release version you would like to install, you can download the individual files by clicking on their name.

Install

PyBUDA releases can be installed using two methods: Python virtualenv or Docker. PyBUDA can be installed from source following the Python virtualenv method.

Python Environment Installation

It is strongly recommended to use virtual environments for each project utilizing PyBUDA and Python dependencies. Creating a new virtual environment with PyBUDA and libraries is very easy.

Prerequisites (detailed sections below) for python envirnment installation are listed here:

• Setup HugePages (below)

• PCI Driver Installation (below)

• Device Firmware Update (below)

• Backend Compiler Dependencies (below)

• Additional PyBUDA Compile Dependencies (below)

• Tenstorrent Software Package (below).

To install a PyBUDA release, follow these steps:

• Step 1. Navigate to https://github.com/tenstorrent/tt-buda/releases and download the latest release pybuda and tvm wheel files

• Step 2. Create your Python environment in desired directory

python3 -m venv env

• Step 3. Activate environment

source env/bin/activate

• Step 4. Pip install PyBuda and TVM

If you have downloaded the latest release wheel files, you can install them directly with pip.

pip install pybuda-<version>.whl tvm-<version>.whl

To compile PyBUDA from source, follow these steps:

• Step 1. Clone PyBUDA from https://github.com/tenstorrent/tt-buda/

• Step 2. Update submodules

cd tt-buda
git submodule update --init --recursive

• Step 3. Compile. PyBUDA’s make system will automatically create the needed venv

make
source  build/python_env/bin/activate

Docker Container Installation

Alternatively, PyBUDA and its dependencies are provided as Docker images which can run in separate containers.

Prerequisites (detailed sections below) for docker installation are listed here:

• PCI Driver Installation (below)

• Device Firmware Update (below)

• Step 1. Setup a personal access token (classic)

Create a personal access token from: https://github.com/settings/tokens. Give the token the permissions to read packages from the container registry read:packages.

• Step 2. Login to Docker Registry

GITHUB_TOKEN=<your token>
echo $GITHUB_TOKEN | sudo docker login ghcr.io -u <your github username> --password-stdin

• Step 3. Pull the image

sudo docker pull ghcr.io/tenstorrent/tt-buda/<TAG>

• Step 4. Run the container

sudo docker run --rm -ti --shm-size=4g --device /dev/tenstorrent -v /dev/hugepages-1G:/dev/hugepages-1G -v `pwd`/:/home/ ghcr.io/tenstorrent/tt-buda/<TAG> bash

• Step 5. Change root directory

cd home/

Installation Prerequisites

Setup HugePages

NUM_DEVICES=$(lspci -d 1e52: | wc -l)
sudo sed -i "s/^GRUB_CMDLINE_LINUX_DEFAULT=.*$/GRUB_CMDLINE_LINUX_DEFAULT=\"hugepagesz=1G hugepages=${NUM_DEVICES} nr_hugepages=${NUM_DEVICES} iommu=pt\"/g" /etc/default/grub
sudo update-grub
sudo sed -i "/\s\/dev\/hugepages-1G\s/d" /etc/fstab; echo "hugetlbfs /dev/hugepages-1G hugetlbfs pagesize=1G,rw,mode=777 0 0" | sudo tee -a /etc/fstab
sudo reboot

PCI Driver Installation

Please navigate to https://github.com/tenstorrent/tt-kmd and follow the readme to install the kernel mode PCI driver.

Device Firmware Update

Please navigate to https://github.com/tenstorrent/tt-flash and https://github.com/tenstorrent/tt-firmware-gs to download a utility for flashing device firmwares and the firmware itself. Follow respective readmes for setup and installation.

Backend Compiler Dependencies

Instructions to install the Tenstorrent backend compiler dependencies on a fresh install of Ubuntu Server.

You may need to append each apt-get command with sudo if you do not have root permissions.

apt-get update -y && apt-get upgrade -y --no-install-recommends
apt-get install -y software-properties-common
apt-get install -y python3.8-venv libboost-all-dev libgoogle-glog-dev libgl1-mesa-glx ruby
apt-get install -y build-essential clang-6.0 libhdf5-serial-dev libzmq3-dev

Additional PyBUDA Compile Dependencies

OS Level Dependencies

Additional dependencies to compile PyBUDA from source after running Backend Compiler Dependencies

You may need to append each apt-get command with sudo if you do not have root permissions.

apt-get install -y libyaml-cpp-dev python3-pip sudo git git-lfs
apt-get install -y wget cmake cmake-data
pip3 install pyyaml

Package Level Dependencies

In addition, if you intend to utilize torchvision for your model development, we strongly recommend installing it using a specific method that ensures optimal compatibility with PyBUDA. This method involves building and installing torchvision from its source code, which allows for the necessary dependencies and configurations to be correctly set up.

To do this, you should use the following commands:

export TORCH_VISION_INSTALL=1
make torchvision

The export TORCH_VISION_INSTALL=1 command sets an environment variable that our Makefile script uses to determine whether to install torchvision. By setting this variable to 1, you’re instructing the script to proceed with the torchvision installation.

The make torchvision command then triggers the build and installation process. This process includes cloning the torchvision repository, checking out the desired version, and building torchvision using its setup.py script.

By following these steps, torchvision will be installed in a way that ensures it works effectively with PyBUDA.

NOTE

The TORCH_VISION_INSTALL flag is not limited to the make torchvision command. It can also be used with the standard make build command. When this flag is set to 1, the build process will include the installation of torchvision, regardless of the specific make command used. This allows for flexibility in your build process, enabling you to include or exclude the torchvision installation as needed.

NOTE

For your convenience, the torchvision wheel file is already included in the PyBUDA release bundle. This means that if you’re using the release bundle, you won’t need to build torchvision from source unless you want to use a different version or need to modify the source code. Simply install the provided wheel file using pip to add torchvision to your Python environment.

Here’s an example of how you can install the torchvision wheel file:

pip install /path/to/your/wheel/file/torchvision*.whl

Replace /path/to/your/wheel/file/torchvision*.whl with the actual path to the torchvision wheel file in the PyBUDA release bundle.

NOTE

To run the existing unit tests of PyBUDA components, e.g. after compiling it from source, you need to install the following packages.

apt-get install -y wget libgtest-dev libgmock-dev

TT-SMI

Please navigate to https://github.com/tenstorrent/tt-smi to get Tenstorrent’s System Management Interface tool. A command line utility to interact with all Tenstorrent devices on host.

Tenstorrent Software Package

Acquire pybuda and associated software from the aforementioned Download section.

Relevant files:

pybuda-<version>.whl   <- Latest PyBUDA Release
tvm-<version>.whl      <- Latest TVM Release

Smoke Test

With your Python environment with PyBUDA install activated, run the following Python script:

import pybuda
import torch


# Sample PyTorch module
class PyTorchTestModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.weights1 = torch.nn.Parameter(torch.rand(32, 32), requires_grad=True)
        self.weights2 = torch.nn.Parameter(torch.rand(32, 32), requires_grad=True)
    def forward(self, act1, act2):
        m1 = torch.matmul(act1, self.weights1)
        m2 = torch.matmul(act2, self.weights2)
        return m1 + m2, m1


def test_module_direct_pytorch():
    input1 = torch.rand(4, 32, 32)
    input2 = torch.rand(4, 32, 32)
    # Run single inference pass on a PyTorch module, using a wrapper to convert to PyBUDA first
    output = pybuda.PyTorchModule("direct_pt", PyTorchTestModule()).run(input1, input2)
    print(output)


if __name__ == "__main__":
    test_module_direct_pytorch()