Testing

This page describes how to run different kinds of tests in the tt-forge-onnx project. If you haven’t built the project yet, please refer to the Build page.

Unit tests

To build the unit tests, run the following command:

cmake --build build -- build_unit_tests

To run the unit tests (this will also build the tests if they are not built):

cmake --build build -- run_unit_tests

Note: The unit tests are built in the build/forge/csrc/test directory. From there, you can run targeted tests directly.

  • For example, to run all the tests defined in forge/csrc/test/passes/ use: ./build/forge/csrc/test/test_passes

  • You can further filter the tests by using the --gtest_filter flag:

    ./build/forge/csrc/test/test_passes --gtest_filter=MMFuseBias/MMFuseBias.mm_fuse_bias/3

End to end tests

For running the end-to-end tests we use the pytest framework. To run these tests, you need to be on a machine with a Tenstorrent Wormhole device. Also, we are still in the process of cleaning up the old tests, so not all tests are working. For a list of green tests, consult pytest.ini.

Note: Make sure that you have activated the python environment before running the tests.

To run all tests defined in /test/mlir/test_ops.py use:

pytest -svv forge/test/mlir/test_ops.py

To run a specific test, use the following:

pytest -svv forge/test/mlir/test_ops.py::test_add

  • The -svv flag is optional and used to display more information about the test run.

Single operator E2E tests

Single operator E2E tests consists of pre configured collections of in-depth tests for each operator according to test plan. Tests include small models consisting of single operator with or without combination with few other operators. More details about test plan available on Test template page

To start interacting with test sweeps framework load helper commands via

source forge/test/operators/pytorch/test_commands.sh

Available commands

Command

Description

print_help

Print commands and current query parameters.

print_query_docs

Print docs for all available query parameters.

print_params

Print current query parameters values.

select_test_query

Select test_query pytest function.

select_test_push

Select test_push pytest function.

pytest

Run all tests or subset of test plan based on a query parameters.

with-params pytest

Print params before and after test run.

export_tests

Export tests from test plan to JSON file based on a query parameters.

Full list of supported query parameters

Parameter

Description

Supported by commands

OPERATORS

List of operators

test_query

FILTERS

List of lambda filters

test_query

INPUT_SOURCES

List of input sources

test_query

INPUT_SHAPES

List of input shapes

test_query

DEV_DATA_FORMATS

List of dev data formats

test_query

MATH_FIDELITIES

List of math fidelities

test_query

KWARGS

List of kwargs dictionaries.

test_query

FAILING_REASONS

List of failing reasons

test_query

SKIP_REASONS

List of skip reasons

test_query

RANGE

Limit number of results

test_query

RANDOM_SEED

Seed for random number generator

test_query

SAMPLE

Percentage of results to sample

test_query

TEST_ID

Id of a single test to run containing all test parameters

test_query

ID_FILES

Paths to files containing test ids instead of tests from test plan

test_query

ID_FILES_IGNORE

Paths to files containing test ids to be ignored

test_query

Test configuration parameters

Parameter

Description

Supported by commands

SKIP_FORGE_VERIFICATION

Skip Forge model verification including model compiling and inference

all

To check supported values and options for each query parameter please run command print_query_docs.

Usage examples

Run all tests

with-params pytest

Run all tests for few operators

export OPERATORS=add,div
with-params pytest

Run subset of tests based on query criteria

export OPERATORS=div
export FILTERS=HAS_DATA_FORMAT,QUICK
export INPUT_SOURCES=FROM_HOST,FROM_DRAM_QUEUE
export DEV_DATA_FORMATS=Float16_b,Int8
export MATH_FIDELITIES=HiFi4,HiFi3
export KWARGS="[{'rounding_mode': 'trunc'},{'rounding_mode': 'floor'}]"
with-params pytest

Print representative tests ids of all operators with examples for kwargs values

FILTERS=UNIQUE_KWARGS with-params pytest --collect-only

Print representative tests ids of few operators

OPERATORS=add,div FILTERS=UNIQUE_KWARGS with-params pytest --collect-only

Each test can be uniquely identified via a test id. Format of test id is {operator}-{input_source}-{kwargs}-{input_shape}[-{number_of_operands)-]{dev_data_format}-{math_fidelity}.

Kwarg is a mandatory or optional attribute of an operator. See framework (PyTorch, Forge, …) operator documentation for each operator or use filter UNIQUE_KWARGS to find examples.

Run single test based on a test id. Test id may be from a test plan or constructed custom by specifying custom values for kwargs and input_shapes.

TEST_ID='ge-FROM_HOST-None-(1, 2, 3, 4)-Float16_b-HiFi4' with-params pytest