Running Performance Benchmark Tests
You can use forge/test/benchmark/benchmark.py
to run performance benchmark tests:
python forge/test/benchmark/benchmark.py [options]
Available Options:
| Option | Short | Type | Default | Description |
|--------|-------|------|---------|-------------|
| --model
| -m
| string | required | Model to benchmark (e.g. bert, mnist_linear). The test file name without .py extension |
| --config
| -c
| string | None | Model configuration to benchmark (e.g. tiny, base, large) |
| --training
| -t
| flag | False | Benchmark training mode |
| --batch_size
| -bs
| integer | 1 | Batch size, number of samples to process at once |
| --loop_count
| -lp
| integer | 1 | Number of times to run the benchmark |
| --input_size
| -isz
| integer | None | Input size of the input sample (if model supports variable input size) |
| --hidden_size
| -hs
| integer | None | Hidden layer size (if model supports variable hidden size) |
| --output
| -o
| string | None | Output JSON file to write results to. Results will be appended if file exists |
| --task
| -ts
| string | "na" | Task to benchmark (e.g. classification, segmentation) |
| --data_format
| -df
| string | "float32" | Data format (e.g. float32, bfloat16) |
Example:
python forge/test/benchmark/benchmark.py -m mobilenetv2_basic -ts classification -bs 8 -df bfloat16 -lp 32 -o forge-fe-benchmark-e2e-mobilenetv2_basic.json
Alternatively, you can run specific model tests using pytest
:
pytest [model_path]
Example:
pytest -svv forge/test/benchmark/benchmark/models/yolo_v8.py