mirror of
https://github.com/codeflash-ai/codeflash.git
synced 2026-05-04 18:25:17 +00:00
107 lines
3.8 KiB
Text
107 lines
3.8 KiB
Text
---
|
|
title: "Optimize Performance Benchmarks with every Pull Request"
|
|
description: "Configure and use benchmark integration for performance-critical code optimization (Python only)"
|
|
icon: "chart-line"
|
|
sidebarTitle: "Benchmarks (Python)"
|
|
keywords:
|
|
[
|
|
"benchmarks",
|
|
"CI",
|
|
"pytest-benchmark",
|
|
"performance testing",
|
|
"github actions",
|
|
"benchmark mode",
|
|
]
|
|
---
|
|
|
|
<Info>
|
|
**Performance-critical optimization** - Define benchmarks for your most
|
|
important code sections and let Codeflash optimize and measure the real-world
|
|
impact of every optimization on your performance metrics.
|
|
</Info>
|
|
|
|
Benchmark mode is an easy way to define workflows that are performance-critical and need to be optimized and run fast.
|
|
Codeflash will run the benchmark, understand how the current code change in the Pull Request is affecting the benchmark.
|
|
It will then try to optimize the new code for the benchmark and calculate the impact of any optimization on the speed of that benchmark.
|
|
|
|
## Using Codeflash in Benchmark Mode
|
|
|
|
<Note>
|
|
Benchmark mode currently supports Python projects using pytest-benchmark. JavaScript/TypeScript benchmark support is coming soon.
|
|
</Note>
|
|
|
|
1. **Create a benchmarks root:**
|
|
|
|
Create a directory for benchmarks if it does not already exist.
|
|
|
|
In your pyproject.toml, add the path to the 'benchmarks-root' section.
|
|
|
|
```toml
|
|
[tool.codeflash]
|
|
# All paths are relative to this pyproject.toml's directory.
|
|
module-root = "inference"
|
|
tests-root = "tests"
|
|
benchmarks-root = "tests/benchmarks" # add your benchmarks root dir here
|
|
ignore-paths = []
|
|
formatter-cmds = ["disabled"]
|
|
```
|
|
|
|
2. **Define your benchmarks:**
|
|
|
|
Codeflash supports benchmarks written as pytest-benchmarks. Check out the [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/stable/index.html) documentation for more information on syntax.
|
|
|
|
For example:
|
|
|
|
```python
|
|
from core.bubble_sort import sorter
|
|
|
|
def test_sort(benchmark):
|
|
result = benchmark(sorter, list(reversed(range(500))))
|
|
assert result == list(range(500))
|
|
```
|
|
|
|
Note that these benchmarks should be defined in such a way that they don't take a long time to run.
|
|
|
|
The pytest-benchmark format is simply used as an interface. The plugin is actually not used - Codeflash will run these benchmarks with its own pytest plugin.
|
|
|
|
3. **Run and Test Codeflash:**
|
|
|
|
Run Codeflash with the `--benchmark` flag. Note that benchmark mode cannot be used with `--all`.
|
|
|
|
```bash
|
|
codeflash --file test_file.py --benchmark
|
|
```
|
|
|
|
If you did not define your benchmarks-root in your pyproject.toml, you can do:
|
|
|
|
```bash
|
|
codeflash --file test_file.py --benchmark --benchmarks-root path/to/benchmarks
|
|
```
|
|
|
|
4. **Run Codeflash with GitHub Actions:**
|
|
|
|
Benchmark mode is best used together with Codeflash as a GitHub Action. This way,
|
|
Codeflash will trace through your benchmark and optimize the functions modified in your Pull Request to speed up the benchmark.
|
|
It will also report the impact of Codeflash's optimizations on your benchmarks.
|
|
|
|
Use `codeflash init` for an easy way to set up Codeflash as a GitHub Action.
|
|
|
|
After that, you can add the `--benchmark` argument to codeflash to enable benchmarks optimization.
|
|
|
|
```bash
|
|
codeflash --benchmark
|
|
```
|
|
|
|
## How it works
|
|
|
|
1. Codeflash identifies benchmarks in the benchmarks-root directory.
|
|
|
|
2. The benchmarks are run so that runtime statistics and inputs can be recorded.
|
|
|
|
3. Replay tests are generated so the performance of optimization candidates on the exact inputs used in the benchmarks can be measured.
|
|
|
|
4. If an optimization candidate is verified to be correct, the speedup of the optimization is calculated for each benchmark.
|
|
|
|
5. Codeflash then reports the impact of the optimization on each benchmark.
|
|
|
|
Using Codeflash with benchmarks is a great way to find optimizations that really matter.
|