codeflash-admin/codeflash
1
0
Fork 0
mirror of https://github.com/codeflash-ai/codeflash.git synced 2026-05-04 18:25:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Revamp and polish the docs more

Browse source 
Signed-off-by: Saurabh Misra <misra.saurabh1@gmail.com>
This commit is contained in:
Saurabh Misra 2025-08-19 22:00:00 -07:00
parent fbb4bbefa5
commit 678b55aa90
13 changed files with 166 additions and 180 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/codeflash-concepts/how-codeflash-works.mdx
View file

@ -57,7 +57,7 @@ Codeflash runs tests for the target function using either pytest or unittest fra
#### Performance benchmarking #### Performance benchmarking
Codeflash implements [several techniques](/codeflash-concepts/benchmarking.md) to measure code performance accurately. In particular, it runs multiple iterations of the code in a loop to determine the best performance with the minimum runtime. Codeflash compares the performance of the original code against the optimization, requiring at least a 10% speed improvement before considering it to be faster. This approach eliminates most runtime measurement variability, even on noisy CI systems and virtual machines. The final runtime Codeflash reports is the minimum total time it took to run all the test cases. Codeflash implements [several techniques](/codeflash-concepts/benchmarking) to measure code performance accurately. In particular, it runs multiple iterations of the code in a loop to determine the best performance with the minimum runtime. Codeflash compares the performance of the original code against the optimization, requiring at least a 10% speed improvement before considering it to be faster. This approach eliminates most runtime measurement variability, even on noisy CI systems and virtual machines. The final runtime Codeflash reports is the minimum total time it took to run all the test cases.
## Creating Pull Requests ## Creating Pull Requests

47
docs/docs.json
View file

@ -25,27 +25,26 @@
] ]
}, },
{ {
"group": "🚀 Getting Started", "group": "🚀 Quickstart",
"pages": [ "pages": [
"getting-started/local-installation" "getting-started/local-installation"
] ]
}, },
{ {
"group": "🔧 Continuous Optimization", "group": "⚡ Optimizing with Codeflash",
"pages": [
"getting-started/codeflash-github-actions",
"optimizing-with-codeflash/optimize-prs",
"optimizing-with-codeflash/benchmarking"
]
},
{
"group": "⚡ Optimization Workflows",
"pages": [ "pages": [
"optimizing-with-codeflash/one-function", "optimizing-with-codeflash/one-function",
"optimizing-with-codeflash/trace-and-optimize", "optimizing-with-codeflash/trace-and-optimize",
"optimizing-with-codeflash/codeflash-all" "optimizing-with-codeflash/codeflash-all"
] ]
}, },
{
"group": "✨ Continuous Optimization",
"pages": [
"optimizing-with-codeflash/codeflash-github-actions",
"optimizing-with-codeflash/benchmarking"
]
},
{ {
"group": "🧠 Core Concepts", "group": "🧠 Core Concepts",
"pages": [ "pages": [
@ -62,21 +61,7 @@
} }
] ]
} }
], ]
"global": {
"anchors": [
{
"anchor": "GitHub",
"href": "https://github.com/codeflash-ai",
"icon": "github"
},
{
"anchor": "Discord",
"href": "https://www.codeflash.ai/discord",
"icon": "discord"
}
]
}
}, },
"logo": { "logo": {
"light": "/images/codeflash_light.svg", "light": "/images/codeflash_light.svg",
@ -85,8 +70,14 @@
"navbar": { "navbar": {
"links": [ "links": [
{ {
"label": "Contact", "label": "Discord",
"href": "mailto:contact@codeflash.ai" "href": "https://www.codeflash.ai/discord",
"icon": "discord"
},
{
"label": "GitHub",
"href": "https://github.com/codeflash-ai/codeflash",
"icon": "github"
}, },
{ {
"label": "Blog", "label": "Blog",
@ -113,7 +104,7 @@
"footer": { "footer": {
"socials": { "socials": {
"discord": "https://www.codeflash.ai/discord", "discord": "https://www.codeflash.ai/discord",
"github": "https://github.com/codeflash-ai", "github": "https://github.com/codeflash-ai/codeflash",
"linkedin": "https://www.linkedin.com/company/codeflash-ai" "linkedin": "https://www.linkedin.com/company/codeflash-ai"
}, },
"links": [ "links": [

93
docs/getting-started/local-installation.mdx
View file

@ -6,8 +6,6 @@ icon: "download"
Codeflash is installed and configured on a per-project basis. Codeflash is installed and configured on a per-project basis.
You can install Codeflash locally for a project by running the following command in the project's virtual environment:
### Prerequisites ### Prerequisites
Before installing Codeflash, ensure you have: Before installing Codeflash, ensure you have:
@ -15,7 +13,6 @@ Before installing Codeflash, ensure you have:
1. **Python 3.9 or above** installed 1. **Python 3.9 or above** installed
2. **A Python project** with a virtual environment 2. **A Python project** with a virtual environment
3. **Project dependencies installed** in your virtual environment 3. **Project dependencies installed** in your virtual environment
4. **Tests** (optional) for your code (Codeflash uses tests to verify optimizations)
<Warning> <Warning>
**Virtual Environment Required** **Virtual Environment Required**
@ -23,14 +20,15 @@ Before installing Codeflash, ensure you have:
Always install Codeflash in your project's virtual environment, not globally. Make sure your virtual environment is activated before proceeding. Always install Codeflash in your project's virtual environment, not globally. Make sure your virtual environment is activated before proceeding.
```bash ```bash
# Example: Activate your virtual environment
source venv/bin/activate # On Linux/Mac source venv/bin/activate # On Linux/Mac
# or # or
#venv\Scripts\activate # On Windows venv\Scripts\activate # On Windows
``` ```
</Warning> </Warning>
<Steps> <Steps>
<Step title="Install Codeflash"> <Step title="Install Codeflash">
You can install Codeflash locally for a project by running the following command in the project's virtual environment:
```bash ```bash
pip install codeflash pip install codeflash
``` ```
@ -39,37 +37,32 @@ pip install codeflash
**Codeflash is a Development Dependency** **Codeflash is a Development Dependency**
We recommend installing Codeflash as a development dependency. We recommend installing Codeflash as a development dependency.
It doesn't need to be installed as part of your package requirements. Codeflash is intended to be used in development workflows locally and as part of CI.
Codeflash is intended to be used locally and as part of development workflows such as CI. Try to always use the latest version of Codeflash as it improves quickly.
<CodeGroup> <CodeGroup>
```toml Poetry
[tool.poetry.dependencies.dev]
codeflash = "^latest"
```
```bash uv ```bash uv
uv add --dev codeflash uv add --dev codeflash
``` ```
```bash pip ```bash poetry
pip install --dev codeflash poetry add codeflash@latest --group dev
``` ```
</CodeGroup> </CodeGroup>
</Tip> </Tip>
</Step> </Step>
<Step title="Generate a Codeflash API Key"> <Step title="Generate a Codeflash API Key">
Codeflash uses cloud-hosted AI models to optimize your code. You'll need an API key to use it. Codeflash uses cloud-hosted AI models and integrations with GitHub. You'll need an API key to authorize your accesss.
1. Visit the [Codeflash Web App](https://app.codeflash.ai/) 1. Visit the [Codeflash Web App](https://app.codeflash.ai/)
2. Sign up with your GitHub account (free) 2. Sign up with your GitHub account (free)
3. Navigate to the [API Key](https://app.codeflash.ai/app/apikeys) page to generate your API key 3. Navigate to the [API Key](https://app.codeflash.ai/app/apikeys) page to generate your API key
<Note> <Note>
**Free Tier Available** **Free Tier Available**
Codeflash offers a **free tier** with a limited number of optimizations per month. Perfect for trying it out or small projects! Codeflash offers a **free tier** with a limited number of optimizations. Perfect for trying it out on small projects!
</Note> </Note>
</Step> </Step>
@ -77,10 +70,6 @@ Codeflash offers a **free tier** with a limited number of optimizations per mont
Navigate to your project's root directory (where your `pyproject.toml` file is or should be) and run: Navigate to your project's root directory (where your `pyproject.toml` file is or should be) and run:
```bash ```bash
# Make sure you're in your project root
cd /path/to/your/project
# Run the initialization
codeflash init codeflash init
``` ```
@ -89,30 +78,31 @@ If you don't have a pyproject.toml file yet, the codeflash init command will ask
<Info> <Info>
**What's pyproject.toml?** **What's pyproject.toml?**
`pyproject.toml` is a configuration file that is used to specify build tool settings for Python projects. `pyproject.toml` is a configuration file that is used to specify build and tool settings for Python projects.
pyproject.toml is the modern replacement for setup.py and requirements.txt files. `pyproject.toml` is the modern replacement for setup.py and requirements.txt files.
It's the new standard for Python package metadata.
</Info> </Info>
When running `codeflash init`, you will see the following prompts: When running `codeflash init`, you will see the following prompts:
```text ```text
1. Enter your Codeflash API key: 1. Enter your Codeflash API key:
2. Which Python module do you want me to optimize going forward? (e.g. my_module) 2. Install the GitHub app.
3. Where are your tests located? (e.g. tests/) 3. Which Python module do you want me to optimize going forward? (e.g. my_module)
4. Which test framework do you use? (pytest/unittest) 4. Where are your tests located? (e.g. tests/)
5. Which test framework do you use? (pytest/unittest)
6. Install GitHub actions for Continuous optimization?
``` ```
</Step> </Step>
</Steps> </Steps>
After you have answered these questions, Codeflash will be configured for your project. After you have answered these questions, the Codeflash configuration will be saved in the `pyproject.toml` file.
The configuration will be saved in the `pyproject.toml` file in the root directory of your project. To understand the configuration options, and set more advanced options, see the [Manual Configuration](/configuration) page.
To understand the configuration options, and set more advanced options, see the [Configuration](/configuration) page.
### Step 4: Install the Codeflash GitHub App ### Step 4: Install the Codeflash GitHub App
{/* TODO: Justify to users Why we need the user to install Github App even in local Installation or local optimization? */} {/* TODO: Justify to users Why we need the user to install Github App even in local Installation or local optimization? */}
Finally, if you have not done so already, Codeflash will ask you to install the Github App in your repository. The Codeflash GitHub App allows access to your repository to the codeflash-ai bot to open PRs, review code, and provide optimization suggestions. Finally, if you have not done so already, Codeflash will ask you to install the GitHub App in your repository.
The Codeflash GitHub App allows access to your repository to the codeflash-ai bot to open PRs, review code, and provide optimization suggestions.
Please [install the Codeflash GitHub Please [install the Codeflash GitHub
app](https://github.com/apps/codeflash-ai/installations/select_target) by choosing the repository you want to install app](https://github.com/apps/codeflash-ai/installations/select_target) by choosing the repository you want to install
@ -128,30 +118,29 @@ Once configured, you can start optimizing your code immediately:
# Optimize a specific function # Optimize a specific function
codeflash --file path/to/your/file.py --function function_name codeflash --file path/to/your/file.py --function function_name
# Or optimize locally without creating a PR # Or optimize all functions in your codebase
codeflash --file path/to/your/file.py --function function_name --no-pr codeflash --all
``` ```
<Tip>
**Pro tip**: Start with a single function to see how Codeflash works before running it on your entire codebase.
</Tip>
</Tab> </Tab>
<Tab title="Example Project"> <Tab title="Optimize Example Project">
<Card title="🚀 Try Our Example Repository" icon="github" href="https://github.com/codeflash-ai/optimize-me"> <Card title="🚀 Try optimizing our example repository" icon="github" href="https://github.com/codeflash-ai/optimize-me">
Want to see Codeflash in action? Check out our **optimize-me** repository with real examples. Want to see Codeflash in action and don't know what code to optimize? Check out our **optimize-me** repository with code ready to optimize.
**What's included:** **What's included:**
- Sample Python code with performance issues - Sample Python code with performance issues
- Tests for verification - Tests for verification
- Pre-configured `pyproject.toml` - Pre-configured `pyproject.toml`
- Before/after optimization examples in PRs
</Card> </Card>
<Steps> <Steps>
<Step title="Clone the Repository"> <Step title="Fork the Repository">
Fork the [optimize-me](https://github.com/codeflash-ai/optimize-me) repo to your GitHub account by clicking "Fork" on the top of the page. This allows Codeflash to open Pull Requests with the optimizations it found on your forked repo.
</Step>
<Step title="Clone the Forked Repository">
```bash ```bash
git clone https://github.com/codeflash-ai/optimize-me.git git clone https://github.com/your_github_username/optimize-me.git
cd optimize-me cd optimize-me
``` ```
</Step> </Step>
@ -159,7 +148,7 @@ cd optimize-me
<Step title="Set Up Environment"> <Step title="Set Up Environment">
```bash ```bash
python -m venv .venv python -m venv .venv
source .venv/bin/activate # or venv\Scripts\activate on Windows source .venv/bin/activate
pip install -r requirements.txt pip install -r requirements.txt
pip install codeflash pip install codeflash
``` ```
@ -193,7 +182,9 @@ codeflash --all # optimize the entire repo
</Accordion> </Accordion>
<Accordion title="🧪 No optimizations found or debugging issues"> <Accordion title="🧪 No optimizations found or debugging issues">
Use the `--verbose` flag for detailed output: Do know that not all functions can be optimized as no optimization opportunities may exist for them. This is fine and expected.
To investigate further, use the `--verbose` flag for detailed output:
```bash ```bash
codeflash optimize --verbose codeflash optimize --verbose
``` ```
@ -203,13 +194,7 @@ codeflash --all # optimize the entire repo
- 🚫 Why certain functions were skipped - 🚫 Why certain functions were skipped
- ⚠️ Detailed error messages - ⚠️ Detailed error messages
- 📊 Performance analysis results - 📊 Performance analysis results
<Tip>
**Common Reasons Functions Are Skipped:**
- Function is too simple (less than 3 lines)
- Function has no clear performance bottleneck
- Function contains unsupported constructs
</Tip>
</Accordion> </Accordion>
<Accordion title="🔍 No tests found errors"> <Accordion title="🔍 No tests found errors">
@ -223,7 +208,7 @@ codeflash --all # optimize the entire repo
pytest --collect-only pytest --collect-only
# Check your pyproject.toml configuration # Check your pyproject.toml configuration
cat pyproject.toml | grep -A 5 "\[tool.codeflash\]" cat pyproject.toml | grep -A 8 "\[tool.codeflash\]"
``` ```
</Accordion> </Accordion>
</AccordionGroup> </AccordionGroup>
@ -233,5 +218,5 @@ codeflash --all # optimize the entire repo
- Learn about [Codeflash Concepts](/codeflash-concepts/how-codeflash-works) - Learn about [Codeflash Concepts](/codeflash-concepts/how-codeflash-works)
- Explore [Optimization workflows](/optimizing-with-codeflash/one-function) - Explore [Optimization workflows](/optimizing-with-codeflash/one-function)
- Set up [GitHub Actions integration](/getting-started/codeflash-github-actions) - Set up [Pull Request Optimization](/optimizing-with-codeflash/codeflash-github-actions)
- Read [configuration options](/configuration) for advanced setups - Read [configuration options](/configuration) for advanced setups

16
docs/getting-the-best-out-of-codeflash.mdx
View file

@ -6,14 +6,11 @@ sidebarTitle: "Best Practices"
keywords: ["best practices", "tips", "github actions", "tracer", "optimization", "workflow"] keywords: ["best practices", "tips", "github actions", "tracer", "optimization", "workflow"]
--- ---
# Getting the best out of Codeflash Codeflash is a powerful tool; here are our recommendations based on how the Codeflash team and our customers use Codeflash.
Codeflash is a powerful tool; here are our recommendations based on how the Codeflash team uses Codeflash. ### Install the GitHub App and actions workflow
### Install the Github App and actions workflow After you install Codeflash on an actively developed project, [installing the GitHub Actions](optimizing-with-codeflash/codeflash-github-actions) will automatically optimize your code whenever new pull requests are opened. This ensures you get the best version of any changes you make to your code without any extra effort. We find that PRs are also the best time to review these changes, because the code is fresh in your mind.
After you install Codeflash on an actively developed project, [installing the GitHub App](getting-started/codeflash-github-actions) and setting up the
GitHub Actions workflow will automatically optimize your code whenever new pull requests are opened. This ensures you get the best version of any changes you make to your code without any extra effort. We find that PRs are also the best time to review these changes, because the code is fresh in your mind.
### Find and optimize entire scripts with the Codeflash Tracer ### Find and optimize entire scripts with the Codeflash Tracer
@ -21,13 +18,18 @@ Find the best results by running [Codeflash Optimize](optimizing-with-codeflash/
This internally runs a profiler, captures inputs to all the functions your script calls, and uses those inputs to create Replay tests and benchmarks. This internally runs a profiler, captures inputs to all the functions your script calls, and uses those inputs to create Replay tests and benchmarks.
The optimizations you get with this method, show you how much faster your workflow will get plus guarantee that your workflow won't break if you merge in the optimizations. The optimizations you get with this method, show you how much faster your workflow will get plus guarantee that your workflow won't break if you merge in the optimizations.
###
### Find optimizations on your whole codebase with `codeflash --all` ### Find optimizations on your whole codebase with `codeflash --all`
If you have a lot of existing code, run [`codeflash --all`](optimizing-with-codeflash/codeflash-all) to discover and fix any If you have a lot of existing code, run [`codeflash --all`](optimizing-with-codeflash/codeflash-all) to discover and fix any
slow code in your project. Codeflash will open new pull requests for any optimizations it finds, and you can review and merge them at your own pace. slow code in your project. Codeflash will open new pull requests for any optimizations it finds, and you can review and merge them at your own pace.
It is first recommended to trace your tests to achieve higher quality optimizations with this approach
```bash
codeflash optimize --trace-only -m pytest tests/ ; codeflash --all
```
### Review the PRs Codeflash opens ### Review the PRs Codeflash opens

BIN
docs/images/codeflash_pr_suggestion_1.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 527 KiB

BIN
docs/images/codeflash_social_card.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 34 KiB

24
docs/index.mdx
View file

@ -1,17 +1,17 @@
--- ---
title: "What is Codeflash?" title: "Codeflash is an AI performance optimizer for Python code"
description: "AI-powered Python performance optimizer that automatically speeds up your code while verifying correctness"
icon: "rocket" icon: "rocket"
sidebarTitle: "Overview" sidebarTitle: "Overview"
keywords: ["python", "performance", "optimization", "AI", "code analysis", "benchmarking"] keywords: ["python", "performance", "optimization", "AI", "code analysis", "benchmarking"]
--- ---
Codeflash speeds up Python code by figuring out the best way to rewrite your code while verifying that the behavior of the code is unchanged. Codeflash speeds up any Python code by figuring out the best way to rewrite it while verifying that the behavior of the code is unchanged, and verifying real speed
gains through performance benchmarking.
The optimizations Codeflash finds are generally better algorithms, opportunities to remove wasteful compute, better logic, utilizing caching and utilization of more efficient library methods. Codeflash The optimizations Codeflash finds are generally better algorithms, opportunities to remove wasteful compute, better logic, utilizing caching and utilization of more efficient library methods. Codeflash
does not modify the system architecture of your code, but it tries to find the most efficient implementation of that architecture. does not modify the system architecture of your code, but it tries to find the most efficient implementation of your current architecture.
### Features ### How to use Codeflash
<CardGroup cols={1}> <CardGroup cols={1}>
<Card title="Optimize a Single Function" icon="bullseye" href="/optimizing-with-codeflash/one-function"> <Card title="Optimize a Single Function" icon="bullseye" href="/optimizing-with-codeflash/one-function">
@ -21,6 +21,13 @@ does not modify the system architecture of your code, but it tries to find the m
``` ```
</Card> </Card>
<Card title="Optimize Pull Requests" icon="code-pull-request" href="/optimizing-with-codeflash/codeflash-github-actions">
Automatically find optimizations for Pull Requests with GitHub Actions integration.
```bash
codeflash init-actions
```
</Card>
<Card title="Optimize Workflows with Tracing" icon="route" href="/optimizing-with-codeflash/trace-and-optimize"> <Card title="Optimize Workflows with Tracing" icon="route" href="/optimizing-with-codeflash/trace-and-optimize">
End-to-end optimization of entire Python workflows with execution tracing. End-to-end optimization of entire Python workflows with execution tracing.
```bash ```bash
@ -35,12 +42,7 @@ does not modify the system architecture of your code, but it tries to find the m
``` ```
</Card> </Card>
<Card title="Optimize Pull Requests" icon="git-pull-request" href="/optimizing-with-codeflash/optimize-prs">
Automatically find optimization code changes in Pull Requests with GitHub Actions integration.
```bash
codeflash init-actions
```
</Card>
</CardGroup> </CardGroup>
### How does Codeflash verify correctness? ### How does Codeflash verify correctness?

30
docs/optimizing-with-codeflash/benchmarking.mdx
View file

@ -1,21 +1,21 @@
--- ---
title: "Using Benchmarks in CI" title: "Optimize Performance Benchmarks with every Pull Request"
description: "Configure and use pytest-benchmark integration for performance-critical code optimization" description: "Configure and use pytest-benchmark integration for performance-critical code optimization"
icon: "chart-line" icon: "chart-line"
sidebarTitle: "CI Benchmarks" sidebarTitle: Setup Benchmarks to Optimize
keywords: ["benchmarks", "CI", "pytest-benchmark", "performance testing", "github actions", "benchmark mode"] keywords: ["benchmarks", "CI", "pytest-benchmark", "performance testing", "github actions", "benchmark mode"]
--- ---
<Info> <Info>
**Performance-critical optimization** - Define benchmarks for your most important functions and let Codeflash measure the real-world impact of every optimization on your performance metrics. **Performance-critical optimization** - Define benchmarks for your most important code sections and let Codeflash optimize and measure the real-world impact of every optimization on your performance metrics.
</Info> </Info>
Benchmark mode is an easy way for users to define workflows that are performance-critical and need to be optimized. Benchmark mode is an easy way to define workflows that are performance-critical and need to be optimized and run fast.
For example, if a user has an important function that requires minimal latency, the user can define a benchmark for that function. Codeflash will run the benchmark, understand how the current code change in the Pull Request is affecting the benchmark.
Codeflash will then calculate the impact (if any) of any optimization on the performance of that function. It will then try to optimize the new code for the benchmark and calculate the impact of any optimization on the speed of that benchmark.
## Using Codeflash in Benchmark Mode ## Using Codeflash in Benchmark Mode
1. **Create a benchmarks root** 1. **Create a benchmarks root:**
Create a directory for benchmarks if it does not already exist. Create a directory for benchmarks if it does not already exist.
@ -31,7 +31,7 @@ Codeflash will then calculate the impact (if any) of any optimization on the per
formatter-cmds = ["disabled"] formatter-cmds = ["disabled"]
``` ```
2. **Define your benchmarks** 2. **Define your benchmarks:**
Currently, Codeflash only supports benchmarks written as pytest-benchmarks. Check out the [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/stable/index.html) documentation for more information on syntax. Currently, Codeflash only supports benchmarks written as pytest-benchmarks. Check out the [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/stable/index.html) documentation for more information on syntax.
@ -50,7 +50,7 @@ Codeflash will then calculate the impact (if any) of any optimization on the per
The pytest-benchmark format is simply used as an interface. The plugin is actually not used - Codeflash will run these benchmarks with its own pytest plugin The pytest-benchmark format is simply used as an interface. The plugin is actually not used - Codeflash will run these benchmarks with its own pytest plugin
3. **Run Codeflash** 3. **Run and Test Codeflash:**
Run Codeflash with the `--benchmark` flag. Note that benchmark mode cannot be used with `--all`. Run Codeflash with the `--benchmark` flag. Note that benchmark mode cannot be used with `--all`.
@ -65,13 +65,13 @@ Codeflash will then calculate the impact (if any) of any optimization on the per
``` ```
4. **Run Codeflash in CI** 4. **Run Codeflash :**
Benchmark mode is best used together with Codeflash as a Github Action. This way, with every PR, you will know the impact of Codeflash's optimizations on your benchmarks. Benchmark mode is best used together with Codeflash as a GitHub Action. This way, if new code in a Pull Request touches the benchmark workflow, Codeflash will try to optimize the code for the benchmark, and you will know the impact of Codeflash's optimizations on your benchmarks.
Use `codeflash init` for an easy way to set up Codeflash as a Github Action (with the option to enable benchmark mode). Use `codeflash init` for an easy way to set up Codeflash as a GitHub Action.
Otherwise, you can run the following command in your Codeflash GitHub Action: After that, you can add the `--benchmark` argument to codeflash to enable benchmarks optimization.
```bash ```bash
codeflash --benchmark codeflash --benchmark
@ -84,7 +84,7 @@ Codeflash will then calculate the impact (if any) of any optimization on the per
1. Codeflash identifies benchmarks in the benchmarks-root directory. 1. Codeflash identifies benchmarks in the benchmarks-root directory.
2. The benchmarks are run so that runtime statistics and information can be recorded. 2. The benchmarks are run so that runtime statistics and inputs can be recorded.
3. Replay tests are generated so the performance of optimization candidates on the exact inputs used in the benchmarks can be measured. 3. Replay tests are generated so the performance of optimization candidates on the exact inputs used in the benchmarks can be measured.
@ -97,5 +97,3 @@ Codeflash will then calculate the impact (if any) of any optimization on the per
Using Codeflash with benchmarks is a great way to find optimizations that really matter. Using Codeflash with benchmarks is a great way to find optimizations that really matter.
Codeflash is actively working on this feature and will be adding new capabilities in the near future!

27
docs/optimizing-with-codeflash/codeflash-all.mdx
View file

@ -1,8 +1,8 @@
--- ---
title: "Optimize Your Entire Codebase" title: "Optimize Your Entire Codebase"
description: "Automatically optimize all functions in your project with Codeflash's comprehensive analysis" description: "Automatically optimize all codepaths in your project with Codeflash's comprehensive analysis"
icon: "database" icon: "database"
sidebarTitle: "Entire Codebase" sidebarTitle: "Optimize Entire Codebase"
keywords: ["codebase optimization", "all functions", "batch optimization", "github app", "checkpoint", "recovery"] keywords: ["codebase optimization", "all functions", "batch optimization", "github app", "checkpoint", "recovery"]
--- ---
@ -19,13 +19,34 @@ codeflash --all
This requires the Codeflash GitHub App to be installed in your repository. This requires the Codeflash GitHub App to be installed in your repository.
This is a powerful feature that can help you optimize your entire codebase in one go. This is a powerful feature that can help you optimize your entire codebase in one go. It also discovers and runs any unit tests covering the function under optimization.
Since it runs on all the functions in your codebase, it can take some time to complete, please be patient. Since it runs on all the functions in your codebase, it can take some time to complete, please be patient.
As this runs you will see Codeflash opening pull requests for each function it successfully optimizes. As this runs you will see Codeflash opening pull requests for each function it successfully optimizes.
If you only want to optimize a subdirectory you can run:
```bash
codeflash --all path/to/dir
```
<Tip>
If your project has a good number of unit tests, we can trace those to achieve higher quality results.
The following approach is recommended instead:
```bash
codeflash optimize --trace-only -m pytest tests/ ; codeflash --all
```
This will run your test suite, trace all the code covered by your tests, ensuring higher correctness guarantees
and better performance benchmarking, and help create optimizations for code where the LLMs struggle to generate and run tests.
Even though `codeflash --all` discovers any existing unit tests. It currently can only discover any test that directly calls the
function under optimization. Tracing all the tests helps ensure correctness for code that may be indirectly called by your tests.
</Tip>
## Important considerations ## Important considerations
- **Dedicated Optimization Machine:** Optimizing the entire codebase may require considerable time—up to one day. It's recommended to allocate a dedicated machine specifically for this long-running optimization task. - **Dedicated Optimization Machine:** Optimizing the entire codebase may require considerable time—up to one day. It's recommended to allocate a dedicated machine specifically for this long-running optimization task.
- **Minimize Background Processes:** To achieve optimal results, avoid running other processes on the optimization machine. Additional processes can introduce noise into Codeflash's runtime measurements, reducing the quality of the optimizations. Although Codeflash tolerates some runtime fluctuations, minimizing noise ensures the highest optimization quality. - **Minimize Background Processes:** To achieve optimal results, avoid running other processes on the optimization machine. Additional processes can introduce noise into Codeflash's runtime measurements, reducing the quality of the optimizations. Although Codeflash tolerates some runtime fluctuations, minimizing noise ensures the highest optimization quality.
- **Checkpoint and Recovery:** Codeflash automatically creates checkpoints as it identifies optimizations. If the optimization process is interrupted or encounters issues, you can resume the process by re-running `codeflash --all`. The command will prompt you to continue from the most recent checkpoint. - **Checkpoint and Recovery:** Codeflash automatically creates checkpoints as it identifies optimizations. If the optimization process is interrupted or encounters issues, you can resume the process by re-running `codeflash --all`. The command will prompt you to continue from the most recent checkpoint.

40
docs/getting-started/codeflash-github-actions.mdx → docs/optimizing-with-codeflash/codeflash-github-actions.mdx
View file

@ -1,21 +1,25 @@
--- ---
title: "GitHub Actions Integration" title: "Auto Optimize Pull Requests"
description: "Automatically optimize pull requests with Codeflash GitHub Actions workflow" description: "Automatically optimize new code in pull requests with Codeflash GitHub Actions workflow"
icon: "github" icon: "github"
--- ---
{/* TODO: Add more pictures to guide better */}
Codeflash can automatically optimize your code when new pull requests are opened. Optimizing new code in Pull Requests is the best way to ensure that all code you and your team ship is performant
in the future. Automating optimization in the Pull Request stage how most teams use Codeflash, to
continuously find optimizations for their new code.
To be able to scan new code for performance optimizations, Codeflash requires a GitHub action workflow to To scan new code for performance optimizations, Codeflash uses a GitHub Action workflow which runs
be installed which runs the Codeflash optimization logic on every new pull request. the Codeflash optimization logic on the new code in every pull request.
If the action workflow finds an optimization, it communicates with the Codeflash GitHub If the action workflow finds an optimization, it communicates with the Codeflash GitHub
App through our secure servers and asks it to suggest new changes to the pull request. App and asks it to suggest new changes to the pull request.
This is the most useful way of using Codeflash, where you set it up once and all your new code gets optimized. This is the most useful way of using Codeflash, where you set it up once and all your new code gets optimized.
So setting this up is highly recommended. So setting this up is highly recommended.
## Pull Request Optimization 30 seconds demo
<iframe width="640" height="400" src="https://www.youtube.com/embed/nqa-uewizkU?si=H1wb1RvPp-JqvKPh" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
## Prerequisites ## Prerequisites
<Warning> <Warning>
@ -38,18 +42,18 @@ So setting this up is highly recommended.
codeflash init-actions codeflash init-actions
``` ```
This command will automatically create the GitHub Actions workflow file and guide you through the setup process. This command will automatically create the GitHub Actions workflow file and guide you through the setup process.
Alternatively running `codeflash init` also asks to setup the github actions.
</Step> </Step>
<Step title="Test Your Setup"> <Step title="Customize and Test Your Setup">
Open a new pull request to your GitHub project. You'll see: Open a new pull request to your GitHub project. You'll see:
- ✅ A new Codeflash workflow running in GitHub Actions - ✅ A new Codeflash workflow running in GitHub Actions
- 🤖 The codeflash-ai bot commenting with optimization suggestions (if any are found) - 🤖 The codeflash-ai bot commenting with optimization suggestions (if any are found)
Ensure that your Python environment installation works correctly and codeflash is able to run.
</Step> </Step>
</Steps> </Steps>
<Note>
**Recommended approach** - This automated setup ensures you get the latest workflow configuration with all best practices included.
</Note>
</Tab> </Tab>
<Tab title="Manual Setup"> <Tab title="Manual Setup">
@ -153,4 +157,16 @@ Customize the dependency installation based on your Python package manager:
</Steps> </Steps>
</Tab> </Tab>
</Tabs> </Tabs>
## How the Pull Request Optimization Suggestion looks
Codeflash creates a new dependent Pull Request for you to review with the reported speedups, helpful explanation for the optimization
and the proof of correctness. The pull request has the code change for you to review and accept.
![Codeflash PR Review](/images/codeflash_pr_suggestion_1.png)
Sometimes it also makes an inline suggestion with the optimization.
![Codeflash PR Suggestion](/images/code-suggestion.png)
We hope you enjoy the performance unlock the Pull Request optimization enables.

2
docs/optimizing-with-codeflash/one-function.mdx
View file

@ -2,7 +2,7 @@
title: "Optimize a Single Function" title: "Optimize a Single Function"
description: "Target and optimize individual Python functions for maximum performance gains" description: "Target and optimize individual Python functions for maximum performance gains"
icon: "bullseye" icon: "bullseye"
sidebarTitle: "Single Function" sidebarTitle: "Optimize Single Function"
keywords: ["function optimization", "single function", "class methods", "performance", "targeted optimization"] keywords: ["function optimization", "single function", "class methods", "performance", "targeted optimization"]
--- ---

33
docs/optimizing-with-codeflash/optimize-prs.mdx
View file

@ -1,33 +0,0 @@
---
title: "Optimize Pull Requests"
description: "Automatically optimize code changes in pull requests with GitHub Actions integration"
icon: "code-merge"
sidebarTitle: "PR Optimization"
keywords: ["pull requests", "github actions", "code review", "automated optimization", "dependent PR", "suggestions"]
---
<Info>
**Continuous optimization** - After initial setup, Codeflash will automatically review every new pull request and suggest performance improvements through comments and dependent PRs.
</Info>
## How to optimize a pull request
After following the setup steps in the [Automate Code Optimization with GitHub Actions](/getting-started/codeflash-github-actions) guide,
Codeflash will automatically optimize your pull requests when they are opened.
If Codeflash finds any successful optimizations, it will comment on the pull request asking you to review the changes.
![Codeflash PR Comment](/images/review-comment.png)
Codeflash can ask you to review the changes in two ways:
### Opening a dependent pull request
Codeflash will open a new pull request with the optimized code.
You can review the changes in this pull request, make changes if you want, and merge it if you are satisfied with the optimizations.
The changes will be merged back into the original pull request as a new commit.
![Codeflash PR Review](/images/dependent-pr.png)
### Reviewing the changes in the original pull request
If the suggested changes are small and only affect the modified lines, Codeflash will suggest the changes in the original pull request itself.
You can choose to accept or reject the changes directly in the original pull request.
The changes can be added to a batch of changes in the original pull request as a new commit.
![Codeflash PR Review](/images/code-suggestion.png)

32
docs/optimizing-with-codeflash/trace-and-optimize.mdx
View file

@ -1,26 +1,28 @@
--- ---
title: "Trace & Optimize Workflows" title: "Trace & Optimize E2E Workflows"
description: "End-to-end optimization of entire Python workflows with execution tracing" description: "End-to-end optimization of entire Python workflows with execution tracing"
icon: "route" icon: "route"
sidebarTitle: "Trace & Optimize" sidebarTitle: "Optimize E2E Workflows"
keywords: ["tracing", "workflow optimization", "replay tests", "end-to-end", "script optimization", "context manager"] keywords: ["tracing", "workflow optimization", "replay tests", "end-to-end", "script optimization", "context manager"]
--- ---
Codeflash supports optimizing an entire Python script end-to-end by tracing the script's execution and generating Replay Tests. Tracing follows the execution of a script, profiles it and captures inputs to all called functions, allowing them to be replayed during optimization. Codeflash uses these Replay Tests to optimize all functions called in the script, starting from the most important ones. Codeflash supports optimizing an entire Python script end-to-end by tracing the script's execution and generating Replay Tests.
Tracing follows the execution of a script, profiles it and captures inputs to all functions it called, allowing them to be replayed during optimization.
Codeflash uses these Replay Tests to optimize all functions called in the script, starting from the most important ones.
To optimize a script, `python myscript.py`, replace `python` with `codeflash optimize` and run the following command: To optimize a script, `python myscript.py`, simply replace `python` with `codeflash optimize` and run the following command:
```bash ```bash
codeflash optimize myscript.py codeflash optimize myscript.py
``` ```
To optimize code called by pytest tests that you could normally run like `python -m pytest tests/`, use this command: You can also optimize code called by pytest tests that you could normally run like `python -m pytest tests/`, this provides for a good workload to optimize. Run this command:
```bash ```bash
codeflash optimize -m pytest tests/ codeflash optimize -m pytest tests/
``` ```
This powerful command creates high-quality optimizations, making it ideal when you need to optimize a workflow or script. The initial tracing process can be slow, so try to limit your script's runtime to under 1 minute for best results. If your workflow is longer, consider tracing it into smaller sections by using the Codeflash tracer as a context manager (point 3 below). The powerful `codeflash optimize` command creates high-quality optimizations, making it ideal when you need to optimize a workflow or script. The initial tracing process can be slow, so try to limit your script's runtime to under 1 minute for best results. If your workflow is longer, consider tracing it into smaller sections by using the Codeflash tracer as a context manager (point 3 below).
The generated replay tests and the trace file are for the immediate optimization use, don't add them to git. The generated replay tests and the trace file are for the immediate optimization use, don't add them to git.
@ -32,10 +34,11 @@ The generated replay tests and the trace file are for the immediate optimization
## What is the codeflash optimize command? ## What is the codeflash optimize command?
`codeflash optimize` tries to do everything that an expert engineer would do while optimizing a workflow. It profiles your code, traces the execution of your workflow and generates a set of test cases that are derived from how your code is actually run. `codeflash optimize` tries to do everything that an expert engineer would do while optimizing a workflow. It profiles your code, traces the execution of your workflow and generates a set of test cases that are derived from how your code is actually run.
Codeflash Tracer works by recording the inputs of your functions as they are called in your codebase. These inputs are then used to generate test cases that are representative of the real-world usage of your functions. Codeflash Tracer works by recording the inputs of your functions as they are called in your codebase, and generating
regression tests with those inputs.
We call these generated test cases "Replay Tests" because they replay the inputs that were recorded during the tracing phase. We call these generated test cases "Replay Tests" because they replay the inputs that were recorded during the tracing phase.
These replay tests are representative of the real-world usage of your functions.
Then, Codeflash Optimizer can use these replay tests to verify correctness and calculate accurate performance gains for the optimized functions.
Using Replay Tests, Codeflash can verify that the optimized functions produce the same output as the original function and also measure the performance gains of the optimized function on the real-world inputs. Using Replay Tests, Codeflash can verify that the optimized functions produce the same output as the original function and also measure the performance gains of the optimized function on the real-world inputs.
This way you can be *sure* that the optimized function causes no changes of behavior for the traced workflow and also, that it is faster than the original function. To get more confidence on the correctness of the code, we also generate several LLM generated test cases and discover any existing unit cases you may have. This way you can be *sure* that the optimized function causes no changes of behavior for the traced workflow and also, that it is faster than the original function. To get more confidence on the correctness of the code, we also generate several LLM generated test cases and discover any existing unit cases you may have.
@ -57,15 +60,16 @@ Codeflash script optimizer can be used in three ways:
codeflash optimize path/to/your/file.py --your_options codeflash optimize path/to/your/file.py --your_options
``` ```
The above command should suffice in most situations. You can add a argument like `codeflash optimize -o trace_file_path.trace` if you want to customize the trace file location. Otherwise, it defaults to `codeflash.trace` in the current working directory. The above command should suffice in most situations.
To customize the trace file location you can specify it like `codeflash optimize -o trace_file_path.trace`. Otherwise, it defaults to `codeflash.trace` in the current working directory.
2. **Trace and optimize as two separate steps** 2. **Trace and optimize as two separate steps**
If you want more control over the tracing and optimization process. You can trace first and then optimize with the replay tests later. Each replay test is associated with a trace file. If you want more control over the tracing and optimization process. You can trace first and then optimize with the replay tests later. Each replay test is associated with a trace file.
To first create just the trace file, run To create just the trace file first, run
```python ```bash
codeflash optimize -o trace_file.trace --trace-only path/to/your/file.py --your_options codeflash optimize -o trace_file.trace --trace-only path/to/your/file.py --your_options
``` ```
@ -79,7 +83,7 @@ Codeflash script optimizer can be used in three ways:
- `--tracer-timeout`: The maximum time in seconds to trace the entire workflow. Default is indefinite. This is useful while tracing really long workflows. - `--tracer-timeout`: The maximum time in seconds to trace the entire workflow. Default is indefinite. This is useful while tracing really long workflows.
3. **As a Context Manager -** 3. **As a Context Manager -**
To trace only very specific sections of your codeflash, You can also use the Codeflash Tracer as a context manager. To trace only specific sections of your code, You can also use the Codeflash Tracer as a context manager.
You can wrap the code you want to trace in a `with` statement as follows - You can wrap the code you want to trace in a `with` statement as follows -
```python ```python
@ -89,7 +93,7 @@ Codeflash script optimizer can be used in three ways:
model.predict() # Your code here model.predict() # Your code here
``` ```
This is much faster than tracing the whole script. Sometimes, if tracing the whole script fails, then the Context Manager can also be used to trace the code sections. This is much faster than tracing the whole script. It can also help if tracing the whole script fails.
After this finishes, you can optimize using the generated replay tests. After this finishes, you can optimize using the generated replay tests.
@ -97,7 +101,7 @@ Codeflash script optimizer can be used in three ways:
codeflash --replay-test /path/to/test_replay_test_0.py codeflash --replay-test /path/to/test_replay_test_0.py
``` ```
More Options for the Tracer: More Options for the Tracer Context Manager:
- `disable`: If set to `True`, the tracer will not trace the code. Default is `False`. - `disable`: If set to `True`, the tracer will not trace the code. Default is `False`.
- `max_function_count`: The maximum number of times to trace a single function. More calls to a function will not be traced. Default is 100. - `max_function_count`: The maximum number of times to trace a single function. More calls to a function will not be traced. Default is 100.