# Codeflash Performance Optimization Tasks

105 Python code optimization tasks for training and evaluating RL agents on real-world performance optimization, packaged in [Harbor](https://github.com/harbor-framework) format.

Each task presents an agent with a Python function in the [roboflow/inference](https://github.com/roboflow/inference) repository and asks it to rewrite the function to run faster while preserving identical behavior. The agent is evaluated on correctness (behavioral equivalence) and speedup (wall-clock performance ratio).

## Docker Image

```
codeflash/codeflash-rl-env:latest
```

Pre-built image with the inference repository, all dependencies, and the codeflash evaluation harness (~8 GB). Contains no task data — Harbor mounts task files into the container at each stage.

## Task Structure

Each task directory follows Harbor format:

```
<task-name>/
  task.toml              # Metadata, resource limits, docker_image reference
  instruction.md         # Shown to the agent
  environment/
    Dockerfile           # FROM codeflash/codeflash-rl-env:latest + git checkout
  solution/              # Reference solution (never shown to agent)
    solve.sh
    apply_optimization.py
    config.json
    optimized_code.py
  tests/                 # Mounted at evaluation time (never shown to agent)
    test.sh              # Runs evaluator, writes /logs/verifier/reward.json
    codeflash_eval/
      evaluate.py
      original.py
      *behaviorinstrumented*.py
      *perfinstrumented*.py
```

Harbor mounts the right directories at each stage:
- **Agent phase**: Only the repo at `/workspace/inference` (at the correct commit). Agent sees `instruction.md` but not `/solution/` or `/tests/`.
- **Evaluation phase**: Harbor mounts `tests/` to `/tests/` and runs `test.sh`.
- **Oracle phase**: Harbor mounts `solution/` to `/solution/` and runs `solve.sh`.

The evaluator writes two files to `/logs/verifier/`:
- **`reward.json`**: Single scalar reward for Harbor (e.g., `{"speedup": 13.3}`)
- **`details.json`**: Full metrics for custom reward functions (correctness, per-test speedups, coverage, quality)

## Reward Signal

`test.sh` runs the evaluator and writes `/logs/verifier/reward.json`:

`reward.json` (used by Harbor):

```json
{"speedup": 13.31}
```

`details.json` (for custom reward functions):

```json
{
  "correct": 1.0,
  "speedup": 13.31,
  "speedup_per_test_case": {
    "test__slugify_model_id__unit_test_1::test_very_long_model_id_performance": 88.5,
    "test__slugify_model_id__unit_test_1::test_stress_test_1000_iterations": 12.63,
    "test__slugify_model_id__unit_test_0::test_basic_alphanumeric_preserved": 5.71
  },
  "tests_passed": 1050,
  "tests_total": 1050,
  "coverage_pct": 100.0,
  "optimization_quality": "low"
}
```

| Field                          | File         | Description                                                            |
|--------------------------------|--------------|------------------------------------------------------------------------|
| `speedup`                      | reward.json  | Ratio of original to candidate runtime (>1.0 = faster). 0.0 if wrong. |
| `correct`                      | details.json | 1.0 if all behavioral tests pass, 0.0 otherwise                       |
| `speedup_per_test_case`        | details.json | Per-test-function speedup breakdown                                    |
| `tests_passed` / `tests_total` | details.json | Behavioral test pass count                                             |
| `coverage_pct`                 | details.json | Line coverage of the target function                                   |
| `optimization_quality`         | details.json | AI-assessed quality: `high`, `medium`, or `low`                        |

## Evaluation Pipeline

The evaluator runs:

1. **Coverage** — Line coverage of the target function
2. **Behavioral tests on original** — Runs tests at HEAD~1, captures return values + stdout + exceptions
3. **Behavioral tests on candidate** — Runs tests at HEAD, captures the same
4. **Comparison** — Deep equivalence check using codeflash's comparator
5. **Performance benchmark on original** — 5-1000 loops over 10 seconds
6. **Performance benchmark on candidate** — Same
7. **Speedup** — `original_runtime / candidate_runtime` (stable minimum across loops)
8. **Quality review** — AI assessment of optimization quality

The agent must commit its changes as a git commit. The evaluator uses `git checkout` to switch between original (HEAD~1) and candidate (HEAD).

## Running with Harbor

### Single task (oracle solution)

```bash
harbor run --path cache-_slugify_model_id --agent oracle
```

### Single task with an agent

```bash
harbor run --path cache-_slugify_model_id --agent claude-code --model anthropic/claude-sonnet-4-6
```

### All tasks

```bash
harbor run --path . --agent oracle --n-concurrent 4
```

### Filter tasks

```bash
# Run tasks matching a pattern
harbor run --path . --agent oracle --include-task-name "cache-*"

# Limit number of tasks
harbor run --path . --agent oracle --n-tasks 10
```

## Running Locally (Without Harbor)

```bash
TASK=cache-_slugify_model_id

# Build per-task image
docker build -t task-test -f $TASK/environment/Dockerfile $TASK/environment/

# Run reference solution + evaluation
docker run --rm \
    -v $(pwd)/$TASK/solution:/solution:ro \
    -v $(pwd)/$TASK/tests:/tests:ro \
    task-test \
    bash -c '
        cd /workspace/inference
        bash /solution/solve.sh
        bash /tests/test.sh
        cat /logs/verifier/reward.json
    '
```

## Task Difficulty Distribution

| Speedup Range | Tasks | Description                                                  |
|---------------|-------|--------------------------------------------------------------|
| >100x         | 6     | Major algorithmic improvements (caching, precomputation)     |
| 10-100x       | 12    | Significant optimizations (data structure changes, memoization) |
| 5-10x         | 5     | Strong optimizations (vectorization, early exits)            |
| 2-5x          | 14    | Moderate optimizations (reduced allocations, batching)       |
| 1.1-2x        | 68    | Micro-optimizations (branch reordering, constant hoisting)   |

## Common Optimization Patterns

- **Memoization / Caching**: `lru_cache`, module-level constant precomputation, instance-level caching
- **Algorithmic**: O(n) to O(log n) via heaps, set lookups instead of list scans
- **Vectorization**: Replacing Python loops with numpy/torch operations
- **Allocation reduction**: Reusing objects instead of creating new ones per call
- **Early exits**: Short-circuiting on common cases before expensive computation
- **Data structure changes**: `frozenset` for O(1) membership, `tuple` for immutable sequences

## Image Details

- **Python**: 3.12
- **Repository**: roboflow/inference (full git history)
- **Key dependencies**: PyTorch, OpenCV, Pydantic, supervision, numpy
- **Evaluation harness**: codeflash==0.20.5
- **Size**: ~8 GB
