# Experiment Loop — Async Domain

> Base framework: `../shared/experiment-loop-base.md`
>
> **Note:** Step numbers below match the async agent's inline loop (17 steps), not the shared base (which has extra steps 4 "Capture original output" and 8 "Verify output equivalence" that are not applicable to async domain).

## Reasoning Checklist

Before writing any code, answer these 9 questions. If you can't answer 3-6 concretely, research more before coding.

1. **Pattern**: What async antipattern or missed concurrency opportunity? (see Antipattern Categories in the agent prompt)
2. **Hot path?** Is this code on a critical async path? Confirm with profiling or asyncio debug mode.
3. **Concurrency gain?** What's the expected improvement? (e.g., sequential N*latency -> max(latency), or unblocking event loop)
4. **Concurrency level?** How many concurrent operations are typical in production? A single request doesn't benefit from gather.
5. **Exercised?** Does the target test/benchmark actually trigger this code path with representative concurrency?
6. **Mechanism**: HOW does your change improve throughput or latency? Be specific.
7. **API lookup**: Before implementing, use context7 to look up the exact API you plan to use — query the project's actual libraries for the specific function/class you'll call. Get the correct signatures, defaults, and edge cases. Do NOT implement from memory when docs are a tool call away.
8. **Production-safe?** Does this change error handling, connection pool usage, or backpressure behavior?
9. **Verify cheaply**: Can you validate with a micro-benchmark before the full benchmark run?

## Domain-Specific Loop Steps

**Step 2 — Choose target** sources:
- **asyncio debug mode output**: Slow callback warnings pointing to blocking calls.
- **Static analysis**: Grep-based pattern detection — sequential awaits, await-in-loop, blocking calls, @cache on async.
- **Deep source reading**: Cross-function concurrency opportunities, connection management issues, architectural bottlenecks. Use Explore subagents for this.

Print: `[experiment N] Target: <description> (<pattern>, <est. impact>%)`

**Step 6 — Benchmark fidelity (async-specific):** If your change involves wrapper parameters (e.g., `thread_sensitive`, `Semaphore` bounds, pool sizes, driver config), verify the benchmark uses the same parameters. A benchmark testing `sync_to_async(fn)` does NOT validate a fix that uses `sync_to_async(fn, thread_sensitive=False)`. A benchmark using default pool settings does NOT validate a fix that changes pool configuration.

**Step 7 — Benchmark**: Run at the agreed concurrency level. Print `[experiment N] Benchmarking at concurrency=<N>...`.

**Step 9 — Read results**: Print `[experiment N] Latency: <before>ms -> <after>ms (<Z>% faster). Throughput: <X> -> <Y> req/s`. Include whichever metrics are available.

**Step 11 — Noise threshold**: If speedup is <10%, re-run 3 times to confirm not noise. Async benchmarks have higher variance than sync.

**Step 16 — Debug mode validation (optional)**: After keeping a fix for a blocking call, re-run with `PYTHONASYNCIODEBUG=1` to confirm the slow callback warning is gone.

**Step 17 — E2E benchmark (after KEEP)**: If `codeflash compare` is available (check `.codeflash/setup.md`), run `$RUNNER -m codeflash compare <pre-opt-sha> HEAD` for authoritative isolated measurement. See `../shared/e2e-benchmarks.md`.

**Step 18 — Milestones**: Create `codeflash/async-<tag>-v<N>` branch. At milestones, run `$RUNNER -m codeflash compare <baseline-sha> HEAD` for cumulative e2e measurement. Print `[milestone] vN — <total kept>/<total experiments>. Latency: <baseline>ms -> <current>ms. Throughput: <baseline> -> <current> req/s`.

## Keep/Discard Thresholds

```
Test passed?
+-- NO -> Fix or discard immediately
+-- YES -> Latency or throughput improved?
    +-- Latency improved >=10% (p50 or p99) -> KEEP
    +-- Throughput improved >=10% (req/s) -> KEEP
    +-- Either improved <10% -> Re-run 3x to confirm not noise
    |   +-- Confirmed -> KEEP
    |   +-- Noise -> DISCARD
    +-- Latency improved but throughput regressed (or vice versa) -> Evaluate tradeoff, ask user if unclear
    +-- Blocking call removed (debug mode confirms) -> KEEP (correctness improvement even if benchmark doesn't show yet)
    +-- Neither improved -> DISCARD
```

Note: Async changes often show larger gains under higher concurrency. If a change removes a blocking call but the benchmark uses low concurrency, keep it anyway — it's a correctness fix that will matter under load. Always test at the agreed concurrency level, not just sequential.

## Strategy Rotation

If 3+ consecutive discards on the same type of optimization, switch strategy:
- Sequential await gathering -> Blocking call removal -> Connection management -> Architectural restructuring

## Architectural Changes

When the bottleneck is a driver or infrastructure limitation (not a code pattern), the incremental experiment loop doesn't apply directly. Examples: migrating database drivers (psycopg2 -> psycopg3), swapping HTTP clients, changing connection pool implementations.

1. **Discover**: Profile to confirm the infrastructure limitation. Write a benchmark that isolates the exact constraint (e.g., connection sharing, thread contention).
2. **Benchmark the limitation**: Run the benchmark to quantify the current state. This is your baseline.
3. **Implement the migration**: Change the infrastructure component.
4. **Re-run the same benchmark**: The benchmark must use the same parameters as the production change. If you changed `thread_sensitive=False` in production, the benchmark must use `thread_sensitive=False` too.
5. **Audit related config**: Infrastructure changes often make existing configuration flags dead or inconsistent. Check all related settings.
6. **Run full test suite**: Infrastructure changes have wide blast radius — all tests must pass.

Do NOT present benchmarking as optional or ask the user whether to benchmark. Always benchmark — that's the whole point.

## Plateau Detection — Domain-Specific

3+ consecutive discards across all strategies = check if remaining issues are:
- I/O-bound by network latency (can't be reduced by better concurrency)
- Already fully concurrent
- Dominated by third-party/framework overhead
- Limited by external service rate limits

If the top 3 remaining issues are all non-optimizable, **stop and report to user** with what's left and why.

## Logging Format

Tab-separated `.codeflash/results.tsv`, 13 columns:

```
commit	target_test	baseline_latency_ms	optimized_latency_ms	latency_change	baseline_throughput	optimized_throughput	throughput_change	concurrency	tests_passed	tests_failed	status	pattern	description
```

- `target_test`: test name, endpoint name, or `micro:<name>`
- `baseline_latency_ms`: baseline p50 latency in milliseconds
- `optimized_latency_ms`: optimized p50 latency in milliseconds
- `latency_change`: percentage improvement (e.g., `-63%` means 63% faster)
- `baseline_throughput`: baseline req/s or ops/s
- `optimized_throughput`: optimized req/s or ops/s
- `throughput_change`: percentage improvement (e.g., `+172%`)
- `concurrency`: number of concurrent operations in the benchmark (e.g., `50`)
- `status`: `keep`, `discard`, or `crash`
- `pattern`: antipattern category (e.g., `sequential-awaits`, `blocking-call`, `await-in-loop`)
