# Experiment Loop — Memory Domain

> Base framework: `../shared/experiment-loop-base.md`

## Reasoning Checklist

Before writing any code, answer these 8 questions. If you can't answer 3-6 concretely, research more before coding.

1. **Category**: What type of allocation? (see Allocation Categories in the agent prompt)
2. **Visible?** Made INSIDE the benchmarked code path, or at import/setup time? Import-time = **skip**.
3. **Reducible?** Can it be made smaller, freed earlier, or avoided?
4. **Persistent?** Does it persist after its operation returns? **Don't assume — verify with micro-bench.**
5. **Exercised?** Does the target test actually execute this code path?
6. **Mechanism**: HOW does your change reduce peak? Be specific.
7. **Production-safe?** Does this hurt throughput, latency, or caching?
8. **Verify cheaply**: Can you validate with a micro-benchmark before the full benchmark run?

## Domain-Specific Loop Steps

**Step 1 — Choose target** sources:
- **Peak profiling**: Highest-memory reducible allocation from profiler output (tracemalloc snapshots or memray stats). This is the primary source of truth.
- **Code-path reading**: Cross-reference profiler output against source to understand WHY the allocation happens. If source looks clean but delta is high, investigate deeper — the overhead is real, the source is just hiding it (e.g., many small attrs per object, per-item dict overhead, shared-reference copies).

Print: `[experiment N] Target: <description> (<category>, <size> MiB, <est. impact>%)`

**Step 6 — Read results**: Print `[experiment N] <before> MiB -> <after> MiB, <delta> MiB`.

**Step 8 — Noise threshold**: If `target_mb` improved <5 MiB, re-run to confirm not noise.

**Step 10 — E2E benchmark (after KEEP)**: If `codeflash compare` is available (check `.codeflash/setup.md`), run `$RUNNER -m codeflash compare <pre-opt-sha> HEAD` for authoritative isolated timing measurement alongside memory profiling. See `../shared/e2e-benchmarks.md`.

**Step 11 — Milestones**: Create `codeflash/<tag>-v<N>` branch. At milestones, run `$RUNNER -m codeflash compare <baseline-sha> HEAD` for cumulative e2e timing. Print `[milestone] vN — <total kept>/<total experiments>, cumulative reduction <X> MiB`.

## Keep/Discard Thresholds

```
Test passed?
├── NO → Fix or discard immediately
└── YES → target_mb improved?
    ├── YES (≥5 MiB) → KEEP
    ├── YES (<5 MiB) → Re-run to confirm not noise
    │   ├── Confirmed → KEEP
    │   └── Noise → DISCARD
    ├── NO, but micro-bench improved >10 MiB or >10% → KEEP (micro-only)
    └── NO → DISCARD
```

## Strategy Rotation

3+ failures on same tier = move to the next higher tier (more memory headroom for optimization: C → B → A → S).

## Plateau Detection — Domain-Specific

**Absolute overhead check (memory-specific)**: After fixing the dominant allocator, re-profile and compare peak memory to input data size. If peak is still >2x the input data, there are more issues — rebuild the candidate list from the new profile and keep going. Don't stop just because the biggest issue is gone.

3+ consecutive discards on a tier = check if top 3 allocations are all irreducible (model weights, framework internals). If >85% of peak is irreducible, **stop and report to user** with what's left and why.

## Logging Format

Tab-separated `.codeflash/results.tsv`, 10 columns:

```
commit	target_test	target_mb	peak_memory_mb	total_allocs	elapsed_s	tests_passed	tests_failed	status	description
```

- `target_test`: test name, `all`, or `micro:<name>`
- `target_mb`: memory of the targeted test — primary keep/discard metric
- `status`: `keep`, `discard`, or `crash`
