codeflash-agent

Monorepo for the Codeflash optimization platform: Python packages, Claude Code plugin, and services.

Case Studies

Active case study data lives in .codeflash/{org}/{project}/ (status, bench scripts, raw data, VM infra). Summaries are built out of .codeflash/ into case-studies/{org}/{project}/.

Active case studies in .codeflash/:

microsoft/typeagent
unstructured/core-product
netflix/metaflow
coveragepy/coveragepy
textualize/rich
pypa/pip

Directory conventions

Target repos live in ~/Desktop/work/{org}_org/{project}:

microsoft_org/typeagent
unstructured_org/core-product
netflix_org/metaflow
coveragepy_org/coveragepy

Optimization flow

Make changes in the target repo on a perf/<description> branch
Run tests locally to verify nothing breaks
Commit and push to the fork
Benchmark on the VM via ssh -A azureuser@<ip> "cd ~/<project> && git fetch origin && ..."
Record results in .codeflash/{org}/{project}/data/results.tsv
Update status.md in .codeflash/{org}/{project}/
Open a PR on the fork with VM benchmark numbers

VM access

VMs use SSH agent forwarding -- always connect with ssh -A:

Project	VM IP	Size	Resource group
core-product	40.65.91.158	Standard_E4s_v5	core-product-BENCH-RG
typeagent	40.65.81.123	Standard_D2s_v5	typeagent-BENCH-RG

If SSH times out, check:

VM is running: az vm start --resource-group <RG> --name <vm>
NSG IP is current: update AllowSSHFromMyIP source address in the Azure portal or via az network nsg rule update

PR strategy

Individual PRs on the fork (KRRT7/<repo>) -- one per optimization on a perf/<description> branch. Each is self-contained with its own benchmark numbers.
Stacked draft PR (optional) on the fork (--base main --head optimization) -- accumulates all optimizations, shows cumulative gain.

Benchmarking

codeflash compare for internal benchmarks (fork PRs) -- worktree-isolated, per-function breakdown, structured markdown. Does NOT handle import time yet -- use hyperfine for that.
hyperfine for upstream PRs and import time measurements -- portable, no codeflash dependency for maintainers to install.
Keep the VM running during optimization sessions -- don't deallocate between benchmarks
Cloud-init must use ASCII only -- Azure CLI chokes on non-ASCII (em dashes, etc.)

Runner convention

Use $RUNNER in docs and scripts to refer to the Python runner. The value depends on context:

Context	`$RUNNER` value	Why
VM benchmark scripts	`.venv/bin/python`	Accuracy -- uv run adds ~50% overhead and 2.5x variance
Upstream PR reproducers	`uv run python`	Portability -- matches how the target team works
Setup / verify steps	`uv run python`	Measurement accuracy doesn't matter

2.9 KiB Raw Blame History