codeflash-internal/experiments/optimization-factory/README.md

Optimizer Factory for Codeflash — EC2-backed

What this is

A minimal pipeline to run Codeflash optimizations across many Python repositories using on-demand EC2 instances. Configure a CSV, launch from the UI, stream logs directly from the instance, and approve results in Codeflash Staging.

Prerequisites

• AWS account with permissions for EC2 and IAM
• AWS CLI installed and configured (aws configure with an IAM user/role)
• GitHub Personal Access Token (classic) with public_repo scope
• Codeflash API key from app.codeflash.ai
• Python 3.10+ and venv

Project Structure

• config/ - Repository configuration and analysis results
  • repos.csv - List of repositories to process with their configuration
  • analysis/ - LLM analysis results for each repository
  • jobs.json - Maps repository URLs to EC2 instance IDs for job tracking
  • analysis_jobs.json - Tracks analysis job status and results
  • custom_run_settings.json - Configuration schema for custom optimization runs
• scripts/ - Core optimization and utility scripts
  • run_optimization.sh - Main optimization script that forks/clones repos, detects roots, and runs Codeflash
  • detect_roots.py - Simple heuristics for detecting module and test root directories
  • llm_setup_helper.py - LLM-powered setup assistant for fixing repository dependencies (Deprecated - now using Claude Code CLI directly in run_optimization.sh)
  • entrypoint.sh - Container entrypoint for dependency installation and optimization
• server/ - Web interface and API
  • app.py - Flask API serving static UI and EC2 job management
  • analyzer.py - Anthropic-powered analyzer to extract per-repository environment configuration
  • static/ - Plain HTML/CSS/JS UI to manage repositories and jobs
• tools/ - Local dependencies and utilities
  • requirements.txt - Python dependencies for the server (Flask, boto3, paramiko, anthropic)
• env.example - Environment template for EC2 configuration and API keys

Features

Standard Optimization Runs

• Repository Management: Configure and manage multiple repositories through a CSV file
• EC2 Instance Management: Launch and manage EC2 instances for optimization jobs
• Log Streaming: Real-time log streaming from EC2 instances
• S3 Log Archiving: Automatic log archiving to S3 for permanent storage
• Job Monitoring: Track optimization progress and status
• Auto-termination: Configurable automatic instance termination after job completion

Custom Run Functionality

• Dynamic Configuration: Configure Codeflash optimizations with custom parameters through a web interface
• Multiple Optimization Modes:
  • Single Function: Optimize individual Python functions
  • Trace & Optimize: End-to-end workflow optimization with execution tracing
  • Optimize All: Comprehensive codebase optimization
• Flexible Settings: Override default module roots, test configurations, and optimization flags
• Custom pyproject.toml: Specify custom locations for Codeflash configuration files
• Real-time Validation: Validate configurations before execution
• Comprehensive Logging: Detailed logging and monitoring for custom runs

For detailed information about the Custom Run functionality, see docs/CUSTOM_RUN.md.

Setup and Installation

1. Clone the repository:

   git clone <repository-url>
   cd optimization-factory
   

2. Create a virtual environment:

   python3 -m venv venv
   

3. Activate the virtual environment:

   source venv/bin/activate
   

4. Install dependencies:

   pip install -r tools/requirements.txt
   

5. Configure environment:

   • Copy env.example to .env
   • Fill in your actual values for AWS configuration, API keys, and SSH key path

Running the Application

From the project root directory, run:

python -m server.app

Then open the web interface at http://localhost:5000.

Step-by-step setup

1. Configure AWS

• Copy env.example to .env and fill it (AWS region, EC2 key pair, security group, AMI ID, SSH key path).
• Important for WSL users: See SSH Key Configuration section below for proper setup.

2. Install dependencies

• pip install -r tools/requirements.txt

3. Provide tokens locally

• Set env vars CODEFLASH_API_KEY and GITHUB_TOKEN in your shell or .env.

4. Ensure networking

• The security group must allow outbound HTTPS and inbound SSH from your IP if you want direct access. The instance will reach GitHub and PyPI over the internet.

5. Configure repositories to process

• Edit config/repos.csv and add rows:

  repo_url,module_root,tests_root,resource_tier https://github.com/psf/requests,requests,tests,small https://github.com/pallets/flask,src/flask,tests,medium https://github.com/numpy/numpy,numpy,numpy/tests,large https://github.com/user/small-util,auto,auto,small

• Columns:

  • repo_url — upstream repository URL
  • module_root, tests_root — path or auto to auto-detect
  • resource_tier — small | medium | large (selects job definition)

6. Run jobs

• Install local deps:

  • pip install -r tools/requirements.txt

• Start server:

• python -m server.app

• Open UI: http://localhost:5000

• From the UI you can:

  • Add/update/delete repos (edits config/repos.csv)
  • Run optimization for a repo or run all (each launches a dedicated EC2 instance)
  • Check job status (instance state and exit code)
  • View logs (tail of /var/log/codeflash-optimization.log on the instance)
  • Analyze a repo via LLM (Anthropic) and apply proposed config to CSV

7. Monitor and review

• EC2 console: see instances launching/terminating
• UI logs panel: streams the remote log file
• Codeflash Staging: approve optimizations

Retries and tuning

• If a job fails with OOM, change the resource_tier in config/repos.csv to a larger tier and re-run the launcher.
• For more automation (e.g., automatic tier escalation), consider adding AWS Step Functions later.

How it works (under the hood)

• The server launches an EC2 instance per job and waits for SSH.
• It uploads scripts/run_optimization.sh and scripts/detect_roots.py, exports env with analyzer hints, and starts the optimization.
• The job writes logs to /var/log/codeflash-optimization.log; the server tails this file.
• A background watcher terminates the instance after completion.

Notes

• Ensure the Codeflash GitHub App is installed for your account/org so forks are covered.
• Provide sufficient EC2 instance size; default is c7i.2xlarge but adjust as needed.

LLM-powered Repo Analysis (optional)

• Purpose: Suggest per-repo configuration (module root, tests root, resource tier) and optional safe setup commands.
• Requirements:
  • ANTHROPIC_API_KEY set in environment for the server
  • pip install -r tools/requirements.txt (includes anthropic, jsonschema)
• How it works:
  • UI → Analyze (🧠) calls /api/analyze_repo and shows results once ready
  • Results are stored as config/analysis/<org>-<repo>.json
  • You can selectively apply module_root, tests_root, and resource_tier to the CSV
  • On job submit, if analysis exists, BE passes sanitized overrides to the container via env:
    • SYSTEM_PACKAGES: allowlisted apt packages
    • PRE_INSTALL_CMDS, INSTALL_CMDS, POST_INSTALL_CMDS: safe, filtered commands joined with &&
    • Non-secret env vars if provided
  • scripts/entrypoint.sh executes these overrides before running the default detection path

SSH Key Configuration

Critical for WSL Users: If you're running this on Windows Subsystem for Linux (WSL), you must configure your SSH key properly to avoid permission errors.

1. Copy SSH key to WSL filesystem:

   # Copy your SSH key from Windows to WSL home directory
   cp /mnt/c/path/to/your/key.pem ~/.ssh/your_key_name.pem
   

2. Set correct permissions:

   # Set restrictive permissions (required by SSH)
   chmod 600 ~/.ssh/your_key_name.pem
   

3. Update .env file:

   # Use WSL path, not Windows path
   SSH_KEY_PATH=~/.ssh/your_key_name.pem
   

Why this is necessary: SSH requires strict file permissions (600) for private keys. Windows file permissions don't translate correctly to WSL, causing "Permissions are too open" errors. By copying the key to the WSL filesystem and setting permissions with chmod, you ensure SSH can read the key properly.

Troubleshooting SSH Issues:

• If you get "Permissions are too open" error: Ensure the key is in WSL filesystem (~/.ssh/) not Windows filesystem (/mnt/c/)
• If you get "No such file or directory": Verify the path in .env matches the actual key location
• If you get "Permission denied": Check that chmod 600 was applied successfully with ls -la ~/.ssh/

Security and safety

• Commands from LLM are pared down via allowlist; risky patterns are dropped.
• Only non-secret env vars are passed through; secrets stay in AWS Secrets Manager.
• If analysis is unavailable, the system falls back to current heuristic detection.