Optimizer Factory for Codeflash — EC2-backed

What this is

A minimal pipeline to run Codeflash optimizations across many Python repositories using on-demand EC2 instances. Configure a CSV, launch from the UI, stream logs directly from the instance, and approve results in Codeflash Staging.

Prerequisites

- AWS account with permissions for EC2 and IAM
- AWS CLI installed and configured (`aws configure` with an IAM user/role)
- GitHub Personal Access Token (classic) with `public_repo` scope
- Codeflash API key from `app.codeflash.ai`
- Python 3.10+ and `venv`

Project Structure

- `config/` - Repository configuration and analysis results
  - `repos.csv` - List of repositories to process with their configuration
  - `analysis/` - LLM analysis results for each repository
  - `jobs.json` - Maps repository URLs to EC2 instance IDs for job tracking
  - `analysis_jobs.json` - Tracks analysis job status and results
  - `custom_run_settings.json` - Configuration schema for custom optimization runs
- `scripts/` - Core optimization and utility scripts
  - `run_optimization.sh` - Main optimization script that forks/clones repos, detects roots, and runs Codeflash
  - `detect_roots.py` - Simple heuristics for detecting module and test root directories
  - `llm_setup_helper.py` - LLM-powered setup assistant for fixing repository dependencies (Deprecated - now using Claude Code CLI directly in run_optimization.sh)
  - `entrypoint.sh` - Container entrypoint for dependency installation and optimization
- `server/` - Web interface and API
  - `app.py` - Flask API serving static UI and EC2 job management
  - `analyzer.py` - Anthropic-powered analyzer to extract per-repository environment configuration
  - `static/` - Plain HTML/CSS/JS UI to manage repositories and jobs
- `tools/` - Local dependencies and utilities
  - `requirements.txt` - Python dependencies for the server (Flask, boto3, paramiko, anthropic)
- `env.example` - Environment template for EC2 configuration and API keys

## Features

### Standard Optimization Runs

- **Repository Management**: Configure and manage multiple repositories through a CSV file
- **EC2 Instance Management**: Launch and manage EC2 instances for optimization jobs
- **Log Streaming**: Real-time log streaming from EC2 instances
- **S3 Log Archiving**: Automatic log archiving to S3 for permanent storage
- **Job Monitoring**: Track optimization progress and status
- **Auto-termination**: Configurable automatic instance termination after job completion

### Custom Run Functionality

- **Dynamic Configuration**: Configure Codeflash optimizations with custom parameters through a web interface
- **Multiple Optimization Modes**:
  - **Single Function**: Optimize individual Python functions
  - **Trace & Optimize**: End-to-end workflow optimization with execution tracing
  - **Optimize All**: Comprehensive codebase optimization
- **Flexible Settings**: Override default module roots, test configurations, and optimization flags
- **Custom pyproject.toml**: Specify custom locations for Codeflash configuration files
- **Real-time Validation**: Validate configurations before execution
- **Comprehensive Logging**: Detailed logging and monitoring for custom runs

For detailed information about the Custom Run functionality, see [docs/CUSTOM_RUN.md](docs/CUSTOM_RUN.md).

## Setup and Installation

1. **Clone the repository**:

   ```bash
   git clone <repository-url>
   cd optimization-factory
   ```

2. **Create a virtual environment**:

   ```bash
   python3 -m venv venv
   ```

3. **Activate the virtual environment**:

   ```bash
   source venv/bin/activate
   ```

4. **Install dependencies**:

   ```bash
   pip install -r tools/requirements.txt
   ```

5. **Configure environment**:
   - Copy `env.example` to `.env`
   - Fill in your actual values for AWS configuration, API keys, and SSH key path

Running the Application

From the project root directory, run:

```bash
python -m server.app
```

Then open the web interface at `http://localhost:5000`.

Step-by-step setup

1. Configure AWS

- Copy `env.example` to `.env` and fill it (AWS region, EC2 key pair, security group, AMI ID, SSH key path).
- **Important for WSL users**: See SSH Key Configuration section below for proper setup.

2. Install dependencies

- `pip install -r tools/requirements.txt`

3. Provide tokens locally

- Set env vars `CODEFLASH_API_KEY` and `GITHUB_TOKEN` in your shell or `.env`.

4. Ensure networking

- The security group must allow outbound HTTPS and inbound SSH from your IP if you want direct access. The instance will reach GitHub and PyPI over the internet.

5. Configure repositories to process

- Edit `config/repos.csv` and add rows:

  repo_url,module_root,tests_root,resource_tier
  https://github.com/psf/requests,requests,tests,small
  https://github.com/pallets/flask,src/flask,tests,medium
  https://github.com/numpy/numpy,numpy,numpy/tests,large
  https://github.com/user/small-util,auto,auto,small

- Columns:
  - `repo_url` — upstream repository URL
  - `module_root`, `tests_root` — path or `auto` to auto-detect
  - `resource_tier` — `small` | `medium` | `large` (selects job definition)

6. Run jobs

- Install local deps:
  - `pip install -r tools/requirements.txt`
- Start server:

- `python -m server.app`
- Open UI: `http://localhost:5000`
- From the UI you can:
  - Add/update/delete repos (edits `config/repos.csv`)
  - Run optimization for a repo or run all (each launches a dedicated EC2 instance)
  - Check job status (instance state and exit code)
  - View logs (tail of `/var/log/codeflash-optimization.log` on the instance)
  - Analyze a repo via LLM (Anthropic) and apply proposed config to CSV

7. Monitor and review

- EC2 console: see instances launching/terminating
- UI logs panel: streams the remote log file
- Codeflash Staging: approve optimizations

Retries and tuning

- If a job fails with OOM, change the `resource_tier` in `config/repos.csv` to a larger tier and re-run the launcher.
- For more automation (e.g., automatic tier escalation), consider adding AWS Step Functions later.

How it works (under the hood)

- The server launches an EC2 instance per job and waits for SSH.
- It uploads `scripts/run_optimization.sh` and `scripts/detect_roots.py`, exports env with analyzer hints, and starts the optimization.
- The job writes logs to `/var/log/codeflash-optimization.log`; the server tails this file.
- A background watcher terminates the instance after completion.

Notes

- Ensure the Codeflash GitHub App is installed for your account/org so forks are covered.
- Provide sufficient EC2 instance size; default is `c7i.2xlarge` but adjust as needed.

LLM-powered Repo Analysis (optional)

- Purpose: Suggest per-repo configuration (module root, tests root, resource tier) and optional safe setup commands.
- Requirements:
  - `ANTHROPIC_API_KEY` set in environment for the server
  - `pip install -r tools/requirements.txt` (includes `anthropic`, `jsonschema`)
- How it works:
  - UI → Analyze (🧠) calls `/api/analyze_repo` and shows results once ready
  - Results are stored as `config/analysis/<org>-<repo>.json`
  - You can selectively apply `module_root`, `tests_root`, and `resource_tier` to the CSV
  - On job submit, if analysis exists, BE passes sanitized overrides to the container via env:
    - `SYSTEM_PACKAGES`: allowlisted apt packages
    - `PRE_INSTALL_CMDS`, `INSTALL_CMDS`, `POST_INSTALL_CMDS`: safe, filtered commands joined with `&&`
    - Non-secret env vars if provided
  - `scripts/entrypoint.sh` executes these overrides before running the default detection path

SSH Key Configuration

**Critical for WSL Users**: If you're running this on Windows Subsystem for Linux (WSL), you must configure your SSH key properly to avoid permission errors.

1. **Copy SSH key to WSL filesystem**:

   ```bash
   # Copy your SSH key from Windows to WSL home directory
   cp /mnt/c/path/to/your/key.pem ~/.ssh/your_key_name.pem
   ```

2. **Set correct permissions**:

   ```bash
   # Set restrictive permissions (required by SSH)
   chmod 600 ~/.ssh/your_key_name.pem
   ```

3. **Update .env file**:
   ```bash
   # Use WSL path, not Windows path
   SSH_KEY_PATH=~/.ssh/your_key_name.pem
   ```

**Why this is necessary**: SSH requires strict file permissions (600) for private keys. Windows file permissions don't translate correctly to WSL, causing "Permissions are too open" errors. By copying the key to the WSL filesystem and setting permissions with `chmod`, you ensure SSH can read the key properly.

**Troubleshooting SSH Issues**:

- If you get "Permissions are too open" error: Ensure the key is in WSL filesystem (`~/.ssh/`) not Windows filesystem (`/mnt/c/`)
- If you get "No such file or directory": Verify the path in `.env` matches the actual key location
- If you get "Permission denied": Check that `chmod 600` was applied successfully with `ls -la ~/.ssh/`

Security and safety

- Commands from LLM are pared down via allowlist; risky patterns are dropped.
- Only non-secret env vars are passed through; secrets stay in AWS Secrets Manager.
- If analysis is unavailable, the system falls back to current heuristic detection.
