codeflash-internal/experiments/optimization-factory/docs/CUSTOM_RUN.md

Custom Run Functionality

The Custom Run feature allows users to configure and execute Codeflash optimizations with custom parameters, settings, and optimization modes. This provides a flexible framework for running Codeflash optimizations with desired settings without modifying the core project configuration.

Overview

The Custom Run functionality enables users to:

• Choose from three optimization modes: single function, trace & optimize, or optimize all
• Configure all Codeflash settings dynamically through a web interface
• Set custom pyproject.toml locations
• Override default module and test roots
• Enable/disable various Codeflash flags and options
• Execute optimizations with custom parameters

Features

1. Optimization Modes

Single Function Optimization

• Purpose: Target and optimize individual Python functions for maximum performance gains
• Use Case: When you want to optimize a specific function in a specific file
• Required Parameters:
  • file_path: Path to the Python file containing the function
  • function_name: Name of the function to optimize
• Command: codeflash --file <file_path> --function <function_name>

Trace & Optimize E2E Workflows

• Purpose: End-to-end optimization of entire Python workflows with execution tracing
• Use Case: When you want to optimize an entire script or workflow
• Required Parameters:
  • script_path: Path to the Python script to trace and optimize
• Optional Parameters:
  • trace_file: Custom output file for trace data
  • tracer_timeout: Maximum time in seconds to trace the workflow
  • trace_only: Only perform tracing, do not optimize
• Command: codeflash optimize <script_path>

Optimize All Codebase

• Purpose: Automatically optimize all codepaths in your project
• Use Case: When you want to optimize your entire codebase
• Optional Parameters:
  • target_directory: Specific directory to optimize (defaults to entire codebase)
  • benchmark: Run benchmarks during optimization
• Command: codeflash --all [target_directory]

2. Configuration Options

Basic Codeflash Configuration

• Module Root: The Python module you want Codeflash to optimize
• Tests Root: Directory where your tests are located
• Test Framework: pytest or unittest
• Benchmarks Root: Directory where benchmarks are located (required for --benchmark)
• Ignore Paths: List of paths to ignore during optimization
• Pytest Command: Custom pytest command with arguments
• Formatter Commands: Commands to run code formatters/linters
• Disable Imports Sorting: Disable automatic import organization
• Disable Telemetry: Disable telemetry data collection

Optimization Flags

• Async Mode: Run Codeflash commands asynchronously (requires codeflash[asyncio])
• Verbose Output: Enable verbose logging
• No Pull Request: Optimize locally without opening GitHub PRs
• Run Benchmarks: Run benchmarks during optimization
• Trace Only: Only perform tracing, do not optimize
• Tracer Timeout: Maximum time in seconds for tracing

Advanced Settings

• Custom pyproject.toml Location: Specify custom path to pyproject.toml
• Custom Trace File Path: Specify custom output file for trace data
• Max Function Count: Maximum number of times to trace a single function
• Replay Test Path: Path to existing replay test file

Usage

Web Interface

1. Access Custom Run: Click the "⚙️ Custom Run" button next to any repository in the main interface
2. Configure Settings: Fill out the custom run modal with your desired settings
3. Validate Configuration: The system will validate your configuration and show any errors or warnings
4. Execute: Click "Execute Custom Run" to start the optimization

API Endpoints

Get Custom Run Settings

GET /api/custom-run/settings

Returns the available custom run configuration schema.

Validate Custom Run Configuration

POST /api/custom-run/validate
Content-Type: application/json

{
  "optimization_mode": "single_function",
  "config": {
    "module_root": "src",
    "tests_root": "tests",
    "file_path": "src/main.py",
    "function_name": "process_data"
  },
  "flags": {
    "async": true,
    "verbose": true,
    "no_pr": false
  },
  "advanced": {
    "pyproject_location": "custom/pyproject.toml"
  }
}

Execute Custom Run

POST /api/custom-run/execute
Content-Type: application/json

{
  "repo_url": "https://github.com/user/repo",
  "optimization_mode": "single_function",
  "config": { ... },
  "flags": { ... },
  "advanced": { ... }
}

Get Custom Run Status

GET /api/custom-run/status/<instance_id>

Returns the current status of a custom optimization run.

Configuration File

The custom run settings are defined in config/custom_run_settings.json. This file contains:

• Codeflash Optimization Modes: Available optimization modes and their descriptions
• pyproject.toml Settings: All configurable Codeflash settings with descriptions and types
• Optimization Flags: Available Codeflash flags and their options
• Advanced Settings: Advanced configuration options
• UI Sections: Organization of the web interface

Implementation Details

Backend

The custom run functionality is implemented through several components:

1. API Endpoints (server/app.py):

   • /api/custom-run/settings: Returns configuration schema
   • /api/custom-run/validate: Validates user configuration
   • /api/custom-run/execute: Executes custom optimization
   • /api/custom-run/status/<instance_id>: Returns run status

2. Configuration Processing (server/app.py):

   • Uploads custom configuration to EC2 instance
   • Sets environment variables for custom settings
   • Modifies instance setup for custom runs

3. Script Integration (scripts/run_optimization.sh):

   • Detects custom run mode
   • Loads custom configuration
   • Overrides default settings
   • Enhances Claude Code CLI prompts with custom run context
   • Executes appropriate Codeflash commands

Frontend

The custom run interface is implemented in server/static/app.js:

1. Modal Management: Dynamic modal creation and management
2. Form Handling: Dynamic form generation based on configuration schema
3. Validation: Real-time configuration validation
4. Execution: Custom run execution with progress tracking

Logging and Monitoring

Custom runs include comprehensive logging:

• Configuration Loading: Logs when custom configuration is loaded
• Parameter Override: Logs when default parameters are overridden
• Command Building: Logs the built Codeflash command
• Execution: Logs execution start, progress, and completion
• Error Handling: Logs errors with detailed context

Claude Code CLI Integration

The custom run functionality integrates with Claude Code CLI for intelligent setup assistance:

• Enhanced Prompts: Custom run context is automatically added to Claude Code CLI prompts
• Mode-Specific Guidance: Claude receives specific instructions based on the optimization mode
• Target Awareness: Claude knows about target files, functions, and scripts for focused setup
• Dependency Management: Claude ensures all required dependencies are installed for the specific optimization mode
• Configuration Validation: Claude verifies that custom configurations are properly set up

The Claude Code CLI integration ensures that custom runs have the same intelligent setup assistance as standard runs, but with awareness of the specific custom configuration requirements.

Examples

Single Function Optimization

{
  "optimization_mode": "single_function",
  "config": {
    "module_root": "src",
    "tests_root": "tests",
    "test_framework": "pytest",
    "file_path": "src/calculations.py",
    "function_name": "calculate_fibonacci"
  },
  "flags": {
    "async": true,
    "verbose": true,
    "no_pr": false
  }
}

Trace & Optimize Workflow

{
  "optimization_mode": "trace_and_optimize",
  "config": {
    "module_root": "src",
    "tests_root": "tests",
    "script_path": "scripts/train_model.py"
  },
  "flags": {
    "async": true,
    "verbose": true,
    "trace_only": false,
    "tracer_timeout": 300
  },
  "advanced": {
    "trace_file": "custom_trace.trace"
  }
}

Optimize All Codebase

{
  "optimization_mode": "optimize_all",
  "config": {
    "module_root": "src",
    "tests_root": "tests",
    "benchmarks_root": "benchmarks"
  },
  "flags": {
    "async": true,
    "verbose": true,
    "benchmark": true
  },
  "advanced": {
    "target_directory": "src/core"
  }
}

Best Practices

1. Start Simple: Begin with basic configurations and add complexity as needed
2. Validate First: Always validate your configuration before executing
3. Use Appropriate Mode: Choose the optimization mode that best fits your use case
4. Monitor Progress: Use the status endpoint to monitor long-running optimizations
5. Check Logs: Review logs for detailed execution information and debugging

Troubleshooting

Common Issues

1. Missing Required Parameters: Ensure all required parameters for your chosen optimization mode are provided
2. Invalid File Paths: Verify that file paths exist and are accessible
3. Configuration Errors: Use the validation endpoint to check for configuration issues
4. Execution Failures: Check logs for detailed error information

Debug Information

• Configuration Logs: Check logs for configuration loading and parameter override information
• Command Logs: Review the built Codeflash command for correctness
• Execution Logs: Monitor execution progress and any errors
• Status Endpoint: Use the status endpoint to get current run information

Integration with Existing Features

The Custom Run functionality integrates seamlessly with existing project features:

• Repository Management: Uses existing repository configuration and management
• EC2 Instance Management: Leverages existing EC2 instance lifecycle management
• Log Archiving: Integrates with existing S3 log archiving system
• Monitoring: Uses existing job monitoring and status tracking
• Security: Inherits existing security and access control mechanisms

This ensures that custom runs benefit from all existing infrastructure while providing additional flexibility and customization options.