mirror of
https://github.com/codeflash-ai/codeflash-internal.git
synced 2026-05-04 18:25:18 +00:00
Feature/optimizer updates (#1871)
- Introduced new API endpoints for downloading individual archived logs and all logs for a repository as a ZIP file. - Enhanced the CSV handling in the application to include new fields for auto-termination and analysis configuration. - Updated the frontend to support new features, including a modal for auto-termination settings and improved UI for log management. - Updated styles for modals and tables to enhance visual consistency and usability." - Added Dumpy security code in the FE with "123456" - added ability to customize and view the repo analysis by the LLM results - adding deployment folder for the whole deployment flow along with the dogs - fixed some permission errors and ssh access by www-data user while deployment - added s3 bucket saving logs based --------- Co-authored-by: Ashraf <ashraf@rapiddata.io> Co-authored-by: Mohamed Ashraf <mohamedashrraf222@gmail.com> Co-authored-by: Kevin Turcios <106575910+KRRT7@users.noreply.github.com> Co-authored-by: Sarthak Agarwal <sarthak.saga@gmail.com>
This commit is contained in:
mashraf-222
GitHub
parent
93d612340c
commit
ebd1755359
15 changed files with 9578 additions and 1409 deletions
|
|
@ -10,20 +10,94 @@ Prerequisites
|
|||
- AWS CLI installed and configured (`aws configure` with an IAM user/role)
|
||||
- GitHub Personal Access Token (classic) with `public_repo` scope
|
||||
- Codeflash API key from `app.codeflash.ai`
|
||||
- Python 3.10+
|
||||
- Python 3.10+ and `venv`
|
||||
|
||||
Project layout
|
||||
Project Structure
|
||||
|
||||
- scripts/
|
||||
- run_optimization.sh — forks/clones, detects roots, runs Codeflash
|
||||
- detect_roots.py — simple heuristics for module/tests roots
|
||||
- server/
|
||||
- app.py — Flask API serving static UI and EC2 job actions
|
||||
- static/ — plain HTML/CSS/JS UI to manage repos and jobs
|
||||
- analyzer.py — Anthropic-powered analyzer to extract per-repo env config
|
||||
- config/repos.csv — list of repos and resource tiers
|
||||
- tools/requirements.txt — local deps for the server (Flask, boto3, paramiko)
|
||||
- env.example — environment template for EC2 and secrets
|
||||
- `config/` - Repository configuration and analysis results
|
||||
- `repos.csv` - List of repositories to process with their configuration
|
||||
- `analysis/` - LLM analysis results for each repository
|
||||
- `jobs.json` - Maps repository URLs to EC2 instance IDs for job tracking
|
||||
- `analysis_jobs.json` - Tracks analysis job status and results
|
||||
- `custom_run_settings.json` - Configuration schema for custom optimization runs
|
||||
- `scripts/` - Core optimization and utility scripts
|
||||
- `run_optimization.sh` - Main optimization script that forks/clones repos, detects roots, and runs Codeflash
|
||||
- `detect_roots.py` - Simple heuristics for detecting module and test root directories
|
||||
- `llm_setup_helper.py` - LLM-powered setup assistant for fixing repository dependencies (Deprecated - now using Claude Code CLI directly in run_optimization.sh)
|
||||
- `entrypoint.sh` - Container entrypoint for dependency installation and optimization
|
||||
- `server/` - Web interface and API
|
||||
- `app.py` - Flask API serving static UI and EC2 job management
|
||||
- `analyzer.py` - Anthropic-powered analyzer to extract per-repository environment configuration
|
||||
- `static/` - Plain HTML/CSS/JS UI to manage repositories and jobs
|
||||
- `tools/` - Local dependencies and utilities
|
||||
- `requirements.txt` - Python dependencies for the server (Flask, boto3, paramiko, anthropic)
|
||||
- `env.example` - Environment template for EC2 configuration and API keys
|
||||
|
||||
## Features
|
||||
|
||||
### Standard Optimization Runs
|
||||
|
||||
- **Repository Management**: Configure and manage multiple repositories through a CSV file
|
||||
- **EC2 Instance Management**: Launch and manage EC2 instances for optimization jobs
|
||||
- **Log Streaming**: Real-time log streaming from EC2 instances
|
||||
- **S3 Log Archiving**: Automatic log archiving to S3 for permanent storage
|
||||
- **Job Monitoring**: Track optimization progress and status
|
||||
- **Auto-termination**: Configurable automatic instance termination after job completion
|
||||
|
||||
### Custom Run Functionality
|
||||
|
||||
- **Dynamic Configuration**: Configure Codeflash optimizations with custom parameters through a web interface
|
||||
- **Multiple Optimization Modes**:
|
||||
- **Single Function**: Optimize individual Python functions
|
||||
- **Trace & Optimize**: End-to-end workflow optimization with execution tracing
|
||||
- **Optimize All**: Comprehensive codebase optimization
|
||||
- **Flexible Settings**: Override default module roots, test configurations, and optimization flags
|
||||
- **Custom pyproject.toml**: Specify custom locations for Codeflash configuration files
|
||||
- **Real-time Validation**: Validate configurations before execution
|
||||
- **Comprehensive Logging**: Detailed logging and monitoring for custom runs
|
||||
|
||||
For detailed information about the Custom Run functionality, see [docs/CUSTOM_RUN.md](docs/CUSTOM_RUN.md).
|
||||
|
||||
## Setup and Installation
|
||||
|
||||
1. **Clone the repository**:
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd optimization-factory
|
||||
```
|
||||
|
||||
2. **Create a virtual environment**:
|
||||
|
||||
```bash
|
||||
python3 -m venv venv
|
||||
```
|
||||
|
||||
3. **Activate the virtual environment**:
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
```
|
||||
|
||||
4. **Install dependencies**:
|
||||
|
||||
```bash
|
||||
pip install -r tools/requirements.txt
|
||||
```
|
||||
|
||||
5. **Configure environment**:
|
||||
- Copy `env.example` to `.env`
|
||||
- Fill in your actual values for AWS configuration, API keys, and SSH key path
|
||||
|
||||
Running the Application
|
||||
|
||||
From the project root directory, run:
|
||||
|
||||
```bash
|
||||
python -m server.app
|
||||
```
|
||||
|
||||
Then open the web interface at `http://localhost:5000`.
|
||||
|
||||
Step-by-step setup
|
||||
|
||||
|
|
@ -65,7 +139,7 @@ Step-by-step setup
|
|||
- `pip install -r tools/requirements.txt`
|
||||
- Start server:
|
||||
|
||||
- `python server/app.py`
|
||||
- `python -m server.app`
|
||||
- Open UI: `http://localhost:5000`
|
||||
- From the UI you can:
|
||||
- Add/update/delete repos (edits `config/repos.csv`)
|
||||
|
|
|
|||
602
experiments/optimization-factory/deployment/README.md
Normal file
602
experiments/optimization-factory/deployment/README.md
Normal file
|
|
@ -0,0 +1,602 @@
|
|||
# Optimization Factory Deployment Guide
|
||||
|
||||
This deployment solution provides two distinct deployment paths for the Optimization Factory application, with the ability to seamlessly switch between them on an already-deployed machine.
|
||||
|
||||
## Overview
|
||||
|
||||
The deployment script (`deploy.sh`) offers two modes:
|
||||
|
||||
- **Simple Mode**: Quick setup for testing, accessing the application directly via the server's IP address and a configurable port (e.g., `http://203.0.113.55:8080`)
|
||||
- **Production Mode**: Full-featured deployment with a custom domain and free SSL certificate (e.g., `https://www.example.com`)
|
||||
|
||||
The script intelligently handles cleanup of the old mode before setting up the new one, making it safe to switch between deployment modes.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Ubuntu 22.04 EC2 instance with root/sudo access
|
||||
- For production mode: A registered domain name
|
||||
- For production mode: DNS management access to create A records
|
||||
- **AWS Configuration**: Properly configured AWS credentials and permissions
|
||||
|
||||
### AWS Configuration Setup
|
||||
|
||||
Before deploying the application, you must configure AWS credentials and permissions:
|
||||
|
||||
#### Option 1: IAM Role (Recommended for EC2)
|
||||
|
||||
1. **Create an IAM Role**:
|
||||
|
||||
- Go to AWS IAM Console → Roles → Create Role
|
||||
- Select "EC2" as the service
|
||||
- Attach policies with necessary permissions (e.g., `S3FullAccess`, `EC2FullAccess`, etc.)
|
||||
- Name your role (e.g., `OptimizationFactoryRole`)
|
||||
|
||||
2. **Attach the role to your EC2 instance**:
|
||||
- Go to EC2 Console → Instances
|
||||
- Select your instance → Actions → Security → Modify IAM Role
|
||||
- Choose the role you created
|
||||
- Apply changes
|
||||
|
||||
#### Option 2: AWS Credentials File
|
||||
|
||||
1. **Install AWS CLI** (if not already installed):
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install awscli -y
|
||||
```
|
||||
|
||||
2. **Configure AWS credentials**:
|
||||
|
||||
```bash
|
||||
aws configure
|
||||
```
|
||||
|
||||
Enter your:
|
||||
|
||||
- AWS Access Key ID
|
||||
- AWS Secret Access Key
|
||||
- Default region (e.g., `us-east-1`)
|
||||
- Default output format (e.g., `json`)
|
||||
|
||||
3. **Verify configuration**:
|
||||
```bash
|
||||
aws sts get-caller-identity
|
||||
```
|
||||
|
||||
#### Required AWS Permissions
|
||||
|
||||
Ensure your AWS credentials/role have the following permissions:
|
||||
|
||||
- **S3**: Full access to buckets used by the application
|
||||
- **EC2**: Instance management permissions
|
||||
- **IAM**: Basic read permissions (if needed)
|
||||
- **CloudWatch**: Log access (if using logging)
|
||||
|
||||
#### Environment Variables
|
||||
|
||||
Add these to your `.env` file:
|
||||
|
||||
```bash
|
||||
# AWS Configuration
|
||||
AWS_ACCESS_KEY_ID=your_access_key
|
||||
AWS_SECRET_ACCESS_KEY=your_secret_key
|
||||
AWS_DEFAULT_REGION=us-east-1
|
||||
```
|
||||
|
||||
### SSH Key Configuration Setup
|
||||
|
||||
The application needs SSH access to connect to optimization instances. The deployment script automatically handles SSH key setup, but you need to ensure your SSH key is available.
|
||||
|
||||
#### Required SSH Key
|
||||
|
||||
You need an SSH key pair that matches the key pair used when launching EC2 instances for optimization jobs. The deployment script will automatically detect the SSH key name from your `.env` file's `SSH_KEY_PATH` setting.
|
||||
|
||||
#### Automatic SSH Key Setup (Recommended)
|
||||
|
||||
The deployment script automatically:
|
||||
|
||||
1. **Creates SSH directory**: Sets up `/var/www/.ssh/` for the `www-data` user
|
||||
2. **Sets proper permissions**: Ensures security with correct file permissions (700 for directory, 600 for key)
|
||||
3. **Auto-detects key**: Extracts the SSH key name from your `.env` file's `SSH_KEY_PATH` and looks for it in common locations:
|
||||
- `/home/ubuntu/.ssh/[your-key-name].pem`
|
||||
- `/root/.ssh/[your-key-name].pem`
|
||||
4. **Copies key**: Automatically copies the key to the correct location for `www-data` user
|
||||
5. **Updates .env file**: Automatically sets `SSH_KEY_PATH=/var/www/.ssh/[your-key-name].pem` in your `.env` file
|
||||
6. **Provides guidance**: Shows helpful instructions if the key isn't found
|
||||
|
||||
#### Manual SSH Key Setup (If Needed)
|
||||
|
||||
If the automatic setup doesn't find your SSH key, you can manually copy it:
|
||||
|
||||
```bash
|
||||
# Create SSH directory for www-data user
|
||||
sudo mkdir -p /var/www/.ssh
|
||||
|
||||
# Copy your SSH key (replace with your actual key path and name)
|
||||
sudo cp /path/to/your/[your-key-name].pem /var/www/.ssh/
|
||||
|
||||
# Set proper ownership and permissions
|
||||
sudo chown -R www-data:www-data /var/www/.ssh
|
||||
sudo chmod 700 /var/www/.ssh
|
||||
sudo chmod 600 /var/www/.ssh/[your-key-name].pem
|
||||
```
|
||||
|
||||
#### Verify SSH Key Setup
|
||||
|
||||
After deployment, verify the SSH key is properly configured:
|
||||
|
||||
```bash
|
||||
# Check if the key exists and has correct permissions
|
||||
sudo ls -la /var/www/.ssh/
|
||||
|
||||
# Test SSH connection (replace with actual optimization instance IP and your key name)
|
||||
sudo -u www-data ssh -i /var/www/.ssh/[your-key-name].pem ubuntu@optimization-instance-ip
|
||||
```
|
||||
|
||||
#### SSH Key Requirements
|
||||
|
||||
- **Key Name**: Any name (automatically detected from `SSH_KEY_PATH` in your `.env` file)
|
||||
- **Key Format**: PEM format (standard AWS EC2 key pair format)
|
||||
- **Permissions**: 600 (readable only by owner)
|
||||
- **Location**: Must be accessible to the `www-data` user
|
||||
- **Matching**: Must match the key pair used when launching optimization EC2 instances
|
||||
|
||||
## Path A: Simple IP-Based Deployment
|
||||
|
||||
This mode is perfect for testing, development, or quick deployments where you want direct access via IP address.
|
||||
|
||||
### Step-by-Step Instructions
|
||||
|
||||
1. **SSH into your server**:
|
||||
|
||||
```bash
|
||||
ssh -i your-key.pem ubuntu@your-server-ip
|
||||
```
|
||||
|
||||
2. **Clone the repository**:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/your-org/optimization-factory.git
|
||||
cd optimization-factory
|
||||
```
|
||||
|
||||
3. **Navigate to deployment directory and make script executable**:
|
||||
|
||||
```bash
|
||||
cd deployment
|
||||
chmod +x deploy.sh
|
||||
```
|
||||
|
||||
4. **Create the environment file**:
|
||||
|
||||
```bash
|
||||
cp ../env.example ../.env
|
||||
nano ../.env
|
||||
```
|
||||
|
||||
**Important**: Set the `PORT` variable in the `.env` file. For example:
|
||||
|
||||
```
|
||||
PORT=8080
|
||||
```
|
||||
|
||||
Also configure your other required variables like `CODEFLASH_API_KEY`, `GITHUB_TOKEN`, etc.
|
||||
|
||||
5. **Run the deployment script in simple mode**:
|
||||
|
||||
```bash
|
||||
sudo ./deploy.sh simple
|
||||
```
|
||||
|
||||
6. **Access your application**:
|
||||
The script will output the URL where your application is accessible. It will be something like:
|
||||
```
|
||||
http://203.0.113.55:8080
|
||||
```
|
||||
|
||||
### Expected Output
|
||||
|
||||
```
|
||||
[INFO] Starting deployment in 'simple' mode...
|
||||
[INFO] Performing common setup...
|
||||
[INFO] Installing common dependencies...
|
||||
[INFO] Copying application from current project directory...
|
||||
[INFO] Application directory exists, preserving data files...
|
||||
[INFO] Backed up jobs.json
|
||||
[INFO] Backed up config directory
|
||||
[INFO] Backed up .env file
|
||||
[INFO] Restoring preserved data files...
|
||||
[SUCCESS] Restored jobs.json
|
||||
[SUCCESS] Restored config directory
|
||||
[SUCCESS] Restored .env file
|
||||
[INFO] Cleaned up temporary backup files
|
||||
[INFO] Setting proper permissions...
|
||||
[INFO] Creating symlinks for data file access...
|
||||
[SUCCESS] Created symlink for jobs.json
|
||||
[SUCCESS] Created symlink for config directory
|
||||
[SUCCESS] Created symlink for .env file
|
||||
[INFO] Setting up Python virtual environment...
|
||||
[INFO] Installing Python dependencies...
|
||||
[SUCCESS] AWS credentials verified successfully
|
||||
[INFO] Setting up SSH keys for www-data user...
|
||||
[INFO] Found SSH key name in .env: Deployed_Optimization_Factory_key_pair.pem
|
||||
[INFO] Found SSH key 'Deployed_Optimization_Factory_key_pair.pem' in ubuntu user directory, copying to www-data...
|
||||
[SUCCESS] SSH key copied successfully
|
||||
[INFO] Updating .env file with correct SSH key path...
|
||||
[SUCCESS] SSH key path updated in .env file: /var/www/.ssh/Deployed_Optimization_Factory_key_pair.pem
|
||||
[INFO] Configuring for Simple IP-Based Deployment...
|
||||
[INFO] Cleaning up any existing production configuration...
|
||||
[INFO] Using port: 8080
|
||||
[INFO] Configuring firewall...
|
||||
[INFO] Creating systemd service...
|
||||
[INFO] Starting service...
|
||||
[SUCCESS] Simple deployment completed!
|
||||
[SUCCESS] Your application is now accessible at: http://203.0.113.55:8080
|
||||
[INFO] Service status: active
|
||||
[SUCCESS] Deployment completed successfully!
|
||||
```
|
||||
|
||||
## Path B: Production Deployment (Domain + SSL)
|
||||
|
||||
This mode provides a production-ready deployment with Nginx reverse proxy, custom domain, and SSL certificate.
|
||||
|
||||
### Step-by-Step Instructions
|
||||
|
||||
#### Step 3.1: Point Your Domain
|
||||
|
||||
1. **Get your server's public IP**:
|
||||
|
||||
```bash
|
||||
curl ifconfig.me
|
||||
```
|
||||
|
||||
2. **Create an A record in your DNS settings**:
|
||||
- Log into your domain registrar or DNS provider
|
||||
- Create an A record pointing your domain (e.g., `www.example.com`) to your server's public IP
|
||||
- Wait for DNS propagation (usually 5-15 minutes)
|
||||
|
||||
#### Step 3.2: Run the Deployment Script
|
||||
|
||||
1. **SSH into your server**:
|
||||
|
||||
```bash
|
||||
ssh -i your-key.pem ubuntu@your-server-ip
|
||||
```
|
||||
|
||||
2. **Clone the repository** (if not already done):
|
||||
|
||||
```bash
|
||||
git clone https://github.com/your-org/optimization-factory.git
|
||||
cd optimization-factory
|
||||
```
|
||||
|
||||
3. **Navigate to deployment directory and make script executable**:
|
||||
|
||||
```bash
|
||||
cd deployment
|
||||
chmod +x deploy.sh
|
||||
```
|
||||
|
||||
4. **Create the environment file**:
|
||||
|
||||
```bash
|
||||
cp ../env.example ../.env
|
||||
nano ../.env
|
||||
```
|
||||
|
||||
Configure your required variables. Note: The `PORT` variable is not used in production mode as Nginx handles the port.
|
||||
|
||||
5. **Run the deployment script in production mode**:
|
||||
```bash
|
||||
sudo ./deploy.sh production
|
||||
```
|
||||
|
||||
#### Step 3.3: Configure Your Domain in Nginx
|
||||
|
||||
1. **Edit the Nginx configuration**:
|
||||
|
||||
```bash
|
||||
sudo nano /etc/nginx/sites-available/optimization-factory
|
||||
```
|
||||
|
||||
2. **Replace the placeholder domain**:
|
||||
Find the line:
|
||||
|
||||
```
|
||||
server_name your_domain.com; # <-- CHANGE THIS TO YOUR ACTUAL DOMAIN
|
||||
```
|
||||
|
||||
Replace `your_domain.com` with your actual domain:
|
||||
|
||||
```
|
||||
server_name www.example.com;
|
||||
```
|
||||
|
||||
3. **Restart Nginx**:
|
||||
|
||||
```bash
|
||||
sudo systemctl restart nginx
|
||||
```
|
||||
|
||||
4. **Test your domain**:
|
||||
Visit `http://www.example.com` in your browser. You should see your application.
|
||||
|
||||
#### Step 3.4: Enable HTTPS with Certbot
|
||||
|
||||
1. **Install Certbot**:
|
||||
|
||||
```bash
|
||||
sudo apt install certbot python3-certbot-nginx -y
|
||||
```
|
||||
|
||||
2. **Obtain SSL certificate**:
|
||||
|
||||
```bash
|
||||
sudo certbot --nginx -d www.example.com
|
||||
```
|
||||
|
||||
3. **Follow the prompts**:
|
||||
|
||||
- Enter your email address
|
||||
- Agree to terms of service
|
||||
- Choose whether to share your email with EFF
|
||||
- Certbot will automatically configure Nginx for HTTPS
|
||||
|
||||
4. **Test automatic renewal**:
|
||||
|
||||
```bash
|
||||
sudo certbot renew --dry-run
|
||||
```
|
||||
|
||||
5. **Access your secure application**:
|
||||
Visit `https://www.example.com` - you should see your application with a valid SSL certificate.
|
||||
|
||||
## How to Update the Application
|
||||
|
||||
To deploy new code or updates:
|
||||
|
||||
1. **SSH into your server**:
|
||||
|
||||
```bash
|
||||
ssh -i your-key.pem ubuntu@your-server-ip
|
||||
```
|
||||
|
||||
2. **Navigate to the deployment directory**:
|
||||
|
||||
```bash
|
||||
cd optimization-factory/deployment
|
||||
```
|
||||
|
||||
3. **Re-run the deployment script with your current mode**:
|
||||
|
||||
```bash
|
||||
# If you're currently in simple mode:
|
||||
sudo ./deploy.sh simple
|
||||
|
||||
# If you're currently in production mode:
|
||||
sudo ./deploy.sh production
|
||||
```
|
||||
|
||||
### Data Preservation During Updates
|
||||
|
||||
The deployment script automatically preserves your data during updates:
|
||||
|
||||
- ✅ **jobs.json**: Preserves running job states and instance mappings
|
||||
- ✅ **config/analysis/**: Preserves all analysis results and configurations
|
||||
- ✅ **config/repos.csv**: Preserves repository configurations
|
||||
- ✅ **.env**: Preserves your environment variables and settings
|
||||
|
||||
### Symlink Access
|
||||
|
||||
After deployment, you can access data files from your original repository directory:
|
||||
|
||||
```bash
|
||||
# Access jobs.json from your repository
|
||||
cat server/jobs.json
|
||||
|
||||
# Access config files
|
||||
ls -la config/
|
||||
|
||||
# Edit repository configurations
|
||||
nano config/repos.csv
|
||||
|
||||
# View analysis results
|
||||
ls -la config/analysis/
|
||||
```
|
||||
|
||||
The script creates symlinks so you can edit files in your familiar repository location while the application uses the deployed version.
|
||||
|
||||
The script will automatically pull the latest code and restart the services.
|
||||
|
||||
## Switching Between Deployment Modes
|
||||
|
||||
This is a powerful feature that allows you to reconfigure your server from one mode to another without manual cleanup.
|
||||
|
||||
### Switching from Simple to Production Mode
|
||||
|
||||
If you're currently running in simple mode and want to switch to production mode:
|
||||
|
||||
1. **Ensure your domain is pointing to your server** (see Step 3.1 above)
|
||||
|
||||
2. **Run the production deployment**:
|
||||
```bash
|
||||
sudo ./deploy.sh production
|
||||
```
|
||||
|
||||
The script will automatically:
|
||||
|
||||
- Stop and disable Nginx (if running)
|
||||
- Remove the simple mode firewall rules
|
||||
- Install and configure Nginx
|
||||
- Set up the production firewall rules
|
||||
- Configure the application to use Unix socket instead of TCP port
|
||||
- Start both the application and Nginx services
|
||||
|
||||
3. **Configure your domain** (see Step 3.3 above)
|
||||
|
||||
4. **Enable SSL** (see Step 3.4 above)
|
||||
|
||||
### Switching from Production to Simple Mode
|
||||
|
||||
If you're currently running in production mode and want to switch to simple mode:
|
||||
|
||||
1. **Configure your .env file** with the desired PORT:
|
||||
|
||||
```bash
|
||||
sudo nano /opt/optimization-factory/.env
|
||||
```
|
||||
|
||||
Set: `PORT=8080` (or your preferred port)
|
||||
|
||||
2. **Run the simple deployment**:
|
||||
```bash
|
||||
sudo ./deploy.sh simple
|
||||
```
|
||||
|
||||
The script will automatically:
|
||||
|
||||
- Stop and disable Nginx
|
||||
- Remove Nginx configuration files
|
||||
- Remove production firewall rules
|
||||
- Configure the application to use TCP port instead of Unix socket
|
||||
- Set up simple mode firewall rules
|
||||
- Start the application service
|
||||
|
||||
3. **Access your application** at `http://your-server-ip:8080`
|
||||
|
||||
## Managing Services & Viewing Logs
|
||||
|
||||
### Service Management Commands
|
||||
|
||||
**Check service status**:
|
||||
|
||||
```bash
|
||||
sudo systemctl status optimization-factory
|
||||
sudo systemctl status nginx # (production mode only)
|
||||
```
|
||||
|
||||
**Restart services**:
|
||||
|
||||
```bash
|
||||
sudo systemctl restart optimization-factory
|
||||
sudo systemctl restart nginx # (production mode only)
|
||||
```
|
||||
|
||||
**Stop services**:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop optimization-factory
|
||||
sudo systemctl stop nginx # (production mode only)
|
||||
```
|
||||
|
||||
**Enable/disable services**:
|
||||
|
||||
```bash
|
||||
sudo systemctl enable optimization-factory
|
||||
sudo systemctl disable optimization-factory
|
||||
```
|
||||
|
||||
### Viewing Logs
|
||||
|
||||
**Application logs**:
|
||||
|
||||
```bash
|
||||
# View recent logs
|
||||
sudo journalctl -u optimization-factory
|
||||
|
||||
# Follow logs in real-time
|
||||
sudo journalctl -u optimization-factory -f
|
||||
|
||||
# View logs from last boot
|
||||
sudo journalctl -u optimization-factory -b
|
||||
|
||||
# View logs with timestamps
|
||||
sudo journalctl -u optimization-factory --since "1 hour ago"
|
||||
```
|
||||
|
||||
**Nginx logs** (production mode only):
|
||||
|
||||
```bash
|
||||
# Access logs
|
||||
sudo tail -f /var/log/nginx/access.log
|
||||
|
||||
# Error logs
|
||||
sudo tail -f /var/log/nginx/error.log
|
||||
|
||||
# Combined logs
|
||||
sudo tail -f /var/log/nginx/access.log /var/log/nginx/error.log
|
||||
```
|
||||
|
||||
**System logs**:
|
||||
|
||||
```bash
|
||||
# View system logs
|
||||
sudo journalctl -f
|
||||
|
||||
# View boot logs
|
||||
sudo journalctl -b
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**If the application won't start**:
|
||||
|
||||
1. Check the service status: `sudo systemctl status optimization-factory`
|
||||
2. View logs: `sudo journalctl -u optimization-factory -f`
|
||||
3. Check the .env file: `sudo nano /opt/optimization-factory/.env`
|
||||
4. Verify Python dependencies: `sudo /opt/optimization-factory/venv/bin/pip list`
|
||||
|
||||
**If optimization jobs fail with SSH errors**:
|
||||
|
||||
1. Check SSH key exists: `sudo ls -la /var/www/.ssh/`
|
||||
2. Verify key permissions: `sudo ls -la /var/www/.ssh/Deployed_Optimization_Factory_key_pair.pem`
|
||||
3. Test SSH connection: `sudo -u www-data ssh -i /var/www/.ssh/Deployed_Optimization_Factory_key_pair.pem ubuntu@instance-ip`
|
||||
4. Ensure key matches EC2 key pair: Check AWS EC2 Console → Key Pairs
|
||||
5. Re-run deployment script: `sudo ./deploy.sh simple` (or `production`)
|
||||
|
||||
**If AWS credentials are not working**:
|
||||
|
||||
1. Check AWS credentials: `aws sts get-caller-identity`
|
||||
2. Verify .env file has AWS variables: `sudo nano /opt/optimization-factory/.env`
|
||||
3. Check IAM role attachment: AWS EC2 Console → Instances → Your Instance → Security → IAM Role
|
||||
4. Test AWS CLI: `aws ec2 describe-instances --region us-east-2`
|
||||
|
||||
**If Nginx won't start** (production mode):
|
||||
|
||||
1. Check Nginx status: `sudo systemctl status nginx`
|
||||
2. Test configuration: `sudo nginx -t`
|
||||
3. View error logs: `sudo tail -f /var/log/nginx/error.log`
|
||||
|
||||
**If SSL certificate issues** (production mode):
|
||||
|
||||
1. Check certificate status: `sudo certbot certificates`
|
||||
2. Test renewal: `sudo certbot renew --dry-run`
|
||||
3. View Certbot logs: `sudo tail -f /var/log/letsencrypt/letsencrypt.log`
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- The application runs as the `www-data` user for security
|
||||
- Firewall rules are automatically configured based on the deployment mode
|
||||
- Security headers are included in the Nginx configuration
|
||||
- SSL certificates are automatically renewed by Certbot
|
||||
- Regular security updates should be applied: `sudo apt update && sudo apt upgrade`
|
||||
|
||||
## File Locations
|
||||
|
||||
- Application directory: `/opt/optimization-factory`
|
||||
- Environment file: `/opt/optimization-factory/.env`
|
||||
- Systemd service: `/etc/systemd/system/optimization-factory.service`
|
||||
- Nginx configuration: `/etc/nginx/sites-available/optimization-factory`
|
||||
- Application logs: `journalctl -u optimization-factory`
|
||||
- Nginx logs: `/var/log/nginx/`
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
|
||||
1. Check the logs first using the commands above
|
||||
2. Verify your configuration files
|
||||
3. Ensure all prerequisites are met
|
||||
4. Check that your domain DNS is properly configured (production mode)
|
||||
5. Verify firewall rules are correct for your deployment mode
|
||||
566
experiments/optimization-factory/deployment/deploy.sh
Normal file
566
experiments/optimization-factory/deployment/deploy.sh
Normal file
|
|
@ -0,0 +1,566 @@
|
|||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Function to print colored output
|
||||
print_status() {
|
||||
echo -e "${BLUE}[INFO]${NC} $1"
|
||||
}
|
||||
|
||||
print_success() {
|
||||
echo -e "${GREEN}[SUCCESS]${NC} $1"
|
||||
}
|
||||
|
||||
print_warning() {
|
||||
echo -e "${YELLOW}[WARNING]${NC} $1"
|
||||
}
|
||||
|
||||
print_error() {
|
||||
echo -e "${RED}[ERROR]${NC} $1"
|
||||
}
|
||||
|
||||
# Check if running as root
|
||||
if [[ $EUID -ne 0 ]]; then
|
||||
print_error "This script must be run as root (use sudo)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if argument is provided
|
||||
if [ $# -eq 0 ]; then
|
||||
print_error "Usage: sudo ./deploy.sh [simple|production]"
|
||||
echo ""
|
||||
echo "Available modes:"
|
||||
echo " simple - Deploy for direct IP access (e.g., http://203.0.113.55:8080)"
|
||||
echo " production - Deploy with Nginx reverse proxy for domain access"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
MODE=$1
|
||||
APP_DIR="/opt/optimization-factory"
|
||||
SERVICE_NAME="optimization-factory"
|
||||
|
||||
print_status "Starting deployment in '$MODE' mode..."
|
||||
|
||||
# Common setup for both modes
|
||||
print_status "Performing common setup..."
|
||||
|
||||
# Update package list
|
||||
apt-get update -y
|
||||
|
||||
# Install common dependencies
|
||||
print_status "Installing common dependencies..."
|
||||
apt-get install -y git python3-pip python3-venv curl awscli
|
||||
|
||||
# Copy or update the application from current directory
|
||||
print_status "Copying application from current project directory..."
|
||||
# Get the current script directory and go up one level to get the project root
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||
|
||||
# Pre-copy: remove repo symlinks that could cause circular references on copy
|
||||
print_status "Cleaning repository symlinks (pre-copy)..."
|
||||
if [ -L "$PROJECT_ROOT/server/jobs.json" ]; then
|
||||
rm -f "$PROJECT_ROOT/server/jobs.json"
|
||||
print_status "Removed repo symlink: server/jobs.json"
|
||||
fi
|
||||
if [ -L "$PROJECT_ROOT/config" ]; then
|
||||
rm -f "$PROJECT_ROOT/config"
|
||||
print_status "Removed repo symlink: config"
|
||||
fi
|
||||
if [ -L "$PROJECT_ROOT/.env" ]; then
|
||||
rm -f "$PROJECT_ROOT/.env"
|
||||
print_status "Removed repo symlink: .env"
|
||||
fi
|
||||
|
||||
# --- START: NEW DATA PRESERVATION AND FILE COPY LOGIC ---
|
||||
|
||||
# First, check if the source .env file exists. This is a critical pre-deployment check.
|
||||
if [ ! -f "$PROJECT_ROOT/.env" ]; then
|
||||
print_error ".env file not found in your project directory ($PROJECT_ROOT/.env)"
|
||||
print_error "Please create it by copying 'env.example' and configuring your settings."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PRESERVE_DIR="" # Initialize preservation directory variable
|
||||
|
||||
# If the application is already deployed, preserve its live data.
|
||||
if [ -d "$APP_DIR" ]; then
|
||||
print_status "Existing deployment found. Preserving live data..."
|
||||
PRESERVE_DIR="/tmp/optimization-factory-preserve-$(date +%s)"
|
||||
mkdir -p "$PRESERVE_DIR"
|
||||
|
||||
# Move the 'config' directory (which contains 'analysis' folder) if it's a real directory
|
||||
if [ -d "$APP_DIR/config" ] && [ ! -L "$APP_DIR/config" ]; then
|
||||
mv "$APP_DIR/config" "$PRESERVE_DIR/config"
|
||||
print_status "Preserved 'config' directory."
|
||||
fi
|
||||
|
||||
# Move the 'jobs.json' file if it's a real file
|
||||
if [ -f "$APP_DIR/server/jobs.json" ] && [ ! -L "$APP_DIR/server/jobs.json" ]; then
|
||||
mkdir -p "$PRESERVE_DIR/server"
|
||||
mv "$APP_DIR/server/jobs.json" "$PRESERVE_DIR/server/jobs.json"
|
||||
print_status "Preserved 'jobs.json' data file."
|
||||
fi
|
||||
|
||||
# Remove the old application directory for a clean copy
|
||||
print_status "Removing old application code..."
|
||||
rm -rf "$APP_DIR"
|
||||
fi
|
||||
|
||||
# Copy the entire project to the target directory. The `/.` ensures hidden files like .env are included.
|
||||
print_status "Copying new application files to $APP_DIR..."
|
||||
mkdir -p "$APP_DIR"
|
||||
cp -r "$PROJECT_ROOT"/. "$APP_DIR"/ || { print_error "Failed to copy application files"; exit 1; }
|
||||
cd "$APP_DIR"
|
||||
|
||||
# Restore the preserved live data, overwriting the template versions from the source repo
|
||||
if [ -n "$PRESERVE_DIR" ]; then
|
||||
print_status "Restoring live data..."
|
||||
|
||||
if [ -d "$PRESERVE_DIR/config" ]; then
|
||||
rm -rf "$APP_DIR/config" # Remove the template config dir just copied
|
||||
mv "$PRESERVE_DIR/config" "$APP_DIR/config"
|
||||
print_success "Restored 'config' directory."
|
||||
fi
|
||||
|
||||
if [ -f "$PRESERVE_DIR/server/jobs.json" ]; then
|
||||
# Ensure the server directory exists before moving the file back
|
||||
mkdir -p "$APP_DIR/server"
|
||||
mv "$PRESERVE_DIR/server/jobs.json" "$APP_DIR/server/jobs.json"
|
||||
print_success "Restored 'jobs.json' file."
|
||||
fi
|
||||
|
||||
# Clean up the temporary preservation directory
|
||||
rm -rf "$PRESERVE_DIR"
|
||||
print_status "Cleaned up temporary backup files."
|
||||
fi
|
||||
|
||||
# --- END: NEW DATA PRESERVATION AND FILE COPY LOGIC ---
|
||||
|
||||
# Set proper ownership and permissions for www-data user
|
||||
print_status "Setting proper permissions..."
|
||||
chown -R www-data:www-data "$APP_DIR"
|
||||
chmod -R 755 "$APP_DIR"
|
||||
|
||||
# Ensure www-data can write to necessary directories
|
||||
mkdir -p "$APP_DIR/server/logs"
|
||||
mkdir -p "$APP_DIR/config/analysis"
|
||||
chown -R www-data:www-data "$APP_DIR/server/logs" "$APP_DIR/config"
|
||||
chmod -R 775 "$APP_DIR/server/logs" "$APP_DIR/config"
|
||||
|
||||
|
||||
# Create virtual environment
|
||||
print_status "Setting up Python virtual environment..."
|
||||
if [ ! -d "$APP_DIR/venv" ]; then
|
||||
python3 -m venv "$APP_DIR/venv"
|
||||
fi
|
||||
|
||||
# Activate virtual environment and install dependencies
|
||||
print_status "Installing Python dependencies..."
|
||||
source "$APP_DIR/venv/bin/activate"
|
||||
pip install --upgrade pip
|
||||
pip install -r "$APP_DIR/tools/requirements.txt"
|
||||
pip install gunicorn
|
||||
|
||||
# Check if .env file exists
|
||||
if [ ! -f "$APP_DIR/.env" ]; then
|
||||
print_warning ".env file not found at $APP_DIR/.env"
|
||||
print_warning "Please create it by copying env.example and configuring your settings"
|
||||
print_warning "For simple mode, ensure PORT is set (e.g., PORT=8080)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Verify AWS configuration
|
||||
print_status "Verifying AWS configuration..."
|
||||
|
||||
# Check if AWS credentials are in .env file
|
||||
AWS_ACCESS_KEY_ID=$(grep "^AWS_ACCESS_KEY_ID=" "$APP_DIR/.env" | cut -d'=' -f2 | tr -d '"' | tr -d "'")
|
||||
AWS_SECRET_ACCESS_KEY=$(grep "^AWS_SECRET_ACCESS_KEY=" "$APP_DIR/.env" | cut -d'=' -f2 | tr -d '"' | tr -d "'")
|
||||
|
||||
if [ ! -z "$AWS_ACCESS_KEY_ID" ] && [ ! -z "$AWS_SECRET_ACCESS_KEY" ]; then
|
||||
print_success "AWS credentials found in .env file"
|
||||
|
||||
# Test AWS credentials by setting environment variables
|
||||
export AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID"
|
||||
export AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY"
|
||||
export AWS_DEFAULT_REGION=$(grep "^AWS_DEFAULT_REGION=" "$APP_DIR/.env" | cut -d'=' -f2 | tr -d '"' | tr -d "'" || echo "us-east-1")
|
||||
|
||||
if aws sts get-caller-identity >/dev/null 2>&1; then
|
||||
print_success "AWS credentials verified successfully"
|
||||
else
|
||||
print_warning "AWS credentials in .env file are invalid"
|
||||
print_warning "Please check your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY values"
|
||||
fi
|
||||
elif aws sts get-caller-identity >/dev/null 2>&1; then
|
||||
print_success "AWS credentials verified successfully (system-wide configuration)"
|
||||
else
|
||||
print_warning "AWS credentials not configured or invalid"
|
||||
print_warning "Please configure AWS credentials using one of these methods:"
|
||||
print_warning "1. Attach an IAM role to your EC2 instance (recommended)"
|
||||
print_warning "2. Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your .env file"
|
||||
print_warning "Continuing deployment, but the application may not function properly without AWS access..."
|
||||
fi
|
||||
|
||||
# SSH Key Setup for www-data user
|
||||
print_status "Setting up SSH keys for www-data user..."
|
||||
|
||||
# Extract SSH key name from existing SSH_KEY_PATH in .env file
|
||||
SSH_KEY_NAME=""
|
||||
if grep -q "^SSH_KEY_PATH=" "$APP_DIR/.env"; then
|
||||
SSH_KEY_PATH=$(grep "^SSH_KEY_PATH=" "$APP_DIR/.env" | cut -d'=' -f2 | tr -d '"' | tr -d "'")
|
||||
SSH_KEY_NAME=$(basename "$SSH_KEY_PATH")
|
||||
print_status "Found SSH key name in .env: $SSH_KEY_NAME"
|
||||
else
|
||||
print_warning "SSH_KEY_PATH not found in .env file, using default key name"
|
||||
SSH_KEY_NAME="Deployed_Optimization_Factory_key_pair.pem"
|
||||
fi
|
||||
|
||||
# Create .ssh directory for www-data user
|
||||
mkdir -p /var/www/.ssh
|
||||
chown -R www-data:www-data /var/www/.ssh
|
||||
chmod 700 /var/www/.ssh
|
||||
|
||||
SSH_KEY_COPIED=false
|
||||
|
||||
# Check if SSH key exists in ubuntu user's directory
|
||||
if [ -f "/home/ubuntu/.ssh/$SSH_KEY_NAME" ]; then
|
||||
print_status "Found SSH key '$SSH_KEY_NAME' in ubuntu user directory, copying to www-data..."
|
||||
cp "/home/ubuntu/.ssh/$SSH_KEY_NAME" "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
chown www-data:www-data "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
chmod 600 "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
SSH_KEY_COPIED=true
|
||||
print_success "SSH key copied successfully"
|
||||
|
||||
# Convert RSA key to OpenSSH format if needed
|
||||
print_status "Checking SSH key format compatibility..."
|
||||
if head -1 "/var/www/.ssh/$SSH_KEY_NAME" | grep -q "BEGIN RSA PRIVATE KEY"; then
|
||||
print_status "Converting RSA key to OpenSSH format for Paramiko compatibility..."
|
||||
|
||||
# Create a backup of the original key
|
||||
cp "/var/www/.ssh/$SSH_KEY_NAME" "/var/www/.ssh/${SSH_KEY_NAME}.backup"
|
||||
|
||||
# Method 1: Try ssh-keygen conversion
|
||||
if ssh-keygen -p -m OpenSSH -f "/var/www/.ssh/$SSH_KEY_NAME" -N "" 2>/dev/null; then
|
||||
print_success "SSH key converted to OpenSSH format successfully (method 1)"
|
||||
else
|
||||
print_warning "Method 1 failed, trying alternative conversion..."
|
||||
|
||||
# Method 2: Try ssh-keygen with different parameters
|
||||
if ssh-keygen -p -m OpenSSH -f "/var/www/.ssh/${SSH_KEY_NAME}.backup" -N "" 2>/dev/null; then
|
||||
mv "/var/www/.ssh/${SSH_KEY_NAME}.backup" "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
print_success "SSH key converted to OpenSSH format successfully (method 2)"
|
||||
else
|
||||
print_warning "Method 2 failed, trying manual conversion..."
|
||||
|
||||
# Method 3: Generate new OpenSSH key (safer than manual conversion)
|
||||
if ssh-keygen -t rsa -b 4096 -f "/var/www/.ssh/$SSH_KEY_NAME" -N '' -m OpenSSH 2>/dev/null; then
|
||||
print_success "SSH key converted to OpenSSH format successfully (method 3 - new key generated)"
|
||||
print_warning "Note: A new SSH key was generated. You may need to add the public key to your EC2 instances."
|
||||
print_status "Public key: $(ssh-keygen -y -f "/var/www/.ssh/$SSH_KEY_NAME" 2>/dev/null || echo 'Unable to extract public key')"
|
||||
else
|
||||
print_warning "All conversion methods failed, but continuing with original key"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Clean up backup
|
||||
rm -f "/var/www/.ssh/${SSH_KEY_NAME}.backup"
|
||||
|
||||
# Ensure correct ownership after conversion
|
||||
chown www-data:www-data "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
chmod 600 "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
|
||||
elif head -1 "/var/www/.ssh/$SSH_KEY_NAME" | grep -q "BEGIN.*PRIVATE KEY"; then
|
||||
print_success "SSH key format is already compatible"
|
||||
else
|
||||
print_warning "SSH key format may be invalid - check the key file"
|
||||
fi
|
||||
|
||||
# Verify the key format and permissions
|
||||
print_status "Verifying SSH key setup..."
|
||||
if [ -f "/var/www/.ssh/$SSH_KEY_NAME" ]; then
|
||||
KEY_PERMS=$(stat -c "%a" "/var/www/.ssh/$SSH_KEY_NAME")
|
||||
KEY_OWNER=$(stat -c "%U:%G" "/var/www/.ssh/$SSH_KEY_NAME")
|
||||
print_status "SSH key permissions: $KEY_PERMS, owner: $KEY_OWNER"
|
||||
|
||||
# Test SSH key with ssh-keygen
|
||||
if ssh-keygen -y -f "/var/www/.ssh/$SSH_KEY_NAME" >/dev/null 2>&1; then
|
||||
print_success "SSH key validation passed"
|
||||
else
|
||||
print_warning "SSH key validation failed - may cause connection issues"
|
||||
fi
|
||||
fi
|
||||
elif [ -f "/root/.ssh/$SSH_KEY_NAME" ]; then
|
||||
print_status "Found SSH key '$SSH_KEY_NAME' in root directory, copying to www-data..."
|
||||
cp "/root/.ssh/$SSH_KEY_NAME" "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
chown www-data:www-data "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
chmod 600 "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
SSH_KEY_COPIED=true
|
||||
print_success "SSH key copied successfully"
|
||||
|
||||
# Convert RSA key to OpenSSH format if needed
|
||||
print_status "Checking SSH key format compatibility..."
|
||||
if head -1 "/var/www/.ssh/$SSH_KEY_NAME" | grep -q "BEGIN RSA PRIVATE KEY"; then
|
||||
print_status "Converting RSA key to OpenSSH format for Paramiko compatibility..."
|
||||
|
||||
# Create a backup of the original key
|
||||
cp "/var/www/.ssh/$SSH_KEY_NAME" "/var/www/.ssh/${SSH_KEY_NAME}.backup"
|
||||
|
||||
# Method 1: Try ssh-keygen conversion
|
||||
if ssh-keygen -p -m OpenSSH -f "/var/www/.ssh/$SSH_KEY_NAME" -N "" 2>/dev/null; then
|
||||
print_success "SSH key converted to OpenSSH format successfully (method 1)"
|
||||
else
|
||||
print_warning "Method 1 failed, trying alternative conversion..."
|
||||
|
||||
# Method 2: Try ssh-keygen with different parameters
|
||||
if ssh-keygen -p -m OpenSSH -f "/var/www/.ssh/${SSH_KEY_NAME}.backup" -N "" 2>/dev/null; then
|
||||
mv "/var/www/.ssh/${SSH_KEY_NAME}.backup" "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
print_success "SSH key converted to OpenSSH format successfully (method 2)"
|
||||
else
|
||||
print_warning "Method 2 failed, trying manual conversion..."
|
||||
|
||||
# Method 3: Generate new OpenSSH key (safer than manual conversion)
|
||||
if ssh-keygen -t rsa -b 4096 -f "/var/www/.ssh/$SSH_KEY_NAME" -N '' -m OpenSSH 2>/dev/null; then
|
||||
print_success "SSH key converted to OpenSSH format successfully (method 3 - new key generated)"
|
||||
print_warning "Note: A new SSH key was generated. You may need to add the public key to your EC2 instances."
|
||||
print_status "Public key: $(ssh-keygen -y -f "/var/www/.ssh/$SSH_KEY_NAME" 2>/dev/null || echo 'Unable to extract public key')"
|
||||
else
|
||||
print_warning "All conversion methods failed, but continuing with original key"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Clean up backup
|
||||
rm -f "/var/www/.ssh/${SSH_KEY_NAME}.backup"
|
||||
|
||||
# Ensure correct ownership after conversion
|
||||
chown www-data:www-data "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
chmod 600 "/var/www/.ssh/$SSH_KEY_NAME"
|
||||
|
||||
elif head -1 "/var/www/.ssh/$SSH_KEY_NAME" | grep -q "BEGIN.*PRIVATE KEY"; then
|
||||
print_success "SSH key format is already compatible"
|
||||
else
|
||||
print_warning "SSH key format may be invalid - check the key file"
|
||||
fi
|
||||
|
||||
# Verify the key format and permissions
|
||||
print_status "Verifying SSH key setup..."
|
||||
if [ -f "/var/www/.ssh/$SSH_KEY_NAME" ]; then
|
||||
KEY_PERMS=$(stat -c "%a" "/var/www/.ssh/$SSH_KEY_NAME")
|
||||
KEY_OWNER=$(stat -c "%U:%G" "/var/www/.ssh/$SSH_KEY_NAME")
|
||||
print_status "SSH key permissions: $KEY_PERMS, owner: $KEY_OWNER"
|
||||
|
||||
# Test SSH key with ssh-keygen
|
||||
if ssh-keygen -y -f "/var/www/.ssh/$SSH_KEY_NAME" >/dev/null 2>&1; then
|
||||
print_success "SSH key validation passed"
|
||||
else
|
||||
print_warning "SSH key validation failed - may cause connection issues"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
print_warning "SSH key '$SSH_KEY_NAME' not found in common locations:"
|
||||
print_warning "- /home/ubuntu/.ssh/"
|
||||
print_warning "- /root/.ssh/"
|
||||
print_warning "Please ensure your SSH key is available for the application to connect to optimization instances"
|
||||
print_warning "You can copy it manually:"
|
||||
print_warning " sudo cp /path/to/your/$SSH_KEY_NAME /var/www/.ssh/$SSH_KEY_NAME"
|
||||
print_warning " sudo chown www-data:www-data /var/www/.ssh/$SSH_KEY_NAME"
|
||||
print_warning " sudo chmod 600 /var/www/.ssh/$SSH_KEY_NAME"
|
||||
print_warning " # If you get 'encountered RSA key, expected OPENSSH key' error, convert the key:"
|
||||
print_warning " sudo ssh-keygen -p -m OpenSSH -f /var/www/.ssh/$SSH_KEY_NAME -N ''"
|
||||
fi
|
||||
|
||||
# Update .env file to use absolute SSH key path for www-data user
|
||||
if [ "$SSH_KEY_COPIED" = true ]; then
|
||||
print_status "Updating .env file with correct SSH key path..."
|
||||
|
||||
# Update SSH_KEY_PATH to use absolute path for www-data user
|
||||
NEW_SSH_KEY_PATH="/var/www/.ssh/$SSH_KEY_NAME"
|
||||
if grep -q "^SSH_KEY_PATH=" "$APP_DIR/.env"; then
|
||||
# Update existing SSH_KEY_PATH
|
||||
sed -i "s|^SSH_KEY_PATH=.*|SSH_KEY_PATH=$NEW_SSH_KEY_PATH|" "$APP_DIR/.env"
|
||||
else
|
||||
# Add SSH_KEY_PATH if it doesn't exist
|
||||
echo "SSH_KEY_PATH=$NEW_SSH_KEY_PATH" >> "$APP_DIR/.env"
|
||||
fi
|
||||
|
||||
print_success "SSH key path updated in .env file: $NEW_SSH_KEY_PATH"
|
||||
fi
|
||||
|
||||
# Mode-specific configuration
|
||||
case $MODE in
|
||||
"simple")
|
||||
print_status "Configuring for Simple IP-Based Deployment..."
|
||||
|
||||
# Cleanup any existing production configuration
|
||||
print_status "Cleaning up any existing production configuration..."
|
||||
systemctl stop nginx 2>/dev/null || true
|
||||
systemctl disable nginx 2>/dev/null || true
|
||||
rm -f /etc/nginx/sites-enabled/optimization-factory 2>/dev/null || true
|
||||
rm -f /etc/nginx/sites-available/optimization-factory 2>/dev/null || true
|
||||
ufw delete allow 'Nginx Full' 2>/dev/null || true
|
||||
|
||||
# Read PORT from .env file
|
||||
PORT=$(grep "^PORT=" "$APP_DIR/.env" | cut -d'=' -f2 | tr -d '"' | tr -d "'")
|
||||
if [ -z "$PORT" ]; then
|
||||
print_error "PORT not found in .env file. Please set PORT=8080 (or your preferred port) in $APP_DIR/.env"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
print_status "Using port: $PORT"
|
||||
|
||||
# Configure firewall
|
||||
print_status "Configuring firewall..."
|
||||
ufw --force enable
|
||||
ufw allow OpenSSH
|
||||
ufw allow "$PORT/tcp"
|
||||
|
||||
# Create systemd service file
|
||||
print_status "Creating systemd service..."
|
||||
cat > /etc/systemd/system/$SERVICE_NAME.service << EOF
|
||||
[Unit]
|
||||
Description=Optimization Factory Web Application
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=www-data
|
||||
Group=www-data
|
||||
WorkingDirectory=$APP_DIR
|
||||
EnvironmentFile=$APP_DIR/.env
|
||||
ExecStart=$APP_DIR/venv/bin/gunicorn --workers 3 --timeout 300 --bind 0.0.0.0:$PORT server.app:app
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
# Reload systemd and start service
|
||||
print_status "Starting service..."
|
||||
systemctl daemon-reload
|
||||
systemctl enable $SERVICE_NAME
|
||||
systemctl restart $SERVICE_NAME
|
||||
|
||||
# Get server IP
|
||||
SERVER_IP=$(curl -s ifconfig.me || curl -s ipinfo.io/ip || echo "YOUR_SERVER_IP")
|
||||
|
||||
print_success "Simple deployment completed!"
|
||||
print_success "Your application is now accessible at: http://$SERVER_IP:$PORT"
|
||||
print_status "Service status: $(systemctl is-active $SERVICE_NAME)"
|
||||
;;
|
||||
|
||||
"production")
|
||||
print_status "Configuring for Production Deployment with Nginx..."
|
||||
|
||||
# Cleanup any existing simple configuration
|
||||
print_status "Cleaning up any existing simple configuration..."
|
||||
if [ -f "$APP_DIR/.env" ]; then
|
||||
SIMPLE_PORT=$(grep "^PORT=" "$APP_DIR/.env" | cut -d'=' -f2 | tr -d '"' | tr -d "'")
|
||||
if [ ! -z "$SIMPLE_PORT" ]; then
|
||||
ufw delete allow "$SIMPLE_PORT/tcp" 2>/dev/null || true
|
||||
fi
|
||||
fi
|
||||
|
||||
# Install Nginx
|
||||
print_status "Installing Nginx..."
|
||||
apt-get install -y nginx
|
||||
|
||||
# Configure firewall
|
||||
print_status "Configuring firewall..."
|
||||
ufw --force enable
|
||||
ufw allow OpenSSH
|
||||
ufw allow 'Nginx Full'
|
||||
|
||||
# Create systemd service file
|
||||
print_status "Creating systemd service..."
|
||||
cat > /etc/systemd/system/$SERVICE_NAME.service << EOF
|
||||
[Unit]
|
||||
Description=Optimization Factory Web Application
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=www-data
|
||||
Group=www-data
|
||||
WorkingDirectory=$APP_DIR
|
||||
EnvironmentFile=$APP_DIR/.env
|
||||
ExecStart=$APP_DIR/venv/bin/gunicorn --workers 3 --bind unix:/run/gunicorn.sock -m 007 server.app:app
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
# Create Nginx configuration
|
||||
print_status "Configuring Nginx..."
|
||||
cat > /etc/nginx/sites-available/optimization-factory << EOF
|
||||
server {
|
||||
listen 80;
|
||||
server_name your_domain.com; # <-- CHANGE THIS TO YOUR ACTUAL DOMAIN
|
||||
|
||||
location / {
|
||||
proxy_pass http://unix:/run/gunicorn.sock;
|
||||
proxy_set_header Host \$host;
|
||||
proxy_set_header X-Real-IP \$remote_addr;
|
||||
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto \$scheme;
|
||||
proxy_connect_timeout 60s;
|
||||
proxy_send_timeout 60s;
|
||||
proxy_read_timeout 60s;
|
||||
}
|
||||
|
||||
# Security headers
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header X-XSS-Protection "1; mode=block" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header Referrer-Policy "no-referrer-when-downgrade" always;
|
||||
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
|
||||
}
|
||||
EOF
|
||||
|
||||
# Enable the site
|
||||
ln -sf /etc/nginx/sites-available/optimization-factory /etc/nginx/sites-enabled/
|
||||
rm -f /etc/nginx/sites-enabled/default
|
||||
|
||||
# Test Nginx configuration
|
||||
print_status "Testing Nginx configuration..."
|
||||
nginx -t
|
||||
|
||||
# Reload systemd and start services
|
||||
print_status "Starting services..."
|
||||
systemctl daemon-reload
|
||||
systemctl enable $SERVICE_NAME
|
||||
systemctl enable nginx
|
||||
systemctl restart $SERVICE_NAME
|
||||
systemctl restart nginx
|
||||
|
||||
print_success "Production deployment completed!"
|
||||
print_success "Next steps:"
|
||||
print_status "1. Update your domain DNS to point to this server's IP"
|
||||
print_status "2. Edit /etc/nginx/sites-available/optimization-factory and replace 'your_domain.com' with your actual domain"
|
||||
print_status "3. Restart Nginx: sudo systemctl restart nginx"
|
||||
print_status "4. Install Certbot and enable SSL:"
|
||||
print_status " sudo apt install certbot python3-certbot-nginx"
|
||||
print_status " sudo certbot --nginx -d your_domain.com"
|
||||
print_status "Service status: $(systemctl is-active $SERVICE_NAME)"
|
||||
print_status "Nginx status: $(systemctl is-active nginx)"
|
||||
;;
|
||||
|
||||
*)
|
||||
print_error "Invalid mode: $MODE"
|
||||
print_error "Usage: sudo ./deploy.sh [simple|production]"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
print_success "Deployment completed successfully!"
|
||||
print_status "To view logs: sudo journalctl -u $SERVICE_NAME -f"
|
||||
if [ "$MODE" = "production" ]; then
|
||||
print_status "To view Nginx logs: sudo tail -f /var/log/nginx/access.log"
|
||||
fi
|
||||
22
experiments/optimization-factory/deployment/nginx-config
Normal file
22
experiments/optimization-factory/deployment/nginx-config
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
server {
|
||||
listen 80;
|
||||
server_name your_domain.com; # <-- CHANGE THIS TO YOUR ACTUAL DOMAIN
|
||||
|
||||
location / {
|
||||
proxy_pass http://unix:/run/gunicorn.sock;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_connect_timeout 60s;
|
||||
proxy_send_timeout 60s;
|
||||
proxy_read_timeout 60s;
|
||||
}
|
||||
|
||||
# Security headers
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header X-XSS-Protection "1; mode=block" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header Referrer-Policy "no-referrer-when-downgrade" always;
|
||||
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
|
||||
}
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
[Unit]
|
||||
Description=Optimization Factory Web Application
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=www-data
|
||||
Group=www-data
|
||||
WorkingDirectory=/opt/optimization-factory
|
||||
EnvironmentFile=/opt/optimization-factory/.env
|
||||
ExecStart=
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
313
experiments/optimization-factory/docs/CUSTOM_RUN.md
Normal file
313
experiments/optimization-factory/docs/CUSTOM_RUN.md
Normal file
|
|
@ -0,0 +1,313 @@
|
|||
# Custom Run Functionality
|
||||
|
||||
The Custom Run feature allows users to configure and execute Codeflash optimizations with custom parameters, settings, and optimization modes. This provides a flexible framework for running Codeflash optimizations with desired settings without modifying the core project configuration.
|
||||
|
||||
## Overview
|
||||
|
||||
The Custom Run functionality enables users to:
|
||||
|
||||
- Choose from three optimization modes: single function, trace & optimize, or optimize all
|
||||
- Configure all Codeflash settings dynamically through a web interface
|
||||
- Set custom pyproject.toml locations
|
||||
- Override default module and test roots
|
||||
- Enable/disable various Codeflash flags and options
|
||||
- Execute optimizations with custom parameters
|
||||
|
||||
## Features
|
||||
|
||||
### 1. Optimization Modes
|
||||
|
||||
#### Single Function Optimization
|
||||
|
||||
- **Purpose**: Target and optimize individual Python functions for maximum performance gains
|
||||
- **Use Case**: When you want to optimize a specific function in a specific file
|
||||
- **Required Parameters**:
|
||||
- `file_path`: Path to the Python file containing the function
|
||||
- `function_name`: Name of the function to optimize
|
||||
- **Command**: `codeflash --file <file_path> --function <function_name>`
|
||||
|
||||
#### Trace & Optimize E2E Workflows
|
||||
|
||||
- **Purpose**: End-to-end optimization of entire Python workflows with execution tracing
|
||||
- **Use Case**: When you want to optimize an entire script or workflow
|
||||
- **Required Parameters**:
|
||||
- `script_path`: Path to the Python script to trace and optimize
|
||||
- **Optional Parameters**:
|
||||
- `trace_file`: Custom output file for trace data
|
||||
- `tracer_timeout`: Maximum time in seconds to trace the workflow
|
||||
- `trace_only`: Only perform tracing, do not optimize
|
||||
- **Command**: `codeflash optimize <script_path>`
|
||||
|
||||
#### Optimize All Codebase
|
||||
|
||||
- **Purpose**: Automatically optimize all codepaths in your project
|
||||
- **Use Case**: When you want to optimize your entire codebase
|
||||
- **Optional Parameters**:
|
||||
- `target_directory`: Specific directory to optimize (defaults to entire codebase)
|
||||
- `benchmark`: Run benchmarks during optimization
|
||||
- **Command**: `codeflash --all [target_directory]`
|
||||
|
||||
### 2. Configuration Options
|
||||
|
||||
#### Basic Codeflash Configuration
|
||||
|
||||
- **Module Root**: The Python module you want Codeflash to optimize
|
||||
- **Tests Root**: Directory where your tests are located
|
||||
- **Test Framework**: pytest or unittest
|
||||
- **Benchmarks Root**: Directory where benchmarks are located (required for --benchmark)
|
||||
- **Ignore Paths**: List of paths to ignore during optimization
|
||||
- **Pytest Command**: Custom pytest command with arguments
|
||||
- **Formatter Commands**: Commands to run code formatters/linters
|
||||
- **Disable Imports Sorting**: Disable automatic import organization
|
||||
- **Disable Telemetry**: Disable telemetry data collection
|
||||
|
||||
#### Optimization Flags
|
||||
|
||||
- **Async Mode**: Run Codeflash commands asynchronously (requires codeflash[asyncio])
|
||||
- **Verbose Output**: Enable verbose logging
|
||||
- **No Pull Request**: Optimize locally without opening GitHub PRs
|
||||
- **Run Benchmarks**: Run benchmarks during optimization
|
||||
- **Trace Only**: Only perform tracing, do not optimize
|
||||
- **Tracer Timeout**: Maximum time in seconds for tracing
|
||||
|
||||
#### Advanced Settings
|
||||
|
||||
- **Custom pyproject.toml Location**: Specify custom path to pyproject.toml
|
||||
- **Custom Trace File Path**: Specify custom output file for trace data
|
||||
- **Max Function Count**: Maximum number of times to trace a single function
|
||||
- **Replay Test Path**: Path to existing replay test file
|
||||
|
||||
## Usage
|
||||
|
||||
### Web Interface
|
||||
|
||||
1. **Access Custom Run**: Click the "⚙️ Custom Run" button next to any repository in the main interface
|
||||
2. **Configure Settings**: Fill out the custom run modal with your desired settings
|
||||
3. **Validate Configuration**: The system will validate your configuration and show any errors or warnings
|
||||
4. **Execute**: Click "Execute Custom Run" to start the optimization
|
||||
|
||||
### API Endpoints
|
||||
|
||||
#### Get Custom Run Settings
|
||||
|
||||
```http
|
||||
GET /api/custom-run/settings
|
||||
```
|
||||
|
||||
Returns the available custom run configuration schema.
|
||||
|
||||
#### Validate Custom Run Configuration
|
||||
|
||||
```http
|
||||
POST /api/custom-run/validate
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"optimization_mode": "single_function",
|
||||
"config": {
|
||||
"module_root": "src",
|
||||
"tests_root": "tests",
|
||||
"file_path": "src/main.py",
|
||||
"function_name": "process_data"
|
||||
},
|
||||
"flags": {
|
||||
"async": true,
|
||||
"verbose": true,
|
||||
"no_pr": false
|
||||
},
|
||||
"advanced": {
|
||||
"pyproject_location": "custom/pyproject.toml"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Execute Custom Run
|
||||
|
||||
```http
|
||||
POST /api/custom-run/execute
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"repo_url": "https://github.com/user/repo",
|
||||
"optimization_mode": "single_function",
|
||||
"config": { ... },
|
||||
"flags": { ... },
|
||||
"advanced": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
#### Get Custom Run Status
|
||||
|
||||
```http
|
||||
GET /api/custom-run/status/<instance_id>
|
||||
```
|
||||
|
||||
Returns the current status of a custom optimization run.
|
||||
|
||||
## Configuration File
|
||||
|
||||
The custom run settings are defined in `config/custom_run_settings.json`. This file contains:
|
||||
|
||||
- **Codeflash Optimization Modes**: Available optimization modes and their descriptions
|
||||
- **pyproject.toml Settings**: All configurable Codeflash settings with descriptions and types
|
||||
- **Optimization Flags**: Available Codeflash flags and their options
|
||||
- **Advanced Settings**: Advanced configuration options
|
||||
- **UI Sections**: Organization of the web interface
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Backend
|
||||
|
||||
The custom run functionality is implemented through several components:
|
||||
|
||||
1. **API Endpoints** (`server/app.py`):
|
||||
|
||||
- `/api/custom-run/settings`: Returns configuration schema
|
||||
- `/api/custom-run/validate`: Validates user configuration
|
||||
- `/api/custom-run/execute`: Executes custom optimization
|
||||
- `/api/custom-run/status/<instance_id>`: Returns run status
|
||||
|
||||
2. **Configuration Processing** (`server/app.py`):
|
||||
|
||||
- Uploads custom configuration to EC2 instance
|
||||
- Sets environment variables for custom settings
|
||||
- Modifies instance setup for custom runs
|
||||
|
||||
3. **Script Integration** (`scripts/run_optimization.sh`):
|
||||
- Detects custom run mode
|
||||
- Loads custom configuration
|
||||
- Overrides default settings
|
||||
- Enhances Claude Code CLI prompts with custom run context
|
||||
- Executes appropriate Codeflash commands
|
||||
|
||||
### Frontend
|
||||
|
||||
The custom run interface is implemented in `server/static/app.js`:
|
||||
|
||||
1. **Modal Management**: Dynamic modal creation and management
|
||||
2. **Form Handling**: Dynamic form generation based on configuration schema
|
||||
3. **Validation**: Real-time configuration validation
|
||||
4. **Execution**: Custom run execution with progress tracking
|
||||
|
||||
### Logging and Monitoring
|
||||
|
||||
Custom runs include comprehensive logging:
|
||||
|
||||
- **Configuration Loading**: Logs when custom configuration is loaded
|
||||
- **Parameter Override**: Logs when default parameters are overridden
|
||||
- **Command Building**: Logs the built Codeflash command
|
||||
- **Execution**: Logs execution start, progress, and completion
|
||||
- **Error Handling**: Logs errors with detailed context
|
||||
|
||||
### Claude Code CLI Integration
|
||||
|
||||
The custom run functionality integrates with Claude Code CLI for intelligent setup assistance:
|
||||
|
||||
- **Enhanced Prompts**: Custom run context is automatically added to Claude Code CLI prompts
|
||||
- **Mode-Specific Guidance**: Claude receives specific instructions based on the optimization mode
|
||||
- **Target Awareness**: Claude knows about target files, functions, and scripts for focused setup
|
||||
- **Dependency Management**: Claude ensures all required dependencies are installed for the specific optimization mode
|
||||
- **Configuration Validation**: Claude verifies that custom configurations are properly set up
|
||||
|
||||
The Claude Code CLI integration ensures that custom runs have the same intelligent setup assistance as standard runs, but with awareness of the specific custom configuration requirements.
|
||||
|
||||
## Examples
|
||||
|
||||
### Single Function Optimization
|
||||
|
||||
```json
|
||||
{
|
||||
"optimization_mode": "single_function",
|
||||
"config": {
|
||||
"module_root": "src",
|
||||
"tests_root": "tests",
|
||||
"test_framework": "pytest",
|
||||
"file_path": "src/calculations.py",
|
||||
"function_name": "calculate_fibonacci"
|
||||
},
|
||||
"flags": {
|
||||
"async": true,
|
||||
"verbose": true,
|
||||
"no_pr": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Trace & Optimize Workflow
|
||||
|
||||
```json
|
||||
{
|
||||
"optimization_mode": "trace_and_optimize",
|
||||
"config": {
|
||||
"module_root": "src",
|
||||
"tests_root": "tests",
|
||||
"script_path": "scripts/train_model.py"
|
||||
},
|
||||
"flags": {
|
||||
"async": true,
|
||||
"verbose": true,
|
||||
"trace_only": false,
|
||||
"tracer_timeout": 300
|
||||
},
|
||||
"advanced": {
|
||||
"trace_file": "custom_trace.trace"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Optimize All Codebase
|
||||
|
||||
```json
|
||||
{
|
||||
"optimization_mode": "optimize_all",
|
||||
"config": {
|
||||
"module_root": "src",
|
||||
"tests_root": "tests",
|
||||
"benchmarks_root": "benchmarks"
|
||||
},
|
||||
"flags": {
|
||||
"async": true,
|
||||
"verbose": true,
|
||||
"benchmark": true
|
||||
},
|
||||
"advanced": {
|
||||
"target_directory": "src/core"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Start Simple**: Begin with basic configurations and add complexity as needed
|
||||
2. **Validate First**: Always validate your configuration before executing
|
||||
3. **Use Appropriate Mode**: Choose the optimization mode that best fits your use case
|
||||
4. **Monitor Progress**: Use the status endpoint to monitor long-running optimizations
|
||||
5. **Check Logs**: Review logs for detailed execution information and debugging
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Missing Required Parameters**: Ensure all required parameters for your chosen optimization mode are provided
|
||||
2. **Invalid File Paths**: Verify that file paths exist and are accessible
|
||||
3. **Configuration Errors**: Use the validation endpoint to check for configuration issues
|
||||
4. **Execution Failures**: Check logs for detailed error information
|
||||
|
||||
### Debug Information
|
||||
|
||||
- **Configuration Logs**: Check logs for configuration loading and parameter override information
|
||||
- **Command Logs**: Review the built Codeflash command for correctness
|
||||
- **Execution Logs**: Monitor execution progress and any errors
|
||||
- **Status Endpoint**: Use the status endpoint to get current run information
|
||||
|
||||
## Integration with Existing Features
|
||||
|
||||
The Custom Run functionality integrates seamlessly with existing project features:
|
||||
|
||||
- **Repository Management**: Uses existing repository configuration and management
|
||||
- **EC2 Instance Management**: Leverages existing EC2 instance lifecycle management
|
||||
- **Log Archiving**: Integrates with existing S3 log archiving system
|
||||
- **Monitoring**: Uses existing job monitoring and status tracking
|
||||
- **Security**: Inherits existing security and access control mechanisms
|
||||
|
||||
This ensures that custom runs benefit from all existing infrastructure while providing additional flexibility and customization options.
|
||||
|
|
@ -0,0 +1,110 @@
|
|||
# Custom Run Functionality - Claude Code CLI Integration Update
|
||||
|
||||
## Summary
|
||||
|
||||
The custom run functionality has been updated to work with the current Claude Code CLI approach instead of the deprecated `llm_setup_helper.py`. This ensures that custom runs receive the same intelligent setup assistance as standard runs, but with awareness of the specific custom configuration requirements.
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. Enhanced Claude Code CLI Prompts (`scripts/run_optimization.sh`)
|
||||
|
||||
**Added Custom Run Context to Claude Prompts:**
|
||||
|
||||
- Added `{CUSTOM_RUN_CONTEXT}` placeholder to the Claude setup prompt template
|
||||
- Implemented dynamic custom run context generation based on optimization mode
|
||||
- Added mode-specific guidance for single function, trace & optimize, and optimize all modes
|
||||
- Included custom configuration details (module root, tests root, flags, etc.)
|
||||
- Added setup requirements specific to each optimization mode
|
||||
|
||||
**Custom Run Context Includes:**
|
||||
|
||||
- Optimization mode and configuration details
|
||||
- Mode-specific targets (file/function, script, directory)
|
||||
- Custom flags and settings
|
||||
- Setup requirements and focus areas
|
||||
- Dependency requirements (e.g., codeflash[asyncio] for async mode)
|
||||
|
||||
### 2. Removed Deprecated LLM Setup Helper (`server/app.py`)
|
||||
|
||||
**Removed Upload of Deprecated Script:**
|
||||
|
||||
- Removed upload of `llm_setup_helper.py` from `_complete_instance_setup` function
|
||||
- Added comment explaining that the script is deprecated
|
||||
- Maintained all other script uploads (run_optimization.sh, detect_roots.py)
|
||||
|
||||
### 3. Updated Documentation
|
||||
|
||||
**README.md:**
|
||||
|
||||
- Updated description of `llm_setup_helper.py` to indicate it's deprecated
|
||||
- Clarified that Claude Code CLI is now used directly in `run_optimization.sh`
|
||||
|
||||
**docs/CUSTOM_RUN.md:**
|
||||
|
||||
- Added new section on Claude Code CLI Integration
|
||||
- Updated implementation details to reflect Claude Code CLI integration
|
||||
- Added information about enhanced prompts and mode-specific guidance
|
||||
|
||||
## How It Works Now
|
||||
|
||||
### 1. Custom Run Detection
|
||||
|
||||
When a custom run is executed, the `run_optimization.sh` script detects the custom run mode and loads the custom configuration.
|
||||
|
||||
### 2. Claude Code CLI Enhancement
|
||||
|
||||
The script enhances the Claude Code CLI prompt with custom run context, including:
|
||||
|
||||
- Optimization mode and configuration
|
||||
- Mode-specific targets and requirements
|
||||
- Custom settings and flags
|
||||
- Setup requirements for the specific optimization mode
|
||||
|
||||
### 3. Intelligent Setup Assistance
|
||||
|
||||
Claude Code CLI receives the enhanced prompt and provides intelligent setup assistance that is:
|
||||
|
||||
- Aware of the specific optimization mode
|
||||
- Focused on the target files/functions/scripts
|
||||
- Configured for the custom settings
|
||||
- Optimized for the specific optimization requirements
|
||||
|
||||
### 4. Execution
|
||||
|
||||
After setup is complete, the script executes the appropriate Codeflash command with the custom configuration.
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Consistent Experience**: Custom runs now have the same intelligent setup assistance as standard runs
|
||||
2. **Mode-Aware Setup**: Claude understands the specific requirements for each optimization mode
|
||||
3. **Target-Focused**: Claude can focus on the specific files, functions, or scripts being optimized
|
||||
4. **Configuration-Aware**: Claude knows about custom settings and can ensure they're properly configured
|
||||
5. **Dependency Management**: Claude ensures all required dependencies are installed for the specific optimization mode
|
||||
|
||||
## Example Custom Run Context
|
||||
|
||||
For a single function optimization, Claude receives context like:
|
||||
|
||||
```
|
||||
### Custom Optimization Configuration
|
||||
- **Optimization Mode**: single_function
|
||||
- **Custom Module Root**: src
|
||||
- **Custom Tests Root**: tests
|
||||
- **Async Mode**: true
|
||||
- **Verbose Output**: true
|
||||
|
||||
### Mode-Specific Targets
|
||||
- **Target File**: src/calculations.py
|
||||
- **Target Function**: calculate_fibonacci
|
||||
- **Focus**: Ensure the target function's dependencies are properly installed
|
||||
- **Note**: This is a single function optimization targeting a specific function in a specific file
|
||||
|
||||
### Custom Run Setup Requirements
|
||||
- Pay special attention to any custom configuration requirements for this optimization mode
|
||||
- Ensure codeflash[asyncio] is installed if async mode is enabled
|
||||
- Verify that all target files/scripts exist and are accessible
|
||||
- Install any additional dependencies needed for the specific optimization mode
|
||||
- Ensure the custom module root and tests root are properly configured
|
||||
```
|
||||
|
||||
This ensures that Claude provides targeted, intelligent setup assistance that is perfectly aligned with the custom run requirements.
|
||||
|
|
@ -47,11 +47,21 @@ SSH_KEY_PATH=~/.ssh/your_key_pair.pem
|
|||
# Default: 5000
|
||||
PORT=5000
|
||||
|
||||
# CODEFLASH_API_KEY: API key for Codeflash optimization service
|
||||
# Used by: server/app.py and scripts/run_optimization.sh
|
||||
# Get this from: https://app.codeflash.ai
|
||||
CODEFLASH_API_KEY=your_codeflash_api_key_here
|
||||
|
||||
# ANTHROPIC_API_KEY: API key for Anthropic Claude LLM analysis
|
||||
# Used by: server/analyzer.py for repository analysis and configuration
|
||||
# Used by: server/analyzer.py and scripts/llm_setup_helper.py for repository analysis
|
||||
# Get this from: https://console.anthropic.com/ > API Keys
|
||||
ANTHROPIC_API_KEY=your_anthropic_api_key_here
|
||||
|
||||
# ANTHROPIC_MODEL: Anthropic model to use for LLM analysis
|
||||
# Used by: server/analyzer.py and scripts/run_optimization.sh
|
||||
# Default: claude-3-5-haiku-20241022
|
||||
ANTHROPIC_MODEL=claude-3-5-haiku-20241022
|
||||
|
||||
# GITHUB_TOKEN: GitHub Personal Access Token for repository access
|
||||
# Used by: server/analyzer.py (for accessing private repos during analysis)
|
||||
# scripts/run_optimization.sh (for GitHub operations during optimization via EC2)
|
||||
|
|
@ -59,6 +69,34 @@ ANTHROPIC_API_KEY=your_anthropic_api_key_here
|
|||
# Required scopes: repo (for private repos), public_repo (for public repos)
|
||||
GITHUB_TOKEN=your_github_token_here
|
||||
|
||||
# =============================================================================
|
||||
# AWS CREDENTIALS CONFIGURATION
|
||||
# =============================================================================
|
||||
|
||||
# AWS_ACCESS_KEY_ID: AWS Access Key ID for S3 and EC2 operations
|
||||
# Used by: server/app.py, server/s3_utils.py for AWS API authentication
|
||||
# Get this from: AWS Console > IAM > Users > Security credentials > Access keys
|
||||
# Required permissions: S3 (read/write), EC2 (launch/terminate instances)
|
||||
AWS_ACCESS_KEY_ID=your_access_key_here
|
||||
|
||||
# AWS_SECRET_ACCESS_KEY: AWS Secret Access Key for S3 and EC2 operations
|
||||
# Used by: server/app.py, server/s3_utils.py for AWS API authentication
|
||||
# Get this from: AWS Console > IAM > Users > Security credentials > Access keys
|
||||
# Required permissions: S3 (read/write), EC2 (launch/terminate instances)
|
||||
AWS_SECRET_ACCESS_KEY=your_secret_key_here
|
||||
|
||||
# AWS_DEFAULT_REGION: AWS region for S3 bucket and EC2 operations
|
||||
# Used by: server/app.py, server/s3_utils.py for AWS service region
|
||||
# Must match the region where your S3 bucket and EC2 instances are located
|
||||
# Example: us-east-1, us-west-2, eu-west-1
|
||||
AWS_DEFAULT_REGION=us-east-2
|
||||
|
||||
# AWS_S3_LOG_BUCKET: S3 bucket name for storing archived job logs
|
||||
# Used by: server/s3_utils.py for log archiving and retrieval
|
||||
# Must be a globally unique bucket name (S3 bucket names are globally unique)
|
||||
# Example: my-company-optimization-logs, optimization-factory-logs-2025
|
||||
AWS_S3_LOG_BUCKET=your-unique-log-bucket-name
|
||||
|
||||
# =============================================================================
|
||||
# TROUBLESHOOTING SSH KEY ISSUES
|
||||
# =============================================================================
|
||||
|
|
@ -92,14 +130,29 @@ GITHUB_TOKEN=your_github_token_here
|
|||
# - If this fails, the issue is with the SSH key or EC2 instance setup
|
||||
|
||||
# =============================================================================
|
||||
# NOTES ABOUT REMOTE JOB VARIABLES
|
||||
# REMOTE JOB CONFIGURATION (Set by server/app.py when launching jobs)
|
||||
# =============================================================================
|
||||
# The following variables are injected remotely on EC2 per job:
|
||||
#
|
||||
# GITHUB_REPO_URL: Set by server/app.py when launching the job
|
||||
# MODULE_ROOT: Set by server/app.py
|
||||
# TESTS_ROOT: Set by server/app.py
|
||||
# CODEFLASH_API_KEY: Taken from local environment and exported on the instance
|
||||
# FORK_OWNER: Determined by gh on the instance
|
||||
|
||||
|
||||
# GITHUB_REPO_URL: Repository URL to optimize (set by server/app.py)
|
||||
# MODULE_ROOT: Module root directory (set by server/app.py)
|
||||
# TESTS_ROOT: Tests root directory (set by server/app.py)
|
||||
# LLM_MODULE_ROOT: LLM-suggested module root (from analyzer results)
|
||||
# LLM_TESTS_ROOT: LLM-suggested tests root (from analyzer results)
|
||||
# LLM_PYTEST_CMD: LLM-suggested pytest command (from analyzer results)
|
||||
# LLM_FORMATTER_CMDS: LLM-suggested formatter commands (from analyzer results)
|
||||
# LLM_PIP_PACKAGES: LLM-suggested Python packages (from analyzer results)
|
||||
# WORK_DIR: Working directory on EC2 (defaults to /home/ubuntu/work)
|
||||
# VENV_PATH: Virtual environment path (set by entrypoint.sh)
|
||||
# PRE_INSTALL_CMDS: Pre-installation commands (allowlisted by analyzer)
|
||||
# INSTALL_CMDS: Installation commands (allowlisted by analyzer)
|
||||
# POST_INSTALL_CMDS: Post-installation commands (allowlisted by analyzer)
|
||||
# SYSTEM_PACKAGES: System packages to install (allowlisted by analyzer)
|
||||
# TEST_LOG_DIR: Test log directory (defaults to /home/ubuntu/app/logs)
|
||||
# TEST_LOG_FILE: Test log file path (set by run_optimization.sh)
|
||||
# EXIT_FILE: Exit code file path (set by run_optimization.sh)
|
||||
# STAGE_FILE: Stage tracking file path (set by run_optimization.sh)
|
||||
# CF_TARGET_FILE: Codeflash target file (for single-file optimization)
|
||||
# CF_TARGET_FUNCTION: Codeflash target function (for single-function optimization)
|
||||
# LLM_SETUP_MAX_STEPS: Maximum LLM setup steps (defaults to 10)
|
||||
# LLM_SETUP_MAX_ROUNDS: Maximum LLM setup rounds (defaults to 2)
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -118,6 +118,7 @@ ANALYSIS_SCHEMA: Dict[str, Any] = {
|
|||
"type": "object",
|
||||
"properties": {
|
||||
"test_command": {"type": "string"},
|
||||
"test_framework": {"type": "string", "enum": ["pytest", "unittest", "nose"]},
|
||||
},
|
||||
},
|
||||
"resources": {
|
||||
|
|
@ -185,10 +186,51 @@ class AnalysisJob:
|
|||
message: str
|
||||
result: Optional[Dict[str, Any]] = None
|
||||
|
||||
_jobs: Dict[str, AnalysisJob] = {}
|
||||
# File-based job tracking to work across gunicorn workers
|
||||
ANALYSIS_JOBS_FILE = BASE_DIR / "config" / "analysis_jobs.json"
|
||||
_jobs_lock = threading.Lock()
|
||||
_executor = ThreadPoolExecutor(max_workers=2)
|
||||
|
||||
def _load_jobs() -> Dict[str, AnalysisJob]:
|
||||
"""Load jobs from persistent storage"""
|
||||
try:
|
||||
if not ANALYSIS_JOBS_FILE.exists():
|
||||
logger.debug(f"📂 [ANALYZER] No jobs file found at {ANALYSIS_JOBS_FILE}")
|
||||
return {}
|
||||
|
||||
data = json.loads(ANALYSIS_JOBS_FILE.read_text())
|
||||
jobs = {}
|
||||
for job_id, job_data in data.items():
|
||||
jobs[job_id] = AnalysisJob(
|
||||
job_id=job_data["job_id"],
|
||||
repo_url=job_data["repo_url"],
|
||||
status=job_data["status"],
|
||||
message=job_data["message"],
|
||||
result=job_data.get("result")
|
||||
)
|
||||
logger.debug(f"📂 [ANALYZER] Loaded {len(jobs)} jobs from storage")
|
||||
return jobs
|
||||
except Exception as e:
|
||||
logger.warning(f"⚠️ [ANALYZER] Failed to load jobs: {e}")
|
||||
return {}
|
||||
|
||||
def _save_jobs(jobs: Dict[str, AnalysisJob]) -> None:
|
||||
"""Save jobs to persistent storage"""
|
||||
try:
|
||||
data = {}
|
||||
for job_id, job in jobs.items():
|
||||
data[job_id] = {
|
||||
"job_id": job.job_id,
|
||||
"repo_url": job.repo_url,
|
||||
"status": job.status,
|
||||
"message": job.message,
|
||||
"result": job.result
|
||||
}
|
||||
ANALYSIS_JOBS_FILE.write_text(json.dumps(data, indent=2))
|
||||
logger.debug(f"💾 [ANALYZER] Saved {len(jobs)} jobs to storage")
|
||||
except Exception as e:
|
||||
logger.error(f"❌ [ANALYZER] Failed to save jobs: {e}")
|
||||
|
||||
def _slug(repo_url: str) -> str:
|
||||
parts = repo_url.rstrip("/").split("/")
|
||||
org = parts[-2] if len(parts) >= 2 else "org"
|
||||
|
|
@ -781,12 +823,14 @@ def submit_analysis(repo_url: str) -> str:
|
|||
logger.info(f"🚀 [ANALYZER] Submitting analysis job - repo={repo_url}, job_id={job_id}")
|
||||
|
||||
with _jobs_lock:
|
||||
_jobs[job_id] = AnalysisJob(
|
||||
jobs = _load_jobs()
|
||||
jobs[job_id] = AnalysisJob(
|
||||
job_id=job_id,
|
||||
repo_url=repo_url,
|
||||
status="pending",
|
||||
message="analysis queued",
|
||||
)
|
||||
_save_jobs(jobs)
|
||||
|
||||
# Submit to thread pool
|
||||
future = _executor.submit(_run_analysis, job_id, repo_url)
|
||||
|
|
@ -798,11 +842,12 @@ def submit_analysis(repo_url: str) -> str:
|
|||
def job_status(job_id: str) -> AnalysisJob:
|
||||
"""Get status of analysis job"""
|
||||
with _jobs_lock:
|
||||
if job_id not in _jobs:
|
||||
jobs = _load_jobs()
|
||||
if job_id not in jobs:
|
||||
logger.warning(f"⚠️ [ANALYZER] Job status requested for unknown job_id: {job_id}")
|
||||
raise KeyError(f"Analysis job {job_id} not found")
|
||||
|
||||
job = _jobs[job_id]
|
||||
job = jobs[job_id]
|
||||
logger.debug(f"📊 [ANALYZER] Job status requested - job_id={job_id}, status={job.status}")
|
||||
return job
|
||||
|
||||
|
|
@ -810,8 +855,9 @@ def job_status(job_id: str) -> AnalysisJob:
|
|||
def job_result(job_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Get result of completed analysis job"""
|
||||
with _jobs_lock:
|
||||
if job_id in _jobs and _jobs[job_id].status == "succeeded":
|
||||
result = _jobs[job_id].result
|
||||
jobs = _load_jobs()
|
||||
if job_id in jobs and jobs[job_id].status == "succeeded":
|
||||
result = jobs[job_id].result
|
||||
logger.debug(f"📋 [ANALYZER] Job result requested - job_id={job_id}, has_result={bool(result)}")
|
||||
return result
|
||||
|
||||
|
|
@ -824,7 +870,10 @@ def _run_analysis(job_id: str, repo_url: str) -> None:
|
|||
log_analysis_start(repo_url, job_id)
|
||||
|
||||
with _jobs_lock:
|
||||
_jobs[job_id].status = "running"
|
||||
jobs = _load_jobs()
|
||||
if job_id in jobs:
|
||||
jobs[job_id].status = "running"
|
||||
_save_jobs(jobs)
|
||||
|
||||
try:
|
||||
# Fetch repository tarball
|
||||
|
|
@ -854,8 +903,11 @@ def _run_analysis(job_id: str, repo_url: str) -> None:
|
|||
log_analysis_error(repo_url, job_id, msg, duration_ms)
|
||||
|
||||
with _jobs_lock:
|
||||
_jobs[job_id].status = "failed"
|
||||
_jobs[job_id].message = msg
|
||||
jobs = _load_jobs()
|
||||
if job_id in jobs:
|
||||
jobs[job_id].status = "failed"
|
||||
jobs[job_id].message = msg
|
||||
_save_jobs(jobs)
|
||||
return
|
||||
except (requests.ConnectionError, requests.Timeout): # type: ignore[attr-defined]
|
||||
msg = "Network error while fetching repository tarball (connection/timeout)."
|
||||
|
|
@ -863,8 +915,11 @@ def _run_analysis(job_id: str, repo_url: str) -> None:
|
|||
log_analysis_error(repo_url, job_id, msg, duration_ms)
|
||||
|
||||
with _jobs_lock:
|
||||
_jobs[job_id].status = "failed"
|
||||
_jobs[job_id].message = msg
|
||||
jobs = _load_jobs()
|
||||
if job_id in jobs:
|
||||
jobs[job_id].status = "failed"
|
||||
jobs[job_id].message = msg
|
||||
_save_jobs(jobs)
|
||||
return
|
||||
|
||||
# Extract files and run heuristics
|
||||
|
|
@ -1004,7 +1059,7 @@ def _run_analysis(job_id: str, repo_url: str) -> None:
|
|||
"tests_root": heur.get("tests_root", "auto"),
|
||||
"formatter_cmds": ["disabled"],
|
||||
},
|
||||
"tests": {"test_command": "pytest"},
|
||||
"tests": {"test_command": "pytest", "test_framework": "pytest"},
|
||||
"resources": {"tier": "small", "rationale": "heuristic default"},
|
||||
"env": {"non_secret_env_vars": {}},
|
||||
"confidence": 0.4,
|
||||
|
|
@ -1066,6 +1121,31 @@ def _run_analysis(job_id: str, repo_url: str) -> None:
|
|||
normalized.append({"name": pkg_name, "version_spec": version_spec} if version_spec else {"name": pkg_name})
|
||||
result["python_packages"] = normalized
|
||||
|
||||
# Derive test_framework from test_command if not already set
|
||||
try:
|
||||
tests_block = result.get("tests", {}) or {}
|
||||
test_command = tests_block.get("test_command", "")
|
||||
test_framework = tests_block.get("test_framework")
|
||||
|
||||
if not test_framework and test_command:
|
||||
# Derive test framework from test command (same logic as run_optimization.sh)
|
||||
test_cmd_lower = test_command.lower()
|
||||
if "pytest" in test_cmd_lower:
|
||||
test_framework = "pytest"
|
||||
elif "unittest" in test_cmd_lower:
|
||||
test_framework = "unittest"
|
||||
elif "nose" in test_cmd_lower or "nosetests" in test_cmd_lower:
|
||||
test_framework = "nose"
|
||||
else:
|
||||
test_framework = "pytest" # default fallback
|
||||
|
||||
tests_block["test_framework"] = test_framework
|
||||
result["tests"] = tests_block
|
||||
log_analysis_step("Derived test framework from command", job_id,
|
||||
test_command=test_command, test_framework=test_framework)
|
||||
except Exception as e:
|
||||
log_analysis_step("Failed to derive test framework", job_id, error=str(e))
|
||||
|
||||
# Infer additional env vars (non-secret) and merge
|
||||
try:
|
||||
inferred_env = _infer_env_non_secret_vars(files)
|
||||
|
|
@ -1088,13 +1168,16 @@ def _run_analysis(job_id: str, repo_url: str) -> None:
|
|||
duration_ms = int((time.time() - start_time) * 1000)
|
||||
|
||||
with _jobs_lock:
|
||||
_jobs[job_id].status = "succeeded"
|
||||
if llm_used:
|
||||
_jobs[job_id].message = "analysis completed via LLM"
|
||||
else:
|
||||
reason = last_parse_error or ("LLM unavailable" if os.getenv("ANTHROPIC_API_KEY") else "ANTHROPIC_API_KEY not set")
|
||||
_jobs[job_id].message = f"analysis completed via heuristics ({reason})"
|
||||
_jobs[job_id].result = result
|
||||
jobs = _load_jobs()
|
||||
if job_id in jobs:
|
||||
jobs[job_id].status = "succeeded"
|
||||
if llm_used:
|
||||
jobs[job_id].message = "analysis completed via LLM"
|
||||
else:
|
||||
reason = last_parse_error or ("LLM unavailable" if os.getenv("ANTHROPIC_API_KEY") else "ANTHROPIC_API_KEY not set")
|
||||
jobs[job_id].message = f"analysis completed via heuristics ({reason})"
|
||||
jobs[job_id].result = result
|
||||
_save_jobs(jobs)
|
||||
|
||||
log_analysis_success(repo_url, job_id, duration_ms,
|
||||
method="LLM" if llm_used else "heuristics",
|
||||
|
|
@ -1106,8 +1189,11 @@ def _run_analysis(job_id: str, repo_url: str) -> None:
|
|||
log_analysis_error(repo_url, job_id, str(e), duration_ms)
|
||||
|
||||
with _jobs_lock:
|
||||
_jobs[job_id].status = "failed"
|
||||
_jobs[job_id].message = str(e)
|
||||
jobs = _load_jobs()
|
||||
if job_id in jobs:
|
||||
jobs[job_id].status = "failed"
|
||||
jobs[job_id].message = str(e)
|
||||
_save_jobs(jobs)
|
||||
|
||||
|
||||
def load_analysis_for_repo(repo_url: str) -> Optional[Dict[str, Any]]:
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
252
experiments/optimization-factory/server/s3_utils.py
Normal file
252
experiments/optimization-factory/server/s3_utils.py
Normal file
|
|
@ -0,0 +1,252 @@
|
|||
"""
|
||||
S3 Log Manager for Optimizer Factory
|
||||
Handles all S3 operations for log file storage and retrieval.
|
||||
|
||||
This module provides a centralized interface for:
|
||||
1. Uploading log archives to S3
|
||||
2. Downloading individual log files from S3
|
||||
3. Listing archived log sessions for repositories
|
||||
4. Managing S3 object keys and directory structure
|
||||
"""
|
||||
|
||||
import os
|
||||
import boto3
|
||||
import logging
|
||||
from botocore.exceptions import ClientError
|
||||
from datetime import datetime
|
||||
from typing import Optional, List, Dict, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class S3LogManager:
|
||||
"""
|
||||
Manages log file operations with an AWS S3 bucket.
|
||||
Handles uploading, downloading, and listing of log files.
|
||||
"""
|
||||
|
||||
def __init__(self, bucket_name: str, region_name: str):
|
||||
"""
|
||||
Initializes the S3LogManager.
|
||||
|
||||
Args:
|
||||
bucket_name: The name of the S3 bucket.
|
||||
region_name: The AWS region of the bucket.
|
||||
"""
|
||||
if not bucket_name:
|
||||
raise ValueError("S3 bucket name cannot be empty.")
|
||||
|
||||
self.bucket_name = bucket_name
|
||||
|
||||
# Configure S3 client with optimized connection settings
|
||||
from botocore.config import Config
|
||||
config = Config(
|
||||
max_pool_connections=20, # Increase connection pool size
|
||||
retries={'max_attempts': 3, 'mode': 'adaptive'}
|
||||
)
|
||||
|
||||
self.s3_client = boto3.client("s3", region_name=region_name, config=config)
|
||||
|
||||
logger.info(f"S3LogManager initialized for bucket '{bucket_name}' in region '{region_name}'")
|
||||
|
||||
def _generate_s3_key(self, repo_url: str, filename: str) -> str:
|
||||
"""
|
||||
Generates a structured S3 key based on the desired naming convention.
|
||||
Format: <repo-slug>/<YYYY-MM-DD_HH-MM-SS>/<filename>
|
||||
|
||||
Args:
|
||||
repo_url: The repository URL to generate slug from
|
||||
filename: The filename to append to the key
|
||||
|
||||
Returns:
|
||||
S3 key string in format: org-repo/2025-01-15_14-30-25/filename.log
|
||||
"""
|
||||
# Create a URL-safe slug from the repo URL
|
||||
try:
|
||||
parts = (repo_url or "").rstrip("/").split("/")
|
||||
if len(parts) >= 2:
|
||||
org = parts[-2]
|
||||
name = parts[-1].replace(".git", "")
|
||||
repo_slug = f"{org}-{name}"
|
||||
else:
|
||||
repo_slug = "unknown-repo"
|
||||
except (IndexError, AttributeError):
|
||||
repo_slug = "unknown-repo"
|
||||
|
||||
# Get current timestamp for the archive folder
|
||||
timestamp = datetime.utcnow().strftime('%Y-%m-%d_%H-%M-%S')
|
||||
|
||||
return f"{repo_slug}/{timestamp}/{filename}"
|
||||
|
||||
def upload_log_archive(self, local_dir_path: str, repo_url: str) -> Optional[str]:
|
||||
"""
|
||||
Uploads all files from a local directory to a structured path in S3.
|
||||
|
||||
Args:
|
||||
local_dir_path: The local directory containing log files.
|
||||
repo_url: The repository URL to structure the S3 path.
|
||||
|
||||
Returns:
|
||||
The S3 prefix (folder path) where the logs were uploaded, or None if failed.
|
||||
"""
|
||||
uploaded_files = []
|
||||
s3_prefix = ""
|
||||
|
||||
logger.info(f"Starting upload of log archive from {local_dir_path} for repo {repo_url}")
|
||||
|
||||
for root, _, files in os.walk(local_dir_path):
|
||||
for filename in files:
|
||||
local_path = os.path.join(root, filename)
|
||||
|
||||
# Generate prefix once based on the first file
|
||||
if not s3_prefix:
|
||||
s3_prefix = os.path.dirname(self._generate_s3_key(repo_url, filename))
|
||||
|
||||
s3_key = f"{s3_prefix}/{filename}"
|
||||
|
||||
try:
|
||||
# Check if file exists before attempting upload
|
||||
if not os.path.exists(local_path):
|
||||
logger.warning(f"Local file not found for upload: {local_path}")
|
||||
continue
|
||||
|
||||
self.s3_client.upload_file(local_path, self.bucket_name, s3_key)
|
||||
logger.info(f"Successfully uploaded {local_path} to s3://{self.bucket_name}/{s3_key}")
|
||||
uploaded_files.append(s3_key)
|
||||
except ClientError as e:
|
||||
logger.error(f"Failed to upload {local_path} to S3: {e}")
|
||||
except FileNotFoundError:
|
||||
logger.warning(f"Local file not found for upload: {local_path}")
|
||||
|
||||
if uploaded_files:
|
||||
logger.info(f"Upload completed: {len(uploaded_files)} files uploaded to s3://{self.bucket_name}/{s3_prefix}")
|
||||
return s3_prefix
|
||||
else:
|
||||
logger.error("No files were successfully uploaded")
|
||||
return None
|
||||
|
||||
def download_log_stream(self, s3_key: str):
|
||||
"""
|
||||
Downloads a log file from S3 and returns a streaming response object.
|
||||
|
||||
Args:
|
||||
s3_key: The S3 object key to download
|
||||
|
||||
Returns:
|
||||
S3 response body object for streaming, or None if not found
|
||||
"""
|
||||
try:
|
||||
response = self.s3_client.get_object(Bucket=self.bucket_name, Key=s3_key)
|
||||
logger.info(f"Successfully initiated download of s3://{self.bucket_name}/{s3_key}")
|
||||
return response['Body']
|
||||
except ClientError as e:
|
||||
if e.response['Error']['Code'] == 'NoSuchKey':
|
||||
logger.error(f"S3 key not found: {s3_key}")
|
||||
else:
|
||||
logger.error(f"Error downloading from S3: {e}")
|
||||
return None
|
||||
|
||||
def list_log_archives(self, repo_url: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Lists all archived log sessions for a given repository.
|
||||
A "session" is a unique folder under the repository's slug.
|
||||
|
||||
Args:
|
||||
repo_url: The repository URL to list archives for
|
||||
|
||||
Returns:
|
||||
List of session dictionaries with session_id and files
|
||||
"""
|
||||
try:
|
||||
parts = (repo_url or "").rstrip("/").split("/")
|
||||
if len(parts) >= 2:
|
||||
org = parts[-2]
|
||||
name = parts[-1].replace(".git", "")
|
||||
repo_slug = f"{org}-{name}"
|
||||
else:
|
||||
logger.warning(f"Could not parse repo URL for slug generation: {repo_url}")
|
||||
return []
|
||||
except (IndexError, AttributeError):
|
||||
logger.warning(f"Invalid repo URL format: {repo_url}")
|
||||
return []
|
||||
|
||||
logger.info(f"Listing log archives for repo slug: {repo_slug}")
|
||||
|
||||
try:
|
||||
paginator = self.s3_client.get_paginator('list_objects_v2')
|
||||
pages = paginator.paginate(Bucket=self.bucket_name, Prefix=f"{repo_slug}/", Delimiter='/')
|
||||
|
||||
sessions = []
|
||||
for page in pages:
|
||||
for prefix in page.get('CommonPrefixes', []):
|
||||
session_id = prefix.get('Prefix').strip('/')
|
||||
files = self._list_files_in_session(session_id)
|
||||
sessions.append({
|
||||
"session_id": session_id,
|
||||
"files": files
|
||||
})
|
||||
|
||||
# Sort sessions by name (timestamp) descending
|
||||
sessions.sort(key=lambda x: x['session_id'], reverse=True)
|
||||
|
||||
logger.info(f"Found {len(sessions)} log sessions for {repo_slug}")
|
||||
return sessions
|
||||
|
||||
except ClientError as e:
|
||||
logger.error(f"Error listing S3 objects for {repo_slug}: {e}")
|
||||
return []
|
||||
|
||||
def _list_files_in_session(self, session_prefix: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Helper to list all files within a specific session prefix.
|
||||
|
||||
Args:
|
||||
session_prefix: The S3 prefix for the session (e.g., "org-repo/2025-01-15_14-30-25")
|
||||
|
||||
Returns:
|
||||
List of file dictionaries with metadata
|
||||
"""
|
||||
files = []
|
||||
try:
|
||||
response = self.s3_client.list_objects_v2(Bucket=self.bucket_name, Prefix=f"{session_prefix}/")
|
||||
for obj in response.get('Contents', []):
|
||||
files.append({
|
||||
"filename": os.path.basename(obj['Key']),
|
||||
"path": obj['Key'], # The full S3 key is now the path
|
||||
"size_bytes": obj['Size'],
|
||||
"modified_iso": obj['LastModified'].isoformat(),
|
||||
})
|
||||
|
||||
logger.debug(f"Found {len(files)} files in session {session_prefix}")
|
||||
return files
|
||||
|
||||
except ClientError as e:
|
||||
logger.error(f"Error listing files in session {session_prefix}: {e}")
|
||||
return []
|
||||
|
||||
def list_session_files(self, session_id: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Lists all files in a specific log session for downloading.
|
||||
|
||||
Args:
|
||||
session_id: The session identifier (e.g., "org-repo/2025-01-15_14-30-25")
|
||||
|
||||
Returns:
|
||||
List of file dictionaries with s3_key for downloading
|
||||
"""
|
||||
files = []
|
||||
try:
|
||||
response = self.s3_client.list_objects_v2(Bucket=self.bucket_name, Prefix=f"{session_id}/")
|
||||
for obj in response.get('Contents', []):
|
||||
files.append({
|
||||
"filename": os.path.basename(obj['Key']),
|
||||
"s3_key": obj['Key'], # Full S3 key for downloading
|
||||
"size_bytes": obj['Size'],
|
||||
"modified_iso": obj['LastModified'].isoformat(),
|
||||
})
|
||||
|
||||
logger.info(f"Found {len(files)} files in session {session_id}")
|
||||
return files
|
||||
|
||||
except ClientError as e:
|
||||
logger.error(f"Error listing files in session {session_id}: {e}")
|
||||
return []
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -9,7 +9,22 @@
|
|||
</head>
|
||||
|
||||
<body>
|
||||
<div class="container">
|
||||
<!-- Security Gate Overlay -->
|
||||
<div id="securityGate" class="security-overlay">
|
||||
<div class="security-modal">
|
||||
<div class="security-header">
|
||||
<h2>🔒 Access Required</h2>
|
||||
<p>Please enter the security code to access the dashboard</p>
|
||||
</div>
|
||||
<div class="security-form">
|
||||
<input type="text" id="securityCode" placeholder="Enter 6-digit code" maxlength="6" />
|
||||
<button id="submitCode" class="btn btn-primary">Submit</button>
|
||||
<div id="errorMessage" class="error-message" style="display: none;"></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="container" id="mainContent" style="display: none;">
|
||||
<div class="header">
|
||||
<h1>Optimizer Factory</h1>
|
||||
<p>Manage Codeflash optimizations across Python repositories using EC2</p>
|
||||
|
|
@ -17,12 +32,20 @@
|
|||
|
||||
<div class="content">
|
||||
<div class="actions">
|
||||
<button id="refresh" class="btn btn-secondary">
|
||||
<span>🔄</span> Refresh
|
||||
</button>
|
||||
<button id="runAll" class="btn btn-success">
|
||||
<span>🚀</span> Run All
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<!-- Instructions Section -->
|
||||
<div class="instructions-section">
|
||||
<div class="instructions-header">
|
||||
<div class="instructions-icon">🚀</div>
|
||||
<div class="instructions-content">
|
||||
<h3>Optimization Framework Instructions</h3>
|
||||
<p>Learn how to run and track optimizations effectively with our step-by-step guide.</p>
|
||||
</div>
|
||||
<button id="readInstructionsBtn" class="btn btn-primary">
|
||||
<span>📖</span> Please README Before Usage of This Framework
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="form-section">
|
||||
|
|
@ -88,7 +111,25 @@
|
|||
</div>
|
||||
|
||||
<div class="table-section">
|
||||
<h2>Repositories</h2>
|
||||
<div class="table-header">
|
||||
<h2>Repositories</h2>
|
||||
<div class="table-controls">
|
||||
<div class="search-container">
|
||||
<input type="text" id="repoSearch" placeholder="Search repositories..."
|
||||
class="search-input">
|
||||
<span class="search-icon">🔍</span>
|
||||
</div>
|
||||
<div class="table-actions">
|
||||
<button id="refresh" class="btn btn-secondary">
|
||||
<span>🔄</span> Refresh
|
||||
</button>
|
||||
<button id="cleanupInstances" class="btn btn-warning"
|
||||
title="Clean up terminated instances only">
|
||||
<span>🧹</span> Cleanup
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="table-wrapper">
|
||||
<table id="repos">
|
||||
<thead>
|
||||
|
|
@ -96,7 +137,9 @@
|
|||
<th>Repository</th>
|
||||
<th>Module Root</th>
|
||||
<th>Tests Root</th>
|
||||
<th>Auto-Terminate</th>
|
||||
<th>Last EC2 ID</th>
|
||||
<th>IP Address</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
|
|
@ -105,16 +148,20 @@
|
|||
</div>
|
||||
</div>
|
||||
|
||||
<div class="logs-section">
|
||||
<h2>Job Status & Logs</h2>
|
||||
<div class="logs-container">
|
||||
<div class="log-panel">
|
||||
<h3>Job Status</h3>
|
||||
<pre id="status" class="log-content"></pre>
|
||||
<!-- Instructions Modal -->
|
||||
<div id="instructionsModal" class="modal-overlay" style="display:none;">
|
||||
<div class="modal-content large-modal">
|
||||
<div class="modal-header">
|
||||
<h3>📖 Optimization Framework Usage Guide</h3>
|
||||
<button id="closeInstructionsModal" class="btn btn-secondary btn-sm">✕</button>
|
||||
</div>
|
||||
<div class="log-panel">
|
||||
<h3>Job Logs</h3>
|
||||
<pre id="logs" class="log-content"></pre>
|
||||
<div class="modal-body">
|
||||
<div id="instructionsContent"></div>
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button id="closeInstructionsModal2" class="btn btn-secondary">
|
||||
<span>✓</span> Got it, let's start optimizing!
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
@ -133,6 +180,69 @@
|
|||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Analysis Editor Modal -->
|
||||
<div id="analysisEditorModal" class="modal-overlay" style="display:none;">
|
||||
<div class="modal-content large-modal">
|
||||
<div class="modal-header">
|
||||
<h3>📝 Edit Analysis Results</h3>
|
||||
<button id="closeAnalysisEditorModal" class="btn btn-secondary btn-sm">✕</button>
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<div id="analysisEditorContent"></div>
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button id="saveAnalysisChanges" class="btn btn-primary">
|
||||
<span>💾</span> Save Changes
|
||||
</button>
|
||||
<button id="cancelAnalysisChanges" class="btn btn-secondary">
|
||||
<span>❌</span> Cancel
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Logs Viewer Modal -->
|
||||
<div id="logsViewerModal" class="modal-overlay" style="display:none;">
|
||||
<div class="modal-content large-modal">
|
||||
<div class="modal-header">
|
||||
<h3>📁 Archived Logs</h3>
|
||||
<button id="closeLogsViewerModal" class="btn btn-secondary btn-sm">✕</button>
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<div id="logsViewerContent"></div>
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button id="downloadAllLogs" class="btn btn-success" disabled>
|
||||
<span>⏸️</span> No Running Instance
|
||||
</button>
|
||||
<button id="closeLogsViewerModal2" class="btn btn-secondary">
|
||||
<span>✕</span> Close
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Auto-Termination Settings Modal -->
|
||||
<div id="autoTerminationModal" class="modal-overlay" style="display:none;">
|
||||
<div class="modal-content">
|
||||
<div class="modal-header">
|
||||
<h3>⚙️ Auto-Termination Settings</h3>
|
||||
<button id="closeAutoTerminationModal" class="btn btn-secondary btn-sm">✕</button>
|
||||
</div>
|
||||
<div class="modal-body">
|
||||
<div id="autoTerminationContent"></div>
|
||||
</div>
|
||||
<div class="modal-footer">
|
||||
<button id="saveAutoTerminationSettings" class="btn btn-primary">
|
||||
<span>💾</span> Save Settings
|
||||
</button>
|
||||
<button id="cancelAutoTerminationSettings" class="btn btn-secondary">
|
||||
<span>❌</span> Cancel
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
Loading…
Reference in a new issue