codeflash-internal/deployment/onprem-simple/TESTING.md

# Codeflash On-Premise Testing Guide

This guide walks through testing the Codeflash on-premise deployment step-by-step.

## Quick Test (5 minutes)

This is the fastest way to verify everything works:

### 1. Build and Run

```bash
# Build the image (from repository root)
docker build -f deployment/onprem-simple/Dockerfile.unifiedall -t codeflash/unified:latest .

# Run the container
docker run -d --name codeflash-test \
  -e AZURE_OPENAI_API_KEY=your-azure-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-test-data:/var/lib/postgresql/data \
  codeflash/unified:latest

# Wait ~15 seconds for services to start
sleep 15
```

### 2. Verify All Services Are Running

```bash
# Check container status
docker ps | grep codeflash-test

# Check all services
docker exec codeflash-test supervisorctl status
```

Expected output:
```
postgres                         RUNNING   pid 40, uptime 0:00:15
aiservice                        RUNNING   pid 41, uptime 0:00:15
cf-api                           RUNNING   pid 42, uptime 0:00:15
cf-webapp                        RUNNING   pid 43, uptime 0:00:15
```

### 3. Test Service Endpoints

```bash
# Test cf-api
curl http://localhost:3001/cfapi/healthcheck
# Expected: {"status":"ok"}

# Test aiservice
curl http://localhost:8000/health
# Expected: {"status":"healthy"}

# Test webapp
curl http://localhost:3000
# Expected: HTML content
```

### 4. Get API Key

```bash
docker logs codeflash-test | grep "Your API Key"
# Or
docker exec codeflash-test cat /app/API_KEY.txt
```

### 5. Test CLI

```bash
# Install CLI
pip install codeflash

# Configure environment
export CODEFLASH_API_KEY=cf-your-key-from-step-4
export CODEFLASH_AIS_SERVER=local
export CODEFLASH_CFAPI_SERVER=local

# Test connection
codeflash --help
```

### 6. Test Optimization (Optional)

If you have a Python project to test:

```bash
cd your-python-project
codeflash --file path/to/file.py --function function_name --no-pr -v
```

### 7. Cleanup

```bash
docker stop codeflash-test
docker rm codeflash-test
docker volume rm codeflash-test-data
```

---

## Detailed Testing Scenarios

### Scenario 1: Fresh Installation Test

Tests a brand new installation from scratch.

```bash
# Clean slate
docker stop codeflash-test 2>/dev/null || true
docker rm codeflash-test 2>/dev/null || true
docker volume rm codeflash-test-data 2>/dev/null || true

# Build
docker build -f deployment/onprem-simple/Dockerfile.unifiedall -t codeflash/unified:latest .

# Run
docker run -d --name codeflash-test \
  -e AZURE_OPENAI_API_KEY=your-azure-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-test-data:/var/lib/postgresql/data \
  codeflash/unified:latest

# Wait for startup
sleep 20

# Verify
docker logs codeflash-test
docker exec codeflash-test supervisorctl status
docker exec codeflash-test pg_isready -h localhost -p 5432 -U codeflash
```

**Success criteria:**
- ✅ All 4 services show "RUNNING"
- ✅ PostgreSQL is ready
- ✅ API key generated and saved
- ✅ No error messages in logs

### Scenario 2: Persistent Data Test

Tests that data persists across container restarts.

```bash
# Get initial API key
INITIAL_KEY=$(docker exec codeflash-test cat /app/API_KEY.txt)
echo "Initial API key: $INITIAL_KEY"

# Restart container
docker restart codeflash-test
sleep 15

# Get API key after restart
NEW_KEY=$(docker exec codeflash-test cat /app/API_KEY.txt)
echo "API key after restart: $NEW_KEY"

# Compare
if [ "$INITIAL_KEY" = "$NEW_KEY" ]; then
  echo "✅ Data persisted correctly"
else
  echo "❌ Data did not persist"
fi
```

**Success criteria:**
- ✅ API key remains the same after restart
- ✅ All services restart successfully
- ✅ Database data is intact

### Scenario 3: Service Restart Test

Tests individual service restarts.

```bash
# Restart cf-api
docker exec codeflash-test supervisorctl restart cf-api
sleep 5
curl http://localhost:3001/cfapi/healthcheck

# Restart aiservice
docker exec codeflash-test supervisorctl restart aiservice
sleep 5
curl http://localhost:8000/health

# Restart cf-webapp
docker exec codeflash-test supervisorctl restart cf-webapp
sleep 5
curl http://localhost:3000

# Check all services
docker exec codeflash-test supervisorctl status
```

**Success criteria:**
- ✅ Each service restarts without errors
- ✅ Endpoints respond after restart
- ✅ No cascading failures

### Scenario 4: Database Test

Tests database connectivity and data.

```bash
# Check PostgreSQL is running
docker exec codeflash-test pg_isready -h localhost -p 5432 -U codeflash

# Check database exists
docker exec codeflash-test psql postgresql://codeflash:codeflash@localhost:5432/codeflash \
  -c "\l" | grep codeflash

# Check tables exist
docker exec codeflash-test psql postgresql://codeflash:codeflash@localhost:5432/codeflash \
  -c "\dt" | head -10

# Check user exists
docker exec codeflash-test psql postgresql://codeflash:codeflash@localhost:5432/codeflash \
  -c "SELECT * FROM users;"

# Check API key exists
docker exec codeflash-test psql postgresql://codeflash:codeflash@localhost:5432/codeflash \
  -c "SELECT key, suffix FROM cf_api_keys;"
```

**Success criteria:**
- ✅ Database `codeflash` exists
- ✅ All migrations applied (36+ tables)
- ✅ User created
- ✅ API key stored

### Scenario 5: CLI Integration Test

Full end-to-end test with a real Python file.

```bash
# Create test project
mkdir -p /tmp/codeflash-test-project
cd /tmp/codeflash-test-project

# Create a simple Python file
cat > test.py << 'EOF'
def fibonacci(n):
    """Calculate fibonacci number recursively (slow)."""
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

def main():
    print(fibonacci(10))

if __name__ == "__main__":
    main()
EOF

# Get API key
export CODEFLASH_API_KEY=$(docker exec codeflash-test cat /app/API_KEY.txt)
export CODEFLASH_AIS_SERVER=local
export CODEFLASH_CFAPI_SERVER=local

# Install CLI
pip install codeflash

# Run optimization
codeflash --file test.py --function fibonacci --no-pr -v

# Check if file was modified
git diff test.py 2>/dev/null || echo "File modified (no git)"
```

**Success criteria:**
- ✅ CLI connects successfully
- ✅ Optimization completes without errors
- ✅ Function is optimized (or determined not optimizable)
- ✅ Test file shows improvements

### Scenario 6: Port Conflict Test

Tests behavior when ports are already in use.

```bash
# Try to run second container (should fail)
docker run -d --name codeflash-test-2 \
  -e AZURE_OPENAI_API_KEY=your-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  codeflash/unified:latest 2>&1 | grep "address already in use"

# Run with different ports (should succeed)
docker run -d --name codeflash-test-2 \
  -e AZURE_OPENAI_API_KEY=your-key \
  -p 15432:5432 -p 18000:8000 -p 13001:3001 -p 13000:3000 \
  -v codeflash-test-2-data:/var/lib/postgresql/data \
  codeflash/unified:latest

sleep 15
docker exec codeflash-test-2 supervisorctl status

# Cleanup
docker stop codeflash-test-2
docker rm codeflash-test-2
docker volume rm codeflash-test-2-data
```

**Success criteria:**
- ✅ Second container with same ports fails gracefully
- ✅ Second container with different ports works
- ✅ Both containers can run simultaneously

### Scenario 7: Environment Variable Test

Tests various environment variable configurations.

```bash
# Test with minimal config
docker run -d --name codeflash-minimal \
  -e AZURE_OPENAI_API_KEY=test-key \
  -p 25432:5432 -p 28000:8000 -p 23001:3001 -p 23000:3000 \
  codeflash/unified:latest

sleep 20
docker logs codeflash-minimal | grep "Generated SECRET_KEY"
docker logs codeflash-minimal | grep "API Key:"

# Test with full config
docker run -d --name codeflash-full \
  -e AZURE_OPENAI_API_KEY=test-key \
  -e SECRET_KEY=my-custom-secret \
  -e NEXT_PUBLIC_APP_URL=http://custom.domain:3000 \
  -e WEBAPP_URL=http://custom.domain:3000 \
  -p 35432:5432 -p 38000:8000 -p 33001:3001 -p 33000:3000 \
  codeflash/unified:latest

sleep 20
docker logs codeflash-full | grep -v "Generated SECRET_KEY"

# Cleanup
docker stop codeflash-minimal codeflash-full
docker rm codeflash-minimal codeflash-full
```

**Success criteria:**
- ✅ Minimal config works with auto-generated values
- ✅ Full config respects custom values
- ✅ No required variables missing

### Scenario 8: Upgrade Test

Tests upgrading from one version to another.

```bash
# Run old version
docker run -d --name codeflash-old \
  -e AZURE_OPENAI_API_KEY=test-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-upgrade-test:/var/lib/postgresql/data \
  codeflash/unified:latest

sleep 20
OLD_KEY=$(docker exec codeflash-old cat /app/API_KEY.txt)

# Stop old container
docker stop codeflash-old
docker rm codeflash-old

# Rebuild image (simulating new version)
docker build -f deployment/onprem-simple/Dockerfile.unifiedall -t codeflash/unified:latest .

# Run new version with same volume
docker run -d --name codeflash-new \
  -e AZURE_OPENAI_API_KEY=test-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-upgrade-test:/var/lib/postgresql/data \
  codeflash/unified:latest

sleep 20
NEW_KEY=$(docker exec codeflash-new cat /app/API_KEY.txt)

# Compare
if [ "$OLD_KEY" = "$NEW_KEY" ]; then
  echo "✅ Upgrade successful - data preserved"
else
  echo "❌ Upgrade failed - data lost"
fi

# Cleanup
docker stop codeflash-new
docker rm codeflash-new
docker volume rm codeflash-upgrade-test
```

**Success criteria:**
- ✅ New version starts successfully
- ✅ Data from old version is preserved
- ✅ API key remains the same
- ✅ No data migration issues

---

## Monitoring and Debugging

### Watch All Logs

```bash
# Follow all logs
docker logs -f codeflash-test

# Filter specific service
docker logs codeflash-test | grep "cf-api"
docker logs codeflash-test | grep "aiservice"
docker logs codeflash-test | grep "postgres"
docker logs codeflash-test | grep "webapp"
```

### Check Resource Usage

```bash
# Container stats
docker stats codeflash-test

# Disk usage
docker system df
du -sh /var/lib/docker/volumes/codeflash-test-data
```

### Interactive Debugging

```bash
# Access container shell
docker exec -it codeflash-test bash

# Check service logs inside container
docker exec codeflash-test tail -f /var/log/supervisor/cf-api-stdout.log
docker exec codeflash-test tail -f /var/log/supervisor/aiservice-stdout.log
docker exec codeflash-test tail -f /var/log/supervisor/postgres-stdout.log
docker exec codeflash-test tail -f /var/log/supervisor/webapp-stdout.log

# Check process tree
docker exec codeflash-test ps auxf
```

---

## Common Test Failures

### Container won't start
```bash
docker logs codeflash-test | grep -i error
docker logs codeflash-test | grep -i fail
```

### Service shows FATAL
```bash
docker exec codeflash-test supervisorctl tail cf-api
docker exec codeflash-test supervisorctl tail aiservice
```

### PostgreSQL not ready
```bash
docker exec codeflash-test pg_isready
docker logs codeflash-test | grep postgres
```

### API key not generated
```bash
docker logs codeflash-test | grep "API Key"
docker exec codeflash-test ls -la /app/API_KEY.txt
```

---

## Automated Test Script

Save this as `test-deployment.sh`:

```bash
#!/bin/bash
set -e

echo "🚀 Starting Codeflash deployment test..."

# Cleanup
echo "🧹 Cleaning up old test containers..."
docker stop codeflash-test 2>/dev/null || true
docker rm codeflash-test 2>/dev/null || true
docker volume rm codeflash-test-data 2>/dev/null || true

# Build
echo "🔨 Building image..."
docker build -f deployment/onprem-simple/Dockerfile.unifiedall -t codeflash/unified:test .

# Run
echo "🏃 Running container..."
docker run -d --name codeflash-test \
  -e AZURE_OPENAI_API_KEY=${AZURE_OPENAI_API_KEY:-test-key} \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-test-data:/var/lib/postgresql/data \
  codeflash/unified:test

# Wait
echo "⏳ Waiting for services to start..."
sleep 20

# Test
echo "🔍 Testing services..."
docker exec codeflash-test supervisorctl status | grep RUNNING || (echo "❌ Services not running" && exit 1)
curl -f http://localhost:3001/cfapi/healthcheck || (echo "❌ cf-api not responding" && exit 1)
curl -f http://localhost:8000/health || (echo "❌ aiservice not responding" && exit 1)
docker exec codeflash-test pg_isready || (echo "❌ PostgreSQL not ready" && exit 1)

# Get API key
echo "🔑 Retrieving API key..."
API_KEY=$(docker exec codeflash-test cat /app/API_KEY.txt)
echo "API Key: $API_KEY"

echo "✅ All tests passed!"
echo ""
echo "To clean up: docker stop codeflash-test && docker rm codeflash-test && docker volume rm codeflash-test-data"
```

Make it executable and run:
```bash
chmod +x test-deployment.sh
./test-deployment.sh
```

---

## Performance Benchmarks

Expected performance metrics:

| Metric | Value |
|--------|-------|
| Build time | 5-10 minutes |
| Startup time | 15-20 seconds |
| Container size | ~5GB |
| Memory usage (idle) | ~500MB |
| Memory usage (active) | ~1-2GB |
| CPU usage (idle) | <5% |
| CPU usage (optimizing) | 50-100% |

---

## Next Steps

After successful testing:
1. Update to your actual AI provider keys
2. Configure custom domain/URLs if needed
3. Set up monitoring and backups
4. Deploy to production environment
5. Document your specific configuration

For production deployment best practices, see the main README.md.