codeflash-internal/deployment/onprem-simple/TESTING.md

Codeflash On-Premise Testing Guide

This guide walks through testing the Codeflash on-premise deployment step-by-step.

Quick Test (5 minutes)

This is the fastest way to verify everything works:

1. Build and Run

# Build the image (from repository root)
docker build -f deployment/onprem-simple/Dockerfile.unifiedall -t codeflash/unified:latest .

# Run the container
docker run -d --name codeflash-test \
  -e AZURE_OPENAI_API_KEY=your-azure-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-test-data:/var/lib/postgresql/data \
  codeflash/unified:latest

# Wait ~15 seconds for services to start
sleep 15

2. Verify All Services Are Running

# Check container status
docker ps | grep codeflash-test

# Check all services
docker exec codeflash-test supervisorctl status

Expected output:

postgres                         RUNNING   pid 40, uptime 0:00:15
aiservice                        RUNNING   pid 41, uptime 0:00:15
cf-api                           RUNNING   pid 42, uptime 0:00:15
cf-webapp                        RUNNING   pid 43, uptime 0:00:15

3. Test Service Endpoints

# Test cf-api
curl http://localhost:3001/cfapi/healthcheck
# Expected: {"status":"ok"}

# Test aiservice
curl http://localhost:8000/health
# Expected: {"status":"healthy"}

# Test webapp
curl http://localhost:3000
# Expected: HTML content

4. Get API Key

docker logs codeflash-test | grep "Your API Key"
# Or
docker exec codeflash-test cat /app/API_KEY.txt

5. Test CLI

# Install CLI
pip install codeflash

# Configure environment
export CODEFLASH_API_KEY=cf-your-key-from-step-4
export CODEFLASH_AIS_SERVER=local
export CODEFLASH_CFAPI_SERVER=local

# Test connection
codeflash --help

6. Test Optimization (Optional)

If you have a Python project to test:

cd your-python-project
codeflash --file path/to/file.py --function function_name --no-pr -v

7. Cleanup

docker stop codeflash-test
docker rm codeflash-test
docker volume rm codeflash-test-data

---

Detailed Testing Scenarios

Scenario 1: Fresh Installation Test

Tests a brand new installation from scratch.

# Clean slate
docker stop codeflash-test 2>/dev/null || true
docker rm codeflash-test 2>/dev/null || true
docker volume rm codeflash-test-data 2>/dev/null || true

# Build
docker build -f deployment/onprem-simple/Dockerfile.unifiedall -t codeflash/unified:latest .

# Run
docker run -d --name codeflash-test \
  -e AZURE_OPENAI_API_KEY=your-azure-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-test-data:/var/lib/postgresql/data \
  codeflash/unified:latest

# Wait for startup
sleep 20

# Verify
docker logs codeflash-test
docker exec codeflash-test supervisorctl status
docker exec codeflash-test pg_isready -h localhost -p 5432 -U codeflash

Success criteria:

• ✅ All 4 services show "RUNNING"
• ✅ PostgreSQL is ready
• ✅ API key generated and saved
• ✅ No error messages in logs

Scenario 2: Persistent Data Test

Tests that data persists across container restarts.

# Get initial API key
INITIAL_KEY=$(docker exec codeflash-test cat /app/API_KEY.txt)
echo "Initial API key: $INITIAL_KEY"

# Restart container
docker restart codeflash-test
sleep 15

# Get API key after restart
NEW_KEY=$(docker exec codeflash-test cat /app/API_KEY.txt)
echo "API key after restart: $NEW_KEY"

# Compare
if [ "$INITIAL_KEY" = "$NEW_KEY" ]; then
  echo "✅ Data persisted correctly"
else
  echo "❌ Data did not persist"
fi

Success criteria:

• ✅ API key remains the same after restart
• ✅ All services restart successfully
• ✅ Database data is intact

Scenario 3: Service Restart Test

Tests individual service restarts.

# Restart cf-api
docker exec codeflash-test supervisorctl restart cf-api
sleep 5
curl http://localhost:3001/cfapi/healthcheck

# Restart aiservice
docker exec codeflash-test supervisorctl restart aiservice
sleep 5
curl http://localhost:8000/health

# Restart cf-webapp
docker exec codeflash-test supervisorctl restart cf-webapp
sleep 5
curl http://localhost:3000

# Check all services
docker exec codeflash-test supervisorctl status

Success criteria:

• ✅ Each service restarts without errors
• ✅ Endpoints respond after restart
• ✅ No cascading failures

Scenario 4: Database Test

Tests database connectivity and data.

# Check PostgreSQL is running
docker exec codeflash-test pg_isready -h localhost -p 5432 -U codeflash

# Check database exists
docker exec codeflash-test psql postgresql://codeflash:codeflash@localhost:5432/codeflash \
  -c "\l" | grep codeflash

# Check tables exist
docker exec codeflash-test psql postgresql://codeflash:codeflash@localhost:5432/codeflash \
  -c "\dt" | head -10

# Check user exists
docker exec codeflash-test psql postgresql://codeflash:codeflash@localhost:5432/codeflash \
  -c "SELECT * FROM users;"

# Check API key exists
docker exec codeflash-test psql postgresql://codeflash:codeflash@localhost:5432/codeflash \
  -c "SELECT key, suffix FROM cf_api_keys;"

Success criteria:

• ✅ Database codeflash exists
• ✅ All migrations applied (36+ tables)
• ✅ User created
• ✅ API key stored

Scenario 5: CLI Integration Test

Full end-to-end test with a real Python file.

# Create test project
mkdir -p /tmp/codeflash-test-project
cd /tmp/codeflash-test-project

# Create a simple Python file
cat > test.py << 'EOF'
def fibonacci(n):
    """Calculate fibonacci number recursively (slow)."""
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

def main():
    print(fibonacci(10))

if __name__ == "__main__":
    main()
EOF

# Get API key
export CODEFLASH_API_KEY=$(docker exec codeflash-test cat /app/API_KEY.txt)
export CODEFLASH_AIS_SERVER=local
export CODEFLASH_CFAPI_SERVER=local

# Install CLI
pip install codeflash

# Run optimization
codeflash --file test.py --function fibonacci --no-pr -v

# Check if file was modified
git diff test.py 2>/dev/null || echo "File modified (no git)"

Success criteria:

• ✅ CLI connects successfully
• ✅ Optimization completes without errors
• ✅ Function is optimized (or determined not optimizable)
• ✅ Test file shows improvements

Scenario 6: Port Conflict Test

Tests behavior when ports are already in use.

# Try to run second container (should fail)
docker run -d --name codeflash-test-2 \
  -e AZURE_OPENAI_API_KEY=your-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  codeflash/unified:latest 2>&1 | grep "address already in use"

# Run with different ports (should succeed)
docker run -d --name codeflash-test-2 \
  -e AZURE_OPENAI_API_KEY=your-key \
  -p 15432:5432 -p 18000:8000 -p 13001:3001 -p 13000:3000 \
  -v codeflash-test-2-data:/var/lib/postgresql/data \
  codeflash/unified:latest

sleep 15
docker exec codeflash-test-2 supervisorctl status

# Cleanup
docker stop codeflash-test-2
docker rm codeflash-test-2
docker volume rm codeflash-test-2-data

Success criteria:

• ✅ Second container with same ports fails gracefully
• ✅ Second container with different ports works
• ✅ Both containers can run simultaneously

Scenario 7: Environment Variable Test

Tests various environment variable configurations.

# Test with minimal config
docker run -d --name codeflash-minimal \
  -e AZURE_OPENAI_API_KEY=test-key \
  -p 25432:5432 -p 28000:8000 -p 23001:3001 -p 23000:3000 \
  codeflash/unified:latest

sleep 20
docker logs codeflash-minimal | grep "Generated SECRET_KEY"
docker logs codeflash-minimal | grep "API Key:"

# Test with full config
docker run -d --name codeflash-full \
  -e AZURE_OPENAI_API_KEY=test-key \
  -e SECRET_KEY=my-custom-secret \
  -e NEXT_PUBLIC_APP_URL=http://custom.domain:3000 \
  -e WEBAPP_URL=http://custom.domain:3000 \
  -p 35432:5432 -p 38000:8000 -p 33001:3001 -p 33000:3000 \
  codeflash/unified:latest

sleep 20
docker logs codeflash-full | grep -v "Generated SECRET_KEY"

# Cleanup
docker stop codeflash-minimal codeflash-full
docker rm codeflash-minimal codeflash-full

Success criteria:

• ✅ Minimal config works with auto-generated values
• ✅ Full config respects custom values
• ✅ No required variables missing

Scenario 8: Upgrade Test

Tests upgrading from one version to another.

# Run old version
docker run -d --name codeflash-old \
  -e AZURE_OPENAI_API_KEY=test-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-upgrade-test:/var/lib/postgresql/data \
  codeflash/unified:latest

sleep 20
OLD_KEY=$(docker exec codeflash-old cat /app/API_KEY.txt)

# Stop old container
docker stop codeflash-old
docker rm codeflash-old

# Rebuild image (simulating new version)
docker build -f deployment/onprem-simple/Dockerfile.unifiedall -t codeflash/unified:latest .

# Run new version with same volume
docker run -d --name codeflash-new \
  -e AZURE_OPENAI_API_KEY=test-key \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-upgrade-test:/var/lib/postgresql/data \
  codeflash/unified:latest

sleep 20
NEW_KEY=$(docker exec codeflash-new cat /app/API_KEY.txt)

# Compare
if [ "$OLD_KEY" = "$NEW_KEY" ]; then
  echo "✅ Upgrade successful - data preserved"
else
  echo "❌ Upgrade failed - data lost"
fi

# Cleanup
docker stop codeflash-new
docker rm codeflash-new
docker volume rm codeflash-upgrade-test

Success criteria:

• ✅ New version starts successfully
• ✅ Data from old version is preserved
• ✅ API key remains the same
• ✅ No data migration issues

---

Monitoring and Debugging

Watch All Logs

# Follow all logs
docker logs -f codeflash-test

# Filter specific service
docker logs codeflash-test | grep "cf-api"
docker logs codeflash-test | grep "aiservice"
docker logs codeflash-test | grep "postgres"
docker logs codeflash-test | grep "webapp"

Check Resource Usage

# Container stats
docker stats codeflash-test

# Disk usage
docker system df
du -sh /var/lib/docker/volumes/codeflash-test-data

Interactive Debugging

# Access container shell
docker exec -it codeflash-test bash

# Check service logs inside container
docker exec codeflash-test tail -f /var/log/supervisor/cf-api-stdout.log
docker exec codeflash-test tail -f /var/log/supervisor/aiservice-stdout.log
docker exec codeflash-test tail -f /var/log/supervisor/postgres-stdout.log
docker exec codeflash-test tail -f /var/log/supervisor/webapp-stdout.log

# Check process tree
docker exec codeflash-test ps auxf

---

Common Test Failures

Container won't start

docker logs codeflash-test | grep -i error
docker logs codeflash-test | grep -i fail

Service shows FATAL

docker exec codeflash-test supervisorctl tail cf-api
docker exec codeflash-test supervisorctl tail aiservice

PostgreSQL not ready

docker exec codeflash-test pg_isready
docker logs codeflash-test | grep postgres

API key not generated

docker logs codeflash-test | grep "API Key"
docker exec codeflash-test ls -la /app/API_KEY.txt

---

Automated Test Script

Save this as test-deployment.sh:

#!/bin/bash
set -e

echo "🚀 Starting Codeflash deployment test..."

# Cleanup
echo "🧹 Cleaning up old test containers..."
docker stop codeflash-test 2>/dev/null || true
docker rm codeflash-test 2>/dev/null || true
docker volume rm codeflash-test-data 2>/dev/null || true

# Build
echo "🔨 Building image..."
docker build -f deployment/onprem-simple/Dockerfile.unifiedall -t codeflash/unified:test .

# Run
echo "🏃 Running container..."
docker run -d --name codeflash-test \
  -e AZURE_OPENAI_API_KEY=${AZURE_OPENAI_API_KEY:-test-key} \
  -p 5432:5432 -p 8000:8000 -p 3001:3001 -p 3000:3000 \
  -v codeflash-test-data:/var/lib/postgresql/data \
  codeflash/unified:test

# Wait
echo "⏳ Waiting for services to start..."
sleep 20

# Test
echo "🔍 Testing services..."
docker exec codeflash-test supervisorctl status | grep RUNNING || (echo "❌ Services not running" && exit 1)
curl -f http://localhost:3001/cfapi/healthcheck || (echo "❌ cf-api not responding" && exit 1)
curl -f http://localhost:8000/health || (echo "❌ aiservice not responding" && exit 1)
docker exec codeflash-test pg_isready || (echo "❌ PostgreSQL not ready" && exit 1)

# Get API key
echo "🔑 Retrieving API key..."
API_KEY=$(docker exec codeflash-test cat /app/API_KEY.txt)
echo "API Key: $API_KEY"

echo "✅ All tests passed!"
echo ""
echo "To clean up: docker stop codeflash-test && docker rm codeflash-test && docker volume rm codeflash-test-data"

Make it executable and run:

chmod +x test-deployment.sh
./test-deployment.sh

---

Performance Benchmarks

Expected performance metrics:

Metric	Value
Build time	5-10 minutes
Startup time	15-20 seconds
Container size	~5GB
Memory usage (idle)	~500MB
Memory usage (active)	~1-2GB
CPU usage (idle)	<5%
CPU usage (optimizing)	50-100%

---

Next Steps

After successful testing:

1. Update to your actual AI provider keys
2. Configure custom domain/URLs if needed
3. Set up monitoring and backups
4. Deploy to production environment
5. Document your specific configuration

For production deployment best practices, see the main README.md.