# Codeflash Org CI Audit

Comprehensive CI/CD audit of the `codeflash-ai` GitHub org: 200+ forks, 2 main repos (`codeflash`, `codeflash-internal`).

## Background

While working on Unstructured optimization case studies, discovered that codeflash-ai org forks were running GitHub Actions unnecessarily — Dependabot updates, upstream scheduled CI, and failing workflows creating ~960 noise runs/month across 26 active forks out of 200+.

Expanded scope to a full audit of CI hygiene across both main repos.

## Results

### Fork CI

| Metric | Before | After |
|---|---|---|
| Forks with active CI | 26 of 200+ | 0 |
| Wasted runs/month | ~960 | 0 |
| Estimated cost/yr (private rates) | $248 | $0 |

Top offenders before disable:

| Repo | Runs (audit period) | Cost/yr | Pattern |
|---|---|---|---|
| sglang | 4,936 | $4 | CI Failure Monitor (cron, skips on fork) |
| kornia | 308 | $226 | Daily macOS + Windows test matrix (91% of all fork cost) |
| ray | 527 | $3 | Dependabot + Stale PR bot |
| lerobot | 330 | $12 | Dependabot + Full Tests |
| next.js | 217 | $3 | Dependabot only |
| 21 others | 694 | <$5 | Dependabot / misc |

Actions disabled on all 200+ forks on 2026-04-23.

### Main Repo — codeflash

| Finding | Impact | Fix |
|---|---|---|
| Wildcard path triggers (`paths: ['**']`) on 12 E2E workflows | Every PR (including README edits) triggered ~2hrs of E2E tests | Targeted path filters (#2025) |
| Broken claude-code-action (v1.0.90 Bedrock OIDC regression) | 60-100% failure rate on Claude Code workflows | Pinned to v1.0.89 (#2026) |
| Dependabot scanning test fixtures (no `dependabot.yml`) | 70% of Dependabot runs failing on `code_to_optimize/` fixtures | Added explicit config excluding fixtures (#2027) |
| 13 ghost workflows (deleted source files, still active in UI) | Actions dashboard clutter, confusing status signals | `gh workflow disable` on all 13 |
| 17 separate workflow files with individual required checks | Workflow-only PRs stuck at "Pending", required admin merge | Consolidated into single `ci.yaml` with gate job (#2044) |
| Stale `codeflash.yaml` | Superseded by `codeflash-optimize.yaml`, still active | Deleted (#2056) |
| No npm/Maven caching, duplicate workflows | Slow JS/Java CI, redundant compute | Consolidated + caching (#2050) |
| Broken Windows paths, outdated action versions | CI failures on Windows matrix | Fixed paths + upgraded actions (#2052, #2053) |
| Flaky E2E tests | False failures blocking merges | Increased test data size + timeout (#2051, #2057) |

### Main Repo — codeflash-internal

| Finding | Impact | Fix |
|---|---|---|
| Deploy AI Service path filter included `.github/workflows/**` | Any workflow edit triggered a production deploy | Scoped to actual service paths (#2588) |
| Claude Code workflow had no path filters | Fired on every PR/comment regardless of files changed | Added `paths-ignore` (#2588) |
| Publish to PyPI permanently disabled (`if: false`) | Created skipped run noise on every push to main | Disabled via API + workflow fix (#2588) |
| Broken claude-code-action (same v1.0.90 regression) | 85% failure rate | Pinned to v1.0.89 (#2587) |

### Operational Improvements

| Before | After |
|---|---|
| 17 individual workflow files, 13 required checks in branch protection | Single `ci.yaml`, 1 gate check (`required-checks-passed`) |
| Workflow-only PRs stuck at "Pending", required admin merge | Skipped jobs report as "skipped", gate accepts them, self-service merge |
| Branch protection rules | Repository rulesets (modern GitHub standard) |
| ~$1.85 compute burned on every non-code PR | ~$0.001 (change detection only) |
| 22 workflow files in repo | 7 workflow files |

## PRs Merged

### codeflash

| PR | Date | Description |
|---|---|---|
| [#2025](https://github.com/codeflash-ai/codeflash/pull/2025) | Apr 9 | Replace wildcard path triggers on 12 E2E workflows |
| [#2026](https://github.com/codeflash-ai/codeflash/pull/2026) | Apr 9 | Pin claude-code-action to v1.0.89 (Bedrock auth fix) |
| [#2027](https://github.com/codeflash-ai/codeflash/pull/2027) | Apr 9 | Exclude test fixtures from Dependabot |
| [#2044](https://github.com/codeflash-ai/codeflash/pull/2044) | Apr 9 | Consolidate 17 workflows into single ci.yaml with gate job |
| [#2047](https://github.com/codeflash-ai/codeflash/pull/2047) | Apr 9 | Path filters, validate-pr action, fetch-depth, continue-on-error |
| [#2050](https://github.com/codeflash-ai/codeflash/pull/2050) | Apr 9 | npm cache, Maven consolidation, remove duplicate workflow |
| [#2052](https://github.com/codeflash-ai/codeflash/pull/2052) | Apr 10 | Upgrade action versions, add uv cache, fix broken paths, DRY publish |
| [#2053](https://github.com/codeflash-ai/codeflash/pull/2053) | Apr 10 | Fix shell: bash for Windows conditional install |
| [#2056](https://github.com/codeflash-ai/codeflash/pull/2056) | Apr 10 | Delete disabled codeflash.yaml workflow |

### codeflash-internal

| PR | Date | Description |
|---|---|---|
| [#2587](https://github.com/codeflash-ai/codeflash-internal/pull/2587) | Apr 9 | Pin claude-code-action to v1.0.89 |
| [#2588](https://github.com/codeflash-ai/codeflash-internal/pull/2588) | Apr 9 | Fix deploy path filter, Claude Code paths, disable PyPI |

### Direct Actions

| Action | Date | Details |
|---|---|---|
| Disabled 13 ghost workflows | Apr 9 | `gh workflow disable` on codeflash |
| Disabled Publish to PyPI workflow | Apr 9 | `gh workflow disable` on codeflash-internal |
| Closed stale Dependabot PR #2012 | Apr 9 | vite bump to test fixture |
| Migrated branch protection to rulesets | Apr 9 | Modern GitHub standard, single gate check |
| Disabled Actions on 200+ forks | Apr 23 | `gh api --method PUT` on all org forks |

## Methodology

1. **Inventory** — `gh repo list codeflash-ai` to enumerate all 200+ repos, classify as fork vs primary
2. **Fork scan** — Query Actions run counts per fork since Apr 2025, identify 26 active forks
3. **Compute cost** — Sample job-level data (duration, runner type), calculate at GitHub rates ($0.008/min Linux, $0.016/min Windows, $0.08/min macOS)
4. **Main repo audit** — List all workflows, check run history, failure rates, ghost detection, trigger config
5. **Root cause analysis** — Compare working vs broken runs by commit SHA + timestamp to pinpoint regressions

## Monitoring

- **anthropics/claude-code-action#1196** — unpin from v1.0.89 once Bedrock OIDC fix lands upstream
- **Dependabot alerts** — 24 known vulnerabilities (1 critical, 8 high) at time of audit; new `dependabot.yml` targets real deps only
- **Fork re-enable** — if a fork is needed for active development: `echo '{"enabled":true}' | gh api --method PUT repos/codeflash-ai/<repo>/actions/permissions --input -`
