mirror of
https://github.com/codeflash-ai/codeflash-agent.git
synced 2026-05-04 18:25:19 +00:00
Add codeflash org CI audit case study and interactive Dash report
Case study in .codeflash/krrt7/codeflash-ai/ci-audit/ with README, status, and raw data (fork activity, PRs merged). Interactive Dash report in reports/codeflash-ci-audit/ with two tabs: Executive Summary (hero metrics, cost impact charts, before/after) and Full Detail (fork breakdown, findings table, PR inventory, methodology). Key numbers: 71% fewer workflow runs, ~$12K/yr in Enterprise overage savings, 200+ forks disabled, 11 PRs merged across 2 repos.
This commit is contained in:
Kevin Turcios
parent
8221ce32a2
commit
c492164fbf
7 changed files with 1259 additions and 0 deletions
112
.codeflash/krrt7/codeflash-ai/ci-audit/README.md
Normal file
112
.codeflash/krrt7/codeflash-ai/ci-audit/README.md
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
# Codeflash Org CI Audit
|
||||
|
||||
Comprehensive CI/CD audit of the `codeflash-ai` GitHub org: 200+ forks, 2 main repos (`codeflash`, `codeflash-internal`).
|
||||
|
||||
## Background
|
||||
|
||||
While working on Unstructured optimization case studies, discovered that codeflash-ai org forks were running GitHub Actions unnecessarily — Dependabot updates, upstream scheduled CI, and failing workflows creating ~960 noise runs/month across 26 active forks out of 200+.
|
||||
|
||||
Expanded scope to a full audit of CI hygiene across both main repos.
|
||||
|
||||
## Results
|
||||
|
||||
### Fork CI
|
||||
|
||||
| Metric | Before | After |
|
||||
|---|---|---|
|
||||
| Forks with active CI | 26 of 200+ | 0 |
|
||||
| Wasted runs/month | ~960 | 0 |
|
||||
| Estimated cost/yr (private rates) | $248 | $0 |
|
||||
|
||||
Top offenders before disable:
|
||||
|
||||
| Repo | Runs (audit period) | Cost/yr | Pattern |
|
||||
|---|---|---|---|
|
||||
| sglang | 4,936 | $4 | CI Failure Monitor (cron, skips on fork) |
|
||||
| kornia | 308 | $226 | Daily macOS + Windows test matrix (91% of all fork cost) |
|
||||
| ray | 527 | $3 | Dependabot + Stale PR bot |
|
||||
| lerobot | 330 | $12 | Dependabot + Full Tests |
|
||||
| next.js | 217 | $3 | Dependabot only |
|
||||
| 21 others | 694 | <$5 | Dependabot / misc |
|
||||
|
||||
Actions disabled on all 200+ forks on 2026-04-23.
|
||||
|
||||
### Main Repo — codeflash
|
||||
|
||||
| Finding | Impact | Fix |
|
||||
|---|---|---|
|
||||
| Wildcard path triggers (`paths: ['**']`) on 12 E2E workflows | Every PR (including README edits) triggered ~2hrs of E2E tests | Targeted path filters (#2025) |
|
||||
| Broken claude-code-action (v1.0.90 Bedrock OIDC regression) | 60-100% failure rate on Claude Code workflows | Pinned to v1.0.89 (#2026) |
|
||||
| Dependabot scanning test fixtures (no `dependabot.yml`) | 70% of Dependabot runs failing on `code_to_optimize/` fixtures | Added explicit config excluding fixtures (#2027) |
|
||||
| 13 ghost workflows (deleted source files, still active in UI) | Actions dashboard clutter, confusing status signals | `gh workflow disable` on all 13 |
|
||||
| 17 separate workflow files with individual required checks | Workflow-only PRs stuck at "Pending", required admin merge | Consolidated into single `ci.yaml` with gate job (#2044) |
|
||||
| Stale `codeflash.yaml` | Superseded by `codeflash-optimize.yaml`, still active | Deleted (#2056) |
|
||||
| No npm/Maven caching, duplicate workflows | Slow JS/Java CI, redundant compute | Consolidated + caching (#2050) |
|
||||
| Broken Windows paths, outdated action versions | CI failures on Windows matrix | Fixed paths + upgraded actions (#2052, #2053) |
|
||||
| Flaky E2E tests | False failures blocking merges | Increased test data size + timeout (#2051, #2057) |
|
||||
|
||||
### Main Repo — codeflash-internal
|
||||
|
||||
| Finding | Impact | Fix |
|
||||
|---|---|---|
|
||||
| Deploy AI Service path filter included `.github/workflows/**` | Any workflow edit triggered a production deploy | Scoped to actual service paths (#2588) |
|
||||
| Claude Code workflow had no path filters | Fired on every PR/comment regardless of files changed | Added `paths-ignore` (#2588) |
|
||||
| Publish to PyPI permanently disabled (`if: false`) | Created skipped run noise on every push to main | Disabled via API + workflow fix (#2588) |
|
||||
| Broken claude-code-action (same v1.0.90 regression) | 85% failure rate | Pinned to v1.0.89 (#2587) |
|
||||
|
||||
### Operational Improvements
|
||||
|
||||
| Before | After |
|
||||
|---|---|
|
||||
| 17 individual workflow files, 13 required checks in branch protection | Single `ci.yaml`, 1 gate check (`required-checks-passed`) |
|
||||
| Workflow-only PRs stuck at "Pending", required admin merge | Skipped jobs report as "skipped", gate accepts them, self-service merge |
|
||||
| Branch protection rules | Repository rulesets (modern GitHub standard) |
|
||||
| ~$1.85 compute burned on every non-code PR | ~$0.001 (change detection only) |
|
||||
| 22 workflow files in repo | 7 workflow files |
|
||||
|
||||
## PRs Merged
|
||||
|
||||
### codeflash
|
||||
|
||||
| PR | Date | Description |
|
||||
|---|---|---|
|
||||
| [#2025](https://github.com/codeflash-ai/codeflash/pull/2025) | Apr 9 | Replace wildcard path triggers on 12 E2E workflows |
|
||||
| [#2026](https://github.com/codeflash-ai/codeflash/pull/2026) | Apr 9 | Pin claude-code-action to v1.0.89 (Bedrock auth fix) |
|
||||
| [#2027](https://github.com/codeflash-ai/codeflash/pull/2027) | Apr 9 | Exclude test fixtures from Dependabot |
|
||||
| [#2044](https://github.com/codeflash-ai/codeflash/pull/2044) | Apr 9 | Consolidate 17 workflows into single ci.yaml with gate job |
|
||||
| [#2047](https://github.com/codeflash-ai/codeflash/pull/2047) | Apr 9 | Path filters, validate-pr action, fetch-depth, continue-on-error |
|
||||
| [#2050](https://github.com/codeflash-ai/codeflash/pull/2050) | Apr 9 | npm cache, Maven consolidation, remove duplicate workflow |
|
||||
| [#2052](https://github.com/codeflash-ai/codeflash/pull/2052) | Apr 10 | Upgrade action versions, add uv cache, fix broken paths, DRY publish |
|
||||
| [#2053](https://github.com/codeflash-ai/codeflash/pull/2053) | Apr 10 | Fix shell: bash for Windows conditional install |
|
||||
| [#2056](https://github.com/codeflash-ai/codeflash/pull/2056) | Apr 10 | Delete disabled codeflash.yaml workflow |
|
||||
|
||||
### codeflash-internal
|
||||
|
||||
| PR | Date | Description |
|
||||
|---|---|---|
|
||||
| [#2587](https://github.com/codeflash-ai/codeflash-internal/pull/2587) | Apr 9 | Pin claude-code-action to v1.0.89 |
|
||||
| [#2588](https://github.com/codeflash-ai/codeflash-internal/pull/2588) | Apr 9 | Fix deploy path filter, Claude Code paths, disable PyPI |
|
||||
|
||||
### Direct Actions
|
||||
|
||||
| Action | Date | Details |
|
||||
|---|---|---|
|
||||
| Disabled 13 ghost workflows | Apr 9 | `gh workflow disable` on codeflash |
|
||||
| Disabled Publish to PyPI workflow | Apr 9 | `gh workflow disable` on codeflash-internal |
|
||||
| Closed stale Dependabot PR #2012 | Apr 9 | vite bump to test fixture |
|
||||
| Migrated branch protection to rulesets | Apr 9 | Modern GitHub standard, single gate check |
|
||||
| Disabled Actions on 200+ forks | Apr 23 | `gh api --method PUT` on all org forks |
|
||||
|
||||
## Methodology
|
||||
|
||||
1. **Inventory** — `gh repo list codeflash-ai` to enumerate all 200+ repos, classify as fork vs primary
|
||||
2. **Fork scan** — Query Actions run counts per fork since Apr 2025, identify 26 active forks
|
||||
3. **Compute cost** — Sample job-level data (duration, runner type), calculate at GitHub rates ($0.008/min Linux, $0.016/min Windows, $0.08/min macOS)
|
||||
4. **Main repo audit** — List all workflows, check run history, failure rates, ghost detection, trigger config
|
||||
5. **Root cause analysis** — Compare working vs broken runs by commit SHA + timestamp to pinpoint regressions
|
||||
|
||||
## Monitoring
|
||||
|
||||
- **anthropics/claude-code-action#1196** — unpin from v1.0.89 once Bedrock OIDC fix lands upstream
|
||||
- **Dependabot alerts** — 24 known vulnerabilities (1 critical, 8 high) at time of audit; new `dependabot.yml` targets real deps only
|
||||
- **Fork re-enable** — if a fork is needed for active development: `echo '{"enabled":true}' | gh api --method PUT repos/codeflash-ai/<repo>/actions/permissions --input -`
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
repo runs_audit_period cost_yr_usd runner_types pattern
|
||||
sglang 4936 4 Linux CI Failure Monitor (cron, skips on fork)
|
||||
ray 527 3 Linux Dependabot + Stale PR bot
|
||||
lerobot 330 12 Linux Dependabot + Full Tests
|
||||
kornia 308 226 Linux + Windows + macOS Daily scheduled CPU test matrix
|
||||
next.js 217 3 Linux Dependabot only
|
||||
strapi 118 1 Linux Dependabot (lodash across 20+ packages)
|
||||
rasa 85 1 Linux Dependabot
|
||||
Rocket.Chat 80 1 Linux Dependabot
|
||||
nest 73 1 Linux Dependabot (@nestjs/core across 20+ samples)
|
||||
openclaw 68 1 Linux Dependabot
|
||||
appsmith 64 1 Linux Dependabot
|
||||
other_15 206 1 Linux Dependabot / misc
|
||||
|
12
.codeflash/krrt7/codeflash-ai/ci-audit/data/prs-merged.tsv
Normal file
12
.codeflash/krrt7/codeflash-ai/ci-audit/data/prs-merged.tsv
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
pr repo date title
|
||||
2025 codeflash 2026-04-09 ci: replace wildcard path triggers on E2E tests
|
||||
2026 codeflash 2026-04-09 ci: pin claude-code-action to v1.0.89 (fix Bedrock auth)
|
||||
2027 codeflash 2026-04-09 ci: exclude test fixtures from Dependabot
|
||||
2044 codeflash 2026-04-09 ci: consolidate required checks into single gate workflow
|
||||
2047 codeflash 2026-04-09 ci: consolidate improvements — paths, validate-pr action, fetch-depth, continue-on-error
|
||||
2050 codeflash 2026-04-09 Optimize CI: npm cache, Maven consolidation, remove duplicate workflow
|
||||
2052 codeflash 2026-04-10 ci: upgrade action versions, add uv cache, fix broken paths, DRY publish
|
||||
2053 codeflash 2026-04-10 fix(ci): add shell: bash to conditional install step for Windows
|
||||
2056 codeflash 2026-04-10 chore: delete disabled codeflash.yaml workflow
|
||||
2587 codeflash-internal 2026-04-09 ci: pin claude-code-action to v1.0.89
|
||||
2588 codeflash-internal 2026-04-09 ci: fix deploy path filter, Claude Code paths, disable PyPI
|
||||
|
27
.codeflash/krrt7/codeflash-ai/ci-audit/status.md
Normal file
27
.codeflash/krrt7/codeflash-ai/ci-audit/status.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
# ci-audit Status
|
||||
|
||||
Last updated: 2026-04-23
|
||||
|
||||
## Current state
|
||||
|
||||
Complete. All audit findings addressed, all forks disabled, all PRs merged.
|
||||
|
||||
## What was completed
|
||||
|
||||
### Session 1 (Apr 9)
|
||||
- Full CI audit of codeflash-ai org (93 forks at the time + 2 main repos)
|
||||
- 5 PRs merged: path triggers (#2025), claude-code-action pin (#2026, #2587), Dependabot config (#2027), internal fixes (#2588)
|
||||
- 13 ghost workflows disabled
|
||||
- CI gate workflow consolidated (#2044)
|
||||
- Additional CI improvements (#2047, #2050, #2052, #2053, #2056)
|
||||
- Branch protection migrated to rulesets
|
||||
- Audit report written to `~/Desktop/work/cf_org/codeflash-ci-audit/`
|
||||
|
||||
### Session 2 (Apr 23)
|
||||
- Disabled GitHub Actions on all 200+ org forks (previously blocked by missing org admin)
|
||||
- Created case study in `.codeflash/krrt7/codeflash-ai/ci-audit/`
|
||||
|
||||
## Remaining / monitoring
|
||||
|
||||
- Unpin claude-code-action once anthropics/claude-code-action#1196 is fixed upstream
|
||||
- Re-enable Actions on specific forks if needed for active optimization work
|
||||
919
reports/codeflash-ci-audit/app.py
Normal file
919
reports/codeflash-ci-audit/app.py
Normal file
|
|
@ -0,0 +1,919 @@
|
|||
"""Codeflash Org — CI Audit Report
|
||||
|
||||
Two-tab report served at http://localhost:8051/:
|
||||
1. Executive Summary — hero metrics, key findings, before/after
|
||||
2. Full Detail — per-PR inventory, fork breakdown, methodology
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import plotly.graph_objects as go
|
||||
from dash import Dash, Input, Output, clientside_callback, dash_table, dcc, html
|
||||
from theme import (
|
||||
ACCENT,
|
||||
AMBER,
|
||||
BG,
|
||||
BLUE,
|
||||
CARD,
|
||||
CARD_BG,
|
||||
CARD_BORDER,
|
||||
DARK,
|
||||
FONT,
|
||||
GRAY,
|
||||
GREEN,
|
||||
GRID_OVERLAY,
|
||||
LIGHT_GRAY,
|
||||
LIGHT_GREEN,
|
||||
LIGHT_RED,
|
||||
MONO,
|
||||
PURPLE,
|
||||
RED,
|
||||
SLATE,
|
||||
TABLE_CELL,
|
||||
TABLE_DATA,
|
||||
TABLE_DATA_CONDITIONAL,
|
||||
TABLE_HEADER,
|
||||
TABLE_WRAP,
|
||||
WHITE,
|
||||
)
|
||||
|
||||
# ── Data ────────────────────────────────────────────────────────────────────
|
||||
_DATA = json.loads((Path(__file__).parent / "data.json").read_text())
|
||||
|
||||
CODEFLASH_BASE = _DATA["codeflash_base"]
|
||||
INTERNAL_BASE = _DATA["internal_base"]
|
||||
FORK_CI = _DATA["fork_ci"]
|
||||
FINDINGS = _DATA["findings"]
|
||||
PRS_MERGED = _DATA["prs_merged"]
|
||||
DIRECT_ACTIONS = _DATA["direct_actions"]
|
||||
OPS = _DATA["operational_before_after"]
|
||||
RUN_VOL = _DATA["run_volume"]
|
||||
BILLING = _DATA["billing"]
|
||||
|
||||
REPO_BASES = {
|
||||
"codeflash": CODEFLASH_BASE,
|
||||
"codeflash-internal": INTERNAL_BASE,
|
||||
}
|
||||
|
||||
|
||||
# ── Helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def hero_metric(value, label, detail, color=GREEN):
|
||||
return html.Div(
|
||||
[
|
||||
html.Div(
|
||||
value,
|
||||
style={
|
||||
"fontSize": "42px",
|
||||
"fontWeight": "800",
|
||||
"color": color,
|
||||
"lineHeight": "1",
|
||||
"letterSpacing": "-0.02em",
|
||||
"fontFamily": FONT,
|
||||
},
|
||||
),
|
||||
html.Div(
|
||||
label,
|
||||
style={
|
||||
"fontSize": "15px",
|
||||
"fontWeight": "600",
|
||||
"color": SLATE,
|
||||
"marginTop": "8px",
|
||||
},
|
||||
),
|
||||
html.Div(
|
||||
detail,
|
||||
style={"fontSize": "13px", "color": GRAY, "marginTop": "4px"},
|
||||
),
|
||||
],
|
||||
style={
|
||||
"background": CARD_BG,
|
||||
"borderRadius": "16px",
|
||||
"padding": "32px 24px",
|
||||
"textAlign": "center",
|
||||
"flex": "1 1 0%",
|
||||
"minWidth": "0",
|
||||
"border": f"1px solid {CARD_BORDER}",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def section(title, subtitle=None):
|
||||
children = [
|
||||
html.H2(
|
||||
title,
|
||||
style={
|
||||
"fontSize": "22px",
|
||||
"fontWeight": "700",
|
||||
"color": SLATE,
|
||||
"margin": "0",
|
||||
"fontFamily": FONT,
|
||||
"letterSpacing": "-0.01em",
|
||||
},
|
||||
)
|
||||
]
|
||||
if subtitle:
|
||||
children.append(
|
||||
html.P(
|
||||
subtitle,
|
||||
style={
|
||||
"fontSize": "14px",
|
||||
"color": GRAY,
|
||||
"margin": "6px 0 0",
|
||||
"lineHeight": "1.5",
|
||||
},
|
||||
)
|
||||
)
|
||||
return html.Div(children, style={"margin": "56px 0 24px"})
|
||||
|
||||
|
||||
def card(children, **kw):
|
||||
style = {**CARD}
|
||||
for k, v in kw.items():
|
||||
style[k] = v
|
||||
return html.Div(children, style=style)
|
||||
|
||||
|
||||
def _logo_lockup(bolt_size="20px", text_size="24px", gap="12px", offset="4px"):
|
||||
return html.Div(
|
||||
[
|
||||
html.Span(
|
||||
"⚡",
|
||||
style={
|
||||
"fontSize": bolt_size,
|
||||
"position": "relative",
|
||||
"top": offset,
|
||||
},
|
||||
),
|
||||
html.Span(
|
||||
"codeflash",
|
||||
style={
|
||||
"fontSize": text_size,
|
||||
"fontWeight": "800",
|
||||
"color": ACCENT,
|
||||
"fontFamily": FONT,
|
||||
"letterSpacing": "-0.03em",
|
||||
},
|
||||
),
|
||||
],
|
||||
style={"display": "inline-flex", "alignItems": "center", "gap": gap},
|
||||
)
|
||||
|
||||
|
||||
def _badge(text, color):
|
||||
return html.Span(
|
||||
text,
|
||||
style={
|
||||
"padding": "2px 10px",
|
||||
"borderRadius": "999px",
|
||||
"fontSize": "11px",
|
||||
"fontWeight": "700",
|
||||
"background": color,
|
||||
"color": DARK,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ── Charts ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def make_fork_chart():
|
||||
"""Horizontal bar: fork CI runs by repo."""
|
||||
repos = [f["repo"] for f in FORK_CI[:8]]
|
||||
runs = [f["runs"] for f in FORK_CI[:8]]
|
||||
repos.reverse()
|
||||
runs.reverse()
|
||||
|
||||
fig = go.Figure()
|
||||
fig.add_trace(
|
||||
go.Bar(
|
||||
y=repos,
|
||||
x=runs,
|
||||
orientation="h",
|
||||
marker_color=[ACCENT if r > 500 else BLUE for r in runs],
|
||||
marker_cornerradius=4,
|
||||
text=[f"{r:,}" for r in runs],
|
||||
textposition="outside",
|
||||
textfont={"size": 12, "color": SLATE},
|
||||
)
|
||||
)
|
||||
fig.update_layout(
|
||||
plot_bgcolor="rgba(0,0,0,0)",
|
||||
paper_bgcolor="rgba(0,0,0,0)",
|
||||
font={"family": FONT, "size": 13, "color": SLATE},
|
||||
xaxis={"title": "Workflow Runs (audit period)", "gridcolor": CARD_BORDER, "zeroline": False},
|
||||
yaxis={"title": "", "automargin": True},
|
||||
margin={"t": 10, "b": 50, "l": 10, "r": 60},
|
||||
height=320,
|
||||
showlegend=False,
|
||||
)
|
||||
return fig
|
||||
|
||||
|
||||
def make_fork_cost_chart():
|
||||
"""Pie chart: fork CI cost breakdown."""
|
||||
labels = [f["repo"] for f in FORK_CI if f["cost_yr"] >= 3]
|
||||
values = [f["cost_yr"] for f in FORK_CI if f["cost_yr"] >= 3]
|
||||
other = sum(f["cost_yr"] for f in FORK_CI if f["cost_yr"] < 3)
|
||||
if other > 0:
|
||||
labels.append("All others")
|
||||
values.append(other)
|
||||
|
||||
colors = [ACCENT, BLUE, GREEN, PURPLE, AMBER, GRAY, LIGHT_GRAY, RED][:len(labels)]
|
||||
|
||||
fig = go.Figure()
|
||||
fig.add_trace(
|
||||
go.Pie(
|
||||
labels=labels,
|
||||
values=values,
|
||||
hole=0.55,
|
||||
marker={"colors": colors},
|
||||
textinfo="label+percent",
|
||||
textfont={"size": 12, "color": WHITE},
|
||||
hovertemplate="%{label}: $%{value}/yr<extra></extra>",
|
||||
)
|
||||
)
|
||||
fig.update_layout(
|
||||
plot_bgcolor="rgba(0,0,0,0)",
|
||||
paper_bgcolor="rgba(0,0,0,0)",
|
||||
font={"family": FONT, "size": 13, "color": SLATE},
|
||||
margin={"t": 10, "b": 10, "l": 10, "r": 10},
|
||||
height=300,
|
||||
showlegend=False,
|
||||
annotations=[
|
||||
{
|
||||
"text": f"${sum(values)}/yr",
|
||||
"x": 0.5,
|
||||
"y": 0.5,
|
||||
"font": {"size": 18, "color": ACCENT, "family": MONO},
|
||||
"showarrow": False,
|
||||
}
|
||||
],
|
||||
)
|
||||
return fig
|
||||
|
||||
|
||||
def make_before_after_chart():
|
||||
"""Grouped bar: operational metrics before vs after."""
|
||||
cats = ["Workflow Files", "Required Checks", "Failing Fork Runs/mo"]
|
||||
before = [OPS["workflow_files"][0], OPS["required_checks"][0], OPS["fork_failing_runs_monthly"][0]]
|
||||
after = [OPS["workflow_files"][1], OPS["required_checks"][1], OPS["fork_failing_runs_monthly"][1]]
|
||||
|
||||
fig = go.Figure()
|
||||
fig.add_trace(
|
||||
go.Bar(
|
||||
name="Before",
|
||||
x=cats,
|
||||
y=before,
|
||||
marker_color=LIGHT_GRAY,
|
||||
marker_cornerradius=6,
|
||||
text=[str(v) for v in before],
|
||||
textposition="outside",
|
||||
textfont={"size": 13, "color": GRAY},
|
||||
)
|
||||
)
|
||||
fig.add_trace(
|
||||
go.Bar(
|
||||
name="After",
|
||||
x=cats,
|
||||
y=after,
|
||||
marker_color=GREEN,
|
||||
marker_cornerradius=6,
|
||||
text=[str(v) for v in after],
|
||||
textposition="outside",
|
||||
textfont={"size": 13, "color": GREEN},
|
||||
)
|
||||
)
|
||||
fig.update_layout(
|
||||
barmode="group",
|
||||
bargap=0.3,
|
||||
bargroupgap=0.1,
|
||||
plot_bgcolor="rgba(0,0,0,0)",
|
||||
paper_bgcolor="rgba(0,0,0,0)",
|
||||
font={"family": FONT, "size": 13, "color": SLATE},
|
||||
yaxis={"gridcolor": CARD_BORDER, "zeroline": False},
|
||||
xaxis={"title": ""},
|
||||
margin={"t": 20, "b": 60, "l": 50, "r": 20},
|
||||
legend={
|
||||
"orientation": "h",
|
||||
"yanchor": "bottom",
|
||||
"y": 1.05,
|
||||
"xanchor": "center",
|
||||
"x": 0.5,
|
||||
"font": {"size": 13},
|
||||
},
|
||||
height=360,
|
||||
)
|
||||
return fig
|
||||
|
||||
|
||||
def make_run_volume_chart():
|
||||
"""Bar chart: monthly workflow runs before vs after."""
|
||||
months = ["Dec '25", "Jan '26", "Feb '26", "Mar '26", "Apr '26\n(projected)"]
|
||||
runs = [4150, 9391, 21307, 14753, RUN_VOL["codeflash_apr_projected"]]
|
||||
|
||||
fig = go.Figure()
|
||||
fig.add_trace(
|
||||
go.Bar(
|
||||
x=months,
|
||||
y=runs,
|
||||
marker_color=[LIGHT_GRAY, LIGHT_GRAY, RED, AMBER, GREEN],
|
||||
marker_cornerradius=6,
|
||||
text=[f"{r:,}" for r in runs],
|
||||
textposition="outside",
|
||||
textfont={"size": 13, "color": SLATE},
|
||||
)
|
||||
)
|
||||
fig.add_hline(
|
||||
y=RUN_VOL["codeflash_apr_projected"],
|
||||
line_dash="dot",
|
||||
line_color=GREEN,
|
||||
opacity=0.5,
|
||||
)
|
||||
fig.add_annotation(
|
||||
x="Feb '26",
|
||||
y=21307,
|
||||
text="Audit starts (Apr 9)",
|
||||
showarrow=True,
|
||||
arrowhead=2,
|
||||
arrowcolor=ACCENT,
|
||||
ax=40,
|
||||
ay=-40,
|
||||
font={"size": 12, "color": ACCENT},
|
||||
)
|
||||
fig.update_layout(
|
||||
plot_bgcolor="rgba(0,0,0,0)",
|
||||
paper_bgcolor="rgba(0,0,0,0)",
|
||||
font={"family": FONT, "size": 13, "color": SLATE},
|
||||
yaxis={"title": "Workflow Runs", "gridcolor": CARD_BORDER, "zeroline": False},
|
||||
xaxis={"title": ""},
|
||||
margin={"t": 20, "b": 60, "l": 70, "r": 20},
|
||||
height=360,
|
||||
showlegend=False,
|
||||
)
|
||||
return fig
|
||||
|
||||
|
||||
def make_billing_chart():
|
||||
"""Stacked bar: Enterprise minutes allotment vs overage."""
|
||||
cats = ["Before (Feb)", "After (Apr)"]
|
||||
included = [BILLING["enterprise_included_min"], BILLING["enterprise_included_min"]]
|
||||
overage = [BILLING["overage_before_min"], BILLING["overage_after_min"]]
|
||||
|
||||
fig = go.Figure()
|
||||
fig.add_trace(
|
||||
go.Bar(
|
||||
name="Included (50K)",
|
||||
x=cats,
|
||||
y=included,
|
||||
marker_color=BLUE,
|
||||
marker_cornerradius=6,
|
||||
)
|
||||
)
|
||||
fig.add_trace(
|
||||
go.Bar(
|
||||
name="Overage",
|
||||
x=cats,
|
||||
y=overage,
|
||||
marker_color=RED,
|
||||
marker_cornerradius=6,
|
||||
text=[f"{v:,} min\n(${v * 0.008:,.0f}/mo)" for v in overage],
|
||||
textposition="outside",
|
||||
textfont={"size": 12, "color": RED},
|
||||
)
|
||||
)
|
||||
fig.update_layout(
|
||||
barmode="stack",
|
||||
plot_bgcolor="rgba(0,0,0,0)",
|
||||
paper_bgcolor="rgba(0,0,0,0)",
|
||||
font={"family": FONT, "size": 13, "color": SLATE},
|
||||
yaxis={"title": "Billed Minutes/month", "gridcolor": CARD_BORDER, "zeroline": False},
|
||||
xaxis={"title": ""},
|
||||
margin={"t": 40, "b": 60, "l": 70, "r": 20},
|
||||
legend={
|
||||
"orientation": "h",
|
||||
"yanchor": "bottom",
|
||||
"y": 1.05,
|
||||
"xanchor": "center",
|
||||
"x": 0.5,
|
||||
"font": {"size": 13},
|
||||
},
|
||||
height=380,
|
||||
)
|
||||
return fig
|
||||
|
||||
|
||||
# ── Tab: Executive Summary ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _build_summary_tab():
|
||||
return html.Div(
|
||||
id="summary-view",
|
||||
children=[
|
||||
# Hero metrics
|
||||
html.Div(
|
||||
style={"display": "flex", "gap": "16px", "marginTop": "32px", "flexWrap": "wrap"},
|
||||
children=[
|
||||
hero_metric(f"~${BILLING['overage_saved_annual_usd']:,}/yr", "Overage Savings", "Enterprise minutes overage reduced 58%", GREEN),
|
||||
hero_metric(f"{RUN_VOL['codeflash_reduction_pct']}%", "Fewer Runs", f"{RUN_VOL['codeflash_feb']:,} → {RUN_VOL['codeflash_apr_projected']:,}/mo", ACCENT),
|
||||
hero_metric("200+", "Forks Disabled", "GitHub Actions turned off org-wide", BLUE),
|
||||
hero_metric("22 → 7", "Workflows Consolidated", "Single ci.yaml with gate job", PURPLE),
|
||||
],
|
||||
),
|
||||
|
||||
section(
|
||||
"What We Found",
|
||||
"Full CI/CD audit of the codeflash-ai GitHub org: 200+ forks and 2 main repos.",
|
||||
),
|
||||
|
||||
# Key findings grid
|
||||
html.Div(
|
||||
style={"display": "grid", "gridTemplateColumns": "1fr 1fr", "gap": "16px"},
|
||||
children=[
|
||||
card([
|
||||
html.Div("Fork CI Waste", style={"fontWeight": "700", "color": ACCENT, "fontSize": "16px", "marginBottom": "12px"}),
|
||||
html.P(
|
||||
"26 of 200+ forks were running GitHub Actions — Dependabot updates, upstream scheduled CI, "
|
||||
"and failing workflows creating ~960 noise runs/month. kornia alone was 91% of fork CI cost "
|
||||
"due to a daily macOS + Windows test matrix.",
|
||||
style={"color": GRAY, "fontSize": "14px", "lineHeight": "1.6", "margin": "0"},
|
||||
),
|
||||
]),
|
||||
card([
|
||||
html.Div("Wildcard Path Triggers", style={"fontWeight": "700", "color": RED, "fontSize": "16px", "marginBottom": "12px"}),
|
||||
html.P(
|
||||
"All 12 E2E workflows used paths: ['**'] — any file change (README, docs) "
|
||||
"triggered the full E2E suite. A single docs-only PR burned ~2 hours of compute.",
|
||||
style={"color": GRAY, "fontSize": "14px", "lineHeight": "1.6", "margin": "0"},
|
||||
),
|
||||
]),
|
||||
card([
|
||||
html.Div("Ghost Workflows", style={"fontWeight": "700", "color": AMBER, "fontSize": "16px", "marginBottom": "12px"}),
|
||||
html.P(
|
||||
"13 workflow files had been deleted from the repo but their entries remained active in "
|
||||
"GitHub Actions. These cluttered the Actions UI and created confusing status signals.",
|
||||
style={"color": GRAY, "fontSize": "14px", "lineHeight": "1.6", "margin": "0"},
|
||||
),
|
||||
]),
|
||||
card([
|
||||
html.Div("Broken claude-code-action", style={"fontWeight": "700", "color": RED, "fontSize": "16px", "marginBottom": "12px"}),
|
||||
html.P(
|
||||
"v1.0.90 broke Bedrock OIDC auth. Every Claude Code run was failing with 403s. "
|
||||
"60-100% failure rate on codeflash, 85% on codeflash-internal.",
|
||||
style={"color": GRAY, "fontSize": "14px", "lineHeight": "1.6", "margin": "0"},
|
||||
),
|
||||
]),
|
||||
],
|
||||
),
|
||||
|
||||
section(
|
||||
"Run Volume & Cost Impact",
|
||||
"Workflow runs dropped 71%. Enterprise minutes overage cut from ~215K to ~90K/month.",
|
||||
),
|
||||
|
||||
# Run volume + billing side by side
|
||||
html.Div(
|
||||
style={"display": "grid", "gridTemplateColumns": "1fr 1fr", "gap": "16px"},
|
||||
children=[
|
||||
card([
|
||||
html.Div("Monthly Workflow Runs (codeflash)", style={"fontWeight": "700", "color": SLATE, "fontSize": "15px", "marginBottom": "12px"}),
|
||||
dcc.Graph(figure=make_run_volume_chart(), config={"displayModeBar": False}),
|
||||
]),
|
||||
card([
|
||||
html.Div("Enterprise Minutes Billing", style={"fontWeight": "700", "color": SLATE, "fontSize": "15px", "marginBottom": "12px"}),
|
||||
html.P(
|
||||
f"50K included minutes/month. Overage dropped from ~{BILLING['overage_before_min']:,} to ~{BILLING['overage_after_min']:,} min "
|
||||
f"(${BILLING['overage_saved_monthly_usd']:,}/month saved).",
|
||||
style={"color": GRAY, "fontSize": "13px", "lineHeight": "1.5", "margin": "0 0 12px"},
|
||||
),
|
||||
dcc.Graph(figure=make_billing_chart(), config={"displayModeBar": False}),
|
||||
]),
|
||||
],
|
||||
),
|
||||
|
||||
section("Before vs After"),
|
||||
card([dcc.Graph(figure=make_before_after_chart(), config={"displayModeBar": False})]),
|
||||
|
||||
section(
|
||||
"Operational Improvements",
|
||||
"The audit transformed CI from a maintenance burden to a self-service system.",
|
||||
),
|
||||
|
||||
# Before/after comparison table
|
||||
card([
|
||||
html.Div(
|
||||
style={"display": "grid", "gridTemplateColumns": "1fr auto auto", "gap": "0"},
|
||||
children=[
|
||||
# Header
|
||||
html.Div("", style={"padding": "12px 16px"}),
|
||||
html.Div("Before", style={"padding": "12px 16px", "fontWeight": "700", "color": RED, "fontSize": "13px", "textAlign": "right", "width": "200px"}),
|
||||
html.Div("After", style={"padding": "12px 16px", "fontWeight": "700", "color": GREEN, "fontSize": "13px", "textAlign": "right", "width": "200px"}),
|
||||
# Rows
|
||||
*_comparison_row("Workflow files in repo", "22", "7"),
|
||||
*_comparison_row("Required checks in branch protection", "13 individual", "1 gate job"),
|
||||
*_comparison_row("Workflow-only PR merge", "Admin override", "Self-service"),
|
||||
*_comparison_row("Non-code PR compute cost", "$1.85", "$0.001"),
|
||||
*_comparison_row("Fork failing runs/month", "~960", "0"),
|
||||
*_comparison_row("Ghost workflows in Actions UI", "13", "0"),
|
||||
*_comparison_row("Branch protection model", "Legacy rules", "Repository rulesets"),
|
||||
*_comparison_row("Dependabot test fixture noise", "70% failure rate", "Excluded"),
|
||||
],
|
||||
),
|
||||
]),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def _comparison_row(label, before, after):
|
||||
border = f"1px solid {CARD_BORDER}"
|
||||
return [
|
||||
html.Div(label, style={"padding": "12px 16px", "color": SLATE, "fontSize": "14px", "fontWeight": "600", "borderTop": border}),
|
||||
html.Div(before, style={"padding": "12px 16px", "color": LIGHT_GRAY, "fontSize": "14px", "fontFamily": MONO, "textAlign": "right", "borderTop": border, "width": "200px"}),
|
||||
html.Div(after, style={"padding": "12px 16px", "color": GREEN, "fontSize": "14px", "fontFamily": MONO, "fontWeight": "600", "textAlign": "right", "borderTop": border, "width": "200px"}),
|
||||
]
|
||||
|
||||
|
||||
# ── Tab: Full Detail ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _build_detail_tab():
|
||||
pr_rows = []
|
||||
for p in PRS_MERGED:
|
||||
base = REPO_BASES[p["repo"]]
|
||||
pr_rows.append({
|
||||
"PR": f"[#{p['pr']}]({base}/{p['pr']})",
|
||||
"Repo": p["repo"],
|
||||
"Date": p["date"],
|
||||
"Description": p["title"],
|
||||
})
|
||||
|
||||
action_rows = [
|
||||
{"Action": a["action"], "Date": a["date"], "Repo": a["repo"]}
|
||||
for a in DIRECT_ACTIONS
|
||||
]
|
||||
|
||||
fork_rows = [
|
||||
{"Repo": f["repo"], "Runs": f"{f['runs']:,}", "Cost/yr": f"${f['cost_yr']}", "Runners": f["runners"], "Pattern": f["pattern"]}
|
||||
for f in FORK_CI
|
||||
]
|
||||
|
||||
finding_rows = []
|
||||
for repo, items in FINDINGS.items():
|
||||
for f in items:
|
||||
pr_link = f"[#{f['pr']}]({REPO_BASES[repo]}/{f['pr']})" if f["pr"] else "Direct action"
|
||||
finding_rows.append({
|
||||
"Repo": repo,
|
||||
"Finding": f["finding"],
|
||||
"Impact": f["impact"],
|
||||
"Fix": f["fix"],
|
||||
"PR": pr_link,
|
||||
})
|
||||
|
||||
return html.Div(
|
||||
id="detail-view",
|
||||
style={"display": "none"},
|
||||
children=[
|
||||
section("Fork CI Activity", "26 of 200+ forks had active GitHub Actions. Actions disabled on all forks 2026-04-23."),
|
||||
|
||||
# Fork charts side by side
|
||||
html.Div(
|
||||
style={"display": "grid", "gridTemplateColumns": "1fr 1fr", "gap": "16px"},
|
||||
children=[
|
||||
card([
|
||||
html.Div("Runs by Repository", style={"fontWeight": "700", "color": SLATE, "fontSize": "15px", "marginBottom": "12px"}),
|
||||
dcc.Graph(figure=make_fork_chart(), config={"displayModeBar": False}),
|
||||
]),
|
||||
card([
|
||||
html.Div("Cost Breakdown", style={"fontWeight": "700", "color": SLATE, "fontSize": "15px", "marginBottom": "12px"}),
|
||||
html.P(
|
||||
"kornia is 91% of fork CI cost: daily scheduled test matrix across macOS ($0.08/min), Windows ($0.016/min), and Linux.",
|
||||
style={"color": GRAY, "fontSize": "13px", "lineHeight": "1.5", "margin": "0 0 12px"},
|
||||
),
|
||||
dcc.Graph(figure=make_fork_cost_chart(), config={"displayModeBar": False}),
|
||||
]),
|
||||
],
|
||||
),
|
||||
|
||||
# Fork table
|
||||
html.Div(style=TABLE_WRAP, children=[
|
||||
dash_table.DataTable(
|
||||
data=fork_rows,
|
||||
columns=[
|
||||
{"name": "Repo", "id": "Repo"},
|
||||
{"name": "Runs", "id": "Runs"},
|
||||
{"name": "Cost/yr", "id": "Cost/yr"},
|
||||
{"name": "Runners", "id": "Runners"},
|
||||
{"name": "Pattern", "id": "Pattern"},
|
||||
],
|
||||
style_header=TABLE_HEADER,
|
||||
style_cell=TABLE_CELL,
|
||||
style_data=TABLE_DATA,
|
||||
style_data_conditional=TABLE_DATA_CONDITIONAL,
|
||||
style_as_list_view=True,
|
||||
page_size=20,
|
||||
),
|
||||
]),
|
||||
|
||||
section("All Findings", "Categorized by repository."),
|
||||
|
||||
html.Div(style=TABLE_WRAP, children=[
|
||||
dash_table.DataTable(
|
||||
data=finding_rows,
|
||||
columns=[
|
||||
{"name": "Repo", "id": "Repo"},
|
||||
{"name": "Finding", "id": "Finding"},
|
||||
{"name": "Impact", "id": "Impact"},
|
||||
{"name": "Fix", "id": "Fix"},
|
||||
{"name": "PR", "id": "PR", "presentation": "markdown"},
|
||||
],
|
||||
style_header=TABLE_HEADER,
|
||||
style_cell={**TABLE_CELL, "whiteSpace": "normal", "height": "auto"},
|
||||
style_data=TABLE_DATA,
|
||||
style_data_conditional=TABLE_DATA_CONDITIONAL,
|
||||
style_as_list_view=True,
|
||||
page_size=20,
|
||||
css=[{"selector": "p", "rule": "margin: 0"}],
|
||||
),
|
||||
]),
|
||||
|
||||
section("PRs Merged", f"{len(PRS_MERGED)} pull requests across 2 repositories."),
|
||||
|
||||
html.Div(style=TABLE_WRAP, children=[
|
||||
dash_table.DataTable(
|
||||
data=pr_rows,
|
||||
columns=[
|
||||
{"name": "PR", "id": "PR", "presentation": "markdown"},
|
||||
{"name": "Repo", "id": "Repo"},
|
||||
{"name": "Date", "id": "Date"},
|
||||
{"name": "Description", "id": "Description"},
|
||||
],
|
||||
style_header=TABLE_HEADER,
|
||||
style_cell={**TABLE_CELL, "whiteSpace": "normal", "height": "auto"},
|
||||
style_data=TABLE_DATA,
|
||||
style_data_conditional=TABLE_DATA_CONDITIONAL,
|
||||
style_as_list_view=True,
|
||||
css=[{"selector": "p", "rule": "margin: 0"}],
|
||||
),
|
||||
]),
|
||||
|
||||
section("Direct Actions", "Non-PR changes applied during the audit."),
|
||||
|
||||
html.Div(style=TABLE_WRAP, children=[
|
||||
dash_table.DataTable(
|
||||
data=action_rows,
|
||||
columns=[
|
||||
{"name": "Action", "id": "Action"},
|
||||
{"name": "Date", "id": "Date"},
|
||||
{"name": "Repo", "id": "Repo"},
|
||||
],
|
||||
style_header=TABLE_HEADER,
|
||||
style_cell={**TABLE_CELL, "whiteSpace": "normal", "height": "auto"},
|
||||
style_data=TABLE_DATA,
|
||||
style_data_conditional=TABLE_DATA_CONDITIONAL,
|
||||
style_as_list_view=True,
|
||||
css=[{"selector": "p", "rule": "margin: 0"}],
|
||||
),
|
||||
]),
|
||||
|
||||
section("Methodology"),
|
||||
card([
|
||||
html.Ol(
|
||||
[
|
||||
html.Li([html.Strong("Inventory"), " — ", html.Span("gh repo list codeflash-ai", style={"fontFamily": MONO, "fontSize": "13px"}), " to enumerate all 200+ repos, classify as fork vs primary"], style=_li_style()),
|
||||
html.Li([html.Strong("Fork scan"), " — Query Actions run counts per fork since Apr 2025, identify 26 active forks"], style=_li_style()),
|
||||
html.Li([html.Strong("Compute cost"), " — Sample job-level data (duration, runner type), calculate at GitHub rates: $0.008/min Linux, $0.016/min Windows, $0.08/min macOS"], style=_li_style()),
|
||||
html.Li([html.Strong("Main repo audit"), " — List all workflows, check run history, failure rates, ghost detection, trigger configuration"], style=_li_style()),
|
||||
html.Li([html.Strong("Root cause analysis"), " — Compare working vs broken runs by commit SHA and timestamp to pinpoint regressions"], style=_li_style()),
|
||||
],
|
||||
style={"paddingLeft": "20px", "margin": "0"},
|
||||
),
|
||||
]),
|
||||
|
||||
section("Monitoring"),
|
||||
card([
|
||||
html.Ul(
|
||||
[
|
||||
html.Li([html.Strong("claude-code-action"), " — unpin from v1.0.89 once anthropics/claude-code-action#1196 lands upstream"], style=_li_style()),
|
||||
html.Li([html.Strong("Dependabot alerts"), " — 24 known vulnerabilities at audit time; new dependabot.yml targets real deps only"], style=_li_style()),
|
||||
html.Li([html.Strong("Fork re-enable"), " — if a fork is needed: ", html.Code("echo '{\"enabled\":true}' | gh api --method PUT repos/codeflash-ai/<repo>/actions/permissions --input -", style={"fontFamily": MONO, "fontSize": "12px", "color": ACCENT})], style=_li_style()),
|
||||
],
|
||||
style={"paddingLeft": "20px", "margin": "0", "listStyleType": "'\\2022 '"},
|
||||
),
|
||||
]),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def _li_style():
|
||||
return {"color": GRAY, "fontSize": "14px", "lineHeight": "1.7", "marginBottom": "8px"}
|
||||
|
||||
|
||||
# ── Main layout ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
_TAB_BTN_STYLE = {
|
||||
"padding": "10px 24px",
|
||||
"border": "none",
|
||||
"borderRadius": "8px",
|
||||
"cursor": "pointer",
|
||||
"fontSize": "14px",
|
||||
"fontWeight": "600",
|
||||
"fontFamily": FONT,
|
||||
"background": "transparent",
|
||||
"color": GRAY,
|
||||
"transition": "all 0.2s",
|
||||
}
|
||||
|
||||
_TAB_BTN_ACTIVE = {**_TAB_BTN_STYLE, "background": ACCENT, "color": DARK}
|
||||
|
||||
|
||||
def _main_layout():
|
||||
return html.Div(
|
||||
style={
|
||||
"fontFamily": FONT,
|
||||
"background": BG,
|
||||
"color": SLATE,
|
||||
"minHeight": "100vh",
|
||||
"position": "relative",
|
||||
},
|
||||
children=[
|
||||
html.Div(style=GRID_OVERLAY),
|
||||
html.Div(
|
||||
style={
|
||||
"maxWidth": "1100px",
|
||||
"margin": "0 auto",
|
||||
"padding": "48px 32px 80px",
|
||||
"position": "relative",
|
||||
"zIndex": "1",
|
||||
},
|
||||
children=[
|
||||
# Header
|
||||
html.Div(
|
||||
style={"textAlign": "center", "marginBottom": "8px"},
|
||||
children=[
|
||||
_logo_lockup(),
|
||||
html.H1(
|
||||
"CI/CD Audit Report",
|
||||
style={
|
||||
"fontSize": "36px",
|
||||
"fontWeight": "800",
|
||||
"color": WHITE,
|
||||
"margin": "16px 0 8px",
|
||||
"letterSpacing": "-0.02em",
|
||||
},
|
||||
),
|
||||
html.P(
|
||||
"codeflash-ai org — 200+ forks, 2 main repos",
|
||||
style={
|
||||
"fontSize": "16px",
|
||||
"color": GRAY,
|
||||
"margin": "0 0 4px",
|
||||
},
|
||||
),
|
||||
html.P(
|
||||
"April 9–23, 2026",
|
||||
style={
|
||||
"fontSize": "14px",
|
||||
"color": LIGHT_GRAY,
|
||||
"margin": "0",
|
||||
"fontFamily": MONO,
|
||||
},
|
||||
),
|
||||
],
|
||||
),
|
||||
|
||||
# Tab buttons
|
||||
html.Div(
|
||||
style={"display": "flex", "justifyContent": "center", "margin": "40px 0 8px"},
|
||||
children=[
|
||||
html.Div(
|
||||
style={
|
||||
"display": "inline-flex",
|
||||
"background": CARD_BG,
|
||||
"borderRadius": "12px",
|
||||
"padding": "4px",
|
||||
"border": f"1px solid {CARD_BORDER}",
|
||||
},
|
||||
children=[
|
||||
html.Button("Executive Summary", id="btn-summary", n_clicks=1, style=_TAB_BTN_ACTIVE),
|
||||
html.Button("Full Detail", id="btn-detail", n_clicks=0, style=_TAB_BTN_STYLE),
|
||||
],
|
||||
),
|
||||
],
|
||||
),
|
||||
|
||||
_build_summary_tab(),
|
||||
_build_detail_tab(),
|
||||
|
||||
# Footer
|
||||
html.Div(
|
||||
style={
|
||||
"textAlign": "center",
|
||||
"marginTop": "64px",
|
||||
"paddingTop": "24px",
|
||||
"borderTop": f"1px solid {CARD_BORDER}",
|
||||
},
|
||||
children=[
|
||||
html.Div(
|
||||
_logo_lockup("16px", "20px", "10px", "3px"),
|
||||
style={"display": "flex", "justifyContent": "center", "marginBottom": "4px"},
|
||||
),
|
||||
html.P(
|
||||
"CI/CD Audit Report — April 2026",
|
||||
style={"color": LIGHT_GRAY, "fontSize": "13px", "margin": "0"},
|
||||
),
|
||||
],
|
||||
),
|
||||
],
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
# ── App ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
app = Dash(
|
||||
__name__,
|
||||
meta_tags=[
|
||||
{"name": "viewport", "content": "width=device-width, initial-scale=1"},
|
||||
{"property": "og:title", "content": "Codeflash — CI/CD Audit Report"},
|
||||
{
|
||||
"property": "og:description",
|
||||
"content": "CI audit of codeflash-ai org: 200+ forks disabled, 11 PRs merged, 960 failing runs/month eliminated",
|
||||
},
|
||||
],
|
||||
suppress_callback_exceptions=True,
|
||||
)
|
||||
app.title = "Codeflash — CI/CD Audit Report"
|
||||
|
||||
app.index_string = """<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
{%metas%}
|
||||
<title>{%title%}</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;600;700;800&family=JetBrains+Mono:wght@400;600;700&display=swap" rel="stylesheet">
|
||||
{%favicon%}
|
||||
{%css%}
|
||||
<style>
|
||||
.dash-table-container .dash-cell a,
|
||||
.dash-table-container .cell-markdown a,
|
||||
.dash-table-container a,
|
||||
.dash-spreadsheet a { color: #60a5fa !important; text-decoration: none !important; }
|
||||
.dash-table-container a:hover,
|
||||
.dash-spreadsheet a:hover { text-decoration: underline !important; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
{%app_entry%}
|
||||
<footer>
|
||||
{%config%}
|
||||
{%scripts%}
|
||||
{%renderer%}
|
||||
</footer>
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
app.layout = _main_layout
|
||||
|
||||
# ── Toggle callback ──
|
||||
clientside_callback(
|
||||
"""
|
||||
function(summary_c, detail_c) {
|
||||
summary_c = summary_c || 0;
|
||||
detail_c = detail_c || 0;
|
||||
var base = {
|
||||
"padding": "10px 24px", "border": "none", "borderRadius": "8px",
|
||||
"cursor": "pointer", "fontSize": "14px", "fontWeight": "600",
|
||||
"fontFamily": "'Inter', system-ui, -apple-system, sans-serif",
|
||||
"transition": "all 0.2s"
|
||||
};
|
||||
var active = Object.assign({}, base, {"background": "#ffd227", "color": "#09090b"});
|
||||
var inactive = Object.assign({}, base, {"background": "transparent", "color": "#a1a1aa"});
|
||||
var show = {"display": "block"};
|
||||
var hide = {"display": "none"};
|
||||
var mx = Math.max(summary_c, detail_c);
|
||||
if (detail_c === mx && detail_c > 0)
|
||||
return [hide, show, inactive, active];
|
||||
return [show, hide, active, inactive];
|
||||
}
|
||||
""",
|
||||
Output("summary-view", "style"),
|
||||
Output("detail-view", "style"),
|
||||
Output("btn-summary", "style"),
|
||||
Output("btn-detail", "style"),
|
||||
Input("btn-summary", "n_clicks"),
|
||||
Input("btn-detail", "n_clicks"),
|
||||
)
|
||||
|
||||
server = app.server
|
||||
|
||||
if __name__ == "__main__":
|
||||
app.run(
|
||||
debug=os.getenv("DASH_DEBUG", "1") == "1",
|
||||
port=int(os.getenv("PORT", "8051")),
|
||||
)
|
||||
96
reports/codeflash-ci-audit/data.json
Normal file
96
reports/codeflash-ci-audit/data.json
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
{
|
||||
"codeflash_base": "https://github.com/codeflash-ai/codeflash/pull",
|
||||
"internal_base": "https://github.com/codeflash-ai/codeflash-internal/pull",
|
||||
"audit_date": "2026-04-09",
|
||||
"completion_date": "2026-04-23",
|
||||
"org": "codeflash-ai",
|
||||
"total_forks": 200,
|
||||
"active_forks": 26,
|
||||
"fork_runs_audit_period": 7012,
|
||||
"fork_runs_monthly": 23373,
|
||||
"fork_failing_monthly": 960,
|
||||
"fork_cost_yr": 248,
|
||||
"ghost_workflows_disabled": 13,
|
||||
"workflows_before": 22,
|
||||
"workflows_after": 7,
|
||||
"required_checks_before": 13,
|
||||
"required_checks_after": 1,
|
||||
"skip_savings_per_pr": 1.85,
|
||||
"fork_ci": [
|
||||
{"repo": "sglang", "runs": 4936, "cost_yr": 4, "runners": "Linux", "pattern": "CI Failure Monitor (cron, skips on fork)"},
|
||||
{"repo": "kornia", "runs": 308, "cost_yr": 226, "runners": "Linux + Windows + macOS", "pattern": "Daily scheduled CPU test matrix"},
|
||||
{"repo": "ray", "runs": 527, "cost_yr": 3, "runners": "Linux", "pattern": "Dependabot + Stale PR bot"},
|
||||
{"repo": "lerobot", "runs": 330, "cost_yr": 12, "runners": "Linux", "pattern": "Dependabot + Full Tests"},
|
||||
{"repo": "next.js", "runs": 217, "cost_yr": 3, "runners": "Linux", "pattern": "Dependabot only"},
|
||||
{"repo": "strapi", "runs": 118, "cost_yr": 1, "runners": "Linux", "pattern": "Dependabot (lodash x 20+ packages)"},
|
||||
{"repo": "rasa", "runs": 85, "cost_yr": 1, "runners": "Linux", "pattern": "Dependabot"},
|
||||
{"repo": "Rocket.Chat", "runs": 80, "cost_yr": 1, "runners": "Linux", "pattern": "Dependabot"},
|
||||
{"repo": "nest", "runs": 73, "cost_yr": 1, "runners": "Linux", "pattern": "Dependabot (@nestjs/core x 20+ samples)"},
|
||||
{"repo": "openclaw", "runs": 68, "cost_yr": 1, "runners": "Linux", "pattern": "Dependabot"},
|
||||
{"repo": "appsmith", "runs": 64, "cost_yr": 1, "runners": "Linux", "pattern": "Dependabot"},
|
||||
{"repo": "15 others", "runs": 206, "cost_yr": 1, "runners": "Linux", "pattern": "Dependabot / misc"}
|
||||
],
|
||||
"findings": {
|
||||
"codeflash": [
|
||||
{"finding": "Wildcard path triggers on 12 E2E workflows", "impact": "Every PR triggered ~2hrs of E2E tests", "fix": "Targeted path filters", "pr": 2025},
|
||||
{"finding": "Broken claude-code-action (v1.0.90 Bedrock OIDC)", "impact": "60-100% failure rate on Claude Code workflows", "fix": "Pinned to v1.0.89", "pr": 2026},
|
||||
{"finding": "Dependabot scanning test fixtures", "impact": "70% of Dependabot runs failing", "fix": "Added dependabot.yml excluding code_to_optimize/", "pr": 2027},
|
||||
{"finding": "13 ghost workflows (deleted source files)", "impact": "Actions dashboard clutter, confusing signals", "fix": "gh workflow disable on all 13", "pr": null},
|
||||
{"finding": "17 separate workflow files, 13 required checks", "impact": "Workflow-only PRs stuck at Pending, admin merge required", "fix": "Single ci.yaml with gate job", "pr": 2044},
|
||||
{"finding": "No npm/Maven caching, duplicate workflows", "impact": "Slow JS/Java CI, redundant compute", "fix": "Consolidated + caching", "pr": 2050},
|
||||
{"finding": "Outdated action versions, broken Windows paths", "impact": "CI failures on Windows matrix", "fix": "Upgraded actions + fixed paths", "pr": 2052},
|
||||
{"finding": "Stale codeflash.yaml", "impact": "Superseded by codeflash-optimize.yaml", "fix": "Deleted", "pr": 2056}
|
||||
],
|
||||
"codeflash-internal": [
|
||||
{"finding": "Deploy AI Service path included .github/workflows/**", "impact": "Any workflow edit triggered production deploy", "fix": "Scoped to actual service paths", "pr": 2588},
|
||||
{"finding": "Claude Code workflow had no path filters", "impact": "Fired on every PR/comment", "fix": "Added paths-ignore", "pr": 2588},
|
||||
{"finding": "Publish to PyPI permanently disabled (if: false)", "impact": "Skipped run noise on every push to main", "fix": "Disabled via API", "pr": 2588},
|
||||
{"finding": "Broken claude-code-action (same v1.0.90 regression)", "impact": "85% failure rate", "fix": "Pinned to v1.0.89", "pr": 2587}
|
||||
]
|
||||
},
|
||||
"prs_merged": [
|
||||
{"pr": 2025, "repo": "codeflash", "date": "2026-04-09", "title": "Replace wildcard path triggers on 12 E2E workflows"},
|
||||
{"pr": 2026, "repo": "codeflash", "date": "2026-04-09", "title": "Pin claude-code-action to v1.0.89 (Bedrock auth fix)"},
|
||||
{"pr": 2027, "repo": "codeflash", "date": "2026-04-09", "title": "Exclude test fixtures from Dependabot"},
|
||||
{"pr": 2044, "repo": "codeflash", "date": "2026-04-09", "title": "Consolidate 17 workflows into single ci.yaml with gate job"},
|
||||
{"pr": 2047, "repo": "codeflash", "date": "2026-04-09", "title": "Path filters, validate-pr action, fetch-depth, continue-on-error"},
|
||||
{"pr": 2050, "repo": "codeflash", "date": "2026-04-09", "title": "npm cache, Maven consolidation, remove duplicate workflow"},
|
||||
{"pr": 2052, "repo": "codeflash", "date": "2026-04-10", "title": "Upgrade action versions, add uv cache, fix broken paths, DRY publish"},
|
||||
{"pr": 2053, "repo": "codeflash", "date": "2026-04-10", "title": "Fix shell: bash for Windows conditional install"},
|
||||
{"pr": 2056, "repo": "codeflash", "date": "2026-04-10", "title": "Delete disabled codeflash.yaml workflow"},
|
||||
{"pr": 2587, "repo": "codeflash-internal", "date": "2026-04-09", "title": "Pin claude-code-action to v1.0.89"},
|
||||
{"pr": 2588, "repo": "codeflash-internal", "date": "2026-04-09", "title": "Fix deploy path filter, Claude Code paths, disable PyPI"}
|
||||
],
|
||||
"direct_actions": [
|
||||
{"action": "Disabled 13 ghost workflows", "date": "2026-04-09", "repo": "codeflash"},
|
||||
{"action": "Disabled Publish to PyPI workflow", "date": "2026-04-09", "repo": "codeflash-internal"},
|
||||
{"action": "Closed stale Dependabot PR #2012", "date": "2026-04-09", "repo": "codeflash"},
|
||||
{"action": "Migrated branch protection to rulesets", "date": "2026-04-09", "repo": "codeflash"},
|
||||
{"action": "Disabled Actions on 200+ forks", "date": "2026-04-23", "repo": "org-wide"}
|
||||
],
|
||||
"operational_before_after": {
|
||||
"workflow_files": [22, 7],
|
||||
"required_checks": [13, 1],
|
||||
"non_code_pr_cost": [1.85, 0.001],
|
||||
"fork_failing_runs_monthly": [960, 0],
|
||||
"admin_merges_needed": true
|
||||
},
|
||||
"run_volume": {
|
||||
"codeflash_feb": 21307,
|
||||
"codeflash_apr_projected": 6219,
|
||||
"codeflash_reduction_pct": 71,
|
||||
"internal_feb": 2494,
|
||||
"internal_apr_projected": 3864,
|
||||
"total_eliminated_monthly": 16000
|
||||
},
|
||||
"billing": {
|
||||
"enterprise_included_min": 50000,
|
||||
"overage_before_min": 214872,
|
||||
"overage_after_min": 89613,
|
||||
"overage_saved_min": 125259,
|
||||
"overage_saved_monthly_usd": 1002,
|
||||
"overage_saved_annual_usd": 12025,
|
||||
"non_code_pr_before_sec": 780,
|
||||
"non_code_pr_after_sec": 8
|
||||
}
|
||||
}
|
||||
80
reports/codeflash-ci-audit/theme.py
Normal file
80
reports/codeflash-ci-audit/theme.py
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
"""Theme and styling constants for the Codeflash CI Audit report."""
|
||||
|
||||
# ── Colors (Codeflash dark - amber/zinc) ────────────────────────────────────
|
||||
ACCENT = "#ffd227"
|
||||
DARK = "#09090b"
|
||||
CARD_BG = "rgba(16,20,28,0.7)" # dark navy/70 — readable over animated bg
|
||||
CARD_BORDER = "rgba(63,63,70,0.35)" # zinc-700/35
|
||||
SLATE = "#e4e4e7"
|
||||
GRAY = "#a1a1aa"
|
||||
LIGHT_GRAY = "#71717a"
|
||||
BG = "#0d1117" # dark navy (codeflash.ai style)
|
||||
WHITE = "#fafafa"
|
||||
GREEN = "#4ade80"
|
||||
LIGHT_GREEN = "rgba(74,222,128,0.12)"
|
||||
RED = "#f87171"
|
||||
LIGHT_RED = "rgba(248,113,113,0.12)"
|
||||
AMBER = "#fbbf24"
|
||||
BLUE = "#60a5fa"
|
||||
PURPLE = "#a78bfa"
|
||||
PINK = "#f472b6"
|
||||
|
||||
# ── Grid overlay ────────────────────────────────────────────────────────────
|
||||
# Matches the roadmap page's subtle grid pattern.
|
||||
GRID_BG_IMAGE = (
|
||||
"linear-gradient(to right, currentColor 1px, transparent 1px),"
|
||||
"linear-gradient(to bottom, currentColor 1px, transparent 1px)"
|
||||
)
|
||||
GRID_BG_SIZE = "48px 48px"
|
||||
GRID_OVERLAY = {
|
||||
"position": "fixed",
|
||||
"top": 0,
|
||||
"left": 0,
|
||||
"right": 0,
|
||||
"bottom": 0,
|
||||
"backgroundImage": GRID_BG_IMAGE,
|
||||
"backgroundSize": GRID_BG_SIZE,
|
||||
"opacity": "0.05",
|
||||
"pointerEvents": "none",
|
||||
"zIndex": "0",
|
||||
}
|
||||
|
||||
# ── Component styles ────────────────────────────────────────────────────────
|
||||
CARD = {
|
||||
"background": CARD_BG,
|
||||
"borderRadius": "16px",
|
||||
"padding": "28px 32px",
|
||||
"border": f"1px solid {CARD_BORDER}",
|
||||
}
|
||||
FONT = "'Inter', system-ui, -apple-system, sans-serif"
|
||||
MONO = "'JetBrains Mono', 'Menlo', monospace"
|
||||
|
||||
# ── Table styles ────────────────────────────────────────────────────────────
|
||||
TABLE_HEADER: dict[str, str] = {
|
||||
"backgroundColor": "rgba(24,24,27,0.8)",
|
||||
"color": ACCENT,
|
||||
"fontWeight": "600",
|
||||
"fontSize": "13px",
|
||||
"padding": "12px 16px",
|
||||
"borderBottom": f"1px solid {CARD_BORDER}",
|
||||
}
|
||||
TABLE_CELL: dict[str, str] = {
|
||||
"textAlign": "left",
|
||||
"padding": "12px 16px",
|
||||
"fontSize": "13px",
|
||||
"fontFamily": FONT,
|
||||
"border": "none",
|
||||
"color": SLATE,
|
||||
}
|
||||
TABLE_DATA: dict[str, str] = {
|
||||
"backgroundColor": "rgba(24,24,27,0.5)",
|
||||
"color": SLATE,
|
||||
}
|
||||
TABLE_DATA_CONDITIONAL: list[dict[str, object]] = [
|
||||
{"if": {"row_index": "odd"}, "backgroundColor": "rgba(31,31,35,0.6)"}
|
||||
]
|
||||
TABLE_WRAP: dict[str, str] = {
|
||||
"borderRadius": "16px",
|
||||
"overflow": "hidden",
|
||||
"border": f"1px solid {CARD_BORDER}",
|
||||
}
|
||||
Loading…
Reference in a new issue