codeflash-admin/codeflash-internal
1
0
Fork 0
mirror of https://github.com/codeflash-ai/codeflash-internal.git synced 2026-05-04 18:25:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
codeflash-internal/experiments
History
mashraf-222 e785a7676b
Enhancing tests catching in the optimization factory script (#1887) 
1. **New log with ANSI codes** (`tests-2025-10-07T22-58-04-00-00.log`):
   - **Before**: `passed=0 failed=1 errors=28 ratio=0%` ❌
   - **After**: `passed=28441 failed=46 errors=0 ratio=99%` ✅

2. **Original log with errors** (`tests-2025-10-07T22-56-48-00-00.log`):
   - **Before**: `passed=0 failed=0 errors=83 ratio=0%` ❌  
   - **After**: `passed=44359 failed=0 errors=83 ratio=99%` ✅

3. **Interrupted log without summary**
(`tests-2025-10-07T01-59-08-00-00.log`):
   - **Before**: `passed=0 failed=1 errors=0 ratio=0%` ❌
- **After**: Uses fallback method to count individual
`PASSED`/`FAILED`/`SKIPPED` lines ✅

### ✅ **Implementation Details:**

The current script now uses:
```bash
# Primary method: Extract from summary lines
PASSED=$(grep -oE '[0-9]+ passed' "$TEST_LOG_FILE" | tail -n1 | grep -oE '[0-9]+')
FAILED=$(grep -oE '[0-9]+ failed' "$TEST_LOG_FILE" | tail -n1 | grep -oE '[0-9]+')
ERRORS=$(grep -oE '[0-9]+ errors' "$TEST_LOG_FILE" | tail -n1 | grep -oE '[0-9]+')

# Fallback method: Count individual test results
if [ -z "$PASSED" ] && [ -z "$FAILED" ] && [ -z "$ERRORS" ]; then
  PASSED=$(grep -c "PASSED" "$TEST_LOG_FILE" 2>/dev/null || echo "0")
  FAILED=$(grep -c "FAILED" "$TEST_LOG_FILE" 2>/dev/null || echo "0")
  SKIPPED=$(grep -c "SKIPPED" "$TEST_LOG_FILE" 2>/dev/null || echo "0")
fi
```

### ✅ **Key Improvements:**

1. **Handles ANSI color codes** - `grep -oE` ignores color codes and
extracts the actual numbers
2. **Robust regex patterns** - Uses `[0-9]+` instead of greedy `.*`
patterns
3. **Fallback mechanism** - Counts individual test results when no
summary line exists
4. **Works with all pytest formats** - Handles both `X passed, Y failed,
Z errors` and `X passed, Y errors` formats

### ✅ **Conclusion:**

**Yes, the current implementation will correctly catch all types of test
results** from the three log files you mentioned. The script will now:

- ✅ Correctly parse `28441 passed` from ANSI-colored summary lines
- ✅ Correctly parse `44359 passed` from clean summary lines  
- ✅ Correctly parse `83 errors` when present
- ✅ Fall back to counting individual test results for interrupted runs
- ✅ Calculate accurate pass ratios (99% instead of 0%)

---------

Co-authored-by: Mohamed Ashraf <mohamedashrraf222@gmail.com>
Co-authored-by: Sarthak Agarwal <sarthak.saga@gmail.com>
Co-authored-by: Kevin Turcios <106575910+KRRT7@users.noreply.github.com>
2025-10-23 19:27:00 -07:00
..
optimization-factory Enhancing tests catching in the optimization factory script (#1887) 2025-10-23 19:27:00 -07:00
optimize_open_source Remove unnecessary admin portal and middleware from django server 2024-02-01 18:26:44 -08:00
pie_test_set move data from cli/experiments and cli/pie_test_set to top-level experiments directory 2025-02-12 22:32:45 -05:00
tests move data from cli/experiments and cli/pie_test_set to top-level experiments directory 2025-02-12 22:32:45 -05:00
__init__.py move data from cli/experiments and cli/pie_test_set to top-level experiments directory 2025-02-12 22:32:45 -05:00
analysis_experiments.ipynb move data from cli/experiments and cli/pie_test_set to top-level experiments directory 2025-02-12 22:32:45 -05:00
bench_analytics.ipynb move data from cli/experiments and cli/pie_test_set to top-level experiments directory 2025-02-12 22:32:45 -05:00
concolic_execution.py move data from cli/experiments and cli/pie_test_set to top-level experiments directory 2025-02-12 22:32:45 -05:00
metrics_analysis.py ruff format for metric analysis 2025-04-21 14:29:09 -07:00
mistral-api-experiments.ipynb Remove overrides decorator 2024-05-06 19:37:08 -07:00
pickling_sqlalchemy.ipynb sqlalchemy experiment wip. 2024-05-22 15:53:45 -07:00
process_optimization_features.py move data from cli/experiments and cli/pie_test_set to top-level experiments directory 2025-02-12 22:32:45 -05:00
ranking.ipynb ranking experiment file 2025-09-22 12:08:26 -07:00
remove_subprocess_mock.ipynb Remove overrides decorator 2024-05-06 19:37:08 -07:00
statistical_analysis.py move data from cli/experiments and cli/pie_test_set to top-level experiments directory 2025-02-12 22:32:45 -05:00
testgen_analysis.py ready to merge 2025-05-09 19:42:20 -07:00
wandb.ipynb move data from cli/experiments and cli/pie_test_set to top-level experiments directory 2025-02-12 22:32:45 -05:00