codeflash-ai/codeflash
1
0
Fork 0
Code Issues 18 Pull requests 33 Projects Releases Packages Wiki Activity Actions 4.9k
codeflash/tiles/codeflash-skills/evals/scenario-2/task.md
Kevin Turcios ff2abd29f2 chore: add eval scenarios for codeflash-skills tile 
5 scenarios testing: sequential debugging, Result type + effort config,
test patterns, domain type conventions, and deduplication/repair mechanics.
Also adds tessl-labs/tessl-skill-eval-scenarios dev dependency.
2026-02-14 21:24:54 -05:00

1 KiB
Raw Blame History

Add Candidate Timeout Feature

Context

The codeflash optimization engine currently has no per-candidate timeout. Some candidates take too long during verification, wasting the optimization budget. A new feature is needed to skip candidates that exceed a configurable time limit during behavioral testing.

The timeout should vary based on the optimization effort setting — shorter timeouts for low effort runs (to save time) and longer for high effort runs (to allow more complex optimizations).

Task

Implement a check_candidate_timeout function in codeflash/optimization/function_optimizer.py that:

  1. Takes a candidate runtime and returns whether the candidate should be skipped
  2. Uses a configurable timeout threshold that scales with optimization effort
  3. Handles the error case where the runtime measurement is unavailable

Also add the necessary configuration constant to codeflash/code_utils/config_consts.py.

Expected Outputs

  • Modified function_optimizer.py with the new function
  • Modified config_consts.py with the new configuration