codeflash-admin/codeflash-internal
1
0
Fork 0
mirror of https://github.com/codeflash-ai/codeflash-internal.git synced 2026-05-04 18:25:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
codeflash-internal/tiles/codeflash-internal-docs/docs/test-generation-pipeline.md
Kevin Turcios dfc56f19a0 feat: add Tessl tiles for codeflash-internal (rules, docs, skills) 
Three private tiles published to the codeflash workspace:
- codeflash-internal-rules: 6 eager rules (code-style, architecture,
  optimization-patterns, git-conventions, testing-rules, multi-language-handlers)
- codeflash-internal-docs: 8 lazy doc pages (domain-types, optimization-pipeline,
  test-generation-pipeline, context-extraction, aiservice/cf-api endpoints,
  configuration-thresholds, llm-provider-abstraction)
- codeflash-internal-skills: 4 on-demand skills (debug-optimization-failure,
  add-language-support, add-api-endpoint, debug-test-generation)
2026-02-14 22:16:33 -05:00

2 KiB
Raw Blame History

Test Generation Pipeline

Flow from test generation request to instrumented test output.

1. Request Entry (core/shared/testgen_router.py)

The testgen_api NinjaAPI router receives a POST request and dispatches by data.language:

  • "javascript" / "typescript" → core.languages.js_ts.testgen.generate_tests_javascript
  • "java" → core.languages.java.testgen.generate_tests_java
  • Default → core.languages.python.testgen.testgen.generate_tests_python

2. Python Testgen (core/languages/python/testgen/testgen.py)

build_prompt()

Jinja2-based prompt builder for test generation:

  • Constructs system and user prompts from context
  • Handles async/sync function variants
  • Includes source code, dependency code, and test context

instrument_tests()

Applies instrumentation to generated tests:

  • Behavior instrumentation (captures return values, stdout)
  • Performance instrumentation (timing loops)

LLMOutputParseError

Custom exception for LLM response parsing failures — includes raw output for debugging.

3. Instrumentation (testgen/instrumentation/instrument_new_tests.py)

Framework Detection

  • detect_frameworks_from_code() — parses imports to identify ML frameworks (PyTorch, TensorFlow, JAX) and their aliases
  • Used to add GPU sync calls in timing blocks

Device Sync

  • _create_device_sync_precompute_statements() — pre-computes device sync checks (CUDA, MPS) outside timing blocks
  • Ensures accurate timing for GPU-accelerated code

4. Postprocessing (testgen/postprocessing/)

  • Import management: add_missing_imports.py adds from __future__ import annotations
  • Test validation and cleanup

Key Files

File Role
core/shared/testgen_router.py Language dispatch
core/languages/python/testgen/testgen.py Testgen flow
core/languages/python/testgen/instrumentation/instrument_new_tests.py Test instrumentation
core/languages/python/testgen/postprocessing/ Import management, cleanup