---
name: debug-test-generation
description: >
  Diagnose why test generation failed or produced invalid tests.
  Use when testgen returns errors, empty results, or produces tests
  that fail to compile or run. Walks through request validation,
  router dispatch, context building, prompt construction, LLM calls,
  postprocessing, instrumentation, and output validation.
---

# Debug Test Generation

Use this workflow when test generation fails or produces invalid tests. Work through the stages sequentially — stop at the first failure found.

## Step 1: Validate the Request

Check that the incoming testgen request is well-formed.

1. Read the testgen request schema in the relevant testgen module
2. Verify required fields: `source_code`, `trace_id` must be non-empty
3. Check `language` field — must match a supported language
4. Check for valid code — source code should parse without syntax errors

**Checkpoint**: If the request schema is invalid, the error comes from Pydantic validation. Check the 400 response.

## Step 2: Check Router Dispatch

Verify the correct language handler is invoked.

1. Read `core/shared/testgen_router.py` — the `testgen()` endpoint dispatches by `data.language`
2. Supported routes:
   - `"javascript"` / `"typescript"` → `core.languages.js_ts.testgen.generate_tests_javascript`
   - `"java"` → `core.languages.java.testgen.generate_tests_java`
   - Default → `core.languages.python.testgen.testgen.generate_tests_python`
3. Check for `ImportError` — lazy imports may fail if a language module is broken

**Checkpoint**: If dispatch fails, check that the language module exists and imports cleanly.

## Step 3: Check Context Building

Verify the testgen context is constructed correctly.

1. Read `core/languages/python/testgen/testgen.py` — `build_prompt()` constructs the prompts
2. Check that source code and dependency code are passed correctly
3. Verify the Jinja2 template renders without errors
4. Check for async/sync variants — the prompt builder handles both

**Checkpoint**: If context is empty or malformed, check the input `source_code` and `dependency_code`.

## Step 4: Check Prompt Construction

Verify the LLM prompts are well-formed.

1. The `build_prompt()` function uses Jinja2 templates (`.md` files alongside the module)
2. System prompt sets the role and language context
3. User prompt includes the source code and test context
4. Check that prompts are non-empty and contain the function to test

**Checkpoint**: If prompts are empty, check the template files and Jinja2 rendering.

## Step 5: Check LLM Response

Verify the LLM returns valid test code.

1. Read `aiservice/llm.py` — `call_llm()` handles the API call
2. Check for network errors or API key issues (same as optimization debugging)
3. Look for `LLMOutputParseError` — this means the LLM returned unparseable output
4. Check the raw response content — it should contain markdown code blocks with test code

**Checkpoint**: If the LLM returns malformed output, check the prompt quality and model selection.

## Step 6: Check Postprocessing

Verify generated tests survive postprocessing.

1. Read `core/languages/python/testgen/postprocessing/` directory
2. `add_missing_imports.py` — adds `from __future__ import annotations` if needed
3. Check for syntax errors in generated test code — `cst.ParserSyntaxError` means malformed code
4. Verify imports are resolved correctly

**Checkpoint**: If postprocessing fails, the LLM generated syntactically invalid code. Check the raw output.

## Step 7: Check Instrumentation

Verify tests are properly instrumented.

1. Read `core/languages/python/testgen/instrumentation/instrument_new_tests.py`
2. `instrument_tests()` applies behavior and performance instrumentation
3. `detect_frameworks_from_code()` identifies ML frameworks (PyTorch, TensorFlow, JAX)
4. `_create_device_sync_precompute_statements()` adds GPU sync calls for timing accuracy
5. Check that instrumented tests still compile — instrumentation may introduce syntax errors

**Checkpoint**: If instrumented tests fail to compile, check the instrumentation transforms. The issue is usually in import handling or device sync injection.

## Key Files Reference

| File | What to check |
|------|---------------|
| `core/shared/testgen_router.py` | Language dispatch |
| `core/languages/python/testgen/testgen.py` | Testgen flow, prompt building |
| `core/languages/python/testgen/postprocessing/` | Import management, cleanup |
| `core/languages/python/testgen/instrumentation/instrument_new_tests.py` | Test instrumentation |
| `aiservice/llm.py` | LLM calls and client setup |
