# Codeflash RL Environment — Batch Validation Report

## Summary

| Metric | Count | % |
|--------|-------|---|
| Total tasks | 29 | 100% |
| Solve passes | 0 | 0% |
| Eval correct (all behavioral tests pass) | 27 | 93% |
| Faster than original (speedup > 1.0) | 25 | 86% |
| All test cases pass | 27 | 93% |

## Speedup Distribution (correct tasks only)

- **1-1.5x**: 14 tasks
- **1.5-2x**: 3 tasks
- **2-5x**: 2 tasks
- **5-100x**: 6 tasks
- **>100x**: 2 tasks

## Successful Tasks (correct=1.0)

| Task | Function | Speedup | Tests | Coverage | Quality | DB Speedup |
|------|----------|---------|-------|----------|---------|------------|
| introspection-prepare_operators_descriptions | `prepare_operators_descriptions` | 15486.8052x | 1057/1057 | 35.0% | low | 14150.75x |
| camera-_generate_grid_colors | `_generate_grid_colors` | 271.0578x | 1901/1901 | 9.0% |  | 218.30x |
| dynamic_blocks-build_traceback_string | `build_traceback_string` | 20.3020x | 2047/2047 | 16.0% |  | 13.38x |
| bytetrack-bytetrackmanifest-describe_outputs | `describe_outputs` | 18.7971x | 3033/3033 | 90.2% | low | 17.56x |
| managers-try_releasing_cuda_memory | `try_releasing_cuda_memory` | 15.5887x | 1006/1006 | 10.8% | medium | 1.22x |
| workflow_caller-_describe_outputs_from_spec | `_describe_outputs_from_spec` | 14.7656x | 25/25 | 23.1% | low | 12.94x |
| cache-_slugify_model_id | `_slugify_model_id` | 13.0696x | 1050/1050 | 26.1% | medium | 11.21x |
| dynamic_blocks-create_dynamic_module | `create_dynamic_module` | 10.5132x | 142/142 | 27.4% |  | 12.31x |
| core_steps-load_kinds | `load_kinds` | 4.7933x | 1153/1153 | 42.0% |  | 3.68x |
| core-_prepare_workflow_response_cache_key | `_prepare_workflow_response_cache_key` | 2.8879x | 7539/7539 | 2.7% |  | 2.39x |
| cache-is_block_cached | `is_block_cached` | 1.7514x | 53/53 | 27.9% |  | 1.36x |
| dynamic_blocks-assembly_custom_python_block | `assembly_custom_python_block` | 1.6098x | 135/135 | 36.7% | low | 1.61x |
| cache-get_cached_foundation_models | `get_cached_foundation_models` | 1.5838x | 32/32 | 34.7% | low | 1.46x |
| dataset_upload-register_datapoint_at_roboflow | `register_datapoint_at_roboflow` | 1.3550x | 2037/2037 | 38.6% | medium | 1.32x |
| core-get_workflow_specification | `get_workflow_specification` | 1.3160x | 1157/1157 | 3.6% | low | 1.56x |
| cache-_is_model_cached | `_is_model_cached` | 1.2949x | 45/45 | 27.0% | high | 1.24x |
| core-load_cached_workflow_response | `load_cached_workflow_response` | 1.2928x | 12126/12126 | 2.8% | low | 1.38x |
| cache-is_model_cached | `is_model_cached` | 1.2835x | 55/55 | 28.7% | low | 1.22x |
| cache-get_task_type_to_block_mapping | `get_task_type_to_block_mapping` | 1.2790x | 30/30 | 29.6% | low | 1.39x |
| workflow_caller-_fetch_workflow_spec_for_validation | `_fetch_workflow_spec_for_validation` | 1.2450x | 1547/1547 | 23.1% |  | 1.33x |
| workflow_caller-build_workflow_url | `build_workflow_url` | 1.1834x | 6137/6137 | 22.2% |  | 1.29x |
| email_notification-apply_operations_to_message_parameters | `apply_operations_to_message_parameters` | 1.1713x | 44/44 | 29.5% | low | 1.15x |
| workflow_caller-_convert_output_descriptions_to_kinds | `_convert_output_descriptions_to_kinds` | 1.0978x | 37/37 | 24.6% | medium | 1.19x |
| core-record_inference | `record_inference` | 1.0950x | 3033/3033 | 1.6% |  | 1.22x |
| workflow_caller-_resolve_output_kinds_for_run | `_resolve_output_kinds_for_run` | 1.0881x | 1047/1047 | 26.2% |  | 1.12x |
| builder-get_cached_models | `get_cached_models` | 1.0000x | 21/21 | 51.4% | low | 1.80x |
| core-wrap_roboflow_api_errors | `wrap_roboflow_api_errors` | 1.0000x | 438/438 | 3.7% | low | 1.28x |

## Failed Tasks (2)

### compiler-establish_batch_oriented_step_lineage

- **Function**: `establish_batch_oriented_step_lineage`
- **File**: `inference/core/workflows/execution_engine/v1/compiler/graph_constructor.py`
- **Commit**: `90243bdc6278ef7d17b6db09dc1eb5b0d155b4be`
- **Method**: db_code_match
- **DB Speedup**: 1.54x
- **Solve OK**: False

- **Duration**: 16.2s
- **Reward**: correct=0.0, speedup=0.0, tests=33/36

<details><summary>Key errors</summary>

```
  /workspace/inference/inference/core/workflows/execution_engine/entities/types.py:1220: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.11/migration/
  /workspace/inference/inference/core/workflows/execution_engine/entities/types.py:1236: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.11/migration/
FAILED tests/codeflash_generated/test_establish_batch_oriented_step_lineage__behaviorinstrumented_0.py::test_multiple_control_flow_lineages_with_same_min_length_raises_assumption_error[ 1 ]
FAILED tests/codeflash_generated/test_establish_batch_oriented_step_lineage__behaviorinstrumented_1.py::test_empty_lineage_lists[ 1 ]
FAILED tests/codeflash_generated/test_establish_batch_oriented_step_lineage__behaviorinstrumented_1.py::test_missing_dimensionality_reference_property[ 1 ]
FAILED tests/codeflash_generated/test_establish_batch_oriented_step_lineage__behaviorinstrumented_1.py::test_non_batch_oriented_property_raises_error[ 1 ]
FAILED tests/codeflash_generated/test_establish_batch_oriented_step_lineage__behaviorinstrumented_1.py::test_multiple_control_flow_same_min_length_raises_error[ 1 ]
FAILED tests/codeflash_generated/test_establish_batch_oriented_step_lineage__behaviorinstrumented_1.py::test_compound_input_no_batch_oriented_raises_error[ 1 ]
INFO:   INCORRECT: 33/36 passed, 3 diffs
```
</details>


**Reproduce**: `bash docker_e2e_test.sh compiler-establish_batch_oriented_step_lineage --debug`

---

### compiler-get_reference_lineage

- **Function**: `get_reference_lineage`
- **File**: `inference/core/workflows/execution_engine/v1/compiler/graph_constructor.py`
- **Commit**: `HEAD`
- **Method**: db_code_only
- **DB Speedup**: 1.61x
- **Solve OK**: False

- **Duration**: 14.8s
- **Reward**: correct=0.0, speedup=0.0, tests=20/24

<details><summary>Key errors</summary>

```
/errors.pydantic.dev/2.11/migration/
  /workspace/inference/inference/core/workflows/execution_engine/entities/types.py:1294: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.11/migration/
  /workspace/inference/inference/core/workflows/execution_engine/entities/types.py:1310: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.11/migration/
FAILED tests/codeflash_generated/test_get_reference_lineage__behaviorinstrumented_1.py::TestGetReferenceLineageBasic::test_batch_oriented_property_in_simple_input[ 1 ]
FAILED tests/codeflash_generated/test_get_reference_lineage__behaviorinstrumented_1.py::TestGetReferenceLineageEdge::test_compound_input_with_batch_oriented_nested[ 1 ]
FAILED tests/codeflash_generated/test_get_reference_lineage__behaviorinstrumented_1.py::TestGetReferenceLineageEdge::test_compound_input_no_batch_oriented_raises_error[ 1 ]
FAILED tests/codeflash_generated/test_get_reference_lineage__behaviorinstrumented_1.py::TestGetReferenceLineageLargeScale::test_large_compound_input_many_nested[ 1 ]
FAILED tests/codeflash_generated/test_get_reference_lineage__behaviorinstrumented_1.py::TestGetReferenceLineageLargeScale::test_many_input_data_keys[ 1 ]
INFO:   INCORRECT: 20/24 passed, 4 diffs
```
</details>


**Reproduce**: `bash docker_e2e_test.sh compiler-get_reference_lineage --debug`

---
