codeflash-admin/codeflash
1
0
Fork 0
mirror of https://github.com/codeflash-ai/codeflash.git synced 2026-05-04 18:25:17 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch 'main' into fix/js-jest30-loop-runner

Browse source
This commit is contained in:
mohammed ahmed 2026-02-06 17:22:15 +02:00 committed by GitHub
commit f800ae3d92
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
34 changed files with 2871 additions and 115 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
.claude/rules/architecture.md Normal file
View file

@ -0,0 +1,28 @@
# Architecture
```
codeflash/
├── main.py # CLI entry point
├── cli_cmds/ # Command handling, console output (Rich)
├── discovery/ # Find optimizable functions
├── context/ # Extract code dependencies and imports
├── optimization/ # Generate optimized code via AI
│ ├── optimizer.py # Main optimization orchestration
│ └── function_optimizer.py # Per-function optimization logic
├── verification/ # Run deterministic tests (pytest plugin)
├── benchmarking/ # Performance measurement
├── github/ # PR creation
├── api/ # AI service communication
├── code_utils/ # Code parsing, git utilities
├── models/ # Pydantic models and types
├── languages/ # Multi-language support (Python, JavaScript/TypeScript)
├── setup/ # Config schema, auto-detection, first-run experience
├── picklepatch/ # Serialization/deserialization utilities
├── tracing/ # Function call tracing
├── tracer.py # Root-level tracer entry point for profiling
├── lsp/ # IDE integration (Language Server Protocol)
├── telemetry/ # Sentry, PostHog
├── either.py # Functional Result type for error handling
├── result/ # Result types and handling
└── version.py # Version information
```

9
.claude/rules/code-style.md Normal file
View file

@ -0,0 +1,9 @@
# Code Style
- **Line length**: 120 characters
- **Python**: 3.9+ syntax
- **Tooling**: Ruff for linting/formatting, mypy strict mode, prek for pre-commit checks
- **Comments**: Minimal - only explain "why", not "what"
- **Docstrings**: Do not add unless explicitly requested
- **Naming**: NEVER use leading underscores (`_function_name`) - Python has no true private functions, use public names
- **Paths**: Always use absolute paths, handle encoding explicitly (UTF-8)

6
.claude/rules/git.md Normal file
View file

@ -0,0 +1,6 @@
# Git Commits & Pull Requests
- Use conventional commit format: `fix:`, `feat:`, `refactor:`, `docs:`, `test:`, `chore:`
- Keep commits atomic - one logical change per commit
- Commit message body should be concise (1-2 sentences max)
- PR titles should also use conventional format

11
.claude/rules/source-code.md Normal file
View file

@ -0,0 +1,11 @@
---
paths:
- "codeflash/**/*.py"
---
# Source Code Rules
- Use `libcst` for code modification/transformation to preserve formatting. `ast` is acceptable for read-only analysis and parsing.
- NEVER use leading underscores for function names (e.g., `_helper`). Python has no true private functions. Always use public names.
- Any new feature or bug fix that can be tested automatically must have test cases.
- If changes affect existing test expectations, update the tests accordingly. Tests must always pass after changes.

15
.claude/rules/testing.md Normal file
View file

@ -0,0 +1,15 @@
---
paths:
- "tests/**"
- "codeflash/**/*test*.py"
---
# Testing Conventions
- Code context extraction and replacement tests must always assert for full string equality, no substring matching.
- Use pytest's `tmp_path` fixture for temp directories (it's a `Path` object).
- Write temp files inside `tmp_path`, never use `NamedTemporaryFile` (causes Windows file contention).
- Always call `.resolve()` on Path objects to ensure absolute paths and resolve symlinks.
- Use `.as_posix()` when converting resolved paths to strings (normalizes to forward slashes).
- Any new feature or bug fix that can be tested automatically must have test cases.
- If changes affect existing test expectations, update the tests accordingly. Tests must always pass after changes.

10
.github/workflows/claude.yml vendored
View file

@ -62,6 +62,7 @@ jobs:
If there are prek issues:
- For SAFE auto-fixable issues (formatting, import sorting, trailing whitespace, etc.), run `uv run prek run --from-ref origin/main` again to auto-fix them
- For issues that prek cannot auto-fix, do NOT attempt to fix them manually — report them as remaining issues in your summary
If there are mypy issues:
- Fix type annotation issues (missing return types, Optional/None unions, import errors for type hints, incorrect types)
@ -72,6 +73,11 @@ jobs:
- Commit with message "style: auto-fix linting issues" or "fix: resolve mypy type errors" as appropriate
- Push the changes with `git push`
IMPORTANT - Verification after fixing:
- After committing fixes, run `uv run prek run --from-ref origin/main` ONE MORE TIME to verify all issues are resolved
- If errors remain, either fix them or report them honestly as unfixed in your summary
- NEVER claim issues are fixed without verifying. If you cannot fix an issue, say so
Do NOT attempt to fix:
- Type errors that require logic changes or refactoring
- Complex generic type issues
@ -167,7 +173,7 @@ jobs:
2. For each optimization PR:
- Check if CI is passing: `gh pr checks <number>`
- If all checks pass, merge it: `gh pr merge <number> --squash --delete-branch`
claude_args: '--allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh pr checks:*),Bash(gh pr merge:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh api:*),Bash(uv run prek *),Bash(uv run mypy *),Bash(uv run coverage *),Bash(uv run pytest *),Bash(git status*),Bash(git add *),Bash(git commit *),Bash(git push*),Bash(git diff *),Bash(git checkout *),Read,Glob,Grep,Edit"'
claude_args: '--model claude-opus-4-6 --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh pr checks:*),Bash(gh pr merge:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh api:*),Bash(uv run prek *),Bash(uv run mypy *),Bash(uv run coverage *),Bash(uv run pytest *),Bash(git status*),Bash(git add *),Bash(git commit *),Bash(git push*),Bash(git diff *),Bash(git checkout *),Read,Glob,Grep,Edit"'
additional_permissions: |
actions: read
env:
@ -239,7 +245,7 @@ jobs:
uses: anthropics/claude-code-action@v1
with:
use_foundry: "true"
claude_args: '--allowedTools "Read,Edit,Write,Glob,Grep,Bash(git status*),Bash(git diff*),Bash(git add *),Bash(git commit *),Bash(git push*),Bash(git log*),Bash(git merge*),Bash(git fetch*),Bash(git checkout*),Bash(git branch*),Bash(uv run prek *),Bash(prek *),Bash(uv run ruff *),Bash(uv run pytest *),Bash(uv run mypy *),Bash(uv run coverage *),Bash(gh pr comment*),Bash(gh pr view*),Bash(gh pr diff*),Bash(gh pr merge*),Bash(gh pr close*)"'
claude_args: '--model claude-opus-4-6 --allowedTools "Read,Edit,Write,Glob,Grep,Bash(git status*),Bash(git diff*),Bash(git add *),Bash(git commit *),Bash(git push*),Bash(git log*),Bash(git merge*),Bash(git fetch*),Bash(git checkout*),Bash(git branch*),Bash(uv run prek *),Bash(prek *),Bash(uv run ruff *),Bash(uv run pytest *),Bash(uv run mypy *),Bash(uv run coverage *),Bash(gh pr comment*),Bash(gh pr view*),Bash(gh pr diff*),Bash(gh pr merge*),Bash(gh pr close*)"'
additional_permissions: |
actions: read
env:

4
.gitignore vendored
View file

@ -258,6 +258,10 @@ WARP.MD
.mcp.json
.tessl/
tessl.json
# Claude Code - track shared rules, ignore local config
.claude/*
!.claude/rules/
**/node_modules/**
**/dist-nuitka/**
**/.npmrc

49
CLAUDE.md
View file

@ -33,55 +33,6 @@ uv run codeflash init # Initialize in a project
uv run codeflash --all # Optimize entire codebase
```
## Architecture
```
codeflash/
├── main.py # CLI entry point
├── cli_cmds/ # Command handling, console output (Rich)
├── discovery/ # Find optimizable functions
├── context/ # Extract code dependencies and imports
├── optimization/ # Generate optimized code via AI
│ ├── optimizer.py # Main optimization orchestration
│ └── function_optimizer.py # Per-function optimization logic
├── verification/ # Run deterministic tests (pytest plugin)
├── benchmarking/ # Performance measurement
├── github/ # PR creation
├── api/ # AI service communication
├── code_utils/ # Code parsing, git utilities
├── models/ # Pydantic models and types
├── tracing/ # Function call tracing
├── lsp/ # IDE integration (Language Server Protocol)
├── telemetry/ # Sentry, PostHog
├── either.py # Functional Result type for error handling
└── result/ # Result types and handling
```
### Key Rules to follow
- Use libcst, not ast - For Python, always use `libcst` for code parsing/modification to preserve formatting.
- Code context extraction and replacement tests must always assert for full string equality, no substring matching.
- Any new feature or bug fix that can be tested automatically must have test cases.
- If changes affect existing test expectations, update the tests accordingly. Tests must always pass after changes.
- NEVER use leading underscores for function names (e.g., `_helper`). Python has no true private functions. Always use public names.
## Code Style
- **Line length**: 120 characters
- **Python**: 3.9+ syntax
- **Tooling**: Ruff for linting/formatting, mypy strict mode, prek for pre-commit checks
- **Comments**: Minimal - only explain "why", not "what"
- **Docstrings**: Do not add unless explicitly requested
- **Naming**: NEVER use leading underscores (`_function_name`) - Python has no true private functions, use public names
- **Paths**: Always use absolute paths, handle encoding explicitly (UTF-8)
## Git Commits & Pull Requests
- Use conventional commit format: `fix:`, `feat:`, `refactor:`, `docs:`, `test:`, `chore:`
- Keep commits atomic - one logical change per commit
- Commit message body should be concise (1-2 sentences max)
- PR titles should also use conventional format
<!-- Section below is auto-generated by `tessl install` - do not edit manually -->
# Agent Rules <!-- tessl-managed -->

2
code_to_optimize/js/code_to_optimize_vitest/package-lock.json generated
View file

@ -15,7 +15,7 @@
}
},
"../../../packages/codeflash": {
"version": "0.3.1",
"version": "0.7.0",
"dev": true,
"hasInstallScript": true,
"license": "MIT",

14
codeflash/cli_cmds/cli.py
View file

@ -10,7 +10,7 @@ from codeflash.cli_cmds.cmd_init import init_codeflash, install_github_actions
from codeflash.cli_cmds.console import logger
from codeflash.cli_cmds.extension import install_vscode_extension
from codeflash.code_utils import env_utils
from codeflash.code_utils.code_utils import exit_with_message
from codeflash.code_utils.code_utils import exit_with_message, normalize_ignore_paths
from codeflash.code_utils.config_parser import parse_config_file
from codeflash.languages.test_framework import set_current_test_framework
from codeflash.lsp.helpers import is_LSP_enabled
@ -284,16 +284,12 @@ def process_pyproject_config(args: Namespace) -> Namespace:
require_github_app_or_exit(owner, repo_name)
if hasattr(args, "ignore_paths") and args.ignore_paths is not None:
normalized_ignore_paths = []
for path in args.ignore_paths:
path_obj = Path(path)
if path_obj.exists():
normalized_ignore_paths.append(path_obj.resolve())
# Silently skip non-existent paths (e.g., .next, dist before build)
args.ignore_paths = normalized_ignore_paths
# Project root path is one level above the specified directory, because that's where the module can be imported from
args.module_root = Path(args.module_root).resolve()
if hasattr(args, "ignore_paths") and args.ignore_paths is not None:
# Normalize ignore paths, supporting both literal paths and glob patterns
# Use module_root as base path for resolving relative paths and patterns
args.ignore_paths = normalize_ignore_paths(args.ignore_paths, base_path=args.module_root)
# If module-root is "." then all imports are relatives to it.
# in this case, the ".." becomes outside project scope, causing issues with un-importable paths
args.project_root = project_root_from_module_root(args.module_root, pyproject_file_path)

51
codeflash/code_utils/code_utils.py
View file

@ -27,6 +27,57 @@ ImportErrorPattern = re.compile(r"ModuleNotFoundError.*$", re.MULTILINE)
BLACKLIST_ADDOPTS = ("--benchmark", "--sugar", "--codespeed", "--cov", "--profile", "--junitxml", "-n")
# Characters that indicate a glob pattern
GLOB_PATTERN_CHARS = frozenset("*?[")
def is_glob_pattern(path_str: str) -> bool:
"""Check if a path string contains glob pattern characters."""
return any(char in path_str for char in GLOB_PATTERN_CHARS)
def normalize_ignore_paths(paths: list[str], base_path: Path | None = None) -> list[Path]:
"""Normalize ignore paths, expanding glob patterns and resolving paths.
Accepts a list of path strings that can be either:
- Literal paths (relative or absolute): e.g., "node_modules", "/absolute/path"
- Glob patterns: e.g., "**/*.test.js", "dist/*", "*.log"
Args:
paths: List of path strings (literal paths or glob patterns).
base_path: Base path for resolving relative paths and patterns.
If None, uses current working directory.
Returns:
List of resolved Path objects, deduplicated.
"""
if base_path is None:
base_path = Path.cwd()
base_path = base_path.resolve()
normalized: set[Path] = set()
for path_str in paths:
if is_glob_pattern(path_str):
# It's a glob pattern - expand it
# Use base_path as the root for glob expansion
pattern_path = base_path / path_str
# glob returns an iterator of matching paths
for matched_path in base_path.glob(path_str):
if matched_path.exists():
normalized.add(matched_path.resolve())
else:
# It's a literal path
path_obj = Path(path_str)
if not path_obj.is_absolute():
path_obj = base_path / path_obj
if path_obj.exists():
normalized.add(path_obj.resolve())
# Silently skip non-existent literal paths (e.g., .next, dist before build)
return list(normalized)
def unified_diff_strings(code1: str, code2: str, fromfile: str = "original", tofile: str = "modified") -> str:
"""Return the unified diff between two code strings as a single string.

14
codeflash/code_utils/edit_generated_tests.py
View file

@ -361,21 +361,27 @@ def normalize_codeflash_imports(source: str) -> str:
return _CODEFLASH_IMPORT_PATTERN.sub(r"import \1 from 'codeflash'", source)
def inject_test_globals(generated_tests: GeneratedTestsList) -> GeneratedTestsList:
def inject_test_globals(generated_tests: GeneratedTestsList, test_framework: str = "jest") -> GeneratedTestsList:
# TODO: inside the prompt tell the llm if it should import jest functions or it's already injected in the global window
"""Inject test globals into all generated tests.
Args:
generated_tests: List of generated tests.
test_framework: The test framework being used ("jest", "vitest", or "mocha").
Returns:
Generated tests with test globals injected.
"""
# we only inject test globals for esm modules
global_import = (
"import { jest, describe, it, expect, beforeEach, afterEach, beforeAll, test } from '@jest/globals'\n"
)
# Use vitest imports for vitest projects, jest imports for jest projects
if test_framework == "vitest":
global_import = "import { vi, describe, it, expect, beforeEach, afterEach, beforeAll, test } from 'vitest'\n"
else:
# Default to jest imports for jest and other frameworks
global_import = (
"import { jest, describe, it, expect, beforeEach, afterEach, beforeAll, test } from '@jest/globals'\n"
)
for test in generated_tests.generated_tests:
test.generated_original_test_source = global_import + test.generated_original_test_source

4
codeflash/languages/javascript/import_resolver.py
View file

@ -558,6 +558,7 @@ class MultiFileHelperFinder:
"""
from codeflash.discovery.functions_to_optimize import FunctionToOptimize
from codeflash.languages.registry import get_language_support
from codeflash.languages.treesitter_utils import get_analyzer_for_file
if context.current_depth >= context.max_depth:
@ -578,12 +579,15 @@ class MultiFileHelperFinder:
imports = analyzer.find_imports(source)
# Create FunctionToOptimize for the helper
# Get language from the language support registry
lang_support = get_language_support(file_path)
func_info = FunctionToOptimize(
function_name=helper.name,
file_path=file_path,
parents=[],
starting_line=helper.start_line,
ending_line=helper.end_line,
language=str(lang_support.language),
)
# Recursively find helpers

6
codeflash/languages/javascript/module_system.py
View file

@ -416,8 +416,10 @@ def ensure_module_system_compatibility(code: str, target_module_system: str, pro
is_esm = has_import or has_export
# Convert if needed
if target_module_system == ModuleSystem.ES_MODULE and is_commonjs and not is_esm:
logger.debug("Converting CommonJS to ES Module syntax")
# For ESM target: convert any require statements, even if there are also import statements
# This handles generated tests that have ESM imports for test globals but CommonJS for the function
if target_module_system == ModuleSystem.ES_MODULE and has_require:
logger.debug("Converting CommonJS require statements to ES Module syntax")
return convert_commonjs_to_esm(code)
if target_module_system == ModuleSystem.COMMONJS and is_esm and not is_commonjs:

48
codeflash/languages/javascript/parse.py
View file

@ -122,6 +122,7 @@ def parse_jest_test_xml(
# This handles cases where instrumented files are in temp directories
instrumented_path_lookup: dict[str, tuple[Path, TestType]] = {}
for test_file in test_files.test_files:
# Add behavior instrumented file paths
if test_file.instrumented_behavior_file_path:
# Store both the absolute path and resolved path as keys
abs_path = str(test_file.instrumented_behavior_file_path.resolve())
@ -132,18 +133,35 @@ def parse_jest_test_xml(
test_file.test_type,
)
logger.debug(f"Jest XML lookup: registered {abs_path}")
# Also add benchmarking file paths (perf-only instrumented tests)
if test_file.benchmarking_file_path:
bench_abs_path = str(test_file.benchmarking_file_path.resolve())
instrumented_path_lookup[bench_abs_path] = (test_file.benchmarking_file_path, test_file.test_type)
instrumented_path_lookup[str(test_file.benchmarking_file_path)] = (
test_file.benchmarking_file_path,
test_file.test_type,
)
logger.debug(f"Jest XML lookup: registered benchmark {bench_abs_path}")
# Also build a filename-only lookup for fallback matching
# This handles cases where JUnit XML has relative paths that don't match absolute paths
# e.g., JUnit has "test/utils__perfinstrumented.test.ts" but lookup has absolute paths
filename_lookup: dict[str, tuple[Path, TestType]] = {}
for test_file in test_files.test_files:
# Add instrumented_behavior_file_path (behavior tests)
if test_file.instrumented_behavior_file_path:
filename = test_file.instrumented_behavior_file_path.name
# Only add if not already present (avoid overwrites in case of duplicate filenames)
if filename not in filename_lookup:
filename_lookup[filename] = (test_file.instrumented_behavior_file_path, test_file.test_type)
logger.debug(f"Jest XML filename lookup: registered {filename}")
# Also add benchmarking_file_path (perf-only tests) - these have different filenames
# e.g., utils__perfonlyinstrumented.test.ts vs utils__perfinstrumented.test.ts
if test_file.benchmarking_file_path:
bench_filename = test_file.benchmarking_file_path.name
if bench_filename not in filename_lookup:
filename_lookup[bench_filename] = (test_file.benchmarking_file_path, test_file.test_type)
logger.debug(f"Jest XML filename lookup: registered benchmark file {bench_filename}")
# Fallback: if JUnit XML doesn't have system-out, use subprocess stdout directly
global_stdout = ""
@ -179,6 +197,21 @@ def parse_jest_test_xml(
key = match.groups()[:5]
end_matches_dict[key] = match
# Also collect timing markers from testcase-level system-out (Vitest puts output at testcase level)
for tc in suite:
tc_system_out = tc._elem.find("system-out") # noqa: SLF001
if tc_system_out is not None and tc_system_out.text:
tc_stdout = tc_system_out.text.strip()
logger.debug(f"Vitest testcase system-out found: {len(tc_stdout)} chars, first 200: {tc_stdout[:200]}")
end_marker_count = 0
for match in jest_end_pattern.finditer(tc_stdout):
key = match.groups()[:5]
end_matches_dict[key] = match
end_marker_count += 1
if end_marker_count > 0:
logger.debug(f"Found {end_marker_count} END timing markers in testcase system-out")
start_matches.extend(jest_start_pattern.finditer(tc_stdout))
for testcase in suite:
testcase_count += 1
test_class_path = testcase.classname # For Jest, this is the file path
@ -306,7 +339,18 @@ def parse_jest_test_xml(
matching_ends_direct.append(end_match)
if not matching_starts and not matching_ends_direct:
# No timing markers found - add basic result
# No timing markers found - use JUnit XML time attribute as fallback
# The time attribute is in seconds (e.g., "0.00077875"), convert to nanoseconds
runtime = None
try:
time_attr = testcase._elem.attrib.get("time") # noqa: SLF001
if time_attr:
time_seconds = float(time_attr)
runtime = int(time_seconds * 1_000_000_000) # Convert seconds to nanoseconds
logger.debug(f"Jest XML: using time attribute for {test_name}: {time_seconds}s = {runtime}ns")
except (ValueError, TypeError) as e:
logger.debug(f"Jest XML: could not parse time attribute: {e}")
test_results.add(
FunctionTestInvocation(
loop_index=1,
@ -318,7 +362,7 @@ def parse_jest_test_xml(
iteration_id="",
),
file_name=test_file_path,
runtime=None,
runtime=runtime,
test_framework=test_config.test_framework,
did_pass=result,
test_type=test_type,

11
codeflash/languages/javascript/support.py
View file

@ -2165,6 +2165,10 @@ class JavaScriptSupport:
candidate_index=candidate_index,
)
# JavaScript/TypeScript benchmarking uses high max_loops like Python (100,000)
# The actual loop count is limited by target_duration_seconds, not max_loops
JS_BENCHMARKING_MAX_LOOPS = 100_000
def run_benchmarking_tests(
self,
test_paths: Any,
@ -2198,6 +2202,9 @@ class JavaScriptSupport:
framework = test_framework or get_js_test_framework_or_default()
# Use JS-specific high max_loops - actual loop count is limited by target_duration
effective_max_loops = self.JS_BENCHMARKING_MAX_LOOPS
if framework == "vitest":
from codeflash.languages.javascript.vitest_runner import run_vitest_benchmarking_tests
@ -2208,7 +2215,7 @@ class JavaScriptSupport:
timeout=timeout,
project_root=project_root,
min_loops=min_loops,
max_loops=max_loops,
max_loops=effective_max_loops,
target_duration_ms=int(target_duration_seconds * 1000),
)
@ -2221,7 +2228,7 @@ class JavaScriptSupport:
timeout=timeout,
project_root=project_root,
min_loops=min_loops,
max_loops=max_loops,
max_loops=effective_max_loops,
target_duration_ms=int(target_duration_seconds * 1000),
)

244
codeflash/languages/javascript/vitest_runner.py
View file

@ -23,8 +23,13 @@ if TYPE_CHECKING:
def _find_vitest_project_root(file_path: Path) -> Path | None:
"""Find the Vitest project root by looking for vitest/vite config or package.json.
Traverses up from the given file path to find the nearest directory
containing vitest.config.js/ts, vite.config.js/ts, or package.json.
Traverses up from the given file path to find the directory containing
vitest.config.js/ts or vite.config.js/ts. Falls back to package.json only
if no vitest/vite config is found in any parent directory.
In monorepos, package.json may exist at multiple levels (e.g., packages/lib/package.json),
but the vitest config with setupFiles is typically at the monorepo root.
We need to prioritize finding the actual vitest config to ensure paths resolve correctly.
Args:
file_path: A file path within the Vitest project.
@ -34,8 +39,10 @@ def _find_vitest_project_root(file_path: Path) -> Path | None:
"""
current = file_path.parent if file_path.is_file() else file_path
package_json_dir = None # Track first package.json found (fallback)
while current != current.parent: # Stop at filesystem root
# Check for Vitest-specific config files first
# Check for Vitest-specific config files first - these should take priority
if (
(current / "vitest.config.js").exists()
or (current / "vitest.config.ts").exists()
@ -45,27 +52,40 @@ def _find_vitest_project_root(file_path: Path) -> Path | None:
or (current / "vite.config.ts").exists()
or (current / "vite.config.mjs").exists()
or (current / "vite.config.mts").exists()
or (current / "package.json").exists()
):
return current
# Remember first package.json as fallback, but keep looking for vitest config
if package_json_dir is None and (current / "package.json").exists():
package_json_dir = current
current = current.parent
return None
# No vitest config found, fall back to package.json directory if found
return package_json_dir
def _is_vitest_coverage_available(project_root: Path) -> bool:
"""Check if Vitest coverage package is available.
In monorepos, dependencies may be hoisted to the root node_modules.
This function searches up the directory tree for the coverage package.
Args:
project_root: The project root directory.
project_root: The project root directory (may be a package in a monorepo).
Returns:
True if @vitest/coverage-v8 or @vitest/coverage-istanbul is installed.
"""
node_modules = project_root / "node_modules"
return (node_modules / "@vitest" / "coverage-v8").exists() or (
node_modules / "@vitest" / "coverage-istanbul"
).exists()
current = project_root
while current != current.parent: # Stop at filesystem root
node_modules = current / "node_modules"
if node_modules.exists():
if (node_modules / "@vitest" / "coverage-v8").exists() or (
node_modules / "@vitest" / "coverage-istanbul"
).exists():
return True
current = current.parent
return False
def _ensure_runtime_files(project_root: Path) -> None:
@ -97,8 +117,146 @@ def _ensure_runtime_files(project_root: Path) -> None:
logger.error(f"Could not install codeflash. Please install it manually: {' '.join(install_cmd)}")
def _find_monorepo_root(start_path: Path) -> Path | None:
"""Find the monorepo root by looking for workspace markers.
Args:
start_path: A path within the monorepo.
Returns:
The monorepo root directory, or None if not found.
"""
monorepo_markers = ["pnpm-workspace.yaml", "yarn.lock", "lerna.json", "package-lock.json"]
current = start_path if start_path.is_dir() else start_path.parent
while current != current.parent:
# Check for monorepo markers
if any((current / marker).exists() for marker in monorepo_markers):
# Verify it has node_modules or package.json (it's a real root)
if (current / "node_modules").exists() or (current / "package.json").exists():
return current
current = current.parent
return None
def _is_vitest_workspace(project_root: Path) -> bool:
"""Check if the project uses vitest workspace configuration.
Vitest workspaces have a special structure where the root config
points to package-level configs. We shouldn't override these.
Args:
project_root: The project root directory.
Returns:
True if the project appears to use vitest workspace.
"""
vitest_config = project_root / "vitest.config.ts"
if not vitest_config.exists():
vitest_config = project_root / "vitest.config.js"
if not vitest_config.exists():
return False
try:
content = vitest_config.read_text()
# Check for workspace indicators
return "workspace" in content.lower() or "defineWorkspace" in content
except Exception:
return False
def _ensure_codeflash_vitest_config(project_root: Path) -> Path | None:
"""Create or find a Codeflash-compatible Vitest config.
Vitest configs often have restrictive include patterns like 'test/**/*.test.ts'
which filter out our generated test files. This function creates a config
that overrides the include pattern to accept all test files.
Note: For workspace projects, we skip creating a custom config as it would
conflict with the workspace setup. In those cases, tests should be placed
in the correct package's test directory.
Args:
project_root: The project root directory.
Returns:
Path to the Codeflash Vitest config, or None if creation failed/not needed.
"""
# Check for workspace configuration - don't override these
monorepo_root = _find_monorepo_root(project_root)
if monorepo_root and _is_vitest_workspace(monorepo_root):
logger.debug("Detected vitest workspace configuration - skipping custom config")
return None
codeflash_config_path = project_root / "codeflash.vitest.config.js"
# If already exists, use it
if codeflash_config_path.exists():
logger.debug(f"Using existing Codeflash Vitest config: {codeflash_config_path}")
return codeflash_config_path
# Find the original vitest config to extend
original_config = None
for config_name in ["vitest.config.ts", "vitest.config.js", "vitest.config.mts", "vitest.config.mjs"]:
config_path = project_root / config_name
if config_path.exists():
original_config = config_name
break
# Also check for vite config with vitest settings
if not original_config:
for config_name in ["vite.config.ts", "vite.config.js", "vite.config.mts", "vite.config.mjs"]:
config_path = project_root / config_name
if config_path.exists():
original_config = config_name
break
# Create a config that extends the original and overrides include pattern
if original_config:
config_content = f"""// Auto-generated by Codeflash for test file pattern compatibility
import {{ mergeConfig }} from 'vitest/config';
import originalConfig from './{original_config}';
export default mergeConfig(originalConfig, {{
test: {{
// Override include pattern to match all test files including generated ones
include: ['**/*.test.ts', '**/*.test.js', '**/*.test.tsx', '**/*.test.jsx'],
}},
}});
"""
else:
# No original config found, create a minimal one
config_content = """// Auto-generated by Codeflash for test file pattern compatibility
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
// Include all test files including generated ones
include: ['**/*.test.ts', '**/*.test.js', '**/*.test.tsx', '**/*.test.jsx'],
// Exclude common non-test directories
exclude: ['**/node_modules/**', '**/dist/**'],
},
});
"""
try:
codeflash_config_path.write_text(config_content)
logger.debug(f"Created Codeflash Vitest config: {codeflash_config_path}")
return codeflash_config_path
except Exception as e:
logger.warning(f"Failed to create Codeflash Vitest config: {e}")
return None
def _build_vitest_behavioral_command(
test_files: list[Path], timeout: int | None = None, output_file: Path | None = None
test_files: list[Path],
timeout: int | None = None,
output_file: Path | None = None,
project_root: Path | None = None,
) -> list[str]:
"""Build Vitest command for behavioral tests.
@ -106,6 +264,7 @@ def _build_vitest_behavioral_command(
test_files: List of test files to run.
timeout: Optional timeout in seconds.
output_file: Optional path for JUnit XML output.
project_root: Project root directory for --root flag.
Returns:
Command list for subprocess execution.
@ -120,6 +279,14 @@ def _build_vitest_behavioral_command(
"--no-file-parallelism", # Serial execution for deterministic timing
]
# For monorepos with restrictive vitest configs (e.g., include: test/**/*.test.ts),
# we need to create a custom config that allows all test patterns.
# This is done by creating a codeflash.vitest.config.js file.
if project_root:
codeflash_vitest_config = _ensure_codeflash_vitest_config(project_root)
if codeflash_vitest_config:
cmd.append(f"--config={codeflash_vitest_config}")
if output_file:
# Use dot notation for junit reporter output file when multiple reporters are used
# Format: --outputFile.junit=/path/to/file.xml
@ -135,7 +302,10 @@ def _build_vitest_behavioral_command(
def _build_vitest_benchmarking_command(
test_files: list[Path], timeout: int | None = None, output_file: Path | None = None
test_files: list[Path],
timeout: int | None = None,
output_file: Path | None = None,
project_root: Path | None = None,
) -> list[str]:
"""Build Vitest command for benchmarking tests.
@ -143,6 +313,7 @@ def _build_vitest_benchmarking_command(
test_files: List of test files to run.
timeout: Optional timeout in seconds.
output_file: Optional path for JUnit XML output.
project_root: Project root directory for --root flag.
Returns:
Command list for subprocess execution.
@ -157,6 +328,12 @@ def _build_vitest_benchmarking_command(
"--no-file-parallelism", # Serial execution for consistent benchmarking
]
# Use codeflash vitest config to override restrictive include patterns
if project_root:
codeflash_vitest_config = _ensure_codeflash_vitest_config(project_root)
if codeflash_vitest_config:
cmd.append(f"--config={codeflash_vitest_config}")
if output_file:
# Use dot notation for junit reporter output file when multiple reporters are used
cmd.append(f"--outputFile.junit={output_file}")
@ -220,11 +397,20 @@ def run_vitest_behavioral_tests(
logger.debug("Vitest coverage package not installed, running without coverage")
# Build Vitest command
vitest_cmd = _build_vitest_behavioral_command(test_files=test_files, timeout=timeout, output_file=result_file_path)
vitest_cmd = _build_vitest_behavioral_command(
test_files=test_files, timeout=timeout, output_file=result_file_path, project_root=effective_cwd
)
# Add coverage flags only if coverage is available
if coverage_available:
# Don't pre-create the coverage directory - vitest should create it
# Pre-creating an empty directory may cause vitest to delete it
logger.debug(f"Coverage will be written to: {coverage_dir}")
vitest_cmd.extend(["--coverage", "--coverage.reporter=json", f"--coverage.reportsDirectory={coverage_dir}"])
# Note: Removed --coverage.enabled=true (redundant) and --coverage.all false
# The version mismatch between vitest and @vitest/coverage-v8 can cause
# issues with coverage flag parsing. Let vitest use default settings.
# Set up environment
vitest_env = test_env.copy()
@ -251,6 +437,7 @@ def run_vitest_behavioral_tests(
cwd=effective_cwd, env=vitest_env, timeout=subprocess_timeout, check=False, text=True, capture_output=True
)
result = subprocess.run(vitest_cmd, **run_args) # noqa: PLW1510
# Combine stderr into stdout for timing markers
if result.stderr and not result.stdout:
result = subprocess.CompletedProcess(
@ -288,8 +475,7 @@ def run_vitest_behavioral_tests(
logger.debug(f"Vitest JUnit XML created: {result_file_path} ({file_size} bytes)")
if file_size < 200: # Suspiciously small - likely empty or just headers
logger.warning(
f"Vitest JUnit XML is very small ({file_size} bytes). "
f"Content: {result_file_path.read_text()[:500]}"
f"Vitest JUnit XML is very small ({file_size} bytes). Content: {result_file_path.read_text()[:500]}"
)
else:
logger.warning(
@ -297,6 +483,26 @@ def run_vitest_behavioral_tests(
f"Vitest stdout: {result.stdout[:1000] if result.stdout else '(empty)'}"
)
# Check if coverage file was created
if coverage_available and coverage_json_path:
if coverage_json_path.exists():
cov_size = coverage_json_path.stat().st_size
logger.debug(f"Vitest coverage JSON created: {coverage_json_path} ({cov_size} bytes)")
else:
# Check if the parent directory exists and list its contents
cov_parent = coverage_json_path.parent
if cov_parent.exists():
contents = list(cov_parent.iterdir())
logger.warning(
f"Vitest coverage JSON not created at {coverage_json_path}. "
f"Directory exists with contents: {[f.name for f in contents]}"
)
else:
logger.warning(
f"Vitest coverage JSON not created at {coverage_json_path}. "
f"Coverage directory does not exist: {cov_parent}"
)
return result_file_path, result, coverage_json_path, None
@ -350,7 +556,7 @@ def run_vitest_benchmarking_tests(
# Build Vitest command for performance tests
vitest_cmd = _build_vitest_benchmarking_command(
test_files=test_files, timeout=timeout, output_file=result_file_path
test_files=test_files, timeout=timeout, output_file=result_file_path, project_root=effective_cwd
)
# Base environment setup
@ -461,6 +667,12 @@ def run_vitest_line_profile_tests(
"--no-file-parallelism", # Serial execution for consistent line profiling
]
# Use codeflash vitest config to override restrictive include patterns
if effective_cwd:
codeflash_vitest_config = _ensure_codeflash_vitest_config(effective_cwd)
if codeflash_vitest_config:
vitest_cmd.append(f"--config={codeflash_vitest_config}")
# Use dot notation for junit reporter output file when multiple reporters are used
vitest_cmd.append(f"--outputFile.junit={result_file_path}")

37
codeflash/models/models.py
View file

@ -23,7 +23,7 @@ from collections.abc import Collection
from enum import Enum, IntEnum
from pathlib import Path
from re import Pattern
from typing import NamedTuple, Optional, cast
from typing import Any, NamedTuple, Optional, cast
from jedi.api.classes import Name
from pydantic import BaseModel, ConfigDict, Field, PrivateAttr, ValidationError, model_validator
@ -172,7 +172,7 @@ class BestOptimization(BaseModel):
winning_behavior_test_results: TestResults
winning_benchmarking_test_results: TestResults
winning_replay_benchmarking_test_results: Optional[TestResults] = None
line_profiler_test_results: dict
line_profiler_test_results: dict[Any, Any]
async_throughput: Optional[int] = None
concurrency_metrics: Optional[ConcurrencyMetrics] = None
@ -209,7 +209,7 @@ class BenchmarkDetail:
f"Benchmark speedup for {self.benchmark_name}::{self.test_function}: {self.speedup_percent:.2f}%\n"
)
def to_dict(self) -> dict[str, any]:
def to_dict(self) -> dict[str, Any]:
return {
"benchmark_name": self.benchmark_name,
"test_function": self.test_function,
@ -232,20 +232,28 @@ class ProcessedBenchmarkInfo:
result += detail.to_string() + "\n"
return result
def to_dict(self) -> dict[str, list[dict[str, any]]]:
def to_dict(self) -> dict[str, list[dict[str, Any]]]:
return {"benchmark_details": [detail.to_dict() for detail in self.benchmark_details]}
class CodeString(BaseModel):
code: str
file_path: Optional[Path] = None
language: str = "python" # Language for validation - only Python code is validated
language: str = "python" # Language for validation
@model_validator(mode="after")
def validate_code_syntax(self) -> CodeString:
"""Validate code syntax for Python only."""
"""Validate code syntax for the specified language."""
if self.language == "python":
validate_python_code(self.code)
elif self.language in ("javascript", "typescript"):
# Validate JavaScript/TypeScript syntax using language support
from codeflash.languages.registry import get_language_support
lang_support = get_language_support(self.language)
if not lang_support.validate_syntax(self.code):
msg = f"Invalid {self.language.title()} code"
raise ValueError(msg)
return self
@ -272,7 +280,7 @@ markdown_pattern_python_only = re.compile(r"```python:([^\n]+)\n(.*?)\n```", re.
class CodeStringsMarkdown(BaseModel):
code_strings: list[CodeString] = []
language: str = "python" # Language for markdown code block tags
_cache: dict = PrivateAttr(default_factory=dict)
_cache: dict[str, Any] = PrivateAttr(default_factory=dict)
@property
def flat(self) -> str:
@ -408,7 +416,7 @@ class GeneratedTestsList(BaseModel):
class TestFile(BaseModel):
instrumented_behavior_file_path: Path
benchmarking_file_path: Path = None
benchmarking_file_path: Optional[Path] = None
original_file_path: Optional[Path] = None
original_source: Optional[str] = None
test_type: TestType
@ -448,6 +456,19 @@ class TestFiles(BaseModel):
normalized_benchmark_path = self._normalize_path_for_comparison(test_file.benchmarking_file_path)
if normalized == normalized_benchmark_path:
return test_file.test_type
# Fallback: try filename-only matching for JavaScript/TypeScript
# Jest/Vitest JUnit XML may have relative paths that don't match absolute paths
file_name = file_path.name
for test_file in self.test_files:
if (
test_file.instrumented_behavior_file_path
and test_file.instrumented_behavior_file_path.name == file_name
):
return test_file.test_type
if test_file.benchmarking_file_path and test_file.benchmarking_file_path.name == file_name:
return test_file.test_type
return None
def get_test_type_by_original_file_path(self, file_path: Path) -> TestType | None:

28
codeflash/optimization/function_optimizer.py
View file

@ -545,15 +545,24 @@ class FunctionOptimizer:
]:
"""Generate and instrument tests for the function."""
n_tests = get_effort_value(EffortKeys.N_GENERATED_TESTS, self.effort)
source_file = Path(self.function_to_optimize.file_path)
generated_test_paths = [
get_test_file_path(
self.test_cfg.tests_root, self.function_to_optimize.function_name, test_index, test_type="unit"
self.test_cfg.tests_root,
self.function_to_optimize.function_name,
test_index,
test_type="unit",
source_file_path=source_file,
)
for test_index in range(n_tests)
]
generated_perf_test_paths = [
get_test_file_path(
self.test_cfg.tests_root, self.function_to_optimize.function_name, test_index, test_type="perf"
self.test_cfg.tests_root,
self.function_to_optimize.function_name,
test_index,
test_type="perf",
source_file_path=source_file,
)
for test_index in range(n_tests)
]
@ -578,7 +587,7 @@ class FunctionOptimizer:
if not is_python():
module_system = detect_module_system(self.project_root, self.function_to_optimize.file_path)
if module_system == "esm":
generated_tests = inject_test_globals(generated_tests)
generated_tests = inject_test_globals(generated_tests, self.test_cfg.test_framework)
if is_typescript():
# disable ts check for typescript tests
generated_tests = disable_ts_check(generated_tests)
@ -1911,10 +1920,11 @@ class FunctionOptimizer:
return Failure(baseline_result.failure())
original_code_baseline, test_functions_to_remove = baseline_result.unwrap()
if isinstance(original_code_baseline, OriginalCodeBaseline) and (
not coverage_critic(original_code_baseline.coverage_results)
or not quantity_of_tests_critic(original_code_baseline)
):
# Check test quantity for all languages
quantity_ok = quantity_of_tests_critic(original_code_baseline)
# TODO: {Self} Only check coverage for Python - coverage infrastructure not yet reliable for JS/TS
coverage_ok = coverage_critic(original_code_baseline.coverage_results) if is_python() else True
if isinstance(original_code_baseline, OriginalCodeBaseline) and (not coverage_ok or not quantity_ok):
if self.args.override_fixtures:
restore_conftest(original_conftest_content)
cleanup_paths(paths_to_cleanup)
@ -2098,7 +2108,11 @@ class FunctionOptimizer:
formatted_generated_test = format_generated_code(concolic_test_str, self.args.formatter_cmds)
generated_tests_str += f"```{code_lang}\n{formatted_generated_test}\n```\n\n"
<<<<<<< fix/js-jest30-loop-runner
existing_tests, replay_tests, _concolic_tests = existing_tests_source_for(
=======
existing_tests, replay_tests, _ = existing_tests_source_for(
>>>>>>> main
self.function_to_optimize.qualified_name_with_modules_from_root(self.project_root),
function_to_all_tests,
test_cfg=self.test_cfg,

80
codeflash/verification/verification_utils.py
View file

@ -9,17 +9,93 @@ from pydantic.dataclasses import dataclass
from codeflash.languages import current_language_support, is_javascript
def get_test_file_path(test_dir: Path, function_name: str, iteration: int = 0, test_type: str = "unit") -> Path:
def get_test_file_path(
test_dir: Path,
function_name: str,
iteration: int = 0,
test_type: str = "unit",
source_file_path: Path | None = None,
) -> Path:
assert test_type in {"unit", "inspired", "replay", "perf"}
function_name = function_name.replace(".", "_")
# Use appropriate file extension based on language
extension = current_language_support().get_test_file_suffix() if is_javascript() else ".py"
# For JavaScript/TypeScript, place generated tests in a subdirectory that matches
# Vitest/Jest include patterns (e.g., test/**/*.test.ts)
if is_javascript():
# For monorepos, first try to find the package directory from the source file path
# e.g., packages/workflow/src/utils.ts -> packages/workflow/test/codeflash-generated/
package_test_dir = _find_js_package_test_dir(test_dir, source_file_path)
if package_test_dir:
test_dir = package_test_dir
path = test_dir / f"test_{function_name}__{test_type}_test_{iteration}{extension}"
if path.exists():
return get_test_file_path(test_dir, function_name, iteration + 1, test_type)
return get_test_file_path(test_dir, function_name, iteration + 1, test_type, source_file_path)
return path
def _find_js_package_test_dir(tests_root: Path, source_file_path: Path | None) -> Path | None:
"""Find the appropriate test directory for a JavaScript/TypeScript package.
For monorepos, this finds the package's test directory from the source file path.
For example: packages/workflow/src/utils.ts -> packages/workflow/test/codeflash-generated/
Args:
tests_root: The root tests directory (may be monorepo packages root).
source_file_path: Path to the source file being tested.
Returns:
The test directory path, or None if not found.
"""
if source_file_path is None:
# No source path provided, check if test_dir itself has a test subdirectory
for test_subdir_name in ["test", "tests", "__tests__", "src/__tests__"]:
test_subdir = tests_root / test_subdir_name
if test_subdir.is_dir():
codeflash_test_dir = test_subdir / "codeflash-generated"
codeflash_test_dir.mkdir(parents=True, exist_ok=True)
return codeflash_test_dir
return None
try:
# Resolve paths for reliable comparison
tests_root = tests_root.resolve()
source_path = Path(source_file_path).resolve()
# Walk up from the source file to find a directory with package.json or test/ folder
package_dir = None
for parent in source_path.parents:
# Stop if we've gone above or reached the tests_root level
# For monorepos, tests_root might be /packages/ and we want to search within packages
if parent in (tests_root, tests_root.parent):
break
# Check if this looks like a package root
has_package_json = (parent / "package.json").exists()
has_test_dir = any((parent / d).is_dir() for d in ["test", "tests", "__tests__"])
if has_package_json or has_test_dir:
package_dir = parent
break
if package_dir:
# Find the test directory in this package
for test_subdir_name in ["test", "tests", "__tests__", "src/__tests__"]:
test_subdir = package_dir / test_subdir_name
if test_subdir.is_dir():
codeflash_test_dir = test_subdir / "codeflash-generated"
codeflash_test_dir.mkdir(parents=True, exist_ok=True)
return codeflash_test_dir
return None
except Exception:
return None
def delete_multiple_if_name_main(test_ast: ast.Module) -> ast.Module:
if_indexes = []
for index, node in enumerate(test_ast.body):

4
packages/codeflash/package-lock.json generated
View file

@ -1,12 +1,12 @@
{
"name": "codeflash",
"version": "0.7.0",
"version": "0.8.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "codeflash",
"version": "0.7.0",
"version": "0.8.0",
"hasInstallScript": true,
"license": "MIT",
"dependencies": {

2
packages/codeflash/package.json
View file

@ -1,6 +1,6 @@
{
"name": "codeflash",
"version": "0.7.0",
"version": "0.8.0",
"description": "Codeflash - AI-powered code optimization for JavaScript and TypeScript",
"main": "runtime/index.js",
"types": "runtime/index.d.ts",

9
packages/codeflash/runtime/index.js
View file

@ -77,8 +77,13 @@ module.exports = {
incrementBatch: capture.incrementBatch,
getCurrentBatch: capture.getCurrentBatch,
checkSharedTimeLimit: capture.checkSharedTimeLimit,
PERF_BATCH_SIZE: capture.PERF_BATCH_SIZE,
PERF_LOOP_COUNT: capture.PERF_LOOP_COUNT,
// Getter functions for dynamic env var reading (not constants)
getPerfBatchSize: capture.getPerfBatchSize,
getPerfLoopCount: capture.getPerfLoopCount,
getPerfMinLoops: capture.getPerfMinLoops,
getPerfTargetDurationMs: capture.getPerfTargetDurationMs,
getPerfStabilityCheck: capture.getPerfStabilityCheck,
getPerfCurrentBatch: capture.getPerfCurrentBatch,
// === Feature Detection ===
hasV8: serializer.hasV8,

0
tests/code_utils/__init__.py Normal file
View file

209
tests/code_utils/test_normalize_ignore_paths.py Normal file
View file

@ -0,0 +1,209 @@
"""Tests for normalize_ignore_paths function."""
from __future__ import annotations
from pathlib import Path
import pytest
from codeflash.code_utils.code_utils import is_glob_pattern, normalize_ignore_paths
class TestIsGlobPattern:
"""Tests for is_glob_pattern function."""
def test_asterisk_pattern(self) -> None:
assert is_glob_pattern("*.py") is True
assert is_glob_pattern("**/*.js") is True
assert is_glob_pattern("node_modules/*") is True
def test_question_mark_pattern(self) -> None:
assert is_glob_pattern("file?.txt") is True
assert is_glob_pattern("test_?.py") is True
def test_bracket_pattern(self) -> None:
assert is_glob_pattern("[abc].txt") is True
assert is_glob_pattern("file[0-9].log") is True
def test_literal_paths(self) -> None:
assert is_glob_pattern("node_modules") is False
assert is_glob_pattern("src/utils") is False
assert is_glob_pattern("/absolute/path") is False
assert is_glob_pattern("relative/path/file.py") is False
class TestNormalizeIgnorePaths:
"""Tests for normalize_ignore_paths function."""
def test_empty_list(self) -> None:
result = normalize_ignore_paths([])
assert result == []
def test_literal_existing_path(self, tmp_path: Path) -> None:
# Create a directory
test_dir = tmp_path / "node_modules"
test_dir.mkdir()
result = normalize_ignore_paths(["node_modules"], base_path=tmp_path)
assert len(result) == 1
assert result[0] == test_dir.resolve()
def test_literal_nonexistent_path_skipped(self, tmp_path: Path) -> None:
# Don't create the directory - should be silently skipped
result = normalize_ignore_paths(["nonexistent_dir"], base_path=tmp_path)
assert result == []
def test_multiple_literal_paths(self, tmp_path: Path) -> None:
# Create directories
dir1 = tmp_path / "node_modules"
dir2 = tmp_path / "dist"
dir1.mkdir()
dir2.mkdir()
result = normalize_ignore_paths(["node_modules", "dist"], base_path=tmp_path)
assert len(result) == 2
assert set(result) == {dir1.resolve(), dir2.resolve()}
def test_glob_pattern_single_asterisk(self, tmp_path: Path) -> None:
# Create test files
(tmp_path / "file1.log").touch()
(tmp_path / "file2.log").touch()
(tmp_path / "file.txt").touch()
result = normalize_ignore_paths(["*.log"], base_path=tmp_path)
assert len(result) == 2
resolved_names = {p.name for p in result}
assert resolved_names == {"file1.log", "file2.log"}
def test_glob_pattern_double_asterisk(self, tmp_path: Path) -> None:
# Create nested structure
subdir = tmp_path / "src" / "utils"
subdir.mkdir(parents=True)
(subdir / "test_helper.py").touch()
(tmp_path / "src" / "test_main.py").touch()
(tmp_path / "test_root.py").touch()
result = normalize_ignore_paths(["**/test_*.py"], base_path=tmp_path)
assert len(result) == 3
resolved_names = {p.name for p in result}
assert resolved_names == {"test_helper.py", "test_main.py", "test_root.py"}
def test_glob_pattern_directory_contents(self, tmp_path: Path) -> None:
# Create directory with contents
node_modules = tmp_path / "node_modules"
node_modules.mkdir()
(node_modules / "package1").mkdir()
(node_modules / "package2").mkdir()
result = normalize_ignore_paths(["node_modules/*"], base_path=tmp_path)
assert len(result) == 2
resolved_names = {p.name for p in result}
assert resolved_names == {"package1", "package2"}
def test_glob_pattern_no_matches(self, tmp_path: Path) -> None:
# Pattern with no matches should return empty list
result = normalize_ignore_paths(["*.nonexistent"], base_path=tmp_path)
assert result == []
def test_mixed_literal_and_patterns(self, tmp_path: Path) -> None:
# Create test structure
node_modules = tmp_path / "node_modules"
node_modules.mkdir()
(tmp_path / "debug.log").touch()
(tmp_path / "error.log").touch()
result = normalize_ignore_paths(["node_modules", "*.log"], base_path=tmp_path)
assert len(result) == 3
resolved_names = {p.name for p in result}
assert resolved_names == {"node_modules", "debug.log", "error.log"}
def test_deduplication(self, tmp_path: Path) -> None:
# Create a file that matches multiple patterns
(tmp_path / "test.log").touch()
# Same file should only appear once
result = normalize_ignore_paths(["test.log", "*.log"], base_path=tmp_path)
assert len(result) == 1
assert result[0].name == "test.log"
def test_nested_directory_pattern(self, tmp_path: Path) -> None:
# Create nested test directories
tests_dir = tmp_path / "src" / "__tests__"
tests_dir.mkdir(parents=True)
(tests_dir / "test1.js").touch()
(tests_dir / "test2.js").touch()
result = normalize_ignore_paths(["src/__tests__/*.js"], base_path=tmp_path)
assert len(result) == 2
resolved_names = {p.name for p in result}
assert resolved_names == {"test1.js", "test2.js"}
def test_absolute_path_literal(self, tmp_path: Path) -> None:
# Create a directory
test_dir = tmp_path / "absolute_test"
test_dir.mkdir()
# Use absolute path
result = normalize_ignore_paths([str(test_dir)], base_path=tmp_path)
assert len(result) == 1
assert result[0] == test_dir.resolve()
def test_relative_path_with_subdirectory(self, tmp_path: Path) -> None:
# Create nested directory
nested = tmp_path / "src" / "vendor"
nested.mkdir(parents=True)
result = normalize_ignore_paths(["src/vendor"], base_path=tmp_path)
assert len(result) == 1
assert result[0] == nested.resolve()
def test_default_base_path_uses_cwd(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
# Change to tmp_path
monkeypatch.chdir(tmp_path)
# Create a directory
test_dir = tmp_path / "test_dir"
test_dir.mkdir()
# Call without base_path
result = normalize_ignore_paths(["test_dir"])
assert len(result) == 1
assert result[0] == test_dir.resolve()
def test_bracket_pattern(self, tmp_path: Path) -> None:
# Create files matching bracket pattern
(tmp_path / "file1.txt").touch()
(tmp_path / "file2.txt").touch()
(tmp_path / "file3.txt").touch()
(tmp_path / "fileA.txt").touch()
result = normalize_ignore_paths(["file[12].txt"], base_path=tmp_path)
assert len(result) == 2
resolved_names = {p.name for p in result}
assert resolved_names == {"file1.txt", "file2.txt"}
def test_question_mark_pattern(self, tmp_path: Path) -> None:
# Create files matching question mark pattern
(tmp_path / "test_a.py").touch()
(tmp_path / "test_b.py").touch()
(tmp_path / "test_ab.py").touch()
result = normalize_ignore_paths(["test_?.py"], base_path=tmp_path)
assert len(result) == 2
resolved_names = {p.name for p in result}
assert resolved_names == {"test_a.py", "test_b.py"}

4
tests/languages/javascript/test_support_dispatch.py
View file

@ -182,7 +182,9 @@ class TestBenchmarkingTestsDispatch:
call_kwargs = mock_vitest_runner.call_args.kwargs
assert call_kwargs["min_loops"] == 10
assert call_kwargs["max_loops"] == 50
# JS/TS always uses high max_loops (100_000) regardless of passed value
# Actual loop count is limited by target_duration, not max_loops
assert call_kwargs["max_loops"] == 100_000
assert call_kwargs["target_duration_ms"] == 5000

312
tests/test_languages/test_javascript_integration.py Normal file
View file

@ -0,0 +1,312 @@
"""E2E tests for JavaScript/TypeScript optimization flow with backend.
These tests call the actual backend /testgen API endpoint and verify:
1. Language parameter is correctly passed to backend
2. Backend validates generated code with correct parser (JS vs TS)
3. CLI receives and processes tests correctly
Similar to test_validate_python_code.py but for JavaScript/TypeScript.
"""
from pathlib import Path
from unittest.mock import patch
import pytest
from codeflash.api.aiservice import AiServiceClient
from codeflash.discovery.functions_to_optimize import FunctionToOptimize
from codeflash.languages.base import Language
from codeflash.models.models import CodeString, OptimizedCandidateSource
def skip_if_js_not_supported():
"""Skip test if JavaScript/TypeScript languages are not supported."""
try:
from codeflash.languages import get_language_support
get_language_support(Language.JAVASCRIPT)
except Exception as e:
pytest.skip(f"JavaScript/TypeScript language support not available: {e}")
class TestJavaScriptCodeStringValidation:
"""Tests for JavaScript CodeString validation - mirrors test_validate_python_code.py."""
def test_javascript_string(self):
"""Test valid JavaScript code string."""
skip_if_js_not_supported()
code = CodeString(code="console.log('Hello, World!');", language="javascript")
assert code.code == "console.log('Hello, World!');"
def test_valid_javascript_code(self):
"""Test that valid JavaScript code passes validation."""
skip_if_js_not_supported()
valid_code = "const x = 1;\nconst y = x + 2;\nconsole.log(y);"
cs = CodeString(code=valid_code, language="javascript")
assert cs.code == valid_code
def test_invalid_javascript_code_syntax(self):
"""Test that invalid JavaScript code fails validation."""
skip_if_js_not_supported()
from pydantic import ValidationError
invalid_code = "const x = 1;\nconsole.log(x" # Missing parenthesis
with pytest.raises(ValidationError) as exc_info:
CodeString(code=invalid_code, language="javascript")
assert "Invalid Javascript code" in str(exc_info.value)
def test_empty_javascript_code(self):
"""Test that empty code passes validation."""
skip_if_js_not_supported()
empty_code = ""
cs = CodeString(code=empty_code, language="javascript")
assert cs.code == empty_code
class TestTypeScriptCodeStringValidation:
"""Tests for TypeScript CodeString validation."""
def test_typescript_string(self):
"""Test valid TypeScript code string."""
skip_if_js_not_supported()
code = CodeString(code="const x: number = 1;", language="typescript")
assert code.code == "const x: number = 1;"
def test_valid_typescript_code(self):
"""Test that valid TypeScript code passes validation."""
skip_if_js_not_supported()
valid_code = "function add(a: number, b: number): number { return a + b; }"
cs = CodeString(code=valid_code, language="typescript")
assert cs.code == valid_code
def test_typescript_type_assertion_valid(self):
"""TypeScript type assertions should pass TypeScript validation."""
skip_if_js_not_supported()
ts_code = "const value = 4.9 as unknown as number;"
cs = CodeString(code=ts_code, language="typescript")
assert cs.code == ts_code
def test_typescript_type_assertion_invalid_in_javascript(self):
"""TypeScript type assertions should FAIL JavaScript validation.
This is the critical test - TypeScript syntax like 'as unknown as number'
should fail when validated as JavaScript.
"""
skip_if_js_not_supported()
from pydantic import ValidationError
ts_code = "const value = 4.9 as unknown as number;"
with pytest.raises(ValidationError) as exc_info:
CodeString(code=ts_code, language="javascript")
assert "Invalid Javascript code" in str(exc_info.value)
def test_typescript_interface_valid(self):
"""TypeScript interfaces should pass TypeScript validation."""
skip_if_js_not_supported()
ts_code = "interface User { name: string; age: number; }"
cs = CodeString(code=ts_code, language="typescript")
assert cs.code == ts_code
def test_typescript_interface_invalid_in_javascript(self):
"""TypeScript interfaces should FAIL JavaScript validation."""
skip_if_js_not_supported()
from pydantic import ValidationError
ts_code = "interface User { name: string; age: number; }"
with pytest.raises(ValidationError) as exc_info:
CodeString(code=ts_code, language="javascript")
assert "Invalid Javascript code" in str(exc_info.value)
def test_typescript_generics_valid(self):
"""TypeScript generics should pass TypeScript validation."""
skip_if_js_not_supported()
ts_code = "function identity<T>(arg: T): T { return arg; }"
cs = CodeString(code=ts_code, language="typescript")
assert cs.code == ts_code
def test_typescript_generics_invalid_in_javascript(self):
"""TypeScript generics should FAIL JavaScript validation."""
skip_if_js_not_supported()
from pydantic import ValidationError
ts_code = "function identity<T>(arg: T): T { return arg; }"
with pytest.raises(ValidationError) as exc_info:
CodeString(code=ts_code, language="javascript")
assert "Invalid Javascript code" in str(exc_info.value)
class TestAiServiceClientJavaScript:
"""Tests for AiServiceClient with JavaScript/TypeScript - mirrors test_validate_python_code.py."""
def test_javascript_generated_candidates_validation(self):
"""Test that JavaScript candidates are validated correctly."""
skip_if_js_not_supported()
ai_service = AiServiceClient()
# Invalid JavaScript (missing closing parenthesis)
code = """```javascript:file.js
console.log(name
```"""
mock_candidates = [{"source_code": code, "explanation": "", "optimization_id": ""}]
candidates = ai_service._get_valid_candidates(mock_candidates, OptimizedCandidateSource.OPTIMIZE)
assert len(candidates) == 0
# Valid JavaScript
code = """```javascript:file.js
console.log('Hello, World!');
```"""
mock_candidates = [{"source_code": code, "explanation": "", "optimization_id": ""}]
candidates = ai_service._get_valid_candidates(mock_candidates, OptimizedCandidateSource.OPTIMIZE)
assert len(candidates) == 1
assert candidates[0].source_code.code_strings[0].code == "console.log('Hello, World!');"
def test_typescript_generated_candidates_validation(self):
"""Test that TypeScript candidates are validated correctly."""
skip_if_js_not_supported()
ai_service = AiServiceClient()
# Valid TypeScript with type annotations
code = """```typescript:file.ts
function add(a: number, b: number): number {
return a + b;
}
```"""
mock_candidates = [{"source_code": code, "explanation": "", "optimization_id": ""}]
candidates = ai_service._get_valid_candidates(mock_candidates, OptimizedCandidateSource.OPTIMIZE)
assert len(candidates) == 1
def test_typescript_type_assertion_in_candidate(self):
"""Test that TypeScript type assertions are valid in TS candidates."""
skip_if_js_not_supported()
ai_service = AiServiceClient()
# TypeScript-specific syntax should be valid
code = """```typescript:file.ts
const value = 4.9 as unknown as number;
```"""
mock_candidates = [{"source_code": code, "explanation": "", "optimization_id": ""}]
candidates = ai_service._get_valid_candidates(mock_candidates, OptimizedCandidateSource.OPTIMIZE)
assert len(candidates) == 1
class TestBackendLanguageParameter:
"""Tests verifying language parameter flows correctly to backend."""
def test_testgen_request_includes_typescript_language(self, tmp_path):
"""Verify the language parameter is sent as 'typescript' for .ts files."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current
# Set current language to TypeScript
lang_current._current_language = Language.TYPESCRIPT
ts_file = tmp_path / "utils.ts"
ts_file.write_text("""
export function add(a: number, b: number): number {
return a + b;
}
""")
functions = find_all_functions_in_file(ts_file)
func = functions[ts_file][0]
# Verify function has correct language
assert func.language == "typescript"
ai_client = AiServiceClient()
captured_payload = None
def capture_request(*args, **kwargs):
nonlocal captured_payload
if 'payload' in kwargs:
captured_payload = kwargs['payload']
elif len(args) > 1:
captured_payload = args[1]
# Return a mock response to avoid actual API call
from unittest.mock import MagicMock
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"generated_tests": "// test",
"instrumented_behavior_tests": "// test",
"instrumented_perf_tests": "// test",
}
return mock_response
with patch.object(ai_client, 'make_ai_service_request', side_effect=capture_request):
ai_client.generate_regression_tests(
source_code_being_tested=ts_file.read_text(),
function_to_optimize=func,
helper_function_names=[],
module_path=ts_file,
test_module_path=tmp_path / "tests" / "utils.test.ts",
test_framework="vitest",
test_timeout=30,
trace_id="test-language-param-ts",
test_index=0,
language="typescript",
)
assert captured_payload is not None
assert captured_payload.get('language') == 'typescript', \
f"Expected language='typescript', got: {captured_payload.get('language')}"
def test_testgen_request_includes_javascript_language(self, tmp_path):
"""Verify the language parameter is sent as 'javascript' for .js files."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current
# Set current language to JavaScript
lang_current._current_language = Language.JAVASCRIPT
js_file = tmp_path / "utils.js"
js_file.write_text("""
function add(a, b) {
return a + b;
}
module.exports = { add };
""")
functions = find_all_functions_in_file(js_file)
func = functions[js_file][0]
# Verify function has correct language
assert func.language == "javascript"
ai_client = AiServiceClient()
captured_payload = None
def capture_request(*args, **kwargs):
nonlocal captured_payload
if 'payload' in kwargs:
captured_payload = kwargs['payload']
elif len(args) > 1:
captured_payload = args[1]
from unittest.mock import MagicMock
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"generated_tests": "// test",
"instrumented_behavior_tests": "// test",
"instrumented_perf_tests": "// test",
}
return mock_response
with patch.object(ai_client, 'make_ai_service_request', side_effect=capture_request):
ai_client.generate_regression_tests(
source_code_being_tested=js_file.read_text(),
function_to_optimize=func,
helper_function_names=[],
module_path=js_file,
test_module_path=tmp_path / "tests" / "utils.test.js",
test_framework="jest",
test_timeout=30,
trace_id="test-language-param-js",
test_index=0,
language="javascript",
)
assert captured_payload is not None
assert captured_payload.get('language') == 'javascript', \
f"Expected language='javascript', got: {captured_payload.get('language')}"

564
tests/test_languages/test_javascript_optimization_flow.py Normal file
View file

@ -0,0 +1,564 @@
"""End-to-end tests for JavaScript/TypeScript optimization flow.
These tests verify the full optimization pipeline including:
- Test generation (with mocked backend)
- Language parameter propagation
- Syntax validation with correct parser
- Running and parsing tests
This is the JavaScript equivalent of test_instrument_tests.py for Python.
"""
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
from codeflash.discovery.functions_to_optimize import FunctionToOptimize
from codeflash.languages.base import Language
from codeflash.models.models import CodeString, FunctionParent
from codeflash.verification.verification_utils import TestConfig
def skip_if_js_not_supported():
"""Skip test if JavaScript/TypeScript languages are not supported."""
try:
from codeflash.languages import get_language_support
get_language_support(Language.JAVASCRIPT)
except Exception as e:
pytest.skip(f"JavaScript/TypeScript language support not available: {e}")
class TestLanguageParameterPropagation:
"""Tests verifying language parameter is correctly passed through all layers."""
def test_function_to_optimize_has_correct_language_for_typescript(self, tmp_path):
"""Verify FunctionToOptimize has language='typescript' for .ts files."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
ts_file = tmp_path / "utils.ts"
ts_file.write_text("""
export function add(a: number, b: number): number {
return a + b;
}
""")
functions = find_all_functions_in_file(ts_file)
assert ts_file in functions
assert len(functions[ts_file]) == 1
assert functions[ts_file][0].language == "typescript"
def test_function_to_optimize_has_correct_language_for_javascript(self, tmp_path):
"""Verify FunctionToOptimize has language='javascript' for .js files."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
js_file = tmp_path / "utils.js"
js_file.write_text("""
function add(a, b) {
return a + b;
}
""")
functions = find_all_functions_in_file(js_file)
assert js_file in functions
assert len(functions[js_file]) == 1
assert functions[js_file][0].language == "javascript"
def test_code_context_preserves_language(self, tmp_path):
"""Verify language is preserved in code context extraction."""
skip_if_js_not_supported()
from codeflash.context.code_context_extractor import get_code_optimization_context
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current
lang_current._current_language = Language.TYPESCRIPT
ts_file = tmp_path / "utils.ts"
ts_file.write_text("""
export function add(a: number, b: number): number {
return a + b;
}
""")
functions = find_all_functions_in_file(ts_file)
func = functions[ts_file][0]
context = get_code_optimization_context(func, tmp_path)
assert context.read_writable_code is not None
assert context.read_writable_code.language == "typescript"
class TestCodeStringSyntaxValidation:
"""Tests verifying CodeString validates with correct parser based on language."""
def test_typescript_code_valid_with_typescript_language(self):
"""TypeScript code should pass validation when language='typescript'."""
skip_if_js_not_supported()
ts_code = "const value = 4.9 as unknown as number;"
code_string = CodeString(code=ts_code, language="typescript")
assert code_string.code == ts_code
def test_typescript_code_invalid_with_javascript_language(self):
"""TypeScript code should FAIL validation when language='javascript'.
This is the exact bug that was in production - TypeScript code being
validated with JavaScript parser.
"""
skip_if_js_not_supported()
from pydantic import ValidationError
ts_code = "const value = 4.9 as unknown as number;"
with pytest.raises(ValidationError) as exc_info:
CodeString(code=ts_code, language="javascript")
assert "Invalid Javascript code" in str(exc_info.value)
def test_typescript_interface_valid_with_typescript_language(self):
"""TypeScript interface should pass validation when language='typescript'."""
skip_if_js_not_supported()
ts_code = "interface User { name: string; age: number; }"
code_string = CodeString(code=ts_code, language="typescript")
assert code_string.code == ts_code
def test_typescript_interface_invalid_with_javascript_language(self):
"""TypeScript interface should FAIL validation when language='javascript'."""
skip_if_js_not_supported()
from pydantic import ValidationError
ts_code = "interface User { name: string; age: number; }"
with pytest.raises(ValidationError) as exc_info:
CodeString(code=ts_code, language="javascript")
assert "Invalid Javascript code" in str(exc_info.value)
class TestBackendAPIResponseValidation:
"""Tests verifying backend API responses are validated with correct parser."""
def test_testgen_request_includes_correct_language(self, tmp_path):
"""Verify test generation request includes the correct language parameter."""
skip_if_js_not_supported()
from codeflash.api.aiservice import AiServiceClient
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current
lang_current._current_language = Language.TYPESCRIPT
ts_file = tmp_path / "utils.ts"
ts_file.write_text("""
export function add(a: number, b: number): number {
return a + b;
}
""")
functions = find_all_functions_in_file(ts_file)
func = functions[ts_file][0]
# Verify function has correct language
assert func.language == "typescript"
# Mock the AI service request
ai_client = AiServiceClient()
with patch.object(ai_client, 'make_ai_service_request') as mock_request:
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"generated_tests": "// test code",
"instrumented_behavior_tests": "// behavior code",
"instrumented_perf_tests": "// perf code",
}
mock_request.return_value = mock_response
# Call generate_regression_tests with correct parameters
ai_client.generate_regression_tests(
source_code_being_tested="export function add(a: number, b: number): number { return a + b; }",
function_to_optimize=func,
helper_function_names=[],
module_path=ts_file,
test_module_path=tmp_path / "tests" / "utils.test.ts",
test_framework="vitest",
test_timeout=30,
trace_id="test-trace-id",
test_index=0,
language=func.language, # This is the key - language should be "typescript"
)
# Verify the request was made with correct language
assert mock_request.called, "API request should have been made"
call_args = mock_request.call_args
payload = call_args[1].get('payload', call_args[0][1] if len(call_args[0]) > 1 else {})
assert payload.get('language') == 'typescript', \
f"Expected language='typescript', got language='{payload.get('language')}'"
class TestFunctionOptimizerForJavaScript:
"""Tests for FunctionOptimizer with JavaScript/TypeScript functions.
This is the JavaScript equivalent of test_instrument_tests.py tests.
"""
@pytest.fixture
def js_project(self, tmp_path):
"""Create a minimal JavaScript project for testing."""
project = tmp_path / "js_project"
project.mkdir()
# Create source file
src_file = project / "utils.js"
src_file.write_text("""
function fibonacci(n) {
if (n <= 1) return n;
return fibonacci(n - 1) + fibonacci(n - 2);
}
module.exports = { fibonacci };
""")
# Create test file
tests_dir = project / "tests"
tests_dir.mkdir()
test_file = tests_dir / "utils.test.js"
test_file.write_text("""
const { fibonacci } = require('../utils');
describe('fibonacci', () => {
test('returns 0 for n=0', () => {
expect(fibonacci(0)).toBe(0);
});
test('returns 1 for n=1', () => {
expect(fibonacci(1)).toBe(1);
});
test('returns 5 for n=5', () => {
expect(fibonacci(5)).toBe(5);
});
});
""")
# Create package.json
package_json = project / "package.json"
package_json.write_text("""
{
"name": "test-project",
"devDependencies": {
"jest": "^29.0.0"
}
}
""")
return project
@pytest.fixture
def ts_project(self, tmp_path):
"""Create a minimal TypeScript project for testing."""
project = tmp_path / "ts_project"
project.mkdir()
# Create source file
src_file = project / "utils.ts"
src_file.write_text("""
export function fibonacci(n: number): number {
if (n <= 1) return n;
return fibonacci(n - 1) + fibonacci(n - 2);
}
""")
# Create test file
tests_dir = project / "tests"
tests_dir.mkdir()
test_file = tests_dir / "utils.test.ts"
test_file.write_text("""
import { fibonacci } from '../utils';
describe('fibonacci', () => {
test('returns 0 for n=0', () => {
expect(fibonacci(0)).toBe(0);
});
test('returns 1 for n=1', () => {
expect(fibonacci(1)).toBe(1);
});
});
""")
# Create package.json
package_json = project / "package.json"
package_json.write_text("""
{
"name": "test-project",
"devDependencies": {
"vitest": "^1.0.0"
}
}
""")
return project
def test_function_optimizer_instantiation_javascript(self, js_project):
"""Test FunctionOptimizer can be instantiated for JavaScript."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.optimization.function_optimizer import FunctionOptimizer
src_file = js_project / "utils.js"
functions = find_all_functions_in_file(src_file)
func = functions[src_file][0]
func_to_optimize = FunctionToOptimize(
function_name=func.function_name,
file_path=func.file_path,
parents=[FunctionParent(name=p.name, type=p.type) for p in func.parents],
starting_line=func.starting_line,
ending_line=func.ending_line,
language=func.language,
)
test_config = TestConfig(
tests_root=js_project / "tests",
tests_project_rootdir=js_project,
project_root_path=js_project,
pytest_cmd="jest",
)
optimizer = FunctionOptimizer(
function_to_optimize=func_to_optimize,
test_cfg=test_config,
aiservice_client=MagicMock(),
)
assert optimizer is not None
assert optimizer.function_to_optimize.language == "javascript"
def test_function_optimizer_instantiation_typescript(self, ts_project):
"""Test FunctionOptimizer can be instantiated for TypeScript."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.optimization.function_optimizer import FunctionOptimizer
src_file = ts_project / "utils.ts"
functions = find_all_functions_in_file(src_file)
func = functions[src_file][0]
func_to_optimize = FunctionToOptimize(
function_name=func.function_name,
file_path=func.file_path,
parents=[FunctionParent(name=p.name, type=p.type) for p in func.parents],
starting_line=func.starting_line,
ending_line=func.ending_line,
language=func.language,
)
test_config = TestConfig(
tests_root=ts_project / "tests",
tests_project_rootdir=ts_project,
project_root_path=ts_project,
pytest_cmd="vitest",
)
optimizer = FunctionOptimizer(
function_to_optimize=func_to_optimize,
test_cfg=test_config,
aiservice_client=MagicMock(),
)
assert optimizer is not None
assert optimizer.function_to_optimize.language == "typescript"
def test_get_code_optimization_context_javascript(self, js_project):
"""Test get_code_optimization_context for JavaScript."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current
from codeflash.optimization.function_optimizer import FunctionOptimizer
lang_current._current_language = Language.JAVASCRIPT
src_file = js_project / "utils.js"
functions = find_all_functions_in_file(src_file)
func = functions[src_file][0]
func_to_optimize = FunctionToOptimize(
function_name=func.function_name,
file_path=func.file_path,
parents=[FunctionParent(name=p.name, type=p.type) for p in func.parents],
starting_line=func.starting_line,
ending_line=func.ending_line,
language=func.language,
)
test_config = TestConfig(
tests_root=js_project / "tests",
tests_project_rootdir=js_project,
project_root_path=js_project,
pytest_cmd="jest",
)
optimizer = FunctionOptimizer(
function_to_optimize=func_to_optimize,
test_cfg=test_config,
aiservice_client=MagicMock(),
)
result = optimizer.get_code_optimization_context()
context = result.unwrap()
assert context is not None
assert context.read_writable_code is not None
assert context.read_writable_code.language == "javascript"
def test_get_code_optimization_context_typescript(self, ts_project):
"""Test get_code_optimization_context for TypeScript."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current
from codeflash.optimization.function_optimizer import FunctionOptimizer
lang_current._current_language = Language.TYPESCRIPT
src_file = ts_project / "utils.ts"
functions = find_all_functions_in_file(src_file)
func = functions[src_file][0]
func_to_optimize = FunctionToOptimize(
function_name=func.function_name,
file_path=func.file_path,
parents=[FunctionParent(name=p.name, type=p.type) for p in func.parents],
starting_line=func.starting_line,
ending_line=func.ending_line,
language=func.language,
)
test_config = TestConfig(
tests_root=ts_project / "tests",
tests_project_rootdir=ts_project,
project_root_path=ts_project,
pytest_cmd="vitest",
)
optimizer = FunctionOptimizer(
function_to_optimize=func_to_optimize,
test_cfg=test_config,
aiservice_client=MagicMock(),
)
result = optimizer.get_code_optimization_context()
context = result.unwrap()
assert context is not None
assert context.read_writable_code is not None
assert context.read_writable_code.language == "typescript"
class TestHelperFunctionLanguageAttribute:
"""Tests for helper function language attribute (import_resolver.py fix)."""
def test_helper_functions_have_correct_language_javascript(self, tmp_path):
"""Verify helper functions have language='javascript' for .js files."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current, get_language_support
from codeflash.optimization.function_optimizer import FunctionOptimizer
lang_current._current_language = Language.JAVASCRIPT
# Create a file with helper functions
src_file = tmp_path / "main.js"
src_file.write_text("""
function helper() {
return 42;
}
function main() {
return helper() * 2;
}
module.exports = { main };
""")
functions = find_all_functions_in_file(src_file)
main_func = next(f for f in functions[src_file] if f.function_name == "main")
func_to_optimize = FunctionToOptimize(
function_name=main_func.function_name,
file_path=main_func.file_path,
parents=[],
starting_line=main_func.starting_line,
ending_line=main_func.ending_line,
language=main_func.language,
)
test_config = TestConfig(
tests_root=tmp_path,
tests_project_rootdir=tmp_path,
project_root_path=tmp_path,
pytest_cmd="jest",
)
optimizer = FunctionOptimizer(
function_to_optimize=func_to_optimize,
test_cfg=test_config,
aiservice_client=MagicMock(),
)
result = optimizer.get_code_optimization_context()
context = result.unwrap()
# Verify main function has correct language
assert context.read_writable_code.language == "javascript"
def test_helper_functions_have_correct_language_typescript(self, tmp_path):
"""Verify helper functions have language='typescript' for .ts files."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current
from codeflash.optimization.function_optimizer import FunctionOptimizer
lang_current._current_language = Language.TYPESCRIPT
# Create a file with helper functions
src_file = tmp_path / "main.ts"
src_file.write_text("""
function helper(): number {
return 42;
}
export function main(): number {
return helper() * 2;
}
""")
functions = find_all_functions_in_file(src_file)
main_func = next(f for f in functions[src_file] if f.function_name == "main")
func_to_optimize = FunctionToOptimize(
function_name=main_func.function_name,
file_path=main_func.file_path,
parents=[],
starting_line=main_func.starting_line,
ending_line=main_func.ending_line,
language=main_func.language,
)
test_config = TestConfig(
tests_root=tmp_path,
tests_project_rootdir=tmp_path,
project_root_path=tmp_path,
pytest_cmd="vitest",
)
optimizer = FunctionOptimizer(
function_to_optimize=func_to_optimize,
test_cfg=test_config,
aiservice_client=MagicMock(),
)
result = optimizer.get_code_optimization_context()
context = result.unwrap()
# Verify main function has correct language
assert context.read_writable_code.language == "typescript"

516
tests/test_languages/test_javascript_run_and_parse.py Normal file
View file

@ -0,0 +1,516 @@
"""End-to-end tests for JavaScript/TypeScript test execution and result parsing.
These tests verify the FULL optimization pipeline including:
- Test instrumentation
- Running instrumented tests with Vitest/Jest
- Parsing test results (stdout, timing, return values)
- Benchmarking with multiple loops
This is the JavaScript equivalent of test_instrument_tests.py for Python.
NOTE: These tests require:
- Node.js installed
- npm packages installed in the test fixture directories
- The codeflash npm package
Tests will be skipped if dependencies are not available.
"""
import os
import shutil
import subprocess
from pathlib import Path
from unittest.mock import MagicMock
import pytest
from codeflash.discovery.functions_to_optimize import FunctionToOptimize
from codeflash.languages.base import Language
from codeflash.models.models import FunctionParent, TestFile, TestFiles, TestType, TestingMode
from codeflash.verification.verification_utils import TestConfig
def is_node_available():
"""Check if Node.js is available."""
try:
result = subprocess.run(["node", "--version"], capture_output=True, text=True)
return result.returncode == 0
except FileNotFoundError:
return False
def is_npm_available():
"""Check if npm is available."""
try:
result = subprocess.run(["npm", "--version"], capture_output=True, text=True)
return result.returncode == 0
except FileNotFoundError:
return False
def has_node_modules(project_dir: Path) -> bool:
"""Check if node_modules exists in project directory."""
return (project_dir / "node_modules").exists()
def install_dependencies(project_dir: Path) -> bool:
"""Install npm dependencies in project directory."""
if has_node_modules(project_dir):
return True
try:
result = subprocess.run(
["npm", "install"],
cwd=project_dir,
capture_output=True,
text=True,
timeout=120
)
return result.returncode == 0
except Exception:
return False
def skip_if_js_runtime_not_available():
"""Skip test if JavaScript runtime is not available."""
if not is_node_available():
pytest.skip("Node.js not available")
if not is_npm_available():
pytest.skip("npm not available")
def skip_if_js_not_supported():
"""Skip test if JavaScript/TypeScript languages are not supported."""
try:
from codeflash.languages import get_language_support
get_language_support(Language.JAVASCRIPT)
except Exception as e:
pytest.skip(f"JavaScript/TypeScript language support not available: {e}")
class TestJavaScriptInstrumentation:
"""Tests for JavaScript test instrumentation."""
@pytest.fixture
def js_project_dir(self, tmp_path):
"""Create a temporary JavaScript project with Jest."""
project_dir = tmp_path / "js_project"
project_dir.mkdir()
# Create source file
src_file = project_dir / "math.js"
src_file.write_text("""
function add(a, b) {
return a + b;
}
function multiply(a, b) {
return a * b;
}
module.exports = { add, multiply };
""")
# Create test file
tests_dir = project_dir / "__tests__"
tests_dir.mkdir()
test_file = tests_dir / "math.test.js"
test_file.write_text("""
const { add, multiply } = require('../math');
describe('math functions', () => {
test('add returns sum', () => {
expect(add(2, 3)).toBe(5);
});
test('multiply returns product', () => {
expect(multiply(2, 3)).toBe(6);
});
});
""")
# Create package.json
package_json = project_dir / "package.json"
package_json.write_text("""{
"name": "test-project",
"version": "1.0.0",
"scripts": {
"test": "jest"
},
"devDependencies": {
"jest": "^29.0.0",
"jest-junit": "^16.0.0"
}
}""")
# Create jest.config.js
jest_config = project_dir / "jest.config.js"
jest_config.write_text("""
module.exports = {
testEnvironment: 'node',
reporters: ['default', 'jest-junit'],
};
""")
return project_dir
def test_instrument_javascript_test_file(self, js_project_dir):
"""Test that JavaScript test instrumentation module can be imported."""
skip_if_js_not_supported()
from codeflash.languages import get_language_support
# Verify the instrumentation module can be imported
from codeflash.languages.javascript.instrument import inject_profiling_into_existing_js_test
# Get JavaScript support
js_support = get_language_support(Language.JAVASCRIPT)
# Create function info
func_info = FunctionToOptimize(
function_name="add",
file_path=js_project_dir / "math.js",
parents=[],
starting_line=2,
ending_line=4,
language="javascript",
)
# Verify function has correct language
assert func_info.language == "javascript"
# Verify test file exists
test_file = js_project_dir / "__tests__" / "math.test.js"
assert test_file.exists()
# Note: Full instrumentation test requires call_positions discovery
# which is done by the FunctionOptimizer. Here we just verify the
# infrastructure is in place.
class TestTypeScriptInstrumentation:
"""Tests for TypeScript test instrumentation."""
@pytest.fixture
def ts_project_dir(self, tmp_path):
"""Create a temporary TypeScript project with Vitest."""
project_dir = tmp_path / "ts_project"
project_dir.mkdir()
# Create source file
src_file = project_dir / "math.ts"
src_file.write_text("""
export function add(a: number, b: number): number {
return a + b;
}
export function multiply(a: number, b: number): number {
return a * b;
}
""")
# Create test file
tests_dir = project_dir / "tests"
tests_dir.mkdir()
test_file = tests_dir / "math.test.ts"
test_file.write_text("""
import { describe, test, expect } from 'vitest';
import { add, multiply } from '../math';
describe('math functions', () => {
test('add returns sum', () => {
expect(add(2, 3)).toBe(5);
});
test('multiply returns product', () => {
expect(multiply(2, 3)).toBe(6);
});
});
""")
# Create package.json
package_json = project_dir / "package.json"
package_json.write_text("""{
"name": "test-project",
"version": "1.0.0",
"type": "module",
"scripts": {
"test": "vitest run"
},
"devDependencies": {
"vitest": "^1.0.0",
"typescript": "^5.0.0"
}
}""")
# Create vitest.config.ts
vitest_config = project_dir / "vitest.config.ts"
vitest_config.write_text("""
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
globals: false,
reporters: ['verbose', 'junit'],
outputFile: './junit.xml',
},
});
""")
# Create tsconfig.json
tsconfig = project_dir / "tsconfig.json"
tsconfig.write_text("""{
"compilerOptions": {
"target": "ES2020",
"module": "ESNext",
"moduleResolution": "node",
"strict": true,
"esModuleInterop": true
}
}""")
return project_dir
def test_instrument_typescript_test_file(self, ts_project_dir):
"""Test that TypeScript test instrumentation module can be imported."""
skip_if_js_not_supported()
from codeflash.languages import get_language_support
# Verify the instrumentation module can be imported
from codeflash.languages.javascript.instrument import inject_profiling_into_existing_js_test
test_file = ts_project_dir / "tests" / "math.test.ts"
# Get TypeScript support
ts_support = get_language_support(Language.TYPESCRIPT)
# Create function info
func_info = FunctionToOptimize(
function_name="add",
file_path=ts_project_dir / "math.ts",
parents=[],
starting_line=2,
ending_line=4,
language="typescript",
)
# Verify function has correct language
assert func_info.language == "typescript"
# Verify test file exists
assert test_file.exists()
# Note: Full instrumentation test requires call_positions discovery
# which is done by the FunctionOptimizer. Here we just verify the
# infrastructure is in place.
class TestRunAndParseJavaScriptTests:
"""Tests for running and parsing JavaScript test results.
These tests require actual npm dependencies to be installed.
They will be skipped if dependencies are not available.
"""
@pytest.fixture
def vitest_project(self):
"""Get the Vitest sample project with dependencies installed."""
project_root = Path(__file__).parent.parent.parent
vitest_dir = project_root / "code_to_optimize" / "js" / "code_to_optimize_vitest"
if not vitest_dir.exists():
pytest.skip("code_to_optimize_vitest directory not found")
skip_if_js_runtime_not_available()
# Try to install dependencies if not present
if not has_node_modules(vitest_dir):
if not install_dependencies(vitest_dir):
pytest.skip("Could not install npm dependencies")
return vitest_dir
def test_run_behavioral_tests_vitest(self, vitest_project):
"""Test running behavioral tests with Vitest."""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import get_language_support
ts_support = get_language_support(Language.TYPESCRIPT)
# Find the fibonacci function
fib_file = vitest_project / "fibonacci.ts"
functions = find_all_functions_in_file(fib_file)
fib_func = next(f for f in functions[fib_file] if f.function_name == "fibonacci")
# Verify language is correct
assert fib_func.language == "typescript"
# Discover tests
test_root = vitest_project / "tests"
tests = ts_support.discover_tests(test_root, [fib_func])
# There should be tests for fibonacci
assert len(tests) > 0 or fib_func.qualified_name in tests
def test_function_optimizer_run_and_parse_typescript(self, vitest_project):
"""Test FunctionOptimizer.run_and_parse_tests for TypeScript.
This is the JavaScript equivalent of the Python test in test_instrument_tests.py.
"""
skip_if_js_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current
from codeflash.optimization.function_optimizer import FunctionOptimizer
lang_current._current_language = Language.TYPESCRIPT
# Find the fibonacci function
fib_file = vitest_project / "fibonacci.ts"
functions = find_all_functions_in_file(fib_file)
fib_func_info = next(f for f in functions[fib_file] if f.function_name == "fibonacci")
# Create FunctionToOptimize
func = FunctionToOptimize(
function_name=fib_func_info.function_name,
file_path=fib_func_info.file_path,
parents=[FunctionParent(name=p.name, type=p.type) for p in fib_func_info.parents],
starting_line=fib_func_info.starting_line,
ending_line=fib_func_info.ending_line,
language=fib_func_info.language,
)
# Verify language
assert func.language == "typescript"
# Create test config
test_config = TestConfig(
tests_root=vitest_project / "tests",
tests_project_rootdir=vitest_project,
project_root_path=vitest_project,
pytest_cmd="vitest",
test_framework="vitest",
)
# Create optimizer
func_optimizer = FunctionOptimizer(
function_to_optimize=func,
test_cfg=test_config,
aiservice_client=MagicMock(),
)
# Get code context - this should work
result = func_optimizer.get_code_optimization_context()
context = result.unwrap()
assert context is not None
assert context.read_writable_code.language == "typescript"
class TestTimingMarkerParsing:
"""Tests for parsing JavaScript timing markers from test output.
Note: Timing marker parsing is handled in codeflash/verification/parse_test_output.py,
which uses a unified parser for all languages. These tests verify the marker format
is correctly recognized.
"""
def test_timing_marker_format(self):
"""Test that JavaScript timing markers follow the expected format."""
skip_if_js_not_supported()
import re
# The marker format used by codeflash for JavaScript
# Start marker: !$######{tag}######$!
# End marker: !######{tag}:{duration}######!
start_pattern = r'!\$######(.+?)######\$!'
end_pattern = r'!######(.+?):(\d+)######!'
start_marker = "!$######test/math.test.ts:TestMath.test_add:add:1:0_0######$!"
end_marker = "!######test/math.test.ts:TestMath.test_add:add:1:0_0:12345######!"
start_match = re.match(start_pattern, start_marker)
end_match = re.match(end_pattern, end_marker)
assert start_match is not None
assert end_match is not None
assert start_match.group(1) == "test/math.test.ts:TestMath.test_add:add:1:0_0"
assert end_match.group(1) == "test/math.test.ts:TestMath.test_add:add:1:0_0"
assert end_match.group(2) == "12345"
def test_timing_marker_components(self):
"""Test parsing components from timing marker tag."""
skip_if_js_not_supported()
# Tag format: {module}:{class}.{test}:{function}:{loop_index}:{invocation_id}
tag = "test/math.test.ts:TestMath.test_add:add:1:0_0"
parts = tag.split(":")
assert len(parts) == 5
assert parts[0] == "test/math.test.ts" # module/file
assert parts[1] == "TestMath.test_add" # class.test
assert parts[2] == "add" # function being tested
assert parts[3] == "1" # loop index
assert parts[4] == "0_0" # invocation id
class TestJavaScriptTestResultParsing:
"""Tests for parsing JavaScript test results from JUnit XML."""
def test_parse_vitest_junit_xml(self, tmp_path):
"""Test parsing Vitest JUnit XML output."""
skip_if_js_not_supported()
# Create sample JUnit XML
junit_xml = tmp_path / "junit.xml"
junit_xml.write_text("""<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="vitest tests" tests="2" failures="0" errors="0" time="0.5">
<testsuite name="tests/math.test.ts" tests="2" failures="0" errors="0" time="0.5">
<testcase classname="tests/math.test.ts" name="add returns sum" time="0.1">
</testcase>
<testcase classname="tests/math.test.ts" name="multiply returns product" time="0.2">
</testcase>
</testsuite>
</testsuites>
""")
# Parse the XML
import xml.etree.ElementTree as ET
tree = ET.parse(junit_xml)
root = tree.getroot()
# Verify structure
testsuites = root if root.tag == "testsuites" else root.find("testsuites")
assert testsuites is not None
testsuite = testsuites.find("testsuite") if testsuites is not None else root.find("testsuite")
assert testsuite is not None
testcases = testsuite.findall("testcase")
assert len(testcases) == 2
def test_parse_jest_junit_xml(self, tmp_path):
"""Test parsing Jest JUnit XML output."""
skip_if_js_not_supported()
# Create sample JUnit XML from jest-junit
junit_xml = tmp_path / "junit.xml"
junit_xml.write_text("""<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="jest tests" tests="2" failures="0" time="0.789">
<testsuite name="math functions" tests="2" failures="0" time="0.456" timestamp="2024-01-01T00:00:00">
<testcase classname="__tests__/math.test.js" name="add returns sum" time="0.123">
</testcase>
<testcase classname="__tests__/math.test.js" name="multiply returns product" time="0.234">
</testcase>
</testsuite>
</testsuites>
""")
# Parse the XML
import xml.etree.ElementTree as ET
tree = ET.parse(junit_xml)
root = tree.getroot()
# Verify structure
testsuites = root if root.tag == "testsuites" else root.find("testsuites")
testsuite = testsuites.find("testsuite") if testsuites is not None else root.find("testsuite")
assert testsuite is not None
testcases = testsuite.findall("testcase")
assert len(testcases) == 2

94
tests/test_languages/test_javascript_support.py
View file

@ -1605,3 +1605,97 @@ export class MathUtils {
f"Replacement result does not match expected.\nExpected:\n{expected_result}\n\nGot:\n{result}"
)
assert js_support.validate_syntax(result) is True
class TestTypeScriptSyntaxValidation:
"""Tests for TypeScript-specific syntax validation.
These tests ensure that TypeScript code is validated with the TypeScript parser,
not the JavaScript parser. This is important because TypeScript has syntax that
is invalid in JavaScript (e.g., type assertions, type annotations).
"""
def test_typescript_type_assertion_valid_in_ts(self):
"""TypeScript type assertions should be valid in TypeScript."""
from codeflash.languages.javascript.support import TypeScriptSupport
ts_support = TypeScriptSupport()
# Type assertions are TypeScript-specific
ts_code = """
const value = 4.9 as unknown as number;
const str = "hello" as string;
"""
assert ts_support.validate_syntax(ts_code) is True
def test_typescript_type_assertion_invalid_in_js(self, js_support):
"""TypeScript type assertions should be invalid in JavaScript."""
# This is the code pattern that caused the backend error
ts_code = """
const value = 4.9 as unknown as number;
"""
# JavaScript parser should reject TypeScript syntax
assert js_support.validate_syntax(ts_code) is False
def test_typescript_interface_valid_in_ts(self):
"""TypeScript interfaces should be valid in TypeScript."""
from codeflash.languages.javascript.support import TypeScriptSupport
ts_support = TypeScriptSupport()
ts_code = """
interface User {
name: string;
age: number;
}
"""
assert ts_support.validate_syntax(ts_code) is True
def test_typescript_interface_invalid_in_js(self, js_support):
"""TypeScript interfaces should be invalid in JavaScript."""
ts_code = """
interface User {
name: string;
age: number;
}
"""
# JavaScript parser should reject TypeScript interface syntax
assert js_support.validate_syntax(ts_code) is False
def test_typescript_generic_function_valid_in_ts(self):
"""TypeScript generics should be valid in TypeScript."""
from codeflash.languages.javascript.support import TypeScriptSupport
ts_support = TypeScriptSupport()
ts_code = """
function identity<T>(arg: T): T {
return arg;
}
"""
assert ts_support.validate_syntax(ts_code) is True
def test_typescript_generic_function_invalid_in_js(self, js_support):
"""TypeScript generics should be invalid in JavaScript."""
ts_code = """
function identity<T>(arg: T): T {
return arg;
}
"""
assert js_support.validate_syntax(ts_code) is False
def test_language_property_is_typescript(self):
"""TypeScriptSupport should report typescript as language."""
from codeflash.languages.base import Language
from codeflash.languages.javascript.support import TypeScriptSupport
ts_support = TypeScriptSupport()
assert ts_support.language == Language.TYPESCRIPT
assert str(ts_support.language) == "typescript"
def test_language_property_is_javascript(self, js_support):
"""JavaScriptSupport should report javascript as language."""
from codeflash.languages.base import Language
assert js_support.language == Language.JAVASCRIPT
assert str(js_support.language) == "javascript"

16
tests/test_languages/test_js_code_replacer.py
View file

@ -434,8 +434,8 @@ function process() {
# Should be converted to ESM
assert "import x from './module';" in result
def test_mixed_code_not_converted(self, tmp_path):
"""Test that mixed CJS/ESM code is NOT converted (already has both)."""
def test_mixed_code_converted_to_esm(self, tmp_path):
"""Test that mixed CJS/ESM code has require converted to import when targeting ESM."""
package_json = tmp_path / "package.json"
package_json.write_text('{"devDependencies": {"jest": "^29.0.0"}}')
@ -447,10 +447,18 @@ function process() {
return existing() + helper();
}
"""
# Mixed code has both import and require, so no conversion
expected = """\
import { existing } from './module.js';
import { helper } from './helpers';
function process() {
return existing() + helper();
}
"""
# Mixed code should have require converted to import for ESM target
result = ensure_module_system_compatibility(mixed_code, ModuleSystem.ES_MODULE, tmp_path)
assert result == mixed_code, "Mixed code should not be converted"
assert result == expected, "require should be converted to import for ESM target"
def test_pure_esm_unchanged_for_esm_target(self, tmp_path):
"""Test that pure ESM code is unchanged when targeting ESM."""

5
tests/test_languages/test_registry.py
View file

@ -271,11 +271,14 @@ class TestClearFunctions:
# Now Python should not be supported
assert not is_language_supported(Language.PYTHON)
# Re-register by importing
# Re-register all languages by importing
from codeflash.languages.python.support import PythonSupport
from codeflash.languages.javascript.support import JavaScriptSupport, TypeScriptSupport
# Need to manually register since decorator already ran
register_language(PythonSupport)
register_language(JavaScriptSupport)
register_language(TypeScriptSupport)
# Should be supported again
assert is_language_supported(Language.PYTHON)

446
tests/test_languages/test_typescript_e2e.py Normal file
View file

@ -0,0 +1,446 @@
"""End-to-end integration tests for TypeScript pipeline.
Tests the full optimization pipeline for TypeScript:
- Function discovery
- Code context extraction
- Test discovery
- Code replacement
- Syntax validation with TypeScript parser (not JavaScript)
This is the TypeScript equivalent of test_javascript_e2e.py.
Ensures parity between JavaScript and TypeScript support.
"""
import tempfile
from pathlib import Path
import pytest
from codeflash.languages.base import Language
def skip_if_ts_not_supported():
"""Skip test if TypeScript language is not supported."""
try:
from codeflash.languages import get_language_support
get_language_support(Language.TYPESCRIPT)
except Exception as e:
pytest.skip(f"TypeScript language support not available: {e}")
class TestTypeScriptFunctionDiscovery:
"""Tests for TypeScript function discovery in the main pipeline."""
@pytest.fixture
def ts_project_dir(self):
"""Get the TypeScript sample project directory."""
project_root = Path(__file__).parent.parent.parent
ts_dir = project_root / "code_to_optimize" / "js" / "code_to_optimize_vitest"
if not ts_dir.exists():
pytest.skip("code_to_optimize_vitest directory not found")
return ts_dir
def test_discover_functions_in_typescript_file(self, ts_project_dir):
"""Test discovering functions in a TypeScript file."""
skip_if_ts_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
fib_file = ts_project_dir / "fibonacci.ts"
if not fib_file.exists():
pytest.skip("fibonacci.ts not found")
functions = find_all_functions_in_file(fib_file)
assert fib_file in functions
func_list = functions[fib_file]
func_names = {f.function_name for f in func_list}
assert "fibonacci" in func_names
# Critical: Verify language is "typescript", not "javascript"
for func in func_list:
assert func.language == "typescript", \
f"Function {func.function_name} should have language='typescript', got '{func.language}'"
def test_discover_functions_with_type_annotations(self):
"""Test discovering TypeScript functions with type annotations."""
skip_if_ts_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
with tempfile.NamedTemporaryFile(suffix=".ts", mode="w", delete=False) as f:
f.write("""
export function add(a: number, b: number): number {
return a + b;
}
export function greet(name: string): string {
return `Hello, \${name}!`;
}
interface User {
name: string;
age: number;
}
export function getUserAge(user: User): number {
return user.age;
}
""")
f.flush()
file_path = Path(f.name)
functions = find_all_functions_in_file(file_path)
assert len(functions.get(file_path, [])) == 3
for func in functions[file_path]:
assert func.language == "typescript"
def test_get_typescript_files(self, ts_project_dir):
"""Test getting TypeScript files from directory."""
skip_if_ts_not_supported()
from codeflash.discovery.functions_to_optimize import get_files_for_language
files = get_files_for_language(ts_project_dir, Language.TYPESCRIPT)
ts_files = [f for f in files if f.suffix == ".ts" and "test" not in f.name]
assert len(ts_files) >= 1
class TestTypeScriptCodeContext:
"""Tests for TypeScript code context extraction."""
@pytest.fixture
def ts_project_dir(self):
"""Get the TypeScript sample project directory."""
project_root = Path(__file__).parent.parent.parent
ts_dir = project_root / "code_to_optimize" / "js" / "code_to_optimize_vitest"
if not ts_dir.exists():
pytest.skip("code_to_optimize_vitest directory not found")
return ts_dir
def test_extract_code_context_for_typescript(self, ts_project_dir):
"""Test extracting code context for a TypeScript function."""
skip_if_ts_not_supported()
from codeflash.context.code_context_extractor import get_code_optimization_context
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
from codeflash.languages import current as lang_current
lang_current._current_language = Language.TYPESCRIPT
fib_file = ts_project_dir / "fibonacci.ts"
if not fib_file.exists():
pytest.skip("fibonacci.ts not found")
functions = find_all_functions_in_file(fib_file)
func_list = functions[fib_file]
fib_func = next((f for f in func_list if f.function_name == "fibonacci"), None)
assert fib_func is not None
context = get_code_optimization_context(fib_func, ts_project_dir)
assert context.read_writable_code is not None
# Critical: language should be "typescript", not "javascript"
assert context.read_writable_code.language == "typescript"
assert len(context.read_writable_code.code_strings) > 0
class TestTypeScriptCodeReplacement:
"""Tests for TypeScript code replacement."""
def test_replace_function_in_typescript_file(self):
"""Test replacing a function in a TypeScript file."""
skip_if_ts_not_supported()
from codeflash.languages import get_language_support
from codeflash.languages.base import FunctionInfo
original_source = """
function add(a: number, b: number): number {
return a + b;
}
function multiply(a: number, b: number): number {
return a * b;
}
"""
new_function = """function add(a: number, b: number): number {
// Optimized version
return a + b;
}"""
ts_support = get_language_support(Language.TYPESCRIPT)
func_info = FunctionInfo(
function_name="add",
file_path=Path("/tmp/test.ts"),
starting_line=2,
ending_line=4,
language="typescript"
)
result = ts_support.replace_function(original_source, func_info, new_function)
expected_result = """
function add(a: number, b: number): number {
// Optimized version
return a + b;
}
function multiply(a: number, b: number): number {
return a * b;
}
"""
assert result == expected_result
def test_replace_function_preserves_types(self):
"""Test that replacing a function preserves TypeScript type annotations."""
skip_if_ts_not_supported()
from codeflash.languages import get_language_support
from codeflash.languages.base import FunctionInfo
original_source = """
interface Config {
timeout: number;
retries: number;
}
function processConfig(config: Config): string {
return `timeout=\${config.timeout}, retries=\${config.retries}`;
}
"""
new_function = """function processConfig(config: Config): string {
// Optimized with template caching
const { timeout, retries } = config;
return `timeout=\${timeout}, retries=\${retries}`;
}"""
ts_support = get_language_support(Language.TYPESCRIPT)
func_info = FunctionInfo(
function_name="processConfig",
file_path=Path("/tmp/test.ts"),
starting_line=7,
ending_line=9,
language="typescript"
)
result = ts_support.replace_function(original_source, func_info, new_function)
# Verify type annotations are preserved
assert "config: Config" in result
assert ": string" in result
assert "interface Config" in result
class TestTypeScriptTestDiscovery:
"""Tests for TypeScript test discovery."""
@pytest.fixture
def ts_project_dir(self):
"""Get the TypeScript sample project directory."""
project_root = Path(__file__).parent.parent.parent
ts_dir = project_root / "code_to_optimize" / "js" / "code_to_optimize_vitest"
if not ts_dir.exists():
pytest.skip("code_to_optimize_vitest directory not found")
return ts_dir
def test_discover_vitest_tests_for_typescript(self, ts_project_dir):
"""Test discovering Vitest tests for TypeScript functions."""
skip_if_ts_not_supported()
from codeflash.languages import get_language_support
from codeflash.languages.base import FunctionInfo
ts_support = get_language_support(Language.TYPESCRIPT)
test_root = ts_project_dir / "tests"
if not test_root.exists():
pytest.skip("tests directory not found")
fib_file = ts_project_dir / "fibonacci.ts"
func_info = FunctionInfo(
function_name="fibonacci",
file_path=fib_file,
starting_line=1,
ending_line=7,
language="typescript"
)
tests = ts_support.discover_tests(test_root, [func_info])
# Should find tests for the fibonacci function
assert func_info.qualified_name in tests or len(tests) > 0
class TestTypeScriptPipelineIntegration:
"""Integration tests for the full TypeScript pipeline."""
def test_function_to_optimize_has_correct_fields(self):
"""Test that FunctionToOptimize from TypeScript has all required fields."""
skip_if_ts_not_supported()
from codeflash.discovery.functions_to_optimize import find_all_functions_in_file
with tempfile.NamedTemporaryFile(suffix=".ts", mode="w", delete=False) as f:
f.write("""
class Calculator {
add(a: number, b: number): number {
return a + b;
}
subtract(a: number, b: number): number {
return a - b;
}
}
function standalone(x: number): number {
return x * 2;
}
""")
f.flush()
file_path = Path(f.name)
functions = find_all_functions_in_file(file_path)
assert len(functions.get(file_path, [])) >= 3
standalone_fn = next((fn for fn in functions[file_path] if fn.function_name == "standalone"), None)
assert standalone_fn is not None
assert standalone_fn.language == "typescript"
assert len(standalone_fn.parents) == 0
add_fn = next((fn for fn in functions[file_path] if fn.function_name == "add"), None)
assert add_fn is not None
assert add_fn.language == "typescript"
assert len(add_fn.parents) == 1
assert add_fn.parents[0].name == "Calculator"
def test_code_strings_markdown_uses_typescript_tag(self):
"""Test that CodeStringsMarkdown uses typescript for code blocks."""
from codeflash.models.models import CodeString, CodeStringsMarkdown
code_strings = CodeStringsMarkdown(
code_strings=[
CodeString(
code="function add(a: number, b: number): number { return a + b; }",
file_path=Path("test.ts"),
language="typescript"
)
],
language="typescript",
)
markdown = code_strings.markdown
assert "```typescript" in markdown
class TestTypeScriptSyntaxValidation:
"""Tests for TypeScript-specific syntax validation.
These tests ensure TypeScript code is validated with the TypeScript parser,
not the JavaScript parser. This was the root cause of production issues.
"""
def test_typescript_type_assertion_valid(self):
"""TypeScript type assertions should be valid."""
skip_if_ts_not_supported()
from codeflash.languages import get_language_support
ts_support = get_language_support(Language.TYPESCRIPT)
# This is TypeScript-specific syntax that should pass
code = "const value = 4.9 as unknown as number;"
assert ts_support.validate_syntax(code) is True
def test_typescript_type_assertion_invalid_in_javascript(self):
"""TypeScript type assertions should be INVALID in JavaScript.
This test would have caught the production bug where TypeScript code
was being validated with the JavaScript parser.
"""
skip_if_ts_not_supported()
from codeflash.languages import get_language_support
js_support = get_language_support(Language.JAVASCRIPT)
# This TypeScript syntax should FAIL JavaScript validation
code = "const value = 4.9 as unknown as number;"
assert js_support.validate_syntax(code) is False
def test_typescript_interface_valid(self):
"""TypeScript interfaces should be valid."""
skip_if_ts_not_supported()
from codeflash.languages import get_language_support
ts_support = get_language_support(Language.TYPESCRIPT)
code = """
interface User {
name: string;
age: number;
}
"""
assert ts_support.validate_syntax(code) is True
def test_typescript_interface_invalid_in_javascript(self):
"""TypeScript interfaces should be INVALID in JavaScript."""
skip_if_ts_not_supported()
from codeflash.languages import get_language_support
js_support = get_language_support(Language.JAVASCRIPT)
code = """
interface User {
name: string;
age: number;
}
"""
assert js_support.validate_syntax(code) is False
def test_typescript_generic_function_valid(self):
"""TypeScript generics should be valid."""
skip_if_ts_not_supported()
from codeflash.languages import get_language_support
ts_support = get_language_support(Language.TYPESCRIPT)
code = "function identity<T>(arg: T): T { return arg; }"
assert ts_support.validate_syntax(code) is True
def test_typescript_generic_function_invalid_in_javascript(self):
"""TypeScript generics should be INVALID in JavaScript."""
skip_if_ts_not_supported()
from codeflash.languages import get_language_support
js_support = get_language_support(Language.JAVASCRIPT)
code = "function identity<T>(arg: T): T { return arg; }"
assert js_support.validate_syntax(code) is False
class TestTypeScriptCodeStringValidation:
"""Tests for CodeString validation with TypeScript."""
def test_code_string_validates_typescript_with_typescript_parser(self):
"""CodeString with language='typescript' should use TypeScript parser."""
skip_if_ts_not_supported()
from codeflash.models.models import CodeString
# TypeScript-specific syntax should pass when language='typescript'
ts_code = "const value = 4.9 as unknown as number;"
cs = CodeString(code=ts_code, language="typescript")
assert cs.code == ts_code
def test_code_string_rejects_typescript_with_javascript_parser(self):
"""CodeString with language='javascript' should reject TypeScript syntax."""
skip_if_ts_not_supported()
from pydantic import ValidationError
from codeflash.models.models import CodeString
# TypeScript-specific syntax should FAIL when language='javascript'
ts_code = "const value = 4.9 as unknown as number;"
with pytest.raises(ValidationError):
CodeString(code=ts_code, language="javascript")

134
tests/test_validate_javascript_code.py Normal file
View file

@ -0,0 +1,134 @@
"""Tests for JavaScript/TypeScript code validation in CodeString.
These tests ensure that JavaScript and TypeScript code is validated correctly
using the appropriate syntax parser for each language.
"""
import pytest
from pydantic import ValidationError
from codeflash.api.aiservice import AiServiceClient
from codeflash.models.models import CodeString, OptimizedCandidateSource
class TestJavaScriptCodeValidation:
"""Tests for JavaScript code validation."""
def test_valid_javascript_code(self):
"""Valid JavaScript code should pass validation."""
valid_code = "const x = 1;\nconst y = x + 2;\nconsole.log(y);"
cs = CodeString(code=valid_code, language="javascript")
assert cs.code == valid_code
def test_invalid_javascript_code_syntax(self):
"""Invalid JavaScript syntax should raise ValidationError."""
invalid_code = "const x = 1;\nconsole.log(x" # Missing closing parenthesis
with pytest.raises(ValidationError) as exc_info:
CodeString(code=invalid_code, language="javascript")
assert "Invalid Javascript code" in str(exc_info.value)
def test_javascript_empty_code(self):
"""Empty code is syntactically valid."""
empty_code = ""
cs = CodeString(code=empty_code, language="javascript")
assert cs.code == empty_code
def test_javascript_arrow_function(self):
"""Arrow functions should be valid JavaScript."""
code = "const add = (a, b) => a + b;"
cs = CodeString(code=code, language="javascript")
assert cs.code == code
class TestTypeScriptCodeValidation:
"""Tests for TypeScript code validation."""
def test_valid_typescript_code(self):
"""Valid TypeScript code should pass validation."""
valid_code = "const x: number = 1;\nconst y: number = x + 2;\nconsole.log(y);"
cs = CodeString(code=valid_code, language="typescript")
assert cs.code == valid_code
def test_typescript_type_assertion(self):
"""TypeScript type assertions should be valid."""
code = "const value = 4.9 as unknown as number;"
cs = CodeString(code=code, language="typescript")
assert cs.code == code
def test_typescript_interface(self):
"""TypeScript interfaces should be valid."""
code = "interface User { name: string; age: number; }"
cs = CodeString(code=code, language="typescript")
assert cs.code == code
def test_typescript_generic_function(self):
"""TypeScript generics should be valid."""
code = "function identity<T>(arg: T): T { return arg; }"
cs = CodeString(code=code, language="typescript")
assert cs.code == code
def test_invalid_typescript_code_syntax(self):
"""Invalid TypeScript syntax should raise ValidationError."""
invalid_code = "const x: number = 1;\nconsole.log(x" # Missing closing parenthesis
with pytest.raises(ValidationError) as exc_info:
CodeString(code=invalid_code, language="typescript")
assert "Invalid Typescript code" in str(exc_info.value)
def test_typescript_syntax_invalid_as_javascript(self):
"""TypeScript-specific syntax should fail when validated as JavaScript."""
ts_code = "const value = 4.9 as unknown as number;"
# Should pass as TypeScript
cs_ts = CodeString(code=ts_code, language="typescript")
assert cs_ts.code == ts_code
# Should fail as JavaScript (type assertions are not valid JS)
with pytest.raises(ValidationError) as exc_info:
CodeString(code=ts_code, language="javascript")
assert "Invalid Javascript code" in str(exc_info.value)
class TestGeneratedCandidatesValidation:
"""Tests for validation of generated optimization candidates."""
def test_javascript_generated_candidates_validation(self):
"""JavaScript optimization candidates should be validated."""
ai_service = AiServiceClient()
# Invalid JavaScript code
invalid_code = """```javascript:file.js
const x = 1
console.log(x
```"""
mock_candidates = [{"source_code": invalid_code, "explanation": "", "optimization_id": ""}]
candidates = ai_service._get_valid_candidates(
mock_candidates, OptimizedCandidateSource.OPTIMIZE, language="javascript"
)
assert len(candidates) == 0
# Valid JavaScript code
valid_code = """```javascript:file.js
const x = 1;
console.log(x);
```"""
mock_candidates = [{"source_code": valid_code, "explanation": "", "optimization_id": ""}]
candidates = ai_service._get_valid_candidates(
mock_candidates, OptimizedCandidateSource.OPTIMIZE, language="javascript"
)
assert len(candidates) == 1
def test_typescript_generated_candidates_validation(self):
"""TypeScript optimization candidates should be validated."""
ai_service = AiServiceClient()
# TypeScript code with type assertions (valid TS, invalid JS)
ts_code = """```typescript:file.ts
const value = 4.9 as unknown as number;
console.log(value);
```"""
mock_candidates = [{"source_code": ts_code, "explanation": "", "optimization_id": ""}]
# Should pass when validated as TypeScript
candidates = ai_service._get_valid_candidates(
mock_candidates, OptimizedCandidateSource.OPTIMIZE, language="typescript"
)
assert len(candidates) == 1