codeflash-ai/codeflash-internal
1
0
Fork 0
Code Issues Pull requests 39 Projects Releases Packages Wiki Activity Actions 2.0k

Optimize LLMClient.call

Browse source 
The optimization adds an early-exit check in `calculate_llm_cost` that returns zero immediately when all rate fields (`input_cost`, `cached_input_cost`, `output_cost`) are zero, before extracting token counts via `getattr` calls. Line profiler confirms the hot path: the original spent 70.7% of function time (580 ms) in the final return statement's arithmetic, but 99.3% of calls (949/956) had zero-cost models where token extraction was wasted work. The optimized version short-circuits these cases in 1.9 ms total, cutting `calculate_llm_cost` from 821 ms to 29 ms (96.5% reduction). This cascades to `LLMClient.call`, where cost calculation dropped from 50.5% to 4.3% of method time, yielding an 80% throughput gain (6165 → 11,097 ops/sec) despite a 37% concurrency ratio regression caused by spending proportionally more time in non-yielding sync code after eliminating the async bottleneck.
This commit is contained in:
codeflash-ai[bot] 2026-04-05 05:05:52 +00:00 committed by GitHub
parent c60d43d334
commit 1f621c4682
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 9 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

13
django/aiservice/aiservice/llm.py
View file

@ -236,6 +236,15 @@ def calculate_llm_cost(response: ChatCompletion | AnthropicMessage, llm: LLM) ->
usage = response.usage usage = response.usage
# Early evaluate rates to allow short-circuit when all are zero
input_rate = llm.input_cost or 0.0
cached_rate = llm.cached_input_cost if llm.cached_input_cost is not None else input_rate
output_rate = llm.output_cost or 0.0
# If there is no charge for input or output whatsoever, skip all computation
if input_rate == 0.0 and output_rate == 0.0 and (llm.cached_input_cost is None or llm.cached_input_cost == 0.0):
return 0.0
# OpenAI: prompt_tokens is total (cached is subset), Anthropic: counts are additive # OpenAI: prompt_tokens is total (cached is subset), Anthropic: counts are additive
if llm.model_type == "anthropic": if llm.model_type == "anthropic":
cache_read = getattr(usage, "cache_read_input_tokens", 0) or 0 cache_read = getattr(usage, "cache_read_input_tokens", 0) or 0
@ -247,10 +256,6 @@ def calculate_llm_cost(response: ChatCompletion | AnthropicMessage, llm: LLM) ->
non_cached = usage.prompt_tokens - cache_read # type: ignore[union-attr] non_cached = usage.prompt_tokens - cache_read # type: ignore[union-attr]
output = usage.completion_tokens # type: ignore[union-attr] output = usage.completion_tokens # type: ignore[union-attr]
input_rate = llm.input_cost or 0.0
cached_rate = llm.cached_input_cost if llm.cached_input_cost is not None else input_rate
output_rate = llm.output_cost or 0.0
return (non_cached * input_rate + cache_read * cached_rate + output * output_rate) / 1_000_000 return (non_cached * input_rate + cache_read * cached_rate + output * output_rate) / 1_000_000