# Python Profiling Guide for Data Structure Optimization

## cProfile — Function-Level Profiling

### Basic usage

```bash
# Sort by cumulative time (time in function + all callees)
$RUNNER -m cProfile -s cumtime script.py

# Sort by total time (time in function itself only)
$RUNNER -m cProfile -s tottime script.py

# Profile a specific test
$RUNNER -m cProfile -s cumtime -m pytest tests/test_foo.py -k "test_name" -v

# Save for later analysis
$RUNNER -m cProfile -o /tmp/profile.prof script.py
```

### Reading cProfile output

```
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   100000    2.341    0.000    2.341    0.000 utils.py:45(find_customer)
        1    0.523    0.523    3.891    3.891 main.py:12(process_orders)
   100000    0.234    0.000    0.234    0.000 {method 'append' of 'list' objects}
```

| Column | Meaning |
|--------|---------|
| `ncalls` | Number of calls (e.g., `100/50` means 100 total, 50 non-recursive) |
| `tottime` | Time spent in this function only (excludes callees) |
| `cumtime` | Cumulative time including all callees |
| `percall` | Per-call average |
| `filename:lineno(function)` | Location |

### What to look for (data structure issues)

- **High `tottime` on simple functions**: A function doing `if x in some_list` called 100K times — the list should be a set
- **High `ncalls` with small `percall`**: Death by a thousand cuts — many small inefficient operations
- **`{method '__contains__' of 'list' objects}`** or **`{method 'index' of 'list' objects}`**: Direct evidence of O(n) search on list
- **`{method 'remove' of 'list' objects}`**: O(n) removal — consider set.discard or deque
- **Large `ncalls` on `{built-in method builtins.sorted}`**: Repeated sorting — consider heapq or bisect

### Programmatic analysis

```python
import pstats

p = pstats.Stats("/tmp/profile.prof")

# Top 20 by cumulative time
p.sort_stats("cumtime").print_stats(20)

# Top 20 by total time
p.sort_stats("tottime").print_stats(20)

# Only functions matching a pattern
p.sort_stats("tottime").print_stats("mymodule")

# Callers of a specific function
p.print_callers("find_customer")

# What a function calls
p.print_callees("process_orders")
```

---

## timeit — Micro-Benchmarking

### Command line

```bash
# Compare two approaches
$RUNNER -m timeit -s "data = list(range(10000))" "5000 in data"
$RUNNER -m timeit -s "data = set(range(10000))" "5000 in data"

# Multiple statements (use semicolons)
$RUNNER -m timeit -s "from collections import deque; d = deque(range(1000))" "d.appendleft(0); d.pop()"

# Specify iterations
$RUNNER -m timeit -n 100000 -r 5 "sorted([3,1,4,1,5,9])"
```

### In code

```python
import timeit

# Simple comparison
t_list = timeit.timeit("5000 in data", setup="data = list(range(10000))", number=10000)
t_set = timeit.timeit("5000 in data", setup="data = set(range(10000))", number=10000)

print(f"list: {t_list:.4f}s")
print(f"set:  {t_set:.4f}s")
print(f"speedup: {t_list/t_set:.1f}x")
```

### A/B benchmark template

```python
import timeit
import statistics

def benchmark(label, stmt, setup, number=10000, repeats=5):
    times = timeit.repeat(stmt, setup=setup, number=number, repeat=repeats)
    mean = statistics.mean(times)
    stdev = statistics.stdev(times)
    per_call = mean / number * 1e6  # microseconds
    print(f"{label:>20s}: {mean:.4f}s +/- {stdev:.4f}s  ({per_call:.2f} us/call)")
    return mean

setup = """
from collections import deque
data_list = list(range(10000))
data_deque = deque(range(10000))
"""

t_list = benchmark("list.pop(0)", "data_list.pop(0); data_list.append(0)", setup)
t_deque = benchmark("deque.popleft()", "data_deque.popleft(); data_deque.append(0)", setup)
print(f"{'speedup':>20s}: {t_list/t_deque:.1f}x")
```

### Common pitfalls

- **Setup vs statement**: Code in `setup` runs once, code in `stmt` runs `number` times. Put data creation in setup.
- **Mutation side effects**: If your statement mutates data (e.g., `list.pop(0)`), the state changes across iterations. Recreate in setup or account for it.
- **JIT warmup**: First few iterations may be slower. Use `repeat` to get multiple samples.
- **Too few iterations**: For fast operations (<1us), use number=1000000 to get measurable times.

---

## line_profiler — Line-Level Profiling

### Installation

```bash
pip install line_profiler
```

### Usage with decorator

```python
# Add @profile to target functions (no import needed — kernprof injects it)
@profile
def process_orders(orders, customers):
    customer_map = {}
    for c in customers:
        customer_map[c.id] = c

    for order in orders:
        if order.customer_id in customer_map:
            handle(order, customer_map[order.customer_id])
```

```bash
kernprof -l -v script.py
```

### Reading line_profiler output

```
Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    12         1          2.0      2.0      0.0      customer_map = {}
    13      1000       1523.0      1.5      2.1      for c in customers:
    14      1000       1234.0      1.2      1.7          customer_map[c.id] = c
    15
    16     10000      15234.0      1.5     20.9      for order in orders:
    17     10000      45678.0      4.6     62.7          if order.customer_id in customer_map:
    18      9500       9123.0      1.0     12.5              handle(order, customer_map[order.customer_id])
```

| Column | Meaning |
|--------|---------|
| `Hits` | Number of times line was executed |
| `Time` | Total time on this line (microseconds) |
| `Per Hit` | Average time per execution |
| `% Time` | Percentage of function's total time |

### What to look for

- **High `% Time` on a single line**: That's your bottleneck
- **High `Hits` * high `Per Hit`**: Many calls to an expensive operation
- **`in` operator with high time**: Likely O(n) search on list
- **Lines inside loops**: Multiply `Per Hit` by `Hits` to get true impact

### Programmatic usage (no decorator needed)

```python
from line_profiler import LineProfiler

def target_function(data):
    ...

lp = LineProfiler()
lp.add_function(target_function)
lp_wrapper = lp(target_function)
lp_wrapper(test_data)
lp.print_stats()
```

---

## Scaling Tests — Confirming Complexity

The most definitive way to confirm a data structure issue. Run with increasing input size and measure growth rate.

```python
import time

def scaling_test(fn, sizes, setup_fn):
    """Run fn with increasing input sizes and report growth rate."""
    results = []
    for n in sizes:
        data = setup_fn(n)
        start = time.perf_counter()
        fn(data)
        elapsed = time.perf_counter() - start
        results.append((n, elapsed))

    # Print with ratios
    print(f"{'n':>10s} {'time':>10s} {'ratio':>8s} {'expected O(n)':>14s} {'expected O(n^2)':>16s}")
    for i, (n, t) in enumerate(results):
        ratio = f"{t / results[0][1]:.1f}x" if i > 0 else "—"
        exp_n = f"{n / results[0][0]:.1f}x"
        exp_n2 = f"{(n / results[0][0])**2:.1f}x"
        print(f"{n:>10d} {t:>10.4f} {ratio:>8s} {exp_n:>14s} {exp_n2:>16s}")

# Example usage
scaling_test(
    fn=lambda data: process_items(data),
    sizes=[1000, 2000, 4000, 8000, 16000],
    setup_fn=lambda n: generate_items(n),
)
```

### Interpreting results

| Actual ratio | Likely complexity |
|-------------|-------------------|
| ~1x when n doubles | O(1) or O(log n) |
| ~2x when n doubles | O(n) |
| ~4x when n doubles | O(n^2) |
| ~8x when n doubles | O(n^3) |

---

## tracemalloc — Memory Allocation Tracking (stdlib)

No installation needed — part of the standard library. Useful for detecting data structure memory issues.

### Basic usage

```python
import tracemalloc

tracemalloc.start()

# ... code under test ...

snapshot = tracemalloc.take_snapshot()
top = snapshot.statistics('lineno')

print("Top 10 memory allocations:")
for stat in top[:10]:
    print(stat)
```

### Comparing snapshots (find leaks)

```python
tracemalloc.start()

snapshot1 = tracemalloc.take_snapshot()
# ... code that might leak ...
snapshot2 = tracemalloc.take_snapshot()

# Show what grew between snapshots
top = snapshot2.compare_to(snapshot1, 'lineno')
for stat in top[:10]:
    print(stat)
```

### When to use

- Quick memory auditing without installing external tools
- Finding which lines allocate the most memory
- Detecting memory growth between snapshots
- `sys.getsizeof()` for individual object size checks (note: doesn't count referenced objects)

---

## pyinstrument — Statistical Profiler (optional)

```bash
pip install pyinstrument
```

Lower overhead than cProfile, provides more realistic timing by sampling rather than instrumenting every call.

```bash
# CLI usage
pyinstrument script.py
pyinstrument -m pytest tests/test_foo.py -k "test_name"

# HTML output
pyinstrument --html -o /tmp/profile.html script.py
```

```python
# Programmatic
from pyinstrument import Profiler

profiler = Profiler()
profiler.start()
# ... code ...
profiler.stop()
print(profiler.output_text(unicode=True))
```

### When to use over cProfile

- **Production profiling**: Lower overhead, won't distort timing as much
- **Cleaner output**: Focuses on actual bottlenecks, hides fast functions
- **When cProfile shows noise**: Statistical sampling filters out functions called many times but cheaply

### When to use cProfile instead

- **Need exact call counts**: `ncalls` column is useful for detecting "death by thousand cuts"
- **Need callers/callees**: `pstats.Stats.print_callers()` / `print_callees()`
- **No-install environments**: cProfile is stdlib

---

## Workflow: Finding Data Structure Issues

1. **Profile first**: `$RUNNER -m cProfile -s tottime script.py` (or `pyinstrument` for cleaner output)
2. **Identify hot functions**: Top 5 by `tottime`
3. **Check for known antipatterns**: Is the hot function doing list search, repeated sort, string concat?
4. **Run static analysis**: Use Grep to scan for known antipatterns (list.pop(0), deepcopy, membership on lists, etc.)
5. **Confirm with line_profiler**: Zoom into the hot function to find the exact hot line
6. **Micro-benchmark the fix**: `timeit` to verify your alternative is faster
7. **Scaling test**: Confirm the complexity improvement with increasing input sizes
8. **Full benchmark**: Run the actual test suite to measure end-to-end impact
