# Python C Extensions and FFI Optimization

This guide covers the performance boundary between Python and native C extensions -- the conversions, copies, and call overhead that dominate when profiling shows C extension functions as hotspots. For most C extension hotspots, the bottleneck is NOT in the C code itself. It's in what happens crossing the boundary.

## Where Time Actually Goes

When cProfile shows a C extension function eating time, the cost is usually in one of four places:

1. **Data conversion at the boundary**: Python objects -> C-compatible buffers -> Python objects
2. **Memory allocation**: C extension allocates, returns a Python object, Python GC manages it
3. **Call overhead**: Per-call overhead for ctypes/cffi/pybind11 calls (~1us, but adds up in tight loops)
4. **GIL management**: C extensions that release the GIL enable parallelism; those that don't create serialization points

**Rule of thumb**: If profiling shows a C extension call as a hotspot, look at the Python code surrounding it first. The fix is almost always reducing boundary crossings, eliminating unnecessary copies, or batching small calls into fewer large ones.

## Profiling C Extension Boundaries

Standard Python profilers see C extensions as opaque:

| Tool | What it shows | Limitation |
|------|--------------|------------|
| `cProfile` | Total time in C function | Not WHY it's slow |
| `py-spy --native` | Python + C stacks interleaved | Requires `--native` flag, needs debug symbols for C |
| `perf` (Linux) | Full system-level profile | Low-level, requires root or `perf_event_paranoid=1` |
| `tracemalloc` | Python-side allocations | Doesn't see C-side memory (numpy, PIL internal buffers) |
| `memray` | Python + C allocations | Best for seeing total memory including C extensions |

### Diagnosing boundary overhead

```bash
# Profile with cProfile -- look for C functions with high ncalls
$RUNNER -m cProfile -s tottime script.py 2>&1 | head -40

# If top functions are C extension calls (numpy, PIL, lxml, etc.),
# the question is: are they slow because of the C work, or the boundary?

# Profile with py-spy to see into C code
py-spy record --native -o /tmp/profile.svg -- $RUNNER script.py
```

**Key signal**: If cProfile shows a C function called many times with low per-call time but high total time, the bottleneck is call overhead -- batch the operations. If per-call time is high, the C work itself may be expensive, or the input conversion is costly.

## Common Libraries and Their Boundaries

### numpy

numpy is the most common C extension boundary in Python. Almost every numeric Python project crosses it.

#### Buffer protocol: view vs copy

```python
import numpy as np

# ZERO-COPY: returns a view (shared memory)
arr = np.asarray(existing_array)

# ALWAYS COPIES: creates new array with own memory
arr = np.array(existing_array)
```

`np.asarray(x)` returns a view when possible -- modifying the view modifies the source. `np.array(x)` always copies. Use `asarray` when you only need to read, `array` when you need an independent copy.

**Danger**: Views become invalid if the source is freed or resized. In pipelines where objects are passed between stages, either copy explicitly or document the lifetime dependency.

#### Contiguity

Many C functions require C-contiguous memory layout (`arr.flags['C_CONTIGUOUS']`). If the array isn't contiguous (e.g., after slicing or transposing), the C function may silently copy or raise an error.

```python
# Check contiguity
print(arr.flags['C_CONTIGUOUS'])  # True or False

# Force contiguous -- copies only if needed
arr = np.ascontiguousarray(arr)

# Transpose creates a non-contiguous view
a = np.zeros((100, 200))      # C-contiguous
b = a.T                        # Fortran-contiguous view, NOT C-contiguous
c = np.ascontiguousarray(b)    # copies into C-contiguous layout
```

#### dtype conversion

```python
# ALWAYS COPIES: astype creates a new array
arr32 = arr64.astype(np.float32)

# NO-COPY IF DTYPE MATCHES: returns view when no conversion needed
arr32 = np.asarray(arr, dtype=np.float32)  # view if already float32, copy if not
```

In pipelines with multiple stages, dtype mismatches force copies at each stage. Normalize dtype once at the entry point.

#### BLAS dispatch

`np.dot`, `@` operator, and `np.linalg` functions dispatch to BLAS (OpenBLAS, MKL, etc.). The dispatch overhead is ~1us per call -- negligible for large arrays, significant for thousands of small array operations.

```python
# Check which BLAS backend numpy is using
np.show_config()

# BAD: many small BLAS calls
for row in rows:
    np.dot(row, weights)  # ~1us dispatch overhead per call

# GOOD: single BLAS call
result = np.dot(np.stack(rows), weights)  # one dispatch, BLAS-optimized
```

#### Vectorization vs temporary arrays

```python
# Vectorized -- eliminates Python loop overhead
result = np.where(arr > threshold, arr, 0)

# But vectorization creates temporary arrays:
# This creates 3 temporaries (x**2, y**2, x**2 + y**2)
result = np.sqrt(x**2 + y**2)

# In-place operations avoid temporaries:
result = np.empty_like(x)
np.multiply(x, x, out=result)
temp = np.empty_like(y)
np.multiply(y, y, out=temp)
np.add(result, temp, out=result)
np.sqrt(result, out=result)
```

**Crossover**: For very small arrays (< ~100 elements), Python loops can be faster than numpy because the boundary crossing and temporary allocation dominate. Measure with representative sizes.

### Pillow (PIL)

Pillow's internal storage is a C buffer. Every conversion between PIL and Python/numpy crosses the boundary.

#### Lazy loading

```python
from PIL import Image

# open() is LAZY -- doesn't read pixel data
img = Image.open(path)

# Pixel data is loaded on first access (getpixel, convert, save, etc.)
# Force load explicitly if you need to control when I/O happens:
img.load()
```

#### PIL <-> numpy conversions

```python
# ZERO-COPY VIEW into PIL's internal buffer
arr = np.asarray(img)
# WARNING: if img is closed or GC'd, arr becomes dangling

# SAFE COPY: independent array
arr = np.array(img)

# numpy -> PIL: always copies
img = Image.fromarray(arr)
```

**Common pipeline anti-pattern** (from the unstructured case study):

```python
# BAD: copy chain -- 3 boundary crossings
img = Image.open(path)          # lazy load
arr = np.array(img)             # boundary crossing 1: PIL -> numpy (copy)
processed = some_transform(arr) # may create another copy
img2 = Image.fromarray(processed)  # boundary crossing 2: numpy -> PIL (copy)
img2.save(output_path)          # boundary crossing 3: PIL -> encoder (copy to codec)

# BETTER: stay in one domain as long as possible
img = Image.open(path)
arr = np.asarray(img)           # zero-copy view
# ... do ALL numpy processing ...
# convert to PIL only at the final output point
result = Image.fromarray(final_arr)
result.save(output_path)
```

#### Format overhead

| Format | Encode speed | File size | Notes |
|--------|-------------|-----------|-------|
| BMP | Fastest (no compression) | Largest | Best for pipeline intermediates |
| PNG | Slow (zlib compression) | Small | Don't use between pipeline stages |
| JPEG | Medium (lossy) | Small | Quality loss, not for ML preprocessing |
| TIFF | Variable | Variable | Supports uncompressed mode |

For intermediate pipeline stages (e.g., passing rendered PDF pages to OCR), use the cheapest format. BMP skips compression entirely -- 10-50x faster to encode than PNG with no quality loss.

#### Resampling

```python
# NEAREST: ~10x faster than LANCZOS, but lower quality
img.resize((w, h), Image.NEAREST)

# BILINEAR: good enough for ML preprocessing
img.resize((w, h), Image.BILINEAR)

# LANCZOS: highest quality, slowest
img.resize((w, h), Image.LANCZOS)
```

For ML preprocessing pipelines, `BILINEAR` is almost always sufficient. The model doesn't benefit from `LANCZOS` quality.

#### Mode conversion

```python
# Each conversion allocates a new image
img_rgb = img.convert('RGB')    # new image
img_gray = img_rgb.convert('L') # another new image

# Convert once to the final mode -- don't chain
img_gray = img.convert('L')     # single conversion
```

#### Feature detection

```python
from PIL import features

# Check if libjpeg-turbo is available (2-6x faster JPEG)
features.check('libjpeg_turbo')

# Check available image codecs
features.check('webp')
features.check('zlib')  # PNG compression
```

### pdfium / pypdfium2

PDF rendering is a major C extension boundary in document processing pipelines.

#### Rendering

```python
import pypdfium2 as pdfium

pdf = pdfium.PdfDocument(path)
page = pdf[0]

# render() allocates a bitmap in C, returns as PIL image or numpy array
# Format choice affects downstream conversion cost
bitmap = page.render(scale=2)
pil_image = bitmap.to_pil()     # boundary crossing: C bitmap -> PIL
numpy_arr = np.asarray(bitmap)  # may be zero-copy depending on format
```

**Color format matters**: Rendering to BGRA then converting to RGB downstream is wasteful. Render in the format the downstream consumer needs.

#### Memory: process pages sequentially

```python
# BAD: holds all pages in memory
images = []
for i in range(len(pdf)):
    images.append(pdf[i].render().to_pil())

# GOOD: process and free each page
for i in range(len(pdf)):
    page = pdf[i]
    image = page.render().to_pil()
    process(image)
    image.close()
    # page and bitmap can be GC'd
```

Holding all rendered pages simultaneously is the #1 memory issue in PDF pipelines. A single rendered page at 300 DPI can be 30-50 MB.

#### Text vs rendering

```python
# If you only need text, DON'T render
textpage = page.get_textpage()
text = textpage.get_text_range()

# Rendering is 100-1000x more expensive than text extraction
```

### ONNX Runtime

ML model inference crosses the Python/C boundary at session creation and every inference call.

#### Session reuse

```python
import onnxruntime as ort

# BAD: creates session per call (~100ms-1s per creation)
def predict(input_data):
    session = ort.InferenceSession("model.onnx")
    return session.run(None, {"input": input_data})

# GOOD: create once, reuse
session = ort.InferenceSession("model.onnx")

def predict(input_data):
    return session.run(None, {"input": input_data})
```

Each session holds model weights + execution arena in memory. Creating multiple sessions multiplies memory usage.

#### Input dtype

```python
# BAD: dtype mismatch forces conversion inside ONNX Runtime
arr = np.array(data)  # defaults to float64
result = session.run(None, {"input": arr})  # ONNX expects float32, converts internally

# GOOD: match expected dtype
arr = np.array(data, dtype=np.float32)
result = session.run(None, {"input": arr})
```

Check model input spec: `session.get_inputs()[0].type` to determine expected dtype.

#### Execution providers

```python
# Check available providers
print(ort.get_available_providers())

# CPU provider uses multiple threads internally
session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

# Control thread count
opts = ort.SessionOptions()
opts.intra_op_num_threads = 4
session = ort.InferenceSession("model.onnx", sess_options=opts)
```

### lxml

XML parsing and XPath evaluation cross the Python/C boundary at parse time and on every result extraction.

#### Parse once, query many

```python
from lxml import etree

# BAD: re-parse for each query
for query in queries:
    tree = etree.fromstring(xml_bytes)  # builds C tree each time
    results = tree.xpath(query)

# GOOD: parse once
tree = etree.fromstring(xml_bytes)
for query in queries:
    results = tree.xpath(query)  # queries existing C tree
```

#### iterparse for large XML

```python
# BAD: builds entire tree in memory
tree = etree.parse(large_file)

# GOOD: streams elements, constant memory
for event, elem in etree.iterparse(large_file, events=("end",), tag="record"):
    process(elem)
    elem.clear()           # free element memory
    elem.getparent().remove(elem)  # remove from partial tree
```

#### Batch text extraction

```python
# BAD: each .text access crosses boundary
for elem in tree.xpath("//item"):
    name = elem.find("name").text     # boundary crossing
    value = elem.find("value").text   # boundary crossing
    process(name, value)

# BETTER: extract all text in one XPath call
names = tree.xpath("//item/name/text()")    # returns list of strings
values = tree.xpath("//item/value/text()")  # returns list of strings
for name, value in zip(names, values):
    process(name, value)
```

## GIL Patterns

### Which extensions release the GIL

| Extension | Releases GIL | When |
|-----------|-------------|------|
| numpy | Yes | Compute-heavy ops (matmul, FFT, large array ops). NOT for small operations. |
| Pillow | Yes | Image operations (resize, filter, save, convert) |
| ONNX Runtime | Yes | During model inference |
| pdfium | Yes | During page rendering |
| lxml | Partially | During parsing and XPath evaluation |

### Practical implications

```python
import threading

# GIL-releasing extensions + threads = real parallelism
# Useful when mixing I/O with C compute

def process_page(page):
    image = page.render()     # releases GIL -- other threads can run
    arr = np.asarray(image)
    result = model.run(arr)   # releases GIL
    return result

# These run in parallel because render() and run() release the GIL
threads = [threading.Thread(target=process_page, args=(p,)) for p in pages]
for t in threads:
    t.start()
for t in threads:
    t.join()
```

**Caveat**: Python code between C calls still holds the GIL. For mixed Python + C workloads, the Python portions serialize even if C portions are parallel. If your hot loop is 50% Python and 50% C extension, threading gives at most ~2x speedup, not Nx.

For CPU-bound Python code mixed with C extensions, `multiprocessing` avoids the GIL entirely but adds process spawn + IPC overhead. Worth it for long-running tasks, not for many small ones.

## Optimization Patterns

### 1. Minimize boundary crossings

The single most impactful pattern. Every Python <-> C transition has fixed overhead. Reduce the number of crossings.

```python
# BAD: N boundary crossings
results = [np.sqrt(x) for x in python_list]

# GOOD: 1 boundary crossing
results = np.sqrt(np.array(python_list))
```

```python
# BAD: per-pixel boundary crossing
for y in range(h):
    for x in range(w):
        pixel = img.getpixel((x, y))  # C call per pixel

# GOOD: single boundary crossing
arr = np.asarray(img)  # one C call, then pure numpy
```

### 2. Batch small operations

When you have many small C calls, the per-call overhead dominates the actual computation.

```python
# BAD: overhead per call dominates
for row in rows:
    np.dot(row, weights)  # ~1us overhead + ~0.1us compute

# GOOD: one call, BLAS-optimized
result = np.dot(np.stack(rows), weights)  # ~1us overhead + bulk compute
```

### 3. Avoid unnecessary copies at boundaries

Track where copies happen in your pipeline. Each copy is O(n) in data size.

```python
# BAD: 3 copies through PIL and numpy
img = Image.open(path)
arr = np.array(img)              # copy 1
processed = transform(arr)       # copy 2 (if transform allocates)
img2 = Image.fromarray(processed)  # copy 3

# GOOD: 1 view + 1 copy (minimum for PIL -> process -> PIL)
img = Image.open(path)
arr = np.asarray(img)            # view (zero-copy)
processed = transform(arr)       # unavoidable if transform allocates
img2 = Image.fromarray(processed)  # one copy at the output boundary
```

### 4. Pre-allocate output buffers

Avoid repeated allocation in loops by reusing output buffers.

```python
# BAD: allocates result array each iteration
for i in range(1000):
    result = np.dot(a[i], b)

# GOOD: pre-allocate and write into buffer
result = np.empty(output_shape)
for i in range(1000):
    np.dot(a[i], b, out=result)
```

### 5. Choose the cheapest format for intermediates

In multi-stage pipelines, the format used between stages matters more than the format at the edges.

```python
# BAD: PNG between pipeline stages (compress + decompress at each stage)
img.save("/tmp/stage1.png")  # slow: zlib compression
img2 = Image.open("/tmp/stage1.png")

# GOOD: BMP or raw numpy for intermediates
img.save("/tmp/stage1.bmp")  # fast: no compression
# Or skip the filesystem entirely:
arr = np.asarray(img)  # pass numpy array directly
```

### 6. Stay in one domain

Avoid bouncing between Python objects, numpy arrays, and PIL images. Pick one domain and stay in it as long as possible.

```python
# BAD: PIL -> numpy -> PIL -> numpy -> PIL
img = Image.open(path)
arr = np.array(img)          # crossing 1
img2 = Image.fromarray(arr)  # crossing 2
img3 = img2.resize((w, h))   # could have done this in PIL from the start
arr2 = np.array(img3)        # crossing 3

# GOOD: stay in PIL for PIL operations, numpy for numpy operations
img = Image.open(path)
img = img.resize((w, h))     # stay in PIL
arr = np.asarray(img)        # cross once into numpy for numeric processing
```

## Choosing an Extension API

If you need to call custom C code from Python, the API choice affects per-call overhead:

| API | Per-call overhead | Compilation required | Best for |
|-----|-------------------|---------------------|----------|
| **ctypes** | ~1us | No | Quick integration, prototyping |
| **cffi** | ~0.3us (ABI), ~0.1us (API/inline) | API mode needs compilation | Production wrapping of C libraries |
| **pybind11** | ~0.05us | Yes (C++) | C++ libraries, lowest overhead |
| **Cython** | ~0.05us | Yes (to C) | Wrapping C + writing C-speed Python |

**ctypes** is simplest (no compilation), but its per-call overhead is highest. For a function called millions of times in a loop, the difference between ctypes and pybind11 is 20x per call.

**cffi** in inline/API mode compiles once and offers near-C call speed. Good middle ground.

**pybind11** and **Cython** have the lowest overhead but require a build step. Worth it for hot-path code that crosses the boundary frequently.

## Experiment Loop Integration

When profiling reveals C extension boundaries as hotspots, follow this investigation order:

1. **Profile both sides**: Use `cProfile` for the Python side AND `py-spy --native` to see into C code. This tells you which side is slow.

2. **Check for unnecessary copies**: Search for `np.array()` where `np.asarray()` would work. Search for `Image.fromarray()` / `np.array(img)` pairs that could be eliminated by staying in one domain.

3. **Check for unnecessary conversions**: dtype mismatches (float64 where float32 is expected), image mode conversions (RGB -> RGBA -> RGB), contiguity copies.

4. **Count boundary crossings in the hot path**: Map out every Python <-> C transition. Each one is a potential optimization target.

5. **Benchmark with representative data sizes**: Boundary overhead is fixed (microseconds), so it matters more for small data. A 1us call overhead is 50% of a 2us computation but 0.001% of a 100ms computation.

6. **Verify output equivalence carefully**: View vs copy can cause subtle bugs. If you replace `np.array()` with `np.asarray()`, the result shares memory with the source -- modifying one modifies the other. Test that downstream code doesn't mutate the array.

### Red flags in profiler output

| Signal | Likely cause | Fix |
|--------|-------------|-----|
| C function called 10K+ times, low per-call time | Call overhead dominates | Batch into fewer calls |
| `np.array` / `np.asarray` high in tottime | Repeated array creation | Pre-allocate or reuse buffers |
| PIL `save`/`open` in hot path | Serialization as IPC | Pass in-memory objects instead |
| `astype` high in tottime | Repeated dtype conversion | Normalize dtype at pipeline entry |
| Multiple `Image.fromarray` / `np.array(img)` | Domain bouncing | Stay in one domain |

## Pitfalls

- **Don't optimize the C code**: If profiling shows `np.dot` is slow, the fix is almost never "use a faster BLAS." It's "call `np.dot` fewer times" or "pass a larger array."
- **Views are not free of risk**: Zero-copy views share memory. If the source is freed, resized, or mutated, the view becomes invalid or reflects unexpected changes. Document lifetime dependencies.
- **`np.asarray` on non-array inputs**: If the input isn't already an array, `asarray` still copies. It only avoids copies when the input is already a compatible numpy array.
- **PIL lazy loading surprises**: `Image.open()` doesn't read pixels. If you benchmark image processing, call `img.load()` first or the first access pays the I/O cost.
- **Thread safety with views**: numpy views into PIL images are not thread-safe if PIL operations happen concurrently. Copy the data if sharing across threads.
- **memray vs tracemalloc for C extensions**: `tracemalloc` only sees Python-side allocations. Use `memray` to see memory allocated by C extensions (numpy buffers, PIL image data, ONNX model weights).
