# I/O & Serialization Optimization for Python

This guide covers data format choices, serialization overhead, and I/O pipeline design -- the class of optimizations that are not purely CPU, memory, or async. These patterns matter whenever profiling reveals time spent in encoding, decoding, format conversion, or disk/buffer I/O rather than in computation itself.

**Real-world impact**: In the unstructured/core-product case study, switching PDF page rendering from PNG to BMP -- eliminating compression that was immediately decompressed downstream -- produced a 14.9% end-to-end latency improvement on a 10-page scan workload. No algorithm changed. No parallelism added. Just removing wasted I/O work.

## The Decision Framework

Before changing any I/O or serialization path, answer three questions:

1. **Where does this data go next?** If the next consumer immediately decodes/decompresses, you are doing round-trip work for nothing.
2. **Is this a system boundary or an internal pipeline stage?** Compress and validate at boundaries (network, storage, user-facing API). Use the cheapest format internally.
3. **Is this on the hot path?** A single `json.dumps` call at startup does not matter. The same call inside a per-page loop processing thousands of documents does.

If you cannot answer #3 with profiler data, profile first. Never optimize cold I/O.

## File Format Selection

### Image formats

| Format | Encode cost | Decode cost | Size | Use when |
|--------|-------------|-------------|------|----------|
| PNG | High (~90ms/page) | Moderate (~30ms/page) | Compressed | Storing images, sending over network |
| BMP | Negligible (~1ms) | Negligible | Uncompressed (large) | Internal pipeline stages, temporary buffers |
| JPEG | Moderate | Moderate | Compressed (lossy) | Web delivery, storage where quality loss is acceptable |
| Raw bytes (numpy) | Zero | Zero | Uncompressed | Staying in numpy for processing, avoiding PIL entirely |

**Key insight**: Compression is work. If the compressed output is consumed by code that immediately decompresses it, you are paying encode + decode cost for zero benefit. This is the most common I/O antipattern in document processing pipelines.

```python
# BAD: compress to PNG just to decompress 2 lines later
buf = io.BytesIO()
img.save(buf, format='PNG')   # ~90ms compression
buf.seek(0)
img2 = Image.open(buf)        # ~30ms decompression -- total waste: ~120ms

# GOOD: use BMP for internal pipeline stages
buf = io.BytesIO()
img.save(buf, format='BMP')   # ~1ms, no compression
buf.seek(0)
img2 = Image.open(buf)        # ~1ms

# BEST: stay in numpy if both sides are numpy-aware
np_arr = np.asarray(img)      # zero-copy view
# ... process in numpy ...
# only convert back to PIL at the boundary where PIL is required
```

### Document processing formats

- **PDF rendering**: Choose render format based on downstream consumer. If feeding into OCR (tesseract, PaddleOCR), render to BMP or raw pixels -- OCR engines decompress internally anyway.
- **Page-by-page streaming vs full-document load**: Process pages one at a time when possible. Loading all pages into memory simultaneously multiplies peak memory by page count. Use generators or iterators to yield pages.

### Tabular data interchange

| Format | Read speed | Write speed | Size | Python support | Best for |
|--------|-----------|-------------|------|---------------|----------|
| CSV | Slow (text parsing) | Slow (text formatting) | Large (text) | stdlib `csv` | Interchange with non-Python tools, human readability |
| Parquet | Fast (columnar, compressed) | Moderate | Small | `pyarrow`, `fastparquet` | Analytics, column-oriented queries |
| Arrow IPC | Very fast (zero-copy capable) | Very fast | Medium | `pyarrow` | In-process / IPC data sharing |
| SQLite | Fast (indexed queries) | Moderate | Medium | stdlib `sqlite3` | Queryable local storage |

**General principle**: Match format to use case. Internal pipeline stages should use the cheapest format. Only compress, validate, or pretty-print at system boundaries.

## Serialization Overhead

### JSON

`json.dumps`/`json.loads` from the stdlib is pure Python for encoding and C-accelerated for decoding. For hot paths, drop-in replacements provide significant speedups:

| Library | Speed vs stdlib | Notes |
|---------|----------------|-------|
| `orjson` | 3-10x faster | Returns `bytes`, handles datetime/numpy natively, strict output |
| `ujson` | 2-5x faster | Returns `str`, less strict, good compatibility |
| `msgspec.json` | 3-10x faster | Schema-aware, integrates with `msgspec.Struct` |

```python
# stdlib
payload = json.dumps(data).encode('utf-8')

# orjson -- drop-in for most cases, returns bytes directly
payload = orjson.dumps(data)

# orjson with options (datetime as ISO, numpy array support)
payload = orjson.dumps(data, option=orjson.OPT_NAIVE_UTC | orjson.OPT_SERIALIZE_NUMPY)
```

**Datetime/decimal overhead**: `json.dumps` cannot handle `datetime` or `Decimal` without a custom `default=` function that is called per-object. `orjson` handles these natively in C. If your serialized data contains many datetimes, this alone can be a 5x difference.

### pickle

Fast for Python objects, but version-fragile and insecure for untrusted data.

```python
import pickle

# Use protocol 5 (PEP 574) for large bytes/bytearray -- enables out-of-band buffers
data = pickle.dumps(large_object, protocol=5)

# For numpy arrays, pickle protocol 5 avoids copying the buffer
buffers = []
data = pickle.dumps(arr, protocol=5, buffer_callback=buffers.append)
# buffers[0] is a zero-copy view of arr's data
```

**When to use**: IPC between trusted Python processes, caching computed results locally, multiprocessing data transfer.
**When not to use**: Anything that crosses a trust boundary, long-term storage (pickle format is tied to class definitions), cross-language communication.

### MessagePack

Compact binary format, faster than JSON, good for IPC and network protocols:

```python
import msgpack

# Encode -- ~2-3x faster than json.dumps, ~30% smaller output
packed = msgpack.packb(data, use_bin_type=True)

# Decode
data = msgpack.unpackb(packed, raw=False)
```

### struct.pack for fixed-format binary data

When the format is known at compile time, `struct.pack` with `memoryview` achieves zero-copy:

```python
import struct

# Pack fixed-format records
header = struct.pack('!HHI', version, flags, length)

# Unpack from a buffer without copying
mv = memoryview(raw_bytes)
version, flags, length = struct.unpack_from('!HHI', mv, offset=0)
```

### Pydantic model serialization

Pydantic v2 model serialization has hidden overhead. The optimal path depends on the target format:

```python
from pydantic import BaseModel
import orjson

class Record(BaseModel):
    id: int
    name: str
    tags: list[str]

record = Record(id=1, name="test", tags=["a", "b"])

# SLOW: .model_dump() creates a dict, then json.dumps serializes it
payload = json.dumps(record.model_dump())

# FASTER: .model_dump_json() uses Rust-based serializer internally
payload = record.model_dump_json()

# ALSO FAST: orjson with model_dump (mode='python' returns Python types)
payload = orjson.dumps(record.model_dump(mode='python'))

# AVOID: mode='json' converts everything to JSON-compatible types first
# then orjson re-serializes -- double work
payload = orjson.dumps(record.model_dump(mode='json'))  # wasteful
```

**For bulk serialization** (list of models), serialize the list once rather than each model individually:

```python
# BAD: serialize each model separately
payloads = [record.model_dump_json() for record in records]

# GOOD: serialize the whole list at once
payload = orjson.dumps([r.model_dump(mode='python') for r in records])
```

## Buffer Protocol & Zero-Copy

### memoryview -- avoid copies when slicing

`memoryview` wraps objects that support the buffer protocol (`bytes`, `bytearray`, `numpy.ndarray`) and allows slicing without copying:

```python
data = b'\x00' * 10_000_000  # 10 MB

# BAD: slicing bytes creates a copy
chunk = data[1000:2000]  # copies 1000 bytes

# GOOD: memoryview slice is a view -- no copy
mv = memoryview(data)
chunk = mv[1000:2000]  # zero-copy view into same buffer
```

This matters when processing large buffers in chunks (file I/O, network packets, image data).

### numpy buffer protocol

```python
import numpy as np

# COPY: np.array() always copies
arr = np.array(existing_buffer)

# VIEW: np.frombuffer() creates a view -- no copy, but buffer must stay alive
arr = np.frombuffer(raw_bytes, dtype=np.uint8)

# VIEW: np.asarray() returns a view if possible, copies only if necessary
arr = np.asarray(existing_array)
```

### PIL/numpy interop -- views vs copies

```python
from PIL import Image
import numpy as np

img = Image.open("photo.png")

# VIEW (when possible): np.asarray shares memory with PIL
arr = np.asarray(img)  # view -- changes to arr may affect img

# COPY: np.array always copies
arr = np.array(img)    # independent copy

# DANGER: if PIL closes or modifies the image, the view becomes invalid
img.close()
print(arr[0, 0])  # undefined behavior -- the underlying buffer is gone
```

**Lifecycle rule**: If you use `np.asarray(img)`, the PIL image must stay alive and unmodified for the lifetime of the numpy array. If you cannot guarantee this, use `np.array(img)` (copy).

### Avoiding format round-trips

Every PIL-numpy-PIL conversion is expensive. Stay in one format as long as possible:

```python
# BAD: 3 copies, 2 format conversions
pil_img = Image.open(path)
np_arr = np.array(pil_img)        # copy 1: PIL -> numpy
processed = some_fn(np_arr)        # copy 2: if some_fn returns new array
result = Image.fromarray(processed)  # copy 3: numpy -> PIL

# GOOD: minimize conversions, stay in numpy until the boundary
np_arr = np.asarray(Image.open(path))  # view if possible
processed = some_fn(np_arr)             # process in numpy
# only convert back to PIL at the point where PIL is actually needed
```

**Same principle for bytes**: `bytes -> PIL -> bytes` round-trips are common in document processing. If the downstream consumer accepts bytes directly, skip the PIL step entirely.

## Streaming & Chunked Processing

### File I/O strategies

| Method | Memory | Speed | Random access | Use when |
|--------|--------|-------|---------------|----------|
| `f.read()` | Loads entire file | Fast for small files | No | File fits comfortably in memory |
| `f.read(chunk_size)` | Fixed buffer | Steady throughput | No | Large files, streaming processing |
| `mmap.mmap()` | OS-managed pages | Fast random access | Yes | Large files with random access patterns |

```python
import mmap

# mmap: OS manages which pages are in memory -- good for random access
with open(path, 'rb') as f:
    mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    # mm supports slicing, find(), regex -- all zero-copy
    header = mm[:64]  # no copy, reads directly from OS page cache
    mm.close()
```

### Generator pipelines

Process-as-you-go instead of accumulate-then-process. This caps memory at O(1) per pipeline stage instead of O(n):

```python
# BAD: accumulates all pages in memory before processing
pages = [render_page(i) for i in range(num_pages)]  # peak: all pages in memory
results = [ocr(page) for page in pages]

# GOOD: generator pipeline -- only one page in memory at a time
def render_pages(num_pages):
    for i in range(num_pages):
        yield render_page(i)

def ocr_pages(pages):
    for page in pages:
        yield ocr(page)

results = list(ocr_pages(render_pages(num_pages)))
# peak memory: 1 page + 1 OCR result
```

**Tradeoff**: Generator pipelines reduce peak memory but can reduce throughput if the pipeline stages could be parallelized. For CPU-bound stages, consider `concurrent.futures` with bounded queues instead.

### io.BytesIO / io.StringIO reuse

```python
import io

# BAD: create a new BytesIO per iteration
for img in images:
    buf = io.BytesIO()
    img.save(buf, format='BMP')
    process(buf.getvalue())

# GOOD: reuse the buffer
buf = io.BytesIO()
for img in images:
    buf.seek(0)
    buf.truncate()
    img.save(buf, format='BMP')
    process(buf.getvalue())
```

The savings are from avoiding repeated allocation/deallocation of the internal buffer. Meaningful when processing thousands of items in a loop.

### tempfile overhead

`NamedTemporaryFile` creates a real file on disk with filesystem I/O. Avoid it when a buffer works:

```python
import tempfile

# SLOW: writes to disk, reads back from disk
with tempfile.NamedTemporaryFile(suffix='.png') as f:
    img.save(f.name)
    result = process_file(f.name)

# FAST: pass bytes directly if the API supports it
buf = io.BytesIO()
img.save(buf, format='PNG')
result = process_bytes(buf.getvalue())
```

**When tempfile is unavoidable**: Some tools (e.g., tesseract CLI) require a file path. In those cases, use `tempfile.NamedTemporaryFile` but consider whether the format matters -- writing BMP instead of PNG to the temp file still saves compression time even if disk I/O is unavoidable.

## Common I/O Antipatterns

### 1. Unnecessary compression in internal pipelines

The most impactful antipattern. Compression exists to reduce size for storage or transfer. Inside a pipeline where the next stage immediately decompresses, it is pure waste.

**Detection**: Profile shows significant time in `zlib.compress`, `zlib.decompress`, PIL `PngImagePlugin`, or similar encode/decode functions. The encoded data is consumed within the same process.

**Fix**: Use an uncompressed format (BMP, raw bytes, Arrow) for internal stages. Only compress at system boundaries.

### 2. Repeated serialization of the same object

```python
# BAD: serialize the same object N times
for consumer in consumers:
    consumer.send(json.dumps(data))

# GOOD: serialize once, send bytes
payload = json.dumps(data).encode()
for consumer in consumers:
    consumer.send(payload)
```

**Detection**: Profile shows `json.dumps` or `pickle.dumps` called with high `ncalls` for data that does not change between calls.

### 3. File path round-trips when buffers work

```python
# BAD: write to temp file, read it back -- unnecessary disk I/O
with tempfile.NamedTemporaryFile(suffix='.png') as f:
    img.save(f.name)
    result = process_file(f.name)

# GOOD: pass bytes directly
buf = io.BytesIO()
img.save(buf, format='PNG')
result = process_bytes(buf.getvalue())
```

**Detection**: `tempfile.NamedTemporaryFile` appears on the hot path. The temp file is written and read within the same function or tight call chain.

### 4. Copy chains in image/array processing

```python
# BAD: 3 copies, 2 format conversions
pil_img = Image.open(path)
np_arr = np.array(pil_img)        # copy 1
processed = transform(np_arr)      # copy 2 (if transform returns new array)
result = Image.fromarray(processed)  # copy 3

# GOOD: minimize conversions
np_arr = np.asarray(Image.open(path))  # view
processed = transform(np_arr)
# convert back to PIL only at the final boundary
```

**Detection**: Multiple `np.array()` / `Image.fromarray()` calls in the same function or call chain. `tracemalloc` shows high allocation in these conversion functions.

### 5. Decoding then re-encoding unchanged data

```python
# BAD: decode JSON just to re-encode it
data = json.loads(raw_json)  # parse
# ... no modifications to data ...
output = json.dumps(data)    # re-serialize

# GOOD: pass through the raw bytes
output = raw_json
```

**Detection**: `json.loads` and `json.dumps` paired in a function where the data is not modified between them. Same applies to `pickle.loads`/`pickle.dumps`, `base64.b64decode`/`base64.b64encode`, etc.

## Experiment Loop Integration

When the optimizer detects I/O or serialization changes:

1. **Benchmark with representative data sizes**: Unit test fixtures (1 page, 10 records) will not surface I/O overhead. Use production-representative data (100+ pages, thousands of records).

2. **Measure both latency AND memory**: Format changes often trade one for the other. BMP is larger in memory than PNG -- this is fine for pipeline stages but could blow up memory on large batches. Verify peak memory does not regress.

3. **Verify output equivalence**: Format changes can affect quality or precision:
   - Lossy formats (JPEG) alter pixel values -- verify downstream consumers tolerate the difference
   - Float serialization precision varies between JSON libraries (`orjson` and `ujson` may round differently)
   - Binary format byte ordering (endianness) matters for cross-platform data

4. **Test the full pipeline, not just the changed stage**: I/O changes can have cascading effects. A format change in stage 2 might break assumptions in stage 5.

5. **Profile before and after**: Use `cProfile` sorted by `tottime` to confirm the target function actually dropped. A format change that speeds up encoding but triggers slower decoding downstream is a net loss.

```bash
# Before: profile the hot path
$RUNNER -m cProfile -s tottime pipeline.py 2>&1 | head -30

# After: same profile, verify the encode/decode functions dropped
$RUNNER -m cProfile -s tottime pipeline.py 2>&1 | head -30

# Memory check with tracemalloc
$RUNNER -c "
import tracemalloc
tracemalloc.start()
# ... run pipeline ...
snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics('lineno')[:10]:
    print(stat)
"
```
