# Worker Pools and Process Management

This guide covers CPU-aware worker pool sizing, process management, and concurrency infrastructure. The most common mistake is over-provisioning workers in containers — using `os.cpu_count()` (host CPUs) instead of `os.sched_getaffinity(0)` (available CPUs). In the unstructured/core-product case study, fixing this one issue produced a 5.9% latency improvement and 60% memory reduction (2.1 GB savings) on single-CPU pods.

## CPU Detection: The Container Problem

`os.cpu_count()` and `multiprocessing.cpu_count()` report the number of logical CPUs on the **host machine**. In a container with cgroup CPU limits, this is wrong:

```python
# BAD: reports host CPUs, not what's available to the container
os.cpu_count()  # Returns 64 on a machine where your pod has 1 CPU

# BAD: multiprocessing also reports host CPUs
multiprocessing.cpu_count()  # Same: 64

# GOOD: respects cgroup CPU limits AND taskset
os.sched_getaffinity(0)  # Returns {0} on a 1-CPU pod
len(os.sched_getaffinity(0))  # -> 1
```

`os.sched_getaffinity(0)` returns the set of CPUs the current process is allowed to run on. It respects:
- **cgroup limits** (Kubernetes CPU requests/limits)
- **taskset** / cpuset bindings
- **CPU affinity** set by the scheduler

On macOS, `sched_getaffinity` doesn't exist — fall back to `os.cpu_count()`:

```python
try:
    available_cpus = len(os.sched_getaffinity(0))
except AttributeError:
    available_cpus = os.cpu_count() or 1
```

## Worker Pool Sizing

### When to skip the pool entirely

If only 1 CPU is available, a worker pool adds overhead with zero parallelism:
- **Process creation**: ~50-200ms per worker (fork/spawn)
- **Memory duplication**: each worker gets its own copy of loaded models, caches, etc.
- **IPC overhead**: data serialization between main process and workers

```python
cpus = len(os.sched_getaffinity(0))
if cpus <= 1:
    # Serial execution — no pool overhead
    results = [process(item) for item in items]
else:
    with ProcessPoolExecutor(max_workers=cpus) as pool:
        results = list(pool.map(process, items))
```

This is exactly what the unstructured case study did for OCR workers. On a single-CPU Kubernetes pod, the code was spawning 64 workers (host CPU count), each loading 4 ONNX models. Switching to `sched_getaffinity` and falling back to serial execution eliminated the pool entirely on 1-CPU pods.

### Sizing heuristics

| Workload type | Pool size | Why |
|---|---|---|
| CPU-bound, no I/O | `len(sched_getaffinity(0))` | One worker per available core, fully utilized |
| CPU-bound with occasional I/O | `len(sched_getaffinity(0))` | I/O is short, don't over-provision |
| I/O-bound (network, disk) | `len(sched_getaffinity(0)) * 2-4` or use threads | More workers overlap I/O waits |
| Mixed CPU + I/O | `len(sched_getaffinity(0))` for CPU, threads for I/O | Separate concerns |
| ML inference (GPU) | 1 worker per GPU, CPU workers for pre/post | GPU is the bottleneck, CPU workers feed it |

### The model duplication problem

When workers load ML models (ONNX, PyTorch, TensorFlow), each worker gets its own copy:

```
Main process: 500 MB (model loaded)
+ 4 workers x 500 MB = 2.5 GB total

With CPU-aware sizing (1 CPU pod):
Main process: 500 MB (model loaded)
No workers = 500 MB total -> 2 GB savings
```

This is what happened in the unstructured case study: 4 ONNX models duplicated across workers on a single-CPU pod. The 2.1 GB memory reduction came from eliminating worker processes that each loaded their own copies of the models.

## fork vs spawn vs forkserver

### fork (default on Linux)

- **Fast**: ~50ms, copy-on-write memory sharing
- **Dangerous**: inherits ALL parent state (file descriptors, locks, threads). If parent has threads running, child can deadlock.
- **Memory**: Initially shares parent's memory via COW. As workers modify data, pages get copied.
- **Gotcha**: CUDA, some C extensions (OpenSSL), and threaded libraries break with fork. Use spawn instead.

### spawn (default on macOS, Windows)

- **Slow**: ~200-500ms, starts fresh Python interpreter
- **Safe**: clean process, no inherited state issues
- **Memory**: No sharing — each worker loads everything from scratch (imports, models, data)
- **Gotcha**: Everything sent to workers must be picklable. Module-level code runs in every worker.

### forkserver

- **Middle ground**: First call forks a clean server process, subsequent workers fork from that server
- **Faster than spawn**: Server process has imports loaded, workers inherit them via COW
- **Safer than fork**: Server process has no threads/locks
- **Best for**: Long-running applications that create worker pools multiple times

```python
import multiprocessing as mp
mp.set_start_method('forkserver')  # Call once at program start
```

## Memory Sharing Between Workers

### SharedMemory (Python 3.8+)

For large data (numpy arrays, model weights), avoid serializing through IPC:

```python
from multiprocessing.shared_memory import SharedMemory
import numpy as np

# Create shared memory block
shm = SharedMemory(create=True, size=arr.nbytes)
shared_arr = np.ndarray(arr.shape, dtype=arr.dtype, buffer=shm.buf)
np.copyto(shared_arr, arr)  # Copy data once

# Workers access the same memory (no copy)
def worker_fn(shm_name, shape, dtype):
    existing_shm = SharedMemory(name=shm_name)
    arr = np.ndarray(shape, dtype=dtype, buffer=existing_shm.buf)
    # Read arr -- zero copy
    existing_shm.close()
```

### Manager vs shared state

| Mechanism | Speed | Use case |
|---|---|---|
| `multiprocessing.Manager()` | Slow (IPC per access) | Coordination: shared dicts, lists across processes |
| `multiprocessing.Value`/`Array` | Fast | Simple scalars and fixed-size arrays |
| `SharedMemory` | Fastest | Large data (numpy arrays, buffers), requires manual lifecycle |

### Memory-mapped files

For read-only shared data, mmap avoids both copying and SharedMemory lifecycle management:

```python
import mmap
import numpy as np

# Write array to file
arr.tofile('shared_data.bin')

# Workers mmap the file -- OS handles caching
with open('shared_data.bin', 'r+b') as f:
    mm = mmap.mmap(f.fileno(), 0)
    shared = np.frombuffer(mm, dtype=arr.dtype).reshape(arr.shape)
```

## Thread Pools vs Process Pools

### When to use threads (concurrent.futures.ThreadPoolExecutor)

- I/O-bound work (HTTP requests, file I/O, database queries)
- C extensions that release the GIL (numpy compute, Pillow operations)
- Low memory overhead (threads share process memory)
- No serialization cost (objects shared directly)

### When to use processes (concurrent.futures.ProcessPoolExecutor)

- CPU-bound Python code (GIL prevents thread parallelism)
- Isolation needed (crash in worker doesn't kill main process)
- Untrusted code execution
- Accept the memory + IPC overhead

### Hybrid pattern

```python
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor

# CPU-heavy: process pool
with ProcessPoolExecutor(max_workers=cpu_count) as cpu_pool:
    computed = list(cpu_pool.map(cpu_heavy_fn, items))

# I/O-heavy: thread pool
with ThreadPoolExecutor(max_workers=cpu_count * 4) as io_pool:
    fetched = list(io_pool.map(fetch_fn, urls))
```

## Pool Lifecycle Patterns

### Reuse pools across requests

```python
# BAD: create pool per request (~200ms startup each time)
def handle_request(data):
    with ProcessPoolExecutor() as pool:
        return list(pool.map(process, data))

# GOOD: long-lived pool
_pool = ProcessPoolExecutor(max_workers=cpu_count)

def handle_request(data):
    return list(_pool.map(process, data))

# Clean up on shutdown
import atexit
atexit.register(_pool.shutdown)
```

### Warm-up

Workers do lazy imports — the first task per worker is slow. Force initialization:

```python
pool = ProcessPoolExecutor(max_workers=n)
futures = [pool.submit(lambda: None) for _ in range(n)]
for f in futures:
    f.result()  # Wait for all workers to initialize
```

### Graceful shutdown

```python
pool.shutdown(wait=True)   # Wait for in-flight tasks to complete
pool.shutdown(wait=False)  # Cancel pending, don't wait
pool.shutdown(wait=True, cancel_futures=True)  # Python 3.9+: cancel queued + wait for running
```

## Antipatterns

### Over-provisioning in containers

Covered above. Use `sched_getaffinity`, not `cpu_count`. Check `os.cpu_count()` vs `len(os.sched_getaffinity(0))` — if they differ, any pool sized from `cpu_count()` is wrong.

### Pool for single items

```python
# BAD: pool overhead for 1 item
with ProcessPoolExecutor() as pool:
    result = pool.submit(process, single_item).result()

# GOOD: just call it directly
result = process(single_item)
```

### Large objects through IPC

```python
# BAD: pickle 1GB array to send to worker
pool.submit(process, huge_array)  # Serializes entire array

# GOOD: use SharedMemory or mmap
shm = SharedMemory(create=True, size=huge_array.nbytes)
# ... copy to shared memory, pass name to worker
```

### Not handling worker crashes

```python
# BAD: worker crash silently kills the future
future = pool.submit(process, data)
result = future.result()  # Raises BrokenProcessPool if worker crashed

# GOOD: retry with backoff
def robust_submit(pool, fn, *args, max_retries=3):
    for attempt in range(max_retries):
        try:
            return pool.submit(fn, *args).result()
        except Exception as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(0.1 * (2 ** attempt))
```

## Experiment Loop Integration

When profiling shows worker pool overhead:

1. **Check CPU detection**: Compare `os.cpu_count()` vs `len(os.sched_getaffinity(0))`. If they differ, the pool is oversized.
2. **Measure process creation time** separately from work time. If creation dominates, the pool is being recreated too often or the work is too small.
3. **Check RSS of workers vs main process**. If workers are nearly as large as main, look for model/data duplication.
4. **Test serial execution as a baseline**. Sometimes it's faster than pool + IPC, especially for small batches or single-CPU environments.
5. **For the deep agent**: Worker pool issues show up as both memory AND CPU hotspots — memory from duplication, CPU from serialization/IPC overhead.
