# Python stdlib Containers Reference

## `collections.deque`

Double-ended queue with O(1) append/pop on both ends.

```python
from collections import deque

d = deque()
d = deque([1, 2, 3])
d = deque(maxlen=100)  # bounded — auto-evicts oldest
```

### Operations and Complexity

| Operation | Complexity | Notes |
|-----------|-----------|-------|
| `append(x)` | O(1) | Add to right |
| `appendleft(x)` | O(1) | Add to left |
| `pop()` | O(1) | Remove from right |
| `popleft()` | O(1) | Remove from left |
| `extend(iterable)` | O(k) | k = len(iterable) |
| `extendleft(iterable)` | O(k) | Reverses order of iterable |
| `rotate(n)` | O(k) | k = min(n, len(d)-n) |
| `x in d` | O(n) | Linear scan — no hash index |
| `d[i]` | O(n) | Random access is slow! |
| `len(d)` | O(1) | |
| `remove(x)` | O(n) | Linear scan + shift |
| `reverse()` | O(n) | In-place |
| `copy()` | O(n) | Shallow copy |
| `count(x)` | O(n) | Linear scan |
| `index(x)` | O(n) | Linear scan |
| `insert(i, x)` | O(n) | Linear shift |
| `clear()` | O(n) | Deallocate blocks |

### When to use deque vs list

- **Use deque**: FIFO queues, sliding windows, bounded buffers, BFS
- **Use list**: Random access, slicing, sorting, stack (LIFO)
- **Key trap**: `deque[i]` is O(n) — if you need index access, use list

### maxlen behavior

```python
d = deque(maxlen=3)
d.extend([1, 2, 3])  # deque([1, 2, 3])
d.append(4)           # deque([2, 3, 4]) — 1 was evicted
d.appendleft(0)       # deque([0, 2, 3]) — 4 was evicted
```

---

## `collections.defaultdict`

Dict subclass that calls a factory for missing keys.

```python
from collections import defaultdict

dd = defaultdict(list)       # missing key -> []
dd = defaultdict(int)        # missing key -> 0
dd = defaultdict(set)        # missing key -> set()
dd = defaultdict(lambda: "unknown")
```

### Common patterns

```python
# Group by key
groups = defaultdict(list)
for item in items:
    groups[item.category].append(item)

# Count occurrences (prefer Counter for this)
counts = defaultdict(int)
for word in words:
    counts[word] += 1

# Multi-level nesting
tree = lambda: defaultdict(tree)
taxonomy = tree()
taxonomy["animal"]["mammal"]["dog"] = True

# Set of values per key
index = defaultdict(set)
for doc_id, word in word_occurrences:
    index[word].add(doc_id)
```

### defaultdict vs setdefault

```python
# defaultdict — factory called once per missing key
dd = defaultdict(list)
dd["key"].append(value)  # clean

# setdefault — creates default on every call (wasteful if key exists)
d = {}
d.setdefault("key", []).append(value)  # [] created even if key exists
```

defaultdict is preferred in loops. setdefault is fine for occasional use.

---

## `collections.Counter`

Dict subclass for counting hashable objects.

```python
from collections import Counter

c = Counter("abracadabra")     # Counter({'a': 5, 'b': 2, 'r': 2, 'c': 1, 'd': 1})
c = Counter(words)             # Count iterable elements
c = Counter({"a": 5, "b": 2}) # From dict
```

### Key operations

| Operation | Result |
|-----------|--------|
| `c.most_common(n)` | Top n elements as (elem, count) pairs |
| `c.elements()` | Iterator over elements repeated by count |
| `c.total()` | Sum of all counts (Python 3.10+) |
| `c.subtract(other)` | Subtract counts (in-place) |
| `c.update(other)` | Add counts (in-place) |
| `c + d` | Add counters (positive counts only) |
| `c - d` | Subtract, keeping positive |
| `c & d` | Intersection (min of counts) |
| `c \| d` | Union (max of counts) |

### Counter vs manual counting

```python
# Manual — verbose, error-prone
counts = {}
for word in words:
    counts[word] = counts.get(word, 0) + 1

# Counter — cleaner, supports arithmetic
counts = Counter(words)
top_10 = counts.most_common(10)
```

---

## `heapq` — Heap Queue (Priority Queue)

Min-heap operations on a regular list. The smallest element is always at `heap[0]`.

```python
import heapq

# Create heap from list
data = [3, 1, 4, 1, 5, 9]
heapq.heapify(data)  # O(n) — in-place

# Push/pop
heapq.heappush(heap, item)       # O(log n)
smallest = heapq.heappop(heap)   # O(log n)
smallest = heap[0]                # O(1) — peek without removing

# Push and pop in one operation
old_smallest = heapq.heapreplace(heap, new_item)  # O(log n) — pop then push
new_smallest = heapq.heappushpop(heap, new_item)  # O(log n) — push then pop

# Top-k operations
heapq.nsmallest(k, iterable, key=None)  # O(n log k)
heapq.nlargest(k, iterable, key=None)   # O(n log k)

# Merge sorted iterables
merged = heapq.merge(*sorted_iterables)  # O(n log k) where k = number of iterables
```

### Max-heap trick

heapq only provides min-heap. For max-heap, negate values:

```python
# Max-heap via negation
heapq.heappush(heap, -value)
max_value = -heapq.heappop(heap)
```

### When to use heapq vs sorted

| Scenario | Use | Why |
|----------|-----|-----|
| Need top k from n items | `heapq.nlargest(k, data)` | O(n log k) vs O(n log n) for sorted |
| Streaming: process smallest first | `heappush`/`heappop` | O(log n) per operation |
| Need full sorted order | `sorted()` | O(n log n), simpler code |
| Need kth smallest, k close to n | `sorted()` | nlargest overhead not worth it |

---

## `bisect` — Sorted List Operations

Binary search operations for maintaining sorted lists.

```python
import bisect

sorted_list = [1, 3, 5, 7, 9]

# Find insertion point
i = bisect.bisect_left(sorted_list, 5)    # 2 (before existing 5)
i = bisect.bisect_right(sorted_list, 5)   # 3 (after existing 5)
i = bisect.bisect(sorted_list, 5)         # same as bisect_right

# Insert maintaining sorted order
bisect.insort_left(sorted_list, 4)     # [1, 3, 4, 5, 7, 9]
bisect.insort_right(sorted_list, 4)    # [1, 3, 4, 4, 5, 7, 9]
bisect.insort(sorted_list, 4)          # same as insort_right

# All operations accept key parameter (Python 3.10+):
bisect.bisect_left(data, target, key=lambda x: x.score)
```

### Complexity

| Operation | Search | Insertion (total) |
|-----------|--------|-------------------|
| `bisect_left/right` | O(log n) | N/A |
| `insort_left/right` | O(log n) search + O(n) shift | O(n) |

Note: `insort` is O(n) total because of the list insertion. For truly O(log n) insertion, you need a tree-based structure (not in stdlib).

### Common patterns

```python
# Grade lookup (replacing if/elif chain)
breakpoints = [60, 70, 80, 90]
grades = ['F', 'D', 'C', 'B', 'A']

def grade(score):
    return grades[bisect.bisect(breakpoints, score)]

# Range query on sorted data
def find_in_range(sorted_data, lo, hi):
    left = bisect.bisect_left(sorted_data, lo)
    right = bisect.bisect_right(sorted_data, hi)
    return sorted_data[left:right]
```

---

## `array.array` — Typed Numeric Array

Compact array of C-type values. 3-4x less memory than list for numeric data.

```python
from array import array

# Type codes: 'b' (signed char), 'i' (int), 'l' (long), 'f' (float), 'd' (double)
a = array('d', [1.0, 2.0, 3.0])  # array of C doubles
a = array('i', range(1000))       # array of C ints
```

### When to use array vs list

- **Use array**: Large homogeneous numeric data, memory-constrained, I/O (tofile/fromfile)
- **Use list**: Mixed types, non-numeric, need full Python object behavior
- **Use numpy**: Numeric computation (vectorized ops) — array.array is for storage, not math

### Memory comparison

```python
import sys
from array import array

lst = list(range(10000))
arr = array('l', range(10000))

sys.getsizeof(lst)  # ~87,624 bytes
sys.getsizeof(arr)  # ~81,088 bytes (8 bytes per long vs 8 bytes pointer + 28 bytes per int object)
```

The real savings come from not boxing each integer as a Python object.

---

## `itertools` — Iterator Building Blocks

Key functions for efficient iteration without materializing intermediate lists.

```python
import itertools

# Chain multiple iterables (avoid list concatenation)
itertools.chain(iter1, iter2, iter3)           # O(1) memory
itertools.chain.from_iterable(list_of_lists)   # Flatten

# Group consecutive elements
for key, group in itertools.groupby(sorted_data, key=keyfunc):
    items = list(group)

# Cartesian product (replace nested loops)
for a, b in itertools.product(range(10), range(10)):
    ...

# Combinations and permutations
itertools.combinations(items, r=2)    # C(n,2) pairs
itertools.permutations(items, r=2)    # P(n,2) ordered pairs

# Accumulate (running totals)
list(itertools.accumulate([1, 2, 3, 4]))  # [1, 3, 6, 10]

# islice (slice iterators without materializing)
first_10 = list(itertools.islice(huge_iterator, 10))

# takewhile/dropwhile
itertools.takewhile(lambda x: x < 5, sorted_data)
itertools.dropwhile(lambda x: x < 5, sorted_data)
```

### Key principle

itertools functions return iterators — they don't create intermediate lists. Use them when you only need to iterate once and don't need random access.

---

## `functools` — Caching and Reduction

```python
import functools

# LRU cache — memoize pure functions
@functools.lru_cache(maxsize=128)  # None = unbounded
def expensive(n):
    ...

# Cache (Python 3.9+) — unbounded, simpler
@functools.cache
def compute(key):
    ...

# reduce — fold a sequence
total = functools.reduce(lambda a, b: a + b, numbers)
# Usually prefer sum(), min(), max(), etc. for simple cases
```

### Cache traps

- **`@cache` on instance methods**: Captures `self` as key — instances can never be GC'd. Use `@lru_cache` with `cache_clear()` in `__del__`, or cache on the instance itself.
- **Unhashable arguments**: `@cache`/`@lru_cache` require all arguments to be hashable. Lists, dicts, sets as arguments will raise TypeError.
- **Memory growth**: `@cache` (unbounded) will grow forever. Use `@lru_cache(maxsize=N)` for bounded memory.

---

## `weakref` — References That Don't Prevent GC

```python
import weakref

# WeakValueDictionary — cache that auto-evicts when values are GC'd
cache = weakref.WeakValueDictionary()
cache["key"] = some_object  # entry disappears when some_object has no other refs

# WeakSet — track objects without preventing GC
live_connections = weakref.WeakSet()
live_connections.add(conn)  # auto-removed when conn is GC'd

# finalize — cleanup callback (modern replacement for __del__)
weakref.finalize(obj, cleanup_function, arg1, arg2)
```

### When to use

- **Object caches** — `WeakValueDictionary` for caches where you want entries evicted when nothing else references the value
- **Observer/listener registries** — `WeakSet` to track listeners without preventing them from being GC'd
- **Resource cleanup** — `finalize()` is more reliable than `__del__` (guaranteed call order, no resurrection issues)

### Caveats

- Not all objects can be weakly referenced. Built-in types like `int`, `str`, `tuple`, `list`, `dict` cannot. Custom classes can by default.
- `WeakValueDictionary` entries may disappear between a `key in cache` check and the subsequent `cache[key]` access. Use `.get()`.

---

## `sys.intern()` — String Deduplication

```python
import sys

# Force string interning — all equal strings share one object
key = sys.intern(input_string)
```

CPython automatically interns identifier-like strings (attribute names, variable names). `sys.intern()` extends this to arbitrary strings.

### When to use

- **Parsed data with many repeated string values** — CSV column names, JSON keys, log levels
- **Dict keys from user input** — interned keys use pointer comparison for dict lookup
- **Large datasets** — if the same 100 strings appear millions of times, interning saves significant memory

### Caveats

- You must keep a reference to the return value: `s = sys.intern(s)`, not just `sys.intern(s)`
- Only useful when the same strings genuinely recur many times
- Adds overhead for strings that are unique — only intern known-repeated values

---

## `operator` — C-Speed Callables

```python
from operator import itemgetter, attrgetter, methodcaller

# itemgetter — faster than lambda for item access
sorted(pairs, key=itemgetter(1))          # vs lambda x: x[1]
sorted(rows, key=itemgetter(2, 0))        # multi-key sort

# attrgetter — faster than lambda for attribute access
sorted(objects, key=attrgetter("score"))   # vs lambda x: x.score
sorted(objects, key=attrgetter("a.b"))     # chained attribute access

# methodcaller — faster than lambda for method calls
list(map(methodcaller("strip"), lines))    # vs [l.strip() for l in lines]
```

### Performance note (version-dependent)

These are C-implemented callables that avoid Python frame creation. The speed advantage over lambdas:
- **Python 3.10 and below**: ~30-50% faster
- **Python 3.11+**: ~15-30% faster (PEP 659 specializes lambda calls too, narrowing the gap)

Still a net win on all versions, but the gap is smaller on modern CPython. Prefer for hot paths; use lambdas for readability in cold code.
