# Modularity Guide

## Core Concepts

### Coupling vs Cohesion

**Cohesion** measures how strongly related the entities within a single module are. High cohesion means entities in the same file call each other frequently — they belong together.

**Coupling** measures how many dependencies exist between different modules. Low coupling means modules are relatively independent — changing one doesn't ripple across the codebase.

The goal is **high cohesion, low coupling**. In graph terms: dense intra-module edges, sparse inter-module edges.

### Directed Weighted Modularity (Intuitive Version)

spaghettree (the tool this skill is inspired by) computes a formal modularity metric called DWM. The intuition behind it:

1. Take all call relationships in the codebase
2. For each call A→B, check if A and B are in the same module
3. **Actual intra-module calls** minus **expected intra-module calls if entities were randomly assigned to modules** = modularity score
4. Higher is better — it means entities are grouped better than random

You don't need to compute this numerically. Instead, use the **affinity heuristic** which captures the same signal:

- For each entity, count connections to each module
- If an entity has more connections to module X than to its home module, it's contributing negatively to modularity
- Moving it to module X improves the score

### The Affinity Heuristic

For entity `e` and module `m`:

```
affinity(e, m) = outgoing_calls(e → m) + incoming_calls(m → e)
```

Where:
- `outgoing_calls(e → m)` = number of calls from entity `e` to any entity in module `m`
- `incoming_calls(m → e)` = number of calls from any entity in module `m` to entity `e`

**Decision rule**: If `affinity(e, other_module) > affinity(e, home_module)`, entity `e` is a move candidate. The gap `affinity(other) - affinity(home)` is the improvement signal.

### Confidence Levels

| Confidence | Criteria | Action |
|------------|----------|--------|
| **High** | Affinity gap ≥ 3x, total connections ≥ 5 | Strongly recommend move |
| **Medium** | Affinity elsewhere > home, margin < 3x | Recommend with explanation |
| **Low** | Roughly equal affinity | Mention if it would resolve a structural smell |
| **Skip** | Only 1-2 total connections | Not enough signal — leave in place |

## Structural Smells (Detailed)

### God Module

**Detection**: Count inbound import edges per module. If one module is imported by >50% of all other modules, it's a god module.

**Common names**: `utils.py`, `helpers.py`, `common.py`, `misc.py`, `shared.py`

**Why it's bad**: Every change to the god module is high-risk because so many other modules depend on it. It also makes the dependency graph star-shaped, hiding the true logical structure.

**Fix strategy**:
1. Group the god module's entities by their callers. Which entities are called mostly from `pipeline/`? From `api/`? From `models/`?
2. Create new modules organized by domain, not by "type" of code (avoid creating `string_utils.py`, `dict_utils.py` — that's just splitting the god module into smaller god modules)
3. Move each cluster of entities to the module that uses them, or create a new domain-specific module

**Example**: `utils.py` has `normalize_text`, `validate_email`, `parse_date`, `retry_request`. After analysis:
- `normalize_text`, `validate_email` → called from `pipeline/` → move to `pipeline/text.py`
- `parse_date` → called from `models/` → move to `models/dates.py`
- `retry_request` → called from `api/` → move to `api/resilience.py`

### Circular Dependencies

**Detection**: If module A imports from B and B imports from A (directly or through a chain), there's a cycle.

**Why it's bad**: Circular imports can cause `ImportError` at runtime (depending on import order). They also mean the modules can't be understood independently.

**Fix strategies** (in order of preference):
1. **Extract shared entities**: If A and B both need entity X, move X to a new module C that both import from
2. **Merge modules**: If A and B are so intertwined they can't be separated, they might be one logical module
3. **Defer imports**: Move the import inside the function that needs it (`def foo(): from B import bar; bar()`) — this is a last resort

### Feature Envy

**Detection**: A class in module A has methods that primarily access data from module B's classes.

**Example**: `Formatter` in `output.py` takes a `Record` from `models.py` and accesses `record.name`, `record.date`, `record.tags` in every method. The formatter is "envious" of Record's data.

**Fix**: Move `Formatter` to `models.py` (near the data it uses), or create a shared module if the coupling goes both ways.

### Shotgun Surgery

**Detection**: When you trace the files touched by recent feature additions or bug fixes (via git log), the same change pattern touches many unrelated-sounding modules.

**Fix**: Identify which entities are always changed together and co-locate them.

### Dead Code

**Detection**: Entities with zero incoming edges in the call graph (nothing calls them).

**Caveats**: Check for:
- Entry points (`__main__`, CLI commands, API endpoints, test functions)
- Dynamic invocation (`getattr`, plugin systems, serialization/deserialization)
- Template references (Jinja, Django templates)
- Re-exports in `__init__.py`

Only flag as dead code if none of these apply.

## Safe Refactoring Protocol

### Before any move

1. **Grep exhaustively** for all references to the entity:
   ```
   grep -r "entity_name" --include="*.py"
   grep -r "from.*module.*import.*entity_name" --include="*.py"
   grep -r "module\.entity_name" --include="*.py"
   ```

2. **Check `__init__.py` files**: Is the entity re-exported as part of the package's public API?

3. **Check for string references**: Serialization, logging format strings, dynamic imports

4. **Identify the entity's own dependencies**: What imports does the entity need? These must be added to the target file.

### The move

1. Copy entity + decorators + associated constants to target file
2. Add the entity's import dependencies to the target file
3. Update all import sites (found in step 1) to point to the new location
4. Add re-export in old location: `from new_location import entity  # TODO: remove re-export`
5. Run tests
6. If tests pass, commit. If not, diagnose:
   - Missing import in target file?
   - Circular import created?
   - String-based reference not updated?

### After all moves

1. Re-run the modularity assessment
2. Present before/after module connection matrix
3. Clean up re-exports after confirming all references are updated
4. Run full test suite one final time

## Grouping Moves

When multiple entities from the same source module all have high affinity to the same target, move them together. This:
- Reduces the number of import-update rounds
- Often reveals natural "modules" that should have existed from the start
- Makes the PR easier to review (one coherent theme vs scattered changes)

Present grouped moves to the user:

```
Move from utils.py to pipeline/text.py:
  - normalize_text (affinity: 0 home, 8 pipeline)
  - strip_html (affinity: 1 home, 6 pipeline)
  - truncate_string (affinity: 0 home, 4 pipeline)

Move from utils.py to api/validation.py:
  - validate_email (affinity: 0 home, 5 api)
  - validate_phone (affinity: 0 home, 4 api)
```
