# Database Query Optimization for Java

This guide covers JPA/Hibernate performance patterns, connection pooling, query optimization, caching, batch operations, and EXPLAIN plan verification. For general database verification tiers (EXPLAIN comparison, result diffing, generated integration tests), see the shared database reference patterns.

## JPA/Hibernate N+1 Problem

The N+1 problem is the most common JPA performance issue. Loading a parent entity and then accessing its lazy-loaded children triggers N additional queries -- one per parent row.

### Detection

```java
// BAD: N+1 -- one query per order to load its items
List<Order> orders = entityManager.createQuery(
    "SELECT o FROM Order o WHERE o.status = :status", Order.class)
    .setParameter("status", "ACTIVE")
    .getResultList();

for (Order order : orders) {
    order.getItems().size();  // triggers SELECT * FROM order_item WHERE order_id = ?
    // One query per order -- if 100 orders, 101 queries total
}
```

**Detection signals**:
- Hibernate SQL logging (`spring.jpa.show-sql=true` or `hibernate.show_sql=true`) shows repeated queries with different parameter values
- `hibernate.generate_statistics=true` shows high `prepareStatement` count
- P6Spy or datasource-proxy logs show query count per request

### Fix 1: JOIN FETCH (JPQL)

```java
// GOOD: single query with JOIN
List<Order> orders = entityManager.createQuery(
    "SELECT DISTINCT o FROM Order o JOIN FETCH o.items WHERE o.status = :status", Order.class)
    .setParameter("status", "ACTIVE")
    .getResultList();

// All items already loaded -- no additional queries
for (Order order : orders) {
    order.getItems().size();  // no query -- already fetched
}
```

**Warning**: JOIN FETCH with multiple collections causes a cartesian product. Hibernate limits to one `JOIN FETCH` collection per query (or use `@Fetch(FetchMode.SUBSELECT)` for the second collection).

### Fix 2: @EntityGraph

```java
@Entity
public class Order {
    @OneToMany(mappedBy = "order", fetch = FetchType.LAZY)
    private List<OrderItem> items;
}

// Define the entity graph
@EntityGraph(attributePaths = {"items", "items.product"})
List<Order> findByStatus(String status);

// Or programmatically
EntityGraph<Order> graph = entityManager.createEntityGraph(Order.class);
graph.addAttributeNodes("items");
Subgraph<OrderItem> itemGraph = graph.addSubgraph("items");
itemGraph.addAttributeNodes("product");

List<Order> orders = entityManager.createQuery("SELECT o FROM Order o", Order.class)
    .setHint("javax.persistence.fetchgraph", graph)
    .getResultList();
```

### Fix 3: @BatchSize

```java
@Entity
public class Order {
    @OneToMany(mappedBy = "order", fetch = FetchType.LAZY)
    @BatchSize(size = 25)  // loads items in batches of 25 orders
    private List<OrderItem> items;
}
```

Instead of N queries, Hibernate issues `ceil(N/25)` queries using `WHERE order_id IN (?, ?, ..., ?)`.

### Global batch size

```properties
# application.properties (Spring Boot)
spring.jpa.properties.hibernate.default_batch_fetch_size=25
```

This applies batch fetching to ALL lazy associations globally -- often the single highest-impact Hibernate tuning parameter.

### Fix selection guide

| Scenario | Best fix | Why |
|----------|----------|-----|
| Always need children with parent | `JOIN FETCH` | Single query, minimal overhead |
| Sometimes need children | `@EntityGraph` on specific queries | Selective eager loading per use case |
| Multiple collections on entity | `@BatchSize` or `default_batch_fetch_size` | Avoids cartesian product from multiple JOINs |
| Large result sets | `@BatchSize` | JOIN FETCH with pagination is problematic (Hibernate warns about applying in-memory pagination) |

## Connection Pooling

### HikariCP Configuration

HikariCP is the default connection pool for Spring Boot 2+ and the recommended pool for any Java application.

```properties
# Essential settings
spring.datasource.hikari.minimum-idle=5           # min connections kept open (default: same as max)
spring.datasource.hikari.maximum-pool-size=10      # max connections (default: 10)
spring.datasource.hikari.connection-timeout=30000   # ms to wait for connection (default: 30s)
spring.datasource.hikari.max-lifetime=1800000       # max connection age before recycling (default: 30min)
spring.datasource.hikari.idle-timeout=600000        # max idle time before eviction (default: 10min)
spring.datasource.hikari.leak-detection-threshold=60000  # ms -- log warning if connection not returned
```

### Common misconfigurations

| Misconfiguration | Symptom | Fix |
|-----------------|---------|-----|
| `maximum-pool-size` too small | `SQLTransientConnectionException: Connection not available, request timed out after 30000ms` | Increase pool size. Rule of thumb: `pool_size = (core_count * 2) + effective_spindle_count`. For SSDs, start at ~10. |
| `maximum-pool-size` too large | Database overwhelmed with connections, context switching overhead | PostgreSQL: keep total connections (across all app instances) under `max_connections`. Each idle connection uses ~10 MB of DB memory. |
| `connection-timeout` too short | Spurious timeouts during traffic spikes | Increase to 30-60s. If timeouts persist, the pool is too small. |
| `max-lifetime` not set or too high | Connections go stale, database restarts cause errors | Set to 5 minutes less than database's `wait_timeout` / `idle_in_transaction_session_timeout`. |
| `minimum-idle` = `maximum-pool-size` | Pool never shrinks during idle periods | Set `minimum-idle` lower to release connections during off-peak. |
| No `leak-detection-threshold` | Connection leaks go undetected until pool exhaustion | Set to 60000 (60s). Logs a warning with stack trace when a connection isn't returned within the threshold. |

### Connection pool sizing formula

The PostgreSQL wiki suggests: `pool_size = ((core_count * 2) + effective_spindle_count)`. For most modern servers with SSDs:

- 4-core machine: 10 connections
- 8-core machine: 20 connections
- More is NOT always better -- beyond the optimal point, context switching and lock contention reduce throughput

**Multiple app instances**: If you have 4 app instances each with a pool of 10, the database sees 40 connections. Size accordingly.

## Query Optimization

### JPQL vs Criteria API vs Native SQL

| Approach | Type-safe | Readable | Performance | Use when |
|----------|-----------|----------|-------------|----------|
| JPQL | No (string) | High | Good | Simple queries, most use cases |
| Criteria API | Yes | Low (verbose) | Same as JPQL (same query plan) | Dynamic queries with optional filters |
| Native SQL | No | Medium | Best (full DB feature access) | Complex aggregations, CTEs, window functions, DB-specific features |

### Pagination: OFFSET vs Keyset

```java
// OFFSET pagination: simple but slow for deep pages
// Page 1000 = database reads and discards 999 * pageSize rows
List<Order> page = entityManager.createQuery(
    "SELECT o FROM Order o ORDER BY o.createdAt DESC", Order.class)
    .setFirstResult(999 * 20)  // skip 19,980 rows
    .setMaxResults(20)
    .getResultList();

// KEYSET pagination: constant performance regardless of page depth
// Pass the last seen value from the previous page
List<Order> page = entityManager.createQuery(
    "SELECT o FROM Order o WHERE o.createdAt < :cursor ORDER BY o.createdAt DESC", Order.class)
    .setParameter("cursor", lastSeenCreatedAt)
    .setMaxResults(20)
    .getResultList();
```

**Rule**: Use OFFSET for shallow pages (< 100 pages) or admin UIs. Use keyset pagination for any user-facing infinite scroll, API pagination, or deep result sets.

### Projection: DTO vs Entity

```java
// FULL ENTITY: loads all columns, managed by persistence context
List<Order> orders = entityManager.createQuery(
    "SELECT o FROM Order o WHERE o.status = :status", Order.class)
    .setParameter("status", "ACTIVE")
    .getResultList();
// Each Order is tracked for dirty checking, occupies identity map memory

// DTO PROJECTION: loads only needed columns, not managed
List<OrderSummary> summaries = entityManager.createQuery(
    "SELECT new com.example.OrderSummary(o.id, o.total, o.createdAt) " +
    "FROM Order o WHERE o.status = :status", OrderSummary.class)
    .setParameter("status", "ACTIVE")
    .getResultList();
// Lightweight: no dirty checking, no identity map, less memory

// TUPLE PROJECTION (Criteria API)
CriteriaBuilder cb = entityManager.getCriteriaBuilder();
CriteriaQuery<Tuple> q = cb.createTupleQuery();
Root<Order> root = q.from(Order.class);
q.multiselect(root.get("id"), root.get("total"));
```

**Rule**: Use DTO projections for read-only queries (reports, lists, API responses). Use entity loading only when you need to modify the entity or traverse lazy relationships.

## Second-Level Cache

### Configuration (Ehcache / Caffeine)

```properties
# Enable second-level cache
spring.jpa.properties.hibernate.cache.use_second_level_cache=true
spring.jpa.properties.hibernate.cache.region.factory_class=org.hibernate.cache.jcache.JCacheRegionFactory
spring.jpa.properties.hibernate.javax.cache.provider=org.ehcache.jsr107.EhcacheCachingProvider

# Enable query cache (caches JPQL/HQL query results)
spring.jpa.properties.hibernate.cache.use_query_cache=true
```

### Entity cache

```java
@Entity
@Cacheable
@org.hibernate.annotations.Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
public class Product {
    // Cached after first load. Subsequent findById() returns from cache.
}
```

### Query cache

```java
List<Product> products = entityManager.createQuery(
    "SELECT p FROM Product p WHERE p.category = :cat", Product.class)
    .setParameter("cat", "electronics")
    .setHint("org.hibernate.cacheable", true)  // enable query cache for this query
    .getResultList();
```

### When to cache vs when to avoid

| Cache type | Use when | Avoid when |
|-----------|----------|-----------|
| Entity cache | Read-heavy entities updated rarely (products, configuration, reference data) | Frequently updated entities (orders, events, logs) |
| Query cache | Same query with same parameters runs repeatedly | Queries with frequently-changing underlying data (cache invalidated on any table change) |
| Collection cache | `@Cache` on `@OneToMany` -- collection accessed repeatedly and rarely modified | Large collections or frequently-modified collections |

**Warning**: The query cache is invalidated when ANY entity in the queried table changes. For tables with frequent writes, the cache hit rate drops to near zero and the cache management overhead makes things slower.

## Batch Operations

### Hibernate batch inserts

```properties
# Enable JDBC batching
spring.jpa.properties.hibernate.jdbc.batch_size=50
spring.jpa.properties.hibernate.order_inserts=true
spring.jpa.properties.hibernate.order_updates=true
```

```java
// Batch insert with periodic flush/clear
for (int i = 0; i < 10_000; i++) {
    entityManager.persist(new OrderItem(/* ... */));
    if (i % 50 == 0) {
        entityManager.flush();  // execute batched INSERTs
        entityManager.clear();  // detach all entities (free memory)
    }
}
```

### JDBC batch inserts (bypass Hibernate)

For maximum insert throughput, bypass Hibernate entirely:

```java
@Autowired
JdbcTemplate jdbcTemplate;

jdbcTemplate.batchUpdate(
    "INSERT INTO order_item (order_id, product_id, quantity) VALUES (?, ?, ?)",
    items, 1000,  // batch size
    (ps, item) -> {
        ps.setLong(1, item.getOrderId());
        ps.setLong(2, item.getProductId());
        ps.setInt(3, item.getQuantity());
    }
);
```

### Statement ordering

When batch-inserting entities with multiple types, Hibernate may interleave INSERT statements for different tables, breaking JDBC batching. Enable statement ordering:

```properties
hibernate.order_inserts=true   # group INSERTs by table
hibernate.order_updates=true   # group UPDATEs by table
```

## EXPLAIN Plan Verification

### Running EXPLAIN from Java

```java
// Spring JdbcTemplate
String plan = jdbcTemplate.queryForObject(
    "EXPLAIN (FORMAT JSON) SELECT * FROM orders WHERE status = ? AND created_at > ?",
    String.class, "ACTIVE", cutoffDate);

// EntityManager native query
Query q = entityManager.createNativeQuery(
    "EXPLAIN (FORMAT TEXT) SELECT * FROM orders WHERE status = ?1 AND created_at > ?2");
q.setParameter(1, "ACTIVE");
q.setParameter(2, cutoffDate);
List<Object[]> plan = q.getResultList();
for (Object[] row : plan) {
    System.out.println(row[0]);
}
```

### What to check

| Check | What to look for | Problem if wrong |
|-------|-----------------|-----------------|
| **Scan type** | `Index Scan` or `Index Only Scan` on filtered columns | `Seq Scan` on large table = missing index |
| **Estimated rows** | Should match actual row count (use `EXPLAIN ANALYZE` in dev) | Stale statistics = wrong query plan. Run `ANALYZE table_name`. |
| **Join type** | `Nested Loop` for small result sets, `Hash Join` for large | `Nested Loop` on large joins = O(n*m) |
| **Sort** | `Index Scan` providing order, or `Sort` node | `Sort` on large result set = disk sort possible |
| **Bitmap Heap Scan** | Filter efficiency -- `Rows Removed by Filter` should be low | High `Rows Removed` = index returns too many false matches |

### Common missing indexes

```java
// If you frequently filter by these patterns, verify indexes exist:

// 1. Status + timestamp (range query on created_at with status filter)
@Index(columnList = "status, created_at")

// 2. Foreign keys (JPA does NOT auto-create FK indexes -- unlike Django)
@Index(columnList = "customer_id")

// 3. Composite for multi-column WHERE
@Index(columnList = "tenant_id, status, created_at")

// 4. Partial index (PostgreSQL) for common filter
// Create via native DDL or Flyway migration:
// CREATE INDEX idx_orders_active ON orders (created_at) WHERE status = 'ACTIVE'
```

## Pitfalls

- **N+1 is the default**: JPA loads associations lazily by default. Every access to an unloaded association triggers a query. Use `default_batch_fetch_size` as a global safety net.
- **JOIN FETCH + pagination = in-memory pagination**: Hibernate cannot apply LIMIT/OFFSET when JOIN FETCH produces a cartesian product. It loads ALL rows and paginates in memory, logging: `HHH90003004: firstResult/maxResults specified with collection fetch; applying in memory`. Use `@BatchSize` or DTO projection for paginated queries with eager loading.
- **Entity loading for read-only queries**: Loading full entities for display/API responses wastes memory (identity map tracking, dirty checking) and may trigger lazy-loading cascades. Use DTO projections.
- **HikariCP pool exhaustion during long transactions**: A transaction holds a connection for its entire duration. Long transactions (batch processing, report generation) exhaust the pool. Move long operations to a separate data source or use streaming (`ScrollableResults`).
- **Query cache with frequent writes**: The query cache is table-level -- ANY update to the table invalidates ALL cached queries for that table. For write-heavy tables, the query cache has near-zero hit rate and adds overhead.
- **Missing JDBC batching**: Without `hibernate.jdbc.batch_size`, each `persist()` generates a separate `INSERT` statement. For bulk inserts, this is orders of magnitude slower than batched inserts.
- **Identity generation disables batching**: `@GeneratedValue(strategy = GenerationType.IDENTITY)` forces Hibernate to execute INSERT immediately (to get the generated ID), defeating JDBC batching. Use `SEQUENCE` strategy instead.
