Handling Database Growth
The bookstore took off, the data swelled, queries slowed. This chapter splits one database into many — partitioning it, replicating it, and then confronting the hard problem that follows: when several nodes accept writes at once, whose version wins?
Assumes you know
After this you should be able to
- Explain the core idea in Handling Database Growth
- Apply Handling Database Growth in a representative design
The bookstore took off, the data swelled, queries slowed. This chapter splits one database into many — partitioning it, replicating it, and then confronting the hard problem that follows: when several nodes accept writes at once, whose version wins?
Chapter 1 assumed one database instance. This chapter grows past it in a clear arc: first partition the data (vertical splits, then sharding) so it fits and queries stay fast; then replicate it for availability and read scale; then face the consequence — multi-master write conflicts — and the three families of fixes (clock ordering, CRDTs/OT, and leaderless quorums). Anything marked [+] is context I added; everything else is rewritten from the seven teaching lessons.
Partitioning: split the table before it buries you#
As data grows, queries slow and one instance can't hold it all — so you partition, either vertically or horizontally. Partitioning splits big tables into smaller parts; queries scan less data, so performance improves.
Vertical partitioning — split by column
Take the bookstore's Book
table and split its columns into two tables on the same
instance: BookInfo
(name, description, price) and BookStats
(stock, last-ordered). This maps cleanly to access patterns — BookInfo
is fetched often and is mostly static (cacheable); BookStats
is dynamic with more concurrent writes, so it can get a stricter isolation level. Vertical partitioning also
lets you apply targeted security — sensitive data (card details, passwords) moves to its own table with
encryption and tighter controls. It's the model behind wide-column databases. The limit: vertical partitions
still live on one instance, so there's only so much you can store.
Horizontal partitioning (sharding) — split by row
When data outgrows one machine, shard: split a large table into smaller tables with fewer rows but the same columns, spread across many instances.
The bookstore's Book
table is divided into shards on a key (book_Id
), spreading load across DB instances and cutting the risk any one goes down.
The sharding key is everything: it must spread data evenly. Shards needn't be equal size, but one shard soaking up most traffic defeats the purpose — which makes the sharding strategy the critical decision.
Picking a sharding strategy — and living with it#
| Strategy | How rows are assigned to shards |
|---|---|
| Range-based | By a value range — books $5–10 → Shard A, $11–15 → Shard B, $16–20 → Shard C. |
| List-based | By a defined list — European publishers → Shard A, Asian publishers → Shard B. |
| Hash-based | A hash function on a column decides the shard — random distribution, useful when there's no logical way to group. |
| Round robin | Row 1 → shard 1, row 2 → shard 2, … wrapping around. Simplest; yields roughly equal partitions. |
| Composite | Combines several of the above to get the desired effect. |
Sharding removes the ceiling — there's no limit to how many instances you can add — and adds availability, since a few shards can fail while others keep serving. But it demands planning.
1. Cross-partition access is slow — keep data used together in the same shard; run parallel queries across shards and aggregate in the app instead of cross-shard joins. 2. Replicate hot static data across shards to avoid constant trips to one static-data shard (at the cost of a sync overhead when it changes). 3. Keep a shard map in the app so queries locate the right shard directly — never scan shards at query time to find one.
Study your access patterns first (frequency, result-set size, latency), monitor shards continuously (storage, CPU, bandwidth) for even load, back up critical shards (financial data) more often than others (logs), and re-partition if a shard gets hot. Plan for sharding ahead of time — it's non-trivial and needs careful planning, resources, and time.
Replication: copies for availability and read scale#
Replication copies a database across instances to make it highly available, scalable, and durable — more instances means less risk of total failure, more read capacity, and backed-up data. Relational databases have two main modes.
Master-slave (single-leader)
One master handles reads and writes; slaves handle reads only. This lifts read throughput (and write, indirectly — the master is freed of most reads), and since all writes go to one node there are no write conflicts and coordination is simple. Risk: a write-intensive app can bottleneck on the single master. If a slave dies, others serve reads; if the master dies, the app either elects a new master from the slaves (consensus ) or goes read-only until it returns.
With async replication, if the master dies and a new one is elected without the latest writes, then the original master returns — you now have two masters accepting writes with inconsistent data. The fix is a fencing mechanism: STONITH (Shoot The Other Node In The Head) kills one leader to end the confusion.
Sync vs. async replication: synchronous replicates to all slaves during the write — strong consistency, higher write latency. Asynchronous replicates after the write completes — lower latency, but reads from replicas can be stale. That staleness is the classic "I posted something, refreshed, and it wasn't there yet" bug — your read hit a replica before replication caught up. This is read-your-write consistency; a fix is routing a user's reads of their own just-written data to the master (or to the master for ~2 min after a write), though that can hurt performance under heavy writes. A bonus of master-slave: read-only slave access can be safely handed to other teams for analytics.
Master-master (multi-leader)
Multiple masters accept writes simultaneously. It stays writable even if one master dies, and masters can sit near clients in different regions to cut write latency. The cost: syncing writes across masters while keeping consistency is tricky, ACID gets hard, and write conflicts arise. Making masters write synchronously for consistency spikes latency and defeats the point of independent leaders. Both modes add complexity; heavy writes mean heavy sync overhead.
The conflict multi-master creates#
When two masters accept writes independently and sync asynchronously, their states can diverge — and merging exposes a conflict. The cart example: a user adds A, B, C on master M1 (M2 not yet updated), then adds D on M1 (M1 = A,B,C,D; M2 = A,B,C). Before the sync lands, they add E — this time handled by M2 (M2 = A,B,C,E). When the masters sync, they detect a conflict. This is routine in globally distributed multi-master systems, and unlike Git merge conflicts a human resolves, these must be resolved by the system in real time.
Last Write Wins — the crude default
Keep only the last write on a resource, discard the rest — which means losing data (the cart becomes A,B,C,E; M1's D is dropped). Used by MongoDB and Cassandra for conflicting replica updates. It's not ideal here (we don't want to drop the user's book) and not fully reliable, because "which write was last" depends on event timestamps — which brings us to the clock problem.
Each node has its own clock, running at a slightly different rate — synchronizing them to the smallest unit
is extremely hard, and there's always drift. Even with NTP, accuracy is only as good as the network (congestion wrecks it). So causal transactions can get
mis-ordered: if T1 commits on N1 at time t
and T2 (caused by T1) commits on N2 at t+50ms, but N2's clock lags 100ms, N2 records t−50ms
— making the system think T2 happened before
T1. Databases fix this with methods like Lamport clocks and vector clocks.
Ordering events with synchronized clocks#
Clock-sync algorithms bound the skew — the difference between any two nodes' clocks — guaranteeing it never
exceeds some x
(say 10ms).
The guarantee hinges on network conditions, which is why companies route this traffic over their own private
networks.
Google Spanner's TrueTime
Spanner (a managed relational DB) uses a distributed clock called TrueTime to generate monotonically increasing transaction timestamps. Its twist: instead of a single timestamp, TrueTime returns a time interval, referencing both GPS and atomic clocks — two sources kept for redundancy because they fail differently (GPS: antenna/receiver failures, interference; atomic: frequency drift).
- Implemented via time master machines per data center (GPS Time Masters with dedicated antennas, physically separated to reduce interference/spoofing) plus a time slave daemon per machine.
- Armageddon Masters carry atomic clocks as backup when satellite links drop.
- Redundant masters mean even a network partition leaves both sides with accurate time; nodes must stay connected to a master or their clocks drift.
- Result: skew of just 1–7ms, versus the typical 50–200ms for NTP-based systems.
MongoDB instead uses a hybrid logical clock — combining physical and logical clocks, needing no GPS or atomic hardware — to order transactions across cluster nodes.
Merging without dropping data: CRDTs & OT#
CRDTs — Conflict-free Replicated Data Types
Unlike Last Write Wins, a CRDT drops nothing — it tracks every write and merges them eventually. The cart ends up with A, B, C, D, and E. A CRDT is a data structure replicated across nodes that allows concurrent updates without real-time coordination; a built-in conflict-resolution algorithm reconciles inconsistencies. The trivial example: a boolean flag tracking whether an event occurred anywhere — if any node sets it true, the merged result is true. Real CRDTs use tailored data types (counter, list, stack, queue), each with its own resolution technique. Complex cases still need hand-written, well-tested resolution logic.
Redis implements CRDTs via CRDBs (Conflict-free Replicated Databases) — local DBs in each cluster that replicate through a CRDB Syncer, resolving conflicts by per-data-type rules that infer developer intent, streaming compressed writes and surviving network partitions. Real-world: Riot Games uses CRDTs for League of Legends in-game chat; Bet365 uses CRDTs with Erlang and Riak for scale.
Operational Transformation (OT)
OT resolves conflicts in collaborative editors like Google Docs, where many users edit the same content
at once.
Each user works on a local copy lock-free; changes propagate asynchronously, but are
transformed
before display so everyone converges on a consistent view. The example: string abc. User A inserts z
at position 0 → zabc
(synced to server). User B, still seeing abc, deletes position 2 (c
); applied naively to the server's zabc, position 2 is now b, giving the wrong zac. OT passes B's operation through a transform function that adjusts the position, yielding the correct
zab. Many OT algorithms exist with different consistency models. MongoDB Atlas
app services use OT with predefined rules — e.g. "deletes always win," so a deleted record stays deleted even if
another user edited it offline.
Leaderless replication & quorums#
In leaderless replication there's no leader — reads and writes go to any node — and strong consistency comes from quorums. Used by Cassandra.
-
Writes go to all nodes.
The app sends the write to every node and waits for acknowledgment from a majority —
n/2 + 1. With 12 nodes, it needs ≥7 acks to call the write successful; fewer means failure. Every record carries a monotonically increasing version number to identify the latest update and resolve conflicts. -
Reads go to all nodes too.
The app reads from all nodes, waits for a majority (
n/2 + 1) to respond, and takes the value with the highest version number. Because both writes and reads touch a majority, at least one node overlaps between them — guaranteeing the read sees the latest write. These are quorum reads/writes.
Why it works, and how nodes sync
Because a majority holds the latest value, the approach is strongly consistent by default, has no single point of failure, and never loses a write. Even if 3 of 5 nodes holding the latest write go
down, the app still reads from a majority — 2 surviving stale nodes don't form a majority, so stale data can't
win. Nodes converge two ways: read repair
(when the app sees mismatched values, it writes the latest back to stale nodes) and the anti-entropy
process
(the DB syncs in the background, catching rarely-read records that read-repair would miss). The tuning rule:
keep
w + r >n
(write nodes + read nodes >total nodes) to guarantee overlap and strong consistency.
The app can fan out requests to all nodes itself — or hand that job to a coordinator node that receives the request, queries the others, aggregates responses, and returns the result, offloading the work from the app.
Recap#
Every technique here trades a single point of truth for many, and then pays to reconcile them. One master is simple but a bottleneck; many masters scale but must agree on order without a shared clock and merge without losing writes. The whole chapter is the price of distribution — and the cleverness (quorums, TrueTime, CRDTs) is all about paying it without lying to the user.
Further reading#
The lessons cite several external links. Below are the ones that resolve live, grouped by topic. A few (a Redis active-active docs page, and two academic/conference PDFs on hybrid clocks and Bet365's Erlang CRDTs) returned errors or blocked automated access and are omitted.
Clocks & event ordering
Conflict resolution in production