Numbers to Know
What today’s hardware can actually handle — and how those figures keep you from over-engineering in an interview.
Contents
Our industry moves fast, and hardware evolves constantly — so even recent textbooks quote numbers that are off by orders of magnitude. Relying on those stale constraints is one of the biggest giveaways that a candidate has book knowledge but no hands-on experience. Knowing what modern hardware really handles is what lets you decide when to shard, when to cache, and when a single machine is plenty.
01Why these numbers matter
A common failure mode: doing scale calculations with numbers from 2015 (or even 2020) that dramatically underestimate what modern systems handle. You’ll hear worries about database sizes, memory limits, and storage costs that made sense then but would lead to a significantly over-engineered system today. It isn’t the candidate’s fault — they studied — but an accurate sense of hardware is what separates good design decisions from bad ones.
Modern hardware represents a step change, not an incremental one. Many applications that once required distributed systems now run on a single machine. The recurring interview skill is knowing where the real limits are so you don’t reach for complexity you don’t need.
The article states these as typical values for well-tuned systems in 2026; your mileage varies with workload, hardware, and configuration. Cloud providers update offerings constantly, so treat them as order-of-magnitude anchors for capacity planning, not hard limits. Readers specifically noted the numbers will eventually go stale — a reasonable interview approach is to state your assumptions rather than memorize exact figures.
02Modern hardware limits
Memory
Modern servers pack serious power. A general-purpose AWS M6i.32xlarge has 512 GiB of memory and 128 vCPUs. Memory-optimized instances go much further:
This is why many workloads that once demanded a distributed system can now live on one box.
Storage
The days of storage being a primary constraint are largely behind us.
Network & latency
Within a datacenter, 25 Gbps is common, with high-performance instances at 50–100 Gbps or more. Bandwidth across AZs within a region is limited only by instance network capacity. Latency stays predictable:
03Caching
In-memory caches have grown exponentially. Gone are the days of 32–64GB Redis instances needing careful memory management and partial-dataset caching. Today’s caches routinely hold terabyte-scale datasets at single-digit millisecond latency, and a single instance processes hundreds of thousands of operations per second.
Numbers to know
- Memory: up to 1TB on memory-optimized instances (more for specialized cases)
- Read latency: < 1ms within the same region
- Write latency: < 1ms same-AZ, 1–2ms cross-AZ (same region)
- Throughput: 100k–200k+ ops/sec per instance (ElastiCache Redis on Graviton)
When to scale
- Dataset approaching 1TB
- Sustained 100k+ ops/sec
- Read-latency needs below 0.5ms consistently
Being able to cache entire databases in memory — even at hundreds of GB — means you can often skip complex partial-caching schemes entirely. This brute-force approach typically costs less than the engineering time spent on selective caching logic. When you do need to scale, the bottleneck is usually ops/sec or network bandwidth, not memory size — a counterintuitive flip from a few years ago.
A reader asked exactly this. Two reasons emerged from the discussion:
1. Operational limits. Large instances get harder to back up, replicate, and recover from failures (a reader noted a 1TB backup at ~160MB/s would take on the order of 6,000 seconds). The 1TB figure is a soft operational guideline, not a hard technical ceiling.
2. Redis is single-threaded. You’ll likely become CPU-bound before memory-bound, so you may need to shard just to use more CPU. Alternative implementations (e.g. Dragonfly) can help, but plain Redis pushes you toward sharding for CPU regardless of how much RAM the box has.
Someone pointed out AWS advertises 1M+ RPS per node for ElastiCache Redis 7.1. The author’s guidance: that’s achievable under the right conditions with small value sizes, but for interviews stick to the 100k order of magnitude rather than optimizing for the 1M best case.
04Databases
The raw power of modern databases surprises even experienced engineers. Single PostgreSQL or MySQL instances routinely handle dozens of terabytes while maintaining millisecond-level responses, and tens of thousands of transactions per second on a single primary — with the bottleneck often being operational concerns rather than performance.
Numbers to know
- Storage: up to 64 TiB most engines; Aurora up to 256 TiB
- Read latency: 1–5ms cached, 5–30ms disk (RDS/Aurora)
- Write latency: 5–15ms commit (single-node, high-perf)
- Read throughput: up to 50k TPS single-node (Aurora/RDS)
- Write throughput: 10–20k TPS single-node
- Connections: 5–20k concurrent
When to shard
- Dataset approaching/exceeding 50 TiB
- Write throughput consistently > 10k TPS
- Uncached read latency needs < 5ms
- Cross-region replication / distribution
- Backup windows stretching into hours
The largest systems (social networks, e-commerce giants) genuinely need sharding, but many candidates jump there far too soon. For millions or even tens of millions of users, a well-tuned single database often handles the load. When you do scale, name what’s actually driving it: data volume, operational concerns like backup windows, or geographic distribution. The author’s repeated observation: “They have 500GB or a couple of terabytes and start explaining how they’d shard. Slow down, do the math.”
A single instance still runs a primary with read replicas for availability (e.g. Aurora multi-AZ failover). The point is you often don’t need to shard the data. Replication for high availability is a separate concern from horizontal partitioning for scale.
05Application servers
Modern app servers have outgrown the resource constraints that shaped many traditional patterns. They routinely handle thousands of concurrent connections with modest resource usage, and cloud platforms scale them quickly. CPU — not memory or connection limits — typically determines capacity.
Numbers to know
- Connections: 100k+ concurrent per instance (optimized)
- CPU: 8–64 cores
- Memory: 64–512GB standard, up to 2TB high-mem
- Network: 25 Gbps standard, 50–100 Gbps high-perf
- Startup: 30–60s for containerized apps
When to scale horizontally
- CPU consistently > 70–80%
- Response latency exceeding SLA
- Memory trending > 70–80%
- Network approaching instance limits
Stateless services are great for scaling, but don’t forget each server has substantial memory. Local caching, in-memory computation, and session handling can dramatically improve performance. Since CPU is almost always the first bottleneck, not memory, don’t shy away from memory-intensive optimizations. And with 30–60s container startup, aggressive auto-scaling beats over-provisioning.
06Message queues
Message queues have gone from simple task-delegation systems to high-performance data highways. Kafka processes millions of messages per second at single-digit millisecond latency while retaining weeks or months of data.
Numbers to know
- Throughput: up to 1M messages/sec per broker
- Latency: 1–5ms end-to-end within a region
- Message size: 1KB–10MB handled efficiently
- Storage: up to 50TB per broker
- Retention: weeks to months
When to scale
- Nearing 800k messages/sec per broker
- Partition count approaching 200k per cluster
- Consumer lag consistently growing
- Cross-region replication needed
With reliable sub-5ms latency, you can use a queue inside a synchronous request flow — getting reliable delivery and decoupling without forcing your APIs to be async. Combined with near-unlimited storage, queues can back event sourcing, real-time analytics, and data integration that once needed specialized systems.
07Cheat sheet
| Component | Key metrics | Scale triggers |
|---|---|---|
| Caching | ~1ms latency · 100k+ ops/sec · memory-bound (up to 1TB) | Hit rate < 80% · latency > 1ms · memory > 80% · cache churn/thrashing |
| Databases | Up to 50k TPS · sub-5ms read (cached) · 64 TiB+ storage | Write > 10k TPS · read > 5ms uncached · geographic distribution |
| App servers | 100k+ connections · 8–64 cores @ 2–4 GHz · 64–512GB (up to 2TB) | CPU > 70% · latency > SLA · connections near 100k · memory > 80% |
| Message queues | Up to 1M msgs/sec per broker · sub-5ms end-to-end · up to 50TB | Throughput near 800k msgs/sec · partitions ~200k/cluster · growing lag |
08Common mistakes in interviews
Premature sharding
The single biggest mistake: assuming sharding is always necessary and immediately naming a shard column. It shows up constantly with Design Yelp. Do the math:
The same over-reach happens with caches. Take a LeetCode leaderboard with 100k competitions and up to 100k users each:
400GB is more than the Yelp figure, yet it still fits on a single large cache — no sharding needed.
Overestimating latency
Candidates vastly overestimate the latency of an indexed SSD/database lookup — it’s sub-millisecond to a few milliseconds for indexed row lookups. That speed means they often add a caching layer to “reduce latency” when the simple row lookup was already fast enough, adding infrastructure for no reason.
Caveat: this holds only for simple indexed row lookups. It’s still wise to cache expensive queries.
Over-engineering high write throughput
Given, say, 5k writes/sec, candidates often jump to a message queue to “buffer” it. They don’t need to.
A well-tuned Postgres instance with simple writes handles 20k+ writes/sec. What actually limits write capacity is complex multi-table transactions, write amplification from excessive indexes, expensive cascading updates, or heavy concurrent reads competing with writes. Simple indexed inserts/updates at 5k WPS need no queue.
Message queues earn their place when you need guaranteed delivery on downstream failure, event sourcing, write spikes beyond your DB’s capacity (e.g. above ~20k WPS for a single Postgres), or producer/consumer decoupling — but they add complexity. Before reaching for one, try batch writes, schema/index tuning, connection pooling, or async commits for non-critical writes.
09What about costs?
In a real design, cost is often a major factor — you’d estimate usage, multiply by pricing tables, and compare total cost of ownership. In interviews it’s rarely the focus: most people don’t memorize pricing (RAM/SSD prices change constantly), and accurate pricing needs precise estimates when system-design estimates are only after an order of magnitude.
Interviewers aren’t sensitive to exact dollars, but the abstract idea of cost still matters. Running 100 machines when 1 will do, or a bank of in-memory caches when users only need data in hundreds of milliseconds, will raise a flag. Spend your prep time elsewhere than AWS pricing tables.
10The takeaways
Modern hardware has changed the calculus of system design. Distributed systems and horizontal scaling still matter for the world’s largest applications, but many systems can be far simpler than traditional wisdom suggests:
- Single databases can handle terabytes of data.
- Caches can hold entire datasets in memory.
- Message queues are fast enough for synchronous flows — as long as there’s no backlog.
- Application servers have enough memory for significant local optimization.
It’s not that vertical scaling is always the answer — it’s knowing where the real limits are. That lets you avoid premature optimization and build simpler systems that still grow. Demonstrating it shows you can balance theory with practice — especially valued at senior levels.
11Active recall
You have ~2TB of data and 5k writes/sec. Walk through why you probably don’t need sharding or a queue.
A single well-tuned Postgres/MySQL instance handles dozens of TB and 10–20k+ simple writes/sec, so 2TB and 5k WPS sit comfortably inside one primary (with read replicas for HA). Sharding is driven by ~50 TiB+, >10k TPS sustained writes, sub-5ms uncached reads, geo-distribution, or impractical backup windows — none of which apply here. A queue is justified by guaranteed delivery, event sourcing, spikes beyond ~20k WPS, or decoupling — also not here.
For the Yelp example, reconstruct the capacity math and the conclusion.
10M businesses × ~1KB = 10GB; ×10 to fold in reviews stored in the same DB ≈ 100GB. 100GB fits comfortably on a single database — no sharding. The lesson: do the arithmetic before proposing a shard key.
Modern RAM reaches 24TB — so why is the practical cache guideline ~1TB?
Two reasons from the discussion: (1) operational — big instances are harder to back up, replicate, and recover (a 1TB backup at ~160MB/s ≈ 6,000s); (2) Redis is single-threaded, so you tend to hit a CPU bottleneck before a memory one and must shard to use more CPU (or switch to something like Dragonfly). 1TB is a soft operational guideline, not a hardware ceiling.
When you do need to scale, what’s usually the bottleneck — and why is that counterintuitive?
For caches, it’s ops/sec or network bandwidth, not memory size. For app servers, it’s CPU, not memory or connections. It’s counterintuitive because older constraints trained people to worry about running out of RAM/storage first, when modern capacity has moved the ceiling to throughput and CPU.
Why can a message queue now sit inside a synchronous request flow?
Because modern queues deliver consistent sub-5ms end-to-end latency within a region. That’s fast enough to get reliable delivery and decoupling without forcing the API to be async — with the caveat that this only holds when there’s no growing backlog / consumer lag.
How should you treat these numbers, given they’ll go stale?
As order-of-magnitude starting points for capacity planning, not hard limits — they’re stated for 2026 and vary by workload, hardware, and configuration. In an interview, state your assumptions explicitly rather than relying on memorized exact figures.
12Quick revision
The one-line reference (2026)
| Component | Single-instance ceiling | Scale when… |
|---|---|---|
| Cache | ~1TB · 100k–200k+ ops/s · <1ms | ~1TB, 100k+ ops/s, or <0.5ms reads (CPU-bound first — Redis is single-threaded) |
| Database | 50–64 TiB (Aurora 256) · 50k read TPS · 10–20k write TPS | ~50 TiB, >10k write TPS, <5ms uncached, geo, or long backups |
| App server | 100k+ conns · 8–64 cores · 64–512GB (2TB) | CPU >70–80% (CPU is the first bottleneck) |
| Queue | ~1M msg/s/broker · sub-5ms · 50TB | ~800k msg/s, ~200k partitions, or growing lag |
Latency ladder
<1ms same AZ → 1–2ms cross-AZ same region → 50–150ms cross-region.
Must remember
- Do the math before adding infrastructure. 100GB is not a sharding trigger.
- Bottleneck ≠ size. It’s usually ops/sec, CPU, or bandwidth.
- Single instance ≠ SPOF — run replicas for HA; that’s separate from sharding for scale.
- Indexed SSD lookups are sub-ms — don’t cache them reflexively; do cache expensive queries.
- Postgres does 20k+ simple WPS — 5k WPS doesn’t need a queue.
- Queues are sub-5ms — usable in sync flows, if no backlog.
- Numbers are 2026 starting points, not hard limits — state assumptions.
Three interview traps
- Premature sharding — naming a shard column before doing the arithmetic.
- Overestimating SSD latency — adding a cache in front of an already-fast indexed lookup.
- Over-queueing — buffering write throughput a single DB handles easily.