High Availability
Hardware dies, code crashes, and people fat-finger configs. High availability is the set of design choices that keep the service up anyway — and this chapter builds them from the failure up.
Assumes you know
After this you should be able to
- Explain the core idea in High Availability
- Apply High Availability in a representative design
Hardware dies, code crashes, and people fat-finger configs. High availability is the set of design choices that keep the service up anyway — and this chapter builds them from the failure up.
The chapter is one continuous argument: a system will fail, so how do we keep serving through it? Each section adds one tool. Anything marked [+] is context I added beyond the source lessons — everything else is rewritten from the six teaching lessons of the course's High Availability section.
Availability is measured in nines#
High availability (HA) is a system's ability to stay online despite infrastructure-level failures happening in real time. It's the property that keeps uptime high, downtime minimal, and the service reachable even while pieces of it are breaking underneath.
The industry states it as a percentage of total hosting time the service is expected to be up. A promise of
99.99999% availability means the service is designed to be reachable that fraction of the time —
the number you'll find written into a cloud provider's Service Level Agreement (SLA).
Each extra nine is an order of magnitude less downtime, and it gets expensive fast. Three nines (99.9%) allows about 8.8 hours of downtime a year; four nines allows ~53 minutes; five nines only ~5 minutes. The percentages in the lessons look like rounding noise, but each nine is a real jump in engineering cost.
Not every service needs the same target. A social app blinking offline for a moment and bouncing back rarely hurts the business. Mission-critical systems — aircraft avionics, spacecraft, mining machinery, hospital servers, stock-market and finance systems — cannot go down at all, because lives or large sums ride on their continual connection to their network and servers. Those are the systems that justify super-highly-available infrastructure. To hit those targets, systems are deliberately designed to be fault-tolerant and their components are made redundant — the two ideas the rest of this chapter builds on.
Four ways a system goes down#
Before designing around failure, name it. Systems fail for four recurring reasons.
| Failure class | What it looks like |
|---|---|
| Software crashes | Applications crash on phones, desktops, and cloud nodes alike. Corrupt files, the Windows blue screen of death, an OS falling over, or a memory-hogging unresponsive process can take down an entire node. Cloud software crashes unpredictably. |
| Hardware failures | Overloaded CPU and RAM, hard-disk failures, whole nodes going dark, and network outages. |
| Human error | The biggest single cause. Flawed configurations and the like. A tiny Google network-config mistake once took down roughly half the internet in Japan. |
| Planned downtime | Not every outage is a surprise — routine maintenance, software patching, and hardware upgrades all take the system offline on purpose. |
The course singles out human error as the biggest reason systems fail. That's why later sections lean so hard on automation — taking the human out of the recovery loop removes the most common failure source.
Fault tolerance means failing soft, not falling over#
Fault tolerance is a system's ability to stay up despite taking hits. Out of many nodes running a service, a few can go offline and bounce back continuously; the system keeps working at a reduced level instead of collapsing entirely. This graceful degradation is called fail soft — the system absorbs the failure rather than becoming unusable. It's an essential property when designing life-critical systems.
The monolith is the trap; microservices are the fix
The clearest way to make an application fault-tolerant is to stop building it as one giant program. A massive service is broken down into granular, loosely coupled microservices, each owning one feature. When features are isolated this way, one failing feature can't drag down the rest.
Picture a social network with image upload, comments, instant messaging, user tagging, and live streaming. If all of that lives in a single monolith and a bug hits image upload, the whole service can take the hit — live streaming and comments start behaving unexpectedly, because they share the same process. Split those features into separate microservices, deployed independently, and an image-upload bug — however big — leaves live streaming and comments untouched. The network as a whole stays up; only one feature degrades. That's fault tolerance through architecture.
The upsides the course attributes to this split: easier management and maintenance, easier development, the ability to add features to one service without touching others, and — the reason it's here — scalability and high availability. Fault tolerance can be engineered at both the application level (this microservice split) and the deployment level (the redundant, replicated infrastructure covered next).
Redundancy: always keep a spare#
Redundancy is duplicating critical components so a backup can take over the instant an active one fails. When an active component drops, its duplicate steps in and the service keeps serving. This spans redundant servers, storage, and networking gear (switches, routers), plus spreading the service across data centers, cloud availability zones, and regions.
Active-passive mode: the standby backup
Redundancy is commonly implemented as active-passive HA. Active instances serve live traffic in real time; passive instances sit idle on standby. The system continuously monitors the active nodes and automatically swaps in a standby the moment one fails. When a recovered node comes back online, it rejoins as a passive standby — and the cycle continues. The result is zero or minimal downtime, which is how systems like GPS, aircraft, and communication satellites hold effectively zero downtime.
No single points of failure
Distributed systems went mainstream for large-scale apps precisely because they let us eliminate the single points of failure (SPOF) that plagued monoliths. When many redundant server nodes work together toward one synchronous application state, no single node's death matters — a redundant node takes its place and the whole system carries on. A SPOF at the component level is a bottleneck, such as a single database instance serving every application node; the course's guidance is to catch these in performance testing and remove them fast.
Monitoring and automation close the loop
Redundancy only works if failures are detected fast. Systems are monitored in real time, and infrastructure automation lets instances self-recover or be replaced by redundant ones without human intervention. A self-managing cluster balancing its own load in real time also removes the human-error failure mode — the most common cause of outages from Section 02.
Replication: keep copies in sync everywhere#
Replication is duplicating data or resources across multiple locations and keeping every copy in sync. The classic case is replicating data across servers, storage devices, data centers, availability zones, and cloud regions. It buys four things at once: higher data availability, fault tolerance, reliability, and load spreading — query load is balanced across all instances holding a copy instead of hammering one. The same applies to compute: replicating an instance's memory and storage lets a fleet process tasks in parallel, making the system more scalable and available.
Geographically distributing copies is a disaster-recovery play. Replicating data across regions protects against regional disasters — power outages, floods, earthquakes — removing the data center, zone, or region as a single point of failure. It also cuts latency, because data lives near the user.
Active-active mode: everyone works at once
Replication powers active-active HA: many similar nodes run the workload together, replicating data among each other to stay in sync. There are no standby nodes — every instance is live. When one or a few nodes fail, the remaining nodes absorb the load. Contrast this with the active-passive mode from Section 04, where backups sat idle.
Redundancy and replication are partners, not rivals#
They're closely related and both essential, but they answer different questions.
They depend on each other. Passive redundant instances in active-passive mode need replication too — any change on the active instances must be replicated to the standbys in real time or near-real time. If a standby's data is stale, it can't seamlessly take over. Redundancy provides the spare; replication makes sure that spare is up to date enough to actually be useful.
Geographic distribution as contingency
As a hedge against natural disasters, regional outages, and large-scale failures, data center workloads are spread across the globe. If a disaster wipes out data centers in one zone, others in different geographies keep the application running, and proximity to users keeps latency down. Every HA and fault-tolerance decision is ultimately subjective — it depends on how critical the system is and the odds of components failing. Businesses often go a step further with multi-cloud deployments, so if one cloud provider has trouble, they can fail over to another.
HA clusters: nodes that watch each other's pulse#
A high availability cluster — also called a fail-over cluster — is a set of nodes running together to guarantee a service stays available. It's where fault tolerance, redundancy, and replication come together into one operating unit.
- Heartbeat network. The nodes are wired together by a private network called the heartbeat, which continuously monitors the health and status of every node in the cluster.
- Single shared state. All nodes agree on one state using shared distributed memory and a distributed coordination service — the course names ZooKeeper as the example.
- Redundant everything. Clusters use disk mirroring / RAID (Redundant Array of Independent Disks), redundant network connections, and redundant electrical power. Every component that could be a single point of failure is made redundant.
- Multiple clusters per zone. Several HA clusters run together in one geographical zone to keep downtime minimal and service continuous.
Recap#
Failure is a certainty, not a risk — so never let any one thing be the only thing. Split features so one bug can't sink the app, keep spare components so a death is survivable, replicate data so no copy is irreplaceable, and automate recovery so the most fragile part — the human — stays out of the loop.
Further reading#
The section cites one external link. Verified live today; it now resolves to an updated URL.