Distributed Payments
The subscription-payment service — where getting it wrong means double-charging a customer or losing a sale. This chapter traces a card payment through six intermediaries you don't control, then builds a backend on Saga (no locks across banks), event sourcing (an immutable timeline you can replay), and idempotency (click purchase ten times, get charged once).
Assumes you know
After this you should be able to
- Explain the core idea in Distributed Payments
- Apply Distributed Payments in a representative design
The subscription-payment service — where getting it wrong means double-charging a customer or losing a sale. This chapter traces a card payment through six intermediaries you don't control, then builds a backend on Saga (no locks across banks), event sourcing (an immutable timeline you can replay), and idempotency (click purchase ten times, get charged once).
First the domain: what actually happens between clicking "purchase" and the money moving — six players, two stages (authenticate, then transfer funds). Then the backend you build, layered: Saga because you can't lock a bank's database; event sourcing because payments need a replayable timeline; Kafka partitioning because event order matters; and idempotency because networks retry. Anything marked [+] is context I added; everything else is rewritten from the five teaching lessons.
The online card payment flow#
Before designing anything, understand what happens end-to-end. A card payment has two stages — user authentication, then funds transfer — passing through a chain of players, none of which (past your own platform) you control.
| Player | Role |
|---|---|
| Consumer | The user. Clicks purchase → request routes to your platform's payment backend. |
| Platform (Merchant) | You. Your payment module asks the payment processor to process the transaction (order id, amount, …). |
| Payment Processor | Stripe, PayPal, Square, Razorpay. A single point of contact to many banks/financial services; screens for fraud/duplicates, and picks the best acquiring-bank gateway in real time to maximize success rate. |
| Payment Gateway | Provided by the acquiring bank (where you feed card details). The acquiring bank collects funds on the merchant's behalf; the merchant holds an account (usually current) there. |
| Card Network | VISA, MasterCard, RuPay, Amex — connects issuing and acquiring banks (the logo on your card). |
| Issuing Bank | Holds the user's account. Authenticates the user (the OTP page), then debits if funds suffice. |
After the issuing bank debits and notifies the user, the response flows back and the user sees "order successful." But the acquiring bank isn't paid instantly — it submits the day's transactions to the issuing bank in a batch settlement. Until settlement, the card is in the captured state: the debit is authorized and ready, not yet settled.
Why Saga, not two-phase commit#
Which database for payments?
Payments need strong consistency and ACID transactions
— so a relational DB (Postgres/MySQL) is the obvious choice. (Subscription info reuses the existing
User
table, assuming one subscription per user at a time.) Recall the earlier lesson: for strong consistency there's
no difference between SQL and NoSQL — both must lock nodes, and locking a NoSQL setup kills the availability and
performance that made it attractive.
A subscription purchase spans multiple systems — your platform is just one. You cannot acquire a lock across services (your platform can't lock a bank server or the processor's infrastructure), so 2PC is out. The answer is the Saga pattern: a sequence of local transactions, each with a compensating transaction triggered on failure. If anything fails at any stage, the whole thing rolls back from the point of failure. Rollbacks are vital in financial transactions.
You only control the merchant side; everything else is third-party. You interact with the payment processor purely through its API, against which your entire payment backend is written. Every transaction spawned from the merchant carries a unique transaction reference number.
Event sourcing and the payment architecture#
Event sourcing tracks every event in the system in an immutable, sequential log — rather than storing only the final application state. Why bother?
The architecture
- User clicks purchase → request hits the Payments API, which creates an order-purchase event and pushes it to a Kafka queue — the source of events for the whole service.
-
The Payment Module
pulls events, applies business logic, and persists order details to the relational Order
table
(the application state) with
OrderStatus: IN PROGRESS. TheUsertable has a one-to-many withOrder(a user places many orders — subscriptions plus one-off content: podcasts, articles, merch). - The Event Handler pulls events in parallel and appends them to the event store, building the timeline.
- The Payment Module forwards the request to the payment processor — and every component pushes its event details back to the queue so the event store keeps a complete sequential log.
From the moment the API receives the request, the payment flow is a Saga: passing the event to the queue, writing to the Order table, writing to the event store, and forwarding to the processor are each separate local transactions with compensating transactions. Any one fails → the entire transaction rolls back.
The event store
An immutable log of events. You can back it with a relational, document, or wide-column store (Cassandra), or a purpose-built event database (EventStoreDB). Whatever you pick must support sequential writes and querying events by id; if you append multiple events at once, it needs transactions too. The author prefers a relational DB (if not a native event store) and, when choosing products, favors maturity — time in market, community, docs — over a shiny new option, to avoid production firefighting.
Event ordering with Kafka partitions#
A customer's actions must be logged in the order they occurred. Kafka guarantees order within a partition
— events in a partition are persisted and consumed in occurrence order. With one payments
topic, which partition an event lands in is decided by hashing the event key:
partition = hashCode(key) % N // N = number of partitions
To keep one customer's events in order, partition by customer id so they all land in the same partition. Without a key, events distribute round-robin across partitions — and a consumer might read a cancel-order event before the purchase event that it cancels, creating chaos. (Event ordering in distributed systems is tricky and needs heavy testing; corner cases remain even with keyed partitioning.)
Idempotency and CQRS#
The duplicate-events problem
Message queues offer delivery guarantees, and neither is free:
| Strategy | Problem for payments |
|---|---|
| At least once | Guarantees delivery — but may deliver more than once. If a consumer-delivery timeout occurs, Kafka can't tell if it arrived, so it retries → duplicate events → the customer charged multiple times. |
| Only once | May drop some messages — also unacceptable when strong consistency is the order of the day. |
Downstream systems (payment module, event handler) own duplicate-handling. For every subscription-purchase request, generate a unique transaction id / reference number — it can combine customer id + transaction id + timestamp. Use it as the primary key in your tables. Now a duplicate can't be persisted twice — the DB throws a duplicate-key exception, weeding out duplicates. The customer can click purchase n times; all events are idempotent and they're charged once. (The payment processor always receives a unique transaction request.)
CQRS with the event-store data
Recall CQRS (Command Query Responsibility Segregation) from the sharding chapter: besides writing to the primary database, an async process writes to a dedicated query store built for specific reads. The payment service does the same shape — the Order table holds application state (the command side), while the event store serves insights into user activity (the query side). External services read the event store for batch validations, and its data can stream into ELK for analytics.
Payment/banking systems are eventually consistent by their distributed nature. Companies run daily batch operations over event data to validate the day's transactions — catching a customer erroneously over-charged, an inconsistent DB entry, etc., and fixing them. Batch reconciliation is a vital part of any payment system, not an afterthought.
Recap#
A payment is the hardest kind of distributed transaction because the stakes are money and you own almost none of the systems involved. You can't lock what you don't control, so you give up on atomic locking entirely — Saga lets each step fail and compensate. You can't trust the network to deliver exactly once, so you make every step safe to repeat — a unique id turns retries into no-ops. And because money must be auditable and eventually reconciled, you don't just store the final balance — you store the whole story as an immutable log you can replay. Design for failure, repetition, and scrutiny, because in payments all three are guaranteed.
Further reading#
The one external product reference resolves live (now under its new brand name). The author's own course cross-links, and one blog source that failed to load (a scaleyourapp.com article on how Razorpay handled IPL transaction bursts), are omitted.