Distributed Content Search

The goal: let users search videos, articles, or both by keyword — in natural language, with minimal latency, in near real-time (content is searchable as soon as it's persisted). Video data lives in the VideoMetadata table; articles live in the content graph.

The no-brainer architecture is to search those two databases directly — hit VideoMetadata for videos, the content graph for articles, run both in parallel for a mixed query and merge. It works for basic search, but fails at personalized, relevant results: databases are built to store data, not to run complex natural-language queries, and they lack the features search engines have.

Both primary databases stream to Elasticsearch in real time via Change Data Capture — and Elasticsearch becomes the source of truth for site search (a derived database). Since data flows continuously, keeping the primary DBs and the derived store consistent is essential — and that's a Kafka queue's job.

On the read side, a user's search hits a backend search service that holds the business logic to build Elasticsearch queries, fetches data, then prepares custom objects for display the way the business wants before returning them.

Elasticsearch's vocabulary maps to the relational world: document = row, field = column, index = table (a field can hold a list of same-type values; queries run against an index).

An inverted index stores, for each unique word, the list of documents it appears in — which is what makes full-text search fast. Take three video titles:

1. Lionel Messi Opens the New Season with a Bicycle-Kick Goal
2. Football World Reacts to the Messi 's Bicycle-Kick
3. Messi Looking to Bounce Back With PSG and Peak for World Cup

Break each title into terms (tokens), build a sorted list of unique terms, and record which documents each appears in:

messi → [1, 2, 3]
bicycle-kick → [1, 2]
world → [2, 3]
psg → [3]
goal → [1]...
                

Now searching "Messi" instantly returns documents 1, 2, and 3.

Analyzers, synonyms & querying

• Normalization — some terms are uppercase, some lowercase; normalize to a standard form for case-insensitive search.
• Synonyms — index "leaped/jumped/pounced" as one term (jump ), shrinking the index and making results more relevant (a search for "jump" also surfaces its synonyms).
• Analyzers — built-in ones remove punctuation, standardize case, split on whitespace, detect language, and index accordingly. You can specify custom analyzers, but Elasticsearch is detailed with a steep learning curve — you typically index, then play with the API to see how it analyzes text and computes relevance.
• Querying — a Query API runs queries with endless combinations. Elasticsearch computes a relevance score per matched document and returns results in descending order of score by default.

Scatter-gather: how a query fans out

1. A request hits Node 3; the matching documents are spread across all three nodes.
2. Node 3 broadcasts the request to the other nodes.
3. Every node runs the search in parallel, keeping a local priority queue ordered by relevance score.
4. All nodes return document IDs to Node 3, which merges them with its own priority queue and returns the final results to the user.

Here's the counterintuitive core: the inverted index written to disk is immutable — it never changes. Updates don't modify the original; they add new supplementary index segments holding new and updated data. Why give up in-place updates?

Updating dynamically with segments (from Lucene)

Instead of rebuilding, Elasticsearch appends new segments of recent data to the original, like a log. A query scans the whole log of segments and combines the results. In Lucene, an index is a collection of segments plus a commit point (a file listing all segments). New data flows in-memory buffer → filesystem cache → disk; on a write, pending filesystem-cache writes are flushed to disk to guarantee they're physically written, then the in-memory buffer clears for new data.

Segment-based search shrinks the gap between indexing a document and seeing it in results to within minutes. But there's a bottleneck: writing a new segment to disk needs an fsync to guarantee it's physically persisted, and fsync is costly — every one is a performance hit.

When the system is write-heavy: merging

Frequent writes → proportionally more segments → more CPU cycles to process them and larger scans per query, slowing search down. Elasticsearch fixes this by auto-merging segments in the background: small segments merge into bigger ones, which merge into even bigger ones — and data marked deleted in .del files is excluded from the new merged segments (so deletes are physically reclaimed here).

All three external references resolve live. The author's own course cross-links are omitted.