Graph RAG in Production → DSPy

Section 1 · Copy this by hand

Notebook Summary#

Telegraphic revision notes. Box the ⭐ / ⚠ / 💡 lines. Everything here is derived from the Detailed Notes below.

Section 2 · Primary learning material

Detailed Study Notes#

Standard RAG vs. Graph RAG: stuffing vs. mapping#

Graph RAG answers a question by handing the model a small map of how facts connect, instead of a pile of text and hoping the model connects them itself.

In ordinary retrieval-augmented generation (standard RAG), the strategy is simple and greedy: grab as many relevant text chunks as you can, cram them into the prompt, and let the model sort it out. It works for "what does this document say about X" but falls apart on questions whose answer is spread across several facts that were never written down together.

Graph RAG changes what you put in the context window. Instead of unstructured paragraphs, you feed the model the topology, the literal map of which things relate to which. Put another way: you build a "surgical theater" of a context window and tell the model, "here is a map of relationships, read the map and tell the user what you see."

Putting vectors inside the graph#

Neo4j stores each node's embedding as a property attached directly to that node, then searches those embeddings with an approximate nearest-neighbor structure called HNSW.

Earlier, vectors lived in FAISS, a separate flat index held in memory, completely divorced from any graph. Neo4j does it differently: it attaches the numerical vector (that 768-dimensional float array) directly to the node entity, on disk. The node "Microsoft" carries both its label/properties and its embedding. The embedding captures the semantic meaning of the node's original text, e.g. Microsoft's text "a tech giant" holds far more searchable meaning than the bare token "Microsoft".

How the search zooms in — HNSW#

To find the closest vectors fast, Neo4j uses HNSW (Hierarchical Navigable Small World). You don't scan every node; you descend through layers of increasing detail.

The similarity metric#

Underneath every layer, "closeness" is measured with cosine similarity between the user's prompt vector and each node's embedding:

cosine(A, B) = (A · B) / (‖A‖ · ‖B‖)

Neo4j calculates this between the incoming query vector and node embeddings, and HNSW just decides which nodes are worth calculating it for.

Architecting the index (the DDL rules)#

You cannot just shove arrays into Neo4j and expect fast search, you must declare a formal vector index with a Cypher DDL statement. Three configuration rules matter:

CREATE VECTOR INDEX alpha_fund_embeddings IF NOT EXISTS
FOR (n:Company|Executive|Document) ON n.embedding
OPTIONS { indexConfig: {
  `vector.dimensions`: 768,
  `vector.similarity_function`: 'cosine'
}}

Embedding at scale with UNWIND#

To embed thousands of nodes without melting your API quota, you pull a batch of un-embedded nodes, embed them all at once, and write them back in a single bulk operation.

The naive ("junior") approach loops one node at a time: MATCH … RETURN, call the embedding API, write it back, repeat, 10,000 times. That takes hours and slams straight into API rate limits. The production ("senior") approach is batch processing: pull the chunks, embed them in bulk, push them back.

# fetch un-embedded nodes, batched
MATCH (n) WHERE n.original_text IS NOT NULL AND n.embedding IS NULL
RETURN id(n) AS id, n.original_text AS text LIMIT $batch

# ...embed all texts in one API call, then bulk write:
UNWIND $rows AS row
MATCH (n) WHERE id(n) = row.id
CALL db.create.setNodeVectorProperty(n, 'embedding', row.vector)

The three-phase hybrid retrieval#

Answer a multi-hop question by first finding where to start (vector), then walking the relationships (graph), then letting the model read the result (LLM).

Here is the core insight of the whole session: for a question like "the startup Microsoft just bought," neither tool works alone. Pure vector search (FAISS) can't follow relationships. Pure Cypher can't be written because you don't know the exact node names yet. So you go hybrid, in three phases.

Phase ① — Vector Jump#

Embed the user's vague prompt at runtime, then ask the vector index for the single closest node. This turns "the startup Microsoft just bought" into an exact database identity like Alpha AI, complete with a confidence score.

CALL db.index.vector.queryNodes('alpha_fund_embeddings', 1, $promptVector)
YIELD node, score
RETURN id(node) AS entryNode, labels(node), score

Phase ② — Graph Traversal (the ego graph)#

From the entry node you extract a localized subgraph, the ego graph: everything radiating outward from that "center of the universe" node, out to a fixed number of hops. You don't care about the whole city, only the entry node's neighborhood.

The tool for this is a variable-length path:

MATCH path = (start)-[*1..2]-(neighbor)
WHERE id(start) = $entryNode
RETURN path

• *1..2 = follow between 1 and 2 hops, regardless of edge type.
• path returns the entire journey (node → edge → node → edge → node), not just the destination.
• The dashes -[...]- have no arrows, the traversal is undirected.

Phase ③ — LLM Synthesis#

The extracted subgraph is serialized into clean text and posed to the model alongside the user's question. The model reads the map and writes the final answer (covered in detail under Serialization).

Choosing the hop depth#

Two hops is the empirically-chosen sweet spot: deep enough to capture indirect relationships, shallow enough not to drag in the whole database.

Why not just set five hops "to be safe"? Because path count explodes with depth in a connected graph. The trade-off, explicitly:

Serializing the graph for the LLM#

Flatten the retrieved paths into simple linear text so the model spends its reasoning on the answer, not on parsing brackets.

The model "has no eyes", it can't look at a graph visualization, and it shouldn't have to. If you dump raw Python dictionaries or JSON into the prompt, the LLM burns reasoning power parsing brackets instead of answering. Serialization is the translation layer: turn each path into a clean, human-readable line.

Context-window pruning: taming super nodes#

Before serializing, trim the retrieved subgraph so a highly-connected node can't drown the model in tens of thousands of irrelevant paths.

The 2-hop traversal works beautifully for a niche node like Alpha. Now point it at Microsoft, a super node defined by its enormous degree (number of connected edges): thousands of employees, hundreds of acquisitions, tens of thousands of press releases. A 2-hop traversal from Microsoft can return ~50,000 paths. Stuff those in the prompt and you hit the token limit, dilute the model's attention, hallucinate, and burn API credits, all in one query.

The "junior" move is to shrug, "Gemini has a 1M-token window, stuff it all in." That's wrong. Two layers of pruning fix it:

Layer A — Topological pruning (blunt)#

Instruct Cypher to ignore over-generic, over-connected nodes by capping neighborhood size:

WHERE size(neighbors) < 100

This chops off the noisy hubs and keeps only reasonably specific relationships. It's crude but cheap.

Layer B — Semantic edge filtering (scalpel)#

This is the advanced technique: use vector math twice in the same query. Step 1 already used the prompt to find the entry node. Step 2 walks the paths and computes cosine similarity between the user's prompt vector and each neighbor's embedding. If a branch's meaning doesn't match (e.g. below a ~0.55–0.60 threshold), silently drop it.

# step 4 of a pruned query
WITH neighbor, vector.similarity.cosine(neighbor.embedding, $promptVector) AS relevance
WHERE relevance > 0.55
RETURN neighbor ORDER BY relevance DESC LIMIT $topK

Instead of 5,000 Microsoft connections, you return the ~5–10 paths that actually align with "competitor."

Wrapped in Python, a "surgically pruned" retriever guarantees the LLM receives at most ~10 curated, semantically-relevant paths no matter how massive the graph, and returns "no highly relevant relational paths found" when nothing clears the threshold.

Hands-on: the Trinity Router#

A production-grade agent that routes each question to one of three engines, SQL, document vector search, or the live graph traversal, and lets the model chain them to answer.

The demo build (called Trinity Router because it routes to three tools) binds three functions to Gemini 2.5 Flash and runs an agentic loop. Two of the tools are mock stubs; the third is the real pruned graph retriever from above.

Why a loop, not a single call?#

Because Gemini 2.5 Flash runs in thinking mode and may chain tool calls, e.g. run the graph traversal, then a follow-up document search, then synthesize. The loop lets the model refine its strategy mid-flight. It's capped at five rounds (MAX_TOOL_ROUNDS) so it can't run forever.

On the canonical stress-test query, the router fires the graph traversal, walks from the Microsoft acquisition to the competitor to its CEO, and answers, in the demo, "Tim Cook leads Neural Net Labs, the AI company that competes with Alpha." [!] [+] Note: this is illustrative mock data from a toy graph, not a real-world fact; treat the names as placeholders for the mechanism.

Topic 2 · Declarative Optimization

Why Prompt Engineering Doesn't Scale, Enter DSPy#

Prompt engineering is a "vibes-based" hack#

Hand-writing prompts is unscientific tuning that's brittle, model-specific, and doesn't behave like real software engineering.

When you write a giant prompt telling the model exactly how to think, behave, and format, you're really hard-coding the system to one model's tokenization, biases, and attention mechanism. It works, until something changes. Three failure modes make manual prompting unscalable for production:

1. Model drift / latent-space shift. A prompt tuned for Gemini fails on Claude 3.5 Sonnet because each model represents concepts in a slightly different latent space. "You are an elite extraction AI" may trigger analytical output on one model and markdown-heavy filler on another.
2. Vibes-based debugging loop. Fixing a broken prompt means sitting in a playground, CAPITALIZING words, adding "seriously, do NOT do this," and tweaking few-shot examples until it "feels" right. That's not engineering.
3. Context-sensitivity trap. A prompt that works on 100 documents breaks the moment a slightly differently-formatted document arrives, the model's attention shifts, your instructions get drowned out by the new context, and the pipeline crashes.

What DSPy is#

DSPy is a compiler framework that treats your natural-language instructions and few-shot examples as tunable parameters, so an optimizer, not you, writes the prompts.

DSPy (Demonstrate, Search, Predict), created at Stanford, flips the paradigm. You stop hard-coding prompt strings. Instead you declare the pipeline logic (the architecture) and let an automated optimizer tune the instructions and few-shot examples as if they were mathematical weights.

The building block: a Signature [+]#

The lesson ended just as it introduced the first hands-on concept: the DSPy signature, a declarative spec of a task's inputs and outputs (e.g. question -> answer) that formalizes what a step does without you writing how to prompt for it. [+] A full hands-on with signatures is beyond the scope of this note.

Section 2 · Test yourself

Interview & Practice Q&A#

Click each question, then say the answer aloud before expanding.

Key Takeaways#