Stream Processing
IoT sensors, social apps, and machines emit data endlessly. Stream processing is the pipeline that collects that flood, standardizes it, analyzes it, and shows it — deciding along the way whether to answer fast or answer accurately.
Assumes you know
After this you should be able to
- Explain the core idea in Stream Processing
- Apply Stream Processing in a representative design
IoT sensors, social apps, and machines emit data endlessly. Stream processing is the pipeline that collects that flood, standardizes it, analyzes it, and shows it — deciding along the way whether to answer fast or answer accurately.
The chapter follows the life of a data point: why it's generated (data-driven systems) → how it's collected and standardized (ingestion and its layers) → the fundamental fork of real-time vs. batch and why ingestion is hard → where it's used → the pipelines and ETL that move it → the distributed engines that crunch it → and finally the two architectures (Lambda, Kappa) that decide how fast vs. how accurate the answer is. Anything marked [+] is context I added; everything else is rewritten from the eight teaching lessons.
Everything now emits data — constantly#
The world is increasingly data-driven, and the Internet of Things (IoT) has given machines a degree of self-awareness — they generate and transmit data online at an unprecedented rate, even making decisions without human intervention. The main sources: industrial and smart-city sensors, drones, smartwatches, phones, and wearable health sensors.
Businesses depend on this flood. Customer data drives plans and projections; understanding user needs and behavior lets companies build better products, make smarter decisions, run sharper ad campaigns, recommend products, and read the market — which loops back into more customer-centric products and loyalty. A second use is tracking service health: the steady "everything is okay" signal from millions of deployed IoT devices tells a business its products are up and functioning. To handle all of it, you need a backend sophisticated enough to gather the meaningful data and archive or purge the rest — and the more data you study, the better your systems evolve.
Ingestion: collecting the flood and making it usable#
Data ingestion is the whole process of collecting data streaming in from many sources and getting it ready for the system to process. Data is pulled from IoT devices and other sources, routed through pipelines to different layers, run through algorithms, and eventually archived. A processing setup has several layers:
- Collection & standardization. Data arrives heterogeneous — different speeds, sizes, and semantics from web services, social networks, IoT, and industrial machines. It's collected and converted to a uniform, standardized format in the collection/preparation layer to avoid downstream processing issues.
- Processing. The standardized data is classified into different flows and routed to different destinations per business requirements.
- Analytics. Models run on the data — predictive modeling, statistical analytics, text analytics.
- Visualization. The resulting intel is presented to stakeholders, usually in a web dashboard. Kibana is a widely used example.
- Storage & security. The storage layer persists data; the security layer keeps it safe as it moves (moving data is highly vulnerable to breaches).
Real-time or batch — and why ingestion is hard#
There are two ways to ingest data — in real-time, or in batches at regular intervals — and the business need decides which.
| Approach | Best for | Trade-off |
|---|---|---|
| Real-time | When time, lives, or money are on the line — medical data from wearables (heartbeat, blood pressure), stock-market events. | Analytics runs on only a limited slice of data, so results are less accurate/holistic. |
| Batch | Trends over time where instant data isn't critical — e.g. a sport's popularity in a region over a period. | Takes longer, but the entire dataset is considered, so results are more accurate and comprehensive. |
Fast vs. accurate. Real-time gives quick insight on partial data; batch gives comprehensive insight but slowly. This exact trade-off is what the Lambda and Kappa architectures (Section 07) are built to resolve.
Why ingesting massive data is genuinely painful
- It's slow. Data from each source has a different format, syntax, and metadata — it's heterogeneous, and must be transformed into a standard format (like JSON) the analytics system can understand. That conversion is tedious and resource-heavy; data is staged, processed, authenticated, and verified at every step. Traditional cleansing (legacy ETL ) could take weeks or months and couldn't even do real-time.
- It's complex and expensive. The whole flow is resource-intensive and needs a dedicated team. Off-the-shelf tools often fall short, forcing custom builds — LinkedIn had 15 ingestion pipelines and wrote its own tool, Gobblin, in-house to manage them. And because external data sources aren't under your control, their semantics can change and break your backend code.
- Moving data is risky. Every hop through a staging area is a breach opportunity, so teams spend extra effort meeting security standards throughout.
Real-time results aren't as accurate or holistic as batch, because they run continuously over a limited data slice. Batch, by taking the whole dataset, is more accurate — the more time spent studying the data, the better the results.
Where streaming actually gets used#
Data pipelines — and how they beat old ETL#
Data pipelines are the core of a data-processing infrastructure: they move data efficiently from point to point and let developers apply filters and business logic to the stream in real time. They run on rules predefined by the engineering team, so the entire flow — extraction, transformation, combination, validation, and the convergence of multiple streams into one — is automated with no manual intervention. What they give you:
- Smooth data flow, with no bottlenecks or redundancy.
- Filters and business logic applied to streaming data.
- Parallel processing by managing multiple streams at once.
- Protection against data corruption.
ETL, and why pipelines surpass it
ETL = Extract, Transform, Load. Extract pulls data from one or many sources; Transform converts that heterogeneous data into a standardized format per business rules; Load moves the transformed data into a warehouse or storage for further processing. The ETL flow mirrors data ingestion — the key difference is that ETL moves data in batches, whereas pipelines can stream it in real time, which legacy ETL couldn't do.
Real-time doesn't make batch obsolete — companies use both to get the best of each (the Lambda/Kappa payoff ahead). The popular tools — Apache Flink, Storm, Spark, Kafka — all share one trait: they process data in a cluster, in a distributed environment, via pipelines.
Distributed data processing: divide and conquer#
Distributed data processing diverges large amounts of data to many nodes running in a cluster for parallel processing. The nodes execute their allotted tasks in parallel, coordinated by a node-coordinator — Apache ZooKeeper is the common example. Because work is parallel and distributed, the setup is scalable (horizontally and vertically) and highly available, with data replicated across the cluster to prevent loss. The speed win is structural: a distributed system splits tasks across nodes, whereas a centralized system queues them to be processed one by one.
| Technology | What it is & best use |
|---|---|
| MapReduce / Hadoop | A programming model for distributed processing: map sorts data by a parameter, reduce summarizes it. Apache Hadoop is its popular open-source implementation — used by Twitter for analytics and Facebook for storing big data. Best for batch. |
| Apache Spark | Open-source cluster-computing framework; high performance for both batch and real-time in-stream processing. Has a cluster manager (node communication) and distributed storage; integrates with Cassandra, HDFS, MapReduce FS, Amazon S3. |
| Apache Storm | Distributed stream -processing framework for massive streaming data — real-time analytics, ML, distributed RPC. |
| Apache Kafka | Open-source distributed stream-processing and messaging platform (Java/Scala, built by LinkedIn). Its storage layer is a distributed, scalable pub-sub message queue. Used for notifications, stream management, activity/metrics monitoring, messaging, log aggregation. |
The rule of thumb: Hadoop for batch; Spark, Kafka, and Storm for real-time streaming.
Lambda & Kappa: resolving fast-vs-accurate#
Lambda — run both, then merge
Lambda is a distributed data-processing architecture that uses both batch and real-time streaming, then joins their results — beating the latency of batch-only while keeping its accuracy. Batch is slow but comprehensive and accurate; real-time is fast but runs on a small slice, so less accurate. Lambda takes the best of each via three layers:
- Batch layer — results from batch processing (accurate, comprehensive).
- Speed layer — results from real-time streaming (fast).
- Serving layer — combines the batch and speed results before presenting them.
Kappa — one pipeline for everything
In Kappa, all data flows through a single streaming pipeline — unlike Lambda's separate layers that converge. Both real-time and batch processing run through that one pipeline, cutting the complexity of managing separate layers. Kappa has just two layers: Speed (the stream-processing layer) and Serving (the final layer).
Both have their use cases. Pick Kappa when the batch and streaming analytics results are fairly identical (one pipeline suffices). Pick Lambda when they differ and you need both a fast approximate answer and a slow accurate one.
Recap#
Stream processing is one long negotiation between speed and accuracy. Every choice in the chapter — real-time vs. batch ingestion, pipelines vs. ETL, Lambda vs. Kappa — is really answering the same question: how fresh does the answer need to be, and how much accuracy will you trade to get it that fast? Lambda's refusal to choose (run both, merge) is the tell: when you can afford it, you buy speed and accuracy by paying for two paths.
Further reading#
The lessons cite two external links. One resolves live; the other blocked automated access and is omitted (noted in the footer).