How Netflix built a real-time graph of what its members are doing

Netflix runs on a microservices architecture - hundreds of small, independent services, each owned by its own team, each with its own database. That's genuinely useful for building and scaling software: teams can ship independently, and each service can pick the database that actually fits its job.

But it creates a side effect nobody wanted: a member's activity ends up scattered across dozens of separate, siloed databases that were never designed to talk to each other. Figuring out that a phone login, a TV viewing session, and a tablet game session all belonged to the same continuous story meant manually stitching data together from a data warehouse and a pile of unrelated databases - slow, error-prone, and not remotely close to real time.

Netflix's data team could have tried to solve this by building bigger, more connected tables. They chose a graph model instead, for three concrete reasons worth understanding on their own.

First, relationship queries are what a graph is actually built for - hopping from 'this member' to 'shows they watched' to 'games related to those shows' is a fast traversal in a graph, where the same question in a table-based system usually means several expensive joins.

Second, graphs tolerate change well. As Netflix added new kinds of member activity (ad interactions, live events, games), a graph can absorb new node and edge types without the kind of schema rework a rigid table structure would need.

Third, a lot of what Netflix's data scientists actually want to find - hidden relationships, repeating patterns, unusual clusters of behavior - is a much more natural fit for graph traversal than for point lookups scattered across separate databases.

Every action a member takes in the app - logging in, pressing play, finishing an episode - gets published as an event to Apache Kafka, a system built specifically for durable, high-throughput event streaming. Kafka's job here is simple to state and hard to do well at this scale: take an enormous number of small messages arriving continuously, and make them reliably available for other systems to read, in order, without losing any.

To put the scale in perspective: Netflix's team describes individual Kafka topics generating up to roughly a million messages a second. Each event is encoded in a compact binary format called Avro, with the exact shape of each event type tracked centrally in a schema registry, so every service reading the data knows exactly what fields to expect.

Kafka's job is delivering the raw events reliably. Turning those raw events into actual graph nodes and edges is a separate job, handled by Apache Flink - a stream processing framework built for exactly this kind of continuous, low-latency transformation.

The pipeline, in plain terms: a Flink job reads a stream of raw events (say, 'member started watching an episode'), filters out noise, enriches the event with extra context it needs, and transforms it into graph primitives - a node representing the member, a node representing the show, and an edge connecting them representing the 'watched' relationship. Along the way, it also deduplicates near-identical updates that arrive in a short window, so the system isn't wastefully re-writing the same relationship five times because of five closely-spaced events.

Once that transformation is done, the resulting nodes and edges get published downstream to a system Netflix calls Data Mesh, which handles actually persisting them into the storage layer other services can query. Netflix's own numbers describe this pipeline writing more than 5 million node-and-edge records per second at peak.

This is the most instructive part of the whole story for anyone learning system design, because Netflix's team openly describes trying the simpler approach first, and it not working.

Their first attempt used one single Flink job to consume every Kafka source topic. It sounded reasonable - one job, one thing to manage - but different topics have wildly different volumes and traffic patterns throughout the day. Tuning one shared job to handle all of them at once turned into a losing battle over CPU, memory, and parallelism settings that never quite fit every topic simultaneously.

Their fix was to flip the design: one Flink job per Kafka topic, a strict 1:1 mapping, instead of one job trying to do everything. That meant more individual jobs to build, deploy, and monitor - genuinely more operational overhead. But each individual job became dramatically simpler to reason about and tune, because it only had to handle the traffic shape of exactly one topic. This is a specific, real example of a very general lesson: sometimes fewer moving parts is not actually simpler if the one big part now has to handle wildly different situations at once - splitting it apart can trade a little extra operational surface area for a lot less complexity per piece.