How Spotify scaled its backend

The primary language for the large majority of Spotify's backend services, chosen for its mature tooling and performance characteristics across a very large number of independently run services.

Used across a number of Spotify's data engineering and internal tooling services, alongside Java, particularly where rapid iteration mattered more than raw runtime performance.

The destination of Spotify's multi-year migration off its own data centers - compute, storage, and managed services that Spotify's infrastructure teams no longer have to run and maintain themselves.

Used across a number of Spotify's services for data that needs to be written and read at high volume with availability prioritized over strict consistency, similar to its role at other large-scale streaming and social platforms.

Part of Spotify's event delivery system after its cloud migration, used to move events - like a play, a skip, a follow - between services asynchronously, replacing earlier in-house messaging infrastructure built for its own data centers.

Scio is a Scala API for Apache Beam that Spotify built and open-sourced, used to write the large-scale data processing pipelines - like the ones behind Spotify's personalized playlists - that need to run the same logic over both real-time streams and historical batch data.

Spotify's backend services run as containers, packaged consistently regardless of which underlying language or team built them, which is what made a uniform deployment and scheduling story across hundreds of independently owned services possible in the first place.

Following the move to Google Cloud, Spotify runs much of its service fleet on GKE, scheduling and scaling containers across the cloud infrastructure it migrated onto rather than infrastructure it operates itself.