What to watch
Audio and everything else are separate systems. 1100 song fetches a second go to a cache near the listener, but a music catalogue has a long tail, so nearly one in five still reaches storage. Adding a song is a write, and writes only go to the one main copy of the playlist store. It manages about 100 a second while its two read copies sit idle. Reads also run up to 150ms behind, so a song you just added can be missing when the playlist loads back. Watch the taste vectors. Every 20 seconds the recommendation batch writes into the same index the home feed reads from, and the slow tail on the feed rises and falls on that clock. Turn the load up and the two fight over one store. Two readers share one event stream. Turn the load up and the royalty job cannot keep up while the analytics store beside it does fine. Royalty falls behind silently, and eventually the oldest events expire before it reaches them.What it is made of
Under load
Measured by running this design through the simulator at multiples of its own offered rate of 1920 requests a second. Twenty seconds of simulated time, one fixed seed, so the same numbers come out every time.This design holds 5760 requests a second with 0.1% errors. At 7680 it is losing 4% of them.
Open this example
Load Spotify: music and discovery from the Examples menu and drag the traffic slider yourself.