Five Things that Break OTT Services at Peak Load, and What to Test Before Kickoff

The failures that take down a live stream at peak concurrency are not the ones you see in normal operation. They are the ones that only appear when a few million viewers arrive in the same ninety seconds, which is also the one moment you cannot afford them. By then the test window is closed.

Here are five failure patterns we have seen repeatedly across large live events, each with a test you can run before kickoff, the fix that usually applies, and the diagnostic that tells you whether the thing is actually optimised or quietly broken.

1. Origin manifest update spikes

What breaks: at low latency, manifests update constantly, and every player refreshes them on a tight cadence. When concurrency jumps, the manifest request rate to the origin can spike faster than the segment request rate, and the origin starts serving stale or slow manifests. Viewers see stalling that the CDN dashboards do not explain, because the segments are fine. The manifests are late.

Test before kickoff: load-test manifest requests separately from segment requests, at the refresh interval your live config actually uses, scaled to projected peak. Most synthetic load tests hammer segments and under-test manifests, which is backwards for low latency.

Fix: cache manifests at the edge with a short, correct TTL, and confirm the origin shields are absorbing manifest refreshes rather than passing them through.

Diagnostic: watch origin manifest latency at the 99th percentile, not the mean. If p99 manifest latency climbs while segment latency stays flat as concurrency rises, the manifest path is your bottleneck.

2. ABR degradation under shallow low-latency buffers

What breaks: low-latency streaming runs the player on a much shallower buffer, which leaves the adaptive bitrate algorithm less room to absorb network variance. Under congestion, ABR logic tuned for a deep buffer overreacts, oscillates between renditions, or collapses to the bottom of the ladder and stays there. Viewers get a stream that is technically playing and visibly bad.

Test before kickoff: run your real player matrix under simulated network impairment, packet loss and jitter, at low-latency buffer settings rather than defaults. The behaviour you care about only shows up when the buffer is shallow.

Fix: tune ABR parameters specifically for the low-latency profile, and validate per player implementation. The right settings for Shaka Player are not the right settings for ExoPlayer.

Diagnostic: measure rendition switch frequency and time-at-lowest-rendition during impairment. A healthy low-latency ABR holds a stable mid-ladder rendition. A broken one flaps or floors.

3. Ad-marker drift and replay desync

What breaks: SCTE-35 ad markers slightly misaligned with segment boundaries cause ad breaks to start a beat early or late, and on replay or start-over the drift compounds until the ad insertion desynchronises from the content. The live stream looks fine. The start-over and catch-up versions are subtly broken, which is the version a large share of the audience watches.

Test before kickoff: validate ad-marker alignment against segment boundaries across live, start-over, and catch-up renditions, not live alone. The desync is a replay problem, so it has to be tested on replay.

Fix: align splice points to segment boundaries at the packager, and confirm the markers survive the packaging path intact.

Diagnostic: compare marker timestamps to segment-boundary timestamps. Any non-zero drift that grows across a replay window will surface as a visible ad-insertion fault for catch-up viewers.

4. Regional CDN cache exhaustion

What breaks: concurrent viewers for a live event cluster geographically and temporally, so a regional cache can fill and start evicting hot content precisely when every viewer in that region wants the same few segments. The cache hit ratio collapses, requests fall through to the origin, and a region that looked fine at average load browns out at peak.

Test before kickoff: model cache fill for the regions you expect to peak, using realistic concurrency clustering rather than evenly distributed synthetic load. The failure is regional, so a global average hides it.

Fix: size regional cache and origin-shield capacity for clustered peak, not average throughput, and confirm hot content stays resident through the peak window.

Diagnostic: track regional cache hit ratio under rising concurrency. If hit ratio falls while origin egress rises in a specific region, that region is exhausting its cache.

5. STB re-buffer storms after a network blip

What breaks: a brief network interruption causes a population of set-top boxes to re-buffer and reconnect at the same instant. They all request the same segments at the same moment, producing a synchronised demand spike that can be larger than steady-state peak and that hits the origin in a thundering-herd pattern. The original blip was minor. The recovery takes the service down.

Test before kickoff: simulate a synchronised reconnect across a representative device population and measure the recovery demand spike, not just steady state. The danger is in the recovery, not the outage.

Fix: implement jittered reconnect and request back-off on the client where you control it, and confirm origin shields can absorb a synchronised spike. Headend input redundancy reduces how often the blip happens in the first place.

Diagnostic: measure the ratio of recovery-spike request rate to steady-state peak. If a one-second simulated blip produces a recovery spike well above your provisioned peak, you have a thundering-herd risk.

---

The redundancy that prevents several of these at the source

Most of the five failures are made worse by upstream instability, so the cheapest mitigation is often to stop the instability at the headend. A 1+1 active/active statistical multiplexer keeps the encode and mux stage running through a component failure without a visible interruption. SMPTE 2022-7 input redundancy, which protects against the loss of an input feed by running two time-aligned copies, is on the NEA Live roadmap for the same reason: the fewer upstream blips reach the delivery path, the fewer reconnect storms and manifest spikes you have to absorb downstream.

One honest caveat about pre-event testing

No test rig reproduces a real audience exactly. Synthetic load misses the long tail of real device behaviour, real network paths, and the genuinely unpredictable ways a few million people interact with a live event. Pre-event testing has a realistic goal: find the failures that are reproducible in a lab, so the only ones left on match day are the ones that genuinely could not be found in advance. That is a much smaller and more survivable set.

Landing thought

Every one of these failures is invisible at normal load and obvious at peak, which is why they get shipped and then discovered live. The operators who get through a major event cleanly are the ones who tested the peak-specific failure modes on purpose, before kickoff, while there was still time to fix them. NEA Live and NEA CDN are built around these patterns because the patterns came from production events, but the discipline matters more than the product: test the failure that only happens at peak, before peak.

---

Explore related insights

---