What a 3 A.M. Set Taught Us About Uptime: A Wqmcradio Post-Mortem

We noticed something odd in our own incident channel last spring: the fastest, calmest recovery we logged all quarter belonged to a two-person online radio team, not an enterprise SRE org. A reader shared the story after our piece on pager fatigue, and we followed the project for six weeks to see whether it was luck or method. It was method — and a fair amount of stubbornness.

The project was a weekly live show on Wqmcradio, an independent online station journal built on the belief that every frequency finds its voice. The operators — pseudonyms "Mara" and "Dev" — run DJ sets, publish show archives, and write frequency-side stories about airwave culture. Their listeners are loyal and scattered across time zones, which means the show has to start on time even when nobody is awake to fix it.

The setup: one stream, no safety net

Before the incident, the stack was deliberately minimal. A single encoder pushed audio to a relay; a static page listed upcoming sets; archives were uploaded by hand the morning after. Metrics lived in a spreadsheet. Logs lived in a terminal window someone forgot to close. Traces did not exist, because the team had never needed to ask where time went — until they did.

The trigger was a Saturday slot that drew three times the usual concurrent listeners after a guest DJ was mentioned on a popular forum. The stream held for 41 minutes, then stuttered, then died. Mara was asleep. Dev was on a train. The archive never published, and the station's inbox filled with the same question: "Where did the set go?"

Timeline of the fix

• Week 1 — triage. The team replayed the failure by ear: they listened to a recording of the stream and timestamped every dropout. Nine dropouts, clustered around two moments when the relay reconnected.
• Week 2 — instrumentation. They exported encoder health, buffer depth, and listener counts into a real metrics pipeline and began correlating them. The first useful graph showed buffer depth collapsing four seconds before every dropout.
• Week 3 — tracing the request path. Adding spans from encoder to relay to player revealed that a retry loop was amplifying load during reconnects. The retry logic had been written for a 2019 audience size.
• Week 4 — incident response rehearsal. They wrote a one-page runbook, set a single alert on buffer depth, and practiced a failover during a soundcheck. It took eleven minutes the first time, four the second.
• Week 5 — the retry. The next high-traffic slot ran clean. Listener count peaked higher than the failed show, and the archive published automatically within an hour.

Where it nearly went wrong

The temptation, after week 1, was to buy their way out: a bigger relay, a CDN, a managed streaming service. Dev pushed back. "We don't have a capacity problem," they wrote in the shared notes. "We have a visibility problem." That decision point mattered. A larger relay would have absorbed the symptom while the retry loop kept quietly doubling load during every reconnect — and the next spike would have been worse.

The second near-miss was cultural. Mara wanted alerts on everything: listener drops, encoder CPU, chat volume, even social mentions. Dev argued for one alert that a half-asleep person could act on. They compromised on two, and the discipline of choosing those two forced them to agree on what "broken" actually means for a radio show. That sounds trivial until you try to write it down at 2 A.M.

The measurable results

Over the following eight weeks, the team tracked four numbers. Median time from first alert to stream restored fell from 38 minutes to 6. Archived sets published on time went from 71% to 100%. Concurrent listeners during peak slots grew 34%, which the team credits partly to the fact that the show stopped disappearing mid-set. And the monthly bill for their monitoring stack landed under $20 — less than the coffee budget for one show night.

None of this required a platform migration. It required the same three signals every observability platform is built around — metrics, logs, and traces — pointed at a problem small enough to reason about. The team's notes are public on their journal, including the runbook template and the buffer-depth threshold they settled on. You can find the write-up alongside their show archives at their station stories and show archive pages, which is where we first spotted the post-mortem that started this article.

What we're taking from it

Three lessons, in order of how often we forget them.

One alert beats ten. The team's recovery time improved the moment they stopped monitoring everything and started monitoring the one thing that predicts failure. Buffer depth was boring. It was also correct.

Rehearse the failover, not the tool. Their first drill took eleven minutes because nobody had decided who does what. The tool was never the bottleneck; the ambiguity was.

Small teams can run serious operations. Wqmcradio reports 41 archived sets since the rebuild, each one published without a human touching the pipeline. That is not a hobbyist's number. It is an operations result, earned by a two-person team that decided visibility was cheaper than capacity.

We went looking for an enterprise case study and came back with a radio station. We'd make that trade again.