--- title: 'The Day Our App DDoSed Itself' excerpt: 'A backlog of pending uploads, released all at once on startup, turned every GeekBye client into a small denial-of-service attack on our own servers. The fix — and the connection-liveness ladder it forced us to build — is one of the most useful things v2 taught us.' date: '2026-07-04' author: 'Steven' authorAvatar: '/images/blog/authors/steven.jpg' coverImage: '/images/blog/covers/the-day-our-app-ddosed-itself.svg' tags: ['Reliability', 'Networking', 'Engineering', 'GeekBye Releases'] keywords: - 'app overwhelms own server on startup' - 'client side self ddos backlog' - 'rate limit 429 backoff retry' - 'websocket liveness heartbeat reconnect' - 'startup thundering herd' - 'AI notetaker connection reliability' lastModified: '2026-07-04' tldr: 'GeekBye v2.0.1 fixed a self-inflicted denial of service: a queue of pending uploads flushed all at once on launch, hammering our own backend into rate-limiting itself. v2.0.4 followed with a connection-liveness ladder so a dead connection is detected in seconds, not minutes. Both are the boring reliability work that makes an app feel calm — plus a live example from this week of the same failure class, one tier down.' keyTakeaways: - 'A backlog of uploads released all at once on startup turned our own clients into a thundering herd against our backend' - 'The backend rate-limited the flood — correctly — which then risked logging users out and showing false upgrade prompts' - 'The fix paces the backlog with backoff and cooldowns, and hardens the client so a transient 429 is never mistaken for a real auth or quota failure' - 'A follow-up release added a connection-liveness ladder: heartbeats, a network-up reconnect kick, and control ping/pong so a dead connection is caught in seconds' - "The same failure class — a client stampeding shared infrastructure — showed up again this week one tier down, at our speech provider's concurrency limit" --- Every distributed-systems engineer eventually meets the thundering herd: a mass of clients all doing the same thing at the same instant, and the shared resource behind them buckling. Usually it's someone else's clients. In [GeekBye v2.0.1](https://github.com/aiescu/geek-bye-releases/releases/tag/v2.0.1) it was ours, and the attacker was our own app. ## How a notetaker attacks itself GeekBye can upload recordings to your Google Drive. If a recording can't upload immediately — you were offline, the app closed, Drive hiccuped — it goes into a backlog to retry later. Sensible. The failure was in _when_ "later" happened. On the next launch, the app would try to drain the whole backlog **at once**. One user with a week of pending recordings became a burst of simultaneous upload requests the instant they opened the app. Multiply by every user launching in the morning, and our backend was being asked to authenticate, rate-check, and process a wall of traffic in the same few seconds — from clients that were all, technically, behaving. Our backend did the right thing and rate-limited the flood. And that's where the second-order damage started. A rate-limit response is an HTTP `429`, and `429` looks a lot like other failures if you're not careful: - The startup **profile fetch** got a `429` and the app treated it as _auth failed_ — briefly logging users out of a perfectly valid session. - The **transcription connection** got a generic `429` and showed an _audio-limit-reached upgrade prompt_ — telling paying users they'd hit a quota they hadn't. So one root cause — an unpaced backlog — produced three visible symptoms: server strain, false logouts, and false upsells. The classic shape of a self-DoS: the load is bad, but the _misinterpretation_ of the load's symptoms is what users actually feel. ## The fix, in two layers **Stop the stampede.** The upload backlog is now paced — requests spread out with backoff, and a cooldown after any `429` so the client backs away from a strained server instead of leaning harder on it. A thundering herd becomes an orderly queue. **Stop the misinterpretation.** A transient `429` or `5xx` during startup no longer logs you out — the client distinguishes "the server is briefly busy" from "your session is invalid." And a generic rate-limit `429` on the transcription path no longer masquerades as an audio-quota error; only a genuine quota response shows the upgrade prompt. The lesson that stuck: **an error code is not an error meaning.** `429` means "slow down," not "you're unauthorized" and not "you're out of quota" — and the client has to know the difference. ## The liveness ladder that followed Pacing the herd exposed a quieter problem: when a connection _did_ go bad under load, how fast did we notice? Slower than we wanted. So [v2.0.4](https://github.com/aiescu/geek-bye-releases/releases/tag/v2.0.4) built out a connection-liveness ladder for real-time transcription: - **Heartbeats** — the transcription socket is pinged on a fixed interval, so a silently-dead connection is detected in seconds instead of waiting for the next chunk of audio to fail. - **A network-up reconnect kick** — when the OS reports the network is back, the app proactively re-establishes the connection instead of waiting to stumble into the discovery. - **Control ping/pong liveness** — an application-level round-trip that confirms not just "the socket is open" but "the other end is actually answering." - **A service-owned reconnect gate** — one place decides whether to reconnect, instead of several parts of the app racing to do it and stepping on each other. None of this is glamorous. All of it is why a v2 session feels like it just... stays connected. ## It happened again this week — one tier down Here's the part that makes this more than a war story. This same failure class resurfaced days ago, one level below us. A single client fired **21 transcription-session starts in under 400 milliseconds** — and tripped our speech provider's account-wide limit of 15 concurrent requests. A stampede against shared infrastructure, exactly like the upload backlog, just aimed at a dependency instead of our own backend. The shape is identical, and so is the fix: the stampeding client needs a single-flight guard so it can't fire twenty starts for one intent, and the shared resource needs a per-user cap so one client can't consume the pool. We've built the client-side version of this before — the upload pacer _is_ this pattern. Now we apply it to session starts. The thundering herd is not a bug you fix once; it's a shape you learn to recognize everywhere clients meet a shared limit. ## Three things to take away 1. **Your own clients are a load test you didn't schedule.** Anything that batches-on-startup, retries-on-launch, or reconnects-on-wake is a thundering herd waiting for enough users. Pace it before you have them. 2. **Distinguish the code from the meaning.** `429` is the most misread status in existence. "Slow down" is not "log out" and not "pay us." Route each failure to what it actually means. 3. **Liveness is a ladder, not a flag.** "Is the connection alive?" has several honest answers at different layers — socket open, bytes flowing, other end answering, network present. A robust app checks more than one. This is the unglamorous machinery under GeekBye v2's calm — the same discipline that drove the whole rewrite in [what a version 2 actually takes: 206 commits of honest states](/blog/what-a-version-2-actually-takes-206-commits) (v2.0.0). For the reliability features it enabled, see [why your AI notetaker stops on bad Wi-Fi](/blog/reliable-transcription-bad-wifi) and [live transcription when the firewall blocks WebSockets](/blog/live-transcription-when-firewall-blocks-websockets) (v2.0.8). For neighboring releases in this series, [why your AI notetaker stops recording mid-meeting](/blog/why-ai-notetaker-stops-recording-mid-meeting) (v2.0.9). For the v1-era groundwork where a dropped connection first stopped crashing the app, [a dropped connection shouldn't crash your whole app](/blog/a-dropped-connection-shouldnt-crash-your-app) (v1.6.8).