Discussion
April 2026 Outage Post-Mortem
threecheese: > What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request.That’ll do it.
98codes: Ahh, the three relevant numbers in development: 0, 1, and infinity.
templar_snow: Communism confirmed for never working once
lavela: Why?
goekjclo: > The timing of these log spikes lined up with drops in user-facing traffic, which makes sense. Our data plane heavily uses memcached to keep load off our main Scylla database, and if we're exhausting ports, that's a huge problem.I expect this is common.
rvz: Thank you for the post mortem on this outage.
jonstaab: nostr never goes down
pfraze: All support to other decentralizers but nothing never goes down.
electrondood: Great write up... curious about the RCA. Thanks!
