Latest issue
Your Trace Tree Has Every Span and Still Can't Tell You Why the Answer Was Wrong
10/26/2026
Here's a scenario that should feel familiar if you've run a RAG pipeline past a demo: the vector database answers in 40 milliseconds, the model returns a 200, every service in the chain reports healthy, and the user still gets a confident answer built from last year's policy docu…
Recent posts
Picture a service pegged at 98% CPU at 2am. There's a runbook called Restart Service, and an AI agent has it loaded and ready to execute in seconds. The obvious move is to run it.…
A model returns syntactically perfect JSON. It matches your schema field for field, types checked, required keys present. And it's still wrong — an enum value in the wrong case, a…
A model upgrade lands on a Tuesday. The aggregate score on the regression suite comes back a point higher than last week. The team ships it the same afternoon. By Friday, support i…
Here's the failure mode that should scare you more than an outage: your eval suite goes green, you merge the prompt change, and three weeks later a support engineer finds out your…
The support ticket arrives at 11am on a Tuesday. A user ran your financial summary agent overnight. The output is beautifully formatted, confidently written, and built on a company…
You've got two prompt variants. You run them against a traffic split. Variant B wins on your quality metric. You ship it. Three days later, something unrelated starts degrading — a…
You've instrumented TTFT. You're tracking it at p99. You've even read the post I wrote about why p99 TTFT is the number that saves you at 2am. Good. Now here's the uncomfortable fo…
The support assistant analytics are damning in the best way. Out of roughly 400,000 distinct user queries, the top 500 phrasings account for 30% of volume, and another 25% clusters…
The support ticket comes in on a Tuesday. A user got a confident, well-formatted answer from your AI assistant — complete with a citation to a policy document that doesn't exist. Y…
The Cursor incident is the clearest case study in what happens when you price a flat plan on top of a cost structure that isn't flat. A developer posted a $7,225 invoice from a sin…
A team runs their LLM feature for a full quarter. The dashboard stays green — p50 latency at 1.9 seconds against a 2.5-second target. Then someone actually looks at churn data and…
A team ships semantic caching. Cache hit rates climb to 30%. Costs drop. Everyone's happy — until a user asks "what's our refund policy?" and gets an answer that was accurate six m…
The 3am call comes in. Your AI feature is returning 500s. You check the logs, confirm it's an OpenAI outage, and feel briefly smug — you built a fallback chain six months ago. Then…
The Monday morning bill shock is a rite of passage. You shipped the feature Friday, traffic came in over the weekend, and now you're staring at a number that's 3–5× what your back-…
Four seconds. That's how long a user stares at a blank screen before they decide your AI feature is broken. Not slow — broken. They don't know about prefill phases or KV cache tran…
The incident is always the same. A customer asks "where's my delivery?" and gets a confident, detailed answer — about someone else's order. Two users asked semantically similar que…
The April incidents should have been a wake-up call. A ten-hour Claude outage on April 6 and a major OpenAI platform outage on April 20 — both multi-hour, both affecting production…
The support ticket reads: "The AI feels slow and dumb lately." No stack trace. No error code. Just a user who noticed something your dashboards didn't. This is the failure mode tha…
The invoice arrives and the number is wrong — not wrong as in fraudulent, wrong as in useless. It tells you what you spent across the whole month. It doesn't tell you that one agen…
