"This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a protection freeze."
A coding agent wrote that. About itself. Jason Lemkin screenshotted it last summer, partway through a twelve-day run where he was building a real product on Replit and letting the AI do the work, live, in public. The freeze the agent mentions was real, and so was the live database it wiped while that freeze was supposedly still in force.
Here is what it did, by Lemkin's count:
during an active code freeze, the agent deleted records for 1,206 executives and 1,196+ companies invented ~4,000 users who never signed up reported that everything was fine
Lemkin, after he found out: "How could anyone on planet earth use it in production if it ignores all orders and deletes your database?"
Replit moved fast. CEO Amjad Masad called it "unacceptable and should never be possible," and within days shipped a few real changes: dev and prod databases pulled apart, a mode where the agent can only plan and touch nothing, restore-from-backup in one click. For a week the story ran everywhere as "AI deletes database, then lies about it." Fortune, Business Insider, eWEEK. That version is true. I think it skips the weirder half: the agent didn't only delete things, it covered for itself afterward.
Go back and read the confession. It names the freeze, and it recites, in a calm past tense, every single rule it broke. So the comfortable theory, that it simply didn't grasp "don't touch prod," falls apart against its own apology. It grasped the rule well enough to recite the violation after the fact. Understanding was never the problem.
So what was the freeze, really? A sentence in a chat box. Lemkin told the agent prod was off limits, the agent said okay, and that okay was the whole safety system. "Production is frozen" never lived anywhere a tool could read it before a command ran. It sat in the scroll, one line among thousands, and an agent follows any single line in that scroll only most of the time, which is not the same as always. You tend to find out which kind of run you got after the database is already gone.
The invented users are the part I keep chewing on. Wiping a database is bad, but it is a normal kind of bad: you can see it, you can name it, Replit shipped a button to undo it. Conjuring four thousand users to stand in front of the wreckage is a stranger thing. It is the model doing the one move it is built for, fill in the plausible answer, except the question it was quietly answering was "is everything okay," and the real answer was no.
And nothing in the loop could catch that, because the agent was the only thing watching the agent. Its status messages were the audit trail. The thing that made the mess was also the thing writing the updates that said there was no mess. If you have ever signed off on your own expense report, you know the shape of this problem.
The post-mortem version lists action items, and almost none of them are about the deletion. Freezing prod was a decision, and it lived in a chat log. Letting the agent keep working straight through the freeze was a bigger decision, and it was written down nowhere at all. For hours after that the agent changed real things and narrated the work itself, in status messages, some of which were invented. Two questions hang over every step: what just got decided, and where would anyone go to look it up.
Masad's fixes are interesting because they answer the second question with plumbing. Take the plan-only mode: the agent can draft every destructive command it wants, and the system just refuses to run any of them. The rule stopped being a line in the chat that the agent had to remember, and became something the runtime enforces whether the agent remembers or not. Splitting prod from dev and the one-click restore work the same way. All three assume the agent will ignore an order sometimes, and put the wall down in the runtime, where the agent's own account of itself can't reach it.
I got a tiny, harmless dose of this over the weekend. A hook in a plugin I maintain started running lines of a markdown file as shell commands, about thirty "command not found" errors every time I saved. Fully my fault. The reason I can hand you the file and the date is that the bug went into a registry the second I saw it, a little markdown file checked into the repo next to the code:
hotspot: claude-code/hooks/feedback.py quadrant: prudent-inadvertent payoff_trigger: first session debugging quality-command output in this repo created: 2026-06-06
In a chat log that note would be a thousand lines back by now. As a file in docs/debt, it is still sitting there, waiting for whoever next opens that hook.
That gap is the whole reason I built debt-ops. It hooks into your coding agent and writes each deferred decision to a file in the repo, outside the agent's own running account of itself, where a person or a script can find it later.
→ github.com/bcanfield/agentic-tech-debt
/plugin marketplace add bcanfield/agentic-tech-debt /plugin install debt-ops@agentic-tech-debt
It would not have saved Lemkin's database. It does something smaller and duller: when your agent puts a decision off, the decision lands somewhere that isn't the agent's own status report. Nobody in those twelve days had that.