How my IT agent handled the eMudhra CA Keys compromise in 20 minutes

A certificate authority's keys broke and PHP pre-announced two critical bugs on the same day. My part of the response was two messages.

How my IT agent handled the eMudhra CA Keys compromise in 20 minutes

October 20th was one of those days. Twitter (I still call it that) had been buzzing since earlier in the day about two things at once. First, someone had broken the keys of a certificate authority trusted by every major browser and operating system. Second, the PHP security team had pre-announced two critical vulnerabilities, both rated CVSS 9.2, with fixes due the next day.

In my homelab, a day like that used to mean an evening of reading advisories, working out which of my two dozen machines and sixty-odd containers were affected, SSHing into each of them, and hoping I didn't miss one.

This time, I pasted a screenshot into a chat window and typed one question. Twenty minutes later the CA problem was fixed everywhere it could be fixed, and the fix was proven on every machine. Before the hour was up, the one internet-facing PHP application I run was updated too, and it had an automatic update pipeline it had never had before.

This post is about what that felt like from my side of the keyboard, which is to say: not much.

What I actually did

Here is my entire contribution to the CA response, verbatim:

9:20 PM: [screenshot] There was some kind of hack today with CA root certificates. Can you see what our exposure is, especially for WAN facing surfaces?

9:27 PM: Great, let's do 1, the rebuild part of 2 [...]

Two messages. Everything else in this post was done by my IT agent, Claude Code. It works from runbooks it's expected to read before touching anything, and it asks before it changes anything.

Seven minutes: from "some kind of hack" to a full assessment

I didn't know what had happened. The screenshot was just a table of software with "patched versions" next to each row: Go, Electron, Python's certifi, Rust's webpki-roots, NSS. Node.js was "pending".

The agent went and found out. It read the coordinating site and the Mozilla security-policy thread, and came back with the story:

  • Eight intermediate certificates belonging to eMudhra, a certificate authority, had private keys anyone could recover. They were P-384 keys generated in April 2025, and every one was a number smaller than 256. Anyone could "crack" them by trying 256 values.
  • Those intermediates chain to roots that Mozilla, Microsoft, Apple and Chrome all trust. With one of those keys, you can mint a certificate for any domain that most software will believe.
  • Browsers were safe, because they enforce Certificate Transparency and no CT log entries exist for these intermediates. Everything else (curl, Python, Go programs, Docker, package managers) trusts whatever its CA bundle says.

Then it told me, politely, that I'd asked the wrong question.

I'd asked about WAN-facing surfaces, the instinct you have when you hear "security incident". But a compromised CA doesn't attack servers. It lets someone impersonate other servers to your clients. My public sites use Let's Encrypt certificates (it checked), so nothing I serve was affected. The real exposure was every machine of mine that makes an outbound TLS connection.

Then it tested that, rather than reasoning about it. The coordinating site published a test endpoint, dv-g3b-emudhra.we-have-a-problem.com, which serves a certificate chain built on one of the compromised intermediates. If a client connects successfully, it trusts the bad chain. The agent ran that curl on all three Proxmox nodes and inside every running container: 19 machines.

Every one of them returned 200. Every machine I own trusted the compromised chain.

The fixed ca-certificates packages were already in the Ubuntu and Debian security repos. The agent downloaded the Ubuntu one without installing it and read its changelog, which named the root being distrusted: emSign ECC Root CA - G3. It used that root as a fingerprint for the next step, which I wouldn't have thought of: checking the CA bundle inside every Docker container on all seven Docker hosts. Container images ship their own CA bundles, frozen when they were built, and patching the host doesn't touch them. Nearly all of them still trusted the bad root.

The container CA-bundle scan: nearly every container still trusted the compromised root

Then came the part I value most, an honest risk assessment instead of a panic. To exploit this, an attacker has to sit on the network path between my machines and the internet. If someone managed that, the agent listed what would be at stake: container image pulls (a spoofed registry could serve a malicious image), and the secrets my services send over TLS (bot tokens, SMTP credentials, API keys). It explicitly ruled out apt: packages are GPG-signed, so a forged certificate can't sneak one in.

It finished with a short numbered plan, and asked before doing any of it.

Eight minutes: the fix, and the proof

I replied "let's do 1, the rebuild part of 2".

Every host was patched. ca-certificates was upgraded on all three nodes and all sixteen running containers. Then the agent re-ran the test on every machine, this time paired with a known-good site as a control:

moxxy-01 node   forged=REJECTED good=200
  CT101         forged=REJECTED good=200
  CT104         forged=REJECTED good=200
  ...
moxxy-03 node   forged=REJECTED good=200
  CT121         forged=REJECTED good=200

The control is what makes this proof rather than hope. "The forged site fails" could also mean "TLS is broken entirely", and requiring the good site to succeed at the same time rules that out.

Patching the hosts also closed the biggest real risk. The Docker daemon checks registries against the host's trust store, so image pulls could no longer be spoofed.

My own container images were rebuilt. I build four images myself: my alerting agent's three and a DDNS exporter. The agent rebuilt them from fresh bases, confirmed inside each one that Python's certifi was the patched release and that the forged chain was rejected, and redeployed them. At 9:35 my alerting agent's scheduled run completed successfully on the new image.

The images I don't build (Grafana, Prometheus, Plex and so on) get fixed as their maintainers rebuild them. My nightly image check picks those up on its own.

Meanwhile, PHP

Once the CA work was done, I mentioned the PHP pre-announcement and asked whether anything of mine serves PHP.

The agent checked every host and container for running PHP processes, then cross-referenced them against my reverse proxy's routing table to see what the internet can actually reach:

Where What Internet-facing?
Nextcloud (All-in-One) PHP 8.4.25 Yes, through my reverse proxy
TasmoAdmin PHP-FPM No, LAN only
Pi-hole leftover PHP packages No, nothing listening

So there was exactly one thing to worry about, and it had a problem: Nextcloud wasn't set to update itself. Every time I'd tried to turn on automatic updates, Nextcloud All-in-One refused unless I also turned on its built-in backup, and I already back the whole container up nightly with Proxmox Backup Server.

I asked the agent if it could help. It read Nextcloud's own update script and found it already supports updating without the built-in backup. The web interface just never offers that combination. So it built a nightly job around it:

  • It updates only if Proxmox has a backup of Nextcloud less than three days old. If it can't confirm that, it doesn't update. This is the same rule the rest of my auto-updates follow: the question isn't "is this update safe?" but "can I recover if it isn't?".
  • It pulls new images first and does nothing if none changed, so Nextcloud doesn't restart every night for no reason.
  • It checks the result through the reverse proxy, the same way a user would reach it, and it raises an alert if an update fails or the job stops running.
  • Old images get cleaned up. I asked whether they'd pile up. They would have, at about 5 GB per Nextcloud release, so it added Nextcloud to my weekly image cleanup.

Then I asked it to run the first update while I watched. All eleven Nextcloud images updated with about three minutes of downtime, PHP went from 8.4.25 to 8.4.26, and every container came back healthy.

The PHP security release itself was due the next morning, a few hours after that night's update run, so the nightly job picks it up the following night, without me.

The rest of the questions

A security day generates follow-up worries, and I just asked them as they came up:

  • "I am concerned about my vpn, though. I remote in on my phone, so it's outward facing." WireGuard doesn't use certificates at all, and there's no PHP on that box. It's one UDP port that doesn't even reply to packets without a valid key. Neither vulnerability applies.
  • "Let's check haos." (Home Assistant OS) It was two months behind on updates, and neither its OS nor its Core release notes mentioned a CA fix yet. That's on the watch list until Home Assistant ships one.
  • Node.js had no fix at all yet. That's noted and waiting.

What "effortless" actually means

I want to be precise here, because "AI fixed my security problem" is the kind of claim that deserves suspicion.

The work wasn't trivial. Testing nineteen machines against a live endpoint, scanning the trust store in sixty containers, rebuilding and verifying images, reading Nextcloud's update script and building a backup-gated pipeline around it: that's an evening of careful work for a person, and the kind of evening where you miss one container at 11 PM.

What made it effortless for me was where the decisions landed. I made the ones that were mine to make: "yes, patch them", "the VPN worries me", "turn on Nextcloud updates, but I already have backups". The agent did everything else, and showed its work. Every claim in its summaries came with the test that proved it: the forged-chain test re-run on every machine, the image versions read back after the rebuild, Nextcloud's health checked through the same path a user takes.

Twenty minutes for the CA compromise and under an hour including PHP. Most of my part was reading.