Two scanners. One said nothing. One said there was a problem.
That’s where this started — and that gap is exactly the kind of thing that should make you curious rather than relieved.
The Setup
The container registry scanner — the main one, the one whose output the team relied on — flagged nothing. All clear. The secondary scanner, run as part of a broader policy check, flagged a mid-severity CVE from 2024. Old finding, mid-level severity, and the primary scanner wasn’t concerned. The natural read: false positive, secondary scanner being noisy, move on.
But security policy is security policy. The finding had to be resolved, not just dismissed. So the investigation began.
The first pass was obvious and clean. Docker base image — current. Tool versions inside the image — current. Nothing visibly outdated. The kind of check where you expect to find the problem in thirty seconds and don’t, which means you’ve already spent the easy thirty seconds and everything from here is harder.
Asking the AIs
This is 2026. When you’re stuck on something like this, you ask. I asked both IBM Bob and Claude — separately, first in general terms about the CVE, then with specifics: here is the image, here is the finding, please tell me if this image is actually affected.
Both did something I thought was reasonable: they wrote scripts to find the relevant binary on the filesystem, check its version, and compare it against the CVE’s fix date. That’s a defensible approach. Version of the binary, compare to patched version, done.
The scripts ran. The results came back. And here is where it got interesting in the wrong way.
Bob had found a library that looked old — but it predated the CVE entirely. Pre-vulnerability, not post-fix. Claude had found something slightly different and assumed it was older than the fix date, so it flagged it as affected. Two tools, two different answers.
So I did what seemed like the sensible thing: I had them check each other’s work. Bob reviewed Claude’s finding. Claude reviewed Bob’s. They went back and forth, cross-referenced their results, and arrived at a consensus. From a supervisory perspective it looked exactly right — two independent analyses, peer-reviewed, agreement reached. The kind of process you’d trust.
The list of binaries they’d found looked accurate. The version comparisons looked correct. The conclusion: probably a false positive.
But something didn’t sit right. Security findings are rarely arbitrary — scanners flag things for a reason, even when that reason isn’t immediately clear. The right instinct at this point would have been to run the scanner locally: pull the problem as far left as possible in the investigation, see what the tool reports directly, compare outputs, understand the variable creating the disagreement. Not “probably false” and move on. Actually run it.
That approach wasn’t fully available here — one scanner couldn’t be run locally, and doing the equivalent binary-chop by repeatedly pushing images to the registry and waiting for a scan result would have been painfully slow. But it’s worth naming the instinct, because it’s the right one.
The Fresh Set of Eyes
After reporting the likely false positive, a fresh pair of eyes joined the conversation — and checked the image and found the problem. The essential question was:
Had anyone done a full filesystem scan?
Not a targeted scan. Not “find the binary we think it is.” A full walk of the filesystem — looking for anything that matched the vulnerable component, regardless of where it lived, regardless of whether it was on a classpath, regardless of whether it was registered anywhere.
My first instinct was that this would be slow. Full filesystem scans feel like something you only do when you have to — a relic of the era when storage was slow enough that walking it was a genuine cost. That instinct is out of date. Modern hardware, modern SSDs, a container filesystem that is finite and bounded. It was not slow. It took seconds.
The binary was hiding as a plugin to another tool. Not on any major path, not in any system registry, not somewhere a standard “where is this binary?” lookup would surface it. Tucked away in a plugin directory, loaded at runtime by the parent tool, invisible to anything that wasn’t explicitly walking the entire filesystem.
Once you see it, it’s obvious. Painfully obvious. The kind of obvious that only exists in hindsight after spending hours not seeing it.
The Tool That Was There All Along
When I asked Bob about the problem again a few days later — fresh session, no accumulated context — it took a different approach. Rather than writing scripts immediately, it suggested trying Trivy, an open-source vulnerability scanner designed specifically for container images. If that didn’t work, it offered, here’s a fallback bash script.
That fallback script was, in essence, the full filesystem scan. It would have found the issue. But Trivy found it even faster — and with considerably less effort. A single command produced a clean, human-readable table: component, version, CVE ID, severity, fix version, and the exact path of the affected binary. No scripts. No manual inspection. No cross-referencing between two AI outputs.
CVE-2024-24790, for the record, is a Go standard library vulnerability — a flaw in net/netip.Addr classification that causes IPv4-mapped IPv6 addresses to be misclassified, which breaks access control logic that depends on that classification. Any Go binary compiled against Go before 1.21.11 or 1.22.4 is affected. The fix is to rebuild with a patched Go toolchain. The binary in question was bundled as a plugin, built against an older Go version, sitting in a directory that none of the earlier targeted searches had reached.
Trivy reached it on the first try.
There’s a broader observation here too. When you step away from a problem and come back to it — or ask the same question a different day — the shift in perspective can change everything. AIs are consistent in execution but non-deterministic in strategy; the same question on a different day can produce a genuinely different approach. People work the same way. Sometimes the most powerful debugging technique is simply to walk away and come back fresh.
What Actually Went Wrong
The investigation had three participants — me, Bob, and Claude — and all three of us shared the same blind spot. We were looking for the binary as a first-class thing: a system binary, a library on a classpath, something that a version query or a dependency manifest would surface. That search strategy is reasonable for the most common case. It was wrong for this one.
The AIs wrote good scripts for finding what we expected to find, and those scripts ran correctly. The problem is that a correct script operating on the wrong search space produces a confident wrong answer. When Bob and Claude cross-checked each other’s work, they were validating each other’s execution — did the script run correctly, were the version comparisons valid — not each other’s assumptions about where to look. The assumption was shared, so the peer review didn’t catch it. That confidence was further reinforced by the fact that all visible versions were current and the main registry scanner had already given the all-clear.
That’s a subtler failure mode than “the AI was wrong.” The AI was right about everything it checked. It just didn’t know what it wasn’t checking. And neither did I.
The supervisory model — two AIs reviewing each other’s outputs — works well for catching execution errors, logical mistakes, and version comparison bugs. It does not work for catching shared blind spots. For that you need someone whose mental model of the problem is genuinely different. In this case, that was a human with fresh eyes who hadn’t already anchored on the assumption that the binary should be findable by normal means.
There is also a point about experience worth making. A more seasoned security engineer might have reached for Trivy first — or at least asked “what’s the right tool for this?” before writing bespoke scripts. And it reinforces the importance of being able to run diagnostic tools at every stage of the process, not just at the gates where managed tooling has been installed for you.
A Note on Non-Determinism
After the investigation was closed, I asked Bob the same question again with a clean prompt: explain CVE-2024-24790, then tell me how to check a Docker image for it. Same intent. Different day.
This time it reached for external tooling first — attempted to download a scanner directly into the container, the download timed out, then it fell back to a path-based search and installed tools via the package manager. A completely different strategy from the earlier session. Neither approach was wrong, but they weren’t the same — and in a security context, “not wrong but different” is its own kind of problem. You cannot build a repeatable process on top of a tool that approaches the same question differently on each run.
This isn’t a reason to distrust AI tooling. It is a reason to treat AI-generated investigation scripts as a strategy, not the strategy. Verify the approach, not just the output. And when the AI suggests a purpose-built tool like Trivy — actually use it.
It’s also worth being honest about the bigger picture here. Bob, Claude, the registry scanner, and I all got this wrong. As the safety-critical systems principle goes: approach the same problem by different means and you may get different answers. In this case, the different means — a fresh perspective, a different tool, a different day — is exactly what eventually found it. The lesson applies equally to human investigators and silicon ones.
What I’d Do Differently
Run Trivy first. It’s purpose-built for exactly this problem — it walks the full filesystem, understands container layers, and reports CVEs with location and severity in a clean, human-readable table. It should be the starting point, not a suggestion filed away for later.
Treat scanner disagreement as a signal, not a tiebreaker. When one scanner says nothing and another says there’s a problem, the correct read is not “the quiet one is probably right.” It’s “these two tools are looking at different things, and I need to understand what the difference is.” In this case, not being able to run the primary tool locally meant there was no way to understand what it was and wasn’t covering. The fact that a major registry scanner failed to surface a two-year-old CVE should have been a warning sign, not a source of comfort.
Shift left on the right scanner. Running Trivy in CI — against every image build, not just on a periodic policy sweep — would have caught this weeks earlier with no investigation required. That’s the shift-left principle: the cost of finding a vulnerability drops dramatically the earlier in the pipeline you find it. A Trivy run at build time costs seconds. A multi-day investigation costs considerably more.
Shift Left Means Across the Ecosystem, Not Just Earlier in Your Pipeline
There’s one more thing worth saying about “shift left,” because the common reading of it is too narrow.
Shift left usually means: find problems earlier in your delivery pipeline. Catch the CVE in CI rather than in a production scan. That’s correct, but incomplete. The deeper version is: the finding should have been someone else’s responsibility before it ever reached my pipeline at all.
The vulnerable binary arrived as part of a plugin ecosystem — installed and managed by a tool I was consuming, not something I built or packaged directly. In an ideal world, the tool managing that plugin ecosystem would be aware of the CVEs in what it installs, and would carry that signal forward. The plugin, or its installer, should have surfaced this. It shouldn’t be hiding silently inside a container image, waiting for a scanner to stumble across it three layers removed from the thing that originally introduced it.
That team has a gap to close. I made mistakes in tracking the problem down. But the real takeaway is this: shift left isn’t just a property of your own pipeline — it’s a responsibility that extends across the entire ecosystem of tools and components you depend on. When that responsibility is ambiguous, findings fall through the cracks. This one did.
Final Thoughts
Nobody in this story is the villain. The people involved were doing their jobs in good faith, the tools were doing what they were designed to do, and the AIs were working with the information they were given. The only villain is whoever might want to exploit CVE-2024-24790 in a system that never got patched for nefarious reasons.
Mistakes are how we learn. For me the lessons are.
Binaries hide. They hide in plugin directories, in tool ecosystems, in places that no dependency manifest or system registry knows about. The only way to be certain something isn’t there is to look everywhere — which, on modern hardware, is fast enough that there’s no good excuse not to.
If you can run a scanner locally, run it. The single biggest leverage point in this investigation would have been getting any scanning tool running locally against the image — whether Trivy, the secondary commercial scanner, or anything else. A local tool means a tight feedback loop: you can strip components, rebuild, rescan, and binary-chop toward the answer in minutes. Without that, you’re dependent on remote scans, queued results, and other people’s schedules. Push hard to get local tooling working before you need it, not during a live investigation.
Everything else — the AI cross-checking, the peer review that missed the shared assumption, the trust placed in the primary scanner’s silence — flows from those two facts. If you know binaries can hide and you have a local scanner that finds them, the rest of this story doesn’t happen.
It’s a good lesson. It was an expensive way to learn it. And what stings is how obvious the issue was in hindsight :-)