For several months, our team has been rigorously testing a suite of specialized LLMs designed to identify vulnerabilities within our own systems. These models not only help us patch weaknesses before they can be exploited but also provide a window into how attackers might leverage the latest AI advancements. Among these, Anthropic's Mythos Preview has stood out, especially during our involvement in Project Glasswing. We deployed it against over fifty of our private repositories to assess its capabilities and limitations. This article distills the most critical observations from that experiment, highlighting what makes Mythos a game-changer—and where the industry must adapt to harness such tools at scale.
1. A Paradigm Shift in Security AI
Mythos Preview isn't simply an incremental upgrade over previous frontier models; it represents a fundamentally different approach to security auditing. Where earlier LLMs offered refined but familiar capabilities, Mythos introduces a new class of functionality. It moves beyond passive vulnerability scanning to active, reasoned exploitation. This shift makes direct, apples-to-apples comparisons difficult. Instead of benchmarking against general-purpose models, it's more illuminating to examine what Mythos can achieve—specifically its ability to construct complex exploit chains and autonomously generate proofs—as these features redefine the role of AI in cybersecurity.
2. Advanced Exploit Chain Construction
Real-world attacks rarely rely on a single bug. They typically chain multiple small attack primitives into a coherent exploit. For example, a use-after-free flaw might be combined with techniques to gain arbitrary read/write primitives, then hijack control flow via return-oriented programming (ROP) to fully compromise a system. While many automated scanners can identify individual bugs, they struggle to connect them. Mythos Preview, however, demonstrates the reasoning ability of a senior security researcher: it can take several disconnected primitives, analyze their interactions, and stitch them into a working proof-of-concept. The model's step-by-step reasoning during chain construction reveals a level of sophistication previously unseen in automated tools.
3. Autonomous Proof Generation Through Iterative Testing
Finding a bug is only half the battle; proving it's exploitable is equally critical. Mythos excels at both by automating the proof-of-concept creation process. It writes code designed to trigger a suspected vulnerability, compiles that code in a sandboxed environment, and executes it. If the program behaves as expected, the proof is validated. If it fails, Mythos analyzes the error, adjusts its hypothesis, and attempts again—repeating this loop until it either succeeds or exhausts possibilities. This iterative testing closes the gap between suspicion and certainty, transforming speculative findings into actionable exploits. The model's ability to self-correct and refine its approach sets it apart from static vulnerability scanners.
4. Where Other Models Fall Short: The Stitching Problem
During our Project Glasswing trials, we ran other frontier models through the same harness used for Mythos. They identified many of the same underlying vulnerabilities and, in some cases, even performed well on initial reasoning. However, they consistently hit a wall when it came to combining those vulnerabilities into a chain. A model would pinpoint a use-after-free bug here and an information leak there, but fail to connect them into a viable exploit. Mythos Preview, in contrast, excels at this 'stitching' step. This gap highlights a crucial limitation of current AI auditing tools: without the ability to reason about multi-step attack sequences, they remain efficient bug finders but not fully autonomous security analysts. Future models will need to bridge this reasoning divide to truly scale cybersecurity efforts.
5. Architectural Lessons for Scaling AI Security Tools
While Mythos Preview demonstrates impressive capabilities, our experiments also revealed the need for significant changes in the architecture and processes surrounding these models. To use them at scale, organizations must consider how to integrate chain construction and proof generation into existing workflows. This includes designing harnesses that can safely compile and execute generated code, establishing feedback loops for model refinement, and creating interfaces that allow security teams to inspect and validate reasoning chains. Moreover, the compute and time costs associated with iterative testing must be managed. Our experience suggests that the most effective deployments will pair AI models like Mythos with human oversight—automating the heavy lifting of vulnerability discovery and exploit creation while leaving strategic decisions and edge-case handling to experienced researchers.
In conclusion, Project Glasswing has shown that Mythos Preview represents a meaningful leap forward in AI-driven security auditing. Its ability to construct exploit chains and autonomously generate proofs sets a new benchmark. However, the technology is not yet plug-and-play. To unlock its full potential, the industry must evolve both the tools themselves and the infrastructure that supports them. As we continue to refine these systems, one thing is clear: AI is no longer just a helper in cybersecurity—it's becoming an active participant in the cat-and-mouse game between defenders and attackers.