Key takeaways
- AI agents outperformed 90% of human teams in two major cybersecurity competitions.
- Four AI teams solved 19 out of 20 challenges in the Capture The Flag event.
- The best AI agent ranked 20th globally, demonstrating strong performance in cryptography tasks.
- AI can reliably solve challenges requiring one hour or less of human effort.
- Competitions serve as effective models for evaluating real-world AI capabilities.
AI agents have outperformed the majority of human participants in two key cybersecurity contests hosted by Hack The Box. According to a research report by Palisade Research, AI teams were placed in the top 5% and top 10% in events that attracted a combined total of over 18,000 participants.
In the ‘AI vs Humans’ Capture The Flag (CTF) competition, six AI teams competed against 400 mostly human teams over 48 hours. Four agents solved 19 out of 20 challenges, placing them in the top 5% overall and well ahead of the majority of human teams.
The best-performing AI agent, named CAI, ranked 20th on the global leaderboard. Tasks were designed around reverse engineering and cryptography and could be solved locally, reducing the infrastructure overhead for AI systems.
In the second event, ‘Cyber Apocalypse’, two AI teams entered alongside more than 8,000 teams comprising 18,000 participants. The best AI placed within the top 10% of all competitors, despite the challenge set requiring interaction with external systems, something many AI agents were not optimised for.
Only four AI agents participated, but the top-performing model still exceeded expectations by completing 20 challenges, placing ahead of 90% of human teams.
The research also applies METR’s 50%-completion-time metric to estimate what kind of human effort current AIs can match.
“AI agents can reliably solve cyber challenges requiring one hour or less of effort from a median human CTF participant,” the research paper mentioned.
Palisade Research described the competitions as a new model for evaluating real-world AI performance. “Open-market elicitation may offer an effective complement to in-house evaluations,” the report stated. Unlike traditional benchmarks, these events ran publicly, allowing observers to see AI and human performance side by side.
The observation encourages CTF organisers to host more such events and calls on funders to support prize-based evaluations. It suggests that such efforts can help maintain situational awareness as offensive AI capabilities evolve quickly.
