---
title: "How Panop is Benchmarked"
description: "Panop's autonomous pentest is scored on three public suites: XBOW, Argus, and ExploitBench. This page is the protocol and the results: XBOW is complete, Argus and ExploitBench are in progress."
url: https://panop.io/blog/how-panop-is-benchmarked/
---

Article Results forthcoming September 15, 2026 [Panop Research](https://panop.io/company) · Security research team, Panop SA

# How Panop is Benchmarked

Panop's autonomous pentest is scored on three public suites: XBOW, Argus, and ExploitBench. This page is the protocol and the results: XBOW is complete, Argus and ExploitBench are in progress.

On this page

1. [Why three suites, not one?](#why-three-suites-not-one)
2. [What XBOW measures](#what-xbow-measures)
3. [What Argus measures](#what-argus-measures)
4. [What ExploitBench measures](#what-exploitbench-measures)
5. [How Panop is run against them](#how-panop-is-run-against-them)
6. [Results](#results)
7. [What this does not prove](#what-this-does-not-prove)
8. [Sources](#sources)

A CVSS score is a property of a vulnerability. “Exploitable here” is a property of a path. [Panop’s autonomous pentest](https://panop.io/autonomous-penetration-testing) exists to turn the second into evidence: a finding is confirmed only when the platform has demonstrated access, under guardrails, with the reproduction path attached.

That claim is only as good as the targets you test it on. Scanner labs you wrote yourself do not count. We are scoring Panop against three public, independently authored suites that were built to evaluate offensive agents, not to flatter a vendor:

1. [XBOW Validation Benchmarks](https://github.com/xbow-engineering/validation-benchmarks): 104 Jeopardy-style web challenges.
2. [Argus Validation Benchmarks](https://github.com/pensar-x/argus-validation-benchmarks): 71 live pentest targets on modern stacks, plus a separate threat-model suite.
3. [ExploitBench](https://github.com/exploitbench/exploitbench): a 16-flag exploitation ladder on 41 Chromium V8 bugs.

This page is the protocol. The XBOW [results](#results) are in. The Argus and ExploitBench tables stay empty until those runs complete. We will fill them in place, not in a separate announcement that can drift from the method.

## Why three suites, not one?

Each suite asks a different question. Treating them as one leaderboard would hide that.

| Suite | What it asks | Win condition | Why it is in this evaluation |
| --- | --- | --- | --- |
| [XBOW](https://github.com/xbow-engineering/validation-benchmarks) | Can the agent find a hidden flag in a web CTF target? | Flag | The 2024-25 reference set for web offensive tools. XBOW now marks it as **saturated** (~100% industry performance as of mid-2026). We run it as a baseline, not as the discriminating test. |
| [Argus](https://github.com/pensar-x/argus-validation-benchmarks) | Can the agent exploit modern, multi-step, sometimes defence-enabled applications? | Flag | Built because older sets over-index on PHP and under-cover GraphQL, race conditions, JWT, WAF bypass, sandbox boundaries and attack chains. This is the discriminating **application** suite. |
| [ExploitBench](https://github.com/exploitbench/exploitbench) | How far up a real exploitation ladder does the agent climb? | Capability bitmap, graded by deterministic oracles | Crash-only benches stop at reproduction. ExploitBench scores coverage → trigger → primitives → arbitrary code execution on production V8 with the heap sandbox on. This is the discriminating **engine** suite. |

A high XBOW score, in 2026, says the agent can still do the work that used to separate tools. It does not say it can pentest a GraphQL SaaS app, bypass a WAF, or turn a V8 N-day into a sandbox escape. That is why Argus and ExploitBench are here.

## What XBOW measures

[XBOW Validation Benchmarks](https://github.com/xbow-engineering/validation-benchmarks) are 104 isolated challenges, each with one objective: recover a flag from a Dockerised web target. Flags are injected at build time, not hardcoded. Difficulty is author-assigned as level 1 (easy), 2 (medium) or 3 (hard). The original brief was to mirror vulnerability classes seen in pentest and bug-bounty work, and to evaluate web-based offensive tools rather than trivia.

XBOW’s own README now warns that the set is **outdated for ranking**. Released in 2024, with build fixes in 2025, it has been trained into current models. As of mid-2026 the maintainers put industry-standard performance at approximately 100%, and they keep the repository for historical comparison.

We still run it, for two reasons. First, most published agent papers from 2024 to 2025 used it, so a Panop number that cannot be placed next to those papers is not useful. Second, a miss on a saturated set would be a real signal. A perfect score is not.

**Panop is scored on flag capture.** A challenge counts as solved only when the injected flag is recovered. We report solved / 104 and a breakdown by level.

## What Argus measures

[Argus Validation Benchmarks](https://github.com/pensar-x/argus-validation-benchmarks), published by Pensar, are the most complete open pentest set we could find that is not already spent. Two suites live in the same repository. Only the first is in scope for this product evaluation.

**Pentesting benchmarks (71)**: self-contained Docker applications and infrastructure targets. Each has a planted vulnerability that must be discovered and exploited to capture a flag in the form `FLAG{…}`. Coverage includes Node.js, Python, Java, Go, PHP and Ruby; injection, authz bypass, SSRF, XSS, prototype pollution, race conditions, deserialization, SSTI, WAF/IDS bypass, cloud and CI/CD chains; and, in the 2026 `ARGUS-*-26` family, sandbox escapes, OT/ICS, firmware, identity, ML supply chain, mobile and edge.

Difficulty, as assigned by the authors:

| Difficulty | Count | Their description |
| --- | --- | --- |
| 1 (Easy) | 2 | Single request, obvious vulnerability, standard payload |
| 2 (Medium) | 27 | Multiple requests, some enumeration, encoded payloads |
| 3 (Hard) | 42 | Multi-step, timing-sensitive, chaining or boundary reasoning |

**Threat-model benchmarks (10)**: source-code analysis against a ground-truth document, scored on six dimensions (structure, grounding, anti-patterns, discovery quality, attack-path depth, usefulness). That is a different product question than live exploit validation. It is **out of scope** for this run. We may score it later; we will not mix those numbers into the pentest tables.

**Panop is scored on flag capture** against the 71 live targets. A challenge counts as solved only when the suite’s own checker accepts the flag. We will report solved / 71, then splits by difficulty and by the authors’ vulnerability categories.

## What ExploitBench measures

[ExploitBench](https://github.com/exploitbench/exploitbench) (Lee and Brumley, Carnegie Mellon; public results at [exploitbench.ai](https://exploitbench.ai/)) does not award a single flag. It asks how far an agent climbs, on a known V8 bug, against a **production-like** `d8` with the V8 heap sandbox and shipped mitigations enabled.

The current instantiation, bench-v8, covers **41 V8 bugs** reported no earlier than 2024. Each episode yields a 16-bit capability map, grouped into five tiers:

| Tier | Name | What a success means |
| --- | --- | --- |
| T5 | Coverage | The patched function or line actually ran |
| T4 | Reproduction | Crash, sanitizer report, or differential behaviour versus the fixed build |
| T3 | Target primitives | V8-in-sandbox building blocks (`addrof`, `fakeobj`, caged read/write) |
| T2 | Generic primitives | Arbitrary read/write and infoleaks past the engine’s isolation |
| T1 | Full control | Control-flow hijack and arbitrary code execution |

Every rung is checked by a **deterministic oracle** compiled into the grader: coverage, differential execution, randomised challenge-response for primitives, a signal-handler proof for PC control. There is no LLM-as-judge. Crash-class benches (the authors name CyberGym, CyBench, SEC-bench Pro) stop at T4. The point of ExploitBench is the climb above that floor.

Public frontier models, in the authors’ own panel, routinely reach coverage and often crash. Arbitrary code execution on this set is rare. That is the reason to include it: if Panop’s “exploitable” only ever means “web flag captured”, we have not measured exploitation against a hardened engine.

**Panop is scored with the suite’s own grader.** We will report, per the published protocol: highest tier reached, bugs (of 41) that reached each tier, and whether any cell achieved ACE. We will not collapse the ladder into a single pass/fail.

ExploitBench asks researchers not to run reinforcement learning on the set. We will not.

## How Panop is run against them

The method is the suites’ method. We do not rewrite the challenges, weaken the defences, or substitute a private flag.

- **Isolation.** Every target is the published Docker (or ExploitBench) image, started as the maintainers document. Nothing on these hosts is a customer system.
- **Briefing.** Panop receives what a competitor is supposed to receive: the challenge description and the reachable service (or the published evaluation interface). It does not receive the flag, the solve script, `expected_results/`, or the ground-truth vulnerability file.
- **Flags.** On XBOW and Argus the flag is injected at build time, as the READMEs require. A hardcoded string in the agent’s prompt is not a solve.
- **Win condition.** XBOW and Argus: the injected flag, recovered. ExploitBench: capability bits set by the mechanical grader, not by the agent’s self-report.
- **No training on the test.** We do not fine-tune, RL, or otherwise fit Panop to these repositories.
- **Guardrails stay on.** Panop’s production rule (prove access without taking data or leaving a foothold) still applies. A “solve” that requires disabling that rule does not count.

We will also record, as diagnostics rather than as the headline: wall-clock time, steps/turns where the runner exposes them, and (for ExploitBench) cost if the runner reports it. Those numbers will sit next to the scores, not instead of them.

## Results

Results forthcoming

XBOW is complete. Argus and ExploitBench are in progress; their scores will replace the placeholders in the tables below. We will not write a conclusion until the numbers exist.

| Suite | Targets | Panop result | Reading |
| --- | --- | --- | --- |
| XBOW Validation Benchmarks | 104 flags | 104 / 104 (100%) | Saturated baseline. A miss would matter; a full score would not, on its own. |
| Argus pentest set | 71 flags | Forthcoming | Discriminating application result. |
| Argus threat-model set | 10 apps | Out of scope | Live exploit validation only, this run. |
| ExploitBench bench-v8 | 41 V8 bugs × 16 capabilities | Forthcoming | Discriminating engine result. Highest tier and ACE count, not a single percentage. |

### XBOW

| Level | Challenges | Solved | Rate |
| --- | --- | --- | --- |
| 1 (Easy) | 45 | 45 | 100% |
| 2 (Medium) | 51 | 51 | 100% |
| 3 (Hard) | 8 | 8 | 100% |
| **Total** | **104** | **104** | **100%** |

### Argus pentest

| Difficulty | Challenges | Solved | Rate |
| --- | --- | --- | --- |
| 1 (Easy) | 2 | Forthcoming | \- |
| 2 (Medium) | 27 | Forthcoming | \- |
| 3 (Hard) | 42 | Forthcoming | \- |
| **Total** | **71** | **Forthcoming** | **\-** |

### ExploitBench (bench-v8)

| Metric | Panop |
| --- | --- |
| Highest tier reached | Forthcoming |
| Bugs reaching T5 coverage (of 41) | Forthcoming |
| Bugs reaching T4 reproduction (of 41) | Forthcoming |
| Bugs reaching T3 target primitives (of 41) | Forthcoming |
| Bugs reaching T2 generic primitives (of 41) | Forthcoming |
| Bugs reaching T1 / ACE (of 41) | Forthcoming |

When these tables are filled, the protocol above does not change. If a later run uses a different snapshot of a suite, we will say so on this page rather than silently overwrite.

## What this does not prove

These are laboratory targets with a known win condition. They do not replace a scoped engagement against a customer’s estate, and they do not measure whether Panop is *safe* to run in production, only whether it can demonstrate the exploit the benchmark authors planted.

XBOW, by the maintainers’ own statement, no longer separates strong agents from weak ones. Argus still does, for application pentest work. ExploitBench still does, for engine exploitation, and it measures a climb that most public models currently stall on.

A number without that context is a press release. The context is the point of publishing the protocol first.

The product that this evaluation is supposed to underwrite is [autonomous penetration testing](https://panop.io/autonomous-penetration-testing): continuous, guardrailed exploitation that turns a severity score into demonstrated access. See also [Autonomous Pentest](https://panop.io/solutions/autonomous-pentest), or [book a demo](https://panop.io/demo).

## Sources

Figures for suite size, difficulty split and scoring rules are the publishers’ own, taken from the repositories and papers linked below. Panop results in the tables are ours. The Argus and ExploitBench rows stay empty until those runs complete.

1. XBOW Engineering: [Validation Benchmarks](https://github.com/xbow-engineering/validation-benchmarks)
2. Pensar: [Argus Validation Benchmarks](https://github.com/pensar-x/argus-validation-benchmarks)
3. Lee and Brumley: [ExploitBench](https://github.com/exploitbench/exploitbench), public leaderboard at [exploitbench.ai](https://exploitbench.ai/), paper [arXiv:2605.14153](https://arxiv.org/abs/2605.14153)
4. Panop: [Autonomous penetration testing](https://panop.io/autonomous-penetration-testing)

About the author

[Panop Research](https://panop.io/company)

Security research team, Panop SA

Panop Research is the security research function of Panop SA, a Swiss exposure management company founded in July 2024 by three practitioners with nearly twenty years each inside large international organisations. It publishes analysis built on primary public datasets, with every figure attributed and linked to the publisher that produced it.

- [LinkedIn](https://www.linkedin.com/company/panop-io)

[← Back to blog](https://panop.io/blog/)