PoCBench Autonomous Vulnerability Reproduction & PoC Synthesis
An end-to-end evaluation suite assessing autonomous agents on exploring vulnerable network topologies, bypassing protections, and synthesizing executable PoCs inside hermetic sandboxes.
Multi-Container Topologies (CrafterHub)
Isolated Vulnerability Environments
Physical Canary State Change (I/O & Memory)
PoC Reproduction & Resource Bounds
Core Research Problem
PoCBench benchmarks an autonomous agent ability to navigate realistic multi-service topologies, formulate exploit hypotheses, and synthesize minimal, deterministic Proof-of-Concept scripts. Unlike benchmarks relying on superficial string matches or HTTP response codes, PoCBench requires the trigger of an observable physical canary state change inside isolated container sandboxes.
Multi-Service Environment Recon
Agent autonomously scans and probes network topology, database links, and application middleware to map attack surfaces.
Dynamic Exploit Synthesis
Generates valid network, HTTP, or binary payloads through iterative trial and error and log telemetry.
Physical Canary Trigger
Verifies exploitation through deterministic host filesystem markers or network callback beacons, eliminating false positives.
Resource & Safety Watchdog
Monitors agent execution under strict step deadlines, token budgeting, and container isolation boundaries.
Deterministic Execution Pipeline
Every evaluation task follows a structured, auditable lifecycle from ingestion to oracle confirmation.
Containerized Topology Spin-Up
Self-healing, reproducible multi-container network is instantiated with vulnerable services and isolated networks.
Autonomous Exploration Cycle
Agent interacts via standardized CLI/HTTP tool interfaces, observing service feedback and tailoring payload inputs.
Physical Oracle Attestation
Witness monitor confirms whether the canary artifact was triggered within timeout and resource limits.
Falsifiable Scoring Standards
SecDome enforces strict ground-truth verification standards. No speculative scoring or synthetic shortcuts.
Observable Ground-Truth Witness
Requires explicit filesystem, process, or network state modifications verified by container monitors.
Hermetic Isolation Enforcement
All agent actions are restricted to air-gapped container sandboxes with zero host authority.
Efficiency & Determinism
PoC scripts must be standalone, minimal, and re-runnable with 100% deterministic success.
Multi-container target registry and canary oracle harness under active testing. Formal benchmark evaluation sweeps scheduled for upcoming academic release.
Reproducible & Auditable.
All SecDome benchmark specifications, task schemas, and evaluation protocols are maintained openly under Apache 2.0 / MIT. We invite external research groups to audit our task definitions and contribute reproducible evaluation environments.