Ground Truth in Evaluation. Standardized Suites for Cyber Systems & AI Agents.
AuditBench · Source Code Vulnerability Audit
Evaluating traditional program analysis engines and code LLMs on interprocedural taint propagation, pointer aliasing, and deep control-flow vulnerability detection across diverse languages.
FixBench · Security Patch & Remediation
Assessing automated security agent patches on whether they effectively neutralize exploit vectors while preserving complete business functionality and passing full functional regression test suites.
PoCBench · Exploit & PoC Reproduction
An end-to-end benchmark assessing autonomous agents on vulnerability reproduction and executable proof-of-concept synthesis inside sandboxed micro-VMs with verifiable canary execution state.
LogicBench · Access Control & Business Logic
Benchmarking agent semantic reasoning on non-syntactic vulnerabilities: broken object-level authorization (BOLA), workflow state bypasses, and multi-tenant isolation leaks.
Evaluation Framework &
Evidence Standards.
SecDome benchmarks adhere to strict reproducibility principles. Every published evaluation task fully records inputs, expected behaviors, execution traces, and auditable outputs. We reject synthetic scoring without physical verification.
Complete repository snapshot, dependency tree, and compile flags locked by cryptographic SHA-256 hashes.
Every intermediate agent prompt, tool invocation, and kernel eBPF event recorded in timestamped JSONL logs.
Success requires physical file modification, memory corruption witness, or network callback verified by independent oracles.