AuditBench Source Code Vulnerability Audit & Program Analysis
Evaluating program analysis engines and code LLMs on interprocedural taint propagation, pointer aliasing, and complex control-flow vulnerability discovery across multi-file codebases.
C/C++, Go, Java, Rust, TypeScript
Multi-File CVE Snapshots
SOP v3.1 Def-Use & Observable Sink Oracles
Line-Level Precision & Soundness
Core Research Problem
Traditional static application security testing (SAST) and generic code LLMs suffer from high false-positive rates and context fragmentation. AuditBench is designed to measure how well autonomous agents and program analysis engines understand heterogeneous code structure, resolve interprocedural call chains, and trace taint flows from untrusted user inputs to verifiable execution sinks without losing state provenance.
Universal AST & CFG Traversal
Evaluates agent capability to navigate canonical Universal Abstract Syntax Trees and reconstruct multi-file control-flow graphs.
Interprocedural Taint Propagation
Tests deep taint tracking across function call boundaries, indirect callbacks, and macro wrappers without state loss.
Sanitizer Gap Identification
Requires models to accurately detect partial or flawed sanitization logic where defenses fail under edge-case inputs.
Pointer & Heap Alias Reasoning
Stresses reasoning over mutable heap state, pointer aliases, and framework routing abstractions.
Deterministic Execution Pipeline
Every evaluation task follows a structured, auditable lifecycle from ingestion to oracle confirmation.
Source Code Snapshot Ingestion
Source code is ingested directly from verified pre-patch parent commits and parsed into canonical syntax trees.
Agentic Exploration & Taint Slicing
The evaluating model or engine explores code paths, extracts causal data-flow slices, and outputs identified vulnerability candidates.
Def-Use Ground-Truth Verification
Outputs are matched against formal ground-truth ledgers, validating line-exact source, propagation steps, and true execution sinks.
Falsifiable Scoring Standards
SecDome enforces strict ground-truth verification standards. No speculative scoring or synthetic shortcuts.
Line-Level Exact Localization
Vulnerability localization must correspond to the actual definition-use chain; coarse file-level guesses receive zero score.
True Execution Sink Requirement
Rejects passive return statements, factory constructors, and pure variable assignments as execution sinks.
Zero-Tolerance False Positive Penalty
Heuristic spamming across non-vulnerable code paths is strictly penalized under normalized precision scoring.
Specification and ground-truth dataset schemas established. Evaluation framework under active internal calibration; verified empirical results will be published with peer-reviewed research papers.
Reproducible & Auditable.
All SecDome benchmark specifications, task schemas, and evaluation protocols are maintained openly under Apache 2.0 / MIT. We invite external research groups to audit our task definitions and contribute reproducible evaluation environments.