HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance Remediation
In this paper, we introduce HardEval, an execution-verified benchmark for evaluating language-model-assisted security hardening.
ProofPaper ↗
Key points
- Implementing Security Technical Implementation Guide (STIG) controls remains a demanding operating-system administration task.
- HardEval is built from Red Hat Enterprise Linux 8 STIG controls that fail on an unhardened baseline system.
- The model produces a Bash script, which is executed on a live virtual machine and graded from fail to pass using the corresponding OpenSCAP/OVAL compliance check.
- Evaluation of fourteen frontier and open-weight configurations reveals large and uneven capability differences across security domains.
Sources (1)
- [1]HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance RemediationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 04:14 PM
In this paper, we introduce HardEval, an execution-verified benchmark for evaluating language-model-assisted security hardening.
Implementing Security Technical Implementation Guide (STIG) controls remains a demanding operating-system administration task.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026openai/codex rust-v0.163.0: 0.163.0
- Oct 15, 2026capy-flash: Qwen3.8-Flash-Next (125B MoE) in full NVFP4 on two 16 GB consumer GPUs
- Oct 15, 2026What a £300, 35W Snapdragon mini PC can actually do 👀: 35B coder at ~30 tok/s, and an 80B model that doesnt even fit in RAM running at 17 tok/s
- Oct 15, 2026openKylin Outlines Its Vision for Reimagining Linux in the AI Era at Open Source Summit Europe 2026
- Oct 15, 2026Best Minimax H3 workflow for Upscaling?
- Oct 14, 2026Deep Reinforcement Learning for HPC Job Scheduling: A Statistical Evaluation