Reliability of Hands-On Cybersecurity Skill Assessments: A Generalizability Study

What Did This Cybersecurity Research Find?

This cybersecurity assessment quality study examined the reliability of hands-on skill tests (virtual labs, CTF challenges) used in hiring and certification. Cybersecurity hands-on assessments showed acceptable reliability when they included at least 8 tasks spanning multiple difficulty levels, but shorter assessments produced inconsistent results.

Key Findings

• 1Assessments with 8+ tasks across difficulty levels achieved reliability above 0.80
• 2Shorter assessments (3-4 tasks) had reliability below 0.60, making them unsuitable for high-stakes decisions
• 3Task type diversity (not just difficulty) improved measurement reliability
• 4Time pressure reduced reliability for complex analytical tasks but not for procedural ones
• 5Rater agreement for open-ended tasks was moderate (ICC = 0.72), suggesting rubric improvement is needed

How Does This Apply to Cybersecurity Careers?

Candidates can evaluate the quality of assessments they encounter. Organizations can design hands-on tests that actually produce reliable, defensible results.

Who Should Read This?

Frequently Asked Questions