Evaluating threat assessment for multi-stage cyber attacks

Current practices to defend against cyber attacks are typically reactive yet passive. Recent research work has been proposed to proactively predict hacker's target entities in the early stage of the attack. With prediction, there comes false alarms and missed attacks. Very little has been reported on how to evaluate a threat assessment algorithm, especially for cyber security. Because of the variety and the constantly changing nature of hacker behavior and network vulnerabilities, a cyber threat assessment algorithm is, perhaps more susceptible that for other application domains. This work sets forth the issues on evaluating cyber threat assessment algorithms, and discusses the validity of various statistical measures. Simulation examples are provided to illustrate the pros and cons of using different metrics under various cyber attack scenarios. Our results show that commonly used false positives and false negatives are necessary but not sufficient to evaluate cyber threat assessment.