An Evaluation of Cost-Benefit Using Security Requirements Prioritization

This article describes a comparison of six security requirements prioritization methods: analytical hierarchy process (AHP), accelerated requirements method (ARM) prioritization, priority poker, cost-benefit model, security investment decision dashboard (SIDD), and COCOMO-II security extensions. OVERVIEW When building complex systems, stakeholders must often prioritize requirements as part of the requirements engineering process. The development team may not implement all requirements due to lack of time, lack of resources, or changing or unclear project goals. In these cases it is important to define which requirements should be given priority over others. Security Quality Requirements Engineering (SQUARE) is a requirements engineering process developed by the Software Engineering Institute’s (SEI) CERT® Program at Carnegie Mellon University (CMU). SQUARE is a nine-step process that delivers categorized, prioritized, and validated security requirements. SQUARE prioritizes requirements based on completed risk assessment and requirement categorization steps [Mead 2006a] [SQUARE 2010]. Various prioritization techniques may be and have been used in SQUARE step 8. Though the official SQUARE tool uses the Analytical Hierarchy Process (AHP), SQUARE does not prescribe any specific method. This article describes a comparison of six prioritization methods using a previous SQUARE case study, which described the Accelerated Requirements Method (ARM) and Analytical Hierarchy Process (AHP) method, for context [Hough 2006]. The case study applied SQUARE to three large-scale software applications with security requirements. One of the projects, the basis for the current evaluation, provides nine security requirements and a risk assessment, the inputs to SQUARE prioritization. For reference, Table 1 shows the security requirements. Nancy Mead Travis Christian June 2010 Table 1. Security requirements SR-1 The system shall implement access control via a secure login screen. SR-2 The system shall identify and authenticate all users who attempt to access it. SR-3 The server-side components and files contained therein shall have their access restricted to authorized personnel. SR-4 Fault tolerance shall be provided for the asset management system’s essential services (IIS server, GIS server, and network lines). SR-5 The system shall maintain data integrity via logged modifications and user access control. SR-6 An access control system shall be configured for optimal information gathering for auditing purposes (access log and application log). SR-7 The system shall recover from attacks, failures, and accidents in less than one minute. SR-8 A backup shall consist of a complete reproduction of every file on the server. SR-9 The system shall be able to provide full functionality from backup. Each of the six prioritization methods was performed on this same set of requirements. For the methods involving the collaboration of multiple stakeholders, students familiar with software engineering and basic security concepts played the role of stakeholders in a software project that had yielded the given requirements. They ranked the requirements according to their own judgment, by whatever criteria the method called for. METHODS EVALUATED 1. Analytical Hierarchy Process (AHP) 2. Accelerated Requirements Method (ARM) Prioritization 3. Priority Poker 4. Cost-Benefit Model 5. Security Investment Decision Dashboard (SIDD) 6. COCOMO II Security Extensions 1 | AN EVALUATION OF COST-BENEFIT USING SECURITY REQUIREMENTS PRIORITIZATION AHP The Analytical Hierarchy Process (AHP) is a general decision-making method for situations involving multiple decision factors. It uses pair-wise comparison to estimate the relative values of each pair of options and the consistency of the responses. By breaking the problem down into a comprehensive set of individual comparisons, AHP provides a consistent model for prioritizing many elements [Mead 2006b]. The existing SQUARE support tool implements AHP to prioritize security requirements [Tool 2010]. For each pair of requirements, users indicate their relative cost and relative value on 5-point scales. After each pair of requirements is compared, the tool calculates the average score for each requirement as a composite of both factors and prioritizes them accordingly. In the SQUARE implementation, the lead requirements engineer may adjust the final rank after discussing the results with the team. AHP has a number of advantages that made it an excellent choice for implementation in the SQUARE tool. The method has already been proven effective in a previous case study as well as in the wider context of industry use. It splits the problem into specific criteria, in this case cost and value, that can be measured independently. While AHP does not specifically require multiple participants, it scales to any number, averaging their results into a final ranking. The method’s disadvantages are minimal and can be easily mitigated. AHP relies heavily on calculations for pair-wise comparison, but with tool support this is not an issue. However, since the number of comparisons grows quickly as the number of requirements increases, it may be unwieldy for a complex project. AHP’s results may be partially subjective because they rely on relative rather than absolute scoring, but relative scores could be determined with some formality to obtain more objective results. It is worth noting that the mathematical model behind pair-wise comparison can also reflect how consistently a requirement has been scored across all of its pairings in relation to the other requirements. For this study, participants performed AHP using the existing SQUARE tool [Tool 2010]. For each pair of requirements, the tool presents users with two 5point scales, one to indicate which requirement is more valuable and one to indicate which is more costly. Each participant, playing the role of a stakeholder for a large project, received a SQUARE project at the prioritization step and completed the pair-wise comparison. For the set of nine requirements, each participant compared a total of 36 pairs. The tool automatically calculated and averaged the results shown in Table 2. 2 | AN EVALUATION OF COST-BENEFIT USING SECURITY REQUIREMENTS PRIORITIZATION Table 2. AHP results Rank Requirement