Evaluate alternative solutions using established criteria and methods.

Evaluating alternative solutions involves analysis, discussion, and review. Iterative cycles of analysis are sometimes necessary. Supporting analyses, experimentation, prototyping, piloting, or simulations may be needed to substantiate scoring and conclusions.

Often, the relative importance of criteria is imprecise and the total effect on a solution is not apparent until after the analysis is performed. In cases where the resulting scores differ by relatively small amounts, the best selection among alternative solutions may not be clear. Challenges to criteria and assumptions should be encouraged.

Example Work Products

1.    Evaluation results

Subpractices

1.    Evaluate proposed alternative solutions using the established evaluation criteria and selected methods.

2.    Evaluate assumptions related to the evaluation criteria and the evidence that supports the assumptions.

3.    Evaluate whether uncertainty in the values for alternative solutions affects the evaluation and address these uncertainties as appropriate.

For instance, if the score varies between two values, is the difference significant enough to make a difference in the final solution set? Does the variation in score represent a high-impact risk? To address these concerns, simulations may be run, further studies may be performed, or evaluation criteria may be modified, among other things.

4.    Perform simulations, modeling, prototypes, and pilots as necessary to exercise the evaluation criteria, methods, and alternative solutions.

Untested criteria, their relative importance, and supporting data or functions can cause the validity of solutions to be questioned. Criteria and their relative priorities and scales can be tested with trial runs against a set of alternatives. These trial runs of a select set of criteria allow for the evaluation of the cumulative impact of criteria on a solution. If trials reveal problems, different criteria or alternatives might be considered to avoid biases.

5.    Consider new alternative solutions, criteria, or methods if proposed alternatives do not test well; repeat evaluations until alternatives do test well.

6.    Document the results of the evaluation.

Document the rationale for the addition of new alternatives or methods and changes to criteria, as well as the results of interim evaluations.