What you will learn
- Define mandatory and weighted criteria.
- Separate sourced facts from judgment.
- Test sensitivity to weights and missing evidence.
- Explain trade-offs and decision ownership.
What you need
- Two or more options.
- A real decision context and budget, policy or technical constraints.
Make the decision model visible
NIST’s AI RMF promotes contextual risk management, measurement and documentation rather than treating AI output as an isolated answer.
Microsoft’s responsible AI principles include transparency and accountability, both relevant when AI supports choices that affect people or resources.
Begin with disqualifiers: legal, security, compatibility or budget requirements an option must meet. Then add weighted criteria such as cost, usability and support. Do not let the model choose weights after seeing the options because that can rationalize a preferred result.
Define criteria before collecting scores
Write each criterion so two reviewers would interpret it similarly. State the evidence needed and the scoring scale. Separate verified facts, estimates and subjective ratings. Assign weights only when they reflect the real decision.
- 1
State the decision and owner.
- 2
List mandatory requirements.
- 3
Define weighted criteria.
- 4
Set a scoring scale.
- 5
Name the evidence source for each criterion.
- 6
Record assumptions and unknowns.
Build the matrix without hiding trade-offs
Ask AI to structure the matrix and identify missing evidence, not to invent scores. Populate factual fields from documentation, contracts, tests or quotes. Add notes explaining why each score was assigned. Run at least one alternate weighting scenario to see whether the recommendation is stable.
- 1
Create one row per option.
- 2
Add evidence links or document references.
- 3
Mark unknown values.
- 4
Score only supported criteria.
- 5
Apply agreed weights.
- 6
Test an alternate weighting.
- 7
Write the advantages, disadvantages and unresolved risks.
Explain the recommendation and its limits
The final note should state which criteria drove the result, where evidence is weak, how sensitive the result is to assumptions and who approves the choice. If two options are close, a pilot may be more honest than declaring a universal winner.
A comparison becomes more defensible when criteria are weighted before the options are scored. Otherwise, reviewers may change the importance of cost, security or usability after seeing a favored result. Record the evidence behind every score and run a sensitivity check: change one important weight and see whether the recommendation changes. A fragile winner should be presented as a trade-off, not an obvious answer. Record what evidence would change the decision later.
- Mandatory requirements are evaluated separately from scores.
- Every material score has evidence or is labeled judgment.
- The recommendation changes only for understandable reasons.
Compare three tools with a transparent matrix
Use a real purchase or workflow decision.
- 1
Define five criteria and two mandatory requirements.
- 2
Collect official evidence.
- 3
Build the matrix.
- 4
Apply weights.
- 5
Run a sensitivity scenario.
- 6
Write a recommendation with risks and owner.
Common mistakes to avoid
- Asking AI to invent criteria and weights after seeing options.
- Treating unknown as zero.
- Using a total score to bypass a mandatory failure.
- Publishing a recommendation without evidence links.
Key takeaways
- Criteria precede scores.
- Evidence and judgment must be distinguishable.
- Sensitivity analysis reveals fragile recommendations.
Frequently asked questions
Should every criterion have a numeric score?
No. Mandatory requirements and qualitative risks may be clearer as pass/fail or narrative evidence.
Can AI choose the weights?
It can suggest questions, but the accountable decision owner should set and approve weights before reviewing the result.
Sources and further reading
Ready to continue?
Mark the lesson complete so your Learning Path progress stays current on this device.