What you will learn
- Recognize common factual and reasoning failure modes.
- Distinguish missing context from hallucinated detail.
- Test references, quotations, dates and scope.
- Rewrite output to preserve uncertainty and limitations.
What you need
- An AI answer containing at least five factual claims.
- Access to official or primary sources for the topic.
Learn the failure modes you can actually observe
OpenAI documents that ChatGPT may generate incorrect facts, fabricated quotations, studies or citations, and overconfident answers to ambiguous questions.
NIST’s Generative AI Profile identifies risks that include confabulation, harmful bias, privacy, information integrity and over-reliance, and recommends managing them in context.
A hallucination is not the only reason an answer fails. The model may lack the latest information, misread an ambiguous term, rely on an unstated assumption, apply a rule from the wrong country, or compress a nuanced debate into one confident paragraph. Naming the failure mode tells you what correction is possible.
Mark claims before judging the prose
Read the answer once for meaning, then a second time as an auditor. Highlight every date, number, quotation, named study, product capability, legal requirement and causal statement. Also mark vague phrases such as “experts agree” or “research shows,” because they hide the evidence you would need.
- 1
Copy the answer into a review document.
- 2
Split combined statements into atomic claims.
- 3
Mark claims that are current, quantitative or high impact.
- 4
Identify terms that could have multiple meanings.
- 5
List assumptions the answer made about location, version, audience or time.
- 6
Flag every citation that has not been opened.
Use the smallest test that can disprove the claim
Verification should be adversarial enough to detect a confident mistake. Search the exact title of a cited document, compare the publication date, open the original page and inspect whether the wording supports the same scope. For calculations, reproduce the formula. For product capabilities, check current documentation and account availability. If the claim cannot be tested, label it as an opinion, estimate or unresolved assertion.
- 1
Start with the highest-impact claim.
- 2
Locate the responsible organization or original publication.
- 3
Check the title, author, date and version.
- 4
Compare population, conditions, units and exceptions.
- 5
Recalculate quantitative statements when possible.
- 6
Seek a second independent source for disputed claims.
- 7
Remove or narrow claims that remain unsupported.
Correct the record, not only the wording
When a claim is partly supported, rewrite it to match the evidence rather than keeping the original and adding a vague disclaimer. Preserve dates, versions and uncertainty. If the answer mixed sourced fact with generated interpretation, separate those layers. For consequential use, record what was checked and by whom.
- Every important citation opens and supports the nearby claim.
- Current facts include a date or version.
- Uncertainty is visible instead of buried in a final sentence.
Run a limitation audit
Audit one AI answer and produce a corrected version with an evidence log.
- 1
Extract at least eight atomic claims.
- 2
Rank them by consequence.
- 3
Verify the top three with primary sources.
- 4
Test one quotation or reference independently.
- 5
Identify one hidden assumption.
- 6
Rewrite the answer with verified wording and labeled uncertainty.
Common mistakes to avoid
- Equating detail with accuracy.
- Checking only whether a source exists, not whether it supports the claim.
- Ignoring version, date or jurisdiction.
- Leaving a false claim in place and adding “may vary.”
Key takeaways
- Hallucinations are one of several observable failure modes.
- Atomic claims are easier to verify than fluent paragraphs.
- A correction must change the claim to match the evidence.
Frequently asked questions
Can search-enabled AI still make mistakes?
Yes. Retrieval can improve freshness and traceability, but sources may be misunderstood, incomplete or irrelevant. Open the evidence.
What if reliable sources disagree?
Describe the disagreement, compare methods and dates, and avoid presenting one side as settled unless the evidence supports that conclusion.
Sources and further reading
Ready to continue?
Mark the lesson complete so your Learning Path progress stays current on this device.