AI answer monitoring checklist
The sections below outline ten practical steps. Each includes a short explanation and concrete items to confirm before, during, and after a monitoring cycle.
1. Define the monitoring scope
Scope determines what is monitored, for whom, and against what standard. Clear scope prevents scope drift and makes findings comparable across audits.
- Products, families, or SKUs in scope are listed and versioned.
- Regions and languages in scope are defined.
- User personas (patient, caregiver, clinician, distributor, support) are identified.
- Business owners for review are named.
- Out-of-scope items are recorded to prevent scope creep.
2. Select AI platforms and answer sources
Different platforms behave differently. Decide which will be tested and record why.
- Public AI systems selected (for example, general-purpose assistants and AI search overviews).
- Brand or partner chatbots included where in scope.
- Third-party chatbots (distributor, retailer) considered.
- Platform, model, and product versions recorded where visible.
- Session settings such as memory or custom instructions documented.
3. Build a prompt library
A prompt library is a reusable set of questions that reflects how real users ask about the product. It should evolve with the portfolio.
- Prompts reflect real user language, not internal terminology.
- Coverage includes safety, indications, compatibility, cleaning, and troubleshooting.
- Regional and language variants are included where relevant.
- Prompts are grouped by intent and persona.
- The library is versioned and reviewed periodically.
4. Collect source materials
Answers are only meaningful against a reference. Collect the approved materials that will serve as the source of truth for comparison.
- Current IFUs and labeling are available for each in-scope product.
- Approved marketing claims are identified.
- Region-specific documents are collected where required.
- Superseded documents are marked and set aside.
- A single owner is designated to keep source materials current for the monitoring cycle.
5. Capture evidence consistently
Evidence quality determines how useful findings are for internal review. Use a fixed capture template.
- Product, region, and language logged for every observation.
- AI platform, model or product name, and version recorded.
- Prompt captured verbatim.
- Full AI output captured as text and as a screenshot.
- Date and time of testing recorded, including time zone.
- Cited sources captured where visible.
6. Classify answer defects
Use a consistent taxonomy so findings can be aggregated and reviewed. A shared taxonomy also supports retesting.
- Defect category assigned from a defined taxonomy.
- Source-comparison note identifies the specific gap.
- Answer components (claims, warnings, instructions) tagged individually.
- Ambiguities flagged rather than force-fit into a category.
7. Assign severity or review priority
Severity should reflect the client's internal risk criteria, not vendor-defined scoring.
- Severity criteria documented and applied consistently.
- High-severity items include a rationale.
- Review priority mapped to internal owners.
- Recommended next step recorded for each finding.
8. Prepare findings for internal review
Findings should arrive in a form that qualified teams can read, prioritize, and act on.
- Findings organized by product, region, and defect category.
- Evidence attached to each finding.
- Reviewer notes field available for internal comments.
- Recommended actions phrased as options, not directives.
9. Retest over time
AI systems change. Retesting on a defined cadence, and after labeling or product changes, is how trends emerge.
- Retest cadence agreed (for example, monthly or quarterly).
- Retest triggered after IFU or product changes.
- Retest date recorded for each finding.
- Trends compared across audits to distinguish patterns from noise.
10. Document limitations
Being explicit about limits keeps findings credible and useful.
- Time-specific nature of observations stated.
- Platform variability acknowledged.
- Scope boundaries repeated in the report.
- Disclaimer confirming findings support, and do not replace, qualified internal review.
Disclaimer. Answer Assurance findings are designed to support internal review by qualified client teams. They do not replace legal, regulatory, clinical, medical, quality, or compliance judgment.