What matters for your decision
Set warning and stop thresholds by severity, especially for permission breaches or irreversible mistakes; don’t average those away in overall satisfaction or accuracy scores.
Make the metric auditable
- Define success and fail conditions for specific intents before opening the pilot.
- Review low-volume severe incident categories separately from percentage-based quality metrics.
- Keep metrics attributable to a prompt, source, workflow, or model version so you can diagnose a sudden change.