Response quality scoring and feedback

Omnifox scores your agent's replies and flags weak ones; review them and leave your human feedback.

Jun 26, 2026

Watch this step in the video · Reply quality

What quality scoring is

Omnifox automatically reviews your AI agents' replies with a quality judge (another AI that grades them), so you catch weak or risky answers without reading every conversation by hand. Each reply gets an overall score broken down into four dimensions: relevance, safety, coherence and helpfulness. Replies that fall below the threshold get flagged for your review.

How to open the Quality panel

  1. In the side menu open AI Agents.
  2. Open the Quality section (page /ai-agents/{id}/quality).
  3. At the top you'll see the totals: scored replies, flagged %, reviewed % and average overall score, along with the active flagging threshold.

How to review and give feedback

  1. Use the filters: by agent, by range (1 day, 7 days, 30 days or all), and the Flagged only toggle to focus on what matters most.
  2. Open a reply to see the per-dimension breakdown and the judge's comment.
  3. Click Review and set your human verdict: good, bad or neutral. You can add a comment explaining why.
  4. Save. Your decision is recorded and counts toward the reviewed %.

This human feedback is gold: it confirms (or corrects) the automatic judge and surfaces patterns you can fix in the agent's instructions.

How to verify it's working

After a few hours of traffic, return to the panel: you should see scored replies and, if there were weak cases, some flagged ones. Turn on Flagged only and confirm you can open and rate each one.

If it doesn't work

  • No scored replies: the agent hasn't produced replies in that range yet, or scoring is still processing. Widen the range to 30 days.
  • Too many flagged: this is usually an instructions or knowledge-base problem, not the judge. Read 5–10 flagged cases and look for the pattern.
  • No Quality section: you need the ai-agents.view permission.

Tip: spend 10 minutes a day reviewing only the flagged ones. It's the fastest way to raise your agent's real quality.

Tip: when you mark several replies "bad" for the same reason, edit the agent's instructions and re-measure a week later.

Related articles

Was this helpful?

Related articles

Message us on WhatsApp