Check whether your AI chatbot's answers are actually accurate

Tests your AI assistant's answers against its source documents to catch made-up or unsupported claims.

How the work actually flows

A straight line. Runs once per each test question.

Pattern: Sequence (1) ยท Multiple Instances without a priori Run-Time Knowledge (15)

flowchart TD trig(("user runs test questions")):::human s0[["collect answer and source documents"]]:::mi s1["compare answer to source documents"]:::svc s2["produce groundedness score"]:::task trig --> s0 s0 --> s1 s1 --> s2 out[/"groundedness score with flagged answers"/]:::out pay{{"confidence the chatbot isn't making things up"}}:::pay s2 --> out out --> pay classDef task fill:#e7f6fe,stroke:#34b8f0,color:#2c2a29 classDef svc fill:#f6f8fa,stroke:#7c8795,color:#2c2a29 classDef mi fill:#e7f6fe,stroke:#0079a8,color:#2c2a29,stroke-width:2px classDef human fill:#fff,stroke:#0079a8,color:#0079a8 classDef store fill:#f6f8fa,stroke:#0079a8,color:#2c2a29 classDef trig fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigtime fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigdata fill:#8ad4f5,stroke:#0079a8,color:#06314c,font-weight:bold classDef gate fill:#fff,stroke:#e8a23d,color:#6b4708,font-weight:bold classDef out fill:#1f9d6b,stroke:#167a53,color:#fff,font-weight:bold classDef pay fill:#06314c,stroke:#021f33,color:#fff
A stepAn outside serviceRuns once per itemA personResultPayoff
Build size
Advanced

A larger build with multiple systems, AI reasoning, and custom rules.

Business functions
AI Agents & Autonomous SystemsAI Chatbots & AssistantsKnowledge Base & RAGReporting & Analytics
Connects
OpenAI

The problem it solves

You've built an AI assistant that answers questions using your own documents, but you can't always tell if it's making things up. If it starts inventing facts that aren't in your source material, customers or employees could get bad information without anyone noticing.

Who it fits

Businesses running an AI chatbot or assistant that answers from their own documents and need to verify accuracy.

How it works

  1. Starts when you run a set of test questions through your AI assistant
  2. Collects the assistant's answer along with the documents it pulled from
  3. A second AI compares the answer against those documents
  4. Flags any part of the answer that isn't actually backed up by the source material
  5. Produces a score showing how well-grounded the assistant's answers are
What you get

Chatbot answers you know you can trust

You get a groundedness score showing how well your AI assistant's answers are actually backed by your documents.

What you get

A groundedness score and flagged examples showing where the AI assistant strayed from its source documents.

What you need

An OpenAI API key and a vector store of your documents.

We can build this. But should you?

The hard question is not how to build it. It is whether this is the right thing to build first.

That is what a Fractional Chief AI Officer figures out with you, before anyone writes a line of code.

Let's Talk Strategy

Related automations

Manage your Google Calendar and get a daily AI briefing by chat

Book, reschedule, and cancel meetings by chatting or talking, and get a morning summary of your day.

Advanced

Get instant crypto market answers through a Telegram chatbot

Ask about token prices and trading pairs in Telegram and get real-time crypto market data back instantly.

Advanced

Summarize YouTube videos and chat about them in Telegram

Send a YouTube link to Telegram and get an AI summary, then ask follow-up questions about the video.

Advanced

Turn your documents into a searchable AI knowledge assistant

Upload PDFs, images, or web links and get an AI chatbot that answers questions only from your own documents.

Advanced
Back to the AI Playbook