Check whether your AI assistant's answers address the question

Scores how closely your AI assistant's answers actually relate to what was asked, flagging off-topic responses.

How the work actually flows

A straight line.

Pattern: Sequence (1)

flowchart TD trig(("team runs relevance check")):::human s0["assistant answers test question"]:::task s1["AI generates question from answer"]:::task s2["compare generated to original question"]:::task s3[("calculate and record relevance score")]:::store trig --> s0 s0 --> s1 s1 --> s2 s2 --> s3 out[/"relevance score for assistant answers"/]:::out pay{{"catches off-topic ai answers"}}:::pay s3 --> out out --> pay classDef task fill:#e7f6fe,stroke:#34b8f0,color:#2c2a29 classDef svc fill:#f6f8fa,stroke:#7c8795,color:#2c2a29 classDef mi fill:#e7f6fe,stroke:#0079a8,color:#2c2a29,stroke-width:2px classDef human fill:#fff,stroke:#0079a8,color:#0079a8 classDef store fill:#f6f8fa,stroke:#0079a8,color:#2c2a29 classDef trig fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigtime fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigdata fill:#8ad4f5,stroke:#0079a8,color:#06314c,font-weight:bold classDef gate fill:#fff,stroke:#e8a23d,color:#6b4708,font-weight:bold classDef out fill:#1f9d6b,stroke:#167a53,color:#fff,font-weight:bold classDef pay fill:#06314c,stroke:#021f33,color:#fff
A stepA personA record or sheetResultPayoff
Build size
Advanced

A larger build with multiple systems, AI reasoning, and custom rules.

Business functions
AI Agents & Autonomous SystemsAI Chatbots & AssistantsKnowledge Base & RAG
Connects
OpenAIGoogle Sheets

The problem it solves

Your AI assistant might sound confident while drifting off-topic, padding answers with irrelevant information, or only partially answering the question you actually asked. Without a way to measure this, off-topic or bloated answers can slip through unnoticed.

Who it fits

A business running a question-and-answer AI assistant that wants to catch answers that miss the point.

How it works

  1. The AI assistant answers a test question
  2. AI generates a new question based on that answer
  3. The generated question is compared to the original question for similarity
  4. A relevance score is calculated from that comparison
  5. You receive a score showing how on-topic the assistant's answers are
What you get

On-topic scores you can track over time

You get a clear score showing how closely your AI assistant's answers actually match what was asked.

What you get

A relevance score showing how closely the assistant's answers match the original questions.

What you need

An OpenAI account and a place to store your test questions, such as a spreadsheet.

We can build this. But should you?

The hard question is not how to build it. It is whether this is the right thing to build first.

That is what a Fractional Chief AI Officer figures out with you, before anyone writes a line of code.

Let's Talk Strategy

Related automations

Back to the AI Playbook