Test two different chatbot prompts and compare their performance

Each new chat conversation is randomly given one of two AI prompts, so you can see which one works better.

How the work actually flows

It branches. Exactly one path is taken.

Pattern: Exclusive Choice (4) · Simple Merge (5)

flowchart TD trig>"new chat conversation starts"]:::trig s0[("check for existing prompt assignment")]:::store s1["respond using assigned prompt"]:::svc trig --> s0 gx{"× already assigned a version"}:::gate s0 --> gx p00["reuse existing prompt version"]:::task gx -->|"already assigned"| p00 p10["randomly assign prompt version"]:::task gx -->|"not yet assigned"| p10 jn{"○ before responding"}:::gate p00 --> jn p10 --> jn jn --> s1 out[/"conversation tagged with prompt version"/]:::out pay{{"reliable ab test data"}}:::pay s1 --> out out --> pay classDef task fill:#e7f6fe,stroke:#34b8f0,color:#2c2a29 classDef svc fill:#f6f8fa,stroke:#7c8795,color:#2c2a29 classDef mi fill:#e7f6fe,stroke:#0079a8,color:#2c2a29,stroke-width:2px classDef human fill:#fff,stroke:#0079a8,color:#0079a8 classDef store fill:#f6f8fa,stroke:#0079a8,color:#2c2a29 classDef trig fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigtime fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigdata fill:#8ad4f5,stroke:#0079a8,color:#06314c,font-weight:bold classDef gate fill:#fff,stroke:#e8a23d,color:#6b4708,font-weight:bold classDef out fill:#1f9d6b,stroke:#167a53,color:#fff,font-weight:bold classDef pay fill:#06314c,stroke:#021f33,color:#fff
Starts itAn outside serviceA record or sheetOne path onlyPaths rejoinResultPayoff
Build size
Advanced

A larger build with multiple systems, AI reasoning, and custom rules.

Business functions
AI Agents & Autonomous SystemsAI Chatbots & AssistantsSpreadsheet & Database Ops
Connects
SupabaseOpenAI

The problem it solves

You want to know if a different chatbot prompt performs better, but testing it usually means manually switching versions and losing track of which customer saw which one. Without a controlled test, you're just guessing which wording works best.

Who it fits

A business running an AI chatbot that wants to compare different versions of its instructions.

How it works

  1. A new chat conversation starts
  2. The system checks whether that conversation has already been assigned a prompt version
  3. If not, it randomly assigns either the standard or the alternative prompt
  4. The chatbot responds using whichever prompt was assigned, consistently for that conversation
What you get

The better-performing prompt identified for you

You get a clear read on which chatbot prompt performs better, based on how real conversations actually play out.

What you get

A record of which prompt version each conversation used, so you can compare results between the two.

What you need

A Supabase account and an OpenAI account.

We can build this. But should you?

The hard question is not how to build it. It is whether this is the right thing to build first.

That is what a Fractional Chief AI Officer figures out with you, before anyone writes a line of code.

Let's Talk Strategy

Related automations

Handle sales chat conversations automatically with an AI agent

An AI agent chats with prospects, remembers context, and pulls answers from your own knowledge base.

Advanced

AI assistant that searches your data and the web to answer questions

One AI assistant searches your documents, spreadsheets, and the live web to give complete, accurate answers.

Advanced

Answer customer questions instantly from your own knowledge base

A chat assistant that answers questions using your own documents and remembers the conversation as it goes.

Standard

Build a chatbot that answers from your own documents

This AI chatbot remembers your conversation and searches your private knowledge base to answer accurately.

Advanced
Back to the AI Playbook