Test two AI chatbot prompts against each other automatically

Randomly shows customers one of two chatbot scripts and keeps the conversation consistent so you can see which performs better.

How the work actually flows

It branches. Exactly one path is taken.

Pattern: Exclusive Choice (4) · Simple Merge (5)

flowchart TD trig>"customer starts new chat"]:::trig s0[("Check for existing script assignment")]:::store s1["Assign script version"]:::task s2["Reply using assigned script"]:::svc trig --> s0 s1 --> s2 gx{"× was script already assigned"}:::gate s0 --> gx p00["Reuse existing script"]:::task gx -->|"Already assigned"| p00 p10["Randomly assign script"]:::task gx -->|"New customer"| p10 jn{"○ continue conversation"}:::gate p00 --> jn p10 --> jn jn --> s1 out[/"consistent script assigned per customer"/]:::out pay{{"enables reliable chatbot wording tests"}}:::pay s2 --> out out --> pay classDef task fill:#e7f6fe,stroke:#34b8f0,color:#2c2a29 classDef svc fill:#f6f8fa,stroke:#7c8795,color:#2c2a29 classDef mi fill:#e7f6fe,stroke:#0079a8,color:#2c2a29,stroke-width:2px classDef human fill:#fff,stroke:#0079a8,color:#0079a8 classDef store fill:#f6f8fa,stroke:#0079a8,color:#2c2a29 classDef trig fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigtime fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigdata fill:#8ad4f5,stroke:#0079a8,color:#06314c,font-weight:bold classDef gate fill:#fff,stroke:#e8a23d,color:#6b4708,font-weight:bold classDef out fill:#1f9d6b,stroke:#167a53,color:#fff,font-weight:bold classDef pay fill:#06314c,stroke:#021f33,color:#fff
Starts itA stepAn outside serviceA record or sheetOne path onlyPaths rejoinResultPayoff
Build size
Advanced

A larger build with multiple systems, AI reasoning, and custom rules.

Business functions
AI Agents & Autonomous SystemsAI Chatbots & AssistantsSpreadsheet & Database Ops
Connects
SupabaseOpenAIPostgreSQL

The problem it solves

You know your AI chatbot's wording matters, but you have no reliable way to compare one version against another without confusing customers mid-conversation. Testing changes by gut feel means you never really know which script gets better results.

Who it fits

Businesses running a customer-facing AI chatbot who want to improve its wording over time.

How it works

  1. A customer starts a new chat conversation
  2. The system checks if this customer has been assigned a script before
  3. If not, it randomly assigns them the standard or the alternative script
  4. The AI chatbot replies using that script for the rest of the conversation
  5. The conversation history is kept so the AI stays consistent
What you get

Which script keeps customers talking longer

Your chatbot conversations reveal which script keeps customers engaged, giving you clear evidence for choosing the better wording.

What you get

A chatbot that consistently uses one of two prompt versions per customer, so you can compare their performance.

What you need

A Supabase account, an OpenAI account, and a PostgreSQL database for chat history.

We can build this. But should you?

The hard question is not how to build it. It is whether this is the right thing to build first.

That is what a Fractional Chief AI Officer figures out with you, before anyone writes a line of code.

Let's Talk Strategy

Related automations

Back to the AI Playbook