Run side-by-side tests to see which AI setup performs best

The system randomly splits incoming requests between two setups so you can see which one gets better results.

How the work actually flows

It branches. Exactly one path is taken.

Pattern: Exclusive Choice (4) · Simple Merge (5)

flowchart TD trig>"new request arrives"]:::trig s0["Receive incoming request"]:::task s1["Run assigned setup"]:::task s2[("Track results for comparison")]:::store trig --> s0 s1 --> s2 gx{"× which setup gets assigned"}:::gate s0 --> gx p00["Run setup A"]:::task gx -->|"Setup A"| p00 p10["Run setup B"]:::task gx -->|"Setup B"| p10 jn{"○ results tracked"}:::gate p00 --> jn p10 --> jn jn --> s1 out[/"performance comparison data"/]:::out pay{{"data-backed proof of best setup"}}:::pay s2 --> out out --> pay classDef task fill:#e7f6fe,stroke:#34b8f0,color:#2c2a29 classDef svc fill:#f6f8fa,stroke:#7c8795,color:#2c2a29 classDef mi fill:#e7f6fe,stroke:#0079a8,color:#2c2a29,stroke-width:2px classDef human fill:#fff,stroke:#0079a8,color:#0079a8 classDef store fill:#f6f8fa,stroke:#0079a8,color:#2c2a29 classDef trig fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigtime fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigdata fill:#8ad4f5,stroke:#0079a8,color:#06314c,font-weight:bold classDef gate fill:#fff,stroke:#e8a23d,color:#6b4708,font-weight:bold classDef out fill:#1f9d6b,stroke:#167a53,color:#fff,font-weight:bold classDef pay fill:#06314c,stroke:#021f33,color:#fff
Starts itA stepA record or sheetOne path onlyPaths rejoinResultPayoff
Build size
Standard

A mid-size build with several tools working together.

Business functions
AI Agents & Autonomous Systems
Connects
OpenAIAnthropicGoogle Gemini

The problem it solves

You suspect one version of your automated process works better than another, but you have no easy way to prove it. Testing changes one at a time is slow, and relying on gut feel isn't a strategy.

Who it fits

Any business running an AI-powered process who wants data-backed proof of which version performs better.

How it works

  1. A new request comes in and needs to be handled
  2. The system randomly assigns it to one of your chosen setups based on weights you set
  3. That setup runs and produces its result
  4. Results are tracked so you can compare which version performs better over time
What you get

Setups compared on real results

Your incoming requests get split automatically between two AI setups, so you can see which one performs better over time.

What you get

A side-by-side performance comparison of two or more automated approaches.

What you need

Whatever accounts already power your existing automated process, plus a place to store the comparison results.

We can build this. But should you?

The hard question is not how to build it. It is whether this is the right thing to build first.

That is what a Fractional Chief AI Officer figures out with you, before anyone writes a line of code.

Let's Talk Strategy

Related automations

Track your brand's visibility across AI search assistants

Automatically asks multiple AI assistants your key search terms and tracks whether and how your brand gets mentioned.

Advanced

Track AI usage costs automatically in a spreadsheet

Logs how many tokens your AI agent uses each run and calculates the estimated cost in Google Sheets.

Advanced

Get daily AI stock analysis and alerts sent to Telegram

Combines stock data, news, and sentiment through three AI models and sends a daily consensus alert to Telegram.

Advanced

Automatically test and score your AI assistants for accuracy

Runs test datasets through your AI assistants and scores their accuracy, tone, and helpfulness automatically.

Advanced
Back to the AI Playbook