Test how accurately an AI legal search tool finds answers

Runs a batch of legal questions through an AI search system and measures how often it finds the right answer.

How the work actually flows

It branches. Every path runs; all paths must finish before it continues; runs once per each test question.

Pattern: Parallel Split (2) · Synchronisation (3) · Multiple Instances without Synchronization (12)

flowchart TD trig[\"batch of test questions loaded"\]:::trigdata s0["search legal document database"]:::task s1["rank combined results"]:::task s2[["check against correct answer"]]:::mi s3["calculate accuracy score"]:::task trig --> s0 s1 -->|"one per each test question"| s2 s2 --> s3 gx{"+ which search method to use"}:::gate s0 --> gx p00["keyword matching"]:::task gx -->|"keyword search"| p00 p10["ai based matching"]:::task gx -->|"semantic search"| p10 jn{"+ combine results"}:::gate p00 --> jn p10 --> jn jn --> s1 out[/"accuracy score for search tool"/]:::out pay{{"confidence before trusting real cases"}}:::pay s3 --> out out --> pay classDef task fill:#e7f6fe,stroke:#34b8f0,color:#2c2a29 classDef svc fill:#f6f8fa,stroke:#7c8795,color:#2c2a29 classDef mi fill:#e7f6fe,stroke:#0079a8,color:#2c2a29,stroke-width:2px classDef human fill:#fff,stroke:#0079a8,color:#0079a8 classDef store fill:#f6f8fa,stroke:#0079a8,color:#2c2a29 classDef trig fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigtime fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigdata fill:#8ad4f5,stroke:#0079a8,color:#06314c,font-weight:bold classDef gate fill:#fff,stroke:#e8a23d,color:#6b4708,font-weight:bold classDef out fill:#1f9d6b,stroke:#167a53,color:#fff,font-weight:bold classDef pay fill:#06314c,stroke:#021f33,color:#fff
Starts itA stepRuns once per itemEvery pathWaits for allResultPayoff
Build size
Advanced

A larger build with multiple systems, AI reasoning, and custom rules.

Business functions
Knowledge Base & RAG
Connects
Qdrant

The problem it solves

When you're building an AI tool to search legal documents, you need proof it actually works before you trust it with real cases. Testing accuracy by hand, question by question, is slow and hard to repeat consistently.

Who it fits

Legal tech teams or firms building an AI-powered document search tool who need to measure its accuracy.

How it works

  1. A batch of legal questions and their known correct answers is loaded in
  2. The system searches a legal document database using both keyword and AI-based matching
  3. The two search methods are combined to rank the best matching passages
  4. Each result is checked against the correct answer
  5. An accuracy score is calculated across all the questions
What you get

Accuracy numbers for your search tool

You get a clear accuracy score showing how often your legal search tool finds the right answer.

What you get

An accuracy score showing how often the search tool retrieves the correct answer.

What you need

A Qdrant vector database account and access to an embedding and search API.

We can build this. But should you?

The hard question is not how to build it. It is whether this is the right thing to build first.

That is what a Fractional Chief AI Officer figures out with you, before anyone writes a line of code.

Let's Talk Strategy

Related automations

Recommend products to customers with an AI chat assistant

An AI chat assistant searches your product catalog and instantly recommends the best matches to each customer.

Advanced

Find the most relevant document for any search term

Type a keyword and AI scores your whole document library to surface the single best-matching document.

Advanced

Manage and edit your AI chatbot's knowledge base directly

Lets you add, update, or delete the information stored in your AI chatbot's knowledge base without writing any code.

Standard

Turn your documents into a searchable AI knowledge assistant

Upload PDFs, images, or web links and get an AI chatbot that answers questions only from your own documents.

Advanced
Back to the AI Playbook