QA Suites & Active Learning

divinci.qa (Scored QA) is a production-grade evaluation, calibration, and optimization suite built for domain-specific knowledge bases and Retrieval-Augmented Generation (RAG) pipelines. Rather than relying on generic, uncalibrated LLM benchmarks, Scored QA bridges the gap between offline developer-led evaluation and online user feedback. It implements active-learning loops, human-anchored judge calibration, and autonomous agent-driven optimization (Auto-Fix/AutoRAG) to continually audit and heal LLM applications.

Below is the comprehensive technical reference for the Scored QA ecosystem, explaining how every system works in detail, along with their architecture, APIs, mathematical models, and deployment configurations.

1. Architectural Blueprint & Life Cycle

The Scored QA system is designed around a self-improving feedback loop. It bridges offline calibration (verifying prompts, configurations, and models against a gold-standard benchmark) and online production (real user interactions in the RAG Arena).

 ┌────────────────────────────────────────────────────────┐
 │                      RAG ARENA                         │
 │  (Online User Interacting with Model & Custom RAG)     │
 └───────────┬────────────────────────────────────────────┘
             │                               ▲
             │ [Promote Arena Impression]    │ [Graduate Route/Variant]
             ▼ (#234 promoteFromArena)        │ (#237 graduateToArena)
 ┌───────────────────────────────────────────┴────────────┐
 │                     CALIBRATION                        │
 │  (Human Expert Gold Ratings vs. Automated LLM Judges)  │
 └───────────┬────────────────────────────────────────────┘
             │                               ▲
             │ [Select Highest ρ Judge]      │ [Auto-Apply Prompt Patches]
             ▼                               │ (Iteration & Patching)
 ┌───────────────────────────────────────────┴────────────┐
 │                      EVALUATION                        │
 │  (QA Suites, Scorer Weights, & Multi-Release Runs)     │
 └───────────┬────────────────────────────────────────────┘
             │                               ▲
             │ [Calculate Escrow Budget]     │ [Tune parameters: k, chunk]
             ▼                               │ (AutoRAG Agent Cycle)
 ┌───────────────────────────────────────────┴────────────┐
 │                 AUTONOMOUS AUTO-FIX                    │
 │  (LLM Healing Agents Diagnostic Optimization Loops)   │
 └────────────────────────────────────────────────────────┘

The cycle consists of four primary stages:

Promote from Arena: Real-world user interactions and “winner picks” from live arena sessions are captured. High-disagreement cases (where human choice differs from automated scoring) are flagged via active learning and promoted into the QA Suite as a calibration corpus, preserving their original RAG routing provenance.
Human-Anchored Calibration: Domain experts rate model responses against gold-standard rubrics using a streamlined card-based UI. The system computes statistical agreement (Spearman’s $\rho$ and Quadratic Weighted Kappa) between the human raters and various LLM-based automated judges, nominating the highest-correlation judge as the workspace’s “defensible default.”
Scored QA Evaluation: Automated test runs evaluate multiple pipeline variants, prompt revisions, or models against the suite, using the calibrated judge to score the output on modular metrics like relevance, faithfulness, and completeness.
Autonomous Auto-Fix / AutoRAG: Autonomous optimization loops analyze failures, flag low-performing chunked sources, propose and deploy prompt patches, and run grid searches over RAG retrieval parameters (such as $k$, chunk sizing, and rerank thresholds) to maximize the QA score.

2. Core Evaluation Engine (`QASuite` & `QATest`)

At the base of the system are QA Suites (QASuite) and QA Tests (QATest). A suite is a logical grouping of test prompts, gold-standard reference answers, and scorer configurations.

2.1 Test Cases and Golden Answers

Each QATest within a suite represents an evaluation item consisting of:

Prompt: The user input/query to be tested.
Golden Answer (Optional): A reference ground-truth response written by a domain expert or validated during calibration.
RAG Source Document (Optional): Links back to the original source chunk or vector document from which the question was generated.

SDK: Programmatic Suite Creation & List

import { divinci } from "@divinci-ai/sdk";

// Initialize and create a suite
const suite = await divinci.qa.createSuite("workspace_abc", {
  name: "Legal Intake Verification",
  description: "Evals for handling complex legal intake prompts and routing",
  scoreGenerators: [
    { source: { type: "llm-correctness" }, weight: 2.0 },
    { source: { type: "llm-completeness-coverage" }, weight: 1.0 },
    { source: { type: "llm-faithfulness" }, weight: 1.0 }
  ],
  suggestionGeneration: {
    merger: { id: "llama-3-3-70b-instruct" }
  }
});

// Load all suites in a workspace
const suites = await divinci.qa.listSuites("workspace_abc");

2.2 Execution & Cost Accounting

Running a large QA suite on every single prompt or model alteration can be slow and extremely expensive. The Scored QA execution engine incorporates several features to make this scalable.

Cost & Latency Estimation (`/estimate`)

Before a multi-release or Arena run is executed, clients should hit the estimation endpoint. This returns a dry-run cost projection based on the current pricing structure of target LLM APIs, input/output token sizes, and the count of score generators:

const estimate = await divinci.qa.estimateRun("workspace_abc", suite._id, {
  releaseId: "rel_12345",
  maxTests: 150
});

console.log(`Estimated Cost: $${estimate.totalCostUSD}`);
console.log(`Estimated Latency: ${estimate.latencySeconds}s`);

Online vs. Asynchronous Batch Runs

To safeguard against rate-limits, network timeouts, and performance bottlenecks, the system dynamically switches execution modes based on suite scale:

Online Execution: Small runs (e.g., less than 10 tests) are run in a concurrent thread pool (Promise.all with a customizable concurrency window) and return the results immediately.
Asynchronous Batch Jobs (should-use-batch.ts): For larger runs, the API queues a batch job, allocates a budget escrow, and delegates running to an background worker (advance-batch-job.ts). This worker processes requests sequentially or in metered chunks, saving up to 60% in costs by utilizing batch API endpoints (such as Google Vertex Batch or OpenAI Batch API) when available, and persisting intermediate states to handle cold container restarts.

Ejecting Active Runs (`/cancel`)

If a run behaves poorly (e.g., the prompt is garbled and returning empty strings, causing a massive score drop), developers can programmatically eject. The cancellation worker (cancel-run.ts) immediately flags the running job. The batch worker polls for this flag on its next inter-batch boundary (~6-second frequency) and exits cleanly, releasing the remaining budget escrow.

2.3 Suite Configuration Serialization

For teams practicing GitOps or strict version control, suites can be exported and imported as round-trippable YAML or JSON files:

// Export a suite schema to YAML for version control
const yamlData = await divinci.qa.exportSuite("workspace_abc", suite._id, { format: "yaml" });

// Import a suite schema to seed another environment
const newSuite = await divinci.qa.importSuite("workspace_abc", {
  format: "yaml",
  content: yamlData
});

3. Score Generators & Grading Rubrics

The evaluation of a response’s quality is handled by modular Score Generators (graders). These judges use structured prompting to output granular evaluations, scores, and natural language reasoning.

3.1 Rubric Structure

Each score generator attaches to a specific rubric version. When a judge receives a model’s output, it grades it against the rubric template. The rubric prompt contains strict JSON output instructions, returning a score between 0 and 1 (typically quantized in steps of 0.25 on the frontend for rating ease, but evaluated continuously on the backend).

Available Out-of-the-Box Scorers:

Scorer Key	Evaluation Focus	Output Profile
`llm-correctness`	Compares model response against the golden answer for factual consistency.	Binary/Continuous
`llm-completeness-coverage`	Evaluates if all specific instructions in the prompt or reference guidelines were met.	Ordinal Scale
`llm-faithfulness`	Measures RAG hallucination: determines if the response is fully supported by the retrieved context.	Binary/Continuous
`llm-relevance`	Verifies if the model answered the user’s question directly without extraneous fluff.	Continuous
`llm-tone-alignment`	Grades response tone against predefined brand guidelines.	Continuous

4. Human-Anchored Calibration & Statistics

While automated judges are essential for continuous evaluation, they suffer from judgment drift—LLM judges routinely disagree with each other (empirical Spearman correlation rho is approximately 0.66 to 0.75). To make evaluations defensible, Divinci anchors judges to human domain experts.

4.1 The Calibration Data Model (`ScoredQAHumanRating`)

The ScoredQAHumanRating document captures human judgments of a model’s response compared to automated scoring:

{
  "_id": "660c1d2e5a420b12cd53ef91",
  "workspace": "ws_12345",
  "rater": {
    "userId": "auth0|6123abc456",
    "displayName": "Dr. Sarah Jenkins",
    "role": "domain-expert"
  },
  "testResultId": "660c1d1a5a420b12cd53ef22",
  "scorerKey": "llm-completeness-coverage",
  "rubricVersion": "a1f9e2b8c7d6e5f4",
  "score": 0.75,
  "reasoning": "Missed the secondary warning instruction, but correct overall.",
  "durationMs": 14200,
  "createdAt": "2026-07-06T09:12:44.200Z"
}

The rubricVersion is a server-side sha256(scorer.prompt).slice(0, 16) hash of the evaluator’s prompt template. If an administrator edits the grader’s system instructions, historical ratings under the old version are marked as stale, preventing them from skewing the calibration statistics.

4.2 Inter-Rater Reliability & Agreement Statistics

When multiple raters score responses, or when we compare human raters against automated LLM judges, the system evaluates calibration metrics via /agreement and /inter-rater.

Mathematical Formulations

1. Spearman’s Rank Correlation Coefficient (`rho`)

Spearman’s rho is the primary metric for continuous/ordinal ratings, measuring how closely the rank ordering of outputs by an LLM judge matches that of a human expert.

Given n paired ratings, the raw scores are converted to ranks, and Spearman’s rho is computed as:

rho = 1 - (6 * sum(d_i^2)) / (n * (n^2 - 1))

where d_i is the difference between the ranks of each pair.

2. Quadratic Weighted Kappa (`wk`)

While Spearman measures monotonic correlation, it does not adjust for chance agreement or penalize large-scale rating gaps effectively (e.g., human rates 1.0 but judge rates 0.0). For this, we calculate Quadratic Weighted Kappa:

kappa_w = 1 - (sum(w_i,j * O_i,j)) / (sum(w_i,j * E_i,j))

Where:

O_i,j is the observed proportion of times rater 1 assigns score i and rater 2 assigns score j.
E_i,j is the expected proportion of agreements under chance (the outer product of the raters’ marginal distributions).
w_i,j is the quadratic weight of disagreement, defined as:

w_i,j = (i - j)^2 / (k - 1)^2

with k being the number of possible rating levels.

Defensible Judge Nominations

If an LLM judge’s correlation with the domain expert’s ratings achieves rho >= 0.85 across a minimum sample size (n >= 30, recommended n >= 50), the system nominates it as the Defensible Default Judge. All future automated runs are scored under this judge.

// Fetch agreement metrics for a suite
const metrics = await divinci.qa.getCalibrationAgreement("workspace_abc", {
  suiteId: suite._id,
  scorerKey: "llm-correctness"
});

console.log(`Recommended Judge: ${metrics.recommended_default_judge}`);
console.log(`Spearman Rho (ρ): ${metrics.rho_per_judge[0].rho}`);

4.3 Next-Priority Calibration Selection (`/next-priority`)

To optimize domain expert time, the calibration interface does not present tests randomly. The /next-priority endpoint selects the next 10-20 items to rate using an active-learning selection algorithm:

It prioritizes high-disagreement items: test cases where different automated judges (or different raters) show high variance.
It prioritizes low-confidence items: cases where the judge was closest to a boundary threshold.
It excludes items already rated by the current user under the current rubricVersion to prevent duplication.

5. RAG Arena × Scored QA Active-Learning Loop

A key innovation of the Divinci platform is the unification of offline calibration and live RAG Arena environments. Rather than keeping them isolated, data flows bidirectionally between them.

       ┌────────────────────────┐
       │   RAG Arena (Online)   │
       └───────────┬────────────┘
                   │
                   │ [Promote winner-picks]
                   ▼
       ┌────────────────────────┐
       │   Scored QA (Offline)  │
       └───────────┬────────────┘
                   │
                   │ [Calibrate & evaluate]
                   ▼
       ┌────────────────────────┐
       │  Graduate High-Winners │
       └────────────────────────┘

5.1 Route-Aware Performance Aggregation (`routePerformance`)

To tie evaluation directly to architectural routing, QATestResult documents store routing snapshots taken at runtime:

ragVectorGroupId: The group of vector indexes active during generation.
ragVectorIndexId: The specific vector index searched.
releaseSlugVersion: The prompt version or deployment variant used.

The /route-performance endpoint groups and aggregates QA test scores by these fields. This lets developers pinpoint exactly which vector indexes or prompt variants are causing regressions, and which ones are excelling.

5.2 Promoting Arena Impressions (`promoteFromArena`)

When a real user picks a variant output in a live RAG Arena session (a “winner pick”), this interaction holds high signal. Under /promote-from-arena:

The system freezes the prompt, response, and active RAG index state.
It transforms the winner-pick impression into a QATest within the specified calibration suite.
This continuously populates the calibration corpus with real-world edge cases.

5.3 Active-Learning Ranker (`activeLearningCandidates`)

The active-learning candidate ranker (activeLearningCandidates) scans live arena logs to surface the highest-EV messages to add to the suite. It calculates judge-vs-human disagreement:

EV_improvement is proportional to:
abs(HumanWinnerScore - max(JudgeScore(j) for j in RejectedVariants))

If a human user picked a response that the automated judge scored poorly, or if the judge assigned its highest score to a response the human rejected, this indicates a massive gap in the judge’s understanding. Adding this case to the calibration suite ensures the judge gets trained on its blind spots.

5.4 Pre-filtering Arena Candidates (`arenaPrefilter`)

To minimize the substantial cost of generating responses for $N$ different variants during active testing, the /arena-prefilter endpoint acts as a gatekeeper:

It runs a lightweight, fast scoring model to predict the weighted-mean score of proposed variants.
It returns a recommendedKeep array: keeping the Top-K Warm (historically high-performing) variants, while keeping All Cold (new, exploratory) variants active.
This prunes clearly underperforming variants before they trigger costly downstream generations, cutting LLM operational costs by up to 40%.

5.5 Graduation & Demotion (`graduateToArena`)

Once an offline variant or routing configuration consistently achieves superior scores on the calibrated QA Suite, it can be programmatically graduated into the live RAG Arena preset. Conversely, live variants that have dropped in performance are automatically demoted. Running graduateToArena in apply mode automatically updates the workspace’s active routing configuration.

6. Autonomous Auto-Fix & AutoRAG Agent Loop

Evaluating models is only half the battle; fixing them is the other. The Scored QA suite features an autonomous healing agent pipeline designed to self-correct prompt templates and RAG configurations.

6.1 The Auto-Fix Engine & Escrow

An Auto-Fix Loop is a long-running, autonomous agent run. Because agents can easily spiral out of control and consume thousands of dollars in API calls, the system implements a strict Escrow Budgeting System (calculate-escrow):

Before a loop starts, the system calculates the maximum possible cost based on maxIterations, test counts, and the token bounds of the models used.
This amount is frozen as an escrow. If the agent runs out of escrow mid-run, it pauses and fires a webhook requiring manual escrow resolution (resolveEscrowFailure).

6.2 Diagnostic & Iteration Cycles

During an iteration (iteration.ts), the agent executes the following loop:

┌─────────────────────────────────────────────────────────┐
│                    RUN QA EVALUATION                    │
│      Evaluate current release against the QA suite       │
└───────────────────────────┬─────────────────────────────┘
                            ▼
┌─────────────────────────────────────────────────────────┐
│                    DIAGNOSE FAILURES                    │
│  Identify low-scoring test items and parse reasoning    │
└───────────────────────────┬─────────────────────────────┘
                            ▼
┌─────────────────────────────────────────────────────────┐
│               IDENTIFY FLAG CHUNKS & PATHS              │
│ Locate problematic RAG chunks or prompt phrasing defects│
└───────────────────────────┬─────────────────────────────┘
                            ▼
┌─────────────────────────────────────────────────────────┐
│                 PROPOSE & APPLY PATCHES                 │
│ Propose prompt template adjustments or chunk deletions │
└───────────────────────────┬─────────────────────────────┘
                            ▼
┌─────────────────────────────────────────────────────────┐
│                    VALIDATE PASSES                      │
│ Verify that the patches do not introduce regressions    │
└───────────────────────────┬─────────────────────────────┘
                            ▼
                  [Repeat or Complete]

Flagged Chunks (`flagged-chunks.ts`)

If certain text chunks in a custom RAG index consistently lead to low faithfulness scores (e.g., outdated pricing sheets or conflicting instructions), the diagnostic step flags those vector chunks (getFlaggedChunks). The agent can mark them for active exclusion or propose a patch-chunk update.

Resolving Obstacles & Human Intervention

If an iteration loop runs for multiple cycles without achieving the designated targetScore, or if it reaches a performance plateau, the agent pauses, saves its snapshot state, and flags a humanReviewRequired status. Operators can resolve the blocker via /human by writing custom prompt guidelines or accepting/rejecting the agent’s proposed patches before resuming.

6.3 AutoRAG Parameter Tuning

While Auto-Fix focuses primarily on prompt editing and content pruning, AutoRAG (autorag.ts) focuses on systemic search space tuning. It executes grid and Bayesian search algorithms over:

Retrieval $k$-value: The count of documents retrieved.
Chunk Size / Overlap: The sliding boundaries of the text segments.
Reranking Thresholds: The minimum score threshold required to pass a document to the LLM context.

It saves snapshots of optimized vector parameter structures, allowing the workspace to hot-swap to the top-performing configuration.

6.4 Streamlined Quick-Start Wizard (`quick-start.ts`)

To dramatically lower the barrier to entry, the /quick-start endpoint exposes a streamlined, single-call wizard. In one POST request, it:

Creates a new QASuite with default score generators.
Uses the RAG file parser to generate representative evaluation test cases from provided document files.
Spawns an active AutoFix loop to tune the default assistant against these files.

This turns a multi-step setup into a single, automated command.

7. Post-Fine-Tune QA Validation Gate

For teams fine-tuning models (e.g., LoRA, SFT), the QA suite serves as a final quality gate.

                  ┌──────────────────────┐
                  │  Fine-Tuning Source  │
                  └──────────┬───────────┘
                             │
                             ▼
                  ┌──────────────────────┐
                  │ Export Run to SFT    │
                  └──────────┬───────────┘
                             │
                             ▼
                  ┌──────────────────────┐
                  │ Run SFT / LoRA Job   │
                  └──────────┬───────────┘
                             │
                             ▼
                  ┌──────────────────────┐
                  │ Post-FT Validation   │
                  │         Gate         │
                  └──────────┬───────────┘
                             │
            ┌────────────────┴────────────────┐
            │                                 │
     [Passes Gate]                     [Fails Gate]
            ▼                                 ▼
   ┌──────────────────┐              ┌──────────────────┐
   │ Deploy to Prod / │              │ Block Deployment │
   │  Graduate Arena  │              │  & Alert Team    │
   └──────────────────┘              └──────────────────┘

7.1 Post-FT Regression Verifier (`post-finetune-qa.ts`)

Once a fine-tuning job is completed (e.g., trained on Modal), the /post-finetune-qa endpoint is invoked prior to staging deployment:

It spins up a temporary container running the fine-tuned candidate.
It executes the designated QASuite against it.
If the model’s scores meet or exceed baseline criteria (and do not introduce catastrophic regressions), the gate is cleared, allowing safe automated deployment.

7.2 Fine-Tune Export Bridges (`exportRunToSft` & `trainFromQA`)

If high-scoring runs are verified on a suite, those (prompt, response) pairs can be exported into an SFT JSONL format:

exportRunToSft: Converts a completed suite run’s top scoring items into a clean training format, ignoring any low-scoring or flagged hallucination items.
trainFromQA: Handles the entire process in one click. It compiles the training dataset, uploads it to storage, creates a FineTuneAI object, and kicks off a Modal-backed training job automatically.

8. TrustRuns: Cryptographic Verifiable Runs

To allow enterprises to prove their model’s compliance, faithfulness, or accuracy to external auditors or third parties, Scored QA supports TrustRuns (publish-to-trust.ts).

A TrustRun aggregates and packages a completed suite run into a signed manifest:

It compiles all prompts, model responses, intermediate RAG contexts, and judge-assigned scores.
It uploads the full audit payload to Cloudflare R2 storage using standard eval-harness formatting.
It produces a cryptographically signed public verification URL.
Third parties can inspect this verifiable manifest to audit the model’s performance without requiring access to the internal MongoDB database.

9. API & Method Reference

Below is a quick reference table of the primary REST endpoints exposed by the Scored QA and Calibration routers:

HTTP Method	Route	Description	Primary Controller
POST	`/scored-qa/suites`	Creates a new QA suite with scorers.	`root.ts -> createQASuite`
GET	`/scored-qa/suites`	Lists all suites in the current workspace.	`root.ts -> listQASuites`
POST	`/scored-qa/suites/copy-from`	Copies a suite and its tests from a source workspace.	`copy-from.ts -> copyQASuiteHandler`
POST	`/scored-qa/suites/:suiteId/multi-release-run`	Runs a suite against multiple releases concurrently.	`multi-release-run.ts -> multiReleaseRun`
POST	`/scored-qa/suites/:suiteId/arena-run`	Evaluates a suite across every variant in an arena preset.	`arena-run.ts -> arenaRun`
POST	`/scored-qa/suites/:suiteId/estimate`	Computes a dry-run cost & latency estimate.	`estimate-run.ts -> estimateRun`
GET	`/scored-qa/suites/:suiteId/calibration-report`	Combines latest run aggregates + Spearman correlation metrics.	`calibration-report.ts -> calibrationReport`
GET	`/scored-qa/suites/:suiteId/route-performance`	Groups test results by active RAG vector group/index.	`route-performance.ts -> routePerformance`
POST	`/scored-qa/suites/:suiteId/tests/bulk`	Bulk creates and attaches test items to a suite.	`bulk-create-tests.ts -> bulkCreateTestsForQASuite`
POST	`/scored-qa/suites/:suiteId/promote-from-arena`	Promotes a live RAG Arena winner-pick into a QA test item.	`promote-from-arena.ts -> promoteFromArena`
POST	`/scored-qa/suites/:suiteId/arena-prefilter`	Ranks and filters active variants before generation.	`arena-prefilter.ts -> arenaPrefilter`
POST	`/scored-qa/suites/:suiteId/graduate-to-arena`	Promotes high-performing offline configurations to live.	`graduate-to-arena.ts -> graduateToArena`
POST	`/scored-qa/calibration-ratings`	Logs or upserts a human calibration rating.	`create-rating.ts -> createCalibrationRating`
GET	`/scored-qa/calibration-ratings/triples`	Returns a feed of available test result triples to rate.	`list-triples.ts -> listCalibrationTriples`
GET	`/scored-qa/calibration-ratings/agreement`	Calculates Spearman $\rho$ and weighted kappa for judges.	`compute-agreement.ts -> computeCalibrationAgreement`
GET	`/scored-qa/calibration-ratings/next-priority`	Identifies priority items requiring human evaluation.	`next-priority.ts -> nextPriorityForCalibration`
POST	`/scored-qa/auto-fix`	Initiates an autonomous prompt/RAG optimization loop.	`root.ts -> startAutoFixLoop`
POST	`/scored-qa/auto-fix/autorag`	Starts an agent-driven RAG parameter tuning cycle.	`autorag.ts -> startAutoRAGCycle`
POST	`/scored-qa/quick-start`	Streamlined, single-call suite + test + loop bootstrap.	`quick-start.ts -> quickStartAutoFix`
POST	`/scored-qa/post-finetune-qa`	Automated quality gate validation post fine-tuning.	`post-finetune-qa.ts -> postFineTuneQA`