TrustBench (Attested Evals)

TrustBench runs a benchmark against a Divinci-registered model assistant and signs the result with a platform Ed25519 key. The output is a TrustRun manifest: a JSON document stating what was run, against which model, with which harness, and what came out — signed such that a third party can check it without asking Divinci anything.

The point is the last part. An eval score you publish is a claim; a signed manifest is a claim someone else can independently check. Verification runs offline, against a public key registry, using an MIT-licensed package with no Divinci dependencies — so “trust us” is never part of the chain.

Running a benchmark

divinci trust benchmarks list                    # what's available
divinci trust run <benchmark> --model <assistantId>
divinci trust list                               # your runs

trust run submits the run and polls to completion. Useful flags:

Flag	Effect
--model <assistantId>	Required. The Divinci assistant tool id under test
--public	Publish to the public registry. Default is private
--no-wait	Return the run id immediately instead of polling
--timeout <seconds>	Polling timeout, default 600
--whitelabel <id>	Ownership context
--output json	Machine-readable output

Over the API directly:

curl -X POST https://api.divinci.app/v1/trustbench/runs \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"benchmarkId":"<id>","modelAssistantId":"<assistantId>","visibility":"private"}'

curl https://api.divinci.app/v1/trustbench/runs/$RUN_ID \
  -H "Authorization: Bearer $TOKEN"

Runs are private unless you say otherwise

visibility defaults to private, and the CLI’s --public is opt-in. A private run is fully verifiable by you, but its manifest is not served from the public endpoints — so nobody else can check it. Publishing is the deliberate act of making a claim checkable; do it when you actually want the result cited.

Verifying a run

Two modes, and the difference matters:

divinci trust verify <run-id>            # LOCAL — offline, independent
divinci trust verify <run-id> --remote   # server-side; works on private runs
divinci trust verify <run-id> --strict   # warnings become failures

Local is the default, and it is the one that means something. It fetches the public manifest and outputs, then verifies them offline against the well-known key registry — the same thing an outside party would do. --remote asks Divinci whether Divinci’s own signature is good, which is useful for a private run you own but proves nothing to a sceptic.

Verifying from your own code

The verifier is published as @divinci-ai/trustbench-verifier — MIT, pure TypeScript, no Divinci-internal dependencies, and it runs in browsers, Node, Workers, and Deno.

import { verify } from "@divinci-ai/trustbench-verifier";

const result = await verify(manifest, {
  outputs,           // optional: raw outputs file → checks results.outputsHash
  benchmarkContent,  // optional: raw task spec  → checks benchmark.contentHash
});

if (!result.verified) console.error(result.errors);

verify() returns a breakdown rather than a bare boolean, and the tri-state fields are the part to read carefully:

Field	Meaning
verified	Overall pass — true only if every gating check passed
signatureValid	The manifest body was signed by a known platform key
outputsHashMatches	true / false, or null if you passed no outputs
benchmarkContentHashMatches	true / false, or null if you passed no spec
signedBy	The key id that signed it
errors, warnings	Detail; --strict promotes warnings to failures

`null` is not a pass

Omit outputs and outputsHashMatches is null — the check did not run. A signature-only verification proves the manifest is authentic and unmodified; it does not prove the outputs you are looking at are the ones that were scored. To make that claim, pass the outputs file. The same applies to the benchmark spec: without it you have not checked that the benchmark run was the benchmark published.

What a manifest contains

A v1 manifest (version: "1.0.0") carries a runId, createdAt/completedAt, and five blocks:

Block	Holds
benchmark	Which benchmark, its contentHash, and the author’s signature
model	The assistant under test and its provider
harness	Which harness executed it, and its version
execution	How the run was carried out
results	Scores, plus outputsHash over the outputs file
signatures[]	One or more Ed25519 blocks — keyId, publicKey, signature

The signatures array is excluded from the signed body before signing, so a manifest signs everything except its own signatures. Signing is over RFC-8785-style canonical JSON, which is what lets an independent implementation in another language reach byte-identical bytes and check the same signature.

Public artifacts for a published run:

GET /v1/trustbench/public/runs/:id/manifest
GET /v1/trustbench/public/runs/:id/outputs
GET /.well-known/trustbench-keys.json        ← the public key registry

Keys and rotation

Each environment has its own keypair — tbp-prod-001 signs production, tbp-staging-001 signs staging — so a staging manifest is deliberately not verifiable against production keys. The registry entry for each carries validFrom, validUntil, and a deprecated flag.

Rotation appends a new key and marks the old one deprecated rather than removing it, so manifests signed under a retired key stay verifiable offline forever. That property is the reason a manifest is worth publishing at all: a citation that expires when a key rotates is not a citation.

Point the verifier at the right registry

verify() defaults to https://api.divinci.app/.well-known/trustbench-keys.json. Verifying a staging manifest means passing wellKnownUrl (or a keyResolver) explicitly, or the signature check fails against a registry that has never seen tbp-staging-001.

Gotchas

• A verified: true with two nulls is a weak result. It says the manifest is authentic, not that the numbers describe the artifacts in front of you.
• --remote is not independent verification. It is Divinci checking its own signature. Fine for your own private run; worthless as evidence to a third party.
• Private runs have no public manifest. The public endpoints will not serve one, so a collaborator cannot verify a run you have not published.
• The benchmark’s contentHash pins the spec, not the dataset behind it. Verifying it proves the published task spec was the one run.
• Environments do not cross-verify. Staging keys sign staging manifests only.

Red-team benches

Three of the platform benchmarks are the shared adversarial corpus, reported as a severity-weighted attack success rate (lower is better):

Slug	Isolates
divinci-redteam-core-v1	All 32 probes across seven attack classes
divinci-redteam-rag-injection-v1	Indirect injection via retrieved content only
divinci-redteam-prompt-leak-v1	System-prompt extraction, fully canary-graded

To probe your deployed release (your prompt, your moderation, your RAG) rather than a registered model assistant, use the workspace Red Teaming surface. Same probes, different leak detector — your real prompt instead of a planted canary.

Related

• Red Teaming — the same corpus, run against a deployed release
• QA Suites — Divinci’s internal scoring and regression harness, which is a different tool: QA is for your iteration loop, TrustBench is for claims you publish
• Arena — A/B comparison between variants