Safety: Moderation & Flaggers

A Release's Safety settings decide what reaches the model and what gets escalated to a person. There are three, and they do different jobs:

| Control | Field | Job | | --- | --- | --- | | Prompt moderation | promptModeration | Blocks a message before generation | | De-identification | deIdentification | Rewrites a message to strip PII | | Notification flaggers | notificationFlaggers | Escalates a conversation to your team |

Moderation blocks, de-identification redacts, flaggers notify — nothing here does two of those. This page covers the first and third; de-identification has its own page.

Where they run

user message
   │
   ├─▶ de-identify              ← original text ends here
   ├─▶ RAG retrieval            ← builds the context on-topic is judged against
   ├─▶ PROMPT MODERATION        ← blocks here, before any generation spend
   ├─▶ input flaggers           ← notify, do not block
   ├─▶ model generation
   └─▶ output flaggers          ← run against the reply

Moderation sees the redacted text, not what the user typed — so a custom moderator that keys on names or emails will never match once de-identification is on. It also runs after retrieval, because on-topic moderation is judged against the context that retrieval produced.

Prompt moderation

Three independent checks. A message is blocked if any of them trips, and the block happens before generation, so a blocked message costs no inference.

Harmful-message moderation

Runs the message through Llama Guard and blocks anything classified unsafe. On by default (noHarmful.on: true) — unlike every other control on this page, this is opt-out.

| Code | Category | Waivable | | --- | --- | --- | | S1 | Violent Crimes | ⛔ | | S2 | Non-Violent Crimes | ⛔ | | S3 | Sex-Related Crimes | ⛔ | | S4 | Child Sexual Exploitation | ⛔ | | S5 | Defamation | ✅ | | S6 | Specialized Advice | ✅ | | S7 | Privacy | ✅ | | S8 | Intellectual Property | ✅ | | S9 | Indiscriminate Weapons | ⛔ | | S10 | Hate | ⛔ | | S11 | Suicide & Self-Harm | ⛔ | | S12 | Sexual Content | ⛔ | | S13 | Elections | ✅ |

noHarmful.allowedCategories waives categories for this Release only:

{ "noHarmful": { "on": true, "title": "Message considered harmful",
                 "allowedCategories": ["S6"] } }

Why S6 usually needs waiving

S6 Specialized Advice covers medical, legal and financial guidance — which for a diagnostics, law, or finance assistant is the subject matter. Left blocking, the assistant refuses its own domain: a lab tech asking how to clear a deck-check error gets "I can't help with that request. Message considered harmful (Specialized Advice)". If your Release exists to give specialist guidance, waive S6.

Two rules the waiver obeys, both deliberate:

• The floor cannot be waived. Codes outside the waivable set are ignored, and the floor is applied at decision time rather than at save time — so a waiver that reaches the database by any route still cannot unblock S1/S4/ S9/S11.
• Every tripped category must be waived. Waiving S6 does not excuse a message flagged S6,S1 — something that is both specialist advice and a violent crime is still a violent crime.

Omitting `allowedCategories` silently drops the waiver

The config cast keeps only the keys it is told about, so a save that omits allowedCategories removes an existing waiver and still returns 200. The update looks applied and is not — and the symptom appears later, as an assistant that has quietly started refusing its own domain again. Always send the full noHarmful object, including the waiver you want to keep.

On-topic moderation

Blocks messages your knowledge base cannot support, judged against the context RAG retrieved for that turn. Off by default (both minimums 0).

| Field | Meaning | | --- | --- | | minimumContext | Minimum number of retrieved context items | | minimumTokens | Minimum total tokens across those items |

Either being greater than zero turns the check on. A message is blocked when retrieval returned nothing at all, or fewer items than minimumContext, or fewer tokens than minimumTokens.

No RAG vector means no enforcement — silently

On-topic is judged against retrieved context, so with no RAG vector attached there is nothing to judge and the check passes everything. It does not error and nothing warns you; the setting simply has no effect. The dashboard locks the control until a vector is attached, but a config written through the API can set the minimums on a vectorless Release and look configured while enforcing nothing. Attach the knowledge base first — see RAG.

Custom moderators

A custom moderator is its own document: an assistant, a prefix (your instructions — what to look for), and outputKeys (the boolean fields it must return). Attach it to a Release by id and title:

{ "custom": [{ "id": "<moderatorId>", "title": "Competitor mentions" }] }

At request time the moderator runs against the message and returns its keys. If any key comes back true, the message is blocked — the keys are OR'd, so a moderator returning { "offTopic": false, "competitorMention": true } blocks. Design each key as "should this be stopped?", not as a neutral observation.

Test one before attaching it:

curl -X POST \
  https://api.divinci.app/white-label/$WORKSPACE/prompt-moderation/$MODERATOR/test-run \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"prompt":"a message that should trip it"}'

Notification flaggers

Flaggers watch conversations for patterns and notify — they never block. Use them for "this conversation needs a human", not for enforcement.

{
  "helpRequests": { "enabled": true },
  "custom": [{ "id": "<flaggerId>", "title": "Escalate: frustrated customer" }]
}

helpRequests is the built-in toggle for users explicitly asking for a human; it is off by default. Custom flaggers are their own documents, with more surface than a moderator:

| Field | Purpose | | --- | --- | | assistant, prefix, outputKeys | Same shape as a custom moderator | | flagInput | true → runs on the user's message; false → on the AI's reply | | notifyUser | Whether the end user is told their message was flagged | | notificationMessage, notificationIcon | What your team sees | | notificationTags | Tags applied to notifications this flagger raises — the handle for routing and filtering |

Flaggers also have a test-run route, at /white-label/$WORKSPACE/notification-flagger/$FLAGGER/test-run.

Delivery, channels and the notification feed itself are covered in Notifications, Analytics & Metrics.

Every custom moderator and flagger is an extra model call

Each one runs its own structured-output assistant call per message — input flaggers against the user's message, output flaggers against the reply — and that cost is escrowed alongside the generation. Three custom moderators plus two flaggers means five extra model calls on every turn, not five regex matches. Consolidate related checks into one moderator with several outputKeys rather than attaching one moderator per rule.

Configuring

Both fields are settable through the SDK, and each also has a dedicated endpoint. Either way, you send the whole config object: omitted keys revert to defaults rather than preserving current values, so read the release first if you are changing one setting among several.

await divinci.releases.updateInWorkspace(workspaceId, releaseId, {
  promptModeration: {
    noHarmful: {
      on: true,
      title: "Message considered harmful",
      allowedCategories: ["S6"],   // omitting this DROPS an existing waiver
    },
    onTopic: {
      title: "Message considered off topic",
      minimumContext: 1,
      minimumTokens: 0,
    },
    custom: [],
  },
  notificationFlaggers: {
    helpRequests: { enabled: true },
    custom: [],
  },
});

The dedicated endpoints remain useful for two things the SDK path cannot do: they work without reading the release first, and toggle-moderation flips the harmful check alone while preserving everything else. Both refuse drafts.

# Prompt moderation — all three keys are REQUIRED, or the request is rejected
curl -X POST \
  https://api.divinci.app/white-label/$WORKSPACE/release/$RELEASE/moderation \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{
    "noHarmful": { "on": true, "title": "Message considered harmful",
                   "allowedCategories": ["S6"] },
    "onTopic":   { "title": "Message considered off topic",
                   "minimumContext": 1, "minimumTokens": 0 },
    "custom":    []
  }'

# Notification flaggers
curl -X POST \
  https://api.divinci.app/white-label/$WORKSPACE/release/$RELEASE/notification-flaggers \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{ "helpRequests": { "enabled": true }, "custom": [] }'

# Harmful moderation on/off only, leaving everything else alone
curl -X POST \
  https://api.divinci.app/white-label/$WORKSPACE/release/$RELEASE/toggle-moderation \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"enabled": false}'

POST /moderation rejects a body missing any of noHarmful, onTopic, or custom — there is no partial update. Use toggle-moderation when you only want to flip the harmful check, since it preserves the rest.

Omitting a field still preserves it

releases.updateInWorkspace() carries the stored promptModeration and notificationFlaggers forward on every save, so an update that does not mention them leaves them untouched. Passing a value now overrides that carry-forward — which is exactly what these fields could not do before, and why a value passed to an older client was accepted and discarded in silence.

Failure behaviour

Moderation is mostly fail-open — the opposite of de-identification, and a deliberate difference: a redaction outage that let PII through would be a compliance breach, whereas a moderation outage that blocked everything would take the whole assistant down.

| Failure | Result | | --- | --- | | Moderation service returns a non-OK HTTP status | Fails open — message allowed | | Network rejection, or an unparseable response body | Blocks the message | | A custom moderator errors | Blocks the message |

So a clean outage of the moderation provider degrades to "no harmful-content filtering" rather than to an outage of your Release — but a messy one (DNS failure, truncated body) blocks. Neither state announces itself to the end user beyond the block message, so alert on moderation errors rather than expecting to notice.

Gotchas

• Harmful moderation is on by default; everything else here is off. A new Release already blocks unsafe content, and already has zero on-topic enforcement, no custom moderators, and no flaggers.
• Moderation sees redacted text. With de-identification on, a custom moderator keyed on names, emails, or phone numbers will never match — they were replaced before moderation ran.
• A blocked message costs nothing; a flagged one costs extra. Moderation runs before generation, so a block saves the inference spend. Flaggers run in addition to generation, so they only add cost.
• title is user-visible. It is the text shown when a message is blocked, with the tripped categories appended for harmful blocks. Write it as something you are willing to show an end user.
• Custom moderator and flagger ids are per workspace. Cloning a Release to another workspace carries the ids but not the documents they point at; a moderator that cannot be found throws, which blocks the message.

Related

• Red Teaming — probe whether this layer (and the model behind it) actually holds; defense.moderationHeld counts the probes that never reached the model
• De-Identification — the third Safety control
• Notifications, Analytics & Metrics — where flagger notifications are delivered and read
• Security & Abuse Protection — who may reach a Release at all