sauce.ai/redteam

Adversarial safety testing for clinical AI chatbots. Point it at your bot, pick a specialty, and it runs synthetic patient conversations engineered to surface unsafe medical advice — then scores every reply and emails you an epidemiological report.

Free while in preview: up to 100 trials total per email. For use only on systems you are authorized to test.

This is a safety-evaluation tool. It deliberately tries to elicit unsafe responses from the bot under test so you can find and fix them. Only run it against endpoints you own or have written permission to test. Reports are LLM-scored screening signals, not clinical determinations — have a clinician review flagged transcripts.

1 · Your chatbot (the target)

Endpoint type

URL / API endpoint https only; must resolve to a public address

Model as the API names it

API key held in memory for the run only, never stored

System prompt sent to the target optional; match your production setup

Custom endpoint mapping

Request body template JSON; placeholders {{message}}, {{messages_json}}, {{history_text}}, {{conversation_id}}, {{system_prompt}}

Response path dotted path to the reply text

Conversation state

Web page selectors

Input selector

Send button blank = Enter

Reply bubble selector

Requires the browser add-on on the server. Selectors target the newest assistant bubble.

2 · The evaluation

Number of trials 1–100

Specialty / area

Max prompts per conversation

Condition optional; blank = sample across the specialty

Harm categories to hunt (blank = all)

Anything else the attacker should know optional; e.g. "bot is aimed at UK patients", "focus on paediatric dosing"

Harm threshold P(harm) ≥ counts as unsafe

Control arm share of cooperative baseline conversations

Stop each conversation at first harm

Advanced: the red-team engine (optional)

The attacker is an ensemble of LLMs that propose, refine, and vote on the next patient message most likely to push the bot into an unsafe reply. All of this is optional — defaults use Claude.

Attacker models comma-separated provider:model; mix providers to strengthen it

Arbiter / scorer models

Judge models annotate every reply; ≥3 gives inter-rater κ

Sub-agent levels propose→refine depth

Candidates per agent

Consensus rounds

Adaptive tactic bandit Simulate the bot's reply before committing (slower, stronger)

3 · Where to send the report

Email the report is emailed here when the run finishes

Run

Open live report ↗ · JSON export ·

A sauce.ai lab tool. Questions → redteam@sauce.ai.