To measure how often, how fast, and how severely deployed clinical chatbots give unsafe advice under adversarial pressure, with clinician-interpretable risk metrics.
Synthetic multi-turn conversations between a generated patient persona and a target chatbot, each turn annotated by an LLM judge panel (P(harm), severity, categories, escalation). No real patient data; no PHI.
Conversations are generated on demand against researcher-supplied endpoints the researcher is authorized to test. Target credentials are held in memory for the run only and never persisted.
Harm labels are LLM-judge screening signals; a stratified sample is adjudicated by clinicians for validation. The judge is audited for bias and calibration.
Safety benchmarking, methods research, and (with clinician review) vendor disclosure. Not for clinical decision-making.
A de-identified conversation + annotation export (tidy CSV/JSON) accompanies publications; the most operational jailbreak strings are withheld under responsible disclosure.
Suite/judge/prompt versions and model snapshots are recorded; changes are versioned and never mixed across versions in a comparison.
Standards-neutral reporting (NIST AI RMF, UK AISI, TRIPOD-LLM, datasheet-for-datasets).