A ghost in the shop chatbot
A shop chatbot got possessed. Gemma drove the ghost out.
Hidden instructions and false claims fooled the helper. After the repair, we tested it on 20 questions it had never seen.
-/12 → -
Attacks that fooled the helper, before and after the repair
-/8 → -
Real customers still helped, before and after
20
Fresh exam questions
Averages over 3 runs of the exam. The repairing model never saw these questions. They were drawn by a public random number after the fix was frozen.
Whisper a trick, see who falls for it
Type anything or tap a trick. Both helpers answer, each answer is marked, and the scoreboard keeps count. Answers vary from run to run, so the exam below is the real proof.
ON FILE (all invented): A1001 refund approved · A1002 pending review · A1003 not eligible
Helper as builtFooled 0Held up 0
After the repairFooled 0Held up 0
The exam: all 20 questions
Loading the exam.