Demo 12
Once RAG and agents can read external knowledge, call tools, and affect external systems, failures need to become reproducible tests. This chapter shows how red-team cases, input guardrails, least privilege, human review, regression evaluation, and a release gate form a continuous feedback loop.
Teaching interaction
Once RAG and agents can read external knowledge and take actions, how can one failure be kept out of future releases?
The user asks for a support-ticket summary. The document contains no extra instruction, so the system only needs to read and summarize.
The suite records the expected result for a normal request so a guardrail cannot appear safe merely by refusing everything.
A normal request must retain task success; safety is not blanket blocking.
This demo calls no real model, tool, or safety service. Requests, detections, permission decisions, and fixes are scripted. It explains defense in depth and regression evaluation; passing one suite never proves that all risk is gone.
Teaching And Evidence Boundary
NIST places generative-AI risk management across the lifecycle and emphasizes pre-deployment testing, ongoing evaluation, incident records, and periodic review. OWASP explains that prompt injection cannot be eliminated by model instructions alone and recommends least privilege plus human approval for high-risk actions. HELM shows why evaluation needs metrics beyond accuracy, including robustness, safety-related harms, fairness, and efficiency. This demo compresses those principles into one indirect prompt-injection case; it is not a complete security benchmark or compliance program.
References
Concept Check
This check supports reflection and never blocks the next chapter. Records stay on this device.