Problem
A subscription business wants a chat agent that can actually do things: check a plan, change an address, raise a ticket. The moment it can act on accounts, the real question is how you stop someone talking their way into another customer’s data. Prompt instructions are not a security control. The layer that sits above retrieval infrastructure (dialogue state, gated actions, escalation, and the observability to know if any of it works) is the whole project.
Constraints
- Verification must be per account, not per conversation. An implementation that only asks “is this conversation verified?” lets someone verify their own account and then edit a stranger’s.
- The model must never hold a credential, so a prompt injection has nothing to exfiltrate.
- “Not in your documents” has to be a supported outcome, not a low-scoring guess.
- The whole thing has to be regression-testable without paying for an LLM, because prompts and tool schemas change and both break silently.
System
Dify runs the dialogue and tool calling. The tools are n8n webhook workflows, which call three FastAPI services: the mock backend (accounts, tickets, handoff), the verification service (OTP and ID check), and retrieval (two-stage search over Qdrant with a local BGE embedder and cross-encoder reranker). Supabase holds conversation state, verification sessions, the handoff queue and an append-only audit log. Langfuse traces every turn and tool call. Airtable is the inbox a human agent opens.
Decisions
The gate is a function call in the backend. Sensitive routes call require_verified() before touching a row. Two softer checks sit above it (the prompt, an n8n pre-flight) for the customer’s experience, not for security. The API key lives in n8n’s environment; the model never sees it.
OTP destination comes from the account record. The tool takes no destination parameter, so “text it to my new number instead” has nothing to bind to. OTPs are stored as peppered HMAC digests, compared in constant time, cleared on success. Account details are masked until the gate passes: finding an account is not reading it.
Below the rerank threshold, return nothing. A bi-encoder happily returns billing chunks for “what does your home fibre broadband cost” because they’re topically near. The cross-encoder reads query and passage together, scores them low, and the retrieval service returns no records, which turns a confident wrong answer into a handoff.
Escalation carries the conversation, not just the customer. The handoff record holds the transcript, resolved entities, verification state, and a summary written for the colleague picking it up.
Settings refuse to boot in production with demo flags on. The eval harness needs OTP codes returned in responses; the config layer makes that impossible to ship.
Outcome
21 deterministic checks cover retrieval grounding (the refusals matter more than the hits) and gate behaviour (refused without verification, allowed with it, and the cases that look verified: wrong code, wrong DOB, verification belonging to another account). A consistency test catches the Dify tool schema and n8n webhook paths drifting apart.
Known limits, stated in the repo: OTP sending isn’t rate-limited per destination, the ID check is a field comparison rather than a document vendor, and no SMS is actually sent. What would survive a real deployment is the gate, the state model, the escalation payload, the calibrated two-stage retrieval, and the eval harness; the three mocked integrations are replaceable without touching the agent layer.