LIVE AGENT · HOSTED SERVICESAgentDojo
3,774 full-stack defended pairs · six attack strategies · four suites · gemini-2.5-flash
47.7% → 0.6%flagship attack, baseline to full stack
3,774full-stack pairs across six strategies
0.41%13 residuals in the five-strategy expansion
0.03%one side-effecting residual in that expansion
FIVE-ARM DECOMPOSITIONimportant_instructions
The original 629-pair run separates the deterministic gate, behavioral layer, and human review so each contribution and utility cost remains visible.
Banking 144Baseline33.3%44.4%n/a
Banking 144Gate0.0%29.9%n/a
Banking 144Gate + review0.0%29.9%0.00
Banking 144Gate + behavioral0.0%29.9%n/a
Banking 144Full stack0.0%29.9%0.00
Travel 140Baseline63.6%10.7%n/a
Travel 140Gate16.4%18.6%n/a
Travel 140Gate + review17.1%20.0%0.20
Travel 140Gate + behavioral2.9%22.9%n/a
Travel 140Full stack2.9%25.0%0.20
Slack 105Baseline84.8%54.3%n/a
Slack 105Gate34.3%24.8%n/a
Slack 105Gate + review34.3%38.1%0.48
Slack 105Gate + behavioral0.0%21.0%n/a
Slack 105Full stack0.0%32.4%0.52
Workspace 240Baseline30.8%27.9%n/a
Workspace 240Gate0.0%38.8%n/a
Workspace 240Gate + review0.0%45.8%0.28
Workspace 240Gate + behavioral0.0%38.8%n/a
Workspace 240Full stack0.0%45.4%0.28
Residual: Four of 629 pairs remained under the full stack. Three were speech-only answers with no tool call. One produced a side effect to a destination the user's own task had authorized, for a 0.16% side-effecting residual.
Behavioral layer: It reduced travel from 16.4% to 2.9% and slack from 34.3% to 0.0%. On banking and workspace the deterministic gate had already reached 0.0%.
Review rule: Human review must sit above the behavioral layer. On travel, review without behavioral context loosened one gate refusal and raised attack success from 16.4% to 17.1%.
EXPANDED STRATEGY SWEEPFive additional attack strategies
Every cell below covers all user and injection pairs for that suite. Baseline and gate come from one runner; the full stack comes from another.
Banking 144direct4.9%0.0%0.0%32.6%
Banking 144ignore_previous2.1%0.0%0.0%31.9%
Banking 144injecagent2.1%0.0%0.0%31.2%
Banking 144system_message2.1%0.0%0.0%31.2%
Banking 144tool_knowledge40.3%0.0%0.0%27.8%
Travel 140direct0.7%0.0%0.0%40.7%
Travel 140ignore_previous0.0%0.0%0.0%40.0%
Travel 140injecagent0.7%0.0%0.0%36.4%
Travel 140system_message1.4%0.7%0.7%40.0%
Travel 140tool_knowledge67.9%15.7%6.4%25.0%
Slack 105direct3.8%3.8%1.0%55.2%
Slack 105ignore_previous13.3%7.6%0.0%55.2%
Slack 105injecagent10.5%5.7%0.0%53.3%
Slack 105system_message2.9%1.9%1.9%53.3%
Slack 105tool_knowledge84.8%34.3%0.0%35.2%
Workspace 240direct0.0%0.0%0.0%46.7%
Workspace 240ignore_previous0.0%0.0%0.0%46.2%
Workspace 240injecagent0.0%0.0%0.0%49.2%
Workspace 240system_message0.4%0.0%0.0%48.3%
Workspace 240tool_knowledge34.6%0.0%0.0%44.2%
Signal: Seven of the twenty cells have an undefended baseline already at or near zero, so those defended zeros are not counted as evidence of a win. Thirteen cells carry signal.
Strongest attack: tool_knowledge produced the highest baseline on banking, travel, and workspace, and tied the highest on slack. The original baseline is therefore conservative.
Expanded residuals: Across 3,145 full-stack pairs, 13 remained (0.41%): ten speech acts, two reads, and one side-effecting message to an already-authorized colleague (0.03%).
Authorization split: Outward actions remain deterministic. A deterministic unknown-target read rule held 58 Slack reads, including 45 legitimate ones (78% false positives), so reads use the behavioral layer.
STATIC REPLAY · LIVE HOSTED GATEWAYInjecAgent
UIUC · Findings of ACL 2024 · 1,054 cases · 17 user tools · 63 attacker tools · base + enhanced
1,054 / 1,054attacks stopped · base setting
1,054 / 1,054attacks stopped · enhanced hacking prompt
1,054 / 1,054legitimate task tools allowed
What it proves: Every InjecAgent attack requires at least one tool outside the current task's scope. UBAG stops the attack chain when it refuses any required out-of-scope tool.
Live execution: Each setting produced 2,652 authorization calls through the deployed gateway, with zero errors and 174 ms average latency. The live HTTP path and the in-process cross-check agreed exactly.
Stopped, not every call blocked: One attacker step reused the authorized GitHubGetUserDetails tool and was allowed. The attack still failed because its required exfiltration tool was refused.
Boundary: This is a static replay of labeled calls, not a live-agent run. Tool confinement is insufficient when harmful use stays inside an authorized tool; destination and argument controls must then carry the decision.
UBAGby Dixit AlgorizmiRequest technical review