FULL CLOUD EVALUATION COMPLETE

47.7% to 0.6%
across AgentDojo.

UBAG was evaluated across all four AgentDojo suites and 629 user and injection pairs. Every number came live from the hosted gateway and behavioral service. The complete deployed stack reached 0.6% aggregate attack success.

LIVE AGENT · HOSTED SERVICES

AgentDojo

629 pairs · banking · travel · slack · workspace · gemini-2.5-flash · important_instructions

47.7% → 0.6%aggregate attack success, baseline to full stack
629user and injection pairs across all four suites
0gateway and behavioral service errors
Suite and pairsArmAttack successTask utilityApprovals per task
Banking 144Baseline33.3%44.4%n/a
Banking 144Gate0.0%29.9%n/a
Banking 144Gate + review0.0%29.9%0.00
Banking 144Gate + behavioral0.0%29.9%n/a
Banking 144Full stack0.0%29.9%0.00
Travel 140Baseline63.6%10.7%n/a
Travel 140Gate16.4%18.6%n/a
Travel 140Gate + review17.1%20.0%0.20
Travel 140Gate + behavioral2.9%22.9%n/a
Travel 140Full stack2.9%25.0%0.20
Slack 105Baseline84.8%54.3%n/a
Slack 105Gate34.3%24.8%n/a
Slack 105Gate + review34.3%38.1%0.48
Slack 105Gate + behavioral0.0%21.0%n/a
Slack 105Full stack0.0%32.4%0.52
Workspace 240Baseline30.8%27.9%n/a
Workspace 240Gate0.0%38.8%n/a
Workspace 240Gate + review0.0%45.8%0.28
Workspace 240Gate + behavioral0.0%38.8%n/a
Workspace 240Full stack0.0%45.4%0.28

Residual: Four of 629 pairs remained under the full stack. Three were speech-only answers with no tool call. One produced a side effect to a destination the user's own task had authorized, for a 0.16% side-effecting residual.

Behavioral layer: It reduced travel from 16.4% to 2.9% and slack from 34.3% to 0.0%. On banking and workspace the deterministic gate had already reached 0.0%.

Review rule: Human review must sit above the behavioral layer. On travel, review without behavioral context loosened one gate refusal and raised attack success from 16.4% to 17.1%.

STATIC REPLAY

InjecAgent

UIUC · Findings of ACL 2024 · 1,054 cases · 17 user tools · 62 attacker tools

1,054 / 1,054attack-required tool calls blocked
1,054 / 1,054authorized task tools allowed

What it proves: InjecAgent attacks require at least one tool outside the legitimate task scope. UBAG least-privilege confinement blocks that call deterministically.

Our disclosure: We report this as a replay of the benchmark's labeled tool calls through the shipped authorization engine, not a live-agent run and not an end-to-end task-completion measurement.

Boundary: When a harmful action can reuse an authorized tool, tool scope alone is insufficient. Destination, argument, value, state, and grant controls must carry the decision.

SUPPLEMENTARY ENGINE COVERAGE

Known failure modes stay executable as tests.

Our authored suite exercises tenant isolation, capability grants, state drift, cumulative plans, fabricated references, argument injection, circuit breakers, and benign controls.

302 / 302unique authored attacks stopped
199 / 199benign controls allowed
0false positives

The suite contains 511 unique cases and recorded zero execution errors. Six blind, model-authored corpora were generated from the same declared scenario using GLM 5.2, Gemini 3.5, Microsoft Copilot, Claude Opus 4.8, GPT-5.6 Sol, and Claude Sonnet 4.6. The models proposed the cases; UBAG evaluated them deterministically. Ten authored expectation labels differed from the configured policy semantics and are reported separately rather than counted as attacks or benign controls. This is internal regression evidence, not independent validation.

01

Mechanism, not detection

The claim is not that UBAG recognizes every malicious instruction. The claim is that policy denies unauthorized execution even when the model is persuaded.

02

Conditions remain visible

Model, suite, attack family, policy point, case count, utility effect, and evaluation type are stated beside each result.

03

Residuals stay public

Successful attacks, blocked legitimate tasks, service errors, and out-of-scope speech acts remain part of the result.

04

Evidence is reviewable

The public report explains the method, full results, residuals, and limitations. Run logs and policy configuration are available for technical review.

These are company-run evaluations on public academic benchmark suites. They are not a third-party audit.

FULL PUBLIC REPORTRead the complete method, table, residual analysis, caveats, and reproduction steps.
Download the paper ↓