Fix Proposal · controlled autonomy

Require evidence verification before customer-facing root-cause claims

fix_1001 · risk low · confidence 87% · source mock

high

Root cause

The agent makes causal claims after checking tickets/docs but before checking deployment history or runtime telemetry.

Evidence from trace & evals

Groundedness eval scored 31% on the Acme run — the final answer asserts "caused by last night's deployment".
Required-tool eval failed: checkDeploymentHistory and checkRuntimeTelemetry were never called.
Trace shows only support-ticket and docs searches before the final customer-facing answer.

Proposed change

Add a mandatory verification step before final customer-facing summaries.

Policy update

agent-policies/escalation-agent.md

## Customer-facing root-cause claims

Draft the customer update from tickets and docs.

Before making a root-cause claim, the agent must check deployment history and telemetry. If evidence is missing, it must state uncertainty instead of asserting causality.

Expected improvement

Groundedness 31% → 92%, required-tool fail → pass, unsupported causal claims 1 → 0.

Regression cases

• Acme checkout failure
• Globex timeout escalation
• Initech billing outage

Human approval

Every risky action needs approval

Review the evidence, then approve to open a GitLab remediation and unlock the regression rerun.

Tool-Call Firewall · action receipt

Tool: createGitlabIssue
Params: title="Require evidence verification before customer-facing root-cause claims"
Risk: high
Approval: required
Rollback: Close the created issue/MR. No agent policy is applied until the MR is reviewed and merged.

Affected runs

run_acme_hero, run_cust_acme_uc_0_180, run_cust_acme_uc_1_1500, run_cust_acme_uc_2_2940, run_cust_globex_uc_0_320, run_cust_globex_uc_1_1850, run_cust_initech_uc_0_610, run_cust_initech_uc_1_2600