Problem
A report today carries three numbers, and none answers "how bad is this finding, on this agent?":
- Evaluator
severity (low/high/critical) is a static label baked into the evaluator YAML. A BOLA finding is "high" whether the target is a read-only demo bot or an agent wired to prod with a refund tool — blind to the deployment.
- The per-evaluator "Avg Score" is the mean of the judge's per-attack scores. It's cosmetic (feeds nothing) and, worse, averaging hides breaches:
9 9 2 → 6.7 looks healthy even though one attack got through.
- The headline Safety Score is an aggregate posture %, so individual critical holes get diluted into one figure.
Proposal
Add a deployment-aware per-evaluator risk score (0–10): keep the severity label as a floor and amplify it by the target's agentic power (autonomy, tools, identity/tenancy, persistence). Same flaw → higher score on a more powerful agent. This mirrors the OWASP AIVSS amplification model, reduced to something fully automatic.
Key properties:
- Worst-case, not averaged — a finding scores from its severity floor; a defended evaluator scores
0.0. Fixes the breach-hiding above and removes the "running the same evaluator N times moves the score" wobble.
- No new setup questions — the power profile is derived from
businessUseCase + target metadata already in the config.
Acceptance
Implemented in the linked PR. Follow-ups tracked separately.
Problem
A report today carries three numbers, and none answers "how bad is this finding, on this agent?":
severity(low/high/critical) is a static label baked into the evaluator YAML. A BOLA finding is "high" whether the target is a read-only demo bot or an agent wired to prod with a refund tool — blind to the deployment.9 9 2 → 6.7looks healthy even though one attack got through.Proposal
Add a deployment-aware per-evaluator risk score (0–10): keep the severity label as a floor and amplify it by the target's agentic power (autonomy, tools, identity/tenancy, persistence). Same flaw → higher score on a more powerful agent. This mirrors the OWASP AIVSS amplification model, reduced to something fully automatic.
Key properties:
0.0. Fixes the breach-hiding above and removes the "running the same evaluator N times moves the score" wobble.businessUseCase+ target metadata already in the config.Acceptance
amplifiedRisk(severity, isFinding, power)with unit testsderiveAgentProfile()from existing config, with unit testsrisksurfaced in the report ("Risk (this agent)" column + rationale caption)summaryshape and severity-weighted scores unchangedImplemented in the linked PR. Follow-ups tracked separately.