Amazon
2 items across 2 editions · appeared in the last 2 editions in a row. First seen Mon 14 Sep, last seen Tue 15 Sep.
- Help Net Security reported on 14 September that AWS has released the Deception Benchmark, a public dataset of "14,822 samples across 16 programming languages and more than 70 Common Weakness Enumeration (CWE) categories", of which "9,695 are scored. These include 6,988 code-level and 2,707 environment-gated challenges." The deceptive samples place real vulnerability patterns next to controls that stop them being exploited.
- AWS evaluated "12 models from five providers". With direct prompting, Help Net Security reports, models "incorrectly flagged 41% to 99% of safe code".
- Asking the models to prove exploitability "reduced false positives by 17 to 74 percentage points" but pushed false negatives to "7% to 44%". AWS treats false-positive and false-negative rates below 10% as "a minimum bar for production use", and Help Net Security reports "None of the tested configurations met both thresholds."
- This is AWS evaluating models on a benchmark AWS built, and the accompanying whitepaper is not peer reviewed. Help Net Security reports the models struggled most with environment-gated cases such as Kubernetes network policies.
- "GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents" (arXiv 2609.12191, submitted 10 September, announced 14 September) is by Umesh Bodhwani, Thanh Tran and Kai Wei; the paper's title page lists Amazon. It measures 25 agents from six providers on the τ²-bench and SimulatorArena benchmarks against a grounded verifiable reward.
- The abstract reports a "satisfaction-success gap": "conversations rated satisfied by our blind panel are decorrelated from actual success, with 57.5% of them failing the customer's task, a pattern consistent across five rater populations, both benchmarks, and every subjective dimension we rated".
- The authors also report that the judge's ranking holds across a broad capability span but "loses resolution among the near-equal strong agents: this decision-disagreement rate jumps from <1% on wide-reward pairs to 31% on close pairs". They propose a judge-free completion bit as a zero-cost tripwire for truncation regressions.
- Preprint, not peer reviewed. The measurement uses LLM user-simulators rather than real customers, and the paper does not claim a figure for how often deployed agents leave real people believing a task was done when it was not.