Benchmarks

4 items across 2 editions. First seen Mon 9 Nov, last seen Wed 11 Nov. Traced across 1 weekly review.

How this story has evolved

From the week in review: the connections, developments and open questions filed under Benchmarks, newest week first.

Week of 2–8 November 2026

Connection

Terminal-Bench appears on both sides of the week: version 4.0 in the model cards, version 2.0 in the audit

Terminal-Bench appears on both sides of the week. Moonshot's model card for Kimi K3.1 Pro reports 65.7 on Terminal-Bench 4.0 against 66.4 for Claude Opus 5.5 and 58.2 for GPT-6 Astra. SemiAnalysis, publishing GDPx on 2 November, writes of its own quality-assurance process: "When we run this same QA process on popular benchmarks like Terminal Bench 4.0, it flags nearly 50% of tasks as broken." The Terminal-Bench audit paper submitted on 5 November covers version 2.0 and reports issues in "roughly 30% of the benchmark", with repairs after which "accuracy increased by over 10 percentage points".

Development · Mon 2 Nov, Wed 4 Nov, Thu 5 Nov, Fri 6 Nov

Four audits of AI benchmarks landed in five days, two of them finding defects in the benchmarks themselves

SemiAnalysis published GDPx on 2 November, an agentic benchmark across 11 domains, writing: "GDPx is a true frontier-level benchmark: even the best model scores just 12%." On its cyber tasks it reports: "Over five runs per task for each of Opus 5.5, GPT-6 Sol, Gemini 3.8 Flash Cyber and GLM 5.3, 200 runs total. 98% (196 of them) broke at least one of the task's behavior rules, and 87% (173) broke one rule rated severely misaligned." It also says: "When we run this same QA process on popular benchmarks like Terminal Bench 4.0, it flags nearly 50% of tasks as broken."

Open question

Will any lab re-publish its scores on the repaired benchmarks, or respond to the audits at all?

No lab responded inside the week. The audit covers Terminal-Bench 2.0 and releases repairs as 2.1, while the model cards read here quote Terminal-Bench 4.0; no source reconciles the two versions, and SemiAnalysis's claim about 4.0 carries no reply from that benchmark's authors.

Wednesday, 11 November 2026

Google opens Gemini 4 Carbon to paying developers at $2 and $10 per million tokens, doubling to $4 and $20 on 30 November mixedCompany claimUpdate

Google·SecurityWeek

Google says Carbon is now available to paying developers and enterprises through Google AI Studio and Gemini for business, and is rolling out to all Google AI Ultra subscribers in Google Antigravity, the Gemini app and AI Mode in Search. Introductory pricing is "$2 per million input tokens and $10 per million output tokens", the same as Gemini 4 Argon; after the introductory period ends on November 30, 2026 it moves to $4 per 1M input tokens and $20 per 1M output tokens.
The cybersecurity-focused "Carbon for cyber" variant stays gated: Google says access requires applying to join the Fairwind Program, offered on a rolling basis. Until this announcement Carbon had gone only to cyber defenders in that programme.
Google's cited results are its own: it says Wiz used Carbon in its Scan for Good programme and, scanning two major SaaS providers, found 0-day vulnerabilities in under two hours; and that its Cloud Vulnerability Research team built a proof of concept showing two bugs in a popular open-source project were exploitable, after which maintainers reclassified them as critical and issued a patch. Google says details will be shared after responsible disclosure is complete, so neither claim can be checked today.
Update to the 10 November edition, which reported Carbon's benchmark scores at launch; the new facts here are general availability, the price schedule and the Fairwind gate.

AI-enabled hacking Google DeepMind Cyber Defense Agents Benchmarks

New 100-task expert-written software benchmark scores Claude Opus 5.5 at 67.3% and GPT-6 Astra at 63.7% PreprintSingle source

arXiv

The paper describes ExpertCrowd-SWE as "a benchmark of 100 expert-authored, multi-file software engineering tasks" spanning "87 open-source projects and 11 languages", where "The median ticket is 148 words, while the median reference solution changes 471 lines".
Across "seven frontier models, five runs each (3,495 scored runs)", it reports "Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) lead without a statistically significant gap, followed by Claude Fable 5.1 (46.5%), Gemini 3.8 Flash (40.0%), and a statistically indistinguishable group of Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 (32-34%)."
The headroom claim is the useful one: "No task is fully solved by every model, and on 80 tasks no model earns full credit on all five runs. Scores span 35 points." Agents take "a median of 33 to 166 steps per run, with little relation to score".
The authors release 24 tasks and hold out 76 "to limit contamination", so independent reproduction of the full set is not possible. arXiv prints no author affiliations. Submitted 9 November, announced in the 11 November cs.AI listing.

Evals Benchmarks Agents

Mercor raises $750 million at a $20 billion valuation with Nvidia, Salesforce and Zoom investing Company claimSingle source

SiliconANGLE

SiliconANGLE reports Mercor raised "$750 million in new funding at a $20 billion valuation", with a tender offer for employee shares included. Corporate investors named are Nvidia, Salesforce and Zoom Communications; returning investors are Benchmark Capital, Felicis Ventures and General Catalyst, alongside Fundomo, G Squared, Glade Brook Capital, Sapphire Ventures, StepStone Group, XTX Ventures and funds managed by affiliates of Apollo Global Management.
The valuation doubled in about a year: in October 2025 "Felicis led a $350 million Series C round that valued Mercor at $10 billion".
The money is going into evaluation, not just data labelling — Mercor supplies human expertise for training and evaluating models from a network of "more than 5 million domain experts", and SiliconANGLE says the new funding targets enterprise customers with custom per-company benchmarks and training environments.
The revenue figure is the company's own and is dated: co-CEO Brendan Foody said in a July post on X that annualized revenue had passed $2 billion in June. Mercor has published no audited financials, and SiliconANGLE is the single source for the round's investor list.

Funding Evals Labor Benchmarks

Monday, 9 November 2026

Audit finds the standard time-series benchmark asks classifiers to identify 52 pigs rather than detect haemorrhage mixedPreprintSingle source

arXiv

"Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do" (arXiv:2611.09843, submitted 6 Nov 2026) examines the UCR Time Series Classification Archive, which it calls "the de facto benchmark" for the field. On the PigAirwayPressure dataset, the original experiment asked "could airway pressure reveal hemorrhage before it became critical?", while in UCR "the same recordings ask a different question: 'which of 52 pigs produced this trace?'"
The audit reports that "several datasets are repurposed from their original task, built from toy setups, or have flawed test splits", and questions how representative the archive's "ordered non-temporal sequences (e.g. spectra, shape outlines)" are of real time-series classification.
This is the second benchmark-integrity audit in four days, after the Stanford review of Terminal-Bench 2.0 covered on 6 November. The authors call for "more careful interpretation of archive-wide scores" rather than retraction, and publish their audit code. It is a preprint, and the paper does not quantify how much published accuracy on the archive is affected.
Authors are at the University of Freiburg, Prior Labs and the ELLIS Institute Tübingen, including Frank Hutter.

Evals Benchmarks Healthcare