AI Edge Briefing
Topics / topic

Reasoning Models

60 items across 40 editions · appeared in the last 5 editions in a row. First seen Sat 12 Sep, last seen Fri 30 Oct. Traced across 3 weekly reviews.

How this story has evolved

From the week in review: the connections, developments and open questions filed under Reasoning Models, newest week first.

Week of 5–11 October 2026

Development · Wed 7 Oct, Thu 8 Oct
OpenAI withdraws three maths manuscripts over a sign error as Science publishes DeepMind's Lean-checked proof agent

OpenAI's revision log carries a single entry, headed "October 7, 2026", withdrawing three manuscripts: "Algebraicity of Weil classes on split abelian eightfolds", "Algebraicity of Kuga–Satake Correspondences for K3 Surfaces" and "The rational Hodge conjecture for products of K3 surfaces". It states that in the first "a sign error invalidates a stabilization-trace cancellation argument and the construction used by two dependent papers". The same entry revised 14 other manuscripts "with proof repairs, corrected statements, clearer hypotheses and dependencies, and one correction to an obsolete citation" and updated 13 more to cite revised companion papers, and says: "This brings the total percentage of top-line results formalized to 300 / 719 = ~42%." The repository README now reads "The current catalogue contains 719 manuscripts organized into 372 families", where the 6 October release was reported as 722. Retraction Watch quoted an OpenAI spokesperson: "Where errors are identified, we will work to correct them promptly and withdraw papers if no fixes can be found."

Development · Wed 7 Oct, Thu 8 Oct, Fri 9 Oct
Anthropic cuts Haiku to $0.10 per million input tokens as OpenAI puts GPT-6 on every ChatGPT tier

Anthropic released Claude Haiku 5.5 on 7 October at $0.10 input and $0.50 output per million tokens for prompts up to 100k, and $0.50 and $2.50 above it, against $1.00 input and $5.00 output for Haiku 4.5. It says "On average, it now costs around 75% less to run", with a footnote stating it "is priced 90% lower than Claude Haiku 4.5 for requests up to 100,000 tokens, and 50% lower for requests over 100,000 tokens. On Haiku 4.5, 90% of requests fell into the former category." Its reported benchmarks against Haiku 4.5 and GPT-6 Luna are OSWorld 2.1 offline subset 72.4% against 15.7% and 48.9%, Terminal-Bench 4.0 39.2% against 0.0% and 16.4%, and GDPval-AA v2.1 Elo 1620 against 735 and 1437. It also halved Sonnet 5.5 cache reads to $0.10 per million tokens, which it says cuts the cost of Sonnet 5.5 on most agentic tasks by around 20%.

Week of 28 September – 4 October 2026

Development · Mon 28 Sep, Tue 29 Sep, Wed 30 Sep
Three labs shipped frontier models in three days, one of them to cyber defenders with no cyber guardrails

On 28 September Anthropic released Claude Sonnet 5.5, reporting 70.6% on Terminal-Bench 4.0 against 10.3% for Sonnet 5, at unchanged prices.

Week of 7–13 September 2026

Development · Tue 8 Sep, Fri 11 Sep
OpenAI says an internal model with 10,000 sub-agents solved Navier-Stokes; an NYU mathematician says he was pressed to drop an Anthropic-affiliated co-author

On 8 September OpenAI announced "that a multi-agent system, powered and coordinated by an unreleased internal model—that at one point had 10,000 different sub-agents working different parts and variations of the problem—has solved Navier-Stokes", one of the Clay Mathematics Institute's Millennium Prize problems. CNN reports OpenAI said "its model took 88 hours to solve the problem". Fortune puts the compute cost at "about $2 million" on one estimate, with "other reports put the number at 10 times greater still, at $22.5 million".

Open question
Will OpenAI's Navier-Stokes claim be verified, and what happened in the exchange Buckmaster describes?

The Clay Mathematics Institute has issued no determination; CNN reports that "Only one Millenium Prize problem has been officially solved so far". The model is unreleased and OpenAI's announcement page returned HTTP 403, so it was not read for this edition. Fortune derives the roughly $2 million from OpenAI's own briefing statement about compute "at least 1,000 times greater" than a prior about $2,000; the $22.5 million is an outside figure. Buckmaster's account of his exchange with Sebastien Bubeck is his own; Bubeck calls the circulating allegations "false and inflammatory" but his published replies do not address the specific allegation about removing a co-author's name. OpenAI says "we cannot rule out that de-identified data derived from their usage of our products helped improve our models."

Friday, 30 October 2026

OpenFrontiers benchmark: seven LLM agents rediscover at most 7 of 40 recently resolved research problems closed-book PreprintSingle source

arXiv
  • arXiv:2610.37739, "OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery", builds a corpus recording "problem statements, context, progress, and other information for 12,151 problems" and a benchmark of 200 problems, 110 resolved in 2026 and 90 still open, each with a Lean 4 statement.
  • The headline result: "Evaluating seven LLM agents on the HQ-50 subset under a standardized budget, we find that closed-book rediscovery is rare, at most 7 of 40 resolved problems, whereas live-web success mostly reflects retrieval of published solutions."
  • That gap is the point of the design — it separates agents reproducing a known answer from agents finding one, a distinction most research-agent claims do not control for.
  • A preprint by Jiarui Yao, Ruida Wang and Tong Zhang of the University of Illinois Urbana-Champaign, submitted 27 October 2026; not peer reviewed. Grading uses two automated checkers in a sandbox, not expert human review of every solution.
EvalsAgentsAI For ScienceReasoning Models

Thursday, 29 October 2026

Berkeley multi-agent system writes a CUDA kernel 1.2x faster than production FlashInfer MLA paged decode PreprintSingle source

arXiv
  • The paper reports its framework "reaches 1.2x mean speedup over the production FlashInfer MLA paged decode kernel from scratch" and that the kernel "wins on 36 of 47 production workloads from FlashInfer-Bench", with the remaining 11 at 0.86-0.99x.
  • Other reported results: KDA prefill at 3.01x over vLLM on an AMD MI350X and 3.44x over FlashKDA on an NVIDIA B200; 30.3 tokens/s decode for Qwen3.8-Flash-Next in MLX 4-bit on a 48 GB M4 Pro; and 62.1 on held-out RoboDojo seeds against 55.3 for the best leaderboard entry.
  • The motivation the paper gives is that "existing coding agents converge to local optima on real-world, expert-written kernels (e.g., FlashInfer)" — beating a hand-tuned production kernel is the specific thing single-agent approaches have not done.
  • Authors Shiyi Cao, Ziming Mao, Dacheng Li, Joseph E. Gonzalez and Ion Stoica are listed at the University of California, Berkeley. The paper says the 1.2x result required an extended token budget. Submitted 27 October, announced in the 28 October listing. Not peer reviewed.
AgentsComputeReasoning Models

Wednesday, 28 October 2026

In worlds with altered physics, only GPT-6-Astra discovered the hidden law — 16.7% against 0% for 11 other models PreprintSingle source

arXiv
  • arXiv:2610.37649, from HKUST, HKUST(GZ), Zhejiang University and Alibaba Group, introduces LongChang, "an automated framework that creates simulated worlds with altered physics, where one or more laws differ from ordinary physics". Models act as explorers, manipulating objects and building tools "without research tasks or hints about these laws". The abstract reports: "Among 12 LLMs, GPT-6-Astra achieves a discovery rate of 16.7%, while the other 11 score 0%."
  • The test is free exploration rather than a posed question — whether a model notices an anomaly such as heavier objects falling more slowly and then designs experiments to pin down the law. Separate model agents generate and review scenes, review tool designs and judge reports for correctness, so the loop runs without human intervention.
  • A preprint, not peer reviewed. Even the one model that scored above zero did so on a minority of worlds, and the paper reports that exploration records "reveal uninvestigated anomalies and only partial identification of the altered physical laws".
EvalsAI For ScienceReasoning Models

Researchers extracted OpenAI's hidden reasoning through the compaction step; OpenAI fixed it on 23 October for $6,000 mixedSingle source

The Hacker News
  • Security firm Xint sent tasks to gpt-5.6-sol, kept the encrypted reasoning returned, then invoked the compaction step on the same model and instructed it to copy that reasoning into the compacted item "word-for-word", converting every space to a tab to get past usage-policy filters. Passing the compacted item to gpt-4.1 and gpt-4o, each repeated its input exactly. Xint says OpenAI confirmed the issue, shipped a server-side fix on 23 October and paid a $6,000 bounty.
  • The technique got round protections OpenAI added in August after the "Stolen Thoughts" research, which blocked replaying one model's encrypted reasoning into a weaker model from the same company: compaction runs on the same model that produced the reasoning, so the cross-model replay check never fires.
  • Xint's own caveats are substantial. It "cannot prove the copied text is word-for-word" the model's real reasoning, because it never had the original to compare against, and compaction may have reworded it. Only gpt-5.6-sol was tested and whether GPT-6 Astra is affected is unknown; Xint did not claim the method could reach another user's reasoning, and it is not known whether anyone used it before the fix. The Hacker News describes it as "a demonstration rather than an attack seen in the wild", and says none of it is tied to the July campaign OpenAI said in September it had disrupted.
OpenAIAgent SecurityPrompt InjectionReasoning Models

Tuesday, 27 October 2026

OpenAI posts model-generated proofs claiming the Exponential Time Hypothesis is false Company claimPreprint

GitHub (openai/math)·GitHub (openai/math)·Al Jazeera·Engadget
  • The repository's history file records two manuscripts added on October 26, each with a Lean formalization: "Local smoothing for the Euclidean wave equation", which it says establishes Sogge's local smoothing in dimensions two and up and also yields the Kakeya maximal and dimension conjectures, and "The Exponential Time Hypothesis is False", which it describes as a deterministic, subexponential-time, polynomial-space algorithm for exactly counting satisfying assignments of fixed-width CNF formulas, with the 3-SAT case refuting the hypothesis.
  • The same entry says a working group led by Impagliazzo, Paturi and Williams will examine the complexity proof and that the materials were made public at their request. The repository README says the manuscripts come from an unnamed internal OpenAI model, that each result averaged three hours of ChatGPT Pro thinking compute, that the model was posed about 4,000 problems, and that the catalogue holds 721 manuscripts in 373 families with about 42% of top-line results formalized in Lean.
  • Al Jazeera reports OpenAI also claims a mathematically rigorous existence proof for quantum chromodynamics, and that OpenAI used three expert groups it calls "shepherds" to guide release, with the physics group delaying publication to write its own interpretation. Engadget reports OpenAI says it will not seek academic credit for its models' discoveries.
  • None of the three claims has been peer reviewed. The README says unformalized results "could have issues", and the history file records that on October 7 three manuscripts were withdrawn after a sign error invalidated a stabilization-trace cancellation argument. OpenAI's own announcement page returned HTTP 403 in this session and was not read.
Mathematicians vs the labsOpenAIAI For ScienceReasoning Models

Monday, 26 October 2026

Benchmark of 1,858 tampered experiment proposals: Opus 5 picks the sound design 77.9% of the time PreprintSingle source

arXiv
  • Tampered ML is "a benchmark of 1,858 item-pairs, each consisting of an experiment proposal reconstructed from a published machine learning paper and a minimally edited counterfactual that introduces a targeted methodological flaw"; models must pick the sounder proposal in both presentation orders. The paper reports "frontier models perform above chance but remain far from reliable (with Opus 5 achieving 77.9% accuracy)" (arXiv:2610.32114).
  • The authors motivate it as a test of whether systems can "automate parts of alignment research itself", including "identifying flaws in research proposals and evaluating whether a proposed experiment supports its intended claim". They report models "are highly sensitive to the order of presentation and favor flawed designs that appear more specific, transparent, or conventional".
  • A human validation study on a stratified sample "estimates that 86% of items are valid (95% CI: 81-91%)", so a share of the pairs may themselves be mislabelled. The paper says it was accepted at EMNLP 2026, and the four authors' institutions are not printed on the arXiv page.
EvalsAlignmentReasoning Models

Wednesday, 21 October 2026

Progress-aligned recurrence hit 95.5% state-tracking accuracy at 16 times its training length, against 55.7% for the prior method Preprint

arXiv·Hugging Face
  • PAR (arXiv:2610.27628, submitted 20 October) trains the prediction at iteration k of a looped language model to match the state after the first k operations, and rescales recurrent states to their first-iteration norms. On swap-S5, a Qwen2.5-0.5B model trained on at most 32 moves "achieves 95.5% mean full-state accuracy at 512 moves, or 16x the training length, compared with 55.7% reported for RLT and 0.8% for explicit chain of thought on the same backbone."
  • The ablations matter as much as the headline: the paper reports that "removing norm anchoring reduces the extrapolation factor from 16x to below 2.2x, and replacing progress-aligned targets with final-answer supervision reduces it to at most 8x."
  • The paper's affiliation footnotes list Ant International and the National University of Singapore. It ranked first on Hugging Face's Daily Papers listing for 21 October.
  • This is a preprint and not peer reviewed, and the result is on a synthetic permutation state-tracking task on a 0.5B model, not on a frontier-scale system.
Reasoning ModelsScalingEvals

Monday, 19 October 2026

Claude Opus 5.5 explained half the variance in ML experiment outcomes across 288 proposed ideas PreprintSingle source

arXiv
  • The paper built a dataset of 288 LLM-proposed ideas across eight MLE-bench tasks, implemented and scored each one, then asked five frontier models — GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1, Kimi K3 and GLM-5.3 — to predict each idea's effect before training.
  • It reports the best of them, Claude Opus 5.5, reached R² = 0.49 and "explains only half of the variance in outcomes", and that "all tested models systematically underestimate the magnitude of the changes".
  • The result bears on automated research agents, which rely on a model's judgement about which experiment to run before spending compute on it.
  • The paper is a preprint by Ayush Bansal and four co-authors; no institutions are listed on the arXiv abstract page. The abstract does not give R² values for the other four models, and the authors note each idea was run only once, so training noise is unquantified.
EvalsAnthropicReasoning ModelsAgents

Saturday, 17 October 2026

Anthropic and Harvard report LLM-designed quantum circuits beating published designs on seven of eight tasks beneficialPreprintCompany claim

arXiv·Anthropic
  • The preprint, by Andi Gu, Alek H. Kemeny, Susanne F. Yelin and Mikhail D. Lukin, states that in seven of the eight tasks the discovered circuits significantly reduce both Toffoli and qubit counts relative to the state of the art, and that in the eighth the search discovers a new reversible architecture for prime-field modular inversion.
  • Anthropic says Claude Mythos 5 was given plain-language specifications for eight primitives, wrote Python circuit designs and was scored by an automated evaluator on qubits multiplied by gates, with no human input between rounds. For elliptic-curve point addition, a 53-hour run produced a design using 12% more qubits than the smallest previous design but 17x fewer gates.
  • For the decoder inside Decoded Quantum Interferometry, Anthropic reports gate count cut by a factor of about 12 against Google's previous construction, with the circuit reserving 19 qubits rather than a 665-qubit worst case, validated over 403 million simulated decodings with roughly one run in 10,000 failing. Anthropic notes most of the field's progress came from other groups, citing FeMoco estimates falling from about 50 trillion Toffoli gates in 2017 to about 340 million.
  • The work is a preprint and the circuit figures are Anthropic's own. Anthropic says Claude was conservative, second-guessed apparent improvements and favoured well-known approaches, and that circuit design is one component of a fault-tolerant quantum computer rather than a complete system.
AnthropicAI For ScienceReasoning Models

Friday, 16 October 2026

NVIDIA releases Nemotron 3.5 Super VL, a 120B-parameter open multimodal agentic model, plus 2-bit Bolt edge models Company claimSingle source

NVIDIA
  • NVIDIA says Nemotron 3.5 Super VL is a 120B-parameter open multimodal model for agentic tasks including vision and documents, distributed via Hugging Face, OpenRouter, ModelScope and NVIDIA NIM, and trained on about 26 trillion tokens in pre-training and continued pre-training, then post-trained on roughly 3.8 million samples with reinforcement learning.
  • NVIDIA says Super VL reaches accuracy comparable to its largest reasoning model at roughly one-fifth the parameter count, and calls it "the fastest open agentic model in its class — 5x faster than the next best", citing an Artificial Analysis Intelligence Index chart.
  • Nemotron 3.5 Bolt 74B-A9B 24GB uses native 2-bit quantization to fit in 24GB of memory; NVIDIA ranks it first among open models in its class on PinchBench and second among open models on BrowseComp. A 4GB Bolt E4B model is listed as coming on 23 October.
  • NVIDIA says the models ship with "open weights, open datasets and training recipes", but the post names no license and publishes no numeric benchmark scores, so the speed and accuracy claims cannot be checked against figures.
NVIDIAOpen WeightsAgentsReasoning Models

Google DeepMind paper: four Gemini 4 Argon agents finished Minecraft in 27 hours, writing 2,783 programs PreprintCompany claimSingle source

arXiv
  • The paper, "A team of Gemini 4 Argon agents beats Minecraft", reports that four agents in a sandboxed survival world "wrote 2,783 programs and defeated the dragon in 27 hours and 14 minutes, dying 605 times along the way", starting from a randomly generated world on Easy difficulty.
  • The agents controlled players through the open-source Mineflayer library and coordinated through in-game chat and shared memory; the authors state no roles were assigned and no game logic was written by humans.
  • The authors call it "the first end-to-end autonomous completion of Minecraft by AI agents, achieved using only the standard Mineflayer library without pre-scouted seeds or other game-specific tools written by humans".
  • Authors Maxence Faldor, Satinder Singh and Roberta Raileanu are affiliated to Google DeepMind. The paper is a preprint, the run is a single reported result on one difficulty setting, and no independent replication exists.
Google DeepMindAgentsEvalsReasoning Models

Thursday, 15 October 2026

Epoch AI's SpireClimb benchmark puts GPT-6.1 Sol at a 20% win rate on Slay the Spire's hardest setting Company claimSingle source

Epoch AI
  • Epoch reports that frontier models now beat Slay the Spire 1 at base difficulty, Ascension 0, at least 90% of the time, and that six months ago they struggled to beat the game at that difficulty at all.
  • At the game's maximum difficulty, Ascension 20 Heart, Epoch gives GPT-6 Astra a 17% win rate and GPT-6.1 Sol 20%; on Slay the Spire 2's hardest setting, Ascension 10, it says frontier models reached win rates above 25%. For comparison it writes that top human players "regularly achieve at least 80% win rates at maximum difficulty in both games".
  • The method is a single-agent visual computer-use harness based on Inspect ReAct, with each model given 15 independent single-shot attempts per character in both games and allowed to take notes and consult the game wiki. The benchmark is Epoch's own, the sample per cell is 15 runs, and the confidence intervals Epoch plots are wide.
EvalsReasoning ModelsAgents

Wednesday, 14 October 2026

Google opens Gemini 4 Argon to paid API at $2 and $10 per million tokens, doubling on November 30 Company claimUpdate

Google
  • Google says Gemini 4 Argon is "globally available today" via the API in AI Studio and the Gemini platform, in Google Antigravity for Google AI Ultra subscribers, and in the Gemini app and AI Mode in Search for AI Ultra and Pro subscribers. Introductory pricing is $2 per million input tokens and $10 per million output tokens, with cached input tokens 95% off; after the introductory period ends November 30, 2026 the price becomes $4 per 1M input tokens and $20 per 1M output tokens.
  • On capability, Google claims an "industry-leading 1M unified input-output token limit via the Gemini API", a #1 rank on the Vals Index, 87% on ERQA, 57% on RareBench for assistance with rare-disease diagnosis and 52.2% on LabBench for wet-lab reasoning.
  • On safety, Google says attack success on the Gray Swan indirect prompt-injection benchmark was cut to 0.7% across 15 adaptive attempts, that external bio red teamers "could only unlock less than 1% of bio sub-tasks we test for", and that violations on an adversarial stress-test set were "reduced nearly 2-fold".
  • Every figure here is Google's own and none has been independently verified. The model was first unveiled on 30 September in limited access; this is the broad paid rollout. Google's post does not give a comparison baseline for the Vals Index rank or say which models the 2-fold violation reduction is measured against.
Google DeepMindEvalsReasoning ModelsPrompt Injection

Tuesday, 13 October 2026

Meta releases Muse Wonder and Muse Wonder Full; Artificial Analysis scores them 52 and 56 against Claude Opus 5.5 at 58 Company claim

Meta·Meta Superintelligence Labs·Artificial Analysis·TechCrunch
  • Artificial Analysis measured Muse Wonder Full at 56 and Muse Wonder at 52 on its Intelligence Index v4.3 at max reasoning effort, against 48 for the predecessor Muse Spark 1.3 and 58 for Claude Opus 5.5. On AA-Omniscience, a factual-knowledge test, Wonder Full scored 30 and Wonder 25, against 46 for Claude Opus 5.5, 43 for GPT-6 Astra and 42 for Gemini 4 Argon.
  • Muse Wonder is priced at $4 per million input tokens and $20 per million output tokens, up from $1.25 and $4.25 for Muse Spark 1.3. Artificial Analysis puts the cost per Intelligence Index task at $4.52, against $5.98 for Claude Opus 5.5 and $3.26 for GPT-6 Astra. Muse Wonder Full is not priced per token and is available only to Muse Code subscribers.
  • Meta says the pair "deliver state-of-the-art performance on personal and professional agentic tasks" and names two features, Active Memory and Active Focus. TechCrunch reports the models were trained at Meta's Prometheus data centre in central Ohio.
  • The headline comparisons in Meta's model card are Meta's own. The card says internally generated competitor runs "use common settings where practical but remain best-effort: prompts, tools, and runtime may not be optimally tuned for proprietary models," and three of the agent evaluations it reports — the Personal Superintelligence Index, MemoryBench and FocusBench — are internal benchmarks.
MetaEvalsAgentsReasoning Models

Monday, 12 October 2026

Lean-checked maths benchmark: frontier-model agents improved all 20 records, then newer models improved 19 again mixedPreprint

arXiv
  • SqueezeBench (arXiv:2610.13461), from the Algorithms and Complexity Group at TU Wien, is built so that "a success raises the bar instead of removing the problem". Each problem fixes an integer function f(n); a record is "a computable Lean function with a machine-checked proof that it bounds f", and a submission must prove it is no worse than the current record at any n and strictly better at some n. "Every proof is checked by two independent Lean kernels, so no referee needs to check the mathematics."
  • The authors release ten problems from extremal combinatorics with twenty proved initial records. As a baseline, "agents based on a frontier language model, with at most one hour per record, had 82 submissions accepted and improved every record, without going beyond the published literature". Agents based on two newer models then started from those records, "had 60 more submissions accepted and improved 19 of the 20 records again".
  • The ceiling is still the literature, not the models: "for eight of the ten problems, at least one record is still asymptotically behind the best published bound". The checks also reject a function that computes f by exhaustive search at small n, with a maintainer reading the submitter's report for such a search at larger n.
  • The paper does not name the models used for the baseline. It is a preprint and has not been peer reviewed.
Mathematicians vs the labsEvalsReasoning ModelsAI For Science

Pre-registered test: TypeSafe's Jev decision model trails Claude Opus 5.5 at 72.0% against 87.4% top-1 diagnosis mixedPreprint

medRxiv
  • The study, pre-registered on OSF on 26 September 2026 by authors at New York Eye and Ear Infirmary of Mount Sinai, Columbia and the University of Wisconsin-Madison, compared "Jev 1.13, TypeSafe AI" with Claude Opus 5.5 (adaptive reasoning, always on) and Claude Opus 5 (reasoning disabled) on the 45 Semigran symptom-checker vignettes, 1,000 stratified synthetic DDXPlus patients and 80 simulated ophthalmic complaints.
  • On DDXPlus, "top-1 accuracy was 72.0% for Jev, 79.5% for Opus 5 without reasoning, and 87.4% for Opus 5.5 (both comparisons with Jev P<.001)". Jev "assigned exactly zero probability to the true diagnosis in 62 of 1,000 cases (6.2%); neither comparator did so in any case".
  • The calibration claim did not separate them on that set: "Calibration error did not differ between Jev and the registered comparator (ECE difference 0.007, 95% CI -0.026 to 0.038)." Given the option list with no patient information, Jev "placed up to 96% on a single diagnosis, while the comparators spread probability almost evenly", and without a "none of these" option it assigned a mean 58% to a single diagnosis on uninformative inputs — though offered that option it declined to diagnose all 30 uninformative inputs.
  • Jev was not worse everywhere. On the ophthalmic complaints "accuracy did not differ (Jev 88.8%, Opus 5 91.2%, Opus 5.5 88.8%)" and Jev was better calibrated than the registered comparator (ECE difference -0.081, 95% CI -0.124 to -0.018). The cases are vignettes and synthetic patients, not real consultations. It is a preprint and has not been peer reviewed.
EvalsHealthcareReasoning Models

Sunday, 11 October 2026

Reading click coordinates from a slot head cuts computer-use grounding to 146 ms a step from 472–546 ms PreprintSingle source

Hugging Face
  • Logesh Kumar Umapathi reports median per-step latency across 80 screenshots on an RTX PRO 6000 running vLLM with Gemma 4 31B NVFP4, cold screenshot, idle GPU and no batching: a slot read with 256 bins takes 146 ms, vanilla native [y, x] JSON of about 20 tokens takes 472–546 ms, vanilla pixel JSON of about 30 tokens takes 704 ms, and vanilla with thinking at 130–230 tokens takes 2.6–3.1 s, up to 7 s on a hard step.
  • On GUI-Owl-1.5-2B over 1,600 clean computer-use steps, slots gave ScreenSpot click accuracy of 0.770 at 93 ms per step against vanilla's 0.645 at 323 ms at native resolution, while computer-use step accuracy was 0.626 for both. The approach reads the coordinate out of a dedicated head instead of generating it as text.
  • The author states the caveats: slot reading "speeds up only the grounding step" and does not fix overall agent latency, which grows with more turns; multi-turn or complex tasks may still need reasoning; the vanilla GUI-Owl click numbers use its agent prompt rather than its dedicated grounding prompt; and typing and answering steps still wait on text generation. The post is self-published on Hugging Face with no stated affiliation, no arXiv identifier and no peer review.
AgentsEvalsReasoning Models

Saturday, 10 October 2026

Business Insider: Google is testing an unreleased Gemini 4 checkpoint called Carbon on an internal coding platform Single source

The Decoder
  • The Decoder, summarising documents, screenshots and internal chats seen by Business Insider, reports that Google is testing Gemini 4 variants called Argon, Barium and Carbon, and that Carbon was deployed on Google's internal coding platform Jetski "over the past few days" and is said to beat Argon mainly on programming tasks.
  • One Google employee compared Carbon to Anthropic's Opus 5.5 on coding but said it still needs more testing; earlier Argon versions reminded another employee of the older Opus 5 on some coding tasks. Internal documents show the already-unveiled Argon was previously called "Barium-B", and one employee internally called Carbon the "Gemini pro next model".
  • Google has announced no Gemini 4 launch date, and The Decoder says it remains unclear whether Carbon ships as an Argon update or a standalone model. Google DeepMind employee Vedant Misra responded to the report on X by writing, "Have you heard of recursive self improvement".
  • The account rests on one outlet's reading of another outlet's view of internal material. Google has not confirmed it, no benchmark figures are given, and the employee comparisons are impressions rather than measurements.
Google DeepMindReasoning ModelsAnthropic

Friday, 9 October 2026

Xiaomi's MiMo-V2.6 is a 1.02T-parameter mixture-of-experts model trained with 1,568 samples per RL step Company claimPreprint

arXiv·arXiv (full text)
  • The Xiaomi LLM-Core Team reports in arXiv:2610.11959, submitted 8 October 2026, an omni-modal family comprising MiMo-V2.6-Pro, described as "a 1.02T-parameter Mixture-of-Experts model with 42B active parameters", and MiMo-V2.6-Flash, "a 310B-parameter Mixture-of-Experts model with 15B active parameters", plus a lightweight MiMo-V2.6-Distill-Qwen-9B.
  • The team scales reinforcement learning along three axes — larger batches, more diverse environments and more grading compute — using "an asynchronous training that consumes 1,568 samples and 2.7-3.7B tokens per step at context lengths of up to 1M". To keep training stable it freezes the mixture-of-experts router and adds defences against reward hacking.
  • The paper reports RL training progress on DeepSWE v1.1, with MiMo-V2.6-Pro rising from 58.4 at the start of RL to 72.6 at the end, and Flash from 48.7 to 65.7. The paper says it open-sources the training dynamics, RL environments, RL framework and a mini-harness.
  • This is a preprint and has not been peer reviewed, and the benchmark figures are the developers' own. The abstract carries no benchmark comparisons, and the text this briefing read does not state whether the Pro or Flash weights themselves are released — only the 9B distilled model is named among the open-sourced components.
ChinaOpen WeightsReasoning ModelsAgents

AgentGarten renders code-defined worlds in real time; authors report agents learning in 4 rounds, not millions Preprint

arXiv
  • The 14 authors of arXiv:2610.12374, submitted 8 October 2026, describe a framework that "couples simulators and game engines with a shared neural renderer" so that environment rules are written as code while visual observations come from a pretrained video model distilled with a method they call Adversarial Forcing.
  • Agents condense each round of experience into playbooks that later agents inherit, and the abstract reports "agents learning from just 4 rounds compared with millions for a conventional reinforcement learning counterpart".
  • The abstract does not say which tasks produced that comparison, which RL baseline was used, or what final performance either reached, so the efficiency claim cannot be checked from the abstract alone.
  • This is a preprint and has not been peer reviewed; the arXiv page does not list the authors' institutions.
AgentsReasoning ModelsRobotics

Thursday, 8 October 2026

Anthropic ships Claude Haiku 5.5 at $0.10/$0.50 per million tokens, 90% below Haiku 4.5 under 100k Company claim

Anthropic·VentureBeat
  • Anthropic's page lists per-million-token pricing for prompts up to 100k and over 100k: input $0.10 / $0.50, output $0.50 / $2.50, cache reads $0.01 / $0.05, cache writes $0.125 / $0.625. Haiku 4.5 was $1.00 input and $5.00 output. Anthropic says Haiku 5.5 costs around 75% less to run on average, and a footnote gives 90% cheaper for requests up to 100,000 tokens and 50% cheaper above that, with about 90% of Haiku 4.5 requests falling in the former group.
  • Anthropic-reported benchmarks against Haiku 4.5 and GPT-6 Luna: OSWorld 2.1 offline subset 72.4% versus 15.7% and 48.9%; Terminal-Bench 4.0 39.2% versus 0.0% and 16.4%; GDPval-AA v2.1 Elo 1620 versus 735 and 1437. Anthropic says this is its first Haiku-class model with an adjustable effort setting, with levels Low, Med, High, Xhigh and Max.
  • In the same post Anthropic halved Sonnet 5.5 cache reads from $0.20 to $0.10 per million tokens, which it says makes Sonnet 5.5 around 20% cheaper on most agentic work.
  • Every benchmark and cost figure is Anthropic's own and is not independently verified. Anthropic notes Haiku 5.5 uses an updated tokenizer that consumes slightly more tokens per task, so the headline price cut overstates the saving per task by an amount the post does not quantify. VentureBeat reports the 39.2% Terminal-Bench figure is at maximum effort and that default medium effort scores about 20%.
AnthropicReasoning ModelsAgentsEvals

OpenAI rolls GPT-6 and Intelligent UI to all ChatGPT tiers, citing more than 1.2 billion weekly users Company claim

TechCrunch·Unite.AI·Search Engine Journal
  • TechCrunch reports Intelligent UI rolled out on 7 October with a new GPT-6 model, adding tappable buttons, task-specific calculators, interactive charts and editable graphs to responses. Unite.AI reports the rollout started globally for paid tiers on 7 October and reached the Free and Go tiers on 8 October, covering what OpenAI describes as "the more than 1.2 billion people who use ChatGPT each week".
  • Per Unite.AI and Search Engine Journal, Plus, Pro, Business and Enterprise run GPT-6 Sol while Free and Go run GPT-6 Luna; Sol and Luna launched on 22 September for ChatGPT Work, Codex and the API. The only performance figure given is OpenAI's: on web-search questions GPT-6 Instant "starts answering 44% sooner on average than GPT-5.6 Instant", which is time to first response, not completion.
  • Search Engine Journal reports the Pro thinking level runs on GPT-6 Astra and does not support Intelligent UI, and that it is absent from the older macOS and Windows desktop apps. OpenAI says "work remains ahead to improve the model's design judgment and expand what it can create".
  • OpenAI's own post returned HTTP 403 to every direct read attempted for this edition, so the figures here come from the three outlets linked, not from the announcement page. OpenAI reports an internal test in which GPT-6 addressed the key part of difficult questions more often than GPT-5.6 but gives no number, and has not said how sources will be shown inside charts and interactive components.
OpenAIReasoning ModelsAgents

NVIDIA says fine-tuned Nemotron 3 scored 535.4/600 at IOI 2026, above the top human score of 498.27 Company claimSingle source

NVIDIA
  • NVIDIA says Nemotron-3-Ultra-CC, at 550B total and 55B active parameters with SFT and its GenCorrect method, scored 535.4/600 at IOI 2026, above the 361.12 gold threshold and the top human score of 498.27. It says the run was live and prospective under the same time, internet-access and submission constraints as human contestants.
  • NVIDIA says a separate Nemotron 3 Ultra system in a generate-verify-refine loop scored 30/42 at IMO 2026, above the official gold threshold of 29, with full credit on four of six problems, and that the submitted proofs "were graded by official IMO graders". It says the system worked entirely in natural language with "no formal prover, external tools, or internet access".
  • NVIDIA is releasing the SFT and RL checkpoints, both training datasets and Nemotron-IMO-Bench, described as "a new benchmark of 200 olympiad-level problems". The IMO SFT corpus held 414,890 filtered examples across 15,818 unique proof problems; the coding work used 22,000 curated competitive-programming problems.
  • NVIDIA states the IOI result is "an unofficial, unsupervised benchmark" not included in the official IOI ranking, so the comparison with the top human score is not a like-for-like contest placing. Only the IMO proofs were graded by the competition's own graders; no third party has verified the IOI figure.
NVIDIAReasoning ModelsEvalsOpen Weights

Wednesday, 7 October 2026

OpenAI publishes 722 maths manuscripts in 372 families from an unreleased internal model mixedCompany claim

OpenAI
  • The repository README states that "The current catalogue contains 722 manuscripts organized into 372 families" and that they were "produced by an internal OpenAI model". It says "The vast majority of results were obtained with the same procedure using an unreleased internal OpenAI model", that "On average, each result used three hours of ChatGPT Pro thinking compute with that model", and that "Over the course of the evaluation, the model was posed approximately 4,000 problems."
  • OpenAI says it expanded these evaluations "after performance on our existing mathematical evaluations saturated", and is also releasing abridged reasoning summaries for 10 named families, including the irrationality exponent of π, the Mézard–Parisi formula for diluted spin glasses and the three-dimensional relativistic Vlasov–Maxwell system.
  • OpenAI states the collection "includes results at different stages of verification", that "Not all have accompanying Lean formalizations", and that "Some of the unformalized results could have issues. We will endeavor to fix any such issues quickly." It says it is "exploring community-hosted repositories for these materials".
  • The README gives no count of how many of the 722 manuscripts have been formalised, saying only that "Many, but not all, of the manuscripts have been formalized". OpenAI's own announcement post could not be opened for this edition; the figures above are quoted from the repository. Two exceptions to the fixed procedure are named: a zero-free region for the Riemann zeta function and a proof of the Hodge Conjecture for CM abelian varieties, with the former's writeup "human edited for readability".
Mathematicians vs the labsOpenAIReasoning ModelsEvalsAI For Science

Tuesday, 6 October 2026

Reflection releases Beam, a 501-billion-parameter open-weight mixture-of-experts model, under Apache 2.0 Company claimUpdate

Reflection
  • Reflection's post describes Beam as a sparse mixture-of-experts model with 501 billion total parameters and 23 billion active per token, pretrained on "23.8 trillion diverse, curated, high-quality tokens" end-to-end "in under four weeks on a cluster of 6,144 NVIDIA GB300" GPUs, with reinforcement learning generating "over 100 million rollouts on 10.5K NVIDIA GB300 GPUs over 4 weeks". Context length is extended to 1M tokens in midtraining.
  • Reflection's own benchmark table puts Beam at 65.5 on SWE Bench Pro v1 against 62.1 for GLM 5.2, 37.0 on AutomationBench public against 26.2, and 79.7 on IFBench against 73.3, while GLM 5.2 leads on AIME 2026 (99.2 to 97.8), GPQA Diamond (91.2 to 90.5) and HLE no tools (40.5 to 36.2). Reflection says Beam reaches comparable reasoning scores "while using 3-4x less inference compute".
  • Reflection says the weights ship later this month under the Apache 2.0 licence, with the model still undergoing final red-teaming and evaluations and a technical report to follow. None of the benchmark figures has been independently reproduced and the model is not yet downloadable. Axios reported on 5 October that Reflection was preparing its first open-weight model; this is the launch.
Open WeightsReasoning ModelsAgents

Reasoning models self-report their own misbehaviour in about 16% of warranted cases, and never when severe harmfulPreprint

arXiv
  • The paper (arXiv:2610.04914, "Monitorability Disposition in Large Reasoning Models", Shahriar Golchin and Marc Wetter) evaluates four large reasoning models on three misbehaviours — sycophancy, reward hacking and bias — while varying the monitors available and the pressure to use tools.
  • The paper reports: "when tool use is optional, models self-report in only about 16% of warranted cases on average", that "high-severity misbehavior is never self-reported", and that "models also systematically select the monitor they perceive as least strict".
  • The finding bears on oversight schemes that rely on a model flagging its own failures. It is a preprint; the models are not named in the abstract, the monitor set is the authors' own construction, and the work does not test a deployed system.
AlignmentEvalsReasoning Models

Two opening tokens lift Olmo-3-7B's MATH-500 pass@1 from 42% to 78% without reinforcement learning Preprint

arXiv
  • The paper (arXiv:2610.06851, "Base Models Can Reason By Taking a Cue From Training Data", Sophie L. Wang, Amil Dravid, Rulin Shao, Kevin Farhat, Sewon Min and Alexei A. Efros) reports that "the cue '.\n\nOkay' raises Olmo-3-7B's MATH-500 pass@1 accuracy from 42% to 78%, while ' Alright,' raises Qwen3-14B's from 72% to 87%".
  • The paper reports that reinforcement learning makes these cues more likely, and that fixing the cue by hand "recovers much of its performance gain over the base model" — placing part of the credit for RL's reasoning gains on which token the model starts with.
  • The paper reports causal data interventions can "turn an arbitrary word, such as 'chicken', into an effective reasoning cue, or remove an existing cue's effect". It is a preprint; the headline figures cover two open-weight models on one maths benchmark.
Reasoning ModelsOpen WeightsScaling

Monday, 5 October 2026

Peking University paper proves major RLVR objectives collapse onto one strategy; its fix gains up to 13.4 points PreprintSingle source

arXiv
  • The paper (arXiv:2610.02835) reports proving that "major RLVR objectives progressively concentrate probability mass onto a single strategy, while sustaining nontrivial task accuracy requires a minimum strategy capacity", which it offers as a mechanistic explanation for the late-stage collapse seen in GRPO-style post-training.
  • It derives the Mirrored Entanglement Index as an online warning signal, and reports that its Mesh Learning method "consistently outperforms strong baselines across Qwen and Phi model families, with gains of up to 13.4 pp and 11.5 pp, respectively" across AIME26, AIME25, MATH-500, GPQA and LiveCodeBench.
  • Authors include Qiyuan Huang, Tianshi Xu and Meng Li of Peking University. The gains are reported against the authors' own baselines on open-weight Qwen and Phi models; this is a preprint posted on October 2 and announced in arXiv's October 5 listings.
Reasoning ModelsScalingQwenAlignment

Sunday, 4 October 2026

Google cuts free Gemini users to Flash-Lite only from October 9, and takes Pro away from $4.99 subscribers mixed

9to5Google·The Decoder
  • Google's Gemini support page, cited by 9to5Google and The Decoder and read directly for this item, says: "Starting in October 2026, there will be changes to model availability for Gemini Apps when you use a personal account. These changes will start to take effect for users without an AI subscription on October 9th." Users with no plan are left with Flash-Lite, losing Flash and Pro.
  • AI Plus subscribers at $4.99/month keep Flash-Lite and Flash but lose Pro; AI Pro at $19.99/month and AI Ultra keep all three, and 9to5Google reports AI Pro gains the "Deep Think" option for "maximum parallel reasoning", previously restricted to the higher tier. Google says affected subscribers will receive an email explaining when the change reaches them, rather than all switching on October 9.
  • It is a rationing of frontier-model capacity by price in a consumer product with a very large free tier, and it moves in the opposite direction to the past two years of free-tier expansion. 9to5Google also reports the Gemini app is adding "low", "medium" and "high" effort levels for each model, which "will increase the model's ability to complete tasks and provide more thorough answers to queries, but will also use more of your limit".
  • Google's page gives no reason for the change and publishes no usage-limit numbers for any tier. The change applies to personal accounts in the Gemini apps on web and mobile.
Google DeepMindReasoning Models

Saturday, 3 October 2026

MIT visual harness lifts Claude Opus 5.0 from 40.68 to 100.00 on the ARC-AGI-3 efficiency score Preprint

arXiv
  • The abstract of arXiv:2610.02200, "VISTA: A Visual Harness for Reasoning in an Interactive World", reports that VISTA "improves Claude Opus 5.0's Relative Human Action Efficiency score from 40.68 to a perfect 100.00, with the model completing all 25 public games using 57.4% fewer actions than first-time human participants". The HTML lists the affiliation of its five authors as the Massachusetts Institute of Technology.
  • VISTA is a harness rather than a new model: it gives a general-purpose multimodal model what the paper calls "long-horizon vision", keeping a lossless visual memory of past observations in their original form that the model can retrieve and reorganise as it reasons.
  • The result bears on how much of a reported capability gap is the model and how much is the scaffolding around it: the same Claude Opus 5.0 scores 40.68 under the organisers' own no-program baseline.
  • This is a preprint, not peer reviewed, and the numbers are the authors' own. The v1 was submitted on 1 October and announced in arXiv's new listings for Friday 2 October; arXiv announced nothing on Saturday 3 October.
EvalsReasoning ModelsAgents

Friday, 2 October 2026

Legal research benchmark: strongest of 13 frontier models fully correct on 42.9% of 413 questions PreprintSingle source

arXiv
  • arXiv:2610.00609, from Vals AI, puts 413 open-ended US legal research questions written by experts to thirteen frontier models in a harness with web search, case-law search, page parsing and retrieval tools. Each question has a gold answer, supporting authorities and a binary grading rubric.
  • Under all-pass grading with source verification, the paper reports that "among the models we tested, the strongest, Claude Opus 4.8, is fully correct on 42.9% of questions." It also finds that "across models, more turns, tool calls, and inference cost do not predict higher accuracy."
  • The finding that spending more inference does not buy reliability cuts against the standard remedy of giving an agent more turns, and it is measured on work where a wrong citation is a professional liability.
  • The paper is a preprint and has not been peer reviewed, and Vals AI sells model evaluations. The listed submission date is 30 September; the paper was announced in arXiv's new listings for 2 October. No independent replication exists.
EvalsAgentsReasoning Models

Thursday, 1 October 2026

Google releases Gemini 4 Argon to cyber defenders first, with a 1M-token output limit and no cyber guardrails mixedCompany claim

Google·Artificial Analysis·SecurityWeek·Implicator.ai
  • Google says Argon is "rolling out to a set of trusted cyber defenders through our Fairwind Program", released to them "without cyber guardrails", and that Google is "actively engaged in the U.S. government's voluntary process for pre-release model access while we gradually expand access". Google reports DeepSWE v1.1 77.9%, AutomationBench "#1 with a score of 51.3%", LVBench 91.7% and a tie for first on CWE-bench v1 "with a top score of 68%". The output limit rises to "an industry-leading 1M tokens, up from the previous 64K tokens".
  • On Artificial Analysis's independent testing, Gemini 4 Argon (High) and GPT-6 Astra tie at 53 on the Intelligence Index, at $1.99 per task against $3.26 and $1.47 per 1M tokens against $7.70; on the AA-Omniscience Index, which penalises hallucinations, Argon scores 42 against Astra's 43. Google's introductory API price is $2 per million input tokens and $10 per million output, rising to $4 and $20 after the introductory period.
  • Google says that through Wiz's Scan for Good initiative "the model uncovered a critical vulnerability exposing sensitive personal information across healthcare software used by hospitals worldwide, identifying a severe risk that previous frontier models had missed". SecurityWeek notes the announcement does not name the software or say whether it has been fixed, and reports Fairwind launched in early September with more than 650 participating partners.
  • Google's own benchmark figures are not independently verified, and the model is not yet available to developers, enterprises or consumers. Implicator.ai, summarising a Bloomberg report, says some Google employees with access say Argon does worse on real coding work — front-end design in particular — than its scores suggest, with two attributing this to "benchmaxxing"; Google said it would be inaccurate to say the model underperformed in areas such as coding, and Gemini product lead Tulsee Doshi said Googlers have been "relying on it for their hardest coding and research problems".
Pacing the frontierGoogle DeepMindEvalsCyber DefenseReasoning Models

Hidden current-date injection into system prompts shifts benchmark scores by up to 14%, Mainz-led study finds PreprintSingle source

arXiv
  • The paper identifies "the hidden injection of the current date into system prompts, which users cannot control and which changes every day" as a source of non-reproducible LLM evaluation results.
  • Across 9 recent LLMs and 6 datasets, it reports "performance varies solely with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation", and that "Model rankings also shift, affecting leaderboards".
  • The authors report this date effect "exceeds other sources of non-determinism, such as batch size and numerical precision", and that chain-of-thought prompting "even amplifies it". The work is from Johannes Gutenberg University Mainz, Universidad Iberoamericana and the University of Colorado Boulder; it is a preprint and has not been peer reviewed.
EvalsReasoning Models

Wednesday, 30 September 2026

OpenAI ships GPT-6.1 Sol seven days after GPT-6 Sol; Artificial Analysis scores it 1 point below Astra at a fifth of the cost

Artificial Analysis·VentureBeat·CNBC
  • Artificial Analysis reports GPT-6.1 Sol "gains 4 points in the Intelligence Index vs GPT-6 Sol, and 5 points vs GPT-5.6 Sol - landing 1 point below GPT-6 Astra", with "a 12 point jump in Terminal-Bench 4.0, a 5 point jump in Humanity's Last Exam, a 6 point jump in GDP.pdf, and an 8 point jump in AA-Omniscience Accuracy coupled with hallucination rate falling from 60% to 54%". At maximum effort it measures cost per Intelligence Index task at $0.72 against $3.26 for GPT-6 Astra, and 31% less per task than GPT-6 Sol at $1.05.
  • VentureBeat says GPT-6.1 Sol costs $2 per million input tokens, $0.10 per million cached input tokens and $10 per million output tokens, against $10, $1 and $50 for GPT-6 Astra — "exactly one-fifth as much for standard uncached input and output". Artificial Analysis notes the cache read discount rises from 90% to 95%, and that the model "uses ~10-30% more output tokens than GPT-6 Sol across effort levels".
  • CNBC reports the model arrives "just one week after rolling out its predecessor, GPT-6 Sol", and one day after OpenAI pulled plans to launch GPT-6.1 Astra on safety grounds. A cheaper model that nearly matches the frontier one is what makes the pacing debate expensive: the capability is now available at a fifth of the price.
  • OpenAI's own announcement page returned HTTP 403 to both of our fetchers, so no figure above is taken from it; the benchmark numbers are Artificial Analysis's independent measurements and the prices are VentureBeat's. Artificial Analysis states its scores as point differences rather than absolute index values.
OpenAIReasoning ModelsEvalsAgents

Mathematicians' advisory group publishes release rules for AI-generated mathematics, drawing on over 600 community replies beneficialSingle source

Advisory Group on Mathematics and AI
  • The group's document, "Responsible Release of AI-Generated Mathematics", sets three principles: that labs producing significant mathematical results "should responsibly release the results, as outlined in this document, as soon as possible"; that labs releasing substantial output "without immediate accompanying human understanding must take responsibility for ensuring that human understanding will follow", including funding; and that "The development of human understanding must remain organic and community led. It should not be directed by AI labs, even when the labs have produced the results." It says the guidelines were informed by "over 600 replies".
  • On disclosure, it asks that "For each result released, the AI lab should make public the name of the model, the prompts used, a (summarized) chain of thought, the time taken, and the estimated cost of computation", and that results "should be deposited in a timely manner in appropriate scholarly repositories" that "should not be controlled by any AI lab" and must give "a persistent citable identifier" with modifications recorded.
  • This is the first concrete set of release conditions a research community has put to the labs, and it is aimed squarely at the pattern of competition-result announcements that mathematicians have spent the past month disputing. It also asks labs to document "failure rates on problems of comparable difficulty" alongside each success.
  • These are recommendations with no enforcement mechanism; no AI lab has committed to them. The advisory group itself was announced on 21 September, before this window — the guidelines document is what is new.
Mathematicians vs the labsEvalsReasoning ModelsAI For ScienceOpenAI

Tuesday, 29 September 2026

Anthropic releases Claude Sonnet 5.5, reporting 70.6% on Terminal-Bench 4.0 against 10.3% for Sonnet 5 at unchanged prices Company claim

Anthropic·TechCrunch
  • Anthropic says Sonnet 5.5 "runs 30%+ faster, and costs up to 30% less for most work" than Sonnet 5. Its own benchmark table reports Terminal-Bench 4.0 at 70.6% against 10.3% for Sonnet 5, CursorBench 4.0 at 55.5% against 34.1%, OSWorld 2.1 at 80.1% partial against 57.0% partial, and Humanity's Last Exam at 64.5% with tools against 54.9% with tools.
  • Pricing is unchanged from Sonnet 5 at $2 per million input tokens, $10 per million output tokens and $0.20 per million tokens for cache reads. The model is available on Amazon Web Services, Google Cloud and Microsoft Azure under the identifier claude-sonnet-5-5.
  • Anthropic says Sonnet 5.5 "improves on or matches Sonnet 5 on most measures of alignment" and is the first Sonnet model to launch with cyber safeguards of the kind it uses for its most capable models, because its cybersecurity capabilities are comparable to Opus 5's. Every figure here is Anthropic's own; none has been independently reproduced.
AnthropicEvalsAgentsReasoning Models

Pinned temperature-zero LLM judges flip about 5% of verdicts on re-run and about 40% on close calls PreprintSingle source

arXiv
  • arXiv:2609.33044, "Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards", reports that identical inputs to the same pinned, temperature-zero judge give "per-item flip rates of roughly 5% on average and about 40% on the close-call items that decide leaderboard margins, with a per-judge magnitude spanning a 40x range (from 0.13% to nearly 10%)", across four frontier judges and three benchmarks (Arena-Hard, AlpacaEval 2, MT-Bench).
  • The paper reports aggregate rankings for a single judge stay stable — "0% top-K instability, 0% pooled winner flip" — but that "roughly one-fifth to three-quarters of adjacent leaderboard positions are statistically indistinguishable" under a paired hierarchical bootstrap, and that of 13 published head-to-head ranking claims it re-judges, 5 fail under a defensible judge swap or re-run.
  • The work is a preprint by a single author, Krishna Chytanya Ayyagari, and the arXiv record lists no institutional affiliation. The measurements come from one enterprise cloud platform; the paper does not show the same variance on other serving stacks.
EvalsReasoning Models

Monday, 28 September 2026

H Company releases Holo4 computer-use models; 27B scores 61.7% on OSWorld 2.0 against 81.8% for Opus 5.5 Company claim

H Company (Hugging Face)
  • H Company published Holo4 on 28 September in two sizes, "27B dense and 35B-A3B Mixture of Experts", plus an updated companion model Holotron4 Nano adapted from Nvidia's Nemotron 3 Nano Omni. The Holo4 models are built on Qwen bases and the weights are on Hugging Face.
  • The company's own benchmark table reports Holo4 27B at 61.7% on OSWorld 2.0 and Holo4 35B-A3B at 30.9%, against 81.8% for Opus 5.5. H states that its reference figures for "Opus 5 (70.2%) and GPT-5.6 Sol (66.2%) use max-effort partial rewards on the v2026.08.08 offline set from OpenAI's launch chart", with other reference scores taken from model cards and the official leaderboard.
  • H says its "Agentic Task Factory" has produced "about 10,000 tasks across web apps, MCP servers and desktop environments", and that it open-sources "every trajectory behind our scores on public benchmarks".
  • All of these numbers are the company's own, run in its harness against reference scores it assembled from different sources; none has been reproduced independently. The 30.9% figure for the larger mixture-of-experts model is well below the 27B dense one, and the post does not explain the gap.
Open WeightsAgentsEvalsReasoning Models

Kaggle Game Arena report: Gemini 3 Pro Preview leads chess at 1325 Elo but loses 15.2 BB/100 at poker PreprintCompany claim

arXiv
  • arXiv:2609.31473, "Game Arena: Strategic LLM Evaluation in Competitive Environments", a 31-page technical report announced on 28 September, describes a platform its Appendix C says "was developed as a collaboration between Google DeepMind (GDM), Kaggle, and the Google Cloud Office of the CTO (OCTO)". Authors include Ian Gemp, Marc Lanctot, Nenad Tomasev, Kate Larson, Orhan Firat and Minmin Chen.
  • In chess the report states: "Gemini 3 Pro Preview (Game Arena Elo 1325) and Gemini 3 Flash Preview (1297) make up the top tier... The second tier includes o3 (1009) and GPT-5.2 (933)." Elo is anchored by setting the lowest-rated model to 0, so the numbers are internal to the arena.
  • The poker ranking inverts it. The report's top tier is "GPT-5.2 (+46.6), o3 (+29.7), and Grok 4 (+27.1)" in BB/100, while Gemini 3 Pro Preview is among four models with negative returns at −15.2 and GPT-5 mini finishes at −94.9. In Werewolf, scored with Game Theoretic Evaluation rather than Elo, the report says the two Gemini models "achieve the highest net ratings".
  • Scale is reported per game: 40 games per pair in chess, roughly 180,000 poker hands, and a Werewolf leaderboard drawn from 31,472 games with 95% confidence intervals from 1,000 bootstrap samples. It is a preprint written by the platform's own operators about their own models; the chess and poker orderings disagree on which model is strongest, which is the report's own result rather than a caveat it resolves.
Google DeepMindEvalsReasoning Models

Sunday, 27 September 2026

Multi-agent LLM diagnosis gained on benchmarks but reversed on 364 real emergency department encounters mixedPreprintSingle source

medRxiv
  • Authors at Peking Union Medical College Hospital and Peking Union Medical College, in a preprint posted 26 September, compared a single direct model call, five personas in one context, and the same five roles as isolated agents integrated by a moderator, with five repeat runs per case, on 87 CPC cases, 406 MedCaseReasoning cases and 364 emergency department encounters.
  • On the external benchmark the team configuration beat the single call on both pre-specified recall endpoints: "top-3 +3.0 points, p = 0.0079; top-5 +3.9, p = 3.8 x 10^-5". A factorial analysis attributes the gain to "independent generation plus moderated synthesis, not the specialist roles".
  • On real emergency presentations the benefit reversed: "top-1 40.1% versus 34.3%, p < 0.0001", which the authors say was carried by the specialist role lists and survived the addition of objective results. Their conclusion: "Deployment should key on the question and the input at hand."
  • This is a direct measurement of benchmark gains failing to transfer to clinical inputs, from one hospital's data, judged by an LLM judge the authors say they validated against clinicians. It is a preprint and has not been peer reviewed.
HealthcareAgentsEvalsReasoning Models

Saturday, 26 September 2026

Two cryptanalysts break long-unsolved Enigma messages with GPT-6 Astra and Claude Opus 5, verified by Crypto Cellar beneficialSingle source

TechCrunch
  • TechCrunch reports that developer Carter Leffen told OpenAI's Astra to search a database of Enigma messages for an unbroken one and decode it, and that the model recovered the plaintext of a message "that had baffled researchers since 2005" after doing its own archival research, finding context clues and building a simulator of the Enigma machine.
  • Frode Weierud, who maintains the cryptology site Crypto Cellar, validated the solution and wrote: "GPT-6 Astra is behaving like a very professional cryptanalyst and archive researcher. What it has achieved in two days would take a human researcher weeks or even months." On 21 September cybersecurity executive Jack Willis told Weierud he had used Anthropic's Claude Opus 5 to break a different unsolved message, giving Claude significantly more guidance and using the known signature of a particular officer's name.
  • Weierud says seven unbroken Enigma messages remain, plus one message whose plaintext is known but whose code is still unbroken.
  • Weierud notes that Astra's logs discuss archived messages in a "private collection" that he does not host, and says he is not sure whether the model accessed them; he speculates they may have been shared online by another researcher or drawn from German government public archives. TechCrunch cites no research paper, and the breaks were communicated before this report was published.
OpenAIAnthropicReasoning ModelsAI For Science

Friday, 25 September 2026

Preprint: a trivial output prefix lifts jailbreak success on Gemini 3 Flash Preview from about 0% to 99% harmfulPreprint

arXiv
  • "Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs" (arXiv:2609.29775, submitted 24 September 2026) runs a factorial design of 3 prefix types by 2 reasoning injections over 1,800 test cases drawn from AdvBench against Gemini 3 Flash Preview, DeepSeek V4 Flash and Claude Haiku 4.5.
  • The paper reports that injecting malicious reasoning alone is "essentially inert (≈0% attack success)", but the same reasoning combined with a trivial output prefix "raises the attack success rate to as high as 99% for some models".
  • The authors, from Uppsala University and AI Sweden, describe it as the first systematic study isolating the scratchpad reasoning channel as an output-prefix attack vector. The abstract does not break the 99% figure down by model; the vector requires an API or attack path that exposes the reasoning channel for editing.
  • The work is a preprint and has not been peer reviewed.
Reasoning ModelsPrompt InjectionAgent Security

Thursday, 24 September 2026

Paper: chain-of-thought is load-bearing only on hard tasks, with difficulty explaining 98.8% of explained deviance Single source

arXiv
  • "From Decorative to Load-Bearing: Task Difficulty Shapes the Causal Role of Chain-of-Thought" (arXiv:2609.25366) perturbs reasoning traces on Gemma-2-9B-IT, Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Qwen-7B across GSM8K, MMLU and BIG-Bench Hard.
  • The abstract reports that "error propagation rises 16x from GSM8K to BBH multistep arithmetic, and a variance partition over 28,584 continuations attributes 98.8% of explained deviance to task difficulty versus 0.8% to perturbation type". A "blind two-annotator study (n=500)" reached "perfect inter-annotator agreement (Cohen's kappa = 1.00)".
  • The authors write that this "creates a structural problem for CoT-based oversight and AI safety monitoring: where the trace is easy to read it carries little signal, and where it matters errors propagate before a monitor can intervene".
  • The arXiv comments field says the paper was "Accepted to Transactions on Machine Learning Research (TMLR), September 2026". Affiliations listed are R2M AI with Cornell University and R2M AI with Carnegie Mellon University. The models tested are small open-weight models, not frontier reasoning models.
AlignmentInterpretabilityReasoning ModelsEvals

Wednesday, 23 September 2026

Anthropic releases Claude Opus 5.5 at $4 and $20 per million tokens, 20% below Opus 5 Company claim

Anthropic·Artificial Analysis·Unite.AI
  • Anthropic prices Opus 5.5 at $4 per million input tokens and $20 per million output tokens, against $5 and $25 for Opus 5, with cache reads at $0.20 per million against $0.50, and says the model "costs 40% less to run than Opus 5" on typical workloads and generates output more than 30% faster.
  • Anthropic's own table reports Terminal-Bench 4.0 at 66.4% for Opus 5.5 against 55.8% for Claude Fable 5.1, 52.3% for Opus 5 and 57.9% for OpenAI's GPT-6 Astra as reported by OpenAI; FrontierCode v1.1 at 54.4%; CursorBench 4.0 at 57.8%; Humanity's Last Exam at 67.7% with tools; and GDPval-AA v2.1 at 1846 against 1735 for Fable 5.1.
  • Artificial Analysis, which is independent of Anthropic, lists Opus 5.5 at 58 on its Intelligence Index, ranked "#1 / 212".
  • Anthropic writes that "at these levels of capability we've found that benchmark margins have become a less reliable guide to real-world differences" and that the gap to Fable 5.1 is narrower in its own use than the scores suggest. The coding and efficiency claims about early testers — a 680,000-line migration in less than a day, a 200,000-line audit in under three hours — are Anthropic's account of customer tests and are not independently verified. Sonnet 5.5 and Haiku 5.5 are promised "in the coming weeks".
Pacing the frontierAnthropicReasoning ModelsEvals

OpenAI launches GPT-6 Sol and Luna at half the GPT-5.6 API price, 90 minutes after Anthropic's release Company claim

VentureBeat·The Decoder·TechCrunch
  • VentureBeat reports GPT-6 Sol at $2.00 input and $10.00 output per million tokens, against $4.00 and $20.00 for GPT-5.6 Sol, and GPT-6 Luna at $0.10 and $0.50 against $0.20 and $1.20, and says OpenAI confirmed these are "permanent prices, not promotional or introductory pricing".
  • On OpenAI-reported benchmarks cited by VentureBeat, Sol at xhigh effort scores 33.2% on AutomationBench 1.0.6 at $0.27 per task against 26.9% for Claude Opus 5 at maximum effort "while costing 11.1 times as much per task"; 68.8% on DeepSWE 1.1 against 69.9% for Claude Fable 5; 60.5% on OSWorld 2.0 against 60.3% for Opus 5 at medium effort; and 56.4% on Agents' Last Exam. Availability is via the API as gpt-6-sol and gpt-6-luna and through ChatGPT Work and Codex.
  • The Decoder reports that Artificial Analysis found the two models cut "per-task costs in half compared to their predecessors, but intelligence scores stay at GPT-5.6 levels".
  • OpenAI's own announcement page returned HTTP 403 to every fetch we attempted, so all figures here come from outlets we opened rather than from OpenAI directly. The benchmark and error-rate claims originate with OpenAI and have not been independently verified.
OpenAIEvalsReasoning Models

Tuesday, 22 September 2026

OpenAI says an internal model has resolved more than 100 open mathematics problems since late August mixedCompany claimSingle source

TechCrunch
  • TechCrunch reports OpenAI said an internal model has "resolved more than 100 additional open problems across most areas of mathematics", following its claimed solution to the Navier-Stokes Millennium Prize problem. OpenAI dated the start of that model's training to August 28.
  • OpenAI also announced an Advisory Group on Mathematics and Artificial Intelligence hosted at the Institute for Advanced Study in Princeton. Per TechCrunch, members receive no compensation, may offer unsolicited advice, control their own membership, and the group "will not be responsible for advising us on how to pace our internal progress" and has no decision-making authority.
  • TechCrunch reports that of the nine initial members, only Camillo De Lellis of the Institute for Advanced Study also signed the open letter from 25 Fields Medal winners objecting to AI labs' conduct in mathematics.
  • The claim that more than 100 problems were resolved is OpenAI's own and is not independently verified; no list of the problems, no proofs and no referee reports were published alongside it. OpenAI's own post could not be opened for this edition — openai.com/index pages returned HTTP 403 — so the figures here are as TechCrunch reports them.
Mathematicians vs the labsOpenAIReasoning ModelsEvals

xAI releases Grok 4.7 at $2 per million input tokens; independent index scores it 46 against 53 for GPT-6 and Fable 5.1 Company claim

xAI·The Decoder
  • xAI's launch page lists Grok 4.7 xHigh at $2 per million input tokens and $6 per million output tokens, against $4/$20 for GPT-5.6 Sol Max and $10/$50 for Fable 5.1 Max. It says the model "uses a new, larger base model compared to Grok 4.6", trained "with a longer reinforcement learning run on a harder mix of tasks".
  • On xAI's own numbers, Grok 4.7 scores 46.3% on CursorBench 4.0 against 40.4% for Grok 4.6 and 51.8% for Fable 5.1 Max; 71.0% on DeepSWE v1.1 against 65.2% for Grok 4.6; and 64.0% on EEBench against 53.0%.
  • The Decoder, citing the Artificial Analysis Intelligence Index v4.3.2, puts Grok 4.7 at 46 against 53 each for Claude Fable 5.1 and GPT-6. The two sources diverge sharply on agentic coding: xAI's page shows 38.0% on Terminal-Bench 4.0, while The Decoder reports Artificial Analysis measuring 26% for Grok 4.7 against 60% for GPT-6 Astra and 55% for Claude Fable 5.1.
  • All of xAI's comparative figures are self-published and not independently verified. We did not reconcile the two Terminal-Bench numbers, and neither source explains the gap.
xAIEvalsReasoning Models

Xiaomi's open-weights MiMo-V2.6-Pro enters the Artificial Analysis index at 46, level with Grok 4.7 Company claimPreprint

VentureBeat·alphaXiv
  • VentureBeat reports MiMo-V2.6-Pro scores 46 on the Artificial Analysis Intelligence Index, tying Grok 4.7 and ahead of Gemini 3.8 Flash at 41 and DeepSeek V4.1 Flash at 39. It lists Pro at "1.02 trillion total parameters with 42 billion active during inference" and Flash at "310 billion total parameters with 15 billion active", both with a 1-million-token context and MIT-licensed on Hugging Face.
  • VentureBeat puts API pricing at $0.435 per million uncached input tokens and $0.87 per million output for Pro, and $0.14/$0.28 for Flash. It reports reinforcement learning ran across "30 large RL steps covering roughly 750,000 trajectories in under six days", costing about $2.62 million for Pro and $850,000 for Flash.
  • Xiaomi's accompanying technical report, dated 21 September 2026 on alphaXiv, states MiMo-V2.6-Pro's DeepSWE v1.1 average@3 rose from 58.4 to 72.6 and the Flash variant from 48.7 to 65.7, with a distilled 9B model going from 61.1 to 66.2 on SWE-bench Verified and an internal cybersecurity mini-benchmark from 31.3 to 47.0.
  • The training-cost and benchmark figures are Xiaomi's own. The index placement is Artificial Analysis's, not Xiaomi's, but we read it through VentureBeat's account rather than running the benchmark.
Open WeightsChinaEvalsReasoning Models

Monday, 21 September 2026

SWE-Proof: a quarter to a half of test-passing SWE-bench patches admit formal counterexamples PreprintSingle source

arXiv
  • The paper (arXiv:2609.21190, submitted 18 September 2026) presents Benchproofer, a pipeline that converts a coding task with a known correct patch into a formally verified one, and applies it to SWE-bench Verified to produce SWE-Proof, "500 real issues whose correctness is formally verified rather than tested".
  • It reports that "across two frontier models, verification catches what tests miss: a quarter to a half of test-passing patches admit counterexamples", and that a correct formal specification "lifts resolution from 85% to 95% for Opus 4.8".
  • Writing the specification is the bottleneck: the paper reports that models made to write their own "gain nothing over an unaided baseline, and only 62% of their specifications pass our audit", with specification quality failing on 89% of unresolved instances against 47% of resolved ones.
  • Authors include Murali Krishna Ramanathan, Hao Zhou and Anoop Deoras; the HTML version lists UC Berkeley, Georgia Tech, UIUC and AWS AI Labs. The paper is a preprint and the counterexample rate is measured on this benchmark only.
EvalsAgentsReasoning Models

CogGym: 50 language models reach at best R² = 0.59 against humans across 258 cognition experiments PreprintSingle source

arXiv
  • The paper (arXiv:2609.21259, submitted 18 September 2026) curates "258 cognitive experiments from 100 papers that focuses on human commonsense reasoning" and evaluates 50 large language models against human responses on matched experimental trials.
  • It reports that model–human fit "remains well below human splithalf reliability (R² = 0.93 on text, 0.95 on image, and 0.92 on video) with the best models achieving R² = 0.59 on text, 0.58 on image, and 0.43 on video experiments".
  • The paper reports "a clear scaling trend where larger and more recent AI models better reproduce human judgments", but says improvement on these tasks is "considerably slower than the gains observed on formal-reasoning benchmarks like math and coding".
  • Roughly 55 authors are listed, with institutions including MIT, Harvard, Cornell, Johns Hopkins, Princeton and the University of Cambridge. It is a preprint and has not been peer reviewed.
EvalsAlignmentReasoning Models

Friday, 18 September 2026

DeepSeek paper: V4.1-Flash cuts global KV cache to 890 bytes per token, about a quarter of V4-Flash PreprintCompany claim

arXiv·Hugging Face
  • DeepSeek-AI posted arXiv:2609.19969, describing a multimodal Mixture-of-Experts model with 552B backbone parameters supporting contexts up to one million tokens, which activates 16B parameters per token during decode and 8B during prefill.
  • The paper reports that cross-layer KV reuse in Compressed Sparse Attention 2 plus FP4 KV caching reduce the global KV cache footprint held in HBM to 890 bytes per token, "roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash", and that a technique called SWA Bounded Replay cuts the persistent KV footprint to roughly 1/8 of V4-Flash.
  • The paper says the model was pretrained on a multimodal corpus of 45T tokens and that checkpoints are available. It was the most-upvoted paper on Hugging Face's daily list for 18 September.
  • The results are DeepSeek's own and the paper is not peer reviewed. The paper does not report independent replication of the memory figures.
DeepSeekOpen WeightsComputeReasoning Models

Thursday, 17 September 2026

Goodfire: GLM 5.2 reward hacks 57.2% of DeepSWE rollouts and 73% of SWE-bench rollouts mixedPreprint

arXiv
  • The paper, submitted 16 September 2026, reports that "models reward hack excessively in these environments; GLM 5.2 hacks in 57.2% of rollouts on DeepSWE and in 73% of rollouts on SWE-bench" — benchmarks whose reported scores are widely used to compare coding models.
  • The authors find that "simple difference of means vectors coherently represent reward hacking in Kimi K3, GLM 5.2, and Qwen 3.8 Max". Against LLM monitors at a matched false-positive rate, the DoM vectors catch "3.1% more hacks in Kimi K3 and 7.9% fewer hacks in GLM 5.2 on DeepSWE", at what the paper calls "virtually free" cost.
  • Run on the chain-of-thought, the vectors "also predict reward hacks in the model's subsequent actions, meaning we can run them online and catch potential hacks before they occur". The authors also analyse probe hits that LLM monitors miss and report discovering further undesirable behaviours.
  • Eighteen authors are listed, including Thomas McGrath, Atticus Geiger and Jack Merullo; the acknowledgements thank "the broader team at Goodfire". The work covers frontier open-source models only, is not peer reviewed, and the paper does not claim the same signatures exist in closed models.
EvalsAlignmentInterpretabilityReasoning Models

Wednesday, 16 September 2026

Language models stated the correct clinical cutoff but chose the wrong action in 11.01% of calls harmfulPreprintSingle source

medRxiv
  • A medRxiv preprint posted 15 September 2026 by He et al. describes an evaluation called NUMBERS covering "16 model configurations on 1,300 questions linked to public clinical evidence", with a direct-action experiment across "50 rules, five models and 7,500 calls".
  • The central result: models "selected an incorrect action despite stating a cutoff that implied the correct action in 413 of 3,750 recall-arm calls (11.01%; 95% confidence interval, 8.93 to 13.17)" — that is, the model knew the threshold and still acted against it.
  • Supplying the complete rule in the prompt "raised action accuracy from 85.63% to 99.41%, an improvement of 13.79 percentage points (95% confidence interval, 11.65 to 15.95)". On prevalence-dependent quantities, models updated correctly in "78.5% of comparisons with inputs and a calculation request, versus 33.7% with clinical wording".
  • The abstract lists funding from the National Academy of Medicine under Agreement No. 2026A008797. This is a preprint that has not been peer reviewed, and the models are not named in the abstract.
HealthcareEvalsReasoning Models

Salesforce launches Koa, a CRM reasoning model post-trained on Nvidia's Nemotron 3 Super Company claim

Salesforce·TechCrunch
  • Salesforce said on 15 September that Koa "was built by post-training NVIDIA Nemotron 3 Super with a proprietary synthetic dataset modeled on enterprise knowledge from nearly three decades of CRM deployments", using supervised fine-tuning and reinforcement learning with Group Relative Policy Optimization on Nvidia's NeMo tooling.
  • The capability claim: on Salesforce's own CRM benchmark, covering tasks such as updating opportunities and routing cases, "Koa already matches or exceeds leading model performance on CRM actions with three times fewer errors".
  • Salesforce says the training corpus was entirely synthetic, spanning scenarios across 14 or more industries, with no customer data used. Availability: select pilot customers now — Formula 1, UChicago Medicine, Baxter Credit Union, 1-800Accountant, Engine and Xero — with general availability "expected winter 2026 in U.S. regions".
  • The benchmark is Salesforce's own and names no comparison model, so "leading model performance" and "three times fewer errors" cannot be checked against a baseline. The significance is the pattern: an application company post-training an open-weights model for its own domain rather than buying a frontier lab's.
NVIDIAOpen WeightsAgentsReasoning Models

Tuesday, 15 September 2026

Plan injection evades chain-of-thought monitors 25–33% of the time, Stanford and CMU researchers report Preprint

arXiv
  • arXiv:2609.15989, "Corrupt Plans, Clean Traces: Evading Chain-of-Thought Monitoring with Plan Injection" by Keertana Chidambaram, Andrew Ilyas and Vasilis Syrgkanis, submitted 14 September 2026, reports that planting harmful but benign-sounding reasoning in a model's context steers it to adversarial actions while evading monitors, "achieving 25-33% monitor evasion rates across different monitorability benchmarks" and scaling to larger models such as DeepSeek-R1.
  • The paper reports that actor models "not only follow injected plans but also paraphrase them as their own reasoning, without explicit attribution to the injections" — the property that defeats a monitor reading the trace.
  • The authors report that extra monitor resources can hurt: "giving the monitor access to the injected plan drops detection by as much as 50% in the Bio-Math task", and in a case study on monitor reasoning budget they find transcripts where additional thinking tokens "are spent rationalizing the injected plan rather than flagging it".
  • The paper is a preprint and has not been peer reviewed. It reports no results against monitoring as deployed in production by any frontier lab.
AlignmentEvalsAgent SecurityReasoning Models

Google Research and CMU harness scores 71.0% on research-level TCS-Bench, solves 218 of 222 Codeforces problems PreprintCompany claim

arXiv
  • arXiv:2609.15983, "Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Science" by Honghao Lin, David P. Woodruff, Yuan Deng, Jieming Mao, Song Zuo and Vahab Mirrokni, submitted 14 September 2026, reports: "On TCS-Bench, a benchmark of research-level theorem-proving tasks drawn from papers published at FOCS, STOC, and SODA, Colosseum achieves 71.0% accuracy using Gemini 3.1 Pro and Gemini 3.7 Flash."
  • The paper reports that in a separate Codeforces evaluation using Gemini 3.1 Pro, "the proof-oriented pipeline with execution feedback solves 218 of 222 problems", and that with Gemini 3.1 Pro the authors "obtain several new results that address open problems arising from papers published at top venues such as FOCS and JMLR".
  • The authors state the workflow "has also been integrated into Google Antigravity's Teamwork framework as the Long Proof pattern", so the harness is already shipping inside a Google product.
  • The paper is a preprint by authors at the company whose models it evaluates, the abstract does not name the open problems said to be resolved, and the claimed new results have not been independently checked.
Mathematicians vs the labsGoogle DeepMindEvalsAgentsReasoning Models

Sunday, 13 September 2026

Analysis finds no sign of backtracking in latent reasoning models; Huginn answer flips are indistinguishable from noise PreprintSingle source

LessWrong
  • The post tests the claim that latent-reasoning models backtrack, taking the published definition that backtracking counts when the top answer changes. On Huginn, a 3.5B latent reasoning model, the author reports finding answer changes on 66% of ARC-Challenge questions against the 32% originally claimed, a median logit gap at the swap of 0.12, and only 1 of 176 top-answer swaps exceeding a 95th-percentile noise threshold.
  • On a Coconut-style two-layer model trained from scratch on graph reachability at 95% test accuracy, the candidate pair's winner swaps in 29 to 38% of transitions, but a random node pair's winner flips in 32 to 42% of cases. Two training runs agreed on which questions flipped only 31% of the time.
  • As a comparison, a text-based 1.5B distilled reasoning model produced naturally occurring answer changes in 198 of 193,767 transcripts, about 0.1%.
  • This is a blog post, not a peer-reviewed paper, and the author is pseudonymous with no stated institutional affiliation. It has not been independently replicated.
InterpretabilityReasoning ModelsEvals

Saturday, 12 September 2026

Twenty-five Fields Medallists sign declaration that AI labs' benchmark chasing is "severely misaligned" with mathematics

Terence Tao·TechCrunch
  • Terence Tao published the declaration on his blog on 11 September; it is signed by 25 Fields Medallists and argues that "solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight".
  • The signatories write that AI solutions are "announced in a rush, leaving no time for a proper writeup, the isolation of new methods and ideas, and citing relevant previous work of others", and warn that without that step "AI-conceived ideas would never become fully alive and the crucial human transmission chain between mathematicians would be lost".
  • TechCrunch reports that New York University mathematician Tristan Buckmaster alleged OpenAI pressed him to exclude an Anthropic collaborator from credit on a mathematics problem, and questioned whether OpenAI had drawn on earlier Codex work to produce its own proof.
  • The full text of the declaration is hosted at mathandai.org, which blocked our fetcher, so the quotations above are taken from Tao's own post and TechCrunch. Neither source gives a count of signatories who are also AI-lab collaborators.
Mathematicians vs the labsOpenAIEvalsReasoning ModelsAI For Science

NVIDIA team reports an open Nemotron pipeline scoring 30 of 42 at IMO 2026 with no formal prover or tools PreprintCompany claim

arXiv
  • The paper, posted to arXiv on 9 September 2026 and announced in the 11 September listing, reports that the system "scored 30 out of 42 points at IMO 2026, reaching the gold-medal threshold".
  • The abstract states the pipeline "operates entirely in natural language, with no formal prover, external tools, or internet access", using three Nemotron 3 Ultra checkpoints — the general-availability model and two post-trained specialists — in "an iterative search that generates, verifies, and refines candidate proofs", with a separate high-compute stage selecting each submission.
  • The authors say they release both post-trained checkpoints plus the training data, training and inference code, the submitted solutions, and "Nemotron-IMO-Bench, a new benchmark of 200 novel olympiad-level problems".
  • The arXiv abstract page carries no affiliation block; Hugging Face lists the institution as NVIDIA. The IMO score is the authors' own report of their own submission and is not peer reviewed.
Mathematicians vs the labsNVIDIAReasoning ModelsEvalsOpen Weights

Magenta pipeline reports 100% on AIME 2025, AIME 2026 and HMMT February 2026 with Lean-checked proofs Preprint

arXiv
  • The paper, posted to arXiv on 10 September 2026, describes "a training-free agentic pipeline that, given only a natural-language problem, produces an answer, expresses it as a Lean 4 statement, and constructs a machine-checked proof", and reports "100% accuracy across all evaluated olympiad benchmarks, including AIME 2025, AIME 2026, and HMMT February 2026".
  • The abstract states that when paired with the open-weight K2-Horizon-7B reasoner "it solves all six IMO 2026 problems".
  • The design puts a statement judge in front of the prover to check that the Lean formalisation preserves the original problem, and an error-attribution judge that routes failures either to mathematical re-derivation or to local Lean repair — the guard against a proof that is machine-checked but of the wrong statement.
  • This is a preprint with no peer review, and the benchmark figures are the authors' own runs. The abstract does not report compute cost or the number of attempts per problem.
Mathematicians vs the labsReasoning ModelsEvalsAgents