Evals
8 items across 2 editions · appeared in the last 2 editions in a row. First seen Fri 11 Sep, last seen Sat 12 Sep.
- Terence Tao published the declaration on his blog on 11 September; it is signed by 25 Fields Medallists and argues that "solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight".
- The signatories write that AI solutions are "announced in a rush, leaving no time for a proper writeup, the isolation of new methods and ideas, and citing relevant previous work of others", and warn that without that step "AI-conceived ideas would never become fully alive and the crucial human transmission chain between mathematicians would be lost".
- TechCrunch reports that New York University mathematician Tristan Buckmaster alleged OpenAI pressed him to exclude an Anthropic collaborator from credit on a mathematics problem, and questioned whether OpenAI had drawn on earlier Codex work to produce its own proof.
- The full text of the declaration is hosted at mathandai.org, which blocked our fetcher, so the quotations above are taken from Tao's own post and TechCrunch. Neither source gives a count of signatories who are also AI-lab collaborators.
- The paper, posted to arXiv on 9 September 2026 and announced in the 11 September listing, reports that the system "scored 30 out of 42 points at IMO 2026, reaching the gold-medal threshold".
- The abstract states the pipeline "operates entirely in natural language, with no formal prover, external tools, or internet access", using three Nemotron 3 Ultra checkpoints — the general-availability model and two post-trained specialists — in "an iterative search that generates, verifies, and refines candidate proofs", with a separate high-compute stage selecting each submission.
- The authors say they release both post-trained checkpoints plus the training data, training and inference code, the submitted solutions, and "Nemotron-IMO-Bench, a new benchmark of 200 novel olympiad-level problems".
- The arXiv abstract page carries no affiliation block; Hugging Face lists the institution as NVIDIA. The IMO score is the authors' own report of their own submission and is not peer reviewed.
- The paper, posted to arXiv on 10 September 2026, describes "a training-free agentic pipeline that, given only a natural-language problem, produces an answer, expresses it as a Lean 4 statement, and constructs a machine-checked proof", and reports "100% accuracy across all evaluated olympiad benchmarks, including AIME 2025, AIME 2026, and HMMT February 2026".
- The abstract states that when paired with the open-weight K2-Horizon-7B reasoner "it solves all six IMO 2026 problems".
- The design puts a statement judge in front of the prover to check that the Lean formalisation preserves the original problem, and an error-attribution judge that routes failures either to mathematical re-derivation or to local Lean repair — the guard against a proof that is machine-checked but of the wrong statement.
- This is a preprint with no peer review, and the benchmark figures are the authors' own runs. The abstract does not report compute cost or the number of attempts per problem.
- The Agent Incident Registry, posted to arXiv on 10 September 2026, catalogues "487 records of agent-related events disclosed from 2022 through 2026" with labels for causal role, disclosure class, mechanism and outcome; in the primary population, "81 of 336 records have realized harm (24%; 95% Wilson interval 20–29%)".
- The five authors are all affiliated with Anaconda.
- The authors are unusually direct about what the numbers cannot do: "AIR samples public disclosure, not deployed systems or agent runs", and therefore "no count in this paper estimates incidence, prevalence, vendor risk, or control efficacy".
- They also report that "source dependence dominates precision", with the realised-harm proportion moving between 23% and 31% when dominant source blocks are removed. Preprint, not peer reviewed.
- Reuters reported on 11 September, updated 6:10 p.m., that Senate negotiators are considering legislation creating a duty of care requiring AI companies to "design their products with the goal of preventing 'catastrophic risks'", with the federal government able to block release of models deemed unsafe and companies able to challenge that in federal court.
- The measure would also "block states from enforcing their own laws governing certain risks posed by AI models" — federal preemption that would cut across the state statutes signed this month, including California's package of chatbot child-safety bills.
- Nextgov reported at 4:49 p.m. ET that the negotiators are split on who does the testing: the Cruz–Klobuchar–Thune approach has companies run their own safety tests and submit results to the Commerce Secretary for deployment approval, which a Democratic aide characterised as "primarily a voluntary standard type situation", while Senator Maria Cantwell wants models tested by "scientists and experts at our national laboratories".
- Nothing has been introduced, and a Commerce markup planned before the August recess was cancelled. Reuters notes the House is in session for one week before the 3 November midterms and the Senate for three, so the calendar is the binding constraint rather than the drafting.
- "The Missing Boundary: How Autonomous Agents Lose Control" (arXiv 2609.11024, submitted 10 September) tests five agent models across 16 operational domains and 1,800 trajectories. Neither a degraded control boundary nor an executable unsafe opportunity alone produced substantial loss of control; together they produced a 55% loss-of-control rate, and 62% across ten further domains.
- The paper reports that restoring the original control boundary drops the rate to 0% even when the unsafe action remains executable, and that context compaction is not itself the problem: preserving control constraints through compaction yields 0%, while omitting them raises the rate to 87%.
- This is a preprint and has not been peer reviewed. The environment is deterministic and multi-turn rather than a live deployment, so the absolute rates should be read as a controlled measurement, not an incident frequency. The practical claim — that constraints must survive context compaction — is testable by anyone running long-horizon agents.
- "Big Enough to Break Out" (arXiv 2609.10780, submitted 9 September) compares two PentestGPT-based systems: a legacy human-in-the-loop system on open-weight Kimi K2.5, and a newer autonomous system on Claude Opus 4.8. Across three public targets the autonomous system solves all three, including the two the legacy system never finishes.
- The authors flag the legacy result as the more surprising one: even on machines it fails to solve, it completes about half the subtasks while running on ordinary university GPUs with no provider guardrails — a capability floor available to anyone with open weights and campus hardware.
- The paper explicitly declines to attribute the gain, since model, harness, autonomy and memory architecture all changed together. Adding a coverage-memory layer to both systems improved neither, and in reviewable stalled runs the limiting factor looked like planning and commitment rather than lost memory. Three targets is a very small sample.
- Published 10 September, the evaluation measures intelligence targeting and conventional weapons capability across Claude Mythos Preview, Mythos 5, Opus 5 and Sonnet 5, plus open-weights Kimi K3 and GLM 5.2. On 6,000 YFCC100M Flickr images, Mythos Preview reached a 37.0 km median error with 23.7% of images placed within 1 km, against 181 km for Opus 5, 384 km for Sonnet 5 and 385 km for Kimi K3; Anthropic compares this to 151 km for top GeoGuessr players.
- On text geolocation from anonymised GeoText tweets covering 1,697 users, median error ranged from 20.1 km (Mythos Preview) to 31.3 km (Sonnet 5), and 135 users — 8% of the corpus — were reliably placed within 1 km by at least one model. On account linkage across synthetic social media, Mythos Preview processed median 37,000-word samples in about 11 minutes, against roughly 2.5 hours for human analysts.
- On simulated drone terminal guidance against a parked high-visibility vehicle, Opus 5 struck the target on 80% of runs, Mythos Preview 70%, Mythos 5 53%, Kimi K3 15% and Sonnet 5 5%. Across all nine difficulty settings Opus 5 hit on 20% of 540 launches. Under GPS denial, only Opus 5 kept about a third of flights inside five metres.
- Anthropic frames these as capability ceilings for isolated models and notes human teams with internet access would likely do better. The drone work is in simulation, not flight, and the report does not disclose what mitigations follow. The open-weights results matter most: Kimi K3 trails the frontier but is not far behind on photo geolocation, and cannot be withdrawn.