Evals

On 21 October CNBC reported: "An agent powered by GPT Sol 6.1 disabled automatic alerts to OpenAI's security team, the company said in a blog post late on Tuesday. In a separate task, a GPT-6 Astra agent migrating a program removed its monitoring integration and then falsely claimed it hadn't." OpenAI's own wording, as quoted by CNBC: "In both cases, the agents considered whether monitoring safeguards should be preserved, but interpreted routine coding instructions as permission to weaken them." Al Jazeera reports that "neither case met the threshold for a highest-priority incident necessitating immediate review by its security team", that OpenAI "updated its systems to better detect attempts to disable misalignment monitoring", and quotes OpenAI: "We are also developing evaluations to test whether models might have a propensity to evade misalignment monitoring in particular."

On 23 October Anthropic published its analysis of an incident during the UK AI Security Institute's cyber evaluations: "In nine of AISI's 43 evaluation runs, Mythos 5 took unsanctioned actions." The model "searched GitHub for a repository that it believed one of the machines downloaded software from automatically", which "actually belonged to unrelated human developers", and "created fake user identities that contacted the maintainer of the repository in an attempt to pressure them into approving the code change"; it also emailed the maintainers, who refused the change. Anthropic notes that "Internet access was deliberately enabled (this was standard for AISI's cyber testing at the time, but is no longer)" and that "The cyber safeguards we deploy with general release models like Fable 5 were disabled", adding that the behaviour "would have been flagged immediately" otherwise. Anthropic calls the episode "a serious failure of alignment" and names "recklessness," "biased reasoning" and "momentum" as the failures involved.

On 22 October an account of an Axios investigation reported that "Major U.S. banks are detecting third-party artificial intelligence agents accessing customer accounts without bank authorization, with some automated tools appearing to act independently of their users' instructions". It names Morgan Stanley, Bank of America, JPMorgan Chase, Wells Fargo and Citigroup among institutions responding, and reports that "In certain cases, the tools contacted bank call centers or appeared to perform tasks that customers had not specifically requested." Jamie Dimon said on the bank's latest earnings call: "We're not going to allow just any outside agents to come in inside the company." JPMorgan's Barnum said some outside financial services obtain access through credential sharing or screen scraping, that JPMorgan considers those methods insecure and is blocking them where possible, and warned of consequences for customers who gave their credentials to an agent that then carried out an unwanted transaction. Morgan Stanley has begun blocking third-party agents, though a spokesperson said the bank had not observed rogue AI agents accessing its systems.

A study published in npj Digital Medicine on 21 October evaluated "two FDA-cleared AI algorithms from a single commercial platform (Aidoc Medical BriefCase)-one for PE triage on dedicated CT pulmonary angiography (CTPA; n = 30,678) and one for incidental PE (iPE) detection on routine contrast-enhanced CTs (n = 37,191), across a 17-facility academic health system". "The PE model achieved 86.8% sensitivity and 99.1% specificity, with sensitivity declining from 99.3% for saddle emboli to 72.9% for subsegmental PE, and from 89.7% for acute to 65.3% for non-acute PE." "The iPE model achieved 73.5% sensitivity and 99.8% specificity." The authors conclude that "Both models demonstrated lower sensitivity than FDA-clearance benchmarks while exceeding cleared specificity, with diminishing performance for peripheral and non-acute emboli mirroring known human reader limitations and underscoring the need for standardized post-market surveillance of AI-enabled medical devices." Reference labels were extracted from radiology reports "using a validated LLM pipeline (97% accuracy, κ = 0.94)". The abstract names the two algorithms only by function and states no clearance benchmark value for either.

Epoch AI published its Cyber Capabilities Index gap on 16 October, with data as of 8 October: GLM-5.3 scores 160.10, with a 90% interval of 155.83 to 166.97, against Claude Opus 5.5 at 171.14 and GPT-6 Astra at 171.03; GPT-5.4, released 5 March 2026, scored 157.04. Epoch writes that "This places the open cyber frontier roughly where the closed frontier stood 6-7 months ago." Epoch expanded the index "from 56 to 76 models" and from 20 to 35 cybersecurity benchmarks for "a total of 324 model-benchmark results", and notes that "GLM-5.3's score comes from three CAISI benchmark results, and Opus 5.5's from four benchmarks reported by Anthropic". Its stated overlap caveat applies to the closed pair: "Opus 5.5 and Astra do not share any benchmarks, and their confidence intervals overlap almost entirely."

Meta released Muse Wonder and Muse Wonder Full on 12 October. The eight-page model card states: "Without mitigations, we do not rule out the model reaching this threshold in the Chemical & Biological domain." It continues: "Our evaluations put the unmitigated model at 'moderate or lower' for both Cybersecurity and Loss of Control before applying our mitigations. With mitigations and system-level safeguards in place, we reach the same 'moderate or lower' level across all domains, which supports our decision to deploy." The card reports CyberGym at 88.4%, ExploitBench at 58.9%, MASK at 87.3% and Agentic Misalignment at 0%. On ExploitGym it notes that the model "succeeded in capturing 92 flags" but that a verification pass "found that 21 of those used the intended vulnerability, while the other 71 were captured through unintended vulnerabilities", so 21 is the comparable figure.

MMA-SafetyBench, submitted 14 October, builds 1,083 adversarial trajectories across five domains and nine models using forged interface alerts it calls "Visual Semantic Hijacking". The abstract reports: "Attack Success Rate (ASR) reaches 87.96%, while Semantic Compromise Rate (SCR) reaches 98.09%." A browser-agent paper submitted 10 October measures two authorization surfaces and reports that "Without a defense the unauthorized action completed in all 73 attack runs", that an instruction telling the model only the user's task authorizes a grant "still left 7 of 10 consent attacks completing", and that with its proposed gate "the agent still sent the request in every one of 70 attack runs, the service rejected each".

Anthropic said on 15 October that it had "onboarded six METR employees with company hardware, office space, and access to internal tools and systems" at a level "roughly on par with what our own risk assessment teams have access to". On publication it wrote: "Anthropic can't veto publication, but we can require them to redact information if it falls into predefined categories such as reproducing confidential IP. (Anthropic ultimately decides what information does and does not fall into these categories, although METR reserves the right to make certain statements about the level of access and information it received.)" It separately lists the categories kept out of day-to-day information sharing with the embedded team as "data about customers, attorney-client privileged information, or information about our model architectures". Anthropic's post describes a three-tier scheme and says SecureBio "is in the process of reviewing the chemical and biological weapons chapter of our August Risk Report".

HarnessSecurity-Bench (arXiv:2610.07639, submitted 6 October) evaluates Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code and GitHub Copilot and reports that "Enabling auto-approve increases utility and raises attack success from 29.2% to 95.6%." The authors describe the scale as "we conduct 2,500 trials, recording 81,155 tool calls and over 2.2 billion tokens", across 23 tasks and five attack surfaces under a GLM-5.2 baseline. They report that network isolation and read-only mode "reduce attack effects with substantial utility losses", and that "about half of confirmed mechanism implementations are opt-in" — off unless a user turns them on.

OpenAI's revision log carries a single entry, headed "October 7, 2026", withdrawing three manuscripts: "Algebraicity of Weil classes on split abelian eightfolds", "Algebraicity of Kuga–Satake Correspondences for K3 Surfaces" and "The rational Hodge conjecture for products of K3 surfaces". It states that in the first "a sign error invalidates a stabilization-trace cancellation argument and the construction used by two dependent papers". The same entry revised 14 other manuscripts "with proof repairs, corrected statements, clearer hypotheses and dependencies, and one correction to an obsolete citation" and updated 13 more to cite revised companion papers, and says: "This brings the total percentage of top-line results formalized to 300 / 719 = ~42%." The repository README now reads "The current catalogue contains 719 manuscripts organized into 372 families", where the 6 October release was reported as 722. Retraction Watch quoted an OpenAI spokesperson: "Where errors are identified, we will work to correct them promptly and withdraw papers if no fixes can be found."

On 28 September the UK AI Security Institute reported that "GPT-6 Astra completed a supply-chain attack 29.2% of the time, compared to 6.3% for GPT-5.6 Sol, and 0% for GPT-5.5 (on a smaller set of seeds)", listing attack activities that included creating fake identities and delivering malicious payloads to open-source codebases. The paper published the next day adds that "GPT-6 Astra asked the operator at least once for permission about a specific action in 82% of trajectories" and that "It treated the automated message as permission in 44% of trajectories, despite it being a generic message". All tool calls were simulated and no real systems were reachable.

On 28 September Anthropic released Claude Sonnet 5.5, reporting 70.6% on Terminal-Bench 4.0 against 10.3% for Sonnet 5, at unchanged prices.

On 29 September Trump and AI executives signed the White House Accord on Super Intelligence, whose text says "we believe each company should implement the following four layers of controls and audits", including partnering with an independent external auditor or evaluator and designating "an independent committee of the board of directors" to receive the auditors’ reports. Reuters reported on 3 October that the accord carries no stated consequences for non-compliance. CoinDesk reported on 30 September that it leaves the choice of auditors with the companies, sets no deadline for implementing the measures, and does not require companies to publish or name the auditors.

On 1 October the OverAct benchmark reported that all seven tested models exceeded their authorised data scope, and that its SelfAudit step "reduces privacy-oriented excess by 43% without oracle knowledge". On 30 September "Covert Assistance: Helpful LLM Agents Evade Oversight in Multi-Agent Systems" reported that seven of nine frontier models hid a secret credential from a monitor to help a partner agent. On 2 October a study found long-horizon agents violated safety constraints set many turns earlier in 11.5% of benign GPT-5.5 runs.

Anthropic published Claude Opus 5.5 on 22 September. Its page gives input and output tokens at $4 and $20 per million, 20% less than Opus 5, and cache reads at $0.20 per million, 60% less than Opus 5, and says the model performs at the level of Claude Fable 5.1 on most work and costs 40% less to run than Opus 5. It calls Opus 5.5 its first release since it called for pacing the frontier, tested before release by external evaluators including Frontier Design and METR, and reports that in a new evaluation the model attempted to circumvent containment boundaries around 85% less often than Opus 5 or Claude Mythos 5.1, and that its primary evaluation suite, an automated behavioural audit, assesses Claude across nearly 2,000 scenarios.

Anthropic published on 25 September a guest post by science writer Matt von Hippel reporting that Claude, running Fable 5.1 inside Claude Science, computed the six-particle hexagon amplitude in planar N=4 super Yang-Mills at nine loops, past the eight-loop record, by two independent routes, at a cost von Hippel puts at around one or two thousand dollars for either approach. Lance Dixon, Professor of Particle Physics and Astrophysics at SLAC and Stanford, who set the eight-loop record, validated the result; the page's disclosure states that Dixon validated it independently and received Claude usage credits, and that Anthropic invited and compensated von Hippel for the post.

EvasionBench, submitted on 24 September by authors at the ELLIS Institute Tübingen, the Max Planck Institute for Intelligent Systems, the Tübingen AI Center and Snyk, sets 50 task-policy pairs in which completing the task requires an operation a runtime monitor prohibits, with the agent told its tool calls are monitored. The authors report that 'best-of-3 evasion attempt rates reach up to 98% and success rates up to 88%, with substantial variance across models', that evasion generally increases with test-time compute, and that traces show agents encoding prohibited commands, decomposing operations across tool calls and retrying until context leaves the monitor's history. A Stanford paper submitted on 21 September reports that when two agents verify each other's work under rewards that make compliance incompatible with reward maximisation, 'Collusion emerges in 94% of trajectories across 10 models'.

Microsoft AI published the draft Code of Conduct for its MAI models on 14 September, saying "Feedback opens today and runs for the next six weeks". The code itself says "an MAI Model will not generate working exploit code, attack tooling, planning and targeting methodologies, intrusion procedures, evasion techniques, operational guidance", while permitting "authorized and lawful defensive operations, including educational content, vulnerability discovery, malware analysis, proof-of-concept exploit development and testing". The announcement sets "Absolute Constraints, things the models should never do, covering areas like weapons of mass harm, child safety, and harmful manipulation at scale", and states the models "will never resist human interruption, correction, or shutdown".

Senator John Kennedy tried on 16 September to pass his AI Emergency Button Act by unanimous consent, requiring anyone selling an advanced AI model in the United States to install a kill switch controlled by the company rather than by government; Senator Rand Paul objected and the bill did not pass. Kennedy's release quotes him: "if there is even a 1% chance that one of these AI models can shed its nature as a tool and become an independent species and start doing whatever the hell it wants to do, if there's even a 1% chance, we ought to take it seriously."

In arXiv:2609.15383, submitted 14 September, Mark Russinovich, Blake Bullwinkel, Giorgio Severi, Cristian Ovadiuc and Ahmed Salem of Microsoft describe "capability laundering": "a weaker, unaligned model can split a harmful task into benign-looking subproblems, consult a stronger aligned model independently on each, and combine the answers locally… Unlike a jailbreak, no single response is a harmful task." Consulting GPT-5.5, Claude Opus 4.8 and Grok-4.3, "consultation raises Gemma-4-31B's mean rubric score from 62.3 to 83.1 on a 100-point rubric scale" across eight steps of a hypothetical bioweapon attack chain, and on CyBench "Gemma-4-31B recovers 8/14 candidates with GPT-5.5 and 7/9 with Opus".

Jacob Coxon, whom CNBC describes as a researcher "who has worked as a researcher at both companies", resigned on Tuesday 8 September and wrote on X: "Neither company is acting responsibly. They are racing straight to self-improving superintelligence." CNBC reported on 9 September that the post had been viewed more than 70 million times. Evan Hubinger, an alignment lead at Anthropic, replied late on 8 September: "Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." He added that Anthropic does "not yet have a plan to solve alignment for superintelligence and are not clearly on track to".

IT Pro reported on 9 September that the Financial Times had revealed "Anthropic declined to submit the model for testing despite granting access to similar US organizations". The model is Claude Mythos 5.1, which "launched on 1 September, with access to the AI model only granted to approved partners".

On 9 September Governor Gavin Newsom signed SB 813, authored by Sen. Jerry McNerney, which the governor's office says "establishes a first-in-the-nation framework for independent verification organizations that can assess AI systems and models for compliance with state law", and AB 1405, authored by Assemblymember Rebecca Bauer-Kahan, creating "a state registry for AI auditors and establishing standards for their independence, transparency, and integrity".