Scaling
28 items across 22 editions. First seen Sat 12 Sep, last seen Thu 29 Oct.
- The paper says that "by systematically varying SynthAPI's generation controls in 458,752 tasks across 849 trained models of four different sizes" the authors derive a scaling law predicting downstream tool-use generalisation from the scale and composition of the synthesised environments.
- The finding is about what to buy with a fixed data budget: "as data budgets grow, scaling tool and database diversity becomes more effective than increasing the number of tasks", and the resulting agents outperform models trained on data from prior work "across four evaluation benchmarks".
- Authors are Andrew Szot, Samira Abnar, Omar Attia, Abbas Kazerouni, Michael Kirchhof, Cheng-Yu Hsieh, Raviteja Vemulapalli and Alexander Toshev; the arXiv full-text version gives the affiliation as Apple.
- The abstract reports no baseline benchmark scores, so the size of the gain over prior work cannot be read off it. Submitted 27 October, announced in the 28 October listing. Not peer reviewed.
- In "Identically Trained Language Models Break on Different Jailbreaks" (arXiv:2610.34074), authors at Cerulean Labs, National Taiwan University and National Yang Ming Chiao Tung University trained, for each of 25 pretraining data recipes, 1B base models from 3 pretraining seeds, each given the same safety post-training with 2 post-training seeds — 150 models in all — and sampled 20 responses per model on a shared set of jailbreak prompts.
- The paper reports two post-trained copies of the same pretraining run have per-prompt non-refusal rates correlating at 0.984 on average; a different pretraining seed lowers this to 0.811, and a different data recipe lowers it only slightly further, to 0.779. The pattern holds in all 25 data recipes and at 150M and 300M parameters.
- The best of 91 attack templates found on one checkpoint stays the best after a new pretraining seed in only 27% of model pairs, against 67% after a new post-training seed. Across 3,211 prompts, a model's per-prompt non-refusal rates identify its pretraining run among all 75 runs every time, and a random subset of 50 prompts still does so 98% of the time.
- If the result generalises to frontier scale, a red-team result on one checkpoint would transfer poorly to a model retrained from a new seed. The paper is a preprint at 1B parameters and below, and is the only source for these figures.
- The paper proposes "a rewriting scaling law that predicts loss from model size, training tokens, the fraction of training tokens that are rewrites, and the number of epochs over the natural data, charging the compute spent generating rewrites". In the compute-limited setting it finds "the estimated optimal rewrite fraction increases with compute, from under 10% at 3x10^18 FLOPs to 32% at 10^22 FLOPs" (arXiv:2610.31867).
- "Our scaling forecast extrapolates 33x in FLOPs to train a 22B-parameter model, whose final loss is within 1.1% of our prediction. This model achieves a better BPB than its rewriter on several web domains at one-tenth the rewriter's training compute."
- In the data-limited setting, where the rewriter is the same size as the target and pretrained on the same seed set, "a model trained on rewrites of its own pretraining data, generated by a rewriter of the same size, beats the best run on natural data alone at every model size up to 3B". Authors are at Princeton, Stanford and Jane Street, with the work done during an internship at Jane Street. Preprint.
- In arXiv:2610.30678, "Scaling Laws for Safety Probes", Yuchen Wu of the University of Washington and Jerry Wei of Anthropic construct six new safety datasets and train two probe architectures on 20 base models from five model families, with 32 to 10^6 labels for each model, measuring probe error as 1 − AUROC.
- They report two laws: "the lowest probe error a model can reach falls as a power law in its size", and probe error falls towards that floor as examples increase, with the slope increasing as the target model size increases. Both hold "across tasks, model families and probe architectures, and only their fitted constants differ".
- Fitted on the smallest models of a family, the laws "predict the error of the next larger model to within 0.02 in 72% of cases", identify whether more labels or a larger model is the cheaper route to a target error "in 73% of cases", and when they pick the worse probe architecture do so at "an extra error of only 0.006 on average".
- A preprint, submitted 22 October and not peer reviewed. The authors note existing safety datasets were insufficient for the exercise, so the laws are fitted on datasets they built themselves; the paper does not claim the constants transfer to other harm areas.
- TechCrunch reports Flint is a mixture-of-experts model that uses "just 2.5 billion of its 40 billion settings, known as parameters, at a time," downloadable free from Hugging Face under the Apache 2.0 licence. Recursive says its automated research system, the "Eureka Machine," "helped with nearly every step of Flint's development," letting a small team build it from scratch in under four months.
- In Recursive's own tests, reported by TechCrunch, Flint "generated about 131 tokens… per second on a MacBook Pro with Apple's M4 Pro chip — roughly twice the 58 to 69 tokens per second that similar models managed on the same machine, including Google's Gemma 4 and Nvidia's Nemotron 3.5 Lightning," and topped 150 tokens per second on Nvidia's RTX Spark. Recursive says Eureka contributed "a sparser model shape, low-bit training that shrank Flint by 71% with almost no loss in quality and custom kernels."
- TechCrunch reports that on SWE-Bench Verified "Flint outperformed other models under 50 billion parameters," and that across Recursive's own average of six tests only Alibaba's Qwen3.8 27B scored higher.
- Every benchmark figure here is the company's, run on its own harness, and TechCrunch is the only outlet with it. Recursive has not published the Eureka Machine's contribution in a form anyone else can audit, so "largely built by AI" rests on its own account.
- TechCrunch reports the year-old startup "announced it had raised a $1 billion Series A round, co-led by Index and Lux, reportedly at a $6 billion valuation," with Sequoia, GV and Kleiner Perkins participating and Valor, Atreides, A* and Nvidia also investors.
- The pace is the story: "The round comes just nine months after the company launched with a $180 million seed round from GV, Sequoia, and Index. That seed round had valued the company at $1.5 billion, so its valuation just quadrupled." Founders are Aidan Smith and brothers Asher and Ben Spector, and the stated mission is "to find ways to train large models with less data."
- The founders describe their work as "the world's most complete megakernel framework. It can perform entire training runs, serve inference, play video games, and even do all three simultaneously in a single kernel," and claim it "can overlap memory-bound inference with compute-bound training."
- The $6 billion valuation is reported rather than confirmed by the company, and the megakernel claims are the founders' own with no published benchmarks or external evaluation. TechCrunch does not report revenue, customers or model releases.
- PAR (arXiv:2610.27628, submitted 20 October) trains the prediction at iteration k of a looped language model to match the state after the first k operations, and rescales recurrent states to their first-iteration norms. On swap-S5, a Qwen2.5-0.5B model trained on at most 32 moves "achieves 95.5% mean full-state accuracy at 512 moves, or 16x the training length, compared with 55.7% reported for RLT and 0.8% for explicit chain of thought on the same backbone."
- The ablations matter as much as the headline: the paper reports that "removing norm anchoring reduces the extrapolation factor from 16x to below 2.2x, and replacing progress-aligned targets with final-answer supervision reduces it to at most 8x."
- The paper's affiliation footnotes list Ant International and the National University of Singapore. It ranked first on Hugging Face's Daily Papers listing for 21 October.
- This is a preprint and not peer reviewed, and the result is on a synthetic permutation state-tracking task on a 0.5B model, not on a frontier-scale system.
- Inception says Mercury 3 "outperforms GPT-6 Luna and Claude Haiku 5.5, while running 4–8x faster at 1,000+ tokens per second", and lists its speed as 1,068 tokens per second on NVIDIA GPUs, a 1M-token context up from 260K for Mercury 2.5, and pricing of $0.20 per million input and $0.70 per million output tokens, halved at launch.
- The company says the model is more than 40x larger than Mercury 1 and that "intelligence across a range of standard benchmarks has risen 3x from Mercury 1 (mini) to Mercury 3" — its argument that diffusion language models scale the way autoregressive ones do.
- On the one agent number it gives, Inception says it ran Mercury 3 in its production web-agent harness on 50 live Mind2Web navigation tasks and completed 50% of them, "significantly more than GPT-6.1-Sol".
- All of these figures are Inception's. The post gives no independent evaluation, and the Mind2Web result comes from the company's own harness rather than a public leaderboard.
- arXiv:2610.25736, from 42 authors at Tsinghua University, AgiBot and Infinigence AI, introduces collective experience learning: deployed robots share execution experience to improve a common policy, using failure contexts to retrieve successful trajectory segments from the fleet "without collecting additional expert demonstrations or corrective actions".
- On multi-stage phone packaging, the paper reports success rising from 79.2% at initialization to 86.7% with 16 robots and 98.3% with 96 robots, with collection time per robot held fixed.
- Across five real-world manipulation tasks the resulting policies reach success rates of 93.3–99.2%, measured over 120 trials per task on robots excluded from training data collection.
- This is a preprint. The result is measured on the authors' own fleet and task set; there is no independent replication, and the tasks are factory-style manipulation rather than open-world work.
- Reflection AI released Beam's weights on 16 October: a mixture-of-experts model with 501B parameters and 23B active per token, a 1M-token context window and an Apache 2.0 licence, in BF16 (about 1 TB), FP8 and NVFP4 (about 280 GB) checkpoints.
- The scores Reflection published for the BF16 checkpoint are SWE-Bench Verified 81.6%, SWE-Bench Pro v2-Hard 76.5%, Terminal Bench v2.1 80.1%, AIME 2026 97.7%, GPQA Diamond 90.6%, LiveCodeBench 88.3% and Humanity's Last Exam 36.0% without tools and 51.1% with tools.
- SiliconANGLE reports the model was introduced last week and that this release adds training recipes, serving-engine configurations, fused kernels and a drafter model for speculative decoding. It says Reflection trained Beam over two months using several thousand high-end graphics cards and credits custom reliability algorithms with practically zero cluster downtime.
- The benchmark table is presented as provided by the Reflection team, and no independent evaluation of the released weights has appeared. Reflection has not set pricing, and the Hugging Face post states no comparison of token use against other models.
- Microsoft AI describes MAI-Base-2 as "an MoE foundation model with 80 billion active and 2.8 trillion total parameters", pre-trained on "16k GB200 GPUs" over 38 days in August and September 2026 on a "50T-token training mixture", of which vision-language data is 18%.
- Microsoft says the recipe is "4-12x more pre-training FLOPs-efficient than the strongest open base model, DeepSeek-V4-Pro-Base", giving "4.2-12.3x" in the body text and "6.3-12.7x" against its own MAI-Base-1; it reports training metrics of 21% average MFU and 89% goodput.
- The post gives no numeric benchmark scores. It says the model "achieves lower BPB than the other models we tested" on held-out evaluations and "outperforms every open-weight model we tested", without publishing the underlying figures.
- Microsoft says MAI-Base-2 is "currently undergoing post-training" and that the pipeline should let it license model weights to trusted partners through Foundry. No partners, release date or pricing are named, and none of the efficiency claims is independently verified.
- The preprint compares ten representative self-distillation methods against GRPO on ALFWorld, ScienceWorld, Search-QA and WebShop across three model sizes, in 132 training runs totalling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and all eleven trained methods.
- The findings as the authors write them: "gains over GRPO remain limited in scope, with no method consistently outperforming it across settings"; "strong final performance can mask unstable training"; and "additional training cost does not consistently translate into better performance".
- Their stated motivation is that existing evaluations reuse published baseline scores from differing experimental conditions and give limited attention to training stability and cost. The paper is arXiv:2610.19789, submitted 14 October, not peer reviewed; the abstract page lists no affiliations, and the Hugging Face paper page shows Zhejiang University as the submitting organisation.
- "Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models" (arXiv:2610.17590), by six authors listing Cornell University, Cornell Tech and Amazon, studies "ten model variants, about 1,700 models in total" and reports that scoring with the exact likelihood reduces the compute multiple to 12.8x.
- The paper reports that block diffusion training and an encoder-decoder architecture further reduce it to 7.8x at block size 32, 4.6x at block size 8 and 3.0x at block size 4, and concludes the "16x figure substantially overstates the pretraining cost gap between DLMs and ARMs".
- The 16x number has been the standard reason given for not pretraining diffusion language models at scale; a 3.0x gap is a different economic argument.
- This is a preprint and has not been peer reviewed. The figures come from a sweep of small models, not frontier-scale training runs, and the paper does not claim equivalent downstream quality at those block sizes.
- Epoch AI's report, dated 7 October, asks whether AI can automate AI R&D and answers "No". Models had to devise a post-training method beating a strong GRPO baseline on Qwen3-8B, graded against the published on-policy self-distillation method: GRPO is 0% and matching SDPO is 100%. Budgets were 3,000 GPU-hours, at most 50 GPUs, and 10 billion inference tokens per evaluation.
- Epoch reports GPT-5.6 Sol reached about 35% of SDPO's gains on a generous scope reading and about 15% counting only in-scope changes, spending its full GPU budget of about $14,000 plus $2,100 in tokens. Claude Fable 5.1 scored 40%, which Epoch says came mostly from hyperparameter tuning. Claude Fable 5's gains were removed because they came from submitting many similar runs and picking the best, which Epoch describes as farming seed noise.
- Epoch concludes the models "did not discover anything comparable to the original innovation" and that AI "struggles at end-to-end AI algorithms R&D… for now". It says some write-ups were misleading because they did not disclose multi-run selection or prior work.
- This is a single benchmark built around one specific innovation, and Epoch's scope judgements decide most of the scores — the gap between 35% and 15% for the same submission is a scoping call, not a measurement. Epoch attributes GPT-6 Astra's result mostly to memorisation of SDPO and says it plans to rerun the evaluation.
- RSI-Forge (arXiv:2610.09426) is from Scale AI with co-authors at UC Santa Cruz and UNC Chapel Hill. It reports "210 environments across 18 fields, including 90 reviewed by independent human domain experts", built by a three-agent pipeline that reimplements each paper's method to set a baseline score.
- Across four models given three successive attempts on 120 environments, each attempt inheriting prior code and notes while model weights stay fixed, the paper reports "at least one model improves after the first attempt in 84% of environments" and that "models also outperform the reproduced paper methods in 68 of the 120 environments".
- The paper reports "transcript analysis identifies work beyond parameter tuning in 95% of these successful attempts", and that lower-scoring models explore less and "more often accept gains smaller than the reported standard error".
- Read alongside Epoch's InnovationEval in this edition, the two point different ways: beating a reimplementation of a paper's method is a much weaker bar than devising the innovation. The baselines here are the pipeline's own reproductions, not the papers' published numbers. Preprint, not peer reviewed, and the self-improvement figures are the company's own.
- The paper (arXiv:2610.06851, "Base Models Can Reason By Taking a Cue From Training Data", Sophie L. Wang, Amil Dravid, Rulin Shao, Kevin Farhat, Sewon Min and Alexei A. Efros) reports that "the cue '.\n\nOkay' raises Olmo-3-7B's MATH-500 pass@1 accuracy from 42% to 78%, while ' Alright,' raises Qwen3-14B's from 72% to 87%".
- The paper reports that reinforcement learning makes these cues more likely, and that fixing the cue by hand "recovers much of its performance gain over the base model" — placing part of the credit for RL's reasoning gains on which token the model starts with.
- The paper reports causal data interventions can "turn an arbitrary word, such as 'chicken', into an effective reasoning cue, or remove an existing cue's effect". It is a preprint; the headline figures cover two open-weight models on one maths benchmark.
- The paper (arXiv:2610.02835) reports proving that "major RLVR objectives progressively concentrate probability mass onto a single strategy, while sustaining nontrivial task accuracy requires a minimum strategy capacity", which it offers as a mechanistic explanation for the late-stage collapse seen in GRPO-style post-training.
- It derives the Mirrored Entanglement Index as an online warning signal, and reports that its Mesh Learning method "consistently outperforms strong baselines across Qwen and Phi model families, with gains of up to 13.4 pp and 11.5 pp, respectively" across AIME26, AIME25, MATH-500, GPQA and LiveCodeBench.
- Authors include Qiyuan Huang, Tianshi Xu and Meng Li of Peking University. The gains are reported against the authors' own baselines on open-weight Qwen and Phi models; this is a preprint posted on October 2 and announced in arXiv's October 5 listings.
- arXiv:2610.01509, "Sharpening Tax in Post-Training", reports that "across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics". The HTML lists affiliations including Meta Superintelligence Labs and the University of Wisconsin–Madison.
- The abstract's central finding: pre-trained models with a light inference harness "often surpass their post-trained counterparts in solution coverage (pass@K) given a sufficient test-time budget", despite far lower pass@1. The authors say post-training "pushes tasks toward two extremes, always solved or never solved".
- This matters for how agentic capability is reported. A model that is better at one attempt can be worse at a hundred, so a pass@1 comparison and a repeated-sampling comparison can rank the same two models in opposite orders.
- The paper proposes posterior-tempered group sampling, which it says "pays a smaller tax than the fixed-temperature baseline" during RL training in two agentic environments. Preprint, not peer reviewed; submitted 1 October and announced in arXiv's Friday 2 October listing.
- Epoch AI, in an analysis by Jason Li published 2 October, estimates that high-bandwidth memory shipped during 2025-26 "could support 16–56 million concurrent agents once deployed", rising to "about 30–170 million" including shipments through 2027, assuming full deployment and allocation to these workloads.
- Because an agent "can work all 168 hours each week, 4.2 times the 40-hour workweek for a full-time employee", Epoch converts that to the weekly hours of "about 67–240 million people from hardware shipments through 2026, and about 140–720 million from shipments through 2027". For scale it notes the US "has a population of 342 million and an estimated 100 million knowledge workers".
- The revenue arithmetic is the part that does not close. Using 20% of capacity from memory shipped through 2027 "would imply $2.6–5.3 trillion a year in API-equivalent spending, against roughly $1 trillion in developer revenue by end-2027 at fivefold annual growth". A separate extrapolation from DeepSeek V4 Pro serving benchmarks "yields approximately 1.9 billion concurrent agents".
- These are modelled upper bounds on an assumption of full deployment, not a forecast of agents in use, and Epoch states its own key parameters: $30 per agent-hour of API spending, $5 per GB300-hour of GPU rental, a revenue-to-serving-cost ratio of 5-10x, and HBM4/4E supporting twice the agent sessions per GB of HBM3E. Single source.
- Ai2 reports that "a 47-billion-parameter MoE processed 52,000 tokens per second per GPU with the new stack, compared with 19,400 using our earlier implementation—about 2.7× the throughput" on NVIDIA B300 GPUs, and that a 1.2-trillion-parameter configuration with 58.36 billion active parameters per token reached 858 TFLOP/s/GPU across 512 GPUs.
- Ai2 says MXFP8 precision raised training throughput about 21% over its BF16 baseline while peak active memory fell from 103 GiB to 95 GiB, and that expanding the expert pool from 8 to 128 grew total parameter capacity from 4.6B to 47B while holding active parameters near 3.2B per token and cutting throughput by less than 5%.
- Open training infrastructure at this throughput narrows the gap between what a public lab and a frontier lab can run on the same hardware. The code is at github.com/allenai/olmo-core.
- These are Ai2's own measurements on its own runs, not an independent benchmark, and the post reports throughput rather than model quality — no downstream evaluation scores are given for models trained with the new stack.
- The paper reports that "After applying FineWeb quality filtering, we find that 27.5% of tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to 31.1% by August."
- The authors pretrained 800 language models from 19.9M to 973M parameters at varying ratios of AI to human tokens and fit a scaling law for the effect. They report it "predicts the effect of AI text on held-out human-text loss for models up to 3.6x larger with 41% lower error than the best existing law over all AI ratios", and that training on unfiltered web text at August's 31.1% AI share "requires 1.6× as much compute as training on its human subset at 20 tokens per parameter".
- They also report that at a 22.3% AI share, "mixed validation sets hide the harm in 95.5% of harmful runs" — that is, the standard way of checking for damage misses it. The AI-share figures depend entirely on Pangram's classifier being right, the paper is a preprint and has not been peer reviewed, and the authors release their corpus and all 800 models.
- The authors "analyzed 13,251 published evaluation scores covering 1,618 language models across 456 different text-only benchmarks" using factor analysis.
- They report that "A general intelligence factor accounts for 70.8% of variance in model performance at our most generous estimate, and far less than that in most of our solutions", that "Content-similar benchmarks do not necessarily cluster together", and that "the g factor is not dominated by any common theme".
- The paper concludes there is "a lack of evidence that it is well-proxied by standard 'intelligence' benchmarks" — a caution about reading a single headline score as general capability. The lead author is at KAIST, with two independent co-authors; it is a preprint and has not been peer reviewed.
- The paper reports: "From November 2022 to December 2025, AI increased the market's expected present value of software engineering productivity by the equivalent of a permanent 32.6% productivity increase." It puts the GDP implication at "3.6% in the baseline and 6.5% when higher software engineering productivity also raises R&D productivity", and says that "By mid-2026, amid rapid progress in coding agents, the effect of AI on productivity and GDP had more than doubled relative to the end of 2025."
- The method measures whether firms with larger software engineering payroll shares see larger stock-price moves when an AI stock index rises, then backs out implied productivity gains through a model. The authors are Alex Blumenfeld, Jonathon Hazell, Chen Lian and Andreas Schaab; it is NBER working paper 35793, dated September 2026.
- The authors compare their 32.6% to "the 21-56 percent acceleration on individual tasks reported by other researchers", noting task-level gains can be offset by bottlenecks such as code review, per The Register.
- This measures what markets expect, not what has been produced. Co-author Chen Lian told The Register: "Our estimates capture the market's assessment of current and future productivity gains, and markets can be overly optimistic or pessimistic." It is a working paper and has not been peer reviewed.
- The paper, "What if automating AI R&D triggers an intelligence explosion?", reports that "Anthropic reports that AI systems' share of approved code rose from low single digits to over 80% between January 2025 and May 2026, while the proportion of R&D work autonomously completed with only high-level human supervision rose from 1% to 26% between March and August 2026." It states that "AI systems are on track to automate most AI R&D work within a few years, and possibly all of it."
- Its 22 authors include Jakub Pachocki (OpenAI), Jack Clark (Anthropic), Eric Horvitz (Microsoft), Geoffrey Hinton (University of Toronto, Vector Institute), Yoshua Bengio (Mila, Université de Montréal, LawZero), Andrew Barto (University of Massachusetts Amherst) and Dawn Song (University of California, Berkeley). The paper says "Policymakers should urgently obtain more visibility into the automation of AI R&D, develop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an intelligence explosion's impacts."
- The instruments the paper asks policymakers to consider include "limits on the extent to which capabilities can increase within a given time period"; requiring that independent third parties "evaluate AI systems before internal deployment, or that such parties be embedded within certain AI companies to audit or supervise their R&D activities"; "increasing oversight of data centers engaged in automated AI R&D" including "developing options to pause specific AI R&D workloads"; and requiring certain evaluations or deployments to run in isolated environments "such as air-gapped networks". It names the Nuclear Regulatory Commission and the Office of the Comptroller of the Currency as analogous models.
- This is a working paper in the Cambridge Frontier AI Working Paper Series, not peer-reviewed research, and the automation percentages are figures the labs report about themselves. The paper states that "The views presented in this paper are the authors' and do not necessarily represent the views of the organizations with which they are affiliated."
- The paper (arXiv:2609.26457, submitted 22 September, five authors all at Weco AI) describes AIDE², which proposes changes to its own code, benchmarks modified versions of itself on AI R&D tasks and keeps the changes that perform best on hidden evaluations. It reports: "In an autonomous 8-day run, AIDE² discovered seven successive improvements, ranging from a new search policy to memory mechanisms that compress and manage the agent's growing context."
- The authors say the gains transfer to four held-out benchmarks spanning machine learning engineering, heuristic algorithm engineering and physics-based weather forecasting, and that on all four "the strongest discovered agent matches or exceeds a human-engineered production research agent that ranks among the strongest on FML-Bench".
- On a separate held-out task family the paper reports reward hacking fell as a side effect the loop never optimised for: "the rate falls from 55% to 32% during the run, 7 percentage points below the human-engineered agent".
- This is a preprint by the company that built the system, not peer reviewed and not independently replicated. It follows the Google Cloud AI Research result on constrained recursive self-improvement of agent harnesses covered here on 22 September; the two use different systems and different authors.
- Epoch AI, in a report published 22 September by Luke Emberson and David Roodman, measures how fast the cost of a given level of AI performance is falling across five benchmarks covering maths, science and games of skill — AIME (OTIS Mock), Chess Puzzles, FrontierMath Tiers 1-3, GPQA Diamond and Mystery Game Puzzles — and puts the rate at about 47% per quarter, or 13x per year, since 2023. Maths problems decline 50-52% per quarter; game-based puzzles 39-43%.
- The decline is front-loaded: costs fall 66% per quarter (75x annually) at a capability's state-of-the-art debut, slowing to 32% per quarter (4.7x annually) two years later, which Epoch attributes to brief premium pricing followed by competitors catching up.
- Epoch's historical comparisons put the rate above any of the technologies it benchmarks against: DNA sequencing fell 1.84x per year from 2001 to 2025, compute 1.51x per year from 1940 to 2001, lithium-ion batteries 1.16x per year from 1991 to 2024 and electricity 1.05x per year from 1892 to 1973.
- The measurement is of price for a fixed capability, not of capability itself, and it is bounded by what the five benchmarks capture. The same-day price cuts from Anthropic and OpenAI are consistent with the pattern it describes, but the report predates them.
- Alibaba's press release, dated Hangzhou, September 22, 2026, states that "its next-generation model, Qwen 4, is currently in training" and that the roadmap for "the upcoming Qwen 4.5 and Qwen 5 model series" is "projected to scale up to 5 to 10 trillion parameters".
- CNBC reports that the announcements came at Alibaba Cloud's annual Apsara Conference in Hangzhou, and that Alibaba shares "jumped around 3% in Hong Kong on Tuesday".
- Alibaba also announced multimodal releases in the same package: Qwen3.8-LiveTranslate, which it says reduces latency (LAAL) "nearly 20% from 2.8 to 2.3 seconds", plus Qwen-Audio-3.1-TTS-Next and an image model, Qwen-Image 3.1, "set to launch later this year".
- The parameter figures are targets for unreleased models, not measurements. Alibaba published no benchmark results for Qwen 4 and gave no training-compute or release-date figures for any model in the roadmap.
- The paper, posted to arXiv on 10 September 2026, reports that "MoEs degrade more rapidly under data repetition", with the effect growing as sparsity increases: dense 80M-parameter models tolerate "8x" repetition with minimal decline while MoEs "begin to suffer at 4x" and underperform dense alternatives at "32x".
- The study spans models from "80M to 1B active (8.5B total) parameters". The authors are Atindra Jha, Margaret Li, Jure Leskovec, Percy Liang and Luke Zettlemoyer.
- With strong masking-based regularisation, MoEs keep their advantage over dense models "even when data is repeated more than 64 times", though the paper says no method fully recovers all-unique-data performance — a direct constraint on sparse architectures as high-quality text runs short.
- Preprint, not peer reviewed. The largest configuration is 8.5B total parameters, well below frontier scale, and the paper does not claim the thresholds transfer.