The video has 904 views. Its still frame is a dim Stanford lecture hall: a long wooden demonstration bench, a wall of whiteboards, and one man in dark clothes standing at the far end of the bench. YouTube dates the upload November 16, 2022. The talk itself happened the day before, a Tuesday, in the Applied Physics/Physics Colloquium, under the title “Lessons from scale for large language models and quantitative reasoning.” The speaker was Ethan Dyer, a theoretical physicist turned machine-learning researcher on Google’s Blueshift team, and the audience was a room of physicists who had come to hear what scale had taught the people who build language models.
Now count forward from that Tuesday. On November 28, OpenAI released text-davinci-003 and began grouping its recent models into what it called the GPT-3.5 series. On November 30, it put a chat window on a sibling model and called it a research preview. That was ChatGPT. Thirteen and fifteen days after Dyer finished talking, the public met GPT-3.5. His talk is a snapshot of how researchers inside the labs measured progress in the last two weeks before everyone else started measuring it in chatbot demos.
What he was selling to the physicists
Dyer’s pitch had a physicist’s shape. Neural networks, in his framing, are not just software; they are systems with regularities you can measure and sometimes derive. The talk’s own abstract says it plainly: some measures of progress show robust power-law improvement across many orders of magnitude in dataset, model and compute scale, while other capabilities remain hard to extrapolate. He had written the theory side of that argument with Jared Kaplan, Yasaman Bahri and others, in the paper that later appeared in PNAS as “Explaining Neural Scaling Laws,” which splits scaling into variance-limited and resolution-limited regimes. Both men had started their careers in high-energy theory. Now they were asking what a loss curve says about a model’s future.
The empirical half of the talk was BIG-bench, the Beyond the Imitation Game benchmark that Dyer helped coordinate: 204 tasks from 450 authors at 132 institutions. As models grew, a small slice of those tasks looked like they flipped overnight from guessing to competence, the effect people were beginning to call emergence. The analysis this piece builds on ties Dyer’s talk to the counterargument that Rylan Schaeffer and colleagues later formalized: much of that suddenness lives in the metric, not the model. Grade with exact-match or multiple-choice accuracy and the curve looks like a cliff. Grade with a continuous score such as Brier or token edit distance and the same capability climbs smoothly along the line the scaling laws predict.
The other half was Minerva, the model that made the subject concrete. Minerva took Google’s PaLM at 8B, 62B and 540B parameters and kept training it on arXiv papers and mathematical web pages. Its trick at answer time was to stop trusting a single attempt. It wrote out step-by-step solutions many times, then took the most common final answer, a technique called majority voting. The published numbers show what that bought. On the MATH benchmark of high-school competition problems, base PaLM 540B scored 8.8 percent. Minerva 540B scored 33.6 percent on a single attempt and 50.3 percent with voting. On college-course problems drawn from OpenCourseWare, the voting score was 30.8 percent; on the STEM subset of MMLU, 75.0 percent. Minerva 62B, tried on Poland’s national math exam, landed at 57 percent, which happened to be the national average that year, and the 540B model reached 65 percent.
That was the frontier of AI doing quantitative science in November 2022: a system that earned its answers by arguing with itself, and passed a national high-school exam at roughly the average student’s level. It was extraordinary. It was also a long way from a paper.
The question from the back of the room
Then came the Q&A. At 56:15, per the transcript, Dyer reached for an example of the opposite problem: a contest “from Anthropic” offering a prize for the best examples where scaling does not help. Some tasks get worse as the model gets bigger, he went on, often because what you want is poorly matched with, or even anti-correlated with, what the model is trained to imitate. Replicating biases found in the training data is the standard example.
The precise version of that story is better than the loose one. The Inverse Scaling Prize launched June 27, 2022, with a $250,000 pool, and was organized by researchers on behalf of the Fund for Alignment Research, including Ethan Perez, Ian McKenzie and Sam Bowman, with NYU at the center. It was not an Anthropic internal challenge. Anthropic’s part was supplying private, held-out language models on which submissions were evaluated, and Perez and Bowman both carry Anthropic affiliations on the prize’s final paper. The second and final round closed October 27, 2022, which means the contest Dyer name-checked had ended nineteen days before he mentioned it to a Stanford room. It drew 99 submissions over two rounds and paid prizes to 11 tasks. Among the winners were Prompt Injection and Pattern Matching Suppression, which showed that a bigger model can get worse at following an instruction precisely because it is better at continuing a pattern.
The paper that summarized the prize delivered a conclusion that cuts against the tidy version of scaling. Its authors reported that the winning tasks helped uncover U-shaped and inverted-U trends, where a trend reverses direction at larger scale, so trends measured on smaller models are less reliable predictors of larger-model behavior than previously understood. A Google team had come at it from the other side in early November 2022, days before the talk, showing that some of the Round 1 winners turned U-shaped at PaLM scale. There is a small irony worth keeping: Dyer, then at Google, praised a prize whose held-out evaluators were Anthropic’s models, and the PNAS version of his scaling-laws paper now lists Anthropic as his present address.
Three years, two months
Fast-forward. On January 26, 2026, Quanta Magazine ran Natalie Wolchover’s first column in its new Qualia series, “Is Particle Physics Dead, Dying, or Just Hard?” Most of it is a reporter checking in on a field that has spent thirteen years without new physics from the Large Hadron Collider. Partway through, she calls Jared Kaplan, the Anthropic co-founder who in the 2000s was a Harvard graduate student working with Nima Arkani-Hamed on the amplitude research physicists are still pursuing, and who left physics in 2019. Kaplan told her he went into AI because it seemed plausible that it would make progress faster than almost any field in science ever had.
Then he made the forecast. Planning a collider on a ten-year timescale is kind of irrelevant, he said, because if a collider gets built in ten years, AI will be building it. He put roughly a 50 percent chance on theoretical physicists being mostly replaced by AI in two or three years, with systems generating papers as good as those of Arkani-Hamed or Ed Witten “pretty autonomously.” Read that against Minerva. In 2022, the headline was that a model matched a national average on a high-school exam if you let it vote. In 2026, the claim, from a man who co-wrote the scaling-law theory with Dyer, is that the output will be indistinguishable from the best theorists’ papers.
From the Stanford talk to that interview is 1,168 days, three years and two months, which is close to the horizon Kaplan is now forecasting. The window he named closes between early 2028 and early 2029, and 252 days of it have already elapsed as of this week. The same column carries the counterweights. Wolchover notes that AI is already sorting collision events at the LHC more accurately than hand-built algorithms, and Adam Falkowski, who predicted in 2012 that the field would slowly die out, says talented postdocs are leaving for AI. CERN postdoc Cari Cesarotti, meanwhile, argues that chatbots are making students worse at physics and that humans need to read textbooks and think. Kaplan’s forecast arrives as one voice among several, which is how it should be handled.
The ruler
That returns the story to the lesson Dyer’s talk was built around. His argument about emergence was that apparent leaps are frequently artifacts of what you measure. A cliff in exact-match accuracy becomes a slope when you switch to a continuous score. “As good as Witten’s papers” is the exact-match version of a research claim: it asks for a binary verdict, from a handful of expert readers, on an output that has no loss curve. If Dyer is right about metrics, a forecast phrased that way can look like it is arriving suddenly or not at all, depending entirely on who is grading.
The inverse scaling result adds a second caution. Scaling laws are forecasts with error bars and regimes, and the 2021 paper Dyer and Kaplan co-wrote is explicit that which regime a system is in decides which line you may extrapolate. A trend measured at one size can reverse at the next. Kaplan’s career is built on drawing those lines well, and he may be right. But a forecast of papers as good as theirs needs a measured trajectory leading up to it, and nobody has published one.
That absence is the white space. Between Minerva’s percentage on a benchmark and a theorist’s paper there is no agreed, continuous, graded measure of research quality: no Brier score for physics. The labs have every reason to claim the top of that scale and skeptics every reason to dispute it, and anyone pricing the next three years of this industry is implicitly pricing the answer. Whoever builds the ruler first, one that moves smoothly from a national-average math exam to a paper Witten would sign, will decide whether the next 1,168 days look like a leap or a line.






