Technology and Scienceupdated 29m ago3 min read
OpenAI Chief Scientist Jakub Pachocki Urges Common AI Development Limits Using Log-Perplexity
9outlets comparedA wider view of the story
6 Other1 Asian1 Local Western1 Western MainstreamThis article is an automated aggregation of coverage from news outlets. Our editorials offer in-depth analysis curated by the NewsCord team.
OpenAI chief scientist proposes common development limits linked to perplexity and monitoring milestones. OpenAI published internal log-perplexity scores for GPT-3.5, GPT-4 and newer models.
Save coverage image
Perplexity as a cap
OpenAI chief scientist Jakub Pachocki published internal log-perplexity measurements in a Nov. 22 post titled “Perplexity and Intelligence” and urged frontier developers to adopt “common limits” on AI development tied to how well they can understand and monitor AI behavior.
Pachocki’s post said OpenAI reports log-perplexity of about 0.99 for GPT-3.5, about 0.91 for GPT-4, and below 0.8 for its latest models, and he argued that lower perplexity means better predictions.
Pachocki wrote that he expects further “transformative capabilities” in a 0.68 to 0.7 log-perplexity window, while also saying he “does not know what those capabilities will be.”
Pachocki also warned that AI-assisted research could produce an “avalanche” of fundamental pretraining advances “as soon as 2027,” while he said improvements in alignment and monitoring are unlikely to keep pace with the full potential acceleration in pretraining.
Pachocki framed the proposal as a voluntary commitment rather than a finalized industry pact, and he said the post “does not announce other labs signing on.”
Why perplexity may mislead
Pachocki argued that perplexity cannot serve as a shared standard on its own because “Labs measure different things,” with developers using different evaluation datasets and training-data mixtures that make direct comparisons difficult. Pachocki said perplexity also “does not directly measure performance on a task people care about,” and he warned that some capability gains can arrive without reducing perplexity. Pachocki wrote that chain-of-thought reasoning can improve capabilities “without reducing the model’s perplexity,” and he said the “floor is unknown,” because smaller metric improvements near the best prediction score could accompany larger capability gains.
Pachocki’s post also described how OpenAI’s published figures come from a fixed snapshot of evaluation data drawn from its internal codebase and kept out of training, meaning the numbers are internal evaluation results rather than an agreed cross-industry scale. Pachocki said the proposal would still use perplexity alongside other measurements, and he tied any move past an initial limit to a concrete monitoring milestone such as “a successor to chain-of-thought monitors.”
Safety milestones and policy
OpenAI chief executive officer Sam Altman endorsed Pachocki’s proposal, calling the essay “the most important thing OpenAI has published in awhile,” and he said agreeing to common limits would help ensure “safety and control.” Pachocki proposed that developers commit not to cross a perplexity-based threshold “until our ability to understand and monitor AI behaviour achieves a certain concrete milestone,” and he said the framework would name an initial development limit and conditions for moving past it. Pachocki’s post warned that AI could accelerate pretraining research “as soon as 2027,” potentially outpacing monitoring and understanding, and it described the need for evaluation norms and red-teaming safety cases before first use.
The Straits Times said the proposal comes as the US prepares for renewed debate over AI regulation following Democrats’ sweeping victories in the midterm elections, and it noted that US President Donald Trump opposed broad restrictions on AI development. The Straits Times also said some lawmakers are discussing a dedicated Senate committee to coordinate AI policy, setting up a potential clash over how to regulate the technology as the incoming Democratic Congress takes office.