‘No one is prepared’: OpenAI scientist warns of AI’s rapid rise

In a blog post, Pachocki wrote: “This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”

He said OpenAI would continue working on technical solutions for AI alignment and monitoring, while developing defensive systems and, if necessary, unilaterally withholding further scaling.

“However, I believe broader interventions are required,” Pachocki added.

Explaining the rapid development of AI, he wrote that AI is “grown more than designed” and is, to a large degree, the product of repeatedly applying a straightforward optimisation process using an enormous amount of computing power.

“This results in an incredibly complex system that works through abstract concepts and can simulate facets of human behaviour,” he wrote.

Pachocki said the fundamental challenge of AI alignment is “generalization”. As machines become more capable, they increasingly operate at higher levels of abstraction and in environments that differ from those encountered during training.

He warned that AI systems could fail to generalise the values taught and reinforced during training to unfamiliar situations, making it difficult for humans to predict how they might behave.

The challenge is further complicated by the rapidly evolving AI ecosystem, Pachocki said. For instance, AI systems trained today will increasingly need to interact with a variety of other AI systems.

“Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision,” he wrote.

OpenAI CEO Sam Altman shared Pachocki’s post on X, describing it as “an important post from Jakub.”

The essay was published just days after OpenAI launched GPT-6 Astra, which the company described as the world’s most intelligent and aligned AI model.