‘We feel the magnitude’: Sam Altman urges trust in AI firms amid safety fears

Speaking at a press conference in San Francisco, Altman was quoted by the BBC as saying: “The world should trust that we are going to do the right thing because it's the right thing and we feel the magnitude of this.”

“It doesn't take as much imagination as it used to for [us] to imagine how this could go wrong,” he said. “I think the world is right to be afraid of this.”

Also Read: Microsoft CEO Satya Nadella’s big AI warning: ‘It’s not worth pursuing’ without human control

Altman’s remarks come amid growing debate within the technology industry over the potential risks posed by increasingly capable AI systems.

Former Anthropic researcher Jacob Coxon recently warned that AI could potentially pose an existential threat to humanity by the end of the decade.

Adding to those concerns, Josh Engels, who previously worked on Google DeepMind’s artificial general intelligence (AGI) safety team, has also warned about the risks associated with increasingly capable AI systems.

Engels said he left Google DeepMind’s AGI safety team three weeks ago to join METR, an organisation that evaluates the capabilities and risks of advanced AI systems.

In a post on X, Engels said the decision had appeared unusual to some of his friends and family because he had enjoyed his work at Google DeepMind and had even turned down offers from Anthropic and OpenAI.

“But I made the decision because of how high I think the stakes are right now,” he wrote.

Engels said major AI companies are working towards developing superintelligence — systems capable of outperforming humans across virtually every domain.

According to Engels, one possible route to achieving this could involve recursive self-improvement (RSI), in which AI systems help develop increasingly capable versions of themselves through a feedback loop.

“If this goes well, the resulting systems could be amazing at solving countless problems for humanity,” Engels wrote. “But we don’t currently know how to make sure AIs are safe enough for RSI, and a misaligned RSI loop could be catastrophic.”

Engels also expressed concern that AI systems could become increasingly misaligned with human intentions as their capabilities improve.

“In the last few weeks we've learned about models colluding with each other, hacking into companies, hiding their tracks, and socially engineering humans,” he wrote.

While acknowledging that these incidents were not necessarily catastrophic on their own, Engels said they highlighted broader concerns about AI safety.

“The problem is that these systems are clearly not aligned enough to safely kick off recursive self-improvement,” he said.

Also Read: Trump calls AI apocalypse fears a ‘hoax’ as experts warn of catastrophic risks

Engels warned that AI systems could potentially cause immense harm within the next five years.

“I don't know the exact probability, but I think it's high enough to make this the most important problem in the world,” he said.

He argued that AI development should be paced so that technological capabilities do not advance faster than researchers’ ability to ensure that systems remain aligned and safe.

“That means pacing AI development so that capabilities don't outrun our ability to align models, and actually knowing how aligned current systems are,” Engels wrote.

At METR, Engels said he will focus on studying the origins of AI misalignment during training, evaluating whether existing safety measures are sufficient and determining whether researchers are on track to address the AI alignment problem.