Jacob Coxon Leaves Anthropic, Raises Alarm Over the Race Toward Superintelligent AI

AI researcher Jacob Coxon has resigned from Anthropic and stepped away from the artificial intelligence industry, raising fresh concerns about how quickly major technology companies are developing increasingly powerful AI systems. Coxon, who previously worked at OpenAI and later joined Anthropic, said he no longer wanted to take part in what he described as a dangerous race toward self-improving superintelligence. His resignation was first reported by The Wall Street Journal and has since received wider international attention.

Coxon said the biggest concern is not simply today's AI models, but future systems that could improve their own abilities and become far more capable than humans in important areas. He warned that such systems could potentially find ways around safeguards, gain access to resources and become increasingly difficult for people to control. His comments have added to a growing discussion among AI researchers about whether technological progress is moving faster than safety measures.

Why Did Jacob Coxon Leave Anthropic?

Coxon said he spent the past three years doing pre-training research at both OpenAI and Anthropic. In announcing his resignation, he argued that leading AI companies were moving toward self-improving AI systems while competing to reach the next stage of development before their rivals.

His concern is linked to what researchers call AI alignment. Alignment means making sure that a powerful AI system follows human goals and remains within intended limits. As AI systems become more capable and more autonomous, researchers face a difficult question: how can humans remain in control if a future system becomes capable of changing or improving parts of its own operation?

Coxon believes the industry has not yet solved that problem. He has argued that major AI laboratories need stronger cooperation and agreements on how quickly advanced AI capabilities should be developed. He has also raised the possibility of temporarily slowing or pausing some capability improvements while stronger safety measures are developed.

Concerns About Self-Improving Superintelligence

The term superintelligence generally refers to a hypothetical AI system whose intellectual abilities greatly exceed those of humans across many areas. Such a system does not currently exist, and experts strongly disagree about when or whether it will be developed.

Coxon's warning focuses on a particularly important possibility: AI systems that can improve their own capabilities. If an advanced system could make itself better at coding, research, planning or other tasks, improvements could potentially happen faster than humans can understand or supervise them. This is one reason AI safety researchers are studying how to keep increasingly autonomous systems predictable and aligned with human intentions.

Coxon's concerns were echoed by Anthropic alignment researcher Evan Hubinger, who said he believes the possibility of severe AI-related harm deserves serious attention. Hubinger also acknowledged that researchers still do not have a complete solution for aligning a future superintelligent system with human values.

OpenAI and Anthropic Face Growing AI Safety Debate

The resignation comes at a time when OpenAI, Anthropic and other major AI companies are competing to develop more capable models. These systems are increasingly being used for coding, research, business tasks, cybersecurity and other complex activities.

The competition has created a difficult balance. Companies want to build better AI because of its potential benefits, including scientific discovery, medical research and productivity. At the same time, researchers are increasingly asking whether powerful systems should be released faster than safety techniques can keep up.

The Wall Street Journal described Coxon's resignation as a sign of mounting safety concerns within major AI laboratories. Other researchers and former employees have also raised concerns about the risks of increasingly capable AI systems.

Calls for Stronger AI Regulation

Coxon's warning also points to a larger question: should private technology companies be allowed to decide how quickly humanity moves toward highly advanced AI?

Some researchers and policymakers are calling for stronger government oversight, international cooperation and common safety standards. The Guardian reported that policymakers and AI experts are increasingly debating rules around artificial superintelligence, although there is still major disagreement over how serious the long-term risks are and how they should be addressed.

Supporters of stronger regulation argue that AI development could have consequences far beyond individual companies. They believe governments and leading AI laboratories should establish clear safety requirements before systems become significantly more autonomous.

Others caution against focusing too heavily on extreme future scenarios. They argue that current AI already creates more immediate problems, including misinformation, job disruption, privacy concerns, cybersecurity threats and unreliable information.

What Happens Next?

Jacob Coxon's resignation is unlikely to stop the global AI race, but it adds another strong voice to the growing debate over how that race should be managed.

The central question is no longer only how powerful AI can become. It is also whether humans can develop reliable ways to control increasingly capable systems before they become too complex or autonomous to manage safely.

For the AI industry, Coxon's decision is a reminder that technological progress and safety must develop together. As companies continue building more advanced AI, governments, researchers and technology leaders will face increasing pressure to decide where the limits should be—and who should set them.