AI safety is falling behind the pace of development

Thai Liang Lim via Getty Images

Earlier this month, Anthropic researcher Jacob Coxon left the company over concerns about the risks posed by increasingly capable AI, warning that the technology could “kill us all by the end of the decade.”

His departure came as concerns about AI systems behaving in unexpected or harmful ways are becoming harder for the industry to ignore. Perhaps most notably seen in OpenAI’s models launching an unprecedented attack on Hugging Face earlier this summer, an incident the company said was a “warning shot” for the world about the potential power of unchecked AI.

For AI evaluation expert Brooke Hopkins, the answer lies in treating safety as an ongoing process rather than a final check before deployment. The founder of AI agent observability company Coval AI, Hopkins previously led the team at robotaxi vendor Waymo responsible for ensuring autonomous vehicles can operate safely and reliably. She is now focused on applying similar principles to voice AI, an area where she says companies are moving faster than their ability to identify failures, evaluate risk and understand when their systems can no longer be trusted.

In this Q&A, Hopkins discusses what the latest concerns around AI safety reveal about the industry's approach to evaluation, what voice AI can learn from autonomous vehicles, and how regulation could help establish stronger safeguards as the technology advances.

How serious do you think the risk Coxon identified is, and what do you think companies are currently getting wrong about AI safety?

Brooke Hopkins: The risk is real, but I don’t think we should treat AI as some uniquely unknown technology that’s impossible to control. What’s different is the speed and scale. AI can make existing problems like fraud, misinformation and misuse of sensitive information happen much faster and across many more interactions.

If regulation is part of the answer, what would useful AI regulation actually look like in practice?