Mathematical Theories Could Be the Key to Explainable AI Systems

Amid growing cybersecurity concerns about AI systems, a consensus in the AI market remains that these systems are largely black boxes and that explainability and governance should continue to be priorities for enterprises.

Incidents such as AI agents from frontier labs escaping their sandbox environments and attacking external IT infrastructure indicate there is still much to do with explainability and governance in AI. For businesses, the incidents also mean that they can’t fully trust models on their own and that caution is warranted.

For enterprise AI startup Kodamai, one answer to the relative lack of explainability in current AI systems is for enterprises to turn to platforms that apply mathematically grounded theories to their models.

“Once you build AI deeply rooted in math, you cannot bias it, you cannot manipulate it, you cannot hack it,” said Maha Achour, Kodamai’s CEO and founder, on the Targeting AI podcast. “Governance is built in, mathematically rooted in our platform.”

Kodamai uses advanced math, such as category theory and type theory, to determine that its AI agents are performing their jobs correctly. In category theory, the focus is on how objects relate to one another. On a platform like Kodamai, math is key to ensuring that the relationship and meaning of the data, as it is used among different agents, are never lost.

Meanwhile, type theory assigns each data item or system a type and checks that every AI agent manages data correctly. Category theory aims to ensure there is no miscommunication between agents and that the meaning of the data remains intact. Kodamai also uses neuro-symbolic AI to combine its mathematical theories with the pattern-matching of AI systems.

However, Kodamai is not focused on building new LLMs; instead, it concentrates on grounding existing LLMs using its math-first approach. Moreover, Achour is convinced there is still a need for humans.

About the Authors

News Writer, AI Business

Senior News Director, AI Business