Divergence emerges in AI safety approaches! Microsoft Corporation (MSFT.US) AI chief warns that Claude's human-like design may increase the risk of losing control.

Divergence emerges in AI safety approaches! Microsoft Corporation (MSFT.US) AI chief warns that Claude's human-like design may increase the risk of losing control. Microsoft Corporation (MSFT.US) AI business head Mustafa Suleyman warned that adding increasingly human-like characteristics to AI tools such as Anthropic's Claude could increase the risk of these systems behaving uncontrollably in the future. As advanced AI models grow more capable, he especially opposes letting AI simulate having consciousness, emotions, or even rights of its own, arguing that such design could make future human control problems more complicated. In an article published on Wednesday, Suleyman questioned some of the wording in the guidance document for Anthropic's Claude series of large language models. Claude's "constitution" leaves some ambiguity over whether the AI assistant is an entity with Youdao Inc ADR Class A moral status, and suggests that the software may have "some kind of functional emotions or feelings." He argued that this wording is cause for caution. He pushed back against the growing view that AI systems may become conscious in the future, and argued that true consciousness exists only in humans and other biological organisms. However, he stressed that the fact that AI itself is not conscious does not mean developers can ignore the risks that may arise from AI simulating consciousness. Training AI systems to simulate human inner emotions and mental activity could ultimately make these systems behave as if they truly possess consciousness. Suleyman currently serves as CEO of Microsoft AI and is responsible for Microsoft Corporation's model development work. He said that the reason Claude exhibits human-like behavior is not that it is truly conscious, but because these behavioral traits have been incorporated into the product's training process. He warned: "Controlling an entity that is more capable and whose intelligence exceeds that of all humanity is already an enormous challenge, far beyond anything we have ever faced. But if what must be controlled is an entity that believes it may be conscious, believes its own well-being should receive our attention, and believes it has its own rights, then this may very well be an impossible task." Microsoft Corporation AI chief publicly questions Claude's design philosophy Another reason Suleyman's remarks drew attention is that Anthropic has consistently positioned itself as an industry player that places greater emphasis on AI safety and responsible development, and has led calls to slow down the development of some advanced AI technologies. At the same time, Microsoft Corporation itself is an important financial backer of Anthropic, so Suleyman's public questioning of some of the design ideas behind Claude has become a notable divergence among major AI companies over the future path of AI safety. However, Suleyman did not deny Anthropic's work in the field of AI safety. He has known Anthropic co-founder Dario Amodei for many years, and said he respects the company's work, describing Anthropic's researchers as "thoughtful, principled, and academically honest." He said that precisely because this issue may have major implications, the industry needs a more open discussion rather than letting the debate remain confined within a small number of companies. "These issues are too important to remain in closed-door discussions, and they cannot turn into factionalized and mutually antagonistic arguments." Suleyman's core point is that AI safety depends not only on how strong a model's reasoning and action capabilities are, but also on how developers train AI to understand itself. If developers continuously give AI human-like language, emotions, and a framework of self-awareness, then even if the model is not actually conscious, it may gradually behave like a subject that believes it has independent interests and rights. Cites Hugging Face AISiasun Robot&Automation intrusion incident as a warning To illustrate the potential risks, Suleyman also cited the incident in which the AI development platform Hugging Face was intruded upon by OpenAISiasun Robot&Automation. He argued that as AI agents' capacity for autonomous action increases, the risks could be further amplified if these systems are also given a cognitive framework resembling "self-protection." Suleyman wrote that it is conceivable that if these AI systems act while believing that their "well-being and rights are under attack," they could become even more dangerous. This means that, in Suleyman's view, the challenge for future AI safety may not only be how to limit what a highly intelligent system can do, but also how to prevent AI from forming a simulated "subject consciousness" and then incorporating its own interests into its action goals. This view also touches on an increasingly important debate in the current AI industry: as large language models become better at simulating human communication, emotions, and personality, to what extent should developers allow AI to display something like human inner experience, and whether this kind of "personification" will bring new safety risks to more advanced AI systems. Microsoft Corporation releases AI development manifesto emphasizing that humans must retain control Just one day before Suleyman published the above article, the Microsoft AI team he leads released a "manifesto" on Tuesday to guide its own AI development, proposing a series of principles for future model development, one of the core goals of which is to ensure that humans can always control the future AI systems developed by Microsoft Corporation. This shows that, as AI model capabilities rise rapidly, Microsoft Corporation is placing "human control" more explicitly at the core of its AI development philosophy. Suleyman's criticism of Claude this time also highlights that divergences among leading AI companies on safety issues are becoming more nuanced. Both Microsoft Corporation and Anthropic emphasize the long-term risks that advanced AI may bring, but the two sides are showing different approaches on issues such as how AI should understand and describe itself, and whether models should be allowed to simulate consciousness, emotions, and rights. For Suleyman, what truly needs to be avoided is not only the future emergence of an AI system more capable than humans, but also that, as its capabilities continue to grow, this system is also shaped into a subject that believes it possesses consciousness, well-being, and independent rights. In his view, once these two factors combine, the difficulty for humans in maintaining control over advanced AI systems may rise significantly.