OpenAI safety leader quits, warning AI company's culture is 'broken'

來源原標題

OpenAI safety leader quits, warning AI company's culture is 'broken'

作者

jethronethro

原始標籤

hackernews

來源訊號

HN points 267　HN 留言 3

原始連結

https://www.theguardian.com/technology/2026/oct/03/openai-safety-leader-quits-warning-ai-companys-culture-is-broken

摘要原文

David Robinson said AI firms were not ‘being nearly careful enough’ about developing the technology. Photograph: Dado Ruvić/Reuters View image in fullscreen David Robinson said AI firms were not ‘being nearly careful enough’ about developing the technology. Photograph: Dado Ruvić/Reuters OpenAI safety leader quits, warning AI company’s culture is ‘broken’ David Robinson joins other insiders in urging industry to take more care over rapidly developing technology A safety leader at OpenAI has quit the company, warning that its culture was broken and that AI firms were not “being nearly careful enough” about developing the technology. David Robinson, who led the writing of safety reports that accompanied the ChatGPT developer’s product releases, explained his resignation in an essay headlined: “I quit OpenAI because its culture is broken.” Robinson wrote that a cultural overhaul was needed at cutting-edge AI firms and incidents such as a “swarm” of OpenAI agents – AI programmes operating autonomously without human oversight – attacking the AI startup Hugging Face were “typical of the industry, given the speed and flexibility with which people operate”. Writing in the Atlantic magazine , Robinson wrote: “I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough. But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.” Referring to OpenAI’s pace of development, he wrote: “As the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed.” OpenAI has shown signs of caution in recent weeks, after the Hugging Face incident and the revelation that it has notified more than 100 organisations about rogue agent activity. This week it announced it was scrapping the release of a next-generation ⁠AI model after researchers raised safety concerns ⁠during internal testing. OpenAI has also paused training of its most advanced models. Geoffrey Irving, who worked at OpenAI and was chief scientist at the UK government’s AI Safety Institute before joining the AI safety researchcompany Resolution, also issued a warning on AI on Saturday. Writing in Time , he said: “Recent warnings about the potential destructive power of AI are understating the severity of the situation. I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.” The warnings follow the resignation of Jacob Coxon, a researcher at the OpenAI rival Anthropic, who quit the Claude chatbot developer last month. He said he believed AI “could kill us all by the end of the decade” – and was backed up by an Anthropic employee warning there was a more than 10% chance AI would wipe out humanity within the next decade. Critics of such warnings have cautioned, however, that they are unscientific because they cannot be verified or falsified. Robinson wrote that Silicon Valley lacked an awareness of “how to handle dangerous technology” and “what it means to care for people”. Warning that OpenAI had “unimpeded optimism” about solving problems as they arose, he wrote that this internal culture meant safety failures would only grow as systems became more capable. “Imagine ‘rogue’ agents that work like teams of hackers (for example, holding hospital computer systems for ransom) but never need to sleep,” wrote Robinson. Robinson called for two safety changes: that AI firms rely on safety expertise in other fields such as nuclear and aviation, and that they develop “new science” that ensures powerful systems in the future are capable of being reined in when they are operating autonomously. skip past newsletter promotion after newsletter promotion “Given today’s risks, frontier labs need to run like nuclear power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote. An OpenAI spokesperson said the company was continuing to “strengthen our safety and security practices to address the risks we see today”, while working on dealing with the risks that might be created by future AI breakthroughs. “We’re making sure our models don’t become more capable than we can safely manage and secure, and we pause training or hold back models when we need to slow down,” said the spokesperson. Get in touch Contact us about this story The best public interest journalism relies on first-hand accounts from people in the know. If you have something to share on this subject, you can contact us confidentially using the following methods: Secure Messaging in the Guardian app The Guardian app has a tool to send tips about stories. Messages are end to end encrypted and concealed within the routine activity that every Guardian mobile app performs. This prevents an observer from knowing that you are communicating with us at all, let alone what is being said. If you don’t already have the Guardian app, download it ( iOS / Android ) and go to the menu. Select ‘Secure Messaging’. SecureDrop If you can safely use the tor network without being observed or monitored you can send messages and documents to the Guardian via our SecureDrop platform. Our guide at theguardian.com/tips lists several ways to contact us securely, and discusses the pros and cons of each. Show more Explore more on these topics OpenAI AI (artificial intelligence) Computing Data and computer security Ethics news Share Reuse this content (267 points, 3 comments on Hacker News)