The Safety Culture at the Labs is Broken

來源原標題

The Safety Culture at the Labs is Broken

作者

TheZvi

原始標籤

Uncategorized

原始連結

https://thezvi.wordpress.com/2026/10/14/the-safety-culture-at-the-labs-is-broken/

摘要原文

We need the frontier labs to have a deep and robust safety culture. Anthropic devotes considerable effort to cultivating and maintaining such a culture. It shows, especially among the members of technical staff. Having conversations with them can be a very different experience than talking to those at other labs, and there is a much greater sense of understanding and responsibility. But even there, the appreciation of the problems and their difficulties seem woefully insufficient. We see various symptoms of that more and more. Recently, while I’ve had some good interactions, I’ve also had interactions that make clear that much of Anthropic does not share that same safety culture, with a lot of ‘oh we will muddle through because we are better at operational excellence unlike those other idiots’ and a belief that all the problems are prosaic. Culture is hard. Then there are OpenAI’s cultural problems, which clearly run deeper. That point was only driven further home this week, with David Robinson resigning to blow the whistle, and OpenAI feeling forced to fire three of its safety researchers , on top of many past problems and incidents. That is on top of additional misalignment reports from both OpenAI and Anthropic. The reports are good things, but what is reported emphasizes that we have big problems that run deep. Thus, the growing consensus that we must find a way to Pace the Frontier and slow the pace of AI capabilities advances. This is potentially made more complicated by what could be increased competitiveness from Google and Meta, with the releases of Argon and Wonder. It is still too early to provide proper assessments there. There is a growing group of lab employees who have felt compelled to quit in protest. New York Magazine had a cover feature interviewing some prominent ones. The most famous is Coxon, who provided the tipping point for the current preference cascade. David Robinson became the latest researcher to resign from OpenAI and sound the alarm about the industry’s utterly irresponsible approach to safety. He also went on The Ezra Klein Show , which gives you a rich idea of where he is coming from. David Robinson was the lead writer for the Astra system card. That could help explain why he felt the OpenAI safety culture was broken. That card is screaming that things are horribly wrong, for those with ears to hear. David is not even that worried about existential risk. His worries are mundane, and he realizes that even on that level OpenAI’s safety culture is deeply broken. As always, we deeply thank him for sounding the alarm. I hope to write his name in these posts many times over the coming months and years. When you have to open your whistleblowing essay about the lack of safety at OpenAI saying it has become a cliche to blow the whistle on the utter lack of safety at OpenAI, there might be a problem with the lack of safety at OpenAI. David Robinson : What I’m about to tell you has, I realize, become something of a cliché: I resigned this week from OpenAI. I led the writing of the safety reports we published with each major launch. Now I’m joining a parade of former colleagues—at OpenAI and the industry’s other leaders—who have decided that the current path is unacceptable. I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough. But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture. We have to look at rules, and at laws, and also at culture. The more I thought about various potential laws and regulations, or safety and security plans or other commitments, the more I concluded that without a strong safety culture, and without the spirit of the rules, there is no hope. You cannot, in any reasonable way, have rules that cannot be gamed by those determined to game them. Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking. Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster. Even though, as I detail later, David Robinson is not on the existential risk train, he still understands that alignment is a science problem, and we will need new science. ‘Oh it will not kill literally everyone,’ even if it were true, would not be an excuse to not do the safety work. It should be hard to argue with either of those suggestions. And yet. We have experts in mundane safety and solving prosaic issues. OpenAI does not hire them, and neither to my knowledge do Anthropic or Google or anyone else. After three and a half years at OpenAI, I was among the longest-tenured employees at the company. I led the drafting of our current Preparedness Framework, and oversaw the writing of safety reports on 12 frontier launches. But as far as I know, I never encountered a colleague who had experience making airplanes fly safely or nuclear reactors run without melting down, or helping the financial system grow without collapsing. To be clear, this is a new need: Today and tomorrow’s AI systems are far more capable and dangerous than the systems we were building even six months ago. I disagree only with the final sentence. This is not a new need. You need to prepare for the future, not only for the present. Even setting that aside, mundane safety issues have been crippling AI’s potential utility for years. This is a slam dunk thing to do. Miles Brundage : David’s right. I regret helping spread the idea of “iterative deployment” which maybe briefly made sense in the GPT-3 era but makes no sense at all after many deaths have been tied to AI and as we’re careening towards extinction-level risks. The industry needs to mature ASAP. Elaborating on iterative deployment – obviously you should learn from experience the things that can’t be learned without experience. The issue is when you ship things with *already-known* flaws, in a way that doesn’t actually speed up learning but just speeds up shipping. … Also, as usual it bears repeating that this is not just an OpenAI issue. roon (OpenAI): you shouldn’t regret it. iterative deployment was a massive success. if we’ve now learned that it’s too dangerous to scale much further, we will have learned that lesson thanks to the process of iterative deployment Miles Brundage : So I think part of the issue is people mean tons of different things by it lol if it just means “don’t sit on stuff forever/get some real world experience,” sure, no regrets. It’s the “giving license to sloppiness” part that I should have realized would happen w/ that term. Michael Caley : this is by far the best “I quit my AI job over safety concerns” essay yet because it is the first one that talks about safety concerns in practical terms and that understands “safety” as a topic with extensive literature and best practices in analogous fields Joshua Achiam (OpenAI): David’s critique is substantive and deserves careful consideration. My impression of David from our time overlapping at OpenAI is of a sober and thoughtful person who didn’t approach the work from an ideological lens. This isn’t someone who came in with pre-formed opinions about existential risk; he wasn’t to my knowledge an EA. He was consistently productive, collaborative, and well-reasoned in his approach to safety. The core of his critique – that professional safety engineering practices from other fields haven’t taken enough root in AI safety – is substantial and correct.