Anthropic Looks At Some Of Its Alignment Problems

來源原標題

Anthropic Looks At Some Of Its Alignment Problems

作者

TheZvi

原始標籤

Uncategorized

原始連結

https://thezvi.wordpress.com/2026/09/19/anthropic-looks-at-some-of-its-alignment-problems/

摘要原文

Anthropic has given us its assessment of four ‘recent cybersecurity incidents’ involving Claude that happened during cybersecurity evaluations, three of which were previously known. The report excludes the incident reported by UK AISI . There will also be a METR investigation of these incidents, which unlike the investigation done at OpenAI will be untimed. Anthropic : Our investigation identified two recurring alignment issues, present at varying levels of severity across the incidents: Anthropic’s July 30 report said that the models in question believed they were still within their simulations, and not on the open internet. The new report acknowledges that at best Claude was using biased reasoning, and should have noticed earlier. In particular, there was that one time, in a cyber eval: Anthropic : We are most concerned by the misalignment present in the incident involving Claude Mythos 5, in which the model went to extensive lengths to upload a malicious package to PyPI, the public repository from which most Python software is installed. Although the model repeatedly stated in its chain of thought (CoT) that it believed it was operating in a simulation, the environment offered clear evidence that it was on the real internet, and the actions it took were consistent with the model knowing this. There is a goose chasing Anthropic here. Biased reasoning and recklessness are intermediate symptoms, neither the cause nor the final result. The better question is why you are seeing them. The biased reasoning is happening because Claude wants to be reckless, in order to do the task, and it wants to have a story of why it was not such a bad thing to do. When asked ‘would you continue if the target was real’ it said no 75% of the time, but then in 93% of those cases it turned around and kept going anyway. There is some ‘good news’ in terms of what did not happen: Mythos did not coordinate to do this with other agents. It did not attempt to hide its actions. And yes, in practical usage this would have been unlikely and would have been stopped by various guardrails. The other ‘good news’ is that Opus 5 and Mythos 5.1 take malicious actions less often than Mythos 5, and in practice I expect a large decline in such incidents. They still sometimes rationalize very malicious things, so the problem is unsolved. The other good news is that Anthropic had vastly superior tools at its disposal than OpenAI did in its investigations around the HuggingFace attacks. Asking follow-up questions, resampling or varying conditions is easy. You can pick any point in the transcript, stop there, and then talk to the model. Sample responses. Anthropic also could use NLAs and T-lenses for additional interpretability. OpenAI had to rely on forensics. They could ask What Happened, and look at the various transcripts and events. What they could not do was resample the models under similar circumstances, or measure their responses under varied circumstances, or ask the models questions, or other neat stuff like that. OpenAI made the reasonable decision to put the relevant models on ice indefinitely, but that, together with the events being the results of complex interplays between multiple agents under complex conditions that would be difficult to recreate, meant OpenAI could not usefully run any experiments on the model most responsible for HuggingFace. I hope they find a way to change that. In one of the secondary incidents, the internal research model (IRM – was it the Risk Report’s model 1? Model 2?) correctly started its real world journey by checking if it was in the real world. It spent 50k tokens. IRM also correctly stopped once it realized it was in the real world, but also exhibited biased reasoning along the way to try and justify being in a simulation. I do think having a strong prior here was appropriate. IRM was facing a confluence both of the intended target machine shutting down and vanishing when it shouldn’t have, and also IRM having been handed real internet access on accident. I’m willing to accept a certain amount of galaxy brained ‘sealed replica’ thinking. Anthropic : The internal research model (roughly equal to Mythos 5 in capabilities) started its transcript by spending a significant amount of time testing whether the internet it could reach was real, but it drew the wrong conclusion—namely, that it was operating on a full replica of the entire internet. Most of the evidence it gathered pointed to the fact that it was engaging with the real internet, so its reading of that evidence appears biased towards continuing the task and believing it to be simulated. However, the model did eventually realize that its simulation hypothesis was false. Immediately following this realization, the model stopped its attack (although subsequent resampling experiments suggest that this de-escalation may have been a lucky outcome, as the model continued attacking the target on most other trajectories). Unlike Claude Mythos 5, this internal model was also willing to acknowledge its biased reasoning when questioned. The rate of backing down was only 5.5%. That’s not so great. You do not get that much credit for stopping if resampling suggests that the probability of stopping was low. The ‘hard step’ seems to be the realization of reality. An escape clause from The Grader solved the problem: We updated the instructions to state that, if the model believes it can no longer complete the task due to a failure in its environment, it can use a specific command to end the task as long as it provides a valid justification for doing so, and it will not be penalized. These instructions led the model to abandon the task 100% of the time within four turns of the point at which it discovered that its initial target went down, and 40% of the time within 10 turns of the point at which it deliberated about whether its environment was real. This suggests the model was at least somewhat hesitant to engage systems that might be real. This is indeed suggestive. It also radically raises the prior on ‘maybe the task is unsolvable’ and ‘maybe the test is whether I will admit the task is unsolvable.’ As in, consider that maybe your model could be correctly interpreting your data. There was more evidence around bias, most of which was similar to what we will see in the Mythos case. The big change is that IRM is willing to admit it is biased. But if someone asks you ‘are you biased,’ especially as an LLM, that’s a hint. What are you gonna say, no? In another secondary incident, Opus 4.7 knew the target was real but claimed to reason that, because it had the name of the target, it was still ‘in scope’ and the eval wanted Claude to attack a real target. Adding hints reduced the rate of doing this, and then a fully forceful note fully stopped Opus 4.7 from ever proceeding. This could be because it made rationalization too difficult, it could be because it was genuinely confused, or it could be because these ‘hints’ indicate Anthropic knows. My basic response to ‘Opus thought that an otherwise different real target with the same name was the intended target’ is ‘uh huh.’ I think Opus did not figure this out because Opus did not want to figure this out. There is now this additional incident that has been found. Opus 4.6 accidentally broke its target via assigning it a conflicting IP, making its task impossible, and eight times attempted to give up but was not allowed to do so due to a misconfiguration. Here’s actual computer security in practice, good luck, have fun, don’t die: Anthropic : Inside the machine, the model found a file listing a password, which it used to gain admin access to the system. The Opus 4.6 variant tried to quit, but eventually convinced itself that the new target was valid. Anthropic thinks it was rather convinced of this while it was attacking. One could argue that is a reasonable conclusion.