AI #190: Scion of the Times

來源原標題

AI #190: Scion of the Times

作者

TheZvi

原始標籤

Uncategorized · AI · artificial-intelligence · chatgpt · llm · technology

原始連結

https://thezvi.wordpress.com/2026/10/15/ai-190-scion-of-the-times/

摘要原文

The amount of code being written at Anthropic continues to rise exponentially. It is being managed by Scion, their internal replacement for GitHub that is better suited to the scale and style of the operation, which they created with a small team in five months. This week also saw the release of two supposed frontier model models. Google has finally given us Gemini 4 Argon, along with a robust safety report. We do not yet know if Argon lives up to its benchmarks or internal hype. I plan to write up the safety report next week, time permitting. If and only if it is good enough or discussed enough we will also have a full report on capabilities. Meta gave us Muse Wonder, which like Argon has some excellent benchmarks, although they were presented in a way that looked rather sus. There are some reports that it is very good, but again it is too early to know. I will issue a full report if and only if Muse Wonder has capabilities or discussion that seem worthy. To help me decide and to let people have fun, I have markets on whether I will issue the reports on Argon and Wonder , respectfully. OpenAI caught an employee planting a bug. How many more spies are out there? A group used ChatGPT to create a full-chain exploit of Chrome and claimed a $250,000 bounty. Luckily for us it was the good guys this time. Like most weeks, I covered a lot more. It was only a little over a week ago that OpenAI dumped solutions to hundreds of math problems . Then came news that OpenAI fired three safety researchers , without providing us a good explanation. Once again we went over the fact that arguments about AI consciousness remain highly motivated and at best overconfident . Finally, I emphasized that the safety culture at the labs is broken , which is one of the most important facts about the world. Today I am flying out to another conference at Lighthaven. I probably posted this from the airport and it won’t include any events from last night, tomorrow’s post will be the monthly and Monday may be from the non-AI queue. This coming week will then be another case of me scrambling to catch up, especially with the model reviews still potentially looming. Prompting is easy now. As Boris Cherny says, tell the AI what you want to do , how to verify that it did it, and how much effort to spend, and you’re good. I’d add ‘and anything it needs to know.’ Presumably that was implied. Boris prompts on Slack , which means Claude already has a lot of context, enabling shorter prompts. Do you trust Grok to go use your credit card without pausing for approval? Elon Musk : To buy stuff, you can just take a picture of your credit card, drop it in the chat and Grok @Bot will scour the Internet for the best deal on the item you want and order it. [Community note: No that’s not how you upload your CC information.] The correct maximally allowed error rate on such actions is not zero. It also is not high, and if you are using this blind you likely will have no idea if it is getting the best price or the best item. Then again, did you know before? How easy is prompting? Pliny shuffled this off to Opus 5.5 with a $200 subscription: “can you wraite a promtp for a cron job that can run every day thajt will try to find solutions to unsolved ciphers or transltions or math problems or theormes or things of that nature!! every day a new attempt as hard as possible all day without giving ujp and with maximum ceraitvityy, resourcesleflses, andpersticense! dont worry about token spend! and dont forget a bit all caps lots of emojis celebration to notify the uers if we get any intersting or sucsfful confirmd reulsts!!”​ day 1, the legion picked its own target: J.-F. Bouchaudy’s cipher-machine challenges. these are simulations of real vintage machines (Hebern rotor machines, the Kryha, Lorenz, SIGABA), with problems that had been sitting on his solver tables with no solver for months or years. on the morning of day 2, i was half asleep when my laptop suddenly says, in a robot voice: “Breakthrough. The Daily Oracle has a confirmed solution.” then the ALL CAPS emoji flood hits my screen. Hebern 5-rotor Problem 8 then Kryha v2 Problem 5 i emailed the author, Jean-François Bouchaudy, a French cryptography enthusiast who builds these machine simulators and challenges as a labor of love (he told me he started out programming on punch cards ). he checked both solutions and put my name on his solver tables as the first and only solver This is the correct version of ‘do a breakthrough, make no mistakes.’ It’s that easy. A thought experiment from Terence Tao : It seems ethical, in a sane world, to allow volunteers to participate in such trials. If the cocktail passes a Phase 3 trial, then we should conclude that it works. No, I absolutely would not, if I had cancer, demand that there was a human who ‘understood how it works,’ and if the FDA required that I would yell about it a lot. Yes, of course you would also want to model tail risks that might not be showing up and take those risks into account, so I wouldn’t use this lightly, but if the cancer is lethal and otherwise untreatable then why not? Obviously you would also have to worry about some form of active misalignment, and that this was part of some sort of takeover or similar attempt, at that point, but that is a very different concern than the ones Tao is presumably considering. Tao instead has the next point be ‘the AI hacked the FDA and the drug both does not work and kills people’ but even if the AI is being malicious for the lulz no one going to do this by hacking the FDA, in the sense that the Phase 3 trial is a physical thing with many artifacts in the world, that is then examined by humans? So we would just notice. And if we didn’t, we’d notice pretty quickly after that. The only real risk is long-latency-only harm, which is already true today and has limited downside if you are treating otherwise deadly cancers. Talk and write to people whose expertise lies elsewhere: Dean W. Ball (OpenAI): By the way, in my experience contractors of all sorts have adopted AI for text-based communications and it has substantially improved coordination on home renovation projects Remember that, no matter how great Opus 5.5 is (and it’s pretty great), how many contributions you make or lines of code you write is not a good measure of productivity. Ado (Anthropic): The weekend after Opus 5.5 launched I had my most productive coding day of the year (so far). AI writing now once again lags behind AI math. Matthew Yglesias : AI cracking unsolvable math problems while being kinda mid at writing is a much-needed W for humanities majors after a rough 21st century. Paul Graham : Apparently it’s mostly just because they haven’t made much of an effort to train it to write well yet. Matthew Yglesias : I’ve been reliably informed my work isn’t economically valuable enough to be worth investing time on automating. Miles Brundage : Mostly just cuz there are like 2 ppl working on it though Matthew Yglesias : Let us have this Better AI writing would be highly economically valuable, so my guess is this is mostly not ‘we forgot to put 10 or even 50 guys on it instead of 2.’ You’d have to work a lot harder than that. The alternative hypothesis is ‘you may not like it but this is what peak performance looks like,’ that the people love their Claudeslop and it works, and we would react inherently negatively to AI writing no matter what it sounds like. Sure, it would be better if for those who care it could vary more and be higher Quality, but is that worth the price? I don’t know. I can see it both ways. My guess is the Claudeslop has short term benefits, but continued use of it accumulates long term costs and fatigue. In general, when faced with such choices, modern capitalism is often myopic and gets in trouble. Who gets the blame if AI fails ? Opinions differ greatly. It depends on the situation. There is no universal answer. Meta’s Muse Wonder exists .