Noam Brown – Agent swarms, alignment, & recursive self-improvement

來源原標題

Noam Brown – Agent swarms, alignment, & recursive self-improvement

作者

Dwarkesh Patel

原始連結

https://www.dwarkesh.com/p/noam-brown

摘要原文

We talk about multi-agent, Navier-Stokes, and what the current explosion of maths progress tells us about what happens once you automate AI research. And we also discuss how we will know if the models are actually aligned before we kick off RSI. Watch on YouTube ; listen on Apple Podcasts or Spotify . Jane Street has been interested in AI for a lot longer than you’d think, and not just for trading. In 2011, a full year before AlexNet and over a decade before ChatGPT launched, they hosted the first FOOM Debate between Eliezer Yudkowsky and Robin Hanson on whether AI would lead to an intelligence explosion. Now Jane Street is revisiting the question with a new panel: Daniel Kokotajlo, Ege Erdil, Ryan Greenblatt, and Jaime Sevilla, hosted by Ron Minsky in San Francisco this October. I expect it to be a truly excellent conversation. Register at janestreet.com/dwarkesh Grok Bot has made handing off work super easy. It runs on its own cloud computer, where it installs the tools it needs to handle tasks end-to-end. For the podcast, we use Grok Bot to help produce our videos. You may have noticed that our ads feature animations of real websites. Getting these pixel-perfect used to mean running a convoluted, multi-step workflow ourselves. Now we just let Grok Bot handle it. Best of all, Grok Bot has learned all of our specs and preferences, so we don’t have to redescribe the task each time! Try Grok Bot for yourself at x.ai/bot Antithesis gives you the confidence of a giant test suite without actually having to write one. Say you’re doing a major backend refactor: building enough tests to trust it could take weeks. Antithesis solves this by running your software through countless simulated worlds, injecting faults and hunting for failures. On any PR, you can turn a dial to decide exactly how much testing you want. And because every run is fully deterministic, agents can branch off the moment a bug appears, rewind it, inspect memory, and replay it, all while the original test keeps running. Learn more at antithesis.com/dwarkesh (00:00:00) – Multi-agent and Navier-Stokes (00:15:28) – How will AI firms work? (00:22:02) – What math progress tells us about recursive self improvement (00:40:22) – Hugging Face and alignment (01:01:18) – The internal/external model gap (01:08:34) – Chain of thought is degrading (01:14:12) – How will we know when alignment is solved? Today, I’m chatting with Noam Brown , who is a researcher at OpenAI. He was one of the foundational contributors to what became o1 and the reasoning models. Now he’s working on multi-agent systems . Speaking of which, you guys announced last week that you solved one of the Millennium Prize Problems with a system of 10,000 different AI agents that spent 130 billion tokens over 88 hours. One of the reasons I’m interested in talking to you is that you were among the first people, maybe two or three years ago, who were thinking about how the reasoning models would allow us to see into the future. Because if you scale up inference compute, you can see what the base capabilities of the models will be a few years in the future. I feel like you’re in a similar position now to help us understand what future capabilities will look like, given the enormous scaling of agent sizes that we can do right now. The way I think about it, when you plot the performance of these reasoning models with test-time compute on the x-axis and performance on basically any reasoning benchmark on the y-axis, you see a very clear pattern where the longer these models take to think about their answer, the better they do. This is a very natural thing. It’s the same thing with people. If you’re taking the SATs and you have five minutes to go through the entire exam, you’re not going to do very well. If you have five hours, you’re probably going to do a lot better. The AI models are pretty similar. They’ll spend that time doing this monologue to themselves, figuring things out, going through different cases, ruling out different possibilities, building on some of their previous discoveries. The problem is that as you push that further and further, you hit a latency bottleneck. You don’t want to sit around for three years waiting for a response. So what you can do is what a lot of people do. They parallelize. They just get a team of people. If you’re going to found a company, you want to get a group of people together so you can go faster. It’s the same thing with these AI models. It helps to just have multiple agents working on something because they can go faster. So multi-agent is a way of scaling test-time compute in parallel instead of purely serially. It is less efficient, because it’s not like a single agent has all the context to itself. But it is a very effective way of scaling test-time compute if it’s done well. I’m going to ask a bunch of naive questions. This is an unreleased model, so we haven’t publicly seen how these systems work. I just have a bunch of ways in which I’m confused about what the qualitative properties of such systems are. I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. Think about what 130 billion tokens are. If it were a single human thinking as a full-time job, stretched back to back, 130 billion tokens would be a human thinking for 4,000 years. Eight hours a day, working a normal work week. Starting from ancient Sumeria up till today, a single sequential human thinking that long, concentrated in 88 hours. I feel like qualitatively, that is a super important consideration. I’m surprised that there isn’t a bigger parallelization penalty. You can just have 10,000 agents collaborate. Maybe because the agents are better at collaborating than humans might be, they’re going much faster. They can actually productively collaborate at such a big scale. Or maybe there is a big parallelization penalty. Let’s talk about the parallelization penalty, and then we can talk about the qualitative stuff. The truth is that we don’t have very good science on multi-agent scaling up to this kind of scale. When we released 5.6 , I think that was the first time that we had a proper multi-agent system in our models. We actually did show some plots in the blog post of the scaling performance of multi-agent systems, because we have it as an option. It’s Ultra Mode. The default is four agents, but you can set that higher. In the plot, we show what the performance looks like on some benchmarks for one agent, for four agents working together, for 16 agents working together. It depends on the benchmark, but for some of the benchmarks, what you see is that if you have four agents working on the problem, it is done twice as fast. Because there are four agents working for half as long, you’re paying 2x more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern. It’s a little less efficient, but you continue to see that performance. Is it a linear serial time speedup or a sublinear speedup as you increase the number of parallel agents? It’s slightly sublinear, though it does depend a lot on the problem. Math, for example, is quite parallelizable. It’s not the most parallelizable thing, but it is very parallelizable. Web search, things like doing a Deep Research report where you have to look through a bunch of sources, is extremely parallelizable. I suspect that something like writing a novel would be very unparallelizable. You would probably not see a big benefit from having 10,000 agents working on a novel together, in the same way that you’d probably not get a big benefit from having 10,000 people work on a novel together. So the performance does depend on the domain. We do measure it up to 16 or so agents in our published blog posts. The problem is that it’s very hard to push that science to 10,000 agents because it’s just so expensive. You guys just did it over a weekend. But that’s one data point.