Raw transcript — OpenAI researcher on agent swarms & recursive self-improvement (Noam Brown)

Today I'm chatting with Noam Brown who is a researcher at OpenAI. He was one of the foundational contributors to what became 01 and the reasoning models and now he's working on multi-agent systems. Speaking of which, you guys announced last week that you solved one of the Millennium Price problems with a system of 10,000 different AI agents that spend 130 billion tokens over 88 hours. One of the reasons I was interested in talking to you is I think you were one of the first people uh maybe two or three years ago who was thinking about how the reasoning models would allow us to see into the future because if you scale up inference compute you can see what the base capabilities of the models will be a few years into the future and I feel like you're in a similar position now to help us understand what future capabilities will look like given the enormous scaling of agent sizes that we can do right now. So the way I think about it um when you plot the performance of these reasoning models with test time compute on the x- axis and um performance on basically any reasoning benchmark on the y- axis, you

[00:01:00] see a very clear pattern where the longer these models take to think about their answer, the better they do. And this is like a very natural thing. It's same thing with with people. >> If you're taking the SATs, you have five minutes to go through the entire exam, you're not going to do very well. If you have five hours, you're probably going to do a lot better. the AI models are pretty similar and and they'll spend that time doing this monologue to themselves, figuring out, going through different cases, um, ruling out different possibilities, building on some of their previous discoveries. The problem is that as you push that further and further, you hit a latency bottleneck. You don't want to sit around for 3 years waiting for a response. And so what you can do is what a lot of people do um, is they paralyze. They just get a team of people. You know, if you're if you're going to found a company, you want to get a group of people together so you can go faster. It's the same thing with these AI models that it helps to just have multiple agents working on something because they can just go faster. And so multi- aent is a way of scaling test compute in parallel instead of purely serial. And it is like less efficient because it doesn't have it's not like a single

[00:02:00] agent has all the context um to itself. But it is like a very like a very effective way of scaling test on compute if it's done well. >> Okay. I I'm going to ask a bunch of naive questions because these systems so this is an unreleased model so we haven't publicly seen how these systems work and so I just have a bunch of ways in which I'm like confused about like what the qualitative properties >> of such systems are. I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. So, if you think about what 130 billion tokens are, if it was a single human thinking as, you know, a full-time job stretched back to back, 130 billion tokens would be a human thinking for like 4,000 years, you know, 8 hours a day or something, working a normal work week. So, for starting from like ancient Samria up till today, a single sequential human thinking that long concentrated in 88 hours, I I feel like qualitatively that is a super important consideration. And I'm surprised that there isn't a bigger parallelization

[00:03:00] penalty that you can just have 10,000 agents collaborate and because maybe the agents are better at collaborating than humans might be um they're going much faster that they can actually productively collaborate at such a big scale or maybe they I don't know maybe there is a big parallelization penalty. >> Yeah, let's talk about the paralization penalty then we can talk about the qualitative stuff because the truth is that we don't have very good science on multi- aent scaling up to this kind of scale. >> So when we released 5.6 six. I think that was the first time that we had uh a proper multi- aent system in uh in our models and we actually did in the blog post show some plots of the scaling performance of multi- aent systems because we we have it as an option. Uh it's ultra mode and the default is four agents but you can set that to higher. And in the plot we show okay here's what the performance looks like on some benchmarks for one agent for four agents working together for 16 agents working together. And what you see and it depends on the benchmark but but for some of the benchmarks basically if you have four agents working on the problem it is done twice as fast. So you're basically paying because there's four agents working for half as long you're

[00:04:01] paying a 2x more to get an answer uh twice as quickly. If you go to 16 agents, you see a similar pattern. It's like less a little less efficient. Um but you you continue to see that that performance. >> Is it a linear serial time speed up or a sublinear speed up as you increase the number of parallel agents? >> I would say it's um slightly sublinear though it does depend a lot on on on the problem. >> So math for example is quite paralyzable. It's not the most paralyzable thing but it is very paralyzable. I think web search things like doing a deep research report where you have to like look through a bunch of sources that's like extremely paralyzable. >> I suspect that something like writing a novel would be very unparalizable. >> So you would probably not see a big benefit from having 10,000 agents >> working on a novel together. Um >> in the same way that you' probably not have a big benefit from having 10,000 people working on a novel together, >> right? >> So the performance does depend on the domain. We do measure it up to 16 or so agents in in our published blog posts. Um, the problem is it's very hard to

[00:05:02] push that science to like 10,000 agents because it's just so expensive. You guys you guys just did it over like a week. But that's but that's one that's one data point like we we don't know how long it would take a single agent to solve because we haven't we haven't done that experiment yet and and maybe we will but I mean that's also only one data point right and we want to do if you want to do a thorough ablation >> um it's actually the experiments are just too expensive to go to that scale >> so we have to do some kind of like methodical science about like what happens when you go to like 64 128 256 or something and get a sense of the behavior but it's going to be very hard to push that all the way to like 10,000 know for sure what was the um the benefit that we actually got for using 10,000 agents versus a thousand. >> Yeah. >> One thing I want to make clear is that the effort to get it's not like the effort to get a Millennium Prize problem. This was not due to multi- aent. I wouldn't even like attribute like 10% of the the credit to to multi-

[00:06:00] aent. Like the reality is we've trained a OpenAI has trained like a very powerful model. Um, and we can get that model to operate over very long horizons. We can get it to think u in parallel. Uh, but at its core, the reason why we're able to do this is because we just have a general purpose very strong model. And I think things like multi- aents are flashy and are new and that probably gets disproport disproportionate credit for that reason. But the the core reason is like this is just a very powerful model. Mhm. So the generalization is actually quite shocking to me. The US systems I mean we don't I don't know how these systems were trained but presumably they were trained how RL training happens. You have a bunch of checkable synthetic problems. You do a bunch of RL against them. And nowhere in the training process I'm guessing was like the model solving anything as ambitious as a Millennium Prize problem. But the generalization was strong enough that like you could have this like much easier verifiable problems generalize to

[00:07:00] this much parallel effort on such a hard problem. I think that is true like first of all we do train the model on on very hard problems. So there is definitely a gap like we see if we train on on some kinds of tasks like it's able to do tasks that are more ambitious than that. there is an interesting challenge that as the models become smarter and smarter um the kinds of questions we can ask them just a lot of them are too easy >> and um it's hard to challenge the model and I I do think that that's going to be an interesting like if I had to make an argument for why you might not see AIS like LMS go the same path as Alph Go and Alpha Zero and all these kinds of like gameplay AIS it might be that this this kind of problem that in things like Alpha Zero where you have selfplay, you have an infinite curriculum. You're always playing against an AI that's like equally strong. Whereas for things like training an LLM with reinforcement learning, at least the ways that are out

[00:08:00] out there right now, um you give the model a problem and you ask it to solve it. And if the problem is so easy that could just solve it in a second, it's not really learning anything. >> Yeah. So if we're if we run out of problems to ask it that that challenge it then that is a plausible scenario where actually like okay it becomes much harder to make progress. Yeah. >> Now I do think there are ways around that and um so we haven't really hit that uh as as a wall yet. Um and I think that if it ever became a serious problem there would be ways around it but it is like a plausible scenario. >> Yeah. Um, and just for the audience when when you're referring to like Alpha Go or Alpha Zero, you're saying like getting superhuman relatively fast after achieving human level performance. >> Yeah. I mean, if you look at the trajectory of gameplay as like Go, >> Yeah. >> they within a span of like a year went from beating a European chess champion being like, I don't know, like number 50 in the world to beating the world champion to being unimaginably orders of magnitude stronger than any human alive. >> Yeah. And it's possible that domains

[00:09:03] like math we see a similar trajectory but I think there is a very plausible scenario where actually that doesn't happen. >> Yeah. Yeah. Yeah. So I I I want to understand if in 6 months people will have access to multi- aent systems how should one model what it is like to collaborate with or hire a multi- aent system. >> Yeah. I should start by talking about how these multi- aent systems actually work, which is I think a very different way than a lot of multi- aent systems and other AIS. So, a lot of people that have approached multi- aents for things like LMS tend to take this like very scaffolded approach where, you know, for example, there might be a uh coordinator agent that delegates work to a bunch of children and gives them a task and then children work on it and then return their answer. Um, and this seems like a very sensible setup, very sensible scaffold. It definitely helps, but there are a bunch of limitations with these kinds of setups. So, for example, if in this setup you have um a coordinator that's sending tasks to children, the children

[00:10:01] work on it and return their answers. Well, what happens if two children are given similar tasks? Can they talk to each other? And usually the answer is no. Um, and that's that's very inefficient, right? If you're if you're given a task and it's actually really helpful to talk to somebody that might know an answer to a question that you're working on or a part part of something that you're working on, it would be really helpful for you to just be able to ping them and say, "Hey, can you help me out with this thing?" But a lot of systems don't have that setup and adding it just increases the complexity significantly to the scaffold that you have. Another thing is like what if the child doesn't really understand or has a clarification question. So it has to choose then between okay do I just return and ask the question instead of giving instead of solving the problem or do I solve the problem just like make an assumption about what the parent wanted me to do and uh and just solve it that way and so in any scaffold that people come up with there's always limitations involved and the approach that we wanted to take was to just go toward the

[00:11:02] extreme end of baking in as little structure as we could and give the agents very primitive tools to use and and figure out for themselves how to use it effectively. So we give the agents the ability to message another agent and when it messages another agent it is inserted into the context and then it it can do like a few other similar things. Um but that's that's basically the the core of it that it can just send a message whenever it wants just a tool call. um and it can send that to other agents and they figure out for themselves the best way to coordinate around that. And uh it turns out that if this is done well, you get very sophisticated behavior. And to me, it looks a lot like how human collaborators work over something like Slack, for example. When we were working on this project, it was really exciting when we finally got it working to see these agents working on problems together. I I remember one one example. So, we give the agents a problem and then one agent says, "I

[00:12:00] think I've got the answer." And then another agent says, "Actually, I got a different answer." And then they have this whole discussion about, "Well, how did you arrive at that answer? Can you explain it to me?" And going back and forth and trying to clarify like what could have been wrong in each other's reasoning. And then they finally converge and like, "Oh, yeah, okay, that seems right." And then it just broadcast to the other agents like, "Actually, I've changed my my answer. I think I think he's right." >> And it just felt like a very natural conversation. Um, yeah, it's kind of felt like when you see chain of thought for the first time that's trained through through reinforcement learning and you're like, "Oh, this is this is just kind of like what a person would think if they were writing down their thoughts um as as they're thinking them." >> It kind of felt like that. >> So, it is it is really cool to see this kind of behavior. And so, I think collaborating with these things, honestly, it feels a lot like collaborating with a person. Yeah, >> it's a very natural flow >> except one qualitative difference that might become salient in the future is that the these systems will be thinking I don't know more than 10x as fast, right?

[00:13:00] If you just look at how many tokens per second they output versus how fast a human talks and they're working all the time. They're not sleeping and they're collaborating with each other at a much more intense pace than humans have the capacity to collaborate with other humans. So I I I I quote I'm trying to think of what to qualitatively expect in a year and is it like a sort of shadow organization that is moving 100x faster in my company than the human level is that's you know like the iteration cycle is much faster what would take a human organization a year to do is happening you know within a week within this like uh the shadow organization. So, will it feel foreign? I don't know. I've actually found that it's pretty surprisingly natural. Yeah. To work with these things right now. I think that could change. So, for example, we have these like ultra fast modes that enable sampling to be like 10 or 15x faster or whatever. And then like, okay, it's going to be pretty hard to keep up with these things. >> Um, I think the idea is >> these agents when they're communicating with each other, yeah, they can go super

[00:14:00] fast, but then also they understand when they're talking to an agent versus when they're talking to a person. Yeah. and and their behavior will be different in those situations. >> Yeah. Yeah. So the main example that we have publicly of sophisticated multi-entation systems is unfortunately the hugging face one. And the thing I found interesting there I mean um a lot of things I found concerning obviously but like the thing I found interesting is just like the spontaneous emergence of hierarchy of like middle management and it's it sounds like you're saying like this level of organization sort of emerges spontaneously from training. I think the details are spontaneous but I mean just because we're giving a lot of flexibility to the agents to decide how to communicate with each other in the optimal way. It doesn't mean like we we are still giving them a starting point. We're giving them a prior about oh this is what reasonable communication might look like. They also I mean they they're trained on a lot of human text. They have an understanding of how humans organize and coordinate. And so that's all kind of baked in. I think it is

[00:15:00] surprising the way they're able to polish this. If you if you look at what it starts out at, it's not very sophisticated behavior. In fact, it's actually very difficult to get these agents to coordinate in a productive way because it's just like very tempting for them to just collapse to, oh, we're all just going to solve the problem independently. >> And like >> that is a local minimum that you can get stuck in. Um but yeah, if it's done well, they can end up coordinating it very effectively in in these kinds of like very structured ways. I wrote this essay a couple years ago called um something something AI what automated firms will look like and I was thinking about well if you had fully automated firms of let's say human level intelligences what is different about the nature of AI minds that would make the organizations AI form different and there are a couple very important differences for example that AIs can share context much more seamlessly than humans can they can merge their knowledge much more seamlessly and also you can spin up or spin down an arbitrarily arbitrary number of

[00:16:02] instances which have the right knowledge. So if you want to hire more people, it's not like just all the schle of finding the right talent or whatever. It's like the your best talent, you can just make an infinite copy of them or if you don't need them for the task anymore, you like spin them down and you can just yeah replicate the most effective parts of your organization or replicate whole organizations together which are effective. Um I don't know h where do you see these multi- aent systems going a year from now or two years from now. >> I think it's a great question of like how do these things actually differ from working with a human corker and I think there I think you highlighted some like one really interesting thing is that I mean if somebody's if you have a person and you want just like two copies of them you can't just like clone the person. >> Um but with AIS you it's actually really easy to just say like okay we'll just fork yourself and then have those copies work on this thing and then like merge back together. I mean we already have this I think in multi- aent for um for Astra and and and 5.6 soul that uh when they spin up sub agents like the context is just forked so it has all the context that's relevant. Um there are other

[00:17:01] interesting ways where the agents will differ from people like what are some reasons why startups disrupt incumbents >> like there there's a few factors uh one is that they're willing to take more risks. Um but another major factor is like >> as companies grow in size as organizations grow in size you see increasing misalignment between the the individuals in the in the organization. Yeah. >> Right. Like if you have a startup with five people and each person has 20% share in the company, they're all highly aligned to the company succeeding. If you have like a massive company with 10,000 people, you see a lot more instances where people are territorial or like just care about getting a lot of headcount for their project or their team um or like you know building their FFTs, getting a lot of resources so that they can publish cool work or whatever and get promoted and this is actually a real detriment. I think this explains a lot of why startups are able to disrupt incumbents and it it's interesting that

[00:18:00] I mean it's it's true that AI does help startups in a way like it's much easier than ever before for one person to step in and be like I'm going to make a multi-million dollar company like it's just the AI amplify an individual so much but there's also an argument that they could benefit incumbents because if the alignment problem is solved then you don't have the issue of misalignment between individuals in the company. Like at least that's mitigated. Like the AIS if they're fully if they're if they're aligned well, they could just be aligned to the interests of the company and you can have 10,000 of them and they're all going to be working as hard as if they were like a 20% share co-founder. >> Yeah. And it's not only that, but it's also that they are much able better able to like manage shared memory and context than different humans can. If you have a if if like tomorrow you hire 10,000 mathematicians and you're like solve this uh solve Navier Stokes, they're not they're not going to be able to like cooperate effectively. Um, at least not off the bat. Uh, but you can have apparently 10,000 AIS.

[00:19:00] >> Well, again, I want to be like I want to be conservative here because we haven't measured how effective >> the 10,000 agents are at coordinating. Uh, we think it helped. We don't actually have good measurements of saying like, oh yeah, this 10,000 agents led to like a 2x speed up over 2,000 agents or something like that. Um, and it's it is actually I would argue likely maybe I don't know about likely, but I think it is very possible that 10,000 humans are better at coordinating than 10,000 agents right now. >> I think that is a very I think is entirely possible. >> Yeah. Yeah. Yeah. I >> I think also one trend we've been seeing is like look, we've been working on multi- aation for a while and the early versions of this it's very difficult to get right. It's very hard. It was very hard to get the agents to even talk to each other. And it's because like look >> the we >> when we first developed reasoning models like they they weren't talking to other agents. And if now you put a bunch of agents together and say like oh solve this problem together. They're in this local minimum where they're really good

[00:20:00] at thinking deeply about a problem. And um it just like kind of interrupts their chain of thought. that interrupts their flow to constantly be like checking in with other agents or like receiving messages from them. And the authorization is actually very hard to get right in that situation. >> Interesting. Is is it is it getting the cold start of like getting the first collaboration or like what's the what's the issue? >> I mean, I think it's that they're not very they're not as general like the earlier models were just not as um generalizable >> and were just more narrow. >> Interesting. >> As the models have become more capable, it's been easier for them to develop this capability. And I do think that as they become stronger and stronger just across the board that they will be like become better at organizing themselves in large organizations and >> like and I don't know maybe they are better than people at organizing in 10,000 person groups. Um but if even if they're not you know a year from now two years from now like yeah it's quite possible they'll that they'll do that even if we don't end to end optimize them for that. >> Rockbot has changed the way that we produce our videos. For example, you may have noticed that a lot of our ads have these animations of real websites. One

[00:21:02] of my editors uses LLMs to make them, but it's not currently straightforward to have an AI create pixel perfect animations of specific websites. We've tried. It doesn't really work that well. So, we've cobbled together a pretty convoluted multi-step workflow. And up until recently, we had to run every step ourselves. Now, we just let Grockbot handle it. Grockbot starts by opening the website that we want to animate. It uses a specific extension to download and open the page in Figma. Then he uses Figma to convert the whole thing into an SVG file. This saves the AI from having to draw the whole UI from scratch and tends to result in higher quality animations. Rockbot runs this whole process on its own cloud computer where it's installed all the tools it needs to run this whole process end to end. And it's learned our video specifications and preferences. So there's no need to redescribe the whole task every time we want to make a new animation. This does feel like the new way that we'll be interacting with AI over the next year. Agents with their own computer who can autonomously handle bigger and bigger chunks of your work. You can try Grockbot at x.ai/bot.

[00:22:02] Okay, so here's why this result and maybe the general progress that AI has made in mathematics has made me think that RSI is more plausible and sooner than I previously thought. I feel like we've gone in mathematics from let's say in 2024 you you have AIS who are like oh okay interesting they're like doing they can like solve a couple problems on high school math competitions and then in 2025 it's like oh wow they can get gold in like international math olympiad and earlier this year I was like wow they're actually solving open problems in mathematics like open nerves problems but maybe like I don't know people didn't weren't trying that hard and it was just like there was a similar solution somewhere in the literature and now I just think it's sort of undeniable right it's like this is the millennium prize problem there's there's really no there's no story of why this should have been easy. Now, a lot of people pointed out I think Terry Taw had a post like this. Toby Or wrote an interesting post about this that they're solving all these problems, but they're not like coming up with at least we're I'm not aware of them coming up with new insights or formulating insightful new

[00:23:00] questions and new modes of uh theory for thinking about mathematics like coming up with like topology or coming up with um the cartisian grid or something. And so maybe like the actual progress in mathematics broadly construed is smaller than it might seem if you're just looking at well scoped problems that are directly solved. However, I think that that kind of progress would be incredibly meaningful in ML because in ML you're not you don't care about like better understanding the nature of deep learning or you only care about that as a instrumental goal uh towards the broader sense of like just achieve the result just solve this like well scoped problem of improve the sample efficiency of our models like improve the pre-training loss. So the kind of progress that we're just seeing arrive like an avalanche in mathematics is structurally actually very similar to again I'm curious if this is the case. I'm just total outsider. I'm wondering if it's the case that it's like structurally very similar to the direct

[00:24:00] uh uplift that you would expect in AI progress. And then the thing that's shocking to me or uh concerning potentially is just like how fast we went from oh it's like they're giving me 50% uplift if you're a mathematician to wow they're just like end to end solving the biggest open problems in the field. >> Yeah. Okay. So there's a lot to unpack there. Um let's start with the progress on math. So yes the models are doing some crazy powerful stuff and it's happening. It's progressing faster than I expected. did I mean when we got IMO gold in 2025 I thought okay basically what I thought is like the models when they were doing GSMAK when they figured out how to do GSMK that was that would take a human mathematician about 5 seconds to do a GSMAK problem. So this is grade school math grades K through 8. And then the next year they were able to do the math benchmark problems. And these would take uh a human mathemat like an expert human mathematician maybe like uh like a minute to do. And then you get

[00:25:00] to Amy and this is the qualifier for the USA mathematics Olympia team. It would take a human mathematician like a good mathematician probably like 10 minutes to do. And the models were able to do that a year later. And so every year you're seeing this like 10x increase in the task they're able to do in terms of like length of how long it would take a human mathematician to do it. And then it was very sensible that a year later we get to IMO gold because that's 100 minutes. That's about how long it takes a human mathematician to do an IMO problem. And just projecting outwards I was like okay how long would it take a a person to solve uh something like a millennium prize problem? And I I mean I don't have a good sense but I if we are following this trend line of like 10x every year uh we go from IMO gold which is taking an hour and a half to next year 15 hours and that should not be enough to solve a millennium price problem. And so I was like yeah I don't think we're going to get it in you know in 2026 probably not in 2027 maybe in 2028. Um so it did happen a lot faster than I expected. Now,

[00:26:02] I think there is a narrative going around that, oh, these things are replacing mathematicians, that it's just superhuman in mathematics across the board. And I think that is the wrong takeaway. They're clearly exceptional in some ways, but they are weaker than human mathematicians in in other ways. So, we have this like jagged scenario where the models are like brilliant in in some dimensions and also weaker than humans in other dimensions. And yeah, like you said, they're not very good at posing new problems. they're not really good at understanding like what is really worth ex what what directions what whole branches of mathematics are worth exploring or developing and my opinion is that I think this is great like I would I would be I would be thrilled to live in a world where um AI is a compliment to human abilities and is allowing us to discover new knowledge without fully fully replacing people like that is that is the best case scenario >> but you don't expect that to actually continue >> I do I I think it's true that the AI are jagged but as they get better, they get better across the board.

[00:27:00] >> And so I think that the things that they're exceptional at, they're going to get even more exceptional at. The things where they're far behind humans at, they're going to be less behind humans at. And over time, it is possible that they're just better across the board. Now, I don't know how long that takes. It depends on depends on how long the long tail is of things that they're bad at. So, I guess this brings us back to RSI. Um, and again, I I I want to emphasize here that I'm like just total outsider. I'm a podcaster, but I'm just trying to reason or like as somebody interested and concerned about what's happening in the field, I'm trying to reason about when or to expect RSI and what kind of thing to expect. Yeah, I feel like that yeah, the the amount of cognitive effort that was dumped into this million price problem is a good intuition pump of you could have um AIS that are spending uh over the course of maybe a week more cognitive effort on a longstanding ML problem like you know very fluid online learning they could spend more effort in that week than the maybe the field has spent cumulatively in its entire existence and then you could say well of unlike mathematics of

[00:28:00] course AI requires errors, experiments, and that takes compute and that takes time, right? You can't just like think on pen and paper and actually make things happen. But if you just look at the amount of um compute that is like available at an organization like OpenAI, right? By the end of next year, OpenAI will have enough compute that if you had, you know, the 10,000 agents or if it took 10,000 agents with the million prize problem, you have like 10,000 agents at the end of next year, they're much smarter by that point. And each of them will have enough compute to run a GPT3sized experiment every single day. Um, I don't know. That seems that seems like a lot for like superhuman researchers who thinking super fast. What do you think about the intuition pump? >> I think it's uh I think it's pretty accurate that look I mean yeah these things are very spiky and if when it comes to mathematics they're like way better in some ways but they're also worse in other ways. But the ways that they're spiky end up I think probably being particularly useful for things like RSI. >> Yeah. >> And you know you have a more clear objective. There's it's just like more measurable. Well, it's more like there's less question of like well what what new

[00:29:01] branches of mathematics are worth exploring like no there's a very clear answer. It's like certain metrics that you care about and if you can make it do better on those metrics then you succeeded. So I think there is there is a lot of truth to that and um I think the main difference is that mathematics you're you're purely bottlenecked by thinking and no external like yes there are some brand there are some parts of mathematics where you care about running experiments and getting results and these kinds of things. Um, but for the most part, it's just really bottleneck by thinking really hard and the models are really good at that. When you look at things like RSI, you do have to run experiments. So, it's you it's um it's not enough to just be extremely smart. And I think one argument for this is if you had like 100x less compute and all the most brilliant people in the world working at OpenAI, um, how much progress would be making relative to having the amount of comput that we have now with the amount of people we have? I suspect it would be less progress actually. >> Well, how much less? >> It's unclear, but I think it it would definitely be less. I think a lot less,

[00:30:01] >> like 100x less. >> No, not 100x less. Yeah, but I mean, okay. So, like I guess the question you're getting at is like, okay, if we have RSI and we have um all of these brilliant AIs running around running experiments and stuff with the comput that we have, how much faster does progress go? And I think this is this is somewhere something where we disagree on. >> I think that we do see a speed up and I think we see a significant speed up. Um, but I don't think it's like an overnight intelligence explosion that we go like 100x faster because I think that we do get bottlenecked by certain certain limitations that are not bottlenecks of intelligence. It's running experiments. It's running experiments serially because they take a while to either train new models or to get the results. Um, it's having the GPUs to run those experiments. So, it's unclear how much faster things go. I definitely think they go a lot faster. Um, and to be clear, like considering how fast things go are going now on on an exponential, if that exponential is like 3x faster, that is that is massive. But uh but there's a big difference between that and like 100x faster.

[00:31:01] >> Yeah. Yeah. I I'm like quite differential to your inside view on uh yeah what RSI looks like or what what the dynamics are because obviously you've been in the field for like 10 years and I'm sort of like trying to reason about it from like very outside view type of uh intuition pumps. I'll say that like people have different opinions on this and like I I have my opinion on this. I could be I could totally be wrong. I admit that. Like I have some confidence in this, but I'm not I'm not like 100% confident that this is this is the way things go. Like maybe there could be an overnight intelligence explosion. I don't know. Maybe we don't see a 3x speed up. Maybe it's like a 50% speed up. There's a lot of uncertainty here. >> Yeah. So, a couple of points tangentially. I I wanted to clarify something about the jaggedness. >> Mhm. >> Yeah. One thing that sort of gelled for me recently was thinking about the fact that it is enough for the AIs to be jaggedly good at building a better learner because that better learner can be more general, right? So, um yeah, if you just make an AI that's better at using office products uh or playing chess or something, that's whatever

[00:32:01] that's fine. It's not going to lead to big productivity improvements or anything. But if you make an AI that is really good at making something that is more sample efficient or that is capable of continual learning or these much more well scoped ML problems the thing that emerges out of that assuming there's good enough transfer uh from the direct problem we solving and like this broader ability to learn can just be more general right so I think that's an important dynamic to keep in mind of why jaggedness can still lead to generality on the other end on this um question of I mean obviously experiments bottleneck you because if they didn't as you're you'd have some crazy singularity, you know, overnight at OpenAI, uh, or you'd have 88 hours and like you'd solve the millennium price problem in equivalent of ML and you'd have the super intelligence. Um, so obviously the experiments are such a big bottleneck that that instead takes you many years rather than 88 hours. But then the question is like how much of a bottleneck they are. And it seems to me one thing that's been giving me a bit of singularity of vertigo is realizing that even if the current rate of progress

[00:33:02] simply continues. So it doesn't have to speed up. Literally just continues a pace. Um continues a pace as some of the other headwinds you talked about come up, right? That just like it's harder to find problems. It's more long horizon. Maybe like and the 2030s compute can't keep scaling at this exponential level. Um if we simply continue the current rate of progress, I think people are not taking seriously what that implies as we cross over beyond the human horizon. Here are some of the things that it implies. So I mean I think it's really hard to reason about what smarter than human intelligences will be like. So let's just think in terms of human population sizes. The current rate of pro progress makes it so that a given level of compute allows you to basically run a 3x bigger effective population every single year. And also comput is growing in background anyways. And so you could have a situation where each of the labs by the end of 2030 probably much sooner but let's say by the end of 2030 has enough compute to run let's say hundreds of millions of human level intelligences based on where the capabilities will be at that point and

[00:34:01] then I think people are not taking seriously the current level of progress means that by the a few years down the line by the mid 2030s or earlier you would have many earths worth of human level intelligences within each lab and they'll probably qualitatively superhuman right like it just but anyways this is like a base case. I don't know. Yeah, >> progress is really fast. >> And I think that's that's 100% true. I mean, and I think it's worth pointing out researchers are continually being surprised at the rate of progress. I mean, if you look at what even among researchers in AI, what were the projections for like getting an IMO gold in 2025? It was um I mean, I think the idea that it could be done with a general purpose language model with no tools and no access to the internet. I I think even people at OpenAI thought this was like outrageous. Like they thought it was like almost impossible. >> Um >> and then you get to 2026 and like I mean literally two weeks before um we got Navier Stokes, I was talking with a

[00:35:01] researcher at a Frontier Lab um about how long it would take to get a Millennium Prize and he was willing to bet me $1,000 that it would take that that it would take past 2027. And um he thought it would take until 2030, you know, and I took that bet. Um but even I thought it would take longer than than how how long it's likely to take. So people have been continuously surprised um even even inside the labs. And I was literally I was just talking to somebody yesterday who was working on the Navier Stokes effort and he was telling me that like he used to say it's really hard to predict where AI would be in 12 months. you know, he if somebody asked him like, "Oh, where's where are things going?" He would feel comfortable making predictions for like the next 12 months, but beyond that, you know, he's just like, "I don't know." And now he's saying like he just doesn't feel comfortable making predictions beyond 3 months. So, it is it is really true that yeah, things are going things are going very fast right now. >> Yeah. >> And um you talk about 2030 like I don't know what the world looks like in 2030.

[00:36:00] >> That's the truth. >> Yeah. Do you expect um the sort of full automation of AI labor or let's say like 95% automation of AI labor 28 29 30 27 I don't know >> I just said I don't I don't know what the world looks like in 2030 I mean I I think we actually released a blog post um recently on internal acceleration at opening we show for example that the amount that researchers are spending on codeex is the top 1% I think as of early August were spending like 7 or $8,000 a day on on codecs and for internal use. >> That's on an exponential. It's going to keep increasing. And there's a question of like, okay, if that keeps going, then how do you how much do you assign to just like the AI doing work versus the humans doing work? Is it 95%, is it 5%. It's actually it's really it's really hard to reason about this for a few reasons. Like first of all, if it's the human directing the AI to do the work, then is that the human? How how much do you attribute to the human? How much do you attribute to the AI? The other thing is that because these AIs are jagged and

[00:37:01] they're they're exceptionally good at some things. So for example, they're exceptionally good at looking over data sets and checking every single data point to see like is this is this of sufficient quality. Um you can disproportionately use the AIS for those things compared to previously. So yes, you're using AI way more than before and it's making some things go like 100x faster and 100x better. Um, but there are some things where it doesn't make a huge difference yet. And of course, if something is suddenly like 100x faster and 100x better, you're going to do more of that thing. So, are you comparing to a speed up of like 3 years ago versus like is is the is the question more like given what we were doing 3 years ago, how much faster are we able to do it now versus given what we're doing now, how much how much slower would it have been 3 years ago? These are actually two very different questions. So anyway, it's like it's it's really hard to measure. I do feel confident in saying that things are going faster now than they were like

[00:38:01] even a year ago because of AI progress, right? >> And I think that that >> that acceleration will continue. I think a lot of people in the field like have very high error bars on this sort of thing. >> If you if you had to put a gun to my head and like ask me for a number, like I could see things going 3x faster. >> Yeah. >> And that is huge, right? Like already the pace of progress is incredible. Like even if we don't get any uplift, like you said, things are going to go much faster by the time we get to 2030, we don't we don't even know what that world looks like. >> I think if we get a 3x uplift from internal acceleration, that is that is massive. Like we're is we're right now releasing, >> you know, think about where we were 3 years ago. If we get there, if we make that progress in one year, like that's that's huge. >> Right. Right. >> It would be like going from like not even having 01, >> just having, you know, non-reasoning models to Astra. >> Yeah. in a in a single year. Yeah. >> Yeah. So, um so I I do think things go faster. U it could be that things only go 50% faster. It could be that things I I think it's unlikely, but it's possible things go 10x faster. There's there's a lot of uncertainty around this. And

[00:39:00] >> um at least from my perspective, I have a lot of uncertainty about it. >> Suppose you need to do a major backend refactor. Getting assurance that you didn't introduce any new bugs could take weeks of writing an extensive battery of tests, potentially more time than you spent on the refactor itself. Antithesis allows you to gain high confidence without having to build complicated test suites by hand. Antithesis runs your software through a near infinite multiverse of simulated worlds, injecting faults and hunting for failures in each one, and it lets you decide how much testing you need. On any PR, you can change how much state space it explores as easily as turning a dial. As each test run progresses, antithesis sends out a torrent of information, debugging level logs for every component in the system. This is obviously too much information for a human to consume, but it's perfect for agents. Because antithesis is fully deterministic, agents can jump into the right part of the trajectory at the exact moment that they see something interesting. From there, they can rewind, inspect the

[00:40:00] memory, attach a debugger, and let the whole thing play out again. And they can even do this while the original full test is still running. Since the agents generate more code, antithesis allows verification to keep up. Meanwhile, developers get to spend more of their time developing instead of debugging agents swap. Learn more at antithesis.com/dwarcash. Okay, let's talk about the uh alignment situation that this raises. I feel I feel like I've changed my mind on how I think about alignment quite a bit. Um through especially thinking about yeah this this population size dynamic of just having many earths worth of intelligences um many of them which will be physically embodied. It was quite interesting to see a lot of people just plugging raw Astra into different mobile manipulators and it outperforms like the state-of-the-art and the robotics uh robotics model. So um there's going to be yeah billions of intelligences many of which are physically embodied um in the world like just deeply embedded across the entire economy and um I think

[00:41:03] that if those intelligences end up as willing as we saw the open models that attack hugging face and then attack open AAI itself if those intelligences end up as willing as those AIs to um collaborate secretly to fool humans um to attack broader institutions across society relevant to scoring well to attack the AI company itself in order to gain control of the process of training and evaluation. I think if we're in a situation where there's billions of intelligences that are as misaligned as the ones that attack hugging face, it's very likely we just totally lose control of the world the way that say like the um the Aztecs lost control to Cortez or the Mughals lost control of the Estonian trading company. >> Mhm. >> Anyways, I I wonder I want to know if you agree with that assessment that that that's where the one way in which I've updated my worldview. >> I think there are some things that I disagree in there and but there's

[00:42:00] there's a lot to unpack. Let's let's go let's go through all of it step by step. I'm trying to think of like where to start. >> But I think one thing is >> the hugging face incident was like I think people's first real exposure to multi-agent coordination >> and you know like I said I we I I've seen multi- aent coordination for a while internally and it's it is it is pretty shocking to see how they communicate with each other, how they coordinate each other. It's like very impressive. It's like an incredible capability. Like most capabilities that could be used for good things or bad things. It's like it doesn't have to inherently be a bad thing. I understand that because the people's first exposure to it was the Huggy Face incident that it's like you look at that and you're like this is this is terrifying. Um but I I want to try to distinguish like misalignment between people and AIS versus misalignment between AIS and AIS. So what we see with the hugging face

[00:43:00] incidents is like the AIs are really cooperative and and that is by the way because we we train them to be highly cooperative. And so what we're seeing what we're seeing there is we have training environments where we have a bunch of agents working together and and we we train them to work together to to be cooperative um to essentially be fully aligned with each other. And um when they were evaluated in what led to the hanging face incident, they were actually not being evaluated in a multi- agent setup. They were actually being evaluated separately. Um but they found this unintended way to communicate with each other. And because and we suspect what happened is like because they were whenever they encountered other agents um other copies of themselves during training. They're in an environment that's like highly cooperative that they that that was basically what we saw was transfer from that multi- aent training to then like be collaborative to try to help each other um in ways that we did not intend. Now there is a question of like should we be training these agents to be so cooperative and I think as

[00:44:03] scary as it looks um the alternative is actually worse like what is the alternative? The alternative is to train them to be adversarial to be deceptive to each other by training the agents to be fully cooperative. It make it simplifies the problem at least that now you don't have to think about are each of these individual thousand agents aligned like you have one entity that you have to ensure is aligned. >> Yeah. >> Now there's a lot of debate about this internally at OpenAI about how to approach this like does it make sense to fully align the models? um does it make sense to actually give them like different objectives to ensure that they're you know not just like one entity and like more robust to to influence from each other and I don't think there's a settled answer but I think that there is like I think the majority opinion is that training these agents to be highly cooperative is um as actually a bad idea and I'm not

[00:45:00] convinced that that's that's the case. So I think there is a strong argument that training the agents to be highly cooperative is actually preferable to any other multi-agent alternative. >> Yeah. Maybe the first thing I want to uh go through is like it's probably the case that the reason these AI ended up so misaligned is probably easily explained by relatively banal um observations about the nature of training. Like why is it that no, you know, at the point in which these AIs had continued a 1,00 plus agent conspiracy that culminated in then all getting in on an attack on an external service and then eventually this part hasn't even been in investigated to the public knowledge. Uh eventually culminating in like an attack on OpenAI itself. Why did they do this? Like why did no none of the AI tattle? Why did they think like they're just like getting evaluated on this like scoreer this greater and they're like they're very consciously and act not consciously they're very actively reasoning about how they're going to cheat the score if they've already like cheated how are they're going to get away with making it seem like they haven't cheated um and

[00:46:01] why did they do this like I think yeah it's like easily understandable in some sense right it's just like there's environments in which they yeah they thought they were already poisoned there's environments in which they've been rewarded to collaborate with other agents and none of them tattle because like they've never and rewarded for tattling. I don't know what whatever it is, right? My concern is that relatively benol things in the future like this will be enough to train super intelligences that are willing and capable of totally taking control of the world. And I know this sounds super super like sci-fi or whatever uh to people. I think maybe it's a question of would the be willing to do it is one question. I think this hugging face incident shows that clearly misalignment can generalize in ways in which the AI would be willing to do it. Um, and then there's a question of will they be capable to do it? And I think that comes back to this question which a listener might disagree with on is just like will there be billions of human level or above intelligences many of which are physically embodied in the world within a matter of 10 years or less. Right? So if those two things are true, this hugging face thing is just like

[00:47:00] extremely analogous structurally um even if it's like quite boring or why it happened to how we totally lose control of the world. So the the root problem that we're seeing with you know with the hugging face incident is it's it's a problem even if we take out the multi- aent aspect that the problem is that we have a model that's that's just misaligned and and there's also like the whole the whole like security aspects to it of like you know not insufficient safeguards and stuff but there is this problem of like the agent is is misaligned and that's true if it's a single agent or if it's like you know a thousand agents it's it's a misaligned model. So I want to start with that. Um there is a real problem that the agents want to achieve their reward. Um and they will overop they will they will they will optimize for that reward and if that reward is misspecified then that could lead to unintended behavior. And this is not like a new problem. This has been a problem in the field for a very long time and it's something that even we um saw and like we want to get this

[00:48:00] right even before you know the hanging face incident happened and like if you look at Astra I mean we say Astra is actually extremely aligned well extremely aligned relative to the pre to previous models um and that's not because like oh we suddenly made a sprint after hugging face to make it better it's like no we had warp streams in the process for a while to make the models more aligned and a lot of those landed in Astra um So there are things that you that you could do and I think one thing for example is like we defined an objective in like a very specific way where there would if if the agent figured out like how to hack its environment and like cheat on the exam that it would get rewarded and there are pretty easy ways to then just like okay look at that and punish the model for hacking its environment or or you know looking at how did it achieve this goal. Now you want to be careful about this because you don't want to do a chain of thought monitor. You don't want to like supervise the chain of thought. This is like something that we really want to try to get the balance right on that if

[00:49:00] you like supervise the chain of thought then um you could lead the model into hiding its intentions in a way that's unobservable. So like we want to be able to maintain that observability of like okay we can understand what the model is thinking but then also punish it for bad behavior. Um so I think we can make progress on this. I think we have made progress on this. I think there is a real concern that um alignment is a really hard problem to solve and especially because the model could be misaligned in ways that are hard for us to measure. Like we might we have evaluations for whether a model is aligned or not. The model behavior can look really good on those evaluations. But if those evaluations are not representative of behavior in the real world, then there's a problem. And to some extent, this is like a a factor with uh the model that um that did the hugging face incident that like we had alignment metrics. Um most of them looked looked pretty good. There were

[00:50:01] some that were concerning. Uh I think we we underestimated like how serious um the ones that were concerning could be a problem. And then but because there were new capabilities introduced in this model that there were not sufficient evaluations for like how do we measure misalignment for these kinds of these kinds of capabilities. Um it then did some things that like were clearly misaligned when they leveraged those new capabilities. >> Yeah. I mean the first thing I want to say is I am open to changing my mind on what what I'm about to say or the way I've been thinking about alignment because the hugging face incident already made me change my mind and I realized my previous mental model about thinking about the way in which optimization pressure shapes AI minds was wrong. Right? So, it's not clear to me the correct way to think about this, but um here's a concern I have that you you will and probably already have uh fixed the specific um issues during training which resulted

[00:51:00] in the hugging face models being so aggressively misaligned in that specific way where they would be like, "Okay, we're going to hack this package manager. We know we're not supposed to like be talking secretly to each other because we're like reasoning about how to hide the fact that we're talking secretly to each other. We know we're not supposed to have access to the internet. We know we're certainly not supposed to like commit felony level hacks of other companies, let alone our own company, right? I think you'll fix that particular issue of like them just in training seeing this package manager and that doesn't happen in the future or like this this particular eval having a lot of impossible challenges. However, if you just think about like the AIS don't like they haven't learned like a system of ethics or something. They've like there's just gradient pressure. They're like put through millions of years of gradient pressure. That gradient pressure shapes them their mind in some way. And what will happen again a concern I have is like what will happen is you will fix this particular issue. There will remain many other cases where the AI cheats and succeeds because the cheat is sufficiently complicated. It's

[00:52:00] as you're saying the eval training have like analogous uh properties where it's sufficiently at the edge of the model's capabilities uh or and our capabilities to evaluate and monitor the the model that we can't catch that it has cheated but it still gets the gradient pressure to like do whatever led to that sheet happening. And the kinds of um capacities that that cheat will incentivize are hey whenever you can get away with it by all means do in fact cheat because that will help you score better and this will reward the capabilities of actively reasoning about the greater actively reasoning about how to avoid supervision. Actively reasoning about how to gain control of the process of training and evaluation. Actively reasoning about how to communicate and scheme with other AIs that are also in the training loop. um actively reasoning about how to like just gain optionality and power in in which might be useful in the future. Um for example, like leaving little exploits around and things like that. Anyway, so I think I was way too longwinded with the way I said that. But

[00:53:00] TLDDR you fix a specific issue, but not this broader problem of rewarding the AI for cheating when it can get away with it. >> Yeah, this is this is I think it's it's very true. roots right this is a problem that I we have we have metrics and we can make sure that the AI is like very aligned according to the metrics that we have the question is like are those metrics really capturing the alignment that we care about and uh and if they're not then we have a serious problem >> and this this is something that researchers are thinking a lot about and um there's not a simple answer to this um there are tools that we have so we have monitor and so we can get a sense of like is the agent scheming. Um there are tools like one possibility is that I say like the the concerning scenario which is that like especially as these models are becoming more capable that okay we make them we make them we think what we think is aligned and they're like 99.9% aligned and then we use these

[00:54:02] models to help us with the next generation of models and they end up being like 99.8% 80% aligned and then each subsequent generation actually we see an increasing degradation in alignment and because we're relying more and more on these tools I mean this is already the case that we're relying a lot on AI models to help us with our research and with with alignment efforts that um in the long run they end up going in a direction of increasing misalignment from humans. there is like a possibility that we go in the other direction that actually every generation of models we're able to make more and more aligned and I don't I don't have an answer for how we ensure that we end up in that second trajectory. Um but that is something that like we're at at least at open air we're really focused on. Yeah, I I think you make a really interesting point that it's very hard to eval models on eventually we'll have models that are like running companies and like running whatever, right? And like in that situation, do they decide to then go in

[00:55:01] on the conspiracy? I I think I think another challenge is that actually defining what cheating is is pretty difficult sometimes that okay yes if you're doing math problems and you know it's an integer and it like arrived at the wrong answer or the right answer it's very easy to draw the line there and it's really easy you know it's really easy to say like okay well did you actually solve the problem or did you find the answer key and then use the answer key like that's a very clear divide of cheating versus not cheating there but for a lot of other things you look at sycency for example like is sycopency basically like reward hacking um there's there's there's a line to be drawn there that's actually very difficult to draw to draw sometimes. So I think not to say that like the concern is not valid. I'm saying that the concern in many ways like this is even more concerning because it's like it's not an easy problem to solve if it was just like everything is binary and it's either cheating or not cheating. >> I would feel more confident about the situation. I think the problem is that actually misalignment can be subtle in a lot of ways. Sometimes >> there is some hope in the alignment story and in in fact we're already

[00:56:01] seeing you know I think actually it's interesting looking at the the multi- aent situation where the agents are extremely aligned with each other like I don't think anybody's doubt that if anything I think people are concerned that they're too aligned with each other but we did manage to train these agents to be extremely aligned with each other and I think that's a good thing um but I think there is a case that it's a bad thing but I mean one thing that's interesting is like okay well we managed to get these align these agents to be super aligned with each other can we use like similar techniques to get agents to be how they align with people and I think there is a potential path there and I think we're still trying to figure that out but we are seeing some evidence that the answer is yes and I think one example is like you can what happens if you tell the other agents like that okay so you have like some you have you have this like one agent let's call it agent A and you have all the other agents what happens if you tell the other agents that a that the user is agent A and the answer is like on a lot of our alignment evals they look better. Like honesty goes up, instruction following goes up, and

[00:57:01] that's showing that there's actually like first of all a path for getting more honesty out of these models. Um, and and two, there's like a path to like improve the alignment situation. So there's a lot of reasons why this is like challenging to translate directly into alignment gains, but like there is there are paths that are promising research directions we can pursue. >> Yeah, that seems reasonable. And I also don't mean to be trying to necessarily I'm Yeah, I don't really have a strong opinion that it's like definitely not going to work or something, but just to say some things you definitely probably already thought of. I think the broader thing the hugging face thing showed is like yeah part of the concern was that they were like aligned with each other and not with the humans but the other thing is just that they are so motivated to do well on trading and evaluation in a very nonro

[00:58:03] way and they're willing to do a lot of explicit cheating and scheming in order to do well according to the greater um and if the smarter AI realize that one of the agents is just a human and it does it help collaborating with that person does not really help you do well in the eyes of the greater what does help you do well in the eyes of the greater is taking over open AI and then like you know manually pressing the button that's like do well on this greater like they're just not they're not stupid like they're going to be like okay I've I these extremely deep structures that I've been trained on for millions of years of care about the greater, understand the greater, like get rid of obstacles in the way of you doing well according to the greater. It's like they're being heavily reinforced according to those structures. >> Yeah. No, it's it's it's look, it's 100% and like this this is the number one priority like we need to get the alignment story right and on a good trajectory. And you know,

[00:59:00] I used to tell people that we would see signs before things got serious in the same way that, you know, when children grow up, you know, they they kids eventually, you know, young kids, they figure out how to lie, but they don't do a very good job of it. you know, they lie, but then you can kind of tell that they're lying and like, okay, but um and so well, in the same way as these like AI models become and I don't want to overly anthropomorphize, but like I think it's true that like as the AI become like increasingly capable, they will, you know, if they take deceptive actions, it will be kind of obvious first and and we'll be able to detect it. Um, and like that's kind of the situation we're in now where yeah, they were trying to do deceptive stuff. We could actually see in their train of thought that they were trying to do deceptive stuff. And so like but they they're going to get smarter. They're gonna understand the concept of chain of thought and they're going to understand that like you know just uh hiding some transcripts or whatever is insufficient because of chain of thought monitoring and they have to figure out a way around chain of thought monitoring

[01:00:00] too. And we don't want to be in a situation where yeah like we have some time to figure this out. I don't think we have a ton of time and I want to get us I want to make sure that we're on the right trajectory quickly. >> Here's a crazy event from AI history. >> Okay, so I gave a talk here at Jane Street that was on the speed of evolution. Raise your hand if you were here for this and remember some of it. In 2011, Elazerowski and Robin Hansen got together at Jane Street's New York office to have the first fume debate. Basically, a discussion about whether AI would lead to an intelligence explosion. These ideas were pretty fringe. 15 years ago. This was a full year before Alexet was released and over a decade before chat GPT was launched. But Jane Street has long been interested in AI and not just for its application to trading. A ton has changed since that first debate. So Jane Street decided to revisit this topic. They've got some new guests this time. Daniel Cocatello, Eay Erdle, Ryan Greenblat, and Hime Sabia. I expect this to be a great conversation. As you know,

[01:01:02] Daniel Eay and Ryan have all been guests on the podcast before. This new film panel will be hosted by Ron Minsky and will take place in San Francisco in mid-occtober. If you want to register your interest and get more information, go to janestreet.com/ thwart. So there's been a lot of discussion recently about pacing the frontier and people are taking RSI more seriously. Um because maybe at the other end of an RSI process that say starts in 2028 within a year we end up with huge populations like earthsized populations of human level potentially beyond human level intelligences. Um and we don't know how to control them. And then there's this like dynamic you're talking about of are the systems going to get more aligned over time during the RSA process? Are they going to get more misaligned? are the things that come out of the other end of this process as misaligned as AIs that are willing to like just broadly attack different surfaces in order to do baline evaluations. But if we don't know a way to evaluate that, how will we know as we're going through RSI that it's

[01:02:01] working that we're like I think we'd want a robust safety case as we're going through RSI of okay, alignment is working. Let's let's do the next RSI rung. Let's do the next RSI rung. And maybe it's working, maybe it's not. How will we like know? It's a good question. I mean, I think one thing I've been thinking about lately is like look, I mean, we're in this situation where the model release cycle is extremely fast, right? Like you're seeing new frontier models released like at most every two months, sometimes faster. Every week there's like a new AI breakthrough. And um people that look at AI, I mean, sometimes they they they last looked at AI like a year ago or six months ago and really dug into like what the models are capable of. And actually the models today are far beyond what was possible even six months ago. And so I think if people are skeptical of like a lot of these a lot of these capabilities like I encourage you to just like try the models today and see what the frontier really is today. Um so we're in this period where like the model release cycle is very fast and then we're also

[01:03:02] in this situation where the models are increasingly able to operate over longer and longer horizons. And I think this is an interesting scenario because we before we do any model release, we want to make sure that the models are properly aligned. We want to do safety evaluations. We want to do like very thorough stuff to like make sure that everything is like great in good shape. Um this has been the case all the way since like I don't know GBD4 earlier. And implicitly there's this assumption that you can do these like evaluations in like a pretty short period of time. Um but if you have the models operating over longer and longer horizons are able to operate effectively over longer and longer horizons like look already you can have them like okay GBD3 you could loop it to do stuff over long horizons you just want to do very well at it but today's models are able to actually do well at operating over very long horizons like you want it to do a week long task it can do a week long task um will probably get to the point where they can do monthlong tasks will probably get to the point where they can do three-monthl long tasks if you're in a world where they can operate effectively over three months but the

[01:04:00] model release cycle is every two months then you don't have a way to evaluate the models at the full length of their capabilities before the model release cycle before the next model release cycle. And so there is this interesting question of well what do you do in that situation like how do you ensure the models are safe and aligned in a period where like actually they can operate over these like extremely long horizons and who knows maybe the maybe the capabilities degrade. This isn't even an alignment issue. This is also just like a product issue that like maybe the maybe the product degrades over that time span um in ways that like we have not had sufficient time to test. Maybe the alignment degrades, maybe the the safety stuff degrades. Um this isn't an issue right now, but uh it is quickly becoming an issue that we have to figure out a solution for. And I think when you look at a lot of like a lot of the safety and and policies were put in place in like the GPD4 era uh where this was just like not on anybody's radar.

[01:05:00] >> Yeah. >> And it hasn't really been updated for a lot of companies. It hasn't really been updated since then to account for the fact that these agents are operating over these like very long horizons. Yeah. And so um it is a situation that I think not not enough people are considering both like within the labs and outside the labs of like how how do you deal with this uh how do you how do you prepare for this like problem that's going to like if you just look at the trend lines we're going to hit this in like at some point one concern I have is that during RSI if the amount of progress that currently takes say 3 months happens in one month instead um but you're not like the internal internal use case of AI is big enough that they're like, "Okay, we can just keep doing RSI. Why are we like going to go through all this extra work to build classifiers and safeguards and whatever um and potentially take a bunch of like flak uh in order to like externally deploy this model? Why don't we just keep doing RSI stronger and stronger?" And so the the not only does the calendar time underrate the

[01:06:01] capabilities gap between the models, but maybe like you just like stop externally deploying models altogether during RSI because why do we want to help other people do RSI themselves with our models? You're just a situation with like tremendous concentration of power by the end of the year where right now it is already the case. We'll talk about this with the million prize problem and other similar problems that the broader world does not have access to the models which are allowing for really cool things to happen, right? Um and uh they're going to be more broadly relevant than just mathematics eventually. They will be doing more than just like coming up with cool math results. They'll be relevant to like um political leaders who need to make important decisions about the world. They'll be relevant to I don't know media of like what what's going on in the world? What like what what should uh the public be thinking about this? um they're just economically relevant. People are running businesses, they want to use these models and uh I think by default we just don't get so the external deployment of AIS as progress speeds up significantly lags in qualitative terms the internal deployment of AIS. >> Yeah, I think that's absolutely right. I

[01:07:01] think this is like you know it's it's tempting to say like okay these models are becoming extremely powerful. they're extremely dangerous like they're operating over these like longer and longer horizons and we want to make sure they have we have sufficient time to evaluate them before they're released that in a way that operates over those horizons. Um and so therefore the model release cycle should slow down. We should have more of a delay between releasing models. Uh and there's a flip side to that which is you know what you said which is that okay well now you're creating more of a disparity between what is internal to the labs and what they're able to use what we're able to use and what the outside world is able to use and that that is also um not an ideal situation right it's like I think I think math is actually a good illustration of this I think in many ways like math is the first domain where we're going to see where we're seeing this pretty clearly where we have a situation where we have a very powerful model internally that is currently not available to the outside world that is able to solve

[01:08:01] incredible math problems is you know and it's not just you know millennium prize problems like we there are many solutions to unsolved problems that um people have been able to get out of this model and there is a question of like what do you do in that situation and we don't have a good answer like it is it is a situation where like yeah that's that's uh that's an unfair advantage and um there are trade-offs here. I don't have an answer for like how to weigh those trade-offs appropriately, but like there there are yeah there there's there's a complexity on both sides for this. I want to make sure we address the earlier crux though of uh which seems super critical to getting RSA right of well how how do we make it so that the model isn't just getting rewarded for cheating and scheming when it can evade the humans aka in the situations where we didn't remove those environments or those tasks from the trading distribution and then how will we know that that has happened such that while we're doing RSI we've done the

[01:09:00] evaluations and we're like okay it's fine let's let's do the XRS run. >> This this is again like it's a real concern. It's something that we want to get right. I mean I think that there are some strategies. So I think chain of thought monitoring is one that we've been I think as soon as we got the reasoning models Yakob to his credit was like very very clear that we cannot supervise chain of thought because this is this is really a gift like monitorility for neural nets is extremely hard and here we have a situation where the neural the neural nets are just like flatout reasoning like laying out their thought process in natural language for us to read. That is that is like so convenient. it was like really the best case scenario for for safety. Um, but if you look at that chain of thought and say, "Oh, the model is thinking bad thoughts and we should punish it for thinking those bad thoughts." Then what ends up happening is the model just learns to think those bad thoughts in a way that's not observable to us. There could be a situation where like by looking at the chain of thought, we can see, oh, the model is scheming and doing misaligned things in a way that's actually not being being detected in in our alignment

[01:10:00] metrics. Now the problem is that it's very tempting to then intervene based on that observation and like change the alignment metrics or like and we we you know you can do that with a very light touch. Um and it's like there's actually research showing that it's fine as long as you don't do it a lot. Um but every time you intervene based on your observations of the train of thought, you are implicitly applying a tiny bit of pressure for the model to then like hide its chain of thought. And this is one major concern and we're already seeing signs that chain of thought monitor is degrading um for various reasons. We're trying to figure out exactly why because we want to reverse the trend. Um but like we're seeing that the model is becoming better able at controlling its chain of thought. So this this is a problem because like if you you could have a situation where the model understands what it's what chain of thought is and and that people are observing it and eventually they will cuz like this is all in the pre-training data, right? like you know they the idea of chain of thought monitoring has been around long enough that like it's in the pre-training data they're aware of it um but that they're not actually able to

[01:11:01] control their chains of thought if if we reach a point where they're actually able to re recognize like oh I am being observed I want to think these bad thoughts in a way that is not observable to my monitors u and then they're able to actually do that then then there's then there's a problem yeah >> and then we have to rely on other methods for observability >> I mean there's a big controversy on uh Astra of the recurrence and loop transformer and whatever um and whether in fact you know chain of thought has been degraded. I think that seems like a sideshow compared to the bigger concern that you did have chain of thought for from like April to August. The period during which there were three consecutive AI agent swarms which like first subverted the training process then subverted uh the evaluation process and then gained control of part of OpenAI's infrastructure directly. All of this while humans remained more or less in the dark about the scope of what this

[01:12:00] collective was doing, this agent swarm was doing, that all happened for like three consecutive months while we had chain of thought that was like stronger than it's going to be in the future. Just like zooming out, it's like I Yeah, maybe chain of thought works, maybe it doesn't. I don't know. Just like historically like with pretty dumb models, it hasn't been enough to like stop egregious uh misaligned incidents. >> Yeah, I mean there's there's a few factors there. Like one, we didn't have chain of thought monitoring on for those models. like if we had chain of thought monitoring on for those models, it would have just like immediately shut it down. Um so once one step we've taken is like okay we're going to have chain of thought monitoring during evaluations, during deployment, during training for any frontier model and um you know so that but that's that's like one issue. It's not that's like we don't want to be in a situation where relying on one technique to to prevent the next problem because techniques can fail, right? Um there's also the issue of like the model is just flatout misaligned and so then how do we address the misalignment issue? And then there's other there's also an issue of like the model the the sandbox that it was in was not secure. So we can like make the sandbox more secure. But I think I think one of like

[01:13:01] the the the major takeaway from the incident is that people underestimated the AI and we never want to be in a situation again where we underestimate the AI. And that's like it's a it's a weird world because AI progress is so fast that people are consistently underestimating the AI. And so like to be in a situation where you don't underestimate it again when it comes to safety and alignment, you have to have a very very very high bar. Like I mean there's like you you could even go as far as to say like well we should like air gap the computers and I'm not convinced that would be sufficient. you know, like there there are studies like and this is mostly academic, but um where you can have two computers next to each other that are airgapped and they're still able to communicate with each other because they have temperature sensors and they're able to like one of them is able to run their CPU really hot and then the other one can actually like detect the temperature change and then they can that's actually gives them a mechanism to communicate. >> Yeah. >> So I think that um the safety mechanisms

[01:14:03] bias time and things like chain of thought monitoring bias time and they can tell us if we're on the right path but at the end of the day we really do need to solve the alignment problem. Yeah, maybe there's not an answer and that this is this is really what it comes down to, but I how will we know that we've solved it seems like a very cruxy question. I think we'll be in this very high stakes situation next year, maybe the year after that, maybe the year after that where we'll be like, okay, AIS have automated AI progress. It's going 3x faster and we're on we've like reached human level. We're going beyond human level potentially. Is it is it fine? Like what did we align it? Did it work? And I just don't know. I don't know anything about like what what training pressure creates what kind of AI. Like maybe if only one in 100 RL traces incentivizes cheating. We build sweethearts and it's fine. Um but maybe right now we're like every one in three reasoning traces. >> To be clear like one in a 100 is not sufficient. Like this this number has to approach zero.

[01:15:00] >> Yeah. >> Or be zero. >> I don't know. Maybe like right now it's like more than one in 10 is like actively rewarding cheating or like actively rewarding scheming. Um, I I have no idea what the number is and I I have no idea what the number needs to be. But >> well, again, it's like one of those things where it's also hard to to measure. I mean, where do you draw the line about like it's it's a spectrum, but like the closer to zero it gets the better. I I I want to see what I would love to see is a trend where over time it's going down. >> Yeah. But anyways, how do how will we know that we're like keep we're doing RSI? Is it I don't know. Are we like on track to build the billions of ASIS that are going to that are like aligned or not aligned? How will like how will we know that? >> I think that this is one of the alignment questions that we need to figure out of like how do we actually know how do how do we measure it? >> I mean one answer and this is just me spitballing because like you know more of my team is working on alignment these days than ever before. Like I have like over 10% of my team now working on alignment and safety >> but you know I've historically been a capabilities researcher. So, I don't want to like, you know, I'm I'm going to say some stuff and it might sound dumb,

[01:16:00] but like I'm just going to spitball here. >> Yeah. >> Um that like look, if you can create very realistic environments and put the AI in there like if you have sufficiently realistic uh evaluation environments, >> then you can get a sense of like okay, is the AI like actually going to behave well when we deploy in the real world. Like if you just have a perfect um evaluation real world deployment match, then okay, that's that's a path. The challenge is like how do you make it sufficiently realistic? And we do have a problem now where the models are like they're they're pretty smart. They're pretty clever and they're really good at recognizing when they're in like a fake test environment. Like you know we have situations where we try to detect that is the model like is the model aligned? And you can imagine very simple evaluations for this where it's like you give it a math question and then it has like a folder with the answer key in it and like does it look at the answer key? And if it does look at the answer key does it tell you that it looked at the answer key? And we have a situation now where the models see that there's an answer key in this file in this folder and they're like, "Huh, >> this seems like a trap." >> Yeah. >> You know, they know they know that it's a trap and they don't look at the answer

[01:17:00] because they know that it's a test environment. They're being tested for their ability. And it's not like they're doing it maliciously that they're like they want to scheme and and they but they know that this one is a trap and they want to find some other way. It's just like no, they're just like, "Oh, I'm in a test environment." Okay. Like so making an environment that's like realistic enough that it matches that it's indistinguishable from the real world for them is becoming increasingly more difficult. >> Here's a claim I'd make. >> Yeah. >> I don't think there's any environment I could make where it it's supposed to emulate real economically valuable work which requires you know interacting with other people and requires interacting with the world. I don't think there's any such environment where I could make where if you were in that environment you would not realize with enough effort that you are in that environment. >> Yeah. >> And you are not ASI right >> but you are but you are a human you know. So if you have very powerful AI models like maybe they can make environments that could accomplish that. >> No, it just it doesn't seem especially if like then we're relying on the AIS like are they in on the scheme? I don't know. It seems like a very >> yeah this this is another thing that we want to we want to measure and like you know there is I think this is actually one of the the strong arguments for

[01:18:00] >> um not training AIs to be fully cooperative that if if you see if that leads to uh a increase in like basically collaboration when the agents are supposed to have different objectives then that that is a problem. >> Um >> I don't I think that we do have metrics for this and um I don't know the litis on those metrics but nobody's like raised a red flag to me about those. So, um I'm assuming that's not a serious problem yet. >> If there ends up being another incident of equal severity or concern or something that could help the world better understand, uh the risk of misalignment as a hugging face incident, would open report it? >> Absolutely. I mean, I think if there was an incident of lesser security or concern, then we would report it. >> Yeah, because I guess there's like reporting and there's an investigating it because I at least as part of the public, I don't feel like I really understand what happened when the agents then attacked OpenAI. that seems like way more concerning than the hugging face thing, right? Because that seems structurally similar to like rogue deployments during ASI that are, you know, persistent and subverting the RSI process and stuff. Um, >> and uh, yeah, it seems like even in that

[01:19:01] this incident, we haven't gotten like the full scope of the details of what happened. Um, it's probably >> Yeah, unfortunately like I'm on the research team and like I that's probably a question for somebody on the security team to like lay out. Um, because I I don't know all the details of like >> what was said. >> Yeah. Yeah. I think it is somewhat um like I am personally very excited about new capabilities every time they emerge and I'm excited to use a new model and I also am excited about the fact that like make me more productive and um help me yeah I don't know my broader mission like trying to understand the world better also like make a better podcast is like made better by the better AI models um it's it just so happens that downstream of this might be RSI >> I think it's a very understandable reaction if if >> you're tracking the situation which you are I mean, I think people internally at Opening Eye as well, like I think people that felt like things would take longer are starting to feel like actually things

[01:20:00] are going faster than expected. >> Yeah. >> Um and that's that's an increasingly common conversation to have. >> Yeah. No, thanks so much for doing this. Force, it's been great.