dwarkesh ai researchers recursive self improvement debate transcript

Today I'm chatting with three of my AI researcher friends from whom I learn a lot every time we talk and who also happen to be at somewhat openish uh labs and companies so you guys can actually um say things on the record. I'm joined by Baron Militch who is the CTO of Zyra which is developing open source models. John Schulman who is the chief scientist at thinking machines previously the co-founder of OpenAI led the RLF work that led to CHPT and Charlie O'Neal who is head of model training at base 10. The first question I have, if we're in 2036, it's been 10 years, and we don't have like crazy billions of crazy super intelligences that are running around that like radically transform the world, what is the most likely reason that that doesn't end up being the case other than sort of exogenous political shocks or like there's a war or they ban AI or something, but what is the most likely technical reason that we don't like 2036 isn't like a crazy alien super intelligence world? I mean like my reason would just be like it's got to be the sort of like there's been a classic

[00:01:01] thing almost like Marvik's paradox right where like we see like you know we think of the AI being like if it can do this it's going to be amazing right like if it can solve these hard math problems if it can win a chess blah blah and then it solves these things and then it's like not that impactful obviously it's somewhat impactful but like not everything it's like if somehow that continues and like there's never like the true like spark of generalization that occurs I think that could lead to like the AI just being like extremely good at kind of everything that people like put into a benchmark, put into an environment, but like there are still some persistent like sim to real which is somehow blocking everything. I think this is kind of unlikely. I think we do actually see this kind of generalization even from RL in practice already. But like if it is just like ridiculously hard to like generalize metalarning plus like we don't solve continuous learning, it is just like super hard and impossible. Yeah, >> like this would be my like default scenario in that case. >> Yeah, I agree with that. Humans uh have a lot of advantages over models now and uh each time a new model comes out, it'll sort of uh it'll catch up in some of these areas. Um but uh like you end

[00:02:02] up getting bottlenecked by the places where the model is weaker and where it has uh worse judgment or um the models can't check themselves well enough. Yeah. So so there's this uh cycle that keeps repeating where people think uh where a new model comes out and people are blown away and they're like this is it. This is the this is AGI. but then uh they use it a bit and and then it starts to feel dumb after a month or so. So that cycle just might keep going and it's hard to predict how many times it's it's going to repeat. And uh like right now you don't get explosive growth um in capabilities because uh you still get bottlenecked enough when you're trying to do research and engineering that even if the model can write way more code than a person, it doesn't make you like a 100 times more productive. Uh but yeah, so maybe maybe they're just more of these cycles than we would than we would expect. >> For me, it's like a question of how far off like this global optimum of a learner you could have on a chip is like the transformer plus like RL basically

[00:03:02] like the current recipe. So like I think people imagine that even like once you have a a an agent which is better than all humans at AI research even if it's like.1% better than all humans then the fact that you can run like you know hundreds of thousands if not millions of these in parallel um you can run them much faster like chip's going to speed up that's going to outweigh every other like bottleneck and like you're eventually just going to like hit this like very fast takeoff with recursive self-improvement. I could imagine that if we continue along the trajectory that we're currently on with that paradigm where you know it's basically just like self attention RL scaling up RL environments the I guess like if you think about what happened with Mo's law right like we had this very like nice straight line and that held for a really really long time but there were so many like discrete like discontinuities and innovations that had to happen to keep that scaling law going and the same thing has kind of happened with LM like we had this pre-training like scaling law and then that was kind of like you know hitting the diminishing returns and then we came up with like RL and solved

[00:04:00] that and then we got this new like you know diminishing returns curve to hit that made it keep looking like a straight line going up and so like if it requires another one of those discontinuities to solve like I'm not sure that like the current method of like training LM with these RL environments even like RSI targeted RL environments would be able to discover that discontinuity and if not like we're probably going to hit this like asmtoic like curve where like >> sorry but do do you think discontinuity will be harder than anything that's come since 2012. >> If if we had the answer to that that that we'd kind of have the ability to implement it, but like maybe there's the distinguish we should distinguish between a discontinuity which adds to the current paradigm. Again, it's like cumulative like there's some thing beyond the RL that we have to discover and maybe they're capable of like you know connecting the dots in that straight line >> or like like again how far off the the global optimum are we? Do we have to go back and throw out like you know gradient descent and like neural nets in general? And I don't think like if you continue to scale up the current paradigm an LLM no matter how many LM

[00:05:01] you're running are capable of necessarily discovering that if it's too far away. >> Yeah. The only hope really is if deep learning just can't get us to an AI which is at least uh can dominate human research and human development including the human ability to come up with new paradigms and so forth or like I don't know maybe humans would also never have discovered the the next learning architecture but um to the extent humans could have discovered it eventually but it just seems like I don't know if if you just look at the progress that's happens in 2012 till now and you just continued that on. I mean, I know it's been powered by huge amounts of comput scaling and so forth, but um uh it would be weird if like it just didn't get to the point where it could like dominate humans, at least in R&D, especially over the next few years, there's going to be Ryan Greenbl was on the podcast recently and he made this point that you could imagine as AI get more and more capable and are capable of making progress on

[00:06:01] simulations which incentivize getting better at not only AI R&D, but generally at science. So this is a thing that all the labs are targeting, many startups are targeting or the another intuition pump is if you look at the ELO score of chess bots since the 80s, there's just like a very linear increase in ELO over time, but there's this huge discontinuity as they cross the human range of human experts always win against AIS to like human experts never win against AIS as this linear increase in happen. You could think I agree with your point that the so far AI capabilities have not been that big of a deal in terms of their end economic impact in the world. But that is because like they're slowly rising in ELO relative to humans. >> Yeah. >> Yeah. I mean I agree it would be very I mean the only way for this to not happen is if like as you say somehow asmto like just before basically cuz we're already pretty close in my opinion to like where we'll start crossing like the human elo score and so we'll need to asent before that and like that's the only way you know in the scenario you pose where like somehow we're sitting here in 2035 and like everything is normal for this to happen. I think I mean the only other

[00:07:00] way is like there's like some dramatic like regulation on AI. It's like this is kind of what I see as like the most likely way for this scenario to happen actually rather than the technical thing. Yeah, I I think there's different kinds of research. There's like research where it's like the auto research style where the objective is already specified very cleanly and you're optimizing that objective. And I think everyone is picturing like >> if we continue along this path of like, you know, making pre-training loss go down, making our environments, making water go up, that's going to lead to like improvement. But like, you know, maybe what Ryan is talking about is like this much more open-ended type of science which is required for like paradigm shifts where we can't specify the objective and the AIS are definitely not able to specify that objective either. like we have to be really really careful about how we specify objectives for any of these things. >> And maybe your point is that like the nature of the breakthroughs that have happened since 2012 is that we have found like in 2012 people weren't saying or I'm assuming I don't know you guys were there or at least John you were there but I was >> I was in primary school. Um, actually, John, I'm curious for your like sort of uh wi wisdom of the ages of or wisdom of like being in the trenches

[00:08:00] way back when, but presumably a big breakthrough was realizing that next token prediction is the like you wouldn't have thought that nano nano GPT speedrun is the thing to be optimizing for in 2014. Um, but now that we have come to this new paradigm, that's you wouldn't think to do a speedrun on that and have you guys get really good at that. But maybe there's like a next inner loop to optimize that the AIS wouldn't anticipate. And there's an outer loop of like revenue or something that eventually should be strong. But it's a very slow outer loop. >> Yeah. In fact, I remember in the early OpenAI days uh having the int intuition that um actually just uh do like minimizing log loss wasn't going to get you to intelligence uh because uh >> like the important bits are accounting to for such a small fraction of the loss uh that like it was going to be overwhelmed by noise. So just training a a language model uh on next token prediction just wasn't going to learn the interesting things you wanted to learn. And uh we needed to craft better

[00:09:02] objectives that would put more emphasis on the important things. And uh like you can make all sorts of arguments for this and you could say um oh humans probably don't learn how to like uh we don't learn how to model everything in our environment. uh we can't uh most like people can't uh create a photorealistic uh reproduction of uh some kind of scene they've looked at. Uh so um there must be we must need a better objective. But then it turned out that it it just worked anyway. >> And as you were pointing out the inner loop even in current AI research of like post-training benchmarks or whatever doesn't necessarily translate into what users like. >> Oh yeah. I mean the whole field relies a lot on generalization and it's very hard to uh predict when when you're going to get generalization or when you're going to get some kind of out of distribution generalization. So we know that if you train on the task you care about you're going to do better. >> Yeah. >> Uh but like the most important advances are often uh the opp are are um types of

[00:10:03] generalization that we have no right to expect. So for example from uh just pre-training on this um very naive um next token prediction objective um to like various tasks of interest uh where uh some very uh that require understanding of the input in some deep way or learning some skill from pre-training that's like very uh rare and like not very like heavily represented. Um and then like also generalization from these verifiable tasks to um less verifiable ones. This is also a type of generalization that >> there's no reason a priori to expect it. >> Yeah. Yeah. So this is an interesting question because one intuition pump that you could have for why you would see um some sort of singularity very rapidly without even scaling up the inputs to AI progress that are not just AI labor is that before every single experiment you run that's you know like

[00:11:00] a seven figure experiment you spend an equivalent amount of compute on AI labor and so you just have automated versions of you guys spending a century uh thinking about like what is the optimal experiment to run um doing like small scale ablations developing literally like in a century's worth of theory. So going back even before like deep learning it in before you decide what experiment to run doing extremely optimal like setting up of the experiment. Then you do a century of thinking after the experiment is over where you're like analyzing what happened and what the next experiment to run is. >> Yeah. Well, I think if if you think hard enough, you probably could have um expected some of these things beforehand. Like uh like there is probably some very clever way to do a small scale experiment that'll let you build the theory that then will generalize to the large scale experiments. So I would expect that like we're nowhere near the ceiling of how well you can do research. And like I would imagine a future where um where AI is like um is doing a lot of uh like

[00:12:01] analysis and theory building um like spending a comparable amount of compute to the amount that you're spending on the experiments themselves uh doing various kinds of analysis and building a theory around what we've seen so far. >> I think there's like really concrete examples of this when the objective is well specified. So again like all thinking can do is like update your posterior based on like you know the bits that you've gotten since you formed your prior like you can't gain any new bits from like just thinking >> but when the objective is well specified and there is like this data sitting around like I imagine there will be this big speed up in the current paradigm we're in and like good example of this is like you know if you got an AI to think about like the cap plan scaling walls like an AI at this point would have noticed that like you know oh like they've just taken these intermediate checkpoints and didn't account for like the analing and so like this is wrong and that would have caught that like years earlier. We would have made like you know progress like like would have cut off a year or two of progress just from that like observation from an AI. Um and like again once the objective is well specified which is like lower

[00:13:00] pre-training loss or or whatever like there's many many good examples where if you just thought about it a bit more you would have been able to like cut down a significant on things that you've done. So like mu and like how learning rate scales with like model size and like realizing that model width is important in that as well. Um, like I feel like you can you can really back out a lot of these things and cut off like a lot of low hanging fruit. So I would imagine like a 10 times speed up if our thing is just like maximize the objective we're currently on. >> But I don't see that how that generalizes at all to you know come up with the right objective in the first place. Like just thinking doesn't necessarily buy you the right objective in the first place. I mean, yeah, I think this is really the key question to like any kind of like very rapid RSIs like from current AI is like how well can AI generalize to like learning their own objectives because to have any kind of like self-propelling automated loop, you need the AI to like propose objectives, optimize them, figure that out, propose a new objective and like have this like not go off the rails at like any point for like a long long time. Coming back to MVX products, there might be like a case of MX products where like we think this kind of like autonomy and sort of like being like

[00:14:00] self- encapsulated so we can, you know, think of what we should do ourselves and then go do it and like have this loop is like super easy because we always do this and like obviously evolution needs to create creatures that can like survive on by themselves like long periods of time and like this just might be something that for some reason is like really hard for the AI in the same way that like locomotion stuff is really hard. Like math is super easy despite being super hard for us. >> I don't know. Doesn't the time horizon increasing suggest that that's >> you would Yeah, exactly. I mean this is another possibility which like but I agree like there's no obviously evidence for this like in fact >> the fact that you know agent is now like super persistent and it's quite easy to do this is kind of evidence against this >> but like this would be you know potentially like one of the reasons why like we just don't get this like immediate takeoff is like if this is hard >> if if you look back from 2012 till now or maybe from when you started doing your research till now what part of of all the innovations that have happened since that time, including purely engineering ones, including purely conceptual ones. What seems like the thing that is the thing that would

[00:15:00] be the last things humans would have to do before AI is totally automate AI R&D? >> Probably just like iteratively asking the right questions like if you can get the AI to like do any experiment, but like you need to decide what experiments to do. And like right now, I think AI are not very good at this compared compared to coding the experiments at all. >> Yeah. Yeah, like whenever we talk about research, they proposed like a bunch of like miscellaneous things which are like very very tiny steps >> or even going from like you know deep mind's approach of like we're going to solve intelligence by learning to play games at a superhuman level. that's going to be the approach to like one random researcher like Radford being >> like I'm going to try and just predict the next token off a very wide swath of data like >> and then even once Radford had discovered that right like it took a while before people decided to scale it up because we had to come up with the idea of scaling walls and the fact that like you could very reliably predict these things. >> Yeah. Yeah. I would say that the last um job for humans uh or the the the role for humans that'll last the longest is like defining the objective. uh and like deciding what we actually want. Um so

[00:16:02] like in that vein uh something like uh deciding what um how like the AI the assistant should behave or what what it means to be helpful or what what's like the objective when we're doing our all from human feedback >> is one such thing and then like then later like defining like constitutions and model specs >> is another one and I think uh even if the AIs can do all the technical work uh we'll have to still do a lot of that and decide what we actually want. >> Yeah, alignment is the final job. >> Yeah, alignment is sort of the answer, but it's also um alignment itself can be kind of decomposed into um into like uh specification of the objective or figuring out what the right objective should be. And then uh like actually uh like achieving or optimizing the objective you've defined and uh I think the first one is not going to go any away anytime soon. And like if I think about like a post-training team and uh

[00:17:01] why you need a lot of people um to be on the team, it's it's it's just because there are a lot of different uh like areas where you have to figure out like how the model should behave and uh like there's no way of um like there's no way of uh it would be very hard to automate the whole thing just because someone has to think about how how should the model behave in this area. Jane Street started using antithesis to test our software in early 2025 and they were so impressed by the product that they decided to invest in the company. I recently caught up with Ron Minsky who co-leads Jane Street's tech group to ask about how antithesis actually plugs in. The thing that I think is most impressive about antithesis is we started using it in a team that was building high assurance software and being really careful and nonetheless it was able to shake out bugs that were otherwise going to be really hard to find. And that's important both because it helps make those systems more reliable, but also because it helps the teams that build it to just move faster. >> This matters more and more as code

[00:18:01] production is increasingly automated. >> I think in general as we've been using agents more and more, the key problem that you run into is the verification bottleneck. Just the the time it takes from people to look at code and figure out is that actually something you want to accept in your production software. And tools that make testing better are just incredibly helpful there. you just ease the verification bottleneck and make it possible for you to get more stuff done and move faster because you can have more confidence that the code generated by the agent is actually not introducing new problems. To see how antithesis fits into your development process, go to antithesis.com/bash. What is the story for why there there isn't huge consolidation in model providers? There's just so many things that point to centralization here. Is there is it yeah if you if we step back over the course of years is there something that is gonna prevent that? >> Yeah I think um distillation is the main thing that uh fights against the centralizing force uh because basically

[00:19:00] anything that can be learned uh through RL can be distilled very easily because uh um like it's a small number of bits uh it's something that you can learn from a small amount of data. So if you can coll if you can get uh trajectories from the model that uh show a behavior you can easily distill it. So I think um I think distillation is one of the uh things that fight central centralization. uh there's also um I mean there is a possibility that there'll be company specific u models that it'll it'll be possible to learn from deployment um and uh h have a company continually improving its its own model >> uh and um such a system could be provided by the uh current oligopoly of model providers or some other uh currently smaller company uh but I think that'll that'll change the game a bit >> yeah and I also want to point out that like continual learning and So it doesn't stop distillation, right? Like even if your model is improving every day, like you people could be distilling

[00:20:00] it every day. So it's like the loops could just operate at the same pace, >> right? That makes sense. Okay, so copying model behavior um you I I guess you need to know yourself what the right distribution to prompt is in order to get like the relevant model behavior. >> Oh yeah, for um just distilling uh with supervised learning um the prompt distribution is extremely important. So it's uh very non trivial to distill a model even if you have full access to it and have the cot the chain of thought and everything. Uh yeah it's it's non-trivial to distill all of the all of the useful capabilities from it. Um because you need to prompt the model with something and you need to prompt it with uh like realistic prompts. Uh you you need to have a really wide distribution of realistic prompts. So yeah, one thing that's been coming out recently is um some of the some of the Chinese companies are probably using these uh router services which are designed to allow people in China to use

[00:21:01] uh the US frontier models which would otherwise be blocked in China. Uh but there are all these uh router or proxy services that allow people in China to use these models mostly for coding and uh and these router services are collecting and selling some of the data. So I think this is uh like a very useful data set for uh distillation because it gives you the perfect prompt distribution. >> Yeah, I think this is one of those things where AI help a lot here. Like if you actually look at like you know the frontier pipelines of say like the Chinese models that they actually put in their papers, it's a lot of like humans or like they get seed prompts from somewhere which is some combination of humans this kind of data and then they like synthesize a vast coverage from those seed prompts using their existing models or like the other frontier models. And so it's like you can automate like an awful lot of this like prompt distribution gathering and like environment creation. It's just like humans need to provide like increasingly few amounts of bits. It's like the models get better, >> right? But it seems it still seems you're bottlenecked by um like having a service which has users or users are

[00:22:00] going through. So like >> not necessarily I mean like yeah that's obviously very helpful but like theoretically you can just think about like what users want or like >> like the whole point is that we don't the user says make me an application like this. Oh that didn't I actually want you to make this new feature but actually let's step back and do this other thing and capturing that whole trace is the or to the extent you could have done that anyways then you just have like RSI any >> yeah I mean like ultimately like if you have this like fully automated loop that is basically RSI right like the AI is deciding the data it's deciding the training that that is the loop but yeah I mean like it depends how much human information you need like at some point if you're just like I want traces that look like this you prompt that to the model the model will be able to like come up with like a pretty good approximation >> but what if you want to do like make me a really good politician in the like anticipate denovo like how how would discussion in like the Senate halls go or something. I just feel like there's going to be a lot of things. >> Ironically, this is actually I think easier for the distillers than the frontier labs, right? Because the distillers just like I want a good politician, they go to the like frontier model. The frontier model already knows how to be a good politician. So it just like generates those traces. Whereas

[00:23:01] like if you actually want to build the first model that does this, you have to like actually somehow like get data on like what politicians do every day and like build that. So it's actually much easier to like say like I want something like this and then like get like the AI to produce like a billion variations to actually create the thing to begin with. >> I think you can actually make a really concrete prediction based off like this observation that the Chinese labs have this router data. Like I think the thing that that just stated this originally was I was saying isn't it weird how set 5 and opus 5 are like like almost objectively worse models than like GLM 5.3 Kim K3 even though they've had access to like not only distillation but logic distillation from like mythos and so the counter here was that like okay the prompt distribution really really matters um like you need to see what users are doing so you can distill like kind of these behaviors and things in I think the prediction from this is that the frontier labs don't necessarily have much of an advantage if at all in Ral environments now because yes like user distribution matters for like general like behavior and and so on but like the

[00:24:00] best measure of a capability is the very very hard RL environments you've made at the frontier and so if you have access to those RL environments as anthropic and you have access to logic distillation and you've still made a worse model then maybe like >> then real world deployment matters more than the environment that's really interesting so but they they had to incentivize those capabilities in the first face in um in Fable or the Frontier model. >> And so it's weird that they can't incentivize them again >> uh or with a smaller model or something. >> Maybe like maybe we're just in this weird like uncanny valley where you know >> actually trying to copy that frontier model too much like the the student teacher gap or whatever it is is just like too large. And like I think people people made this point with Opus is it's like the difference between Opus 4.6 six and Opus 5 is that Opus 5 really feels like it's got this like AI as a judge checking every possible thing it's done and it's like that's why it uses so many tokens. It like tries to think about all these things but it doesn't necessarily have the big model smell of fable to know when to like stop doing that or like when's the good path to go down or

[00:25:00] whatever. >> The reach exceeds the grass. >> Yeah. >> Yeah. I would offer a slightly different uh hypothesis. So I would say there are a couple of different axes for uh the environments you can create and uh like one of them is difficulty and the other is uh realism. >> It's sort of easy to create or it's comparatively easy to create a lot of difficult environments uh like um >> that are just um like involve uh like doing a much more complicated task or doing something that requires a lot more cleverness. And uh you could you could say this is like the benchmaxing distribution because a lot of the most prominent benchmarks just involve doing some very hard puzzle-like task uh that's easy to verify. And then there's sort of uh like the realism axis where you want the model to be good in the realistic coding agent setting where there's like multiple back and forth. is the human and there's like multiple objectives and uh like I'd say um like the people like the labs who are crafting the model behavior for the

[00:26:01] first time need to push in both directions and to get good model behavior you need to really push on the realism axis and have like rubrics or some kind of human feedback that's informing uh the reward function you use there. Um but uh I think when if you try to do distillation naively, you end up just sort of matching the teacher on the benchmaxing distribution. And um uh but if you don't have enough of the environments that really exercise the capabilities in these um like trickier realistic settings, then you're not going to you're not going to get those into your student model. And I think maybe one thing that's happening is the big models uh generalize better from the uh like the tricky narrow tasks uh to these sort of uh more realistic tasks. Um so if you have a really good um like realistic uh prompt distribution for distillation, you can match uh the big model really well. But if you only have this uh like uh this distribution of

[00:27:01] easily verifiable tasks, then you can match the big model on all the benchmarks. um but uh you do worse on um this broader distribution. So you might uh that might even explain um something about the uh smaller anthropic models uh like Sonnet 5, though it's hard to predict exactly what uh what they're doing to post-train those models. It could also be that they're always changing their post-raining uh stack and they just um made they just got a few things wrong in some of these models. So they uh like I I don't know they they turned up like some um something too high and created some quirks that people really don't like. So it's like really easy to screw up post training in some way that's uh doesn't show up in benchmarks. I mean just one other sort of very basic point is just like the AI like the frontier AI labs buy all their data from big data companies and like the Chinese can also just buy the same data from data companies and they are like they are exactly there's a lot of people like you know being annoyed about this but like if they have exactly the same data and like they can buy that

[00:28:00] they can also distill it's like it's it means it's quite easy to like keep up really. >> Yeah. Yeah. Yeah. Okay. The the other question I had is how the first models that are capable of automating AI R&D will actually be trained because there's a toy version which is this thing that Ryan was talking about which is you just have GPT8 try to build GPT3 size models that are really good at like inner loop type challenges of beating video games that require continual learning or um just get getting to a certain loss with like the least amount of compute etc. But John, I think you had an interesting point that maybe that's not the way it actually will happen in practice. So I'd be curious about yeah, by the point which you have that are actually capable of automating R&D, how are they probably trained? >> Yeah, I think we'll probably do some combination of learning from human feedback to absorb uh like the researchers taste uh and uh just like creating a lot of practice environments uh which involve like doing multi-step research projects. So I think uh yeah

[00:29:03] people will in practice do some combination of those two things and uh just um each iteration like patch whatever seems to be most broken in the last iteration. So uh like researchers will be using uh the AIS uh a lot and um and we'll notice that they have some consistent weaknesses and then uh those things will either be patched by like collecting human feedback or uh like creating environments. >> Yeah. Makes sense. >> Maybe maybe useful way to think about this is like how much of the lineage we roll back and then let self play from there. Like I think in the limit like you're picturing like you know just giving them like a GPU and maybe neural nets or something and saying like okay figure out how to train a model to like do these particular tasks. Like the way it currently works is like we go up to the very like edge of the of the lineage and say okay like here are the bugs like you know Anthropic has found in their training stack in the last few months. tenants into environments like you need to train and get better on the frontier and so you obviously lock in all the

[00:30:00] previous history of the lineage but you could imagine a world in which you roll back to like you know before GRPO or something and then you have environments which like trying to get it to discover like the best will form to like oral models on and then maybe roll further and further back but I think we will be still so computer bottlenecked that like people will just keep like staying at the frontier and like diffing essentially the bugs and whatever improvements they found since the last model version turning those into training environments >> which is also really good for having non-stale like new data between model generations. is just again this is basically continual learning within the AI lab of distilling the last 3 months of AI research progress through environments and like RLHF's type stuff back into the model itself >> and and it is distilling right and that's maybe why some of us feel like it's asic is like you're always like just trying to get the last 3 months of progress and that that progress is being contributed to bys of course but it also still has humans in the loop and it feels like you know you're just constantly inching closer and closer to what the human researchers are like finding capable apable of doing.

[00:31:00] >> Yeah. >> I mean, the one thing I will say though is like obviously if you're just distilling on like trajectories, you can never go above it, but environments can go quite a far way above what a humans can do. Like it's very easy to design environment that like no human can solve, but the AI can obviously still try and solve it. >> And so that would be the path to like go ahead of just like what the human AI research. >> Do you have like an example of like >> in terms of RSI or like like you know doing it even faster than a human speedr runner? >> Yeah. I mean, I feel like in AI research especially, it's very easy to define like goals which like, you know, you could say like the loss needs to be like 1.3 or something and like no human can get there, you know, now, but like that's a very extremely measurable verifiable task and the if the AI gets there then then great, >> right? I don't know. Building like a 100 million parameter model that beats Minecraft, that's maybe too easy, but like be like a much more complicated game or something. >> Isn't it crazy that 100 million parameter models to beat Minecraft? We're calling that too easy. Like imagine you said that like five years ago. I would say a lot of research is not exactly like that though where it's like hill climbing on a well- definfined goal. It's sort of more like uh here's

[00:32:00] an intuition we have about uh some way model should be better and then we also have uh some idea for an algorithm that seems to go a little bit in this direction. So let's come up with a task that is uh sort of designed to show signs of life on this approach and uh like see if we get some uh get those signs of life and then if we do we can make successively more realistic versions of the task, right? It's like a lot more guided by intuition and then the the out the the inner loop is to elicit the uh or make test for that intuition rather than like the the the test itself leading to the insight, >> right? Like you're not directly optimizing uh for the eventual objective you care about or the practical >> like production objective. It's it's sort of uh you're um you're relaxing your objective a little bit. You're saying, "Yeah, let's relax on the realism axis a little bit and find uh some methods that actually work and then

[00:33:00] like then try to get back to realism later after the method matures a little bit." >> Yeah. >> And then there's also like more uh there's research that's more oriented towards explaining things and like uh developing a theory or a sort of Yeah. often we don't have like mathematical theories in machine learning that are that um predictive but we have like a lot of um like more informal theories for what's going on. >> Yeah. I mean like presumably the models will be trained on like some combination of all of these tasks and like some will be very easily verifiable some will be like l judge or like just ask the human like does this look reasonable and then you will the hope would be that like these would all generalize to like these much sort of hard sort of more vague fuzzy kind of tasks and like it probably will to some extent whether it generalized enough that like we could the loop can become like self-sealing without humans being in the loop at all it's like unclear. >> Yeah. Yeah. Yeah. Wait, maybe taking a step back. Here's what I here's what it seems to me that the plan uh for AI research going forward is and you tell me if you think it's going to work or if

[00:34:01] you agree with this characterization. So the bet is that we will scale up RLVR training across millions of diverse environments across hundreds of different kinds of domains. And what will emerge at the other end is an agent which has like learned these basic skill or less than basic skills around being persistent um being able to triage information in context uh eventually having like end to- end optimization of working with other agents and things like that. And such an agent will be very sample efficient within the context. You know, you've done research on how you actually scale up in context learning to make it like arbitrarily long, but you keep scaling it up. And so what comes out the other end will something will be something that it basically functions like a drop in remote worker over the course of a week or a month. First of all, do you agree that that is a bet the labs are making? And second, is it is that enough like basically learning how to learn within these simulacura within a data center uh and then getting deployed into the real

[00:35:00] world? but not actually like learning from real world deployment, only learning these meta skills from the the simulated environments in the data center. >> Yeah, I I think it's now hard to separate out like how much of the lab's effort is going towards like direct RSI versus like making generally intelligent models they can continue to deploy to collect revenue to fund the next big training run. I think for the um latter like yes that's probably just the bet they're making like and it's very clear like the pattern of like where these environments are going over the last few years. I mean like anthropics lineage of of environments is like a very clear example of this like you know first like we just focus on coding and like we're going to get really really good at that and then the task horizon that we've got from coding which is probably the lowest hanging fruit in terms of like data available on the internet to create environments like their own internal stuff that they can turn into environments. then we're going to generalize. We're going to go after finance next and like literally like just so much Excel data and and all that sort of stuff in the in the training and then you know it's powerpoints. It's like this long tale of like the working economy and like that seemed to work

[00:36:01] really well and like a lot of the other labs and thing even the open source labs have now realized that that was the correct >> that but what is the implication from that when I had Dario on the podcast the thing I asked him was if you truly expect models which will be humanlike in their ability to learn on the job why would you try to bake in all these uh skills of like working with PowerPoint or something wouldn't you just expect the model to be able to pick that up on while it's deployed and so yeah there's multiple different explanations one is just that this we expect models to get there soon, but they're not there yet. So, why not amortize these skills um into the model training. Another is that we're not concentrated on making it really good at widely deployed work. We just want it really good at RSI. Um, and this is just like a way for us to like get revenue so that we can pour it back into a model that is actually like really good at doing um RSI development and then like once the singularity happens the thing that comes out the other end will be really good at all the things which seem like bottlenecks to the current generation of models. Yeah, John I don't know if you have takes on like what um

[00:37:00] how how once you construe why there is so much task specific knowledge in these models if the if the path is like this kind of generalization. Yeah, I mean if the models were good enough at learning in context, then in theory you wouldn't be a you wouldn't need to train them on finance. Uh they would just be able to figure out uh read all the um all the books on the fly and uh figure out how to >> how to do everything in the appropriate jurisdiction. >> Um yeah, and you could argue that um you need to do um a lot of this domain specific training um just to make the uh to make them more efficient. Uh so even if they they were smart enough to figure this out on the fly, you still might want to do a bunch of RL and bake the uh bake all these intuitions into the weights. Uh so the model would be more efficient at runtime. >> Yeah. >> Yeah. I'd say in practice it does seem like um model providers are um going domain by domain and trying to strengthen the models in the highest value domain. And I I'd say that that's one of the answers to why the models

[00:38:00] have gotten so much better. It's just because um the model providers have covered a lot of the highv value domains and the most common types of skills. >> I mean I think another thing is just that like it's not that expensive to do both at the same time, right? Because like the models are massive. they can easily afford in terms of their parameters to like learn everything and like there is likely some transfer in sort of even just even if like finance is not specifically like the information is important for like RSI just the general like metal learning of like how to figure out what's important how to have taste how to like do long horizon work is potentially generalizable and like there's not that much RSI like data in the world as well like it's kind of hard to generate and like that requires a lot of effort so like if you can of amatize in this other data get some transfer from it you already have masses of comput masses of parameter space so like why not do that as well as like obviously the direct like commercial incentive of like selling a model. >> Yeah, it makes sense. >> Oh yeah, I'll add that. Um I mean there's one question about whether um this uh current paradigm uh of doing like sim to real uh will be the dominant one forever. So basically you you look

[00:39:00] at what the real world tasks uh are like and then you try to create a bunch of environments that can be simulated uh like in the data center and uh you can do RL on them and I think um obviously this has been very successful successful but it has a lot of weaknesses uh because a lot of things are just kind of hard to simulate um especially if they involve like interact interacting with a bunch of humans in real time. Yeah. So there's some question about uh like sim like whether sim to real will be the dominant framework forever. >> I think sim to real has to be the dominant framework while like sample efficiency is kind of low cuz like right now you need like you know thousands and thousands of interactions with the humans and no human is going to sit there and like deal with this basically be in the loop of RL training >> and so like we kind of have to simulate that now to like get the samples you need but like obviously if sample affinity improves a lot you'd expect learning from deployment to like become like a much bigger part of it. Though there are also other things you could do like you can learn um off policy so you can take all the traces and even without reimulating everything you can potentially learn something from them.

[00:40:01] >> Jane Street just launched a new competition and it's their most ambitious one yet. Design a protocol emulator ASIC. Basically if you have a chip that you want to test you can connect it to this ASIC and then this ASIC will simulate realistic traffic. That way you can see how the chip responds without having to plug it into a live system. Jane Street is looking for flexible generalpurpose designs, not single protocol emulators. When I was chatting with them, they suggested that I start off by trying to implement what are apparently three very common protocols, UART, SPI, and I squared C. Jere also mentioned that they hoped that more ambitious designs will also tackle low-speed USB and Ethernet and any other protocols that flex your chip specific architecture. Importantly, your design should be reprogrammable rather than smashing a bunch of specific protocols onto a chip. If a new protocol comes out after your ASIC is taped out, your chip still needs to be able to handle it. How exactly it does it is up to you, but there is one hard constraint. Your design must target an open-source 130 nm

[00:41:03] process node. That's because Jane Street will pay to tape out the most novel submissions and send the physical copies to the winners. The competition is open till January 18th, 2027 and working in Teams is highly encouraged. Go to janestreet.com/scash to download the template code and get started. I want to ask more about this cuz it it's sort of weird that you have 50% of compute that's spent on inference that is not directly helping the model become better. Like one of the key advantages you'd expect eventually digital minds to have is unlike a human who gets to have 50 years of like real world experience, a model will get to through all its instances will get to experience I don't know millions of years of deployment across all kinds of economically relevant work in the economy and right now that data is just not in a meaningful sense helping the model get better. Like it just seems so obvious that eventually models should be able to learn from this data and once they do

[00:42:00] you would have something that almost feels like a widely deployed intelligence explosion because the model is assimilating so much information um across all these deployed instances but when do you expect this kind of hive mind kind of crazy to be start happening. >> I think broadly like at a very basic level this is already happening right like just in the next generation of models. So like right now you can obviously take your deployment data and put this in the pre-train or the mid train of like future models especially if you do like some kind of filtering or some kind of like judgment or annotation or like recent you know synthesization of that. >> How much do you think that explains the generation over generation improvement? >> I think it explains like quite a bit. I mean especially like I mean this is you know I don't know whether the labs do this because you know theoretically they claim not to train on people's data but like the Chinese 100% do and like they definitely get this advantage both like obviously deploying this is basically what distillation is like they take out the models they get some of their like deployment data they get some fraction of that by like pinging the model and then they train their next generation models on it and they can suddenly do it on their own models as well like there's no reason not to whatsoever. >> I completely agree with this. I think if you zoom out far enough, this is like definitely happening. Like you're picturing this like and we're all

[00:43:01] picturing this. This is like what continual learning like the holy grail is is like >> this very very organic like live loop of like an individual model like getting an experience and like live updating on the spot and learning from that. And like a lot of things break when you like zoom into that level of granularity. But like yeah the big labs are doing this like the closed the closed models are doing this. There's also early signs of life of like people using open source models doing this at a much faster cadence. So like a good example is probably like composer um like you have some sort of model and you are able to or like you know Harvey's doing the same thing with like legal legal agents like >> it is getting very specific environments from the data that you have for that particular task and things that like you know users are complaining about and like all the feedback that you're somehow extracting from like your specific deployments and a lot of these companies have the advantage over the big labs and that they can use this data really really well and then they will create environments they like you know do a big post train of Kimmy K3 um they will go deploy it they might do some online learning as well like composer did online basically like reinforce for

[00:44:01] a long time um so yeah like there's still a human in the loop there's still a human saying okay these are the signals we care about here's how we're going to create environments from the data that we have and it's like still a longer cadence than maybe the one that you're thinking of but like it really is happening and like eventually that loop will become like faster and faster >> I mean the composer thing is interesting because this is where the model like in cursor people like press tab or they don't press tab tab on the next completion that the model suggests and based on that every single day composer gets better at like predicting the next. >> So that was that was the old tab model like they actually did the same thing for the actual not just like the tab model but the actual like generative model. Oh, it's interesting. >> And they it was it's hard because when you do online reinforcement learning, you don't have groups, right? You just have one user saying one thing and then you get one roll out. And so like you have a big variance reduction problem >> and like curs's kind of fuzzy answer to this was like oh you know we have very good heristics which are able to estimate like >> like what the what how much better than average like this response was or how much worse than average this response was. And then they would do like this big, you know, like reinforce update and

[00:45:01] then their solution to like whether it got worse or not was like if it improved on cursor bench, they would deploy the new model like every five hours and if it didn't, they would like throw that version out. >> Interesting. >> Yeah. I think your biggest problem is actually just not knowing what the reward function should be from natural data. And if you use some kind of superficial signal like did they accept the the code the the edit um you might that might get reward hacked in some way. But is it this seems like a bigger issue with the sim to real thing where the longer and longer horizon tasks get, the harder they are to simulate within a data center, right? It seems to me already potentially at least even in coding, we're getting to over the point where um there there's like not some year-long coding task that doesn't eventually require you to like talk to a client or interact with the company or interact with users. And if you think about the gamut of things we would want AI to be capable at, you want eventually super intelligence should be able to like run a business or like start a new business and make it profitable or like have a profitable day trading in the

[00:46:00] markets or win a court case. And these are all things which are very hard to simulate in a data center like inherent part of the learning there is interacting with the real world. Then so maybe maybe yeah they better learn how to get better at these things from like the transfer between sim to real but alternatively maybe you do need weight updates from these kinds of interactions in order to get better at them and then if that is the case if transfer isn't strong enough and you need do need weight updates then the fact that the models are quite sample inefficient is like maybe a deeper problem and the reason I'm curious about this I feel like by default I I don't see how you don't get some kind of crazy recursive self-improvement within the next 10 years. But the one reason why that might not happen is in terms of like weight updates, the sample efficiency of weight updates, they just seem way far behind humans, right? Like plausibly millionfold behind humans in terms of how much data a human sees from birth to adulthood versus how much a model sees from, you know, like cold start to like finishing training. And so yeah, this is

[00:47:00] all to say first of all, is there is there going to be a good transfer between simulations and extremely long horizon really complicated real that we want the AI to do in the real world? And if not, does that really mean that like the the lack of sample efficiency in these models comes to bite us? >> I think maybe the way I'd break down like the two types of tasks in which models get good and models will like still continue to struggle is whether the task is like cumulative or like you kind of have to you have this like non-stationary distribution you have to keep learning and like relitigating a bunch of stuff. So like maybe an example of a cumulative task might be RSI like it's theoretically possible to maybe like have a less than a million token like you know Python file which like from scratch trains a model that is capable of recursive self-improvement and like every discovery that you make is kind of a line in the sand that you hold like if it's true that you know for RSI we don't need to discover a new attention variant or whatever like once you've discovered attention and then once you've discovered you know mixture of experts and once you discover GPO you just add that to the training stack and like that's that's there and like a good example of this is like you 5.6 all

[00:48:00] training 5.6 ter whichever one openai told it to train like it didn't have to go back and discover attention like it basically probably would have called a bunch of like scripts which is like pre-training.sh and post training.sh and just did that. So like that's an example of like a cumulative task. I think the real world and the reason like people are thinking so much about like continue learning is it's it's not really a cumulative task. Like imagine like in a law firm you have an agent acting as a legal associate. Like that's a very non-stationary distribution. you have to be able to fit in your context like all the relationships between all the important people at that company which are also changing all the time. Yeah. >> Um you have like all these like implicit like ways about how things are done, where to find information etc. And like that's not as as clean of an example of a cumulative task as like RSI is. So I think that there will be this breakdown between tasks but you know like if the labs realize that and they they do believe that RSI is cumulative in the sense that like we don't need to go back and discover some brand new like architecture or whatever >> then maybe more and more effort and comput gets focused on that versus the >> it's so unfortunate that RSI happen to be easier than parallegal.

[00:49:01] >> Yeah. >> Um yeah I don't know if you guys have thoughts on this. Yeah, I would say there's uh like models um are today's models are um weaker than humans in a lot of different ways and uh some of them might have to do with um sample efficiency in a certain regime. Uh where I mean in in some regimes models are very sample efficient like learning in context. Uh but then there might be some like medium length regime where they're less sample efficient because humans can uh do uh some kind of weight update um more efficiently than models. So so I think like being less sample efficient in certain regimes might be one of the sources of weakness. But then I think there are other sources of weaknesses that are completely different than that. for example, uh having lower diversity of thought than humans or um yeah being bad at certain kinds of long horizon judgments. Uh I mean I think a lot of um what people call taste is uh is something about um behavior that um

[00:50:03] works in the long run and that people have realized works in the long run. uh not everything but like some some aspect of taste like especially for something like software engineering like I think a lot of taste is like what are the systems that are going to be maintainable and >> uh work well yeah in the long run of this project. So uh yeah I think the weaknesses of humans which limit um RSI along with other things uh are um yeah there's a variety of them and some of them are related to sample efficiency and some of them aren't. Maybe an interesting thought experiment is like if you were able to give a model like a context window of I don't know a trillion tokens or whatever you would have needed to fit in like your experience prior to like let's say RHF and like it's got all that experience in the context window and has the same sample efficiency and in context learning ability as it does at a million tokens like do you think taste is then solved like would it be able to like

[00:51:00] make the same judgments that you did or is there like something fundamentally missing apart from just a longer context window with the same sample efficiency? Yeah, I mean it would have to be trained to learn from that context. So, uh I'm not sure um yeah, either it would have to be um trained to learn uh the right update um to make from that context or >> you have to just like dump it all in like your whole like life like research experience. >> I mean like you still need the data to train it long context, right? Like even if you could theoretically get like a trillion context, you would need a trillion lengths of data to train it. Like right now you have like context. I'm just asking if you had that >> in theory. I think yes. I mean, this really just comes down to the question of like how metalarnable is taste from like shorter horizon episodes and like I feel like there's no obvious reason it's super long cuz like humans somehow develop taste with not having many long episodes like we don't live to be like 10,000. We have like you know we develop pretty quickly, right? And so like you know if you think about like even like in a PhD the difference between like a first year PhD student and like a final like postto or something that's like

[00:52:01] five years maybe and they've only done like maybe like 10 fifth 30 research projects in total but somehow they develop taste quite quickly from like a relatively short succession of like small things and so like theoretically it's you know possible to develop it like that. The AI obviously will have vastly more experience in which to develop taste to like metal learn it and there's like how well does that generalize to like really long horizon things is I think the question which I think is really unsolved at this point like we don't know going back to this question eventually there should be a regime where EIS are learning a ton from each individual instance of deployment that they have well currently you could say there's a meta fuzzy process by which models do improve from deployment but I feel like it's a very weak uh very weak uh feedback loop Do you see this around the horizon where there's this like hive mind kind of learning that's very rapid and um and if if so how exactly does it happen? >> Actually I would say that around uh will we get a hive mind uh that learns from all of its deployment experience? I mean a big part of that is actually about

[00:53:01] incentives uh rather than um being a technical question. So like companies aren't going to want to um have the model provider learn from all of their deployment because that that might just reduce the uh advantage of their business. >> I think that maybe the economics of this will pressure not necessarily weight updates to one big like common shared model but like kind of like modules that get subbed in. like a very obvious example. This is a Laura, but it might be something else like you know there's been a lot of work to try and fit like an arbitrary context length into a fixed size like this is all the linear tension stuff and all that sort of stuff and like cartridges which are essentially KB cases trained to be very very compressed KB cases to fit in a lot of information. That's another example of like you know something that like companies may be willing to sign up for if that's get subbed into the model and it's not like actually changing the base underlying model itself. So like there's many different versions of like learning from from your data in real time and like the latter ones are not really helping the big labs cuz they are just these modules. But I think the like the like

[00:54:02] the the economic pressure will force like the labs to go down that path first before they can embark on this like you know >> so which economic pressure though cuz I feel like even if you have like a bunch of cartridges or laws or whatnot you can still just like take all these traces and just like distill this dub this to the pre-training of like your next generation of bombers. >> Yeah. So it it may be a more indirect form of learning that the big labs are getting and it's that's obviously still really valuable to them, but I can't imagine a world in which we start off with like you know we're going to just like directly train this one big model on like all the exact data. I think it will definitely like go through stages cuz cuz I mean this is assuming there's like one discontinuous event whereas like suddenly we fix like weight updates continuously and like in practice I think it's much more likely to be like the cartridges and stuff allow you specialize in deployments then you generate traces you put that in your model like 3 months later you come out with a model which is better at this stuff you specialize it again and you like consolidate it again and then eventually we'll just like make this loop faster and faster so instead of like every 3 months we release a model now it's like every week and then every like day and then every hour in which point we basically have obviously solved it. Yeah, and I think this is a good

[00:55:00] point as well because um you asked like kind of how far off the current paradigm we are from being able to do this. I think like we've we've done a bit of research to this and people have done a lot of research like at a really large scale like when you wash out enough noise and you have large enough patches like this outer loop process of like putting data into mid training and creating our environments like it does work in like some sort of continual learning regime. But the problem is like when you zoom in close enough at like a micro level it's like I've got one model and I'm trying to update it again for like a law firm or something. Um, and I'm trying to do that very continuously like with a relatively small amount of data. Um, like all the methods kind of break down a bit. So like if I SFT the model on just like you know successful traces um off policy on policy like eventually in the very iterative regime like when you're doing like you know hundreds of these micro updates you see both catastrophic forgetting you see forgetting of like previous information I've learned on top of the base model that was much earlier on and I see degradation of general like use general capabilities um you know on policy distillation seems to like push this

[00:56:00] horizon out a little bit but it still eventually succumbs to the same Um and RL is not very good at like it it is good at like getting capabilities in but it's not as good as getting like knowledge in and like just this very explicit knowledge of like oh okay like this person does this at this law firm and like this is a very specific process we find and you have to pour in a lot of compute to create the right environments to get the the knowledge in >> do you think that the fundamental issue here why you get worse at any of these other skills or there's forgetting and stuff do you think it's fundamentally an issue of capacity or it's an issue of techniques >> uh a little bit of both I think like SFT and even like distillate like on policy distillation can be like way too destructive. Um like the reason RL is so nice is because like yeah it it changes a very very small amount about the model and there's like a lot of evidence for why this is the case and so like it kind of just like tweaks it in this very very very small like loss value to like get it into the right point. Um, but that also then limits what you can do with RL, like how much you can actually change the model. >> Or sorry, are you're saying like the reason this isn't a winner or take all potentially is that is just like very

[00:57:01] hard to distill that much information into the base model >> without ruining something in an interface. Like it's easy to distill it into like a different base model. Like this is where I think it's mostly technique. It's not like it's definitely not like just like there isn't capacity. Like if you had some model, you know, with all this data and you take like literally the same size model and pre-train it from scratch with like all of this stuff in mid training, it will be better. And I think that's a lot of what's happening today. >> Yeah. And so it's very much like there's, you know, a bottleneck that stops us from just keeping training the same model forever versus just like getting all the data from the old model and like training a new model from scratch. And this is exactly saying like some combination like plasticity and like catastrophically forgetting in that like you know if you just naively train on like non-stationary data because you're adding new data as you go basically this is messing with the data distribution. So like the old stuff is just forgotten and we don't really have good methods to like stop that from happening. >> And so maybe at the in the limit you're like just bottleneck by retraining the model from scratch with all this new information. >> Yes. which of course is like very expensive like training a model from scratch is is expensive >> but you're going to do that anyways and >> I mean not necessarily I mean like maybe eventually if you have continual learning you never train a new model you just like just have a model and it keeps

[00:58:01] learning and like expanding right >> but but there's there might be like some deep technical reason why that's very difficult because of these like >> I mean that's the question I think we have pushed back like how much from scratch we need to do like it is definitely possible now to take like the pre-trained base and like do very good mid training on top of that like kind of continuously plus some RL from like different checkpoints that are later on in the training and like that's looking more like a learning but certainly not the case of like you know take the most recent model apply a couple of very small updates and like iteratively like never lose it. >> So but isn't this like I'm a bit confused cuz isn't this literally what happens during training where during post training or something you just have like you have a model that's already gone through so much training and then you like distill some fork that's been further RL or something. Isn't Isn't that literally what happens? And like >> it's it's still at a large enough scale I think that you're washing out a lot of like the noise and like you're not just focused on one distribution which as Baron said is like you know that is now a very if you're just like focusing on one task right like that's I mean in the eventual regime you'd be doing I don't know there's billions of deployed

[00:59:00] instances you're like doing you're learning from all of them at once and so hopefully there's some washing out of noise and stuff from that right >> maybe that scale. Yeah. >> Yeah. I mean I think like definitely is as I was saying like you can do continual mid training for like a long time and you can like roll back to a checkpoint give a new mid training data >> but at the same time like you can't do this like indefinitely like if you just keep continue mid training the same base forever it just like get it does it's sort of asmmpto like you can't just learn new stuff in that base and this is why people end up training new bases like otherwise you would just keep m training the same base forever whenever I finish recording an interview I immediately brain dump all my thoughts into slack things like what was most interesting and what should get cut this ensures that my editors have all the context they need to start editing the episode. But it's not like these brain dumps have any clear timestamps and my unedited recordings are many hours long. It can take a ton of editor time to even find the exact moments that I was referencing. So, we decided to try adding a Grockbot producer to our chat. Now, whenever one of my editors post a rough cut of the episode, Grockbot opens

[01:00:02] a transcript on its own computer and starts working, usually before I've even seen the message. It takes the notes that I dropped into Slack and it highlights the relevant snippets in the transcript. It also uses a big case file that I've compiled with all my preferences so it can suggest potential edits. And when it's done, it sends me its top clip candidates so that I can review everything from my phone. This has worked really well. Being able to send informal messages like I'm texting my editor and then having the transcript immediately reflect my preferences has just been so helpful. Try Grockbot yourself at x.ai/bot. Okay, let's talk a bit about data now. So, I'm generally interested in this question of how much of AI progress is just explained by data progress. Um, doesn't mean it will be necessarily hard to automate, but that's a separate question. So, is there some data distribution which if you trained current architectures on would result in a super intelligence that totally dominates human experts across every single field. >> Are we talking about like pre-training

[01:01:00] plus post- training data like environments as well? Like I think the existence of this is obvious. It's just like whether we can create the right environments. >> Yeah. >> In the trivial case, we could just train it to output the Python file which like trains the actual super intelligence like just have that memorizing the weights. >> Like yes, there's probably like a ladder of oral environments that is possible to construct such that you would get a AI researcher which is at least as good as a human researcher. But the effort to climb each successive rung grows like kind of exponentially. Um, and that's going to be the two things that you have to trade off against as to whether like like how fast we're going to hit like that final run where it's where it's better. Um, I think that's fairly clear and I think there's like, you know, we're still relatively early in like our environment creation. Like there's a lot of asymmetries that we exploit in order to create good environments. So, one of those asymmetries which we've talked about before is like it's there's environments where it's easier to go backwards and forwards. And like what I mean by that is like it's very easy to define this like complex data generating

[01:02:00] process. And this is like this kind of latent variable you keep hidden from the model. Um you can generate like arbitrarily complex like environments and the model has to do a lot of like irreducible like token spend and irreducible work to figure out what that data generating process was. Um there's asymmetries in terms of like you know you can inject information from the real world. So like anthropic finds a bug um through like you know tens of thousands of human and LMR hours combined and like turn that into a very very neat environment which a single LM could theoretically find within like you know a few million tokens. So like there's all these asymmetries which we're cherrypicking and like we're counting on like kind of this task horizon generalization. Um but I think yeah again there's just going to hit diminishing returns at some point like at some point diminishing returns and how hard it is to create those environments in the first place like coming up with them because you can't necessarily just like have these really like these processes where it's easier to go backwards than forwards like you actually have to sit down and construct like something that looks like with humans like you know a long enough time horizon like it's going to be a really complex task to create. Um, and then there's also going to be like the

[01:03:00] compute and time bottlenecks for the agent to actually do those tasks. Um, so like I think you're just going to start seeing this like curve to flatten out. >> I saw something about how someone fine-tuned the talkie model uh which is only trained on data up to 1930 on this uh like modern coding agent data and um and it did better than claude 3 opus on on Sweepbench. Um so so this model that has uh like no knowledge of code whatsoever can be fine-tuned on a moderate amount of data and uh like behave better as as a coding agent than this much larger pre-trained model is pretty crazy. And it kind of shows you that like once you have an example of uh like the right expert behavior, it's actually surprisingly easy to like copy that into a like relatively weak model. >> Yeah. But a counter example to like that kind of is the there was a paper recently where they trained it up to like fifth grade maths. >> Um and like also like primary school like English and stuff. So it was like a decent language model and they tried to

[01:04:00] RL it to do like you know late high school and college maths and the gap was just too large. Like they couldn't get it to climb at all. Like but if you did like successive runs of like you know your seven maths and then year eight maths and so on like you could obviously climb to to year 12. So like again it's just like what is the distance between the rungs on those ladders and how hard is it to create? >> Yeah. And this just comes back to like the RL signal problem. Like RL is not very good at like exploring right now. And so if you if the model can't like get in like you know 128 rollouts it's very unlikely to get signal to like progress. And this is why like in RL we need like curricular whereas like in pre-training we don't because like it's that's not a problem for pre-training at all. >> Yeah. And and again, pre-training data is different to post-training data. >> And I imagine as we continue on, like yeah, humans will be involved less and less, but that doesn't change the fact that you're bottlenecked on like how much signal you can extract from the the real world. So like there's a lot of signal in the world and that's true like you know there's people doing like spreadsheet tasks, there's people doing like legal tasks and all this sort of stuff, but you know the capability frontier of where the models are at now.

[01:05:00] like how many bits in the world are actually like really relevant to like improving the the model's capabilities like you know how many new maths problems are being solved that like couldn't that are just beyond the reach or grasp of the current models like how many new coding problems are being created or or solved that are beyond the reach of the current models and like I think that's why the diminishing returns kicks in cuz like even the world as a whole is not giving you the bits that are useful for tipping you into the next like basin of of capability. >> Yeah, I I totally agree with this. So it's like really a question of like where the signal is coming from and so like the signal doesn't you know in pre-training the signal is like already in common core right like for the tasks that you care about in pre-training the problem is it's not just like getting signal at all it's like filtering out all the noise that exists and that's quite an automatable process but like as the models get better as we enter mid- training and post- training the signal just like doesn't exist anywhere in the original data we have like no amount of filtering will like get this you know there's no like hidden proof of like the millennium prize problem sitting in common core we can just like filter until we see it And so like at that point you have to get bits some other way either from humans like directly like asking them to like write out their

[01:06:01] reasoning or like by like creating environment where like humans decide like what environment should be created what the objectives of these environments are or like you know some kind of like training on like the human data that exists in deployment like you have to get the bits from somewhere. >> Yeah. Yeah. There's a question of how much of the progress in pre-raining is being driven by data. Y >> I I did this investigation with um Jerry Han who's a student at Princeton where we basically trained all the recipes from 2019 till now pairwise with all the data sets from 2019 to now. So you train like GPT2 on the newest data set like ultra fine web and you train deli which is the newest training recipe or the open source training recipe on like the pile or some old data set and you do like the whole grid and you see the getting to some level of capabilities. How much less compute does it take across this grid? And you see that the the data seems to explain like 9x of a compute efficiency gain. Uh but the the architecture improvements explains like

[01:07:00] a 3x computer efficiency gain at a very small scale. Um and so to the extent that that is true at large scale that most of the pre-training computer efficiency gains are coming from better data how much can that continue like can you keep just filtering data more and more and building more and more synthetic data until yeah do you have a sense of how much this kind of pre-training progress can continue? I think my my prior is that like again the low hanging fruit is like somewhat exhausted with like we got the internet as this big block and like there's it's not like the internet is necessarily like growing at the same like rate. All the useful stuff on the internet is growing at the same rate. So like we've probably got like a bunch of like.1% loss drops to go but like not definitely not as many as have currently occurred. >> But like that's also really interesting that like you know you find this like what cumulative like 27 times improvement across both. I think like it was epoch or someone who estimated like three times a year since 2019 which would imply something like you know 3 to the 7 like over 2,000

[01:08:00] >> like times improvement. So like where's that missing you know 100 times or whatever coming from like that probably gives you a good signal of like how much is this is like post training. I think the explanation has to be that a lot of the comput efficiency gains are scale dependent and we're studying at extremely small scale >> and that raises the question of do the data comput efficiency gains or the algorithmic comput efficiency gains have more scale dependence I don't know if you guys are prior on that we just didn't have enough computer to investigate that question >> I mean like just naively right like the theoretically the scale dependence of the architecture is like fairly wellnown >> and like you can fit a straight line to Whereas like I would have no idea how to do that for like combining pre-training plus post- training data and mid-training data. >> Funny enough, I feel like data is actually more like more important with scale. Like I feel like architectures are kind of like a one time like you know an architect I think like combining like saying just like an x% efficiency gain is kind of misleading because like what an architecture does is like let you reach like a qualitatively new regime which you couldn't reach with the

[01:09:00] old architecture and then within that regime obviously the data is like the primary thing determining it but like you know if we say didn't have like even like GQA we're doing like full attention all day we wouldn't be able to do like a million it would be like ridiculous expenses to do a million context and then like because of that we couldn't we could never use the data which is like actually at a million context and so we couldn't get these capabilities even though like if you just do a naive like how much does this do at like 2k context where the architecture isn't unlocking anything then like the data you know that will look much more important than in some sense it is right it's unclear to me that these things are like really just like multiplicative gains in this way >> I see so but then what is the take away for the scale dependence of data >> so I mean on scale dependence I think like a lot of the like mid-training and post- training data we have now is like actually gets better with scale cuz like a lot of it like the very long context horizon environment really requires like big models to be able to like make use of them. >> Yeah. >> And like this is not, you know, if you try and train like your 100 million parameter model on like three bench traces. It's not going to get anywhere. Like it's not going to show you the same kind of improvement that you would get if you train like an actual sensible

[01:10:00] size model on it. >> Yeah. >> Yeah. And like it's hard as well now because so many of the architecture changes like you look at like Kimmy for instance like or DeepSeek, they're doing these architectural modifications with not just like dropping the pre-training loss in mind but like for instance how the models are going to be used in the real world. like the inference efficiency like having some form of compressed attention in the deepse models is not necessarily geared around you know this is fundamentally like a improvement it's just like okay we're considering how the models are going to be used >> right right what one question I'm curious about to understand the future is how parameter scaling will go as we're getting into more of a RL heavy regime like I don't know I don't know how fast historically yeah you can look at sort of opensource architectures and see how fast parameters have been scaling and maybe it's like roughly 2x every for frontier open source models and to the extent that like even frontier closed source models have like 100b or 200b active parameters. Do you think that like keeps 2xing year over year or now that we're in an RL regime where you also want to conserve compute on um rollouts that and also maybe there

[01:11:00] is like a threshold effect where you have enough capacity and at the point does increasing parameters arbitrarily doesn't matter as much. Do you guys have a sense of in 2030 how many active parameters will a frontier model have? Yeah, I think for the next few years we're going to be like like because we're so focused on doing longer and longer horizon rollouts for RL where like inference efficiency ma matters a lot. Um it feels like the the malls aren't necessarily saturated on their ability to do that where the bottleneck is still the environments and so we might see like a little bit of plateau like I I I have a feeling that you know like mythos and and the GPG models are much smaller than like you know the 10 trillion parameter range that that people are talking about. um even just naively comparing them to open source models, you can probably back out that conclusion. >> Um so yeah, probably for the next few years I wouldn't imagine a huge growth in the number of parameters, but again like there's so many different different things to trade-off here. Like you decide the size of your model based on like how much pre-training data you have and then like the difficulty of the RL environments that you've got to train on. And you ideally want to like get to

[01:12:00] the optimal point where you know you can get like a decent pass at one or something on like the the hardest environments you have and like it wouldn't make sense to like make a big bigger model pass there because then you're just paying like much more inference flop than you need to. Um so there's there's a lot of input to this like depends on how quickly you know like Mccor and then inhouse these these guys can scale up the complexity of the oral environments they're training on. >> I would expect the models to keep getting bigger just because people are scaling up compute and the GPUs are getting bigger. But I would say exactly how much they get bigger depends a bit on the scaling laws in non-obvious ways. So one thing is that I think um like data efficiency is going to be a bigger driver than compute efficiency of uh like the exact architectures people use uh now that we're getting to the uh regime where we're sort of running low on um like high quality pre-training data. So that might affect how sparse you want to make the model. And then I also think we don't uh understand sparsity that well >> and it's like parameters are a different resource than active parameters but it's

[01:13:02] uh and um like sparsity uh has um definitely increased a bit but it's not clear that it's going to keep increasing without bound. There might be some kind of sweet spot. >> There's an argument that sparity should u make data efficiency worse because you might have to learn the same thing on multiple experts. uh though that's debatable. So I think we don't um I don't think we we have a good enough theory of scaling laws that we we really understand why sparity is helping and to what ex how much it'll help and if that'll uh like plateau at some point at a certain level of sparity and >> sorry can you spell out exactly what the implication of uh data efficiency would be on so it sounds like you'd say well it should there should be less sparity but what are the other implications on parameter scaling? I guess uh just that the scaling law you're you're not necessarily looking for the most uh you're not trying to optimize compute efficiency. So you have all your um choices you can make on the architecture and uh each of these gives you a

[01:14:01] different scaling law and then like traditionally you would look at some kind of envelope based on compute. Uh so you would look at performance versus compute and take the envelope of uh like the yeah uh the best models. Uh but uh like if if we're making that decision based on um data so it's like uh yeah we're sort of assuming we can um spend a lot of compute um and uh like we're sort of data is on our x-axis um instead of uh instead of compute uh then we just get a different um set of optima or a different set of models that are on that frontier. >> Yeah. And I also don't think that we've necessarily like you know doubled like the size of the models every year for the last few years. Like the people have been training like one trillion parameter models for at least a few years. Like there was even an open source one called Falcon. But like Liam from Periodic Labs like I think post yesterday on Twitter about how like an early experiment at OpenAI was like training a one trillion parameter model that was very very sparse.

[01:15:00] >> Oh yeah. That was what they did before open AAI that was like at Google the switch. Yeah. So like it was like um you know very very good at like knowledge but terrible at reasoning cuz it was so sparse and so like yeah it feels like we've been playing in this like 100 billion to you know up to two trillion parameter range for like at least a little bit and like it certainly hasn't been. This is like nice linear increase. Yeah I mean I feel like there's two things. So as as Charlie was saying like inference efficiency is super important for RL rollouts and so like this will really push down active parameters quite a lot and then I think the total parameters really depends a lot on the hardware as well. So like you really need to get like very high memory bandwidth and like VRAM size to like actually be able to serve like multi- trillion parameter models. And so like you know right now you people still using a lot of like H100s and stuff and so as everyone moves to GBs and then V Rubins will get like more actual like the ability to scale and like actually serve and like do like large RL inputs at like different at larger scales. The data crush I think is interesting because naively like larger models are much more sample efficient in like the actual data points and so like even if

[01:16:01] you're like not saturating the model it's still better to go bigger because like the models larger models generalize better and like get to a better loss for the same amount of data and so right now I think we kind of have a lot of data and like that's not the constraint rather than computing and so we're having like small models which are like very inference efficient but if computers no longer the bottleneck it might come back to larger models which are like un sort of undersaturated but like they have this generalization ability because they're larger >> if if you just look at like the basic chinchilla scaling law and you just maximize out uh parameters. >> Yeah. >> It actually decreases the amount of data you need to get to the same loss very little. >> Yes. If you go to infinity on parameters, the amount of data you need, I think goes down less than 10x just because the nature of like the power >> we're now on the way too much data side of the chinchilla laws, right? So right now we overtrain models according to chill and so we could easily go back to a point to which as we're running out of data, we move back to like the chinchilla optimal point or even like a bit on the overtraining like you know undertraining model side. But surely like even with these new chips that come online and stuff like we're just going

[01:17:01] to be so comput bottlenecked for the next few years that that won't necessarily >> I mean this could well yeah this depends on like the ratio you with like training and inference comput really. It's like if you're super bottl necked on data not on compute you should go bigger. If you're super bottlenecked on compute you should always go smaller and then like yeah but you can also use computer to generate synthetic data. So it's like one of these very hard things to predict. >> Yeah. I think uh part of the reason it took people so long to figure out the scaling laws in the first place was that if you don't get all these things right then you don't get such a clean relationship and like the uh the beautiful stray lines on graphs like hide a lot of complexity on uh how you have to make sure every like to scale every hyperparameter the right way or like parameterize your optimizer in a way that um scales and where you don't have to change your hyperparameters as you change the model size >> and and bugs have their own clean scaling laws as well, right? Like you know like with Kaplan forgetting the cosine and thing or like even just like not considering embedding parameters I think and so that messed up the estimate at smaller models because embedding

[01:18:00] parameters are a decent size of the model. >> Uh a bit on RL. So I feel like a year ago a lot of people were making this argument that RL will not be super successful at scaling for models. I think John you wrote a research paper where you were pointing out that models learn one bit per episode when you RL basically learn did I get the answer right or did I get it wrong and I wrote some blog post earlier this year I was like it's even worse than that cuz when the pass rate is low when the model is very unlikely to get the answer right it's it learns almost almost nothing at all from an RL episode but we I look at the models today and they seem pretty smart and it seems to be the result of scaling up RL Baron you had a post I think a few weeks ago where you're trying to explain what's going on. But why has RL been more successful than one would have naively thought? >> I mean, so I think the success of RL comes down to a bunch of different things. So first, I think what is slightly underestimated is actually the mid-raining. So an awful lot of like what we see as successes of RL actually

[01:19:00] comes from like very very good mid-training data, which is basically where we're like essentially doing pre-training but on like synthetic reasoning data and like the kind of environments that like get the model warm started for RL. And so this actually takes the model like almost like 80% of the way to like the final RL checkpoint often. And then what RL does on top of that is it does like a lot of you know essentially tweaking to the policy. And so this is one of the reasons why it doesn't need like as many bits as you would naively think. It doesn't have to learn all of these behaviors from scratch. It needs just like a few bits from these episodes which you do get. And then the other thing that I really point out in my blog is that these bits are actually extremely high signal compared to like regular like pre-training which is why you need RL at all versus just like SFTing on like successful reasoning traces >> because it's exactly the bits about how to get the answer right. >> Well, there's two things. So yes, one, it's exactly the bits about how to get the answer right. But like this is not exactly how you think of it because in SFT you have a trace, right? You have like a bunch of math reasoning and then the answer at the end. The bit is still there. Like you still SFT on the answer token. So that bit is still there. What's important is that the objective

[01:20:00] ignores all the other bits. So in SFT you like have like you know to try and match like the exact reasoning tokens that the model produces. So you're essentially getting like too many bits about like the exact way this other model you're training on reasons. For RL you only get the one bit and that means that like this it that signal is not drowned out in the noise of like all the other bits the model has. And so that's what really like it's really super dramatic like increasing to the signal to noise ratio during training which is why like RL is like so dramatically efficient in terms of steps. I don't know if you guys have thoughts on that. >> Yeah, I like there there's been so much debate about like what RL does to the model versus like you know mid-training or SFT or whatever. And like you know everyone talks about how you know passer one will go up but pass 256 will go down like very rare correct reasoning traces will be like down weighted and kind of like outweighed by gradient signal from like easier kind of reasoning traces. And I I think the simple like way to view RL now is that if you have a large enough like a large enough amount of compute to sample a large enough group size such that your probability of getting a bunch of correct answers is

[01:21:01] like past some like not insignificant probability then like it will be upweed and like to to to Von's point like basically mid-training and you know more pre-training like the the pass at one the starting point for RL like scales in a log number of pre-training tokens. >> You can ask some very basic questions. Um I I I guess that answer makes sense and maybe there's empirical research which shows that this is what's happening but then I just look at the models themselves and I don't know what's happened like yeah maybe you can give me a sense of what is the basis of the AI progress over the last year but if if it's yeah maybe it's just upweing the uh the policies which were going to do the correct thinking anyways but it just seems like qualitatively the models have gotten so much more capable and anyways maybe there's no nothing to there's no inherent contradiction there but how do we square square like the relatively small impact this take would

[01:22:01] imply that RL would have from the actual qualitative capabilities the model seem to be gaining >> so like one thing I want to point out here is that like it doesn't necessarily imply that RL has a small like effect right even if you have a few bits and like you only change the parameters a small amount like the actual impact on like function space the model like the input output mapping can still be like super dramatic. >> You know, even if it's like even like one bit can change like your function space a lot and that it can like rule out like half the hypothesis space which is huge. So like I don't think it's necessarily the case like small amounts of bits, small amounts of RL once you're starting from a really good point means that like you don't have dramatic impacts in behavior at least like not necessarily. >> I I think it comes down to two things. cuz I think the first thing is that everyone was hoping that RL would like generalize this reasoning across like all these different domains and I don't think we necessarily got this like horizontal generalization like just training on math doesn't necessarily make you the greatest coder like you do have to do RL on on code environments. I think what we did get though is like horizon generalization. >> Um like the models just learned how to use more tokens for for longer and still

[01:23:00] make progress on some sort of task. And so like you can train on environments where they get longer and longer and longer and then put them into a completely new environment and yes like they may not have generalized the reasoning patterns which allow them to do well in that environment but they've at least generalized the ability to like continue on that task for longer which is correlated with like success. I think like there's a paper called edgebench which showed that the rate at which models can work for longer is like doubling every 3 months. And so that's a clear evidence of generalization. And I think like the the final way to think about it is like um in pre-training there's this idea of like quanta. So you have this very smooth like pre-training loss curve. >> And when you actually look at what's happening in the model like the model is learning all these like very discreet like tasks and there's like all these like emergent points where there's like kind of a phase transition like it didn't have induction heads now it has induction heads and there's like tens of thousands millions probably like hundreds of millions of these things and you average them all together and you get this very like smooth loss curve. I think like to a to an extent like a similar thing is happening happening for RL like there is this very slow ad loop as Baron mentioned of you know we will

[01:24:00] train a model and then RL it and then like the next kind of model iteration of training we will dump a bunch of these synthetic reasoning traces into the mid training data like we're kind of hitting all these quanta for all these different tasks and like on an individual task level it may look like a phase transition and like you're suddenly going from like a.5% pass rate to a 90% pass rate on like a particular like finance task or Excel task. ask or whatever, but you average all these things together and plus the horizon generalization, you kind of will go, wow, we've got like qualitatively better models. >> Yeah. I mean, I think a lot of this as well is just like I think RL does generalize a bit like suddenly you get like some transfer between like math and code or like puzzles and math and this kind of stuff. Also just like the amount the sheer amount of environments I think the people are targeting is just like vastly greater. So like you know before when you try to do you know some task which like you do in your daily life like two years ago like the labs wouldn't really care about this. they wouldn't like train the model for it and now like it's just so much broader. They have a lot of environments targeting this specific thing. >> Early in the conversation we're talking about are all in the context of causing this entropy collapse or just you know

[01:25:00] concentrating probability on solutions the base model were already done. Um and causing relatively sparse updates in the policy. Um, but when I think like when I think about I think the there's also another story about RL which is going back to the Atari games and then AlphaGo coming up with move 37, the super creative move that because it was never initialized on human data, it can like think in ways that humans are not even thinking and come up with extremely creative solutions. Yeah. Do you have a sense on when we should expect or if we should expect RL on LLMs to result in things like move 37 just extreme creativity even beyond human creativity because like there's just denovo uh dova initialization of intelligence. I mean so a couple of things here like first off I think that the alpha go is using MCTS which obviously does like more exploration and like stuff than regular policy gradients but I kind of also think that like RL doesn't necessarily like reduce the creativity and like I mean even if we I think you

[01:26:00] know this is obviously qualitative but if we look at like the you know the open air hinge incident like these models were coming up with like multiple zero days at a time to like break out of the sandbox and like this is clearly like some level of like move 37 creativity I think already which we just get from just like the general generalization properties of the LMS like I don't think it's definitely not the case like RL is like totally destroying their like entropy >> especially on long horizons >> yeah I mean one thing that people call creativity is just uh solving hard search problems so uh and so that's like uh like move 37's obviously an example of that or like uh writing some kind of poem that satisfies a ton of different constraints um so that's something AI is obviously going to be extremely good at uh if if trained for it. Um then uh there's another way in which the models uh like uh the diversity of their outputs uh is a lot lower after RL and they sort of develop these ticks and like um even though the models seem like they're good at writing when you do some

[01:27:01] kind of like distributional analysis you find that like they're reusing uh certain themes uh like all the time and they're using the same character names all the time. So there's actually um it's not like you're getting the same kind of diversity that you get when you like from human authors. You're sort of getting one really good uh like style. So I think that um like that kind of diversity has definitely uh been like cut down by RL a lot. And in fact uh now oh yeah since we were talking about distillation earlier that's sort of uh something yeah one thing that's happening is that so many people are distilling uh mostly from claude that like uh like all the open wade models write the same way as claude and use the same like have the same ticks. So this uh seems kind of concerning to me that we're having this uh like this monoculture emerge. >> Yeah. Again, I don't think this is like fundamental to RL as like a method though. And same with distillation. Like even with distillation, like you're just

[01:28:00] training on the data. It's like just because your data is not like super bored, that doesn't mean like the training method itself is somehow wrong. It's like a problem with the data. And I think a lot of for instance like the RL like entropy collapse is basically due to like exploitation of fairly simple like verifiers when you don't have like a huge diversity of environments cuz like for instance like the writing I think the writing is presumably graded by some judge and like the judge has some specific ticks and like the model is learning to award hack the judge and that's why like it collapses but like this is really a problem with the judge. It's not a problem with like RL in general. >> Um okay super rapid fire predictions about the future. So I want timelines on the following couple questions. By when do we have models which you can here's what the it feels like to a user. You basically hire them as a drop in remote worker for all kinds of white collar work not just coding but I don't know video editing um uh law parallegal etc. Like it's like literally an actual

[01:29:00] remote worker with like full computer use with like literally a month of seamless learning and operation and executing on like complex projects and it required interacting with other people etc etc. It's like everything a human worker could do over a month. >> If you like mandated to use like a browser or whatever rather than like these the again the firm setting up the information to be like programmatically accessible like maybe a couple years. But >> if it's not like browser based like it can send Slack messages, it can do all this stuff. But I'd still probably say around a year. >> Yeah. Yeah. I mean, I would say maybe like for the like full generality maybe like three years, but I think to Charlie's point, we will end up with like a lot of people like making their organizations easier for the AIS to use. And so you get like 80 90% of the way there before that. >> So, but the thing that's the the diff between one year and 3 years there is just literally like >> like I think there's going to be like a long tail of like miscellaneous stuff which like some human can do which like will take the models like quite a while to do. Yeah. Like you mean are you thinking of sort of computer stuff or like basic cognitive capabilities? >> I mean I think this this really comes

[01:30:01] down to a question of like how quickly can we solve this kind of like online learning and like >> whether we can like get like 80 90% of the way there with like compaction and like writing files to yourself and stuff and like that's my big uncertainty. >> I really don't know. >> And and another like maybe an example of something that I wouldn't be good at is like you know if I have to like yell at someone to get something at at work or like really push someone to get something done like the model isn't just going to do that. It's just going to be too nice. >> Yeah. >> Yeah. I'd say there's a wide variation in quality of uh human remote workers. So if you if you try to hire someone like off of Upwork to do a software engineering project, there's going to be a huge variation. It's like often quite hard to get them to do like to do a good job or like pay attention to all the feedback you're getting. And uh like I would guess that in some cases it uh like it'll be worse like the the pre-AI um version of this uh was worse than what you can get now from existing AI. Uh so I think it might end up being a little complicated uh because maybe to some some extent we already have this uh

[01:31:01] like for some like not so high quality of work but then like uh then it's obviously like we're not yeah we're not matching human level in certain like higher quality like um forms of work. So, but I basically agree with Charlie and Baron that maybe yeah, we'll yeah, we'll have some version of this in a year or so that's like okay and it we'll be able to do um maybe we'll have that form factor um and uh it'll be able to do some things really well, some things not so well and >> things will be improving from there. >> Like we ship the goalpost based on the very long tail all the time. Like I think feel like you've used this example before of like doing your taxes or something. Like this year I literally just like told Codeex to like go get everything I needed to do and send it to the accountant and like there was this massive list of stuff it had to use computers to click through and like download some stuff and it didn't it was it was like fine. It was perfect. So like I don't know a lot of this stuff it can already do. >> Yeah. Okay. Um give you 10x total productivity uplift.

[01:32:00] >> Basically if you if it takes you a year to make a breakthrough now you you make a breakthrough every month. >> I think I would just refuse to give you a scaler on this. uh like like we might already be past that in some like types of work uh like let's say you're just trying to prove uh yeah you're trying to do um like certain types of math uh >> I was sorry but for you as AI researchers trying to make um yeah >> like advance you know the state of AI research how much are like AI researchers sped up >> or uplifted >> somewhere between five and 10 years >> oh really that's far away you think it's longer than like for general remote worker Yeah, >> interesting. >> I think you probably I think I'm realizing you probably have very different definitions of fully general remote worker. I could have specified. >> Yeah, this is true because I mean like Yeah, because obviously like an AI researcher can be a remote worker and so like >> Yeah. No, I I'm picturing like you know normal white collar work over the period of a month. >> Yeah, >> I think it starts to diverge a little bit past a month >> like a very competent white collar worker but not necessarily like a super creative researcher.

[01:33:00] >> I would say like two years. >> Two years. Yeah. 10x. Okay. >> How about you burn? I can kind of see that actually because like it really is just like right now it's already like definitely more than 10x of like coding stuff and so it's like if it can do even like one or two loops of like experimental feedback that would actually be massive already. >> So 10x uplift of AI researchers within two years >> if you just plug it into like a very naive model of like AI progress and how much is coming from AI researchers and they're like there's like a 10x increase in their productivity. Yeah. Yeah. You you have like radically accelerated pace of AI progress starting 2 years from now. >> Yeah. I mean I think like this will mean that AI progress doesn't get bottlenecked on like AI researchers ability to run like small experiments. It gets bottlenecked on other things. >> Of course. Of course. But it just like happens 10x faster >> for sure. Yeah. >> Which is a huge deal and that also like helps the next thing which makes gives you 100x speed up happen sooner etc. >> Yeah. I'm happy to stick with longer on that one. >> And what's what's the what's like the crux? uh like my capacity to absorb information and make the like Beijian

[01:34:01] optimal decision on the next >> experiment. Makes sense. >> Yeah. I mean, I'm assuming that like you can delegate some of this to the AI. So like the AI is becoming decent at like deciding, you know, it's run this experiment, it's got this result, it runs like the next experiment and then if it can run like two or three experiments in a row without like crashing, then like that is actually big update like uplift and >> and okay, final question. um an AI which is which dominates top human experts across every single field of work that can be done over a computer. So not only a research but all cognitive work and not just like short horizon work but like literally if it takes like three years or something they will still do better than humans. So this is basically just like ASI. Okay. >> I would say like three or four years. >> The >> I mean that's that doesn't seem wrong. >> I would say like um like AI is obviously being more um getting more attention. So it's like one of the harder things, but it's like uh a lot of energy is being

[01:35:01] put into it and it's also like not one of the hardest things for AI because it's like involves a lot of code and and math which models are really good at. Uh maybe for things that involve like 3D and like spatial stuff and physical stuff, I think uh that that will take a little longer. Um so um especially if it's not like Yeah. Yeah. If it's like mechanical engineering or something and it's not getting like the most attention right now, that might take a little longer. >> But it also has to include fields where there is relatively little data because of the nature of the field and it has to like learn that data on the fly. for example, has to become superhuman at like being an engineer at TSMC or something. >> Oh yeah. So you would have to assume that like the on uh like the onboarding Yeah. You can give the AI the same onboarding material and uh Oh yeah. Then there's some like some something has to be solved about like uh sort of longer horizon learning or Yeah. >> I'd say five to 10. So basically yeah it's it's you think

[01:36:02] automating AI research is like ASI complete or something. >> I yeah I think so. Um yeah I think there's so many things in the world which like even if you have some sort of memory system external to the model >> and even if like context length grows a little bit like there are just fundamentally things like even if you could research the information or write notes yourself like you need more than the context to be able to do. Yeah. >> Yeah. I mean, I kind of agree in like the the 5year range at least for like the stuff that like labs are focusing on, but I think like there's going to be a long tale of stuff which like yeah, I could theoretically go out and learn about, but like no one has bothered to do it and like the the computers being allocated to that, so that might take longer for like literally every single human expert. >> Yeah, it's right by this I also included like the ability to learn as fast as a human, a new domain. >> I mean, I think that's not necessarily necessary actually cuz like the AI will have vastly greater experience than like any human, >> right? Thanks so much for doing this, guys. I feel like this is a great format for getting different experts to disagree and debate and discuss things together was very productive. Cool.