Building AlphaGo from scratch – Eric Jang

[00:00:00] Today I'm here with Eric Jen who was [00:00:02] most recently vice president of AI at 1x [00:00:04] Technologies. Before that, senior [00:00:06] research scientist at what is now Google [00:00:08] DeepMind Robotics. And you've been on [00:00:10] sobatical for the last few months. One [00:00:12] of the things you've been doing is [00:00:14] rebuilding and improving and hacking on [00:00:17] AlphaGo. And so we're today what we're [00:00:19] going to do is you're going to explain [00:00:21] building AlphaGo from scratch and what [00:00:22] it tells us about the future of BI [00:00:24] research and development. But uh before [00:00:26] we get to that, why is AlphaGo [00:00:29] interesting? Why is this why is this the [00:00:30] project you decided to do on sobatical [00:00:32] rather than just hang out at the beach? [00:00:33] >> Sure. Yeah. Um I like making things and [00:00:36] Alph Go and Go AI is one of those things [00:00:38] that really got me into the field. Uh [00:00:40] when I saw the kind of early [00:00:41] breakthroughs um on AlphaGo in 2014, [00:00:44] 2015, 2016 and so forth. It was just [00:00:47] profound to see, you know, how smart AI [00:00:49] systems could become and the the kind of [00:00:52] computational complexity class that they [00:00:54] could tackle with deep learning. Um this [00:00:56] is a problem that has you know long been [00:00:58] understood to be kind of intractable for [00:01:01] search and yet um it was solved um [00:01:04] through through deep learning and so so [00:01:07] that was quite mysterious to me and I've [00:01:08] always wanted to understand that [00:01:10] phenomena a little bit better. My [00:01:11] training is often in deep neural nets [00:01:13] for robotics where it's uh the the [00:01:17] decisions made by the neural networks [00:01:18] are a bit more intuitive. But Alph Go is [00:01:20] a sort of problem where the the [00:01:23] decisions are actually the result of a [00:01:25] very very deep search. And it's always [00:01:27] been very mysterious to me how like a 10 [00:01:28] layer network can sort of amortize the [00:01:31] simulation of something so so uh so deep [00:01:33] in the in the game tree. [00:01:35] >> Yeah. Interesting. So if you plot out [00:01:36] how much compute it took to build [00:01:38] various iterations of strong gobots over [00:01:41] the years, you can see that in 2020 [00:01:42] there was a open- source project called [00:01:44] Kadigo um by David Woo from Jane Street [00:01:47] who who basically achieved a 40x [00:01:50] reduction in compute needed to train a [00:01:52] really strong goat tablea. Um I'm not [00:01:54] certain if it's stronger than alpha go [00:01:56] zero or alpha zero or mu0ero u but it's [00:01:59] very very strong and this is what most [00:02:00] go practitioners today train against [00:02:02] when they're when they're playing an AI [00:02:04] and thanks to LM coding what took a [00:02:06] whole team of research scientists at de [00:02:08] mind and you know millions of dollars of [00:02:09] research and compute can now be done for [00:02:11] you know a few thousand dollars of of [00:02:13] rented compute. [00:02:14] >> Okay I guess we should first discuss how [00:02:15] go works. [00:02:16] >> Great. [00:02:16] >> So yeah how does the game work? Um, so [00:02:19] the game of code is a very simple one [00:02:21] that can be implemented uh quickly and [00:02:23] easily in a computer. The the objective [00:02:26] of the game is basically to put down [00:02:27] black and white stones and try to occupy [00:02:29] as much territory in the game as [00:02:31] possible. So I might start by putting [00:02:33] down a black stone. Uh, black always [00:02:35] goes first. Let's go ahead. And so the [00:02:37] way you capture an opponent's stones is [00:02:38] that for every um intersection if you [00:02:40] can surround all four of its neighbors [00:02:43] with um with your stones then um then [00:02:47] this one is sort of cut off from oxygen [00:02:49] if you will and then it uh and it is a [00:02:52] dead dead stone. So so then now I [00:02:54] control these four stones as well as [00:02:56] this empty intersection here. [00:02:57] >> So there's like slight variations [00:02:59] between Chinese, Japanese and uh what is [00:03:01] called Trump Taylor rules. Um, trump [00:03:03] Taylor rules are designed to be [00:03:04] completely unambiguous for go. So this [00:03:06] is what all go AIs train against and [00:03:08] resolve against. So in typical Go like [00:03:11] the humans play, you're actually not [00:03:13] allowed to put this whitest stone down [00:03:14] here. It would be instant suicide. Um, [00:03:16] in Trump Taylor, it's actually fine. You [00:03:18] put it down and then it immediately [00:03:19] resolves to death. So the outcome is [00:03:21] sort of the same. [00:03:22] >> Let's go ahead and start over and and [00:03:24] play play a few stones and then I'll [00:03:25] explain some more. So [00:03:26] >> I'll just start there. [00:03:29] All [00:03:32] right. I'm like basically playing [00:03:33] randomly here, but I'm trying to get [00:03:35] around your stones and see if I can [00:03:37] close them. [00:03:38] >> Yeah. [00:03:45] >> So, this move um basically exposes one [00:03:48] empty neighbor for your white stone. And [00:03:50] it's very akin to a check in chess where [00:03:53] if you don't respond immediately by [00:03:55] putting one here, then I can immediately [00:03:56] capture this. [00:03:57] >> I see. Okay. cuz it is it is sort of the [00:03:58] diagonals that determine whether you're [00:04:00] guarded in [00:04:01] >> the cross-section, not the diagonals. [00:04:03] >> So So this one is surrounded on three [00:04:05] sides. [00:04:05] >> Yeah. [00:04:05] >> And so um you're at threat of losing [00:04:07] that stone if you don't play one [00:04:09] immediately there. [00:04:11] >> Now you can see that I'm starting to [00:04:12] pressure you because by putting a stone [00:04:15] uh here now you are forced to um put one [00:04:18] here. [00:04:19] >> Otherwise you would have this two block. [00:04:21] >> Yes. [00:04:21] >> To yourself. And then if you think think [00:04:23] through like what happens if you were to [00:04:25] respond here, you can probably, you [00:04:27] know, search into the future and deduce [00:04:29] what I'll do in response. Uh once you [00:04:31] once you do that, [00:04:31] >> you have a lot of confidence in my [00:04:32] abilities. But I'm guessing you'd put [00:04:34] the black here. [00:04:35] >> That's right. And then I would capture [00:04:36] all three of these stones. [00:04:37] >> So I should just assume that this is [00:04:38] gone. This little block is gone. [00:04:40] >> Yes. So in go it's actually okay to let [00:04:43] opponent capture um some stones if for [00:04:45] example it allows you to position u to [00:04:48] capture more stones in somewhere else on [00:04:49] the board. And and this is what makes Go [00:04:51] a very beautiful game is that um you can [00:04:53] kind of uh lose the battle but win the [00:04:55] war, right? And and as the board size [00:04:57] increases, the complexity of these kind [00:04:59] of like micro versus macro dynamics uh [00:05:01] gets gets more interesting. [00:05:03] >> But presumably you'd put one here. [00:05:04] >> Mhm. [00:05:06] >> And so now I would capture this entire [00:05:07] group. Okay. [00:05:08] >> And this this would be my [00:05:09] >> Okay. There's one more um uh case that I [00:05:12] want to demonstrate which uh actually I [00:05:13] had a bug in my code uh recently which [00:05:16] is the following situation. So let's [00:05:18] consider a formation like this, right? [00:05:21] And then you know we have other pieces [00:05:22] on the board in play or whatever. Um [00:05:26] and so [00:05:28] um let's talk a little bit about how the [00:05:29] game ends, right? Um in this territory, [00:05:33] who controls these these areas? Is it [00:05:35] white or is it black? [00:05:37] >> White. [00:05:38] >> It's actually black because I have [00:05:39] actually surrounded this whole area. [00:05:41] Yeah. [00:05:41] >> And it's um very assuming I have like [00:05:44] other black stones here. It's actually [00:05:46] very hard for you to break this out of [00:05:48] the control of these stones. [00:05:50] >> So when the final score is tallied, [00:05:51] would would these ones also count as [00:05:53] being in um [00:05:54] >> Yeah, great question. So, so um this is [00:05:56] where different rule sets have different [00:05:58] ways of scoring. And so we should talk a [00:05:59] little bit about how like uh you resolve [00:06:02] scores between humans and how you [00:06:03] resolve scores between uh computer code [00:06:05] >> u because there's actually some [00:06:06] ambiguity in how humans evaluate this. [00:06:09] So most humans would look at this board [00:06:11] configuration and conclude that like [00:06:12] black has kind of totally surrounded [00:06:14] white and so white has no chance of [00:06:16] life. We could play out more here but [00:06:18] then at the end I would capture [00:06:19] everything. [00:06:20] >> Um however if you have a way of breaking [00:06:22] this formation and connecting white to [00:06:24] something outside of it then it can flip [00:06:26] right and so this is where it's you know [00:06:28] a little bit hard for a computer to [00:06:30] decide these kind of things. Right? So [00:06:32] how do humans do it? Right? Like it's [00:06:33] it's worth thinking a little bit about [00:06:34] how humans resolve this because this [00:06:35] will actually map later to how we think [00:06:37] about the deep neural network. Um humans [00:06:40] basically say uh I think the game is [00:06:43] done and then you you have to also say I [00:06:45] think the game is done and then we'll [00:06:46] say like I think these are these are [00:06:48] milestones and then you have to agree. [00:06:50] If you don't agree then we keep playing. [00:06:51] >> Yeah. [00:06:52] >> So um essentially once two humans their [00:06:55] uh so-called value function um agree on [00:06:57] a consensus then the then the Chinese [00:06:59] rules uh result that. [00:07:00] >> Yeah. Interesting. [00:07:01] >> So in Trump Taylor scoring um it's [00:07:04] perfectly unambiguous. So it can be [00:07:06] decided you know algorithmically by a [00:07:08] computer. So if if let's say you you [00:07:10] have this at the end end game the way [00:07:12] you score this is that you first count [00:07:14] how many stones you control and that's [00:07:16] unamiguous. [00:07:17] >> Then you count how many empty [00:07:19] intersections that are not touched by [00:07:21] your opponent's stones. [00:07:23] >> So these intersections would not count [00:07:25] for either player because both of all of [00:07:27] these intersections are connected to [00:07:28] both white stones and black stones. [00:07:30] Right? If um this were like this, then [00:07:34] white would get three points. Now, this [00:07:36] is uh a little odd because a human would [00:07:39] know that white is actually losing these [00:07:41] points. But um Trump Taylor's scoring [00:07:43] would consider white to have all of [00:07:45] these points as well as these points. [00:07:46] >> Got it. Okay. [00:07:47] >> Right. [00:07:47] >> So, so that is a very big difference in [00:07:50] um how computer go scores things and how [00:07:52] humans score things. [00:07:53] >> How does the game end? The game ends [00:07:56] when either a player chooses to resign [00:07:58] or both players pass consecutively. [00:08:00] >> Cool. [00:08:01] >> Yep. So, that's the rules. [00:08:02] >> Nice. All right. Now, help me correct [00:08:04] this with AI. [00:08:05] >> Great. Okay. [laughter] [00:08:06] >> Let's understand how um Alph Go actually [00:08:10] works and how somebody in the audience [00:08:12] might be able to implement it. [00:08:13] >> Great. Yeah. Let's start with um kind of [00:08:15] an intuition about the underlying um you [00:08:18] know search process used to make moves [00:08:21] and we'll layer on uh ideas from deep [00:08:23] learning to make it much more efficient [00:08:25] and tractable. So go is a game where [00:08:28] there's just two players. We're going to [00:08:30] draw a person here and we're going to [00:08:32] draw an AI here. [00:08:34] And um let's say this person is playing [00:08:37] black so they go first. So we're going [00:08:40] to draw [00:08:44] go here. And then now the AI um is going [00:08:47] to make a move based on what it sees [00:08:49] here. So there's a question of like how [00:08:52] you encode these inputs into the AI. [00:08:54] Maybe you could use ones and zeros, but [00:08:56] you want to represent um you know, [00:08:58] black, white, and empty. So So you would [00:09:00] need at least three different values [00:09:02] here, right? So maybe you could use [00:09:03] zero, ones, and twos or something. So, [00:09:05] so the AI might see something like, you [00:09:07] know, 0 0 0 [00:09:12] uh one. [00:09:22] Great. So, so this is the input to the [00:09:24] AI uh on its turn. Yeah. [00:09:26] >> So, so the AI can choose. Let's just [00:09:27] pick three possible random moves that [00:09:29] can go. And I just drew these at random. [00:09:31] And so, which which move is best here, [00:09:33] right? Well, we don't know until the [00:09:34] game ends. Um, there's no Go does not [00:09:36] have any kind of local reward of which [00:09:38] move here is good. And this is what [00:09:40] makes Go a very difficult game is that [00:09:41] you don't actually know who won until [00:09:43] you really get to the end of the game. [00:09:44] So, how deep is this tree, right? Well, [00:09:46] in a 19 by19 um Go board, there are uh [00:09:51] you know roughly to the order of 361 [00:09:53] moves on any given uh move. And of [00:09:56] course, as it fills up, you have less [00:09:57] moves. Um and and the the number of [00:10:01] steps in the game can be somewhere from [00:10:03] 250 to 300 moves. And maybe experts [00:10:06] might uh decide to end the game um well [00:10:08] before that. But uh you know under Trump [00:10:10] Taylor scoring you actually have to play [00:10:11] things all the way to the end. So this [00:10:12] could be like 300 moves or something, [00:10:14] right? So like 300 [00:10:17] um like depth of the tree. [00:10:18] >> Yeah. So if you keep on expanding [00:10:21] possible moves here. So in in this move [00:10:23] the AI is going and then you know here [00:10:25] the human would go [00:10:27] and then you know there's there's some [00:10:35] and so forth [00:10:37] you can find that like essentially what [00:10:39] you end up with is an enormous explosion [00:10:41] in the possible game outcomes [00:10:44] originating from just this one state. So [00:10:48] this is something to the order of like [00:10:49] you know 361 to 300 power of 300 which [00:10:53] is far more than the number of atoms in [00:10:55] the universe right like it's it's just [00:10:57] uh it's just and of course actually [00:10:59] there are redundancies and symmetries so [00:11:01] it's not actually 300 but but that's [00:11:03] sort of the if you were to do a naive [00:11:05] tree where there were no merging of [00:11:06] children then actually you end up with a [00:11:08] tree about this big. [00:11:09] >> What do you mean by merging of children? [00:11:10] >> Right. Let me uh use this board here. So [00:11:13] if we start here and then you play here [00:11:16] and then um I play here and then you [00:11:19] play here that is equivalent to I start [00:11:22] here you play here I play here [00:11:25] >> and then you play here right so so both [00:11:28] of them arrived at the same spot but [00:11:29] through different paths so this child [00:11:31] node can be thought about as a shared [00:11:33] ancestor [00:11:34] >> and I guess it's not 36 it starts at 361 [00:11:36] but it decreases by one each time [00:11:38] >> and the branching factor decreases by [00:11:39] one each time yes but in any case this [00:11:42] is a very very very large tree. And uh [00:11:44] this is also why you know computer [00:11:46] scientists for many years thought that [00:11:48] go was not a tractable problem this [00:11:50] century because the amount of compute [00:11:51] you would need to exhaustively search [00:11:54] every possible possibility um is just [00:11:56] too large. [00:11:57] >> Um if you could go is actually [00:11:58] deterministic game. So um on any given [00:12:01] state you can actually compute what the [00:12:03] uh best possible strategy you can uh you [00:12:06] can make is in order to win the game. [00:12:08] can search all the possible futures [00:12:09] where you win and then just make sure [00:12:10] you always stay in in in that you know [00:12:12] set of futures. [00:12:13] >> Mhm. [00:12:14] >> Um so Alph Go's kind of core conceptual [00:12:18] breakthrough was using neural nets to [00:12:20] make this search problem tractable. So [00:12:23] before we get into you know how neural [00:12:25] networks are involved let's talk a [00:12:26] little bit about how we can you know [00:12:29] assuming we have a powerful enough [00:12:30] computer um search this uh this tree to [00:12:33] find the best move. Right? So in the [00:12:35] beginning um you're not going to build [00:12:37] out the whole tree uh because storing [00:12:39] that tree would be very expensive. [00:12:41] Instead you might do something like [00:12:42] interactively figure out which um which [00:12:45] leaves of this tree are worthy of [00:12:47] exploring and expanding into the future [00:12:48] to see you know what else is there. So [00:12:51] um there are some early algorithms in uh [00:12:54] bandit literature like you know UCB uh [00:12:57] one which is not exactly appropriate for [00:13:00] a you know sequential game like go but [00:13:02] very much inspired the action selection [00:13:05] um algorithm used in uh in Alph Go. So, [00:13:08] so UCB1 looks like on on every move [00:13:11] we're going to take the best action or [00:13:14] you know the arg max over a that [00:13:16] maximizes um um you know the [00:13:22] Q of A and I'll explain what Q of A is [00:13:24] in a moment plus some sort of [00:13:27] exploration bonus. [00:13:34] So on every node we're going to track a [00:13:36] few quantities. So, so let's you know [00:13:38] consider each of these a node. This is [00:13:40] this is the the root node um where [00:13:43] you're making decisions from these are [00:13:46] the children of the root node and um [00:13:49] we're going to say each node is [00:13:51] basically a data structure [00:13:53] that is um it stores a visit count [00:13:59] of this um this node this this child [00:14:02] node [00:14:03] >> is how often the parent visited this [00:14:05] node. [00:14:05] >> Yes. And we'll call this an action. So, [00:14:07] so one thing that is easy to trip on is [00:14:09] like if you come from uh you know [00:14:10] robotics or um other kinds of [00:14:13] reinforcement learning is like where are [00:14:14] the actions right I'm only talking about [00:14:16] nodes um nodes here represent states and [00:14:18] because this is a perfectly [00:14:20] deterministic game with no randomness [00:14:23] you can actually just infer the action [00:14:24] based on the child [00:14:25] >> so so if I go here that implies an [00:14:27] action and this is the state that we [00:14:29] resolve in right so so the LLMs if you [00:14:32] ask to uh you know vibe code a uh MCTS [00:14:35] implementation it'll will most likely [00:14:37] design the right data structure here. [00:14:38] But um you know it's up it's sort of a [00:14:41] chef's choice. You can actually rewrite [00:14:42] the the tree structure however you like. [00:14:44] This was what um Claude 4.6 wrote for me [00:14:46] when I when I asked it and it was a very [00:14:48] reasonable choice. [00:14:49] >> So um [00:14:51] >> so then you know Q represents the um [00:14:54] mean [00:14:56] action [00:14:58] value of this action. [00:15:01] And I'll use a subscript a to denote [00:15:03] that this kind of corresponds to taking [00:15:05] a specific action to to get here right [00:15:07] from from the from the root node. Um so [00:15:09] so like uh if if we have root basically [00:15:13] taking a gets us to this this note here. [00:15:16] >> Um and then we're going to also store [00:15:18] the probability of taking this action [00:15:21] [snorts] [00:15:22] >> again from the parent. [00:15:24] >> From the parent. Yes. Like like what are [00:15:25] the odds that we sample this one? Yeah. [00:15:27] >> And and this will become relevant later. [00:15:29] you know like uh we we've talked about a [00:15:30] deterministic tree for now. So I I'll [00:15:32] bring probabilities into this later. And [00:15:34] then finally we have a sort of uh [00:15:35] dictionary of children [00:15:39] which is just like you know more of [00:15:40] these notes in a in a sort of classic [00:15:42] link list style reference tree. So um [00:15:45] this is the basic data structure to [00:15:47] implement a tree and um in Alph Go um [00:15:51] they use a slightly different action [00:15:53] selection criteria called um pucked and [00:15:56] it's short for predicted upper [00:15:58] confidence with trees and uh this is [00:16:02] basically um when you when you select [00:16:05] which which child to take you do arg Max [00:16:08] A of [00:16:10] Q of S A [00:16:15] plus [00:16:16] uh constant. [00:16:30] So the equation forms are actually [00:16:32] pretty similar. Um these are both [00:16:34] scoring criteria, right? like you want [00:16:36] to argax this quantity and you want to [00:16:37] arg this quantity to determine which [00:16:39] action to take. So let's break down the [00:16:41] intuition of like how you select actions [00:16:43] here. This is the mean action value. So [00:16:46] how good is a given child on average? Um [00:16:49] and and and if you actually you know [00:16:51] knew the whole tree then uh this is all [00:16:53] you need, right, to select the best [00:16:54] action. You don't really need to do more [00:16:55] than that. But if you're interactively [00:16:57] building this tree as you're um figuring [00:16:59] out what the Q values should be, then [00:17:01] what you have have to do is occasionally [00:17:03] try some other actions, you know, as a [00:17:05] sort of explore versus exploit [00:17:06] trade-off. So um in both UCB and uh [00:17:09] puck, there is this term here that [00:17:11] basically rewards um taking actions that [00:17:15] you haven't taken before. So as we [00:17:17] mentioned before, each node stores the [00:17:19] visit count of taking that specific [00:17:21] action, right? So everything is [00:17:22] initialized to zero. And so for a given [00:17:25] action, let's just say like call it n [00:17:26] like action a um initially it's zero and [00:17:30] and so as n is increasing if if let's [00:17:33] say we've already made 10 um 10 uh [00:17:36] action selections from that root node u [00:17:38] but we haven't picked a yet then this [00:17:40] term actually starts to become quite [00:17:42] large for a. [00:17:42] >> Yeah. [00:17:43] >> Right. And conversely if we have chosen [00:17:45] a 10 times out of 10 then now this term [00:17:48] is quite small. It it diminishes very [00:17:50] quickly. And the same thing is actually [00:17:52] true here. [00:17:53] Just make sure I'm understanding it. [00:17:54] Maybe I can uh [00:17:57] put it in my own words. [00:17:59] >> Let's just focus on UCB. What we're [00:18:01] saying here, [00:18:03] you can think of it conceptually as two [00:18:04] different things. The Q and then this [00:18:07] exploration term. [00:18:08] >> Mhm. [00:18:09] >> Let's just be clear about what Q is. Q [00:18:10] is basically saying, hey, once we do [00:18:12] these rollouts. So, you're actually [00:18:13] running all these simulations. You go [00:18:15] down the tree and then you figure out, [00:18:17] okay, if I end up at the terminal value [00:18:19] of this tree, do I win this game or not? [00:18:22] And then you do this, you average [00:18:24] whether I win this game or not across [00:18:25] all the, you know, the leaves of this [00:18:28] tree from starting from this node. That [00:18:31] average you put in Q. [00:18:33] >> Correct. [00:18:33] >> And so you're saying the Q is basically [00:18:34] representing will I win this game or [00:18:36] not? Uh what is the probability that [00:18:38] I'll win this game starting at this [00:18:38] node? That's your sort of um that is [00:18:41] your sort of exploit. That is like [00:18:43] saying I've run these simulations. I [00:18:45] think this is a good move or not. And [00:18:46] then this other term is saying um have I [00:18:49] explored this branch enough yet relative [00:18:52] to the other actions I could be [00:18:53] exploring or I have already explored uh [00:18:56] if I haven't explored this branch yet [00:18:58] you know maybe I think it has a low [00:18:59] score but I just haven't explored that [00:19:00] many branch uh leaves of this uh down [00:19:02] this uh leaves down this uh down this [00:19:05] node in this tree. So I should maybe [00:19:07] like try this even though the Q the sort [00:19:08] of exploit is telling me that this is [00:19:10] not that valuable and because ln of n [00:19:13] grows slower than n. Uh basically as [00:19:17] over time you will move from the arg max [00:19:22] being dominated by this exploration term [00:19:23] which is the second term here to the [00:19:26] argmax being dominated by the q term [00:19:27] which is like okay I've done enough [00:19:29] simulations I'm quite confident that [00:19:31] like this is the branch to go down. [00:19:33] >> Yes that's right. So um the motivation [00:19:36] for UCB was to come up with an algorithm [00:19:39] where if you don't know the payoff of [00:19:41] the arms uh the the different actions [00:19:43] you can select to begin with this uh [00:19:45] strategy basically with given some [00:19:47] exploration term here bounds your regret [00:19:50] >> uh in terms of how wrong you can [00:19:51] possibly be. [00:19:52] >> Um I don't know the proof. I don't also [00:19:54] know if this one is proved to have a [00:19:56] logarithmically or or like uh you know [00:19:58] square root bounded regret or anything [00:19:59] but I think the algorithm was just [00:20:01] derived to look something like this. And [00:20:02] you can tell that these terms are they [00:20:04] grow a little bit differently. And this [00:20:05] is actually just to account for the fact [00:20:07] that go has many more actions in every [00:20:09] given move compared to your standard [00:20:10] bandit problem. [00:20:11] >> Um so uh one small clarification to make [00:20:14] is that you talked a little bit about [00:20:15] simulations on improbability and so [00:20:17] forth. Um we should remember that go [00:20:19] fundamentally is a deterministic game. [00:20:21] So the notion of pro like where does the [00:20:22] notion of probability come from here, [00:20:25] right? Um [00:20:26] >> uh if you had a very powerful computer, [00:20:30] there is no probabilities. you just you [00:20:32] can just compute the true average of [00:20:34] what the the mean action value is. So [00:20:36] where does the probability come in? [00:20:38] Well, it turns out that um uh as in you [00:20:41] know computer go before uh alpho we've [00:20:44] always done some sort of Monte Carlo [00:20:46] method where we have some we we take the [00:20:49] um expected Q value averaged over a [00:20:53] randomly selected tree um and that [00:20:55] randomly selected tree is where [00:20:57] probabilities come in. So the [00:20:58] interpretation of Q is um what is the [00:21:02] expected action value under the um under [00:21:06] the random distribution induced by some [00:21:08] random search process. [00:21:09] >> Makes sense. [00:21:10] >> And so where does the random search [00:21:11] process come in? That's where uh you [00:21:13] know p of action comes in here. So if we [00:21:16] assume a very naive algorithm where you [00:21:18] have a uniform probability of taking any [00:21:19] valid action then this would just be one [00:21:22] over you know the number of valid moves [00:21:24] in uh in in this uh setup and you would [00:21:26] be kind of taking this average over this [00:21:28] very diffuse tree right and and this is [00:21:31] uh this is a valid um integral you can [00:21:33] take but it's very slow because you're [00:21:35] going to consider a lot of trees that [00:21:36] have very low value and uh it's [00:21:39] essentially almost like a important [00:21:40] sampling problem where you want to [00:21:42] there's only a few actions and and sort [00:21:45] of uh paths that can contribute you know [00:21:47] high value and almost everything else is [00:21:49] low value. So so this is sort of a [00:21:50] tricky um problem here. Okay. Um so this [00:21:55] is the action selection criteria for how [00:21:57] you decide which moves to move down. Now [00:22:00] as you move down um in in tree search, [00:22:02] you will eventually run into a node [00:22:04] where um it's quite clear you've won or [00:22:07] lost right at the at the very very end [00:22:09] of the game when when there are no valid [00:22:11] moves to play left under under Trump [00:22:13] Taylor scoring. you can decide whether [00:22:15] you like you know won or lost, right? So [00:22:18] you either win or you lost. [00:22:21] And so this is basically um you know the [00:22:25] the final return of the whole game, [00:22:27] right? [00:22:28] >> Um and so the the question here is like [00:22:31] we we can assign a um a value u to a a [00:22:36] terminal leaf node of the tree, but how [00:22:39] do we assign the values for nodes prior [00:22:42] to that? the parents. And it turns out [00:22:44] um you know what you simply do is you [00:22:46] just take the um your mean action value [00:22:50] is essentially your average. So let's [00:22:52] suppose these were leaf nodes. Um sorry [00:22:54] these were all leaf nodes. The the mean [00:22:57] action value of this node. You know this [00:23:00] action here is just the average of [00:23:03] whether you won or lost at the leaf [00:23:05] nodes. And uh correspondingly you can [00:23:08] kind of walk up the chain and say like [00:23:09] well the mean action value of this node [00:23:11] let's call this like QB and this is [00:23:14] action B is just the average of a [00:23:17] weighted average of these swamps here. [00:23:18] Yeah. [00:23:18] >> Right. And and the weighted average is [00:23:21] um it could be dependent on if you have [00:23:23] a different sampling distribution or [00:23:24] not. But the that the basic intuition is [00:23:26] that you want to resolve the game where [00:23:28] you have a deterministic win or lose and [00:23:30] then you can kind of go backwards. uh [00:23:32] this is called the backup step and [00:23:34] assign values to these uh these these [00:23:36] these nodes or actions um uh [00:23:40] corresponding to the averaged over over [00:23:42] the final terminal leaf. Yeah. Okay. So [00:23:45] um if you were to do this without neural [00:23:47] networks, it would still be intractable. [00:23:49] Um you would you would have a trouble [00:23:50] finding, you know, which um actions to [00:23:53] sample. A lot of the actions would [00:23:55] contribute very low value, especially if [00:23:57] you're like, you know, trying to fight [00:23:58] your way out of a losing position and [00:24:00] only a few actions give you high value. [00:24:02] So the search in practice is still very [00:24:03] very expensive. Um but but the the idea [00:24:07] is that like if you can because go [00:24:09] follows a tree structure you can [00:24:11] actually you know inform a very good [00:24:14] estimate of the value of this node based [00:24:16] on the uh values of uh you know [00:24:18] downstream assuming they're all correct [00:24:20] and assuming you've searched deep [00:24:21] enough. Mhm. [00:24:22] >> Your explanation earlier about the um [00:24:24] the sorts of states where it's obvious [00:24:26] to a human who's going to win, but it's [00:24:28] not obvious to [00:24:29] >> or like you deterministically you still [00:24:31] had to play it out [00:24:32] >> actually drove home the intuition of why [00:24:34] the value function both is trainable and [00:24:38] two why it's necessary in order to [00:24:40] actually h be able to learn this game [00:24:42] effectively. And maybe it's worth [00:24:44] defining value in the first place but [00:24:46] >> sounds good. Yeah. [00:24:47] >> Yeah. So we we talked about uh you know [00:24:49] this U value being you know your final [00:24:51] resolution of whether you won or lost [00:24:52] and this is the terminal leaf node [00:24:54] condition. Um now humans don't play all [00:24:57] the way to the sort of edges of the the [00:24:59] tree the leaves of the tree right they [00:25:00] kind of stop you know some dozens of [00:25:03] moves before maybe maybe even hundred [00:25:05] moves before in in sort of high level [00:25:06] play. So how do they know right? Like [00:25:09] you can think about humans as implicitly [00:25:10] having a neural network called a value [00:25:13] function that basically um you know [00:25:16] takes in uh a board state and then it [00:25:19] kind of evaluates um you know [00:25:23] he went win [00:25:25] and so the human glances at the board [00:25:27] and they know like I'm probably going to [00:25:28] lose, right? and and they're essentially [00:25:30] running a neural network that looks at a [00:25:31] board and implicitly they are [00:25:34] amvertising a huge number of possible [00:25:36] game playouts and and taking that [00:25:39] average and then deciding whether the [00:25:40] board is winnable or not and then [00:25:41] whether they should concede or or you [00:25:43] know keep playing or not. And uh this is [00:25:45] remarkable if you think about like the [00:25:48] um the beauty of something like this. [00:25:49] It's like a a neural network in a in a [00:25:53] human can somehow do all of this [00:25:56] simulation at a glance and then just [00:25:57] know like within a few seconds without [00:26:00] actually playing every single game [00:26:02] logically based on just kind of like [00:26:04] crystallized knowledge and experience [00:26:05] that like they can do this. And so this [00:26:07] gives us a hint that like in games like [00:26:09] Go um there are ways to basically [00:26:12] radically speed up the search process. [00:26:14] And this is one of the fundamental [00:26:15] intuitions behind why AlphaGo works is [00:26:18] that you can train a value function to [00:26:21] look at a board and quickly resolve the [00:26:23] game without playing out all of these [00:26:25] trees into the you know into a very deep [00:26:27] search depth. [00:26:28] >> Yep. Makes sense. I will say for the [00:26:31] audience um I sort of found uh [00:26:36] for previous episodes when I was [00:26:37] prepping and it would seem somewhat [00:26:38] relevant to understand how alpha go [00:26:39] works I would find it very very [00:26:41] confusing and but it's the kind of thing [00:26:43] where once you understand the problem in [00:26:45] this way and then you'll build the next [00:26:47] few pieces it is actually much more [00:26:49] understandable and it will make a lot of [00:26:50] sense and it's okay to be confused right [00:26:53] now uh but it's it's probably simpler to [00:26:55] understand by the end of this lecture [00:26:57] than you anticipate so I'll just make [00:26:59] that note for the audience Uh yeah, the [00:27:01] important intuition at a high level just [00:27:03] to you know step back about where we're [00:27:04] going with all this is that um [00:27:07] >> classically for games like Go you could [00:27:09] build a tree but we don't have computers [00:27:11] powerful enough for that. [00:27:12] >> Yeah. [00:27:12] >> And um estimating the value of every [00:27:17] action that you could possibly take is [00:27:19] also hard because you don't know until [00:27:20] the end of the game. [00:27:22] >> You could take averages uh by playing [00:27:24] them to the end, but that's also hard [00:27:26] because you don't know which actions to [00:27:27] take to sample these averages. So [00:27:30] conceptually there's kind of two [00:27:31] problems there. There's the breadth of [00:27:33] the tree and then there's the depth of [00:27:34] the tree. And AlphaGo gives us a way to [00:27:37] basically u shrink both of those to be [00:27:39] very tractable. [00:27:40] >> Yeah, [00:27:40] >> that's that's essentially the kind of [00:27:42] core idea behind it. [00:27:43] >> Okay. So we uh we take this idea that [00:27:46] like you know humans can glance at a [00:27:47] board and instantly predict whether we [00:27:49] win and maybe that gives us the [00:27:50] opportunity to really truncate the how [00:27:52] deep we we search. And then you know we [00:27:54] also know that humans can look at a [00:27:56] board and um [00:27:59] and uh decide you know [00:28:03] um what what boards you know [00:28:06] like intuitively at a glance what moves [00:28:08] might be good on a go board right so so [00:28:10] these are kind of two things that we can [00:28:12] use deep neural networks for to [00:28:14] accelerate this search process um let's [00:28:16] go back before we talked about neural [00:28:18] nets let's just go back to how this [00:28:19] playout works and we've only talked [00:28:20] about making one move right so so the AI [00:28:23] looks at this encoded goboard. It has a [00:28:25] tree. Um, it searches for, you know, [00:28:28] deeply into the tree to find out which [00:28:30] of its actions might be the best and [00:28:32] then it takes that action. And then now, [00:28:33] you know, it goes back to the human. So [00:28:35] maybe now the human sees a go board that [00:28:38] looks like, you know, like this. And um [00:28:41] and then they um they make their move. [00:28:45] So maybe they put um they put their [00:28:47] stone here. [00:28:49] And then now we [00:28:52] um we go back to the AI [00:28:55] which now looks at a new encoded board. [00:29:07] So I've used two to denote the AI's [00:29:09] playing as white and one to denote the [00:29:11] human playing as black and zero as [00:29:12] empty. And then now on the AI's turn, it [00:29:15] does the MCTS tree search all over again [00:29:18] from scratch. Right? So, so it throws [00:29:20] away this old tree that it searched last [00:29:22] round and now there's a new root node [00:29:24] and it begins to search a new [00:29:27] and then so forth. So, MCTS is basically [00:29:30] a you can think about it like a search [00:29:32] algorithm that is um deciding what moves [00:29:35] to play best aided by neural networks um [00:29:38] and and it's it's done on every every [00:29:40] move. [00:29:41] >> Okay, great. So, let's talk about the [00:29:44] neural network part of this. And while [00:29:46] you're racing, another sort of thing [00:29:48] that was important for me to understand [00:29:49] was this MCTS data structure with nodes [00:29:53] and childrens of nodes and whatever. Um, [00:29:56] this is done per move and reinstantiated [00:30:00] once a move is made. So a human makes a [00:30:03] move then the AI looks at this and is [00:30:05] trying to basically run a bunch of [00:30:07] simulations uh to figure out okay what [00:30:09] should move should I make next and those [00:30:11] simulations just a simulation is [00:30:13] basically like exploring one more node [00:30:15] in this MCTS3 and at the end um once all [00:30:20] these once all this you know you run a [00:30:22] thousand simulations that informs then [00:30:24] this um I guess as you'll explain this [00:30:27] probability of what move to make next [00:30:29] that's what you store you You sort of [00:30:32] choose the best move given those [00:30:33] probabilities. You discard all of that. [00:30:35] Then the next player makes a move and [00:30:37] you restart this process at the [00:30:39] beginning of every move. [00:30:40] >> Correct. One small addendum. You don't [00:30:43] discard all of that. You keep one thing [00:30:45] behind that we'll use later. [00:30:46] >> Yeah. [00:30:47] >> Just like I did for Reiner, I wanted to [00:30:49] make flash cards for this episode so [00:30:50] that people could retain these concepts. [00:30:52] And ideally, an LLM could generate some [00:30:54] candidates for me to then refine. But to [00:30:56] actually get high quality suggestions, I [00:30:58] needed to design a whole pipeline where [00:31:01] the AI could take and ingest screenshots [00:31:03] of [music] the blackboard at the right [00:31:04] time stamps and then make SVG diagrams [00:31:07] in case visuals were helpful and then [00:31:08] run their writing and drawing through a [00:31:10] critic and then revise the card in [00:31:12] response to this feedback. It's very [00:31:13] hard to accomplish this just by stacking [00:31:15] LLM calls. This sort of step-by-step [00:31:17] recipe works much better if you have a [00:31:19] durable agent that's been engaging with [00:31:21] the task across all the previous stages. [00:31:23] So, I use the cursor SDK to spin up an [00:31:25] agent for each card. The cursor hardness [00:31:27] saved me a bunch of work in designing [00:31:29] some custom context scaffold or figuring [00:31:31] out how to design tool calls for taking [00:31:34] screenshots or making animations. These [00:31:36] agents all run in the cloud, so I don't [00:31:38] have to worry about leaving my laptop [00:31:40] open. I just get an email when I have [00:31:41] candidates to review. You can check out [00:31:43] my cards at flashcards.warcash.com. [00:31:47] You can start building with the agents [00:31:49] SDK at cursor.com/sworth. [00:31:53] Okay, so now we have a basic intuition [00:31:55] of how moves are made with search. We're [00:31:57] going to talk about how neural networks [00:31:58] can speed this up uh by providing an [00:32:01] analog to like the human intuition. [00:32:03] >> So there's two networks. There is the [00:32:05] value network [00:32:07] which takes in a state and it predicts [00:32:12] you know am I going to win or lose? It's [00:32:13] a binary classification problem. [00:32:17] Then we're going to have a policy [00:32:19] network which uh induces a distribution [00:32:22] over good actions to take. [00:32:25] >> So um I'm going to draw a [00:32:27] one-dimensional flattened move [00:32:28] distribution, but this is really like [00:32:30] you know a a square kind of grid, right? [00:32:33] So um so maybe like it thinks actions [00:32:37] are like these are the kind of [00:32:39] probability distribution over good [00:32:41] actions. And both of these are uh [00:32:43] categorical classification problems, [00:32:45] right? So you can train this like any [00:32:47] classifier in uh with deep learning um [00:32:50] uh you know cross entropy loss that kind [00:32:52] of stuff. [00:32:53] >> So the um the specific architecture does [00:32:56] not actually matter too much. I I tried [00:32:57] a few different architectures. [00:32:58] Transformers work, ResNets work for [00:33:01] small data regimes. Uh my experience is [00:33:03] that ResNets still kind of outperform [00:33:05] transformers and um and it kind of gave [00:33:07] you more bang for the buck at at lower [00:33:08] budgets. But this may not be true. [00:33:10] >> Why is that? um they they provide the [00:33:12] inductive bias of like local [00:33:13] convolutions and generally transformers [00:33:16] start to outperform uh residual [00:33:18] convolutional networks when you want [00:33:20] more global context. [00:33:21] >> I see. [00:33:21] >> So, so one um interesting finding from [00:33:23] the Kadiggo paper was that they found it [00:33:25] actually quite useful to pull together [00:33:27] global features together um and [00:33:30] aggregate global features like uh [00:33:32] throughout the network um to kind of [00:33:35] give the network a global sense of how [00:33:37] to like connect value from one side of [00:33:38] the board to another side of the board. [00:33:40] But what does it mean to aggregate [00:33:41] global features? [00:33:42] >> Yeah. So if you have a um go a very [00:33:46] large 19 by9 goboard [00:33:48] >> and you you know you've got some some [00:33:50] sort of battles going on here and you [00:33:52] got some battles going on here. [00:33:54] >> Um when you pass this through a [00:33:55] convolutional neural network, [00:33:57] >> the receptive fields of the [00:33:58] convolutional network are going to be [00:34:00] good at computing local things and [00:34:03] making that invariant. [00:34:05] But um they won't be able to kind of [00:34:08] connect these two features easily, [00:34:09] right? They need to sort of be pulled [00:34:11] together and attend to each other [00:34:13] somehow. So the argument about you know [00:34:15] why transformers are good for computer [00:34:18] vision tasks like with uh you know [00:34:20] vision transformers and so forth is that [00:34:22] because they have a sort of global [00:34:23] attention across the whole thing, they [00:34:25] can more easily draw these connect [00:34:27] predictions. But you do need more data [00:34:29] there so that you can kind of uh learn [00:34:32] through data the the sort of invariant [00:34:34] local local features. [00:34:36] >> Um I've tried very hard to make [00:34:37] transformers work for this problem [00:34:38] because I was kind of curious if [00:34:39] transformers would present some sort of [00:34:41] breakthrough in go and just remove a lot [00:34:43] of those tricks. But try as I might I [00:34:45] actually haven't figured out a way to [00:34:46] make transformers better than resonance [00:34:48] for for now. [00:34:49] >> So one uh sorry one more tangential [00:34:50] question. [00:34:52] It it makes sense why transports with [00:34:54] their like global pooling of information [00:34:56] would be better if you need to consider [00:34:58] information that is not just spatially [00:35:01] um [00:35:02] uh yeah CNN's give you a sort of bias [00:35:05] that the things that are next to you are [00:35:08] especially irrelevant [00:35:09] >> and then they're sort of aggregated up. [00:35:11] Yes. [00:35:11] >> But suppose okay so for games where it [00:35:14] it isn't that relevant what is happening [00:35:16] locally you just kind of have to [00:35:16] consider the whole thing [00:35:18] >> you're saying transformers would work [00:35:19] better. How about games where so talking [00:35:21] about the spatial dimension? How about [00:35:23] the temporal dimension where right now [00:35:25] we're only considering the previous move [00:35:27] because it is a deterministic full [00:35:29] information game where um uh but what if [00:35:33] it was something like poker or diplomacy [00:35:35] where [00:35:36] really a bluff they made a while back is [00:35:38] sort of relevant to understanding now [00:35:40] and isolating to decide to make your [00:35:42] next movements you need to consider all [00:35:43] those previous states. Would that then [00:35:45] change into the consideration of what [00:35:46] inductive bias is most relevant and what [00:35:48] architecture is most relevant? [00:35:49] >> Right. Great question. So, Go is a [00:35:52] perfect information game. Yeah. And in [00:35:54] perfect information games, um there does [00:35:57] exist a Nash equilibrium strategy for [00:36:00] which you can do no worse than any other [00:36:02] strategy. M so um if you know that your [00:36:05] opponent has a particular bias like they [00:36:08] they love to play aggressively, you can [00:36:10] actually in principle counter that [00:36:11] specific strategy better than a Nash [00:36:13] equilibrium policy. But um to counter [00:36:16] any given strategy um there does exist a [00:36:19] single um Nash equilibrium that can be [00:36:21] decided solely using the current state. [00:36:24] So um that that is a design choice that [00:36:27] most Go agents AlphaG go chose to do [00:36:29] which in hindsight turned out to work [00:36:30] very well because the uh Nash [00:36:32] equilibrium seems to be superhuman like [00:36:35] like no human strategy seems to be able [00:36:37] to beat it. [00:36:38] >> Now there are variations of this where [00:36:39] you would actually need to consider [00:36:41] temporal history. So, and and this is a [00:36:43] very exciting research area that I I [00:36:44] would encourage people to kind of fork [00:36:46] my repo and try these things out, which [00:36:48] is um if you were to play, let's say, [00:36:49] 2v2 go, then you actually need to model [00:36:52] your partner's uh behavior and you like [00:36:54] you may not have information on how they [00:36:56] play. So, you need to aggregate some [00:36:57] information on like how they play so [00:36:59] that you can respond accordingly. Yeah. [00:37:01] Right. Like these are uh situations [00:37:03] where it's no longer a perfect [00:37:05] information game. And then in those [00:37:07] cases in in games of imperfect [00:37:08] information or partial observability [00:37:10] then you do need some context to build a [00:37:11] model. [00:37:12] >> Yeah. [00:37:12] >> Yeah. And and and I think that's a place [00:37:14] where things get very very exciting in [00:37:16] terms of like selfplay or you know [00:37:18] diplomacy style. [00:37:19] >> Yeah. Interesting. [00:37:20] >> Okay. So uh returning back to the neural [00:37:23] network the architecture again is not [00:37:24] super important. You can get it to work [00:37:25] with transformers. You can get it to [00:37:26] work with resets. I found that for [00:37:29] lowbudget experiments uh resets work a [00:37:31] little better. Um, you can also use kind [00:37:33] of a Carpathy style auto research [00:37:35] hyperparameter tuning to make make your [00:37:36] architecture pretty good. And so, so you [00:37:39] don't have to worry too much about that. [00:37:40] You just need to sort of set up the [00:37:41] problem so that you have a sort of [00:37:43] target optimization. [00:37:44] >> Yeah. [00:37:44] >> Okay. So, we're going to pick just a [00:37:47] somewhat arbitrary architecture that [00:37:48] worked for for you know what I did. But [00:37:50] again, this part is not super important. [00:37:52] um you have your encoded board state and [00:37:56] uh we're going to just choose to let's [00:37:57] say do three uh three like you know [00:38:00] similar to an RGB we're going to have [00:38:01] three kind of channels uh one channel to [00:38:04] encode black, one channel to encode [00:38:06] white and then um and then uh one [00:38:09] channel maybe to encode like um uh [00:38:12] empties or um maybe like a masked region [00:38:15] if you want to train on multiple board [00:38:17] sizes. I'm actually not going to talk [00:38:18] about multiple board sizes for now. [00:38:19] That's a little bit too complicated. So [00:38:21] we'll just say like you know we've got [00:38:22] this two or three channel uh RGB like [00:38:25] image and then we go into a you know a [00:38:28] ResNet [00:38:33] and then we have two branching heads. Um [00:38:36] one head predicts the the value function [00:38:39] and this is like a single loget. So this [00:38:43] is like R1 and then we have the policy [00:38:47] which is you know R 361. [00:38:52] So [00:38:54] um this is the architecture and uh we're [00:38:57] going to basically train this to predict [00:39:00] the outcomes of games given the board [00:39:02] state and we're also going to train this [00:39:04] to predict what are good moves. Yeah. [00:39:06] >> Right. So the OG AlphaGo paper uh or [00:39:09] called AlphaGo Lee um initialized this [00:39:12] network with a supervised learning data [00:39:14] set of expert human play. Um later they [00:39:16] remove this restriction by having the [00:39:18] model teach itself how to play well. But [00:39:20] I find it actually from a matter of like [00:39:22] implementation for your audience. Super [00:39:24] super nice to always kind of initialize [00:39:26] your your experiments to something [00:39:28] that's easy and then like you know get [00:39:30] the problem working before you know [00:39:32] trying to bite off the whole thing and [00:39:34] learn a tablet resin. you you generally [00:39:36] want to kind of initialize just as in [00:39:37] deep learning init in initialization is [00:39:39] everything right. Um you always want to [00:39:42] initialize your research project to [00:39:44] something as close to success as [00:39:45] possible. Um especially if you're you [00:39:47] know doing something new that you [00:39:48] haven't done before. Like always pick [00:39:49] something that works and then get it to [00:39:51] do something better rather than start [00:39:52] from something that doesn't work at all [00:39:53] and then you know um try to make it [00:39:55] work. So um under that philosophy it's a [00:39:58] great idea to start from something that [00:40:00] like you know has a good initialization. [00:40:02] So we're going to take human expert [00:40:03] plays um and train this model to predict [00:40:07] um you know good actions right so we're [00:40:08] going to take all of the winning games [00:40:10] all all the moves in which a a human won [00:40:13] and um sorry an expert won and then [00:40:16] predict those actions and then uh [00:40:18] regardless of board state like you know [00:40:19] whether you won or lost you're going to [00:40:21] predict the outcome. Yeah. So, you might [00:40:22] be wondering like, okay, well, some of [00:40:24] the early boards, you know, where [00:40:25] basically only one stone has been put [00:40:27] down, how could you possibly know [00:40:29] whether who who the winner of this game [00:40:31] is, right? Well, if you have uh, you [00:40:33] know, hundreds of thousands of games, [00:40:35] then in on average, you'll probably see [00:40:37] that boards that start like this have a [00:40:39] sort of half of the games that branch [00:40:41] off from this will win and half of the [00:40:42] games that branch off of this will lose. [00:40:44] So, that'll actually be fine. When you [00:40:46] train this model to predict those, the [00:40:47] loget will sort of converge to uh, you [00:40:49] know, 0.5. Um and and so so for these [00:40:52] for these things it's sort of expected [00:40:54] that once you train the model a starting [00:40:56] board state will look like 0.5 and then [00:40:58] as you progress towards the end of the [00:40:59] game it'll actually look something like [00:41:02] you know if this is 0.5 the the win [00:41:05] probability will sort of either go like [00:41:06] this or it'll it'll go like this right [00:41:09] and um and this is sort of your move [00:41:12] number. [00:41:13] >> Yeah. And so as you, you know, get [00:41:16] hundreds of steps into the game, it [00:41:18] becomes much more clear like who's more [00:41:20] likely to win or who's more likely to [00:41:22] lose under your expert data [00:41:23] distribution. [00:41:24] >> I I didn't understand the significance [00:41:26] of why the this way of thinking about [00:41:28] value is especially relevant to the [00:41:29] expert data. [00:41:30] >> It is not relevant to the expert data. [00:41:32] It's true for any data that you train [00:41:34] on. Yeah. So if you were to learn a [00:41:36] tablet Raza, you would also expect this [00:41:38] to fall out. Yeah. So um if you just do [00:41:42] this like so imagine you know you're [00:41:43] vibe coding Alph Go and you um you you [00:41:46] gather some expert data sets from like [00:41:48] how to go online um or you you know you [00:41:51] have a data set of human players and you [00:41:53] train this model actually it turns out [00:41:54] this model is already a pretty good go [00:41:56] player. it'll most likely beat most [00:41:57] human players, right? So, so like if you [00:42:00] just take this policy recommendation and [00:42:02] take the ARG max over, you know, it's uh [00:42:07] if this is the, you know, probabilities, [00:42:09] if you take the ARG max and you just [00:42:11] take this action as your go play, um [00:42:13] it'll be a very very fast go player that [00:42:15] doesn't think in terms of like reasoning [00:42:17] steps. it just kind of shoots from the [00:42:19] hip and it'll be a very strong go player [00:42:21] which is already quite miraculous if you [00:42:23] think about like you know 10 neural [00:42:25] network layers maybe under like 3 [00:42:28] million parameters can already do [00:42:30] something that impressive [00:42:31] >> um [00:42:33] >> um and so you can start this way and [00:42:36] it's important when implementing this to [00:42:37] kind of just verify that this is [00:42:39] probably true. It's good to verify that [00:42:40] your Go rules are implemented correctly, [00:42:42] that like you know you can run these [00:42:44] simulations relatively quickly. Uh and [00:42:46] and just as almost like a a sort of a um [00:42:49] a checkpoint that like you want to make [00:42:51] sure that you can actually do this basic [00:42:52] step before you try to layer on more [00:42:54] complex uh things like search. [00:42:55] >> Yeah. [00:42:56] >> Um so but yeah, we can do a lot better [00:42:59] than taking the raw neural network and [00:43:01] playing the moves. And uh this is how we [00:43:02] can apply it to Monte Carlo research. So [00:43:05] let's uh apply the neural network to um [00:43:08] to improve Monte Carlo research. So we [00:43:11] start with our root node [00:43:18] and we now have a four-step uh iterative [00:43:22] process to do MCTS. So this tripped me [00:43:25] up when I was first reading the paper [00:43:26] and trying to understand it. But um uh [00:43:29] essentially what we're going to do is [00:43:31] we're going to choose a number of [00:43:32] simulations. So like you know numbum [00:43:36] simulations [00:43:38] and this number varies. This can be you [00:43:40] know somewhere between 200 to uh 2048. I [00:43:44] believe in um in the AlphaGo Lee match [00:43:47] they use tens of thousands of [00:43:48] simulations per move because they really [00:43:50] wanted to boost the strength of the [00:43:51] model as much as possible. [00:43:52] >> Yeah. [00:43:53] >> Um but in training you don't actually [00:43:54] need too many. And Katigo I think uses [00:43:57] something on this order as well. [00:43:58] >> Do you know if they used uh if you watch [00:44:00] the documentary they had a laptop out [00:44:01] during the game. They didn't use a [00:44:03] laptop itself. It was like on some [00:44:04] >> It was on some TPU pod, I think. Yeah. [00:44:06] Um but now [00:44:08] >> honestly kind of unfair. [00:44:09] >> Well, uh [00:44:10] >> like Lee is not using like one E22 flops [00:44:13] to do a move, you know. [00:44:14] >> Fair enough. Um interestingly enough, [00:44:17] modern Go bots don't need that much [00:44:19] compute at test time. And what we'll [00:44:22] actually find out uh as we talk about [00:44:23] how the um MCTS policy improvement works [00:44:27] is that over time the raw network [00:44:29] actually takes all of the burden of that [00:44:32] big TPU pod and just push pushes it into [00:44:34] the network and and you can do all of [00:44:36] that work with one you know neural [00:44:38] network pass. [00:44:39] >> Um but but the TPU pod will always add [00:44:42] the extra oomph on top. And so that's [00:44:43] what they wanted for the match. [00:44:45] >> So so we're gonna pick this kind of like [00:44:47] numbum simulations thing. And uh for [00:44:49] every simulation, we're going to [00:44:51] basically do several things [00:44:53] simultaneously. We're going to see which [00:44:56] which moves are the best in the current [00:44:57] tree. We're going to add extra leaves to [00:45:01] the tree if we get to a point where we [00:45:02] need to add a leaf. And we're going to [00:45:04] update the action values for for the [00:45:06] tree. So that's that's what every every [00:45:08] simulation involves these kind of like [00:45:09] four-step process. So, so the four-step [00:45:12] process is basically selection [00:45:16] um expansion, [00:45:19] evaluation [00:45:22] and backup. [00:45:25] So, so at the beginning of our um Monte [00:45:28] Carlo tree search, our tree is very uh [00:45:30] basic. It only has the the root node or [00:45:33] our current board that our um AI wants [00:45:35] to play at. And so we're going to [00:45:38] basically select the best action for [00:45:40] this. So when this root node is created, [00:45:43] we're also we also know that we can [00:45:45] evaluate this under our neural network [00:45:46] and get the quantities um you know v [00:45:49] theta as well as our um probability over [00:45:53] uh actions [00:45:56] and I'm going to say root. [00:45:59] So for all of the actions here we can [00:46:02] create a bunch of children, right? [00:46:06] So, so this one has um well in this case [00:46:09] I'm drawing a 3x3 board with one one [00:46:11] board missing. So basically there are um [00:46:14] you know eight possible children uh [00:46:16] associated with this root node. [00:46:21] So like [00:46:32] and each of these has an associated [00:46:34] probability of taking that action. [00:46:35] Right? So so there's P8, [00:46:38] P1, P2, etc. Okay. So at the beginning [00:46:42] of our Monte Carlo tree search, we have [00:46:44] our root node and we can initialize it [00:46:46] with some children, right? because we [00:46:47] know it's uh the the policy network [00:46:49] evaluated on the root node gives us on a [00:46:51] 3x3 board with one existing uh stone [00:46:53] placed eight possible children that this [00:46:56] uh AI could take. Um so with each of the [00:46:59] children their policy network also gives [00:47:01] us the probability of selecting that [00:47:02] child. So the um first step is to do the [00:47:06] selection of the tree and again this is [00:47:07] a very shallow tree. All we have so far [00:47:08] is a tree of depth one essentially [00:47:10] right. So our our first move is to [00:47:12] select by maximizing or arg maxing the [00:47:16] pucked criteria which is basically you [00:47:18] know q um qs a plus um you know c pucked [00:47:26] * p of a divided by n over 1 + n a. [00:47:36] So for each of these we're going to uh [00:47:39] you know na is zero for for all the [00:47:42] actions initially n is zero and um and [00:47:46] so we're going to basically just you [00:47:48] know pick according to this um [00:47:52] initially [00:47:54] uh what is going to be the you know [00:47:57] chosen action here is most likely going [00:47:59] to be biased towards um you know the [00:48:01] highest likelihood action here right [00:48:03] because these are sort of uniform for [00:48:04] every So [00:48:07] um let's suppose P1 was the highest [00:48:09] probability node. So you you you [00:48:11] selected this one here. Now you got to [00:48:14] this node and you realize that it's not [00:48:15] a leaf node, right? There are more g [00:48:17] it's not a terminal game. So you cannot [00:48:19] resolve the the final resolution. So the [00:48:21] next step that you do is um expansion. [00:48:24] >> So you um you will then run this node [00:48:29] this board state through the policy [00:48:30] network. Note that this is the AI's [00:48:33] move, right? like a AI is making this [00:48:35] move. And so when we expand this tree, [00:48:37] we're now thinking about what the human [00:48:39] might do or any opponent might do, [00:48:41] right? So so this is like you know your [00:48:43] your your opponent. [00:48:45] >> Um [00:48:47] the tree expansion process actually is [00:48:49] completely uh so so so when we evaluate [00:48:52] the um the node here, we're going to now [00:48:54] evaluate the the node from the [00:48:56] perspective of this player. [00:48:57] >> Yeah. Right. [00:48:58] Um so then this one has possible actions [00:49:02] that we could take and uh we we expand [00:49:05] basically the the the leaf nodes here. [00:49:07] So for each of these nodes um that we [00:49:09] could you know arrive at we're going to [00:49:11] now check how good those nodes are [00:49:13] right. So, so maybe um [00:49:18] from here like the human could play [00:49:20] here, the human could play here or human [00:49:22] could play here. And we're going to um [00:49:25] store essentially the v theta for each [00:49:28] of these things. So v theta of you know [00:49:31] node one [00:49:33] or like node one um prime v theta node [00:49:39] one [00:49:45] um and so we're basically using our [00:49:48] neural network to make an intuitive [00:49:50] guess of how good is this um board from [00:49:53] the perspective of this player. And uh [00:49:56] fortunately because the u it's a zero [00:49:58] sum game it's easy to deduce that you [00:50:00] know the value for this player at this [00:50:03] this step is just one minus the value [00:50:05] for you know from this perspective. So [00:50:06] it's easy to flip the search process uh [00:50:08] depending on which player you're at. Um [00:50:11] and so so this is the expansion step. [00:50:13] You've taken a non-leaf node and [00:50:15] expanded it and evaluated the value. And [00:50:17] this is essentially a quick guess as to [00:50:20] like if I were to play to the end am I [00:50:22] going to win or not? Right? So you can [00:50:24] almost think about the v theta as a [00:50:26] shortcut for uh searching to the end of [00:50:28] the tree for for any given simulation. [00:50:31] Um and then we're go and this is this is [00:50:34] essentially the evaluation step. We're [00:50:35] we're evaluating the quality of each of [00:50:38] these boards. In original alpha they [00:50:40] actually did something uh kind of [00:50:42] interesting which is that they took this [00:50:44] value and they averaged it with the [00:50:47] value of a real go playout. So they [00:50:50] actually played a real game from here [00:50:52] all the way to the end. So So like I'm [00:50:55] just going to draw this squiggly line to [00:50:57] indicate some path. And uh they kind of [00:51:00] like play this all the way to res Trump [00:51:02] tailor resolution [00:51:05] of a full board. And so this is like a [00:51:07] zero or one, right? [00:51:13] And so they took this value and they [00:51:15] just averaged it with with this one [00:51:16] here. So the the the formula they did [00:51:18] was like uh you know alpha * v theta of [00:51:22] of like you know some some node um plus [00:51:26] uh sort of like 1 minus alpha of a of a [00:51:29] true randomly sampled playout. [00:51:35] And you might be wondering like okay [00:51:36] well how do they play this out right [00:51:38] like it would be very very costly to do [00:51:40] another search on on this playout like [00:51:42] almost like a tree within a tree. So [00:51:44] they don't do this. Instead they just uh [00:51:46] take the policy network and play it [00:51:48] against itself. So they just take this [00:51:49] as both players and they just play it [00:51:50] all the way to the end. And and um this [00:51:53] is something that helps ground the um [00:51:56] the estimates here in in reality because [00:51:58] you can get a single sample estimate of [00:52:00] like whether you win or not. You can [00:52:02] think about in the end game where the [00:52:03] board is almost resolved that this one [00:52:05] actually becomes quite useful because [00:52:07] the random the the play according to the [00:52:09] policy will most likely decide a pretty [00:52:12] reasonable guess of the game and so [00:52:14] you're not you know facing a problem [00:52:15] where this one kind of becomes [00:52:16] untethered from from reality. It turns [00:52:18] out this is totally unnecessary. So in [00:52:21] all subsequent papers after Alph Go Lee [00:52:23] they just got rid of this. Yeah. And so [00:52:24] in my implementation I also did the same [00:52:26] and it speeds things up a lot because [00:52:27] you don't have to roll these games out [00:52:29] on every single [00:52:30] >> simulation. Yeah. Okay. So uh again just [00:52:32] to uh reinforce my own understanding and [00:52:35] just to reexlain it for the items by the [00:52:37] way in case it's not obvious the P there [00:52:40] in the select that is the probability [00:52:42] coming from the network in this case [00:52:44] >> correct the policy network here [00:52:46] >> yeah okay so fundamentally [00:52:50] a simulation just think of it as like [00:52:54] rolling out one more node in the search [00:52:56] process [00:52:57] >> um almost so a simulation is easy to [00:53:00] Think about when the whole tree already [00:53:02] exists, right? You just walk down the [00:53:05] tree um using the puck selection [00:53:07] criteria and you you uh and then and [00:53:10] then you keep going. [00:53:11] >> Yeah. [00:53:12] >> Now, uh in Alph Go the data structure is [00:53:15] such that we begin with a tree that has [00:53:17] no like basically only depth one which [00:53:20] is its only children and you want to [00:53:22] iteratively build out the tree as you're [00:53:25] also selecting actions down the tree. So [00:53:27] that's the kind of core thing here is [00:53:29] that because Go is such a combinatorily [00:53:30] complex game, you cannot afford to build [00:53:32] the tree in advance and then search it. [00:53:34] You must search while building the tree, [00:53:36] >> right? Okay. [00:53:37] >> So um let me just finish up with [00:53:39] actually the last step which is the [00:53:40] backup. Right. So once you've scored [00:53:42] these things, you basically take the [00:53:45] mean the the value the Q value assigned [00:53:48] to the node here for taking this action [00:53:50] is now just the average across your [00:53:53] evaluated values. It's you take a [00:53:56] running mean over over uh all of the um [00:53:59] the the simulations that you've taken [00:54:01] and they average the values of the [00:54:03] children nodes. [00:54:04] >> So so that's what is known as the backup [00:54:06] step and once you evaluate this you can [00:54:08] actually kind of recursively go back. So [00:54:09] if you know the you know the action [00:54:11] value of this node you can then take the [00:54:13] average on its parent and so on and so [00:54:15] forth. So so you have this kind of [00:54:17] four-step process where you are choosing [00:54:19] the best action that you know of so far. [00:54:21] Then you may run into a node where you [00:54:24] uh you you haven't been to before. So [00:54:26] you need to grow the tree a bit. And [00:54:28] then you run it through the network to [00:54:30] guess whether you're going to win or [00:54:31] not. And then you walk all the way back [00:54:33] up to the to the root node to update [00:54:35] your values on what the best moves are. [00:54:38] So as you do this iteratively, this [00:54:40] selection criteria will cause you to [00:54:42] visit the because you're always [00:54:44] selecting according to this criteria, [00:54:46] you're always going to be selecting the [00:54:48] best action you think at any given [00:54:49] branch. Right? So, so the final um visit [00:54:53] counts of like how often you chose these [00:54:55] things will reflect your correct policy [00:54:59] distribution as induced through this [00:55:01] search process. Um and so so the visit [00:55:04] count that we store in the node earlier [00:55:05] actually becomes the sort of vote for [00:55:07] like which way we should finally select [00:55:09] an action here. [00:55:10] >> Yep. [00:55:11] >> So um you know as a sort of test of [00:55:13] understanding it's worth thinking a [00:55:15] little bit about whether we could make [00:55:17] this even simpler, right? like could we [00:55:18] actually maybe even get rid of this one [00:55:20] and still make the thing work? Um so [00:55:24] recall that you know when you do an [00:55:25] expansion and then an evaluation at [00:55:27] let's say this node you you are checking [00:55:29] the sort of win probability of each of [00:55:31] the child nodes right and so if this one [00:55:34] is you know like one and these are zero [00:55:37] um you do kind of know something about [00:55:40] which action might be better to take and [00:55:42] so why would you need still need this [00:55:44] right like why not just uh normalize [00:55:47] this one into some distribution and call [00:55:49] that your your policy distribution Um [00:55:52] this is fine. You can do this and um [00:55:55] this probably does work. But in practice [00:55:57] having a single forward pass that gives [00:56:00] you a pretty good guess is um is how the [00:56:03] uh the breath is is uh is pruned out. Um [00:56:07] the there is a sort of duality here. [00:56:09] Like it would be weird if let's say the [00:56:11] policy recommended an action that [00:56:14] disagreed with the value, right? If if [00:56:15] let's say policy said this was very high [00:56:17] probability but this one said it was you [00:56:19] know low value then there's actually [00:56:20] something kind of fundamentally wrong [00:56:22] between uh between your policy head and [00:56:24] your value head. So, so they are linked [00:56:25] and uh you probably could get rid of [00:56:27] this if you came up with a different way [00:56:29] to recover this from just the value [00:56:31] evaluations, [00:56:32] >> right? But, but just to make sure I [00:56:35] understand the reason you don't do that [00:56:37] is so that you don't have to do 360 [00:56:38] independent [00:56:40] uh forward passes to like hey here's the [00:56:42] value of everything. Let's arg max over [00:56:43] instead you can just do one forward pass [00:56:44] and get like the probabilities of all of [00:56:46] them. [00:56:46] >> Um you can usually batch these somewhat [00:56:50] efficiently. Uh so it probably is not a [00:56:53] huge computational burden in practice [00:56:55] but yes you would have to pass 361 board [00:56:59] like up to 361 boards into a single mini [00:57:01] batch update to evaluate all the values [00:57:03] here then normalize them. [00:57:05] >> Um now there's actually a more uh [00:57:07] important reason why we still do this [00:57:09] which is how Monte Carlo tree search is [00:57:11] used to feed back on itself [00:57:13] >> um and and sort of recursively improve [00:57:15] its own predictions and search [00:57:17] capabilities. And that's where this this [00:57:18] one having this as an explicit entity [00:57:20] you're modeling rather than an implicit [00:57:23] normal normalization over your value is [00:57:25] is a is a good idea. [00:57:26] >> Makes sense. Okay. [00:57:27] >> Okay. So um so we talked about the [00:57:29] simulations and basically you know what [00:57:31] you end up with as you roll out the [00:57:32] number of simulations is a tree that [00:57:34] kind of looks like [00:57:39] I'm I'm I'm drawing a very [00:57:40] lowdimensional version of this. Of [00:57:41] course, it's in in in the real game, [00:57:43] it's it's much more high dimensional, [00:57:45] but like you'll end up with basically a [00:57:47] tree structure that like has um [00:57:51] a lot of leaves that kind of terminate [00:57:53] and are not visited again because their [00:57:54] value is deemed to be too low. But then, [00:57:56] you know, along one path there will be a [00:57:59] set of actions with very very high visit [00:58:01] counts that kind of gravitate towards [00:58:04] that one set of decisions as you [00:58:06] increase uh n here. So, so this is kind [00:58:09] of like the mental picture of what the [00:58:11] tree in Monte Carlo research looks like. [00:58:13] And you should contrast this with like [00:58:14] an exhaustive tree like in tic-tac-toe [00:58:17] where you could say like, you know, [00:58:18] there's there's nine actions and then [00:58:20] eight and then seven and six and so it's [00:58:22] a sort of like nine factorial sized um [00:58:25] tree. Um the Monte Carlo tree search in [00:58:28] Go is very very sparse, right? It only [00:58:30] considers the paths that you've expanded [00:58:33] ch children nodes on. Okay. So, um now [00:58:37] that we have the search algorithm that [00:58:39] applies the value function as well as [00:58:41] the policy function, um we can now talk [00:58:45] about how the Monte Carlo tree search [00:58:48] algorithm can actually act as a [00:58:50] improvement operator on top of these [00:58:52] guys here. [00:58:54] >> 20 years ago, Jane Street's data center [00:58:56] fit in the corner of an office. Ron [00:58:58] Minsky, who co-leads the tech group [00:59:00] there, told me about how it all got [00:59:01] started. One of our comput clusters we [00:59:03] called the Hive. And I remember the [00:59:04] first mission of the hive was literally [00:59:06] like six Dell boxes stacked on top of [00:59:08] each other at the end of the row. And [00:59:10] the trading systems themselves we also [00:59:12] had there because we actually wanted the [00:59:15] ability to make sure we could turn the [00:59:17] damn thing off. I mean there were ups [00:59:18] and downs like literally at some point [00:59:20] you know one of the people who was [00:59:21] cleaning the office unplugged one of the [00:59:23] trading systems in the middle of the day [00:59:25] as they were vacuuming. So you know in [00:59:27] the end it is in fact better to have it [00:59:29] all in a data center. Jane Street's data [00:59:30] centers have come a long way since those [00:59:32] six cells and I got to tour one of them [00:59:33] in Texas with Ron and Dan Pontto Cororvo [00:59:35] who leads Jane Street's physical [00:59:37] engineering team. [00:59:38] >> You know these cabinets, these GB300 [00:59:39] cabinets consume at peak about 140 kW. [00:59:42] Compare that to tra traditional air [00:59:44] cooled you're talking about 10 to 40 KW. [00:59:47] So a lot more. [00:59:47] >> We got deep into the details of running [00:59:49] one of these data centers. Things that I [00:59:50] had never considered before. [00:59:52] >> It's filled with a liquid uh a mix of [00:59:54] distilled or deionized water and uh [00:59:56] propyline glycol. 25% of propyline [00:59:58] glycol. Um, that's to inhibit any [01:00:01] bacteria or algae growth. [01:00:02] >> I don't love the world where we have to [01:00:04] worry about bacteria growing in our [01:00:06] servers. [01:00:06] >> I got to see way more of what actually [01:00:08] happens in a data center than I've ever [01:00:10] seen before. Jane Street was willing to [01:00:12] literally pull up the floorboards and [01:00:13] take out the racks and take me to the [01:00:15] back where all the chillers are. You can [01:00:16] check all of this out at [01:00:18] janestreet.com/wcash [01:00:20] where we posted the full tour. Okay. So, [01:00:23] um we now talk about the RL part of like [01:00:26] how this thing gets stronger by playing [01:00:28] itself, right? Um let's say we play a [01:00:31] game where [01:00:33] the AI so you make a move [01:00:39] AI AI um will will kind of compute the [01:00:42] search and then this is this sort of [01:00:44] visit count distribution. Um let's say [01:00:47] this is your policy your policy initial [01:00:49] policy recommendation at the at the at [01:00:51] this node [01:00:52] >> and then after MCTS [01:00:55] it uh gets more confident about one of [01:00:57] these actions right and and so maybe the [01:01:00] the distribution looks a bit more peaky [01:01:01] like this based on the the search. Now [01:01:04] of course you can tune the search [01:01:05] process so that it ends up more diffuse [01:01:07] but that's probably not a good idea. [01:01:08] MCTS should get more confident about [01:01:10] specific actions um than others, but it [01:01:13] of course might place a lot of weight on [01:01:15] you know other actions initially and [01:01:16] then as you increase the number of sims [01:01:18] it should converge to a very peaky [01:01:19] distribution. Um so so this is your new [01:01:22] uh let's call this like pi [01:01:25] let's wrap this in like a MCTS operator [01:01:30] of you know a given s right so after [01:01:35] applying mcts process your your policy [01:01:38] recommended distribution looks like this [01:01:39] it's it's a bit more peaky than than the [01:01:41] previous one um and so then you take the [01:01:45] arcmax or maybe you just sample from [01:01:47] this uh it doesn't have to be the arm [01:01:48] max and then you you make your [01:01:50] And then um and then you throw away the [01:01:51] tree and then you you begin a new on the [01:01:53] next move, right? So [01:01:56] again like you um you know you compute a [01:02:00] new distribution. [01:02:08] So initially maybe your guess looks like [01:02:10] this and then you refine it through [01:02:12] MCTS. [01:02:16] >> There should be one more X on the board, [01:02:17] right? [01:02:18] >> I'm sorry. That's correct. Yes. [01:02:21] Um to something that looks like [01:02:26] right. So so on every move you have your [01:02:29] initial guess from your policy network. [01:02:31] Um and then the search process that [01:02:33] combines your policy network and your [01:02:34] value network arrives at a more [01:02:36] confident action that you take [01:02:38] >> and and then so and so forth and then [01:02:40] the game ends and one person wins and [01:02:41] one person loses. So a um the way that [01:02:45] uh the beauty of of how AlphaGo trains [01:02:48] itself is that it actually can take this [01:02:51] final search process the outcome of the [01:02:54] search process and tell the policy [01:02:56] network hey like you know instead of [01:02:59] having MCTS do all this you know leg [01:03:01] work to arrive here why don't you just [01:03:03] predict that from the get-go right like [01:03:04] why don't you like you know not use this [01:03:06] guess and just predict this to begin [01:03:07] with and if you have this guess to begin [01:03:09] with in your policy network then MCTS [01:03:11] has to do a lot less work to to get [01:03:13] things to work. And so if we draw like a [01:03:14] sort of test time scaling plot um so so [01:03:18] let's say like this is like number of [01:03:19] simulations. [01:03:22] um let's say you know at at zero [01:03:24] simulations your your sort of implicit [01:03:26] win rate is like um is like I don't know [01:03:30] here and then and then um without any [01:03:32] simul if you just take this raw action [01:03:34] this is your what your win rate is and [01:03:36] let's say as we increase the number of [01:03:38] sims um maybe maybe you kind of have a [01:03:40] win rate that looks like this right so [01:03:44] um when you search for let's say a [01:03:46] thousand simulation steps that gets you [01:03:50] to a um a policy here that gets you to [01:03:54] here, which is great. But if you were to [01:03:57] distill this MCTS policy network back [01:04:00] into your sort of uh shoot from the hip [01:04:02] policy network, then you could actually [01:04:05] um uh you know, start here. Like if [01:04:10] let's say this was, you know, zero um [01:04:12] for uh by by distillation, then if you [01:04:14] spend another 1,000 sim steps, then you [01:04:18] actually kind of get to here. It's [01:04:19] almost like if you could just, you know, [01:04:22] am um amvertise the the first 10,00 [01:04:24] steps actually into the policy network [01:04:26] instead of the search process, then you [01:04:28] could begin at a much better starting [01:04:29] point and then get a much better result [01:04:31] for for your uh for the number of sims [01:04:33] that you put. [01:04:34] >> The sigmoid type nature of test time [01:04:36] scaling as the number of simulations [01:04:37] increases the the increase in win rate [01:04:39] is uh smaller. Is that true even for the [01:04:42] distilled network? That is to say, is [01:04:45] there some gain of like, okay, we start [01:04:46] from the distilled, we get these early [01:04:47] gains again, or is that just inherent to [01:04:49] like the nature of [01:04:50] >> yeah, [01:04:50] >> MCTS? [01:04:51] >> To be honest, I actually don't know the [01:04:52] test time scaling behavior of MCTS [01:04:55] simulations. And I I believe it might [01:04:56] actually be quite sensitive to how [01:04:58] strong this one is in practice. I'm just [01:05:00] drawing a monotonically increasing [01:05:01] function that gets to one. So yeah, so [01:05:03] so don't pay too much attention to the [01:05:05] shape of the curve. Just know that it's [01:05:06] monotonic with respect to um Okay. So um [01:05:11] so so the idea of MCTS is very brilliant [01:05:14] which is like we're gonna we got [01:05:15] something better by applying search. [01:05:17] >> Yeah. [01:05:18] >> And um we're going to now on our next [01:05:21] iteration of updating this network just [01:05:23] train this to approximate the outcome of [01:05:26] a thousand steps of search. Y [01:05:27] >> and so instead of starting here we get [01:05:29] to now have a neural network start here [01:05:31] and then and then you know the play gets [01:05:33] stronger once we then apply another [01:05:34] thousand steps on top of it and you can [01:05:36] keep going. Right. So the training [01:05:38] algorithm for Alph Go is to basically [01:05:40] take the games where you've applied the [01:05:42] search on every move that the policy [01:05:44] encountered uh whether you won or lost [01:05:46] and that's quite important um and you're [01:05:48] just going to train the model to imitate [01:05:51] the search process. [01:05:53] >> So um there's an analogy to robotics [01:05:55] actually which is the dagger algorithm. [01:05:57] >> Um uh f first I'm going to draw like a [01:06:00] uh schematic of like let's say you know [01:06:02] the states right? So S0, S1, S2, S3. So [01:06:07] let's say, you know, we we took a series [01:06:09] of actions in an MDP to [01:06:13] to get uh a trajectory. [01:06:17] And these actions may be sub-optimal, [01:06:19] right? Maybe we lost at the end of this [01:06:20] game. [01:06:24] So um there is a family of algorithms [01:06:27] that basically take trajectories uh and [01:06:30] relabel the actions to better [01:06:32] trajectories. So maybe a better action [01:06:35] here would have been to take you know a [01:06:37] z prime. A better action here would have [01:06:39] been to take a1 prime and yet another [01:06:42] one like a2 prime [01:06:45] a3 prime. So um [01:06:50] what MCTS is doing is basically saying [01:06:52] like you play this game where you [01:06:53] eventually lost but on every single [01:06:55] action I'm going to give you a strictly [01:06:57] better action that you should take [01:06:59] instead. It does not guarantee that you [01:07:01] are going to win but it does guarantee [01:07:03] that you know if you take these tpples [01:07:06] as training data so that you retrain [01:07:09] your uh your your policy network to [01:07:11] predict these ones instead of these ones [01:07:12] you're going to do better. And uh this [01:07:15] is very related to dagger in u robotics [01:07:18] and in imitation learning where you want [01:07:19] to collect a intervention here and even [01:07:22] if you're in a in a not great state for [01:07:24] example like a self-driving car that you [01:07:26] know veers off the side of the road [01:07:27] there is still a valid action that kind [01:07:29] of corrects you and brings you back. [01:07:30] >> Yeah. Okay. So um pedantic question but [01:07:34] is there a guarantee that MCTS must be [01:07:36] better than the policy? For example, you [01:07:37] could imagine early on in training [01:07:39] because MCTS is informed by the value [01:07:42] network. Yeah. early on in training uh [01:07:45] when the value network hasn't been well [01:07:47] trained on finished uh games um that [01:07:50] like MCTS is worse than sort of randomly [01:07:52] initialized policy. So is it just like a [01:07:55] heristic that MCTS is better than policy [01:07:56] or is that like is there some guarantee [01:07:58] >> right in in practice it is a heristic um [01:08:01] and it does work in also in practice but [01:08:03] let me illustrate a example where MCTS [01:08:05] can give you a worse distribution than [01:08:07] your policy network. So um and this can [01:08:10] often happen if your self-play algorithm [01:08:12] um has trained to a good point but then [01:08:14] somehow it's uh it collapses because [01:08:16] it's um it's not trained on diverse data [01:08:18] or something right so um let's say we [01:08:20] have a board state where um the policy [01:08:23] recommendations here are very good so so [01:08:25] like you know pi of as is like great but [01:08:31] um somehow because maybe we're playing [01:08:33] on a lot of games where the the bots [01:08:35] just resign instead of playing all the [01:08:37] way to the Trump Taylor resolution, they [01:08:39] kind of forget um how to evaluate those [01:08:42] kind of late stage plants, right? Like [01:08:43] in the in the case that we showed with [01:08:45] the corner play, maybe like 100% of our [01:08:48] training data in our replay buffer has [01:08:50] lost examples of how to evaluate the [01:08:52] value function at those states. So you [01:08:54] might end up in a scenario where your [01:08:56] terminal value um is like very bad. And [01:09:01] if the terminal values of the leaves are [01:09:04] not good, then this will actually [01:09:05] propagate all the way up and cause your [01:09:08] um your your puck selection criteria and [01:09:11] your backups to be off and then you'll [01:09:13] end up visiting a very very different [01:09:14] distribution than what your policy [01:09:16] initially recommended. [01:09:17] >> Um also if your number of sims is low [01:09:20] then you might also have a variance [01:09:21] issue where you just don't explore [01:09:22] enough, right? Like like a it's only [01:09:24] guaranteed to converge when you kind of [01:09:26] take n to infinity. Um so so variance in [01:09:30] you know your search process as well as [01:09:32] inaccuracies in your evaluation can [01:09:34] definitely screw with the quality of [01:09:36] your policy validation and so that's why [01:09:39] it's not a guarantee to improve but and [01:09:41] that is why I think I I suspect why [01:09:43] AlphaGo Lee had the uh playouts to the [01:09:46] end in their training algorithm so they [01:09:47] could ground this thing in real plays. [01:09:50] >> Yeah. Um, in practice, what you could [01:09:51] also do is just like for 10% of the [01:09:53] games, you you prevent the bots from [01:09:55] resigning and you just say like resolve [01:09:57] it to the end. So, you get some training [01:09:58] data in your replay buffer to really [01:10:00] resolve those kind of like latest stage [01:10:02] playouts that normal human players would [01:10:03] would kind of uh not play to. [01:10:06] >> Yeah. Yeah. [01:10:07] >> So, um, so this is why MCTS kind of if [01:10:09] you assume that the value functions are [01:10:12] correct, uh, why it gives you a better [01:10:14] policy is because yeah, assume and it's [01:10:16] a very critical, you know, chain of [01:10:17] assumptions. Assuming that this is [01:10:18] accurate, then uh your search process [01:10:21] should give you a better recommendation [01:10:23] than your initial guess, [01:10:24] >> right? Okay. So, if you have a cold [01:10:25] started policy, uh you have an alpha [01:10:28] zero type thing, really what's happening [01:10:29] for the first few epochs is the policy [01:10:32] is kind of useless. And what you're [01:10:33] really just doing is hey, let's play [01:10:35] full games. And uh once we have a played [01:10:38] full games for the preceding moves, [01:10:41] we'll have labeled [01:10:43] who won, who didn't win, and the loss [01:10:47] for Alpha Zero has two components, which [01:10:49] is like how good is the policy relative [01:10:51] to MCTS and how good is the value [01:10:54] prediction relative to who actually won [01:10:56] the game from this move. And this is [01:10:57] this sort of like you can think of this [01:10:59] being applied to every single action or [01:11:00] every single move. [01:11:01] >> Correct? And really what's happening at [01:11:03] the beginning of alpha zero training is [01:11:05] just like we're trying to get the value [01:11:06] function to actually predict who will [01:11:08] win the game if you're if you find [01:11:09] yourself in this state and you're this [01:11:12] player uh and functionally that's all [01:11:15] that's happening and later on once [01:11:16] that's well trained now the policy is [01:11:18] also improving [01:11:19] >> correct [01:11:19] >> okay [01:11:20] >> one trick I did find to be pretty useful [01:11:22] and this is not a peer-reviewed claim so [01:11:24] just like take this with a grain of salt [01:11:25] is like I I found it useful in my own [01:11:27] imple implementation to do do the [01:11:29] following um you want to first make sure [01:11:31] that this is good before you invest a [01:11:33] lot of cycles doing MC tests, right? [01:11:35] Like like it it doesn't really make a [01:11:36] lot of sense to do search on garbage [01:11:39] value predictions. Um so so you want to [01:11:41] kind of start at a good place where this [01:11:43] works. Alph does a very good thing where [01:11:45] it just takes human games and then you [01:11:46] you like uh trade on it and it just [01:11:48] works, right? [01:11:48] >> Totally works. Um you could also take an [01:11:50] open source go play against itself um [01:11:53] generate data also works. Uh so so if [01:11:55] you have some like offline data set that [01:11:58] um that uh has realistic good play you [01:12:02] can easily learn the latest stage um [01:12:05] value functions pretty well and that's [01:12:07] the that's what you kind of need to [01:12:08] start the search process. [01:12:09] >> Sorry can you just read this sentence [01:12:10] one more time? [01:12:10] >> Sure. So so um it it's quite easy to [01:12:13] evaluate a late stage go game like when [01:12:16] almost all the pieces are on the board [01:12:18] >> like it's almost like a decidable [01:12:19] problem right because it's the lower and [01:12:20] lower uncertainty as to like the depth [01:12:22] of the tree. So um most games played to [01:12:25] the end by reasonable people um will be [01:12:28] good training data to train a good value [01:12:29] function at terminal parts of the tree. [01:12:31] >> Got it. Okay. [01:12:32] >> Then as you play more games the um the [01:12:35] the search will back up good values into [01:12:37] the the sort of intermediate nodes of [01:12:39] the tree. And then like as you increase [01:12:41] the amount of data your your value head [01:12:43] gets a good intuition of like what is a [01:12:45] healthy board state versus a not healthy [01:12:47] board state. Yeah. That that those are [01:12:48] much more subtle to judge in the midame [01:12:50] than the beginning or the end. So, so [01:12:52] the most difficult part to score is like [01:12:54] not the beginning or the because the [01:12:56] beginning is just like obviously 0.5 and [01:12:57] then at the end it's like pretty obvious [01:12:59] who's winning. So, so the hard part that [01:13:00] you want to learn in the value function [01:13:02] is like who is winning in the middle [01:13:03] >> and so this is this is actually very [01:13:05] analogous to TD learning. [01:13:06] >> Yes. And there's a a beautiful [01:13:07] connection to TD learning that we can [01:13:09] you know talk about in a bit uh as [01:13:10] opposed to you know contrasting with [01:13:12] Monte Carlo research. So um so you first [01:13:15] want to get good value functions and [01:13:17] expert data can kind of give you a quick [01:13:18] shortcut. I recommend for you know [01:13:20] practitioners just do that first just to [01:13:22] you know initialize to a good starting [01:13:23] point and then if you want to do the [01:13:24] alpha zero thing or um or kadigo kind of [01:13:27] tab raza learning um then what you can [01:13:30] try to do is on a small board play [01:13:32] random games just take a random agent [01:13:34] and if you play like uh you know 50,000 [01:13:37] games you'll actually learn a pretty [01:13:39] good value function as well because on a [01:13:40] 9 by9 board there's actually um you can [01:13:43] see enough of the common patterns with [01:13:44] random play and then if you train a [01:13:46] model that kind of can train on both 9 [01:13:48] by9 in 19 by9 data um and Katago was a [01:13:51] was a proposed one of these [01:13:52] architectures then um there's some [01:13:54] pretty good transfer learning from the [01:13:56] value uh value head evaluated at 9 by9 [01:13:59] to the 19 by9 [01:14:00] >> right because this this unlike other [01:14:01] games has much like a very much a sense [01:14:03] of like there's not like a new kind of [01:14:05] piece that is introduced when you [01:14:07] increase the size or something [01:14:08] >> if if we take it to its limit and [01:14:09] consider like a very tiny like 4x4 [01:14:11] goboard like if you play 50,000 games [01:14:14] you're going to have a lot of end states [01:14:15] that look like human play right like [01:14:16] it's just like tic-tac-toe at that point [01:14:18] So, so if you like broaden this a little [01:14:20] bit to like 9 by 5 x 5 or 9 by 9, it's [01:14:22] not unrealistic to imagine that like [01:14:24] purely random play will actually [01:14:26] generate pretty reasonable looking [01:14:27] boards. Um, and then so you can score [01:14:29] those pretty easily. And so that is what [01:14:31] gives you the bootstrapping to be able [01:14:32] to then improve your policy with search. [01:14:35] >> But it's very very critical that MCTS [01:14:37] has accurate value uh estimates and you [01:14:40] need to ground the value. Ultimately [01:14:42] MCTS will fall apart if you don't have a [01:14:44] grounding uh function for um for the [01:14:46] value. I I'd be curious how much compute [01:14:49] you save by training the value and [01:14:50] policy on the same network that because [01:14:53] they share the same representations, how [01:14:54] much more efficient learning is because [01:14:56] that would be interesting if um they're [01:14:58] basically kind of we've just talked [01:15:00] about how they're kind of making similar [01:15:02] predictions or they should be in line [01:15:03] with each other. Yeah. [01:15:04] >> And so I'd be curious if like actually [01:15:06] yeah you just you're you're like having [01:15:07] the amount of compute you had to do by [01:15:08] keeping the same network. [01:15:09] >> Right. AlphaGo Lee, the original AlphaGo [01:15:12] paper had two separate networks and then [01:15:14] um in all subsequent papers um they they [01:15:16] merged them into two heads and [01:15:18] presumably this saves compute. But [01:15:19] answering that question in a very [01:15:21] rigorous scientific way is actually uh [01:15:23] it's it's a simple question but but in [01:15:25] practice actually takes like if you [01:15:26] really want to chase that question down [01:15:27] to its its limit. It's uh it takes quite [01:15:29] a bit of work to you know really solve [01:15:31] that. [01:15:31] >> Um but intuitively yes they share a lot [01:15:34] of representations. So, so and you know [01:15:36] as we mentioned there is there is a sort [01:15:37] of like your your policy network and [01:15:39] your value network when doing evaluation [01:15:41] should kind of agree right so there [01:15:43] there really should be this sort of [01:15:44] consistency between them. [01:15:45] >> Yeah. Tell me if this is the wrong way [01:15:46] to think about it. I feel like when I [01:15:49] learn how an LLM works and how simple [01:15:52] RLVR is, at least [01:15:55] as an algorithm, how simple it is, I'm [01:15:58] sort of stunned by the kinds of things [01:15:59] it can do. Uh that it can learn how to [01:16:03] build very complicated code repositories [01:16:04] and whatever simply from getting like a [01:16:07] yes no. And here I feel like the if you [01:16:09] understand it more deeply of like just [01:16:11] predicting MCTS and [01:16:14] it actually seems AlphaGo seems less [01:16:16] impressive in retrospect the more you [01:16:17] understand it because you're like oh [01:16:19] you're putting in a lot of bias by just [01:16:21] saying how much you you're like telling [01:16:22] it how we should titrate exploration as [01:16:25] things go on you're building this very [01:16:27] explicit uh tree search for it and so I [01:16:30] don't know if you show that intuition [01:16:32] where it actually the more you [01:16:34] understand it the less impressive the [01:16:35] accomplishment in 2017 seems themes. [01:16:38] >> I uh I personally disagree. I I think [01:16:40] they're profound for different reasons [01:16:42] and I don't understand the LM RL like [01:16:45] enough to like kind of comment on your [01:16:47] podcast about it. Um but um [01:16:50] >> I think AlphaGo so so yeah, why is it a [01:16:52] profound accomplishment? I think maybe [01:16:53] it's worth stepping back a little bit [01:16:54] and just like [01:16:55] >> uh it is different than modern RL and we [01:16:57] can talk a little bit about like some of [01:16:59] the algorithmic choices there but um I [01:17:02] think the most profound thing here is [01:17:04] that a um a 10 layer neural network pass [01:17:08] so basically uh 10 steps of 10 steps of [01:17:12] reasoning. [01:17:13] >> Yeah. [01:17:13] >> And of course the reasoning is not just [01:17:15] one trail of thought. It could be like [01:17:16] the distributed representations and a [01:17:18] lot of thoughts going on at the same [01:17:19] time. But uh by construction, let's say [01:17:22] a 10- layer neural network can only do [01:17:24] 10 sequential steps of thinking, right? [01:17:26] >> Um 10 steps of neural network paralyzed [01:17:29] distributed representation thinking is [01:17:32] able to amortize and approximate to a [01:17:35] very very high fidelity a nearly [01:17:37] intractable search problem. Yeah. [01:17:39] >> So, so this was a breakthrough that I [01:17:42] think most people don't even understand [01:17:44] today like fully comprehend like how [01:17:46] profound that accomplishment is. And [01:17:49] then this is what also girds like um [01:17:51] alpha fold for example, right? Like um [01:17:53] uh where you have a very very difficult [01:17:57] physical simulation process that you [01:17:58] would need to roll out so many [01:18:00] microscale simulations and yet like 10 [01:18:03] steps of a somewhat small neural network [01:18:06] can somehow capture what feels like a [01:18:09] you know MPL class uh problem into a [01:18:11] single um problem. And and so I I it it [01:18:15] actually makes me wonder if you know our [01:18:18] understanding of problems like P equals [01:18:20] NP or you know these very fundamental [01:18:22] like computational hardness problems are [01:18:25] >> incomplete right like like it's not like [01:18:27] you know obviously this is not a proof [01:18:30] of like P= MP or anything but but [01:18:32] there's something to it that like kind [01:18:33] of is very disturbing where like what [01:18:35] felt like a very hard problem can fall [01:18:37] to a very very simple macroscopic [01:18:39] simulation. [01:18:40] That is a very interesting insight that [01:18:42] a lot of problems which are proven to be [01:18:46] np hard like I don't know if go is [01:18:47] proven to be np hard but okay protein [01:18:49] folding etc have been like neural [01:18:53] networks can solve them because they're [01:18:54] np hard in the worst case but we're not [01:18:55] dealing with the worst we're usually not [01:18:57] concerned about the worst case we're you [01:18:58] know these problems have a lot of [01:18:59] structure to them usually [01:19:00] >> yeah I I think that the the the kind of [01:19:03] question we should be asking ourselves [01:19:04] is like we've been formulating you know [01:19:07] solutions to MP hard problems as in like [01:19:10] kind of worst case complexity [01:19:11] >> and I wouldn't say you know this solves [01:19:13] go right it doesn't give us a exact [01:19:15] solution of the optimum [01:19:16] >> but in practice like it is extremely [01:19:19] useful [01:19:19] >> and the same thing has been shown in [01:19:21] like alpha tensor alpha fold where like [01:19:24] >> yes there is a very hard problem that in [01:19:26] the worst case seems intractable and yet [01:19:28] we're able to make like almost arbitrary [01:19:30] amounts of progress so so here's a sort [01:19:32] of like uh you know in the limit what is [01:19:33] what what what might this look like [01:19:35] right well um if you want to simulate [01:19:37] you know something very complex like [01:19:39] weather or um predict the future like [01:19:41] you know do we live in a simulation or [01:19:43] not. Um the computing resources you need [01:19:45] to build a very complex simulation might [01:19:48] be much smaller than you think based on [01:19:52] you know our ability to amortize a lot [01:19:54] of that computation into the forward [01:19:56] pass of a single network. [01:19:57] >> Interesting. So to me, yeah, Alpha Gold [01:19:59] was the first paper that kind of like [01:20:00] really showed this like profound level [01:20:03] of, you know, simulation being [01:20:05] compressed into a small amount of [01:20:07] >> um I feel totally not at all qualified [01:20:10] on the computational complexity of the [01:20:12] math to comment on this, but I wonder if [01:20:14] um there's an important role of chaos [01:20:17] here where [01:20:19] if under what is the problem with [01:20:21] weather and why does it take 10x the [01:20:24] amount of resources to predict weather a [01:20:26] day out? uh and continually so for every [01:20:29] more day out is because it's a chaotic [01:20:30] system and so small perturbations can [01:20:33] totally change the final estimate as as [01:20:36] time goes on and um I guess it's [01:20:39] interesting well I guess you would [01:20:40] expect that for go and protein folding [01:20:42] as well [01:20:42] >> so here's an analogy to weather that [01:20:44] that might be relevant in go so um the [01:20:46] problem of like you know here's our [01:20:48] current board state [01:20:53] >> um [01:20:55] given what we know about both players [01:20:57] what is the board state in the future? [01:20:59] >> Yeah. [01:20:59] >> What is the exact board state in the [01:21:01] future? Right? This is [01:21:03] >> uh this is extremely sensitive to [01:21:05] initial conditions like a single stone [01:21:07] place here can kind of disrupt the [01:21:08] entire prediction. Yeah. Right. [01:21:09] >> So this is hard. This is kind of [01:21:11] intuitively the chaotic problem. Um [01:21:14] >> and yet somehow [01:21:17] so this is this is hard. Somehow we can [01:21:19] predict who's going to win. [01:21:21] >> Yeah. [01:21:21] >> Like and this captures a lot of [01:21:24] possibilities here. M [01:21:25] >> and so there's this more macroscopic [01:21:27] quantity that we really care about which [01:21:29] is the average or expectation or some [01:21:31] sort of global macro structure over a [01:21:34] lot of like you know possible futures. [01:21:36] >> Interesting way to think about it. [01:21:37] >> And so so in in weather it could be the [01:21:38] same thing right like we don't exactly [01:21:40] care like what the you know velocity of [01:21:42] wind 6,000 uh feet above a specific [01:21:46] latitude longitude is. We kind of care [01:21:48] like where's the hurricane or you know [01:21:51] you know things like that. And and I [01:21:52] would say like in chaos, you know, [01:21:53] there's a classic like Lorenza tractor [01:21:55] which kind of looks like this, right? Um [01:21:57] yes, you you don't if you start anywhere [01:21:59] on the Lorenza tractor, you don't know [01:22:00] where you're going to end up, [01:22:02] >> but you you do know that the thing looks [01:22:04] like this. [01:22:04] >> Yeah. Yeah. [01:22:04] >> Right. And and so there's this kind of [01:22:06] beauty of like sometimes we don't [01:22:08] necessarily care about the microscale [01:22:09] things. We actually care about the [01:22:11] macroscopic structure. [01:22:12] >> Interesting. [01:22:12] >> And and these things can be predictable. [01:22:15] >> And contrast that say to something like [01:22:17] a hash function which is also incredibly [01:22:19] dependent on initial conditions. but [01:22:21] doesn't have a macro structure at least [01:22:22] hopefully if like the work. Yes. [01:22:25] >> Um and so there's like no equivalent of [01:22:28] a value function or like broadly how's [01:22:30] the weather going to be that is [01:22:32] interesting there. It's really just [01:22:34] about what is the move what is a board [01:22:36] going to look like 100 moves from now. [01:22:37] Exactly. [01:22:38] >> Uh yes intuitively that seems correct. I [01:22:40] I uh and then again this is also out of [01:22:42] my my area of expertise but I um I find [01:22:46] it interesting that like cryptography [01:22:48] has not been able to like um the tools [01:22:51] of cryptography and uh you know um [01:22:54] hashing have also not been able to prove [01:22:56] that like uh you cannot come up with [01:22:59] fast approximation like you cannot come [01:23:00] up with fast approximations right if [01:23:02] that if they were able to do that then [01:23:03] you could prove p is not equal to m [01:23:05] >> yeah yeah in fact we know that there's [01:23:06] structure in many cryptographic [01:23:08] protocols obviously like uh uh rsa [01:23:10] cryptography there is structure and that [01:23:12] structure is what quantum computers [01:23:13] exploit to to break them. Right. [01:23:15] >> I see. [01:23:16] >> Um Reiner has a very interesting blog [01:23:18] post which we talked about in the [01:23:19] episode where he uh talks about how if [01:23:22] you look at at a high level what [01:23:24] cryptographic protocols look like and [01:23:25] what neural networks look like. It's [01:23:27] extremely similar [01:23:28] >> where you have sequential layers of [01:23:32] jumbling information together. And it's [01:23:34] because there's been this convergent [01:23:36] devolution in the algorithms where in [01:23:38] cryptography you want the final state to [01:23:41] be incredibly sensitive to initial [01:23:43] conditions so that it can come out sort [01:23:45] of looking jumbled based on if you [01:23:47] change anything and then neural networks [01:23:49] you similarly want everything to be [01:23:51] dependent on all the information because [01:23:53] you want to process all the information [01:23:54] and consider how it relates to itself. [01:23:56] >> Yeah, you have the maximum power of a [01:23:57] neural network at the edge of chaos. Um [01:23:59] I think there's some like research [01:24:01] papers from uh Joshua Schlick's team on [01:24:03] on on this. [01:24:04] >> Yeah. [01:24:04] >> Yeah. Like like there's something kind [01:24:06] of quite fundamental about like chaos [01:24:08] that is it's not just like hopeless [01:24:10] noise. It's like there's something kind [01:24:11] of useful, right, in in um in chaotic [01:24:14] systems at least at that boundary. Um [01:24:17] >> but yeah, this is just my like think [01:24:19] about as a philosophy. I don't I don't [01:24:20] actually know the math uh well enough to [01:24:22] comment on it. Anyway, if we um go back [01:24:24] to um we'll talk about LMRL in a little [01:24:27] bit because there's some connections [01:24:28] there, but let's just go back to like [01:24:29] the MCTS like what is it doing? It is [01:24:31] not um crucially it is not saying we're [01:24:34] going to uh increase the probability of [01:24:37] winning directly. It's not going to say [01:24:39] like we're going to um upweight all [01:24:41] actions that won and downweight all [01:24:43] actions that didn't win. Yeah. Um, [01:24:45] importantly, what it is doing is saying [01:24:47] for every action we took, we did a [01:24:50] pretty exhaustive search uh on MCTS to [01:24:53] see if we could do better and we're just [01:24:55] going to make every action that we took [01:24:56] better by predict like having the policy [01:24:58] network predict that outcome instead. [01:25:00] And and so this is um a very very nice [01:25:03] idea because you have one supervision [01:25:05] target for every single action. [01:25:06] >> Yeah. [01:25:07] >> So the the variance of your your [01:25:08] learning signal is very low compared to [01:25:11] the alternative naive RL thing. So, so [01:25:13] let's actually consider what let's [01:25:15] consider a very naive algorithm uh uh [01:25:17] that looks a lot more like you know [01:25:19] modern LM RL today where where we do [01:25:21] something like um let's take the winner [01:25:24] of a self-play game and encourage it to [01:25:26] do more of that. Okay. So, uh it's worth [01:25:29] kind of thinking a a little bit about [01:25:30] like, okay, what are some alternatives [01:25:32] that we could do to train self-play [01:25:33] agents instead of MCTS, right? Like, you [01:25:35] know, we use a lot of uh LM style RL [01:25:38] these days. Like, is that relevant? [01:25:39] Could we do that instead? Um so, so [01:25:41] let's think through this a little bit. [01:25:43] Let's suppose we have a very naive [01:25:45] algorithm where we take a league of [01:25:47] agents of different checkpoints and we [01:25:48] play them against each other. And um for [01:25:51] for the games where a single player [01:25:53] wins, uh we're going to reinforce those [01:25:55] actions up and then and then uh and and [01:25:57] train retrain the policy network to [01:25:59] imitate those those guys instead of uh [01:26:01] instead of the MCTS objective. Um so [01:26:05] what ends up happening is let's say you [01:26:08] have a chain of actions that led to a [01:26:12] win and you're you're you have a matchup [01:26:15] between two agents that are basically [01:26:17] the same. So in fact, let's just assume [01:26:20] that like uh you know policy A and [01:26:23] policy B are like evenly matched, right? [01:26:26] So they're true their true win rate is [01:26:28] is like 50%. [01:26:31] Um so let's say you play a 100 games [01:26:38] and then uh each game let's say lasts [01:26:40] you know 300 um moves [01:26:46] and um you're doing some sort of like [01:26:49] evolution strategy or some way to [01:26:51] perturb these things to get them get [01:26:53] them to do different things or maybe you [01:26:54] don't and you just play them against [01:26:55] each other and you see like occasionally [01:26:57] this one might actually have a better [01:26:59] strategy than this one, right? And so so [01:27:00] let's say um you know 51 games um policy [01:27:06] A wins [01:27:09] and then 49 games policy B wins. And [01:27:13] this is just due to random luck or maybe [01:27:14] you perturbed policy A in some way that [01:27:16] let it do this. And just to have a very [01:27:18] very simple model let's pretend that for [01:27:20] u for like uh 49 of the games they [01:27:25] played exactly equally. Um, I'm sorry. [01:27:28] For for 50 of the games, they they [01:27:30] played exactly equally, right? Um, and [01:27:34] on that one game where this one won, it [01:27:36] it played slightly differently. It made [01:27:38] like one critical move that like, you [01:27:40] know, normally it would have done [01:27:41] differently, but due to some exploration [01:27:43] or some random noise, it just happened [01:27:44] to make a smarter move than it did [01:27:45] previously. [01:27:46] >> So, you have one supervision signal, [01:27:48] like one true supervision signal for [01:27:50] your policy network. Um, [01:27:54] and then you have uh 99 games times 300 [01:27:58] moves for which uh imitating those [01:28:01] actions gives you exactly the same [01:28:03] policy you had before. [01:28:05] >> And so the um the scale of your variance [01:28:09] is actually very bad because it's like [01:28:10] you only have one label out of this [01:28:12] enormous data set of of actions of [01:28:14] supervision actions where you want [01:28:16] actually sorry let me let me let me [01:28:17] clarify a little bit. Okay, so we were [01:28:20] just talking about how the good move, [01:28:22] the out of distribution move is a small [01:28:24] fraction of all the moves that are [01:28:25] played across all the games on which [01:28:27] you'd want to train. And um this of [01:28:30] course reminds me of how LLMs are [01:28:33] trained with policy gradient methods. Uh [01:28:35] Karpathi was when he was on the podcast [01:28:37] called it like sucking supervision [01:28:39] through a straw. Um and uh and so yeah [01:28:42] it's interesting that this like this [01:28:43] thing you're saying which would be [01:28:44] intractable and prevents you from [01:28:46] actually getting beyond a certain level [01:28:48] in go is just by default how LM are [01:28:50] trained question mark [01:28:51] >> right so um in this case this is not to [01:28:54] say it doesn't work right like if you [01:28:55] imagine increasing the number of games [01:28:57] to like you know millions of samples you [01:29:00] actually can get some meaningful [01:29:01] supervision like samples so long as you [01:29:04] find a way to sort of mask out the [01:29:06] supervision from these guys and then [01:29:08] this is where things start to get pretty [01:29:09] related to RL in terms of advantage and [01:29:12] baselines and so forth. So, so let's um [01:29:14] let's look at the, you know, the [01:29:16] gradient variance of a very naive [01:29:18] approach like this where um I'm just [01:29:20] going to call it like gradient RL and [01:29:23] it's basically the you know sum of [01:29:26] rewards. [01:29:38] Okay, I see what you're saying. [01:29:52] So, so the sum of rewards is the return, [01:29:53] right? So, so like uh in in our naive [01:29:56] setup here, we only have a indicator [01:29:58] variable for the return where either you [01:30:00] won or lost. So [01:30:04] um so in the case where you lost well [01:30:07] you just don't train on your gradient is [01:30:08] zero you don't train on those examples [01:30:10] and when you won you try to predict [01:30:12] those those things right so so you can [01:30:13] think about this setup as a as a special [01:30:16] case of this general formula here [01:30:18] >> um the um the trouble here is that this [01:30:22] is very high variance because when you [01:30:25] multiply these terms out when you take [01:30:27] when you try to compute the variance of [01:30:29] this and so so variance of the gradient [01:30:34] is equal to expectation of [01:30:39] squared minus [01:30:42] and just for simplicity we can pretend [01:30:44] this is like you know on average zero or [01:30:45] something if if you're centering it at [01:30:47] at you know no signal. Um and uh the [01:30:51] variance here basically means that [01:30:52] you're you know taking the square of [01:30:54] this product term and so you end up with [01:30:57] uh a term that kind of grows [01:30:59] quadratically with the with t. So so [01:31:03] variance um when you have a setup like [01:31:06] this this thing acts as a coupling [01:31:08] effect on top of of these terms here. [01:31:12] So, um [01:31:14] um let's actually map this to an LM case [01:31:17] and we can answer like why do LLMs only [01:31:20] do onestep RL instead of a multi-step RL [01:31:23] scenario. [01:31:25] Um in LM you have a decoder that might [01:31:29] you know predict some words like hello [01:31:31] world. [01:31:33] And so in current LMRL they treat this [01:31:36] entire sequence as a single action just [01:31:40] a t and big t is just one right. And so [01:31:44] yes it is true that you know the uh [01:31:46] because of how you know transformers are [01:31:48] formulated uh through the sort of [01:31:50] product of conditional probabilities. Um [01:31:52] we do have uh you know probability of [01:31:57] this sequence is equal to the sort of [01:31:59] sum of log probability of the whole [01:32:01] sequence is equal to the sum of the [01:32:03] probabilities of like you know uh [01:32:05] individual tokens right so so in this [01:32:07] case I would um [01:32:09] I would say something like you know log [01:32:11] hell plus log [01:32:15] low [01:32:17] plus log world. So um this is true [01:32:25] and if this term were one then they [01:32:27] would be the same thing. However, um in [01:32:31] sampling things, if you have a reward [01:32:33] term assigned to every specific token, [01:32:37] now you have these interaction effects [01:32:39] between the cross multiplication of [01:32:40] these terms and these terms, [01:32:42] >> right? [01:32:42] >> And so the problem becomes how do you [01:32:45] ascribe the credit associated with every [01:32:48] episode to all these different terms [01:32:50] here? [01:32:50] >> I guess the thing I'm confused on is [01:32:51] what would that even look like to do it [01:32:53] that way in um [01:32:55] >> in LMS? in LLMs because you do you only [01:32:58] do get a reward at the end of the [01:32:59] episode. So [01:33:01] >> you could imagine a reward that says [01:33:03] like I'm gonna give you some process [01:33:05] supervision. Yeah. [01:33:06] >> Uh where you get a reward for each of [01:33:08] these actions on every step. [01:33:11] >> Okay. So you're saying if instead of [01:33:13] doing it that way where you um well I [01:33:16] guess the way you've written it, it [01:33:17] would be a sum at the end anyways. So it [01:33:18] would they wouldn't have to be [01:33:20] multiplied. But you're saying instead of [01:33:22] doing it that way, you would just add up [01:33:23] this process rewards at the end and then [01:33:26] treat that as one single reward signal. [01:33:27] >> Correct. For one single log action. [01:33:29] >> But um isn't that how it's written to [01:33:32] begin with anyways? Like the sum of the [01:33:34] uh rewards. [01:33:36] >> So So the the thing that's a little bit [01:33:37] hidden here in the math is that we're [01:33:39] assuming that when you decompose the [01:33:40] problem to a multi-step problem that [01:33:42] you're now introducing kind of [01:33:43] correlations between your actions [01:33:45] through the computation of this guy. And [01:33:48] uh so if you separate these things out [01:33:50] then there will be um this this will [01:33:52] magnify the variance of of this one [01:33:55] >> right [01:33:55] >> so in in the case where you don't [01:33:57] separate it out if you just have t= [01:33:59] equals 1 you just have a single estimate [01:34:01] of log prop and a single estimate of [01:34:04] reward. [01:34:05] >> Now um there are this this term still [01:34:08] shows up in L. So in LM it looks a [01:34:10] little bit more like the naive [01:34:12] reinforced estimator looks a bit like [01:34:14] return of the single action plus uh [01:34:18] times you know [01:34:25] it looks kind of like this. This is sort [01:34:27] of the very uh basic form here but this [01:34:30] is still a contributor to variance. Um, [01:34:32] so you want to make sure that like you [01:34:34] don't uh similar to how in this case we [01:34:36] were training on a lot of neutral [01:34:38] labels. You want to make sure that [01:34:40] you're subtract you're sort of [01:34:41] penalizing the labels that don't help [01:34:44] and only rewarding the ones that [01:34:46] actually make you better. Right? [01:34:47] >> Right. So intuitively the analogy here [01:34:48] is like can we find a a term in our [01:34:51] training objective such that it's [01:34:53] actually kind of discouraged from doing [01:34:55] this or you know these don't have any [01:34:57] effect on the gradient and this has an [01:34:59] effect on the gradient. [01:35:00] >> Right? I I guess if you applied that [01:35:02] there, the only thing you could do is [01:35:04] eliminate [01:35:06] 49 of the games. So at least the way you [01:35:10] have earned there, you would be um 51 [01:35:13] times actually the optimal case is to [01:35:15] pull out discard all of these moves and [01:35:18] only get a gradient on that single move [01:35:21] that you got better. [01:35:22] >> Yeah. H but how would you do that? [01:35:24] >> Right. So this is a a pretty tricky [01:35:26] problem in practice. Um and so this is [01:35:28] where advantage estimation happens in [01:35:30] reinforcement learning. So you want to [01:35:33] subtract [01:35:34] um you know a term from [01:35:39] from um [01:35:41] from your your multiplier instead of an [01:35:44] indicator function of like one and zero. [01:35:46] You want something that kind of behaves [01:35:48] like a zero for all of these guys [01:35:50] >> and then a one for all these ones. [01:35:52] >> Yeah. But you so you could do that if [01:35:53] they're um if you can say, "Hey, I won [01:35:56] in this game, so this is slightly above [01:35:58] baseline performance." [01:36:00] >> Well, you won on a lot of games. Uh [01:36:02] >> exactly. [01:36:02] >> But but you don't know which ones let [01:36:04] you win because they were truly better [01:36:06] versus winning on access. [01:36:07] >> How would you design a baseline where [01:36:09] it's truly better? [01:36:10] >> Yeah. So this is where in RL people use [01:36:13] things like TDarning to better [01:36:15] approximate the quality function, the Q [01:36:17] that we mentioned earlier. So you can [01:36:19] try to subtract that from your um your [01:36:22] your your return. So so ideally what you [01:36:24] really want to do is in RL you want to [01:36:27] um push up the actions that make you [01:36:30] better than the average and um push down [01:36:33] the actions that make you worse than the [01:36:35] average. And they call this advantage. [01:36:37] There are multiple ways to compute it. I [01:36:39] highly recommend John Schulman's general [01:36:41] advantage estimation paper as like a [01:36:43] good, you know, treatment on how to um [01:36:45] how to like think about various ways to [01:36:47] compute it. But the at the end of the [01:36:49] day you know um you you want to reduce [01:36:51] variance by trying to make this smaller [01:36:53] and so it doesn't magnify the variance. [01:36:56] >> So but um this requires you to have a [01:37:00] very good estimate of what average [01:37:01] performance from a state would look [01:37:02] like. And this is this gets gets us back [01:37:05] to the value function thing we're [01:37:06] talking about earlier, [01:37:07] >> right? And and so this uh keep in mind [01:37:09] that in this case this model free RL [01:37:11] setting uh is trying to solve a credit [01:37:13] assignment problem where you don't know [01:37:15] which actions were actually good and [01:37:16] which ones were bad. [01:37:18] >> Monte Carlo research is doing something [01:37:19] very fundamentally different which is [01:37:20] it's not trying to do credit assignment [01:37:22] on wins. It's trying to improve the uh [01:37:27] the label for any given action you took. [01:37:29] And so we can actually think about a [01:37:31] completely different algorithm called [01:37:32] neural fictitious selfplay which was [01:37:34] used uh to great effect in uh systems [01:37:36] like Alphaar and um and OpenAI's Dota. [01:37:40] >> Um so so let me talk a little bit about [01:37:42] how um how you can kind of unify some of [01:37:45] these RL ideas in the model free setting [01:37:47] as well as the selfplay setting. [01:37:49] >> Okay. So what happens if you don't have [01:37:53] the ability to easily search a tree, [01:37:55] right? Like in go it's a perfectly [01:37:57] observable game. you can easily [01:37:59] construct a pretty deep tree that [01:38:00] completely captures the game state in a [01:38:02] game like Starcraft where you don't have [01:38:04] really complete control over the binary. [01:38:06] It's it's a little bit hard to do this [01:38:07] and I'm not even sure if it's a it's a [01:38:09] deterministic game, right? So So that [01:38:11] makes this uh kind of difficult from a [01:38:13] data structures perspective. So um what [01:38:16] is done instead is that the basic idea [01:38:19] of supervising your actions with a [01:38:24] better teacher is still there, right? [01:38:26] So, so if in a given neural fictitious, [01:38:29] so we're going to talk a little bit [01:38:30] about how neural fictitious selfplay [01:38:32] works. [01:38:40] Same idea. We're going to like come up [01:38:42] with better labels for each of the [01:38:44] actions we took just like in MCTS. But [01:38:47] how do we derive the better labels? [01:38:56] In MCTS, we uh perform search to and [01:38:59] assuming we have a good value function, [01:39:01] the search will kind of give us a better [01:39:03] result than our initial guess. In um in [01:39:06] a game where you can't easily simulate a [01:39:08] search process, what they do instead is [01:39:10] train what is known as a best response [01:39:12] policy. [01:39:18] So you fix your opponent. So let's say [01:39:20] you you're you're currently training pi [01:39:23] A against [01:39:25] um a strong opponent pi B. Uh in [01:39:28] Starcraft maybe like you know the are [01:39:30] the zergs and you're playing protos or [01:39:31] something. Um [01:39:34] so you fix your opponent and you treat [01:39:37] this as a classic model free RL [01:39:39] algorithm where your goal is just to [01:39:40] beat this guy. And so here you use your [01:39:43] standard TD learning style tricks or use [01:39:45] PO or any actually like you know model [01:39:48] free RL algorithm to try to hill climb [01:39:49] against winning this player. And so um [01:39:53] you train you train uh basically you you [01:39:55] have a reward function that's like you [01:39:57] know return is like you know [01:40:01] one if wins [01:40:04] against [01:40:06] IB. So this is no longer a self-play [01:40:08] kind of problem, right? This is just [01:40:10] like a fixed opponent and you're just um [01:40:12] solving trying to maximize um a score [01:40:15] against against that and then you know [01:40:16] zero otherwise. And so you're you have a [01:40:20] sort of fixed environment where all you [01:40:21] care about is just beating this guy. And [01:40:24] um once you have a good policy that you [01:40:27] trained with uh you know pick your [01:40:29] favorite model free R algorithm PO or [01:40:31] SAC or you know any kind of mixture of [01:40:34] the or you know uh VMPO or whatever um [01:40:37] you now have a good policy that gives [01:40:38] you a good label for what this one [01:40:40] should do when playing against that [01:40:42] player. And when you train multiple best [01:40:45] response policies you can basically then [01:40:47] distill the RL algorithms into the [01:40:50] labels for a given opponent. So you [01:40:52] might have let's say a best response [01:40:53] policy against pi B and then maybe you [01:40:55] have a colleague of you know um of [01:40:58] opponents like pi B pi C pi D and you're [01:41:02] going to take the best response policy [01:41:04] that you train against each of these [01:41:05] fixed opponents and for this one you're [01:41:08] going to uh supervise them with the [01:41:10] label that this one would provide. So it [01:41:12] is kind of like this is almost like a [01:41:14] proxy for your M MCTS teacher, right? [01:41:16] Instead of MCS teacher, you use a model [01:41:18] free RL algorithm to find the best [01:41:21] search action that you could do to uh to [01:41:23] to kind of beat your opponent. And then [01:41:26] you're finally you're distilling the um [01:41:28] the policy here into what is known as [01:41:31] like a a mix strategy where it's trying [01:41:33] to basically average across all possible [01:41:34] opponents you could play against. And [01:41:36] this is what gives you something that [01:41:37] can do no worse than like you know an [01:41:39] averagely average selected opponent from [01:41:41] the league. And and so this gets around [01:41:43] the problem of having to derive a [01:41:45] teaching signal from MCTS, but it still [01:41:47] fundamentally is about relabeling your [01:41:50] your your states with better actions so [01:41:53] that they improve your policy. [01:41:54] >> And just make sure you understand this [01:41:55] is like if you win against win a game [01:41:58] against this other policy, you sort of [01:42:00] reinforce all the actions. [01:42:01] >> Yes. [01:42:02] >> On that trajectory. [01:42:03] >> Yeah. So here you can use a number of [01:42:05] algorithms like PO VMPO [01:42:08] um you know Q-learning even if you want [01:42:10] like [01:42:11] >> uh the specific algorithm here um can be [01:42:14] you know it's usually a model free thing [01:42:16] because you don't have search but [01:42:17] there's an interesting connection from [01:42:19] MCTS and Q-learning that I I want to you [01:42:21] know bring up so in MCTS you do [01:42:23] something where you have a tree [01:42:28] and through the resolution of your your [01:42:32] value function at the at the leaves of [01:42:33] the tree or you know your approximate [01:42:35] leaves of the tree, you can kind of back [01:42:37] up through through the you know the the [01:42:40] sequence of many sequences and then [01:42:42] obtain some sort of mean value estimate [01:42:44] right your Q is kind of derived from the [01:42:46] average of a bunch of simulations. Um in [01:42:49] model free algorithms [01:42:52] there is often a component uh of [01:42:54] estimating a Q value and so um and Q Q [01:42:58] values are often learned through TD [01:42:59] learning although in PO the the way that [01:43:01] they do advantage estimation is not [01:43:03] necessarily through a Bellman backup but [01:43:05] um but in Q-learning there's this kind [01:43:06] of very cool trick where um you do [01:43:12] you know QSA is backed up as R plus you [01:43:17] know some discount factor times the max [01:43:21] a Q of your next step. So intuitively [01:43:26] how this works is like if you have a MDP [01:43:32] and then this is like you know terminal. [01:43:38] What this is sort of saying is that like [01:43:40] the best action you can take at this uh [01:43:43] state is equal to the reward you take [01:43:46] for you know taking this action plus the [01:43:49] best that you can do at the next state. [01:43:51] So there's a sort of recursive and [01:43:53] dynamic programming property of of MDPs [01:43:56] and you can train neural networks to [01:43:58] basically try to enforce this this con [01:44:00] this uh consistency right so you can say [01:44:02] like well once I know the Q value of [01:44:04] this action I can then use that to kind [01:44:06] of compute something about the Q value [01:44:08] >> so when earlier I was like hey why are [01:44:10] we training a policy why don't we just [01:44:11] train the value alone that that is what [01:44:13] this is [01:44:15] >> um this is a algorithm for recovering [01:44:18] value estimates of intermediate steps [01:44:20] when you don't have the ability to do [01:44:22] forward search. [01:44:23] >> So you must collect a trajectory first [01:44:25] of like n steps before you're able to do [01:44:28] this trick. [01:44:30] >> Um but the intuition is kind of the same [01:44:31] which is that like knowing something [01:44:33] about the Q value here [01:44:35] >> can tell you something about the Q value [01:44:36] here and indeed you can recover a policy [01:44:38] from a Q value right. So so the um you [01:44:40] don't need to explicitly model the [01:44:42] policy distribution you can actually [01:44:44] recover the policy distribution by doing [01:44:46] argmax over your um your Q values. Yeah. [01:44:49] So, so Q uh Q-learning or um you know [01:44:51] this kind of like uh approximate dynamic [01:44:53] programming kind of propagates what you [01:44:56] know about the future cues backward like [01:44:58] this, right? And you can see that [01:45:00] there's a sort of similar structure that [01:45:01] goes on here where uh in in this case [01:45:03] you're planning over trajectories your [01:45:06] agent hasn't actually been to yet. [01:45:07] Whereas in this case you're planning [01:45:09] over trajectories your your agent has [01:45:10] visited. [01:45:11] >> Um so so importantly why does Q-learning [01:45:14] you know why was Q-learning a big deal? [01:45:16] Right? like it's because historically we [01:45:18] just haven't had the ability to do [01:45:20] search on fairly high dimensional [01:45:22] problems like robotics or whatever. So [01:45:24] for a long time we kind of make the [01:45:25] assumption that like okay well if we [01:45:27] can't model the dynamics with like a [01:45:29] world model or something we're going to [01:45:30] instead just collect trajectories [01:45:32] >> and then plan with respect to the only [01:45:34] number that really matters which is [01:45:35] reward. [01:45:36] >> Okay so this is very interesting and [01:45:37] then to unify this with our discussion [01:45:39] of LLMs. So with LLMs you're doing [01:45:42] something you don't have Q values but [01:45:44] you're doing this sort of backwards [01:45:46] learning where hey let's find the [01:45:48] trajectories which pass some unit test [01:45:51] in some coding environment and then [01:45:52] let's reinforce those trajectories [01:45:54] >> and then there's a huge difference [01:45:55] between that and this forward approach [01:45:58] with MCTS and the reason you can do MCTS [01:46:01] and it's much more preferable to do MCTS [01:46:03] because you can do it per move uh and [01:46:05] make each move better rather than having [01:46:06] to learn per trajectory um and hope you [01:46:09] know as Karpathi hope to learn this like [01:46:11] straw. [01:46:12] >> Yeah. To get the supervision through a [01:46:13] straw. Uh basically just upgrade all the [01:46:15] tokens in a trajectory that might or [01:46:17] might not have been relevant to getting [01:46:18] the answer right. The reason you can do [01:46:20] this much more sort of sample efficient [01:46:22] uh much more favorable thing with go is [01:46:25] that because MCTS works in go you [01:46:29] basically know that hey if I just do [01:46:31] search locally here and this search is [01:46:34] sort of truncated at the end by this [01:46:36] value function that uh works even even [01:46:39] if I haven't unfolded my whole [01:46:41] trajectory I can just say this is my new [01:46:44] policy um and I can improve in a more [01:46:47] iterative like local way rather than [01:46:50] having to [01:46:52] h having to unfold all these [01:46:53] trajectories. [01:46:54] >> So there was some research I think from [01:46:57] Google in 2030 2023 2024 where they did [01:47:00] try to apply tree structures to [01:47:02] reasoning. [01:47:03] >> Yeah. [01:47:03] >> And I think it's you know the jury is [01:47:05] still out as to whether this can ever [01:47:07] work. So I I would say like uh it we [01:47:11] probably will see like you know [01:47:13] revisiting of this idea of forward [01:47:15] search um in in the future. But there's [01:47:19] two things that make MCTS very simple [01:47:21] for go which is that value estimation is [01:47:23] kind of concrete and you can determine [01:47:25] it for real and then you can kind of uh [01:47:28] uh sort of use it to truncate depth as [01:47:29] you said and then the uh breath is also [01:47:32] uh determined [01:47:33] >> and what's kind of critical is that the [01:47:35] action selection algorithm where you [01:47:36] iteratively visit and grow the tree is [01:47:40] um well suited for the size of problem [01:47:43] that go is and the depth of the problem [01:47:45] but for something like LLM reasoning um [01:47:47] you know puck might not be a good enough [01:47:49] heristic. It might be too greedy with [01:47:51] local tokens and it might do something [01:47:53] like oh only give you, you know, uh sort [01:47:55] of obvious thoughts that are correct but [01:47:58] not really solve your final problem. [01:47:59] >> So I I would say the jury is probably [01:48:01] still out on how like what the final [01:48:03] instantiation of reasoning for LM would [01:48:06] look like. And I wouldn't rule out that [01:48:07] like this stuff could, you know, come [01:48:08] back, but it's a bit hard. [01:48:09] >> Don't LM sort of natively learn to do [01:48:13] MCTS where they'll try an approach and [01:48:15] be like, "Oh, that doesn't work. let's [01:48:17] back up, let's try this other thing and [01:48:19] then go in the direction that proves to [01:48:21] be more fruitful. [01:48:22] >> Uh yeah, c certainly I think the LLM's [01:48:24] managed to do something that looks like [01:48:26] real human reasoning without having to [01:48:27] do an explicit tree structure. [01:48:29] >> Um [01:48:30] >> that being said, I think the idea of [01:48:31] doing forward search and simulation to [01:48:34] get a better sense of what is valuable [01:48:37] might make a comeback. um even though [01:48:39] not exactly in the same instantiation as [01:48:41] as uh Alfa [01:48:42] >> but u just to make sure I understand the [01:48:45] crux of it like the the breadth from the [01:48:48] number of legal actions being wider and [01:48:51] the depth from being able to not being [01:48:53] able to train a value function as easily [01:48:55] because [01:48:56] >> so here's an example where LM's break [01:48:58] down the cuck rule involves you know [01:49:00] square root of n over 1 + n a [01:49:03] >> in an LLM like you're most likely never [01:49:06] going to sample the same uh child more [01:49:09] than once, right? So, if you have, let's [01:49:10] say, multi-steps of thinking, um because [01:49:12] language is so broad and open-ended, [01:49:15] it's uh a sort of uh discrete set of [01:49:18] actions is not really an appropriate [01:49:19] choice for an LLM. Uh even though [01:49:21] they're discrete tokens, um it's just [01:49:23] such a large number that this type of [01:49:25] exploration heristic is probably not the [01:49:27] right thing to do to guide how to search [01:49:29] down a tree, [01:49:30] >> right? But I I guess the crux comes down [01:49:32] to the fact that in go you know that the [01:49:36] MCTS is almost certainly better than [01:49:38] your current policy even though you [01:49:39] haven't gotten even though you haven't [01:49:41] explored the end of any trajectory. [01:49:43] >> Correct. [01:49:43] >> And then in u in normal reasoning for [01:49:46] LLMs or robotics there's no way to just [01:49:49] locally evaluate and improve your next [01:49:52] move in a way that doesn't result in in [01:49:55] a way that's independent of actually [01:49:56] like solving the problem. [01:49:58] >> Uh no way is a strong word. I think lots [01:50:00] of people have thought about how to try [01:50:01] to apply MCTS or its kind of successors [01:50:04] like new zero to continuous control [01:50:06] spaces and I'm sure you know very cool [01:50:08] research work is still ongoing to try to [01:50:09] crack that problem. Um but yes, the the [01:50:12] seeming challenge right now is that like [01:50:15] most problems in much higher uh [01:50:17] dimensional um you know action spaces or [01:50:20] something that's combinatorally much [01:50:22] bigger like language they they don't [01:50:24] seem as amendable to the kind of [01:50:26] discrete action selection heristics as [01:50:28] well as uh kind of game evaluation type [01:50:30] stuff that uh go does. Um but that's not [01:50:33] to say the idea of like you know [01:50:35] thinking into the future along multiple [01:50:37] parallel tracks might not give you some [01:50:39] information about like which way to [01:50:41] search right like if you think about [01:50:42] mathematics I think mathematics often [01:50:44] occupies a little bit more of like a [01:50:46] logical search kind of procedure where [01:50:48] you kind of can back up you can kind of [01:50:50] see like which paths seem good or not [01:50:52] there's a more of a rigid structure [01:50:53] there whereas maybe like in a uh you [01:50:55] know business negotiation or something [01:50:58] um it's less of a tree and maybe you [01:51:00] know something a bit [01:51:02] Okay, so we're now seated so I can ask [01:51:04] you some more questions about AlphaGo [01:51:06] and about AI research more generally. [01:51:08] Um, in 2021, Andy Jones had a paper [01:51:12] called scaling scaling loss reward [01:51:13] games. And um, he basically anticipated [01:51:16] inference compute or inference scaling [01:51:18] by showing that you can trade off test [01:51:19] time compute and uh, training compute. [01:51:22] That is to say that you can spend more [01:51:23] compute on the forward the searching [01:51:26] through the MCTS. Um and if you do that [01:51:28] you can get the equivalent performance [01:51:30] as having spent more time training the [01:51:32] model. Um and so if you you know if you [01:51:35] see this pattern you might think okay [01:51:36] well with LLMs you might do something [01:51:38] like that in the future and in fact [01:51:39] that's what had ended up happening. Okay [01:51:41] so what is a kind of fun exploration one [01:51:44] could do now to explore other axes of [01:51:47] scaling in toy settings which will be [01:51:51] important to understanding what AI [01:51:53] development might be like in a few [01:51:54] years. [01:51:54] >> Sure. Yeah. Um I think that indeed test [01:51:57] time scaling and uh reasoning and how it [01:52:01] interacts with model size are quite [01:52:04] profound when it comes to like how much [01:52:07] uh needs to be actually done as explicit [01:52:09] search versus how much can be packed [01:52:11] into the forward pass of a neural [01:52:12] network right and and um how does a [01:52:15] forward pass of a neural network sort of [01:52:17] learn how to do something that should be [01:52:18] a sort of sequential and you know [01:52:20] recursive step that's quite interesting [01:52:22] um so the yeah the Andy Jones scaling [01:52:24] laws for board games paper is quite [01:52:25] Cool. There's another really nice result [01:52:27] from that paper where they where he [01:52:28] showed that um not only can you predict [01:52:31] scaling loss of like you know the sort [01:52:33] of LM variety where um as you increase [01:52:36] parameters you can decrease the amount [01:52:37] of compute for search or vice versa. Um [01:52:40] >> he also showed that you can actually [01:52:42] predict uh how much compute is needed to [01:52:44] solve a uh larger version of the board [01:52:47] game. uh for example and and so with go [01:52:49] you know which can scale from you know [01:52:51] uh 3x3 to infinitely sized you know uh [01:52:54] go board you might actually be able to [01:52:56] sort of revisit this question and try to [01:52:57] reproduce uh whether this shows up u you [01:53:00] know I actually started this project [01:53:01] with this sort of a motivation that like [01:53:03] does the bitter lesson or does our [01:53:04] knowledge of scaling laws allow us to [01:53:06] kind of execute a lot better on a sort [01:53:08] of compute optimal gobot and can we can [01:53:10] we kind of build a strong gobot without [01:53:12] all the katago tricks right just just by [01:53:14] really focusing on the bitter lesson and [01:53:15] scaling laws um I have not been [01:53:17] successful so far, but I think it's it's [01:53:19] sort of a a fact that like usually when [01:53:21] you want scaling laws to work, you want [01:53:23] to be in the regime where um the the [01:53:26] recipe already works and the data sets [01:53:27] are good rather than trying to kind of [01:53:29] figure out how to do scaling while also [01:53:31] trying to figure out what the the right [01:53:32] data set are. Okay. So, so this is like [01:53:34] the scientific understanding component [01:53:35] in research often follows a step where [01:53:38] you get something to work first and then [01:53:40] you use that um system to collect data [01:53:43] that then helps you build a mental model [01:53:45] of how things work such as scaling laws, [01:53:47] right? And and so usually actually if [01:53:48] you want to build a strong gobot using [01:53:49] scaling laws, you you actually have to [01:53:51] make a strong go first and then use the [01:53:53] scaling laws to kind of extrapolate a [01:53:55] bit farther into the future. [01:53:56] >> Say more just so I understand. First of [01:53:58] all, you're saying scaling laws did not [01:54:00] work or you could not there was no [01:54:01] scaling loss pattern that you could see [01:54:03] in your robot. [01:54:04] >> Yeah. So, um a mistake I made initially [01:54:06] when I had some bugs around how MCTS [01:54:08] labeling was working was I would um I [01:54:11] would collect a bunch of data with an [01:54:12] expert policy and then treat it as a [01:54:14] supervised learning problem and try to [01:54:15] identify scaling laws with uh expert [01:54:18] data sets. um you can indeed plot things [01:54:21] that look kind of like this. But if [01:54:23] you're in a regime where you know your [01:54:24] policy is not working well, you might be [01:54:26] just studying scaling laws on like bad [01:54:28] data, right? So so just like one [01:54:30] important implementation detail is that [01:54:31] if you want to study a scaling laws [01:54:33] problem, you kind of have to have a [01:54:34] problem for which like the data is good, [01:54:36] the architecture is good and there's no [01:54:38] bugs and then like you you you solve it [01:54:40] there. um x anti I wasn't able to like [01:54:43] apply scaling laws to direct what what [01:54:45] to look on look look at um until you [01:54:48] know I had the rest of the system [01:54:49] working and and this sounds obvious like [01:54:51] to to researchers of course you want to [01:54:53] have like a working bug-free system [01:54:54] before you study scaling but uh just as [01:54:56] a sort of advice for practitioners on [01:54:58] like where I actually tripped up when I [01:54:59] started this project was you don't [01:55:01] necessarily want to kind of jump into [01:55:02] the science of studying your man-made [01:55:04] artifact before your man-made artifact [01:55:06] is like interesting enough to be studied [01:55:08] >> speaking of compute so if You can look [01:55:11] at these charts of compute used to train [01:55:14] the best AI model in the world over time [01:55:17] going back 10 years and it's a very [01:55:20] smooth line in log space uh that is [01:55:22] exponentially growing year-over-year uh [01:55:25] except there's this huge aberration and [01:55:27] that aberration is off like a zero uh [01:55:30] which is trained on way more compute [01:55:31] than any other AI model at the time. It [01:55:34] was like uh three E23 flops and they're [01:55:37] sort of comparable to like a Frontier [01:55:39] LLM. I mean, orders of magnitude off, [01:55:42] but still. Um and so yeah, the question [01:55:45] is especially with you being able to get [01:55:47] something off in your own [01:55:49] >> uh I got a donation from Prime [01:55:50] Intellect. Okay. [01:55:51] >> For like about 10K and then I spent um I [01:55:54] spent maybe the first 4K doing um kind [01:55:57] of exploratory research and then uh [01:55:59] about 3K on the kind of final run. Yeah. [01:56:02] Um and then uh some some of it remaining [01:56:04] for serving the model for [01:56:05] >> cool. [01:56:06] >> Yeah. Is your sense that they were just [01:56:08] did a badge up train for it if you can [01:56:09] do it in 10k now? [01:56:10] >> Uh the compute required to be the first [01:56:12] to do something is always like much [01:56:14] larger than the compute it takes to [01:56:16] catch up. And it's the same story [01:56:17] playing out in LMS, right? Like once [01:56:18] someone else has done it, um you could [01:56:20] use tricks like distillation. You could [01:56:22] use um uh all sorts of like kind of [01:56:25] crutches to kind of bootstrap your way [01:56:26] to success. So with my own bot that I'm [01:56:28] I'veo hosted online um I actually used [01:56:31] uh sort of best response training [01:56:32] against the Kadigo models to kind of get [01:56:34] a strong level performance and um you [01:56:37] know as a time of recording I'm I'm [01:56:38] validating whether this can be uh I can [01:56:40] kind of do that first step which is to [01:56:42] do the tabular raza play um but [01:56:44] importantly for research you often want [01:56:46] to start from a good init right so so [01:56:47] the kind of simple thing I did first was [01:56:48] train best response agents against [01:56:50] kadago [01:56:51] >> um alpha zero team uh they did not have [01:56:53] any policy that they could train against [01:56:55] right because they were trying to do [01:56:57] So um and being the first to do it means [01:56:59] that you're prioritizing the thing [01:57:01] rather than like let's say the most [01:57:02] compute efficient uh possible [01:57:04] implementation. So this actually plays [01:57:06] out in robotics as well. Like if you [01:57:08] look at the kind of frontier of large [01:57:10] models trained for robotics um there the [01:57:12] scatter plot is all over the place and [01:57:13] there isn't a very clean line the way [01:57:15] that there is for frontier LMS. And and [01:57:17] that is because the folks training these [01:57:19] models often are not, you know, at the [01:57:22] scale where every flop counts and they [01:57:25] need to like kind of squeeze out the [01:57:26] performance of every single flop as the [01:57:28] dominating decision deciding factor in [01:57:30] pre-training, right? Instead, their [01:57:31] focus is more like we want a certain [01:57:33] capability to to show up. So we optimize [01:57:35] the uh training setup to kind of make it [01:57:37] easy to derive that capability. And once [01:57:39] you have that capability, well um [01:57:41] invariably if you scale up the compute, [01:57:43] you are forced to kind of make it [01:57:44] compute efficient because this is like [01:57:45] hundreds of millions of dollars we're [01:57:47] talking about. But um but in the past [01:57:50] when when compute for experiments was [01:57:52] kind of more plentiful or you know not [01:57:54] not uh not um accounted in a way that [01:57:57] the researcher was really responsible [01:57:58] for then you kind of end up with people [01:57:59] optimizing for things besides kind of [01:58:01] being on the compute optimal prio [01:58:02] frontier. [01:58:03] >> I see like speed or something. [01:58:04] >> Yeah. Like time to result or just [01:58:06] getting it to work. I think the first [01:58:07] AlphaGo like probably they had lots of [01:58:09] compute and they didn't need to be um [01:58:11] they didn't need to worry too much about [01:58:13] making it the most comput optimal thing. [01:58:14] >> And how much of the improvements to [01:58:16] compute efficiency are methods that did [01:58:18] not exist as of 2017 versus things which [01:58:21] you they could have done in 2017 but um [01:58:23] >> yeah great question. So, so going into [01:58:25] this project, I kind of knew in the back [01:58:27] of my mind that like things always get [01:58:29] easier to do over time and I wanted to [01:58:30] see like where where is go at given that [01:58:32] like it didn't seem like there has been [01:58:34] any major open- source you know uh [01:58:36] strong bot after Katago in 2020. [01:58:39] >> Uh and then you know reading the Katigo [01:58:40] paper there's a lot of clever ideas. I [01:58:42] was kind of wondering like okay uh let's [01:58:44] look let's see if the bitter lesson has [01:58:45] happened where like a lot of these kind [01:58:47] of tricks just sort of go away because [01:58:48] the Nvidia made faster GPUs right and [01:58:50] and so uh roughly where are we on that? [01:58:53] So um again this is not a peer-reviewed [01:58:55] claim. So this is just my preliminary um [01:58:57] you know vibe guess on like what I've [01:58:59] seen based on my own experiments. But it [01:59:01] seems like um you know architecture [01:59:05] choices don't matter that much. You know [01:59:06] transformer versus ResNet. [01:59:08] >> We're we're at the sort of speed of GPU [01:59:10] where the size of the model is not so [01:59:12] big that this really matters. Um [01:59:15] um you can actually simplify the setup [01:59:17] quite a lot. So instead of doing a [01:59:19] distributed asynchronous RL setup with [01:59:21] replay buffers and pushers and [01:59:22] collectors, you can kind of do a a dumb [01:59:24] synchronous thing where you like collect [01:59:26] you just train a supervised learning [01:59:28] model and then you collect again and and [01:59:30] so that there's like opportunities to [01:59:31] simplify infrastructure. [01:59:32] >> Um NVIDIA GPUs have indeed got faster. [01:59:34] So whereas Kadigo was trained on V100s, [01:59:37] you can train on like half the number of [01:59:39] you know desktop Blackwell GPUs and it [01:59:41] still works. Um and um some of the kind [01:59:46] of auxiliary supervision objectives that [01:59:48] Katiggo developed aren't really [01:59:49] necessary if you have a strong [01:59:51] initialization, [01:59:52] >> right? So so if you're initializing [01:59:53] against you know best uh response [01:59:55] training against Kadigo itself, then [01:59:57] your own model actually needs none of [01:59:58] the tricks that Katiggo needs. [02:00:00] >> So so then the core thing is like how [02:00:02] can you get as quickly as possible to [02:00:03] some strong opponents and um that [02:00:06] matters a lot more than the specific [02:00:07] architectural innovations but there are [02:00:09] still some nice comput multipliers. So I [02:00:11] found that training on 9x9 boards was [02:00:13] very nice for resolving endgame value [02:00:15] functions and then like if you can code [02:00:17] train that on a architecture that can [02:00:18] transfer between 9 by9 and 19 by9 then [02:00:21] you can really cut down the warm start [02:00:22] time to learn that from scratch. I think [02:00:24] uh Alph Go Zero their plot was um first [02:00:27] 30 hours or so are spent basically [02:00:29] catching up to the supervised learning [02:00:30] baseline. Um and you can cut down that [02:00:34] time a lot by kind of pre-training on a [02:00:36] small board and then and then like you [02:00:37] know warm starting that into your you [02:00:39] know 19 by9 board play. Uh there were [02:00:42] some other stuff like you know varying [02:00:44] the number of sims between episodes. [02:00:46] this turns out to be not that sensitive [02:00:48] actually like you can kind of you know [02:00:50] fix it or increase it doesn't matter too [02:00:53] much um but so anyways it's kind of just [02:00:55] nice from a scientific perspective just [02:00:56] revisiting like an old paper and seeing [02:00:58] like what really matters [02:00:59] >> this is sort of a tangential question [02:01:00] but why is it okay to have a buffer in [02:01:02] alpha go because every time I talk to [02:01:04] any researcher they're telling me about [02:01:05] how bad it is to be off policy [02:01:07] >> but then the way a naive implementation [02:01:09] of alpha go zero would work is that most [02:01:12] of the moves in a given backward step or [02:01:16] in a batch of backward steps um would be [02:01:20] not not among the ones that were made by [02:01:24] the most recently trained model. So why [02:01:25] is that? Okay, [02:01:26] >> great question. Yeah, and this this gets [02:01:28] into the sort of fundamental off policy [02:01:30] versus on policy uh reinforcement [02:01:32] learning kind of questions. So uh as you [02:01:35] recall in MCTS you take actions that you [02:01:38] took and you reabel them to uh to take [02:01:42] different actions on the same states, [02:01:43] right? So, so the off policy part here [02:01:45] comes where um what if you're relabeling [02:01:48] states that your new policy would never [02:01:50] visit, right? Like what's the point? [02:01:51] You're kind of wasting capacity. And in [02:01:53] the extreme limit, imagine your [02:01:54] distribution of states in your training [02:01:56] buffer are all states that you would [02:01:57] never visit. Then you're basically [02:01:59] supervising them uh to uh take good [02:02:03] actions on states you would never [02:02:04] achieve and therefore your policy can [02:02:06] get really bad. Right? So this is where [02:02:07] off policy can really hurt um uh [02:02:10] AlphaGo. Um, however, if you interpret [02:02:13] this sort of from like the dagger [02:02:14] perspective, which is basically saying [02:02:16] like a way to kind of correct yourself [02:02:18] back to the optimal trajectory, uh, [02:02:19] given some some data, what what you kind [02:02:22] of want in an algorithm like this is to [02:02:24] have mostly states that you would visit, [02:02:27] but then you have a small percentage or [02:02:29] maybe a reasonable percentage of states [02:02:31] in this kind of highdimensional tube [02:02:33] around your optimal, [02:02:34] >> you know, trajectories. And any of those [02:02:36] states are given a supervision target to [02:02:38] kind of uh sort of funnel you back into [02:02:41] your optimal trajectory. Uh so maybe I [02:02:43] can just draw quickly here. [02:02:45] >> Great. So in [02:02:47] um sort of a Dagger style setup, what [02:02:49] your kind of optimal training data [02:02:51] distribution is is that here is your [02:02:53] optimal states and actions. So this is [02:02:56] like you know you want to be in this [02:02:57] state, you want to be in this state, you [02:02:59] want to be in this state and then you [02:03:01] win here. Um and then these are your [02:03:03] optimal policy actions. So these are the [02:03:05] these are the things that you definitely [02:03:07] want to train on. But to make it robust [02:03:09] to disturbances, um you want to make [02:03:11] sure that if you happen to drift off [02:03:13] into some other states, [02:03:15] >> you can kind of funnel yourself back [02:03:17] into [02:03:17] >> But why isn't this a fully general [02:03:18] argument for off policy training? [02:03:20] >> This is actually why you want to do off [02:03:22] policy training sometimes is that like [02:03:24] you you don't want to have a compounding [02:03:26] error where if you make a mistake, you [02:03:28] don't have the data of how to return [02:03:29] back to your optimal distribution. [02:03:30] >> Yeah. And so, um, optimal control does [02:03:33] not really say too much about like, uh, [02:03:36] you know, how to, uh, you know, not [02:03:39] accidentally get here because it's sort [02:03:40] of making the assumption that like once [02:03:42] you learn the policy, you're going to [02:03:43] get here. But in applications like [02:03:44] robotics, right, like like I don't know, [02:03:47] uh, a gust of wind blows you slightly [02:03:48] off and then now you need to like [02:03:50] correct, right? Um, or the friction on [02:03:51] one of your tires is kind of a little [02:03:53] bit like lower than the other wheel and [02:03:54] then now your car is drifting and you [02:03:56] got to got to like correct it. So, so [02:03:57] these kind of things in in like more [02:03:59] real environments often happen where [02:04:01] like um actually there's a funny uh [02:04:03] quote about chess and also go is like [02:04:06] the problem with uh the problem with go [02:04:08] and chess is that the other player is [02:04:09] always trying to do some right? [02:04:11] Like uh so so like you know things can [02:04:12] kind of drift off. Yeah. [02:04:14] >> And you always want to be able to [02:04:15] correct uh back to your back to your [02:04:17] winning condition. So, so your replay [02:04:20] buffer really should have like your, you [02:04:22] know, the states that your policy would [02:04:23] visit plus some distribution of states [02:04:26] that you might drift to and then how to [02:04:28] return back to your optimal states. [02:04:30] >> Yeah. Now, if you take this to the [02:04:32] extreme and you say like, well, let's uh [02:04:35] we don't have any of this data [02:04:41] and we're going to just like be labeling [02:04:43] with MCTS [02:04:45] um you know, states that are so far away [02:04:48] from our optimal behavior like this this [02:04:51] bag of states over here. Well, like now, [02:04:54] yeah, I mean, like each of them gets a [02:04:56] MCTS label [02:04:58] >> and your policy learns how to do take [02:04:59] sort of the best possible action here, [02:05:01] but you never get here. So, like you're [02:05:03] training your model on states you would [02:05:04] never reach. Uh like like this is this [02:05:07] is not there. So then this is a problem, [02:05:09] right? And and this is where off policy [02:05:10] can really hurt. [02:05:11] >> Yeah. Um so actually as as part of this [02:05:14] project I did try an experiment where I [02:05:16] took a bunch of trajectories and to try [02:05:18] to saturate the GPU as much as possible [02:05:20] what I did was I took uh you know random [02:05:23] states from the data set [02:05:25] >> and uh reran MCTS on just those states. [02:05:28] Right? So instead of playing a whole [02:05:29] game where I'm doing MCTS on every move, [02:05:31] I just ignore the sort of causality of [02:05:33] moves and just pick random board states [02:05:36] and I just label those with my current [02:05:37] network. and uh and I might revisit old [02:05:39] states that I've labeled before and [02:05:40] reabel them again with my current [02:05:42] network. Right? And so in practice, this [02:05:44] actually does work. You can actually say [02:05:45] like let's take some states that are [02:05:47] reasonable and constantly be relabeling [02:05:50] them um in uh in um while we're [02:05:53] training. and and so this actually [02:05:55] starts to converge on a very robotics-l [02:05:56] like setup which is very common which is [02:05:58] you have your your data set of [02:05:59] trajectories. Um and then you have [02:06:02] something like a replay buffer pusher [02:06:10] and these are off policy offline [02:06:12] trajectories, right? Your replay buffer [02:06:13] pusher pushes transition tpples [02:06:19] to to to the replay buffer [02:06:24] and then you have some job that's kind [02:06:27] of continuously [02:06:29] um replplanning [02:06:33] what the best action you should have [02:06:35] done instead of taking this action is [02:06:36] right and so in robotics it's actually [02:06:38] very common to use a a uh the sort of [02:06:40] minimized TD error so like your bellman [02:06:43] updater [02:06:45] um constantly is pulling things from [02:06:47] here and trying to satisfy, you know, [02:06:49] the QSA. [02:06:59] So, so um and then and then from here [02:07:01] you have your trainer which is trying to [02:07:04] fit the S to A or or or uh um fit the [02:07:10] you know Q to the Q target. So, so here [02:07:12] you can think about this as a sort of [02:07:14] planner, right? You vi revisit old um [02:07:16] states that you've been to and you take [02:07:18] your current model and you rethink like [02:07:20] what could I have done better if I [02:07:22] visited this and um and so this is [02:07:25] actually how like kind of off policy [02:07:27] robotic learning systems are usually [02:07:28] trained. Um these days there's a sort of [02:07:30] simpler recipe but but like you know in [02:07:32] the Google QT op days we kind of did did [02:07:34] things like this. [02:07:35] >> So what is the trainer? Oh yeah, the [02:07:37] trainer is uh you try to you try to [02:07:39] minimize [02:07:41] uh QSA [02:07:43] and Q target. [02:07:45] >> So can you explain the whole setup again [02:07:47] like at a high level? [02:07:48] >> Yeah. So you have your off policy data [02:07:50] that came from various policies. [02:07:52] >> You're constantly pushing uh transitions [02:07:54] that you saw before to a replay buffer. [02:07:56] Yeah. And then you've got this thing [02:07:57] called a a Bellman updater which [02:07:59] basically replans instead of this action [02:08:02] what action should I have taken at S to [02:08:04] have a better you know value and and the [02:08:06] way you enforce that is you try to [02:08:07] minimize the TD error. So, so actually [02:08:09] you given this you have S prime, right? [02:08:12] You you compute Q of S prime and you [02:08:15] find the action that should go with S [02:08:17] prime that makes this Q value as high as [02:08:19] possible [02:08:20] >> and then you add that to the reward here [02:08:22] and that gives you your actual target. [02:08:24] Right? So for this current SNA your Q [02:08:27] target is this. [02:08:31] So now you have a now now you send back [02:08:33] the Q target to to this this transition. [02:08:35] So, so with this tpple, you pair with [02:08:37] that a uh a Q target [02:08:41] and then here on the trainer you simply [02:08:43] just use supervised learning and you [02:08:45] minimize your current network's QSA with [02:08:47] its target. [02:08:48] >> Got it. Okay. So, in the background, [02:08:50] you're just like, hey, let me let me [02:08:52] basically think through how valuable [02:08:54] were all these actions actually. [02:08:55] >> Yeah. In a in a more optimal policy [02:08:57] where you're trying to maximize this, [02:08:59] what is the Q target of this transition? [02:09:01] >> It's sort of like basically daydreaming. [02:09:02] >> Exactly. Yeah. You can think about it's [02:09:03] like you're kind of going back in [02:09:05] hindsight and being like like given what [02:09:08] I've seen in the historical buffer um [02:09:10] like was there a better action I could [02:09:12] have taken? Now the connection to go [02:09:14] here that I I tried and it was you know [02:09:16] moderately successful but too complex to [02:09:18] kind of like open source was um you [02:09:20] replace this with like a MCTS reabler [02:09:26] where um instead of doing this kind of [02:09:28] target network computation you uh run [02:09:32] MCTS on your transition right so in in [02:09:35] this case you have uh your state your [02:09:37] action and then whether you won or not [02:09:38] at the game um and actually you can just [02:09:40] toss these two you don't you don't care [02:09:41] about these ones you just take your [02:09:43] state and you just plan MCTS [02:09:46] >> to get your best policy, you know, pi [02:09:52] >> uh on your current network, right? Not [02:09:54] not the network that took this action, [02:09:56] but your current best policy network, [02:10:00] you just rerun your search offline on [02:10:02] these transitions. And um if these are [02:10:04] transitions that your policy can get to, [02:10:06] then this actually acts as a very nice [02:10:08] stabilizing effect. And also the one [02:10:10] other benefit is that you can like kind [02:10:11] of fully saturate your GPU better [02:10:13] because you're not like blocking on the [02:10:16] go game to kind of like give you board [02:10:18] states. You just simply search across [02:10:20] all board states at any depth in [02:10:22] parallel. [02:10:22] >> Yeah. [02:10:23] >> So u and then here the trainer would be [02:10:25] just you know predict the MCTS label as [02:10:26] possible. So, so again like this kind of [02:10:29] works and this is quite relevant in [02:10:30] robotics where you're really you just [02:10:32] have have a lot of offline data and you [02:10:34] can't simulate things like MCTS but um [02:10:36] in practice like it does run into the [02:10:38] problem where you know like if the [02:10:40] current model is looking at states that [02:10:42] it would never reach then it's kind of [02:10:44] wasting capacity and so you have to be a [02:10:46] little bit careful here so um the on [02:10:48] policy thing and also much of RL has [02:10:50] kind of converged to a much more on [02:10:51] policy setup where they don't really try [02:10:53] to like directly train on off policy [02:10:55] data at best they use off policy data as [02:10:57] a way to reduce variance but not [02:10:58] directly influence the objective. [02:11:01] >> Oh sorry why have they convert to that? [02:11:03] >> It's just more stable. [02:11:04] >> Yeah. [02:11:05] >> Yeah. So so like you you might use the [02:11:07] off policy Q as a way to do like you [02:11:09] know advantage computation um like uh [02:11:12] you know Q minus like sum of Q. Yeah, [02:11:16] >> that's kind of like your your or sorry [02:11:18] like you know sum of uh like if there's [02:11:20] n actions and then yeah so so like uh [02:11:25] >> so so like this is your value and then [02:11:26] this is your your kind of current q [02:11:27] value. So your advantage for that action [02:11:29] is like the average value minus your [02:11:31] current one. So so like people can try [02:11:33] to estimate Q in an off policy way and [02:11:35] then like just use advantage here and [02:11:37] then and then the the sort of if there's [02:11:39] a problem in these dynamics the it [02:11:41] doesn't like blow up your loss as much. [02:11:43] Um, and so in robotics, there's a kind [02:11:45] of convergence towards more like a using [02:11:47] off policy data to just shape your [02:11:48] rewards, but not actually be directly [02:11:50] your [02:11:51] >> I'm reminded now of our earlier [02:11:53] conversation of why MCTS is so favorable [02:11:56] as compared to the kind of, you know, [02:11:58] reinforce a policy gradient kind of [02:11:59] thing LLM do. And this might be totally [02:12:02] wrong, but I wrote a blog post a few [02:12:03] months ago about um how RL at least [02:12:05] policy gradient RL is even more uh [02:12:08] inefficient than you might think. And so [02:12:10] the inefficiency one thinks about [02:12:12] naively is the fact that you have to [02:12:13] roll out a whole trajectory in order to [02:12:16] get any learning signal at all. And so [02:12:18] as these trajectories become longer and [02:12:20] longer as an agent has to instead of [02:12:22] just [02:12:24] previously like complete the next word [02:12:26] in the sentence, it has to go instead to [02:12:28] hey do two days worth of work to figure [02:12:30] out even if you even did this project [02:12:32] correctly. the amount of information per [02:12:36] uh flop has been decreasing as you had [02:12:39] to unroll two days worth of thinking in [02:12:40] order to see if you even did something [02:12:41] correctly to like did I implement this [02:12:43] feature. The amount of samples per flop [02:12:45] has been decreasing. But so you can [02:12:48] think of um uh you're trying to maximize [02:12:51] as you're learning bits per flop, right? [02:12:54] Um [02:12:57] and this is you can think of bits of per [02:12:59] flop as [02:13:02] um samples [02:13:04] per flop [02:13:07] times um uh bits per sample. [02:13:14] And what I just mentioned uh a second [02:13:16] ago is that the samples per flop go down [02:13:18] as RL becomes more and more long [02:13:20] horizon. [02:13:21] But um at least this kind of naive RL is [02:13:25] also terrible from a bits per sample [02:13:26] perspective. And here's what I mean. At [02:13:28] least compare it to supervised learning. [02:13:30] So early on in training, let's say you [02:13:32] have a [02:13:34] uh vocabulary size for an LLM that is [02:13:37] 100k long. So there's 100k possible, you [02:13:39] know, tokens that one could answer. And [02:13:41] you have a totally untrained model. and [02:13:43] you have a prompt like [02:13:46] the sky [02:13:48] is [02:13:50] um with supervised learning you what [02:13:54] would happen is that the model would [02:13:55] have some probability distribution over [02:13:56] all the things it could say um there's a [02:13:58] label that says actually the term here [02:14:00] is blue and it would figure it would [02:14:02] learn basically for cross entropy loss [02:14:04] exactly how far its distribution is from [02:14:06] correctly saying blue now if you were [02:14:08] doing this through RL um you would say [02:14:12] the model would Try the sky is alacon. [02:14:16] Nope, that's wrong. The sky is told. [02:14:20] Nope, that's wrong. This is a totally [02:14:21] untrained model, right? And so you would [02:14:22] have to do this on the order of 100,000 [02:14:25] times in order to just stumble on blue, [02:14:28] then get some learning signal off of [02:14:30] that. So if you're in the supervised [02:14:31] learning regime and you just get you [02:14:33] have your distribution of probabilities, [02:14:35] you get told uh that it's blue and you [02:14:38] figure out how far off you are, the [02:14:39] amount you learn is um is a function of [02:14:43] your pass rate. So like the further away [02:14:45] you are from blue, the more you've [02:14:46] learned to go towards blue uh using [02:14:49] cross entropy loss. And so you can think [02:14:50] of it as like your pass rate, your like [02:14:52] prior probability of having said blue. [02:14:54] And um as a function of that like in [02:14:56] supervised learning uh through cross [02:14:59] entropy loss you would you would learn [02:15:01] negative log p being pass rate [02:15:05] >> uh bits [02:15:06] >> once you get this label [02:15:08] >> whereas in RL [02:15:11] if you're just randomly guessing [02:15:13] and seeing if it works or not that's um [02:15:16] that's just basically going to be the [02:15:18] entropy of a binary random variable [02:15:21] which is [02:15:25] And what's also tough here is that [02:15:26] actually the distribution that you're [02:15:28] sampling under is your policies [02:15:30] distribution. [02:15:32] >> Right. So so it's like if your policy [02:15:33] has no chance of sampling blue, then you [02:15:35] will never get a signal. Exactly. Right. [02:15:37] Right. So that's that's being modeled by [02:15:39] the fact that your probability of [02:15:42] sampling blue is extremely low. If you [02:15:44] do sample it, you do learn as much as [02:15:46] you would have learned in a supervised [02:15:47] learning. In all other cases like you [02:15:49] know 99.999% of in an untrained model [02:15:52] you're um you're just learning [02:15:54] incredibly little from like seeing how [02:15:56] is not the correct word or told is not [02:15:58] the correct word. Um and that's what [02:16:00] happens most of the time. So you're just [02:16:01] like um learn very little. So if you try [02:16:04] to graph um if you put on the x-axis [02:16:08] your pass rate [02:16:10] um [02:16:11] and uh here you put the like sort of the [02:16:16] bits you bits you're learning from a [02:16:17] sample [02:16:19] if you have like 0% here 50% here and [02:16:24] 100% here. So the end of trading you're [02:16:26] here. Um if you have um [02:16:31] supervised learning negative log pass [02:16:33] rate would look something like this. And [02:16:36] then the uh entropy binary random [02:16:40] variable would look like this. [02:16:43] Um and this is uh depending on whether [02:16:47] you're doing knots or bits. [02:16:49] >> Yeah. If you do bits, it's like one [02:16:50] right here at the at the peak. Um this [02:16:53] is like a coin flip. You learn the most [02:16:54] from a coin flip. [02:16:55] >> Mhm. [02:16:56] uh this is supervised learning this is [02:16:58] RL [02:17:00] however [02:17:01] the problem is you spend most of [02:17:04] training in this regime right like in [02:17:08] the in the low pass rate regime and um [02:17:12] in fact of how fast you're learning is a [02:17:14] function of how many bits per sample [02:17:15] you're getting uh and you're getting [02:17:17] very little signal here if you chart the [02:17:21] pass rate on a log scale so you put the [02:17:25] x axis on a log scale where like at the [02:17:27] beginning of training with a vocap size [02:17:29] of 100k the pass rate is 100 one over [02:17:32] 100,000 then one over 10,000 [02:17:36] one over 1,000 [02:17:38] uh one over 100 and then um okay what [02:17:43] this graph looks like here where [02:17:45] supervised learning would look like this [02:17:48] and [02:17:52] and then RL if you just basically crunch [02:17:54] Right. I just showed there it would look [02:17:58] like that. [02:17:58] >> Yeah. And arguably you spend all your [02:18:01] time here [02:18:03] >> potentially never even getting a single [02:18:04] success, right? Like uh so so it's it's [02:18:07] a sort of depressing plot in the sense [02:18:08] that like once you're here it's not at [02:18:10] all obvious how you get to here. [02:18:12] >> Yeah. [02:18:12] >> Um you know once you're here you have [02:18:14] something but like you actually in many [02:18:15] RL problems spend all the time here. [02:18:18] Yeah. Uh so so there's a sort of [02:18:19] question of like how do you initialize [02:18:21] so you're at least not at zero but like [02:18:22] at a nonzero pass rate. Yeah. [02:18:25] >> Um one more thing I'd like to add about [02:18:26] bits per sample that's very relevant to [02:18:28] um uh you know you know any kind of [02:18:30] machine learning problem is that um [02:18:34] >> and there's a connection to soft targets [02:18:36] and distillation where if you have [02:18:37] access to the logets right not just the [02:18:39] one hot like this this is a sort of one [02:18:41] hot uh token answer. [02:18:42] >> Yeah. um if you have access to the soft [02:18:45] targets um the entropy of this [02:18:48] distribution is far far higher than than [02:18:50] the oneh hot. So there's actually way [02:18:52] more uh there's way more information in [02:18:54] bit and bits per sample um in a soft [02:18:57] label. [02:18:58] >> So that's why distillation is so [02:19:00] effective per sample is that it's [02:19:01] actually giving you way more information [02:19:03] per sample. [02:19:03] >> Ah yeah well I wonder what the equation [02:19:05] would be but obviously [02:19:07] >> it would just be the entropy of this [02:19:08] distribution like so the entropy of this [02:19:10] is zero. Yeah. Um the entropy of this is [02:19:12] like you know the the entropy equation [02:19:14] and this is also why like you know [02:19:15] AlphaGo go is quite beautiful in AlphaGo [02:19:17] you don't train the policy network to [02:19:20] imitate the MCTS action you train it to [02:19:23] imitate the MCTS distribution [02:19:25] >> interesting [02:19:25] >> but both of these are actually valid and [02:19:27] if you wanted to do a scientific [02:19:28] experiment of like how important are [02:19:30] this kind of soft label dark knowledge [02:19:32] distillation you can run an experiment [02:19:34] where you you uh retrain the policy [02:19:36] network on the action MCTS selected [02:19:38] rather than the software [02:19:39] >> interesting [02:19:41] Earlier I was sort of stumbling around [02:19:43] this intuitively. Why is this ability to [02:19:46] do um [02:19:50] iterative search where you don't [02:19:52] necessarily need to be able to win the [02:19:54] game in the beginning? You just need to [02:19:55] be able to improve your current policy. [02:19:57] Why is that so powerful a capability in [02:19:59] learning as compared to how LLM's [02:20:01] currently run learn RL? And um and yeah, [02:20:04] it's exactly this thing of uh this is [02:20:06] considering your pass rate of the entire [02:20:09] trajectory. I actually don't know a [02:20:11] formal way to think about this. Maybe [02:20:12] you should help me out here. [02:20:14] >> Why is AlphaGo an elegant RL algorithm? [02:20:16] Like so um uh the major reason is that [02:20:20] you never have to initialize at a 0% [02:20:23] success rate and solve the exploration [02:20:25] problem of how to get a non-zero success [02:20:27] rate. And and this is what allows you to [02:20:28] hill climb this beautiful supervised [02:20:30] learning signal where and if you look at [02:20:32] the actual implementation of AlphaGo um [02:20:34] every step of the way there's no uh [02:20:36] there's actually no um you know TD error [02:20:40] learning or dynamic programming uh at [02:20:42] least explicitly um it's just supervised [02:20:45] learning on a value classification as [02:20:47] well as a policy uh you know KL [02:20:49] minimization. So it's just a super [02:20:52] supervised learning problem on improved [02:20:54] labels. And so the training is very [02:20:55] stable, right? You can train like as big [02:20:57] of a network as you want. You can kind [02:20:59] of retrain this on the data set. [02:21:00] Everything will just go stably. The [02:21:01] infrastructure is very simple to [02:21:03] implement as well. Um you don't need a [02:21:04] complex distributed system to kind of [02:21:06] keep everything on policy. Um at the end [02:21:08] of the day, you're just saying like I [02:21:10] have some improved labels. Let's retrain [02:21:12] my supervised model on these targets. [02:21:15] Yeah. [02:21:15] >> And and so you're always in this [02:21:16] beautiful regime where you're just [02:21:18] trying to improve the policy [02:21:19] >> rather than uh escape this kind of like [02:21:22] uh sort of local minima where every [02:21:24] every signal is flat all around you. Um [02:21:26] so so one way to draw the the curve is [02:21:27] like if you draw the sort of win rate of [02:21:29] an MCTS policy versus the raw network. [02:21:32] Um let's say this dotted line is the raw [02:21:34] network. The MCTS policy kind of looks [02:21:37] like like this. And so every step of the [02:21:40] way this supervision signal is very [02:21:42] clean, [02:21:43] >> right? You're never in a situation [02:21:45] where, you know, the MCTS is kind of [02:21:47] like giving you no signal. Yeah. Unless [02:21:49] your MCTS distribution converges to [02:21:52] exactly what your policy network [02:21:53] predicts. [02:21:54] >> Yeah. Yeah. Yeah. [02:21:55] >> Okay. That that that's that's a great [02:21:57] way to explain it. [02:21:58] >> Um, [02:22:01] cool. Okay. Maybe we sit down and I ask [02:22:03] some questions about automated research. [02:22:05] >> Sounds good. One thing I really wanted [02:22:06] to talk to you about is that you did a [02:22:09] bunch of the research for this project [02:22:13] through this kind of automated uh LLM [02:22:16] coding assistant loop. And um there's an [02:22:21] idea that if you fully automated AI [02:22:23] research, you could have some sort of [02:22:24] singularity. Uh obviously we're not [02:22:26] there yet, but to the extent that we [02:22:27] have early indications of what this [02:22:28] process might look like, I am curious [02:22:30] what your observations about um what the [02:22:34] AI is good at, what it's not good at, [02:22:36] what you think about this scenario, it's [02:22:38] likelihood eventually, [02:22:40] what thoughts you have about this in [02:22:42] general. [02:22:42] >> For sure. Yeah, I think automated [02:22:44] scientific research is one of the most [02:22:45] exciting um uh skills that you know the [02:22:49] Frontier Labs are developing right now [02:22:50] and I think it's important for everyone [02:22:51] who's doing any kind of research to get [02:22:54] a good intuition of like what it can do [02:22:55] now and what it can't and how might the [02:22:57] sort of science process work in the [02:22:59] future once we're having AI automating a [02:23:01] lot of this this investigation. Um so uh [02:23:04] in brief I mostly use Opus 4.6 6 and 4.7 [02:23:07] throughout the working on this and um [02:23:10] what works is that the models can do a [02:23:14] very good job of doing hyperparameter [02:23:15] optimization. So in the past people [02:23:17] would kind of come up with a search base [02:23:18] of hyperparameters like learning rate [02:23:20] and you know weight decay and maybe how [02:23:22] many layers are in your network and um [02:23:24] they would just kind of do a grid search [02:23:25] or a sort of basian hyperparameter [02:23:26] optimization uh approach and then it [02:23:29] would find some tuned parameters. Um the [02:23:31] kind of really cool thing that automated [02:23:34] uh you know uh coding can do now is that [02:23:37] it can search a much more open-ended set [02:23:39] of problems right it can say like well [02:23:41] um I've identified that like the [02:23:43] gradients are kind of small on this [02:23:44] layer so let me change it up here let me [02:23:46] rewrite the code so the data later data [02:23:48] loader has a new augmentation I came up [02:23:49] with let's uh let's uh sort of try to [02:23:52] find the the best way to kind of fit the [02:23:53] constraints of the optimization problem [02:23:55] and and you end up with this much more [02:23:56] flexible and kind of highlevel almost [02:23:58] like grad studentlike uh ability ility [02:24:00] to just you know grind a performance [02:24:02] metric and and so this can squeeze out [02:24:04] quite a lot of performance. you can you [02:24:06] know on a fixed data set with a fixed [02:24:07] time budget um improve perplexity by [02:24:10] quite a lot on on a sort of [02:24:11] classification problem like LMS or um or [02:24:14] go um and uh it is also fantastic now at [02:24:18] basically executing any experiment right [02:24:20] so I have a a clawed skill that I wrote [02:24:22] called experiment where um I give it a [02:24:24] description of what I wanted to plot and [02:24:26] like I just describe here's the x-axis I [02:24:28] want here's the y- axis answer this [02:24:30] question for me and it'll go run off and [02:24:32] do all the experiments compile the plot [02:24:33] make a report and suggest like you know [02:24:35] what might have caused it or or so [02:24:37] forth. Um so so that's what works quite [02:24:39] well today and I think we can expect [02:24:40] that these abilities get better in the [02:24:42] future but it's also kind of useful to [02:24:44] know you know what what is it not doing [02:24:45] so well today. Um so on my uh blog [02:24:48] version of this tutorial I have a a plot [02:24:51] of basically all the kind of experiments [02:24:52] I did grouped in a sort of tree where um [02:24:55] you know every node kind of represents a [02:24:56] failed successful or sort of mixed [02:24:58] experimental result and then from there [02:25:00] it branches off into a child where it's [02:25:02] like the follow-on experiment. Um, [02:25:03] occasionally I'll kind of rabbit hole [02:25:05] down a track like this off policy MCTS [02:25:07] reabeling, do a few experiments, and [02:25:09] then realize it's probably not worth it. [02:25:10] So then I'll kind of jump to a [02:25:12] completely different track, right? And I [02:25:13] call these kind of things like rows, [02:25:14] right? So So what what I find is that [02:25:16] current uh, you know, closed models that [02:25:18] we can access, the public can access [02:25:20] today, um, they don't seem to be that [02:25:23] great at selecting what the next [02:25:25] experiment should be in a given track. [02:25:27] and they don't seem to be able to kind [02:25:30] of step back and do the lateral thinking [02:25:31] of like wait a minute this track doesn't [02:25:33] really make sense like let's go back to [02:25:35] sort of first principles and and think [02:25:37] about you know what the bottleneck might [02:25:39] be or like what are we trying to achieve [02:25:41] right and and so often I had to catch [02:25:42] infra bugs myself by prompting the right [02:25:44] question to cloud to like investigate [02:25:46] you know why why what what is causing [02:25:48] this discrepancy and then it'll answer [02:25:49] the question um I think with like you [02:25:51] know mythos class models or mythos++ [02:25:53] models coming online um maybe this just [02:25:55] completely changes and these these [02:25:57] problems just fall to to just improve [02:25:59] scaling. Um but at the same time I think [02:26:01] there's a lot of like rich opportunity [02:26:03] to um develop RL environments that might [02:26:06] incentivize this kind of lateral [02:26:07] thinking and and so one of the [02:26:08] motivations for setting up this Go [02:26:10] environment was that I think that you [02:26:12] know Go captures a lot of very [02:26:13] interesting research problems often [02:26:15] overlapping with you know LLMs or [02:26:17] robotics and yet it's like very quick to [02:26:19] verify. Um the outer loop is ultimately [02:26:22] like does the agent do what I think it [02:26:23] does in and you can kind of check the [02:26:25] outcome of a go game quite easily. Um [02:26:27] and then the inner loop involves all [02:26:28] this kind of like you know research [02:26:30] engineering around distributed systems [02:26:32] uh predicting whether an idea is going [02:26:34] to work or not. um predicting the you [02:26:36] know the difference a particular [02:26:37] modification to your training algorithm [02:26:39] might make um and I think there's a rich [02:26:41] library of subtasks and sub environments [02:26:43] that you can kind of train an automated [02:26:45] scientist to work on uh with go as a [02:26:47] sort of outer verification loop that [02:26:49] then once you acquire these skills maybe [02:26:50] you can apply them to like other domains [02:26:52] like uh you know um biosciences or [02:26:55] robotics [02:26:55] >> or automating AI research [02:26:56] >> or automating II research [02:26:57] >> which is which is the real crux or the [02:27:00] um scary slash uh incredible thing for [02:27:04] just making AI making future versions of [02:27:06] AIS and you're suggesting the outer loop [02:27:07] here could just be your win rate against [02:27:09] Katago basically. [02:27:10] >> Um that's one of them. Um I think [02:27:13] there's a lot of deeper questions that [02:27:14] one could tackle, right? So for example [02:27:17] um let's say you have an idea on um how [02:27:20] to improve a scaling law compute [02:27:21] multiplier. Yeah. [02:27:22] >> Um the outcome isn't necessarily like I [02:27:25] I uh achieve the best gobot ever. The [02:27:28] outcome might just be like can I predict [02:27:30] what the win rate of my gobot will be? [02:27:32] Yeah. [02:27:33] >> Or can I predict the scaling law plots [02:27:35] that emerge from my idea? But then you [02:27:36] can verify that you haven't kind of [02:27:38] reward hacked anything by using a very [02:27:39] verifiable game like go on the outer [02:27:41] loop. [02:27:41] >> I I think there's a couple of [02:27:43] interesting follow- on questions. [02:27:44] There's questions on the inner loop and [02:27:46] the outer loop. On the inner loop, [02:27:48] there's a question of how locally [02:27:51] verifiable any modification you might [02:27:53] make is. That is to say, could would you [02:27:56] know whether something is actually an [02:27:58] improvement or a degradation, some idea [02:28:00] you try out? Would you know that if [02:28:02] something isn't working as a result of [02:28:04] um a bug or is it the result of the idea [02:28:06] itself being wrong? Um Ilia was talking [02:28:09] about why having one of the reasons he [02:28:12] he thinks he's a good researcher is he [02:28:14] is a good researcher. One of the things [02:28:15] he thinks makes him a good researcher is [02:28:18] that um he has intuition about he has [02:28:23] strong belief in what the correct idea [02:28:25] is and he is able to persevere through [02:28:29] bugs and know which things are bugs [02:28:30] versus mistakes in the fundamental idea [02:28:33] based on his high level belief about [02:28:34] this idea should work so therefore it's [02:28:36] this there has to be bug versus the [02:28:37] other way around. Why don't we start [02:28:39] with that question actually? Yeah. How [02:28:40] locally verifiable are things which are [02:28:43] good ideas? Yeah, I think as in the case [02:28:46] of the success story for deep learning, [02:28:48] you can think about this as like a [02:28:49] decadesl long idea that took like took a [02:28:51] lot of faith to get it to work. [02:28:54] >> Um, and so this presents a very [02:28:56] challenging long horizon, you know, RL [02:28:58] problem where you know, every every step [02:29:00] of the way you have like a committee [02:29:02] telling you that this is a bad idea and [02:29:04] then ultimately you break it through, [02:29:05] right? And so like how do you design RL [02:29:08] environments that maybe give you some [02:29:10] feedback uh uh earlier? Um and and I [02:29:13] think this is a very tough open question [02:29:15] that I don't have an answer to, but um [02:29:18] but you know ultimately to play a very [02:29:19] strong gobot you probably did need to [02:29:21] discover deep learning. [02:29:22] >> Yeah. [02:29:22] >> Right. And so um um I think that like [02:29:26] having a challenging game that cannot be [02:29:29] you know cheated easily on the outer [02:29:31] loop could be used as a sort of outer [02:29:33] loop signal for something like [02:29:35] discovering the principles of deep [02:29:37] learning. Now of course like to make it [02:29:38] tractable and this is where research [02:29:40] taste really matters um like you have to [02:29:42] come up with ways to initialize your [02:29:45] problem so that you don't solve a sort [02:29:47] of very intractable problem right like [02:29:48] like maybe you can leverage LLMs as as a [02:29:51] sort of a universal grammar in the [02:29:53] middle to kind of give give you some [02:29:54] sort of local feedback um um the fact [02:29:57] that LM are universal grammar means that [02:29:59] they can kind of move at almost any [02:30:00] level of the stack right they can think [02:30:02] very locally as well as step back and [02:30:04] think like in very broad steps and I [02:30:07] think that's where a lot of u um the the [02:30:09] lateral thinking ability of humans kind [02:30:11] of come from like like how to know if [02:30:13] the track that you're pursuing or the [02:30:14] objective that you're pursuing is not [02:30:15] right and you should be asking a [02:30:16] different question. [02:30:18] >> The uh the other question is how [02:30:19] stackable local improvements are in the [02:30:22] attempt to get to a better result on the [02:30:24] outer loop. Um I've heard rumors that at [02:30:27] some AI labs the thing that has gone [02:30:28] wrong is that people will individually [02:30:30] pursue good ideas. um but those don't [02:30:33] end up stacking well and so the training [02:30:35] run falls because of some weird uh [02:30:38] interaction between two seemingly good [02:30:40] ideas and having a single top down [02:30:42] vision of how things should work is very [02:30:43] important. Um having worked at uh [02:30:46] different AI labs and also playing [02:30:47] around with I guess parallel agents [02:30:50] trying different ideas what is your [02:30:51] sense of how parallelizable um AI [02:30:54] innovation is? [02:30:55] >> Yeah, great question. Um [02:30:58] I think the research taste for executing [02:31:01] well on you know the bitter lesson is [02:31:03] that you need to know how much the [02:31:06] bitter lesson can buy you and uh how [02:31:08] much is too much to ask for at any given [02:31:10] moment right like of course in the [02:31:12] fullness of time compute kind of is the [02:31:14] single most important determinant on [02:31:16] like how things work and uh and and uh [02:31:20] it's almost like inevitable that as you [02:31:21] scale up energy and compute uh and [02:31:23] parameters intelligence will just fall [02:31:25] out of that and that's super super [02:31:26] beautiful, super profound. No [02:31:28] algorithmic detail really matters beyond [02:31:30] that. But um in present day we don't [02:31:33] have infinite compute and parameters and [02:31:34] and in arbitrarily good initialization. [02:31:37] So we have to come up with like [02:31:39] heristics that kind of give us that but [02:31:41] these heristics are probably somewhat [02:31:43] redundant. So that's probably why you [02:31:45] see this effect where like a lot of [02:31:47] these compute multipliers don't [02:31:48] necessarily stack is that like they [02:31:50] might have some correlated benefit. Um [02:31:52] and then and then you know three years [02:31:54] down the line when the Nvidia GPUs have [02:31:55] gotten even stronger maybe maybe they [02:31:57] stack even less well right like maybe [02:31:59] like at any given point in time the the [02:32:01] the sort of benefit of any given compute [02:32:04] multiplier is transitory which is what I [02:32:06] sort of suspected with the Kadiggo paper [02:32:08] like there was many [02:32:09] >> algorithmic ideas kind of applied and [02:32:11] then you can see that like with you know [02:32:12] modern blackwell GPUs and ADA class GPUs [02:32:15] that are much better than the sort of [02:32:16] V100 uh grade GPUs that that paper used [02:32:19] um you can see that like some of these [02:32:21] algor algorithmic tricks to speed up [02:32:22] convergence just don't matter so much [02:32:25] compared to something else and I think [02:32:26] that's a matter of taste in a in the [02:32:28] present time [02:32:29] >> yeah interesting how about the outer [02:32:32] loop how verifiable for making AI [02:32:36] smarter with go you do have this outer [02:32:39] loop of um win rate against the best [02:32:42] open source model out there and even [02:32:44] there as you were saying there are other [02:32:46] outer loops of did you discover a new [02:32:48] phenomenon which is actually very hard [02:32:50] to if you didn't know scaling laws were [02:32:53] important if you're back in when was [02:32:54] chinchilla or Kaplan scaling laws [02:32:56] released like 2019 [02:32:57] >> yeah so if you're back in 2015 would you [02:33:00] there's not an automated procedure one [02:33:01] can easily imagine of uh knowing which [02:33:05] paper is the scaling loss paper versus [02:33:07] which is just like another random plot [02:33:09] and so that even in the go case is a [02:33:12] hard to verify outer loop and the whole [02:33:15] the whole idea of an outer loop is to [02:33:16] have like some backs stop on um on [02:33:20] improvement. Uh but let alone for [02:33:22] general AGI where of course we have a [02:33:25] bunch of these benchmarks but there's a [02:33:27] problem that like we know the things we [02:33:29] can measure and we improve on the things [02:33:31] we can measure but we're we care about [02:33:33] this broader ability to do economically [02:33:35] useful work which is um at least until [02:33:37] you automate everything easy and not not [02:33:39] super easy to measure. Um so yeah [02:33:42] there's a there's a question of okay how [02:33:44] how good is the outer verification loop [02:33:45] for uh for AI self-improvement and does [02:33:49] that matter? [02:33:49] >> Yeah. Um I'm going to give a [02:33:52] non-rigorous argument but one that I [02:33:53] kind of intuitively believe which is [02:33:55] that you know um DeepMind the AI [02:33:57] research lab um they started as a sort [02:33:59] of focus on games right like they they [02:34:01] kind of use games as their outer loop [02:34:03] and then the researchers learned uh from [02:34:05] experience of solving games and then [02:34:07] like now they're working on LMS and [02:34:09] presumably there was some positive [02:34:11] transfer from their time working on [02:34:13] games and like Atari and Go and and uh [02:34:15] you know Starcraft that like now helps [02:34:17] them make good LMS like I assume that [02:34:20] there's like positive transfer in some [02:34:21] regard whether it's coding or general [02:34:23] research ability or project management [02:34:25] right like all these things kind of like [02:34:27] probably help them do well um and so if [02:34:31] that's the case why wouldn't it also be [02:34:33] true for automated AI researchers like [02:34:36] they should be able to positively [02:34:37] transfer experience tackling quick to [02:34:40] verify uh quick to iterate on [02:34:42] environments to something more ambitious [02:34:44] and economically useful like uh you know [02:34:46] automating drug discovery or so forth [02:34:48] >> I mean I don't know isn't the hasn't the [02:34:50] issue with historically until Gemini 3 [02:34:54] or whatever been a couple years ago [02:34:57] people were saying look Google hasn't uh [02:35:00] isn't catching up in LLMs because [02:35:03] they're too tight to the old approach [02:35:06] and yeah there's gains but there's also [02:35:08] um there's ways in which actively [02:35:11] hinders you um so it's actually not [02:35:13] obvious to me that there's like [02:35:14] >> the jury is still out right like I I [02:35:16] think like who knows if the you know [02:35:18] let's say currently Hey, Google's doing [02:35:19] quite well. Who knows if the uh [02:35:21] initialization on training on games is [02:35:24] ultimately going to hobble their ability [02:35:25] to be the winner in the long term, [02:35:27] right? Like like uh it's it's hard to [02:35:29] say for sure. Um [02:35:30] >> and uh you know, likewise, who knows if [02:35:33] the late seeming late start was really [02:35:36] just them kind of pre-training for [02:35:37] longer on how how to like scale up TPUs, [02:35:40] right? they invested all their tech tree [02:35:41] in like uh getting TPUs to be good which [02:35:44] seemed not that useful in the short term [02:35:45] but then in the long term it becomes [02:35:46] maybe like a uh so it's it's even hard [02:35:49] for humans to reason about what the [02:35:50] optimal research strategy should be [02:35:52] right even with the the data we have [02:35:54] today. [02:35:55] >> Yeah. Yeah. Cool. Um okay we should let [02:35:58] people know how they can find out more [02:35:59] about this project whether to fork it [02:36:02] themselves whether to check out your [02:36:03] blog where you do an excellent job [02:36:04] explaining many of these ideas. Um uh [02:36:07] where do people go next? [02:36:08] >> Great. Yeah. So my my website is [02:36:10] evjang.com. The there's a blog post that [02:36:13] kind of links to a interactive version [02:36:14] of this tutorial. Um and on my GitHub uh [02:36:17] which is the username is just eric Jang. [02:36:18] Uh there's a there's a autogo repo that [02:36:21] people can fork and reproduce the uh [02:36:23] training results. And I also highly [02:36:24] recommend people check out this blog [02:36:26] post as rocks may think which we touch [02:36:27] on some of the ideas in this [02:36:28] conversation but it's this grander [02:36:32] you know um thesis of what happens when [02:36:34] you have uh thinking as a primitive in [02:36:38] >> computer science. [02:36:39] >> Exactly right. Uh and so I highly [02:36:41] recommend people check out that blog [02:36:42] list as well. [02:36:42] >> Yeah. And I encourage to the you know [02:36:43] the audience to you know think about the [02:36:46] relationship between thinking and go you [02:36:48] know via MCTS and search and how it [02:36:50] relates to LMS. I think there's [02:36:51] something quite like profound there. um [02:36:53] and probably underexplored just because [02:36:55] Go has been relatively underexplored [02:36:57] compared to you know the boom in LMS. Um [02:36:59] it's not to say that I think we should [02:37:00] have trees in our in our LMS but um but [02:37:03] but there is some very interesting [02:37:05] duality between them and you can [02:37:06] actually do a lot of research on go um [02:37:09] MCTS and and reasoning with you know [02:37:11] very small budgets. So that's very [02:37:12] exciting. [02:37:13] >> Cool. Awesome. Eric, thanks for doing [02:37:15] this. It's it's an honor to be on the