Transcript — "Chip design from the bottom up – Reiner Pope" (Dwarkesh Patel)

I'm back with Rhiner Pope who is the CEO of Maddox which is a new AI chip company. Last time we were talking about what happens inside a data center. Now I want to understand what happens inside an AI chip. How does a chip actually work? Full disclosure by the way I am an angel investor in MaddX. Um so hopefully you have designed a good chip. [laughter] Okay. >> So I'll start with uh sort of the very smallest fundamental unit of of of chip design. And then we'll sort of build up into what an overall um like actual production chip uh what are the components of that. >> Yeah. >> At the very bottom level of a chip, the primitives that we work with are uh logic gates um which are very simple things like and or not. Um and then these are connected together by by wires that have to be laid out uh physically as metal traces on a chip. The main function that that AI chips want to compute is uh multiplication of matrices. And really inside that is the fundamental primitive is multiply accumulative just like of of pairs of numbers. So we're going to sort of demonstrate

[00:01:00] what that calculation looks like by hand and then sort of infer what what a uh circuit would look like for that. it'll turn out to be sort of easiest if if I do multiplication uh accumulator something like a um a four- bit number um with another four- bit number. Um and then we're going to the actual clearest primitive is actually um multiply accumulate. So there's a multiply these two terms and then we're going to add in so product of these two terms and then we're going to add in an 8 bit uh number and and can I ask a clarifying question? Why is this the um natural primitive for um you know whatever computation happens inside a computer? >> Yeah. So um there there's a few reasons for this. Uh it it's a little bit more efficient. Um but the the reason it's natural for AI chips is that if you look

[00:02:01] what's happening during a matrix multiply the uh what is matrix multiply in very short it is um uh there's a for loop over I and over J. >> Yeah. and over K of output I K plus equals to um input I J times other input J K and so multiply accumulate happens at every single step of of a matrix multiply >> makes sense >> and then the other observation is that um the precision will almost always be higher in the accumulation step than in the multiplication step. Um, this is maybe specific to AI chips. Uh, but you you're multiplying low precision numbers, but then when you accumulate, errors accumulate quickly, and so you need more precision here. So, this is why we've chosen to do a 4-bit multiplication and an 8 bit addition. Let me make sure I understood that.

[00:03:00] There's two ways to understand that. One is that the value will be larger than the inputs. And the other is that if it was a floatingoint number it would be maybe that that part is like less intuitive to me but it's maybe the same principle. It >> it is really the same principle. Um I guess the sort of I mean I guess the the separate principle is that >> as you are summing up this number you are summing up a whole bunch of numbers and so you got a lot of rounding errors accumulating whereas in this case there's like there's there's only one multiplication in that chain and so there's not a lot of rounding errors accumulating in the multiplication. >> Why are you summing up a whole bunch of numbers? Is it just two numbers? just I mean the summation happens it's repeated many >> times so yeah any errors accumulate I see makes sense >> so how would we perform this calculation by hand um I mean as a human we would probably separate it into two but we can sort of do it all all in one um using long multiplication so the multiplication term first we're going to multiply this number this four- bit number here by every single bit

[00:04:00] position in the other four bit number so we write that out Um first one 0 0 1 multiplied by this bit position. Um that is this number itself. Then shifted across by one we're multiplying by zero. That gives us an all zeros number. Shifted uh across even one more to multiply by this one. We get one 01. And then finally for this left last bit position we get an all zero's uh number again. So this this sort of gives us a bunch of terms that we're going to have to add for the um for the multiplication. And then while we're doing that uh summation of this, we might as well add in the the the actual accumulator term as well. And so we just copy that directly across. So So this is the sum. It's a it's a

[00:05:00] fiveway sum that we're going to want to compute. So firstly, what did what logic gates did it take us to even get to this intermediate step? We needed to uh produce all 16 of these um partial products. >> Mhm. >> How do I produce one of these partial products? So let's take this this number one for example. Here it is one. So what was it? How do we produce this number? By multiplying this number by this one over here. We can actually produce that by an andgate. This this number is one. If both this bit is one and this bit is one. If either of them is zero, then the multiplication of one times one of of 0 time anything is zero. >> So to produce all of this stuff, we ended up consuming 16 and gates. >> Mhm. Or in the general case if I were doing a like a P bit multiply times a Q bit multiply

[00:06:00] then this will be like P * Q many ants >> exactly finally I sum them actually most of the work is going to happen in the summing um and so let me describe uh sort of the other logic gate that we use here and is almost the simplest logic gate that exists on a chip. It's it's it's it's almost the smallest. Um at the other extreme, typically the very largest logic gate that you'll use is something called a full adder. And what this does is sort of it does like coming from software, you might think that a full adder like adds 32-bit numbers together. Uh in this case, it just adds three single bit numbers together. And so you can think of it as like adding 0, one, and one together. Now when I add these together the result can be 0 1 2 or three. So I can express that in binary using just two bits. So it it as input it has three bits and as output it has it has two bits which in

[00:07:01] this case are one and like uh the number two in binary is one zero. So, so this is also known as a 3 to2 compressor because it takes three bits of input and produces two bits of output. >> The two inputs are an x and a y-value and then some carry that that came in from like >> the sorry the the the three inputs are um all all bits that are in the same sort of bit position like three bits that are in a column here. >> Yeah. Yeah. And then the two outputs um I I have sort of drawn them vertically here and horizontally here to kind of match this vertical versus horizontal layout here which is expressing that um things that are in the same column are in the same like bit position um whereas things in that are in adjacent columns like this is a carry out whereas this was a sum. >> So if the inputs in the full ladder are let's say like 1 0 1 then the output would still be 1 0. If it was 1 one it'd

[00:08:00] be 1 one. It was 0 0 it'd be 0 0. it was like 0 1 0 it'd still be 0 1. >> So yeah, it's just counting essentially the number of uh the number of things and expressing that in binary. >> So this circuit actually can sort of capture what we as humans naturally do um when when we're when when we're doing summing along a column. So I can show that sort of I'll show sort of one iteration of using the full adder to sum. Um the way I sum here is going to be a little bit unnatural for humans. Humans we would sort of sum along the column and then remember the carry. Um but uh instead of remembering the carry we'll actually just explicitly write it out. So in this we proceed from the rightmost column towards the left. On the rightmost column we sum the one and the one and that produces like a a zero here and a carry of one. So we've we sort of used a this full adder circuit on this pair of bits and produced a pair of bits as output. Now we can do the same thing with this

[00:09:00] column. and we've got a column of 1 2 3 four numbers. And so maybe we'll like take the first three of them. Um run a full adder on them and that gives us a zero and a zero as output. Um so like the sum of these is is 0 0. So that's the full compressor uh full full adder applied to all of these bits. As I've used up bits, I'm going to sort of just cross them out to indicate that I've I've handled them. Let's just keep going a little bit more. Um so we we'll go here. I take these three numbers, I add them. That gives me a one and a zero. I've dealt with these three numbers. And now I take one, two, and I can even take these three numbers for example, right now, and add them. And that gives me a one and a zero. And I've dealt with these numbers. So I can sort of like the way I should view this is that I have this whole grid of numbers that need to be added. Uh I'm going to just keep applying full adders uh to all the bits that are here. Constantly removing three numbers from a column and then writing out uh two

[00:10:01] numbers as output. Um keep going with this over and over and over and again until I eventually get like some just one single number coming out here. Um something like that. Uh this is probably the wrong sum. So, so this approach that I've described here, this is called a um data multiplier. Um uh and this is sort of like the standard for how you do uh area efficient multipliers using uh full adders. Let's try and quantify the circuit size of this just so we have got a sense of like how big things are so we can compare to them later. How many full adders do I did I use? I started with how many numbers I have the 16 partial products which is the product of all of these terms with all of these plus the eight um terms that I'm adding

[00:11:01] here. So I I I started off with 24 bits and then on I produced eight bits on the output eventually and in every step I was sort of crossing off three numbers and writing two numbers out as in result. And so every single use of a full ladder eliminates one of the bits here. >> Mhm. >> And so how many full adders? It must be the 24 minus the 8. So that there were 16 full ladders in this circuit. In general um this is true in the general case as well there will be p * q many um full adders um uh in this circuit. >> You sure I understand the logic of that? So the input bits 24 is p * q plus p plus q. >> That's right. >> And the output bits is just uh p + q. And so p * q plus p plus q minus p plus q equals p * q. That's right. So I think this explains sort of

[00:12:00] uh or at least hints at the second reason why we chose to do a um multiply accumulate. Um first reason being that's actually what shows up in matric matrix multiplication but second reason being it gave us this very slick um p * q very simple algebra. >> Mhm. So we've sort of described like this this whole procedure every single uh atomic step that I took here becomes a logic gate and then sort of the wires connected together like when I had these three inputs that that I salvaged to produce these two outputs like if I think of mapping this to a physical device there would be a wire that runs sort of connecting all three of these things together into a logic gate that produced this output. Okay. So this is the the main primitive at different bit widths um that that is that is inside an an AI chip. Um uh we're going to build up from here to uh how would you use that to run all the other operations you may want. >> This might be the wrong time to ask this question, but whenever Nvidia reports

[00:13:01] that this chip can do x many FP4 or half as many FP8, it seems to imply that those circuits are funible. uh that there's not a dedicated like FP4 versus FP8. Um but the way you you're mapping it out here, it seems like you would need if it has to be mapped out in the logic, you would need a dedicated um FP4 multiply accumulate and then a dedicated FP8 accumulate. Basically, can you can you funge them as drawn? They're actually not particularly funible. Um this is actually one of the main choices you have to make when designing a chip which is how much of FP4 how much of FB8 do I have and then sometimes I'll make a consideration from the point of view of like uh what do I think the is the customer requirement another way to take an angle on that is to say um what is the what is the power budget for equalize the power budget between FP4 and FP8 >> but so then when they report those numbers and they just happen to be the case that like it does 2x as many FP4 as

[00:14:00] FP8 they just happen to choose like give um equivalent die areas to all the floating points and as a result that ended up being >> like why is the ratio exactly 2x? >> Yeah, exactly. >> Yeah. So um part of it is I mean that surely that won't be exactly like exactly equivalent to die area. Um there's uh there's a data movement reason actually and we'll maybe come back to this when we sort of look through how it goes into and out of memories. Um there's something really nice just from a software level of the fact that I can pack two four-bit numbers into into the same storage as an 8- bit number. And so when I store that to a memory or something like that, uh it the the sizing of the um the buses that I wire within the chip um actually makes that work out really really nicely. >> Actually, come to think of it, it's not just 2x. It would be uh the the amount of area it takes it sounds like is um quadratic. >> It's quadratic in fact >> with a with a bit length. So uh that's why smaller precision is like even more

[00:15:00] favorable than your magazine. This is a really big reason. So, um, in fact, Nvidia made a change. Um, historically, uh, up until B 100 or B200, every time you have the bit precision, you you double the, um, the flop count. Um, that ratio is exactly like for the reason you said because of this quadratic scaling, that ratio is actually slightly wrong. Um, it should be like an even bigger you should get even bigger speed up than than you than you might otherwise think. um Nvidia's like product specs have sort of started acknowledging that in B3 at 100 and beyond where the FP4 is three times faster than the FP8 >> though it should be 4x >> yeah what I've shown here is like the simplest case of integer multiply when you're when you're dealing with floating point as you do in FP4 and FB8 um there's this sort of other term which is the the the exponent that just complicates this this calculation >> so so what can we see already from this it's like I think the the big observation we've made is that there's this quadratic scaling with bitwidth um which which is like very effective and

[00:16:01] and is the single reason why um low precision arithmetic has worked so well for neural nets. Um but the other thing we're going to do now is we're going to compare sort of the area spent on the multiplication itself with all of the circuitry that is around it. So we'll we'll walk back in time a little bit and and see how did GPUs prior to tensor cores work um as which is the same way as the way that CPUs worked in fact. M >> so which is like where do we stick this multiply accumulate unit so generically I'll describe like a CUDA core or a CPU you'll have some register file which stores some number of entries maybe maybe it's like eight um eight entries um uh of of like in this case I guess four bit numbers but typically like 32-bit numbers or something like that of uh which are numbers Um so this is the like

[00:17:03] inside the CUDA core I'll have some register file of some depth and then I will have my multiply accumulate circuit multiply uh and accumulate circuit and what it's going to do it's going to like it's going to take three arbitrary registers from this register file perform the multiply accumulate and then write back to the register file. So it's going to maybe write to this one, but it it was able to read from this one, this one, and and another random one. So it'll take three inputs like this. >> So this is the core data path of of many um processes. Most processes look like this. You've got some set of registers and then you've got some set of uh logic units or alus. We want to analyze the cost of the data movement from the register file to to the ALOU and back. So ultimately there's going to be some circuit that says well I I don't always

[00:18:00] have to select this guy. I might select any of the registers in any point in time. And so sort of a first question is how can I build a circuit? The circuit that I'm going to look for is uh is a MX. So um in this case it's going to have eight inputs one from each entry of the register file um and it's going to have one output which is actually um producing this output and then like what is the cost of this thing? It's it's like all we have to to build it out of is and or uh and so how do we build it? We we do the dumbest thing possible. we like form a mask saying we okay when we want to read like the third entry um we're going to and every single entry with either one or zero based on whether that's all we want to read and then we're going to order all of them together >> okay just to make sure I understand the basics what the MX is doing is it just like selecting >> just selecting >> just selecting an input >> yeah so like invisible to software is like you say I want input number three

[00:19:00] that means there's a max here and so like what is the cost of this max so an input uh operating on p bits. Well, I'm going to so I have n rows. That's this eight rows and I've got like each row is p bits wide. Well, I have to and every single bit. So, I get n * p many and gates. Every single input, I have to say, am I going to like mask it out or not? Um, and then and then I'm going to ore them all together. And so, uh, there's going to be like n minus one * p many ore gates, which is saying, uh, I've got all of these different things. Almost all of them are zeros, but I need to sort of collapse them down into into like from my eight options down into one option. And so, every step I need to ore like one row into into an existing row. >> Got it. Yeah. It's actually kind of

[00:20:00] funny that you would sort of um you don't think at the level of hardware, you sort of just think like, oh, I'll just select element three and something as simple as that is uh a sort of like in and of itself a quite complicated circuit. >> Yeah. I mean, this is the first step of all of the hidden data movement costs that the shop. >> Yeah. Yeah. Um, and so like the thing like we're we're just going to like compare like I have to pay this cost and I've got one marks here and then in fact I have two more copies of that for each of the three inputs to my multiply accumulate operation. And so I have this cost which is like like 3 * n * p and gates over here um compared to this p * q um like uh sort of gates in the actual circuit that I that is doing the thing I care about. And if we plug in actual numbers like this n being 8 um like I get like 24 * p gates over just just in the data movement compared to like if q is four like four * p gates just in the in in

[00:21:01] the adder multiply adder >> and sorry where is the three coming from >> three different inputs here. >> Got it. Okay. So the case like really just what I'm hinting at here is that like all of this work which scales like as as the size of the register file and this is a very small register file. All of this work just moving the data from the register file to the to the to the logic unit um is many many times more expensive than the logic unit. In the most recent cluster max report semi analysis ranked almost 100 different GPU clouds. Cruso was one of only five that made the gold tier. Semi- analysis found that gold tier providers like Crusoe had a total cost of ownership that was 5 to 15% lower than silver tier ones, even when they had identical GPU pricing. This makes sense because total cost of ownership is downstream of a bunch of different things that don't necessarily show up in the sticker price, but that Crusoe has optimized things [music] like how well you detect faults and how quickly you replace failed nodes. For example, Crusoe was one of the first

[00:22:00] clouds to adopt Envy Sentinel, [music] Nvidia's own GPU monitoring and self-healing software for enhanced GPU uptime, utilization, and reliability. This lets Crusoe make use of everything that Nvidia has learned about why chips fail across all their different fleets and deployments so that Crusoe can catch faults earlier in the process. And once they identify a failure, Crusoe can swap in a healthy node in less [music] than 10 minutes. Because they're not running bare metal, Cruso doesn't have to spend time installing an operating system or configuring drivers. They can just spin up a new VM on an already running and pre-qualified host. If you want to learn more about this or the other reasons that Cruso made Goldier, go to cruso.ai/dorcache. It may be helpful to just see what a MX looks like. Maybe like a two bit or a 4-bit MX. >> Yeah, great. So, we'll take some inputs. we'll we'll have maybe um uh like we'll just do a a two-way look. So we've got um two different numbers um uh we've got these two inputs and then

[00:23:01] we have a um so these are the inputs um that are being selected between and then we have a selector which says which can either be like I want this one or or it could be I want the other one. So this is a one hot encoding. Um so so this is all what we all start with and then we the output we want to produce like let let's focus on this case. So so this is the actual input we got. >> Yeah. >> Um we just want to produce this guy as as as the result. Um and so like sort of very laboriously what we do is we and this bit with all of these. And so um that produces like ending this bit with this row. And likewise we end this bit with this row that produces all zeros. So this was the there's four ands here.

[00:24:03] There's four ends here. And then finally we just or these two together. This gives a one. We order these two together. This gives a one. We these two together gives a zero. We these two together and it gives a one. And so this is the four or so like this actually ends up looking a little bit like addition in fact like we we did exactly the set of same same set of ands here um sort of we've added all of these things together but then instead of collapsing it by using these full ladder circuits we just get a very simple collapsing with or gates >> and how but that I guess that doesn't look like n * p >> um so so yeah so this was this was with um n equals two um inputs. >> Ah okay. >> Um in the general case we will have um n uh like and so this this is n rows um and then we'll have uh p bits um per row um

[00:25:03] so that gives us the um the n pi n * p many and gates. So this circuit I've described here almost all of the cost like uh 78s of the cost uh is in the reading and writing the register file and only a tiny fraction of the cost is in the logic unit itself. So this is the problem to solve. Um this this essentially was the the state of play um prior to the VA generation of Nvidia GPUs. This is what what this kind of thing is what was inside the CUDA cores. Um and this and this sort of problem statement is uh what motivated the introduction of tensor cores which are more generically called systolic arrays. >> So if we think about how are we going to solve this problem like we we're spending almost all of our circuit area on something that we just really don't care about and is and is like hidden to the software programmer and and the thing that we actually care about is is is not most of the area. Um well make this one bigger somehow while while keeping this at the same size. That's the goal. So sort of the evolution was

[00:26:02] like uh we had baked this much into hardware um in this stage that this single line is a multiply accumulate um and this was this single thing was baked into hardware. Um the idea of a systolic array is to sort of go two levels of loop up and uh and bake uh this entire loop out here in into hardware. And so the idea being that if we have a much bigger granularity fixed function piece of logic um maybe the taxes we pay on the input and output are much smaller. It sounds like you're suggesting that if you have um if you go up one step in the uh in the matrix multiply loop that there's some you you can tilt the balance more towards compute than communication. >> That's right. Sort of there's two effects that we're going to take advantage of here. One is just that we can do more stuff before per every trip through a register file. Um and then the other thing we're going to take advantage of is in fact um uh in in in some of this loop we can take advantage of for example

[00:27:01] uh some certain things staying fixed. So uh let's sort of visually we're going to look at this matrix multiplication. So so this portion of the loop corresponds to a matrix vector multiplication in fact. So we'll we'll take a matrix um and multiply it by uh a vector. So how do we do this? We take every column gets multiplied by the vector and then summed. So we're going to sum sort of along columns. Um and so this 0 and three gets multiplied by the three and seven and gets summed and then the one and two gets multiplied by the three and seven and gets summed. So there is a multiply accumulate associated with every single one of these um entries in the matrix. So we'll just draw out these four multiply accumulates. And just make sure I understand why there's four multi accumulates. So if each entry in the uh column that

[00:28:02] corresponds to the output vector is uh a dotproduct and in this case it'll be like two multiplications and then the addition of those two multiplications. So like you're accumulating. >> Yeah. So the addition So really there's only one addition per dot productduct but like >> the initialization of zero. >> Yeah. >> So the the what we're going to aim for is to have um so we've got to we want to have quadratically more compute. We do we have we've got sort of um x * y uh as much compute as as we had before. Um um but we're going to want to somehow aim for having only x times as much um like uh communication and and this is sort of the the intention so that we get this uh advantage uh term going as y. So we've laid down the multiplications

[00:29:00] bringing in like we're going to want to bring in a vector of size two and so that sort of already is in line with our comm's target. that's fine. However, we need to somehow manage the communication of this matrix which which which does which exceeds our budget of of X. And so, uh the idea is that in the in an AI context, this this matrix is actually going to stay fixed for a long period of time. And so, instead of like bringing it in from the outside, so we've got some register file sitting over here, we we don't want to have like the amount of stuff coming out of this register file. This is the term that we want to go uh sort of as as x in some sense. Um we don't want to bring this full matrix in from the register file every every cycle because we don't have enough um that would cost too much in in terms of wiring from the register file. And so instead we're going to store um our key trick is that this matrix can be stored locally uh to the systolic array. And so

[00:30:02] um where we will store these numbers 0 1 2 and three in in just like a gate called a register that like physically stores these numbers and we're going to reuse these numbers uh over and over again for for a large number of different uh different vectors. So the optimization here is that like the nature of matrix multiplication is you can store this like uh square quadratic thing directly where the logic is happening. >> Mhm. >> Um and which is like higher dimension than the uh or has an extra dimension compared to the inputs which you keep swapping in and out. >> That's right. I mean this is the nature of what a matrix multiplication is is that you do uh a lot of multiplication to get one value out like a dot product is the result of a lot of multiplications and so that optimization means that you you're just like you can stuff a lot of like multiplication in

[00:31:01] before you get some value out of it. >> That's right. That's right. >> Yeah. >> So like just to complete the picture here of concretely how that looks. Um uh I I swapped the three and the two here. Um three and two. Um so just like this 0 and three is going to multiply by the three and seven and so um uh we're going to form a dot productduct sort of along columns here. So somehow we're going to feed a three and a seven in here. These participate in sort of this feeds into this multiplication and also feeds into this multiplication. Likewise the three feeds into here and also into here. And then we're going to sum uh sort of along here with like starting at the top of a column we feed in zeros and then coming out the bottom we get results coming out. So sort of just to visually see what we've got um there's a dot product that is performed along

[00:32:00] columns in a matrix and that that sort of maps exactly to what is done spatially in the systolic array here. Um so this is one dotproduct summed vertically and this is a second dotproduct also summed vertically. Um and then what is the data that needs to go into and out of the register file? We have x amount of data that's coming out here on the output. Um and we also have sort of uh this data coming from the input x amount of data from the input. Um and so with respect to the input and output vectors at least um we we've sort of met our goal of having only x as uh as much data going in and out of the um uh the register file. >> This leaves open the question of like I said that the the weight matrices weight matrix is stored locally in the systolic cray. How did it get there in the first place? Um is sort of a like at some point you need to boot your chip and populate this data and so where did that come from? The trick is just we just do

[00:33:01] it very slowly. So we we very slowly trickle feed it in into the systolic array. The um sort of the simplest strategy is that we we sort of run this daisy chain that says like feed a number into here and then and then on the next clock cycle it'll move down to the next entry of the systolic array. And so we can do that in every um in in every column in parallel. And that gives us um sort of uh this is also going to come from here and this is going to be another factor of approximately x units of of of bandwidth coming in. >> Can can you just would you mind repeating that sentence one more time? >> So like we're sort of like we know that we're going to be bringing in uh numbers only rarely into the matrix. Um and so we just want to come up with any construction at all such that the amount of wiring that actually feeds into sort of crosses this boundary of this array like this boundary right here. We just want to keep that bounded to x and not be not go as xy. And so a particularly simple strategy is that we um

[00:34:00] >> we sort of bring in a a number into the top row of the systolic array. That's what we can do in one clock cycle. And then for like uh for y consecutive clock cycles, we're going to be bringing in the top row every time and then sort of shift all of the other rows down by one. And that keeps the the sort of the wiring that needs to come from this expensive register file uh only down to a factor of x rather than x. >> I see. Okay. So, uh there's two questions in terms of communication. There's like communication time and then there's communication bandwidth. Yes. And you're saying since we're only going to be loading this in once, let's maxim let's minimize bandwidth because bandwidth equals die area >> and let's just like load it in slowly over like smaller lanes because we're just going to keep this value in there for a while. >> Exactly. >> Interesting. So it's interesting to me that uh when we were talking last time about inference across many chips, the big highlevel thing we're trying to optimize for is increase the amount of compute

[00:35:00] >> per memory bandwidth. that is to say per communication. And here also we're trying to increase the amount of like uh actual multiplies or actual additions relative to transporting um information from registers to uh the logic. So in both cases you're trying to maximize compute relative to communication. >> Yeah. Um this this shows up sort of all the way up and down the stack. Um this is sort of close to the bottom um sort of like to to the gates. Um there's sort of a a version that's maybe even closer to the gates of just like um even the precision of number format that you choose to use. Um we saw that same effect. Uh there's like a square cube law or like squared versus linear term going on both in just purely the precision of this um alou but then also in terms of the size of this the the matrix. >> Yeah. Very interesting. >> So um so this unit is sort of the next bigger unit. we had like the multiplication circuit and then uh on top of that we have um a like a pretty

[00:36:00] large systolic array. I drew it as 2x2 but in like um for example older TPUs they were described as 128 by 128 uh of of this circuit uh shown here. Um and this circuit ends up being um this is the the most efficient known mechanism for uh [clears throat] circuit for implementing a matrix multiply. >> I see. So we we've talked about um sort of it seems obvious that you should try to maximize compute relative to communication. Um what are nonobvious trade-offs that um actually you are you know keep you up at night about should we do X or should we do Y and it's not obvious what the answer is. >> Yeah. So I mean I think most of the decisions in chip design are um sizing >> and so uh already in what we've drawn so far um like so AI chips all have this this circuit in it. They have a a systolic array and then somewhere near it a register file uh providing inputs

[00:37:01] and outputs. Um the two sort of uh like even within this scope sizing questions that you have are how big should I make my systolic array and how big should I make the um the the register file the um so and and then the trade-off for the size of size of the systolic array actually these two questions are coupled is um one way to think of it is to say I'm going to like have a budget for um uh how much what percentage of my chip area I want to spend on data movement. So maybe I just say that I want this to be 10% and the system array to be 90%. Um and then sort of I can size my register file. Bigger register files are more flexible. I they allow me to run sort of more I can get more application level performance out. Um but uh but then they sort of take away from this area spent on the systemic career. >> Yeah, makes sense. >> I recently ran an essay contest where I asked people to write about what I consider to be some of the biggest open

[00:38:00] questions about AI. The submission window closed last week, so I use cursor to create a couple of different interfaces to help me review the entries. One interface anonymizes submissions and hides unnecessary information. It lets me group responses by question, add notes, and record my scores. The other interface helps me review entrance who also want to be considered for the researcher role that I'm hiring for. The UI puts the applicant's essay right next to their resume and their personal website so that I can see everything at once. Cursor's harness is really good at helping these models see and improve their UIs. I watched it render these interfaces in the built-in browser, take screenshots, click through sections, and keep iterating. At this point, Cursor is where I do most of my work. Whether I'm reading and visualizing a bunch of research papers or coding up an interface to review applications or making flashcards for my Blackboard lectures, cursor just makes it very easy for an AI to look at whatever I'm looking at and help me understand it and work with me on it. So, whatever you're working on, you should do it in Cursor. [music] Go to cursor.comash.

[00:39:00] Where does the clock cycle of a chip come in? What what determines what that is? >> Yeah. And what is a clock cycle of a chip? >> So, I guess at at baseline, it's it's sort of worth observing that chips are incredibly incredibly parallel, right? You've got 100 billion transistors in a chip. >> A key thing that you need to do whenever you have uh like massive parallelism is you need to synchronize between the different parallel units. Um in software typically you like you uh you have these very expensive synchronization methods like a a mutx. So one thread will finish what it's doing. It will in like it will uh grab a lock somewhere stored in memory and then notify the other thread that it's done. Um on chips we take a very different approach and say that uh uh every like nancond or so um all all circuitry in the chip will kind of pause for a moment and then and then synchronize every so it actually synchronizes every single ncond or so and so that is the clock cycle. >> Um the entire chip typically all in sort

[00:40:00] of one fell swoop um goes in lock step to the next operation that happens. And so what this looks like in circuitry is that um you will have this typically drawn. So the clock is sort of mediated by registers which are these storage devices that we've drawn elsewhere. Um and the way to think of it is that I have I have some storage uh which is storing like a bit which might be zero or one and then I have some sort of cloud of logic which maybe is like this systolic array or this multiplier or something like that. Um, and then I've got some and that's going to produce some output. So my inputs, I've got a bunch of inputs feeding to this cloud of logic. And then eventually later there's going to be some output register that this writes to. There is a global clock signal which drives all of these registers and it says at a certain instance in time when the clock uh uh strikes um uh whatever value

[00:41:04] happens to be on this wire at that instant that's what's going to get stored in there. And so the the sort of the challenge here is like I I would like to have my clock speed run as fast as possible because if I can run at 2 GHz, I can get twice as many operations done per per second than if I run at 1 GHz. Um but what that ends up meaning is that uh I'm very sensitive to the delay through this cloud of logic because uh any computation that is going to happen in here needs to sort of finish before the next clock cycle hits, >> right? So uh a major point of sort of optimization on on any chip then is is to make is to sort of make uh this delay um delay from here uh uh as short as possible. >> Interesting. And um is there ever because the constraint here seems to be that if you add too much logic then you

[00:42:03] might risk missing the clock cycle >> but if you don't add enough then you're you know leaving potential compute on the table. Um, is there ever a situation where you're like you'd take a probabilistic chance that a computer computation finishes and or is it just like no either was going to finish by clock cycle or not? >> Yeah, in standard chip design you you margin it such that I mean there is a probability but it's like many many standard deviations like way standard deviations out such that for all intents and purposes it is a reliable part. It will it will always meet the clock. Um there are some weird exceptions to that. There are clock domain crossings where you go from one clock to the other clock and then you actually do have to reason about this probability. But interesting >> um uh in the main path uh you just like you margin such that you'll get there like 25% of the clock cycle in advance um uh so that it's very unlikely that >> and this in this um the clock uh

[00:43:03] where the uh clocks synchronize I guess where the registers are. This is not something you determine as a chip designer. This is sort of just like an artifact of hey I want whatever sequence of logic and then the software you use to convert your veril log into the thing you send to TSMC that just determines like hey in order to make this work you got to kind of you got to put a register here here and here to make sure that there's a there's no one step that is like too long such that it makes the whole clock cycle of the entire chip longer than it has to be. >> Yeah. So this is actually a huge part of the work of of designing a chip actually is is inserting them. So it is done in a combination of manually and automatically. >> Um so I mean like just like to show the very sort of dumb uh version of like what you can do here, you can take this logic and split it in half. And so like say actually instead of just one uh cloud of logic, I'm going to um have two smaller clouds of logic um which do the

[00:44:01] same thing but split them up by a register, >> right? Um feeding in like this. Um, and this is like like if you split it like in the middle, you can hit twice the clock frequency. That's great. You get twice the performance. Um, at the cost of this extra register and so at the cost of some more uh storage. >> And so stepping back, why do we need to synchronize the whole chip? Like if you have like if you imagine playing Factorio or something, there's no like global clock cycle. Just is done when it's done. There's iron on the plate. You can take it if you want. >> Yeah. So uh taking that analogy um the the the the thing that you need to be mindful of is if I've got two different paths through some logic. So I have to do computation like F here um and then computation G here. Um and then they're going to come and meet for computation H somewhere here. >> Um and so uh there's going to be manufacturing variance here. Uh in some chips F will

[00:45:00] take a little longer. Maybe in some chips G will take a bit little bit longer. And so if I've got some signal that's propagating through here, um, and the result from F and G have to sort of meet up at H, um, what can the the thing that can go wrong is that F can get there early and it meets like the previous value of G or the next value of G or something like that. And age needs to know when to start. >> Exactly. >> Like when when has this next iteration of And so this explains why different chips made at the same process node, the same like TSMC uh technology um can have different clock cycles. Like two chips made at 3 nometer might have different clock cycles based on >> uh whether they were able to optimize making sure that like there's no one critical path that is so long that um it it slows down the whole ship's clock cycle. >> That's right. This optimization that I that I showed here, this is just the uh this is sort of um pipeline register insertion it's called. Yeah. Um we've inserted in the middle of the pipeliner register here. Um this is a sort of pure

[00:46:02] trade-off between clock speed and and an area. >> Yeah. Um this is the easy case. There is a harder case too which is um uh sort of drawn it as a pipeline of logic here. But in in other cases you may have some uh uh some calculation which actually feeds back in on itself. So uh it runs some function f and then writes back to uh itself like this. So for example, this might be this addition like you've got some number that you're adding into every clock cycle. Um, and so this this could be like a um this could be like plus uh we're adding in some number every clock cycle. >> So this like this little circuit uh um it essentially it's just going to sum all of the numbers that get presented on on different clock cycles. >> And the challenge is if this plus takes too long, what can I do? If I like split it in if I try and put a pipeline register like right in the middle of it um like like here in the middle of it.

[00:47:02] this will end up changing the computation that's done. Instead of forming a running sum of everything that comes here, I will actually have two different running sums. I'll end up having a running sum of the even numbers and a running sum of the odd numbers. So sort of this constraint where I have um a loop in my logic which all chips have somewhere. Um this is actually the thing that is the hardest thing to to to address and sets the clock cycle. >> I don't understand why it'd be a problem to have that or I'm not sure even what it would mean to have a register there like is it it's a sort of uh atomic operation right >> yeah well so plus is not really atomic uh like I think um >> as you just demonstrated >> yeah yeah it took a whole lot of work to do a summation and so like you can take the early parts of that work and then and then stick a register in the middle and then do take the late parts of that work >> good okay yeah uh and I guess it's then up to so TSMC offers a uh PDK which specifies hey here's the primitives of um logic that we can grant you in the in

[00:48:00] the chip And it's up to them to determine that no no primitive is bigger than like the clock cycle they're hoping a process node targets. But other than that, is there like what further optimiz can you just say like hey here's all the primitives from TSMC and if keep adding registers in between the primitives as much as is needed until you get to your desired clock cycle. >> Yeah. As a logic designer like the chip architects set the clock cycle. So just for for one example the primitives you get from TSMC are on the order of like and gates or full ladders. >> Um they depends a lot on voltage and frequent and and and which library you choose and so on but generally they you can typically have about like 10 or 20 or 30 of these in in a in a clock cycle sequentially. So these primitives are very very fast like 10 picosconds or something like that. Uh, and so as a logic designer, I mean like [clears throat] in principle, if you literally just had like um like register

[00:49:01] and then and ANDgate um kind of in a loop like that, you could get an insanely fast clock speed like more than four or five six GHz something like that. Um but if you take this this like really sort of like simple circuit um and you look at the area you're spending here like um this is maybe like one I mean this is this is called one gate equivalent um in size so like unit of one in area and this thing is like unit of eight in area or something like that and so like this is just again almost all of your cost has been this like synchronization or communication cost compared to the actual logic and so uh so this could be a case where you've gone too far. You've made your clock speed really really fast at the cost of spending almost all of your area on on on pipeline registers. >> Interesting. So what you're hinting at is a dynamic where you can have really fast clock speed but you're not getting that much work done. >> Yeah. Yeah. Yeah. >> Um and so you can have like low latency but or but low bandwidth or throughput

[00:50:01] rather. >> Yeah. the it hurts your throughput in fact because like uh like the the throughput of your chip you can think of as the product of um how much I get get done per clock cycle which is based on this area efficiency thing times how many clocks I get per second. This is this is actually so similar to the thing we were discussing last time about like batch size um where if you have a low batch size then you can any one user can receive their uh next token really fast but the total number of tokens that are processed in say an hour will be kind of lower than it could otherwise be. >> Yeah, exactly. You get you get less parallelism out if you if you drive your clock speed up really high. >> Language models are starting to compete against the best human forecasters. I sat down with two senior Jane Streeters, Ron Minsky and Dan Ponttoorvo, and asked, "At some point, does AI just do what Jane Street does?" >> There's a world that we should take seriously where, you know, we're going to build large language models or some other AI systems that are like strictly smarter than all humans on the planet and more capable at all cognitive tasks.

[00:51:02] Trading in particular feels to me as like kind of AGI complete, sort of like NP complete because at the end of the day, trading involves figuring out what things are worth, which means making predictions about the future. >> Jane Street isn't betting against AI. They just signed a $6 billion computer deal, but Ron's view is that the edge keeps moving. >> I have never been more desperate to hire more engineers and more traders than I am today. You know, you have the usual thing of like the other hard parts that we don't yet know how to automate. Well, that ends up being where the competitive edge lies. >> You can find these open positions and watch the full interview at janestreet.com/thorcash. Okay. So, I remember talking to um an FPGA uh engineer at Chain Street, Clark, um who actually helped me prep for your the the previous interview we did together, and he was explaining why they use FPGAAS. And um uh I I imagine that for high frequency trading, the throughput is less important than

[00:52:00] latency. And so having very specific control over the clock cycle in a deterministic way is the most important thing. I um maybe it'd be interesting to talk about how you why you can't just achieve that with an ASC or why FPGA is the yeah why you might use an FPGA to have determinance to clock cycles for like high high frequency trading. >> Yeah. So I mean firstly like let's consider the sort of the business case for an FPJ versus an ASIC. um FPJs and AS6 use largely the same sort of uh conceptual model which is that I have um a series of gates built from ANS or exors those like very small primitives um connected together with a fixed clock cycle um uh and connected together with wires that are running in a fixed clock cycle so anything you can express in an FPGA you can express in an ASIC 2 and it'll be about an order of magnitude cheaper and and like better energyffic on an ASIC than an FPGA. Uh the

[00:53:02] trade-off is that the first FPGA costs you $10,000 whereas the first ASIC you make costs you $30 million because uh because of uh it requires an entire tape out. So sort of the business use case for an FPJ would be that I want something that has this very deterministic latency and and and and fast runtime and high parallelism. uh but um uh but you know is is I I'm going to change it very frequently, change what I do every month or something like that. Uh and so that I don't want to pay the tape back cost every time. Now, how does an FPGA actually implement it's sort of like it emulates the ASIC programming model but in in in a fixed piece of hardware. And so how does that work actually? So what it has uh at the at the base is um it's got a it's got a um it's got the two components we just talked about. It's got these registers as as as uh as storage devices and then it's got these

[00:54:00] these are called lots lookup tables um uh which actually provide all of the gates. M so and then and then we're going to see even the the sort of the third component we we then have like sort of a a swarm of these um registers and lots um and all of these are available and then they're connected by sort of this big set of um sort of uh moxes. So in front of uh every single one of these we've got uh something like one of these moxes which selects one input from everywhere else. Um sort of selecting from all of these different things. Um you know we've got a whole bunch of different options feeding into into into all of these things. So um so what what what this allows is essentially a uh when I program my FPGA I can say that I'm going to take all of

[00:55:01] these components and I'm going to sort of superimpose on top of this a particular wiring which like goes through this lot and then feeds into this lot and then goes to this register and then feeds into this lot or something like that. So what I've drawn in orange is like how you like FPGA means field programmable gate array. This is the the orange is what has been programmed in the field whereas the white is all of the wires that must exist in the FPJ in order to actually take to make the device in the first place. >> What does it mean to be programmed into a field >> like programmed in the field? So like the device is being deployed in a data center. It's sitting in the field and then you can come and program >> not field as like electric field or field as in like out there in the world field. Okay. Um, and so if I see look at the how the uh the field programming comes out of the first uh lookup table and goes in a second one. >> How is it? Yeah. How >> like like where are the wires that made that happen? I guess. Yeah. So I I I got

[00:56:00] a little bit like lazy in drawing all of these. Every single device here has a Mox sitting in front of it. um uh uh which can select from all of the like nearby um like uh circuits that are available. >> Yeah. >> Um >> and so uh the actual configuration of the FPGA is like amounts to it is the MX control. So like we in this Mox here we have the data inputs >> and then we have like the control that selects uh and so like there's a little storage device sitting next to every single one of these um uh marxes saying this is where you're going to source your input from. >> Got it. >> Um and so programming it consists of like configuring every single one of these marxes. >> So that makes sense. What is happening inside of the lookup table? >> Yeah. So the purpose of the lookup table, so it's going to also have a little bit of control feeding telling it what to do as well. The purpose of the lookup table is to function to be able

[00:57:00] to configurably take the role of an ANDgate, orgate, exor, any of those different things. >> So there's many ways you could consider doing that. Um the way it is done in sort of traditional FPGAAS is to say um it will support so it will be a a lookup table will be um it will have four bits of input one bit of output. How many different functions are there from four bits to one bit? There are 16 different functions. Um, and so, uh, you can actually just tabulate this as like 16 different like 16 different numbers. You've got a table of 0 1 1 0 0 1 16 entries. And so what it does is um this this table is stored in this blue configuration bit. Um and then uh

[00:58:00] it views these four bits as binary looks up the relevant row of the table and emits that bit. So this is a truth table view of of lookup tables essentially. >> Okay. So the lookup table if you think about an andgate orgate orate xorgate these are all like take as input >> those are like two input functions. Sometimes we have like more complicut function would be a three-way exor, right? Or a four-way exor. >> And in this case, how many it just depends how how big it is. But >> typical size for lots is four input. Um, which is sort of just a sweet spot between um there's another computer communication trade-off like here like if it if it has too few inputs, then you need to use more lots. Yeah. If it if it has too many. >> But basically, the lookup table is like a truth table. >> It's a truth table. Yeah. >> And with a truth table, you can program in any gate you want. That's right. >> Yeah. >> And so it's a lookup table just thinks like a programmable gate. >> That's right. And so I mean one of the things you can do here is you can see why where the rule of thumb that an FPGA

[00:59:00] is like an order of magnitude more expensive than an ASIC comes from. Um is to count how many gates would be inside this lookup table. >> Yeah. >> So we can view this lookup table essentially as one of these MXes. Um and so it is a MX with has to select between 16 different values. uh and so it is a max with uh sort of n= 16 uh options p= 1 bits and so what we saw way earlier is that um it this circuit costs like n * p many gates and so it's like um so it costs like n * p equals 16 and gates and also 16 ores. >> This circuit being the MX. >> Uh yeah, exactly. The MX the MX is the core. >> The MX that goes into the lookup table. >> Uh so the lookup table itself you can think of as being actually a big MX that

[01:00:00] like selects from all 16 rows down to one output. >> Yeah. Okay. >> That is the lookup table. >> But the way you've drawn it here, there's like a Mox and then a lookup table. >> It's Mox is all the way down. So I mean there's a there is a second Mox that is inside here. This Mox is is this Mox. >> Got it. Okay. And then the the other MX is just saying >> where it came from in this sort of mess of of of cases, >> right? And then the second Mox is okay, now you have one value, but that value is still >> um still a four bit value. >> Yeah. So I've selected four bits from the soup, right? >> Um and then and then I use those four bits to select which entry in the lookup table uh I'm going to use. >> Right. Okay. So, so like suppose in the first Mox there's like eight nearby eight you're you're pulling from eight nearby registers as input. >> Um [clears throat] and so that's like to a total of like 32 bits going in. Um and then out of that uh four bits come out those four bits go into the second mox which is inside the lookup table.

[01:01:00] >> So actually I would say in yeah in this case these registers are single bit registers. So if there are eight like eight nearby registers and lookup tables then I have eight bits total coming in in in nearby. I select from eight down to four different values. So there's actually like four different moxes one associated with each of these input like little marks associated with each of these input bits. Each of them is selecting one out of eight. >> And where are those eight coming from? >> Nearby registers and other lots. >> And each register is one bit. >> Yes. Yeah. And so I guess you AMD or whoever makes these um uh FJS still has to be opinionated about what uh what register should connect to which registers uh and then you can program in the actual gates but they add a wire in the connection like the communication topology right >> yeah so there's the sort of like you get flexibility in a local grain thing there's a sort of nearby neighborhood where you can select from um but then

[01:02:00] more grossly like more coarsely longer distant connections they they form an opinion on that. >> Yeah. And the reason it's 10x slower is why. >> So if you look at the cost of like building this lookup table, it's like 32 gates. >> Yeah. >> And then it can give me the equivalent of like what's one an interesting thing I can do here? I can do a four-way and gate. And so that's like I I'm using 32 gates of lookup table to sort of implement like a four-way and means like what does a four-way and I would do like and and and then and if and so to imple like this is a a circuit that I could implement in an ASIC directly using these three and gates. Um but using a lot I can also implement it but it's going to take like these 32 gates instead of three. And so the overhead is really coming from the like um the fact that the lookup table the mucks in the lookup table >> is there's a more concise way to describe a truth table than listing out every

[01:03:02] single possible combination of inputs >> uh which is just to like write out the gate. >> Yeah. Like to to like place down the polysilicon and the and the >> right. Yeah. Yeah. Interesting. One important point he made to me is that the reason they prefer FPGAs to CPUs is because they get deterministic clock cycles. They know when a packet will come in and go out. Um why is it not a guarantee in CPUs? >> So you can actually design a CPU that has deterministic latency as well. Um uh and in fact like the the processors that are inside a lot of uh AI chips actually also have deterministic latency too. Grock has advertised this. TPUs have that in the core as well. Um the challenge is getting sort of deterministic latency and high speed at the same time. And so um where does the non-determinism in latency come from? Um nondeterministic latency um comes from specific design choices in a CPU. Um

[01:04:03] uh it's actually possible to remove those design choices and make a CPU that has deterministic latency. um those are not very attractive in the market and so people don't make those CPUs anymore. Um but uh but but actually in some sense like deterministic latency is maybe a sort of a simpler designing start starting point. Um and then and then like some ship designers have added uh things in to to be nondeterministic. to take a concrete example of that. Um the probably the most important example is on a CPU just like the the uh the CPU cache itself. So uh in a CPU you have the the CPU this is this is the CPU die itself and then there's a memory uh off on the side. This is the DDR memory um off on the side. And then you have a cache system here. Inside it is the cache

[01:05:02] uh that sort of remembers recent accesses to DDR and and and stores them. And so uh when when I'm running through my CPU instructions, uh every time I uh I have an instruction that accesses memory, it first checks in cache uh was was the data stored in cache and then if not, it goes uh fetches out to DDR. Yep, this is a huge optimization. The cache is like two orders of magnitude faster than the DDR. Um, if you never uh if if you never like use the cache like all basically all programs would run 100 times slower. So, uh the presence of a cache is absolutely necessary for a CPU to to run at reasonable speed. Um but whether or not you get a cache hit is dependent on the sort of ambient environment of the CPU like what other programs are running what has run recently what is the random number generator inside the cache system doing and so so that is a big source of nondeterminism in in the runtime of a CPU. >> Yeah. >> Um so this is sort of the memory system for a CPU. Um

[01:06:01] the the big thing that you can do differently is uh instead of having the hardware say I'm going to like read uh read memory and then decide the hardware decides whether or not it comes comes comes from cache or not you can actually bake this in this decision into software. So uh a different design philosophy is is to um so and you see this in maybe for example TPUs. Um the the TPU instead has I mean I'll draw the same diagram but I'll call it a scratch pad. And so the the main difference is um so this would be like a TPU and then like HBM in this case rather than DDR but it's still an offchip memory. Um and instead of like the software saying first access like memory and then the hardware decides um you've got some instructions that that go here. This is like one kind of instruction and then a totally different kind of instruction that goes to HPM. >> Yeah. >> And so this style is is generically known as scratchpad um in instead of

[01:07:01] cache. The key distinction being that you have like one kind of instruction that says read or write scratchpad and a totally different instruction that says read or write hpm. So scratch pad being the cache. >> Yeah, this this thing here is is the scratch pad. um just be clear. So stepping way back, people say computers have the quoteunquote John Moy uh vonoyman architecture where there's this uh serial processing of information and maybe just because we've been talking about parallel accelerators, but I just don't like the FPGA is super parallel, the um the kinds of AI TPUs are super parallel. >> Um even CPUs are super parallel if you think about all the cores they have. And so is it actually like in what sense is modern hardware actually the vonoyman architecture is that actually a fair way to describe modern hardware? >> Um I think it's a fair way to describe CPUs like just the amount of parallelism like on a CPU the amount of parallelism you get is about 100 cores times maybe

[01:08:00] like 16 way vector unit so 106 uh about a thousand way parallelism on a CPU. >> Yeah. So one question is what is the there is a die that is being used for the CPU >> and if there's fewer threads just as a matter of like transistor uh voltages are like switching on and off is it just that there's like literally one control flow like a small part of the die where like voltages are switching on and off or like in what how how do you actually occupy the die area of a CPU if there's as opposed to >> if there's so few cores like what do you am in there. >> The cores are just much bigger and more complicated. Um, >> so uh I mean like so I guess we we should compare like a CPU core which takes up 1 16 1/100th of the die to like I mean to a lot like a lot is just only these 16 gates. So like it's clear why there are so many more lots in an FPGA than cores in a CPU. Um, but then sort of maybe the the like why are they more

[01:09:03] CUDA cores for example than than than CPU cores I think would be like why a like what's the difference between a CPU and a GPU or something like that would would be a big difference. um inside the CPU you have um so one big use of so the sort of the top unit uh uses of area inside a CPU are the cache um very little is actually the alus um like mostly it's like these register files rather than the logic units um and then the uh both of these things have equivalents in in a GPU and so that's not a big difference but the thing that does not have an equivalent in a GPU is the um sort of this branch predictor and so there's a whole big area in the CPU which is uh sort of uh just a whole bunch of predictors that are saying um when will my next branch be and where the where's the branch target for that and so uh stripping a lot of that out uh

[01:10:00] as as well as sort of making these register files tighter in in a sense um is uh driving a lot of where the like GPU gains >> and what is the purpose of the branch predictor to like execute both branches at once or what does it do? >> So the issue is that when I've got a series of instructions like um like instructions instructions instructions instructions I uh if I have a branch like here if this instruction is branch um the the actual processing step of processing an instruction um takes a really long amount of time. It takes like maybe five nanose or something like that. Um, so like uh the time to actually notice that I've got a branch and then like uh uh evaluate the boolean whe whether it's true and then and then update the program counter to the new target and then read from the instruction memory for that. Um that that could take like like actually five nonds to to finish. And so in reality this may finish somewhere down here. Um

[01:11:01] I don't want to like but like I want to run a clock speed that is much faster than what five nconds allows. Like like five nonds is 200 MHz clock speed. I would like to run at one or two GHz or something like that. And so um I need to run other instructions while the branch is is being evaluated. And so I like I really just want to keep running the following instructions that happen after me. Um but that might have been wrong. like if the branch ended up being taken then I need to know that instead of evaluating these instructions I actually need to like jump to wherever the target is and and run run these instructions instead. And so the purpose of the branch predictor is like like genuinely to predict based on like before you even get to this destruction to be like five cycles earlier to predict there was going to be a branch that's going to happen. So if I think about how the brain works versus what you're describing here, at a high level, the differences might be that while you can do structured sparity in these accelerators and then save yourself some

[01:12:02] um area that you would have otherwise had to dedicate to these gates in the brain. There's unstructured sparity. You know, any neuron can connect to any other neuron and not in like ways where it be column aligned or whatever. Yeah. >> Um then there's the fact that memory and computer colllocated. I guess you could say in a way the memory and computer colllocated on >> this is exactly the collocation in some sense of the memory. >> That's right. Yeah. Yes. Maybe that actually isn't a big difference. And the other maybe a bigger a big difference is that the clock cycle on the brain is much slower than on um on computers. And partly that's to preserve energy because the faster the clock cycle the bigger the voltage uh needs to be in order to identify for the signal to settle and to like identify what state of transistor is in. >> That's right. That's right. >> I don't know if you have other high level takes about like how

[01:13:01] any commentary on you know what the brain might be doing versus how these chips work. >> Yeah. I mean so so let's take the clock speed one uh first actually. Yeah. um the clock speed is quite high uh on on a chip because that I mean drives higher throughput. um like when we compare a like a GPU running some some workload it's running batch size thousand or something like that whereas like the brain is not running batch size thousand there's only one of me and so you could sort of imagine saying well take a GPU and like instead of running at a gigahertz run at a mehz or something like that and and that would start to look maybe a little bit more like like um uh sort of equivalent things that you're talking about in the brain um there is in the way that silicon works. Um there are like that does not give you an 10x advantage in energy efficiency. So um what it ends up looking like is

[01:14:00] you can uh like you sort of just end up running this circuit uh once to stabilization and then it'll sit idle for a long period of time. Um it doesn't consume a lot of energy while it's sitting idle because uh most of the energy is consumed in in sort of toggling uh bits from 0 to one and back. Um uh so actually let's let's talk about the energy consumption uh of of a circuit like this. The way to think of uh a bit being stored is you've actually deposited some charge in in a capacitor somewhere sitting somewhere in in the chip uh implicitly. So it becomes charged when it is um bit becomes a one and then it becomes discharged when it next goes to a zero. >> Um and that cycle of like charging the capacitor and then dumping that charge out to ground that is where the energy is consumed. >> Yeah. This is called the dynamic or switching power. Uh this is most of the energy consumption of a chip. There is some other energy consumption just coming from the fact that uh um uh insulators aren't perfect insulators but we'll dis discuss that most of the

[01:15:01] energy consumption actually comes from just the charging and discharging of like toggling from 0 to one and back to zero, >> right? >> Um so if you run a chip much slower and you only clock it once every thousand clock cycles or something, you will have a thousand times fewer transitions. it'll be about a thousand times less energy consumption. Um but but like but not a substantial advantage in energy efficiency. >> Um okay so you described how a TPU works at a high level. Um what is the difference uh at high level between how a GPU and a TPU work? >> Yeah. So I mean I think there's sort of a high level organization principle that is different. Um and then there's sort of inside the cores what are different but we'll look sort of uh outside the uh like at the high level um so we'll take a GPU um and a TPU and what does like sort of the top level block structure look like um if you think of this as the whole chip um in each case um the organization of of the GPU um uh is

[01:16:01] mostly a bunch of uh almost identical units which are these um these are the SM M um and then they've got a an L2 memory in the middle and then a bunch more of these SMS on the bottom. Um and so uh there's sort of this fairly regular grid of of cores. Um uh and then like if we look at a at a TPU in in comparison um you you uh you end up with much coarser grained units of uh logic. And so you you end up with something like um uh some large number of um maybe like maybe just a few m uh matrix units. These are the um these are the big like um sisteric arrays. And then in the middle you've got some vector unit and then you've got your um matrix units at the bottom. So um now sort of like matrix units with a

[01:17:02] vector unit in the middle sort of this is the whole TPU chip. You can sort of think of take scaling this thing down into a really tiny unit with a smaller matrix unit smaller vector unit and that is sort of what an SM is. So sort of at a very high level point of view the the GPU has a lot of tiny tiny TPUs um sort of tiled across the whole uh >> interesting. So like you're suggesting the tensor core within a streaming SM is anal analogous to an MXU. >> Yeah, it's very very similar. Yeah, >> I see. And so if you had more like more lack of structure, having a bunch of tiny TPUs makes a lot of sense. Whereas if you kind of just have like huge matrix multiplications, you're like, why don't we just why don't we avoid the co the cost of having the individual SMS with their own registers and warp schedulers and things like that. Why don't we just like make a huge thing and like amortize those costs across the

[01:18:01] whole thing? >> And I mean I think this shows up in how large you can grow things. We've we've sort of seen this theme like especially with a systolic array where larger systolic array amortizes the register file costs better. Um this sort of design allows you to have larger systolic arrays. This whereas the sort of GPU design constrains you to having small units of everything. Um there is a trade-off however um the there ends up being um because of this sort of coarse grained separation of things there you need to move a lot of data from the vector unit to the matrix units. Um and so like you need to move a lot of data through through a sort of like two two lines of parameter here. Um whereas if you sort of look at the equivalent thing here, you've got vector units everywhere and you need to move data through this line, through this line, through this line, through this line. So the amount of data you can move between a vector unit and a matrix unit is actually much higher in in a GPU than

[01:19:02] in a in a TPU because because like it's like instead of having to like move all the data through these just two lines, you're moving all these data through like 16 lines or something of of of wiring instead in a GPU, >> right? But also you might have to move across less area >> which I mean is also a saving like it's energy saving. So, so data ends up moving like if you can operate entirely within an SM the data movement is much smaller but then the moment you want to operate across SMS like it becomes sort of more more complicated and expensive. So you don't have comment but one might expect that a thing Maddox might try to do is to get the GPU like smaller structure of um systolic array surrounded by um SRAMM but also at the same time make it so that like the things you need in an SM to support the CUDA architecture uh but take a bunch of bunch of space

[01:20:01] you might discard. >> Yeah. Um we've talked publicly about something which we call a suspplitable systolic array which is sort of in in some sense you can think of as like big systolic arrays that can be small systolic arrays as well. Cool. Um okay I think this is a good note to close on. Riner thank you so much. Thanks Rus.