This animation is previewing an algorithm that can compress large pieces of text down to less than just one bit per character, which is kind of mind-boggling if you think about it. There's actually an open cache prize for text compression. It's known as the Hutter Prize available to anyone who can beat the current best record for compressing a gigabyte of Wikipedia as small as possible. I promise we'll build up to what exactly this animation is showing in due time. The core idea that it conveys underlies, well, I was going to say it underlies several of the winning submissions here, but that's not true. It's actually the backbone for all of them, every single one. And the method is not even specific to text. It is a fully general technique that you could use to compress images or videos, audio, really any data type. And this algorithm also got me thinking differently about that phrase, compression is intelligence, but I'll save the discussion for that until the end. As with any compression algorithm, you can think of this as two complimentary programs. One used by a sender to encode
[00:01:02] text into ones and zeros, preferably as few as possible, and the other used by the receiver to decode that bitstream back into text. And the goal here is lossless compression, meaning that output needs to be an exact replica of the original input. And winners for that prize are not just determined based on the size of this compressed representation. It also takes into account the size of the decoding program. So imagine yourself right now setting out to win this prize. Where would you start? How would you even begin to concoct such an algorithm? One option would be to think deeply about the fundamental limit of compressibility. After all, understanding the nature of that limit might inform how to get close to it. We've covered this in a previous chapter, but it's worth a quick recap. Back in the 1940s, Claude Shannon showed that the number of bits used for an optimal compression of a message should be approximately the negative log base 2 of the probability of seeing that message. This negative log quantity is
[00:02:00] what he defined as the information of an event. And I've said before, I kind of wish we had adopted a convention of calling this the log base 1/2 of P instead since I think that better captures the core property that this expression is trying to convey. Namely, each time a probability gets chopped in half, the information content increases by one bit. So, for example, a message with a probability of 1/2 raised to the power k has exactly k bits of information. Looking at Shannon's result, on the one hand, this is very intuitive. More likely messages require fewer bits and less likely messages require more bits. And the more precise statement itself is also decently intuitive. Each time one more bit is required, it corresponds to a message that's half as likely. On the other hand, though, if you really think seriously about what this whole result is saying, it seems utterly absurd. Take a moment to think about some passage with a large number of highly predictable characters. Say I give you the opening to the preamble of the US
[00:03:01] Constitution. After these first few words, you, an intelligent viewer familiar with famous historical documents, can predict with a very high confidence what's going to follow pretty much as soon as the context of what's being quoted has become clear. Say you give a 50% chance to all of these characters following that first phrase. That would mean that the Shannon information of those remaining characters is just one bit. Even if you gave a one in a thousand chance to those being the specific characters that arise, that's still only 10 bits of Shannon information. But now try to imagine writing a program that converts text into ones and zeros. The prospect of somehow cramming most of a full paragraph down into just one bit or even 10 bits feels well ridiculous. For comparison, the default encoding using ASI would spend over 2500 bits on these words. Some of you may have heard of Huffman encodings, a more efficient way
[00:04:00] to associate individual characters with bitst strings such that more common characters get shorter bit strings. Using the specific Huffman encoding I have illustrated on the left, that only gets you down to around 1,500 bits for this section. Even if you try to get smarter by applying a Huffman encoding to larger chunks of letters instead, something like the tokens used by modern language models, it just seems completely intractable to hope that most of a full paragraph like this, no matter how predictable, could ever fit into just a single bit or even into a small handful of bits. And yet, by the end of this video, you're going to see exactly this happening. The issue with any method like a Huffman encoding that always turns a given character into a fixed bit string is that it does nothing to incorporate context. Claude Shannon's result is telling us compressibility is entirely tied to predictability. But predictability of a given word can vary a lot depending on what came before it. Now sitting here in the 2020s, if I told
[00:05:01] you, hey, we need to leverage the predictability of text based on the context around it. there's a very obvious tool at our disposal, namely a large language model. And actually, the large part here doesn't matter. And if anything, as we'll see later, small ones have a real advantage. A language model is specifically trained to take in an arbitrary piece of text and make a prediction of what comes next, where that prediction takes the form of a probability distribution over a set of all possible choices for what might come next. But even if I just hand you some blackbox prediction machine, it's really not obvious how you would turn that into a compression algorithm. Looking back at our setup with a sender and a receiver, the critical addition we're going to make for this whole video is to assume that each one has a copy of the same language model that can be used as part of that encoding and decoding process. The best way to think about this algorithm is that it's something where you can kind of slot in an arbitrary language model and then the
[00:06:00] predictions that model makes are going to be used many times for the encoding and the decoding processes. And basically the better the model you use, the better the final compression. The 10,000 high-foot view of what this algorithm is actually going to do is that it encodes an arbitrary piece of text into one highly highly specific real number between 0 and 1. I recognize that might sound absurd at first, but it's actually very beautiful. Let me show you how it works. Even though most modern language models process text as a sequence of tokens, which are common chunks of characters like words, illustrating this algorithm is going to be much much easier if we use one that processes text character by character. That is, you feed in an arbitrary string of text and what it spits out is a probability distribution over all characters. In fact, for maximal simplicity and complete reduced visual clutter, I'm going to limit the output to just be 26 letters, ignoring capitalization and neglecting
[00:07:00] punctuation. To be clear, everything about the actual algorithm we'll describe works just as well if you're using a model with a larger token vocabulary. In fact, we have to switch to one later to see how effective it is. This just makes our diagrams much easier to follow. We'll start off with a completely empty input, meaning the distribution it spits out on the other side is the model's prediction for what character starts an arbitrary piece of text. And instead of a bar chart like this, throughout the video, we'll visualize probability distributions as horizontal stacks of sidebyside bars where the width of a given bar represents the probability of seeing a given letter. The total width of this diagram is then one since adding up the probabilities of every possible outcome should be one. In fact, the essence of the whole algorithm is going to be to imagine this stack as living on the unit interval of the number line from 0 to 1. And you think of each of these possible first characters as occupying a range of values on that interval. Notice a more common starting
[00:08:00] letter like t occupies a larger range of numbers, whereas this bar for P is about a fourth as wide. meaning the model sees that as four times less likely to be the starting letter. And then its neighbor Q gets an absolutely puny region by comparison. What we're going to set up here is a way to associate not just individual characters but arbitrarily long strings of text with very narrow regions on this interval. Then encoding the text will look like representing a very specific real number inside the appropriate tiny region and sending it to the receiver. Here, let's step through what it looks like to encode a specific small word like math. If instead of a blank input, you feed in a single letter M to the model, here's what the resulting output looks like for its prediction. And you'll notice it's dominated by the vowels. A gets a 38% chance, E gets a 16% chance, and so on, which makes sense. It would be unlikely to see a consonant following the letter M. And the way I'm drawing things where
[00:09:00] the second probability stack is perfectly lined up under that M gives the width of each bar in the second layer a really nice interpretation. Going from the full width of the diagram down to the width of this M involves scaling by the probability of starting with M. And then to shrink further down to the width of this A requires scaling by the probability of seeing an A given that you started with an M. That product right there is exactly the probability of starting with the two letters m a according to the model. This is the chain rule from probability. Likewise, these other small widths represent the probability that the model gives to starting with ME or MI and so on. And there's no reason to stop at two layers. If you feed into the model the context M A, here's the distribution of what it thinks will follow. And then again the width of any bar in this third layer is telling you the probability that the model gives to starting with a given threelet sequence. Applying the same idea one more time to the prefix m a t,
[00:10:00] we get a distribution over all possible fourth letters. And the width of this little bar here corresponds to the probability that the model gives to the whole text starting with math. And this is what I mean when I say that a narrow region on the interval corresponds to a given piece of text. If I give you some number in this range, really any number, together with the fact that I intend to encode a word with only four characters, you can unambiguously recover what word I'm talking about. Zooming out one layer, you can see that other ranges correspond to other strings starting with mat. Zooming out even more from that, you can see ranges of values corresponding to strings starting just with the letters m a and so on. Once we're zoomed out this far, it's hard to actually see the tiny ranges past a couple of levels given how small those probabilities are. But from up here, looking at the full unit interval, we can see what it looks like to range over all possible four-letter strings. And of course, displaying four layers is completely arbitrary. In principle,
[00:11:00] every infinite string corresponds to a unique real number between zero and one. And so back to our sender and receiver, the idea of the compression algorithm is that I, the sender, am going to package up a real number representing my text together with the number of characters in that text and I pass them along to you. In a moment, we'll dig into representing that real number in binary and how to analyze the number of bits of precision required. But first, it might be worth being clear on what exactly it looks like for you as the receiver to use this number together with the shared language model to decode the text. The decompression algorithm is also something where you imagine slotting in this text predictor. And it's critically important that it be exactly the same text predictor that was used for the encoding. Step one is to run the language model on a blank context to get this initial distribution of characters. And from that you can deduce that the number I gave you lands inside the region for M. You now know to run the model a second time using M as the
[00:12:00] context. That gives you this new distribution. And by looking at where my special number falls in the second layer, you can deduce that the second letter of my message is an A. That in turn tells you what to plug into the model next. And you just repeat this a number of times equal to however many letters I told you are in my word. If you want, you can avoid the need to send that message length explicitly if you introduce a special symbol into the vocabulary that represents end of message. In that case, the decoding procedure would look like zooming in until you hit that symbol. This is a design decision. In many ways, that's more elegant. But for our purposes, I'm going to separate out that message length explicitly, mainly because the introductory examples become cleaner if they're just pure text. When Shannon was describing optimal compression, one of the key ideas was that a perfectly compressed message should look indistinguishable from random noise. And you can get a feel for how the algorithm we're building up here fits this property by choosing uniformly random real numbers between 0 and 1. What
[00:13:01] you'll notice is that the strings these correspond to don't look arbitrary. Even though every possible string is represented in this diagram, almost all of the ones that we're landing on look reasonable. They look like actual snippets of English text. And that's because the widths of the bars in this diagram are defined such that more likely strings, according to the model, take up a larger portion of the number line. Meaning, if you just pick a spot at random, it's very likely to land on a likely string. In fact, this is basically just a way to visualize how it is that language models produce text. And so far, I've been writing these real numbers down as decimals, but to think of this as an actual encoding, we're going to have to represent them in binary, as ones and zeros, potentially a lot of ones and zeros depending on the length of the text. Just to be clear on what I mean by binary here, in the same way that the first digit after the decimal point in our usual base 10 system tells you which tenth of the interval you fall in, the first bit after a point in binary representation
[00:14:01] tells you whether you're in the left half or the right half. The next bit after that tells you whether you're in the left half or the right half of that section. And then the third bit narrows you down to one specific eighth of the interval and so on. Each new bit chops the space of possibilities in half. So the highlighted section here represents the range of real numbers that you could specify starting with these bits or in other words every possible continuation of this binary string. Whenever we choose to stop adding more bits, giving just a finite representation, the number that you get is the one sitting on the leftmost end point of this highlighted region. Now, if we bring back the pieces of text under this diagram that we were just looking at, the binary number that I've chosen to write down here is one that sits inside that narrow region for the string mat. And it uses a total of 10 bits of precision. By zooming out, you can actually see that 9 bits could not be enough to encode this particular word. The closest 9- bit number on the left is over here, well outside the
[00:15:01] target region. And then the next 9- bit number after that overshoots. Including that extra 10th bit narrows down the highlighted region enough so that we can get right in the middle of these two landing in that target range. And in fact, from this point of view, considering the size of that target region compared to the spacing between 10- bit numbers, it kind of feels like we lucked out being able to use only 10 bits in this case. 10 bits for a four-letter word works out to be an average of 2.5 bits per character, which is not terrible, but the real advantage kicks in as we apply this method to longer and longer pieces of text. Let me show you what it looks like for this process to play out over the full word mathematics. You'll notice that each new letter at C's, the E, the M, the A, are highly predictable. And as such, the size of our target region hardly shrinks at all. It's only by the time you get to that final S that it feels like an actual decision was made. And in fact, that same 10- bit string that we used to represent the word math still falls in
[00:16:00] this region for mathematics, bringing our bits per character, down to 0.91. Actually, that's a little disingenuous cuz in addition to sending the real number, we also have to send the message length, that is after all what distinguishes whether we're sending math versus mathematics. But still, this is starting to look like actually good compression. And I was actually really surprised when I saw this particular example. I knew that we should get something good with mathematics. But if using only 10 bits was a little lucky for math, it is extremely lucky for this longer word mathematics. The small region for this word has a width of 2 to the -8. Whereas the points on the number line that are representable with 10 bits or fewer are spaced out by 2 to the -10. In essence, what you're looking at is the fact that our word should have required 18 bits of information, and yet we lucked out being able to represent it with 10 bits. In fact, it's worth taking a moment to think about how many bits in general it should require to represent a point inside a given region.
[00:17:01] Notice how all of the tick marks for values expressable with two bits or fewer are spaced out by 1/4 and those expressible with three bits or fewer are spaced out by 1/8. And in general, all the numbers expressable with k bits or fewer are spaced out by exactly 2 to the k. So if you have some target region of size P and you don't want to rely on luck, you want a guarantee that you'll be able to land inside that target region using a number expressible in K bits or fewer regardless of how this region is positioned. What you need is to be sure that 2 to the K is smaller than P. Rearranging a bit, this is the same as saying negative k is at most log base 2 of p, which in turn is the same as saying k is greater than or equal to the negative log base 2 of p. So in other words, if your text has a probability p and you round up the negative log base 2 of that probability, you are guaranteed to be able to represent your text with this many bits
[00:18:00] and possibly fewer if you're lucky. Now this expression should feel very familiar. It looks just like Shannon's definition for the information of a given message. And that's really the whole beauty here. Shannon proved in principle why no compression algorithm can encode text with fewer bits than this on average. And he even showed why it should be possible in principle to get arbitrarily close to this limit. But what we're looking at here feels at least at first glance like it realizes Shannon's dream. It's a concrete implementable method where the size of a compressed message is basically the same as the information value up to a rounding error. To be more exact, this is only describing the encoded real number. Remember, we also need to send that message length. But that message length will be very small by comparison. For example, if you allocate a fixed header of 32 bits to describe that length, this would leave you safe to encode messages up to 4 billion characters long. So at large scales, essentially all of the encoding size is tied up in the bits of precision for
[00:19:01] that real number. And it's the number of those bits that rubs right up against Shannon's limit. And in fact, you've already seen how there are times when it seems to do a little better than Shannon's limit. Our example of mathematics had 18 bits of information, but it just happened to overlap with a 10- bit value. In a much more extreme case, a message that consists entirely of the letter A would correspond to the leftmost point of the diagram, which in our scheme is actually what the empty bit string would represent. So in that extreme example, all of the actual content is tied up purely in the message length. The reason these examples don't violate Shannon's theorem is because his theorem is all about averages. It's saying that you can't do better than this negative log P expression on average. All these cases of getting lucky are balanced out by the fact that more typically you have to round up that fractional information to a larger whole number. However, there's a much deeper reason that this encoding size does not actually achieve Shannon's limit. This
[00:20:00] quantity the negative log P that we wrote down is not actually the Shannon information of the message because this value P is representing the probability of the text according to the model not the true probability of seeing that text say as sampled by what a typical sender might send. In this sense the final compression rate depends almost entirely on the quality of the language model plugged in and how close its predictions are to the true statistics of language. And this is actually a nice moment to step back and reflect on how one benefit of learning about this whole algorithm is that it offers an interesting perspective on how language models are even trained in the first place. So if you look at this key conclusion we have that the compression size looks like the negative log of the probability that the model gives to a given piece of text, you can expand that probability as a product, namely a product over all the individual probabilities for each successive prediction that the model makes. And whenever you see a product in a log like this, it's always nice to break it up as a sum. In this case, the
[00:21:01] sum of the negative logs of all those individual predictions. Now, you can think of each one of these terms as something like a cost associated with one of the model's predictions. When the model gives a low probability to the true next character, the cost is very high. Whereas, if it gives a high probability to that true next character, the cost is low, approaching zero. I imagine this sounds very familiar to many viewers because in the last chapter we talked all about how models are specifically optimized to minimize this value or rather the average of this value over all tokens that they see during the pre-training process. In fact, the average of this value over all tokens sampled from language even has a special name. It's called the cross entropy. More specifically, it's the cross entropy of the models distribution of language against the true distribution. Now the point I want to make is that there are two ways you can think about pre-training. The typical framing says that the goal is to create a next token predictor and this cross entropy acts as a loss function. It is
[00:22:02] the way to turn that goal into a numerical score that can be minimized. But through the lens of this algorithm that we're talking about today, a mathematically equivalent but I think psychologically distinct framing is to say that the objective is not prediction per se. It's to create the best possible text compressor. In that first framing, that use of cross entropy initially feels like a design decision. You know, you have to choose some loss function and it's one choice out of many. But in the second framing, it's much more obvious what number needs to be minimized. It's the number of bits that each training example gets compressed into. Simple as that. I don't know about you, but I just found this a much nicer way to think about what's really going on during pre-training. What this means for the story of the compression algorithm is that it's only as good as the prediction model you plug into it. And we can actually have some fun here. Let me show you two different encodings of the same example phrase. Namely, the concept of proof and its associated mathematical rigor began in ancient Greek mathematics exemplified in Uklid's
[00:23:00] elements. This first encoding comes from nano GPT, a small 3.7 million parameter toy model that we've been using throughout this video for the sake of illustrations. It uses 254 bits in total, which averages to around two bits per character, which is double what Claude Shannon estimated should be possible. And I want to draw your attention specifically to the last two words. The thing is, if you have a background in math and its history, those last two words are highly predictable. Now, by contrast, this is what the encoding looks like using Quen, an open- source model with half a billion parameters. Notice how those last two words now only cost five bits. And the overall average is less than one bit per character. As an even more extreme example, let me bring back that preamble to the US Constitution. After that first phrase, which itself uses about 35 bits, everything that follows is so predictable that it hardly adds anything at all, only 10 bits in total.
[00:24:00] Contrast that with the Huffman encoding I showed at the very beginning of the video, and that is a two order of magnitude improvement. This one is especially fun to see in the context of our diagram. Quen processes tokens instead of characters, meaning there are many, many more options in each probability distribution. And zeroing in on that first token, Wii, seems to cost about 11 bits. And likewise, getting to that second token, the takes another dozen or so bits. People is a little more predictable, and there's a bit more zooming in still as we fill out of the United States. But after that initial phrase, notice how each new token dominates almost all of the probability space. Meaning we only very rarely have to update the actual real number encoding our position. If you look carefully, it seems like most of the decisions come down to capitalization and spelling choices. For example, the choice to spell defense with a C rather than an S costs us a couple bits. But at a high level, high predictability really has turned into high compressibility.
[00:25:02] Hang on a second. I hear some of you saying, "Isn't it kind of disingenuous to praise an algorithm for compressing this passage so much if it's just calling out to some massive model that's basically memorized that passage somewhere in its hundreds of millions of weights?" And yes, absolutely. In fact, hold on to that thought. We'll come back to it in just a minute. A second practical concern that I imagine is niggling in the minds of some viewers is how if you really keep up this process and you keep zooming in, it is very quickly going to become problematic if you need some ultra precise representation of real numbers. Not just the one real number that we're encoding, but also all of the other values delineating the bounds of each successive probability distribution. And indeed, a naive literal interpretation of what these diagrams show would not yet be a practical algorithm. The idea, as I've shown it so far, dates back at least to 1960 with some unpublished work by Peter Elias, but it wasn't made practical until 1976
[00:26:01] when Jorma Rousan and Richard Pascow independently developed a workable version, one that avoids a need for ever growing numerical precision, giving what's known today as an arithmetic coding. The core idea involves renormalizing the interval you're looking at at each step to always have a length of one. Pretty much analogous to the way that we've been animating it throughout the video to always fill the width of the screen. I'll leave links in the description for anyone curious about implementation details. The upshot is that arithmetic codes offer a nearly perfect way to turn a predictor into a compressor. This is the structure used for all of those Hutter Prize winners that I referenced, where improvements generally came from plugging in better and better predictors. I would be remiss not to mention that there is another completely different approach for turning a predictor into a compressor like this, published in 2013 by Yaric Duda. It's known as an asymmetric numeral system. It's also very beautiful and it avoids any of these precision issues with real numbers by instead encoding a message with a
[00:27:00] large ever growing integer applying a few other tricks as well resulting in a much faster overall encoding process. And if we ended things right here, I think that's a fine lesson. Arithmetic codes are very beautiful on their own. They make a core idea from information theory a lot more concrete and they offer an alternate perspective on how to even think about language models both at inference and in training. But this is part three in a trilogy that I gave the title compression is intelligence. [music] Now this phrase is something that's hard to analyze rigorously. Intelligence is very illdefined. Lots of people will disagree about what precisely it means. I think the better way to engage with it is to treat it like a proposed definition of intelligence. Suppose you wanted to give a mathematical foundation to this otherwise very squishy in human concept. Is compression alone really enough to carry that burden? By analogy, I think of it like when Newton gave the word energy a mathematical meaning. This
[00:28:01] measurable quantity with units of kilogram m/s squared. When he made this definition, it was reductive. It misses out on a lot of what people mean when they use the word energy colloquially. And yet, despite this more limited scope, it was still productive for him to make that definition. And productive though it might have been, it still managed to unify a number of seemingly disperate concepts. Similarly, any attempt to give a word like intelligence a mathematical meaning seems bound to miss out on some of the uniquely human nuance that our everyday usage bakes into that word. But the question is whether any such definition, even if reductive, is still productive and manages to unify seemingly disperate aspects of intelligence. What I want to close with here are three specific ways that compression captures part of what I think I mean when I use the word intelligence where it plays into the topic for today is that in all three cases, having an algorithm like an arithmetic coding freshly in our minds
[00:29:00] makes the connection more concrete. So number one, compression and prediction are really two sides of the same coin. And prediction in turn seems a lot more clearly related to intelligence. Arithmetic codes show one half of this equivalence. Specifically, they offer a nearly perfect way to turn an arbitrary predictor into a compressor. [music] And you can go the other way around, too. Suppose you have an arbitrary text compressor and you want to use it to make predictions about what might follow a given passage. I won't go into the full details, but here's the basic idea. for any potential string that might follow. You ask how much larger the compressed message would be with that addition versus without. If you let delta represent the difference between these two, then 2 to the negative delta gives you a pretty good approximation for the implied predictive model's guess about how probable that next string is. Doing this over all strings in a given token vocabulary, for example, could give you a probability distribution analogous to the type we get out of modern language models. So, in the sense
[00:30:01] that predictors can be turned into compressors and vice versa, compression and prediction really are equivalent. And in so far as good prediction seems to require having a coherent model of what's really going on, this hints at the idea that compression ratios may be a good proxy for intelligence. This also ties to an influential theory in modern neuroscience known as predictive processing. It posits that the brain is continuously generating top- down predictions for sensory signals and then what gets passed back up from the sensory organs is not the full information of what they received. It's just the error between the prediction and the sensed reality. The upshot of this theory is that much of what we experience as intelligence may just boil down to neurological predictions and compression. But there are aspects of intelligence or at the very least what I think I mean when I use that word the prediction alone would miss. Personally, I feel like I understand something better when it becomes smaller in my mind. When a handful of facts or even
[00:31:00] just one are all I need to remember or rederive a wide swath of results. Progress in fields like physics also feels like this. Having fewer shorter laws feels like a step forward even if they make identical predictions. This becomes more measurable in a machine learning context. If you have two models with equivalent predictive abilities, but one requires trillions of parameters and the other is small, it suggests that the smaller one is relying on fewer memorized assumptions. Now, if you plug each one of these into an arithmetic coding, if we assume each has equivalent prediction ability, that would mean that they compress a given large data set, say a gigabyte of Wikipedia, down to the same size. After all, the whole point of this video so far is that that compression ratio is dependent only on the prediction. However, an honest account of the final compression ratio here should really take into account the size of the decoder program, which in this case includes the model itself. After all, if you actually wanted to send somebody a gigabyte of Wikipedia in a compressed
[00:32:01] package, the raw data stream coming out of that arithmetic coding would be useless if you didn't also give them the means of decoding it. This is, as I referenced earlier, how the Hutter Prize scores its results. If you didn't include the decoding program size in this, you could basically cheat by having such a decoding program just include a hard copy of all of Wikipedia in its source code and then the supposed payload could be arbitrarily small. If earlier it felt like cheating to have a huge model kind of memorize the US Constitution and use that for prediction, this right here is how that would get accounted for. When you measure things this way, it means a small model with equivalent predictive capacity to a larger one would count as better compression. And this at the very least rhymes with the idea that a clearer understanding uses a smaller number of general principles instead of a large number of memorized facts. And that to me is what makes the phrase compressionist intelligence a lot more thought-provoking than prediction is intelligence would be. A third aspect of
[00:33:01] what I think I mean when I use the word intelligence is an ability to learn new things quickly. The smartest people I know are able to dive into a new field with relative ease. And again, at first glance, this would seem not at all to be captured by a compression ratio. If we stay focused on our example of a predictive model plugged into an arithmetic coding, even if you measure things the way the Hutter Prize does where you include the size of the decoder program, nothing about that final compression size depends on how long it took to train that underlying model in the first place. Except there actually is a case where learning rate influences this final score. Suppose you start with a setup where the underlying predictive model was trained on some specific data set like Wikipedia, but then later you want to apply the same encoder and decoder to a different kind of text, maybe encoding all of Shakespeare. Since the underlying model was trained in a different context, you would expect it to perform worse in this different setting. Certainly worse than a model that was tailored to Shakespearean English. But here's a
[00:34:01] clever way that we can tweak the algorithm. Rather than having the model stay static during the full compression process, what if you fine-tuned it on the fly? You let it adapt to the particular tokens that it sees along the way. Concretely, this would mean with each new token it sees, you would apply back propagation to tweak all of the weights based on what it saw. What this would mean is that over the course of the compression process, the model would get better and better tailored to that new context. In order for this to work over on the decoding side, you would have to have the decoder do exactly the same thing. As long as all of the updates are deterministic, [music] the way that the model would get updated on the receiving side would be an identical replay of the way that it was updated during encoding. This is important because the predictions at each step have to stay identical. This is actually an idea used in practice. And in fact, you can avoid the need to send any parameters at all if you have the encoder and decoder start off by generating a completely random model, but pseudo random in a way that's
[00:35:00] identical for both of them. In either case, when you do this, the final compression rate depends not just on the static predictive capacity, but also on how quickly the model is adapting to the new context. That is, the better the learning rate, the better the compression. So zooming out, if we look over the full trilogy, the conclusion from part one was that when Shannon began studying compression rigorously, he was forced to engage with intelligence. Pure data analysis was not enough. He could only estimate the compressibility of English by surveying human predictions. The conclusion of part two was that the modern engineering for artificial intelligence uses mathematical tools that were originally developed to study compression. Together, these two alone hint at a connection between compression and intelligence. It would appear that studying one forces you to engage with the other. For this third part, the upshot is that one of the best compression algorithms we have, say, as measured by the Hutter Prize, requires using a model which is effective at
[00:36:00] prediction while being small and which also learns quickly. [music] So, are compression and intelligence just really the same thing? The safe answer is of course no, [music] just as Newton's definition of energy is not the same as Webster's. But if you did need a mathematical definition for intelligence, my own takeaway here is that I was just genuinely surprised that compression ticks off many more boxes than I would have guessed the first time that I ever thought about the seemingly innocent task of making files smaller. If you enjoy these videos and you want to support the team behind them, there are three main ways you can do that. If any of you are looking for a career, we have a set of partner organizations actively recruiting and aiming to hire people like you. You can explore them at 3b1b.com/talent. My main aim with that page is just to give you a little glimpse of each team's culture and what made each one stand out to me enough to recommend them. There is also, of course, the Patreon, a long-standing backbone of the channel, where channel supporters get early views of new videos. That has always been
[00:37:01] helpful for me in getting feedback for the final edits. And last, there's the store. And there's a couple fun items that I just added in there for the holidays, like a vinyl record and a fractal mug. So, take a look if you're interested. >> [music]