Transcript — Dwarkesh Goes Inside Jane Street's Latest AI Data Center

Raw auto-generated transcript via yt-dlp. Stored for internal reference only; not for publication.

Today I'm getting a tour of one of Jane Street's training data centers here in Texas from Ron Minsky who co-heads the technology group and Daniela Corvo who heads the physical engineering team. Thanks for coming out here to do this.

Thank you. Cool. Let's get started.

So here's one where we have our training cluster of GB300s and VL72. Are you allowed to say what is currently happening on this cluster?

What actual training jobs are running right now? high level.

We do a bunch of different kinds of models. Some of the models are for training LLMs. Some of the models are for training all sorts of custom architectures that are more adapted to the trading problems and the trading data sets.

Earlier you were explaining to me that this was originally not a facility that was built to handle 200 kW racks and so you had to retrofit it to be liquid cooled and everything. You know, these cabinets, these GB300 cabinets consume at peak about 140 kW. Compare that to traditional air cooled, you're talking about 10 to 40 kW. So, a lot more. So, on the perimeter here, you can see some of this air cooling equipment here that would have fed a traditional air cooling. Some of it remains. We use some of it for the air cooled load that we have. About 15% of these cabinets are air cooled. And here inside the GPU, you can see how the cooling kind of flows from these quick disconnects in the back, taking a fluid in, routing it to these cold plates that are sitting on top of the GPU, and then coming back out at a warmer temperature. So when you slide this sled in, it automatically connects to liquid supply and return and 54v power. Right? So these just slide in powered and cooling all within the sled. There are still some components in here that are air cooled, but about 85 to 90% is cooled via those cold plates of the heat loads.

To what degree have we had concerns about leaks, right? I feel like you spent decades worrying about not having water in our data centers and now we're like putting it on on purpose. Like how big of a deal is that?

So inside they have something — there's underneath there's these ropes that detect leaks. So there's a management side of the switch of this server that will send out an alert if they sense a leak in there. Furthermore, there's leak detection underneath the floor. If it drips and falls under the floor, we'll be able to sense it there and isolate via a valve. But it is true that if something here fails, you are at risk of destroying the server. How often is there a leak? Not often, but this stuff is new. You know, it's yet to be seen over time how this works out. So, I guess it's surprising to me that it was not a problem to get liquid cooling going in a facility that was originally built for air cooling and lower power densities. I don't know. Why did it work?

That's how you define problem. I mean, it was an engineering challenge.

I feel like you're hearing the version of the story after all the problems have been worked out. These guys have spent a huge amount of effort figuring out — like this place was built in a kind of intermediate point when we knew we had to scale up a lot but we didn't know what the shape of the coming compute was. And so I think one of the things that the guys here did really well was take to heart the importance of optionality of like oh yeah there's a lot of different futures and we need to build some stuff that will work for multiple different ones. And I think that worked really well but required a lot of hard thinking and planning to make it land well. You know, one thing I'll say is there's a couple of ways to do liquid cooling. So we have fluid coming from the roof, from the chillers on the roof down here, maybe about 18° C. We use the same fluid for the air cooling, so it's fungible within the data center. We can move the fluid around.

Oh, that's nice.

So what we do is we send this in this device here, make sure that every single cabinet has the right amount of flow. You don't want the ones at the beginning of the road to receive too much flow and the ones at the end to be starved. So what these devices, these valves, they control how much fluid goes to each cabinet so that they're balanced.

How do they measure whether they have the right amount of fluid?

Ultrasonic flow meter here. So ultrasonically, it's measuring the fluid and measuring how much flow in liters per minute and capping that at some rate that we predetermined based on the heat load that it's rejecting. That liquid comes in, it goes to this heat exchanger here. So inside that heat exchanger is a heat exchanger that transfers heat between this building loop, building cooling loop, and a what we call a technical water loop inside there which needs to be very very clean and filtered down to 25 microns so you don't plug the cold plates on the GPUs. You want very good efficient heat transfer there. It's filled with a liquid — a mix of distilled or deionized water and propylene glycol — 25% of propylene glycol. That's to inhibit any bacteria or algae growth. That bacteria could grow in there if you don't have the right ratios and plug the cold plates and plug the heat exchange between the GPU and the cold plate.

I don't love the world where we have to worry about bacteria growing in our servers.

So you have leaks, you have bacteria, all these different new things to worry about. Making sure you have proper flow between all the devices. Air cooled data centers — you put the cabinet in and just flood the room with air and transfer that heat.

Was there just an area underneath that you could have used?

Raised floors traditionally were used to supply air. So there's ways you could supply air — that air would come out here. A lot of the new solutions are going to overhead piping because it takes time to build these raised floors and it slows down projects. So a lot of the piping systems going above overhead now for speed of deployment. But we like this here because you see that blue wire there. That'll sense a leak. If one of these connections is dripping, it'll touch that rope there and send a signal to say that there's a leak. So we're able to contain a leak under the floor and measure it where overhead it's kind of right into your data center. So we have 4,032 GPUs here in 56 racks. What we try to do on the power side is make sure we balance our power. You can't overload certain areas. So you can see how we have this busway here distributing power. And we're very conscious about how many racks are on each bus. Make sure you don't go over amperage and trip breakers. And you could be in the middle of a training run and overload a breaker and you'd have to go back to some bookmark.

What is the hourly price on a Blackwell rack?

There's two ways of pricing it, right? There's how much does the hardware itself cost and the power and all of that. And then there's the opportunity cost.

That's right. And we actually think about opportunity costs when we think about all of this compute stuff very intensively. And because we're in a world where compute is relatively inelastic, you end up in places where there's a real crunch even internally where it's like — it takes time to get new compute online and available and you can get to a case where people are all kind of bidding for the same compute and you're like wow it just becomes incredibly expensive because the stuff that we get out of this is super valuable to the business. And so the opportunity cost tends to dominate the hardware cost even though the hardware costs are not small.

Yeah. So that's the other question. If this facility is connected to the grid and you presumably had asked the grid beforehand for a certain amount of power and now you move to much denser compute. How are you — and this is still using the grid, it's not behind the meter — how are you able to get the power in here? So because the compute is moving — what if we have some power allocated from the utility, we end up just using a lot less space within the data hall. So everything just gets smaller and you can see in this data hall it's got a lot of space that we don't need. So you're trying to respect that power capacity you have from the utility — whether it's utility or behind the meter you still have to respect that overall value but you want to ride as close to it as possible.

That's why you can afford to put a podcast studio in this. Although there are reasons to want higher density like the networking setups themselves are incredibly complicated and like — I don't know — one thing I always feel when I go into one of our data centers is what a beautiful job people have done getting the wiring right. That's actually quite hard and the more you have stuff spread out the harder it is to get all the wiring done. And it's also worth noting most of the wires you see here out of the cages are fiber but the stuff inside — the fastest stuff is all copper. Light moves more slowly in fiber than electrons move in copper. So you really are at many different levels optimizing for the latency of all of these networking rates.

There's about 8,000 km of fiber in this deployment. And here you go. This is what happens when you go really dense. But you do have enough power to fill everything there out.

So it depends how we move power around. One of the ideas of being flexible and fungible with our power and cooling is that we overbuild our distribution. So while we're limited on the upper end, you're able to move power around by loading up different rows, right? So we have these UPS's that supply power to our site, but when we distribute that power out, if we could move it around, it allows us to say "Hey, we're going to grow some CPU here or we're going to grow some GPU here." So we have some headroom in there to do other things. And this is just opportunity areas for us to do that. We want to be ready to go if the business needs some more compute. We have a place to put it. So you do some amount of pre-building for future opportunities that come up.

So what are these things?

So these are breaker panels. These distribute out to those bus bars that you've seen. So this is where power comes in and you're going to have to break that power out to go in different paths. So there's a lot of distribution. You can see all this overhead conduit that we had to put in. This is all carrying power cables out to the data hall. So you're very conscious when you're distributing power. It's less fungible than cooling. With the cooling, you can kind of oversize the pipes and move it around. With power, you have breakers and current limits. So you have to be very careful with what you load up and where and making sure that you don't end up in a situation where you're tripping a breaker. We have protections in place. So if we trip one of those four buses, we're still good. So there's some redundancy. But still it's an interruption something we want to avoid.

What would cause you to trip a breaker?

Just high current. So adding too much load on a single busway or single connection or pushing too far in the oversubscription and saying oh well you know we think we're going to be oversubscribed by 10% and actually things kind of shoot up to 15% or 20%. There are times where you do have time to respond. But if you're too far over the limit, the breaker's going to trip.

And is the current pattern determined by software or by hardware alone?

It's by hardware. It's controlled somewhat by software, I think. So what Nvidia is doing, they have this kind of LPS systems. It's a load management system that they're rolling out in some of their new cabinets. Really, what they want to do is keep that load profile flat. They're building in more bulk capacitance in those power shells, so capacitors in there. And they're also trying to get the software to allow the peak load and the average load to be much tighter. So you have this flat profile.

A place where software comes in is monitoring. We actually put a huge amount of effort into building our own monitoring tools so that in one pane of glass — they like to say — we can monitor every aspect of the system and look for problems and sometimes even drive reactions to the system where sometimes we might want to shut off a workload if it is drawing too much power. And having like a unified system that can see all the things has been a huge step up in being able to run these things in a highly reliable way.

So that software system Ron mentioned is actually pulling information from these breakers and performing some logic behind the scenes. It's topology aware and what it will do — like Ron said — it can shut down nodes so we don't trip that.

Because you want to run as close to the edge as you can because the hardware is incredibly valuable. So you do want to oversubscribe. You do want to run near that edge but you also need to be safe. And so building these safeguards to let you pull back in a controlled way.

If somebody switched one of these switches, would some training workload stop right now?

Yeah, I mean this site's been —

This site is currently live. So yes, so here you can see a little bit of the scale of our liquid cooling. So these are — we call them buffer tanks. So they're here to help with a situation where maybe there's an interruption in power and the chillers on the roof restart and lose cooling. This is almost like a thermal battery storing some energy for us to keep those GPUs cool while the chillers come back online. Also, as these workloads come up and down, you're going to have a movement of temperature. So this helps dampen that effect out. These are the traditional air cooling units that you'll see in almost every air cooling data center. Really just pulling that hot air back and supplying it back into the data hall. Those wheels up there are valves. So you mentioned leaks, right? What happens if there's a leak? Well, you have places where you can isolate the system to work on it to fix a leak or something. These orange things just have more chain in them. So they're not hitting you in the head now. You can reach them with a ladder, take more chain out and turn the valve closed or open.

That's great. Such a simple solution. You know, it's interesting as these data centers and the compute gets so dense — where the place where you have the compute is getting smaller and smaller and the places where you're supporting that compute is getting larger and larger. The infrastructure, the transformers, the chillers are getting bigger and bigger. So the sites are very much "this much compute and this much infrastructure to support that compute."

What did the analogous system to this 20 years back look like?

So yeah, it was dramatically more primitive. Early on we literally just had like computers sitting out with us in the room with all the people. One of our compute clusters we called the Hive and I remember the first version of the Hive was literally like six Dell boxes stacked on top of each other at the end of the row. That's why they're called Hive box.

Yes, that's right. Yeah. Like Hive was the name — actually the first work that I did at Jane Street was doing quantitative research on trading strategies and I was like oh yeah I guess we need a cluster and that pile of six boxes was our first cluster. And the trading systems themselves we also had there and that actually was more important to have out with us, and it took us time to convince ourselves it was okay to actually go and put it in another room in a rack because we actually wanted the ability to make sure we could turn the damn thing off. Like if something went wrong, just the comfort of "I could unplug it" was there. And it took time to convince people that we had enough control over the systems and we understood enough about where things were and would be able to find them if we had to go in there — that things were cleanly labeled enough that we were comfortable taking these things and moving them in the back.

Yeah. I mean, there were ups and downs. Literally at some point one of the people who was cleaning the office unplugged one of the trading systems in the middle of the day as they were vacuuming. So in the end it is in fact better to have it all in a data center but I don't know — early on it was more of a shoestring operation and we were just kind of figuring things out.

But did it not need to be even back then colocated with the exchanges? So an important thing to understand about our early trading is — we were super not fast, right? Like trading latencies matter at lots of different orders of magnitude and sometimes it matters whether you're responding in seconds or milliseconds, sometimes microseconds. Like these days the very fastest systems we care about — you're talking about whether you can turn around a packet in under 100 nanoseconds.

Yeah okay I definitely want to ask you more about that when we get to the podcast. Yeah interesting.