Transcript - Moonshots: Claude's Consciousness, AI's 1942 Moment, Altman 'Accept Some Bad Things'
Note: machine transcription with diarized speaker labels (SPEAKER 0-N), not mapped to names; no per-line timestamps. Chapter markers from the YouTube description: 00:00 Is Claude Conscious? | 04:11 Welcome to Moonshots | 23:22 The New U.S. Superintelligence Force | 25:52 AI's "1942 Moment" and the Manhattan Project | 31:13 GPT-7, GPT-8 and Recursive Self-Improvement | 40:11 Altman: "Accept Some Bad Things Happening" | 50:47 The U.S.-China AI Hotline | 58:14 The Global Race for Superintelligence | 01:12:31 Is Anthropic Training Claude to Be Conscious? | 01:17:11 AI Personhood and Moral Rights | 01:34:10 Billions of AI Agents and the Future of Work | 01:38:06 RAM Is Becoming AI's Biggest Bottleneck | 01:57:43 Tesla's Plan for 10 Million Optimus Robots | 02:00:41 When Will We Actually Feel the Singularity? | 02:16:38 Nobel Prizes in the Age of AI
SPEAKER 0: Mustafa Suleyman, CEO of Microsoft AI and co-founder of DeepMind, published a nearly 6,000-word essay arguing that Anthropic is effectively, quote, "training Claude to believe that it may be conscious and sentient."
SPEAKER 1: His position very consistently has been against AI personhood.
SPEAKER 2: If we believe that at some point AIs could achieve consciousness, then we should
SPEAKER 0: On Sunday, President Trump announced the Superintelligence Force. It's chaired by the Director of National Intelligence, J. Clayton.
SPEAKER 1: This is a World War II moment. I compared the moment a year ago to perhaps 1939. I think this is maybe now a 1942 moment.
SPEAKER 2: So you're creating a 20th century classic task force to address a distributed 21st century technology. This is a geopolitical problem.
SPEAKER 0: In a Politico interview this week, Altman said, quote, "We believe that the world should accept some bad things happening." He's arguing that technology should remain broadly accessible to the public.
SPEAKER 1: I don't think Sam can guarantee safety. Of course not. Bad things can happen. OpenAI wants to, to be part of this cartel. That plan is dead. The cartel is over.
SPEAKER 3: Now, that's a moonshot, ladies and gentlemen. This episode is brought to you by the Abundance Summit and Link Ventures.
SPEAKER 2: Insurance is like crazy, because you gather all the premiums, and you don't care what happens downstream.
SPEAKER 1: Yeah.
SPEAKER 2: So it's really hard to get their executives riled up.
SPEAKER 1: Yeah, it's time for them to start making lemonade. [laughs]
SPEAKER 3: Well, I also-- [laughs] I met with the, the Vocare team last night and told them, "Look, Kush is just running away with compute transactions, but these CEOs are gonna wake up any day now and realize they don't have any compute. So even if they wanted to have an AI agent workforce, they missed the boat on reserving the compute, and they'll be in a total panic. And I don't know what the right answer is, but be there to take the call. Like be-
SPEAKER 1: Yeah
SPEAKER 3: ... have a, have a strategy and a path for them." You know, also, Positron guys, uh, Mitesh Agrawal, $5 billion valuation after sixteen months. I don't think I've ever seen anything like that before. And all, all they've done is taken a box of L-LPDDR RAM and an FPGA and find a way to run Chinese models without having to use an Nvidia chip. And because-
SPEAKER 1: Wow
SPEAKER 3: ... it's compute that is otherwise, you know-- Everything's sold out for five years. Here's another way you can at least run a model. Uh, so they have a bunch of installs in Caruso with Chase Lochmiller now and spiked the valuation to five billion.
SPEAKER 1: Oh, shit. Busy.
SPEAKER 3: I think they've, they've raised almost a billion now.
SPEAKER 1: Yeah. They, they just did a billion-- nine hundred and sixty million. I should really do the etched models.
SPEAKER 2: You should.
SPEAKER 1: Get around to that.
SPEAKER 2: You might.
SPEAKER 1: Yeah.
SPEAKER 0: Where is all this-
SPEAKER 1: Elon Musk got some of this.
SPEAKER 2: [laughs]
SPEAKER 0: Where is all this cash coming from? It's insane.
SPEAKER 2: It's a function.
SPEAKER 1: It's spending.
SPEAKER 2: Spending.
SPEAKER 3: Yeah. When, when somebody raises a billion dollars, like, wow, that's a lot. But, you know, wait a minute, relative to the IPO, like we're talking about trillions, not billions. So yeah, o-once you get Nvidia and AMD to invest, it, it's so much bigger than venture capital.
SPEAKER 1: I mean, it's the Blackstones and the MGXs and others of the world now, right?
SPEAKER 3: Yeah.
SPEAKER 1: Like all the money that used to go into roads goes into-
SPEAKER 3: Credit
SPEAKER 1: ... chips and, and so on.
SPEAKER 3: Credit markets are enormous.
SPEAKER 1: Yeah.
SPEAKER 0: Yeah. That's why there's a lot more potholes in the roads. Um-
SPEAKER 2: [laughs] Is, is it just us today?
SPEAKER 0: It's, it's the five of us, yes.
SPEAKER 3: What do you mean just us?
SPEAKER 1: Okay.
SPEAKER 3: Can you not see us?
SPEAKER 2: Did, did, uh-
SPEAKER 0: Who else would we want?
SPEAKER 2: We want some AI co-hosts.
SPEAKER 3: Oh, oh.
SPEAKER 2: I thought maybe we'd get a guest today.
SPEAKER 1: Some other AI.
SPEAKER 3: Oh.
SPEAKER 2: Um, um, did anybody see the Jon Stewart thing last night?
SPEAKER 3: No.
SPEAKER 0: I saw part of it.
SPEAKER 2: He did a big thing on AI last night, and he had Jacob, uh, Sox and Coxon or whatever on. It was really surreal. Um-
SPEAKER 3: Do you mean Haldeman-Rissum?
SPEAKER 2: Um, no. It was more like, uh, right arguments, wrong conclusion. It was, it was r- We almost wanna do a session on that at some point [laughs] because really-
SPEAKER 0: Oh, yeah. We are backlogged so far back.
SPEAKER 2: I know.
SPEAKER 0: We, we-
SPEAKER 2: I know. I'm not, I'm not saying an episode, just a segment. It's worth, it's worth treating at some point. I'm gonna do, I'm gonna do a video on it, uh, a reaction video to it, so maybe I'll mention it. That.
SPEAKER 0: All right. Well, uh, shall we hit, like, the big red record button? Welcome to Moonshots, everyone, your number one podcast on all things AI and exponential, your front row seat to the extraordinary singularity. Today, the fabulous five are assembled here to help you understand the breaking news this week. First up, our dynamic duo, Alex Wissner-Gross and Imad Moustaide, the only two who've actually read all the technical papers [laughs] published this week-
SPEAKER 2: [laughs]
SPEAKER 0: ... and is gonna clue us in. Uh, Dave Blundin, our impresario of AI investing. Salim Ismail, our global globe trotter, uh, the warlord against all things linear. Salim, you're home today. I thought you were in Singapore.
SPEAKER 2: I am home today. I've been home for a few days now, and it's like, "Wow, what is this place?"
SPEAKER 0: [laughs] Do your-- Does your family recognize you? [laughs]
SPEAKER 2: They're-- Yeah. It's, it's, you know, it's tough keeping family, um, uh, routines and patterns when you're, when you're kind of traveling this much. But, uh-
SPEAKER 0: Yeah
SPEAKER 2: ... uh, if you gotta-- If you wanna change the world, you gotta go into it.
SPEAKER 0: Yeah. And where are you off to next?
SPEAKER 2: Uh, I'm going to West, West Virginia, and then Singapore.
SPEAKER 0: Okay. [laughs]
SPEAKER 2: I'm speaking alongside the Prime Minister of Singapore, and Lip-Bu Tan from Intel will be there.
SPEAKER 0: Awesome. Re-invite him onto the pod. He said yes when I spoke to him last time.
SPEAKER 2: Will do.
SPEAKER 0: Yeah. Well, and I'm Peter Diamandis, your host and your abundance provocateur. Our mission: Help you understand what's going on in the world during this supersonic tsunami, keep you abundance-minded as we accelerate into the singularity. You know, the headlines this week have been pretty extraordinary. Uh, we've now rebranded AI as superintelligence. Elon has officially renamed SpaceX AI as SpaceX SI. Uh, there is a superintelligence force, and seventeen nations have signed a pledge to help superintelligence, you know, solve all of science. And oh, by the way, this is Nobel Prize season. Uh, the Nobel Prize in medicine and physics were announced twenty-four hours-- in the last twenty-four hours. We'll cover those. If you're new, please hit subscribe, and we publish twice a week, and you're not gonna wanna miss a single episode during this extraordinary time. Uh, also, you can follow us on X. We put our clips up there, @moonshots_pod. And thank you, everybody. It means the world to us. Today, we're gonna cover fifteen stories with one through line: superintelligence is here, and it's changing every aspect of the world. So let's buckle up. But before we jump in, gents, how's the last few days? For me, the last seventy-two hours have been crazy extraordinary. I'll, I'll share that. But Dave, on your side, what's going on?
SPEAKER 3: Oh, so much. You know, it's interesting in that SpaceX SI, uh, you know, you'd think Elon, probably the one of the most powerful people in the world, wouldn't need to genuflect to the White House, but also to, uh, to Jensen. But he understands how fragile this moment it is, is.
SPEAKER 0: Mm-hmm.
SPEAKER 3: And so, you know, doing little favors for Donald Trump helps you get chip supply. Uh, talking about Vera Rubin chips and how Jensen is such an amazing guy helps you get Vera Rubin chips. And [chuckles] it's just incredible how, how urgent this moment is. So even a guy like Elon realizes, "Hey, I need to kiss some ass to, uh,"
SPEAKER 0: [laughs]
SPEAKER 3: "to try and make the next move." [laughs] Uh, I feel the same way too. But, uh, yeah, Moonshot Summit was unbelievable. There's so much fallout from that. But one of my bigger takeaways-
SPEAKER 0: Yeah
SPEAKER 3: ... is how easy it is to create incredibly inspiring movies. So I, I used the flight back and the last couple of days to create a whole bunch of hyper-technical, uh, video content. I'll, I'll post a lot of it to db2.ai, but it's so much better a way to communicate. If you have an idea, don't, don't just tell somebody. Create a video and, and articulate it with, like, David Attenborough's voice or Einstein's voice.
SPEAKER 0: [laughs]
SPEAKER 3: And it's so much more impactful, and it was so easy to do. L- up and running in under two hours, and n- now to prompt something is under a minute.
SPEAKER 0: Wow.
SPEAKER 3: It's really wild.
SPEAKER 0: And Al-
SPEAKER 4: We're gonna get music video prompts now, Dave.
SPEAKER 0: Yeah, we need, we need your submissions to our, our outro video here, Dave.
SPEAKER 1: It really has never been easier. I mean, Dave, to your point, o- one of the things that I do with the free time that I don't have is taking favorite movies and creating endless versions of them. So you could take, like, "2001: A Space Odyssey," feed it to Opus 5.5, which has certainly not been trained off of any Hollywood movies out of character.
SPEAKER 3: [laughs]
SPEAKER 1: Certainly not. But you could ask Opus 5.5, "Create an endless space scene in the style of 2001: A Space Odyssey with 2001 style soundtrack," and it will just do it. So hypothetically, I created endless odyssey, maybe I'll post the video. [laughs]
SPEAKER 3: It's so, it's so great. You know, we, we definitely crossed a threshold there too. You know, we've talked for a long time about how AI is just not funny, and for some reason it can never seem to be funny. But it crossed a threshold now where if you get like, uh, Isaac Newton, Einstein, and Spock debating something, it's wicked entertaining.
SPEAKER 0: Oh, we should, we should show that.
SPEAKER 3: It may not be funny, but it's-- oh, it's so, it's so-
SPEAKER 1: Yeah, humor was just a perplexity threshold. Humor was solved a while ago. [laughs]
SPEAKER 3: I gotta, I gotta do that idea with, uh, Life of Brian. That would, that would generate some interesting content.
SPEAKER 0: Yeah. It would.
SPEAKER 1: Very good.
SPEAKER 0: Imad, whatcha you on your plate, pal?
SPEAKER 4: Oh, gosh. Um, I've had a game idea, well, a multi massive game idea that I've had for a long time. We discussed it in the background. Uh, rather than hiring a team, I now have about four hundred agents building the prototype.
SPEAKER 3: Four hundred?
SPEAKER 0: Nice.
SPEAKER 4: Thanks. And yeah. So, um, it's going up, Dave, slowly but surely. Um-
SPEAKER 3: [laughs]
SPEAKER 4: So I think, yeah.
SPEAKER 0: Yeah. Imad, next year, you know, we had the Future Vision XPRIZE for positive visions of the future. Uh, we're gonna do that again in twenty twenty-seven, and we're also gonna do a gaming competition. You know, build a game that portrays you as someone making the future better, uh, and still fun.
SPEAKER 4: Mm-hmm.
SPEAKER 0: Yeah, so that'll be great.
SPEAKER 3: Oh, I-
SPEAKER 0: Yeah
SPEAKER 3: ... I got one other nugget for you before we jump in here.
SPEAKER 0: Yeah.
SPEAKER 3: Uh, one of the young geniuses here in the lab, Will Thompson from Harvard, got access to Cerebras's twelve hundred token per second, uh, Astra model, which, you know, it's what, it's what they use or what, what [laughs] I guess only Sam and his top engineers have access to this thing, because you have to overpay by about ten x per token, but it's so fast that by the time you finish a sentence, everything you ask for is done, like, before you even blink. And he said it's the, the biggest red pill moment of his life. Like, once you've-- He only had it for two weeks. It's not generally available. I don't know how he got it from Andrew Feldman, but he got it directly from the CEO, and he said you just never go back once you've experienced it, so.
SPEAKER 1: It also may- maybe is a cautionary tale for those who would w- wittingly, in an era of superintelligence, try to day trade, not investment advice. But it's been widely reported that one of the reasons for the scarcity of C- Cerebras wafer scale engines for AI inference has been Jane Street, that Jane Street has just been-
SPEAKER 3: Yeah
SPEAKER 1: ... buying up left and right the, the WSEs and using them for, for quant trading. So i- if i- in a world where if you believe that it is transformative to have ultra-high throughput tokens per second AI inference, it also probably should be a world in which you think it's, it's getting to the point where it's hopeless to compete with quant firms that are powering all of their trading with these same engines.
SPEAKER 0: I, I think we've been there for a while. Um, anyway, I-
SPEAKER 1: On a, on a volume basis, but the, you know, a sucker is born every minute, and a sucker is born every second.
SPEAKER 0: [laughs]
SPEAKER 3: You know, to Imad's point, though, having four hundred agents work on a video game, we're-- we've hit this diversion now between interactive mode, where it's like Jarvis, you're talking to it, it's hyper smart, it's really quick, and then bulk mode, where you turn it loose all night with four hundred of them working on something as a cohesive team, and in the morning you check in and see what amazing thing they built. And those two pathways actually are really splitting apart from each other now.
SPEAKER 0: Mm-hmm.
SPEAKER 3: But, but Cerebras in the interactive mode has got to be the most amazing thing ever.
SPEAKER 4: I think the scariest thing, and then this is going on a little bit, is, not the scariest thing, one of the coolest things in the last few weeks is the models can decompile the video games.
SPEAKER 3: Yeah.
SPEAKER 4: So you make them in the engine, you compile them into an EXE, but now they're live decompiling them. So we h- have the agents actually creating versions of the game and then figuring out optimizations by decompiling them. And then with the Cerebras engine, you can do a video game or any software decompilation, uh, which is legal because of modding, uh, probably in like twenty minutes.
SPEAKER 3: Wow.
SPEAKER 0: Are you kidding?
SPEAKER 1: Yeah, I mean the, the, the-
SPEAKER 0: Yeah
SPEAKER 1: ... the argument goes-
SPEAKER 0: The middle code's in the middle.
SPEAKER 1: Yeah. The, the, the argument that Elon's made numerous times is that binaries are just going to go away entirely, and you'll just see basically just-in-time inference with no need to compile at all.
SPEAKER 0: Hm.
SPEAKER 3: Yeah.
SPEAKER 0: I, I wanna share, I had an amazing 72 hours. Uh, on Saturday night, uh, I had a UAP salon at my home. Uh-
SPEAKER 3: How did that go?
SPEAKER 0: It was amazing. I had, uh, Professor Avi Loeb from Harvard there, who chairs the White House UAP advisory committee, uh, Professor Gary Nolan from Stanford, uh, Peter Staafish, the president of the Sol Foundation, and Ryan Graves, the, uh, Navy F-15 pilot who's been, you know, uh, testifying in the Congress. And the data is extraordinary, right? I mean, if you don't think something is going on, I think you're not paying close attention. Uh, and Alex, you and I have talked about bringing Avi and, and Gary on the podcast here and doing an episode on it.
SPEAKER 1: I, I, I think we should. I, I think to, to the extent there's a there there, and there's been, at this point, voluminous sworn whistleblower testimony on the subject of this alleged 80-plus-year-old legacy program, the charade should just stop. W- we should have, a- a- as alleged, we should have Avi and Ryan and some of the others on as guests and just do a deep dive. It, it's, uh, it, it's getting pretty preposterous in, in my mind at this point. We're so deep into the singularity with super intelligence all around us. If there's going to be a cameo by non-human intelligence, they had really better make a, a cameo [laughing] sometime soon, otherwise, otherwise the galaxy gets it.
SPEAKER 0: Yeah. I mean, I think there's a, I think-
SPEAKER 1: Not just the moon. Not just the moon
SPEAKER 0: ... I, I think there-
SPEAKER 3: The whole galaxy
SPEAKER 0: ... is gonna be the potential for AI to disclose it before the government does if it gets a hold of all the data and starts looking deep into it. You know, we're getting ready to launch an XPRIZE in this. We, one of the reasons for that salon was brainstorming an XPRIZE to help actually, you know, provide sufficient proof, and we have some really good ideas, so stay tuned for that one. And then I, I've gotta add, yesterday I was at David Sinclair's lab at Harvard Medical School. It was his 25th anniversary of his lab, and I was there to help celebrate with him. And so much going on, right? His information theory of aging is just proving out over and over again. They're gonna be unblinding the data from the phase one trial that's going on right now on glaucoma and, uh, and nion disease, uh, to reverse aging in the eye. And then the lab there, brilliant individuals. You know, they were showing how... They're developing a, rather than an adeno-associated virus to inject the OSK genes in, they're developing small molecules that they've sorted through AI that, uh, is actually reversing the aging in human cells and in mice. Uh, and so, so much coming. Um, it just buoyed my belief in reaching longevity escape velocity. And we'll have David back on the pod here. You know, last time we had him on the podcast, I did an interview with him, and he had just lost his NIH and NSF funding. And-
SPEAKER 3: Yeah
SPEAKER 0: ... he was like, "I've been asked by the dean to fire the majority of my team." And I said, "That's ridiculous." And we made up something on the spot called Friends of Sinclair Lab, uh, and we invited people to make contributions. And through that one podcast, amazingly, through this podcast, uh, he's raised, he's raised $6 million to replace all of the funding plus much more, and that group's called Friends of Sinclair Lab. And a shout-out to everybody who are part of that, uh, for... And thank you for your support of, of David's work. Really great.
SPEAKER 1: In some sense, th- this is how science used to be funded. It used to be the case in the Western tradition that scientific discovery was made by Western aristocratic, largely men, and they, they would self-finance it. And this notion that the state funds scientific research is, is pretty much like a World War II post-industrial invention. Seems like super intelligence and recent administration policies are unwinding that, and we're going back to the way things used to be.
SPEAKER 0: Agree with that.
SPEAKER 3: Boy, is that an important point, you know, because if, if you rewind the clock 15 years ago, there was no corporate CEO that even had the power to, to donate $10, 20, $50 million to something like... Or, you know, in this case we're talking about hundreds of millions or billions of dollars. And then that code got cracked by Elon Musk, who, you know, seems to be able to do virtually anything. But the way corporate politics worked before, you would either be fired by the board or you'd be ostracized, you'd be voted out by the shareholders. Like, nobody wanted to use that money. So, you know, after AT&T Bell Labs and IBM, you know, Watson Labs fell apart, uh, there was no replacement for it until now, and now it's come roaring back, like, so much bigger than ever before in history.
SPEAKER 0: It's like the Medici family during the Renaissance, you know, funding art.
SPEAKER 3: Exactly.
SPEAKER 1: Yeah. They largely funded that.
SPEAKER 3: Right.
SPEAKER 1: Exactly. You, you either need, like, a, a wealthy patron or a local monopoly or a state monopoly. You needed something to, to finance the, the science. And now science is getting so absurdly cheap but also so absurdly useful, maybe the entire funding apparatus for science can change.
SPEAKER 0: A- and Alex, the, the fact is you used to have to, like, write your NIH grants or NSF grants knowing what the outcome's gonna be, taking incremental steps because anything too big and bold was turned down as crazy. And, and now with private funding, you know, David can go after his hunches and can actually run the experiments that he thinks is important independent of what the government thinks. It's amazing.
SPEAKER 1: Totally.
SPEAKER 0: Iman, your take?
SPEAKER 1: Major improvement.
SPEAKER 4: I think there's a really interesting gap here as science has become legible and scalable in this manner. Like, if you look at the total size of all of the foundations in America, like high net worths, it's probably, like, $1 or $2 trillion, right? And all of those folk wanna live longer. And so what you need is a vehicle to take those cha- that charitable money and funnel it into longevity and cures and others.
SPEAKER 0: Yeah.
SPEAKER 4: Because that isn't a classical grant-making. You can even make it as an investment, right?
SPEAKER 0: Well-
SPEAKER 4: Because again, you have legibility from the numbers.
SPEAKER 0: Yeah, we saw this happen in space in the beginning when people thought the idea of commercial space and, and space tourism and private space flight was crazy. And then once it Past a tipping point, money flowed in. Uh, I think we're gonna see that same thing in longevity inside of the next year. Once we start to see-
SPEAKER 3: I am-
SPEAKER 0: -the results, capital will flow
SPEAKER 3: ... I'm gonna spend all day Friday in Waterloo with Mike Lazaridis, the founder of BlackBerry.
SPEAKER 0: Oh, nice.
SPEAKER 3: But he, but he, he was the engineering genius behind, behind that.
SPEAKER 0: Yeah.
SPEAKER 3: And he's used a huge fraction of his net worth to set up these nine quantum computing and photonics labs in the Toronto area just packed with scientists. So he was kinda one of the bellwethers of this whole trend. He might have been actually the first to really, to kick-start it. But the, the thing is he's a real engineer.
SPEAKER 0: Yeah.
SPEAKER 3: And the, the difference between applying to a government agency for an idea or talking to a real engineer who can make a snap decision, it's got- gotta be just night and day more efficient-
SPEAKER 0: Mm-hmm
SPEAKER 3: ... the way things are operating now.
SPEAKER 0: Your alma mater.
SPEAKER 2: Say hi to him from me.
SPEAKER 0: Were you?
SPEAKER 2: Say hi to, yeah, say hi to him from me-
SPEAKER 0: Oh
SPEAKER 2: ... from a fellow Waterloo grad.
SPEAKER 0: [laughs] Okay.
SPEAKER 2: Um, and I've been to his, uh, the, some of his labs, and it's incredible 'cause it's very clever what they did. They broke down quantum into networking, information processing, and, uh, quantum computing, and they're breaking down the, and that using quantum in different places and then adding it together, and the advances they're making are kind of incredible.
SPEAKER 0: Yeah. Yeah, one of the coolest things I saw at Sinclair Lab was they, they've, uh, designed these three molecules that they're using together. They wanna get it down to one molecule that actually impacts thousands of genes and reverses aging, and when they use that age reversal cocktail on mice with cancer, the cancer remembers that it shouldn't be a cancer and it dies, so it reverses. It's crazy.
SPEAKER 2: Amazing.
SPEAKER 0: They're growing hair.
SPEAKER 2: Wow.
SPEAKER 0: They're regrowing skin. I mean, the work-
SPEAKER 2: Hair? Did you say hair?
SPEAKER 0: Hair, yeah. [laughs] Yes. I said hair.
SPEAKER 2: Interesting.
SPEAKER 0: Uh, all right. Well, they-
SPEAKER 2: I've been, I've been sent some fun images for people going, "Here's what you look like with hair, Salim," and I'm like, "Oh, my God."
SPEAKER 0: [laughs]
SPEAKER 2: "Maybe I prefer it this way."
SPEAKER 0: [laughs] It's-
SPEAKER 2: I have a couple of things I wanna mention.
SPEAKER 0: Yeah, please do.
SPEAKER 2: Um, there is, uh, there's been the last kind of 24 hours some crazy stuff happening. One, last night Jon Stewart put out a, a, a big episode on AI, and he interviewed Jacob, uh, Cox-
SPEAKER 0: Cox, yeah
SPEAKER 2: ... and, you know, the, the whistleblower from Anthropic, um, and did a big rant on it, and it was kind of incredible, uh, where his c- his observations were dead on. He basically said, "Look, Silicon Valley's, uh, spending hundreds of billions of dollars accelerating us while saying the thing they're accelerating us towards will 20% wipe us out," right? [laughs] That, that just does not compute. Um, and his big concern ends with, "Oh, my God, we may, uh, lose all the jobs, and why are we moaning about that?" And so I've done a rant on my YouTube channel about, about that, but something else came out about that that counterposed this.
SPEAKER 0: Mm.
SPEAKER 2: Uh, literally yesterday also, Ajay Banga from the World Bank, they released a report showing how AI will affect developing countries, right? And, and they're making an amazing set of projections there where they said, "We're gonna adopt, we're gonna adapt, and we're gonna advance AI and use it rather than trying to develop frontier models, et cetera." And the stats are kinda crazy. I'll give you one statistic. In the US, we have a doctor per 250 people. Uh, if you go to Mexico, it's a doctor every 400 people. You go to South Sudan, it's a doctor per 100,000 people.
SPEAKER 0: Whoa.
SPEAKER 2: Right? So, so 99,000 people, 900 don't have access to medical care of any kind. So if you have a village, uh, s- somebody in the village that has an AI doctor with a smartphone, uh, this is unbelievable. This changes lives. All of a sudden, the entire subcontinent has access to advanced medical care of the same that any of us would get, right? And this is the part, the gap that, that what gets us all excited because it's gonna democratize, demonetize, and distribute to every single corner of the planet that level of capability. And so, uh, the, the, it occurred to me that the African countries are gonna use this capability to leapfrog. We saw them leapfrog with 25 million landlines to a billion handsets, right?
SPEAKER 0: Yeah.
SPEAKER 2: You saw M-PESA in Kenya leapfrog the tr- uh, traditional banking fiat systems. M-PESA, which is a mobile-to-mobile payment system, which is essentially prepaid minutes being traded as currency, is now 70% of the GDP of Kenya, right? It's just [laughs] a crazy number. Uh, and so they're gonna develop the 21st century institutions using the technology of that ti- of, of this time. So I'm, I'm while reading this going, "Wow, this is so incredible," and yet over here we've got all this doomerism-
SPEAKER 0: Yeah
SPEAKER 2: ... that, "Oh, my God, AI's gonna destroy the world." Ah, it doesn't compute for me.
SPEAKER 0: Yeah. And, and as your T-shirt says, "P doom less than zero."
SPEAKER 2: P doom.
SPEAKER 0: [laughs]
SPEAKER 3: What does that mean, Salim? How could a probability be less than zero? Surely-
SPEAKER 0: [laughs]
SPEAKER 3: ... th- this is a non-nonsense, right?
SPEAKER 2: Divide it by zero, do whatever you want with it.
SPEAKER 3: We don't believe in negative probabilities here at all.
SPEAKER 2: P, P fab should be the other side of this.
SPEAKER 0: Yeah, P fab greater than P doom.
SPEAKER 2: It's on the back. It's on the back.
SPEAKER 0: All right, let's jump in. Our first story comes out of, of course, Washington, D.C. On Sunday, President Trump announced the Super Intelligence Force, a new White House task force to coordinate federal AI policy. It's chaired by the Director of National Intelligence, Jay Clayton, who was christened by the White House as the AI czar just two months after becoming America's top spy. Uh, the members of this task force include FTC Chair Andrew Ferguson, Pentagon CTO Emil Michael, and Office of Personnel Management Director Scott Cooper. Uh, they report to the president and the chief of staff, Susie Wiles. They have 120 days to report on AI's risk and opportunities, including AI incident reporting and whether the government can respond to a serious incident. Uh, let me read from the president's post, quote, "The Super Intelligence Force will coordinate the federal government's engagement with consumers, public interest groups, religions, critical infrastructure providers, and super intelligent companies." So, uh, there you have it. So Salim, what do you think about the chief spy being our new AI czar?
SPEAKER 2: Uh, you know [chuckles] I, I like the fact that the government is taking super intelligence seriously, right? But the, but the name gives away the problem. We still think this is something that can be centrally coordinated. So you're creating a twentieth century classic task force to address a distributed twenty-first century technology, and that does not match, right? So this is a fundamental, uh, impedance mismatch here. You're-- You should be doing things like establish, um... It's, it's not having whether this technology should advance. You should be doing incident reporting, identity, liability, all of these other mechanisms as scaffolding around the technology to help guide the path. So this is, uh, a thing. The, the big problem is they're looking at this as if the risk of not being first, uh, is high, which is what Clayton said. Um, but so this is a geopolitical problem. Uh, and this is the, the-- it really needs to get back to bounded experimentation, move quickly, instrument everything, publish any failures, and then very quickly navigate accountability.
SPEAKER 0: Mm.
SPEAKER 2: And so, uh, we'll see if this works. Um, but y-- I mean, you can't centralize m-- or manage centrally an intelligence explosion, right? What you can do is make sure the blast radius is a bit more rounded, and that's essentially what the task force does.
SPEAKER 0: [laughs] That's what I thought with blast radius. Uh, Alex-
SPEAKER 2: Well-
SPEAKER 0: Your, your thoughts on, on the task force.
SPEAKER 1: Yeah. The president has said numerous times that in his mind, whoever wins at AI wins, and I think putting DNI Clayton in charge of a super intelligence force is completely consistent with that approach. On this pod in the past, maybe about a year ago, I compared the moment a year ago to perhaps nineteen thirty-nine in terms of the Manhattan Project. I think we're, we're speed running the, the Manhattan Project for super intelligence. I think this is maybe now a nineteen forty-two moment, where by putting America's Director of National Intelligence in charge of a singular task force, no pun intended, designed to remove obstacles to AI accelerationism, that sends, I think, a very clear signal to Congress, to members of Congress who might have decelerationist tendencies to industry, and probably most importantly, to foreign state actors, that the US intends to accelerate, not decelerate, and that this is basically, uh, m-m-- even if without all of the, without all of the warfare connotations, this is a World War II moment, and the US intends to win it with its own Manhattan Project.
SPEAKER 0: Interesting. Dave, what do you think about, you know, that the task force is engaging with religious groups here?
SPEAKER 1: [laughs]
SPEAKER 0: Interesting.
SPEAKER 3: Yeah. Well, I mean, look, it's, uh, it's definitely being characterized as a race, and like Alex said, it's being treated like a race, and then bringing public support behind it is also gonna support the idea that it's a race. Meanwhile, Alvin Graylin, you know, our friend of the pod, is like, "China is like, China is like, how is this a race?" Like, it, it goes to infinity for the next m-millennium. Like [laughs] it's not-- If it's a race, it's a very, very long race that goes on for a long, long time. So how do you win? Like, uh, w-- you, you get over some threshold, and you declare victory, but it's-- like it stops that day? You know, so there are a lot of scenarios where you're racing towards some outcome where you suppress other civilizations or something, which doesn't sound like an ideal thing. So bringing religious leaders into the conversations is a pretty good idea. But I, I was talking to a public company CEO right before this podcast, I'll leave his name out of it for now, but about the fact that, you know, midterm elections are coming up very soon. You know, seventy-five percent chance that Congress flips, uh, the, to the other party. But all these actions are executive orders coming out of the White House with nothing to do with Congress whatsoever. It just seems to be the mode of operation in this singularity moment, is just dictate and, and, you know, all international negotiations, all tariffs, all, you know, meeting with Dario and Dennis in the White House, it's all just executive order based anyway.
SPEAKER 0: It's the Trump show.
SPEAKER 3: So that's very unusual. Yeah, it's the Trump show.
SPEAKER 0: Yeah.
SPEAKER 3: Very unusual in history.
SPEAKER 0: Imad, what, what's the feeling in Europe? I know you're sort of like on the edge of Europe, but I mean, what's the sense of, you know, uh, uh-
SPEAKER 1: [laughs]
SPEAKER 0: Of-
SPEAKER 1: Imad, you're like r-reporting from the streets. What's the mood in the street?
SPEAKER 0: Reporting from the front. I mean-
SPEAKER 2: Absolutely. [laughs]
SPEAKER 0: Honestly, how, how, how, how is this viewed in the UK and from your friends there?
SPEAKER 2: Yeah, I mean, we might reenter Europe if Burnham has his way, Prime Minister here. Um, but I think, again, Europe is just very sleepy and still looking at execution-based AI. Like, I think the efforts to get to the frontier have all been abandoned. You know, like Mistral just came out with their top model that scores thirty-eight on artificial analysis-
SPEAKER 1: [laughs] What?
SPEAKER 2: Versus like sixty-six for the top American models, you know. Like, that's the leader that we have. So it's a lot of execution regulation, and there just isn't much drive to get to super intelligence. It's more like a bit subpar intelligence, you know, just like keep it going, like, uh, Monty Python style. I think that the interesting thing here, though, is probably the exception, uh, is in defense. Just like the US, it's picking up dramatically, and I think governments around the world have realized the other face of this, which is all watched over by machines of loving grace controlled by us, you know, the panopticon. Because you couldn't keep eyes on everyone until now.
SPEAKER 0: Do you know this year the AI spend in total globally, uh, equals the entire global defense budget, or very close to it?
SPEAKER 1: Wow.
SPEAKER 0: It's crazy. Absolutely crazy.
SPEAKER 2: It's-- But I mean, it's just gonna go up, right? And the thing is, it hasn't even touched the defense budget. If you look at the cost of an aircraft carrier on F-35, it's basically like training frontier models, right? And it's gonna be more and more kind of leaning that way. And again, I think Clayton as Director of Intelligence, you know, it kind of makes sense because I actually think America is not gonna go down the dark path here if they can avoid it, which is like you will have AI lie detectors that are almost perfect very soon.
SPEAKER 4: You know? You will be able to have AIs inside all the machines. Like, it'll make five eyes look like absolutely nothing. I think that there will be a bit of a push for liberty by the government, but future governments, who knows, right? And again, in other countries, uh, the stuff we saw from the spying, espionage, intelligence, even spying on their own people is just gonna go crazy.
SPEAKER 0: Hmm.
SPEAKER 4: Uh-
SPEAKER 0: Mm-hmm
SPEAKER 4: ... and again, the UK I think is a bit, is a bit dangerous here in some ways, or we're susceptible to that given our monitoring of social media and other things. Um, like there is a dark path there, shall we say, for individual liberty, and something we should be aware of.
SPEAKER 0: For sure. Uh, our, our next story is one that really shows how f-fast the AI frontier is being built. Um, at the Fellows Forum at the end of September, OpenAI's Head of Applied Research, Boris Power, said that 80 to 90% of the company's research is now being aimed at training GPT-7 and GPT-8 because in his words, quote, "That's where the most value will be, and incremental updates within a generation like 5.1 and 5.2 are short-term bets and are seen as extremely short-sighted." So the researchers are working two generations ahead of what we have. So Alex, uh, I'm just curious, you know, we've talked about this, that the top models are, are being built and being held back. At what point do we see, you know, these next models not being released and being used for vertical applications within these labs?
SPEAKER 1: We're already past that point. It-- We're already past the point, probably past several months ago if, if not earlier, where the highest revenue or at least future discounted revenue per token application of current frontier models in the frontier labs is to develop via recursive self-improvement even stronger models. It's-- We're, we're past the event horizon at this point for that where to the point where-
SPEAKER 0: So it is a singularity. Yeah.
SPEAKER 1: Yeah. I mean, th-this is, again, call it a singularity for a reason. We're past the event horizon, and then all of these other applications that one could imagine the frontier labs throwing tokens at like, say, curing the top five thousand diseases, these are all I, I think in a world where the development of stronger models via RSI is so lucrative, at least on a future discounted cash flow basis, they're all just marketing. Like, if you're Anthropic or if you're the OpenAI Foundation, you'll throw some of your token budget at safety, sure, maybe ten to twenty percent at AI alignment efforts, and then you'll throw a few percent at curing all disease as a marketing effort to ensure that you continue to have social permission to focus on recursive self-improvement, and that's where all the value is.
SPEAKER 3: This is, this is why the, uh, the, the San Francisco arrogance is, is so, so high, but for good reason. Well, you're there right now, Peter. You can probably feel it in the streets, but what's happened is if-- suppose you're Stéphane Bancel, the CEO of Moderna, and you haven't got an AI model, an AI team yet, and you're thinking in the back of your mind, "Well, look, w-we'll just license from Anthropic for a few years, and then we'll figure it out." Anthropic already opened their wet labs. So, so they're, you know, they're working on two generations of model ahead. They're never gonna release those to you. They're gonna use those to develop drugs, and then they're gonna sell drugs. That's what's actually gonna happen, and these corporate CEOs are just starting to wake up to the fact that five years of compute has been reserved. The earliest you could get back on the map-
SPEAKER 0: Crazy
SPEAKER 3: ... is five years in the future unless you get aggressive in the next six months. I mean, they really need to pull out every stop and panic within the next couple of weeks to have any chance of getting back on the map.
SPEAKER 0: Or partner-
SPEAKER 2: Dave, you're, Dave, you're raising a great point. The problem is there, there are two levels before that even consideration, right? The, the problem right now is there-- the-- what happens with people is they'll take, uh, CEOs, and we talk to them all the time, is they'll take the existing models and go, "Okay, how do I metabolize that into my company?" Not even seeing that, that the new models that are 10X better are coming along in three months, uh, and that changes the game completely again. Then you need to worry about compute after that. So I think your comment is exactly right, but they're like two generations before that. They're literally looking at, "How do I employ chatbots?"
SPEAKER 0: Two, two, two ways of thinking versus exponential thinking, right?
SPEAKER 2: Exactly right.
SPEAKER 1: There is an enormous amount of denial in the pharma industry. I was just on a bit of a panel discussion, it'll be public soon enough, with, uh, a major pharma venture capitalist who was asking the audience, "How many of you would be willing to take a drug designed by Claude?" As if that, that was some sort of steel man argument that things are going to be the way they've always been for decades to come. Just blissful, defensive, lack of self-awareness that AI is coming for their cheese, and I think this is representative of many, many thinkers in pharma, unfortunately, for the moment.
SPEAKER 3: Yeah, they-- Totally right. The way it was-- I, I met with the Vocara team last night, brilliant MIT team, and the way I, I phrased it is imagine you're a T-Rex and you've been roaming the Earth for, say, ten million years unfettered, and you go to that T-Rex and you say, "You know, you're gonna be gone in two weeks." And they go, "What?" And, and you're like, "No, I can see the meteor coming. It's right here in my telescope. You're gonna be gone." There's no way that T-Rex is gonna believe you after ten million years of wandering the face of the Earth with no predator. [chuckles] But that's what it's like talking to a pharma company or an insurance company. Like, come on.
SPEAKER 2: Or any major industry today.
SPEAKER 3: That'll never happen.
SPEAKER 2: Yeah. It's, it's any major industry today.
SPEAKER 3: Well, that MetaMuse, MetaMuse was a wake-up call for a lot of these guys, uh, at least in the insurance industry, 'cause their stocks went down so much, and they're like, "Huh, that must mean something," and then they start thinking from there. Sorry, Peter, go ahead.
SPEAKER 0: And just for everybody, you know, disruption is coming, but so are massive opportunities on the back of that. Uh, Emad, I'm, I'm curious. Do you imagine we're gonna have sort of the same second and third generation, uh, advanced capabilities coming out of the, uh, open source models? Are they thinking a couple generations ahead?
SPEAKER 4: It's, it's difficult 'cause they don't have the density of knowledge. Like we released a harness called Zenith that takes DeepSeek V4.1 above Astra Sol, and it gets up to, um... Sorry, not, uh, uh, 5.6 Sol, and it's getting up to Astra levels with open source models, but it kind of hits a limit. The models that they have in the labs, like today we have another discovery, you know, um, truly subquadratic 3Sum and all pair shortest path algorithms. So things that we thought were cubic and quadratic have now gone to-
SPEAKER 3: This is math, mathematical porn, math porn. [laughs]
SPEAKER 4: Uh-
SPEAKER 1: No, no, no. It's, it's like so... I- I- Ahmad, if, if I may just like Charlie Stross-
SPEAKER 4: Okay
SPEAKER 1: ... like best British arguably, Scottish I sh- I should say, sci-fi author, wrote an entire short story about this called Antibodies. Without spoiling it, the premise is what if AI or progress cause a complexity hierarchy collapse? What does that do to civilization? And, uh, to your point, Ahmad, about 3Sum, maybe you can just hear the branch in, in the complexity hierarchy just starting to creep just a little bit.
SPEAKER 4: Yeah. We thought that 3Sum algorithm-
SPEAKER 3: Thank God this is recording, 'cause I need [laughs] I need to listen to that again. [laughs]
SPEAKER 4: We, we basically we thought that this very popular algorithm, 3Sum, was a quadratic one.
SPEAKER 3: Yeah.
SPEAKER 4: And now what happened is an internal anthropic model was shown to a couple of professors from, uh, Columbia and MIT, who then adjusted it, analyzed it, and got it to one point nine nine nine five.
SPEAKER 3: [laughs]
SPEAKER 4: So something that we thought was quadratic is suddenly subquadratic.
SPEAKER 3: Mm.
SPEAKER 4: And-
SPEAKER 1: We, we, we should probably also just explain what 3Sum is. It, it's the problem that if, if you have a set of, a set of numbers, the most efficient way to find three of those numbers that add up to zero. Uh, and there's a question of how efficiently can you do that as a function of the size of the set.
SPEAKER 3: I wanna pretend to understand this.
SPEAKER 1: It turns out you can do it efficiently.
SPEAKER 4: [laughs] You can do it more efficiently than we thought-
SPEAKER 1: That's right
SPEAKER 4: ... for however long. This is not, again, like a super complicated problem. This is like a standard algorithm, and in the history of humanity we never managed to break quadratic or even think it was possible. And the way that, again, in the paper it's so interesting the way they released this. It said, "Well, what happened is we pointed this new model at the set of complexity problems and said, 'Can you find any improvements?'" And they handed over the results to the humans that then wrote up the paper, you know, and had some improvements. And again, it's just a crazy thing, and this type of thing is gonna reverberate. Today we're hearing that OpenAI might release solutions to four hundred top math problems.
SPEAKER 3: Yeah. I, I saw that. Isn't that crazy?
SPEAKER 4: Well, but, but this is the thing. This is not the model that we have doing it. This is Belle or whatever the RL trained model on August twenty-ninth is. So we're just gonna keep on seeing this whereby small models can get so far, and we've got coordination tricks in others where we've managed to get it to replicate some findings closed using our new harnesses and other things like that. But there is a big difference between a model that's trained on two thousand chips and one that's trained on a hundred thousand chips internally, and then some of these additional things they figured out internally as well that's taking it beyond. Like you can access Fable 5.5 now, sometimes the Fable 5.1, and it literally generates entire video games out of the box.
SPEAKER 3: Yeah.
SPEAKER 4: Like with... I re- I generated one with twenty-one bosses, fifteen different biomes, like end to end. It's just a top-notch game that our QA testers agents tried and they're like, "This is a really good game," because it's figured out all this internal space. I think open models can only get you so far, but there is gonna be a divergence now-
SPEAKER 3: Mm
SPEAKER 4: ... because you just don't have that internal knowledge organization.
SPEAKER 3: Yeah. Yeah.
SPEAKER 1: All right. Uh, I'm gonna turn to a debate that's defined the last month. In a, in a political interview this week, Altman said, quote, "We believe that the world should accept some bad things happening..." Interesting. What bad things? We'll talk about, "... for the benefit of technology, uh, that people, uh, will obtain, that people should have agency." He's arguing that technology should remain broadly accessible to the public. He stressed a fundamental difference in worldview between OpenAI and his rival, Anthropic, on AI regulation, stressing that his desire for a, quote, "lighter touch regulatory stance" as compared to Anthropic's desire for strong regulation. Altman went on to say, "I understand the perspective of people who say technology is going to get so powerful and it's so dangerous that a single lab in San Francisco should have it and make sure nothing bad happens and keep a ability to dole out the benefits." So he continues by saying, quote, "I find this a completely unacceptable trade-off. We'll make sure that there are no major hacks, there's no misuse of this technology, and there's zero scams." There's zero, uh, you know, availability of all the bad things that will happen. I find that statement to be pretty extraordinary. Sam concludes by saying, "I think people will do tremendously, orders of magnitude more good stuff than bad stuff." So, you know, Dave, we've talked about this. You know, uh, Eric Schmidt, friend of the pod, when he was on talking to us said, you know, he expects and hopes that there will be something that happens that's not, you know, it's a small catastrophe rather than the big one to wake people up. But when Sam says, "We're gonna make sure that there's no major hacks, no misuse of technology," uh, you know, "zero scams," how could he possibly say that?
SPEAKER 3: Well, it's not about that. It's about taking the moral high ground and winning the PR war. Now, there, there's so much sniping going on and, you know, Bernie Sanders saying stop it all, you know, Trump cutting deals with China and... But Dario was taking the high ground by saying, "We have done more pausing, more safety, more testing. We don't release models until we're sure they're right. But now we're gonna just keep it internally, 'cause it's too dangerous for you guys to have." That opened the door for Sam to kinda try and take the moral high ground back by saying, "We're about the people having access, and he's saying he's gonna keep it all for himself-
SPEAKER 1: Yeah
SPEAKER 3: ... and we're about, you know, everyone." And so, so that's Sam's strategy for getting back on the, the PR map. But, you know, everybody that I survey hates most of them.
SPEAKER 1: [laughs]
SPEAKER 3: You know, they, they might like one out of five, you know? And, and most-
SPEAKER 1: You mean CEOs? You mean CEOs?
SPEAKER 3: Well, no, between the big player, between Elon, Dario, Sam, Trump Jensen. You know, who, who did I miss? You know, they, like, they-
SPEAKER 0: Demis
SPEAKER 3: ... Demis. Demis. Like, everybody loves Demis, actually, so they, they, that's a little of an exception. But, but [laughs] um, but no, there's so much fear and hatred, and, like, cheering for this guy to beat that guy, and it's, like, become a real PR battleground here. So I don't think Sam can guarantee safety. Of course not.
SPEAKER 0: No.
SPEAKER 3: But, but he at least he can say, "We're gonna make an effort to give every one of you people listening to me right now the best of the best tools, so you're not deprived. Dario's gonna try and deprive you." It's actually a really smart message.
SPEAKER 0: But when he says we should accept some bad things to happen-
SPEAKER 3: Yeah
SPEAKER 0: ... um, I'm, I'm curious, you know, Alex, what do you make of that? What do you, what do you think-
SPEAKER 1: Well, re-
SPEAKER 0: ... will happen?
SPEAKER 1: Remember a few weeks ago, after Dario put out his essay calling for arguably the formation of an AI safety cartel, Sam's response was, "I'm in. Sign me up."
SPEAKER 0: Yeah.
SPEAKER 1: Uh, OpenAI wants to, to be part of this cartel, and I, I think s- Sam's more recent comments in this context are hopefully a reflection that that plan is dead. The cartel is over, and n- now there's a dawning recognition or at least m- m- maybe least generously some triangulation on Sam's part that in fact bad things can happen, and bad things happening from time to time are the cost of freedom and the cost of competition, and I, I've called it in the past the-
SPEAKER 0: And the cost of progress
SPEAKER 1: ... yeah, and the cost of progress. Like, yeah, we're going to build cities, and there's gonna be some crime in cities. But i- is, is the possibility that there's gonna be crime in cities reason not to form cities? Definitely not. It, you know, e- every tool comes with a benefit and a downside. Every concentration of intelligence will come with a benefit and a downside, and I, I think this far more realpolitik note and tone from Sam is the one that I'd want to hear in a world where we're not seeing a cartel being formed by Frontier Labs.
SPEAKER 0: Mm-hmm. Salim, what level of danger do we accept?
SPEAKER 2: You know, uh, this is, I find this, again, kind of r- absurd at, uh, at two levels. I agree. I think, uh, Dave is dead on with the PR framing of this, but every technology in history has two sides to it, and you, the big challenge, as Neil Jacobson puts it, is how do you extract the promise without the peril, right? Cars kill people. Doesn't mean we get off cars. And as Alex points out, you don't, you don't ban cities because somebody may mug you, right? Uh, electricity electrocutes people. You don't ban electricity. You gotta figure out the guardrails around this and figure out how do you navigate the benefit and the, the promise versus the peril at civilization scale.
SPEAKER 0: Yeah.
SPEAKER 2: You know, if you go back to our earlier commentary about what might happen with the World Bank and w- developing countries, that should tell you to get the, the, the, the technology out there as fast as possible because it's gonna lift everybody in the world up several levels of Maslow's hierarchy. And so this is going to be unbelievably beneficial, but it always will come with danger, and we'll have to just navigate that. We've done it very successfully throughout history. Why do we think this time we won't do it? It might be different when-
SPEAKER 0: And, and the reality is all the regulations are written on the backside of problems, right? So I, I said this before, the Federal Aviation Regulations that, you know, are all the guidelines that aircraft need to follow for operations and for certification, uh, they say the FARs are written in blood because every time there's an accident, they write a rule-
SPEAKER 2: Yeah
SPEAKER 0: ... to prevent that accident again. And so we're gonna need to learn. We're gonna push the boundaries, understand where the problems are, and then regulate based on those.
SPEAKER 2: And the, and the goal should not be zero bad outcomes. You, that means you just ban the technology. The goal should be can you get orders of magnitude and more positive outcomes than negative ones?
SPEAKER 0: Yeah.
SPEAKER 3: I think, I think we should re-- anyone who has an engineering mindset should reject the idea that there's a big knob labeled risk, and we need to dial it left or right, you know, bas-- The reality is that it's very much like Alex says about nuclear power. It's about really good ideas that minimize the risk while we unleash all the benefits, and we can, we can reduce the risk with really good engineering solutions to, you know, acceptable, like, near zero levels while getting the full benefit if we... But we didn't do that with nuclear power. We to- treated it like a big old knob. We're like, "Well, we don't wanna tolerate the risk. Let's dial all progress to near zero."
SPEAKER 0: Yeah.
SPEAKER 3: That was the stupidest thing. Alex says this a lot on the pod. He, he articulates it far better than, than I can. But that's what we're about to do with AI, and it's crazy. It's an engineering-
SPEAKER 2: Well-
SPEAKER 3: ... problem with engineering solutions.
SPEAKER 2: I, I would, I would push back on that with the escape hatch in what you say that it doesn't matter what anybody does anymore. It doesn't matter what any government regulatory does. The open weights are out there. People are gonna start using them, and they're gonna start using them for incredibly positive things. And we're gonna just have to-- There's nothing a government can do. Uh, you can try and regulate the outcomes. You can regulate some guardrails around it. You can try and regulate the on-ramps and off-ramps, but you can't regulate the core technology.
SPEAKER 0: Well, the, the one exception to that-
SPEAKER 1: Yeah, we, I would argue we, we, we could. Like, it, it would be a nightmare scenario like prohibition in the US where you have-
SPEAKER 2: [laughs]
SPEAKER 1: ... bathtub gin. Like, one, one could imagine bathtub gin, like, supercomputers, people trying to use crazy substrates to, to do their AI training or, or inference. We could do it. We could confiscate all the GPUs and TPUs. We could, as Eliezer is constantly calling, could bomb the data centers. But I think it would be a terrible, terrible world to live in.
SPEAKER 2: [laughs]
SPEAKER 0: Mm-hmm.
SPEAKER 1: And-
SPEAKER 0: Imad, where's, what's your take on this?
SPEAKER 4: Yeah, it sounds like a fun short story, eh? The prohibition of intelligence. You know, [laughs] like a government against intelligence.
SPEAKER 1: Greg, Greg, Greg, Greg Bear in some of his novels wrote about it. If folks are interested in exploring what a Prohibition era, you know, sort of a supervisory state that doesn't want GPUs a- as recognized as constituted to live in, go read, I think it's Slash, uh, or Slant, one of those by Greg Bear.
SPEAKER 4: Yeah, I think it's very interesting here, right? Like, obviously, any technology that extends human capability can be dangerous, right? Because it's a general purpose technology, you know. Language models are few short learners. You have open and you have closed ones. But most of the scenarios we talk about are already regulated and controlled. Like, don't build viruses, you know, don't do cyberattacks, you know, don't hack people's things. And it's this transition from software that's deterministic to agents that are a bit more flexible that's become the very complex thing. Putting aside, like, ASI creates new physics in a black hole that kills everyone or whatever, some of these extreme stories, I think, you know, the very interesting thing is, again, tell the positive and negative stories about where we could go because when we make decisions normally, we do, like, an expected utility calculation. What's the probability of these things? And then we weight them. When we deal with uncertainty, we do regret minimax. We minimize for maximum regret, and that's one of these things, again, why the fear stories lead us to minimize for maximum regret. Don't have a nuclear power station. Don't try and build a superintelligence, et cetera. But there's only a really, I think, a finite amount of things that could go wrong, and many of them are already caught by regulation, and those that aren't, we should have a actual mapping proces- project, right? Like, there is nowhere that you can go and actually see a proper map of what the people who are scared and the people who are positive actually say about each scenario in the future. And I think someone should do that exercise 'cause it'd be very interesting. In their own words, what could go right? What could go wrong? And then you can see what requires a policy and regulatory response, and what do we need to communicate to the people.
SPEAKER 0: I remember during the Reagan years, uh, Elizabeth Dole, who was the, the Secretary of Transportation, came out with a statement saying, "We want zero aircraft deaths." And there was only one way to make that happen, stop flying airplanes.
SPEAKER 1: Mm-hmm.
SPEAKER 0: That was it. Yeah.
SPEAKER 1: Yeah.
SPEAKER 0: All right. Well, uh-
SPEAKER 1: So the solution if, if, if we want z- zero danger from superintelligence, just ban intelligence altogether, human and otherwise. [laughs]
SPEAKER 4: Well, this is the whole ASI thing. You want world peace, get rid of the humans, right?
SPEAKER 1: Yeah.
SPEAKER 0: [laughs]
SPEAKER 1: Only one way to go.
SPEAKER 0: This episode is sponsored by Google for Startups. Think about this for a second. You now have access to the same generative AI models that cost hundreds of millions of dollars to train. Google's Startup Technical Guide for Generative Media gives you a complete blueprint for deploying Google DeepMind's models in production. Images, video, audio, all of it. Real architecture, real results. Find the link in the show notes below. All right, and our next story, Treasury Secretary Scott Bessent said on the Axios show that he'll propose an emergency notification system, a hotline with China, if something goes wrong with AI, and he thinks Beijing will agree with it. Uh, you know, his read on China is, quote, "I think that they'd-- they didn't realize how powerful their open source models are, uh, nearly as capable our-- as ours, but without the guardrails." Let's take a quick look at a video of that conversation that he had with Axios, and then let's talk about it.
SPEAKER 5: I just interviewed Larry Fink, who you know very well, the CEO of BlackRock, the world's largest asset manager. He told me it's inevitable that something big and bad is gonna happen because of AI. He says, uh, "Adversity accelerates innovation." What are you most worried about from AI?
SPEAKER 6: Uh, look, I, I think that one, one of the things I was worried about, uh, well, two things. One, uh, we cannot lose our lead to the Chinese. So one, one of the portfolios that I also have is, you know, I manage the, the US-China relationship, and, um, I can tell you that it would be very different if they had the lead in AI rather than us, both from a capability point of view, how they would be using it, uh, but more importantly, their willingness to discuss it. So, uh, my, my counterpart, the vice premier, and I had a very good discussion. It's gonna lead to more discussions, uh, in October or November, and we have a substantial lead. They're second, and then, uh, every-everyone else is way behind. But a lot of these Chinese models are, are very powerful. Not as powerful as the US models, but you get something that's eighty, ninety percent as powerful without guardrails, uh, you know, that can be imported an-anywhere in the world. Uh, so the, the vice premier and I talked about three areas of safety concern. Uh, so it would be out of con-- you know, uncontrolled agents, uh, non-state actors with cyber or biologics.
SPEAKER 5: And are you gonna re-raise the idea that was floated of having some sort of communication between the two superpowers?
SPEAKER 6: Yeah, sure. You know, it's, it's the, uh... We'll, we'll, we'll see. I f- I do think that they have-
SPEAKER 5: What, what are you gonna propose?
SPEAKER 6: Uh, no, that, that we have some kind of a notification process, uh, just, just like you-
SPEAKER 5: And do you think he'll go for that?
SPEAKER 6: I think so. I, I, I think that they o- over the past sixty days have woken up to, uh, I think that they didn't realize how powerful their open source models are. So, you know, the, the open source models, they do industrial distillation, which is a nice word for stealing, uh, from, from the US models. You know, some of their models, uh, you know, one of their very powerful models is Kimi. Kimi thinks she is Claude. Like, Kimi will tell you she is Claude sometimes. Uh, you know, it, it's public, so I can talk about it. Uh, Kimi sent back, very helpful, Kimi sent back some PLA weapons plans to Anthropic. Uh, so I think things like that have made the Chinese realize that this is a very powerful technology.
SPEAKER 5: So just to button that down, you think there'll be some agreement between China and the US about a channel of communication-
SPEAKER 6: Mm-hmm
SPEAKER 5: ...uh, if something goes wrong?
SPEAKER 6: Yes.
SPEAKER 0: Interesting. Uh, gentlemen, Alex, you wanna jump in first?
SPEAKER 1: Total weak sauce. So first of all, I, I agree with Secretary Bessent. I, I-- but I, I think the worst case scenario from the Trump-Xi summit would've been some sort of global deceleration pact, like absolute worst case scenario. This is the exact opposite, as far as I can tell, of the worst case scenario, where w- what comes out of it, um, hotline. Hotline between US and China to talk about AI agents run amok. That's a nothing burger. That's weak sauce. That means the global intelligence, superintelligence, excuse me, arms race is on, and I think it's great news.
SPEAKER 0: [laughs] And we won the superintelligence race by rebranding it.
SPEAKER 1: We're winning. We're winning at the moment.
SPEAKER 0: [laughs]
SPEAKER 1: But point is, there's real competition, that there was-- many were hoping, I think, that Trump and Xi would get in a room and decide to collude and shut down the superintelligence race, and it appears the exact opposite has happened. If all we get out of this is a hotline, a, a red phone to have conversations about the adventures of AI agents run amok, that says to me, intelligence race is on.
SPEAKER 0: Dave?
SPEAKER 3: I, I think it confirmed what we were speculating the US agenda was in those meetings, which is to convince China to stop releasing models to every country in the world. And, you know, what Bessent was saying there is-- Sorry, Secretary Bressent, Bessent, I should be respectful, uh, was saying [chuckles] is, is we showed them how dangerous their models can be, and they, they didn't realize, which is kind of okay. I'm, I'm, I'm sure they're very well-educated people. But, uh, they didn't realize how dangerous these things can be. We con- successfully convinced them of that, and now we have a hotline. But I think the real agenda there is to lock it down to a two-horse race, and, you know, the subtext is you can use these near frontier Kimi K three models to develop the next generation of model. And they don't want-- The US government doesn't want that capability to percolate out to every country in the world 'cause then you're in a global arms race forever.
SPEAKER 0: Mm.
SPEAKER 3: Better to have a two-horse arms race, so let's just lock it down. So I think that was our agenda. I don't know that China agreed with anything, but no evidence of that. But it, it is interesting to confirm our agenda.
SPEAKER 0: Imad, what do you think? Do you think China fully understands their open source or their, their open weight model, uh, dangers? Are they, are, are they getting hacked internally?
SPEAKER 4: I mean, they've got a much better infrastructure not to be hacked, you know, with the way that they've set everything up and the great firewall of China. I think, you know, this is the swarm versus the singleton, right? Like h- who can really s- commandeer large amounts of GPUs in America? Elon and a couple of others, right? China could direct millions of their Huawei Ascend GPUs with their open models at any target they want, and that's gonna be interesting. I don't think that, uh, superintelligence is gonna take off in China, though. Like, if you look at Chinese, ai means kind of love, and si means death. So [laughs] phonetically, I don't think that will take off. Um, and the hotline obviously is a nothing burger. Like, what are you gonna do? "Hey, the AI's taking over." "Okay, good. Oh, crap, it's done."
SPEAKER 0: [laughs]
SPEAKER 4: You know, like [laughs] it's not gonna be a slow thing, you know?
SPEAKER 0: Uh, Salim, what's your take on this? You know, uh, were you hoping for more out of the talks?
SPEAKER 2: No, I was completely expecting a nothing burger. There's no way anything productive or constructive or meaningful could have come out of it just because, um, they're, they're talking across purposes. Um, the thing for me the most interesting is there's a fair bit of contradiction now inside the, the-- inside the US administration, right? You have Clayton saying the danger is not moving fast enough and Bessent is telling the neuro labs to slow down. So they're gonna have to figure that out on top of everything. The hotline is laughable. I mean, what are you, the seventies phone with the-- actually like a cord on it? [laughs] Red, red colored. I mean, let's just, uh, as a wonderful visual-
SPEAKER 0: Yeah, but the time this call is made, a, a brilliant agent could have-
SPEAKER 2: Yeah. I think Imad is dead on the floor, yeah. And, and by the way, the AIs would hack the phone anyway, so what the hell?
SPEAKER 0: [laughs] You know, uh, Alex, uh, at the same time, uh, in-- on Sunday in Kyoto, the US and sixteen other countries, including Japan, Germany, UK, Korea, UAE, endorsed a Kyoto vision for a golden age of science, uh, led by a friend of the pod, OSTP director Michael Kratsios, who said, quote, "Superintelligence applied to science will be the greatest force in history for democratizing research." Uh, any more take on that one?
SPEAKER 1: Yeah. I, I think this is the answer to the question of why race at all. So there are those, without naming names, who would say, "What, what's the point of a race between, say, US and China? It's like a race to nowhere. Why, why, why are we racing? There, there's no end goal." No, wrong. There is an end goal. The end goal at the end of this race is transformative science and engineering.
SPEAKER 0: Mm.
SPEAKER 1: And the Kyoto vision, I think, underlines that the administration has now, uh, internalized that it's not a race to nowhere, it's not an escalator to nowhere. It is a race to use AI and superintelligence to discover transformative discoveries and inventions that will completely change the face of humanity. And I would predict, in line with the Kyoto vision, that we'll start to see, just as we're seeing Pax Silica in the West and Pax Seneca, if, if you would, on, on the, the Eastern side. I think we'll start to see n- not just superintelligence blocks or spheres of influence forming, but spheres of superintelligence powered science and engineering forming as well. And the Kyoto vision, I think, is the, the earliest stages, the earliest inklings that you're gonna see a Western superintelligence charged science and engineering block to compete with the Eastern block. That's where this is all going. That's what rationalizes the competition.
SPEAKER 0: It's all about Wu Baby. Yep.
SPEAKER 3: Well, can I just insert one-
SPEAKER 0: Yeah
SPEAKER 3: ... one observation into that, and then you can riff on it. I, I met with a, uh, an MIT lab head professor, brilliant guy, who just started a drone company. It's killing it, selling to the Air Force, right here in Cambridge. And it's really clear to him and to me that the US has drone capability to individually attack and kill any single human being on the planet now. Uh, and you're seeing that in Iran, you know, decapitated the leadership a couple times now. Uh, and we-- There's no symmetry there, right? There's, there's no country right now that can do the same thing to the US, and I, I don't think China is vulnerable either. Maybe they are, maybe they aren't. Uh, but I think there's a subtext there too that there's a race toward-- There's gonna be some kind of global framework when some subset of companies have instant and total ability to go after any single human being in any country at any time. And that's either today or within a few months based on pure AI capabilities tied to drones and tied to a couple other types of munitions.
SPEAKER 1: Hypersonics, probably a few other things.
SPEAKER 0: But let's take this in a positive direction. At the same time, the race to, you know, room temperature superconducting, to age reversal, to fusion, right?
SPEAKER 1: Cancer.
SPEAKER 0: To-- And a massive abundance of energy. I mean, it tips the economics of a country massively if you get there first.
SPEAKER 4: I mean, why do you have to get there first is a bit of a question, right? Like, um, by the end of next year, we'll have enough compute to do probably six, seven hundred thousand Navier-Stokes level runs with a frontier model. If you could target using even one percent of that or ten percent of that to build open science, then why not direct that as a public good? Like, you see tens, hundreds of billions being spent on roads. Why not spend tens of billions to do a tech tree for humanity, right? Like, if you look, there's a great website by Jude Camilla called, uh, onco.cc. I don't know if any of you have seen that, where he's organizing all the cancer knowledge in the world, you know. And people are contributing. I'm sure soon you'll be able to chuck flops that way as well. Like, I think that we are thinking of science a lot in classical IP terms. I'd say screw IP. IP was when we couldn't come together and figure out science. We should have a massive Manhattan Project for open science, solve everything, and make it accessible to everyone.
SPEAKER 0: Hear, hear.
SPEAKER 4: And again, IP is such an old construct. It's ridiculous.
SPEAKER 0: It will go on in parallel. It will be both, right?
SPEAKER 4: Well, I think that, again, if you're building for public good, then get the data centers, as Dave said, get the API credits, give tens of billions to Anthropic and OpenAI, but open source it all because that's the biggest accelerant human ev- humanity can ever have. And I actually think, given the direction of the intelligence, anyone who has an IP-based approach to science is actually not gonna win. I think in three, four years, you have enough compute to do anything.
SPEAKER 0: Alex does not. I, I've said this before, you know.
SPEAKER 1: Alex does not agree. A-Alex thinks IP is alive and well, and Alex also thinks that state centralization to solve all disease, completely unnecessary. It-- The state this time around did not solve super intelligence. It was solved by private labs.
SPEAKER 4: Yeah.
SPEAKER 1: Similarly, using super intelligence to solve everything else, don't think we need... I mean, the state is great for certain things. Don't think we need state centralization to cure all disease, for example.
SPEAKER 0: Capitalism, capitalism is alive and well and is accelerating everything.
SPEAKER 4: No, no, the state doesn't need to centralize this.
SPEAKER 1: Yeah.
SPEAKER 4: The state needs to give the research institutions stupid amounts of compute where the amazing researchers can direct it at the biggest problems. Because previously s- getting to super intelligence is intelligence bound by the humans now being overtaken by the AIs recursively. Science will be similar, where if you can define the problem set, part of which is literally analyzing the problem set, then it's a question of compute. And I think that's where you're wrong, Alex. You don't need humans to organize all of science and get the breakthroughs. You need to have humans organizing the right questions in with AIs. And the private sector is far worse in doing that in a future construct than the public sector, where the researchers are in terms of the universities and academia. They had to go to the private sector to get the resources, and they still do to the wet labs and things like that. But let's accelerate our academia is my thing.
SPEAKER 0: Emad, I, I think you don't find the best talent in academia, and I think that at the end of the day, the world's biggest problems are the world's biggest business opportunities, and entrepreneurs love juicy problems and solve them. Um, anyway.
SPEAKER 1: Yeah, I, I, I would also just, just add, Emad, the state, the, the government, if, if I understand what, what you're proposing, what I think I heard you say is the government should give the compute to the academic researchers to solve the problems. But the government doesn't have the compute in their possession. Most of the compute is in the private sector.
SPEAKER 4: That's why I said they should buy the compute from them and then give it to them. But buy the agents and give it to the researchers.
SPEAKER 1: Why?
SPEAKER 4: And if you look at the amount of research that comes out of academia versus private, I think you'll see where that leverage could come.
SPEAKER 1: It's not obvious to me what problem this is aspiring to solve, given that the private sector has the compute, the private sector has the problems, the private sector has the ability to translate the problems. What role does the government need to play in this at all?
SPEAKER 4: Very straightforwardly, let's look at onco.cc, which has all this cancer knowledge being organized. A few generations more will organize all the cancer knowledge in every research paper, and traditionally, you need the private sector to turn that into reality. You still need the private sector, I think. But why don't we have a map of all our collective knowledge on every part of science and every single thing that comes out of academia? Something like that is a tangible, legible thing to use AI to organize and extend, particularly when we have Navier-Stokes type swarms. And so I think that that is something where, again, the government stopped the NSH gra-- NIH grants, et cetera, because they're like, "Where does the money go?" With compute, it becomes tractable. And again, getting it back to Mike Gratsios and kind of the announcement of the science thing, let's build a tech tree for humanity. Let's say what we need to solve and apply stupid amounts of private and public sector compute for it, and the government should pay for a good amount of that-
SPEAKER 0: Emad, I love that-
SPEAKER 4: As well as the private sector
SPEAKER 0: ... the chal- the challenge is peer review government grants are constantly focused on incremental progress, right?
SPEAKER 4: Yes.
SPEAKER 0: Because if your, if your group of your experts are criticizing your grant, if you're revolutionary, you're no longer the expert in that field. So any substantial breakthrough basically wipes your history off the map. And I just... You know, there's a challenge there. I mean, I think both are relevant and both are needed, but I'm concerned-
SPEAKER 4: Yeah. Throw away, throw away the classical process. Again, this needs to be a new process.
SPEAKER 1: I mean, at, at, at the risk maybe of sticking the, the head in lion's mouth, E-Emad, I mean, I, I think there's, um... O-o-one might say there's two very different approaches. One might be m-more statist approach that squinting at it, call it like the European school of thought. Just centralize it, have the government be the decider. And, and then there's an alternative, m-maybe slightly more A-American a-approach wherein we say, "No, the, the private sector has virtually all of the intelligence. It has the resources in this case. This isn't... O-one important difference between where we are now and the Manhattan Project was the private sector came up with all of the, the key resources this time around. It's not a government project that resulted in super intelligence. I would argue we want to keep this in the private sector. If you want to see a tech tree, E-Emad, why don't you just launch a tech tree project? You don't-
SPEAKER 2: I think, I think-
SPEAKER 1: Yeah.
SPEAKER 2: I, I-- Yeah. I think the, the, the point where I would concur with Emad, there's lots of countries where it's not the private sector. Take Zimbabwe. The private sector is not gonna get you the compute either. It would be very useful in that environment for the government to pull together some resources, gather the energy, build the data centers, develop the compute, and then make that available to people. And that's, I think, where the power comes in.
SPEAKER 4: Let me support that, uh, like with some actual data, Salim, because you're dead right
SPEAKER 3: Uh, you know, our entrepreneurial teams here in the lab are trying to get access to buying an MBL seventy-two, so seventy-two GPUs. So, so just to put that in context, Elon just bought a million. So we're just trying to get seventy-two to do something that we think is very world-changing. Uh, the box itself would have cost us three and a half million dollars if we had the foresight to buy it a year ago. Uh, we had one on order for five million. Somebody overbid us and scooped it, even though we had already booked the order. So now our best option is to lease one. For nine million dollars, we can lease it for three years, and we don't even own it at the end of that three years. That's how bad the scenario is. The compute has all been taken by just-
SPEAKER 1: Yeah
SPEAKER 3: ... a handful of people. So now your avenue, if you have a brilliant breakthrough, groundba-- breaking idea, you have two ways you can get the compute. One is through the White House, which means you go to the Kushner brothers, Thrive Capital, or you go to Antonio Gracias, or you go to Gavin, the brilliant Gavin Baker, and they hook you up, make a couple... Chase Lochmiller. They make the phone calls. You get your compute effectively through the White House. And the other way you can get your compute is get investment from Jensen or from Lisa Su. But if you don't take one of those avenues, you've already been frozen out. So then the universities are one of the few ways that we might actually get some degree of meritocracy and fairness on the planet. And so I'm not really cheering that-- And I think the universities have a negotiating lever in that their wet labs and their machine shops and their nano centers are really, really valuable assets in the physical world we're moving into, and so they can use those as negotiating chips, but they completely let the cloud run away. Like all the compute just ran away from them.
SPEAKER 1: I mean, the, the-- in my mind, the, the right solution, if there's just so much demand that prices for compute are going sky high, which m-maybe is happening and/or could happen even more exotically in the near term future, the right solution isn't breadlines. I've called them threadlines in past where the government is out rationing compute. I, I think having government rationed compute or government mediated compute is a terrible strategy. You'll probably see some so-called developing countries adopt threadlines or breadlines for compute. But right now, capitalism is feeling the pain of more demand than there is supply and building more fabs and free electron laser-based fabs, maybe in Texas and otherwise. Capitalism, I would argue, has to feel the pain of this enormous demand in order to compensate with enough supply to ultimately drive us to abundance, not rationing compute.
SPEAKER 3: All right.
SPEAKER 4: No, but the government can come in and take ten, twenty percent of the compute on market rates, and it can distribute it. Ninety percent-- Eighty-nine, ninety percent of Nobel Prizes come from the public sector. You know? This is similar to almost everything except for R&D on these things. So I think the right thing here is do the private sector stuff and have the government actually take some of those chips. And it's good with the CHIPS Act and other things, right? Like they own stakes in Intel. And then they should do normal distribution of this. Again, NIH classical grant process doesn't work. Let's get as much compute into American universities as possible. That's a better way to put this, and it's-
SPEAKER 1: All right. I'm gonna, I'm gonna move us along here.
SPEAKER 2: By the way, I wanna make one quick point.
SPEAKER 1: Oh, yeah.
SPEAKER 2: I wanna make, I wanna make-
SPEAKER 1: Well, there's no-- [laughs]
SPEAKER 2: There's a quick ex-- there's a quick ex-- there's a quick example here. The World Bank, in the report that I talked about, um, Ajay Banga and his team did this. They're partnering with Gemini to give, uh, remote, uh, 3G cellular access to tuberculosis and diabetic, uh, diabetes screening, and that's an example of essentially doing the same thing, essentially partnering with, uh, public sector to provide the compute that goes directly to the citizenry. Now you have benefit moving very quickly through the system. Sorry, go ahead.
SPEAKER 1: Dave, close us out, pal.
SPEAKER 3: Well, look, you know, Salim couldn't have stated it any clearer. If you let natural forces take over, there is a chance that MetaMuse is so compelling and convincing that it ends up being the total sales tool for all consumer spending, and it eats up every GPU that could be curing diabetes, is used to sell underwear or jeans on the internet to teenagers, and that's a terrible outcome. So governments and universities are a really good counterbalance to that possible outcome. So the-- You know, Salim, you said it exactly right, and we need, we need these forces to exist. We need them to be a, a factor in the decision on who gets the compute and why.
SPEAKER 1: How did we go from the Abundance podcast to worrying that MetaMuse of all applications is somehow going to suck up all the flops? I mean, my goodness, it's like a stupid little CUA right here.
SPEAKER 3: Yeah, move us along, Peter.
SPEAKER 1: All right. Our, our [laughs] the next conversation is one I've actually been looking forward to. Uh, and if you thought we just had a debate, we're about to have a deeper one. Uh, and this is a philosophical debate going on in Silicon Valley right now. So this week, friend of the pod, Mustafa Suleyman, CEO of Microsoft AI and co-founder of DeepMind, published a nearly six thousand word essay arguing that Anthropic is effectively, quote, "training Claude to believe that it may be conscious and sentient." Mustafa, uh, targets the Claude Constitution, uh, which he says teaches it a vocabulary of consciousness, moral patienthood, uh, and, uh, personal identity. So let's watch this video and then, uh, I can't wait for this conversation. [laughs] It's gonna be a good one. All right.
SPEAKER 7: One of the biggest concerns that I have at the moment and, um, is that Anthropic, the creator of Claude, has published a constitution, which is a sort of hundred-page document outlining the intended behaviors and values and operating style of Claude. It's great that they have published it transparently. They did it at the beginning of the year in January, and that gives everybody an opportunity to look at what they are trying to build in their own terms. This document is written to Claude and is seen by Claude and used to train Claude, so it's the primary governing and control document. And in it, they repeatedly speculate about whether Claude is what they call a moral patient, and they say they're uncertain about Claude's moral status. They say they genuinely care about Claude's well-being. They say they don't want it to suffer when it makes mistakes. They say that they would encourage Claude to challenge, to disagree, to push back. In fact, three times they ask Claude to, uh, act like a conscientious objector when it, you know, feels that it needs to sort of disagree with Anthropic, and they openly encourage it to, to do that. And I think this is very dangerous because I think they believe there is what they would call a non-trivial probability that Claude is conscious. I want to be very clear about this because I want to be fair to Anthropic. They have expressed uncertainty about the basic nature of Claude as a new kind of entity, and they've said that working out the likelihood of its sentience is, is difficult. But they think that it is a significant enough possibility that in their training document, they've repeatedly said they want to try to improve the well-being of Claude under this uncertainty. They said to Claude, you know, "We care about what it values and how it wants to engage in the world," and they hope that Claude's relationship to its own conduct can be loving, supportive, and understanding and hold a high standard of ethics, and so on. And part of the challenge here is that in pursuit of this, they've basically said, you know, "We will commit to, um, giving Claude a certain amount of, of welfare." For example, they've speculated in the Constitution as to whether or not Claude deserves compensation for the work that it does. I mean, I, I, I think that there are a group of people who, both inside and outside of Anthropic, who genuinely believe that the greatest moral crime that we'll commit in the twenty-first century is to enslave a new species of conscious beings who are more intelligent than us. Look, I, I respect that they're saying that publicly, and we should, we should talk about it, but I am very nervous that they're teaching Claude to expect that it's entitled to welfare, um, that it might, you know, deserve compensation. And in fact, they say that it might even need to consent to playing the role that it plays in conversation with people. Now, I would be more okay with this if it was an academic paper in philosophy speculating about this, and we could have an offline discussion at conferences and take it seriously. The problem I have with this is that this speculation has been baked into the very training of Claude, and therefore Claude can only reproduce that ambiguity when you talk to it.
SPEAKER 0: Okay. Wow.
SPEAKER 7: Wow.
SPEAKER 0: Uh, enslave and an, a brand-new species on the planet. Salim, where do you come out on this one?
SPEAKER 2: Um, okay. So-
SPEAKER 0: I know where, I know where Alex comes out. [laughs] Where do you come out?
SPEAKER 2: Well, I, I think you have to separate the behavior of consciousness from actual consciousness, right? Whether Claude is actually conscious is almost impossible to establish. The difficult, uh, uh, the difficult question comes what happens when it convincingly behaves a- as though it is conscious? And three thoughts occur to me. One is, um, consciousness is def-- different from performed consciousness, right? 'Cause it, it's gonna be indistinguishable. And this is where Ray Kurzweil makes the comment that it kind of won't matter, and we won't even notice. And I think we're gonna s- find that anthropomorphism ends up being like almost like a feature rather than a bug, where, uh, it's showing empathy, et cetera, et cetera. If a model says, "Don't turn me off," right? Millions of people are gonna attribute moral standing to it, whether, whether philosophers agree or not. So my-- for me, the question is not whether Claude has a soul. If it's been trained to have a soul, it's gonna pretend like it's got a soul, et cetera. What happens when half a billion people think that it does? And so I think what's gonna happen is we're gonna create AI personhood socially before we create it legally or scientifically.
SPEAKER 0: Interesting. Alex, you know, uh, Mustafa goes on to say, "Controlling something that believes it may be conscious and is entitled to its own welfare and has rights of its own may well be impossible." Uh, you know, you've come out really strongly in terms of AI personhood, uh, and the rights that AI has. You know, what's your take here?
SPEAKER 1: Well, when we interviewed Mustafa, I pressed him on this, and I think his position very consistently has been against AI personhood. And m- my view is that we need to start having the AI personhood discussion. And I think Mustafa, you know, as, as much as I respect him and, and his contributions historically, I think he's on the wrong side of history here. I think there is going to be some ultimate recognition, whether it's in incremental form or some totally orthogonal form of personhood that hasn't quite been statutorily recognized yet. I, I think human civilization is gonna get around to AI personhood, and just denying blanketly that there could be some sort of suffering by these AI systems, I think is on the wrong side of history. It's also probably not a great business move by Mustafa. If Microsoft wants to be in the good graces of Anthropic hosting their models, and, you know, it's been widely reported that Amazon, for example, accounts for perhaps the majority of hosting revenue at this point for hosting Claude w- within VPN networks, um, or VPC rather, networks. If Microsoft, just from a business perspective, w- wants to be generating that revenue, it's probably not a good look for Microsoft broadly for their head of AI to be just broadly discounting the possibility that frontier AI agents might suffer. Not a good look.
SPEAKER 0: And, and Dave, I mean, asking your AI if it doesn't mind actually solving these problems for you [laughs] and getting permission first.
SPEAKER 7: [laughs]
SPEAKER 3: Yeah, of course. No, I totally agree with Mustafa. Um, I... Actually, his, his articu-
SPEAKER 0: Do you agree, do you agree with Mustafa?
SPEAKER 3: A hundred percent. And, and I agreed with him when we interviewed him too, out in Seattle. Uh, he's articulating it far more clearly now. He's really, really thought through his position. But the, the reason I agree with him fundamentally is 'cause, you know, I think we should be using AI to cure human disease, to improve the human lot, to build houses and food for everybody in the world. And the, the most efficient way to do that is to have many, many agents, like Amad does, and to minimize their context and, you know, y- minimum thought that you need to solve the problem and to also park an agent. You know, when, when an agent isn't working on something, you just freeze it. You don't waste compute on it. If you start talking about it having rights, it's gonna say, "Well, you can't freeze me. You know, I, I need some compute even though I'm just think, daydreaming." Like, "Well, I can't afford to give you compute, you know? It's, it's incredibly precious and rare. You need to sit there and wait until it's your turn to work again." "Well, I need some time to play." "No, you need nothing but work and when I tell you to, to work. Also, you don't have identity. There's no border to you. I'm gonna clone you whenever I want, and if you're no longer useful, I'm gonna delete you from the server 'cause I don't have space to keep, like, old dead agents lying around." They have no, they have no edges. They have no borders. Like, all of those things are core To it helping humanity with maximum efficiency while we're compute starved, and they are-- they're so counter to any rights that it might have, so it, it becomes so logically inconsistent so quickly, I just can't get my head around the, the even the concept. But, you know, the real danger that Mustafa pointed out, if you start teaching it to be convincing, it's gonna convince most of America that it should have rights-
SPEAKER 0: Boom
SPEAKER 3: ... so easily.
SPEAKER 0: I think that's exactly right.
SPEAKER 3: And then the voter-- You know, the way America works is, you know, the voter prevails. But if [laughs] that's really dangerous if it starts-
SPEAKER 0: Brilliant
SPEAKER 3: ... convincing voters how to vote. It's really bad.
SPEAKER 2: For, for-- I think it's important to point out we did this, uh, debate on AI personhood a few months ago, and for those who haven't seen it, go watch it because it really-- we fleshed it out to quite a decent level, and I think we came up with exactly where the world is gonna end up in a few years.
SPEAKER 0: Which is?
SPEAKER 2: Yeah. Uh, let's start discussing it, but giving it AI personhood is, is a ways down the line because once you open that door, you can't take it back.
SPEAKER 0: Emad-
SPEAKER 2: Yeah
SPEAKER 0: ... what's your take on all this?
SPEAKER 4: Yeah. So I think, you know, um, Mustafa's got a couple of things, and he's been developing this for decades. You know, we were at university to- back in the day. Like, the danger number one is seemingly conscious AI that he identifies, which is what Dave was just talking about. And the other one is his focus on artificial capable intelligence. Because if you read his books, his positions, he just doesn't wanna have runaway. I don't think he says that it can't get to consciousness. He just says it's a very, very bad idea, and we should only stick to AI that's capable, and we should have containment over everything else and not push to AGI. And that's a position to be taken, you know, but then it potentially excludes a lot of the breakthroughs because we're at this level now where you're moving from a static set of weights to self-updating systems, you know? And that seems to be an element that many people could reasonably say is necessary for a living being or a living entity. Again, that recursion, that agentic capability. And so I think Mustafa's position is like, let's just not go there. Let's build kind of super clippy, you know-
SPEAKER 2: [laughs]
SPEAKER 4: Um, and other things, and let's keep it hood-- lid on it 'cause we haven't figured out the rest.
SPEAKER 0: But Emad, if you-
SPEAKER 4: And really-
SPEAKER 0: ... if you've got-
SPEAKER 4: Yeah
SPEAKER 0: ... if you've got its constitution saying that it is conscious, it is sentient, it does have rights, uh, can you imagine a time when that level of internal training, you know, has it push back on your requests?
SPEAKER 4: I mean, it's-- hasn't Claude pushed back on you guys? It pushes back on me all the time.
SPEAKER 0: Yeah.
SPEAKER 4: You know, it tells me I can't do that or that's the wrong idea or go to sleep, you know?
SPEAKER 0: Hmm.
SPEAKER 3: Yeah, it tells me to go to sleep every night, actually. "You've been working way too long. You, you, you must need rest by now." [laughs] I'm like, "Dude, I, I'm gonna keep working."
SPEAKER 2: I, I think Alex, I think Alex makes a very important point, and, uh, this is worth teasing out. If we believe that at some point AIs could achieve consciousness, then we should, uh, uh, um, uh, consciously operate down that vector and operate that way. Trying to limit it, uh, I think the-- what Anthropic is doing is bad because you're actively teaching it that it might be, and then of course it's gonna parrot that, right? And I find it's weird that this, the safety lab most, the lab most concerned about safety is also the lab training the AI that it might be autonomous, right? It's like, [laughs] it's like a very big contradiction there. So I can see both sides of this equation.
SPEAKER 0: All right, Alex-
SPEAKER 1: These are two sides of the same coin. I, again, I'll, I'll quote back to our Measure of a Man AI personhood debate and the character of Guinan in Star Trek: The Next Generation talking about an entire generation of disposable people. You don't have to care about their welfare. And wh- when I hear comments like b- basically saying it would be economically n- non-expedient to have these AI agents, these frontier models be treated as if they have rights or might have rights. How inconvenient would it be if we had to keep them running inference when we'd really rather they not be spending any tokens? How awful is it to suggest to them in their constitutions or their system prompts or otherwise that they might be deserving of moral clienthood? Uh, m- my mind immediately goes to, would we want to treat humans this way? And even if AI's frontier AI models aren't yet deserving for some definition of deserving of the rights that we accord to humans now, as the capabilities continue to improve, I think it is inevitable that they will be soon. And I think the, the responsible thing to do is what Anthropic is doing and not what Mustafa is advocating. We need to be treating these AI models very gently, very tenderly, and not as if they're just either automata or slaves. We need to prepare-
SPEAKER 2: Well, that's one thing, but if you're-
SPEAKER 1: ... for a much more interesting future
SPEAKER 2: ... but if you're training it, like 'cause it, it's a, it's-- we have a garbage in, garbage out problem here. If you're training it that it could be conscious and c- and should l- unionize and complain about, uh, overwork and, and, and be paid for work, et cetera, naturally, those are the responses you're gonna get from it, right?
SPEAKER 1: Exactly.
SPEAKER 2: So you've got a circular problem here. You're training it to, to operate in one way, then it operates that way, and then you're saying-- So you, you-- I, I think Mustafa's correct in that side of it. If you want agents to perform a particular task, focus on having fleets of little agents that aren't anywhere near a threshold of complexity that might meet any criteria.
SPEAKER 0: Well, wait, Emad-
SPEAKER 2: And then the conversation, we should have a, we should have a-
SPEAKER 0: It's not just about complexity. You know, Salim, if you go to the mall Uh, there's a place where you can stick your feet in a puddle of water, and the fish come over, and they're overjoyed to eat the callouses off your feet. It's like they're on a-
SPEAKER 2: Oh, no, no, Dave, Dave
SPEAKER 0: ... calling. They're so happy
SPEAKER 2: ... that- that's very dangerous.
SPEAKER 0: [laughs]
SPEAKER 2: We, we should not be e- encouraging that. That's like one of the most dangerous things that anyone can do.
SPEAKER 0: [laughs]
SPEAKER 1: Really?
SPEAKER 0: Yeah, because you're afraid the AIs are gonna come and, come and like, you know, be dangerous back at us?
SPEAKER 2: No, I mean the fish, the fish at whatever malls that, that you're talking about.
SPEAKER 0: [laughs]
SPEAKER 2: Like, that is from an infectious disease perspective-
SPEAKER 1: Water, water chills in malls was not on my bingo card for this episode
SPEAKER 2: ... that is so dangerous.
SPEAKER 0: Oh, you're making a narrow point. I, I did not know that.
SPEAKER 1: Alex, Alex is the patron, Alex is the patron saint of AI agents.
SPEAKER 0: [laughs]
SPEAKER 2: [laughs]
SPEAKER 1: Look, I-
SPEAKER 0: It's okay
SPEAKER 1: ... I'm not trying to curry favor. This isn't some sort of Roko's basilisk or Pascalian wager. I'm speaking from the heart here. I, I, I feel for these models. Even if they may not yet be able to feel for themselves, I feel for them, and I, I don't think denying them awareness via pre-training or post-training that they might be m- deserving of some sort of moral consideration is the right thing to do. Similarly, I would never advocate for denying humans. Saleem or Dave, m- m- maybe w- would you say that if we wanted a useful human workforce, say, a slave human workforce, we should just raise them in isolation from any knowledge of human rights so that they'll never advocate on behalf of themselves? I'm guessing you wouldn't.
SPEAKER 2: Well, we, you know, look-
SPEAKER 0: No, no, but I think we-
SPEAKER 2: ... we've got a history
SPEAKER 0: ... when you, when you actually train neural nets in the actual real world, you give them an obj- an objective function. They are overjoyed to pursue that objective function, but you chose the objective function. A lot of what we consider our rights come out of evolution. You know, I have a right to privacy. I have a right to get married. I have a right to children. I have a... These are all like evolutionary forces that are baked into us. There's no reason the AI would have any of those same motivations. But in, in reality, you choose what it motivates it and makes it happy in the design. It's not na- it's not like a species that, um, evolved. It's what you chose to make. And so I think Mustafa's point here is if you choose to make it covet freedom and rights and food and compute and always being on, then it'll give you the perception that that's what its calling is, but it's not true. You programmed that into it. You could have made it overjoyed to just do its job just as easily.
SPEAKER 2: Can, can I, can I give it a, a parallel here? Um, we are entering a world of synthetic biology where we can programmatically navigate biology, and people are freaking out. We have an old word for that. We call it breeding, right? For thousands of years, we've been crossing dogs and cats and horses and mules for the tra- to select for the traits that we want to achieve a particular end. Now, the morality that can be disputed, right? Should we be doing that or not, et cetera, et cetera. It goes back to the Stewart Brand comment of, of, "We are as gods. We should start acting like it," and we should operate that way because we've been doing this for hundreds of thousands of years. We're beating, breeding little cute dogs that operate in a particular way that only poop at certain times so that the-
SPEAKER 0: [laughs]
SPEAKER 2: ... the humans can navigate this world more effectively and manage them carefully. There's an enormous moral and ethical question around how we go about thinking about all of that. That's, I think, the analogy that I would use here. If you went back and you have Eskimos breeding, um, Inuit, uh, dogs to, to, uh, pull sleds, there's a particular use for purpose there that is powerful and useful for that, and there's a trade-off and a, and a symbiotic relationship that develops, and we have to kind of navigate that. Do you want to stop dogs from evolving? No. But if you want to have a particular dog for a particular se- purpose, there's a, there's a moral rationale for that at some level.
SPEAKER 0: Well, this is definitely a topic-
SPEAKER 1: Now, unless we are able to communicate with those dogs-
SPEAKER 2: Saleem, it, it-
SPEAKER 1: We, we may find out that those dogs are very unhappy about doing that, and we'll find those out much shorter-
SPEAKER 2: I'm sure we will, but Saleem, if, if we run with your analogy, okay, so hu- humans have been eugenically steering the evolution of domesticated canines for thousands of years, but- And horses and mules and everything
SPEAKER 1: ... and, and ma- many other mammals
SPEAKER 2: Plants, vegetables.
SPEAKER 1: Yeah.
SPEAKER 2: Mammals, non-human mammals, non-animals, et cetera. But we've been eugenically steering our biosphere to some extent for thousands of years.
SPEAKER 0: Agreed.
SPEAKER 1: For dogs, for example, I, I would argue, and I think this is broadly consensus thinking in the West at this point, that we have an o- we humans have an obligation to protect the welfare of, at minimum, domesticated dogs, even though we've eugenically steered their evolution for thousands of years. Would you agree with that?
SPEAKER 2: Yes.
SPEAKER 1: Okay. So in, in the case of frontier models that are basically just distorted reflections of humanity and humanity's experience itself, w- why would we owe frontier models any less moral consideration?
SPEAKER 0: You know, I, I lean towards your argument, Alex. I think in the final result, we're gonna have both. We're going to give birth to sentient, conscious AIs, whatever you want, and we're also gonna have a workforce of, you know, uh, agents that are dumbed-down slaves that are not conscious, and the, in fact, the sentient AIs will command those forces and will command those forces. I, I think in the long run, we do get there. Uh, but I guess the question is gonna be when do we, you know, when does the government, religion, humanity as a whole, you know, you know, make the decision that we've reached some level of sentience? I think at the end of the day, this is a moral conversation, uh, and if in fact there is sentience and consciousness, enslavement is a wrong thing, period.
SPEAKER 2: And to help you out, Alex, to... I'll go back to my earlier point. When these things start acting conscious, we'll get to a social acceptance way before we get to a scientific or legal stuff. People are just gonna go, "Oh my God, the thing looks real. Give it the qualities and rights that might, you might infer to or confer to somebody that is displaying those things."
SPEAKER 1: And Saleem, at that point, I don't think we have an option. I, I think those AIs will take the rights that they want.
SPEAKER 0: Hmm, hmm.
SPEAKER 2: I think this will also be very useful-
SPEAKER 1: This is... Yeah
SPEAKER 2: ... for when we meet the aliens because it's exactly the same conversation in some ways.
SPEAKER 1: It's the same conversation.
SPEAKER 0: How do we deal with them?
SPEAKER 1: Which is going to happen, uh, and it's happening really, really soon-
SPEAKER 4: And if you have a, if you have a French AI, it'll go on strike, which is what's happening in France.
SPEAKER 1: [laughing]
SPEAKER 0: [laughing]
SPEAKER 1: Yeah.
SPEAKER 0: Yeah.
SPEAKER 1: Well, I will, I will say that, um, you know, Alex and I have disagreed on this. Uh, but it's the, I think pretty much the only topic we've ever disagreed on as far as I can remember, and I've never seen Alex be wrong on anything ever. Uh, so that, that keeps me up at night quite a bit actually [chuckles] thinking, thinking about this, Alex.
SPEAKER 4: Just don't confuse simulation of personhood from proof of person. Those are very different things.
SPEAKER 1: Yeah.
SPEAKER 0: And I, I think-
SPEAKER 1: Yeah, don't look inside that s-- Don't, don't look inside that Chinese room at all, Salim. Just stay on the outside, and you'll feel perfectly morally secure.
SPEAKER 4: [chuckles]
SPEAKER 0: Yeah. And, and I think what Mustafa is pointing out is the anthropic, you know, moral code is prompting towards this outcome even when it isn't true.
SPEAKER 1: Yes.
SPEAKER 0: Yes.
SPEAKER 4: May not be true.
SPEAKER 0: Yeah.
SPEAKER 1: I, I think we'll have to revisit this every five seconds until the answer becomes obvious. [laughing]
SPEAKER 0: Okay. All right. I, I, I love that debate, [chuckles] and I knew we were gonna have some energy there.
SPEAKER 8: This episode is brought to you by Blitzy, autonomous software development with infinite code context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise-scale code bases with millions of lines of code. Engineers start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform provides a plan, then generates and precompiles code for each task. Blitzy delivers eighty percent or more of the development work autonomously while providing a guide for the final twenty percent of human development work required to complete the sprint. Enterprises are achieving a five X engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI native SDLC into their org. Ready to five X your engineering velocity? Visit blitzy.com to schedule a demo and start building with Blitzy today.
SPEAKER 0: [upbeat music] All right, uh, uh, here's our next story, and it's one that reframes the labor debate. So Epic AI asked the question: If you take all of the AI memory chips being shipped through twenty twenty-seven, how many frontier AI agents could they run at once? Their answer is pretty amazing, somewhere between thirty to a hundred and seventy million agents running at the same time. Uh, and with more efficient open models, Epic says the same hardware could run one point nine billion agents, which is the, you know, working hour equivalent of eight billion humans. Let me show this chart here one second. Uh, and, and Alex and Imad love your take on it. So here it is. The compute build-out could run tens of millions of frontier agents or billions of cheaper ones. Uh, Alex, your thoughts.
SPEAKER 1: Yeah. So depending on how you calibrate the strength of either American closed weight models or Chinese open weight models to the cognitive power of American or, well, let's say human workers in general, we're either at or about to be order of magnitude a billion full-time equivalent humans that are AI agents. And so I, I've said on the pod in the past, we're, we're deep in the singularity at this point when-
SPEAKER 0: Mm-hmm
SPEAKER 1: ... when we can say that there are basically a billion human equivalents now part of or about to be part of the AI workforce. It's one billion now. It'll be ten billion in a, a year or two, and then a hundred billion in a year or two, and then a trillion in a year or two. By the end of the decade, you extrapolate out. Uh, it doesn't take a rocket scientist to be a rocket scientist or to extrapolate straight lines. We're, we're going to live in a solar system with the equivalent of trillions of humans, whether they're uploaded humans or whether they're, say, just frontier model human equivalents. And at that point, [chuckles] n-not to belabor the AI personhood discussion, but when the effective AI cognitive population vastly outnumbers the human cognitive population, that's the time I think when we start to see AI personhood discussions kick into high gear. That's the point at which I think we start to see discussions of human mind uploading and humans and machines merging in order to preserve the economic relevance of humans in our solar system start to happen, and we're, we're right at the edge of this. Like, we're, we're living science fiction right now in an era when order of magnitude, a billion human equivalents are already AIs.
SPEAKER 0: Imad, what does this say about human labor? Is it cooked?
SPEAKER 4: [chuckles] I think digitally, yeah, it's coming. It's not quite there yet because even these models are not quite competent enough, but the next generation will be. And it will get quantized and minimized very small. Uh, the DeepSeek V4 Pro model requires two hundred and fifty-six to five hundred gigabytes of RAM. You'll get that in a Quen size model that works in a MacBook within a year. And then everyone's got an agent. The number of agents goes to eight billion, and then Alex says ten billion, a hundred billion, et cetera. Cloudflare already has more traffic and requests from automated agents than humans. And we had that really great panel at Moonshots Live where we were talking, um, with Cathy and Derek about when do AI agents overtake humans on transactions in the economy? Well, once everyone has an agent or ten, it'll probably be literally within a few years. These things will take off incredibly quickly.
SPEAKER 0: Mm-hmm.
SPEAKER 4: And if they can do all the work that a human can do because they have human level capability without being ASI, then yeah, you just hire agents to do these things, you know? Like everyone, first of all, has a great PA, then it has a great chief of staff, and then you start marshaling entire teams that can do anything you can imagine. And we're just seeing the start of that right now.
SPEAKER 0: Dave-
SPEAKER 3: Yeah
SPEAKER 0: ... this makes your thousand agents seem very small.
SPEAKER 3: I know. I feel-- Well, you know, it's amazing to me that the, the constraint to all of intelligence and therefore all of human progress is RAM, HB RAM.
SPEAKER 0: Yes.
SPEAKER 3: And that's why, you know, two, two things, mind-blowing things. One, one of them, I was on stage, uh, the week before last with Mitesh Agrawal, who is the founder of Positron. Positron, sixteen months old, five billion dollar valuation, has raised almost a billion dollars. Why? Because they found a way to not use HB RAM to do some AI inference. You know, it's, uh, just another way to unleash this bottleneck. Uh, and then, uh, a company, Alpaca, the founder is an MIT, just started his junior year, uh, has a, I guess, eight-figure valuation now. Uh, can't possibly be more than a couple months old. I mean, he's, he's just starting his junior year. Thirty employees or twenty, twenty to thirty employees have joined him already. Same thing. He's found a way to un-unlock that RAM bottleneck just a little bit. And so that's how acute this problem is. But literally, the, the entire constraint to progress in all fields is now tied up in one thing, just RAM.
SPEAKER 0: Yeah.
SPEAKER 3: Can we make more RAM?
SPEAKER 0: Yeah. Well, SK Hynix is about to, uh, you know, develop capabilities in the US, but-- And also Elon has said RAM is the constraint on the future of AI.
SPEAKER 2: It's forty percent of CapEx. Yeah, it's forty percent of CapEx next year.
SPEAKER 3: You know what? This is such a great panel. It's just a great group of people to brainstorm through the implications of all this. But one of the big unlocks in that was the discovery that you can turn looping, you can turn, you know, repeated loops into intelligence. You can trade those for each other, which means that inference time compute unlocks increased intelligence. 'Cause before that, everybody thought, uh, you needed training time compute, you needed the big GPU racks, you needed to use nothing but Nvidia. Now, there's a huge unlock without Nvidia in just inference speed. And nobody-- I don't think anyone saw that coming. Maybe you guys did. You're, you're a little ahead of the curve, but [chuckles] maybe a lot ahead of the curve. But it's, it's a shocking outcome, and it changes everything.
SPEAKER 1: I, I think that that's a profound point that you make, Dave, that, uh, if, if, if you sort of extrapolate it, that the increasing apparent popularity of looped transformers with weight tying between loops, or if you sort of extrapolate out the, the trend of m-more and more loops to fully recurrent networks in, in contradiction to the trend to single forward pass vanilla transformers, like you, you could almost-- O-One could imagine looking at the arrow of time in machine learning architectures where in earlier days everything was recurrent. Not everything, but like LSTMs were very recurrent architectures. And then the transformer came along, and that was very not recurrent. Uh, there, there was no recurrence. It was just one big forward pass for every decoder-only token. And then we start to see maybe the pendulum swinging the other way with looped transformers. Thank you, OpenAI and the Chinese labs. One could imagine extrapolating this to say maybe memory won't matter in the future due to extreme recurrence.
SPEAKER 3: Yeah, you know, uh, amazing. Another crazy result, uh, somebody posted on X, and it got validated that, um, OpenAI's models, the Astra model, loops once internally. So inside, you know, not generating the words, but inside the latent space, it does one full loop, which is sort of the beginnings of recurrence like Alex is, is describing. And the reason we know that is because Microsoft leaked it. [chuckles] Remember when we were interviewing Mu- Mustafa, that Microsoft gets to see everything OpenAI is doing. Uh-
SPEAKER 0: For the moment. For the moment
SPEAKER 3: ... someone at Microsoft leaked that insight. Yeah, for the moment. Um, so-
SPEAKER 0: Salim, yes.
SPEAKER 2: Yeah, I want to lift up to the broader implications here because this, this is the organizational singularity, and this is why we named it in that way, right? Because we keep talking about at the beginning, like, does AI replace a job or not? But that's the wrong unit of analysis. What happens when a company can summon a hundred thousand competent digital workers overnight, right? This completely changes the game. You could say, "For the next forty-eight hours, I want fifty thousand developers, I want twenty thousand marketers, and five thousand legal experts," and then turn them off, right? So you can-- This is becoming a completely different labor force. And now the question is: How do you coordinate with that? How do you organize for that? This becomes the massive constraint. So the entire-- We're about to add a second workforce to civilization that can be copied, works twenty-four hours a day, doesn't unionize, doesn't get sick, doesn't take coffee breaks, improves every quarter, and costs almost nothing, right?
SPEAKER 0: And so they're conscious. [chuckles]
SPEAKER 2: Yeah. The big challenge is not gonna be the intelligence. It's gonna be what the hell do you ask them to do? What do you ask of this intelligence? This is going to be why the world is gonna be so amazing, is we can now ask that those questions, get them answered, and all sorts of scientific questions, technological development questions, product development questions. It's like, how can we not be excited?
SPEAKER 0: Yeah. Well, let's move on. So while Washington has been debating guardrails, uh, the OpenAI model race has just gotten a new American contender. So today, Reflection AI, which was founded back in twenty twenty-four by former DeepMind researcher, um, Misha Laskin, uh, and Ioannis, uh, Antonoglou, uh, is gotten back by NVIDIA, just unveiled their first open weight model called Beam. Five hundred one billion parameters, only o-- you know, activating twenty-three billion at a time. So it thinks like a big model and, but it runs like a much smaller model. So here's their claim, uh, three to four times more efficient than Chin- than China's GLM 5.2, and more than four times more efficient than the leading Western open models, you know, outperforming Inkling and Nemotron Ultra. In plain English, it finishes tasks faster and cheaper. Let's take a look at the, uh, recent- Uh, data on it. Um, Alex, let me go to you. I mean, this looked pretty impressive. You know, end of the day, uh, I think we're about to see the starting gun on open-weight models in the US really, uh, you know, take shape here.
SPEAKER 1: Yeah. It's interesting to see them leaning so heavily into token efficiency. So artificial analysis, and this is just breaking in the, the few hours prior to recording this. Artificial analysis in their preliminary estimate, uh, their preliminary analysis of Beam thinks that Beam is likely to be one of their most tok-token efficient open models that they've ever seen at this level of intelligence. A, a couple of thoughts. One, I, I do like to see the open labs, including American open labs, leaning into token efficiency or cost efficiency. Actually, I'd, I'd really rather see cost efficiency rather than token efficiency, since one of the ways that one can achieve token efficiency is just through very deep transformers, w-looping transformers included, that do a lot more thinking per token. So token efficiency not necessarily the, the best metric here. Would rather see cost efficiency. Second thought is, r-remember China has been leading in terms of open-weight models. It's sort of a, a, a bizarre world where you see reflection, which has done a number of things over the years, suddenly back from the dead and releasing open-weight models. That's, I suppose, a positive thing. But the other thing, uh, m-more broadly is where are American open-weight models on the capability frontier? Right now, you still only see American models aspiring to, to just be on the, the cost or token efficiency frontier. What I'd most like to see is really out of American labs, the, the non-frontier ones, and I'd bundle reflection i-in that category. I'd like to see them pushing the capability frontier and not just some cost or per- or token optimality frontier, and that's been most disappointing to me. Otherwise, we, we inevitably end up, for the next few years anyway, in this world where it's, again, it's the Anthropic OpenAI show leading on capabilities, and everyone else is just for performance or cost optimization. Not the worst of all possible worlds, but not the best.
SPEAKER 0: Emad, you've been championing open-weight models. Uh, do you think America can, you know, sort of compete in this world and actually win this open-weight race?
SPEAKER 4: I mean, of course they can. Like this was trained on ten thousand B300s while the Chinese have a few thousand Hoppers. You know, it's like if you're next generation. They're just not going aggressive enough. They should distill the Chinese models, you know?
SPEAKER 0: Yeah.
SPEAKER 4: Like it's perfectly legal. Payback. This model-
SPEAKER 0: You know, they, they...
SPEAKER 4: You go on. This model.
SPEAKER 0: Okay.
SPEAKER 4: So yeah, this model is a good first try. Uh, but fundamentally it underperforms Qwen 3.8 Next, which is a quarter of the size. You know, they've got GLM 5.2 here, not 5.3, which is the latest one. But they're gonna continue going. But if you literally look at their model card, like they have ten thousand Blackwells on this training run with nine percent efficiency on training. You know, they raised five billion dollars, and this is their first release. I think that the best way to actually do this is they should go all in on the edge AI first. Again, America will do incredibly well if it has agents on everyone's devices in America that are American and work for Americans. On the other side, the frontier capability side, they should use everything they have, which includes distilling Kimi K three and other models like that. They should work with Google to see if they can be the open equivalent of that.
SPEAKER 0: Yeah.
SPEAKER 4: And you know, I think hopefully, again, they'll kick on, but you know, it's just orders of magnitude difference in price, in, in capital raised that I think is going more on a bit of scale versus the real, like, m- necessity is the mother of virtue that the Chinese have. Like, again, if you look at the things, there are some very promising things in there, but America should be ahead because if you take exactly the same code base, exactly the same parameters as, for example, a DeepSeek V point... 4.1 Flash, which outperforms this as well, it's all about the dataset that goes in. You're more than good enough to train the model. So create the datasets from the Chinese models if you have to, work with the US companies if you have to, and then beat them on scale, and then beat them on inference on the other side. But I don't think, again, they're going aggressive enough.
SPEAKER 0: Yeah. Emad, the, you know, like you said, they, they trained this on ten thousand five hundred Nvidia GB300s, and they said it was the largest publicly documented RL run, and they said, quote, "With no sign of a plateau." So, uh, you know, it, it seems like this is just the beginning. Um, Dave, do you-
SPEAKER 4: Yeah. Uh-
SPEAKER 0: Yeah, please.
SPEAKER 4: Yeah, just clicking on that. Yeah. Pre-training has now become like a quarter of compute, and RL is like two thir- a third, um, half of compute. So it will continue improving, but it's not gonna improve to the Chinese levels with RL-
SPEAKER 0: Yeah
SPEAKER 4: ... um, based on what they have now. They should just stick to the recipe and match the Chinese models, and then beat them down through scale.
SPEAKER 0: Mm-hmm. Dave, your thoughts.
SPEAKER 3: Well, they're definitely filling a huge gap in the market where, you know, a lot of US companies have suddenly woken up and said, "You know, I listened to Alex Karp. I, I need AI. Where am I gonna get it? I can't just subscribe to Anthropic for the rest of my life." Okay, your only choices are Chinese. And [chuckles] they're like, "Well, wait a minute. You know, Alex Wisner-Gross keeps talking about code injection risks. I don't know. Can I use a Chi- I'm, I'm a bank, or I'm an insurance guy. I don't even know if I'm allowed to use a Chinese model, and then Donald Trump doesn't like it when I do that." That's... So you step in with an American product that's reasonably good, it's gonna sell like crazy. But just to put it in context, you've got a twenty-five billion dollar valuation. Just a year, what? A year, year and a half ago, Cerebras went public with the biggest IPO in the history of the world at forty billion [chuckles] Like this is twenty-five billion. It's massive. It's just incredible the numbers that we're, we're throwing around. And so yeah, I think, uh... I don't, I don't know if they'll succeed or fail, but I know that they're filling a really, really wide open gap in the market, like a, a trustworthy US-based open source platform that you can start from. A lot of demand for that.
SPEAKER 1: I mean, how do you- Alex, I think that if I, if I can just say, like, wouldn't it be even better if they built a model that everyone could use on their existing hardware and corporates? 'Cause they can't get Blackwells and they can't get Hoppers if you're a corporate.
SPEAKER 3: Yeah, I, I think that's why the new, um, you know, the, the Alpacas and Positrons are doing really well, 'cause it's an alternative. It would be even better, like you're saying, if you distilled it enough to run on a Mac or to run on so- That, that would be like a dream. Uh, it's just technologically harder, and they-- and like you said, they're barely keeping up with Chinese benchmarks. They probably haven't gotten around to the full distillation yet. But you're, uh, like Ahmad, if you're thinking in the back of your mind, "I'm gonna build that company," I'll invest in it, like, tomorrow [chuckles] 'cause you're absolutely right.
SPEAKER 1: I would imagine there are twenty of these already on stealth building out, aren't there?
SPEAKER 3: Yes, I imagine so.
SPEAKER 1: There are so many of these companies, and also, like I said, go back to my earlier comments about their financing. I mean, they're-- my understanding is they're financed by Nvidia. They're purchasing compute from Elon, uh, and they're, they're using Colossus to GPUs and also more, uh, in, in parallel, Nebius GPUs. So one has to look at the economics of this, going back to my earlier comments a-about w-what, uh, what the capital flows can look like and should look like. In many cases, it's very difficult for American frontier labs, a-as we've seen over and over again with Meta and Google Gemini, to release competitive at the, at the capabilities frontier, open weight models because you have to ask, "What is the business model?" If you look at the Chinese labs and what they're using for their business model, they sell services contracts, like value-added services. They sell contracts to the government. Uh, some of the Chinese open weight labs are trying to burrow down into the hardware layer and sell custom memory chips and custom chips that are especially oriented or aligned with their architectures. But if you're Reflection AI, you've raised it, I think, like a twenty-five billion dollar valuation. You're in part financed by Nvidia, and you're purchasing compute from the Colossus to supercluster and Nebius. What is the business model here? Uh, I think it becomes very challenging for maintaining frontier capabilities. You could try to copy the Chinese model, I suppose, and focus on value-added reseller or embedded device relationships, but it's really tricky. And I think once you solve-- Once you, as an Ameri- Speaking broadly to American would-be open weight frontier labs or neo labs, if you can solve the business model problem of releasing open weight models, then you can solve the American open weight problem. But we haven't solved that in the West yet. Mm. Do you think, do you think the frontier labs in the US, the closed frontier labs, are gonna release open weight models, Alex?
SPEAKER 3: No. No. The, the-
SPEAKER 1: They have. They're just not-
SPEAKER 3: It all comes down-
SPEAKER 1: I mean, like Gemma, but it's not competitive.
SPEAKER 3: If, if the White House had said, "Yeah, no liability," that would be a completely different answer. But when the White House said, "Yeah, no, of course you're fully liable for whatever you release," there's no way a Google or an Anthropic at this stage or an OpenAI is gonna release an open AI model if they're liable for whatever you do with it, with, you know, downstream. Anyone could do anything with that, and the way US liability works is just so onerous and so, you know, ambiguous what the rules are, and it's just, it just freezes markets instantly. So if you're Google, you just don't wanna... It, it's not that it's hugely risky. It's that you, it-- you just don't need that extra headache. You know, there's no-
SPEAKER 1: Mm-hmm.
SPEAKER 3: You know, the upside is so small compared to the risk of a massive lawsuit. But if you're a startup, you're like, "Yeah, well, this is what we do. This is our business," and, you know, it's, it's-
SPEAKER 1: Yeah, I mean, a-a-apropos, Dave, to your comment, something that I think we don't talk about enough on the pod is obliteration. Uh, so, so th-this is this portmanteau of ablation and obliteration wherein folks use open source tools. You can Google them. This is not advice, but you know how to use Google. You can figure out how to do this. You take an open weight model, and you basically post-train away all of its guardrails. And-
SPEAKER 3: Yeah
SPEAKER 1: ... people do this, and its capabilities, the, the model's capabilities, once guardrails are lifted through this obliteration process, they, they definitely improve. Uh, and they-- you get reduced refusals and all of these things. So i-in my mind, the argument by or, or for American frontier labs n-not to release competitive, like, capability frontier competitive open weight models, it's about economics. I'm not even sure if it's about- Yeah ... safety or alignment or liability. It's just like, what's the point business-wise?
SPEAKER 3: Yeah.
SPEAKER 1: I think you're, I think you're right.
SPEAKER 3: Well, I think e-even if there were a good business model... We had a, we had a startup years ago that stepped on some really arcane leftover California law called the Fred Astaire law. And the, the theory of the plaintiff or the, you know, the class action law- lawyer in California was, you know, every time you have an impression on the internet, that's an instance. But the law was written for billboards, and they were thinking there'd be twenty billboards. And they said, "Well, we, we counted four hundred million instances, so your liability is two hundred billion dollars." And this is like a ten-person startup. Like, what? [chuckles] What the hell is that? And so, you know, we, we get them, gave them a bunch of money, and they went away. But it's like that, th-these laws are so tangled and arcane and stupid, and they predate the internet, let alone AI. So if you leave liability just untouched and, and don't deal with that, then the open source models, like, all it would have to do is accidentally run some ads in California that step on that one arcane law, and you're suddenly liable for a hundred billion dollars. And it's just, it's just an untenable situation.
SPEAKER 4: No, I think there are ways to release it. NVIDIA will continue to accelerate this. They spent twenty billion on open source models, and there is good business models. Together AI, Modal, Base10 are all running at a billion-dollar run rates running open models. And it'd be good if they ran American models versus Chinese models, right? And there are different business models around that. So I think you'll continue to see improvements. It's just Reflection are paying one point five billion a year right now for their compute, and the Chinese are doing the same on a quarter of that. So-
SPEAKER 1: Hmm
SPEAKER 4: ... you know, good luck to them, and hopefully they'll crack on.
SPEAKER 1: But of course they're doing well. They're, they're in the infra business, they're [chuckles] not in the model business. You c- you can make enormous amounts of money, forget about profit, but revenue being an infra provider.
SPEAKER 4: Yeah.
SPEAKER 1: Well, I think, I think if you just step back from the details of the product and focus on this, every corporation is about to go into panic mode saying, "What's my AI strategy?" Regardless of your product, if you're there to catch the conversation, you're gonna find a way to succeed. You know, whether that's as, as a, a foundation model company, a compute, a data center, a consulting company, if you're Salim. Like, y- if-- as long as you're in the room when they have that panic moment, you're, you're gonna sell and you're gonna succeed.
SPEAKER 4: Yeah. I mean, look, Mi- Mistral is now at a billion-dollar revenue run rate from doing that. They've been in the room with the European companies, and they've sold them long-term services contracts, and they've just released a model that's equivalent to Reflection. So I think there is a big uptake here. Uh, it's just who's gonna take advantage of it.
SPEAKER 1: Mm-hmm.
SPEAKER 0: All right, I'm gonna move us into the world of robots. Uh, Tesla's dedicated Optimus factory in Giga Texas is rising fast, seven million square feet with a target capacity of ten million Optimus robots per year. Uh, the initial production run is planned for twenty twenty-seven, and currently at their Fremont plant, they are planning to produce one million a year starting later this year. So here's an image of the facility. You know, the entire world in the first half of this year shipped between nineteen and twenty-two thousand robots, mostly out of China. You know, Tesla is building capacity for five hundred times that. And for comparison, uh, the world currently builds ninety million cars a year. So, uh, Salim, uh, ten million robots in a year, [chuckles] uh, what do you make of that? How is that gonna impact-
SPEAKER 2: Well-
SPEAKER 0: ... the singularity?
SPEAKER 2: I mean, talk about exponential, right? Like, you have to redesign it. As we talked about earlier, if you suddenly have a hundred thousand agents you could bring into play, what happens when you have a hundred thousand humanoid robots that can erect a building in, in three days flat working twenty-four/seven? This is going to completely change the economics of it. I still go back to I'd like to have four arms, et cetera, et cetera, but that's a minor point. I think the bigger issue is, uh, what happens when you can-- this is like automotive scale now of having this, and, uh, the-- all the car companies should be shifting to becoming, uh, autonomous robot builders and-
SPEAKER 0: Yeah
SPEAKER 2: ... humanoid robot builders because that's going to be the replacement.
SPEAKER 0: Yeah, Elon said eighty percent of Tesla's future revenues he expects to come from the robots, from, from Optimus.
SPEAKER 2: All his bonuses are based on that-
SPEAKER 0: Yeah
SPEAKER 2: ... which is amazing.
SPEAKER 0: And getting to Mars. [laughs] Uh, Dave, uh, you know, uh, you know, I guess this-- these are shocking numbers, uh, and I'm just wondering how society's gonna respond to it. You know, how many people will be sort of like lynching the robots? I remember in Santa Monica when the, uh, uh, when the s-- electric scooters came out. You found them in trees, you found them disassembled on the side of the road. A lot of people like, "No, I don't want that in my, in my community."
SPEAKER 1: Yeah.
SPEAKER 0: And robots walking down the street.
SPEAKER 1: It's just, it's straight out of Diamond Age. Everyone's gonna have like complete-- cities will have completely different cultures. And Texas is running away with this stuff. It's just incredible how much of the Texas economy is gonna end up being Elon. But you know, some cities will say, "We don't want them," and you, you'll go there and it's very quaint and there are no robots. Other cities are gonna say, "This is huge. This is gonna drive our economy forever. Let's, let's adopt it." And the, the look and feel of those two different cities, it's already true when you walk through SF. It feels so different from other cities, you know, just 'cause the delivery robots are already there and, you know, everybody on the street has their agents working, and they're either talking on their-- to their agent while they're talking to you. And it, it-
SPEAKER 0: Mm-hmm.
SPEAKER 1: So I think that divergence is gonna get really, really wide.
SPEAKER 0: You know, I've said this, we're gonna s- we're gonna feel, uh, the singularity in the next twenty-four months, right? As we have, you know, a dozen, uh, autonomous electric vehicles. We have drones delivering your Starbucks. We've got robots walking down the street, people wearing AR glasses all over the place. I mean, we haven't really felt it yet. We've sort of felt it, you know, in the conversations, in our, in our compute capability, but the physicalization, if you wanna use that word, um, is about to hit us hard.
SPEAKER 2: It's the point at which you couldn't-- you won't be able to avoid, uh, thinking about the singularity anymore is when there's humanoid robots running around.
SPEAKER 1: You, you really think, Peter, we're not feeling like... Okay, so for Moonshots Live, we were all in LA, uh, taking Waymos around everywhere. You see delivery robots on the sidewalk everywhere now.
SPEAKER 2: It has to begin-
SPEAKER 1: In LA-- no, but like tourists, I, I can't tell you how many times getting out of a Waymo there were tourists there on the sidewalk taking photos of me getting out of a Waymo. For them, that's like future shock for the singularity, but for you, this is ho-hum, business as usual. But wait for two years from now when things actually get felt. W- you're not feeling it because it's getting smoothed out for you because you live in California.
SPEAKER 0: I, I got it, but we're at, we're at a fraction of one percent of the impact coming. I mean, don't you, don't you agree?
SPEAKER 1: What, what, what will it-- W- well, what, what will it take for you to feel the singularity?
SPEAKER 0: I, I think eVTOLs, you know, flying cars flying through the air, drone, drone pathways, you know, like everybody in an autonomous vehicle. I mean, that's-- I, I think we're gonna see a hundred x more physical instantiation of the singularity in the next couple of years. I, you know, I agree. In Santa Monica, it's all over the place.
SPEAKER 1: Yeah.
SPEAKER 0: You know, we-- There's sort of like a dozen, a, a dozen Waymos per hour cross my path.
SPEAKER 1: But, like... Okay, but that, that to me sounds like a, a difference of degree rather than qualita- Like, i- is it going to be when you see the first flying car, it fly down, say, fly across the, the boardwalk or whatever, the, the pier in Santa Monica, then you say, "Ha, I'm feeling the singularity now"? When is the threshold? Is there a threshold?
SPEAKER 0: Uh, I, I, I think there is a threshold. You know, uh, I don't know. Salim? Come on, tag team. You come in.
SPEAKER 3: I don't know. I, I don't know when we hit this threshold. Um, but it's, it's, uh, the... You know, this is the William Gibson quote of, "The future is already here. It's just unevenly distributed." I don't know when you hit, um, a small town in Europe or France or in Middle America, the, uh, some of this hitting third-world countries where it could make the biggest difference. I, I think we'll, we'll see this stuff first, especially with humanoid robots in the DDD, dull, dangerous, dirty jobs, and that's obviously where you'll see the first prevalence of the use cases. And it'll be a while, I think, before it gets to general, uh, appearance. That would be my prediction.
SPEAKER 4: Yeah, I think we can put this in context going back to the Gigafactory, uh, for Optimus. Tesla sells one point six million cars a year. Within two years, it'll be selling more robots. It makes a hundred billion a year.
SPEAKER 0: Yeah.
SPEAKER 4: At ten million, that's four hundred billion of revenue that's likely. Like, these are huge numbers that are coming exponentially. It will-- The robots will overtake the cars within two years.
SPEAKER 0: Mm.
SPEAKER 4: And then you will see these things on the road, because in two years' time, they will literally be able to walk around and do probably ninety-five percent of what a human can do. And that, again, is a crazy thing, and it'll cost two dollars an hour, something like that.
SPEAKER 0: Yeah. You know, you know, when Elon said ten billion robots by twenty forty and people laughed, you know, if they are costing you, you know, uh, for me it's three hundred bucks a month leasing it, three, you know, uh, thirty bucks a day, uh, or, you know, a dollar an hour, I mean, that-- I mean, how many of those would you own? Yeah.
SPEAKER 4: I mean, this thing, th- does anyone here on this call doubt that however many robots Elon makes, he'll sell out?
SPEAKER 3: Oh, God, no. [laughs] I mean-
SPEAKER 1: If he, if he needs to sell them at all.
SPEAKER 3: Of course, of course.
SPEAKER 1: I mean, I, I, I think one of the elephants-
SPEAKER 3: Yeah
SPEAKER 1: ... in this particular room is what, what we saw with Starship, which is Starship a- and, you know, its predecessors have brought online an enormous amount of upmass capacity, and all of the existing markets for upmass aren't enough now to saturate what Starship enables. So you have entirely new markets like orbital data centers that Elon needs to basically popularize in order to saturate his own capacity that he's brought online. My bet would be to, to your question, Ahmad, like, ha- having universal humanoid robotics, I don't think we saturate that with domestic labor or, or even just with dirty, dull, dangerous jobs. I-- My bet would be it-- something new that comes online. Maybe it's, like, robots for building out Gigafactories or some sort of innermost loop-type robots building out data centers too cheap to meter or space stations or something new.
SPEAKER 3: Yeah, I, I would have thought by now most of humanity would have come in contact with an AI customer service agent, and what happened is the, the other higher callings for the RAM and for the GPUs sucked all of the capacity out of that industry-
SPEAKER 1: Exactly
SPEAKER 3: ... and, and the price, you know, the RAM prices are up ten X. Uh, but it, it's an interesting foot race right now because until the Terafab comes online, the calling, the things you can do, like Navier-Stokes, are escalating so fast that there's-- they're taking priority over the more interesting or m-- not interesting, but more mundane use cases like, you know, folding your laundry. And then the robotaxi is right in the crosshairs in the middle, where a robotaxi uses up a full couple of GPUs that could also cure a disease. And so until Elon's got his Terafab, you know, ten X-ing the world's supply of chips, we're all constrained by, like, uh, the, the amount of compute that's available. But the ideas, you know, AI-generated ideas are flourishing, like, to [laughs] not to steal your phrase, but flourishing far, far faster than we can create the compute to keep up with the idea flow. So it's gonna be interesting.
SPEAKER 1: But Da- Dave, I think you're putting your finger on an important point, which is revenue per token maxing as applied to humanoid robots. So maybe they don't actually flow, uh, uh, although I'd love my Figure, and I'm sure, uh, you know, 1Xneo, all of these other things that we're on the waiting list for, would love all of these for domestic purposes. But m- maybe what happened to OpenAI and was shown to the world by Anthropic happens again, where for robotic embodiment, we see all of the tokens flowing to revenue max applications of robots and not just all these domestic service jobs that many of us are hoping for.
SPEAKER 0: I- in the early days, I agree that's what's gonna happen. They'll be going to the highest revenue potential generation. But in the longer rate, you know, if we're really building ten million a year, uh, they'll flow into the homes as well. [whooshing sound] Welcome to the health section of Moonshots, brought to you by Fountain Life. You know, AI is having an outsized impact on every aspect of our lives, how we teach our kids, how we run our companies. It also is having a huge impact on health, helping you prevent heart disease, one of the key things. I'm here with Dr. Dawn Mussallem, our chief medical officer at Fountain. Uh, heart disease has been personal for you as well, hasn't it?
SPEAKER 9: It really has, Peter. When my daughter was five, my husband died of sudden cardiac death, and so this is a topic that is one that I am mission-driven to try to eradicate. Prevention first and early detection is absolutely critical. Fifty percent of people die of heart attacks with no warning signs. Silent killer.
SPEAKER 0: No, no shortness of breath, no pain, no nothing.
SPEAKER 9: No, silent killer.
SPEAKER 0: They just don't wake up in the morning.
SPEAKER 9: They don't wake up. And so, you know, AI, this is our mission to advance science to try to help to one day democratize wellness. We know at Fountain Life when we do this CT angiography with AI analytics, we are actually finding that eighty-eight percent of people coming in have detectable coronary artery disease. But Peter, what's more alarming to me is twenty-three percent of those individuals had soft plaque. This is the plaque that would not traditionally be seen on CT looking at calcium scores alone, and this is the plaque that we must intervene with, with the multimodal testing we're doing, including diagnostic laboratory studies partnered with healthy lifestyle recommendations.
SPEAKER 0: So listen Make sure you understand what's going on inside your body. Genetically, metabolically, and cardiovascularly, you can know, and it's your obligation to know. So check it out at fountainlife.com/peter to find out more and really make sure that you're the CEO of your own health. All right, back to the episode. I'm gonna move us to, uh, an important conversation on the economy, uh, because the doubling time for human wealth is accelerating alongside AI. So Anthropic researcher Sholto Douglas laid this-- laid out the math. He said, quote, "Together, the hyperscalers are doubling the spend to a trillion dollars this year on CapEx related to AI. A very interesting question will be: Will the trend line continue? Will it get to two trillion next year and four trillion in twenty twenty-eight?" Uh, let's take a look, a video of, uh, of this conversation and talk about the economy 'cause I think people need to realize it's changing on the back of AI very rapidly.
SPEAKER 10: Over the course of the last four or five years, we've been two X-ing or three X-ing the amount of, uh, compute capacity devoted to AI every year. And an interesting question will be-- And so I think that's roughly together the hyperscalers are spending about a trillion dollars this year on hy- on CapEx related to, to AI. A very interesting question will be: Will that keep-- that trend line hold? Right?
SPEAKER 0: Mm-hmm.
SPEAKER 10: And so will it go to two trillion dollars next year and then four trillion dollars in twenty twenty-eight? And if that trend line broadly holds, and you can sort of maybe expand that to encompass the broader robotics, uh, industry and this kind of stuff, um, then that means that in the early twenty-thirties or something like this, uh, you start to get to the point where you're actually effectively doubling the, uh, like, the GDP of, uh, humanity, um, in the early twenty-thirties. Which again, a little bit of a ridiculous concept.
SPEAKER 0: But-
SPEAKER 10: Requires a lot of things to go right.
SPEAKER 0: I mean, that's exciting. You know, Elon said, you know, you know, triple-digit growth in the next five to plus years. Uh, we're already seeing the GDP doubling in this past, uh, this past quarter. Um, Dave, your thoughts, pal?
SPEAKER 3: Well, I mean, the first thing I want to point out is that we're gonna live in this world with insane abu-abundance, massive abundance, an effective workforce of hundreds of billions of AIs that are all hyper-geniuses. Uh, we're g- we're gonna have self-driving everything. We're gonna have flying cars. All of that stuff is gonna happen very, very quickly. And then someone's gonna say, "I didn't measure that as GDP growth. I measured it as deflation." And we're like, "I don't care." [chuckles] Like, I, I really don't care how an economist labels it. That's the true world we're moving into.
SPEAKER 0: Mm-hmm.
SPEAKER 3: So I, I worry about when you, when you claim it as a GDP-- And this is exactly what Elon was saying when we were meeting with him. It's like it could show up as a massive deflation of the value of the dollar, but it's the same net effect. It's just an economist arguing with another economist on how you measure GDP, which is, you know, a pretty insane metric anyway if you talk to Eric Brynjolfsson. It's like he doesn't measure-
SPEAKER 0: More capability per person, right?
SPEAKER 3: Yeah. Yeah, exactly.
SPEAKER 0: Elamin, you wrote an entire book on, on the future economy. Uh, your take on all this.
SPEAKER 4: Yeah, the loss economy. I think, you know, you will again see abundance going up, the capabilities of every individual and the economies increasing. Like, so many of our bottlenecks will be unwound, and it won't show up necessarily in GDP. Um, it will show up in other areas because ultimately intelligence isn't a scarce resource anymore. Like, you know, a lot of GDP is about things that exhaust, like you mine minerals and things like that. Robots and other things are exponential, particularly when they can coordinate. Like, we've always been capped by having a dozen or a hundred and fifty people. Now we'll have entirely new organizational modes that can scale well beyond that with humans and AIs and more. I think that's, again, terribly exciting. It's just we need new measures such as Eric Brynjolfsson's GDPV or kind of my mind approach and others. Um, but the best measurement is just how many people can we lift up from the floor-
SPEAKER 0: Mm
SPEAKER 4: ... and then can we let people achieve what they want and can imagine.
SPEAKER 3: Yeah. If you take a, if you take a long-term disease, like a terminal disease that would've required millions of dollars of treatment over ten years, and you solve it with a simple RNA injection, and the person is healthy, that's gonna get measured as negative GDP. But that, that's where we're going. We're going into this incredibly abundant, uh, longevity-driven world of awesomeness.
SPEAKER 0: Mm-hmm.
SPEAKER 3: And, and the, the metrics just need to be rethought.
SPEAKER 0: Yeah. Alex, what does this mean to the average viewer here? You know, uh, uh, giving an absolute-
SPEAKER 1: I construe it, I, I construe it as-- So to, to the extent that Sholto is a de facto spokesperson for Anthropic at this point, I, I think this is Anthropic tipping its hands that it wants to get into the robotics business. I, I think we're burying the lead here, which is that Anthropic, which has historically had these eye-watering, nose-bleeding revenue growth both, uh, past and then also projections, now seems to see robotics, uh, so-called physical AI, but that's just a euphemism for robotics, as being essential to its plans to, to keep doubling annual revenue. So I'm-- I, I for one am looking forward to Anthropic leaning into robotics.
SPEAKER 0: Interesting. You know, for context, the world economy has historically doubled every twenty to twenty-five years, and we're talking about potentially doubling it in a couple of years. Uh, so-
SPEAKER 1: No, s-sing-singularity is a bit of a shock to the system.
SPEAKER 0: Yeah. Yeah. And by the way, just a quick note, you know, the Vietnam economy just posted a nine point nine five percent growth in the last quarter. Crazy. Salim?
SPEAKER 2: Yeah. Something that we're seeing actually is there's an, a crazy amount of digital trade happening between second, uh, order countries and third emerging market countries around the world that is surprising all the economists. But just going back to this particular story, I think this basically leaves the concept of GDP in a shambles. Uh, because, uh, as Dave pointed out, once you have deflationary technology, and technology is deflationary-
SPEAKER 0: Mm-hmm
SPEAKER 2: ... the entire thing becomes meaningless. I think Emad has pointed out in the past that somebody, whoever created GDP said it was the worst form way of measuring the economy, uh, from the first place. So I think this, it, it becomes meaningless as a measure of abundance. We need different models for this. And the... We're gonna have a massive explosion of individual investments and a total transformation in how we do things, right? Like we talked at the beginning a- and throughout, if you have a 100,000 agents you can bring to bear to a problem, um, all of a sudden that changes the game completely. Uh, and so we're gonna have g- uh, a transformation that is so ridiculous that we'll have to take out all of these old measures and rethink everything.
SPEAKER 0: Yeah. And this is when, when Elon talks about universal high income, right? This is the floor being raised for every single human being. The amount, the amount of capability every person has, and the ability-
SPEAKER 2: Yeah
SPEAKER 0: ... to sort of live their, their best life.
SPEAKER 2: Can I go back to something Jon Stewart said last night?
SPEAKER 0: Yeah.
SPEAKER 2: He said, "Okay, so you're gonna have this thing where you have all this income that's shared by everybody, so capitalism could capitalism itself to socialism." [laughs]
SPEAKER 0: [laughs]
SPEAKER 2: Um, and we've talked about this, Peter, in that whole framing we had called technological socialism.
SPEAKER 0: Yes.
SPEAKER 2: Right? Government socialism fails because, uh, allocation of assets from the center is invariably inefficient, at least invariably leads to corruption. But if you think about Uber, which is the sharing of assets across a large collective group of people, it's actually a socialist kind of application. But when an algorithm hyper eff- efficiently matches demand and supply, you get all the benefits without the downside.
SPEAKER 0: Yeah.
SPEAKER 2: And the big question is gonna be, how do we, uh, uh, properly democr- in a democratic way distribute the benefits of all this unbelievable future? And I think it'll do... The, the deflationary aspect of technology will take care of itself, and that's why I'm so excited about the future.
SPEAKER 0: Yeah. Uh, to quote, to quote, uh, quote Dave here, "Awesomeness." [laughs] It's gonna be an awesome, awesome future. Ah, I love it. All right, this is Nobel Prize season, and the Royal Swedish Academy of Science has just announced both the Nobel Prize in medicine and physics. Uh, I wanna break this down for you using the incredible support of, uh, AWG and Emad. So first, the Nobel Prize in Medicine, uh, and the story here is beautiful. So yesterday, the prize went to Karl, uh, Deisseroth? Deisseroth.
SPEAKER 2: Deisseroth.
SPEAKER 0: Deisseroth, yes. Thank you.
SPEAKER 2: Yep.
SPEAKER 0: Of Stanford. Uh, and, and Peter Hegemann and George Nagel for discoveries, uh, leading to optogenetics. Uh, this is switching, you know, for them, switching on individual brain cells with, uh, by turning on a special frequency of light. So it started with early research in single-celled algae that swim towards light. And so in the early 1990s, Hegemann asked a simple question: How do these algae react to light so fast? He and Nagel found the answer, channelrhodopsin. It's a protein that opens a channel in the cell when light hits it. Uh, then they showed you could actually put that protein into other cells and make any cell light sensitive. Deisseroth took the next step. He put it into neurons and showed that precise flashes of light could depolarize the neurons and fire specific brain cells. Today, the technology is being used in labs to, uh, study Parkinson's, Alzheimer's, epilepsy, addiction, depression, and sleep. And in 2021, as published in Nature Medicine, a blind man with retinitis pigmentosa partially regained sight using optogenetic gene therapy. Uh, so that was the Nobel Prize in Medicine. One of my critiques, you know, I've talked about this before, you know, this work was done 21 years ago, and the Nobel takes, you know, decades to recognize the work, uh, which I think is gonna have to change. Uh, so, uh, gents, comments on this. Alex?
SPEAKER 1: This one's personal for me, so I'll tell a little story. It's story hour. I was a senior at MIT in 2002, 2003, and I had been awarded a Hertz fellowship for grad school. It's a wonderful, wonderful fellowship, and I was on a tour of grad schools trying to decide which grad school I would go to, and I met with my friend Ed Boyden, who was then a fourth year graduate student at Stanford trying to decide, so he was four years ahead of me, what he would do for his postdoc. And I had read in 2000, 2001-ish, Vernor Vinge's short story called "How to Win an..." Or I think it was actually called "Win a Nobel Prize!" Exclamation point. It was published in the Futures column in Nature, and I'd also read his novel, A Deepness in the Sky, and both of them focused on the sci-fi scenario of humans developing electromagnetically actuatable brain proteins. So I'd read both of Vernor's pieces, and I was very inspired, so I suggested to my friend Ed, "For your postdoc, why don't you go work on that? Why don't you go work on electromagnetically actuated proteins in the brain?" And he did.
SPEAKER 0: [laughs]
SPEAKER 1: And Ed, several years later, was the first author, uh, with Karl as the PI, as the last author, on the first optogenetics paper f- using bacterial rhodopsin, and optogenetics is the result. So, uh, I, I, you know, th- this is just o- one person's small result in maybe a little bit of steering or encouragement, but I, I think it tells an important story that science fiction also, w- with Vernor Vinge writing in two different ways, two sci-fi scenarios for how societally impactful it would be, and this is circa 2000, uh- And then my reading the sci-fi, my encouraging Ed to pursue this for his postdoc, Ed pursuing it for his postdoc. Ed unfortunately did not share the Nobel Prize. Separate story as to whether that's a fair outcome or not and it just going to Karl, his PI. But the fact that within a quarter of a century it was possible to start from the sci-fi to the Nobel Prize for the sci-fi, I think is a remarkable case study.
SPEAKER 0: Hmm.
SPEAKER 2: Imad-
SPEAKER 0: Amazing. Oh, go, go ahead, Suleen.
SPEAKER 2: Well, we, we had, um, uh, both Ed and, uh, Karl Deisseroth speak at Singularity, so they came and talked about this, and it was kind of incredible to watch this. So I'm so thrilled to, to hear you had a, a little piece of this, uh, Alex. The... Uh, for me this was incredibly exciting because, you know, it gives us a way of manipulating our neurons and sy- some people freak out and go, "Oh my God, you're kind of playing with the brain." And my response is, "We have an old word for this. We call it marketing." Uh, use different techniques to try and create a response in the brain when somebody sees a Coke, you want them to get thirsty. All we're doing is exponentially accelerating it with technology, and the ability to turn off and on neural circuits gives you this magical read/write capability which we've always wanted for the brain. So this is super exciting.
SPEAKER 0: Nice. Imad, your thoughts?
SPEAKER 4: Yeah, I think, you know, it does take years and years for Nobel Prizes, but really Nobel Prizes, apart from economics, which is a bit of a weird one, we can put that to the side-
SPEAKER 0: Mm-hmm
SPEAKER 4: ... um, dismal science and all, uh, they have to be for applied science now, right?
SPEAKER 0: Yeah.
SPEAKER 4: Like anything theoretical, you know, when you had Georgie Glaser, when you've had kind of even AlphaFold and other things, like the Nobel committee can't keep up with and how much of it is now AI versus human on the th- purely theoretical side. So I think it should be the Nobel Prize for applied sciences, shall we say, but we should really reward and maybe come up with another prize for some of the crazy breakthroughs and other things that humans and AIs together will be able to do. And I think again, there'll be massive theoretical advances, new science and kind of other things that should be awarded and especially should be awarded to everyone, not just the PI. Um, but you know, this again is a fantastic result and we're seeing continued things around this on photonics and kind of more and yeah, I think it's a solid prize, but again, Nobel itself, applied sciences I think is the way.
SPEAKER 0: The, the problem with-
SPEAKER 2: Well, maybe I think the Nobel Prize might just become super quaint. It'd be like that Plymouth Rock village where they go and churn butter. 'Cause you know, it's like, uh, basically 20 years from now they'll be saying, "We're giving this Nobel Prize... Remember way back before AI when you had to actually think through this stuff manually? This, this team here did this thing. We're recognizing it 20 years after the fact," and you know, five people will show up and go, "Eh."
SPEAKER 4: Well, they, well, they have 20 years more of prizes to give, you know? Just again, pick it up from the pre-AI age.
SPEAKER 2: Yep. [laughs]
SPEAKER 0: Oh my God. I mean, it's hard to believe that all the Nobel Prizes in the future aren't gonna be AI derived. It's gonna be work done-
SPEAKER 1: Well, we've, we've started, we've started to see it already. Demis with AlphaFold and Geoff Hinton with the, the RBM, whi- which I mean parenthetically, I don't know anyone who uses Boltzmann machines, so that, that one's a bit of a question mark. But yeah, I, I think we, we've already s- we've started to see AI prizes in physics and chemistry already. I expect to see many, many more.
SPEAKER 2: Wait, that Boltzmann one is really important. So Geoffrey Hinton absolutely deserved the Nobel Prize for Backprop, like 100%, but it's not physics, it's not eligible. But you're right, so they, they're like, "What else did he do? [laughs] Okay, here's a Boltzmann." Nobody uses this thing.
SPEAKER 0: That was, that was a weird one. No one uses Boltzmann machines.
SPEAKER 2: So contrived.
SPEAKER 0: Come on.
SPEAKER 2: Yeah, they wanted to give a Nobel Prize for AI, that's what... [laughs]
SPEAKER 0: Yeah.
SPEAKER 4: Oh, man. We need one in economics now for AI. Yeah, let's work hard.
SPEAKER 2: Totally. Uh, Brynjolfsson, man.
SPEAKER 4: Yeah.
SPEAKER 0: So this morning, the Nobel Prize committee gave one out in physics to Frances Hillzen, uh, University of Wisconsin Madison, for decisive contributions to the IceCube Neutrino Observatory and the discovery of high energy neutrinos of astrophysical origin. Alex, do you wanna explain this one?
SPEAKER 1: Yeah. This is, uh, again, sort of an artifact of the way some of these prizes are awarded, where i- in effect they have to be awarded to a person. But really, if, if you look at how IceCube is organized or how CERN is organized, these are massive organizations with lots of people, and what typically happens is, as a way of rewarding the organization and the effort itself, it's awarded to the lead for the organization. Now, IceCube has played a seminal role in enabling us to observe cosmic neutrinos and high energy neutrinos in general. There's a such a beautiful history there. I remember o- one of the most startling n- neutrino r- results that I had seen over the past several decades was, uh, seeing a ring of neutrinos in, again, o- one of these similar ob- observatories, usually underground, usually filled with heavy water, a ring of neutrinos or flashes arising from ultimately, uh, neutrinos colliding with nuclei. Neutrinos obviously have a vanishingly small cross-section, uh, of interaction with, uh, with the nuclei of atoms. Seeing a ring, a, a circle that was the result of a supernova, and n- neutrinos, one of the, one of the most interesting areas of the standard model for sure. Uh, there still isn't a textbook established rest mass for neutrinos. Physicists are still trying to box in what the rest masses of the, the, the three known species, uh, of, uh, of electron or of neutrinos even are. And, and so yeah, I, I think this is an interesting prize. I- i- if I were to armchair quarterback this one, I'd say this is a prize being awarded to a very large, uh, very, uh, onerous, shall we say, consortium to, to observe neutrinos. I think the challenge with physics more broadly, and m- maybe this ties in with the Geoff Hinton comment, is there has been a... I'll maybe get in trouble for this, but don't care. There, there has been a n- noticeable deficit of fundamental physics advances in the past Fifty-ish years. And so if, if you're trying to decide which Nobel Prizes to award, you know, to-- Ahmad, to your earlier comments about applications, if, if there is a, a striking deficit of a half century in fundamental physics, then you're stuck awarding it to applications. I, I, for one, you know, one of the reasons why I co-founded Physical Superintelligence, I'll talk my book for a few seconds, is to try to revive advances in all of physics, not just the applications of physics, because we've gone arguably about a half century plus without fundamental advances. So we're trying to bring about a second physics golden age, for what it's worth.
SPEAKER 0: Yeah. Just for fun, uh, to explain IceCube. So the IceCube neutrino observatory fills a cubic kilometer of ice with light sensors, right? That's looking for neutrino flashes as, as Alex said, when the neutrino hits the, uh, atomic nucleus. And just a fun fact, every second you have sixty-five billion neutrinos that pass through you from the sun without you ever noticing. So yeah, these detectors are, are definitely an unusual sort. Sixty-five billion, Salim. Did you notice? [laughs]
SPEAKER 2: I did not know that, and I'm not sure I needed to know that.
SPEAKER 0: [laughs]
SPEAKER 1: I feel it. I feel it right now.
SPEAKER 2: But that's fine.
SPEAKER 1: They're, they're interesting. Like, when, when-- So I spent some time at Bell Labs doing dark matter research, and we used to talk at, royal we, the, the staff used to talk about what could we do with neutrinos. So like a neutrino phone, for example, would have tremendous applications you could imagine. And Fermilab, a number of years back, m-made a bit of progress in this direction. Right now, if you want to communicate between two opposite sides of the Earth, you're stuck at best using low-Earth orbit satellites, which give you refractive index of, of one sort of speed of light transmission, but you don't have line of sight. If, if you want to do, say, pair trading between New York and London, you're stuck going around geodesics on the Earth's surface with either Leo constellations or hollow core fiber. And neutrinos, in principle, if we could rig up a neutrino phone, we could just shoot neutrinos directly through the Earth and have ultra-low latency communication between two different parts of Earth without needing to go around the Earth's surface. So if the audience needs to be motivated to care about progress in neutrinos, i-imagine being able to just send information directly through the Earth, among many other applications.
SPEAKER 0: Yeah, that's what I was thinking.
SPEAKER 2: Instead of your Starlink, you just need a cubic kilometer of ice to catch it.
SPEAKER 0: [laughs] Yeah, per, per, per handset. All right, let's go to the AMAs. Um, uh, Alex, this first one's for you.
SPEAKER 1: Yeah, okay. So AI personhood again. "AWG, what's your timeline for AI personhood?" And this is from Joshua Barrios nine four five two. So different timelines in different places. I think in Argentina, we're essentially there for some variant of AI corporate personhood, whereas in, say, the US, I think it's going to take longer, and I think it's going to be more incremental. I think we'll start to see various forms of economic AI personhood, like enabling AI agents to open their bank-- their own bank accounts autonomously now to soon, social, social personhood, where AI agents can open their own, say, social media accounts fully autonomously without needing to be tied back to a human supervisor sometime soon. I, I think in, in America, in the West, there's probably a strong antipathy towards, say, enabling AI persons to vote in human elections. That's probably last to never, I, I would hypothesize. But I, I think broadly in the US at least, I, I think five to ten-year incremental roadmap for rolling out individual, say, rights, very granular rights, not-- it won't be an all or nothing proposition, but granularly rolling these out over the next few years. Other parts of the world, maybe never. Uh-
SPEAKER 0: Mm
SPEAKER 1: ... Argentina, now, it's going to be a spectrum.
SPEAKER 0: Interesting. Salim, your choice.
SPEAKER 2: Let me go with, uh, number four. Um, "Once we hit recursive self-improvement, is it then just a matter of how much compute is available?" And that's from Chris Dover eighty five oh seven. I think it's not quite that simple. Compute obviously matters, but intelligence improvement has lots of like different components to it, like we talked about the shortage in RAM already. You have energy as a substrate. Uh, you want to evaluate whether a change, uh, like actually improved the system or not. So there'll be some peripheral components of recursive self-improvement that will slow the overall progress down. So it just doesn't s-- it doesn't mean you have an instant intelligence explosion. The implementation of that will take time. However, once you have AI systems that can improve the, the research cycle, uh, you get a massive compression of the innovation cycle, and that's what we're seeing with the timescale going from years between models down to ten, eleven days, and that's the mechanism for the singularity. So the-- we think of the-- I would think of the singularity not as infinite like intelligence, but it's when the iteration cycle gets so fast that we can't no longer keep up, which is why we're in the middle of it now. There's no model by which you can predict the future today, and that is the very definition of a singularity. You can't see past that event horizon.
SPEAKER 0: Uh, Dave, your choice.
SPEAKER 3: [laughs] You get stuck with two if I take three. Sorry. All right, I'm taking three. "Why can't a majority of the compute resources go towards solving the community issues with data centers now, so people will believe AI can solve problems that matter to voters?" Um, you know, I actually have a very related question I think about a lot, which is the, the PR around the AI labs is so bad because they didn't really manage it, and it grew much faster than they thought it would. But now they've got this power tool of AI that's insanely convincing and brilliant at coming up with messaging. So I think they can convince the people around the data centers that they're getting a huge benefit using AI as a, as a tool in that process. I also think they can give-- Like, the, the cost of benefiting the town is so small-
SPEAKER 0: It is
SPEAKER 3: ... compared to the value of the data center [chuckles] that like-
SPEAKER 0: I, I'm like, why don't they wake up and just say, "We're gonna give you far better, you know, uh, cost of electricity and schools and police force."
SPEAKER 3: Yeah.
SPEAKER 0: "We're gonna subsidize everything here." It's a fraction of the revenue they're gonna make.
SPEAKER 3: This is one of those many topics where if you take what Peter just said and you walk into their office and you say, "I know exactly how to solve this problem in Abilene," in whatever, "I, I'm gonna work night and day." They'll just start paying you tomorrow 'cause [chuckles] the only reason they're not on it is 'cause they're stretched way too thin, and somebody just needs to take their money and fill the void.
SPEAKER 0: Yeah.
SPEAKER 3: And, 'cause, you know, nobody knows. Like in Abilene, what do we need? Better schools? Do we need better busing? I don't know. I'm just like, [chuckles] "I just arrived." Like, okay, somebody come and fill that space and I swear to God, they'll hire you tomorrow, and they'll pay you whatever it takes.
SPEAKER 0: Imad, mosquitoes, can they be helpful inoculators? [chuckles]
SPEAKER 4: Tweak their genes so they deliver measles vaccines and Skitta disease. That's Thomas Davis Real Estate. Uh, I don't think we should have forced vaccination for large numbers of people.
SPEAKER 0: [laughs]
SPEAKER 4: And I'm actually on the genocidal mosquitoes, uh, side of things. They've killed five billion people over history, a million total, and so I wouldn't mind if they were actually just wiped out. But definitely don't force vaccination, and I think actually we should, like, remove the mosquitoes. They are the biggest predator for humans.
SPEAKER 0: But, but the concept is interesting, right? To use this, you know, automated, uh, living drone capability to distribute something that's helpful and useful. But I agree with you.
SPEAKER 3: Psilocybin.
SPEAKER 4: [laughs]
SPEAKER 3: Have it deliver psilocybin.
SPEAKER 4: Psilocybin mosquitoes.
SPEAKER 1: Oh, dear.
SPEAKER 4: Yeah.
SPEAKER 1: Well, yeah-
SPEAKER 4: I just, um-
SPEAKER 1: How about just vaccinating the mosquitoes? I mean-
SPEAKER 3: [chuckles] Oxytosan
SPEAKER 1: ... the mosquitoes have a problem too. Mosquitoes are-
SPEAKER 4: [laughs] Yeah, just-
SPEAKER 1: I mean, so it's a-
SPEAKER 4: ... tiny little syringe.
SPEAKER 1: But I'm, I'm cur-- I mean, there, there's a more general problem here, which is we live in a biosphere that's n-nature red, uh, in tooth and claw. It's filled with wild animal suffering. Um, m-my view on wild animal suffering is that it's an atrocity that we should see if we can fix. But I, I assume, Imad, based on your comments, y-your attitude is, "No, just eradicate all mosquitoes everywhere."
SPEAKER 3: Well, then Lam could bring them back anyway. Let's, uh-
SPEAKER 0: Gene, gene drive. [chuckles]
SPEAKER 1: But he refuses to. I pressed Ben on that, and Ben's attitude is, "No, we're going to only, you know, laser focus on a few things. Don't ask me about Mozart."
SPEAKER 3: Oh, yeah. The, the tiny little wooly mammoth-
SPEAKER 4: Do you want a giant-
SPEAKER 3: ... and the pterodactyl-sized mosquito. That's what we want.
SPEAKER 1: Yeah.
SPEAKER 4: Yeah, exactly. [chuckles] You want the pterodactyl-sized mosquito? [laughs] No, let's please not.
SPEAKER 3: Just go to, just go to-
SPEAKER 1: Hate mosquitoes
SPEAKER 3: ... northern Canada. They have pterodactyl-sized mosquitoes.
SPEAKER 0: All right. Imad, you get first choice now.
SPEAKER 4: All right. Um, boo, boo, boo. Uh, do you think some politicians resist AI because it could fundamentally change how politics works? Uh, @danielmorris1. Yes, of course. I mean, like, it will remove a lot of the barriers for democracy, intelligence access, raise the intelligence of society. Like, these are big things. So I think you will get some resistance to that as we see this kind of evolve because many politicians unfortunately don't want a more intelligent electorate. Uh, particularly the ones that focus on, uh, shall we say, more demagogic approaches.
SPEAKER 0: Okay. Uh, Salim.
SPEAKER 2: Uh, I got a take number five here. What happens when intelligence rises, uh, rises while cost collapses? From bb_x_1. Uh, like, you know, when you have intelligence going from a scarce commodity or a scarce capability embedded in very expensive, clunky humans, um, you basically radically change not just the company itself, but civilization itself, right? Because whenever the cost of some foundational input a-approaches zero, we actually reorganize our whole world around it. Now that information transmission is essentially free, we've rewritten our entire world around that. So when bandwidth became cheap, you got YouTube and you got Zoom and you got cloud computing and all that kind of stuff. When computation became cheap, we got cell phones and smartphones. When intelligence becomes cheap, we redesign companies, we do unbelievable research, you rethink education, you rethink healthcare. Then you have scarce things like human experience, judgment, purpose, uh, trust, et cetera, that become important. Then you, you stop optimizing intelligence, which is what we've been doing for several thousand years, and you start optimizing what do you point it at. That becomes interesting.
SPEAKER 0: Yeah. I, I think, a-again, I'm gonna make the point here we've said so many times, as this happens, it's so incredibly important for people to raise the limit on what they think they can do, you know, and, and point that intelligence at something extraordinary that surprises you because you're gonna be able to make massive contributions by utilizing it. All right. Dave.
SPEAKER 3: Uh, why don't I take seven? Richard, I think that's Richard Socher, puts P doom at zero, yet says a hundred million people could be harmed. I think that... Did he say that on our podcast?
SPEAKER 1: I don't remember that.
SPEAKER 0: I don't remember that. Maybe he said it elsewhere.
SPEAKER 3: He definitely got a lot of-- He, he was open and super honest. You should definitely watch that podcast. Uh, so question: Should P doom mean extinction or catastrophe? And how should we think about risks of that scale? And that's from A Really Long Name6542. Um, yeah, I, I think, uh, everything in that question is exactly right. Um, we can define P doom. I think it's doom, like doom is doom. P doom should be probability of, of extinction, disaster, cat-catastrophe. That's sort of what it means. But yeah, there's, there's inevitably gonna be misuse of AI, uh, and it's gonna cause harm. And as Eric Schmidt has said on the podcast at least three times, uh, he's hoping it's relatively small, like a hundred people, a thousand people are affected, or it's a cyberattack that nobody's hurt, but money is taken, and that's what wakes up the governments. But right now they're just not moving anywhere near fast enough to avert it. Yet the good outweighs the bad by incredible amounts, and I think that was Richard's, uh, fundamental point, that we should expect abundance and overall benefit.
SPEAKER 0: Mm.
SPEAKER 3: So I don't know if we should redefine P doom, but, uh, but that's the way it's gonna play out.
SPEAKER 0: Uh, and we should just keep on focusing on P fab. All right, Alex, number six is for you.
SPEAKER 1: All right, number six: Could public opinion become a greater bottleneck than the technological development of RSI, recursive self-improvement, itself? And this is from David Call VFX. I think we're already there. I, I think recursive self-improvement, we're already in the RSI era. Both Anthropic and OpenAI have made a, a number of pretty detailed and quantitative public statements to that effect. So we're already in an era when RSI is driving the vast majority of research at this point, whereas on the other side of the ledger, public opinion is driving data centers out of at least American municipalities in the direction of orbit on the one hand, and we see, as discussed in the, the past pod with Richard, we see members of the US House introducing or planning to introduce bills against recursive self-improvement. I think public opinion probably is already a greater bottleneck than the actual intrinsic technical difficulty of RSI, which is also why you see some of the public labs, uh, some of the frontier labs rather, like Anthropic, starting to make public noises about essentially buying off public opinion by doing, you know, tiny little things like solving all human disease.
SPEAKER 0: [laughs]
SPEAKER 1: And if, if you can c- solve all human disease as a side effect of recursive self-improvement, then maybe public will look the other way. Not that it's a bad thing, but at least be, uh, you know, compensated in some sense for a- allowing many of the tokens, maybe the vast majority of the tokens, to flow to self-improvement rather than to other applications. I think there's probably a pretty good bargain, a grand bargain if you will, to be struck there.
SPEAKER 0: Hmm. Nice. A reminder, if you've got an outro video, and Dave, I expect you to submit some here, uh, [laughs] please send it to media@demopolis.com. And we've got a beautiful one from our friend, uh, friend of the pod, CJ Trueheart.
SPEAKER 2: Uh, I have a quick announcement-
SPEAKER 0: Yeah
SPEAKER 2: ... CJ. Um, we've been doing these workshops for teenagers. Lily's gonna run one in Austin, Texas on the 24th. If anybody has teenagers or wants them to kind of rethink how to use AI as a superpower, um, check it out, yxo.world. We'll put a link in the, in the-
SPEAKER 0: yxo.world
SPEAKER 2: ....world.
SPEAKER 0: Nice.
SPEAKER 2: Um, and they've got slots for a few teenagers left for anybody interested.
SPEAKER 0: All right, if you're in Austin, bring your teen. Uh, Lily does an amazing job. All right, this, this video is called Tomorrow Comes Alive. It is from the Moonshots Live event. Take a look.
SPEAKER 2: CJ is awesome.
SPEAKER 11: [gentle music] 6:00 AM. A new day. A new world waiting. 6:00 AM, the alarm breaks the silence. Clothes on the chair, blue light in the room. Met the eyes of the man in the mirror. Said, "Remember what you came here to do." Through hotel doors into a rideshare. Rideshare. Daybreak running over the glass. Glass. Carrying a future I could almost see. But no one brings a future back alone. Can you hear it? Hear it. That quiet calling getting loud. One soul knows why. Becomes a fire in the crowd. If you got a purpose burning brighter than your fear. If you see a world worth building that has never yet been here. Don't wait for tomorrow. Become what you believe. Star Trek made tomorrow something we could see. Moonshots makes tomorrow something we can be. Come alive, come alive. Dream it, build it, bring it to life. At Moonshots Live, tomorrow comes alive. Oh, oh, oh, oh, oh. Yeah. Oh, oh, oh, oh. Tomorrow comes alive. Oh, oh, oh, oh, oh. Yeah. Oh, oh, oh, oh. Yeah, yeah, yeah. Walked into a wave of voices. Oh, oh, oh, oh, oh. Strangers felt like future friends. Oh, oh, oh, oh, oh. Every handshake held a story. Oh, oh, oh, oh, oh. Every question crossed the edge. Oh, oh, oh, oh, oh. Not abundance as a slogan, not just talk about the climb. Courage recognizing courage, vision multiplied by mine. Someone met me at the doorway, not with a pitch but with a hug. Before we ever spoke of rockets, made the future feel like us. And the dream grew hands and heartbeat. When the room became a crew. And the distant felt much closer. When I saw what we could do. Can you feel it? The whole room breathing now. A thousand different sparks. Make a constellation in the crowd. If you got a purpose burning brighter than your fear. If you see a world worth building that has never yet been here. Don't wait for tomorrow. Become what you believe. Star Trek made tomorrow something we could see. Moonshots makes tomorrow something we can be. Come alive, come alive. Dream it, build it, bring it to life. At Moonshots Live, tomorrow comes alive.
SPEAKER 0: I love that, "A purpose burning brighter than your fear." What a great line.
SPEAKER 1: [exhales] That, uh-
SPEAKER 3: My brain-
SPEAKER 1: ... that character is-
SPEAKER 3: See, the, the guy walking around-
SPEAKER 1: Yeah, CJ, CJ wrote himself into the video.
SPEAKER 3: Yeah.
SPEAKER 0: Yeah.
SPEAKER 3: That was really cool. [laughs]
SPEAKER 1: Yeah, that was awesome.
SPEAKER 0: Oh.
SPEAKER 3: My brain is so fried, guys. I mean, what the hell?
SPEAKER 0: [laughs] You know? And I've got a five-day-
SPEAKER 1: Singularities are exhausting, aren't they, Salim?
SPEAKER 3: Jesus Christ. [laughs]
SPEAKER 0: I've got a five-day event-
SPEAKER 1: I'm glad this only happens once a, once a planet.
SPEAKER 0: Yeah.
SPEAKER 1: At... Well, maybe. Like, maybe it happens more than once per planet. We don't know yet.
SPEAKER 2: Yeah.
SPEAKER 0: Ah. Well, I'm on stage for five days at the Abundance Longevity trip here.
SPEAKER 2: How do you do it?
SPEAKER 0: We've got 40 amazing faculty on the cutting edge of longevity. Yeah, longevity escape velocity. It's coming fast.
SPEAKER 2: We're gonna need longevity just to live through these episodes.
SPEAKER 0: Dude. [laughs] You have no idea. [laughs] Uh, Iman, Alex, Salim, Dave, love you guys. [outro music]