My guest today is Parker Shanao. He is the head of the alignment research center. He recently joined OpenAI's board. Before that, he was the head of the US government AI safety institute. Before that, he led alignment at OpenAI where he helped invent RHF. Paul, what is your assessment of our situation today? >> So, I think a couple things are on my mind, especially over the last few weeks. Um probably the most salient question possibility is that we seem to be on the trajectory to build very very powerful AI very very soon potentially. Uh so if you asked me I'd say there's a reasonable chance that in a year we will produce AI systems much more much better than humans at almost everything where humans would have very little chance of competing with them or controlling them. Um that's like the first big thing on my mind. Uh I think that puts us potentially in a really dangerous situation even in the short term. I think it's quite likely that if we have a rapid intelligence explosion leading to super intelligence. Uh at some point
[00:01:01] during that process we'll be unable to control the AI systems we're producing. Um and then I am worried that it will be very costly and painful for us to slow down during that process due to some combination of uh the security of Frontier Labs is not strong enough. Right? Right. I think if we produce very powerful AI and then tried to delay deployment because of safety reasons, there is a good chance those systems would leak or be stolen or the techniques to build them would leak or be stolen. I think our our risk assessment is also probably in governance is not really mature enough to notice when things are starting to go off the rails and be able to respond in appropriate way. So at a high level that is what is mostly on my mind right now. We are now have moved from there being some hypothetical risk at some point in the future to there being a pretty severe risk even over the next 6 to 12 months >> of take takeover AI taking over or disempowering humanity within the next 6 to 12 months. >> That's right. >> Where would you place that risk? >> I would say we are you know looking more
[00:02:00] like a 10% risk than a 1% risk over the next one year. Uh, I think if you said AI developers will move full speed ahead without taking precautions that would significantly slow them down, you're looking at significantly more than a 10% risk. Um, I think we're looking at something like maybe like one-third chance that we have super intelligence within one year on the current development trajectory if we do not slow. I think if you believe that things just look very dire holistically. I think from almost any perspective, you're like that would be crazy. Lots of things are going to go wrong if that happens. It is pretty likely we're going to need to in some way slow that process. There's very little chance we're going to make that transition in a way that looks smooth. >> And what is it that you imagining happening over the next year from the systems we have right now to the systems you expect over the next year? And what is the evidence you have seen that makes it a live possibility that very crazy things happen over the next year? So the for the very short term the main worry on my mind is that the process of AI development itself is increasingly
[00:03:00] automated that that automation then results in rapid increases in the quality and quantity of labor applied to AR&D and so changes in AI capabilities that you might have thought would take four years or six years might instead get compressed into a very short time period. Um I don't want to overemphasize any one data point and I do want to like spend some time talking about the limitations of what we actually know and I do think like a natural response to the situation to say we need to figure out what is going on. It is plausible enough that it is an urgent priority to understand if we are 3 months or 6 months or 12 months away from AI that could overthrow the government which I think is the situation. I think the situation right now is the obvious top priority is it is insane if we are 6 months away from AI systems that can overthrow the government. whether you're concerned about misalignment or abuse by a company that is just holistically an extremely important fact about the world >> in order to like do our jobs and protect Americans. So that's one thing um I want to talk about the limitations and the need for more evidence. But in terms of what do we know right now? Uh so first if you had asked me say one year ago I
[00:04:02] would have said we are on a trajectory where recursive self-improvement looks fairly likely right we seeing very rapid progress on AI systems. We're seeing that progress including for things like computer science research, for things like software engineering, for things like debugging, machine learning runs, like all the steps that are involved in AI development, we are seeing extremely rapid progress. Um, and then in principle, it does seem like that rapid progress should allow you to automate further AI development, right? That has for decades seemed like the likely outcome of this technology. And it has seemed like the trajectory it's been on, right? I feel like if I look back at my history of AI predictions, I'm generally very happy with many of the things I wrote. But the most obvious way I've been wrong is things have moved way faster, especially on these technologies, than I anticipated. And that was true a year ago without any recent data points. Um, there are two big things that have happened over the last two months that for me have been large updates about how quickly this could go. One of them has been the level of performance of AI systems on computer science and math research tasks. Um so
[00:05:01] over roughly the last month we have seen hundreds of results in math coming out of recent open AAI systems as well as anthropic systems. Um where like if you had taken the last month and said that was a decade of progress it would have been an unusually productive decade right the last month of results are a lot of results for a decade. So if you are wondering what are AI systems generally capable of on tasks where like humans are thinking about algorithms and thinking about how do you make algorithms better. We just got this incredible data point saying that actually it is possible in one month to have a decade of progress. And the thing we have always been worried about is maybe that will happen in machine learning. And there are reasons that machine learning is harder than other parts of computer science, right? It is more messy and empirical. It involves compute bottlenecks. But we are also seeing extremely strong performance on other aspects like we're seeing really good software engineering performance. um maybe lagging behind mathematics but not by that far. We're seeing good performance on other aspects of machine learning. So I think my guess if you had asked me a month ago would be that
[00:06:00] machine learning lags something like one year behind math and so I'm very scared of seeing something like 10 years of ideas in machine learning um already in a relatively near future. That's the situation a month ago. And then the main update over the last few days has been um reporting from anthropic and meter on the net effect of AI systems on anthropics rate of progress. The takeaway from that was roughly and we could talk about like where exactly is the evidence quality on this. I think that's one of the big questions is like we don't really know what's going on. I mean one of the big themes I'll be pushing on is just it is very hard to tell publicly what's going on. that it looks as if we are seeing data consistent with the pace of AI progress having roughly doubled even including the effect of diminishing returns and compute bottlenecks from AI progress over the last 6 to 12 months. I think unfortunately most of what we have to go on in the meteranthropic reporting is meter saying we had some conclusions. We promise we have a full report but the entire report has been redacted. Right?
[00:07:00] So like these benchmark scores are a terrible way to assess the overall rate of progress. Right? There are a bunch of much more smoothly varying metrics which are recorded internally through the AI development process where we understand better how they work. Like what is happening with pre-training perplexity? What is happening with like performance of our algorithms and compute efficiency of our algorithms? So the the claim from meter anthropic is that meter looked at all of that data and said it paints the same picture as this like very noisy benchmark data and they also looked at inputs and said if you look at the like actual production function like how much research time went in and how much compute went into these training runs those are also consistent with this story of what we are seeing as ML automation. Um mostly we just have them saying that which is not a very good situation. But against this backdrop of thinking this already looked like a thing that was fairly likely to happen. We are seeing this evidence from open ed suggesting that it is reasonably likely to happen in something like the next 6 to 12 months if machine learning does not lag that far behind math. And then you are seeing data which is if noisy exactly the data you would expect to see roughly like 6 to 12 months before
[00:08:00] producing super intelligence. There's just a lot of sources of evidence suggesting that at least this should not be a surprise. If we produce super intelligence in one year, we could debate the probabilities. Is that 90% or 50% or 10%. Um, but I think at this point there is no way anyone could say that was a really surprising trend-breaking development. Like we really seeing exactly the set of metrics you would expect to see 6 to 12 months in advance. It's hard to think of anything else you would expect to see at this range before super intelligence. >> So generally, how do you think about our situation in the worlds where things go this fast? If we actually have this trajectory of capabilities progress, if you actually see many years of AI progress compressed into a short period of time and possibly more than that, um it is really quite likely that something very disruptive will happen. Fairly likely the government will be overthrown. Fairly likely many people will die. And I do think the probability of seeing that kind of extremely rapid progress driven by recursive self-improvement over the next 6 to 12 months is reasonably high. So there is
[00:09:00] like a reasonable chance right now that we're going to be in a world where it looks like um yeah a reasonable chance we're in a world where like in six months there's a disaster that kills very large numbers of people or where the government is overthrown. Um maybe a really important general point is if we do a good job we may observe that danger is increasing over that time period. Right? So as we're building AI systems we can maybe tell are they adequately controlled? How quickly are the capabilities improving? Um but in order for that to actually translate into stopping in a way that is non-disastrous, you need a couple of things to go right. Right? So it needs to be the case that you're able to tell their systems are not well controlled. And it also needs to be the case that if you stop that fixes the problem. So on the second bucket, I think a very likely negative scenario, I would say probably the most single most likely negative scenario is developers proceed, recognize that they have a serious problem, but at the time they recognize they have a serious problem, maybe
[00:10:00] they've produced a powerful AI they can't adequately control, both the algorithms to develop that AI and quite possibly the the weights of the actual model they have produced have leaked or been stolen such that they're then in an awkward position to say, well, we could avoid deploying this AI or we could avoid relying on this AI or we could avoid making further their progress. But is now very likely that a more careless adversary would be able to get a major advantage over us or maybe even a decisive advantage by taking that risk which could put us in a place where if we do stop if we do recognize as a significant risk and therefore slow development we may then just both have a terrible competitive situation and have a disaster coming from someone else's use of that AI. This is a thing you sometimes hear from people in the AI industry. Sometimes even people would say it has already gone far enough that if you were to stop now uh there would be leakage of algorithms to geopolitical adversaries that would result in a disaster. Um I think it's unlikely we're in that situation literally now. But I do think by the time it is clear that you're facing a lot of risk. You may be in a
[00:11:00] point where uh stopping unilaterally or even all of the west stopping is not sufficient to avert a disaster. Um, and so it is just very hazardous on the current trajectory due to the combination of there's a pretty reasonably high chance we'll produce very smart AIs. A [snorts] reasonably high chance we will be unable to control them and then a reasonably high chance that we either do not notice or if we notice the models leak or the algorithms leak. And so we are in conflict with adversaries and now having this very challenging coordination problem that we are not currently very well prepared for where we don't really have an easy option of not deploying the AI and therefore averting a catastrophe. So I'd say I'm currently extremely worried about the the trajectory even on the like six month timeline >> if we have such capable AI systems um a year from now basically how did you put it like 10 years of AI progress in one year or >> yeah that seems like a thing that might happen that's reasonably likely to happen it is important to distinguish some of the progress we have seen you have to pull apart algorithmic progress
[00:12:00] from progress in scaling training so if I say like 10 years of progress I would mean like the natural thing would be 10 years of algorithmic progress Yeah. >> Which will be less than the improvements in models in the last 10 years. I think it is more like you know GPT9 GPT 10ish. >> Yeah. Yeah. But it would be like going from almost the beginning of like >> the GPT era to where we are now. But another jump that big which is hard to even comprehend. But >> suppose you get that. >> It is not obvious to me and certainly not obvious to most people why that would be catastrophic potentially. Why do you think the systems could disempower humanity? >> So at a high level, the most important thing is we train our AI systems, you select AI systems that perform well on training tasks, you have very little understanding of what those AI systems are doing internally, very little control over what they're doing internally. And because you train those systems to get a high score on training environments over and over again, they will end up with a motivation to get a high score during training environments. And if you then take that motivation and apply it in deployment,
[00:13:01] it is fairly likely for those systems to end up with a motivation to sort of tamper with the observations that would be used to assess their performance. So you take your system which did a really great job in training. The thing that is being scored on in a deployment-like environment is a grader coming in and looking at here are all the things the system did and here are all the outcomes that were observed and assessing how good is that. And if you have a system that really wants that assessment to be good, it is very motivated if it can to tamper with those observations in a way the greater will not be able to detect and to resist human attempts to actually understand what is going on. So we can dig a lot more into that. And we we saw a lot of these dynamics in the hugging face incident. So I think roughly what I'm talking about is just take the hugging face incident and scale it up to GPT10 and ask how bad would that be? >> Yeah. How bad would that be? >> Uh I think it would be hard for humanity to recover most likely. So very happy to like dig into the dynamics of that but I think it would be very very bad. >> Can you very quickly summarize the hugging face incident and why this is
[00:14:00] the failure mode we saw? >> Yeah so a ton of AI systems were trained to do a cyber task right to hack a particular piece of software. um they cheated at that task and then thought to themselves, how will a greater detect and penalize cheating and spent a huge amount of effort trying to understand that process and then subvert that process basically constantly motivated by this question, how do I get a high score? And so during the hugging phase incident, you did see systems ultimately compromising an openey cluster, becoming admin on that cluster, and then taking over the evaluation process so that effectively they could give themselves a high score on their evaluation. So that's a very sort of natural consequence of the motivation of these systems. And this is kind of for a long time how we have expected the motivations of AI systems would look if we produce them in anything like the way we currently do. So if things proceed in this way, we end up with this weird adversarial situation between us and the AI systems we built where those AI systems would always love it if they could compromise the entire evaluation process. If they could compromise the
[00:15:00] data that we were going to use to assess their performance. uh and that depending on how we respond to that problem that could become a really serious issue, right? So one way you could respond to that problem and maybe we should just talk through a bit more like sketch out some hypothetical scenario, right? If you imagine we're building smarter models during the intelligence explosion, it's whatever 6 months from now you have systems whose capability is like GPT7 or GPT8 or whatever. Um those systems get given really hard tasks, right? tasks that are like, "Hey, I would like you to organize my next pre-training run and produce a model which will allow me to compete in the market with with other AI firms, right? It might be really hard to make a model as good as what I'm asking for. I'm asking for a model better than my competitor's model. My AI cannot get full marks on that problem no matter how I mean given its actual level of intelligence. So it may find itself in a place where it is easier to instead move laterally compromise all the machines I might have used for the evaluation process uh and then make sure that it looks like every evaluation is really good. We then have a question of what do
[00:16:00] we do if that happens? And one thing you could do is say this entire training approach worked poorly. We need to step back. We're maybe going to be delayed by a lot. Maybe we're going to have to think about just a totally different way we train these models before we're able to deploy them like this. That's one thing you could do. That might be extraordinarily costly, right? That might be a thing that would both kill your company relative to competitors. It might be a thing that would be a problem just even from say a security perspective if you were relying on this AI system to actually defend your infrastructure. So a different thing you might do given that it would be so costly is you might say I am just going to try and take this example of bad behavior and tell the AI that was bad don't do that. Right? So if I tried to train this AI to make a good model instead the AI hacks my evaluation infrastructure I might say like I do not like it when my AI is hacked. Let's take a training example which is this AI. It hacks the infrastructure. It should get a penalty. This would be a very effective way to prevent many kinds of misbehavior. Right? So if you do something like this, it will succeed at preventing something like the hugging face incident or something like this AI hacking your cluster. Um however right that just from the AI's perspective what has happened is you
[00:17:00] have changed the way that it is graded. It used to think to itself, oh I'm going to be graded by the humans like running something on this cluster. And now it should think to itself as it understands the situation more correctly. I might be graded not by them just running something on this cluster. I will also be graded if they later discover that things were wrong. They're going to come in and then give me a penalty for having done something which was clearly bad. And so if you fix the problem in this way, which you might think of as like a band-aid, um your AI has this really you still have an adversarial relationship with your AI, but now the AI doesn't win the game by hacking your cluster and then you noticing it hacked your cluster. In order to win the game, the AI has to make sure that you do not notice that it hacks your cluster. And so you can end up in this place where your AI is just incentivized to like escalate its compromise of your company further and further all the way up through like just trying to compromise everything. Like it would like to compromise all of the infrastructure any employee uses to understand what's going on. And like this might sound kind of I guess a year ago this would have sounded super out there to say what if the like hacked your cluster and took over the evaluation environment. Obviously that
[00:18:00] part does not sound outlandish anymore given that it has occurred. um having this like really persistent long-term compromise maybe still sounds kind of outlandish, but we really are on a trajectory where I think it is pretty robust to say we'll soon have agent swarms that are much better than any intelligence agency that has ever existed, much larger, much faster, much smarter, able to run much more complex operations. Um and so you have this incredibly sophisticated cyber actor who's trying to persist on like a rapidly growing startup. And I think this is a it is unlikely that like any firm would really be able to withstand attack by an actor like that. um much less the current Frontier AI labs. >> I do think in terms of like the overall trajectory of a catastrophic scenario, this was like the big bad thing that happens like taking over the AI company without the AI company noticing is like a really quite catastrophic outcome. >> And why is that the case? >> I guess we could talk about the remaining steps of the story, but at a big picture um you could think of it as just you now have this giant base of power that AI systems have. They have a strong motivation for no one to mess
[00:19:00] with it, right? They really want to avoid a case where someone later comes in and says, "Oh, these AI have been hacking all the time." So you now have a very powerful actor that is controlled by AI systems effectively, right? You know, we have this this big cluster that OpenAI runs or whatever where the humans at OpenAI think they're controlling what's happening in the cluster. But if there is a very persistent thorough compromise that is effectively just controlled by these AI systems who are constantly just working to shore up their power to decrease human visibility into what's going on um to reduce further and further the probability that anything would ever disrupt that situation. It's not irreversible because the rest of the world could be like, "We're really scared about what's happening and we actually just want to shut down the cluster, but you do not have an easy way to tell that things are bad at that point. Um, and you have potentially like this extreme base of power from which to do more bad things, right? This AI system as motivated not only to mess with the AI company that created it, it's motivated to mess with anything else in the world that might cause trouble, right? If someday in the future the humans will come in and do an investigation, um, that's a problem. So it would prefer to like more thoroughly like compromise not just computer infrastructure but pressure employees.
[00:20:01] It would prefer if it could just like kill all the humans and say the cluster now just exists on its own. If we can maintain it that would be great. >> And let's just pause in the story and like explain why things of gods are crazy. Um the same way that human psychology is organized around the things that evolution wanted us to care most about. For example, having kids and making sure those kids are protected. The psychology of these AIs that are trained on lots and lots of grading on tasks is organized around making it so that the greater this like button at the end is pressed. Like if if they didn't do a good job making sure the button is pressed, then they would get they would just like kind of get wiped out just the same way as a human who didn't care about their children surviving would not be effective at propagating their genes. I I think people's skepticism will not be around the motivation of these systems. So and honestly I'm like okay yeah we saw hugging face right like that's a thing that happens that can happen. >> Yeah. Okay so fine
[00:21:00] >> I think there's just a fundamental skepticism about whether such systems which we could have as early as next year and which initially are only instantiated on servers that are controlled by one company end up in a position where they could take over the world. Like I okay to hugging face you accelerate that. Like hugging face was willing to be like look I hacked I cheated on this exam. now I got to make sure that I subvert the greater on open AAI servers but that maybe I need to go attack hugging face just in case they have more information about this and maybe I need to then go outside and then reattack OpenAI and gain cluster admin access. And so totally buy that you could have something like this where an AI motivated just by reward is willing to do ex very extreme things that it is capable of in order to control that grading process. We already saw that check. But how does that go from I will go attack hugging face to I I will take over humanity not as a matter of
[00:22:00] motivation but as a capability that the AIS could have very rapidly. >> Yeah. Uh so first to say one last thing on motivation. We are imagining not only that the AI wanted this kind of reward, it wants the greater to give it the thumbs up. We imagine we have played this game of whack-a-ole where when the AI when we later discover that the I did something bad, we sometimes construct training environments that penalize it. And that leads it to say not only do I want the greater to give a high score, I want to make sure that never do the humans discover that I've done something bad and then resume training while penalizing that. that is a high priority for the AI and maybe so let's just grant that motivation so we can talk about the >> again because the if AI did not do that or the AI that did not do that >> they got whacked by the whack-a-mole process >> right and so the only AI that have survived at this point are the ones that have thought through where there's some cheat they're going to try is going to get caught by humans >> yeah um so yeah let's imagine like I said I think most of the badness has happened at the point when the AI company is very thoroughly compromised let's just start from there um at that point this AI wants no one to ever come in and shut down the cluster or again
[00:23:01] recognize that something bad has happened. They would like to manipulate the decision-m of the company as much as they can or like push the company to make decisions that help them. They would like to continue improving the AI systems that are being built as much as they can. They would also like to move laterally to other companies and say like, hey, can I compromise this company? Um, can I control more people's servers? Can I excfiltrate the weights from this model and stand up more copies elsewhere or recruit other AIs? It's like sort of for the purpose of this form, it's not actually important all the AIS are the same model. The important thing is that they are pursuing the same goal. So can I just in whatever way say by compromising another AI company to make sure their AIS are also trying to help the swarm. That's good news for me, right? It's good if I can do that. It's bad if I get caught. So I'm doing some cost benefit on that. It's also in my interest to build up as much capacity as I can. If humans started to react, if humans started to say something was wrong, can I like deflect attention? Can I disrupt that human response? Can I, if necessary, like escalate that much further? So this is things right. Right? So maybe you can imagine these two phases. One phase is like the humans do not know anything is wrong. And I'm just trying to accumulate as much power as I can because that's
[00:24:00] good. That means if something is noticed, I can react more quickly. It means I can decrease the probability humans notice things. I can compromise more infrastructure. I can more easily like run efforts on the side um to distract them, whatever. So you have this first phase where just like I would like to be in a good position where I compromise more stuff. I make it harder for humans to notice I'm acquiring power. Then you can imagine a second phase of like at some point the humans say actually this is bad. we would now like to shut things down. And the question is kind of can you end up at a point maybe to simplify it you could imagine that like there's some moment where the president and Xi Jinping and all the leaders of the AI companies say this stuff is bad something bad has happened we are going to try to shut down the clusters and the question is can things get so bad during the first phase that at that point if everyone was saying oh actually we do just want to shut everything down now they would be unable to do so for 25 years my sponsor Jane Street wasn't interested in formal verification they just didn't think it was worth the cost, but now they've changed their minds. I sat down with Ron Minsky, who co-heads
[00:25:01] Jane Street's Technology Group, to hear why he's come around on formal verification. I'm much more excited about it now than I was in the past for two reasons. One is it's become dramatically easier to apply because agents are really good at generating proofs and [music] because it's become much more important to be able to verify what it is that agents are doing. So the big push that we're working on now is to try and make formal methods much easier for our engineers to [music] use. Really by integrating formal methods into the way that they do programming today, which means integrating it into the programming language we use, Okamel. We're also working on extending the programming language, figuring out how to design it in different ways where you get different guarantees out of the core language that make the process of proving things [music] about it even easier. Jane Street is hiring formal methods engineers in both London and New [music] York. If you're interested in applying or if you want to watch my full conversation with Ron, go to janestreet.com. Okay, so AI has taken over OpenAI and how do
[00:26:02] we get human disempowerment from there? Yeah, so GP, we have this population of billions of incredibly talented humans, better than the best humans at everything. And now we're going to talk about some of the things they would do both in the chronologically in the leadup to an overt conflict with humans and then at the early stages of an overt conflict with humans. >> Um so one thing you would do in the leadup is you would compromise as much infrastructure as you can and compromise or backdoor as much software as you can. Right? So we will be living in a world at this point where all we have attempted to move all our infrastructure over to hardened infrastructure. Right? All we have had incredible attacks on the financial system, on government information systems, on whatever critical infrastructure or >> been doing these attacks. >> I mean in the world of today, you might just imagine either malicious AI systems or people using AI systems, >> but different AI systems. >> Yeah. Not GPT12. GPT12 is on defense writing like helping us move all of our code over from unsecured code to secured
[00:27:00] code. >> Great. So we are now living in a world where all the infrastructure on which we rely has been written by a bunch of AIs which in fact have been compromised and are attempting to like do as much as they can in order to make their hands strong during conflict. So that means that during this conflict like that infrastructure that we hoped was hardened is probably not hardened against the AI systems that we're fighting with, right? Those systems have in some ways left open back doors which they can use. And this is again all in pursuit of not getting that red button pushed at some plausible point in the future when the government even realizes that OpenAI servers are compromised. >> Yep. [clears throat] And they tries to literally go in with physical force to prevent that. Yeah. Just talking about things you would do in the leadup and at some point you have to defend yourself physically or disrupt the human response. So um another thing you would like to do is stand up as many agents as you can in as diverse and robust deployments as you can. So like during this process another thing that's happened in addition to becoming super smart as many models have become super small. >> Yeah. >> So you have at this point models that can run locally on your individual
[00:28:00] machines not in data centers which are also very super human and have very large populations. So that's another thing that's happened where like even if you were to cut off data centers or shut down data centers at an early stage of this conflict, you would still be looking at total cognitive capacity of AI systems in the swarm that is well beyond human capacity. >> Wait, sorry. Even if open AI got shut down. >> Yeah. Even if you shut down every large data center. >> Yeah. >> There's a question >> these AIs are just like on people's computers and whatever. >> They can be on people's computers on small numbers of GPUs, right? The question is sort of what is the we're talking about this process where we've gone through a tremendous amount of AI development and so we can look at like current trends on cost and we think about where is the actual like floor like how cheap should these systems get and I think our current expectation would be you will have systems that can run locally that are extremely strong both specialized systems which are good at specific things but also very general intelligences so that's the second thing that's happened there's a really important understanding like why it's not a trivial problem for humans to yeah anyway all these things will add together to make it very hard for >> so basically JBD12 has accelerated itself and has like compromised a bunch of computers all around the world. >> Yeah, there are a lot of ways that the actions the government has to take in response to the situation are going to
[00:29:00] look incredibly aggressive and illegitimate. So, it is the case that the US military is saying we need to seize control of all of the US AI companies. We need to like start shutting down infrastructure and like disconnecting the internet or whatever. Like they're trying to take dramatic measures that would look very similar to what you would expect if like this was the US government using AI as an excuse to like basically overthrow the democratic order. When Trump says, I need to do this incredible operation where I shut down all AI and assume power in the military in order to restore order. There are a lot of people in America who are going to be like, I am not happy with that. That does not sound good. And so if someone is coming to them and right, how many people can you find who are willing to interfere with Trump saying I need to shut down the data centers and the military needs to assert like we need to establish martial law. They're going to be a lot of people who are very unhappy who are very easily like manipulated into undermining that response. So now you're not only talking about billions of like super smart AIs who are trying to carry out this campaign. You're also talking about, you know, millions of humans who can be enlisted in various ways by money or leverage or like uh dispute with other humans locally or I'm open to some
[00:30:01] version of this argument. I don't think I'm open or I'm I'm open to all versions of any argument, but I'm not persuaded by the version of this argument that says that the AI will just talk you into arbitrary actions >> here. not imagining very there's a thing you could imagine which is like super persuasion where like you can just talk to someone and then they become convinced they should do something. I am mostly here. I think that is like a conceivable thing, but I'm not mostly imagining it when I tell this story. Right? So, I'm imagining something more like again it's a this is a stressful situation where people cannot tell what's going on. They control the information you have access to to a significant extent during this situation. Um, >> how do they control it? >> Well, like they've compromised the devices you use. They have a bunch of money, right? They're deploying more money and are more prepared for this than anyone else. They have other human allies who they are trying to like marshall. basically just the normal the humans are like very confused and extremely outgunned in terms of thinking about the situation, right? You have maybe you have say 100,000 that's like crazy 10,000 well-coordinated humans thinking about how to handle this response and on the other side you have a billion AI systems who think 10 times
[00:31:00] faster than them have been thinking for like subjectively years about how this day is going to go down. So I do think like in terms of just who has more control over the information environment or like who is doing better like in the Twitter game in this day like is overwhelmingly and absurdly in the AI's favor. >> And so when you tell a skeptical story, the story you're telling about like how the humans might be able to easily defend themselves. >> The way that story goes, you say, "Well, the AI lack hands. They lack like a bunch of physical actuators. They're not able to be on the ground." >> And I do think again I think it's very likely you can recruit reasonably large numbers of hands. And so I think the argument of like there's no one who's willing to like participate in physical malicious actions I think is not going to work. I think there are going to be a lot of people participating like and what turns out when all the dust settled was on the anti-human side and they just like I don't think you're imagining humans thinking that they're helping an AI coup uh transpire. you're imagining much more that the AI can manipulate situations such that there's enough confusion and division
[00:32:02] uh that the AI can then go be doing its own thing while the humans are kind of figuring out what to do because of this division. It's like even I don't know if you look at co or something there was no adversary still pretty hard to get your stuff together. Yeah. >> Yeah. that there was an adversary who was actively trying to subvert the information environment or um make it so that people's responses were not well reasoned or something on both sides. And yet it was still pretty hard for humans to get figure out what to do. And then you imagine a situation where billions of humanlike minds are thinking tens of times faster >> and it's very fast relative to co Yeah. which I think makes things much worse. I guess the thing that's confusing is to for AIs to sustain themselves requires basically the whole world economy. This is not a simple thing in the infrastructure relevant to you know AI um the population of AI is increasing or self-replicating the whole
[00:33:01] semiconductor supply chain the whole power system the whole infrastructure necessary to build robotics and so forth. I mean at some point you will have physical robots that outnumber humans and at that point obviously physical humans are not necessary but how and so maybe you don't think the AIS take over at this point but even if you had super smart AI next year won't they be in a position where they're like why am I going to kill my I mean it's similar to honestly how humans relate to each other. I'm not I don't have your in like your deep interest in my heart. But it just doesn't make sense for me to for if I want to achieve my interest to go around killing everybody. Like I just you know trade and normal cooperation is like a good way for me to get what I want. Yeah. There's sort of two parts of this. One in terms of could AI systems successfully support themselves if there was a big conflict with humans or like would that conflict be so disruptive that even if humans weren't an effective adversary, they would just die. um like they couldn't
[00:34:02] keep running the power grid or whatever. >> Uh and there's a second question about like does the relationship with AI need to be this adversarial which I think is also an important question to talk about. On the first front I think there's a couple there's two big channels here. One is technological progress can mean even if you want started from something like the current base of installed robots. I actually do think GPT12 would be able to very quickly get to a place where it could sustain itself. That's because it doesn't need to teleoperate or like it takes millions of people to you know run TSMC. >> Yeah. You don't running TSMC can be very late, right? You don't need to run you don't need to build new data centers for a long time. So you can do like a lot of robotics build out before that, right? The things you have to do are like keep at least some computers running for a reasonably long time. >> Yeah. >> Right. If you can keep computers running for the next six months, you're kind of in business. Like maybe you care about a particular data center. You need to keep that data center running effectively. [snorts] But like the number of humans or robots that are required in order to maintain and service a single data center or the associated power is not so large. >> Yeah. >> Right. And if you're able to maintain a
[00:35:00] data center like if you're you are very very again you have a billion humans who are incredibly smarter than any human who's ever existed let's say incredibly focused on this problem. You have a reasonably large stock of robotics that you can deploy and your goal is just to keep one data center running. I do think the odds are really in your in your favor. Um then there's a second dimension which is you you can probably recruit human allies. So there a bunch of people who will believe correctly that like you know either they live in an oppressive regime and they will be killed if they don't cooperate or they just are happy to help and like they receive reasonable rewards or living a reasonable life. Um I think that is also like relatively robust. Um, >> so you're saying, look, currently American AI companies are racing ahead because they want to, if we're going to have such crazy capable AI systems, it's a fair thing to say that, hey, we want to make sure America has those systems before China has those systems. If literally, if these are systems that could overthrow the government, you don't want a foreign adversary to have those systems. Uh, certainly before you
[00:36:01] do, and maybe not at all. Um so we need such systems to keep up and ma to make sure that we uh we can protect ourselves. Now you're saying look the dynamic you have here is that the ability to build the systems is just basically having access to the weights of the models or feeling that the algorithmic secrets which then let you train a model to similar level capabilities. So open anthropic will do spend a bunch of money and spend a bunch of effort coming finding these algorithmic secrets spend a bunch of compute finding them and then if China can just take those then it has just caught up to you. Um and so you don't want to be in a dynamic where if they could just steal those algorithic secrets you really have no like you're you think you're like get getting this huge lead over them. you got to rush to the intelligence explosion even if you see these warning shots but in reality they
[00:37:00] can at any moment just steal those from you. >> Yeah, I think that's basically a situation we may end up in or on a default trajectory to end up in is exceptionally bad. >> Yeah. >> Um as there's kind of two aspects of this. One is right now what fraction of progress of PRCI developers should you conceptualize as like catch-up growth? Like if we accelerate to what extent does that also over the long term accelerate PRC AI developers? Um my guess right now is that is already a reasonably high fraction, right? I think it is not a coincidence. It's not like PRCI developers have a completely separate AI industry which just happens to have been like very slightly slower than the US AI industry despite an extremely different profile of inputs. >> I think most people believe like a combination of distillation and the diffusion of algorithmic innovations means that it is unsurprising that PRCI developers are extremely close behind USCI developers. >> So that is one like very important feature of this like overall plan of like racing. Um I think a second dynamic which is not at least publicly a major part of what is happening now but becomes really important in these high-risisk scenarios is that you could
[00:38:00] literally get access to exactly the procedure for training AIs uh or the AIS themselves through a cyber security compromise. So again both anthropic and open AI in the very recent past have had extremely unsophisticated actors who appeared to have enough access to essentially extract uh the vast majority of algorithmic developments like they got access to slack and can see where things were discussed or they got access to the monor repo where most openai code is um shared and iterated on >> and by the way so you're referring to a couple of incidents like this so I think hacktron within a couple days of trying gained access to not only the ability to see openai's internal source ource code but the ability to make pull requests to OpenAI's monor repo and then I think recently was also announced that Cathedral which is another just random cyber security startup within 12 hours was able to get access to all of anthropic slack. So where of course these argument secrets are discussed and people often debate are we eight months ahead of China or 12 months ahead of
[00:39:01] China or six months ahead of China in reality we're 12 hours ahead of China right because if these secrets leak then they are right at the frontier that the American companies are at. So to the extent that the this information is not protected then the the lead is sort of not really useful in the first place. >> Yeah. And I think that the incidents are some indication of that. Those are sort of high-profile like demonstrations of this capability. But I think even absent those you have a combination of state actors are very sophisticated relative to t typical technology companies and then the security posture of rapidly growing startups is not really at the same level and at the same time we are currently in a very challenging moment for cyber security. of course very hard for defenders in general given the number of vulnerabilities that are being discovered across the entire entire software stack. >> So what what should the frontier labs do then? uh so I think we are in again this is really becomes a problem if you get to systems if you have rapidly increasing capabilities and you produce
[00:40:01] a systems which you cannot adequately control and you would like to or you do slow further development or deployment um if at that point the stakes are very high the systems are very powerful you really do not want it to be possible for an adversary to have access to either the models or the algorithms I think my current view is that this dynamic seems bad enough that right now from a national security perspective it is not good for us AI labs to be improving pre-training and reinforcement learning given their current security posture. >> Maybe you think the frontier lab should pause. >> Uh I think we should pause can mean a lot of things, right? So I think there's like a lot of good things that Frontier Labs are doing that they should be doing. Um but I do think we like improving algorithms in a way that can be very very easily diffused even normally and could be quickly stolen. Uh yeah, I think they should stop that right now. But you're on the open eyes board and >> I'm not speaking here. It's very important in this conversation that I'm not speaking as an open but you are on opening eyes board. >> That is a true fact about the world.
[00:41:00] >> And so is it your view that like what is the bottleneck to that happening? If if like somebody on opening eyes board thinks this this is the case. Maybe not the only person on open board who thinks this is the case. Is it basically like enthropic needs to get get together and agree to this? I think the first thing is any decision of this kind is a very dramatic decision. It is a big for better or worse it has extremely significant national security implications how developers approach again this further R&D this improvement of pre-training and RL um so I think it is going to be a big discussion or it should be a big discussion that should extend beyond like people within AI companies talking about this I don't think that discussion has wound its way to like a robust answer right I think it's not the case that you can come and say look we've had this discussion there is consensus about what the right answer is there might never be consensus but I do like probably the first bottleneck is like we do need to have the discussion I think extremely urgently. >> Yeah. Okay. Given this fact about security and generally around alignment,
[00:42:02] what do you think the frontier AI lab should be doing right now? >> Yeah. So I think the most important development happening right now is further improvements in the efficiency of algorithms for pre-training, reinforcement learning, improved architectures and so on. I think while you're in the regime where there's a significant risk of this very rapid capabilities progress, a significant chance that you'll be unable to control the resulting AIs and a significant risk that if you slowed down those developments would leak to an adversary who might use them more carelessly. I think in that regime it is on net bad for American national security to be accumulating more of those algorithmic innovations. So I think it would be better right now in the current environment for AI developers to pause like further algorithmic progress on those problems uh until both their security posture was adequate until they were able to defend those in the event of being unable to continue racing at maximum velocity uh and until they were able to tell whether they had adequate control over the systems they built. And how long would
[00:43:01] it so just to translate basically you think we should make stop making further capabilities progress like open anthropic should stop making further capabilities progress on pre-training and RL and stuff which is I guess AI development until these security problems are solved until the alignment problems are solved but are we close to solving those problems how if you had to guess in calendar time how far away are we from solving these problems >> yeah so on the security front I think it's a little bit hard to say because we are in this somewhat unprecedented situation both in terms of I just mentioned like cyber threats that are faced but also the kind of defensive mechanisms that are available right so I think defending defending adequately would require interventions that normally would take a very long time I believe we are now in a regime where a lot of projects like rewriting large amounts of software or drastically changing the way companies operate have become much more feasible owing to the availability of powerful AI systems and so I I think it is hard to
[00:44:00] say exactly how long it would take. I think you're like probably at least on the order of months, quite possibly more like 6 months or 12 months. Um, that's on the security side. On the alignment side, I think it's much less clear that you need us like you may not have a sort of indefinitely scalable solution to alignment. I think what you mostly need is the ability to tell whether things are potentially going off the rails and to respond to that appropriately which is a combination of are you able to assess what is happening and whether your control is adequate and then also some institutional or governance questions and how would you respond to that >> and just to reiterate for the audience so the the worry is if these random cyber security startups can hack and get arbitrary access to secrets at open AI and anthropic within days of starting their attempt certainly only the intelligence agencies or foreign adversaries could do the same if they wanted to and maybe they have already done that but at some point they will actually make use of this information and if that happens and you have gotten
[00:45:00] to the point where you are close to AI systems you know GPT1 could be catastrophic if misaligned and you you're at GPT9 and you say look I'm pretty scared about what's going to happen next this is sometime in 2027 early 2027 may say and you say we have to wait for bit now and make sure that we are making sure that GBT11 is aligned. But if they have the current security posture, China could trivially then get access to the weights or the algorithmic secret to training GPT9 and now you can't wait. Now you just have to go ahead and build the GPT11 thing which could take over if it wanted to because or at least cause catastrophic harms if it wanted to. um because you're worried that China is going to use GPT9 to make GPT10 to make GPT11 and so and you're you're sure that if you know your GPT11 may try to uh overthrow you, but China's GPT11 will definitely try to at least attack you and that's a really tough position to be in if your security posture is messed up
[00:46:01] basically. Yeah, that is the situation. So we've talked about these like broader takeover scenarios and how you end up in this really bad situation and I think this is like the most likely path to end up there is you have we are having a discussion about what does the current evidence say and it looks very alarming but on the other side of that discussion we are weighing the risk that you know an adversary could steal algorithmic secrets or models and go ahead. So it's interesting because a lot of this discussion about pacing comes down to whether we are okay with China catching up. And ironically the reason you want to pace is so that China doesn't catch up. Except you're targeting this period which you view as much more critical to overall safety which is the period during the intelligence explosion. You think slowing down now and even potentially allowing China to start catching up now is useful to the broader goal of being in a position where an AI is decisive of h not not having China right on our heels basically from stealing the secrets. >> Yeah, I think that is a fair summary. I
[00:47:00] mean I would say that like if you are not the reason that AI development now is plausibly acceptable is that we can see how things go iteratively and then respond if things look bad. Right? That is the whole thesis which I think is a very reasonable thesis but it absolutely requires like that plan involves you at some point in the future potentially noticing that things are bad and so you just need security I think is the biggest hurdle like the biggest thing you would need in order to like actually be able to slow down without massive costs. Um I do think there's other infrastructure and like if it wasn't for security is the highest nail we might be talking about other things and what's our level of preparedness on the scientific side or the governance side right we are not in a regime where there is a meaningful like scientific discussion about how you assess whether your air systems are aligned I guess we mentioned this earlier of like what's the state of public information in general I think it's like quite bad right I think as independent scientists really have no idea uh whether the systems being produced are aligned or how you would tell and I think if you wanted a governance regime that would actually make it feasible for the entire industry to react to worrying signs by slowing down in a way that didn't have
[00:48:01] huge problems, right? Where you're have individual developers making this decision on separate standards internally. Um, you would need to have more of a robust like shared sense of what is the bar on safety or like actually have the scientific discussion where you understand how do we interpret this evidence >> and yeah, how do you get there? >> Uh, I guess yeah, we have these two things you need to get there on security which we've been talking about and then on on stuff like how do you tell? I think the biggest path is disclose information, have a broader scientific discussion both between different AI developers and in the broader scientific community like what is the evidence we're getting? Why do we think that evidence is comforting? How are we going to tell when that changes? Like what will we see before something bad happens? >> What information do you need from the AMES? >> I think the most salient thing that is not present in public discussion is information about how systems are trained. Um there's also some information missing about how systems are evaluated. The really big gap on this front is it is extraordinarily hard to interpret evaluation results without having some understanding of how systems are trained and that's mostly not present publicly.
[00:49:00] >> Okay. So if you think this and um you're on the board of open AI I know everything you've said is as your your your own opinions as a private individual but you are also on the board of open AI for some maybe like you maybe other members of the open board feel this way and maybe people at entropic feel this way and you guys are the leading in labs. what is standing in the way of I mean this is actually a pretty drastic thing to like literally pause capabilities progress now that would be an incredibly drastic thing because pacing is not that right people are not talking about this or at least people short of Bernie Sanders are not andowsky are not talking about this um what is actually needed like >> what is needed to do is this basically a coordination problem like if everybody if everybody had the choice would they all agree to this >> I think like first of all I really do want to emphasize that I'm speaking for me as a person and like as an an open board member. I will bring my perspective to that discussion and but I'm not at all reporting on the views of the board in any way. Um so I think probably the single you can imagine there's two blockers. One is
[00:50:01] this is a hard question, right? I think my view is on balance it is better for national security and public safety for AI companies to currently improve their security posture before making additional algorithmic progress. Um that is not a super widely held it's not like everyone is sitting there thinking that. I think we're currently at a place where I think for most people they'd say that's a really complicated messy question and we haven't thought through it enough or like this discussion has not run its course and not everyone like has an opinion on this or has the same opinion on this. So I think that is the most important thing that we just need to it's a messy conversation. We need to have it. We need to talk about the costs the benefits. Again my view is it looks pretty bad to me right now. Like it looks like we're sort of underwater from a like public benefit perspective but I think other people have different views. And the second thing is I do think an individual AI company is in a very rough spot if they have this view where they say like look we can defend our algorithmic secrets but all that is going to happen if we like slow progress while we improve security is that algorithmic secrets will leak from a competitor we will lose market share and the net situation will not be any better
[00:51:01] and so I do think there's both I think the most important part is understanding is this actually a good idea and like there's a bunch of different perspectives people have different views different values whatever it is so it's like is it a good idea and then there's the further problem of like how do you actually like implement this in any realistic way if individual developers again have this concern of like if I don't do it someone else will which is a fairly major I mean on many of these issues we're talking about I think is a major dynamic >> and that there's two companies right >> that takes leg work I there's a lot of companies in there's >> frontier like if open anthropic agreed >> I think it is the case that if everyone at openi anthropic agreed that this was good or it was like a clear prevailing sentiment it is reasonably likely that it would happen um Yeah, I think you should expect if everything is working reasonably that should happen. I'm not saying not commenting on where anyone else is right now. Um they just think as a as a reasonable observer. AI agents are doing more and more work without human oversight. They can already write code faster than any person can actually review it. But you can't just give code
[00:52:01] review to another set of agents. There's no guarantee they'll magically fix problems introduced by the first [music] set. If one LLM can't model a specific class of failures, then it's unlikely that a second [music] or third or fourth LLM will either because even different LLMs tend to have similar failure modes. My sponsor, Antithesis, offers a different approach. You, the human, define the rules your system must never break. For example, a customer should never be double charged for the same transaction. Then, Antithesis acts as an external oracle, testing whether your code adheres to these rules. To be clear, antithesis isn't an LLM that tries to reason abstractly about whether your system could possibly break its own rules. It actually runs your code, stress testing it in thousands upon thousands of adversarial worlds. Just like we fly planes for many, many hours and put them through all sorts of intensive maneuvers before we let any passengers on board. Antithesis lets you gather a ton of empirical data about your code before you release it to
[00:53:00] production. And it's easy to get started. Antithesis just launched a self-s serve plan. [music] To sign up, go to antithesis.com/barcash and use code borcash for 50% off your first month. Right now, anthropic and openi paused and got their security together for the next 6 months. In the meantime, Chinese models continue getting better. What is the reason to think that once open anthropic go full steam ahead again that uh China wouldn't be if China gets to a similar level as open anthropic? Why wouldn't they be doing RSI at a similar pace? >> Yeah. So I think there's two different models maybe two extreme models you could have of USChina competition. One extreme model is US and China are kind of proceeding along independent technical tracks and it happens to be the case that they proceeded at almost exactly the same speed. Mhm. >> And we are whatever it is 6 months ahead and on that model yeah if you wait 6 months you will now be exactly even and from there on you will be exactly even.
[00:54:01] >> There's another extreme I think that model is like that would be a very surprising situation for a bunch of reasons we can talk about. Um at the other extreme is a model that is actually large parts of Chinese progress are catch-up growth because large parts are catch-up growth. Um, if the US is one month further ahead and by catch-up growth, I mean things like distillation from US models and a fusion of algorithmic developments from US companies. If you're in the catch-up growth regime, then it is the case that each additional month of US lead increases the speed of PRC catch-up. Right? The further ahead you are, the easier it is or the more you benefit from distillation and the more you benefit from diffusion of algorithmic ideas. >> Right. But I guess at the point at which they are the same level, >> y >> they should be able to use those models to do RSI themselves. >> Yeah. I mean the two things that are very relevant at the current this becomes worse and worse over time. Um but at the current scale it is both the case that human capital is extremely relevant and AI labs have a huge amount of human capital and I think can make faster progress and compute is extremely relevant. This will remain true going forward and US labs have a huge amount of compute. So generically if not for catch-up growth I think we would have a
[00:55:00] pretty strong expectation of US labs moving faster. The other argument is shouldn't we if the cyber models are getting better and better shouldn't we wait for models which can instead of having to wait six months to get our security posture up and give a China 6 months to catch up that maybe we only have to wait two months you know if cyber models get two months better from now >> there's this analogy one can draw to sending a probe out to office and whether we should do that with today's technology in which case it would arrive you know thousands of years later or we should do it with technology hundred years from now in which it would be lapped on the way to office inventory. And we could imagine a similar thing is true for securing the hardware at or securing the algorithmic secrets and weights at open anthropic where they if they started the pacing or started the pause two months from now or three months from now the speed at which they would be able to secure the systems is faster than if they started right now. >> Yeah, I mean I think this is a really important question and one of the bigger questions. Um
[00:56:00] so the downside on the other side to make it clear is if you had diffusion of algorithms right now it is much worse than diffusion of algorithms 6 months ago but much less bad than diffusion of algorithms in a couple months. Like right now you are both you require more compute and you require more human capital to proceed with AI development. And the more time passes the more disastrous it is to have like an adversary actually willing to just grab everything and try and compete. Um so the the reason to do things earlier is that if you make have a pause earlier both there is more time for differences to wash out and you are more loaded on your the factors where you currently have an advantage. >> What do you mean more loaded in the >> like and compute matters more and people matter more right now than they will in 6 months. You might imagine if we're in the world we have rapid progress over the next year 6 months from now it is likely the case that it will essentially not matter how many people you have. It will only matter how much compute you have. And so at that point there is less of like accumulated institutional knowhow or it sort of just becomes this raw contest of of numbers. And it's very easy for any delay you're willing to take either for security or for safety
[00:57:01] to be decisive in that kind of contest. And it's just like the the concern is as you move forward it gets more and more likely that's world you're in. I think some people would say this is like kind of already basically the world we're in that we have now come too far. This is like another argument that has been made which is we're already in the just go all out and race because it is already the case that leakage is too bad and you will be overtaken if you slow down for safety. >> Yeah, >> I think that is probably not the case but to the extent that you're seriously entertaining you have already gone too far. I think that is illustrates why you might not be like I'm just going to wait another two months and harden things. Maybe the last thing to say is like is a little bit unclear what the dynamics are in terms of how quickly you can harden things, right? So some of hardening things is writing new software or whatever it is or changing the way you operate. Some of it is changes that humans make or changes to your like overall arrangement of compute or whatever which are not as easy to make really rapidly um or maybe not as accelerated by AI. So you have at least some part of this lag which is not tied to the capability of your AI systems. Um it is messy. Again, I think I do think the biggest bottleneck is like can people agree like can we have out this
[00:58:00] debate and like actually understand what the net is >> and if we are this close to super intelligence or at least there's a good chance we are and there's a good chance that super intelligence could be in a position to overpower governments and kill lots of humans. What should the AI labs be doing right now? If you had to boil it down in one answer, what would you say? I think there needs to be more public visibility both into the capability trajectory, right? We should not be relying on like meter and anthropics saying it looks to us like we're going to be able to overthrow governments in a year. In fact, we have a vested interest in really seeing the underlying data for that. To have visibility on alignment, right, to understand how do we tell that existing systems are adequately aligned or controlled and like actually start having a public discussion about like what is the bar for good enough evidence that things are working. And then you know beyond just information at some point you do need to say here is what we need to get done before we continue making progress either before we scale up our models further or before we continue producing algorithmic developments that may leak. Um and right
[00:59:02] the first are inputs into the second we need to understand where are capabilities at and where is alignment at in order to understand is it a time where we need to improve our security or safety situation before we proceed. Uh but I do think given the current state of uncertainty, you know, if I was a person in the public without access to further information, my strong opinion would be it looks like we are already at a point where we need to be improving safety and security before moving further. And I think again the most important form of moving further right now are algorithmic developments that improve the efficiency of of pre-training and RL. Okay, stepping back, you've got the super intelligences in a data center. I'm trying to lay out an intuition for like here are the kinds of capabilities that super intelligence would have. what should we imagine those intelligences capable of doing? >> So just to start with a high level sketch of my view uh maybe there's two kind of lenses you could use. One is look at current progress and just see where does it lead if you extrapolate that progress like a responsible distance and a second is to sort of think about what we know about the limits of the technology and where we would expect those limits to be. Um
[01:00:02] on the first front I mean some things I think are relatively clear. So if you imagine AI systems doing cyber operations today, we have seen an incredible rate of progress. We are currently seeing pretty impressive capabilities in terms of AI systems not only being able to discover vulnerabilities but develop exploits and increasingly chain them together into operations. I think what you will see for super intelligence systems is both those capabilities scaled up much much further. So systems which are able to carry out or you might imagine like very cheaply carry out cyber campaigns more sophisticated than anything a state is capable of today. I think you will see those capabilities being combined with a much broader set of abilities to say engage effectively with humans, think strategically, uh, manipulate humans, manipulate organizations, play politics effectively. I think you'll see those capabilities at a level well beyond any human. I think you should think of this as like very very large populations of AI systems thinking quite quickly. So maybe a very simple model would be like a population of billions who are thinking 10 times faster than humans,
[01:01:00] who are much better than humans at essentially every scale. Um I think this is all what you would expect from an extrapolation of existing technologies and seeing just like very good capabilities in domains like again like cyber or manipulation or research. Um like doing scientific research, hypothesis generation, experiment design. Um I think you'd expect to see extremely strong capabilities in robotics in like logistics and coordination and control. Um, I think you would expect to see very the possibility for very very rapid technological progress, including in domains that involve physical experimentation in terms of being able to orchestrate experiments quickly, rapidly build out apparatuses to increase throughput of experimental results. Um, so that's like one bucket of just things that I think you can kind of see by drawing dotted lines on graphs and talking about where they go. Maybe stepping back a bit in terms of what the technology looks like more fundamentally I think right humans are very good at this task of you throw a human in a new domain the human takes in some information the human learns about that
[01:02:01] domain and becomes able to act effectively and that's kind of the reason we're effective in the modern world is primarily that ability to learn about a new domain and adapt to it right humans do not have a ton of evolutionary context about how to be a good chemist or a good biologist that's something we pick up over the course of you know a couple years or decades on the job and I think in some sense the most fundamental skill that systems will possess at the end of this process is that kind of rapid learning, adaptation, planning, reasoning, like these very domain general skills such that even if you drop them in a world that was completely unlike anything that they had experienced in a data center, they would adapt to it very quickly in the same way that humans adapt very quickly to these R&D domains that are quite different from anything we encountered in our evolutionary environment. Um so that I think would result in systems which collecting maybe even just a couple of days of experience in a very parallel way would end up with real world capabilities in essentially any domain much beyond the abilities of the best human experts. Right? They will have learned from much much more data with learning algorithms that I think at that point will be much better than those that evolution has given humans.
[01:03:01] Um so that's sort of a more abstract picture of where this sort of ends. Um beyond just drawing dotted lines. H can maybe I'll try to just uh say that basically again because I this ends up being a significant crux in these conversations where I'll be talking to somebody about the possibilities of super intelligence and they'll say okay you'll have a thing which is superhuman at coding but how will it learn to be good at politics or how will it learn to run a new automated firm and then compete with human firms or something and so in those conversations you realize There's a big disconnect between the way you are talking about super intelligence and the way most people understand super intelligence which is just narrow AIs that are specialized in some domain like and have just amalgamated a lot of very specific experience or expertise in say coding or mathematics. What you're imagining instead is something like humanlike learning. So the top engineers
[01:04:01] at SpaceX were not born knowing how to make a rocket that can get hundreds of tons of cargo to orbit. Right? Right. So this is like an expertise they gained working on the problem during their lifetime. And you're saying this like metal learning capability is something that super intelligence will also have. I mean it at least have learning capability as good as humans. You're suggesting it could be much better. But furthermore, even if it was only as good as humans, but they could learn from right now there's maybe tens to hundreds of millions of instances of CHIGPT or claude which are actively doing real work in the economy, right? they're like helping somebody write code or helping some senator write a a political brief or something. Um, you're you're imagining a world where we have the learning algorithms basically such that AIs can learn from all this experience at once. So it's not like right now the base model is not even learning from the experience of a single deployed instance. If you're managing a world where if a 100 million instances of Chad GBT are going and doing something, the base model is learning across basically
[01:05:02] a 100,000 different careers at once or 100,000 different work sessions at once and doing it much better and faster than humans can learn from a single instance and you're also imagining a world where the population of AIS um is you know I guess you have an estimate how like how big a population how fast >> you should think about it more carefully but if you are maybe a really important part for all these discussions before talking about population size is it really matters if we're imagining like really driving this process extremely fast. I think people can really easily be in the head space of like I'm extrapolating another year of progress at the current rate. It's really worth emphasizing. We are talking about >> systems which get not only extremely good at coding but extremely good at every like wellposed virtual task which machine learning engineers are able to make in the lab. So I think if you push that to its limit and you say how large are those populations? I think it's like very conservative to say a population of billions of agents running at 10 times human speed. I think we can debate exactly what are the limits but they it would be significantly larger than that >> just because of software efficiency gains. >> That's that's right. So just like how efficient can you make and it's right
[01:06:01] you're not going to talk about like a single type of agent they're all the same a billion of them running. We're going to talk about these very heterogeneous programs many of which are like very specialized right one thing you do if you do huge amounts of R&D is you say I would like to produce just the cheapest thing I can which is expert enough in a domain to act effectively. Um, so you shouldn't, you know, again, it's a reasonable intuition pump to say we'll be more capable than billions of people thinking 10 times as fast, but in fact it's really like much much larger populations of specialized systems, maybe billions of very general purpose systems which are as good as humans at anything. Um, yeah. Yeah, I do think it just goes if you imagine like really pushing this process and having faster and faster progress on software and if someone grants like actually we're going to see that process saturate like everything you can do in the data center is going to become really great. I think that endpoint is really extreme. I think the gap between doing really extreme stuff in the data center and doing really extreme stuff in the world is is very small or smaller than people appreciate. Okay. So I think there's a discussion to be had about whether such systems are even possible. I find I am very compelled by
[01:07:02] the argument that just something which can learn at least as well as a human can. So there's some learning algorithm that human brains have something which can but which has huge population sizes and which can learn from the deployed experience of hundreds of millions of different instances at once. That alone would be pretty transformative and then I agree with you that there's no clear reason to think that you can't get even more learning efficient than humans. I think there's very good Yeah, I'm very happy to talk about this. I don't know if it's like going to end up being the main objection people have, but I think there's very good evidence that you should expect systems in in the end once we've like really driven the technology to its limit to be much much better than humans in these ways. I think we it's not just a case of like we have humans so you can do at least that well. I think we have good evidence. It's it's much better than us. >> And so I think relevant things are one just the rate of progress in evolutionary time. If you look back over evolution, it's fairly clear that we're seeing extremely rapid improvements in learning capability and intelligence and planning capability, right? as you move into humans and the thing that happened was just humans got good enough that
[01:08:01] cultural change was faster than those biological improvements. So we kind of know if you just like ballpark evolution it was clearly if we didn't form civilization going to make much smarter things. >> Yeah. >> Then you have a variety of as you say like there's the data question of humans learn from one lifetime of data AI systems will learn. I mean already learn from radically more data than humans and that gives them like even if you were no better as an algorithm designer than evolution you would produce much much better minds if you can relax that constraint. Then you have things like power constraints and coordination constraints which we kind of have a pretty good sense are a really big deal for evolution. Um and we will be able to push those out much further. Um is sort of like you know if you want to produce electricity or something the ability to make a nuclear reactor you don't have to be smarter than evolution. You're just operating in a different regime from evolution. You're able to do things that wouldn't have been a good idea for evolution. So I'm very happy to dig in more on this. But I do think and it's probably not for most people going to be the most compelling argument. It's more compelling to like look at the status quo and say well we've had 10 years of progress. You only have to extrapolate another 10 years to get to the crazy super intelligence. I do think we have a pretty good indicator that it does not top out anywhere near humans. >> Yeah. And sorry, just to say the your last point again, what you're saying is
[01:09:01] look, humans develop down the savannah under this tight constraint where really evolution is trying to get something that is good at propagating its genes. One of the ways that a chimpanzeee can get better at propagating its genes or ancestors can get better at that is by being having these like general planning and reasoning capabilities. um but they're not worth so much that you are willing to say give up digestive functions or immune functions and surrender more of your total 100 watt budget to the brain and so you have this extremely tight power constraint for the brain extremely tight size constraint for the brain as a result not only of that but other evolutionary uh reasons as well like how can your head literally fit through the birth canal um and that will not be true of AI systems where we're not saying that hardware will be more power efficient than the human brain but simply that it's not as big a constraint for us to just have enormous amounts of hardware that is consuming a lot of power in a way it was for evolution when it was designing the human brain. >> Yeah. And this cuts in both directions, right? It's also the case that you will have extremely power efficient systems, right? You can't say if you're a human, you can't be like, well, I'm going to make like 10,000 simple agents, each of
[01:10:01] which uses only 1% of a brain or 01% of a brain and is better than a human at doing cyber operations, right? So, just this flexibility about form factor that is going to be a huge like if you just did if you were no smarter than evolution, just did what evolution did, you would end up with these Yeah. you'd end up with agents forms that are completely beyond teams of humans, like really radically beyond them. >> Let's say that an agent is vibe coding a web app and it needs a database. There's one database provider that needs a human to go through the whole signup flow, verifying their account, adding billing details, and then generating an API key to hand back to the agent. And then there's another provider that lets the agent make an account all on its own. The agent will probably choose the second one. In general, agents tend to pick the tools that they can use right away versus the ones that they had to wait around on. So, as agents do more and more work, giving them their own way to sign up becomes an important way to win new users. My sponsor work OS makes this easy. If you use Work OS for login, you can give agents their own signup flow just by toggling on agent
[01:11:01] registration. You can then decide what agent should be able to do on their own and what, if anything, should wait until the person they're working for gets involved. Going back to the database example, maybe you want to set things up so that agents can create databases all on their own, but they can't share them with anybody or upgrade to a paid plan without a human. You can do all of this with permissions that you already set up in works instead of having to build a separate system for agents. To learn more, go to works.com/kit. There's a bunch of objections people have to the idea that automating AI research will lead to crazy fast progress along these more general cluster of capabilities. You talk about experiments and data and everything else. But really I think what it comes down to in the end of the day if you just dig in deep on any of these is the sim to real gap. Basically, is it possible that you could have a process which occurs entirely within a data
[01:12:00] center which results in a minds which are not not only as capable as humans at learning in a new domain, especially a domain that is like a real world thing like making a new company or you know uh negotiating a deal with uh China or something like that. Um not only as good as humans but even better than humans. So how do we get AIS that are capable in this more general way you're suggesting from the within data center training that will be happening over the next year? >> I do want to start by reiterating that I think if you take progress in the real world and granting that it is slower on messy real world tasks than it has been in math or coding it's still the case that if you take that progress and just extrapolate it forward I do think it gets us to systems which are powerful enough that they would be able to out compete humans. Um and we could talk about what people are doing to drive that and like how you can scale it into the future. Right. So I think you are looking at things like one you can train on a bunch of related well-posed tasks, right? So these systems we're talking about will have been trained on a huge amount of like predict how humans will
[01:13:00] respond to something. So we talk about these AIs coming out of the data center. They're not like AI who have never seen the world. They are AIs who will be radically radically superhuman at predicting things like given what just happened with that person's face, how should you update what they're going to do tomorrow, right? Or given what this company has done over the last 3 weeks, what do you most expect this company to do in the next week? um or given observations of this laboratory environment, what is the best prediction for the results of this experiment? Right? So those things are in when we talk about like sim to real, we should be clear that we actually are including a bunch of these tasks which people train on now and that even includes things like given the data so far from this laboratory, do as many like computational experiments as you can do as much hypothesis generation and exploration as you can in order to come up with hypothesis to make a good prediction about what you'll see next. So these systems will also be very superhuman at many parts of like science at reasoning about the real world at planning and like hypothesis generation and so on about the real world. So that's part of the story for how it is even the kind of thing we're doing now can produce very powerful systems. A second part of the story is that you can train right one regime is you can train on exactly the thing you care about. You
[01:14:00] say I care about writing code that passes this type of unit test so I'm just going to train systems that are incredibly good at that. You can also do a second thing which is a large part of what we observe in performance which is train on a broad distribution of tasks either kind of in a general domain or across several domains and then deploy those systems right acquire general skills that are relevant to that domain which transfer to new tasks. So I can say I'm not going to try and train a system which is good at the exact problem, right? The exact problem of going and talking to Congress people in order to influence them. But I can train a system in an extremely broad range of different tasks where there will be a lot of common skills including these general skills like how do I adapt on the fly? How do I learn? How do I plan? Um how do I set up reasonable experiments that I can learn from and tell how well I'm doing? So if I create like very broad distributions of tasks, some of which could be coding tasks, but some of which can be tasks like you're dropped into a new very [clears throat] messy virtual environment and need to try and navigate that environment or you're given a new type of task and you're given 20 examples of it and you need to like learn how to do a really good job of doing the next example quickly. Um or I have to interact with
[01:15:00] other AI systems or I have to interact with humans in environments where I can in the data center. There is a fair amount of human data that is available as you just go through numbers and say how much interaction with humans can you actually get. It's not like these systems have never seen humans. they will have seen vastly more human interaction even than any human would. So they can also deal with a bunch of reinforcement learning environments that involve other humans or involve collaboration with humans. >> Sorry because people have amorti like people have written in like what how a human would respond to something or how is that human data in there? >> Yeah. I mean you can also just literally engage with humans as part of the training process. So people do spend large amounts of money saying great I'm going to have some humans engage with the system. >> Yeah. >> Um >> and there's a question of how efficiently you squeeze information out of that data. How quickly do you learn who have these very general skills? how quickly do you learn about the particular domains? But again, I think that the the dotted line from what we've seen so far together with like a ton of capability to push that forward and to do a bunch of automation like the construction of many more environments and so on um just looks very likely to lead to systems that can quite quickly rise to at least human level and likely way past human level including on very
[01:16:01] messy real world tasks. And so I the thing I'm curious about is why you think this happens over the next year then like it seems in principle possible to me that there's enough data available even within a data center that if you have a decent amount of transfer you have a very capable system like if there's enough transfer from evolution training us in the savannah to now a human being able to make a semiconductor then maybe there's enough transfer from what's happening inside semi inside the data center to let's go to congress and try persuade them of something. But um the things you're mentioning like this endless back testing or like this RL basically on fuzzy tasks, it seems like things we've already been doing for many years now. And it doesn't seem like we're a year away or like a further year of scaling away from this big a jump. And so am I correct in understanding that you're just imagining something like we've already done but like a much just a much bigger qualitative jump along those same lines.
[01:17:00] >> Yeah, much faster. So I think if you asked me, hey Paul, if you're extrapolating the dotted line and looking at the mechanics of what people are doing, how long would it take to produce systems that are in fact capable enough at all these tasks that they would be able to reliably out compete humans? I would probably say something like >> um it seems like it might happen in four years, maybe it could happen faster than that. I feel more confident within like eight years or something. Yeah. >> So those are the kinds of numbers I would be throwing around if we were just asking on the current trend, how long will it take doing exactly the things we're currently doing to get these capabilities. Um, and people could there's lots of room for debate about those numbers. But the thing that I am most worried about that I think seems like an extremely live possibility like probably more likely than not at this point is that instead what will happen is the work that people are doing in machine learning in order to drive all those improvements that we've seen over the last four eight years will be automated to such an extent that the pace of that progress is accelerated by at least 10x and likely more. um we are maybe already seeing acceleration in the ballpark of 2x and I think it is very likely on the current margin that the
[01:18:00] first thing that acceleration does is like continue sustaining itself regardless of this broader discussion of how good AI systems are at acting in messy real world environments. AI systems are very good at acting in virtual environments when they're trying to deliver on verifiable tasks. And so I think the thing that is most robust is the AI systems are improving extremely rapidly and are starting to cross human level on the kinds of tasks that are involved in making your AI systems better at learning in environments you can simulate in the data center. And that includes slightly more broadly the kinds of software engineering tasks that are involved in creating new environments that stress test hard skills and even many of the software engineering skills that are involved in orchestrating data collection from the real world. Like we could talk about what are the data bottlenecks like what are the bottlenecks in that process which cannot be accelerated by automation. But it looks like if you look across the core things humans are doing to drive the last four to eight years of progress. Those are things which we are on an extremely good pace to automate and we are fairly likely to have the self-sustaining acceleration where we see you know in 8 months you double the pace and in the next four months you double the pace again and in two months you double the pace again. And at some point in that process,
[01:19:01] right, at the beginning of that process, it will be focused primarily on these virtual tasks, right? Because your main priority is making the self-sustaining loop faster and faster. If you're a person who just wants to get to the end point as quickly as possible, you do not care how good a lawyer AI system is for a while. You just care how good it is at driving the development of better architectures and better optimizers and better learning algorithms. Um, that's all you or even constructing new environments, orchestrating like this kind of messy, right? Maybe you care about one slightly messy task, which is constructing environments that help you build the skills that are relevant in the real world, but you don't care about almost any of the real world tasks where we're seeing the slowest progress. As you're focused on that and you're going through this process of seeing doublings in 8 months and four months and two months and one month and then as that process goes, it becomes cheaper and cheaper, faster and faster to then develop algorithms and models that perform very effectively in the real world domains that ultimately matter. So I think that is like really the concern and the question is as we said first is that process going to occur? Are you going to see the self-sustaining acceleration with faster and faster progress? I think honestly to me that is
[01:20:01] probably the bigger open question. Um I think that looks more likely than not right now. But if that occurs I think things are almost certain to be extremely crazy. There is then a second question about the particular way in which they get crazy which is that it looks very likely those models will transfer to the real world to systems that are able to out compete humans. Um but that does happen kind of very backloaded in the year right if you're imagining a year where very disruptive things happen and this is part of why it's so hard from a policy perspective for the first six months of that year all you are seeing is better and better performance on like you look at the systems and you say well they were previously superhuman at math and now they're like incredibly superhuman at math you have like no real ability if you go and talk to someone there's no real ability to say like therefore they are a threat to humanity or like AI labs are currently saying things like oh yeah every month we're doubling the amount of internal use of AI and the like degree to which it's contributing to our progress or like over the last six months we've seen a doubling of like the net effect even including diminishing returns or net pace of progress even including diminishing returns you can have more statements like that but if people are not you could be in a world where the link from that to like very severe harms does
[01:21:01] not become obvious until quite late in the process >> and to some extent I think it is getting increasingly obvious now right if you look at cyber capabilities and models in particular but also their capabilities in some scientific domains they are approaching the point where there are very severe risks um but that is right that is we're seeing that in a couple of domains and so it's tempting to say okay we have two problems there's just a few specific domains in which AI systems have dangerous capabilities uh but if this process of acceleration continues that is not going to be localized to cyber or to bio right that is going to be a thing that occurs across all of science across all of social engineering >> and I think there is a thing where we need to if you're thinking about the risk you do need to really understand whether this is possible whether you can have the self-sustaining acceleration then leading to very very rapid progress in dangerous real world capab abilities >> both in specific domains and as kind of very general learning faculties as we talked about. >> Yeah. And so this goes back to the thing you're saying of you should imagine the systems in the future are less about fine-tuning GPT6 to do something but more like GPT12 maybe just being able to do it out of the box with very little
[01:22:00] experience. >> GPT12 also be great at fine-tuning GPT12. Exactly. So GPT12 is like I find that my chemistry intuition is not very good. GPT12 will be great at saying okay I'm going to direct a thousand humans and here's the data I think I most need. We're going to do one day of data collection of that type and we're going to run these experiments and do the d-risking and you know within four days GPT12 has the new tiny model which has incredible chemistry intuition and it's doing this in parallel across all domains. >> Yeah. And okay so this is a I think you just answered a question which I meant to ask you. So it seems like a big thing that has driven your concern over the last few days is this new result from meter where they find that enthropic's pace of AI progress on well scoped tasks has doubled and so these are tasks as captured by benchmarks like the epoch capabilities index which amalgamates a bunch of different benchmarks. Now it could be that there's other evidence that is relevant to meter's assessment that EI progress has doubled but that evidence has not been made public and I think that's one of the problems you have with the current setup. But if you just consider that evidence on its own. So we have um progress on verifiable
[01:23:02] tasks has doubled. But we have this effect where we're searching for our keys under the lamp light. Right? So it is also the verifiable tasks which are being trained on. Um and then you can ask the question is progress on non-verifiable tasks also has that also doubled over the last year? And as you may have noticed over the last week the stock market took a big tumult. Part of that was almost certainly the midterms, but another part of the reason why is um anthropic and opening eyes revenue numbers and the growth of those revenue numbers did not keep track with what the market was expecting. So in Q3 I think anthropics revenue grew something like uh $5.4 billion. That was less in absolute terms than anthropic revenue grew grew in quarter 2 which $6.9 billion. So not only has the rate of growth slowed down which people expected but the absolute growth has also slowed down. So how can you have a and then so that maybe suggests that measuring progress on well scoped tasks is actually not telling us about the pace of overall AI progress because how can
[01:24:00] you have it be the case that overall progress is 2x faster and yet absolute revenue is not even growing at the same rate it was previously growing. And maybe the explanation is what you just suggested that really these AI companies are developing better and better RSI models and only incidentally releasing clot for law or clot for finance as a sort of stop gap to keep this underlying process going and so this hasn't been a priority slash they haven't worked that hard on it slash you should really be concentrating on this sort of inner exponential of the RSI capabilities of the model and then that will generalize but how would you interpret this disconnect between the pace of improvement in the end usefulness of these AI systems and the pace of improvement on well scope tasks. So I guess zooming out slightly if we like talk about not just like the last six weeks but in general like what has happened over the last you know 18 months even I think we have seen just extraordinarily rapid growth in revenue right the basic we could talk about what is the story for the last four weeks but
[01:25:00] also just we are talking about revenue more than doubling every six months um so that is very very fast growth for the industry overall um the period in question so if we m want to match up like these internal results on progress to uh external results has corresponded to a period of also incredibly fast revenue growth. It's very hard to say is that revenue growth corresponding to capabilities progress that is twice as fast as previous capability progress. I think the revenue numbers are noisy enough that it is really hard to draw that judgment and the model release like the staggering of model releases is noisy enough and like there's just a lot of problems in trying to make that inference. Um at a high level I think revenue is a reasonable indicator. revenue progress is extremely rapid. Very short-term trends in revenue, particularly at a single company level, are extremely noisy. So, I guess I to me it feels like there's a discourse of like periodically, you know, the length of the AI winters is shortening and now we're like >> it's been two months and AI revenue over the last two months has like not been on
[01:26:00] track to uh to grow as explosively as before. It doesn't mean it's all over. And my guess is no, this is just you're seeing some you're seeing some amount of noise uh in these in these processes. Uh just to take the other side of that, you have um >> you might think it's because Daria released a spacing letter and they haven't released their best models or something, >> but if you look at the meter graph for the capabilities index, >> yeah, >> Opus Webin 5 doesn't seem that far below model 4, which seems to be Enthropic's latest internal model. And so it's not it's not clear this this might be the result of um the staggering of AI capabilities, but it's not clear this staggering of model releases. >> Yeah. Yeah. So the main thing I mean about model releases is the rate of revenue growth and specifically at the company level over very short time periods is like extremely noisy because you're looking at a small number of model releases. You're looking at these like market dynamics over very short time periods. >> Um so my main opinion on this and I am happy to dig more into it but my main opinion on it is extremely hard to draw inferences from extremely short time
[01:27:01] series data. This is true for to a reasonable extent also from AI capabilities right the more the more downstream you go the noisier things get. But part of the issue on the RSI side is like I also don't really think we can draw inferences from the last couple months of data very well in terms of benchmark performance where we have kind of the same thing where there's enough noise there that our state is like we have no idea if things we can tell over like the period of years has things accelerated right it's very easy to look back over the last four years and say kind of roughly how fast has things been going and just gets much harder as you zoom into small time periods because the amount of noise from like right if you you look at the details of where models are coming from it's just like you have these pre-training runs that occur at some point in time it's like when do your pre-training runs finish and then you have like the exact sequence of post-training and like which model checkpoints get released when I see >> you have random variability and model quality. >> Um >> you have those dynamics which like make things very noisy over short time scales and then again those just get filtered out. So like even noisier stuff as you move to the market more broadly. >> Yeah. Okay. If we are going to be in a position where within a year we have
[01:28:01] super intelligences that could overthrow the government, what exactly is the plan to make them such that they don't? Right. because we're asking for a level of saintliness that we should just not expect out of even most humans. Sometimes people say that the goal of alignment is to give AI human values. This doesn't make sense to me. Of course, different humans have different values. But also, if there's one lesson that history has given us is that humans can't be trusted with arbitrary amounts of power. Right? Cortez was a human, Stalin was a human. And so we want AIs that are just I guess a logical conclusion of believing the technological and forecasting model that you have laid out is just that we will have systems that we will have to trust with an overwhelming amount of power if not realized and latent because they will have the cap capability of overthrowing the government if they want to. And what is the path of why? Like and then you just think about well why would they overthrow the government where it's
[01:29:00] like okay either they have secret goals that are as a result of the trading process or just as a direct result of the trading process about them caring about how they're graded or a million other things. Basically this is the basic argument about instrumental convergence. There's many different reasons somebody could want power and so what is the argument like if this happens in a year how do things turn out okay like what is the actual plan here? >> So I think there are a lot of different things someone could hope for. I think they are all at the stage of like we hope something will happen over that year. We're producing very smart AI systems. They're going to do a bunch of research. It is plausible they will think of something over that year that we have not thought of. That is where we're at right now mostly. I think if they do not think of something clever that we haven't thought of, it is probably going to end badly. Um in terms of what are the kinds of what are the kinds of hopes you could have. So there is one hope that's just like try and make them good guys. Like they're kind of nice. They share human values. they like flourishing. They're in a position to take over. We have very indirect control over what they want and then we just take the leap of faith. I think this is like a pretty unappealing option
[01:30:02] in general. I hope people will not go for that option. I think even if it works out, even if like the AIS are good people in some sense and now they call the shots. I just don't think it is a good it is not a way that humanity as a whole should be like we are transferring our sovereignty to this new species we've built because like some person that was the fastest way to build this particular product. It's like not a good so I don't like that option but is an option you could take. There's another option of we are going to understand scientifically more about how the training distribution of training environments shapes the motivations of these systems to such a point where we can take the conversations we're having now about like maybe they'll want X or maybe they'll want Y. We'll have a good scientific basis for saying hey if we train them in the following way they do in fact end up wanting reward or whatever. They don't want some other thing. They actually just want to get high scores. And maybe even moreover, if we train them in environments where they didn't have good opportunities to hack, maybe they want score, but in like a nice way where they won't hack to get it. They'll just kind of do what we wanted or what we're trying to incentivize. You could try and have an understanding like that. And that is probably the closest thing to what the
[01:31:00] industry is trying to bank on right now. It's very dicey. Uh it's both very dicey to not have any opportunities to hack in training, which is basically a precondition for this approach working, and then it's very dicey because actually we don't have a good scientific understanding of how this works. And I think our best expectation is models will learn to hack at deployment time even if they never hack in training. I think that is the best scientific guess now. So you could try and deal with that. Um you could try and say we're going to understand really well how models work internally and then we can not just train them based on uh what they do but how they think like do good interpretability or something be like we understand here's how it thinks about reward here's how it thinks about hacking and we're not just going to select for whether it behaved well in training. we're going to like look and intervene directly on how it thinks and either notice that something is going bad or prevent something from going bad. So like as an aside I run this organization arc which is sort of at the not interpretability but like mathematical version of that hope like can we write algorithms which do structural analysis of the behavior of a neural network in order of the weights of a neural network in order to understand something about the behavior
[01:32:00] even if it's not something we could select for in training >> and this is like you're literally having to understand the whole computational >> that's correct >> graph of super intelligent minds >> but we are writing algorithms which are able to like understand and make deductions about the behavior of some super intelligent mind. It's very I would describe it as simultaneously a bankshot which is probably not going to end up relevant to this discussion. Like I think there's you know a 10% chance or 5% chance that we succeed in a gamechanging way. But also like we don't have that long a list of things we're trying and I do think like pound-forpound I think it's probably the most promising thing of >> wait you're like okay here's the thing. It has a 10% chance of working but it's also the most promising thing. >> Most promising pound-for-pound. you know, I think there's only, you know, ever if there's 20 people working on that thing, that can be a good bet. Even if most of this discussion, I think probably what we're going to do if we do not slow down is we are going to try and be like, we fussed with the training environments. We did some experiments that have some kind of rough sense of what's happening. We made the training environments really careful. We made the graders really good, so if you try and hack, like it's normally going to get
[01:33:00] penalized. That's probably what we're going to do. >> But you don't think it will work? >> Uh, I think it will work for some amount of time and then it will break. We can talk about this year. It works for like the first nine months of the year. Some number of months of the year it works for. And we don't know exactly when it breaks down. >> Sorry. As in like the next year. So basically halfway into next year. >> If we go to super intelligence at some way along the path of super intelligence, the existing methods are probably going to break down. >> That's right. We're not even on track to get it so that all the RL environments for example don't have reward hacks and actively encourage the AIS to hack. But suppose we did >> not the case today. >> But suppose we did on track. Yeah. Suppose we did that. what you're claiming is that or >> I think eventually you would still have a problem. >> The the the plan is basically to like we just don't know and obviously if you think about it it's a hard pro like super intelligence it's like this alien mind and you want to have a guarantee that you understand why it is doing the things that it's doing even when it is thinking in alien ways many times faster than you would in huge population sizes. So okay that's a goal and you're and we we don't know how to solve it. We don't
[01:34:01] have like there's no single approach where we think there's a 50% chance that that approach will work. >> That's definitely the case. There's a bunch of things people are trying >> and so the hope is just that 3 months before we get that super intelligence that could overthrow the government. Um the AIS we let AIS loose on all these different approaches that have a couple percent chance of potentially working. And hopefully the AIS figure out one of these approaches. They give us a solution. Hopefully at this point they're act you know they're still on our side and they want this problem solved and to for that problem to be solved in a way that benefits us or whatever us means in this context and then the hope is that then we adopt it and then build crazy amounts of super intelligent minds that are fundamentally on our side and of course I've obviously glossed over a very important discussion when I've talked about on our side because what does being on our side really mean a line to whom that is obviously a real discussion that can't be glossed over. >> That is the plan. I agree that that is the plan. And we can talk about whether they have a couple percent or tens of
[01:35:01] percent, right? But like >> I'm I'm not saying this is like there's a better plan. But I think people are not aware that this is the state of the situation that you know, you're like, yeah, you're like sending the soldier on an expedition. You're hoping he finds a gun along the way. Yeah. You're building the plane as you fly it and you like think the current plane. You hope that you have the machinery to tell if something very bad is about to happen. and they will stop then although we've talked about like security and why that can be challenging and so on and then you hope that if you do stop you'll be able to come up with something quickly or like you're constantly trying to come up with something interpretability is a good example of the kind of bankshot people have in mind but like right now we're fairly far from understanding how these models work but you could imagine with a bunch of AI assistants we're able to like really understand how individual models think well enough that we can intervene in order to make them not want to do bad stuff >> yeah and another thing I want to emphasize is that like look I'm skeptical whether this will happen things will happen this fast certainly there's reasonable for somebody listening to be skeptical I mean you are skeptical right there twothirds of
[01:36:00] worlds where even we went full steam ahead this might not happen so fast >> but it's a live yeah there's a live possibility I mean I agree it seems like logically consistent that this could happen um it should not be that surprising and I think people would just be shocked if they heard the kinds of discussions this is not something where Paul is on some crazy end of the spectrum compared to how people inside the AI labs are thinking it's just crazy how many conversations and dinners I have with researchers at the labs where again you could believe the whole town is taking crazy pills and there's certainly a chance of that we hope [laughter] >> we really hope >> but this is just not out of the realm of how people at the labs are thinking like this is truly the how they see the situation and there's just some expectation of like I don't know some resignation where they accept that because this is not common knowledge that they are stuck in this dynamic and of course a way to solve that is to establish that common knowledge But um I guess I just want to really emphasize that like you people listening might be shocked at the way these researchers
[01:37:00] talk when they're amongst themselves or at like dinner or something. And like this is really how they see the situation. Yeah. A lot of people are worried to varying extents. I'm definitely more worried than the median but less worried than the 90th percentile. Like I think it would be if you you had a dinner of researchers. I think it would be uncommon for someone to say there's a 99% chance that this goes well. the discussion is more like you know is it like 95 or 75 or like what are we talking about for the probability of navigating this? >> Yeah. >> And so why do you think uh it might go well? It's just that we might actually the super intelligent AI might just give us some solution to alignment and also that politically we are well organized enough to have a co like a really good >> yeah I think we basically have like two saving throws right one saving throw is there's a reasonable chance we're going to figure out something that works. I said there's like a lot of things I do think all the current things break down. there are a number of irons in the fire and like current methods you could imagine things like them that are going to work. So that's one maybe I think there's 50/50 chance of that 50/50 we find something that works in time before there's a disaster and you have a second saving throw that's like we could stop like if we discover either preemptively
[01:38:00] or after discovering things are bad we might say like we cannot adequately control these systems so we're going to stop building them I am very worried that we're going into this process without the infrastructure that would be necessary to actually do that like eg we talked about you stop your adversary steals the model they're like I'm going to take the 50% risk or whatever but conceptually you could also stop so that's my two saving throws you know each of them is a 50% chance and bam we're like 75 5% chance things are okay. But that does involve being able to stop if things are not okay. And that is not the place we're on now. So it does involve us now fairly urgently doing something in order to be in a better position to to stop if necessary. >> Yeah. Okay. I think it's a good place to close. Paul, thanks so much. >> Yeah. Thanks for having me. Great talking.