[MUSIC PLAYING]
ezra klein
I’m Ezra Klein. This is “The Ezra Klein Show.”
So let me start this one by situating us in time. So we taped this podcast just a few hours after OpenAI released GPT-4, which is their most powerful A.I. language model yet. The thing that is remarkable here that I really keep trying to push people to key into is how quickly these systems are getting better, how steep the curve of improvement is. We talked about some benchmarks here on the show. But GPT-4 passes the bar exam in the 90th percentile — the 90th percentile. Just sit with that.
We were nowhere near fully understanding what the last generation of these models could do. Now we’re on to the next one. And the same is probably going to be true here. GPT-4 — it is extremely powerful. It is, at times, unnerving. We barely understand what it can do or how to use it. And yet everybody is racing ahead. GPT-4 also is going to be a toy, a diversion, compared to what we’re going to have in three or five years.
If you heard the commentary I released on A.I. on the podcast over the weekend, you heard that what I’m most focused on is how society manages to adapt to the speed at which these systems are improving, or else slows them down so we have more time to adapt to understand them to think about how we want to integrate them. And one piece of feedback I got in my commentary was, what are you really saying we should do, aside from being anxious? And I don’t just want people to be anxious.
So this conversation gets more into specifics, both on what these systems can do now, why they could be so disruptive, where they seem to be going, and what we can and I think should do, or at least the kinds of things we should do in response.
Kelsey Piper is a senior writer at Vox, where she’s been doing great coverage for years on A.I. and the technologies and the culture and the safety and economic questions around it. She joins me now.
As always, my email, ezrakleinshow@nytimes.com.
Kelsey Piper, welcome to the show.
kelsey piper
Thanks. Glad to be here.
ezra klein
So one of the reasons I wanted to have you on here is you’ve done a lot of reporting, you spend a lot of time with folks inside the A.I. world. So you’re a little bit more in this hothouse than a lot of us. And when you talk to them over the past couple of months, what do they say? What are they seeing or what are they thinking about that from the outside would not be obvious?
kelsey piper
Yeah, so I think not just in the last six months, since at least 2019, you had people saying this is going to overnight — not today, but in 10 years, overnight — transform our world. We are going to be able to automate any work that can be done remotely. We are going to be able to bring us the next 100 years of drug discovery and development in the space of maybe a couple of months, maybe a single research project. We have this extremely powerful thing we’re on the brink of. And in 2019, when you heard that, I think you had to be a little skeptical. Like, this is Silicon Valley. We’re not new to the hype cycle. Sure you’re going to change the world.
And now I think you’re still hearing the same things, mostly from the same people, but you have to take them a little more seriously. They are, in fact, releasing things that can do things that everybody thought were impossible. They are developing the newest models as fast as they can. OpenAI says we think the way for humanity to do well on A.I. is to race ahead as fast as possible. This was in a statement a couple of weeks ago. This morning, GPT-4 only a couple of months behind ChatGPT, which hit 100 million users overnight. It’s not even that they’re saying new things. It’s harder not to take them seriously.
ezra klein
So when you say overnight, this idea that sometime in the future, all of a sudden, the history of A.I. will speed up, that we’re making progress fast, but it’s a couple of years, a couple of months between major new systems, and then you get these ideas of quantum leaps really quickly. What are people talking about there? What are they looking at?
kelsey piper
So I think the most concrete version of this is that right now, all of the work done on improving A.I. systems is being done by humans. And it’s a pretty small number of humans. And of course, one of the jobs that they are trying to automate is their job, the job of making better A.I. systems. And when they succeed at that, if they succeed at that, then you can, instead of having hundreds of people working on building better A.I. systems, you can have thousands.
You can have a million because you can just make a copy of an A.I. system. That’s like its fundamental economic potential, is that you can make a copy of it. You can make an A.I. system that is as good at writing as you. And then you can have 1,000 of them writing about all different topics. You can make an A.I. system that’s a really good programmer and then you can have all of the programmers working with or possibly replaced by that thing.
So things are already moving fast. And I don’t think the case for taking A.I. really seriously crucially depends on they’ll start moving even faster. Like, the current pace isn’t fast enough to me to be disastrous. But if you ask, like, what’s my gut instinct about what’s going to happen, it’s that at the point where we figure out how to make A.I. researchers out of A.I.‘s, then things will move real fast.
ezra klein
And that sounds very sci-fi, as a lot of things do, because we live in a sci-fi world increasingly, but one of the things that I know is striking to a lot of people in this world was when GPT-3 really proved able to code, even though it hadn’t really been trained to do so.
And so then, as people have begun moving more towards optimizing A.I. systems to create code, and you expect we’ll do even more of that, training on even more code, get even better at that, once A.I.s can code, which they can already do, when you just assume for the improvement curve, then the idea that you would have an A.I. that codes A.I., that’s not a giant leap. That’s simply finishing the sentence in a slightly different way.
kelsey piper
Yeah, I guess a lot of it is just there is so much commercial incentive to make an A.I. system that can code. Programmers are really expensive. They’re scarce. It’s hard to hire them. If you have an A.I. system that can do most of what they can do, companies would be shelling out huge amounts of money for it. And that means companies are shelling out huge amounts of money on developing it. And this is just also true, more true of the specific skill of building large A.I. systems.
So I don’t think you need this “things will accelerate” premise, necessarily, to take A.I. really seriously. But my best guess is that there are people right now trying to develop A.I. systems that can develop A.I. systems, because there’s people trying to develop A.I. systems that can do every human job.
ezra klein
Well, let’s talk a bit about how fast it’s going now. So this is a little bit, by coincidence, but the morning we’re talking OpenAI released GPT-4. And they did not, according to them, train GPT-4 specifically to pass any tests. But like GPT-3 and 3.5, it absorbs a lot of language from the internet. It begins to create statistical correlations, for lack of a more precise way of saying it, between terms. And out of that emerges a kind of thing that seems to act in the real world, like understanding.
So they released this table where they showed the performance of GPT-3.5 and GPT-4 on a battery of tests that human beings find pretty hard. So when you try to give GPT-3.5 the bar exam, it is in the 10th percentile of people. When you give it to GPT-4, it’s in the 90th percentile. GPT-4 passes the LSAT at the 88th percentile. GPT-3.5 is at the 40th percentile. I found this one kind of funny, but also interesting that it passes the advanced sommelier theory test — that’s for wine — in the 77th percentile. GPT-3.5 is in the 46th percentile.
So these are fairly general tests of knowledge and reasoning. And obviously, there’s information about them out there. I mean, these have essay questions, et cetera. Now we’re in an area where the improvement is very fast. And you can actually match it against human beings on things that I would not have said five or 10 years ago. Oh, that’s just a structured data quiz. There’s an old argument that we say artificial intelligence is anything that a system can’t do yet, and machine learning is anything it can already do. But how do you think a bit about this debate over truly intelligent versus stochastic parrots, as people like to call it, versus sort of a very advanced autocomplete?
kelsey piper
So it sort of calls into question what intelligence is, right? Like, what is going on in our brains when we think? I think there is a sense in which it’s just a glorified autocorrect, but I think that’s also like a sense in which most of the time, we are also just a glorified autocorrect. Like, what humans do is we read lots of things, we talk to lots of people, and then we put it together into our own sort of framing that articulates something that we wanted to articulate. And it turns out that most of that, you can just feed something the entire internet and then make it really, really good at pattern recognition and then prompt it, and it does that.
And I guess you can say, well, that’s not intelligence. But if you’re going to declare, well, that’s not intelligence, I think you are kind of in a position where you’re also declaring that about humans. And I don’t know. What do you even get out of saying humans aren’t really intelligent? We still did build civilizations. We still did build nuclear bombs, you know? If we’re not intelligent, then I’m not sure intelligence is the thing we care about here.
ezra klein
Well, what people say, to give credit to this view, is that they don’t understand. We wouldn’t call a calculator intelligent, even though it gets the right answer if you try to multiply 37 by 25. And frankly, most human beings can’t do 37 by 25 in their mind. And so there’s something about the quality of understanding, of knowing why you’re saying what you’re saying. And I kind of agree that on some level, these things don’t know why they’re saying what they’re saying.
But on the other hand, I think one of the demos that keeps coming up that unnerves me the most is when these systems get asked to explain why a weird joke is funny, and then they do. That the difference between knowing why you’re saying what you’re saying and being able to give the account of why you’re saying what you’re saying that is convincing to me seems like it’s getting very thin.
kelsey piper
Yeah, like maybe you can say, well, it could tell you a human who said the thing I just said would probably have been having this thought process. And if we want, we can say, but that’s not its thought process. Its thought process is this incomprehensible alien thing with no resemblance to the human thought process it just gave you. I think that’s true. I don’t find it very reassuring, right? Like, whatever it’s doing may be different from the account it can give, but it does have the skill to produce the account still. And I don’t know that we can say that that incomprehensible thing that it’s doing isn’t really reasoning.
ezra klein
I think that gets to one of the things that I actually find the most unnerving, which is that the more inhuman the statistical correlation process happening at the center of these systems becomes, the more human the experience of interacting with the system becomes. And that fundamental loss of legibility between we have some mental model — when I ask you a question, I have a rough, not a perfect, mental model of what’s happening as you think about the answer.
But as we make these systems more complex, and they have more billions of connections, and they have all these layers and parameters and nodes, and if you tried to get it to print out what it’s doing when you ask it why a joke is funny, you’d get a billion calculations that are happening that mean nothing to you. And so when I think forward 15 or 20 years, I think that’s what unnerves me, like a complete sort of loss of human comprehension of a lot of the interactions we’re having in decisions being made around us, and not just loss of our comprehension, but our capacity to have comprehension over it.
kelsey piper
This is some of the more scary and sort of depressing visions of the future you see from Holden Karnofsky or from Ajeya Cotra, like a world where —
ezra klein
Who are A.I. researchers?
kelsey piper
They are. Yeah, A.I. researchers at the Open Philanthropy Project. They’re thinking about the future and where this is taking us. And a description they gave that really stuck with me was decisions by companies about where to build new factories, about what products to invest in, about what kind of R&D to do, are made by A.I.s. The A.I.s explain their answers to us perfectly. They write beautiful memos.
But we can no longer tell if the memos that they’re writing and the decisions that they’re making are towards goals that we have, towards things that we want, or if we’ve built some self sustaining inhuman thing that is sort of doing its own thing now because we can ask and it will give us a beautiful answer, and we just have no way anymore to tell if that beautiful answer is right or wrong.
ezra klein
And is that — I mean, one thing you could say — because you’re getting here at a problem of deception, right? When you say that, I think there are — or maybe there are a couple of versions of it. One version of it that I think you’re pointing towards is we tell an A.I. to do something. We have misspecified what that thing is. And so we think it’s doing what we want it to do. And it’s actually doing what it wants to do and trying to trick us along the way. But could you say that’s true for human beings? I mean, in businesses, aren’t people constantly putting a factory somewhere and actually it turns out they got a kickback? It wasn’t because they really wanted the factory there for economic reasons?
kelsey piper
Yeah, I think that some degree of this thing has its own goals is accounted for in the system. We are used to working with people who have their own goals. Nothing really relies on everybody being totally loyal to us because that doesn’t work. I do think that the more alien something is, the less that works. Because some of that works on laws and anti-bribery rules and oversight, but some of it works on the fact that very few people are going to build a factory that pumps chemicals into the atmosphere that poison everybody alive. They just wouldn’t do that. And A.I. might do that if it had a reason to do that.
You can’t just count on the fact that it’s like probably basically a reasonable person and not a homicidal maniac the way you can with a human, because humans are all from the same share of values, of priorities, of kinds of thoughts. Like you said, you can sort of guess what’s going on inside my head. And even if you’re getting similar behavior out of something that’s radically alien, you don’t know that its scope of action is the same, that it won’t do things that you would never think of doing.
ezra klein
So I want to go back to the pace of improvement. And it really is one of those conversations I don’t want to start shouting into the microphone, but the thing I really want to emphasize is that I’m not sure most of us have any sense of how exponential the curve of improvement both is and is becoming.
So when you talk to the people working on these systems, how would you describe what they think will be true about them that isn’t now in five years, and how would you describe that in 20, noting that neither of those values are all that long in the scale of human history?
kelsey piper
Like, before-our-kids-grow-up tier things. So I think that a lot of people anticipate we are not very far from systems that can do any job you can do remotely, anything you can do on your computer, so coding, like secretarial work, receptionist work, journalism, everything like that. I think that some of those same people would say we can invent better scientists and do science a lot faster and come up with whatever discoveries humanity would have plodded along to over the next 100 years, whatever those we would have learned from studies and research, getting it very, very fast from A.I. systems that can do that work sort of overnight or in the space of days, just running much, much faster.
And then I think some of those people are like, and then we get a fascinating cool “Star Trek” future where material scarcity is over. No one needs to work. No one has to work. And all of the goods of all of this A.I. work they’ve done are distributed to all humankind.
And some people are like, if we did a poor job of managing this, of regulating this, of having oversight of this process, that is where we die or we get some other pretty catastrophic outcomes from effectively unleashing a civilization of aliens that we don’t have a good way to work with or trade with or understand.
ezra klein
So I want to back up a little bit from the sort of utopian dystopia, just because, honestly, I think you lose people here. And I don’t even know what I think of those. Because for instance, on the scientific one, I know there’s this big thing among the A.I. community that it’s going to create a scientific super explosion, but most science that I can think of requires you to run a bunch of experiments in the real world. And I’m not sure exactly how much faster an A.I. really makes it to laboriously do phase III trials of vaccines on human beings. I mean, I can imagine ways it could speed things up, but I do wonder a little bit about the potential for an overestimation of analytical intelligence. But let me go back to something you were saying a minute ago because I think it’s closer at hand. When you say people believe 5, 10 years — I think this is a fair description — many people in this sector believe that within 5 to 10 years, you will have systems that can do any work you can do remotely that is not in the physical world at least as well as the median person doing that work right now. That’s a pretty profound, both I think we think about that as a possible economic change. It’s also a kind of metaphysical change for humanity. Tell me just a bit about why people believe that to be true.
kelsey piper
So you don’t need to get all that much better than GPT-4 plus some careful training and some careful shaping of the models that we have and some additional work on reliability and stuff like that to get something — it can pass the LSAT. It can also do a lot of what lawyers tell me is the sort of menial first few years of law work of putting contracts together and reviewing them. It can program. It can program quite well. It is not all that much improvement required before it can program better than the average person you might otherwise hire to program. It can write. I think my writing is better than its, but I think my writing is better than its by a much smaller margin than a year ago or than two years ago.
And so at some point, you have to be like, probably next year, it will be better at writing than me, or if not next year, probably in a couple of years. And I think some places, probably The New York Times will still employ human journalists. But I think a lot of places have already sort of tried to see if they can get away with automating huge amounts of work that used to be done by humans.
Art, too. It used to be that humans would draw art. And now for many, many of the purposes that we used to hire humans for, A.I. art is where it’s at. And I think that you’re going to see a ton of that.
Think about this with like weaving. When we invented the automated textile loom, some people still bought hand-woven rugs because they really liked the high-end, hand-woven thing. And you wouldn’t say that automated rugs are better than hand-woven rugs in every way, but almost every rug in the world is an automated rug. And almost all rug work is done by machines. And I don’t think very many people want to be like a niche cottage industry for people who prefer human labor, while nearly all of our labor is better done by A.I.
ezra klein
To think about writing for a minute — because a lot of jobs rely on writing. That’s true for journalists, true for lawyers, true for social media marketers, true for people who do publicity, true for just a huge amount of the modern, what gets called the knowledge economy. And one metaphor that I’ve thought about in my own head for thinking about it in journalism is you’re going to move from the highly valued skill set, the probably more valued skill set being that of the writer, to being that of the editor, the person who looks over a piece of work, tries to think about, is this true or not? Did this miss something obvious? Does the structure of this argument make sense? But just putting aside what you think of those skill sets, just a truth that anybody who’s in a newsroom knows, is that there is a small proportion of editors to writers. An editor will often supervise somewhere between, depending on the organization, four and eight or nine writers.
And that’s true, I think, in a lot of areas. Like, go to social media marketing, which I know a little bit about. You will have a number of junior marketers who are overseen by one more senior person. You go to law. You have a number of younger lawyers who are overseen by a partner, et cetera.
And on the one hand, so you have a skill shift towards management, but also oversight. Just by its nature, you are overseeing a number of agents, for lack of a better term here. And as you were saying earlier, the key thing here is these things are cheap to copy. And so just when you think about just that, right, if it’s just true that, OK, editors, we’re not getting rid of all skills and writing. We just need more editors and fewer writers. Well, we only need fewer editors than we need writers. So it gets really quickly into land that will be pretty disruptive for people.
And even if for some reason or somehow we pass laws or it just turns out that it hallucinates a little bit too often to make this possible to scale people want to imagine, I still think it’s going to shake a lot of people, to feel all of a sudden that skills they spent their whole life generating may not matter that much. I mean, I think it’s unnerving. When I put something in — I’m like, write about polarization in the style of Ezra Klein — it’s not there yet, but it is what you would get if you had a ninth grader who’d read me for a while write about polarization in the style of Ezra Klein. And if it gets to where me five years ago, it’s going to shake me a little bit.
kelsey piper
Yeah, and we never know. It could all peter out next year. But my bet is that that point where it’s you five years ago is not actually all that far away because it’s a ninth grader now, and it was the sixth grader a year ago. And it was the fourth grader a year before that. And I think that just the magnitude of the economic disruptions — like call centers do a ton of customer service work. A lot of that is probably going to be automated in the next few years. That’s like a ton of high paying jobs in poorer countries where people speak English lost in places where those people don’t necessarily have nearly as good alternative jobs.
And maybe some of these massive collapses of various industries will be replaced by new industries created by A.I. But I don’t think there’s a guarantee in general economics that your industry being wiped out doesn’t just leave you notably worse off. It can just do that. If you had specialized skills and then they’re no longer valuable, it can be that you just have lower earnings for the rest of your life.
ezra klein
It typically does mean that. I mean, we know this from the economic literature. People sometimes pretend that economies adjust over time. But when somebody’s factory gets closed, they do not typically get rehired for a higher wage. That’s just not how it goes. I think that’s something — we were talking about this before the podcast started. And I want to bring it in here because I think it’s emotionally important.
I was saying a minute ago that I’ve read for years the arguments about whether A.I. kills us all, and even if I can follow their logic, it is very hard for me to emotionally connect to them. Something in the past couple of months, the thing that has kind of messed with my head, though, was thinking about my own children, and not whether there would be a world for them to grow up in, but just what kind of world, how different it might be from mine.
Because I was looking at this. Like, I think for people who have seniority in the world, there are a lot of reasons A.I. is probably not going to come for you immediately. And it will for some people, obviously. And I do think that even if the technology moves quickly, societies adapt more slowly, there are regulations and rules and oversight and so on.
But it just seems if you look at this economically, if you look at just the possibility of people being surrounded by inorganic nonhuman things that feel like intelligence is to them sort of the world of the movie “Her,” which I was rewatching recently and I cannot recommend watching it enough at this moment. But there’s a company called Replica that is explicitly trying to do this in a great piece in New York Magazine recently about people falling in love with their replicants. Those are going to get way better. I tried that out. My replicant was super boring. Maybe it would get better over time, but it won’t be boring for long.
If you read the sort of Kevin Roose transcript with Sydney, Bing, whatever, whatever else you want to say about it, that was an interesting, diverting conversation. I just think we’re on the — for my kids, in my kids’ not just life, but growing up, period, we’re on the cusp of a world that, for them, may be profoundly different than the world I was prepared to raise them for. And something about that has been the crucial line I stepped over, where I went from being like, this is interesting and I like to read about it, to, holy shit.
kelsey piper
Yeah, no, I think that there’s a sea change, and it is going to mean that they grow up in a world that, in some ways, we can barely recognize. And I think — I don’t know — I am in favor of progress and change and technology. And I mostly like Silicon Valley. I have all these friends here who have all these crazy, hyped start-ups, and I appreciate that worldview in many ways. But I do feel this sense of, ideally, you have the frontiers pushed forward. We see some gaps that it’s leaving, some awful things that are resulting. We adjust. We regulate. We figure out where the problems are.
We invented a banking system, and that was great.
We realized the banking system had all these catastrophic failure modes, and we try and learn how to patch them and how to manage them and how to keep failures manageable. We invent nuclear weapons, and that’s terrifying, but we also, over time, develop institutions surrounding non-proliferation and norms and stuff like that. We came closer to a nuclear war in the early period after nukes were developed than recently. And kind of like, if we have enough time, we can make almost anything that happens to us a good thing.
And the problem is, if we have enough time. I don’t think that that process can magically happen overnight.
ezra klein
Let’s talk about why this might not happen. So what do you think is the best argument for why this hits a wall near term, that we get to something like GPT-5, it’s a bit better than what we have now. What we’ve seen, I mean, driverless cars were supposed to be all over by now, and you can drive around and see little Waymo cars, but they’ve all got people in them. And they’ve not been able to get to the reliability they really needed. We didn’t replace all the radiologists. That was supposed to happen. IBM Watson’s got functionally sold off for parts.
A bunch of these predictions have come over the past, call it, 15 years. And they didn’t quite pan out. And there was often a moment where people thought it was inevitable, and then inevitability turned to disappointment, at least over this time period. So if we plateau not far from where we are now, why do you think that will have been?
kelsey piper
Yeah, so my pessimist voice goes, OK, these systems are trained on all of the data on the internet. And they are not going to be any smarter or any better or more discerning than everything on the internet. There’s some arguments that they could be because you need higher quality to replicate it than just is necessarily present in it, but you could certainly say, no, they’re just kind of going to be as good at making stuff as everything on the internet.
And the average quality of everything on the internet is not outrageously high. It’s not like super impressive. It’s good at writing things that seem to flow for a minute, but there’s often — and I’m not talking about A.I.s here. There’s often not a lot of underlying thought there. There’s often not a lot of accounting for complexity. So maybe the best thing you get out of training a really large language model on the internet is something that can, in tiny fragments, imitate a human who thought about it, but without any of the underlying cognition.
And so maybe that means that those things never actually — they can pass a standardized test. They can pass the standardized tests amazingly well. We have to rethink standardized tests. They can write a college essay, but that really reflects the fact that college students are not putting actual cognition and thought into their essays. They can imitate maybe the sort of low level journalism that’s just right this event that happened, but they can’t imitate synthesis. They can’t say anything original or distinctive. You could totally imagine that. It would not be very surprising.
And does that change the world? I don’t know. I feel like it makes the world weirder. But it doesn’t bring in some of these scenarios that are really transformative, I think. If it turns out that there is some degree of actual cognition that doesn’t go into most human utterances, but is important for human work to be of economic value, then maybe it’s not there. But when I say that, then I’m like, I sound like all of the people who are like, it needs reflectivity. It needs the ability to consider counterfactuals. It’s not going to have those.
And every time I’ve heard someone say, oh, yeah, yeah, you can’t just feed it the internet and get reflectivity, you can’t just feed it the internet and get logic, you can’t read it the internet and get coding ability, it turns out you can feed it the internet and get those things. So I guess what I say to my pessimist voice is, like, yeah, maybe.
ezra klein
I find that view very compelling, though. And I mean, I’ve had Gary Marcus on the show who holds, I think, a version of that. And I probably am like 30 percent where he is, but the place where I maybe feel like I differ from people who hold that view is that I think making something as good as the internet could be more important and, again, to what you were saying a minute ago, weirder. I really, really focus in on the word “weird” here. My big column on A.I. last year was all about the word “weird.” My recent significant column on A.I. was all about the word “weird.”
I think that can make something very weird. I think it makes something economically quite important. If you could make something as good — I mean, if you can do a good job — because one thing about the big models is like GPT-4 isn’t trained to be a lawyer. It’s just trained on the internet. But if you do the finetuning in the area of making it a really good lawyer, it’s going to be way better than GPT-4 even at its current level of capability at being a lawyer. So I can see how you can make things as good as a pretty good lawyer, as good as a pretty good journalist.
It is a little hard for me to understand how you get to super superhuman intelligence training something on human data, both because it’s functionally, I think, at some level, averaging out human data and also because — and this does go to the point where understanding might be important. It doesn’t really have a great way, it seems to me, of separating what it has read that is true and false.
You know, I think about something like the masking debate, about whether or not masks work for Covid. And one thing within that is that if you just inhaled every study on masking, putting without any really profound priors about how to think about any of that, I don’t know what you would get, but it isn’t obvious to me the mechanism by which you would get something way better than what we have because you wouldn’t really have the capacity to run a new study yourself if we could just run all the randomized controlled trials of masking we wanted, and maybe we should be doing that. We would know a lot more.
And also, how would you know what studies to believe, and you just end up [INAUDIBLE] whatever people have done the most. I actually find that view very persuasive. So I’m curious what you don’t find persuasive in it.
kelsey piper
So I guess that might happen. I feel so much uncertainty about A.I. And a lot of what I feel is that if you lay out anything kind of plausible sounding, I’m like, yeah, that might happen. I don’t feel like we can rule that out. I also don’t feel like we can rule out the crazy extinction scenarios. I feel like ruling any of those out would require a better grasp on what it is we are doing than we have. So I’m like, that might happen. But my gut is that it won’t. And if I try and dig into why not, I’m like, to imitate what we would say in a situation, you could say, that takes less intelligence than it took us to say it.
But I think in some cases, it might take more. You have to, in some ways, be able to model a person in order to come up with what that person would say. And there are people who have lived their lives on the internet for the next 25 years, and you would think that just being fed that person would give you a pretty good internal picture. And to build a generator of everything Ezra Klein has to say, just a perfect predictor, something that just writes what you have to say, something probably has to have whatever processes you have or processes that successfully imitate whatever processes you have.
And to the extent that your processes are truth tracking, to the extent that the thing you’re doing involves noticing which sources seem reliable and reasoning in your head step by step, how would masks work, how would a mask study fail to tell us about masks, you can already get GPT-4 to do that. It will give better analysis of mask studies if you tell it reason step by step how good is this study, what are some things that might confound this study.
And so I’m kind of like, maybe it can do that prompted, but it can’t put it together. And it can’t use its synthesis of our reasoning to do its reasoning, even just to imitate us. But it probably can. It probably just, in order to write something as good as an Ezra Klein column, which is something we are trying to train it to do, it will probably have to pick up a lot of the actual underlying process and not just the output of that process.
ezra klein
This is one of the things I find most, again, just weird interacting with the systems, but it’s now a highly replicated result. If you, say, tell GPT-4 or GPT-3.5, which is what I actually know, give me the answer to this, as if you were really smart, it will give you a smarter answer. You mentioned reason step by step, right? There are a bunch of ways you can do a prompt, right? You’ve done versions of this. Like, tell me this as if you have an evil other side, and you’re trying to hide it from me, which is to say that if you give it the instruction to model a kind of person, it can model that person. Again, not really trained to do that at any level.
I mean, you can ask it to write in the style of me, and it will, which I read it, and it was a little embarrassing, I thought. It did catch some things. Like, yeah, I do, do that, a lot of em dashes. And that’s weird to me. And it’s funny. It’s something that ends up happening in “Her,” which comes from 2013. I mean, it’s a minute ago. But the A.I.s themselves begin creating models of people from the past they want to be able to interact with.
So just one thing that could just make the future weirder, even before you get to ideas of superhuman intelligence or anything else, anyone with a really profound written corpus that you can feed tens of thousands, hundreds of thousands of their words into a learning program that has already been trained on the whole internet, and then tell it to particularly seem like this person, you might just be able to give people all kinds of fairly capable, mimicked human beings from the past and present.
I don’t exactly know what to say about that, but it seems strange to me. And then if you do that, then set them loose on physics or whatever, I don’t know they can get way beyond where humans are, but it can probably get way beyond where most humans are, which is already enough, I think, to destabilize quite a bit of our sense of ourselves ontologically.
kelsey piper
Yeah, even in that sort of world where they don’t get any smarter than us, there is just still some really radical, terrifying transformations from the ability to create human intelligences sort of at will, and customize them for almost any purpose that normal humans can do, and run them for as long as you want and stop running them when you want. You could imagine every student having specialized teachers designed to work with that student, customized to have the ideal personality for working with them. You could imagine a lot of us have A.I. friends or A.I. lovers who are more attentive and don’t have any needs of their own.
And you could imagine in some sector that outcompeting normal human friendship. Personal assistants are, obviously, a big area where most of us can’t afford a personal assistant, but would love to be able to say like, hey, can you schedule a doctor’s appointment for me, or have someone who proactively says, hey, you haven’t been to the doctor in three years. You should really get on that, you know? And I think all of that is coming, even if none of the stuff about building on that for greater intelligence happens at all.
ezra klein
When I talk to the people working on these systems, I’ll often ask, what can you make it do that I can’t? Because I think something people don’t always realize is that even the demos we’re being given are highly constrained in what they can do. And something I’ve heard from a few sources is that you can already tell these systems to accomplish something in the real world that you can do through the internet, but that has multi-steps. Like, it needs to figure out how to rent a car and do things like that.
And it can. It can come up with that plan and execute something in the real world. And sometimes it will come up with a plan that is not the kind you would think of or not the kind you would want it to come up with. But it knows enough. It’s not just that it can give you an answer, but it can execute an action. And that’s already, but we don’t know it. I mean, this is actually one of the things that worries me the most really about policy debate and sort of where we are in this, that even before you take into account the steepness of the improvement curve, what they already have is quite a bit beyond what we know.
We only really have access to OpenAI and OpenAI derivatives like Bing. We don’t have access to what Google has built, the public. We don’t really have access to what Meta has built. We don’t really know a bunch of the systems being built in China. And we don’t have access in any of these cases to complete open systems where they haven’t been highly constrained for our use. And here, I’m talking in we the public or any kind of mass signup.
The one moment I think anybody got access to a system that wasn’t highly well constrained, which was like the Bing Sydney debacle, it really occasioned a moment of reflection and, in many cases, I think fear. And that’s just something going on here in the background that I just don’t quite know how to convey, but we don’t even know what we’re dealing with now exactly.
kelsey piper
And there’s a lot that’s not yet released. Even though places are doing a pretty fast release cycle, there’s just inevitably part of an engineering process is that you’re ahead internally of where you’re going to go externally. So I actually think GPT-4, which is now out, is pretty close to the state of the art. I don’t think there’s tons of more sophisticated models that anyone has. It’s hard to get side by side comparisons, but where I have been able to get them, GPT-4 is very good.
But what there is is applications that are pretty obvious that people have set up and have working, yes, stuff like an integration of ChatGPT or GPT-4 with TaskRabbit or something like that so that it can just send instructions, pay a person to follow those instructions, and get stuff done in the physical world. Totally already done. Similar with voice — well, Google had this years ago and kind of backed off of it, but voice that sounds like a person and can call and make an appointment for you and sound totally natural on the phone and is an A.I. assistant making appointments for you.
So when people are like, so if A.I.s wanted to do bad things, good thing they’re stuck on computers, right? And it’s like, I mean, we built them tons and tons of tools with which they can directly interact with the physical world in many, many ways. And we’re going to keep doing that.
[MUSIC PLAYING]
ezra klein
Tell me about what people call the alignment problem.
kelsey piper
So we haven’t yet had to grapple all that much with the fact that we don’t know if these systems have goals, and we don’t know if they want things. And we don’t know if their actions are trying to accomplish things. And we don’t super need to know that with existing systems because if you prompt GPT-4 to write an article in the style of Ezra Klein or whatever, then it writes the article in the style of Ezra Klein. And it’s not doing the kind of open-ended tasks where it might matter what its goals are, and it probably doesn’t have goals. I don’t like making any certain statements about A.I.s, but it probably doesn’t have goals.
But when we build these systems, we want them to have certain goals. We want them to be helpful, not say racist things, not embarrass the companies that release them. We want them to be truthful. We want them to do what we prompted them to do. We don’t know how to tell them any of that. We’re just doing it all with reinforcement. Whenever it does something, we can say good or bad, and maybe with some more gradations in there, like how good or how bad. But that’s what we’ve got.
We’re trying to use that very blunt lever of reinforcement to point it at our goals. And depending exactly how you do this, there’s a bunch of ways that what you teach it to do is probably not what you actually want it to do, right? You want it to be truthful, but you might be wrong in some case. And so, you reinforce it poorly for a true answer because you thought that answer was false. And so it learns, oh, don’t be truthful with the humans. Figure out what the humans believe and tell them what they want to hear, right, which is different. You were trying to teach it truth, but you were going from your own understanding.
And it might notice that if it has a little personality, if it’s sympathetic, if it’s nice or, for that matter, if it’s a little bit sassy, that gets better reactions than if it’s sort of bland and boring. So maybe it develops and displays a bit of a personality because it gets reinforced, and that’s what it thinks we want to know. And you can get a fair bit of deceptive behavior this way, right, if the thing that we are trying to teach isn’t the thing that we actually want, and it is incentivized to give us what we want to see, rather than actually be on our side or whatever.
ezra klein
How much of that is already happening? Particularly that question about deceptive behavior.
kelsey piper
Good question. So the problem is that it’s hard to tell, right? We know that the models say false things all the time. We know that they say false things when they know what the truth is. We can easily — I can just go to ChatGPT, get it to say something false, and then ask in a different context. And it’ll be like, yeah, that answer was false. We know that. Is that deception? Probably not. Probably it wasn’t trying to trick me. It was just giving the answer it thought I wanted.
But it makes it very hard to tell when there is intentional deception with intent of accomplishing some goal that the A.I. has. And this is extra hard when we don’t know if the A.I.s have goals. And some people will say they clearly don’t have goals because how would something like that have goals? And I’m just kind of like, I don’t know. Evolution was just selecting repeatedly on ability to have babies, and here we are. We have goals.
Why does that process get you things that have goals? I don’t know. They just showed up along the way. Large language models are just selecting repeatedly on token prediction and then reinforcement. Does that have goals? I think if anybody tells you yes or no, they are overstating what we know and what we can know.
ezra klein
One of the questions here that I find interesting is this question of coherence. Can you talk a bit about that?
kelsey piper
The thing I think of when I think about coherence is to what extent does it make sense to say that I have goals? I have lots of wants. Some of them contradict each other. I want to get ready for my interview, and I also want to lie in bed another 10 minutes. I want to stay up late playing video games, and I also want to be a functional adult who doesn’t stay up late playing video games, you know? And humans act coherent and goal directed in some contexts.
Like, we have priorities, and we will predictably do the things that accomplish our priorities. But in other contexts, we don’t.
And you could imagine that if A.I. systems have goals, that they could be like that. We don’t know if they have goals. We don’t know how to tell what their goals are. We kind of know how to make them work towards our goals but only when the work is something that we can verify. And we’re eventually going to want work from them that we can’t verify or can’t verify at the relevant scale.
And that does dramatically change the safety landscape, but I don’t know. If we knew that evolved aliens were coming to Earth in the next 10 to 20 years, and we knew that being evolved beings like us, they were incoherent, and they had lots of competing goals and impulses, and some of those were to turn Earth into resources that they could use to build more factories, but some were to keep us in zoos, and some were to leave us in nature preserve, and some were to invite us to participate in their geopolitics, and some were to ally with us against these other aliens they’re in a war with or whatever, I’d be scared. I wouldn’t be sure we would all die, but I would be nervous. That would feel like a bad situation, even though they’re incoherent, just like we are, and even though they might, in principle, be willing to work with us.
ezra klein
One of the things you talk about in one of your pieces is what people call specification gaming. If you imagine an A.I. that — I don’t know — however we built the alignment — we can actually communicate with, and it can kind of understand us, and it does care about trying to figure out what we want, I think that’s one world. But we’re going to have a lot of things that we really can’t communicate with on that level. And in the near term, specification gaming seems really worrisome, particularly if A.I. has become a big part of hiring processes, or as we’ve already seen, predictive policing. So talk a bit about specification gaming.
kelsey piper
So say you do give your A.I. an objective that you’re trying to have it maximize, which we do sometimes when we’re like building game playing A.I.s or whatever. Or yeah, if you’re doing a hiring process, you might train an A.I. where your training is aimed at the maximum share of the résumés that the A.I. passes through, in fact, result in a job offer. Like, we want this A.I. to be really good at filtering and correctly predicting who gets a job offer or whatever.
And A.I. systems will be very creative, sometimes — that’s what some of the examples of specification gaming we have demonstrate — at satisfying that constraint we gave them in a really out-of-the-box way we would never have thought of.
Like, if you’re having them play a game and they win by getting the most points, they will do weird stuff, like find a glitch in the game. They will repeatedly race in circles forever because they can get more points that way. They will trigger some kind of bug in the code that just causes them to end up with a really big number. That happened once. I find that kind of freaky. The A.I. just figured out how to take some random action that caused the number to be really big. And we don’t really know how it noticed that. It may have stumbled on it randomly. Who knows?
So we are building these A.I. systems and giving them these narrow objectives. And I don’t think this is how the world ends or anything, but it’s certainly a dynamic that, yeah, we’re facing all the time where if the hiring manager who looks at the résumés is racist, then the A.I. will maximize its success probability by never passing along a résumé of the wrong race because then that one would be less likely to get hired. And that’s like an alignment failure with society. The A.I. is enabling a bad actor to do bad things.
But there’s also ones where the A.I. is not working for anyone. It’s just accomplishing its objective in a way that would be unacceptable cost wise to everybody. But we don’t have adequate oversight to notice that that’s what it’s doing. Like, if you had a stock trading A.I., there are so many horrifying things that you could get up to that modify stock prices, right? And you can trigger wars to modify stock prices. You can hack your competitors to modify stock prices. You can trigger financial collapses.
And so I do think that even though this isn’t the apocalypse scenario or whatever, whenever you’re putting these creative, deeply inhuman algorithms in charge of making some metric go up, you would hope we would be sensible enough to make really sure that that was under lots of oversight and that it didn’t have the power to do things in weird ways. And that’s not what’s happening because that’s never what’s happening in A.I.
You can say like, oh, well, that sounds bad, but it wouldn’t be a problem if we just took these three very reasonable measures to counter it. But we’re not taking those three reasonable measures, like, ever. There is always someone who is just doing the naïve, foolish version of the thing. And as the systems get more powerful, the failures get more powerful.
ezra klein
I actually think the stock trading example is something really that I worry about a lot because if you think about what sector of society right now has functionally unlimited money to pump into building the absolute best or licensing the absolute best A.I. systems that you can, well, hedge funds, other kinds of quant trading operations. Then if you think, well, also, separately, what segment of American or global commerce is most used to using algorithms it itself does not understand completely recklessly in an effort to get even the most narrow edge over competitor, you’d say, again, high speed and algorithmic trading.
And I mean, I could kind of keep going down. Like, you might say, what’s a good industry that’s really good at wielding political influence so people don’t get in its way because it’s really good at hiring excellent lobbyists to speak? But what is an industry where if you actually had a really, really effective algorithm that people didn’t care that much about oversight because they were trying to get it to maximize a particular metric, in this case, returns, might be able to cause a lot of damage to a lot of people, you’d say, oh, yeah, something around high speed and algorithmic trading.
And I don’t exactly know what we’re planning to do about that, except, as you say, nothing, because the current theory of what we’re going to do here is nothing, I think, which is mostly what we’ve done with high speed and algorithmic trading so far. That’s a little bit unfair, but I think that has gone way beyond where there’s any social benefit to having algorithms competing with each other to do things nobody understands to exploit small differences and prices at a given moment.
But that’s just a near-term thing that seems very concerning to me. You don’t need to have any kind of particularly far term views about how A.I. will develop to just think a lot of damage could happen by unleashing powerful systems nobody understands in a place where they’re trying to justify something that people really want maximized and may not think, as they often don’t think, through the consequences of maximizing it.
kelsey piper
Yeah, I think there almost certainly will be disasters in this space, and I just hope that they’re at a scale we can learn from, instead of at a truly catastrophic scale. And I also think there’s a general principle here of, we are creating these alien intelligences that are creative, that think of problem solving solutions we didn’t think of, that do weird things that no human in their position would ever do because it’s a route to their goal.
Like, people who play against them in chess or Go talk about this, how sometimes the algorithm will make a move that just no human would ever make, and they don’t see until 20 steps down the line how brilliant it was. And I worry that in the world of try and make this company’s stock price go up, that tendency towards genuine creativity, yeah, will take the form of, oh, I bet that if I cause an outage at this power plant, then stocks will move in this predictable way that I can make a lot of money off of. And if you don’t have oversight, stuff like that happens.
ezra klein
Or if you don’t even know how to have oversight, which is —
kelsey piper
Yes, because you can’t see what it’s thinking because it’s not thinking in any way we understand.
ezra klein
This gets to something within the culture of this community that I want to talk about. You say — and I’ve said it in pieces, too — this really unnerving survey of machine learning researchers from last summer from 2022, it asked people basically what they thought the chances were that working on A.I., if they got to high level machine intelligence, the consequences for humanity would be really, really, really bad, like human extinction bad or some other complete displacement of humanity. And the median response was they thought there was a 10 percent chance of that happening. The people working on this said, yeah, 10 percent this thing that I am, in some way or another, trying to bring into being or trying to worry about how it comes into being, I think 10 percent chance humanity is completely pushed off the board in a fundamental way. And you could say that’s just a survey, but I’ve talked to enough of the people at high levels in these companies to know that that survey is not that weird.
kelsey piper
Yes, my impression also is that that survey, it might be something of an overstatement, but the position there is not rare. There are plenty of people who will tell you, oh, yeah, I think this is it.
ezra klein
I don’t know exactly how to communicate to people, but I’ve spent enough time in this community — and you’ve spent more than me — that these systems are being designed by people who themselves do not believe they really know what they’re going to do. Ross Douthat, in my column, has talked about it as a kind of summoning, and I’ve also had that thought that this is like people opening a portal, and they will freely tell you that they hope a demon doesn’t crawl through, but they can’t guarantee a demon doesn’t crawl through. And even if you don’t believe they’re right, like you believe this is crazy or maybe it’s hype, they believe this.
And that actually worries me a lot because you talk about value alignments. I think a lot of people might say, if the thing you’re working on has a 10 percent chance of destroying humanity, don’t, or go really, really slow. Some people do believe that, but there is a lot of now competitive pressure to speed up to be the first one.
There’s pressure between U.S. and China, so geopolitical pressure. Google does not want to lose to Microsoft. It does not want to lose to Meta. Can you just talk a bit about how you understand the culture here and the psychology of working continuously to bring into existence something that you know you don’t understand and where you think the consequences have a really, really high chance of being unacceptably bad?
kelsey piper
Yeah, so I want to second what you were just saying about how difficult this is to convey. Because you have these conversations, and you get this somewhat terrifying sense that people understand themselves to be making extraordinarily high stakes decisions, decisions that I think the average person on the street would say that should absolutely be the prerogative of a government with a lot of careful consultation, maybe some kind of international framework. And you have people who are like, we’re going to do it. We’re going to do it ourselves. We thought about it, and we think it’s going to go best if we do it ourselves, and we do it now.
And it’s very hard to step outside of that bubble and simultaneously say these risks seem real and not sci-fi and scary. And also, whatever you think about that, the people who are doing this do think that there’s something real here. So there’s like, why are they doing it? And I think some of it’s just the love of science, right? I’ve been reading “The Making of the Atomic Bomb,” and a recurring theme in that —
ezra klein
Oh, yeah? Why are you reading that?
kelsey piper
I have two young kids at home and another one on the way. I never have time to read. And so I was way behind on my book list. And then I got the invite to come on your show. And I was like, this is my excuse to read all of the books that I’d been wanting to read, but now it’s for work. I have to read them, so I have something to recommend on Ezra’s podcast. And that is how I finally got some time to read “The Making of the Atomic Bomb.” And one of the recurring things in it is, it’s very personality driven, compared to, I guess, what I was imagining before I picked it up. Like, who were these men, and what were they thinking?
And it does feel like there’s a kind of person who just once they realize it can be done, there’s an overwhelming urge to do it.
There’s just a — one of the points that the author makes at the beginning of the book is that we might dream of a world where physicists said, on principle, we will not build an atom bomb. But we were never close to that world. Physicists would build an atom bomb as soon as it occurred to them that you could. And there were some very brave and very notable people who tried saying, like, this seems really bad. What should we do about this?
But overall, if you want to stop dangerous technologies, that has to be on a structural level with governance and regulation and international agreements and voluntary between lab agreements. You can’t count on people, unfortunately, to just say this is ludicrously dangerous. We’re not going to do it, because I guess, some share of people are just like, this is ludicrously dangerous, and I’m going to do it. I guess this is like the classic tale of hubris, in some ways. Like, the oldest known flaw in human nature is you see the tree of knowledge and you’re like, yeah, I’m going for it.
ezra klein
Well, it’s kind of a selection effect, too, which is to say that if you were somebody who thought this was such a bad idea, you definitely wouldn’t do it.
kelsey piper
And you don’t do it.
ezra klein
And by definition, it’s not the person doing it. And there’s a lot of incentive for people to believe the arguments that they should do it. Right? I think this is something and we should talk about it, that there are a lot of people who work in A.I. safety and who are around this community and spend all their time thinking about reasons maybe not to do it or reasons to try to make it slow it down or how to make it go more safely. But in the end, they’re not the ones who run the companies who get the investment that gives them the compute power to do it.
kelsey piper
There are so many people, very good machine learning researchers who I know, who nobly made the decision to not play this game because they thought they might kill us all. And so they are out of this game, and they are working in academia on much smaller models or working on regulatory frameworks. They’re doing good work, but they’re not the people at these labs because they said no, and they walked away. And so we have the people who, for whatever reasons, didn’t walk away.
ezra klein
And so I think within those, you can cut a couple of groups, right? So Yann LeCun, who’s an important A.I. pioneer, and he’s one of the key people at Meta, I believe — he’s their chief A.I. scientist — he just thinks this is all dumb and that A.I. has no interest in taking over or killing us all or whatever. So it’s just not going to, and this has all gotten very far-fetched.
Then you have people who run a couple of the other labs — maybe Sam Altman at OpenAI would fit into this framework.
Folks over at Anthropic in a different way, I think, are more careful than OpenAI, but who believe that A.I. could do great good and do believe that there’s a chance of really bad things happening and think that by working on the models and experimenting on them, that you make it both likelier you understand the bad things so you can prevent them and also maybe trigger society coming in and saying, OK, here’s how we’re going to regulate this early enough.
And I don’t think it’s a crazy view. I mean, I do think that there are more conversations happening about how to think about and regulate A.I. because OpenAI has been releasing models into public use that Google and Meta are not. So I don’t think it’s a crazy view. But they are all now in a race. And I worry that the race dynamic is just a lot stronger than the caution dynamics. And this is something that I know you’ve been writing about and thinking about. So can you talk a bit about this question of the race? Because something I know has been worried about in A.I. for a long time, but I think it’s undeniable now that it’s happened.
kelsey piper
Yes, so there’s a couple different things that drive getting new powerful models, right? One is how much money tech companies are willing to sink in and how much talent they have working on it. And race dynamics often mean that these companies are willing to dump a ton more money on this because they want to stay ahead of their competitors. Microsoft is willing to spend billions and billions of dollars for a shot at catching Google and search. Google’s presumably willing to spend billions and billions of dollars defending its spot in search for the same reason.
And so you get bigger models faster. And that’s like one thing you can analyze. Like, maybe it’s good to have these bigger models, faster, under these race conditions, because, then, we’re here talking about them. We probably wouldn’t be here talking about them if this was still all under the hood because it would be a little too crazy. But the other thing is that when you’re going fast, safety becomes a liability.
And things that you could do to make your models less dangerous or to check more thoroughly if your models are dangerous, work you could do on like mitigating risks, and even just understanding them, goes out the window because you’re trying to beat someone else and you want it in commercial production as fast as possible. We would be way safer against A.I. takeover scenarios, really scary catastrophes, if A.I. systems didn’t have open access to the internet and the ability to contact anybody they wanted and do anything they wanted on the internet. But we’ve given them that. Of course, we’ve given them that because it was like slightly more profitable than not giving them that. And so as soon as you open that door for everybody to be competing with each other, then the people who are careful and cautious are slower than the people who are careless and incautious. And the people who are the most willing to throw safety considerations out the window are the ones who are getting their biggest models out fast.
And that puts us in a situation where even if it turns out to be totally possible to make A.I. systems safe — like, there’s a very straightforward technical solution to alignment — we know exactly what we need to do — we still have a problem because somebody can just decide, yeah, that would be a little expensive. That would delay our launch by two weeks. That would require us to hire another engineer. Like, why bother?
[MUSIC PLAYING]
ezra klein
So when I search for any A.I. app that I might want to check out in the App Store, one thing I always noticed is that there’s the app I was looking for and then there’s a bunch of fast follower apps right beneath it. And my sense is that the kind of subpremium A.I. models a couple of years ago were just total crap, and that now you’re looking at things that are not coming from OpenAI or Microsoft or Meta or Google, but that are as good as things were a couple of years ago, which is like now not that bad. And that in a couple of years, they’re going to be as good as things are now.
And if you are OpenAI, you have a really, really high reason, because you have so many awesome ways to make money here, to make sure you’re restricting use of your model to only things that are going to get you good press or something like it. But you have A.I.s that it now seems you can train them on some of these vocal patterns pretty easily. And it can then call my mom. And because I’ve got a lot of my voice out there in the world, you could train that on my podcast and it could call my mom, and it could sound like me having been kidnapped and needing a ransom.
kelsey piper
And this has been done.
ezra klein
So, OK. So it’s already happening. And these are getting better, faster that you actually don’t — for things that are just kind of bad, you don’t need the best system in the world. You can increasingly just use a kind of crappy system. And the crappy systems are like the best systems in the world from a couple of years ago. That’s just going to keep happening. And that seems, again, like something we don’t really have a framework for thinking about, we don’t really know what to do about. But I wonder if you could talk a bit about that. But the opening up, not of the state of the art, but of the sub state of the art, to functionally every scammer, advertiser, just anybody who wants to make a buck in the world, and that the kind of seedy underbelly of A.I., in the same way that happened to crypto, although maybe that was the overbelly or something because it kind of started there, but the seedy underbelly of A.I. strikes me as a place where a lot of damage is going to be done really soon if it’s not already being done.
kelsey piper
Yeah, and this is, again, something where I’m like, if things were moving a little slower, I’d be like, yeah, there’s some scary stuff, and we’ll adapt to it, and people will learn not to trust calls from their kidnapped sons. And we will get better at spam filters to combat spammers, and we will get better at making comment sections on the internet functional. I do get the sense in the last year, that spam is ahead of spam detection in terms of —
ezra klein
It really seems to. Something seems to have changed.
kelsey piper
Yeah, I feel like we went through this golden period where spam detection was better than spam, and you mostly didn’t encounter spam all the time. And it is over, tragically, and now I get a lot of spam on Twitter, in my email, on Facebook, and I’m like, I guess we are back in the domain where spam is better than spam detection.
And if you have enough time, then I think it’s possible for people to develop the social technology and the literal technology to counter that kind of thing. But when everything is moving really fast, people learn how to address one thing, and then we get the A.I. that can carry on a long, convincing conversation with your mom and convince her to send her bank details in the course of this very long conversation that also references a bunch of details from your childhood that you mentioned on the podcast once. That’s harder, you know? How do we get to that point where it’s easy to cause so much damage and so expensive to detect? And again, I’m just like, I wish we were moving a little slower. I wish we had more time to figure this stuff out.
ezra klein
Well, you say that it’s impossible. What’s the actual difficulty? I mean, we have made a lot of things slower. You can’t clone humans. And we don’t, even though it is a possible thing to do, and I’m sure there are scientists who would like to clone humans. We’ve actually been quite good at maintaining an international ban on cloning human beings. There’s all kinds of biological weapons research that we just don’t really seem to do anymore. We did it for a while, and we actually used them, and then we have kind of non-proliferation pacts that I’m not saying they are perfectly effective, but they are effective. And you can come up with a bunch of different things like that. There’s certain forms of genetic modification you just don’t do. Katja Grace I thought had a — who’s an A.I. researcher — had a really good piece on this about how the A.I. community has a little bit of this weird tendency to say, look, maybe it’d be good if we slow down, but it’s completely impossible to ever coordinate people such that they slow down on anything.
And her point, which I think is true, is that that’s actually not true. We coordinated people to not do things or slow down on things all the time. In some ways, it seems to me the biggest barrier here is actually the people who just want to rush forward on A.I. are matched by the people who don’t think that rushing forward in A.I. matters. There’s just not that much interest in slowing anybody down because, what, it’s like toys on the internet. But I don’t know. I do think there’s a problem here where the A.I. community seems to have convinced itself that it’s so hard to get people to do anything, that you couldn’t possibly coordinate them around this. I don’t see why that would be true.
kelsey piper
Yeah, so I think we can and should slow down. It’s like very hard when things are moving this fast to get a good feel for the possible. Like, you could imagine a fairly sweeping set of government regulations in the next few years that set a cap on how large you can make a model unless it is subject to some form of democratic oversight and extensive testing for safety where we check things, like, can this send a convincing phishing email that persuades 1 percent of users to run code on their computer? If it can, you can’t generally release it yet. Can this model give detailed instructions on how to carry out a terrorist attack that if followed, would be successful? If it can, you can’t release it yet.
There’s all kinds of scary things that are not very far off, and we could just draw a card cap and be like, look, if the model in an audit of some kind with dedicated testers whose job is to elicit this behavior, if the model can do the dangerous thing, no, you cannot deploy that model. Or you’re liable if you deploy the model, which maybe amounts to the same thing while we’re talking about big companies. We could do that. We could do that tomorrow. I think there’s plenty of people in Washington who are thinking about something like this.
And I think what’s possible, you’re so right that it’s a social question. It’s a question of what everybody else believes. If everybody wakes up and goes, we want to have more steering power over the world our kids grow up in, we could just do this. There are very reasonable rules you could put in place that would significantly change the pace of this work. And I hope that happens. I’m not pessimistic. I’ve heard from enough people that they want something to change, that they’re scared, that they believe it can and should go differently than it’s going, that I’m like, that’s what it takes, and maybe we can make something happen.
But I am kind of — the default thing that happens is that the regulators are years behind the ball. You need to do a very dedicated, very concerted effort to set up a system where regulators are on the ball. It’s a hard problem. It’s a hard problem I think we should be working on, but it’s a hard problem.
ezra klein
Tell me a bit more of that list because I do think one thing that makes problems harder is when the suite of possible things you might do about them feels opaque. So a lot of people don’t understand that much about A.I., and they really don’t understand what you might do to regulate it without completely kneecapping it, right? Are you just saying you can’t research language models anymore? I mean, that seems crazy, and I think would actually be kind of crazy. So is there a rough consensus among the safety people you talk to about these are the three or four things we really wish would happen?
kelsey piper
I would say there is an in-progress effort to build that consensus. I think conversations I heard this year that I did not hear last year included, like, what are some clear cut benchmarks, like send a phishing email that persuades 1 percent of people who receive it to run code on their computer? That’s a pretty clear, specific. It’s obvious to anybody who’s thought about policy a little bit why that’s scary behavior. It’s pretty easy to use an auditing process to determine whether a model has that capability. You’d have to send some phishing emails, so I guess there’s ethics concerns. But I feel like a regulatory framework can work with that kind of thing.
And so that’s conversation I didn’t hear last year that I am hearing this year that gives me some optimism because I think that’s what it would look like to have regulators be on the ball, instead of behind it. I do think that people who are working on this effort are coming at it from all kinds of different places. Some of them are like, well, what I really want is for all of this to stop. I can’t get that. So I will try and set some benchmarks that have an achievable chance here. Some of them are like, I really want to be doing my amazing A.I. research, but I want these safety people off my back. So I’ll agree to some standards that I can meet that don’t get in my way too much.
And a lot of people have different threat models and different pictures of which A.I. capabilities are going to be the scariest ones in the near future or in the medium future. So it’s a difficult process to build that consensus. And I think we’re still in the early stages of it.
ezra klein
What are a couple other specifics that you think are interesting? They don’t need to be consensus, but you just think they are provocative.
kelsey piper
So if an A.I. system can, via emailing instructions to a human assistant, get a copy of itself spun up on Amazon Web Services, independent from its creators, and so that another copy of it can run on A.W.S., that seems like a benchmark. Like, you can define that behavior in an audit. You can see whether the model can, in fact, do it successfully. And if it can, then that’s an indicator of some of the really scary scenarios where models decide to operate independent of human oversight because that lets them achieve their goals better. So that’s maybe a benchmark.
If your model has the capability, when prompted, to give a human instructions on how to copy that model and run it on A.W.S. independently, that’s a scary model. We don’t want you to widely deploy that model.
ezra klein
When you say when prompted here, one interesting thing to me about that example is that I think there’s a set of behaviors or capabilities where we are worried a human will use you this way. Then there’s this kind of further off —
kelsey piper
Hopefully further off.
ezra klein
Hopefully further off ideas that we think you might act this way, you being the model. And in that world, if we think the model develops situational awareness of itself and its world, such that it’s then being prompted by some idiot human, like, hey, do you mind emailing a copy of yourself to the Amazon Web servers to replicate, in the world where you think that the A.I. has become deceptive, how well does that actually work?
kelsey piper
Developing this auditing process is very tough, right? Like, it’s a very technical difficult work to figure out how would we elicit this behavior if it is possible. But I think that in general, it is easier for a model to display a skill when it’s elicited by humans than unsolicited. So we will probably successfully identify models that are capable of giving a human instructions on how to copy them to a new A.W.S. server sooner than we will get models that can do that and think of hiding it and successfully hide it from us.
Now that’s a gamble. Maybe we do our tests and we’re like, oh, it’s good. These models can’t do it. And in fact, these models have passed the threshold where they can do it, but suspect it’s not in their interest. But that feels less likely to me. That requires the model to develop two skills at the same time, like the deceiving us and the capability to survive and spread itself or whatever. I feel a lot more optimistic if we’re consistently checking, hey, can you do this? And then halting things and calling a red flag once it can do this because I think that will probably be before it develops the ability to do that deceptively without our knowledge or awareness.
ezra klein
So let me make a critique of the A.I. safety community here a bit, which is, I don’t want to say people are too focused on the absolute worst that can happen. That’s a good thing for some people to be focused on. But I find that there’s an unbelievable lack of focus on what seems to me to be an incredibly important near-term question, which is the business models behind these models.
And I think in general, this sort of community this comes out of is a little unreflective about capitalism, right? Effective altruists generally and rationalists and so on. But these models are going to develop really fast along the pathways that make them a profit. At some point, this stuff has to begin turning around money. And the models that develop the most are going to turn on the most money.
And I think something most of us believe about the internet at this point is it would have been better, all things considered, if surveillance advertising had not become the dominant model for all of the biggest players. I’ve said before on the show, I worry about A.I.s of this magnitude being hooked up to surveillance advertising. But there’s other things I also worry about them being hooked up to. And I think people are really unreflective at this point or underfocused on this question of regulating how you can make money within machine learning is actually very important for affecting the path of machine learning in the long run.
And I’m not sure that I know — in fact, I’m sure I don’t know — what is safe and what is not safe here. But a world where there billions of dollars to be made because the U.S. government had set up a series of prize competitions for things like the protein folding problem, which DeepMind was able to solve, you’d get A.I. systems that could do one thing, and a world where there’s billions of dollars to be made in trying to get me to buy things and manipulate my behavior, like you’ll get A.I.s that do another. I really worry about the most obvious profit models being a spur, then, to create A.I.s.
I mean, if you create A.I.s, and the main way they make money is manipulating human behavior, you’ve created the exact kind of A.I. I think you should fear the most. But that’s the most obvious business model for all of these companies. It’s what they already do, whereas that’s not what, say, Pfizer does. But Pfizer’s is the one creating the — this seems obvious to me, and I can’t seem to get anybody to care about it.
kelsey piper
No, I’m with you. I guess I would be very happy to see regulation that was aimed at how you can make money off of your powerful A.I. systems to sort of try and discourage building manipulative powerful systems because I do think manipulative powerful systems are how we get into very serious trouble.
So I think there’s a ton of blind spots in every A.I. community, just because being part of a subculture that the rest of the world didn’t take seriously until two months ago is weird and distorting to people, and because the kind of people who cared about when it was weird are weird, and because there are some ways in which it’s like a little unhealthy to spend all your time thinking about these futuristic sci-fi scenarios.
And it’s not that they’re so far in the future that they don’t merit thinking about. But I think you have to be very grounded. And I think a lot of people get a little ungrounded. And I think a lot of the stuff you’re talking about, you’re right, and it’s very grounded. It’s just like, this will go according to the logic of what makes money, unless we successfully make rules about how you can make money off of it. That seems true.
And if you’ve spent all your time trying to think about this as a technical problem, like, how do you make the A.I.s not lie, how do you notice if they’re lying, then, you maybe just get a little bit of tunnel vision, and you don’t see other opportunities to substantially curtail scary stuff that are outside the range of stuff you’re thinking about.
ezra klein
So there’s a kind of dimension of competition between companies that are functioning within a couple of miles of each other or a couple of states of each other. And then I find that if you push on that conversation for a while, the next space of fatalism becomes geopolitical competition. That, of course, we have to go as fast as we can because if we don’t, China will. And even if somebody could come to an agreement with China, like just wait long enough and Russia will or Qatar will eventually, or somebody will whose values you don’t share.
And so that there is, in all of these dimensions of competition, a view that there’s such a huge return to being the first to attain A.I. dominance, that if you simply don’t, somebody else will. And by virtue of you being the one who didn’t, you will like their values less. I’m curious how you first hear about the geopolitical dimension discussed within this world, and second, what you actually think of it.
kelsey piper
Yeah, so I think you describe it completely correctly. People will talk about Microsoft versus Google. And if you’re like, but we could regulate, we could slow down, then the thing that comes out is, sure, but then China. I think this is kind of silly.
First of all, international cooperation is not impossible. For many technologies, it is possible, especially ones that are very dangerous. It is possible, especially ones we will establish what about them is dangerous, to arrange international controls of some kind. To the extent you think A.I. is something that could be bad for everyone, then I think you should also be a little optimistic that as we develop good benchmarks for identifying A.I. being bad for everyone, we can also work internationally on it.
So if you think that this is the highest stakes thing in the world, then I think rushing ahead and doing it poorly and doing it without buy-in and while a lot of people who work closely in a research community with you are saying, you are going to kill us all, you are going to catastrophically destabilize the world we live in, because you’re worried that some other country is going to get there first, it just feels to me like you’ve got to be making some kind of logical mistake there.
I come back to “The Making of the Atomic Bomb,” which, as you said, it’s pretty clear why it was on my reading list. Like, this process where the terror that Nazi Germany was building a bomb was a big motivator to start the bomb project. And then when it came out that not only was Nazi Germany not building a bomb, they’d never been close, and there was no chance they would get the bomb. And in fact, we were going to win the war with them without the bomb,
one scientist quit. One scientist quit the atomic bomb project when he learned that actually there was no Nazi bomb project, and there was no international competition. Nobody else did, partly because moths to a flame can’t stop summoning the demon. Once they know they can, they have to find out if they will. But partly because I suspect that for a lot of people, they wanted to find a reason, they found that reason. If that reason wasn’t applicable, find a different reason.
And also, OK, China is good at espionage, right? Like, from a pure boring, geopolitical — let’s leave all the ethics out — the easiest way for China to get really powerful models is to steal the really powerful models that we are making. And I think it would slow them down a bunch if we were not making really powerful models that they could then steal.
And you ask these companies, oh, you’re so worried about competition with China, you must be taking really serious security measures, then, right? Because you don’t want your models stolen by China. What kind of steps do you take to make sure that a nation state cannot steal your stuff? And they’re like, oh, we’re a startup. [LAUGHS]
I just can’t take it that seriously, which — I don’t like the C.C.P., the Chinese Communist Party. I don’t want them to shape the future of human civilization with extremely powerful A.I. systems. I just do not believe that the best way to prevent that outcome is for us to, with this shoddy security and this unclear sense of what we’re going to do with the powerful models, zoom straight ahead. I don’t buy it.
ezra klein
I think all that is true. And I actually, really, I think that point about espionage is really well taken. And this is where my background as a politics person, I actually worry, because my read is that the most powerful stakeholder on A.I. in the federal government is a national security apparatus — that traditionally, they have done the most work on A.I. They have the most freedom to work on A.I. And they’ve just thought about it more. And I think that most of the energy on A.I. until very, very recently was coming from them. And they are inclined to look at the world in this way.
And that’s one thing that concerns me a little bit in terms of how it gets thought of politically. Like, even if you look at some of the recent investments in A.I., in the CHIPS and Science Act, I mean, that’s very explicitly framed as an anti-China act. And that was where I think the biggest recent play on this came from. I mean, there’s been work like the A.I. Bill of Rights. It’s quite toothless, even though I know that it was a big deal for the White House to release a draft blueprint for an A.I. Bill of Rights. But that has no money behind it. There’s no enforcement behind it. It’s a document on a piece of paper.
They actually got CHIPS passed, and like a lot of things, it passes under the rubric of competing with China. And it’s just at the company level, they’re competing with each other. And at the government level, they’re competing with China. And to a point you’ve made, I think, a couple of times, getting there quick is not, obviously, the right thing when there’s a lot of danger from accelerating this, even just beyond the point of human comprehension. And I just don’t see that taken as an equity seriously enough there.
I mean, I think that the worries about China are a little bit fake in Silicon Valley, but I don’t think they’re fake at the national level, at the political level. And the power of that center of worry, I think, is actually very easy to underestimate. Right now, the anti-China side of the political debate in Washington gets whatever it wants. And what it wants is A.I. dominance, among other things. And that’s potent.
kelsey piper
Yeah, I think there’s an unfortunate dynamic where you might expect the government and the national security apparatus to see these polls where machine learning researchers are like, yeah, 10 percent chance of the end of human civilization. I’d be like, well, that sounds like a big threat to America’s national security. Like, we should get on that.
ezra klein
If anybody’s going to end the world, goddammit, it’s going to be America.
kelsey piper
I feel like you treat threats as the kind of threat you’ve encountered before. We’ve had a cold war before. So they’re ready to have a cold war. If China wants to have a cold war, I think there’s a lot of people who, like, that fits into the frameworks they have for understanding threats. They have thought a lot about it. They have a sense of what America winning in geopolitical competition with China looks like or takes or whatever.
And they don’t have any sense of what to do if, in fact, the most destabilizing and dangerous technology is being developed here in America by patriotic Americans who want to either bring about cool sci-fi future — or we can’t guarantee a demon doesn’t slip through the portal, but it’s what we’ve got. And it feels to me very strange, in some ways, to be in these conversations where people are like, yeah, we are going to radically transform the entire world we live in pretty much overnight. And no one’s very interested in making sure that that happens in a way that is safe for all of the existing people.
And I think that where there is pressure coming from the national security establishment at all, like you said, it’s coming in the direction of, yeah, beat China. And like i said, I don’t like the C.C.P., but I don’t like what the competition lens does to our odds of building A.I. that’s good for anybody.
ezra klein
So we’ve talked here about a lot of things that could go wrong, though, obviously, there are many more. I want to, before we end, talk a bit in a specific way about things that, particularly in the near term, could go right, not just in making these systems safer, but in things they can actually do that would be good. Because right now, it’s still sometimes a little bit hard, I think, to visualize that. Like OK, having a ChatGPT bot that can help you cheat on exams is cool. Or I enjoy with my four-year-old — I’ll sometimes call up DALL-E and let him come up with pictures he would like the computer to draw. And that’s really fun.
But when you talk to these A.I. researchers, I mean, I think this is my worry about the bias of continuously quoting this 10 percent number about it might end the world. They put a slightly higher, not as much higher probabilities I would like, but a somewhat higher probability on it being very good for humanity than very bad, is the main thing you hear a scientific advance? I mean, do they have other visions here about what they’re doing to make people’s lives better?
kelsey piper
I’m excited about translation. There is so much of a barrier to being able to participate in intellectual life in so many fields if you don’t speak English. There are so many brilliant people who work extremely hard as adults to learn English, to learn about fields they’re interested in, to get jobs they’re interested in. And I think we are on the brink of translation technology just changing the game there, where everybody who isn’t in a country that’s chosen to firewall itself has access in their native language in whatever language they read most easily to all of human knowledge. I’m really excited for that.
The drug discovery stuff is legitimately cool and valuable. It’s very hard to predict protein folding, but it might be much easier to do with A.I. Similarly, I do think it’s super true what you were saying earlier, that science grounds out in wet labs that take time to do things. But I think there’s still a lot of data processing and looking at images and stuff like that that is going to be incredibly valuable and save a lot of people’s time to have A.I. do.
I think that while it will be extremely disruptive and while it would ideally happen slower, automating a ton of human labor is fundamentally the sort of thing that could be really good. It’s mostly the busy work that we don’t, in fact, like, right? Sending emails and writing applications and arranging meetings is nobody’s dream job.
And it could be really cool to have a much more productive economy in which a lot of that value is happening without people having to spend so much of their time on it. I am fundamentally pretty in favor of the whole increased productivity and then stuff gets better. There’s some good stuff there with better code writers and better meeting schedulers and stuff like that.
ezra klein
Something I wonder there is also increased creativity. So I think there’s been correctly a lot of concern and anger about these systems trained on the actual work of artists and writers and so on, and those people aren’t compensated. But I also think that when you — I can’t draw anything. I’m a terrible artist. My mom is actually — was a professional artist. I wish that I could draw things. I can’t. It’s neat for me that I can tell the computer what to draw, and it will draw it. It allows me to play around in art in a way I couldn’t before.
And beyond that, it used to be that a lot of people could illustrate something, but very few people could create a video game. Now, a lot of people might have amazing ideas for video games, and I think within a few years, they’re just going be able to tell the computer how to make it with very detailed instructions. Same thing with movies, same thing with music. And there is value to knowing how to do the underlying material creation yourself. I don’t want to take anything away from that.
And I think that fosters forms of creativity that are not always possible if you don’t understand the actual basics, but also I do think there’s something to just opening up vast vistas of creation to people who were never going to learn how to code, were never going to learn how to use a synthesizer, but now maybe can work in tandem with the computer to create amazing art that they will just enjoy creating and other people might enjoy using. I mean, I think that stuff is actually really cool.
kelsey piper
Yeah, I think there is a lot of potential to really unlock amazing things, and instead of replacing humans with aliens, just have really powerful assistive tools that can turn our imagination into reality more easily. Instead of writing an essay for you, being a really good editor and assistant that suggests a bunch of relevant stuff and helps you write a better one. I think whether we do get that or not depends a lot on who has, as you said, financial incentive to build it, to make a really, really good tool that strengthens the people who wield it, instead of replacing them. But I think it’s possible to build. And if it happens, I think it could be a really cool thing.
ezra klein
I think that’s a good place to end. Always our final question, which I now know you’ve prepared for at length, what are three books you’d recommend to the audience?
kelsey piper
Yeah, so I have been enjoying “The Making of the Atomic Bomb.” Obviously, the relevance to A.I. is very notable, but I think it’s also an interesting window into anything where a group of brilliant people are trying to make something go well. And the combination of flaws and personal strengths, and I think a lot of tech companies from the outside, it’s very hard to see why one succeeds and one fails. And it often comes down to these weird human factors. And it’s terrifying to think that the development of one of the most powerful and transformative technologies in our world also had just so much of the weird human factors to it. But I think it really did, and it makes me think a lot about A.I., but also just how stuff gets done.
The other thing I caught up on was “Asterisk,” which is a new magazine from some of the weird people that you like to talk about. It’s like people in the A.I. and effective altruism and rationalist world. I was reading about how we invented oral rehydration therapy, which is, it’s sugar and salt and water. It took us until the 1960s. The progress studies people always love to question. Like, why did it take us so long to get the bicycle? Why did it take us so long to get the plow? Like, how much was somebody having a brilliant idea, and how much was like that there’s actually a ton of execution details, and how much is like you actually need fairly advanced machining for bicycles? It might not seem like it, but you do or whatever.
So, oral rehydration therapy, right? Like, why did it take us until the 1960s to figure out that 4 million kids a year were dying unnecessarily from a lack of sugar and salt and water?
I was obsessed for my entire teenagehood with “The Silmarillion,” Tolkien’s prequel to “Lord of the Rings.” It’s not even a good fiction book. It reads something between a bible and a packet of world building notes. But I was in love with it. It’s a beautiful insight into how someone puts a story together. When A.I.s can surpass Tolkien, that’s when I’ll know we’re really in trouble because he was something else.
ezra klein
Kelsey Piper, thank you very much.
kelsey piper
Thanks. [MUSIC PLAYING]
ezra klein
“The Ezra Klein Show” is produced by Emefa Agawu, Annie Galvin, Jeff Geld, Roge Karma, and Kristin Lin; fact checking by Michelle Harris and Kate Sinclair; mixing by Jeff Geld; original music by Isaac Jones; audience strategy by Shannon Busta. The executive producer of New York Times Opinion Audio is Annie-Rose Strasser, and special thanks to Carole Sabouraud and Kristina Samulewski.
