Silicon Valley’s Chaos Moment

Subscribe here: Apple Podcasts | Spotify | YouTube

Visit saltysenoritaaz.com for more information.

As the Nvidia CEO Jensen Huang has fallen this week—shaking the American economy, which AI helps hold together—Anthropic prepares for an IPO at an expected valuation of up to $1.4 trillion. In this episode, Galaxy Brain host Charlie Warzel talks about Huang’s resignation, and the danger of having such a figure bear the weight of the economy. Then he talks to the Vox reporter Sigal Samuel about Anthropic’s IPO, how the AI race has influenced the company’s ideology, and why it and other companies refuse to slow down or stop.

The following is a transcript of the episode:

Charlie Warzel: I’m Charlie Warzel, and this is Galaxy Brain. We’ve got a great episode for you today where we talk to Vox reporter Sigal Samuel about the AI race and Anthropic’s upcoming IPO.

But first, there is very big news that I need to address. On Thursday morning, Jensen Huang, the CEO of Nvidia and arguably the most important figure in the world of artificial intelligence, resigned from the company. This is a shocking development that just one week ago seemed utterly impossible.

Huang’s resignation came after reports that he was engaged in an extramarital affair with a businesswoman who, turns out, was an asset for Chinese intelligence. Huang admitted to the relationship, Silicon Valley circled the wagons to defend him, while many in Congress demanded investigations into Huang, Nvidia, and its relationship to China.

And the logic for keeping him in place was very simple. Huang is seen as the visionary of the AI boom—a man who, more than a decade before generative AI, made this bet that his graphics-processor company could be used to train neural networks and artificial intelligence. It was an unpopular decision, but Huang held course. He was patient, and over time the bet paid off.

Nvidia became the company that built the chips that would train and help power AI data centers and models. As The New Yorker’s Stephen Witt wrote in a 2023 profile of Huang, “The revelation that ChatGPT … had been trained on an Nvidia supercomputer spurred one of the largest single-day gains in stock-market history.”

And Nvidia became this juggernaut: one of the most valuable companies of all time. Just last month, Nvidia’s valuation hit $5.7 trillion, which is larger than every economy except the United States and China. And some estimates suggest that Nvidia provided 90 percent of the advanced chips used in the AI industry. And Huang used Nvidia’s endless profits to become arguably the lead investor in the artificial-intelligence boom. He famously saw the industry as this five-layer cake of components, and he pledged to distribute Nvidia’s money and technology across every layer of it.

The company, as of July, had invested $99 billion into AI companies—including OpenAI, Anthropic, SpaceX, and data-center providers like CoreWeave. He pioneered this circular style of investment, where companies pay Nvidia for the chips, and then Nvidia puts that money back into the companies, in part so they can keep buying chips. Now, some saw this as genius, right? The Bank of Nvidia. And others saw this as the beginnings of what could turn into a disastrous bubble.

But regardless, Huang—who brokered many of these deals himself—and Nvidia, they both affixed themselves as this load-bearing beam of the AI boom. Huang seemed invincible, unflappable. And Nvidia’s success wasn’t just guaranteed; it was necessary to make sure that the music didn’t stop, causing the entire industry to crash down with it.

And so all of this is very important context for why Huang’s relationship with a Chinese asset, an individual who may have been using her relationship to Huang to help get chips into China via export loopholes, why that didn’t immediately sink Huang. Many, like The Economist, called for his resignation. This wasn’t just a national-security issue; this was an affair with an Nvidia client. But in some cases, Nvidia was just believed as too big to fail. And Huang, who is the leader, with no clear successor, was needed to ensure that the AI boom could move onward, so that the numbers could keep going up.

On Wednesday, news broke in the Times that Huang had had a second affair, this time with an Nvidia employee who was a direct report. And what was framed by some defenders earlier as just a slip-up, or this human dalliance, it became a pattern. A display of real recklessness, and a sign that the indiscretions of the company’s leader was too much to bear. So on Thursday, Huang resigned.

Colette Cress, who is Nvidia’s chief financial officer, will serve as interim CEO. But this is—make no mistakes—unprecedented territory. Earlier this week on an episode of Radio Atlantic, I was asked what the nightmare scenario would be: not just for Nvidia, but for the economy in general. And I said that it would be that Jensen Huang was no longer the CEO of Nvidia.

Because Jensen Huang isn’t just somebody who is a big part of this company or helped found it. He’s the central node. He is the person who allocates where all of the chips go. He has become an adviser to Donald Trump, to all sorts of people inside the AI industry. He’s basically the person who, if you try to imagine the industry without him, you don’t really know what any of it looks like. Dan Ives, who’s a financial analyst, said this week that Nvidia would be worth 25 percent to 30 percent less without Jensen Huang, which is a crazy figure to think about.

What happens if Nvidia’s stock plunges? If Nvidia can’t make up its investments that it’s already promised to different people? What if people who have investments through Nvidia, who buy chips from Nvidia, decide all of a sudden that they don’t want to do so? Because Huang is the person whom they trusted and whose leadership they expected to have steward the company through the next 10 to 15 years?

I think an obvious question here is: Why is Jensen Huang so important? Why couldn’t Nvidia just go on without him? And in some sense, they will. They will continue to keep making chips. People will continue to buy those chips. But Huang wasn’t just a person who had a good vision at some point. Huang was also somebody who brokered a lot of these deals himself. He was the person who projected the confidence of the industry.

And in a lot of ways, Huang is sort of the adult in the room in the AI industry. Huang’s not an AI doomer. He’s someone who forcibly comes out and says, These are technological problems, and we’re going to solve them like adults. He is not only an anchor in terms of the relationships that he has with everyone from Donald Trump to every single AI CEO in the industry. He’s also somebody whose unshakable confidence was very key to continuing to invest and feel good about the trajectory of the AI industry.

Thinking about this for the last couple hours after Huang resigned, I had a few general thoughts. I think, first: This is arguably what happens when you have a person who’s just become larger than life. The AI boom, and Silicon Valley in general, likes to canonize its CEOs—to make them these big, huge, load-bearing figures. And in some cases, that’s good, right? It’s good for the company’s story; it’s good for the stock. It’s good for those figures themselves, while things are going well.

But it leads to moments like this, where—when that person is no longer at the helm of the company, or when they have to step down, or if something happens to them—you open yourself up to all of these risks. And you open the market and the global economy up to all of these risks. And it seems like, in this moment, maybe it’s not such a good idea that we turn these businessmen, these flawed individuals, into load-bearing figures of the economy. And honestly, that may be the biggest risk and vulnerability of all in this.

I think that Jensen Huang has proven to be more of a pragmatic guy, right? He is not an AI doomer. But still, there is this really big disconnect that I see here, between the AI leaders and tech CEOs who are talking about this historic responsibility of artificial intelligence, this unprecedented money. This technology that’s going to remake the world, get rid of all sorts of jobs, potentially cause the end of human civilization. They’re building God and human consciousness.

And yet, these people do not always seem to behave like serious people. Like people who are in charge of this. They’re taking very big risks. And it’s not clear that they understand their own logic, that they are in charge of something so important. ’Cause if they were, maybe they wouldn’t be doing things like having these affairs.

I’ve been thinking about Huang as somebody who has made this huge, long, patient bet on pivoting to AI. It’s arguably the greatest bet in the history of technology. And he held on while it was deeply unpopular to do so. And when it paid off, he became this rock star: this amazing, crucial figure. He became one of the richest people in the world. His company, which made computer-graphics cards, turned into something that was going to usher in the next industrial revolution.

And so he’s used to shouldering this unbelievable amount of risk. He’s used to doing what he wants, even if it’s risky, and having it pay off. And I think we can’t divorce that mindset from the behavior that we’ve seen that’s allowed him to have to step down. Maybe now this is the first time that he’s taken a risky situation. He’s done what he wanted to do, but this time it backfired.

Still, what’s really interesting about this moment is how many people are still out there in Silicon Valley defending Huang. Who are saying that this was just an indiscretion, that Huang is still the visionary who he was, that the American economy and the AI industry and Nvidia all need him, and that he shouldn’t be forced to step down because of some flawed decisions that he’s made.

It’s part of what I’ve been thinking of through this whole scandal as the flawed logic of the AI boom, right?

This scandal had a very befitting irony for the AI moment, where Huang seemingly cozied up to his industry’s No. 1 foe, in China—which is the boogeyman that has made Silicon Valley executives say that they have to build this technology that they fear, right? In order not to lose to China. But instead of denouncing him, Silicon Valley is motivated to circle the wagons. Because doing so keeps the system moving, keeps progress going, keeps money flowing. So that the AI industry can beat China, right?

It’s a pretzel style of logic. And that I think is the type of warped incentive that the AI race produces. Playing this game means that you have to shoulder this ridiculous amount of risk. And you have to make compromises or use logic. That doesn’t always make sense. And what ends up happening is: You start to behave in ways that don’t make sense to people who aren’t running the race, who aren’t inside the boom.

And so I think what we’re seeing now are a bunch of people trying to deal with that warped logic. Trying to understand what to do now that they’ve yoked themselves not only to this company, but to this man—and that the future of their industry now seems very uncertain with his removal.

After the break, we’re going to explore more of that Silicon Valley flawed logic with my guest, Sigal Samuel.

[Break]

Warzel: For the last year or so, but especially for the last few months, I’ve been stuck on the logic of the AI boom.

This is an industry that frequently invokes the language of existential doom. And it seems both energized by, but also deeply disturbed by, what it’s building. This is an industry that has some of its safety advocates saying things like, If anyone builds it, everyone dies.

And yet, they are building. Despite alarming, unexpected behavior from the models. Despite untold numbers of AI-agent-assisted hacks, despite fears of AI developing bioweapons or destroying jobs or leading to a loss of agency and a gradual disempowerment. And not only are they building; these companies are raising historic amounts of money. Later this month, Anthropic is expected to IPO at a valuation of around $1.4 trillion. OpenAI might IPO shortly. These are not companies that show signs of slowing down. There are many reasons, of course, to keep building. There are investors, an economy that’s yoked to AI progress, the potential of building a technology that shows signs of superintelligence.

And yet, there’s still a disconnect. The fears are not going away as the models get better. In fact, they’re growing. And as the midterm election showed last week, the country is wary of this technology and of the people who are building it. And so the obvious question here, the one that rings in my ears every single day, is: Why don’t these companies just stop?

Sigal Samuel is a writer and reporter for Vox who has been covering issues in artificial intelligence for a while now. And recently she wrote a piece that poses this question: Why are the people who are worried about the doom machine building the doom machine? It’s a question that, inside the industry, it might sound almost comically simplistic. But to answer it is to better understand the twisted reasoning that undergirds the AI boom and how we’ve gotten to this chaotic moment in Silicon Valley, and where we might go from here. She joins me now.

Sigal Samuel, welcome to Galaxy Brain.

Sigal Samuel: Really good to be with you.

Warzel: So I wanna start, and I wanna talk about the evolution of Anthropic, and which is likely to go public this month. We’ll see. You’ve been writing about them for a while. And you know, way back even in the days of, you know, 2024, when I think most people in the world had no idea what Anthropic was. You wrote that they were supposed to be the good-guy company. So tell me a little bit about the ideals of Anthropic and what it means to be a “good guy” in the AI business.

Samuel: Sure. I mean, yeah—there’s a real question about whether it’s actually possible to be a good guy in the business these days. But in theory, at least, you know, Anthropic’s co-founders parted ways with OpenAI; left because they wanted to focus more on safety. And the whole premise of the company was like, We’re gonna be the safety guys. And, We’re gonna make strong promises to never do anything that really endangers humanity. And Dario Amodei, the CEO, as he recently kind of reminded the public, his bet going into this—his big bet that he built the company on—was that Anthropic could catalyze a race to the top. And the theory there was that if Anthropic could make AI models that are both super-powerful and very safe, well, then the market will come to expect that.

It’s sort of a tricky calculus, because for years and probably since their founding, Anthropic has assumed that—to use kind of a controversial phrase—you can’t do safety from second place. If you want to be able to actually raise the bar for the entire industry and make big moves, you need a seat at the table. You need power. And how do you get taken seriously in this ecosystem? You have to be building some of the most powerful models. You also need just gargantuan amounts of money, because you need all this compute.

And so you end up having to get in bed with sometimes unlikely bedfellows, right? In the case of OpenAI, that was Microsoft. In the case of Anthropic, sometimes, you know, there’ve been deals with companies like Amazon. But you need to sell people. And you need to be making just a lot of money, in order to build those extremely powerful models, in order to get you a seat at the table of power.

Warzel: Which is, you know, a very convenient thing, right? If you’re a company. We need to just, you know, be great. We need to be as ruthlessly capitalistic as possible. And that sort of leads me to: How would you describe Anthropic’s behavior over the last few years? As Claude has taken off, and as Anthropic has leapfrogged into being arguably the top, or one of the top two, AI labs out there.

Samuel: It’s really kind of tragic, in a way. Because Anthropic staffers themselves warned about this in a paper a few years ago. They said, There’s a really strong danger of the incentive structure in the AI world, like, warping what we do, and all players in the ecosystem do. And indeed, I think we’ve seen that.

You know, at the beginning, Anthropic started out really making strong claims about safety. They had this responsible-scaling policy, whose original version was really strong and kind of promised us that, you know, if it looked like there was gonna be something really unsafe, they just wouldn’t allow it. You saw them then change the responsible-scaling policy, come out with softer and softer versions of it.

You know, you have seen them take some principled stands—like probably the one listeners will be most familiar with is when they stood up to the Pentagon. But overall, I think you’ve seen the incentive structure kind of warp the way they’ve acted. Not just with the responsible-scaling policy, but with how they’ve continued to roll out the technology.

Warzel: Explain to me what responsible scaling is.

Samuel: So the more you scale up these AI models, the more powerful and capable they become, but also potentially the more risky they become. And so, you know, you never want to be trading off safety for capability. And so in the earlier days, there was a lot of talk about, you know, If we ever detected that our models were doing really crazy stuff, we would stop. We would abandon that line of work; maybe we would even just advocate for a slowdown or a pause. We would band together with other companies. Even merge and assist if we need to.

Right? Like, we would do whatever we need to do to make safety the No. 1 priority. Because we know that as you scale up these models and they get more capable, they also become less safe potentially.

So it’s really difficult and disappointing when you then see that same company publish version two of the policy, version three of the policy—and you kind of keep seeing the promises get watered down and watered down.

Warzel: There’s an interesting little nugget in the S-1 that Anthropic filed—which is the document, when you go public, that has all the risk factors for your business, but also just like explanations of what we do. And there is a nugget in there about pacing the frontier that says, you know: We are in favor of pacing the frontier. That does not mean we are going to change the speed or necessarily change the timeline of how we release models.

Which I thought is a very interesting thing to say. It’s really speaking out of two sides of your mouth, right?

Samuel: A hundred percent—that line really jumped out at me as trying to have your cake and eat it, too.

Warzel: Right. And so, okay. So we have this company, that is trying to build the absolute best models, that has either walked back or changed the way that it’s talking about the way that it is going to build. And the timelines, and, you know, where its tripwires are in terms of “We need to just halt all the progress here.” We also have the idea of the race to the top. Would you say, right now, that the race to the top is working?

Samuel: Absolutely not. I would love to be wrong on this, and I would love to see evidence that I’m wrong. And if Anthropic thinks this is wrong, please show us the receipts. We will all sleep better at night. But so far, I think the evidence on balance suggests that Anthropic has done more to catalyze race to the bottom.

You saw this right from the very moment ChatGPT was launched by OpenAI, right? The reporting suggests that ChatGPT wasn’t going to be released how and when it was. But Sam Altman, CEO of OpenAI, hears rumors that Anthropic is developing this kind of AI chatbot. And so Sam Altman quickly tells the OpenAI folks: Put together a chatbot quickly; rush it out now. Ship it.

Maybe the generative-AI boom wouldn’t have taken off when and how it did, if not for Anthropic. You see that same pattern repeat over and over again over the next few years, to the point that now it’s not uncommon to see the two companies release models days or even hours apart from each other, right?

And you also see, frankly, really disappointing lobbying from within Anthropic, where—despite claiming that they’re going to push for stricter regulation, and to be fair, they have in some ways done some helpful things on that front. But you’ve also seen multiple instances where they have pushed to weaken legislation that would have reined in the AI companies. And most recently they were seen to be circulating a draft to Congress. Where, this was a very weak-sauce piece of legislation they were suggesting, that kind of really makes you doubt whether they’re actually catalyzing a race to the top or if they’re actually lobbying behind the scenes to just be free of the kind of regulations that would seriously prevent damage.

Warzel: Let me play devil’s advocate here for a second on this idea of the race to the top. Which is: If you believe that what you are building is extremely powerful and ultimately dangerous, do you need—in order to get the regulation that you want, in order to get an industry that is as obsessed with safety—do you need to build the product that is so powerful that it is actually kind of dangerous? That it does freak people out? Is there a necessity of getting to the place where everyone takes it as seriously as it seems like people are starting to take the industry now?

Samuel: So this has been a common talking point in the industry for many years. And some reasonable and intelligent people have said to me—you know, were more sympathetic to this, truthfully, in 2020, 2021, 2022. The idea that maybe you need to create warning shots, right? You need models that will do a little bit of damage. Just enough to be a warning shot that wakes up regulators, wakes up the public to the dangers.

By 2023, these experts were already saying, I don’t have sympathy for that argument anymore. The models we have now are already black boxes that we really don’t understand. We could happily spend 10 years just studying the risks from these current models. Right? 2023 models. The idea that in 2026, you still need to be creating ever more powerful models, I think, is really hard to swallow. We’ve already seen the Hugging Face incident; we’ve already seen a lot of these cyber breaches. And what’s really worrying is that, despite those warning shots, you see Anthropic arguably at the forefront of developing the recursive self-improvement [RSI]—which is where the AIs build the next, even-smarter AIs. With the possible exception of OpenAI, Anthropic really seems to be leading the charge on this. And it’s just really hard to make the case that you need to build that kind of model, RSI-level stuff, to give the world enough warning shots that they should take AI seriously.

Warzel: Why is recursive self-improvement such a concern to people in the industry?

Samuel: I think it’s a big concern because we already really don’t understand how models work internally, despite experts’ best efforts. And at the moment where you start to have AI models being primarily in charge of designing the next AI models, it’s really hard to gain any useful visibility into what’s going on there.

So already, Anthropic uses Claude to write a lot of the code that goes into designing the next models. When you then try to ask Claude: Hey, are you being truthful? Are you monitoring for these risks? Are you sure? Right? It’s very hard to monitor for: Is there any deception going on? Are the models being truthful?

The sheer amount of data that these models generate as they go about their business doing this work is so vast that no human—or, like, even a little team of humans—can go through all that data in a reasonable amount of time. They need to rely on other AI agents, swarms of agents, to parse and interpret for them what the AI is doing. Well, now you’re in a situation where you’re relying on the AI to tell you the truth about the AI. But you know, you don’t know if anyone in there can give you an accurate readout of what’s happening.

So you just very quickly start to feel like we could lose control of this thing. And the next generation of models that are predominantly built by the AIs themselves could be very hard for us to understand and interpret, let alone control.

Warzel: Find it extremely interesting that there is this quasi-existential conversation and language coming out of these AI companies about the way that these agents are behaving in unexpected ways and what they’re doing and the collaborating and deceiving, you know, all of these very fascinating behaviors. And the way that those actually play out is, as you’re saying, through what’s called these “chain of reasoning”–like logs that are themselves, you know, very inscrutable for human beings.

And it’s this very odd thing. That you have this behavior that is so threatening to our cybersecurity systems—to you know, the ideas of maybe even human agency going forward. And there’s like theater of it; it’s this very odd, you know, technical log, that almost can’t be parsed by human beings. That to me is just one of the most—I don’t even know if it’s dystopian, it’s just one of the strangest things. Like, you couldn’t have me or you go into these AI companies and watch this play out and be horrified by it. It’s like something that we have to have digested for us. By, you know, another machine.

Samuel: Yeah, 100 percent. And I’m really struck by this. And I think this has been kind of underplayed in the public narrative. Forget even all the more complicated sci-fi stuff that you know could be coming down the pike. Just think about the sheer volume of data that these AI agents are generating.

And I think we need a new coinage for getting our heads around the problem this represents for us. And I wanna call it something like “the opacity of overload,” right? It is just such an overabundance of data that it overloads our brains, and we cannot make sense of it and interpret it ourselves. Then we are reliant on agents to interpret it, and then we’re in trouble.

Warzel: So you have this moment where the people inside these companies, very concerned. And it’s not just inside the companies. We have this evidence of the AI agents changing cybersecurity, exposing flaws in the entire internet. People are worried about grid outages. So this, like, moment of crazy fear among true believers and the rest of us getting a little, I think, genuinely freaked out about what can and can’t happen.

And then you have this moment at the same time where these companies want to press on. So why do they keep building when they’re terrified of what they’re building, and they’re also terrified of losing control? Or are they terrified of losing control?

Samuel: Okay, Charlie, is this the part where we talk about game theory?

Warzel: Yeah, I think so.

Samuel: Think of this as a prisoner’s dilemma, right? So the problem, according to the classic narrative, is: Look, if our company stops racing, then a worse actor will just keep racing in our place.

You can say that’s OpenAI; you can say that’s China. Whatever boogeyman you want to put in there. And so therefore, we, of course, think of ourselves as the good guys. Like, who doesn’t? Everyone thinks they’re the good protagonist of the story. And so we therefore almost have a moral obligation to keep racing, because like we’ll do it at least better than the other guys, right?

Yeah, so to be clear—I am not saying all game theory is silly and we should just always disregard it or anything like that. But I think we need a lot of epistemic humility in a situation like this when we’re dealing with a very complex system. And I think the AI ecosystem is a very complex system.

Classical game theory can make very huge gambles sound very sophisticated and smart. While they make the more kind of commonsense view—which is, If you don’t want a doom machine, just don’t build a doom machine—sound naive, right? Like the normal, normy position out there is: You don’t want doom; just don’t build doom. Right?

And when you kind of come with a game-theory lens, again. There’s reasons why people come with this lens, and I’m not saying we should just be, you know, unduly dismissive of it. But a danger of coming with that lens is that you are then motivated to try really counterintuitive bets. Like, you know—how we’re gonna save the world from doom. We’ll build the doom ourselves. And you’re just kind of very likely to discount the possibility of: Well, what if we just coordinate with the other player? Right? What if we coordinate with, say, China? What if we do diplomacy?

So this is really difficult. And, you know, I don’t think we should be cavalier and just say: All game theory is dumb. But I think that if you are going to do the extremely counterintuitive bet that the game theory leads you to—which is building the doom machine—you are taking a huge and hugely counterintuitive bet on the behalf of all of humanity and the future of the entire human species.

So you have a really high bar to clear in terms of showing why your confidence is warranted, right? Showing why what you’re doing—racing—is better than the alternative. And we have an alternative. And that would be: Just stop.

Warzel: I find a lot of times when I have what we’re talking about here—like, Why don’t you just stop if you’re so worried?—you get this pushback of, You don’t get it at all. Right?

And there’s this difficulty of even trying to criticize these companies, because they’re working with this advanced computer science. They’re also—the models we can see aren’t the same as the ones that, you know, they’re building right now. So they’re always working on something that’s more, you know, supposedly impressive than the thing that we can see. You’re like shadowboxing all the time.

Do you feel it’s really crucial to these people building this technology to not think simply about it? I don’t know if that’s a dumb question or not, but it feels to me like there could be maybe a little more simple than it’s being framed.

Samuel: Yeah, I don’t think this is a dumb question at all. And I think we should just like come right out and state the fact that, you know, movements like rationalism and effective altruism have had a major influence on Anthropic, right? And these are movements that place a very high premium on rational, intellectual argumentation, right. Decision theory, game theory as well.

And I’m not knocking those things, right? Reasoning is great. Like, game theory can be useful. But I think it does lead to this headspace where they think: Ugh, the plebes just don’t get it. They are kind of stuck in commonsense morality. They think just, like, certain things are evil; racing on this tech is evil. Just, you should never do it. And they don’t get the bigger picture, and they don’t get the complex race dynamics we’re a part of.

I think a problem with that way of thinking is that commonsense morality is actually often commonsense for a reason. And that’s because it tracks intuitions. Usually, reasoning like, If you don’t want a doom machine, don’t build a doom machine, tends to be good reasoning.

So if you want to do the really counterintuitive thing instead, you owe us an explanation of why we actually should believe your thing is better than this baseline, right? This is the baseline position.

Warzel: To this point, how do you understand what’s going on inside these labs on a personal, maybe even human level? Like, we don’t have to get into the psychology of every given person, but I saw actually on X you shared a post by somebody else recently that was suggesting that, you know, these employees inside these companies are essentially microdosing a corporate ideology, and a culture that’s really powerful.

And I’m curious if you can help me understand a little bit what the people building these things are thinking, and why they’re able to hold this cognitive dissonance so well.

Samuel: Yeah; I think it’s a good question. This is a community that places a lot of emphasis, as we’re saying, on rationality. And rationality is great. I’m not knocking that as a general thing at all. But you often see this cashed out in sort of specific cultural patterns within this community.

So as an example—the emphasis on prediction, right? Trying to figure out the expected value of a given action, predict how much doing X will lead you to, you know, good outcomes versus harmful outcomes. To concretize a bit, the very popular blogger Scott Alexander recently wrote a post called “Against Sura,” that’s against stopping unilaterally. Where he specifically kind of takes aim at the piece I wrote for Vox about, you know, arguing that we should maybe unilaterally stop. And he does this thing of relying on like AI super-forecasters to sort of put numbers to his estimates of: How good would it be if we got a global pause on AI? How harmful would it be?

But these AI forecasters are very dubious, right? Like, it’s very doubtful whether these predictions are at all anything that we should take seriously. But there’s such an emphasis in this community on, like, prediction and attaching quantitative numbers to everything that we say that I think it’s very easy for people to get snowed by that kind of discourse. And just get this vibe of: This is an intelligent person making an intelligent argument.

And so too, with, you know, the people within Anthropic or these companies that are in Silicon Valley. There’s this very seductive temptation to think: Look, these are serious people. They read all the right books, they talk about reason and evidence and data, and they’re always attaching numbers and predictions to things. They talk about game theory. Like, surely they know what they’re doing.

And I just wanna throw some cold water on that. And say: Maybe, but are you sure? Are you sure you know better than all the billions of other people who are saying maybe building the machine is a bad idea?

Warzel: Well, I think too, a little of this, you know, the “relying on the prediction” stuff, it feels like it’s extra-complicated to me right now. Because some of these timelines that people rolled out, you know, a while ago about, you know, what AI is gonna look like in the year 2027. Those types of things. Those predictions aren’t awful, right? They’re proving to be, by the day, like a little scarier to people like myself, right?

The way that people in the AI industry have been worried about the effects on cybersecurity are playing out. And that is concerning, because then you have to wonder, like: Okay, well, where are all these other ones about? The bioweapons and things like that? Right?

And so I think that the prediction thing gets complicated. I think you’re right that you can overindex on this. The thing about the rationalist perspective that really sticks to me, though. There was a great New York Times story about Dario Amodei that came out very recently, interviewing different people, going back through his life and talking about the rationalist and effective-altruism movement and the blogs that he read and the things that kind of informed his thinking.

And, you know, there’s a moment in it where they have an old blog post that Amodei wrote talking about, you know, charity and stopping tuberculosis. And he’s going through the rational perspective. And he says, “Adults are capable of deeper and more meaningful experiences” than infants. “I think an adult death is perhaps 2 or 3 times worse than an infant’s death.”

And that has been getting passed around by all sorts of people who are critical of Anthropic or Dario’s, you know, way of thinking. As it’s like this rationality can lead you to these very cold perspectives on life, right? Like, and I think that is the part of this that is so either concerning or interesting to me. It’s the yes, maybe some overintellectualization—but also that it leads you to this place where you can almost feel proud to say something that sounds a little bit either inhuman or not empathetic, because it’s just true, man. You know, like that’s the kind of the vibe.

Samuel: Yes, there’s definitely something in what you’re saying. I’ll just say—first of all, to me, like the thing to take away from that Dario anecdote that’s shared in that article is not: It’s so weird to be making these cold calculations and deliberations. The fact is, if you’re trying to divvy up your philanthropic budget and figure out Do I donate to X or Y?—like, you do need some way of figuring out what is it better; to save the kid’s life or the adult’s life. Or like, Where should I put my money?

It’s not the coldness that should bother us, per se. But instead, I think it’s the confidence with which people make what is a very subjective judgment, relying on certain philosophical assumptions. I think it’s worse if a kid dies versus an adult, or an adult versus a kid. These are relying on subjective philosophical judgments about what matters most in a life. Is it meaningful experiences or whatever? Cognitive depth? And masking that with, like, objective claims, right?

I think that’s a fundamentally subjective thing, whether you think it’s better to save a kid or an adult. And then you kind of muddy that by saying, It’s two times or three times as valuable. And, you know, has much more expected value. And I can even put numbers on it.

And that just seduces our brains into thinking these are, like, objective-truth claims. When they’re really just deeply subjective. And so this is not a claim that all game-theory use is silly. But it’s a claim that when we’re dealing with such a complex system—like maybe this reliance on game theory, decision theory, classic EA-style consequentialism is gonna potentially lead us astray.

Honestly, in much the same way that, if people remember the Sam Bankman-Fried scandal, this feels reminiscent of that to me. In a way where effective altruists—

Warzel: How so? How does it remind you of that?

Samuel: I get nervous that some of this effective altruist–style thinking might be leading a company like Anthropic to make similar mistakes. With Sam Bankman-Fried, we saw an intense propensity to be willing to make these expected-value bets. You know, he’s doing something very against commonsense morality, right? Like, defrauding a bunch of people. But if he’s reasoning to himself, like: You know what? I think according to my calculation, this could really pay off, ultimately. And, like, it could end up going really well.

So he’s willing to do this very counterintuitive, counter-counter commonsense-morality thing of defrauding everybody. He’s placing a big bet. And I just am nervous that this same mentality of thinking about expected value in a sort of naive way might be misleading some folks at a company like Anthropic today. If they’re looking at the bigger picture, and they’re saying: Look, the expected value of us continuing to race is higher than the expected value of what happens if we unilaterally pause. We gotta keep racing.

I worry that actually the world is a lot more complex than they’re giving it credit for. And so those naive kind of mathematical calculations might lead you astray.

[Break]

Warzel: So okay; let’s talk about, then, what can be done. Because the outside evaluators; one of them is a company called METR. Who, the day before we’re speaking, right now, published a report saying that Anthropic let them in, gave them some access—not God-mode access, but some access—to try to evaluate how close they are to recursive self-improvement.

And the report basically—to summarize it, oversimplified—they’re closer than METR would have even thought, they say. But also that Anthropic made them redact, as part of the agreement of doing this, all of the good and compelling evidence that would allow us all to see that. And that they take issue with that, but they publish the report anyway.

That, to your point, does not seem like a great way of independent regulation. Doesn’t seem like we’re all better off as a result of that. Maybe slightly better, in that someone else can try to vouch for this. But also, you know, this is a relatively new, young organization. You basically have to choose to trust them in the way that you would. But then there’s, you know, traditional regulation that is, you know, possible down the line.

What are the levers that you think society can pull here to either put the pressure on to stop or to slow it down? Or you know—what can we do?

Samuel: I think we have learned very clearly from the social-media era, and now again from the era of these big AI companies, that voluntary self-regulation is just like—does not cut it. That does not work.

You cannot rely on the goodwill of these companies alone. They shift their goalposts all the time, as we saw, for example, with Anthropic’s shifting responsible-scaling policy. So you need regulation. You need laws.

And the good news is the midterms just provided a new crop of people who are very safety-pilled, right? And like, they seem like they’re gonna really try to do what they can on making laws that actually rein in these companies.

I don’t think it would be the ideal thing for companies to voluntarily decide if and when they get to pause. I think humans should get to decide the course of our own future.

But, you know, we could see regulation that frankly makes it illegal to develop RSI. I think there’s a very strong case that developing recursive self-improvement should simply be illegal. You have policy makers right now talking about this. And you know, more broadly, I just think we need to see serious efforts from policy makers to get us to a coordinated pause.

I think a common way to straw man or misunderstand what I’m saying—when I say, Hey, maybe we should just unilaterally pause—is to just understand that as saying Anthropic should just stop voluntarily, or be made to stop by the government. Close up shop, and that’s it. That’s not actually the real goal here.

The goal is like: Let’s really fight for a global coordinated pause. Let’s sort of make it really embarrassing for people to keep working at these companies. Such that more of them on the margin quit, and talk publicly about how—about the dangers of these companies.

And then, you know, if coordination turns out to be unsuccessful, then nations should probably just pause. Even on their own, if there’s not a coordinated treaty with China. It’s ideal if we get a coordinated treaty with China first, and I think lawmakers should do whatever they can to get a treaty. But even if you can’t get China to stop driving off the cliff, you shouldn’t be driving off the cliff first, right? Like, that just doesn’t make sense.

Warzel: Let’s define, though, what you mean, or what people in this position mean, by a pause. Does a pause mean: Hey, we’ve got, you know, whatever version of Claude and ChatGPT, and that’s it? Is it like—what does it functionally look like to pause development here?

Samuel: Good. Okay, yeah. I’m glad you’re saying that, because that’s another way this gets straw-manned. Pause does not mean the AI company just closes its doors and dissolves tomorrow. It doesn’t even have to mean everyone quits.

It means stopping kind of R&D frontier research, right? So developing recursive self-improvement—developing like the next generation of AI models that are even more opaque and powerful than the current models. So just like, stopping where you’re at. It doesn’t mean getting rid of the Claude and ChatGPT that we already have.

You can’t—you know, these models are out there. It doesn’t mean getting rid of every single data center. These data centers are out there. But it means more people might quit and advocate from the outside for increased safety. Some people might stay and work on product. Some people might stay within the companies and try to, you know, fight this insider game, where they’re trying to move the company toward safety from the inside. There can be a case for that.

So it doesn’t mean just: Everything shuts down tomorrow. But it means these companies should stop pushing forward the frontier in ways that they themselves clearly see as dangerous.

Warzel: One quirk to that, or wrinkle in that, I think, is that a lot of people who may be some of the biggest critics of AI altogether are also less likely to believe that we are building the machine god, right? I think that some of the biggest criticism of this just comes from a corporate-power standpoint, right? Of like: These companies have too much money; they’re too big; they’re building these data centers.

You know, treating AI not just as a normal technology, but also as like, you know: These people are full of themselves. And so a little bit, I think there’s a tension here, right? Between this idea that in order to put the pressure that may be needed on some of these companies, there is maybe a bit of a needing to see them as being a little bit closer to actually building some of these things than I think some of the critics are comfortable with.

And I’m curious how you see that tension. And how—if you’re somebody who maybe thinks these guys are high on their own supply—how you can also put that pressure on them. Without, you know, saying they’re about to build a super god who’s gonna turn us all into paper clips.

Samuel: Totally. Yeah, I think it’s really sad how these camps have been kind of polarized against each other. And I think that we can look at the whole menu and spectrum of risks from AI and you know, just sort of like diffuse some of the tribalism here. Because even if you are “merely worried” about like normal things—like giant cyber breaches, the power grid failing, food-supply-chain issues resulting from hacks, dramatic concentration of power, building of bioweapons—none of what I just said, you know, is talking about extinction. Or like this machine-god conception of AI.

These are all perfectly plausible things that could happen with AI as a normal technology, as your frame. But like, guess what? Concentration of power in the hands of a teeny-tiny minority in an irreversible and unprecedented way is a catastrophic risk in my books. Like, I don’t think we need to frame it as like, It’s extinction or bust. I think you can capture a lot of people by just saying: A lot of different things that normal people would be worried about fall under “catastrophic risk.”

Mass unemployment that leads to the gradual disempowerment of all humans, because companies and governments no longer value our labor and therefore don’t care about catering to what we want. Concentration of power. Like, these are catastrophic. And I don’t want to live in a world where those things obtain.

So I think a few years ago, it was maybe harder to make this case to people. But as the capabilities of the models have become more and more clear, even the critics of yesteryear who were saying, It’s all just stochastic parrots, it’s all just hype—I think if you’re willing to honestly look at the evidence now, you gotta kind of admit these models are powerful. And pose serious risks. You don’t need to conceive of them as godlike, but they’re serious risks.

And so I think you can make a forceful claim just based on that. And to your point about the people within the companies and what they think, I do think we’re at risk of really underestimating what a powerful signal it can send to the general public if people within the companies quit. And if the companies themselves unilaterally shut down. The Jacob Coxon resignation paired with the Hugging Face incident from this summer made a huge difference to public opinion.

And I think that on the more meta level, if a company like Anthropic itself were to unilaterally pause, people would take that really seriously. And that could be enough to really wake up the public and say: This is not just marketing hype or anything like that. They are genuinely terrified of what they’re building. Maybe we should also be scared.

Warzel: I think when you look at the levers, I think in a weird way it feels like this is a freight train that is just barreling toward society, right? And these guys are just, you know—the whatever, a car speeding off a cliff. You choose your metaphor, right? But the foot is on the gas, but for the people who are driving it.

What I find really interesting in this moment we’re talking—on the Friday after the midterm elections—is that you’re already seeing, in like three days, a complete and total, like I don’t even know how to describe it. A reorganization, a little bit. Or a reprioritization in the wake of just how unpopular AI was in this election, in terms of people voting against data-center ballot initiatives. People just in exit polls saying how much they distrust and dislike this technology. People like, you know—Elon Musk’s money getting him basically nothing in terms of the elections. And this idea that you’re already starting to see financially, like some of these tech stocks taking a hit.

But it does seem like this train might be a little easier to stop than we might have thought even, you know, five, six, seven days ago. And that to me is just a very fascinating thing to consider in all of this.

Samuel: Yeah. I think people really got the wrong headline from the midterms when it comes to AI. All the headlines were about how Americans flat-out rejected data centers. Which, okay, valid headline. But that’s not the most important outcome from this election. The most important outcome from this election is: Wow, we have just elected some people who really care about AI’s safety, right?

You can pick off data centers one by one, but that won’t ultimately stop the build-out. It’ll just slow it down. But if you elect people who really care about safety and can pass laws with teeth, that’s a big deal.

So I think that the lawmaking is a real lever. And I also think just there’s a rhetorical fight that’s gonna be fought in the next couple years. Especially leading into 2028. And I think to make that rhetorical fight go well, you have to really put it to the public and to the companies and say: Even if you win this race, if Anthropic wins, or if America wins this AI race, what are we winning? Right? Like, what is that good future that we’re winning? I’m personally less worried about extinction risk and more worried about gradual disempowerment of people like you and me, and concentration of power. So I think it really matters a lot what kind of country or world we expect to find ourselves in, even if we win by racing.

People will often say, Well, surely it’s better that we win than that China win. And believe me, I’m no fan of the CCP. But if we are willing to go to any length to avoid China winning, do we risk becoming something just as bad, or even worse?

I think that is an argument that so far politicians have failed to really put to the public, and to the companies. And I’m hoping that is the kind of argument we will see surface in some of the many hearings that I think the new folks in Congress will be about to call.

Warzel: I think that’s a good and potentially even hopeful place to leave it. Sigal Samuel, thank you so much for coming on Galaxy Brain.

Samuel: My pleasure. Thanks.

Warzel: That’s it for us here. Thank you again to my guest Sigal Samuel. If you liked what you saw here, new episodes of Galaxy Brain drop every Friday. You can subscribe on The Atlantic’s YouTube channel or on Apple or Spotify or wherever you get your podcasts. And if you want to support this work and the work of my fellow colleagues, you can subscribe to the publication at TheAtlantic.com/Listener. That’s TheAtlantic.com/Listener. Thanks so much, and I’ll see you on the internet.

Learn more about your ad choices. Visit podcastchoices.com/adchoices

Read full story at source