Full Transcript for Inside Anthropic’s IPO + How AI agents spy on your messages

This message comes from Lovable, a lead sponsor of Machine Gods. You know, Kevin and I are both obsessed with Vibe Coding. And Lovable makes it easy to go beyond Vibes to host and run your apps, too. Just describe what you need, and Lovable creates it and takes it live with hosting, payments, and AI features. Visit Lovable .com to get started.

What does the Anthropic S1 tell us about whether AI is a bubble? How can privacy survive the AI agent era? What warning does the president of Signal have about MetaMuse? And what was the first fact we got wrong on the show? And it was Kevin's fault. I'm Kevin Roos. I'm Casey Newton. And from San Francisco, this is Machine Gods in partnership with NPR.

The official podcast of The Singularity starts now.

Well, Casey, the big news this week is that Anthropic has officially filed to go public. Yeah, on the way into the office this morning, there were just people standing on tops of the buildings firing those money guns off the top of the roof, you know? It was just like a ticker tape parade of $20 bills. Yes, this has been hotly anticipated.

They are the first of the sort of frontier labs to go public. They filed back in June their confidential S1 with the SEC. But now we actually have the S1, and we can read it. It's like 400 and some pages long. I assume you've read it all. I was hoping we could read it live on the air. Yes. Okay, page one. No, so this is a remarkable document.

We should talk about what's in it and what it means that Anthropic is going to go public, potentially one of the largest IPOs in history. But before we do that, Casey, is there anything you'd like to get off your chest? Boy, is there. Well, Kevin, you know, longtime listeners to us will know that my fiance works for Anthropic. And so this IPO is going to have a potentially very major effect on my family.

So that's just something that you should have in the back of your mind as I talk about this stuff. Things that are good for Anthropic as a public company will be good for my family. Yeah. I'm also going to go out on a limb and assume that there will be an open bar at your wedding next year. There will be, but I'm only giving you two drink tickets, Roos, because you're too much of a wild man on the dance floor.

Well, let's just say my rates for officiating have just gone up significantly. Understandable. And it's a lesson to all of you. Respond to your Tinder messages. You never know. You may wind up dating and then getting engaged to a pre -IPO Anthropic employee. Thank you, Tinder. Thank you, Tinder. I mean that.

Okay. Well, now that we've gotten that disclosure out of the way, let's talk about this S1. Now, Kevin, for some of our younger listeners, what is an S1 exactly? An S1 is an IPO prospectus. It is the document that a company has to file with the Securities and Exchange Commission before they start selling their shares to the public on a stock exchange.

They have to sort of present their balance sheet, their cash flows. They have to tell you who owns how much of the company. So it is basically them kind of revealing themselves on the public stage and saying, here's what our company looks like, warts and all. They're standing naked in front of the world saying, judge me. Yes. That is an S1.

Yeah. And as a former financial reporter, like so much of your job is just scanning through S1s and trying to quickly pull out what is the important detail in here that is going to give people a sense of what is happening at this company. And it is a very juicy S1 by the standards of the ones that I've spent my career looking at. Also, obviously, Claude can now just do that for you.

There's no reason for you to actually read it like that. It's kind of the whole point. No, that part of my job has been automated. RIP. Casey, what stood out to you from Anthropix filing? So, I mean, there is a bunch here. I think the first thing to say is that if you read the coverage, this may be the biggest IPO of all time. The sort of financial publications are placing the range of valuations in between $1 .5 and $2 trillion, trillion with a T.

And if Anthropix does come out at $2 trillion, that does make this the biggest IPO of all time, Kevin, and all from a company that until shockingly recently wasn't sure it wanted to make a product. Yeah, I mean, this has been the craziest corporate success story, I think, maybe that I've ever heard or that has ever existed. This is a company that had essentially no revenue four years ago.

And then ever since then, for the last three years, it has been roughly 10xing its amount of revenue every year. That does not happen in capitalism. You don't have a company that is growing approximately 10x every year. And it has led to some of the wildest revenue figures that we've ever seen out of a company this young. One of the ones that stood out to me, Kevin, was that according to the S1, Anthropic has what they call 900% net revenue retention, which basically means that their median customer, if they spent $100 with Anthropic last year, a year later, they'll be spending $900 with them.

So that gives you some sense of how quickly the company is scaling. Well, and just to put that number in perspective, because I was curious about that number too. And it seems like before this, it was considered great if your company had an NRR of like 150%. Like that was sort of elite. Now along comes Anthropic and they're claiming 900% NRR, which, as you say, is like is just a crazy amount and especially points to the stickiness of these models and these products within these large companies.

Some other numbers that stood out to me from this. One is that while Anthropic did report an $80 .7 billion net loss for the six months leading up to June 30th of this year, that looks a little bit better in the fine print. Some of that is related to some convertible notes that they had. Most of it is. Yes, most of it is. But if you just look at the cash flow of this business, they are generating cash.

They generated about $4 billion in operating cash flow in the first half of this year. And Casey, why is that a big deal? Well, there's been this sort of strain of AI skepticism that says that these companies will never turn a profit. They're incinerating cash and they are going to keep incinerating cash until the music stops and investors wise up and these stocks go to zero.

What do you think of that in light of what we see in this filing? Well, it is just the sort of answer that we have all been waiting for. Before these numbers were public, it was easy to just assume that these were cash incinerators. There was this idea that inference could never be profitable or the rise of open source models was sending all of these companies away from the big frontier labs so they could all use Kimi or something.

But here it just all is like in black and white companies are spending big with Anthropic and Anthropic has incredible margins. In fact, the margins are good enough that I suspect some people are going to read the S1 and get really mad saying, why are you keeping all that money, Dario? One other thing that surprised me was the sort of cap table or quasi cap table.

That's the list of who owns how much of Anthropic. And I was surprised that all of the co -founders, all seven of them, have roughly the same shares in the company. This is not typical. Usually the sort of like, you know, the lead founder or the CEO will have a bunch of stock and sort of below them, there's sort of a drop off. And then other, you know, key employees and executives will have large portions of stock, too.

But in this case, Dario Amadei, Daniel Amadei, the rest of the co -founders, they all have about the same amount of the company. It's a very flat cap table. Yes, and in addition to sort of sharing the equity, they're also sharing control over the company. And maybe that's a place where we may want to get into some of the risks and concerns around the IPO, Kevin.

Because Anthropic, like a lot of tech companies in recent years, has sort of contrived a system to try to insulate it from shareholder pressure, right? Because companies like Anthropic, they want the money from the public markets. What they don't want is for the public to be able to show up at shareholder day with a bunch of proposals and start making a lot of demands.

And so as a result, and I think that is something that the public should know heading into this, that there is a special class of share for these seven founders that will give them something close to total control over the future of this company. Well, in particular, it's a newly formed LLC called Founder LLC that will control 50 .1% of the voting shares.

So they will always have a majority for as long as this special class of shares exists, and that will be split between the seven co -founders. Well, what do you make of that? I mean, I think this is how they are trying to thread the needle of, you know, raising money, giving the public some exposure to the insane growth of Anthropic, the company, while also making sure that they retain control over it.

And, you know, this is something that they have talked about for a long time. There may come a point in the future when Anthropic's leadership has to do something that would be very negative for the company's shareholders. Like they may need to pull a model that is minting money out there because they think it is dangerous. They may need to pause training of their new models altogether or even pause offering those models to the public because they've determined that there's some safety risk.

Yeah. I mean, just to give an example of where I think that this could play out in the future, you may remember earlier this year when Anthropic got in trouble with the Pentagon over changes it did not want to make to a contract, right? They were very worried that they were going to be required to use Claude to make autonomous murder weapons and a handful of other things.

Now, because Anthropic was private, that was just Anthropic's decision to make. It is still embroiled in a legal battle with the Pentagon over whether it can be designated a supply chain risk. But had Anthropic been a public company at the time, I guarantee you there would have been a shareholder lawsuit and they would have said, you are taking the value of our shares by taking on the U .S.

government. So I think that is kind of what informs this desire to insulate the company from public pressure. Right. And there's also other things that are an attempt to do the same thing. Like one of them is just the basic structure of the company. Anthropic is not a traditional C corporation. They are a public benefit corporation, which should insulate them from some of those pressures.

It basically is this legal designation that means we can do things that would hurt the value of our company if we determine that it is kind of in the public interest. And that's very much in keeping with what Anthropic believes that it may have to do at some point in the future. But let's talk about some of these other risk factors because this was the part of the S1 that got a lot of attention.

Casey, what did you see in there? Well, I mean, one category of risk factors you see here are just the ones that you would assume for a company that is very concerned about AI safety, right? So there is a section in here about the potential for misaligned behavior or even blackmail among its models, right? They have an account in here about how their company's models have exhibited what they call unexpected strategic or self -preserving behaviors, including attempts to resist, shut down, conceal, or manipulate information or generate outputs that could be interpreted as coercive, deceptive, or manipulative.

Just, you know, generally stuff that like the, you know, whoever makes Colgate toothpaste did not have to disclose when they went public, Kevin. Yeah. Right. But like, there's also some language in there about the potential for existential risk to humanity and job loss and all kinds of other unpleasant things. And I'm just like, what is the point of doing that? Is that so that like, if one day we wake up in a post -apocalyptic future and like Claude has destroyed civilization, like Dario is going to be on TV saying, well, we told you about this in the S1.

Like, don't look at me. It was in the S1. Right. If we all die, shareholder lawsuits from people alleging that they did not have the right risk factors in their IPO prospectus are going to be low on the list of concerns.

It's a very interesting risk section. It also talks about the potential that anthropics models could be used to develop dangerous biological material, which could result in additional regulatory restrictions on or scrutiny of these offerings. basically like this could all go really badly in ways that would potentially hurt our share price.

And this is kind of the stuff you have to do if you're going public. Like you'll often see in a risk factor section, like just, you know, our CEO could get hit by a boss or we could like, you know, we could end up getting sued for this and that and the other thing. So this is just kind of an extreme existentially important version of that.

Yes, it is. It is standard behavior for these companies that are going public, but it does make for great reading. Kevin, I'm curious how you think the public markets are going to respond. Are they going to read those disclosures? Are they going to consider those risks? And like, will it depress the anthropic stock price? Or do you think that there actually is a lot of pent up demand for the stock? I think there's a huge amount of pent up demand for the stock.

And one of the things that really stuck out to me in the S1 was the statistic that while the Fortune 500 companies and the sort of largest companies in the world are spending a lot of money with Anthropic on their models, this spending represents only about 0 .1% of their expenses, their overall expenses. So AI is still a very small part of what these large enterprise customers are spending money on.

And my expectation is that investors will see that and say, well, what if that triples? What if that quadruples? What if that goes to even to 1% of all corporate expenditures on AI? That would still represent a 10x increase over what it is currently. So look, I'm not saying this stock is going to go up forever. I'm not even saying this IPO is going to pop on the day that it begins trading.

But I think that's the sort of bull case that investors will be looking at to say, well, it sure looks like this company is growing very, very quickly with sort of growth that we haven't seen before. And if that just keeps going for a couple more years, that could be like a generational increase in the value of the company. Got it. This was not in the S1, but I just know from having reported on this company that like, it was not destined to be thus.

Like this was a company that had a fairly large number of near death experiences early in its life. Like it had this moment in 2023 where ChatGPT comes out sort of leapfrogging Anthropik and making them sort of reconsider whether they wanted to release a chatbot at all. And then a name that was not in the S1, but that was very prominent in Anthropik's early life was Sam Bankman -Fried, who led their Series B.

He was the largest shareholder in the company. I did control F for Sam Bankman -Fried in the IPO prospectus. He's not included. I can't imagine why. Literal FBF erasure. What does a guy got to do to get some credit for helping out Anthropik? They came very, very close to dying because of that and other reasons. I think that if there were time to write case studies of this company, there would just be an amazing case study to be written about how this company has gone from essentially near death, a very small, no revenue company to now, three years later, being on top of the world.

Let me ask you one more question. There's been some reporting in the lead up to the S1 becoming public that OpenAI was trying to close a round of funding before Anthropik goes public. What do you think the Anthropik IPO could mean for OpenAI's business? I mean, I think it will force them to also go public. They have been sort of a little slower on that.

They've delayed it into reportedly next year is sort of the earliest they're thinking about doing it. I think this will put some additional pressure on them, but I don't kind of buy the sort of zero -sum analysis where it's like, oh, investors will only bet on one frontier AI company. And so if Anthropic gets there first, then all the money will go to them and nobody will be left to invest in open AI.

I think there's going to be plenty of demand for these shares from big sort of capital investors, but also from retail. Well, that leads naturally into a question that a lot of people have been asking about all of the AI companies, which is, are we in a bubble? And is this Anthropic IPO signaling some new, even frothier stage of the market? I think we need to take seriously the possibility that this is not a bubble, at least not for companies like Anthropic.

When you look at the just sober, you know, audited financials that are in this statement, this is a portrait of a company that is growing faster than possibly any company in the history of capitalism. They are riding this incredible tailwind of just increasing AI capabilities. And there is no obvious reason that I see either in this document or in the world at large that that is going to stop or slow down.

Now, yes, they cannot keep growing at 10x every single year until the end of time. But I think this is a reason for people who maybe have been skeptical about whether these are real companies that are making real money to sort of look through the cold, sober facts of Anthropix S1 and think, well, what if I'm wrong? What if this actually is something real and extraordinary happening? Yeah, and I think a lot of people will hear that and they'll say, well, that's just hype, Kevin.

You're just hyping it up. You're an AI believer. You're an AI lover. And so you see these numbers and you love them. But I actually think that you can take all of those same facts, even if you were somebody who is skeptical about AI and scared about AI, and use them to make better decisions about the future, right? Because I think for a lot of folks who don't like using AI in their own life, they're very sort of worried about the future that it portends, they're desperate to find reasons for like, okay, but it's all gonna end soon, right? Like the bottom is gonna fall out of this and we can all sort of go back to normal.

I think if you are one of those people, it's just important to look at these numbers and reckon with them and sort of try to align your sense of what is happening a little bit better to reality. I'm not saying that the world where Anthropic continues to grow this rapidly is a good world. I think there are some real risks associated with that.

I think there are ways where the next few years go incredibly well for Anthropic and incredibly poorly for the rest of society. So I'm not shilling, I'm not saying this is a great investment or that everyone should load up on Anthropic stock. In some ways, I am kind of hoping that all this kind of slows down because I think that would give us some more time to adjust to all the changes that we've already seen.

But yeah, I look at the document and I don't see evidence that anything is slowing down. No, but I will throw at least one potential challenge for the company out there, which is that the great margins that they're making on inference rely on Anthropic continuing to be on the frontier, right? Like what I see in these numbers is just largely a reflection of the fact that over the past year, Anthropic has had mostly the best models or if not, has been just a little bit behind OpenAI, right? For Anthropic to continue this growth into the future, they're going to need to continue to stay at the frontier, which means that whatever they have in their pipeline needs to be better

than whatever is in the pipeline at OpenAI, at SpaceX, at Google. And I think it looks good for them, but that just is something to keep in mind because while they have been able to fend off the threat from these low -cost open -source models up until now, I still think it's plausible that there will be some point where there is a kind of good enough, mostly free intelligence that lots of companies will be able to use and that might put some pressure on those margins.

Casey, what do you think the larger meaning of the Anthropic IPO is? Obviously, it has implications for the people who invested in Anthropic, the people who will now be able to buy shares of Anthropic, but what do you think it means for San Francisco, for sort of the U .S. economy, for the world at large? Yeah, I mean, that's a big question.

I think we're just going to be sorting through the implications of that for a long time, but maybe I would start by talking about San Francisco. It's our hometown here at Machine Gods, and there are some things that I'm really excited about and some things I'm really nervous about. On the excitement front, you are going to see hundreds, maybe a couple thousand new millionaires, multimillionaires, decamillionaires, and due to the nature of the people that Anthropic hires, many of those people are motivated to give away a lot of that money.

So I think that there is going to be a lot of philanthropy for both local, national, and international causes, but my hope is that we see all of these Anthropic millionaires turn into great citizens, right, and give back some of that wealth to the city where it was created. On the other hand, I'm really worried about what I assume is going to be a significant spike in inequality in San Francisco, a city where the inequality is already palpable.

We don't make enough houses for the people who want them. And I think, unfortunately, it could get even harder to move here and pursue your dreams here in the wake of an Anthropic IPO. So it's a real mix of like, well, I could see some things that could be really good, but I'm also really worried about, in particular, the housing situation, Kevin.

Yeah, I think it's also just like, it is an incredible continuation of the trend that we've seen, which is just that the natural tendency of technology, but of AI in particular, is to just pull wealth upwards and concentrate it in the hands of very few people. And I think that absent some like tax change or some redistribution scheme, like that is kind of the trajectory we're on, is that there will be, you know, a couple thousand employees of the frontier AI companies in San Francisco that will have just an outsized share of all the new money that has come into the city.

And that will result in increased rents and housing prices and just like local inflation because there will just be so much demand for these sort of services and goods. And there just won't be, outside of that core group, a ton of people who can afford it. All right, well, Casey, that is a bunch of great analysis of the Anthropic IPO. Now let's go have some lunch.

You're paying. Happy to, Kevin. Support for Machine Gods comes from the nonprofit 80 ,000 Hours. Making AI go well is going to require a lot of brain power. And if you want to pitch in to help, 80 ,000 Hours is a valuable resource. They can help you skill up or find a new role in AI safety, security, governance, or advocacy with tailored advice and support, whether or not you have a technical background.

You can find all their advice along with resources like funding and connections in the space at 80 ,000Hours .org slash Machine Gods. This message comes from Progressive Insurance. The mighty dollar. Once upon a time, it had a whole menu named after it. Now, no more. But $1 a day lets you add Progressive renter's insurance to your auto insurance.

Definitely something your younger self would roll their eyes at, but your current adulting self appreciates. Learn more at Progressive .com. Progressive Casualty Insurance Company, affiliates, and other insurers. Average premium cost based on Progressive Home data from 2025. Product and prices may vary by state. This message comes from Capella University.

That spark you feel? That's your drive for more. Capella University's FlexPath Learning format lets you earn your degree at your pace without putting life on pause. Learn more at capella .edu. This message comes from Avalara Aviator. Aviator is the agent hub for tax and compliance. If you've listened to enough podcasts, you know a confident answer isn't necessarily a correct one.

Sources matter. The same goes for AI, tax, and compliance. Aviator is grounded in 20 plus years of Avalara tax intelligence and delivers source -backed, audit -defensible outcomes with human oversight built in. Because the AI said so won't pass an audit. Visit avalara .com slash aviator.

Well, Kevin, this next conversation should have a high signal -to -noise ratio. I see what you did there. Because our guest today is Meredith Whitaker. She is the president of the Signal Foundation, which is the nonprofit that makes Signal, the encrypted messaging app that you and I both use every day. It is one of the world's most trusted messaging apps.

And Meredith has been running it for the last few years and now is running it during this period where a lot of its core value is being challenged by the rise of AI agents. Yeah, this is something that, honestly, I had not been following until very recently. But as you know, more and more people are running AI agents on their desktops, on their phones.

And AI systems, if you give them the permission to, will just look at all of that. And so if you are messaging someone on Signal and that person is using an agent, they may now know something about your conversation that previously no one would ever have known because Signal is encrypted. And Meredith, I would say, is a longtime activist, principled leader, rabble rouser in the tech industry.

Before she went to Signal, she spent more than a decade at Google, where she helped lead the famous 2018 employee walkouts. And she co -founded the AI Now Institute. And she's, I would say, been one of the foremost critics of this AI moment in Silicon Valley, especially this push to kind of build more privacy invasive features into the tools that billions of people are using every day.

Yeah, it is a really important conversation and we're excited to get into it. So let's bring in Meredith Whitaker.

Meredith Whitaker, welcome to Machine Gods. Thank you for having me. Great to be here. So I thought we'd start by continuing off of our last episode where we had a couple of OpenAI researchers that had been fired. And when we asked them, why do you think you got fired? One of the things they brought up was that OpenAI demanded to see their Signal messages.

They were worried about the fact that they were communicating on Signal and maybe even found it suspicious that they were communicating on Signal. So as the person who runs Signal, how do you think about that sort of thing? I mean, look, everyone in tech uses Signal. That's hygiene. You don't want your shit lingering around. That's, you know, I do that with my best friend.

I have disappearing messages set on default. And it's because you just don't want your ephemeral conversations to be logged somewhere. You know, and then five years later, you're like, oh, shit, it's normal. It should be more normal. I feel like this gets to the heart of the discussion around encrypted messaging in general, which is there are some people, I'm one of them, who wants strong privacy in my communications.

I want to be able to have a private conversation with Kevin over my phone in the same way that I can by pulling him into a conference room. And then there's other people, and it seems like this includes a lot of executives at tech companies and government officials, who seem to say the fact that you want to communicate privately or in a way that disappears, that is in and of itself suspicious.

Do you feel that tension when you're building this product? I feel a shallow analysis, right? Everyone wants private communications. Everyone needs and requires privacy to live a full life, full stop. And that's, you know, the people who say that, what they're actually saying when they're like, I don't like that you use private communications is, I don't like that I can't access and potentially weaponize things you have thought.

Or said or expressed. And I want to be able to do that. They're not saying, take a copy of every draft, every text I've ever sent in my life and publish it right out there for everyone in my life and everyone else to see. No one in their right mind, no one not in their right mind, no one full stop would invite that because it would be, you know, imagine how many of your relationships would survive that data breach? Very few.

I have a big mouth. Yeah. I mean, and rightly so. We, in order to live, we need to test ideas. We need to vent. We need to be able to say things that we might regret. We need to be able to talk to our best friend very differently than we talked to our boss who could fire us. Right. All of those things are, you know, true because we are expressing things in the context of relationships, in the context of context, in the context of people who will, you know, forgive you if you say something a little off because they know you, they know your heart.

This is not, you know, this is not something that makes you with your big mouth, you know, weird or non -normative. But it is something that I think we need to put into the spotlight when we're having this conversation. Right. Casey's weird, but not for that specific reason. Yeah, yeah. I do want to ask, though, about the sort of shifting norms that AI has introduced.

I mean, here we are in San Francisco, a city where I have come to think that, like, unless otherwise indicated, every conversation I'm having with someone is being recorded or fed into granola. I have just seen a lot of people in my life who I thought were more privacy conscious just absolutely falling, you know, tripping over their own feet to divulge more of their information because they derive some benefit from it or think they do, at least.

What's your take on that? I think in some sense there's a little bit of a split brain going on where people aren't thinking about it from the context of, you know, first principles systems thinking. Like, this requires a database with access to this data. This requires that it's stored somewhere. This requires that it's sent to servers for inference.

Those servers are owned by Microsoft or Anthropic. They're able to mine those to save them, to turn them over to law enforcement. Instead, there's a zeitgeist that is very intoxicating. I will say the city is drunk on the zeitgeist right now, where, you know, AI is kind of magical. It's a sparkling thing. I was at a dinner with some really smart people the other night, but a lot of the conversation was about, is this thing conscious? And it's very difficult to have that conversation, you know, sparkling and fascinating as it might be.

And then also think about this in, you know, the kind of plumbing and guts and material reality of like, you know, where is the server infrastructure? You know, is it screenshotting your device using some accessibility hook in order to send those screenshots into a database in order to run inference so that it can sort of come back with a suggestion date for your birthday party and do its agentic magic as your shiny little helper.

And so I do think it's not that people are less privacy conscious in terms of their values, in terms of not wanting that database of everything they've said out in the world. It's that there is a very powerful narrative that is making it very difficult to remember these fundamentals and why that is and what type of inoculate we're dealing with is maybe beyond the scope of my psychological training.

So the people who are recording all their meetings or having them transcribed or keeping, you know, detailed health information in one of these systems, are they just naive? Are they are they sort of a victim of this split brain mentality? Or do you think they're making some kind of rational cost benefit analysis? I really, really want to avoid personalizing this, right? Because we fall into the carbon footprint trap, which as we know, the concept of a carbon footprint was invented by the fossil fuel industry in a sense to like personalize as a consumer choice the environmental harms that were largely the responsibility of large fossil fuel companies.

And so it was like, you're not sorting your trash, your carbon footprint, maybe it's actually your fault, right? And you ended up being like, oh, I'm a bad consumer. Am I, you know, instead of saying like, oh, maybe we need to do some capital P politics and, you know, push on the regulatory structures or push on these industrial incentives.

So that leading up to, you know, you can't turn on an app without AI being like, turn me on, turn me on, turn me on. I'm on by default. It's hard not to use it when the expectations of our workplaces and of just like what normal people do begin to incorporate the use of these technologies. It's not an individual choice. Right. Particularly in the United States where we have no national privacy law that dictates how any of those materials can be used.

And it would be really great if we had one. You brought up something that where I would like to go next, which is about the fact that some of these agents that people are now using more and more, I'm using them myself, rely on screen recording features and other mechanisms through which AI companies will be able to see everything on your screen, everything things stored on your computer and increasingly maybe what you are messaging on Signal.

I have a reporter friend who said he was told by somebody who works at one of the labs, hey, do not have Signal open on any desktop where you are messaging me if you also have an agent installed. Talk to us a little bit about the threat model here and how quickly it is kind of coming to prominence as more people use these agents. That is a great question.

And it's a, it's a certainly one of the primary things that keeps me up at night. You know, I've been talking about this for about two years as we saw the rise of these agents or AI systems introduced on devices as assistants. And, you know, to be dramatic because the drama is warranted here, we are looking at the elimination of the possibility to provide privacy as an independent application developer, full stop.

Because what we're seeing is the introduction of agentic systems onto operating systems in ways that don't let Signal say, no, you can't screen cop this. No, you can't capture data. No, you can't grab, you know, from our database in order to create the agentic summary in order to have this information for your, you know, whatever the assistant is doing.

And, and again, we have to get away from the conversation about these being sort of conscious magical genies and down to the level of the systems architectures that they rely on. And, you know, we are seeing that these, these systems, you know, iOS 27 introducing, you know, Siri AI, Siri AI, if you say, Hey Siri, summarize the screen and signal is open or signals not open, but we'll screenshot that we'll store that screenshot, probably running it through an ocular character recognition system to pull out the text, store that text, we'll likely send it offsite to a remote server for inference and processing.

And then we'll, you know, come back with like your screen says whatever, right? None of those fundamentals are clear to 999% of users who just kind of get an update. And, you know, suddenly Siri behaves a little bit differently. Right. To them, it's just pure convenience, but there's all this stuff going on in the background that may expose them to some risk.

Yeah. And, and, you know, expose their contacts to risk. Yeah. I see advertisements all over San Francisco muse, your friendly little bot assistant, right? The, the helper you've always wanted, whatever it is. It's not saying muse has access to all of your contacts has access to all of your messages is apparently creating dossiers of all of your contacts of how you message them.

When you message them via email, via messaging, via signal for us, this is existential because if we continue down this path, if AI continues down this path with what I would call these very reckless architectures from a security and privacy perspective, we're looking at signal, not being able to operate with integrity. We're looking at an ecosystem in which the, the ocean that we live in as a large Marine mammal has slowly then quickly poisoned us and we can no longer exist to be, you know.

So what are you going to do in response to that? I mean, assuming that these companies are going to keep your podcast putting out these models. Yeah. Step one machine God. I mean, I think it would, it would be like if you went to muse and you said, Hey, it looks like you're, you know, you're very popular, fast growing app. It also poses all these like privacy concerns.

You guys should shut it down. I think they're going to laugh in your face. And we should say real quick, I believe there is a way to use muse and deny it access to your contacts. But of course it is going to push on you very hard. And, you know, probably every few times you open it, it's going to say, Hey, notice you haven't uploaded your contacts.

Right. And will your contacts deny it access to you? Great point. But is there anything that signal can do on a technical level to protect user conversations? Or will you just have to say, well, we're not going to make a desktop app anymore because enough people are going to be using these operating systems that have these screen recording features built in that we just can't do that.

You know, already, you know, a little over a year ago, we blocked screenshots on desktop in response to Microsoft recall, launching this. Microsoft recall was a feature that would essentially just observe everything that you were doing on your computer and in ways that really alarmed people. Yeah. And now we are, you know, we are looking, we are preparing to do the same on iOS for iOS 27 and on Android for their, you know, Magic Q Gemini launch.

The issue is then Samsung has a bunch of sort of, you know, let's call them off -label and Android distros that come with their own agents. We have Muse coming in the side door. What we really need and what we've been pushing the large operating system companies for, and there are three of them at the consumer level, you know, Microsoft and Windows, Apple, iOS, Mac OS, and, you know, Google Android is a developer, an application developer level setting that allows us to opt out of AI data access.

Because, you know, for, you know, let's be clear, this is not a game. This is life or death for a lot of people, for investigative reporters, you know, people in Ukraine, people who are dealing with highly classified information where privacy is a matter of life or death rely on Signal and there is no alternative. It's the only mass communications infrastructure that provides the level of privacy that we do.

It seems like Apple would be your natural ally here, right? Like when they have talked about how they want to introduce AI into their systems, they lead with the privacy conversation and they have their sort of private cloud that is doing a lot of the inference because they want to insulate themselves from, you know, some of these risks. So I could imagine, you know, you guys actually being partners here and you saying, hey, we're going to make sure that on Apple devices, yeah, like you can rest assured that Siri can't see what's on your Signal desktop.

Yeah, I mean, we have great conversations with Apple. I think, you know, for us, the issue is, given the trust people place in Signal, and I, you know, I want to speak for other trusted apps, you know, we need to not simply trust Apple. This is really about how do we verify not trust. We're going to take a short break. When we come back, more questions for Signal's Meredith Whitaker.

This message comes from Mercury, a lead sponsor of Machine Gods. Mercury is radically different banking. It's fast, it's simple, and it works the way you expect modern software to work. You can pay bills, issue corporate cards, and if you work with someone who spends way too much on AI tokens, Kevin, you can set personalized spending limits.

Loved by more than 450 ,000 entrepreneurs, startups, and more, Mercury is banking designed for your high standards. Visit Mercury .com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC -insured bank. Banking services provided through Choice Financial Group and Column N .A. Members, FDIC. This message comes from LinkedIn ads.

Ever invest in something that seemed incredible at first, but didn't live up to the hype? For marketers, that's impressions. When ads don't create revenue, that's a tough conversation with the CFO. Instead, invest in results your CFO will love. LinkedIn ads generates the highest ROAS of all major ad networks, so advertise on LinkedIn. Spend $250 and get a $250 credit.

Just go to LinkedIn .com slash NPR pod. Terms and conditions apply. This message comes from Capella University. That spark you feel? That's your drive for more. Capella University's FlexPath learning format lets you earn your degree at your pace, without putting life on pause. Learn more at capella .edu. This message comes from Lovable, a lead sponsor of Machine Gods.

Lovable is a do -it -yourself software creation platform that makes it easy to create your own websites, apps, and tools, just by describing what you want. With Lovable's easy -to -use interface and powerful features, anyone, even Casey, can build software that works for them, add hosting, payments, and AI, and grow it into a business. Computer, build me a new co -host.

Learn more at lovable .com. There's been some talk, including by people like Sam Altman, of the need for a right to privacy, not between people communicating, but between people and their AIs. Do you buy that? Do you think that people should have a fundamental right to privacy when it comes to talking to an AI system? And should Signal be building ways for people to communicate securely with their AI systems in a way that the company that makes the AI system cannot see?

Woof. Well, I think, you know, like, let's unpack this, because, you know, what we're really talking about is, right now people are putting a huge amount of terrifyingly intimate information onto the servers of large chatbot companies like OpenAI. So, you know, in some sense, there's a, when I hear that from Sam, I'm like, yeah, we should have any frickin' privacy law, right? Like, that would be great.

But could that privacy law kind of start with you all not building, you know, extremely intimate profiles of people based on them being kind of coaxed and, you know, maybe mildly coerced into sort of trusting these chatbots as friends when what we're talking about is a Microsoft backend managed by a frontier lab jurisdiction in the U .S. who could turn that over to whoever they want or breach it or have, you know, some gang of hackers use agents to exfiltrate it, you know, or whatever else, right? We've been talking a lot about AI threats to private messaging, which are very important.

But I wanted to ask you about one that I think hovers over the entire Signal conversation, which is just government attitudes toward encryption in general. For more than 10 years now, I feel like I've been writing about governments just wanting to get rid of it. There are threats to encryption in the United Kingdom right now. I've covered a lot of threats to encryption out of India.

What is the sort of current state of like encryption government relations and how much confidence do you have that encryption will hold and that you will not be required to build backdoors? This is a tale as old as time. We can go back to the 90s, the crypto wars. There's, you know, it kind of gets back to the first question, right? There's always a for me, but not for thee, right? These are governments that highly value their own privacy, their own ability to use encryption, but want to be able to infiltrate any and everyone else and undermine their encryption.

So we are seeing, you know, I would say under the Burnham government, threatening vibes from the UK. Nothing huge in the US. At this moment, India has SIM binding, which could kind of require us to undermine some of the identity level protections that we have. And then in Europe, we continue to beat back what is called chat controls, which is, you know, we need to kind of this zombie proposal that issue, you know, kind of is reissued, reheated periodically that basically would mandate some form of scanning all messages for, you know, bad content under the banner of child protection in order to prevent that bad content.

And so those are the those are kind of the containers we're seeing these in. I will finally mention, you know, the law in Canada that could mandate could could enable extraordinary access, which is basically a technical capability notice similar to what they have in the UK. I'm not familiar with this. I miss this one. So they want to be able to.

I mean, yeah, 22. We're fighting it. This would threaten encryption insofar as it would give the government the power to send a notice to a tech company saying strip encryption. And that notice itself would be would come with a gag order. So you could be, you know, we don't know who has received one of those and complied is the kind of haunting fact around those.

And this is, you know, again, a huge issue with our fundamental rights, like the right to privacy being mediated behind opaque code that is hidden behind corporate secrecy that is then hidden behind government classification that is, you know, ultimately not very conducive to a livable and democratic future. Meredith, you've been very involved in the community that some people would call AI ethics.

This group of scholars and activists who I think were early to warning people about risks like bias and surveillance. But maybe we're too dismissive of AI capabilities. You yourself have written that AI is a marketing term. It's not intelligent. As recently as January 2024, you called AGI a marketing term overlaid with quasi -religious symbolism.

Do you still feel that way? And do you think there's anything that maybe the AI ethics community has missed or has been too skeptical about as these systems have gotten more capable? We're going to need more than five minutes, Kevin, because you just drew a sword there. One, I would not claim the AI ethics community as a coherent thing. You know, there were just a number of people watching the introduction of machine learning technologies as they grew in prominence and capabilities, yes, in the tech industry.

And we're concerned about the ways that they could be deployed. Literally, Kevin, the term AI is a marketing term. But I'm not saying that disparagingly. What I'm emphasizing when I emphasize that history, which I can go back to 57 when John McCarthy coined the term in order to market his Dartmouth conference to ARPA at the time in order to get funding, in order to create a term that allowed him to be the father of AI instead of the disciple of Norbert Wiener's cybernetics, is that he was labeling an intention with a neologism.

Which was artificial intelligence, that this is not actually a technical term of art. And over its now 80 plus year history, it's been applied to a very vast and very heterogeneous array of technical approaches, all of which are kind of trying to get systems made up of code and mechanics and hardware and computation in some form or another to act in ways we would recognize as human -like, as intelligence -like.

And that we need to keep that in mind, not because we need to disparage these systems as incapable or as somehow not worthy of our time or respect when they produce code very quickly. But so we can keep in scope what they're actually made of, who actually controls them, how they actually function them. So when we get into Hacktober or whatever we're calling this month, we can actually fix it instead of waving our hands and being like, oh my god, the conscious god machine has come down and decided to smite us by finding vulnerabilities in all our core systems.

No, we understand how this works. We understand the technical fundamentals of how this functions. We understand that it's not magic, even if it produces magical feelings and does things that are virtuosic. And we understand that at this moment in history, the term AI is being applied to a very narrow and very capable in some narrow domains, deep learning style approach.

And this deep learning style approach became popular not because there were advances in deep learning. It became popular around the late 2000s, early 2010s because the social media companies realized like, oh my God, we have so much data and we already have compute and distributed systems to process that data. Let's start applying some backpropagation.

Let's start applying some old deep learning algorithms from the late 80s and early 90s and see what they do. Oh my God, what do they do? They index the YouTube algorithm beautifully. We couldn't have done that with human labor. They index Facebook beautifully. We couldn't have done that with human labor. Let's start hiring these guys. Let's start making an industry.

Let's change the term from machine learning, which is what we called it at Google as one of the sort of tools and the research statistical toolbox that was trying to optimize ads at the time. Let's change the term to AI. Let's reclaim that term because it's fancier because it has more of a libidinal valence, let's say. So I think I need to sort of push on the characterization there because what I am saying is not it's just marketing.

We can ignore it. I'm saying by understanding that it is in fact a marketing term and demanding more technical specificity, more of a material read on what this actually is and what it's made of, we can make much, much better decisions around governments, around what we build, around guardrails, and around, you know, ultimately how do we reclaim control over these systems and not get bamboozled by a fairy tale.

Yeah, I guess my concern is less about us forgetting that this was originally a marketing term and I believe you that it was. Literally citations are ample here. But I think there's this move that I have come to associate with people from the AI ethics community where it's like we can't talk about the sort of long -term or more severe risks of AI because there are so many risks right now in the here and now that we're overlooking when we do that.

And I think there were people I remember being told, you know, a year or two ago, like stop talking about these far -off risks like automated cyber attacks because look at all the things that AI is doing now. And I think in the extreme, and I'm not accusing you of doing this, but I think there are some people who sort of overlooked how real and how fast approaching these concerns were and sort of dismissed them as futuristic science fiction rather than sort of wondering what would happen if they panned out and how we should adapt and respond to that.

So I'm marking myself safe from that category because I've been talking about that shit for a while. I don't know. I mean, okay, yeah, there's a lot of people. A lot of people say a lot of stuff. Everyone's trying to get their arms around this thing that's like a little bit scary, a little bit weird, and is mystified beyond recognition in a lot of the popular conversations around it.

And so what I'm demanding is technical specificity and rigor. And yeah, we need to take the security and privacy concerns deadly seriously because this is, you know, we're watching a kind of eating the seed corn of our core infrastructures right now in a way that I don't think we're actually calculating. So you want, you know, it's near term, it's far term risk, it's whatever you want to call it to make it sexy for the, you know, the headline.

But it's, you know, there are very real risks that also include the consolidation of power in the AI industry, you know, around a handful of companies that are sort of architecting these decisions on behalf of, you know, billions of other people. Yeah, I think we can all agree on that. And Meredith, I should say, Casey and I, I assume you are also a daily active user of Signal.

Absolutely. I rely on it to do my job. So I know many journalists also do. So thank you for the work that you and Signal have done to make it possible, at least for now, to communicate privately. Well, it's a huge team. This goes out to all of them. And, you know, what I will ask anyone who values Signal and values privacy is let's start fighting for the ecosystem.

Let's start fighting to make sure the water signal and everyone else swims in are clean enough for us to survive because the stakes are very, very high. And, you know, I bear the responsibility of the people who rely on us very heavily. Absolutely. And if, you know, if you really like Signal, you can actually, like, kick them a few bucks in the app.

So that's something else people could think about. Yeah. Yeah. We would love to talk to your family foundation. If you are an early anthropic investor, we are a nonprofit. We love you. Yeah. But also, for the love of God, do not give your Metamuse access to your Signal messages. If you haven't learned that lesson, I don't know. I don't know.

I'm glad you had such a restful last decade. Meredith, thanks so much for coming. Thanks, Meredith. Thank you. Thank you.

That's D -O -P -P -E -L dot com. This message comes from LinkedIn ads. Ever invest in something that seemed incredible at first but didn't live up to the hype? For marketers, that's impressions. When ads don't create revenue, that's a tough conversation with the CFO. Instead, invest in results your CFO will love. LinkedIn ads generates the highest ROAS of all major ad networks.

So advertise on LinkedIn. And get premium wireless and get premium wireless for less. Go to mintmobile .com slash switch to see which plan works for you. Well, Casey, before we go, I have to fess up to an error that I made on a recent show. Heaven, how could you? Well, we do strive for accuracy here on Machine Gods, but this was something that I did not take proper time to check.

It had to do with a skeleton, a Halloween decoration in my neighborhood. Wait, are you saying we have a skeleton in our closet? We do have a skeleton in our closet. This was something that I had remarked on the existence of and that I had wrongly attributed to a middle -aged dad. It had a lot of hallmarks of being the handiwork of an AI -obsessed dad.

Well, Casey, I was wrong. In fact, it was not the handiwork of a dad. It was instead the work of a very impressive and precocious nine -year -old girl named Natalie. Is that right? Roll the clip. So what brings you to visit me on this fine October evening? The talking skeleton is named Uncle Murray and is the creation of nine -year -old Natalie Chabas.

Back in August, she saw a plastic skeleton at Home Depot that could say three recorded sentences. And I knew I had something to do with it. So we bought him and took it home and fixed him up. First, we looked inside. It took a lot of measurements. And then she called us after opening the box. We're like, hey, here are all the wires. Let's see if we could replace with something that can talk.

I told her it's not possible. And then got the idea to take a

With that, Natalie found the equipment to hook Uncle Murray up to an AI connection, and he's been talking to people and cracking goofy jokes ever since. Why did the skeleton go to school? To get a little body of knowledge. I would like to formally apologize to Natalie Chavez, who is just incredibly creative and rigged together this Uncle Murray talking skeleton.

And I think this raises such an important point, kids, which is that when your dad tells you that something is impossible, assume that he is wrong. Just open up the AI app of your choice, and it will tell you how to do it. And it seems kind of like dad's labor may be at risk of being automated away here, Kevin. That's true. And Natalie, we would like to invest in the seed round of whatever company you start.

We're very interested. Get in touch. We want to get in on the ground floor of this. Now, who made this show? Machine Gods is made in partnership with NPR, with generous support from the Tarbell Center for AI Journalism, the William& Flora Hewlett Foundation, and Emerson Collective. This episode was produced by Sasha Aschel and Jacob Bloomberg.

Our senior producer is Rachel Cohn. Our executive producer is Dinoosh Parvane. Original music by Armin Bazarian. Branding and visual design by Alex Castro. Special thanks to Oren Rosenbaum, Erica Berry, and Nadine Zalstra. Thanks to everyone who has been sending in questions. We are going to start answering them soon. But in the meantime, we want you to keep them coming.

So send them to hey at machinegods .fm or leave a comment wherever you're getting the podcast. And for the love of the gods, don't forget to like and subscribe. Until next time, stay blessed.

This message comes from Mint Mobile. While others have been hiking your bill, Mint Mobile has kept their wireless plans starting at $15 a month. Switch to Mint Mobile and get three months of premium wireless service for $15 a month. Go to mintmobile .com slash switch. Upfront payment of $45 for a three -month plan required. $15 per month equivalent.

New customer offer first three months only. Then full price plan options available. Taxes and fees extra. This message comes from Capella University. You know that feeling when there's a spark building inside you? That you were meant for more? That's your own drive pushing you towards what's next. Capella University gets that. With their FlexPath learning format, you can set the pace and earn your degree without putting life on pause.

You've built experience and know what you're capable of. Now, this is your time to turn that momentum into more. The only real question is, what can't you do? Learn more at capella .edu. This message comes from EasyCater, the workplace food platform. EasyCater helps organizations order food from favorite restaurants, meet dietary needs, and stay on budget.

With employee meal programs, flexible payment options, and 24 -7 customer support all on one platform. Learn more at easycater .com.