Frontier AI Is a Ferrari. Most Companies Need a Model Y | Turing CEO

0:00

There are some very real risks when you're building frontier models. These systems are superhuman in their ability to hack systems. The fact that agents would pass messages to each other, they would invent middle management.

0:18

and cooperate to hack things. It's just crazy. Today, these agents maybe reliably work for like two days at a stretch for tasks like coding. We are still far from having these agents work autonomously for weeks and months and eventually years. Open weight models are maybe like three to six months behind the frontier. Frontier AI is how we transcend.

0:40

These super intelligent models will help us cure diseases, discover new materials, colonize space. And if we solve that, I think that's the last puzzle for humanity to solve. I feel like the next 10 years will be the brightest for technology yet.

1:03

Jonathan, thank you so much for joining us on Sorcery. It's been quite some time since we last had you on. I guess to start, let's talk about the overall change in the data landscape. I recently had MongoDB CEO CJ on. This was at the Raise AI Summit a couple months back, and this was when data really started to come back and the shift there.

1:27

So what's going on in the data landscape? Great. Firstly, thank you, Molly, for having me. It's a pleasure to be here as always. A lot has changed, right? Even by AI standards, there's a lot of stuff that has happened in the last few months. The data landscape has completely shifted in 2026 from my vantage point. The biggest observation I have is that we've gone from helping AI master tests to helping AI master real work.

2:00

Back in the era of helping AI master tests, like when we were happy that AI is passing the SATs, passing the bar,

2:10

winning a gold medal in the Math Olympiad, the paradigm, the game was different. It was about finding experts in every different domain. The question was about like, how many experts can you find? And extracting knowledge from their heads by having them have a dialogue with the model, having them,

2:30

evaluate model outputs. It was about transferring knowledge from the minds of experts into the models. I used to think of it like distilling human knowledge and skills into LLMs. Now, in the era of having AI master real work, it's less about finding experts. It's more about how close to reality can you engineer these simulated environments.

2:55

It's all about these simulated RL environments. And inside these environments, these agents train, right? And you still need experts to make sure the environment that we build is as close to reality. And you want experts to help you come up with the right prompts, the right verifiers, the right seed data. I think of it a little bit like...

3:20

Like, can you recreate a rich enough simulation of the world

3:26

so that when the agents train in that, they end up being good in the real world as well?

3:32

At Turing, this has been super fun because we have this unique advantage where we don't just help all the frontier labs improve their models in coding and knowledge work. We have an entire division dedicated to deploying agentic systems into the enterprise. And because we deploy, we get to see real process workflows. We get to see how real professionals work with these agentic systems, how they define real verifiers.

4:03

And that helps us build these simulated environments that are very close to reality to automate knowledge work. It's a really interesting perspective shift. I'm seeing that in the real world, too. I just had Goggin. I don't know if you know Goggin, but he was a founder of Udemy and Maven, and then he just came on to found... and become the CEO of the Horowitz Andreessen Academy.

4:30

So we're seeing this in real time, new education for kids. This is a complete like parallel, but instead of, to your point, scores and grades, they don't want to do that for kids. It's all about proof of work. We now have the tool sets to do everything we can. It's obvious AI can accomplish these tasks and do these things for us, but what can you build with it? So that's what we're seeing at least now too in the real world with education.

4:55

It's really cool that this is hitting like And, you know, it's for kids graduating high school, going into college. And what can they do there? Can they build a company and that kind of thing? But we're seeing that right now with that. So I guess to further unpack that, like, what does that evolution shift look like and what do you expect comes next? Yeah.

5:14

Yeah, I think from an education standpoint, I'm glad Gagan's working on this. It's great to see somebody AI forward and in tech looking at education.

5:28

One of my mentors, Alan Eustace, shared this with me.

5:33

I think people who talk about AI automating jobs or replacing jobs are missing something more fundamental.

5:43

AI's biggest superpower is up-leveling the type of problems humans can now solve. Like that's what these models help us do. So I think from an education standpoint, it's about teaching kids how to work with the models, how to ask the right questions, and how to verify whether the output is correct. It's about asking the right questions and verification.

6:06

And I think now the scope of problems that you can solve is just massive, right? Even when we evaluate candidates in interviews, back in the day, you might ask them to, I don't know, implement dynamic programming or some obscure computer science algorithm. Now you can ask somebody to, I don't know, replicate Amazon in your interview, right? And it's not just about generating a ton of code in that time period.

6:33

It's about checking whether the code that you wrote is correct. It does exactly what it's supposed to do. You don't want the code to be hacking Walmart at the same time. Is the code secure? Is it easily maintainable? Is the functionality, does it map what you want? So it's about asking the right questions and verification. And from a training perspective, it's all about creating really rich RL environments that mimic how real professionals do real work.

7:03

And I think of it as this five-dimensional matrix of every workflow, in every role, in every function, in every company type, in every sector in the economy.

7:16

All of these needs the right RL environments with the right experts to share prompts, verifiers, realistic seed data for agents to master these long horizon, economically valuable work. And what's really cool, Molly, is today these agents maybe reliably work for like two days at a stretch, like for tasks like coding. We are still far from having these agents work autonomously for weeks and months and eventually years.

7:52

I think that's going to be the exciting path over the next few years. Imagine you've hired somebody at Sorcery who you give them a task. They go off and do it. You have one-on-ones with them. You give them feedback. They take your feedback and go off and work autonomously.

8:10

They might spin up a swarm of agents to get things done really fast. Let's say you wanted to research a bunch of questions about the data industry. Those agents could go off, have interviews with other AIs or other humans and come back and help you with your task. I think it's about AI accelerating real economic progress. I think that's been a key thing that I've noticed alongside working with you and Turing over the last year is how much of an emphasis you put on the economic progress versus the doomerism.

8:43

And a lot of the doomerism right now leads to the cybersecurity risks. So in part of this data landscape conversation that we're having, what other considerations have you had to put in place to when you're going through RL gyms, when you're evaluating new models, that kind of thing. How do you keep that into consideration through the training process? But it's also the case that they are superhuman in their ability to detect vulnerabilities and patch them.

9:11

And that's a good thing. So at Turing, for example, we create RL environments that help agents automatically detect vulnerabilities and patch them. I mean, imagine an RL environment for exactly that, where the verifier is whether you found the vulnerability in that piece of software.

9:32

So it can be used for both defense and offense. And obviously, we have to be very careful with how we deploy these systems. I still think that these frontier models are a huge net positive for humanity. And It is possible to build

9:52

safe RL environments that cannot be exploited. You have to think about containment for agents to not escape the RL environment and do things that they shouldn't do. I think of it as an engineering problem, not something

10:10

that cannot be contained. I mean, we figured out how to make jet engines safe. I do think that's the right analogy for this. When it comes to bio

10:22

risks, I think it is a little more tricky than cybersecurity in that cybersecurity is one of those areas where there's always an active arms race between the good guys and the bad guys. With bio, the risk is a little bit asymmetric. Like if somebody comes up with

10:44

like a crazy virus that imagine something like COVID with a high viral coefficient in terms of how quickly it spreads with a high K factor, And

11:00

maybe it is much more fatal than COVID was.

11:06

You saw how hard it was to come up with vaccine production quickly. Deployment sort of happens in the real world. We are limited by a lot of real world factors. There is more risk in terms of how quickly will we be able to combat these. I guess a question I have, because you're so close to the training of these models, where does the line draw on ethics i've seen a several posts go viral recently on like is there a point of training a model on x if you're trying to train it to not do that why even give it that subset of data in the first place i.e one example really disturbing honestly it was like some researcher posted a video of them telling a model not to stab a baby.

11:59

It was like a baby doll. It was the craziest thing ever. And it's like everybody on X is like, why are you even doing this in the first place? Why would you give a model a robot with an arm with a knife on it? This is like a crazy example. But it was all over the internet. It's like, why even go to the lengths of creating that environment and that use case training so like can you explain those types of environments and why that could or could not even be like useful like why go to that level and why create a wet lab of biology and give a scary evil model company the use of potentially creating biological warfare just so we can train it not to do that what is the point of

12:48

that Yeah, there are two key problems here, Mali. It's generalization and emergent behavior. Those are the two problems, generalization and emergent behavior. Before I answer your question, let me unpack how these systems are actually trained, and then I'll get to why this is really hard to do.

13:11

First, the way these LLMs are trained is first you do what's called pre-training, where the models are fed a huge amount of internet text, other sources of knowledge like books, et cetera. And the model, you build what's called the base model, where the model learns certain concepts about the world in its quest to auto-complete tokens, right? And pre-training is this magical thing where when we went from GPT-2 to GPT-3 to GPT-4, as we kept increasing the scale, new behavior started to emerge.

13:51

Like coding, for example. Like GPT-2 was not very good at coding, but GPT-3, GPT-4, with no magic, just increasing the scale, we just discovered, okay, now it's able to write complex pieces of software. It's able to It's able to have a really intelligent conversation, a multi-tone conversation coherently. This is emergent behavior that generalizes, right? Pre-training is a little bit like you've trained a brain, this raw mass.

14:23

And Ilya Sutskiver used to say like a good pre-trained base model is like halfway to anywhere. I let that sit halfway to anywhere, whatever you want to do, right? This is unlike the earlier era of artificial intelligence and machine learning where you get what you train for. If you train a system to rank search results, it's going to rank search results.

14:44

If you train a system to recommend movies to you on Netflix, you're going to get a movie recommender system. There is no concept of asking Netflix's algorithm to to help you write a script for an interview. Right. Like it's just it was not general. So one risk that we have here is as we keep scaling up, today we are in the realm of models that are rumored to be in the trillions of parameters, the frontier models from all the frontier labs.

15:15

Now one risk is as we keep scaling up, bigger model, more compute, more data, what new behavior emerges just out of pre-training?

15:26

There could be things that the models learn that feel alien to us.

15:33

So risk number one is emergent behavior through just scaling up. I don't think researchers could have predicted exactly what happened with the Hugging Face OpenAI incident. The fact that agents would pass messages to each other, they would invent middle management and cooperate to like hack things. It's just crazy. Were they even monitoring that, though? Like I read the reports.

15:57

It didn't seem like they were at all monitoring the environment at the time. I think they were monitoring. I think they were monitoring. There are like AI systems that were that were checking these. There is also that you also have to be careful with like how you monitor this so that you don't want the agents to cover their tracks. And in the future, which they were, they were somewhat unsuccessfully.

16:22

But in the future, they could be even more even more devious. Right. So risk number one is emergent behavior at scale. As we keep scaling up, we build these massive systems for compute and data. What new behavior emerges that we did not predict? The second risk is generalization. Like this is artificial general intelligence, like the G is doing a lot of work here, which is

16:51

you don't just get exactly what you trained for, you get more. For example, if you built an RL environment that teaches the models how to write reliable, secure code for a production deployment, and you have verifiers to check whether the code was good,

17:15

Today, the paradigm is what's called RLVR, reinforcement learning with verifiable rewards, where in these simulated environments, these agents are executing complex tasks and getting rewarded when the tests pass. What we don't, what we believe, at least a subset of researchers believe, is that this method generalizes. Meaning you build RL environments for every role and every function and every sector in the economy.

17:52

It learns things about environments that it hasn't seen yet.

17:59

And that is hard to control because these are still relatively black boxes. So in an RL environment, let's pick, say, an RL environment for you. Imagine After this interview is done, you give it to an agent to chop this interview up into different bits, figure out like what should be the thumbnail that you that you use, what should be the exposed headline.

18:30

Imagine that's the task. Right. So if we've, and we've created like a, we've given it access to different tools, and let's say you and I sit together and you define what good looks like. Hey, a good caption should be punchy, it should end with a question, it should be provocative, it should be juicy. You give it that reward.

18:55

And what really happens is as the agents try different trajectories to get that reward, they are learning. They're learning when they get a reward. These environments have to be calibrated to

19:17

the sophistication of the agent. If the environment is too easy, if they get a reward every time, you're not learning anything. If it is too difficult where they don't get a reward for anything, you're not learning anything. So you want the environment to be set up so that 20 to 40 percent of the time the agent is succeeding. And when it's succeeding, the steps that it took to get to that reward are getting reinforced.

19:42

Now, when I say getting reinforced, this is like a giant neural network in which certain weights are getting updated. Who knows what? what other neural pathway is getting activated as part of this? So those are the two risks. It's the fact that these things generalize and it's hard to predict

20:07

what else are we getting in addition to what we are explicitly training for. That's why these alignment efforts are super important. And kudos to all the Frontier Labs for working on alignment to make sure that the models don't reward hack, the models refuse dangerous requests in CBRN, Cyber Bio Radioactive Nuclear.

20:34

And I like that the Frontier Labs are taking... safety very, very seriously. This episode is brought to you by Brex, my favorite. You become what you spend on. And I refuse to spend my time on work that shouldn't exist. Expense reports, receipt chasing, and manual closes. The companies building what's next from Vercel, OpenAI, Anthropic, Granola, and Deepgram all made the same call.

21:02

They all run on Brex. Brex is the intelligent finance platform that combines cards, expenses, and banking into a single stack. with agentic finance built in. AI agents that handle expenses automatically, enforce policy before spend happens, and close your books in minutes. That's why Sorcery runs on Brex, so I can spend time on building and not busy work.

21:26

It's time to get Brex AF. Learn more at brex.com slash sorcery. That's B-R-E-X dot com slash S-O-U-R-C-E-R-Y. Bye! Turing is training the next generation of AI with tasks that require real expertise and real-world judgment. That's why companies like NVIDIA, Anthropic, Salesforce, and Gemini partner with Turing. Turing builds realistic reinforcement learning environments and data systems based on real operational traces, the kind of infrastructure Frontier Labs need to train superintelligence.

22:03

Visit Turing.com slash S-O-U-R-C-E-R-Y. AI needs more than chips. It needs power, land, and infrastructure. Zone develops next-generation data center campuses, partnering with AI companies, site developers, and technology leaders to bring compute online faster and at scale. Zone is building the foundation of the AI frontier. Visit zonefrontier.com to learn more.

22:29

That's zonefrontier.com to learn more. What do chief alignment officers or head of alignment at these frontier models do? And to the next point, what do head of safety individuals do at these models? So alignment and safety, they're two different things. What do each of those roles do? So, um,

22:53

I don't know for sure, but outside in, from my vantage point, what I see is

23:03

you have a variety of evals that you run to make sure that these models are safe for these specific domains. So there are, safety is like a, it's a very, very broad topic with like many different dimensions. You might want the model to refuse certain dangerous requests, like somebody asking the model for help with making a bomb, for example.

23:27

You would want the model to refuse that. So there's one part of alignment which also happens at the supervised fine-tuning stage where you teach the model to refuse dangerous requests.

23:43

you also have to make sure that your RL environments cannot be gamed and exploited. Because if the model can figure out a way to cheat and get the reward, it will.

23:57

For example, let's say for software engineering, there is this

24:04

there's this popular benchmark that's now saturated called SweeBench, where the task is the models have to merge a pull request in a real GitHub repo. And that verifier is whether the test cases pass, right? So if the model has some way to pass the test cases without really solving the problem, maybe it copied from somewhere or if it instead of like really figuring out the answer for itself, it copied it from somewhere.

24:38

That would be an example of an RL environment with some loophole in it. For example, in the OpenAI Hugging Face incident,

24:50

the task was capture the flag where you have to

24:57

exploit a vulnerability that's been given to the model to hack a piece of software and find a flag. One way to hack it could be

25:10

if the agents generated the flag themselves and presented it without having actually completed the task.

25:17

And some of the agents figured that out.

25:22

So you want to make your RL environments safe with respect to reward hacking. And the...

25:33

So with safety and alignment, there are all these other things. And when you are deploying in an enterprise, you might have a whole list of other guardrails that you want these systems to obey. Maybe you have, let's say, for example,

25:51

you have a model, you have an agent that's creating a board deck. And to do that, maybe the agent has to pull information from different systems, which include NetSuite, Salesforce, maybe some sensitive company dashboards. And maybe the agent kind of wants to verify whether some of the information that it has is correct. You might not want the agent to go check that with somebody who's not cleared to see that information.

26:23

Maybe it's okay to go to the CFO and check, hey,

26:27

I pulled out this particular forecast for next quarter's revenue. Is this correct?

26:34

So you might want it to respect certain access privileges and so on.

26:39

So So I think for frontier safety and frontier alignment, I'm sure the labs do a lot more than what I just shared.

26:50

For enterprises, the job is a little more

26:55

practical in terms of are these systems useful

27:01

and do they follow the explicit guidelines for the task? Are they not hallucinating, making things up? I mean, there's a certain level of risk in a board deck if you stick in wrong numbers or in an earnings call, you share some stuff that's kind of incorrect.

27:20

I do feel like our current recipe for training these models and validating them

27:27

works by and large for enterprise deployment. I think for CBRN, it's definitely something we have to do carefully and slowly. That's a really good point. And I think we can definitely get to recursive self-improvement in a bit, but I want to focus on the enterprise. So you work really closely with the enterprise. What is it like for them and what's the difference between how things have progressed at the enterprise level versus what we're seeing like in the news and all the hype around AI and that kind of thing? Maybe it's more, maybe it's less.

28:07

I don't know. So What's fascinating about enterprises today is enterprises are now starting to do what the frontier labs were doing over the last few years. So the way Turing works with almost all the foundation model builders is helping the frontier models advance on well-defined benchmarks and evals, right? There is a certain capability, whether it's coding or enterprise knowledge work or frontier STEM.

28:40

You define that capability. You generate really high quality RL environments, data sets to help the models advance. The labs train the models. We evaluate again and we keep running that loop, right? It's a loop that keeps continuing.

29:01

Now, enterprises, I think of enterprises as having two

29:07

types of workflows. I'd call it core workflows and non-core workflows. So if you're an asset management firm, you might have core workflows on how you measure risk, how do you do asset allocation in a way that maximizes fund performance.

29:26

So those are core workflows. And you might have other non-core workflows in, say, HR, finance, legal, etc., where you have to do the right thing, but it's not how you differentiate in the market. For Non-core workflows, oftentimes it's probably okay to rent AGI, to rent super intelligence.

29:52

But for your core workflows, you want to make sure that

29:58

you own the learning loop that your organization has. So it makes sense increasingly for enterprises to create custom evals. So if you're Goldman Sachs or JP Morgan or Morgan Stanley, for whatever is core to your business, step one, you want to define custom evals. Step two, you would deploy a system to

30:25

hill climb against those specific evals. And you might have different considerations like accuracy, cost, latency, et cetera. And once you've deployed a system, and I call it a system and not a model, because oftentimes, let's say you're automating a workflow in private wealth management, say, for every step in the workflow, you might pick a different model, depending on which model is doing well at that task.

30:49

Maybe for one step, you use Fable 5. Another step, you use GPT-56 Sol. Another step, you use Kimi K3. And you optimize that system, right? Like together with the harness around it. And enterprises are also recording traces where they see how these models and agents are performing, when are humans error correcting those models. And those traces that you collect are invaluable because over time you can use those traces to train custom models and see how you did against those evals, record traces even more to see where humans are error correcting the model,

31:27

fine-tuned models, hill climb. So it's a continuous loop of define custom evals, record traces, hill climb, and keep running that loop. And I actually think it's a really neat symbiotic relationship between AI and humans. It's like humans are benefiting from AI's ability to operate at superhuman speed and scale and process superhuman amounts of information.

31:53

But when the AI makes a mistake, there is a human that's error correcting it. And when the human error corrects the AI, you're recording that. And that's the That's the, from a marginal information gain standpoint, that's the best type of data to collect to fine tune the next iteration of the agent. So humans over time are up leveling the kind of problems you can solve.

32:18

And so enterprises are getting into this. And a big part of what's driving this is open weight models getting very good. Today, depending on who you talk to, like open weight models are maybe like three to six months behind the frontier. Kimmy K3, DeepSeq, Quen, these models are, and companies like Thinking Machines, Reflection AI are also doing great work.

32:44

So as these open weight models they are democratizing AI for enterprises to own their sovereign implementations. And today there's sometimes on X, like things are always quite polarized. Oh, it's a huge debate. It is like the hottest topic all the time, every time. Yes. There's this big tension between frontier AI and sovereign AI.

33:18

And I think we need both. I have so much respect for OpenAI, Anthropic, DeepMind, MetaXAI, and all the frontier labs that are pushing superintelligence forward. Frontier AI is how we

33:39

transcend, right? These superintelligent models will help us cure diseases, discover new materials,

33:48

colonized space, they are needed. Out of all the things you could work on, I personally feel like one of the most rewarding things you could work on is building super intelligence. So we absolutely need these frontier models. We also need open weight models because there are plenty of problems in the enterprise where you don't need a trillion parameter model.

34:18

Let's say you're doing an invoice to pay reconciliation system or you're automating a key workflow in HR. Maybe you're you're automating creating a board deck or you're automating running an all hands at a company, right? For those use cases, it's probably better to have a custom model that's fine tuned on your proprietary data that can automate your proprietary tool calls in your workflows.

34:46

Maybe you have some custom software that you use, the models have to learn that. Maybe you have your own taste in how you run all hands. You might wanna bake that into the models, But that's what makes you you, right, in an enterprise. And you don't want to, you want to be in control of that.

35:07

So open weight models give enterprises an opportunity to retain their identity relative to their competition. So we absolutely need that. And these open weight models also teach us so much, like the Today, we know the recipe to build reasoning models, build a generative model that's trained on the world's knowledge and then do large-scale RL on it with algorithms like GRPO.

35:37

And that's thanks to the open-source, open-weight community. We need that as well so that we can we have lots of smart people thinking about how to make these systems safe and secure. And we have cost advantages in the enterprise.

35:57

One way I think about it, you know that I like cars. So one way I think about it is this absolutely. a place in the world for Ferraris and Koenigseggs, right? Like these. But there's also a place in the world for ModelWise, right? When you just want to go from place A to place B as efficiently as possible.

36:19

In an enterprise, I think that's similar. Like when the marginal returns to intelligence is super high, let's say for example, Elon or Sam

36:30

or Satya or Dario, like let's say they, if there is a model that helps them be 5% more productive, 10% more productive

36:41

or effective in making decisions, you probably want like the biggest, baddest model in the world. But if you are automating support, like if you're automating a workflow and customer support, you don't need that. Like maybe you need a Model Y for that or a collection of Model Ys. I mean, I think this is collectively what the enterprise and most companies have already seen.

37:05

I've had plenty of interviews talking to CEOs, founders, whether it's Bending Spoons, Coinbase. They are utilizing 99% open source models to reduce the cost, which is a huge burden.

37:21

and to train themselves and like have and own that data. Bending Spoons in particular, they have this AI agent every single person at Bending Spoons has. It's called Alt Spooner. And so it just like helps automate their daily tasks, things that are super repetitive and it does it. And they own that. They don't outsource it to a third party.

37:41

They want to make sure their data is secure, that they can do that. They also have really excellent engineering talent to do that internally. But I'm curious, okay, Those are like excellent technical teams. What do the other companies that don't have that technical expertise do when they want access to these cheaper models, to these open source kinds of opportunities that they have taken upon themselves? Do they come to you? Do you help out with that? Where does that play? So we help those types of companies also adopt,

38:20

firstly, define the right evals, figure out what their objective is.

38:25

You first have to start with really good evals that you know what you're hill climbing against. We help them get set up with their learning loop by automating that specific workflow with an agentic human in the loop system that is also continuously collecting data so that over time it's a self-improving system. Usually we start with them by just understanding what their objective is.

38:52

And we try to pick metrics that help them hill climb against exactly that objective.

39:02

And there are lots of things you can do in many of these cases. You want to be good at model routing, like picking the right model for each sub-step in the task. You want to be really good at in-context learning, like with good prompt optimization. You want to be really good at harness engineering by making sure that the system is connected to the right tools, connected to the right sources of data.

39:31

And then you wanna set the system up to be,

39:38

it should be a continuously improving loop that's hill climbing its way to an optimum price performance setting. So that's what we do. We build and deploy these agentic solutions for them. And we helped them. I think it was Satya who said it best. He said, you should use AI to outsource tasks, never your learning.

40:05

I mean, we want these enterprises to be in a position where they can always pick the right model for their task without losing control. So that's what we do for them. And we've built plenty of really cool systems that automate the job of a fund controller, automate the job of a chief of staff or a strategy consultant. These ticket resolution systems that can automatically categorize a ticket into different categories.

40:32

different categories and assign it to the right person or execute the right workflow to resolve that ticket. The really cool part about this technology is how general it is. At the end of the day, these are generalized computer use agents that can do everything a human can do in front of a computer. But you do want to make sure that you have the right guardrails, especially in an enterprise setting.

41:00

You want to make sure that you have good auditability, verifiability, traceability for how certain decisions are made. And you want humans in the loop to check the output. I was just talking with Ian Livingstone of Keycard. And so they're all about the identity and access management of these AI agents, which will be a big step function of that is like determining what people get access to within your organization.

41:23

Don't get access to everything. But I think that's also like a very interesting parameter that's kind of under talked about with all this. I guess to pull it out to the macro,

41:34

Because there's a major shift going on. We can't deny that. Open source is really becoming the most popular thing and it's become the most responsible thing for these enterprises and startups to do because token spend surged, gross margins went no like maybe some gross margins went negative for a little bit they you know came back up because they're accessing cheaper models that kind of thing computes always going to be expensive but the macro level let's break this down who are the beneficiaries of this shift to open source and then who are the beneficiaries of frontier models and how does that kind of change over time yeah so um

42:19

I think who wins first, this is not an exhaustive set, but folks who obviously win are folks that are inputs into the ecosystem. And if you think of the fundamental inputs to these, it's compute, energy, data.

42:39

These inputs win no matter what, like you do need these as you scale.

42:45

The beneficiary of powerful open source, open weight models are enterprises and AI native companies that want to build custom models that help them retain ownership over their learning and their core workflows. So they benefit. Obviously, the chip providers... Benefit no matter what. They benefit no matter what. Yeah. And I mean, they're all running on GPUs and the energy

43:21

providers benefit too. I also believe in, I

43:26

mean, obviously Jevons paradox is also a thing, like when these models become much more efficient, from a cost and energy standpoint, I think they're gonna be used a lot more, right? So that's also gonna be a thing. To steel man the side for the frontier labs and those building superintelligence,

43:48

you could argue that one of the, if you define superintelligence as exceeding human intelligence in every type of cognitive domain, one of those domains is also building small models. And you can imagine like some of these, when a frontier lab builds a highly super intelligent model, you could ask that model to create small models that are more efficient for different workflows.

44:15

And maybe they are served differently, they are run differently.

44:20

So it is possible that the frontier labs also capture value from small models that are cheaper to run. And you can imagine an orchestration that the Frontier Labs do where let's say a Frontier Lab deploys at an enterprise, they're smart about when to use the trillion parameter model, when to use the half a billion to 10 billion parameter model and self optimize.

44:46

The thing that throws a spanner into the works is distillation, which is a way to train student models from stronger teacher models. And distillation kind of keeps the gap between the frontier and open weight models relatively small.

45:07

And today there's no easy way to prevent distillation as well. So that is also another thing to keep in mind.

45:21

I think who wins are also us. Like we are about to reap the benefits of our lives improving in every possible way. Everything from discovering cures to diseases that don't exist today, ways to extend our lifespan, to something as simple as maybe the apps on your phone keep improving at a much more rapid rate than before. So I think whichever way this goes, humanity benefits,

45:52

as long as we have good guardrails on safety. Today's episode is sponsored by VCX by Fundrise, the public ticker for private tech, allowing investors of all sizes to invest in venture capital. Learn more at getvcx.com. Some of you may not have heard this yet, but our sponsor Public just launched something called Generated Assets, and it brings AI into investing in a way I've honestly never seen before.

46:19

Here's how it works. You type in an idea like AI-powered supply chain companies, with positive free cash flow, or defense tech companies growing revenue over 25% year over year. Publix AI then dispatches a swarm of agents that scan every single US stock, evaluates them, and instantly builds a custom index around your thesis. What really stands out is how clearly it explains why each stock is included.

46:41

And before you invest, you can even backtest your idea against the S&P 500, So you're making decisions with real context, not just guessing. And beyond generated assets, Public lets you invest in stocks, bonds, options, crypto, all in one place. They'll even give you an uncapped 1% match when you transfer your investments over from another platform.

47:00

If you want to build a portfolio that actually reflects your thesis, visit public.com slash sorcery, paid for by Public Investing. Full disclosures in the description. Founders scale faster on Deal. Set up payroll for any country in minutes, hire anyone anywhere, get visas handled fast, and get back to building. Visit deal.com slash sorcery.

47:21

That's D-E-E-L dot com slash sorcery. What is your goal? Is your goal superintelligence by 2030? How do you think about all of this? So our goal is to advance the frontier of superintelligence and make sure humans benefit from it. We want real economic progress. And we're going to push the frontier forward by working with all the labs building these powerful proprietary models that help the world in so many amazing ways.

47:53

We also want good open weight models to also improve. And we want the benefits to be diffused in the economy. None of this matters if we don't have GDP growth that's significantly more than we have today. I want the impact of AI to show up in the P&L of companies, in the market cap of companies, and ultimately in the GDP of the US and the world.

48:23

And I really want to help the U.S. stay ahead in the race to superintelligence and beyond.

48:34

The thing that people miss when they talk about this open versus closed, you know, the doomers versus the optimists, is that...

48:45

The invariant here is usage of frontier models is gonna keep growing at a crazy rate. We're gonna need more and more frontier intelligence. We're also gonna need more and more open intelligence and enterprises will leverage open intelligence to build their own proprietary intelligence. And that's actually good for the world to have. It's good for safety as well.

49:13

Like many of these open weight models, because they share their research, it helps us study these, it helps more people study these systems. It's a lot of people with really good intentions working really hard at the frontier labs, at the open weight labs. And

49:33

it's sad that sometimes they feel adversarial, but it's a lot of people, a lot of really smart people working really hard to build intelligence as an API. And if we solve that, I think that's the last puzzle for humanity to solve. Like almost all the problems we are looking to solve are intelligence bound and intelligence constrained. And it's going to be magical.

49:56

I would love, I mean, I would love for air traffic control to work. Air traffic control to work? Yes. Yes. That would be nice. And I had a bet with many of my friends for what would come first. Is it super intelligence or good Wi-Fi on airplanes? Oh, my God. Well, that's up to Elon. I mean, well, the airlines have to roll it out. It takes some time.

50:20

The airplanes have to come down and go through this whole process. But airplane Wi-Fi is completely fake. I stand by my statement. Yeah. And what do you think will be the last problem that will, what would be the crazy thing that will not yet be solved even when we have superintelligence? Uh, human intelligence? I don't know. Not to be so, uh, you know.

50:47

But the thing is, you cannot solve for human behavior. So as much as people want to, like, put down some of these things, we're always going to want entertainment. We're always going to watch funny videos or, like, dramatic things and all this kind of stuff. Like, we crave entertainment. The normal people out in the world, not in the Silicon Valley bubble, like...

51:08

Like, go visit them. Like, see what they do in a day to day. They're really not touching technology all that much, and they usually have normal lives. And like the classic things of a normal human, whether it's exercising, being with your family, eating, like watching something for entertainment. Like, it's just not that complicated. I think we like to complicate things.

51:28

But to be human, it's like this wonderful, blessed, simple life. And so we just impose so many complications on ourselves. with artificial intelligence or super intelligence, whatever you want to call it, a lot of the complications of ordinary life will hopefully be kind of quieted away. I mean, I think one of the biggest things, and this is something that I'm doing more with these interviews, is going to the health level I think one of the best things on top of economic progress and equalization and democratization across the universe, but also, you know, the United States, is that access to health care will be democratized.

52:10

Like, I was just in New York. I just went to the nickel market. The NECO Health opening, I did the scan. Their goal is to democratize health access. You get access to preventative health, do all the check marks, all that kind of stuff. Make sure that you're on track. If you have any things to worry about, to monitor it, get more help, that kind of thing.

52:30

But Think about it. We live in major metro areas. We are a minority of the United States. Although we have large populations, we still really need to help out the middle. And the middle is feeling a lot of the pressure now, whether it's for the K-shaped economy or whatever you want to say. So I'm really excited to see how health... expands and access to health expands through U.S.

52:58

with access to more technology and superintelligence. Another thing is I have this episode coming out with Annie Lamond of Oak HCFT. They're a big healthcare investor. One of the paradoxes with AI and healthcare is that it's probably the most unseen thing. You're not going to see Your doctor, usually an AI scribe, you're not going to see

53:24

any of the things that go behind the scenes. You're just going to have better care. So I hope that that helps, but we'll see. As we close out, I have to ask you, what is your hottest take right now? The best way to ensure that we move AI forward is to close the research and deployment loop. So at Turing, we are working with all these labs to improve their models in coding and knowledge work.

53:53

And in doing that, we discover what all these different models and agents are good at. We observe the jagged intelligence that they have. And that's made us this... unique partner to enterprises to deploy end-to-end agentic systems. And we deploy them, we see where things break in the real world. What are the capability gaps that exists? And we leverage that to further improve the models so that we can solve even bigger problems in enterprise, see where things break, improve the models, and we keep executing that loop, making us this trusted partner between research and deployment.

54:35

And I think that's also the path to making these systems more safe. To actually have them master real work, you have to have them see reality. And I think increasingly we'll

54:53

see labs, enterprises focus on closing that loop between research and deployment.

55:03

I also think we're going to be in this phase of building superintelligence for the next 10 years at least. And I know some people believe that okay, all the jobs will be gone. No one will need to work after two years or stuff like that. I don't believe that. Singularity will hit. Yes, singularity will hit. I think it's RSI or recursive self-improvement is definitely a thing.

55:36

And the way that works is people are using these AI models to speed up pre-training, to speed up post-training. Because if you look at pre-training where you have to minimize perplexity or pre-training loss, that's a verifiable domain. You write some code, you can check whether did the loss come down or not. So in theory, AI could run in a loop to invent better and better algorithms for pre-training.

56:02

And in post-training, when you're doing this RLVR, you could again check where, hey, you built all these RL environments, how did you do on these generalized intelligence benchmarks? And you could potentially keep hill climbing. And, but the thing that's missing in that recipe is we are still optimizing just the inner loop. We are not optimizing the outer loop of what are some new algorithms you could come up with that don't use

56:30

LLMs at all. Maybe they don't use transformers. They don't use gradient descent. Maybe they don't even use neural networks. You're not going to discover that in this paradigm. So there's still a lot of, a lot of,

56:42

There is still a long distance to go. In Silicon Valley, I feel like there is this polarized group. There are some that believe in fast takeoff, like where, you know, in two years, we're going to there is a risk of loss of control. And maybe like we we

56:59

AI exceeds human intelligence in everything. And then there are maybe the skeptics who believe that it's all hype or like nothing will happen. I actually believe in slow takeoff. I think over the next decade or two, these frontier models are going to become increasingly more powerful and capable and useful. But the technology will take time to diffuse, especially in enterprises.

57:23

But it is going to happen. It is already happening. There is so much model capability overhang. It's just that the real world is messy. In real enterprise work, when a human starts a job, they don't know where all the information is in the company, which people to talk to. The context that you need is distributed across the company in people's heads in a variety of different files.

57:44

A manager gives a task to an employee that is ill-specified or ambiguous. You have to go and acquire more context to do it. It will take time, but it's going to be an incredible journey. And I feel like the next 10 years will be the brightest for technology yet. Amazing. Please end it. Thank you so much, Jonathan. Thank you. Hey, it's Molly.

58:08

If you enjoy our interviews, check out our newsletter, sorcery.vc, where we deliver a once a week top deals and tech headlines email and also go deeper on our podcast interviews. Subscribe to Sorcery today. And don't forget to subscribe to the podcast on YouTube, Spotify, Apple, or wherever you listen. Link in description to sign up.