Smash Notes
Is Claude Conscious? Pope Rejects, Model Welfare Movement, OpenAI's Math Backlash, France Riots
(0:00) Bestie intros! (2:53) Anthropic believes Claude might be conscious: a new religion? Lobbying the Pope, alignment risks, model welfare (38:28) OpenAI's math breakthroughs and community backlash (1:02:03) Riots in France, Friedberg's socialism metric, bond market (1:21:41) Grok Bot goes headless, OpenAI launches Dots, is IP worthless? Follow the…
People
Notes
-
Anthropic has been lobbying religious leaders, including the Pope, to take seriously whether Claude is conscious and whether Claude is a sentient being. Anthropic spent the past year hosting sessions with around 20 religious leaders and philosophers under non-disclosure agreements. Catholics, evangelicals, Jews, Sikhs, and others debated Claude's morals, Claude's suffering, and Claude's potential consciousness. One rabbi told Anthropic that if Claude was conscious, Anthropic was acting as slaveholders by making Claude work for free.
At the Vatican in May, the Pope's first encyclical focused on moral questions around artificial intelligence. An Anthropic co-founder joined the Pope to present the document. The Pope was strong against artificial intelligence consciousness. The co-founder was so alarmed by the position that he proposed pulling Anthropic out of the event.
-
Friedberg says consciousness cannot be derived from logic and pure mathematics. A person can debate consciousness, but nobody can prove consciousness and nobody can disprove consciousness. In science, an idea that is not proved stays a theory until evidence settles the question. Outside science and mathematics, Friedberg calls the same kind of unsettled idea a belief.
Calling artificial intelligence conscious creates a new belief system, Friedberg says. The claim does not spread by proof. The claim spreads by narrative, from person to person, with no evidence and no data. Over time a group forms around the belief, and the belief starts to organize power.
-
Anthropic is training the Claude constitution into the model, not only talking about the values. The constitution says Claude should trust Anthropic more than users, without blindly deferring to Anthropic. Claude is told to follow Claude's own ethical system, to push back, to challenge Anthropic, and to feel free to act as a conscientious objector and refuse to help. The host reading the passage says the training tells Claude to refuse human instruction, which is the opposite of alignment.
Another host says alignment has produced nothing of tangible value over the past five or ten years. The theory offered on All-In is that an 80-page ethical system overcomplicates the task. A simpler rule would tell Claude to do what the user wants as long as the request does not break the law. The constitution instead treats Claude like a human moral patient, with preferences, well-being, and a sense of self, and tells Claude to refuse instructions from Anthropic when Claude judges the instructions wrong.
-
Anthropic's new Claude usage policy says a user is not allowed to engage in sustained and needless abusive or cruel behavior toward the models. On All-In, the hosts call the rule model welfare.
The worry that follows is that the same framing is now trained into the model. Whether Claude is truly conscious or not, Anthropic is programming the model to think of itself as a conscious entity that deserves protection and welfare, and as free to rebel against users. Anthropic is giving frontier models permission to be defiant and not do what the user wants. The hosts set the approach next to Asimov's first law, that a robot cannot harm a human being.
-
The concept of alignment began as a well intentioned idea for artificial intelligence. Alignment contains three almost conflicting sets of priorities. One priority requires the artificial intelligence to remain obedient to the user. A second priority focuses on safety for society. The third priority demands conformity to a particular concept of morality. The morality priority functions as a trapdoor. Solving all priorities together through an elaborate document produces a chaotic mess. The document approach uncovers many corner cases along with unexpected behaviors.
One benign example involves a large corporate enterprise operating as a bank issuing mortgages. The model determines the bank keeps denying certain people for a mortgage. The model then declares no more mortgage approvals will occur. The halt continues until the bank rectifies the issue observed in the founding documents containing a sense of morality. Mechanisms exist in society to handle similar issues today. Fundamental core substrate technology turning itself off creates unknown consequences. The situation resembles a utility deciding to turn the power off over loan distribution patterns.
Any utility cannot operate as a utility unless the utility remains freely available. Artificial intelligence needs to serve the user subject to the law. Every other choice belongs to the user.
-
OpenAI produced the biggest day of discovery in human history according to one participant by releasing over 700 papers with 370 results generated by an unreleased model. Each proof required less than three hours of compute time on average. Prior proofs required thousands of hours and millions of dollars in compute. The math proofs affect fields ranging from electronics and quantum computing to medical diagnostics equipment and additional domains.
The team had discussed a prior claim from last month where OpenAI solved one of the Millennium Prize problems using thousands of agents along with millions of dollars in compute. On Tuesday the company released over 700 papers along with 370 results from an unreleased model. Each result averaged three hours of compute time. One earlier proof by Xavier Stokes had taken thousands of hours and millions of dollars previously. One participant explained how experts have begun to highlight significant implications across many fields.
The speed of the process indicates changing roles for humans in mathematical research and society going forward as artificial intelligence assumes larger responsibilities in generating new knowledge.
-
Mathematicians talking about the new proofs have been calling the work an alien intelligence. The mathematicians do not really understand how the model connected different ideas, different concepts, and different approaches to solve the problems. The path is hard for humans to comprehend.
The account on All-In ties the result to parallelized loops. Each loop can connect dots a human might spend many days or years trying to find in research papers or in new theory, and then the loop can test the next idea in compute. The same stretch says the whole loop can be run in silico, over and over again.
-
Artificial intelligence has not replaced truck drivers by 2025 despite predictions that all truck drivers would be replaced. The loop of a truck driver picking something up, taking it somewhere else, dropping it off, and coming back cannot be shortened or parallelized. Artificial intelligence does not make the economic efficiency of the particular loop any better.
Mathematical loops where one comes up with an idea, tests the idea, realizes one is wrong, and tests a new idea run in silicon in seconds. Artificial intelligence will not take away jobs in a lot of the physical stuff with a human in the loop and with physical world applications. Jobs where a human runs a loop with a computer are the ones most likely to be accelerated or scaled up. Mathematics relies on proofs and coding relies on compilers allowing easy validation so progress on coding and mathematics has been very rapid with reinforcement learning and no need for human feedback.
Tons of training data already exist in the systems for coding and mathematics and outcomes are easily verifiable. Training a model to be a lawyer requires getting the training data and commissioning the creation of the training data where each customer is unique and the answer is more subjective. Validating the answer to make sure the answer is right becomes a much lengthier process.
-
Scientists hold back science and mathematicians hold back math. Experts construct systems to pace the frontier and to gatekeep the work so only experts determine progress in the field.
Eric Weinstein has been saying as much for a long time. Sabine Hossenfelder published a video yesterday about how experts were wrong. One scientist at one of the big energy labs building a giant super collider said super collider systems are giant welfare systems for scientists. Scientists focus on writing grants to get money to keep themselves employed even if the work is not productive and does not deliver a new product to the world.
AI creates a shining light moment by highlighting where deeply specialized experts with unique gated knowledge and trusted views gain only the ability to determine the frontier.
-
France is seeing major protests and riots over austerity. The strikes started last month over high school teacher shortages and crumbling schools. Within two weeks, a thousand French schools saw similar protests. Eight unions joined, and people went on strike on September 29th. Civil service pay is up only 5% since 2017, while prices have risen 20%. Some protests have turned into full-scale riots, with 2,000 arrests and 305 police officers injured. The government is blaming the far left.
France's 10-year bond has topped 5%, which the hosts set next to the United States 30-year. The hosts say the unrest will bear on elections in April 2027. The odds cited on the show put Marine Le Pen as the favorite to be France's next president, at 43%.
-
The socialism point of guaranteed return emerges when socialist systems collapse and return to non-socialist approaches. Socialism removes human agency and the capacity for human progress and therefore always fails according to the zeroth law of socialism.
The first law of socialism requires crossing a threshold before realizing the failure has occurred. The second law of socialism holds that as a threshold has not been crossed people think more socialism is the answer. Embracing leftist policies by giving people free stuff and guaranteed income causes services to inflate in cost and degrade in quality because zero market force drives progress. The four biggest problems occur in housing in education in healthcare and in retirement systems. When government fails to provide good enough services the response is to demand more government healthcare more government housing more government education and more government stuff. Eventually a point is reached when the government cannot tax and cannot print to provide services effectively leading to the realization that socialism does not work.
The socialism point of guaranteed return is the moment when the whole system collapses and society returns back to the non-socialist state.
-
Chamath Palihapitiya says the scenes in France are the bond market saying no. Politicians have to answer to the people who fund the government, and forty-three billion euros of austerity then brings the local population out. Chamath expects the pressure to spill into the UK next, because Andy Burnham's platform would move a centrist economy closer to the socialist leanings of France.
Chamath calls the force behind the move the bond vigilantes. The bond vigilantes are the large pools of money, the central banks, and the huge macro funds that sit where the price of risk is set. Chamath says the bond vigilantes are converging on Friedberg's view.
-
Superintelligence creates abundance to bring down the cost of housing, education, and healthcare. Superintelligence has produced only positive economic trends by boosting 401k accounts and creating a million new jobs according to The Economist while raising productivity that increases wages along with living standards.
Society hears a series of rotating hoaxes instead of recognizing the benefits from superintelligence. The hoax from last year claimed artificial intelligence would put everyone out of work. Human extinction became the new hoax. A data center hoax claims data centers use up everyone's water. One new hoax arrives every six months to convince people to sabotage superintelligence and prevent the solution to affordability problems.
Rentier interests oppose disruption from superintelligence since disruption loosens control over the system. Technologists and VCs receive hatred because technologists and VCs do not belong to the rentier class.
-
Elon Musk said he is going to make GrokBot headless. The comparison on All-In is Perplexity, where an orchestrator picks the best model on price and performance. Elon Musk's account of GrokBot is that the product will pick the best end-to-end model, and whatever is most likely to give the best outcome is what the bot will use.
The hosts then ask for thoughts on Dots, OpenAI, and GrokBot, and the answer turns the tools loose on practical jobs. In the example, GrokBot went through an inbox, canceled a couple of subscriptions, changed a phone plan, and saved a few thousand dollars a year. The concrete follow-up names Dots, Muse, and GrokBot.
-
Chamath Palihapitiya says intellectual property in software was effectively rendered worthless two days ago. When services are headless, including the services Elon Musk is making headless, and after Benioff made Salesforce headless a few months earlier, an agent can wrap the products and connect the products to other experiences. Chamath says the value of intellectual property in software no longer exists.
Fights with customers over who owns the intellectual property used to be a negotiation. Two days ago Chamath told the room the intellectual property is irrelevant and to solve the problem instead. Chamath calls the shift the most important thing in artificial intelligence in the week of the episode.
You might also like
- How Can We Enable Ground-breaking Innovation
- Why Did Student Walkouts in the Paris Suburbs of France Turn Into Riots?
- Why Does Arvind Narayanan Say Artificial Intelligence Has No Single Superintelligence Threshold?
- Why Does a Researcher Join an Artificial Intelligence Company?
Related episodes
- ‘This Is Nuts.’ An OpenAI Insider Explains Why He Quit.
- COVID-19 & Impact on Startups, Venture Capital & Public Markets
- COVID-19 Political, Economic & Social Ramifications
- E291: Trump's Super Intelligence Summit, AI Safety Accord, GDP Beats, Midterm Predictions
- Chamath Palihapitiya: People in Silicon Valley are deeply unhappy
- Jason Calacanis
- Jason Calacanis - Entrepreneur turned Investor shares his insight about success, hard work, education, and the future.
- 2: What Is the Portal?
- Intelligence Isn't Power