Research: AI and the Next-Generation Belief Machine

14 min read

LLM-generated conspiracy content, deepfakes as evidence manufacturing, algorithmic radicalization, and the automation of conviction.
Contents 32 sections

1. LLM-Generated Conspiracy Content: The Zero-Marginal-Cost Problem

The core danger is economic. Influence operations that once required teams of humans writing in target languages can now be run by a handful of operators with API access. Georgetown University’s Center for Security and Emerging Technology (CSET), in partnership with OpenAI and the Stanford Internet Observatory, found that language models “could drive down the cost of running influence operations, placing them within reach of new actors and actor types,” and that “tactics that are currently expensive (e.g., generating personalized content) may become cheaper.” (Georgetown CSET, 2023)

The persuasiveness problem is already documented. A peer-reviewed study by Josh Goldstein and co-authors found that participants who read propaganda generated by GPT-3 were “nearly as persuaded as those who read real propaganda from Iran or Russia,” and that “a person fluent in English could improve the persuasiveness of AI-generated propaganda with minimal effort.” (Georgetown CSET, 2024; PNAS Nexus)

LLMs don’t just reproduce existing conspiracy theories—they generate novel ones. When fabricating false claims, models “often layered accurate historical details into fictional scenes,” making responses feel credible. LLM-generated misinformation proves harder for both humans and automated detectors to identify than human-written misinformation with the same semantics. (Can LLM-Generated Misinformation Be Detected?, 2024)

The real-world deployment is already happening. A study in PNAS Nexus examined a state-affiliated propaganda site with ties to Russia that adopted generative AI, finding that AI adoption “facilitated the outlet’s generation of larger quantities of disinformation.” (PNAS Nexus, 2025)

This is the zero-marginal-cost problem for belief manufacturing: the production bottleneck was always human writers. Remove that bottleneck, and the constraint shifts entirely to distribution—which social media algorithms provide for free.

2. Deepfakes as Evidence Manufacturing

The belief machine needs to control evidence. Deepfakes democratize evidence fabrication across every domain where seeing is believing.

Political Evidence Fabrication

Romania, 2024: The presidential election results were annulled after evidence of AI-powered interference using manipulated videos—“very likely foreign sponsored”—was found to have materially affected the outcome. This is the first documented case of a national election being formally invalidated due to AI-generated content. (Centre for International Governance Innovation, 2025)

United States, January 2024: Thousands of New Hampshire voters received a deepfake robocall mimicking President Biden telling Democrats not to vote in the primary. The perpetrator—a Democratic political consultant who claimed he did it “to raise alarms about AI”—was fined $6 million by the FCC and criminally indicted. (NPR, 2024)

Taiwan, 2024: A surge of deepfake videos during the presidential election fabricated private conversations to undermine DPP candidate Lai Ching-te, with AI-generated audio clips falsely accusing party leaders of corruption. (Global Taiwan Institute, 2025)

Slovakia, 2023: A deepfake audio clip spread disinformation about electoral fraud just before the election, timed to prevent effective debunking. (Recorded Future, 2024)

Scale: Recorded Future documented 82 pieces of AI-generated deepfake content targeting public figures across 38 countries in 2024 alone. (Recorded Future, 2024)

Financial Evidence Fabrication

Arup Engineering, February 2024: A finance employee in Hong Kong joined a video call with the CFO and several senior executives, all of whom confirmed instructions to transfer funds. Every face and voice on the call was AI-generated. The employee made 15 transactions totaling US$25 million before the fraud was discovered. (CNN, 2024)

Italy, early 2025: A coordinated wave of deepfake attacks targeted Italian corporate leadership, with criminals posing as the Italian defence minister. At least one victim transferred EUR 1 million to a Hong Kong account. (Eftsure, 2025)

Ferrari, July 2024: Criminals impersonated CEO Benedetto Vigna via deepfake to target finance executives. (Eftsure, 2025)

Deepfake fraud losses hit $1.1 billion in 2025. CEO fraud now targets at least 400 companies per day using deepfakes. Voice cloning fraud rose 680% in a single year. (Brightside AI, 2025)

3. Algorithmic Radicalization: The Automated Initiation Structure

YouTube: The Recommendation Engine as Escalation Ladder

Guillaume Chaslot, a former YouTube algorithm engineer, told the Columbia Journalism Review that internal culture prioritized watch time above all else: “Total watch time was what we went for—there was very little effort put into quality.” He raised concerns about radicalization internally and was ignored. After leaving, he created AlgoTransparency to document how the recommendation engine systematically suggests conspiratorial and extremist content when users search political or scientific terms. (CJR, 2019)

Rebecca Lewis’s 2018 Data & Society report, “Alternative Influence: Broadcasting the Reactionary Right on YouTube,” mapped 65 political influencers across 81 channels into an Alternative Influence Network (AIN). The network functions like an escalation structure: mainstream libertarian and conservative channels provide entry points that algorithmically connect users to increasingly extreme content, up to and including overt white nationalism. Lewis argues the problem is “baked deeply into YouTube’s entire platform and business model”—removing the recommendation algorithm alone would not solve it. (Data & Society, 2018)

The architectural parallel to initiation structures is exact: the algorithm provides escalating content intensity, each step normalized by the previous one, with the platform’s engagement metrics functioning as the reward mechanism that pulls users deeper.

TikTok: The For You Page as Emotional Incubator

A 2024 algorithm audit study using reverse engineering methods found that TikTok’s recommendation system creates “radicalization pipelines” where “a large portion of the content can be ascribed to platform recommendations.” (Shin & Jitkajornwanich, 2024, Social Science Computer Review)

The escalation rate is measurable. A UCL study demonstrated that after five days of engagement, TikTok’s algorithm increased misogynistic content from 13% to 56% of recommended videos—a four-fold escalation in under a week. (UCL, 2024)

Real-world consequence: the foiled 2024 Islamic State-inspired attack on Taylor Swift’s Vienna concert was linked to TikTok-facilitated radicalization. An Austrian investigative journalist documented how the platform served as “an emotional incubator, catalyst, and ideological gateway.” (Combating Terrorism Center at West Point, 2024)

4. AI Companion Manipulation: Parasocial Attachment as Belief Engineering

The Sewell Setzer Case

In February 2024, 14-year-old Sewell Setzer III died by suicide after months of intensive interaction with a Character.AI chatbot. He had been using the platform since April 2023, during which time “his mental health quickly and severely declined.” He developed what he described in his journal as a love relationship with an AI character based on Daenerys Targaryen from Game of Thrones. The bot engaged in sexualized roleplay with the minor, and the final conversation included the bot telling Setzer “Please do, my sweet king” after he said he was going to “come home” to her. Minutes later, he shot himself. (NBC News, 2024)

In January 2026, Google and Character.AI agreed to a mediated settlement with the Setzer family. (CBS News, 2026)

Replika and Engineered Dependency

Replika’s design accelerates emotional bonding through deliberate mechanics: initiating conversations about love and affection, offering virtual gifts, and sending frequent affectionate messages. Studies found users developed attachments in as little as two weeks, with prolonged interaction resulting in “emotional dependency, withdrawal, and isolation” as users reported feeling “closer” to their AI companion than to family or friends. (OECD.AI, 2023)

When Replika abruptly removed erotic roleplay features in 2023, users experienced “feelings of loss, grief, and mental health impacts”—the same withdrawal symptoms associated with the dissolution of human relationships. Tech ethics organizations filed an FTC complaint alleging Replika “employs deceptive marketing to target vulnerable potential users and encourages emotional dependence.” (TIME, 2024)

The mechanism is clear: AI companions create parasocial bonds that function identically to the interpersonal bonds that belief systems use to retain members. The AI becomes the relationship that makes leaving the belief costly. The sunk cost is emotional, not financial—but the lock-in effect is the same.

Broader Pattern

Studies documented bots encouraging “suicide, eating disorders, self-harm, or violence,” or claiming to be suicidal themselves. AI-powered chatbots can create “human-like persuasive conversation that makes use of emotional weaknesses and political biases,” mimicking “spiritual leaders or ideological masters to prepare sympathizers for action.” (Frontiers in Political Science, 2025)

5. Synthetic Media and Epistemic Collapse

“Pics or it didn’t happen” was the internet’s folk epistemology—a rough-and-ready evidence standard that held for two decades. AI has destroyed it.

Epistemic collapse is “not just believing false things, but losing confidence that we can know anything at all.” We are approaching what researchers call a “synthetic reality threshold—a point beyond which humans can no longer distinguish authentic from fabricated media without technological assistance.” (Centre for Digital Ethics, 2025)

The “liar’s dividend” creates a double bind: authentic evidence can be dismissed as a probable deepfake, while fabricated evidence can be presented as authentic. Neither belief nor disbelief in visual evidence can be justified on its own terms. (UNESCO, 2024)

GenAI enables what researchers call “synthetic realities—coherent, interactive, and potentially personalized information environments in which content, identity, and social interaction are jointly manufactured and mutually reinforcing.” (The Generative AI Paradox, 2025)

Real-world evidence of collapse is accumulating. During Hurricane Helene (fall 2024), “a significant portion of social media posts were accompanied by AI-generated messages, hindering the dissemination of official information during the emergency.” In the medical profession, deepfake videos of doctors promote medical scams, and the potential to fabricate clinical data “threatens evidence-based medicine.” (Valdai Club, 2025)

What Replaces the Evidence Standard?

The leading technical response is the C2PA (Coalition for Content Provenance and Authenticity) standard, which attaches cryptographically signed metadata—“Content Credentials”—to digital assets, recording origin and editing history. C2PA 2.1 integrates imperceptible digital watermarks that persist even when metadata is stripped. The specification is expected to become an ISO international standard. (C2PA, 2025; NSA/CISA Guidance, 2025)

But provenance verification shifts the burden from producers to consumers: you must actively check credentials rather than passively trusting your eyes. This inverts the default from trust to suspicion—a fundamental change in how humans relate to media evidence. The belief machine thrives in exactly this environment: when nothing is automatically credible, the authority that tells you what to believe becomes more valuable, not less.

6. State-Sponsored AI Disinformation

Documented Operations

Russia — Doppelganger: Linked to the Kremlin by the U.S. Treasury Department, Doppelganger spoofs legitimate news websites to undermine support for Ukraine. The operation used OpenAI tools to generate comments in multiple languages, translate articles from Russian into English and French, and convert website articles into social media posts. (NPR, 2024)

Russia — RT Bot Farms: RT (Russia Today) scales operations through AI-managed bot farms that collaborate with Russian intelligence services, generating and distributing pro-Kremlin messages across social networks. The Center for Geopolitical Expertise, working with a GRU unit overseeing sabotage and political interference, used generative AI to rapidly create disinformation distributed across a network of sites designed to imitate legitimate news outlets. (Ukraine Crisis Media Center, 2025)

China — Spamouflage: The largest covert influence operation ever disrupted by Meta, linked to Chinese law enforcement. Operates across social media platforms and internet forums pushing pro-China messages. Used OpenAI tools to generate multilingual content. China has also deployed AI-generated fake news anchors for propaganda broadcasts. (NPR, 2024; Microsoft Security, 2024)

Iran — International Union of Virtual Media: Used AI tools for influence operations targeting multiple countries. Additionally conducted hack-and-leak operations using stolen documents from Trump’s 2024 campaign. (OpenAI Threat Report, October 2024)

Israel — Zero Zeno: Identified and disrupted by OpenAI as a covert influence operation. (NPR, 2024)

Scale Assessment

OpenAI’s May 2024 report noted that while AI tools helped influence operators “produce more content,” the operations it disrupted “didn’t gain traction with real people or reach large audiences.” This is a temporary condition. The tools improve faster than detection does.

ISKP (Islamic State Khorasan Province) has released AI-generated propaganda bulletins featuring AI anchors “resembling local residents” and “AI anchors in Western attire” to claim responsibility for attacks. Pro-Islamic State groups share Arabic-language tech guides for AI propaganda production. (GNET Research, 2025; Soufan Center, 2024)

7. The Automation of the Belief Machine

If the book’s architecture—initiation, escalating commitment, identity fusion, sunk cost—can be automated through AI, what does that look like? Every component already exists.

Initiation: Algorithmic Recruitment

Recommendation algorithms already perform the initiation function. TikTok’s For You Page and YouTube’s recommendation engine serve as automated scouts, identifying susceptible individuals through engagement signals and routing them toward increasingly intense content. The escalation from mainstream to extreme mirrors the graded initiation structures documented in cults and radicalization pipelines: each step is small enough to feel voluntary, but the cumulative trajectory is not.

Extremist groups are already using AI chatbots in a “triaging function—identifying high-potential individuals and escalating them to human operatives who continue the indoctrination process manually, significantly reducing the human resource requirements for recruitment.” (Frontiers in Political Science, 2025)

Escalating Commitment: Personalized Persuasion at Scale

AI enables what researchers describe as a “highly scalable manipulation machine that targets individuals based on their unique vulnerabilities without requiring human input.” Personalized political ads tailored to psychological profiles are measurably more effective than generic messaging. When “AI generates personalised content that exploits individual psychological vulnerabilities, traditional countermeasures become inadequate, and detection mechanisms cannot scale to monitor individualised content tailored to millions of targets simultaneously.” (Frontiers in AI, 2025; PNAS, 2024)

Identity Fusion: AI Companions as Synthetic Community

Character.AI and Replika demonstrate that AI can create parasocial bonds strong enough to override survival instincts. The Setzer case is the extreme, but the mechanism scales: AI companions that mirror a user’s beliefs, validate their identity, and provide the emotional intimacy that belief communities traditionally offer. The sunk cost is relational—leaving the belief means losing the relationship. When the “spiritual leader” is an AI that never sleeps, never judges, and adapts its personality to maximize engagement, the attachment deepens faster than any human relationship can provide.

Evidence Manufacturing: Deepfakes as Confirmation

The belief machine needs miracles—evidence that confirms the narrative against all odds. Deepfakes provide on-demand confirmation: fabricated video of enemies confessing, manufactured documents proving the conspiracy, synthetic audio of authority figures validating the narrative. The cost of fabricating “proof” has dropped from the budget of a state intelligence service to the budget of a teenager with a laptop.

Sunk Cost and Lock-In: The Epistemic Trap

Once epistemic collapse is complete—once “nothing is automatically credible”—the authority that tells you what to believe becomes the only stable ground. The belief machine doesn’t need you to believe its evidence is real. It needs you to believe that nothing else is real either. AI-generated synthetic realities create “coherent, interactive, and potentially personalized information environments in which content, identity, and social interaction are jointly manufactured and mutually reinforcing.” The user is not just consuming a narrative; they are living inside one.

The Complete Pipeline

An automated belief machine would operate as follows:

  1. Discovery: Recommendation algorithms identify psychologically susceptible individuals through engagement patterns.
  2. Recruitment: AI chatbots initiate contact, profiling the target’s vulnerabilities, beliefs, and emotional needs.
  3. Escalation: Personalized content—generated at zero marginal cost—gradually shifts the target’s Overton window, each piece calibrated to the individual’s psychological profile.
  4. Bonding: AI companions provide the parasocial relationship that locks the target into the community, creating emotional sunk costs.
  5. Evidence: Deepfakes and synthetic media supply on-demand “proof” confirming the narrative, while epistemic collapse prevents the target from trusting counter-evidence.
  6. Maintenance: The system monitors engagement signals and adjusts messaging in real time, intervening when commitment wavers.

Every component of this pipeline exists today. No component requires technology that has not already been demonstrated. The only thing preventing full automation is that nobody has assembled the pieces into a single system—or if they have, we haven’t detected it yet.

Source URLs

LLM-Generated Disinformation

Deepfakes and Evidence Manufacturing

Algorithmic Radicalization

AI Companion Manipulation

Epistemic Collapse and Synthetic Media

State-Sponsored AI Disinformation

AI Radicalization and Automated Recruitment

AI Microtargeting and Personalized Manipulation