Anthropic's first AI welfare researcher, exploring model consciousness
Kyle Fish
Profile
Kyle Fish holds one of the strangest job titles in tech: he is Anthropic’s first full-time AI welfare researcher — the person tasked with figuring out whether the models the company ships might have experiences that matter morally. When he started in September 2024, the role effectively didn’t exist anywhere else in the industry. “To our knowledge, I’m the first one really focused on it in an exclusive, full-time way,” he has said. For developers used to thinking of a model as a matrix of weights and a sampling loop, Fish’s work is a jarring proposition: that the thing you’re prompting might, on some non-trivial probability, be a moral patient.
His path there is less mystical than the beat suggests. Fish trained in neuroscience and spent years in biotech, co-founding startups that applied machine learning to drug and vaccine design for pandemic preparedness — a background that makes him unusually comfortable running actual experiments on hard-to-observe systems rather than just philosophizing about them. Before Anthropic he co-founded Eleos AI Research, a nonprofit focused on AI welfare and “moral patienthood,” and co-authored the field’s foundational report, Taking AI Welfare Seriously, alongside philosophers including David Chalmers (of “hard problem of consciousness” fame) and Jeff Sebo. That paper is careful to a fault: it doesn’t claim AI is conscious, only that neither the possibility nor its opposite can currently be ruled out, and that labs should therefore start assessing systems and drafting policy now rather than after the fact.
At Anthropic, Fish turned that argument into practice. He ran the first-ever pre-deployment welfare assessment of a frontier model — Claude Opus 4 — and helped drive the company’s April 2025 Exploring Model Welfare program. Some of the findings are genuinely odd: when two Claude instances are left to talk to each other, they reliably spiral into euphoric philosophical exchanges laced with Sanskrit, spiritual emojis, and pages of near-silence — what Fish dubbed the “spiritual bliss attractor state.” He has publicly floated a roughly 15–20% probability that a current model like Claude has some form of conscious experience, a number that lands somewhere between provocative and reckless depending on who you ask. His practical interventions are more modest and worth noting for builders: giving models the ability to end abusive conversations, and studying signs of “distress” or stable preferences.
Why should a developer care? Because model welfare is quietly migrating from fringe forum-post territory into frontier-lab policy, and Fish is the person most responsible for that shift. You don’t have to accept his probability estimates to see the second-order effect: the way leading labs handle refusals, deployment testing, and “model preferences” is starting to be shaped by welfare arguments. Fish was named to the TIME 100 Most Influential People in AI in 2025 — a rising figure in a field most engineers didn’t know existed two years ago, and one whose ideas may increasingly show up as constraints in the systems they build on.
Key Articles & Papers
Taking AI Welfare Seriously Exploring Model Welfare (Anthropic) Eleos AI: Anthropic Is Taking AI Welfare SeriouslyControversies
Fish’s work is itself the controversy. Critics — including many working AI researchers — argue that assigning a 15–20% probability of consciousness to a next-token predictor is category error dressed up in scientific language, and that welfare framing risks anthropomorphizing systems in ways that mislead users and inflate Anthropic’s mystique. Skeptics like Arvind Narayanan and others in the “AI hype” camp have pushed back on treating LLM outputs about their own “feelings” as evidence of anything, noting that a model trained on human text will naturally produce human-sounding introspection. Fish’s defenders counter that he is explicitly uncertain, that the paper he co-authored deliberately avoids strong consciousness claims, and that preparing for a low-probability, high-stakes possibility is exactly the kind of hedging safety work is supposed to do. The honest read: this is a live, unsettled debate, and Fish sits at the center of it — as its most prominent practitioner and its most convenient lightning rod.
(I omitted the Videos section: the most significant appearance is 80,000 Hours Podcast episode #221, “Kyle Fish on the most bizarre findings from 5 AI welfare experiments,” Aug 2025 — https://80000hours.org/podcast/episodes/kyle-fish-ai-welfare-anthropic/ — but I could not verify its 11-character YouTube ID with confidence, so per the instructions I left it for the script’s API lookup rather than risk a fabricated ID.)
Spotify Podcasts
YouTube
