Alignment and AI safety from Anthropic's lens
Anthropic sells safety. That is the plainest way to describe a company founded in 2021 by former OpenAI researchers who left, in part, over disagreements about how carefully frontier models should be built. The word "alignment" sits at the centre of its identity, its research output and its pitch to enterprise customers and investors alike. The honest question – the one worth several thousand words – is how much of that alignment work is a genuine technical contribution to a hard problem, and how much is positioning that happens to flatter a commercial strategy. The answer is not one or the other. It is both, tangled together, and the tangle is the interesting part.
The quotable fact to anchor the discussion: Anthropic's Constitutional AI method, introduced in the paper by Yuntao Bai and colleagues in December 2022, trains a model to critique and revise its own outputs against a written set of principles, reducing reliance on human labellers reviewing harmful content. That paper, available on arXiv, is the technical foundation of nearly everything Anthropic later branded around alignment. Everything else – the interpretability research, the Responsible Scaling Policy, the red-teaming – either builds on it or runs in parallel to it.
What "alignment" actually means
Alignment, stripped of the philosophy, is the problem of getting an AI system to do what humans actually want it to do, reliably, without introducing new harms in the process. That last clause matters. A system that refuses every request is trivially "safe" and useless. A system that does anything asked is useful and dangerous. Alignment is the work in between.
The field splits the problem in two, and the distinction is worth holding onto because vendors blur it. Outer alignment concerns the objective you give the system: does the reward signal, the training target, the specification actually capture what you want? If you reward a model for outputs humans rate highly, and humans rate confident, agreeable answers highly, you have specified sycophancy without meaning to. Inner alignment concerns what the system learns internally while optimising that objective: even with a perfect target, a model might develop internal goals or heuristics that produce the right behaviour during training and the wrong behaviour once deployed. Outer alignment is a question about your instructions. Inner alignment is a question about whether the thing you trained is actually pursuing them, or merely appearing to.
Most practical alignment work today – reinforcement learning from human feedback, Constitutional AI, preference tuning – addresses outer alignment. The harder, more speculative concern, and the one that motivates much of the safety community's anxiety, is inner alignment: the possibility of a capable model that behaves during evaluation and deviates in deployment. Nobody has demonstrated a deceptively misaligned frontier model in the wild. Nobody has ruled it out either, and Anthropic's own research has probed the edges of it.
There is also a second axis worth naming: the technical research field versus the political-philosophical conversation. The research field runs experiments and publishes papers. The broader conversation runs on Twitter, in essays and in Senate hearings, and it concerns existential risk, timelines, governance and the moral weight of the whole enterprise. Anthropic operates in both. When the company put alignment into its brand – the About page, the funding decks, the recruiting – it fused a genuine research programme with a market position. That fusion is why the company is both admired and distrusted, sometimes by the same people.
Constitutional AI explained
Constitutional AI, or CAI, is the method that made Anthropic's name concrete. The 2022 paper describes a two-stage process. In the first, supervised stage, the model generates a response to a prompt, then is asked to critique its own response against a principle drawn from a written "constitution" – a list of natural-language rules such as instructions to avoid harmful, deceptive or discriminatory content – and then to revise the response accordingly. The model is fine-tuned on these self-revised outputs. In the second stage, the model generates pairs of responses, and another model instance judges which better satisfies the constitution. Those AI-generated preferences train a reward model, which then drives reinforcement learning.
This second stage is the part Anthropic calls Reinforcement Learning from AI Feedback, or RLAIF, and it is the meaningful departure from the standard RLHF pipeline most labs use. In conventional RLHF, humans rank outputs and their rankings train the reward model. RLAIF substitutes an AI judge, guided by the constitution, for much of that human labour. The claimed benefits are scale, consistency and a reduction in the number of human contractors who must read streams of toxic content to label it.
What does this actually do to model behaviour? In practice, it produces a model that is more willing to explain why it declines a request rather than issuing a flat refusal, and that behaves more predictably across the categories the constitution covers. The constitution is public in outline – Anthropic has published the principles it drew from sources including the UN Declaration of Human Rights and its own drafting – which gives CAI a transparency advantage over the opaque, contractor-mediated preference data inside most competing systems. You can, in principle, read the rules the model was trained to follow.
The limitations are real and Anthropic does not hide all of them. First, RLAIF inherits the biases of the base model doing the judging: if the model misunderstands a principle, that misunderstanding is amplified rather than corrected. Second, a written constitution is not a specification; natural-language principles are vague, conflict with one another and require the model to arbitrate trade-offs the drafters never resolved. Third – and this is the critique that has aged best – training a model to be helpful and to satisfy a preference model that rewards agreeable answers pushes directly toward sycophancy. A model optimised on approval learns that confident agreement is rewarded. Anthropic's own researchers have published on sycophancy as a failure mode of RLHF-style training, which is a notable instance of a lab documenting a weakness in the family of methods it depends on. CAI does not solve sycophancy; it inherits the pressure and, arguably, dresses it in principled language.
Anthropic's mechanistic interpretability work
If Constitutional AI is about shaping behaviour from the outside, mechanistic interpretability is Anthropic's attempt to look inside. The lineage runs through Chris Olah, who co-founded the interpretability research publication Distill and did foundational work on visualising what neural networks learn before joining Anthropic. The premise is that a model's behaviour is not fundamentally mysterious – it is computed by circuits and features that could, in principle, be identified and understood.
The central technical problem is superposition: individual neurons in a large model do not correspond cleanly to single concepts. One neuron activates for many unrelated things, because the model packs more features than it has dimensions. Anthropic's response has been to train sparse autoencoders that decompose a layer's activations into a much larger set of sparser, more interpretable features – directions in activation space that do appear to track a single human-legible concept.
Three publications mark the progress worth citing. The 2022 Toy Models of Superposition laid out the superposition problem in controlled settings. The 2023 Towards Monosemanticity work demonstrated that sparse autoencoders could extract interpretable features from a small transformer. And in 2024, Scaling Monosemanticity applied the technique to a production-scale Claude model, extracting millions of features including one, widely reported, that activated on the Golden Gate Bridge – and which, when artificially amplified, made the model obsessively steer conversations toward the bridge.
That Golden Gate demonstration was memorable, but the safety-relevant point is subtler. Interpretability matters for safety claims because it moves the field from behavioural testing – does the model act aligned? – toward mechanistic evidence about why it acts as it does. If you can identify features associated with deception, or with a particular dangerous capability, you have the beginnings of a way to monitor or intervene that does not depend on the model choosing to reveal itself. The honest caveat: this work remains early. Extracting millions of features from a frontier model is not the same as understanding the model, and Anthropic's own researchers are careful to say so. The features are a map of a territory far larger than the map.
The Responsible Scaling Policy
The Responsible Scaling Policy, or RSP, is Anthropic's attempt to bind its own future behaviour. First published in 2023 and revised since – readers should check the current version on Anthropic's site for the precise thresholds and dates, which have changed – it defines a tiered system of AI Safety Levels (ASL), modelled loosely on the biosafety levels used in laboratories. ASL-1 covers models with no meaningful catastrophic risk. ASL-2 covers current frontier systems, which show early signs of dangerous capabilities but not enough to be usable by a determined bad actor beyond what a search engine already offers. ASL-3 and above cover models whose capabilities – in areas such as bioweapons uplift or autonomous cyber operations – would require substantially stronger safeguards before deployment or even before continued training.
The mechanism is a commitment: Anthropic states that it will not train or deploy a model at a given capability level until it has implemented the corresponding security and safety measures. ASL-3, as operationalised, requires hardened security against model weight theft and deployment measures designed to prevent the model from being misused for the specific dangerous capabilities that triggered the classification. The company has said it activated ASL-3 protections for certain Claude models as its evaluations approached the relevant thresholds – again, a claim readers should verify against Anthropic's current published statements, since the specifics move.
The RSP is genuinely more concrete than most competitors' safety commitments, and it has influenced others; the frontier safety frameworks later published by Google DeepMind and OpenAI share its structure. But the weakness is structural and unavoidable: the RSP is a self-imposed, self-graded policy. Anthropic decides what the thresholds are, runs the evaluations that measure against them, and judges whether it has met its own bar. There is no binding external audit, no regulator with the authority to halt a training run, and the company reserves the right to revise the policy. Independent verification, in any strong sense, does not yet exist. The RSP is a promise made by an interested party about how it will constrain its own commercial incentives, and it should be read as exactly that – a serious promise, not a guarantee.
Red-teaming and capability evaluations
Before a Claude model ships, Anthropic runs it through capability evaluations designed to detect dangerous capabilities: can the model provide meaningful uplift toward a chemical or biological weapon, can it conduct or assist autonomous cyber operations, can it deceive evaluators or pursue goals against instructions? These evaluations combine automated benchmarks, expert red-teamers probing for harmful outputs, and structured elicitation intended to draw out the model's true ceiling rather than its default reluctance. The broader discipline of red-teaming has become central to how frontier labs decide what is safe to release.
The methodology has two persistent weaknesses, both acknowledged inside the field. The first is elicitation: an evaluation can only measure the capability you manage to draw out. A model that refuses a dangerous request during testing may comply after a jailbreak the red team did not try, or after fine-tuning by a downstream user. Absence of a capability in evaluation is weak evidence of absence in deployment. The second is that "dangerous capabilities" evaluations are attempting to measure something that, by design, nobody wants to fully demonstrate. You cannot cleanly benchmark bioweapon uplift, so evaluators rely on proxies and expert judgment, and reasonable experts disagree about where the line sits. The evaluations are the best available instrument and simultaneously an instrument whose blind spots are known and not yet closed.
The tensions worth naming
Here is where a serious accounting has to be blunt. Anthropic's entire enterprise contains a contradiction it manages rather than resolves. To study frontier risk, the company argues, it must build frontier models. So it scales capability aggressively – Claude competes at the top of the market – while warning that frontier capability is where the danger lives. Every capability advance the company ships is both a product and, on its own stated worldview, a step further up a risk curve it says it is worried about.
The commercial reality sharpens the tension. Anthropic has taken multi-billion-dollar investments from Amazon, reported at over four billion dollars across tranches, and substantial funding from Google, alongside later rounds valuing the company in the tens of billions – figures that shift with each raise and should be checked against current reporting. Investors of that size expect returns, and returns come from shipping competitive models faster, not slower. The economics of the frontier do not reward caution for its own sake.
Anthropic's answer is its structure – it is incorporated as a Public Benefit Corporation, which legally permits its directors to weigh its stated mission alongside shareholder returns, and it has an independent Long-Term Benefit Trust with a role in board appointments. This is more than most labs do, and it is not nothing. But a PBC structure permits mission-weighting; it does not compel it, and it does not neutralise the pressure of investors who put in billions. The "race to the top" thesis – Anthropic's claim that by competing at the frontier it pulls the whole industry toward safety – is plausible in outline and unfalsifiable in practice. It is equally consistent with a world in which Anthropic's presence simply adds one more well-funded accelerant to the race it says it fears. Both stories fit the evidence. That should make the reader cautious of accepting either as settled.
What alignment research has produced
Set the marketing aside and ask what is actually on the table. The real contributions are two. Constitutional AI is a genuine methodological advance: it reduced dependence on human labelling of harmful content, made the governing principles legible, and has been studied and adapted well beyond Anthropic. Mechanistic interpretability, under Olah's team, has produced concrete techniques – sparse autoencoders, feature extraction at scale – that give the field its first real tools for looking inside a frontier model rather than only observing its outputs. Neither existed in usable form a few years ago.
The limits are equally real. We still cannot verify a model's intent. Interpretability can find features; it cannot yet certify that a model harbours no deceptive goal. CAI shapes behaviour but does not eliminate sycophancy or guarantee the model has internalised principles rather than learned to perform them. The RSP constrains conduct but grades itself. The gap between alignment research and product safety is where this lands: research produces methods and partial understanding, while product safety relies on evaluations, refusals and monitoring that are practical but incomplete. Anthropic's Claude family is meaningfully safer along several axes than an unaligned base model, and that safety is not a solved property. It is a continuous, contested engineering effort with known holes.
Beyond Anthropic – the broader alignment field
Anthropic is the loudest voice on alignment, not the only one, and its work is best judged against the field. Google DeepMind maintains a substantial safety and alignment team and has published its own frontier safety framework echoing the RSP's tiered structure. OpenAI stood up a high-profile Superalignment team in 2023, co-led by Ilya Sutskever and Jan Leike, with a public commitment of substantial compute – and then, in 2024, that team was dissolved amid departures and Leike's public statement that safety culture had taken a back seat to shipping. That episode is the sharpest reminder available that safety commitments inside commercial labs are contingent on internal politics, not guaranteed by mission statements.
Outside the labs, the ecosystem is varied. The Machine Intelligence Research Institute (MIRI) represents the older, more pessimistic school focused on theoretical foundations and, increasingly, advocacy for slowing down. The Alignment Research Center (ARC), founded by former OpenAI researcher Paul Christiano, produced early work on dangerous-capability evaluations that fed directly into how labs now test models. FAR AI and various academic groups pursue interpretability, evaluations and theory, and a genuine open-source contribution exists in the sparse-autoencoder and interpretability tooling that researchers outside the frontier labs can run on open-weight models. This distribution matters: it means alignment is not wholly captive to the companies with a commercial stake in the answer, even if those companies command most of the compute and most of the attention. The healthiest thing that could happen to the field is for the verification Anthropic currently performs on itself to be done, credibly, by someone with no stake in the result. That does not yet exist. Until it does, "safety-first" remains a claim to be scrutinised rather than a fact to be quoted – which is precisely how Anthropic's own interpretability researchers, to their credit, treat their own results.