GPT – what the acronym means and the architecture

GPT stands for Generative Pre-trained Transformer. Each word does specific work: generative because the model produces output rather than just classifying input; pre-trained because the heavy compute happens once on a large text corpus before any task-specific use; transformer because the underlying neural network architecture is the transformer, introduced in 2017 and now the default for every frontier language model. The acronym started as a 2018 OpenAI paper title. By 2026 it is the de-facto label for a category of models that includes Claude, Gemini, Llama and DeepSeek, even though none of them are OpenAI products.

This is the article that connects the three things that usually live separately on the web: what the acronym means, how the architecture actually works, and how the GPT-1-through-GPT-5.5 lineage produced the products you use today. Wikipedia handles each in isolation; we connect them.

GPT in two sentences

GPT is a family of decoder-only transformer language models trained on next-token prediction across a large text corpus, then optionally fine-tuned for instruction-following. The name began as OpenAI’s naming convention for its 2018 paper, expanded through GPT-2, GPT-3, GPT-4 and GPT-5, and is now used generically across the industry to describe any decoder-only transformer trained the same way, regardless of who built it.

The conceptual move that matters is the second sentence. By 2020 the architecture had become a commons rather than a product line. OpenAI’s GPT-3 demonstrated what scaling the recipe could do; every serious lab that followed (Anthropic with Claude, Google with PaLM and then Gemini, Meta with Llama, DeepSeek, Mistral, xAI with Grok) adopted the same decoder-only transformer skeleton and the same pre-training plus fine-tuning pipeline. The architectural details vary; the family resemblance does not. For the company that originated the name, see our OpenAI company profile.

The architecture lineage

The architectural foundation predates GPT-1. The 2017 paper Attention Is All You Need by Vaswani et al., published by researchers at Google, introduced the transformer as a replacement for recurrent neural networks on sequence modeling tasks. The paper’s central claim was that an architecture built entirely from self-attention layers (with no recurrence) could outperform LSTM and GRU networks on machine translation while being more parallelizable, which mattered enormously for training at scale. We unpack the architecture itself in the transformer deep dive.

OpenAI’s GPT-1, published in June 2018 as Improving Language Understanding by Generative Pre-Training (Radford et al.), made two contributions. First, it stripped the original transformer down to just the decoder stack (no encoder), since the task was generation rather than translation. Second, it demonstrated that a model pre-trained on unlabeled text and then fine-tuned on small supervised datasets could outperform task-specific architectures across natural-language inference, question answering, and semantic similarity. The model had 117 million parameters and was trained on the BookCorpus dataset (roughly 7,000 unpublished books). The paper is short. The thesis is what mattered: scale the pre-training, simplify the fine-tuning, and most of NLP becomes a single problem.

GPT-2, released February 2019, scaled the same recipe by an order of magnitude (1.5 billion parameters) and trained on WebText, a curated scrape of outbound links from Reddit. OpenAI initially withheld the full model citing misuse concerns, releasing it in stages through November 2019. The capability that surprised observers was unconditional text generation: given a prompt, the model produced coherent paragraphs of fiction, news, or code without any task-specific training.

GPT-3, released May 2020, scaled again by two orders of magnitude (175 billion parameters) and trained on a filtered subset of Common Crawl, WebText2, books, and Wikipedia. The capability that mattered here was few-shot learning: the model could perform new tasks given two or three examples in the prompt, without any gradient updates. The 2020 paper Language Models are Few-Shot Learners effectively renamed the field. Whether or not GPT-3 was AGI (it wasn’t), the engineering bet of scale is sufficient had returned answers that the prior decade of careful task-specific architectures had not.

GPT-3.5, released November 2022 inside ChatGPT, was the first model in the lineage with reinforcement learning from human feedback (RLHF) layered on top of the pre-training. RLHF is what turned the autocomplete behavior of base models into the helpful-assistant behavior of ChatGPT. The base model could complete “The capital of France is” with “Paris”. The RLHF-tuned model could answer “What is the capital of France?” with “The capital of France is Paris”. The difference is what made the consumer product possible. For the full ChatGPT product analysis, see our ChatGPT pillar.

GPT-4, released March 2023, scaled again, added vision (image inputs), and was the first model in the lineage that OpenAI did not publish architectural details for. The technical report disclosed capabilities and benchmarks but not parameter count, training data, or training compute. The black-box turn started here. We cover the architectural and capability detail in the GPT-4 deep dive.

GPT-4o, released May 2024, was the multimodal-native version: native audio in and out, native image in and out, with a single model handling all three modalities through a shared representation. The “o” stands for “omni”. The latency reduction on voice was the visible win; under the hood, the unification of vision and audio into a single forward pass was the architectural one.

The o-series of reasoning models (o1 in September 2024, o3 in December 2024, o4-mini in 2025) ran in parallel to the main GPT lineage. The o-series used reinforcement learning to teach the model to produce long internal chains of thought before answering, trading inference-time compute for accuracy on hard problems. By 2026 the o-series capability was folded into the main GPT-5 family rather than maintained separately. See the o-series lineage for the detail.

GPT-5, released August 7, 2025, introduced a unified system with an internal router that decided automatically between fast inference and deep reasoning depending on the query. The initial release was rocky: OpenAI deprecated GPT-4o on launch day, faced user backlash, and restored access to older models for paying subscribers within days. The benchmarks presented at launch contained visual errors that contradicted the stated numbers. The 400K-token context window and the per-token price reduction were the substantive improvements.

GPT-5.5 (codename Spud), released April 23, 2026, is the current flagship as of May 2026. It scores 92.4% on MMLU, 88.7% on SWE-Bench Verified, 82.7% on Terminal-Bench 2.0, and 51.7% on FrontierMath tiers 1-3. OpenAI claims roughly 60% fewer hallucinations than GPT-5.4 on internal evaluations. GPT-5.5 Instant followed on May 5, 2026 as the default for broader users. Release cadence has compressed to roughly 60-day cycles between major updates.

The “scale is all you need” thesis that motivated GPT-1 through GPT-3 has not gone away, but it has been complemented. Modern frontier models layer reinforcement learning (post-training reasoning), retrieval augmentation, tool use, and mixture-of-experts architectures on top of the same decoder-only transformer skeleton. The skeleton is now 9 years old. Most of the visible product improvement since 2022 has come from what is built around it, not from changes to the skeleton itself.

How GPT models actually work

A GPT model is, at its most reductive, a function from a sequence of tokens to a probability distribution over the next token. Everything else (the appearance of reasoning, the coherent paragraphs, the helpful-assistant tone) is what falls out when you apply that function repeatedly and sample from its output.

The pre-training objective. Given a sequence of tokens like “The capital of France is”, the model is trained to assign maximum probability to the correct next token (“Paris”) among the roughly 100,000-token vocabulary it knows. The training corpus contains trillions of such sequences extracted from books, web pages, code repositories, and Wikipedia. The model has no special knowledge of facts; it has statistical regularities about which tokens follow which other tokens in human-written text. “The capital of France is” is followed by “Paris” in the training corpus far more often than by “London” or “Brussels”, so the model learns to produce that distribution.

This is “next-token prediction” in one sentence: train the model to guess the next word, on enough text, with enough parameters, and structure and meaning emerge from the statistics.

The fine-tuning step. After pre-training, the base model can complete text but cannot follow instructions. The 2022 InstructGPT paper showed that two further training steps fix this: supervised fine-tuning on examples of human-written instruction-response pairs, then reinforcement learning where a separately-trained reward model evaluates candidate responses and the language model is updated to produce responses the reward model prefers. The combined process is what transforms a base model into a chat assistant. ChatGPT and its analogues across labs all use some variant of this two-stage post-training.

The autoregressive sampling process. At inference time, the model generates text one token at a time. Given the prompt “The capital of France is”, it computes a probability distribution over all 100,000 possible next tokens, samples one (usually with techniques like top-p or temperature to control diversity), appends it to the context, and repeats. The model never sees a full sentence; it produces tokens one by one, each conditioned on every token that came before. This is why streaming output looks the way it does in ChatGPT: the user is watching the autoregressive loop run in real time.

The worked example: ask GPT-5.5 “What is the capital of France?”. The model tokenizes the prompt (roughly 7 tokens). The decoder stack processes the prompt through its attention layers, producing a probability distribution where “The” has the highest probability, conditioned on the question pattern. The model samples “The”, appends it, recomputes, produces “capital” next, then “of”, “France”, “is”, “Paris”, “.”. Total: roughly 8 output tokens. The model never represents Paris as a fact about France; it represents that the sequence “What is the capital of France? The capital of France is” is overwhelmingly followed by “Paris” in the patterns it was trained on. The behavior we read as knowing is a property of the statistics.

This is the foundation that everything downstream is built on. Tool use is the model being trained to emit special tokens that the application interprets as function calls. Reasoning is the model being trained to produce long internal monologues before answering. Multimodality is the model being trained on tokens that represent image patches or audio frames alongside text tokens. All of it sits on top of next-token prediction.

GPT vs other architectures

The transformer paper described three variants: encoder-decoder (the original), encoder-only, and decoder-only. Each found a different use case.

Encoder-only models like BERT (Google, 2018) and its descendants are trained on masked language modeling: a random subset of tokens is replaced with [MASK] and the model is trained to predict them using context from both sides. The bidirectional view makes encoder-only models excellent at understanding text (classification, named-entity recognition, semantic similarity, embedding generation) but unable to generate fluent long-form output. BERT and its successors dominate retrieval and search ranking. They do not power ChatGPT.

Encoder-decoder models like the original transformer, T5 (Google, 2019), and BART (Meta, 2019) keep both halves: an encoder reads the input, a decoder generates the output. The architecture suits tasks where input and output are structurally different (translation, summarization), and it remains the standard in many production ML pipelines for those tasks. It is not what frontier conversational models use.

Decoder-only models are the GPT family. The architecture is trained on causal language modeling: each token can attend only to tokens that came before it (not after). This is the right inductive bias for generation: the model is being trained to produce the next token, so giving it access to future tokens during training would be cheating. Decoder-only models scale unusually well, and as of 2026, every frontier conversational model is decoder-only. Claude, Gemini, Llama, DeepSeek, Mistral, Grok, and the GPT family are all decoder-only transformers. The convergence is not coincidence; it is what the scaling experiments returned. For the full LLM landscape, see our LLM explainer.

The reason decoder-only dominates is roughly this. Encoder-decoder forces a clean separation between understanding the input and producing the output, which is useful for translation but not for the open-ended conversation that turned out to be the killer application. Decoder-only treats everything as one continuous sequence: prompt, response, next prompt, next response, all conditioned on the running context. That uniformity is what makes a single model serve chat, coding, drafting, agentic workflows, voice, and tool use simultaneously without architectural changes per task.

The current GPT lineup

As of May 2026, the OpenAI-shipped GPT lineup is shorter than at any point since 2024, by design. The unification thesis of GPT-5 in August 2025 was that one routed model could subsume the proliferation of GPT-4o, GPT-4o-mini, o1, o3, o4-mini, and the rest.

GPT-5.5 (codename Spud) is the flagship, released April 23, 2026. Available on Plus, Pro $100, Pro $200, Business and Enterprise tiers of ChatGPT, with API access from April 24. Default for all paid users.

GPT-5.5 Thinking ships alongside GPT-5.5 as the extended-reasoning variant. Same underlying model, more inference-time compute spent on internal chain-of-thought before responding.

GPT-5.5 Pro is reserved for Pro $100, Pro $200, and Enterprise tiers, with stronger performance on complex tasks at higher per-token cost.

GPT-5.5 Instant, released May 5, 2026, is the new default for broader users, including Free tier (replacing GPT-5.3). It runs the GPT-5.5 architecture at lower inference cost.

GPT-OSS (gpt-oss-120b and gpt-oss-20b) is the open-weight family released August 5, 2025, two days before GPT-5. These are not “GPT-5”. They are open-weight, mixture-of-experts models that OpenAI released for local deployment, comparable in capability to the older o4-mini. gpt-oss-20b is designed to run on consumer hardware with 16GB+ VRAM; gpt-oss-120b runs on a single H100. See the OpenAI o-series lineage for the relationship to the reasoning model family.

GPT-5.6 is in active development as of May 2026, with public release predicted by mid-2026 based on prediction-market signals and Codex log traces. The compression of release cadence to sub-60-day cycles between major versions is the visible signal that OpenAI now treats the GPT family as continuous iteration rather than discrete generations.

GPT outside OpenAI

The decoder-only transformer architecture is no longer OpenAI’s. By 2026, every frontier model is built on the same skeleton, with each lab differentiating on training data, post-training, alignment, and infrastructure.

Anthropic’s Claude family is decoder-only transformer with Constitutional AI as the post-training method instead of RLHF. Claude Opus 4.7, released in 2026, leads or matches GPT-5.5 on multiple coding benchmarks and on long-horizon agentic tasks. Anthropic’s API revenue surpassed OpenAI’s in mid-2025 despite ChatGPT’s consumer dominance. See the Claude model family explainer.

Google’s Gemini family is decoder-only with Google’s TPU infrastructure as the differentiator. Gemini 3.1 Pro competes with GPT-5.5 on long-context and multimodal benchmarks, and the Google Workspace and Search integration provides distribution that OpenAI cannot match. See the Gemini model family explainer.

Meta’s Llama family is decoder-only, released under permissive licenses, and competitive on most production workloads at a fraction of the per-token cost of GPT-5.5. Llama 4 Maverick scores 91.8% on MMLU. The cost-quality math for many use cases now favors Llama-class open weights over frontier proprietary models. See the Llama explainer.

DeepSeek, Mistral, Qwen, and the broader open-weight ecosystem all use decoder-only transformers with various efficiency innovations (mixture-of-experts, long-context attention variants, distillation). The pace of open-weight progress through 2025 and 2026 has been the structural pressure on frontier labs’ pricing.

The architectural commons is what gives the GPT family its category meaning in 2026. When someone says “a GPT-class model” now, they mean a decoder-only transformer trained on large-scale next-token prediction with instruction post-training. OpenAI ships products called GPT-5.5; the field ships GPT-class models from a dozen labs.

Frequently asked questions

What does GPT stand for?

GPT stands for Generative Pre-trained Transformer. Generative means the model produces output; pre-trained means it is trained on a large corpus before any task-specific use; transformer is the underlying neural network architecture.

Who invented GPT?

The transformer architecture was introduced by researchers at Google in the 2017 paper Attention Is All You Need (Vaswani et al.). OpenAI’s research team, led by Alec Radford, adapted the architecture to language modeling and published GPT-1 in June 2018.

What is the difference between GPT and ChatGPT?

GPT is the family of language models (GPT-1 through GPT-5.5). ChatGPT is OpenAI’s consumer chat product that runs GPT models. The model and the product are distinct: GPT-5.5 also powers the OpenAI API, third-party tools, and Microsoft Copilot, none of which are ChatGPT.

What was the first GPT model?

GPT-1, released by OpenAI in June 2018 in the paper Improving Language Understanding by Generative Pre-Training. It had 117 million parameters and was trained on the BookCorpus dataset of roughly 7,000 unpublished books.

How does GPT actually work?

GPT models are trained on next-token prediction: given a sequence of tokens, the model assigns a probability to every possible next token, samples one, appends it to the context, and repeats. At inference time, this autoregressive loop produces coherent text one token at a time. The appearance of understanding and reasoning is what falls out of running this loop on a model with hundreds of billions of parameters trained on trillions of tokens.

What is the current GPT model?

As of May 2026, the current flagship is GPT-5.5 (codename Spud), released April 23, 2026. GPT-5.5 Instant, released May 5, 2026, is the default model for free and lower-tier users. GPT-5.6 is in active development with expected release by mid-2026.

Is GPT the same as a large language model?

GPT is a type of large language model. All GPT models are LLMs, but not all LLMs are GPT models. Claude, Gemini, Llama, DeepSeek, and others are LLMs that share the same decoder-only transformer architecture but are not part of the GPT family in the proper sense (they were not built by OpenAI). The term “GPT-class” is sometimes used informally to mean “decoder-only transformer LLM” regardless of origin.

What does pre-trained mean in GPT?

Pre-trained means the model has gone through a large initial training run on a general text corpus before any task-specific or instruction-following fine-tuning. The pre-training is the expensive step (millions of dollars in compute); the fine-tuning is comparatively cheap and produces variants of the same base model.

What is GPT used for?

GPT models power conversational AI (ChatGPT), code generation (Codex, Cursor, GitHub Copilot), drafting and writing tools, document analysis, agentic workflows, voice assistants, and multimodal applications. They are general-purpose enough that the use cases are not limited to text generation in any narrow sense.

Is GPT-5 better than GPT-4?

On benchmarks, yes. GPT-5.5 scores roughly 92.4% on MMLU versus GPT-4’s 86%, and shows substantial gains on reasoning, coding, and math benchmarks. On real-world tasks, the improvement is real but task-dependent. The product release of GPT-5 in August 2025 was uneven; the GPT-5.5 release in April 2026 was the version where most users perceived a clear quality step from GPT-4.

What is the difference between GPT and BERT?

GPT is decoder-only and trained on next-token prediction, suited for generating text. BERT (Google, 2018) is encoder-only and trained on masked language modeling, suited for understanding text (classification, search ranking, embeddings). The two architectures have different inductive biases and dominate different use cases.

Will GPT models keep improving by scaling alone?

Scaling continues to deliver returns, but the marginal improvement per dollar of compute has slowed since GPT-3. Most of the visible improvement from GPT-4 onwards comes from post-training innovations (RLHF, reasoning, tool use, agentic workflows) layered on top of the same architectural skeleton, plus better training data curation. The “scale is all you need” thesis was right enough to motivate the field; the 2026 reality is closer to “scale plus a stack of post-training techniques is what works”.