Stable Diffusion – the open image model family
Stable Diffusion is a family of open-weights text-to-image models built on latent diffusion, first released in August 2022 and developed primarily by Stability AI alongside the original research from CompVis (LMU Munich) and Runway. It is the model lineage that commoditised text-to-image generation: because the weights are downloadable and the licence permits commercial use, Stable Diffusion became the foundation for an entire ecosystem of local-inference tools, fine-tuning communities, and commercial services that exists outside the control of any single vendor. As of 2026 it remains the default choice when you need to run image generation on your own hardware, customise a model to a specific visual style, or generate at volume without per-image API fees.
If you are deciding whether to build on Stable Diffusion, the question is not “is it the highest-quality image model?” (it is not – Midjourney and the newer closed models generally produce better single outputs). The question is whether open weights, full customisation, and local inference matter more to you than peak output quality. This article traces how the model family evolved, explains how latent diffusion actually works, evaluates where Stable Diffusion sits against the closed leaders in 2026, and addresses the legal and ethical questions that the lineage has never fully escaped.
What Stable Diffusion is
Stable Diffusion is built on the latent diffusion architecture introduced in the 2022 paper High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach, Andreas Blattmann and colleagues at LMU Munich’s CompVis group. The key insight of that paper – and the reason Stable Diffusion could run on consumer hardware when earlier diffusion models could not – is that the diffusion process happens in a compressed latent space rather than at full pixel resolution. We unpack the mechanics below; the architectural point is that this is what made open, local image generation feasible at all.
What made Stable Diffusion matter was not the architecture alone but the decision to release the weights openly. When the first version shipped in August 2022 under a permissive licence, it broke the pattern every other capable image model had followed. DALL-E 2 was API-only. Midjourney was Discord-only. Stable Diffusion you could download, run on your own GPU, fine-tune, modify, and build products on top of without asking anyone’s permission. That single decision created the open image-generation ecosystem. Within months there were custom interfaces, fine-tuned style models, and a fast-growing community doing things Stability AI had never planned for.
The company behind it has been the most volatile element of the whole story. Stability AI provided the compute for the original training run and became the model’s commercial sponsor, but it has been through severe turbulence. Under founder Emad Mostaque, the company reported losses exceeding $30 million in the first quarter of 2024 against revenues under $5 million, and was widely reported to be facing a cash crunch. Mostaque resigned in March 2024. In June 2024 the company brought in Prem Akkaraju, the former CEO of Weta Digital, as chief executive, with Napster founder Sean Parker as executive chairman, alongside roughly $80 million in new funding from investors including Eric Schmidt, Coatue and Lightspeed, and a reported agreement to forgive about $100 million in existing debt. By December 2024 Akkaraju said the company had “a clean balance sheet, no debt, nothing” and was growing by triple digits. As of early 2026 Stability AI remains independent with a valuation around $1 billion and roughly 190 employees – stabilised, but a fraction of the scale of the frontier labs it competes with. For the company-level detail we go deeper in the Stability AI company profile.
The model family evolution
The defensible way to understand Stable Diffusion is as a lineage with a clear narrative arc: foundational success, a stumble, a recovery, and a hard-won return to form. Most coverage flattens this into “what is Stable Diffusion.” The evolution is the story.
SD 1.4 and 1.5 (2022) – the foundational releases. Version 1.5, released in October 2022, became the most important image model in the open ecosystem and arguably still is in terms of derivative works. Its UNet-based architecture at 512×512 native resolution was small enough to run on a mid-range consumer GPU, and the entire first generation of fine-tuning culture (custom style models, the early LoRA ecosystem, the first ControlNet models) was built on 1.5. Years later, a substantial share of community models on platforms like Civitai are still 1.5 derivatives, because the ecosystem of tooling around it is so deep.
SD 2.0 and 2.1 (late 2022) – the underrated stumble. Stability changed the text encoder from OpenAI’s CLIP to OpenCLIP and re-filtered the training data, partly in response to concerns about copyrighted and NSFW content. The result was a model that, on paper, was an upgrade, but in practice produced worse results on the prompts people actually used – particularly for human anatomy and the artist-style prompts the community relied on. The community largely rejected 2.x and stayed on 1.5. The lesson, which recurs in the SD 3 story, is that changing the training data and text encoder can break the implicit contract with users even when the benchmarks look fine.
SDXL (July 2023) – the recovery. SDXL was the genuine step-change. A larger UNet, a 1024×1024 native resolution, a two-stage base-plus-refiner pipeline, and dual text encoders produced a model that was clearly better than 1.5 while remaining open and runnable on consumer hardware (with more VRAM). SDXL re-established Stable Diffusion as a credible open frontier and became the new base for the fine-tuning community. SDXL Turbo, released later in 2023, used adversarial distillation to generate images in a single step, trading some quality for near-real-time generation.
SD 3 (June 2024) – the second stumble. Stable Diffusion 3 Medium, the first release of the SD 3 series, introduced a new architecture – the Multimodal Diffusion Transformer (MMDiT), replacing the UNet with a transformer backbone – but the initial release landed badly. The 2-billion-parameter Medium model produced notoriously poor human anatomy on certain prompts, and the community reaction echoed the SD 2.0 episode. Stability publicly acknowledged it: the company stated that the SD 3 Medium release “didn’t fully meet our standards or our communities’ expectations” and chose to rework it rather than ship a quick fix.
SD 3.5 (October 2024) – the return to form. The current generation, released 22 October 2024, came in three variants and is the production lineup as of 2026:
- SD 3.5 Large – 8.1 billion parameters, the flagship, MMDiT architecture with three fixed pretrained text encoders, native 1-megapixel output, prioritising quality and prompt adherence.
- SD 3.5 Large Turbo – a distilled version of Large that generates high-quality images in roughly 4 steps, trading some fidelity for speed.
- SD 3.5 Medium – released 29 October 2024, 2.5 billion parameters with an improved MMDiT-X architecture, designed to run on standard consumer GPUs (around 9.9 GB VRAM), with variable resolution from 0.25 to 2 megapixels.
All three ship under the Stability AI Community License, which is free for commercial and non-commercial use below a revenue threshold. We cover the SD 3 architecture in depth in the SD 3 generation deep dive.
How Stable Diffusion actually works
The mechanism is worth understanding plainly, because it explains both the strengths and the limits.
Latent space versus pixel space. A 512×512 RGB image is roughly 786,000 numbers (512 × 512 × 3). Running a diffusion process directly on that many values is enormously expensive – this is why early pixel-space diffusion models needed data-centre GPUs. Latent diffusion solves this with a variational autoencoder (VAE) that compresses the image into a much smaller latent representation – typically a 64×64×4 tensor, roughly 16,000 numbers, a ~48× reduction. The diffusion process runs entirely in this compressed latent space, and only at the very end does the VAE decoder expand the final latent back into a full-resolution image. This compression is the single reason Stable Diffusion runs on a gaming GPU while comparable pixel-space models do not.
The denoising process, plainly. Training works by taking a real image, encoding it to a latent, and progressively adding random noise until it is pure static. The model – historically a UNet, now the MMDiT transformer in SD 3.5 – learns to reverse this: given a noisy latent and a text prompt, predict the noise that was added, so it can be subtracted. At generation time you start from pure random noise in latent space and run the learned denoising process for a number of steps (typically 20–50), at each step nudging the latent toward something that matches the text prompt, guided by the text encoder. After the final step the VAE decodes the clean latent into the output image. “Generating an image” is, mechanically, “removing noise from random static in the direction the text describes.”
The text encoder. The prompt is converted into embeddings by one or more pretrained text encoders. SD 1.x used a single CLIP encoder; SD 3.5 uses three fixed pretrained encoders together, which is a large part of why its prompt adherence and text rendering improved. The choice of text encoder is consequential – it was a major factor in both the SD 2.0 stumble (switching to OpenCLIP) and the SD 3.5 recovery.
The customisation layer. This is where the open ecosystem’s real advantage lives, and none of it would be possible with a closed model:
- LoRAs (Low-Rank Adaptation) are small files (often a few hundred MB or less) that adjust the base model toward a specific style, character, or concept without retraining the whole thing. The community has produced hundreds of thousands of them.
- ControlNet lets you condition generation on a structural input – a pose skeleton, depth map, edge map, or scribble – so you control composition precisely rather than relying on the prompt alone. Stability released Blur, Canny and Depth ControlNets for SD 3.5 Large.
- IP-Adapters let you condition on a reference image, transferring style or subject without a text description.
These three tools – LoRA, ControlNet, IP-Adapter – are the practical reason professionals choose Stable Diffusion. They convert a general image model into a controllable production tool tuned to a specific need. For running this stack locally, see our coverage in the on-device and edge AI silo.
Stable Diffusion versus the closed competitors in 2026
The honest assessment, by dimension:
Peak output quality. Stable Diffusion does not win here. For the best single image from a single prompt, Midjourney (now on v7) generally produces more aesthetically refined results out of the box, and the newer closed models – Google’s Imagen lineage, OpenAI’s image generation in GPT, and Black Forest Labs’ Flux Pro – are strong competitors. Black Forest Labs is itself relevant to this story: it was founded by Robin Rombach and other core members of the original Stable Diffusion research team after they left Stability, and Flux is in many respects the spiritual successor to the SD lineage at the frontier. If your only metric is “best image, least effort,” Stable Diffusion is usually not the answer.
Customisation. Here the Stable Diffusion lead is unassailable, and it is not close. No closed model lets you download weights, train a LoRA on your own images, stack ControlNets, run custom inference pipelines, and deploy the result however you like. For any workflow that needs a consistent custom style, a specific character across many images, or precise structural control, the open ecosystem is the only option that works. This is the durable competitive moat.
Speed and cost. Local inference has no per-image fee. Once you own the hardware, the marginal cost of an image is electricity. For high-volume generation – batch product imagery, large content pipelines, iterative design work – the economics favour self-hosted Stable Diffusion decisively over per-call API pricing. The Turbo variants make near-real-time local generation practical.
Licence clarity. The Stability AI Community License is free for commercial use below a revenue threshold (reported at $1 million annual revenue), above which an enterprise licence is required. This is cleaner than the ambiguity that surrounded earlier releases, but builders should read the current licence and Acceptable Use Policy directly rather than assuming, because both have changed over time. For the comparison against the closed image-generation leaders we go deeper in the Midjourney platform profile.
The Stable Diffusion ecosystem
The models are only half the story. The ecosystem built on top is the other half, and it is what makes the lineage durable independent of Stability AI’s corporate fortunes.
Runtime interfaces. Three dominate. ComfyUI is the node-based workflow tool that has become the professional standard, letting you build complex generation pipelines visually and reproducibly. AUTOMATIC1111 (the “web UI”) was the early community standard and remains widely used for its breadth of extensions. InvokeAI targets a more polished, product-like experience aimed at professional and commercial users. All three are open source.
Distribution. Hugging Face is the primary channel for the official model weights and inference code. Civitai is the centre of the community fine-tuning ecosystem – the place where LoRAs, fine-tuned checkpoints, and embeddings are shared. The scale of Civitai’s library is itself a moat: the depth of available community models for the SD lineage is something no closed model can match.
Commercial services. A layer of businesses runs Stable Diffusion as a hosted service so you do not have to manage GPUs – Replicate, fal.ai, and others provide API access to SD models with per-call pricing. This gives builders a middle path: the open model and its customisation ecosystem, without the operational burden of self-hosting. We cover these inference-provider patterns in the agentic AI silo.
Where Stable Diffusion fits in 2026
Three decisions cover most cases.
Best fit. Stable Diffusion is the right choice when you need a customised, consistent visual style (train a LoRA), when you generate at volume and per-image API cost would dominate (self-host), when you need precise structural control over composition (ControlNet), or when data-path control matters and images cannot leave your infrastructure (local inference). These are the cases where open weights convert directly to value.
Weak fit. It is the wrong choice when you need the single best-quality output with the least effort (Midjourney v7 or a frontier closed model wins), when you have no GPU and no appetite for managing inference, or when you need the model to “just work” without the learning curve that ComfyUI and the customisation stack demand. The open ecosystem trades polish and ease for control and cost.
The realistic pattern. Many production teams use both: a closed model like Midjourney for hero images where peak quality matters, and self-hosted Stable Diffusion for the high-volume, customised, or structurally-controlled work where the open ecosystem is irreplaceable. For the hardware side of self-hosting on consumer GPUs, see the on-device and edge AI silo.
The ethical and legal question
Stable Diffusion has never fully escaped the questions raised by how it was trained, and as an independent analyst portal we treat these as core to the model’s profile, not a footnote.
Training data and the lawsuits. Stable Diffusion’s early versions were trained on LAION datasets – billions of image-text pairs scraped from the public web, including copyrighted images and watermarked stock photography. This triggered the most significant litigation in the field. In the United States, Andersen v. Stability AI, a class action brought by visual artists alleging their works were used without consent, has proceeded through the courts with some claims surviving motions to dismiss, and a separate US case brought by Getty Images is ongoing.
The most consequential ruling so far came in the United Kingdom. On 4 November 2025, the High Court of England and Wales handed down its judgment in Getty Images (US) Inc & Ors v Stability AI Limited, the first UK judgment to examine copyright and trademark infringement in AI model development. The Court largely rejected Getty’s copyright claims, in significant part on jurisdictional and evidentiary grounds – the model was trained outside the UK, and the Court held that the model and its weights were not “infringing copies” imported into the UK because they do not store or reproduce the copyright works. Getty did secure a limited finding on trademark infringement, where Stability AI’s outputs reproduced Getty watermarks. The legal takeaway is narrow and widely regarded as inconclusive: the judgment turned on fact-specific and territorial issues rather than resolving the central question of whether training on copyrighted works is itself lawful. That question remains open, with the US litigation and EU cases still to be decided and UK government consultations ongoing.
The NSFW content problem. Open weights mean Stability cannot control what the community generates. This has been a persistent tension. Stability has progressively restricted what its own released models can do – effective 31 July 2025, Stability updated its Acceptable Use Policy to prohibit the generation of sexually explicit content with its Core Models, which led some community platforms to remove SD 3.5 and derivative models from their hosted generators. But the fundamental tension of open weights remains: once weights are downloadable, the licence governs permitted use, not technical capability.
Watermarking and provenance. As AI-generated imagery becomes indistinguishable from photography, provenance standards matter more. The C2PA standard (Coalition for Content Provenance and Authenticity) is the leading effort to cryptographically attach origin metadata to images. We cover this in detail in the C2PA and content authenticity analysis. For open models the challenge is structural: provenance signals that depend on the generator cooperating are weaker when anyone can run a modified model that omits them.
Frequently asked questions
What is Stable Diffusion?
Stable Diffusion is a family of open-weights text-to-image AI models based on latent diffusion, first released in August 2022 by Stability AI together with the CompVis research group and Runway. Unlike closed models such as Midjourney or DALL-E, its weights can be downloaded, fine-tuned, and run on your own hardware, which is why it became the foundation of the open image-generation ecosystem.
Is Stable Diffusion free to use?
The models are free to download and use commercially below a revenue threshold (reported at $1 million annual revenue) under the Stability AI Community License. Above that threshold an enterprise licence from Stability AI is required. The open-source runtime tools (ComfyUI, AUTOMATIC1111, InvokeAI) are free. You provide your own hardware, or pay a hosting service like Replicate or fal.ai for API access.
What is the latest version of Stable Diffusion?
As of 2026 the current generation is Stable Diffusion 3.5, released 22 October 2024, in three variants: SD 3.5 Large (8.1B parameters, flagship quality), SD 3.5 Large Turbo (distilled for ~4-step fast generation), and SD 3.5 Medium (2.5B parameters, for consumer GPUs). All use the Multimodal Diffusion Transformer (MMDiT) architecture.
How does Stable Diffusion actually work?
It uses latent diffusion. A variational autoencoder compresses images into a small latent space (roughly 48× smaller than pixel space), the diffusion model learns to remove noise from those latents guided by a text prompt, and a decoder expands the final clean latent back into a full image. Generating an image means starting from random noise and progressively denoising it toward something matching the prompt. Running in compressed latent space is what lets it work on consumer GPUs.
Is Stable Diffusion better than Midjourney?
For peak single-image quality with minimal effort, Midjourney (v7) generally produces better results out of the box. For customisation, control, and cost at volume, Stable Diffusion wins decisively because it is open: you can train custom styles (LoRAs), control composition (ControlNet), and run unlimited local generation with no per-image fee. They serve different needs, and many production teams use both.
What are LoRAs and ControlNet?
LoRAs (Low-Rank Adaptation) are small files that adjust the base model toward a specific style, character, or concept without retraining it. ControlNet lets you condition generation on a structural input like a pose, depth map, or edge map for precise compositional control. Both are core to why professionals choose Stable Diffusion, and neither is possible with a closed model.
Is Stability AI going bankrupt?
Stability AI was in serious financial trouble in early 2024 (losses over $30M in Q1 2024, founder Emad Mostaque’s departure), but in June 2024 it raised roughly $80M, brought in former Weta Digital CEO Prem Akkaraju, and reportedly had significant debt forgiven. By December 2024 the company said it had no debt and triple-digit growth. As of early 2026 it remains independent at roughly a $1 billion valuation with around 190 employees – stabilised, though far smaller than the frontier labs.
Did Stability AI lose the Getty lawsuit?
The most significant ruling came on 4 November 2025, when the UK High Court largely rejected Getty Images’ copyright claims against Stability AI, mainly on jurisdictional and evidentiary grounds (the model was trained outside the UK and was held not to store or reproduce the copyright works). Getty won a limited trademark finding where outputs reproduced its watermarks. The ruling was widely seen as inconclusive on the central question of whether training on copyrighted works is lawful. Separate US litigation (including Getty’s US case and the artists’ class action Andersen v. Stability AI) remains ongoing.
Can Stable Diffusion generate any content?
Technically, downloadable weights can be run by anyone, but permitted use is governed by the licence. Effective 31 July 2025, Stability AI’s Acceptable Use Policy prohibits generating sexually explicit content with its Core Models, and some hosted platforms removed SD 3.5 models in response. The structural reality of open weights is that the licence governs use, not technical capability.
What hardware do I need to run Stable Diffusion?
SD 3.5 Medium is designed to run on standard consumer GPUs (around 9.9 GB VRAM). SD 3.5 Large needs more VRAM for comfortable use. Older models like SD 1.5 and SDXL run on a wider range of consumer hardware. For GPU-free use, hosted services (Replicate, fal.ai) provide API access. We cover local deployment in detail in the on-device and edge AI silo.
What is the difference between Stable Diffusion and Flux?
Flux, from Black Forest Labs, is in many ways the spiritual successor to Stable Diffusion: Black Forest Labs was founded by Robin Rombach and other core members of the original Stable Diffusion research team after they left Stability AI. Flux models are widely regarded as among the strongest open and semi-open image models at the frontier as of 2026, competing directly with the SD lineage on quality while sharing the same architectural heritage.