How a Diffusion Model Works: A Practitioner's Read of the 2026 Image Stack

Every image-diffusion explainer I can find stops in 2022. They draw the U-Net, walk through DDPM noise schedules, count fifty denoising steps, and end with a Stable Diffusion 1.5 sample. That mental model is two architecture generations behind what frontier image models are actually doing in 2026. The current stack is a transformer, running flow matching, in four steps or fewer, and almost nothing in the older explainers transfers cleanly.

I've been using image diffusion models in earnest for about a year, long enough to span the SDXL-to-SD3/FLUX transition. Most of my hands-on time is on the API side: Nano Banana Pro and GPT Image for closed frontier, FLUX.1 and FLUX.2 via BFL, fal, and Replicate for the open-weight side, plus the ControlNet / IP-Adapter / LoRA stack for tighter conditioning. Every lab publishes polished marketing pages and almost no architecture documentation. Most published explainers are either two years out of date or paper summaries written for ML PhDs. This post is the in-between read I wanted and couldn't find: a practitioner's mental model of what a 2026-era image diffusion model is actually doing under the hood.

The cast as of May 2026, per the Artificial Analysis Image Arena¹: GPT Image 2 (high) leads at 1337 Elo, followed by GPT Image 1.5, Nano Banana 2 (Gemini 3.1 Flash Image Preview), and Nano Banana Pro (Gemini 3 Pro Image). Seedream 4.0 (ByteDance), FLUX.2 [max] (Black Forest Labs), and Imagen 4 Ultra (Google) round out the upper band. The open-weights frontier is FLUX.2 [dev] (32B) and the Apache-licensed FLUX.2 [klein] released January 2026.² The architecture story below applies across all of them: they are variations on a shared template.

In this post:

1. The "image models are still U-Nets" fallacy: the 2022 mental model doesn't fit
2. The four building blocks: autoencoder, transformer, flow scheduler, conditioning
3. The latent space is where everything happens: no model touches pixels
4. From U-Net to DiT to MMDiT: three architecture generations in four years
5. Flow matching: the straight-line shortcut: why the step count collapsed
6. Conditioning: how a prompt actually steers the model: text, ControlNet, IP-Adapter, LoRA
7. Distillation: how 50 steps became 1: LCM, ADD, and the sub-second frontier
8. What's actually next: VLM-coupled generation, interactive editing, the open frontier

A note before we open the hood: most frontier labs (OpenAI, Google, ByteDance) don't publish architecture details for their image stacks. The architecture-level claims below are inferred from the published academic lineage (DDPM³, latent diffusion⁴, DiT⁵, rectified flow / SD3⁶), the open-source frontier (FLUX.2 has the most-detailed write-up of any current frontier model²), and what's consistent across product surface area. I'll flag where the evidence is thin.

---

The "image models are still U-Nets" fallacy

If you came to image models in the Stable Diffusion 1.5 era, the natural mental model is: "A U-Net iteratively denoises a 512×512 latent over 50 steps. Text comes in via cross-attention. The output is an image."

This is wrong in a way that matters.

Almost none of those four pieces survive at the 2026 frontier:

• The U-Net is gone. Every leading 2025-2026 model uses a transformer backbone (DiT or its multimodal successor MMDiT). The convolutional inductive bias U-Nets relied on turned out to be a constraint, not a help, once data and compute got large enough.⁵
• 50 steps became 4, or 1. Distillation collapsed inference cost by a factor of 10-50×. FLUX.1 [schnell] is the most-cited open-weight example: a single-digit-step model that produces frontier-comparable output at sub-second latency.⁷
• The denoising trajectory got straighter. DDPM's curved noise schedule got replaced by rectified flow's near-straight path between noise and data. The schedule itself was a bottleneck.⁶
• Text injection moved deeper. SD3 and FLUX.2 don't just cross-attend text into the image branch. They run separate transformer streams for image and text and let them attend to each other bidirectionally at every layer.⁶²

The 2022 view: "A U-Net denoises a latent over 50 steps. Text comes in via cross-attention."

The actual 2026 model: "A transformer with separate image and text streams runs flow matching for 4 steps over a latent the VAE will later decode."

That's the whole shift. The rest of this post is the consequences.

---

The four building blocks

Strip away the marketing and a modern image model has four parts:

1. A latent autoencoder (VAE). Compresses raw RGB pixels into a much smaller latent representation. Typical compression: an 8× spatial downsample with 4-16 latent channels, so a 1024×1024 image becomes a 128×128×16 latent. The model never operates on pixels because pixels are too expensive and too redundant. This is the latent-diffusion idea from Rombach et al. 2022.⁴

2. A transformer (DiT or MMDiT). The actual generative engine. It takes a latent grid full of noise and a text prompt, and at each step it predicts a "velocity": the direction to move the noisy latent toward the clean data. Run this for N steps and the latent goes from pure noise to a coherent image.⁵⁶

3. A flow scheduler. The control loop. It defines what noise level the model sees at each step, and how the predicted velocity gets integrated to take the next step. DDPM, DDIM, Euler, DPM++, rectified flow are all schedulers. Modern frontier models almost all use rectified-flow schedulers.⁶

4. Conditioning streams. Text (always, via a frozen text encoder or a full VLM), reference images, ControlNet maps, IP-Adapter embeddings, LoRA weights. Each gets encoded and injected into the transformer.

Every current image model fits this template: GPT Image 2, Nano Banana Pro, FLUX.2, Seedream 4.0, Imagen 4 Ultra, and the open-source SDXL/SD3/FLUX lineage they grew out of. The differences are in the size of each block, the conditioning vocabulary, the training data, and which post-training tricks got bolted on.

The two-stage training recipe is also stable: train the VAE separately on raw images to get a good latent space, then freeze the VAE and train the transformer on (latent, prompt) pairs to learn the noise→data mapping. This is the same pattern as the video stack's autoencoder + DiT split: one transformer, swappable input encoders, post-training doing most of the heavy lifting.

---

The latent space is where everything happens

The VAE's job is to take an RGB image and turn it into a much smaller cube of numbers that the transformer can chew on.

For a single 1024×1024 image, this is well-understood. The encoder runs an 8× spatial downsample plus a channel transform: the output latent is a 128×128 grid with anywhere from 4 to 16 channels per position. That's a ~96× compression in raw element count (1024×1024×3 = 3.1M values → 128×128×16 = 262K values). The decoder reconstructs the image from the latent at the end. The transformer never touches pixels.

	Image (RGB)	Latent (post-VAE)
Shape	1024×1024×3	128×128×4 to 128×128×16
Element count	3.1M	65K to 262K
Compression	n/a	~12× to ~48× in elements
What lives here	Pixels	Compressed perceptual features

The third row is the entire economic story of image diffusion. A modern image model isn't generating 3.1 million pixel values one denoising step at a time. It's generating ~262K latent values, which the VAE expands to pixels at the end. Once you internalize this, every "why is text rendering bad?" or "why does fine detail look smudged?" question becomes obvious. Anything that didn't survive the VAE's compression isn't in the model's working space.

This is also why every frontier model lab quietly ships a new VAE every couple of years. FLUX.2 explicitly re-trained its latent space from scratch and framed it as the "Learnability-Quality-Compression trilemma": you want a latent compressed enough to be cheap, learnable enough that the transformer can model it well, and high-quality enough that the decoder doesn't introduce artifacts.² Pick any two and the third gets hard.

Once the latent grid is the unit of work, the transformer takes it from there.

---

From U-Net to DiT to MMDiT

The transformer is where the field moved fastest. Three architecture generations in four years:

Generation	Backbone	Text injection	Era	Examples
G1: U-Net	Convolutional U-Net	Cross-attention from frozen text encoder into image branch	2020-2023	DDPM, Stable Diffusion 1.5, SDXL (U-Net++), Imagen 1-2
G2: DiT	Transformer over latent patches	Cross-attention or AdaLN injection	2023-2024	DiT, PixArt, partly Imagen 3
G3: MMDiT	Two parallel transformer streams (image + text)	Bidirectional self-attention between streams at every block	2024-present	SD3, FLUX.1, FLUX.2, Seedream, Imagen 4 (likely), GPT Image (likely)

G1 (U-Net): The original DDPM paper³ used a U-Net and basically every diffusion model through 2023 followed. The U-Net's hierarchical encoder-decoder structure was a strong inductive bias for image data: convolutional layers that progressively downsample, then upsample with skip connections, mimicking the structure of natural images. SDXL extended this with a larger U-Net and a second text encoder, but the bones were the same.

G2 (DiT): Peebles & Xie's 2022 paper⁵ showed that you can replace the U-Net entirely with a transformer over latent patches and get better scaling: predictable loss curves as you grow parameters and compute, the same way LLMs scale. The U-Net's inductive bias turned out to be a constraint at scale. DiT became the bridge generation.

G3 (MMDiT): SD3's contribution wasn't just adopting DiT. It was adding a second parallel transformer stream for the text tokens, and letting the image and text streams attend to each other bidirectionally at every layer.⁶ The original quote: "We present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension, typography, and human preference ratings."

This is the architecture every 2025-2026 frontier image model now resembles. FLUX.2 makes the lineage explicit: it couples a Mistral-3 24B vision-language model to a rectified flow transformer, and lets the VLM bring "real world knowledge and contextual understanding" while the transformer "captures spatial relationships, material properties, and compositional logic."² You can read this as MMDiT taken to its logical end: the text stream is a full VLM rather than a frozen text encoder.

The open-weights side mirrors the closed labs almost step-for-step. SDXL was peak G1. SD3 / SD3.5 are G3. FLUX.1 / FLUX.2 are G3 with progressively bigger and better-conditioned text streams.

---

Flow matching: the straight-line shortcut

The transformer is half the story. The other half is what trajectory the model is asked to follow from noise to data.

In classical DDPM, the trajectory from a clean image to pure noise is curved: a Markov chain that adds Gaussian noise at each step on a precisely tuned schedule. To generate, you run that chain in reverse, predicting and removing noise one step at a time. The trajectory is curved enough that you need 50-1000 reverse steps to get a clean image.

Flow matching reframes the problem.⁸ Instead of learning a noise-removal step at every point on a curved trajectory, the model learns a velocity field: at any (latent, time) point, what direction should we move toward the data? With the right loss, this velocity field defines a flow you can integrate from noise to data in fewer, larger steps.

Rectified flow (Liu et al. 2022⁹, built on by SD3) goes further: it explicitly trains the model to make those trajectories nearly straight lines between noise and data. A straight line needs fewer integration steps than a curve to follow accurately.

	DDPM (G1 era)	Rectified flow (G3 era)
Trajectory shape	Curved	Near-straight
Typical steps to converge	50-1000	4-50
What the model predicts	Noise added at this step	Velocity (direction toward data)
Sampler complexity	Higher (DDIM, DPM++ tricks needed)	Lower (Euler is competitive)

Rectified flow plus distillation is why the 2026 leaderboard has FLUX.1 [schnell] generating images in 0.3 seconds and Z-Image Turbo in 2.3, while non-distilled multi-step quality models like GPT Image 1.5 sit at 36 seconds.¹ A 100× spread between distilled few-step and frontier-quality multi-step inference now exists, and the floor keeps dropping.

The math of why straighter trajectories need fewer steps is in Lipman et al.'s flow matching paper⁸ and the SD3 technical report. The practitioner takeaway is: picking your scheduler is no longer a marginal choice. The DDPM/DDIM mental model is obsolete for 2026 frontier inference.

---

Conditioning: how a prompt actually steers the model

A diffusion model with no conditioning generates random images from its training distribution. Useless. Every controllable image model adds conditioning streams that steer the noise→data trajectory.

There are two axes to think about:

• Where in the pipeline the conditioning enters: at the text encoder, at the transformer (cross-attention or self-attention), at the latent (via a parallel adapter network), or at the VAE.
• What kind of signal the conditioning carries: semantic (text), spatial (edges, depth, pose), identity (a reference face), style (a reference image or a LoRA), or composition (a reference layout).

The five most-used conditioning patterns:

1. Text (always). A frozen text encoder (CLIP, T5, or, at the FLUX.2 frontier, a full VLM like Mistral-3 24B) embeds the prompt. Those embeddings get fed into the transformer either via cross-attention (G1/G2) or as a parallel attention stream (G3 / MMDiT). The richer the text encoder, the better the model handles long, structured prompts and rare entities.

2. Classifier-free guidance (CFG). Run the model twice per step, once with the prompt and once unconditional, and extrapolate the difference: pred = uncond + scale × (cond - uncond). Higher scale = more prompt adherence, less diversity. This isn't a separate conditioning stream; it's a sampling trick on top of text conditioning. Almost every diffusion model uses it.

3. Spatial control: ControlNet. Zhang et al. 2023¹⁰ introduced a parallel network ("zero convolutions") that learns to inject spatial maps (Canny edges, depth, OpenPose skeletons, segmentation) into a frozen base model without destroying its priors. Even in a 2026 MMDiT world, the idea (frozen base + adapter for spatial control) is what every modern conditioning system descends from.

4. Identity / style: IP-Adapter and LoRA. IP-Adapter encodes a reference image and injects its features alongside the text features. Useful for "draw this character in this scene." LoRA fine-tunes a small low-rank adapter on top of the frozen base. Useful for "draw in this style" or "draw this specific subject reliably." Both are now baseline expectations for any production image workflow.

5. Multi-reference (FLUX.2 era). FLUX.2 ships with native support for up to 10 reference images in one inference call (character, product, style, layout) combined into a novel output.² This collapses what used to be a ControlNet-plus-IP-Adapter-plus-LoRA workflow into a single model call.

The trend across G1 → G2 → G3 is fewer external adapters, more conditioning baked in. The 2022 ControlNet ecosystem existed because base models were narrow; the 2026 frontier models are wide enough that the adapter ecosystem is mostly consolidating into multi-reference and multi-modal conditioning at the transformer level.

---

Distillation: how 50 steps became 1

A 2022 Stable Diffusion sample took 50 steps. A 2026 FLUX.1 [schnell] sample takes 1-4. The architectures changed, the schedulers changed, but the single biggest reason for the speedup is distillation: training a fast student model to match a slow teacher model's outputs in fewer steps.

Three distillation lineages dominate the modern image stack:

1. Latent Consistency Models (LCM). Luo et al. 2023⁷ train a "consistency" model that directly predicts the solution of the probability-flow ODE in latent space, so a 2-4 step LCM can produce output indistinguishable from a 50-step teacher. The original paper: "a high-quality 768×768 2~4-step LCM takes only 32 A100 GPU hours for training" on top of an existing pretrained latent diffusion model. LCM-LoRA, applying the LCM idea as a LoRA on top of a base model, became the most-deployed open-source few-step trick.

2. Adversarial Diffusion Distillation (ADD). Stability AI's contribution behind SDXL Turbo. Combines the consistency-model idea with a GAN-style adversarial loss against a discriminator that distinguishes student samples from teacher samples. The original BFL team helped invent this; it shows up in their list of foundational contributions.

3. Distilled rectified flow. Once the teacher is a flow-matching model (SD3, FLUX.1), the trajectories are already close to straight, and distillation just has to compress 20-50 Euler steps down to 1-4. FLUX.1 [schnell] is the canonical example. FLUX.2 [klein] is the size-distilled variant: Apache-licensed, smaller than the base model, "many of the same capabilities as its teacher."²

The practitioner consequence: almost every fast model on the leaderboard is a distilled version of a slow teacher. When a model card says "few-step" or "turbo" or "schnell" or "lightning" or "consistency," it's distilled. The teacher does the heavy lifting at training time; the student does the cheap inference at serve time.

This also explains a counterintuitive pricing pattern. On the Image Arena, some of the most expensive frontier models (GPT Image 1.5 at $133 per 1K, Nano Banana Pro at $134, GPT Image 2 at $211) are also the slowest, in the 35-second range. The cheap fast models are distilled. The expensive slow models are running many integration steps to squeeze out the last few percent of quality. Both categories use the same architecture template.¹

---

What's actually next

The seven sections above describe what 2026 image models can do. The capability has moved fast: most of the failure modes a 2024-era practitioner would have called out (broken text, mangled hands, identity drift, factual mistakes about real-world objects) are largely solved at the frontier. What's left is the next-frontier work, and three pieces of it stand out:

VLM-coupled generation, taken further. FLUX.2 pairing a 24B VLM with the rectified flow transformer is the first frontier example, and it's clearly the right direction. The next step is making the VLM a full bidirectional partner: the image model proposes a sample, the VLM critiques it on factual grounds, the image model regenerates the bad regions. This is the diffusion analog of LLM tool use, and nothing is productized.

Interactive, region-level editing at low latency. FLUX.2 [klein]'s January 2026 launch was framed as "interactive visual intelligence": sub-second, region-aware editing on consumer hardware. The frontier is the same as the video region-level editing problem: select a region, change one element, regenerate just that part with the rest preserved. LazyDrag and similar 2025 research lines are credible architectural paths; production-grade tools are not yet here.

The open-weights frontier closing the gap. FLUX.2 [dev] (32B) is the strongest open image model as of May 2026. Closed-weight GPT Image 2 and Nano Banana Pro still lead on the Arena. The open-closed gap on image generation has been narrower than on text generation for the last two years, and the FLUX.2 [klein] release suggests the pattern continues: capable, distilled, Apache-licensed models within 12 months of frontier.

What's not changing: the four-block template is stable. The work for the next two years is in better VAEs, deeper VLM coupling, longer-context conditioning (multi-reference scaling beyond 10), and faster distillation. Not in replacing the transformer or rebuilding from scratch. Image diffusion is in its post-MMDiT phase. The plumbing is settled. The product surface is the action.

---

Related: How a Vision LLM Works · How a Video Model Works · Context Engineering Patterns

This is the third and final post in the multimodal-stack series. The companions: How a Vision LLM Works on the read side and How a Video Model Works on the temporal extension.

Footnotes

1. Artificial Analysis, Text to Image Leaderboard (link). Standings cited as of 2026-05-04; the leaderboard turns over fast and the spread between top models is small. Spot-check before quoting any specific name. ↩ ↩² ↩³

2. Black Forest Labs, FLUX.2: Frontier Visual Intelligence, November 2025 (link). The most-detailed publicly available architecture reference for any current frontier image model: explicitly names the Mistral-3 24B VLM coupling, the rectified flow transformer, the new VAE, and the variant ladder ([max], [pro], [flex], [dev], [klein]). Most architecture-level claims in this post that go beyond product pages are anchored here. See also the FLUX.2 [klein] launch, January 2026. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

3. Ho, Jain & Abbeel, Denoising Diffusion Probabilistic Models, arXiv:2006.11239, June 2020 (link). The paper that established modern diffusion. Defined the U-Net + curved-noise-schedule template that dominated 2020-2023. ↩ ↩²

4. Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, arXiv:2112.10752, December 2021 (link). Established the latent-diffusion paradigm: compress with a VAE, denoise in the much smaller latent space, decode at the end. Every modern image model uses this structure. ↩ ↩²

5. Peebles & Xie, Scalable Diffusion Models with Transformers, arXiv:2212.09748, December 2022 (link). Introduced the Diffusion Transformer (DiT), demonstrating that the U-Net's convolutional inductive bias isn't necessary: a transformer over latent patches scales better and ultimately replaced U-Nets at the frontier. ↩ ↩² ↩³ ↩⁴

6. Esser et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, arXiv:2403.03206, March 2024 (link). The Stable Diffusion 3 technical report. Introduced the MMDiT architecture (separate transformer streams for image and text with bidirectional attention) and demonstrated rectified flow's advantage at high resolution. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

7. Luo et al., Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference, arXiv:2310.04378, October 2023 (link). The breakthrough distillation paper that enabled 2-4 step inference on top of pretrained latent diffusion models. The lineage that produced FLUX.1 [schnell], SDXL Turbo, and the sub-second image models on the current leaderboard. ↩ ↩²

8. Lipman et al., Flow Matching for Generative Modeling, arXiv:2210.02747, October 2022 (link). The simulation-free training framework for continuous normalizing flows that subsumes diffusion as a special case and enables the straighter trajectories used in modern frontier models. ↩ ↩²

9. Liu, Gong & Liu, Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, arXiv:2209.03003, September 2022 (link). Introduced rectified flow specifically: explicitly trains the model to produce nearly straight trajectories between noise and data, which is what enables few-step inference downstream. ↩

10. Zhang, Rao & Agrawala, Adding Conditional Control to Text-to-Image Diffusion Models, arXiv:2302.05543, February 2023 (link). The original ControlNet paper. Introduced the "zero convolutions" trick that lets a parallel network add spatial conditioning (edges, depth, pose) to a frozen base model without destroying its priors. ↩