Text Conditioning via CLIP (and Beyond)

TL;DR CLIP (2021) made text-to-image work by learning a shared embedding space, then injecting those embeddings into the denoising U-Net via cross-attention. Its hard 77-token limit and visual-contrastive bias drove a wave of alternatives: Imagen (2022) showed a frozen T5-XXL outperforms CLIP on compositional prompts [4]; SDXL stacked two CLIPs; SD3 added T5 on top; Flux split duties between pooled CLIP (global modulation) and T5 tokens (fine-grained attention). In 2025 HiDream-I1 uses four encoders including Llama-3.1-8B [12]. For prompt writers: front-load concepts in SD 1.x (token 78 onward is invisible), write naturally in Flux/SD3, and lean on complex syntax only where T5 is present.

1. CLIP: A Shared Text–Image Embedding Space

CLIP (Contrastive Language-Image Pre-training, OpenAI 2021) trains two encoders jointly: a ViT image encoder and a GPT-2-style text transformer, using a symmetric contrastive loss on 400 million image-caption pairs [1]. The objective: maximise cosine similarity between matched pairs, minimise it for all others. The result is a latent space where “a red cube on a blue table” sits geometrically close to images that match.

The text encoder uses byte-pair encoding with a hard 77-token limit (76 content tokens + 1 EOS), inherited from GPT-2’s absolute positional embeddings [3]. This is not a tunable parameter—it is an architectural constraint.

OpenCLIP ⭐ 13.9k [13] is the open reproduction trained on LAION-400M, LAION-2B, and DataComp-1B; SDXL’s CLIP-G encoder uses an OpenCLIP ViT-bigG/14 checkpoint.

2. Cross-Attention: How Text Gets Into the U-Net

Latent Diffusion Models (Rombach et al. 2022) introduced cross-attention layers at multiple resolutions of the U-Net as the mechanism for injecting conditioning [5]:

• Query (Q) — projected from the flattened spatial features of the noisy latent
• Key (K) and Value (V) — linearly projected from the text token embeddings

The attention score matrix (softmax(QKᵀ/√d)) shows which text tokens influence which spatial image regions [20]. “Red cube” activates upper-left pixels; “blue table” activates the lower region. Text conditioning is, at its core, a cross-attention routing problem.

3. Classifier-Free Guidance (CFG)

CFG amplifies the text signal without requiring a separate classifier [17]. During training, 10–20% of captions are randomly replaced with a null token ∅, so the model learns both conditional and unconditional noise prediction. At inference, the two predictions are combined:

ε̂ = ε(x, ∅) + w · (ε(x, text) − ε(x, ∅))

w is the guidance scale. Higher w → stronger prompt adherence, lower diversity. The technique is described as “now as essential as dropout for regularisation” [1].

4. The 77-Token Wall

SD 1.x and 2.x silently truncate any prompt token beyond position 77 [16]. Words at the end of a long prompt simply do not exist for the model. Common workarounds:

• Chunking (AUTOMATIC1111): break into 75-token segments, encode each, concatenate embeddings.
• Compel library: handles chunking transparently for SDXL.
• Long-CLIP: extends positional interpolation to 248 tokens [15].

Practical rule for CLIP-only models: put the most important subject and style words first. “Photorealistic portrait of an astronaut, golden hour lighting, bokeh” beats the reverse ordering when the full prompt exceeds 77 tokens.

5. DALL-E 2: Hierarchical CLIP Conditioning

Rather than conditioning the U-Net directly on the CLIP text embedding, DALL-E 2 inserts an intermediate step [2]:

1. Prior — a diffusion model converts the text embedding into a CLIP image embedding.
2. Decoder — a separate diffusion model generates the final image conditioned on that image embedding.

The CLIP image embedding space has richer visual structure than the text embedding space alone; conditioning on it improves diversity with minimal loss in caption similarity. The hierarchical design also enables image variation (re-run decoder on a real image’s CLIP embedding) and style mixing.

6. Imagen: The T5-XXL Pivot

Google’s Imagen (2022) swapped CLIP for a frozen T5-XXL encoder and reported a striking finding: scaling the text encoder matters more than scaling the diffusion UNet [4]. Human raters strongly preferred T5-XXL over CLIP for image-text alignment, particularly on compositional prompts (“a red cube on top of a blue sphere”).

Why T5 wins on composition:

• T5 is pre-trained on the C4 text corpus—deep language understanding of grammar, syntax, spatial relations.
• CLIP’s text encoder was only ever optimised to be close to image embeddings, not to model language structure.
• T5 token sequences extend well beyond 77, supporting up to 512 tokens in Imagen’s configuration.

The trade-off: T5 has no visual grounding—it cannot leverage CLIP’s visual-semantic alignment for style and aesthetic control. Modern architectures learned this lesson by combining both.

7. The Multi-Encoder Era

SDXL concatenates CLIP-L (768-dim) and CLIP-G (1280-dim) token embeddings to a 2048-dim sequence [6], and additionally passes the CLIP-G pooled embedding alongside the timestep to adaptive normalisation layers for global style and aspect-ratio conditioning [7].

SD3 introduced the Multimodal Diffusion Transformer (MMDiT): text and image tokens flow through parallel streams that interact via multimodal self-attention, using separate weight sets for each modality [8]. T5-XXL is critical for generating text-inside-images; it can be dropped with minor quality loss on other content [9].

Flux.1 uses the same MMDiT-style joint attention but with a cleaner division: the CLIP pooled embedding (single vector) controls global modulation via AdaLN, while T5 token embeddings are concatenated directly with image latent tokens and attend jointly [10] [11]. This means in Flux the CLIP portion handles “what style/vibe” and T5 handles “what subject/scene/composition.”

HiDream-I1 (April 2025, 17B parameters, MIT licence) extends to four encoders: CLIP-L/14, CLIP-G/14, T5-XXL, and Llama-3.1-8B Instruct [12]. The LLM component enables bilingual (Chinese/English) instructions and spatial-relational reasoning (“X behind Y, to the left of Z”) that contrastive encoders struggle with. The encoder evolution is summarised across the SD family in [18].

8. Decoder-Only LLMs as Text Encoders (2025 Frontier)

A CVPR 2025 study [14] systematically evaluated decoder-only LLMs (Qwen2-7B, Mistral-7B, Llama3-8B, Gemma2-9B) as drop-in text encoder replacements. Key finding: using only the final layer output underperforms T5-XXL, but layer-normalized averaging across all layers significantly outperforms it:

The gains are especially pronounced on compositionality: negation (“without any text”), spatial comparison, and differentiation tasks—skills where CLIP fundamentally fails. The cost: these are 7–9B models vs T5-XXL’s 11B, but they encode language structure that CLIP’s 355M-parameter text encoder never learned.

A separate compression result: a distilled T5-base (220M parameters) matches T5-XXL image quality at 1/50th the size [19], suggesting the XXL scale is often wasteful.

9. Practical Prompt-Engineering Takeaways

The progression from CLIP to multi-encoder stacks is a progression from visual tag matching to language understanding. Understanding which encoder your model uses tells you whether to write a tag list or a sentence.