The text graph model has been expanding the model, data, and computing power, but the amount of information Caption trains with images is rarely studied as an independent variable system. The ByteDance Seed team found that natural language Caption increases does not mean that the model receives more available visual supervision; the amount of information attached to the image in Caption is better able to predict the training losses that the scattered model will eventually achieve compared to the length.
Based on this finding, the team came up with Structured Prompt and jointly upgraded the text conditions from both Diffusability and Promptability, ultimately achieving significant improvements in complex combination, reasoning and world knowledge generation tasks.
The length of natural language is rapidly matching; structured conditions continue to increase image information and reduce dispersion training losses along a unified relationship.
Over the past few years, progress has been made in the visual model almost along a familiar path: bigger models, more data, and higher training algorithms.
However, there is an easily overlooked difference between text graphics and language models. Language models can be self-monitored directly from the text sequence; text graphics models rely on pairing images with Caption to learn what type of text corresponds to what type of visual content. Images may contain a large number of objects, properties, locations, movements, and relationships, but only parts that are accurately described and clearly bound by Caption can be passed on to the model as a text condition supervisor.
So there's a fundamental question: besides continuing to expand models, data, and computing, can we make generating models better by adding the image information that Caption carries?
In this new work, the ByteDance Seed team explores this issue. The core conclusions can be summarized in one sentence:
It is not the number of tokens in the Caption that is actually extended with the text conditions, but the image information that can be used by the model.
Title: Scaling Properties of Text Conditioning in Visual Generation
Written by Zilong Chen, Chaorui Deng, Kunchang Li, Hongyi Yuan, Haoqi Fan
The paper:
The main page of the project:https://heheyas.github.io/context-scaling
The code is:https://github.com/heheyas/context-scaling
The model:https://huggingface.co/collections/heheyas/context-scaling
The online demo:
Hugging Face Paper：
Prompt is growing.
Why didn't the model get stronger?
One intuitive approach is to write the training Caption or user Prompt longer and more in detail. More tokens should appear to mean more oversight and should help the model generate more complex images.
Experiments have given different answers. In many existing open source text graphics systems, natural language Prompt increases rapidly with length and output is even lower than the performance of their shortest Prompt. Even if the spreadsheet model is specifically trained on the same set of long text Captions, the benefits are limited.
In order to see the phenomenon more clearly, the team designed an image reconstruction experiment for a fixed-base network. For the same reference image, the team started from the same complete tag, generating four gradually increasingly detailed natural language captions, then using the same Qwen-Image model and random seed attempts to restore the image. These captions describe the same entity and relationship, and subsequent versions increase the length mainly by supplementing and extending natural language expressions.
Unsurprisingly, while Caption has changed significantly, the quality of image reconstruction is almost no longer improved. The new prose is more about interpreting, rewriting, or connecting to existing content, rather than continuously adding new, stable, visual variables.
N.L. Caption continues to grow in size and build up, while the gradual recovery of SP fields continues to improve.
This suggests that the length of the caption is only a weak substitute variable. A text can be written long, but it is still not clear which object a property belongs to, what the relationship between the two objects is, where each is located, and their front and back levels in the image.
How do you measure real image information in Caption?
If the number of tokens is not enough to measure the oversight strength, the information tied to the image in the Caption needs to be directly measured. For this, the team modified two complementary indicators from existing work: Grounded Perplexity Gain (GPG) and Effective Detailness (ED).
GPG: The image makes Caption more predictable.
GPG is a white box indicator of the probability of a model token that needs to be read. For the same Caption, the team respectively allows a frozen visual language model to see and not see the matched image, and calculates the likelihood of the pair of Caption content tokens after the image appears. If the Caption contains a large amount of information that is closely tied to the image, seeing images should significantly improve the predictability of the model for these tokens.
ED: Caption covers how many reliable image properties.
ED is a black box semantic indicator that does not rely on token probability. It extracts the attributes of the object-oriented context from the image and Caption respectively, and then calculates the accuracy of the Caption attributes and the recall rate of the image attributes.
Two metrics illustrate the same problem from different angles: GPG focuses on the statistical dependence between images and text, and ED focuses on whether Caption accurately covers verifiable visual content.
Definition and measurement trends of Figure 6G and ED
The amount of information can predict the training losses of the spread model
Next, the team fixed images, model architecture, initialization methods, optimized configurations and training budgets, changing only the training Caption. The entire experiment included 15 Caption configurations: three different lengths of natural language versions, six Structured Prompt versions of phased field recovery, and six spatial expression or field shielding variables. Each configuration started from the same BAGEL continued-training checkpoint and trained a spreadsheet model independently.
The results show that the token number of the natural language Caption does not have a uniform relationship to the training results; however, when the horizontal axis is converted into the amount of Caption information, the different formats and degrees of detail of the configuration fall on a highly regular curve:
Diffusion loss after convergence is almost linearly related to GPG, Pearson r = -0.984;
Diffusion loss after convergence tends to align with ED, with Pearson r = -0.971 in the log-log space.
GPG is also highly consistent with ED's ranking of different Caption configurations, with Spearman ρ = 0.96.
The team called it the scaling properties of text conditioning. It is not a theoretical law that applies to all models, but is based on the experience gained under fixed architecture and training recipes.
First, it transforms the information quantity of Caption from a vague concept of titanium data to a training variable that can be controlled and measured: the information quantity can predict the training losses that the model ultimately achieves when the model, image and computation remain unchanged.
Second, once the labeling has been completed, it is possible to compare candidate Captions with GPG or ED in the same training recipe before deciding whether to put in expensive scattering model training. Two indicators can still accurately predict convergence losses for the six unmatched Caption variants.
Figure 7 shows a stable correlation between caption loss and caption information in a fixed training recipe.
Structured Prompt：
It's not only more information, it's easier for models to use.
Previous experiments reveal a key point: it is not enough to simply add natural language prose, and new information needs to be organized in a stable and clear way.
Therefore, the team proposed Structured Prompt (SP), which uses structured JSON to represent the visual variables in an image.
At the global level: image intent, scenery, atmosphere, style, lighting and photographic information;
Layer of elements: identity, attributes, movement, location, depth of choice and local photographic information of each subject;
Relationship layer: position, overlapping, interaction and semantic relationships between different elements.
Compared to free text, the key to SP is not just to capture the appearance of JSON, but to allow different visual variables to enter a stable naming field, reducing attribution, spatial relationships, and object-binding ambiguity. In fixed-base reconstruction experiments, the quality of reconstruction continues to improve as SP fields gradually recover; in complete training sweep, the increase in field coverage also simultaneously increases GPG, ED, and reduces diffusion loss after folding.
Figure 5S: Structured Prompt organizes global, elementary, and cross-elemental visual variables into named fields.
In order to generate full SP on large-scale training data, the team constructed an image-to-SP markup stream. General VLM is responsible for the overall semantics and local content, Sapiens supplements human gesture evidence, DepthAnything V2 provides relative depth, SAM 2.1 provides masking and shadowing clues, and finally VLM unifies this information into a consistent full SP.
The paper Figure 8V says that the LM teamed up with experts in gesture, depth, and division to build a complete Structured Prompt.
The team called Diffusability the enhancement of a Caption's ability to expose and organize image supervision to scattered models. SP is precisely this aspect: allowing the model to learn more, clearer visual variables from text conditions without changing the scattered model architecture.
Promptability：
If you have a good structure, you need an LLM to fill it up.
Full SP can be extracted from a pairing of images when training, but only one user word is actually generated, with no reference image or oracle tag. The system also requires an LLM prompter to extend user requests to detailed, coherent, and non-infringing original restrictions of SP.
The team called this ability to instantiate structured conditions from user requests Promptability.
Generation Quality = Diffusability × Promptability
The multiplication here is an organizational perspective rather than a matched mathematical multiplication formula: Diffusability describes what the spread model can learn from Caption, and Promptability describes whether LLM can actually fill in high-quality Caption instances in reasoning.
Figure 4S: Structured Prompt connecting marking, measuring, diffuser training, prompting training and final generation.
First, the team fixed the SP schema and Qwen-Image diffuser, replacing only the zero-sample LLM prompter. With Qwen 3.5 expanding from 0.8B to 397B, GenEval++ in the thinking mode increased from 46.4% to 86.8%. Except for the smallest models, which are easy to replicate in the thinking process and cannot produce effective JSON, chain-of-thought brings further improvement on all other scales. This shows that the general LLM's modeling and reasoning capabilities can be directly converted to better image generation results through Caption interfaces.
However, the zero-sample LLM still tends to generate insufficient information, constructing simpler SPs. To further enhance Promptability, the team uses three stages of training:
SFT learns the expected SP content distribution of the scattering model, not just the JSON format;
Cold-start: How to distill SP from privileged reasoning traces with pairs of images based only on user requests;
RFT continues to optimize on the rollout generated and rendered by the prompter itself, by selecting high-trust trajectories by the verifier, and by providing confidential token supervision through on-policy self-distillation by the image-conditioned teacher.
Dissolving experiments have shown that three phases play a different role: SFT brings the largest single-phase structural enhancement, cold-start strengthens the derivation from user requests to SP, and verifier-gated OPSD achieves the strongest results on the prompter's own distribution.
After the fixed schema and diffuser, the output quality improved with the LLM prompter scale and reasoning mode.
Figure 10S: Cold-start and verifier-gated RFT three-phase prompter training.
Structured says:
It also makes generating processes easier to iterate and modify.
The fielding of SP also has a natural advantage: when an error occurs in generating an image, the system can locate and modify the corresponding object, property, relationship, or layout field instead of rewriting the entire natural language Prompt.
Based on this, the team built the refin-r.