As the agent performs long-range tasks, code, tool output, and interaction records accumulate, as does the context.
Generating a summary, compressing the context, can reduce input, but may also remove the details required for subsequent tasks. A section that seemed irrelevant at the time, reaches the next step, perhaps the key clue. It is difficult to recover it if the model can only see the result of the compression.
DeepSeek-OCR shows another path: allow images to carry text, with fewer visual tokens to represent text information. Compressed pages can cover the entire text, but clarity becomes a new bottleneck. The smaller the characters, the more difficult the model is to identify; too tightly compressed, the answer is on the page, and the model can't see clearly.
Many compression schemes are one-way for the model: after compression, it can't look back on demand.
The Apple Basic Large Model team and Duke University proposed the LensVLM to allow the model to first browse the compression page covering the full text and then open the page related to the problem. In a seven-text Q&A review, it achieved an average accuracy of 68.9% with 4.3 times the effective compression rate, which is close to 72.4% for full text references.
The work was recently picked up by NeurIPS 2026 and gained a lot of attention on X as well, sparking much discussion about how to convert text into images for context-understanding purposes. Victor Mustar, product manager of Hugging Face, presented the study on X, and related posts received about 280,000 views. The public LensVLM 9B Demo has also gone online, allowing readers to experience the model selection page, opening and answering process.
The paper:
The model:https://huggingface.co/apple/LensVLM-9B
The code is:https://github.com/apple-aiml-research/ml-lensvlm
Demo：
The text becomes a graph, what do you miss?
Language models typically break down text into text tokens; visual language models convert images into visual tokens through visual encoders. A page image that contains enough text indicates that the visual token it needs may be less than the text token it needs to directly input the original.
The characters, page size and text density can be used to adjust the compression rate without having to redesign the word table. Compressed images can cover the entire text, allowing scattered evidence at different locations to be found by the model compared to first retrieving a small number of snippets and then reading them.
The cost is also direct: the smaller the compression of the characters, it may exceed the recognition of the visual encoder, and even be difficult to distinguish at the pixel level. In the paper, the basic model directly reads a five-fold compression image, with an average accuracy of only 31.3%, significantly lower than the full text reference.
The higher the compression, the fewer visual tokens, the harder it is to recognize the characters.
Select a page and read it carefully.
LensVLM separates the positioning evidence from the reading evidence. The model first receives the question and compresses the entire image, selects a page worth reading, and then calls the opening tool. For pages rendered in plain text, the tool returns the corresponding original and adds it to the context. After reading, if there is not enough evidence, the model can continue to select the next page.
There is a question in the demo: What government jobs did the woman who played Corliss Archer in the film Kiss and Tell fill? The model faces 15 pages of compression, picks the 10th page, calls the read_page to read the original, and finally answers:
The system needs to retain the original or original image: compress the page to allow the model to browse the full text, while unfolding the contents to fill in the required details.
What page to choose, training is needed
Adding a tool to a model does not mean that it knows which page to read.
Based on the Qwen 3.5-9B-Base, the study first selected samples that it could not directly answer, and then allowed Qwen 3.5-397B to use trajectories based on questions, actual evidence, and standard answer-generating tools.
Subsequently, research using DAPO reinforcement learning continued to optimize the trajectory the model itself generated. The reward was based on the correctness of the answer, and only when the answer was correct, did the tools receive additional allowances.
Under the 5-fold initial compression setting, supervised mitigation will increase the average accuracy rate from 31.3% to 63.7%, and reinforcement learning will further increase to 68.9%.
LensVLM's reasoning and training process. Models alternate between reasoning and opening pages to learn selective reading through supervised mitigation and reinforcement learning.
How much more can you read?
The main ratings cover NQ, HotpotQA, MuSiQue, HELMET, and untrained RULER, Qasper and LongBench, and include single-jump and multi-jump questions.
In the 15-fold initial compression setting, the average accuracy of the base model directly reading the compressed image dropped to 18.1%, while the LensVLM dropped to 52.1%. Although there is still a gap with reference to the full text, opening the page after selection significantly mitigates the decrease in accuracy resulting from high compression.
However, unfolding the original also consumes the input budget, and it is not possible to calculate just how small the original image is. The paper therefore reports that the effective compression rate is lower: using the original token number, except for the input token number of the compressed image and the unfolded content combined.
After accounting for the expense, the 5×, 10×, 15× three initial settings correspond to an effective compression ratio of 4.3×, 7.4×, 10.1× respectively.
The average accuracy and effective compression rates of the seven text questionnaire assessments. The input after the rear axle counting is performed shows the balance between the accuracy and the input budget of the different approaches.
Speed and cost in context
In the long context, 15x initial compression setting of the 32K 140K token, the end-to-end delay for full text input is 3.62 seconds, and the LensVLM is 2.83 seconds. After pre-offline rendering and caching of page images, the delay is further reduced to 2.52 seconds, which is about 30% less than full text input.
Under the same setup, the peak KV cache of the LensVLM is about a quarter of the full text input. In clutter and concurrent testing, the throughput of the pre-cached version of the page is about 3.7 times that of the full text input; at an estimated $6 per GPU hour, the GPU service cost per thousand requests is reduced by about 73%.
In short contexts, direct text entry is still faster. The advantages of this scheme are mainly the long documentation, as well as multiple query scenarios of page images that can be replicated.
End-to-end efficiency and latency. The pre-cached version of the page uses images rendered offline in advance, and the throughput is measured in clutter and simultaneously.
What is PDF going to be?
Plain text pages can be returned directly to the original, while native PDFs also contain tables, graphics, and layouts. Only text is extracted, which may miss the visual information needed to respond.
The study also trained two document understanding variants: one to open the page as OCR text, the other to return the high-definition page and evaluate it on the MMLongBench-Doc.
When the number of page pixels is reduced by 15 times, the accuracy of directly reading compressed pages is 31.1%, the OCR spread variance is 32.1%, and the HD page spread variance is 41.9%, corresponding to 10.3 times the effective compression.
In this evaluation, the graphs and layout information retained by high-definition pages was more helpful than OCR text, but also led to more input costs. Open formats required content selection: pages rendered in plain text are suitable for returning to the original, while documents that rely on visual information are more suitable for returning high-definition pages.
Summary
The core of LensVLM is to allow the model to read compressed content while still viewing the original on demand. The model first browses the entire article and then uses the reading budget on the pages required by the current problem. As the context of the model grows, how to make the model actively allocate the reading budget, determine which content needs to be scrutinized, which needs to be read carefully, and when to revisit, may become as important a research direction as the context window on the extension.
Introduction by the author
Roy Xie, Ph.D. student in computer science at Duke University, specializes in basic model post-training, and the use of long-range mission reasoning and tools. He received the National Science Foundation (NSF) Graduate Scholarship and the Apple AI/ML Ph.D. Scholarship.
© THE