Over the past few years, the core goal of optimizing big-model logic systems has been around a question: how to make GPUs more efficient in weight computing, activation, high-voltage computing such as KV Cache? From parallel strategies, request scheduling, cache management to algorithm optimization, the academic community is constantly improving the efficiency of model reasoning.
However, as model sizes continue to grow, contextual lengths continue to expand, and new applications such as multi-cycle interaction agents rapidly emerge, large model infrastructures are facing a new challenge: how to manage them flexibly and efficiently in the face of the "volume state" of flows of bulk cross-system components, for example:
Large model service resilience expansions require the rapid distribution of TB-level model weight to new instances;
A large amount of GB-level KV Cache is required to be stored in contextual, multi-round conversation scenarios, and quickly migrated and replicated between instances;
In reinforcement learning, it is necessary to quickly synchronize the parameters of the new model at TB level that generate the training node to a large number of rollout logic nodes.
These volumes are no longer just temporary data in the computing process, but have gradually become the core state of large model systems. Around each need, the academic community already has customized optimization solutions. However, a new problem is emerging: different systems are repeatedly solving similar volumes management problems but lack a unified basic abstraction.
In response to this lack of basic capabilities, the University of Beijing, Shijiazhuang and the Beijing Postal University jointly developed TensorCast, which introduces a unified, programmable volume lifecycle management abstraction layer between computing engines, networks, and storage, unraveling model weights, and volume states such as KV Cache from specific systems, enabling new volume management capabilities for different workloads to be replicated and combined in a unified manner.
TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure
Address of the paper:https://arxiv.org/pdf/2608.06007
Code address:https://github.com/tensorcast-ai/tensorcast
The main page of the project:
Today's large-scale model is under construction and is repeating "wheeling".
In today's large model infrastructure, different tasks are usually equipped with independent optimization systems: model loading systems responsible for weight distribution; KV Cache systems responsible for KV storage and migration; Checkpoint systems responsible for model state preservation and recovery. These systems have achieved significant performance improvements, but they are often deeply tied to specific reasoning/training frameworks, network environments, and storage back-end. In essence, these tasks actually involve a set of highly similar operations:
Identify: Identify quantum identity and metadata;
Place: Determine where the volume should be stored;
Move: the amount of voltage transferred between nodes;
Transform: changing the volume layout or parallel form;
Materialisation: Loading logical states into the data available for computation.
However, as shown in Figure 1, these capabilities are now usually enclosed within their respective independent systems, forming a "vertical technical loop" that is isolated from each other.
First, the cost of system development is increasing. When new optimization strategies emerge, developers often need to modify multiple components at the same time, such as a scheduler, caching system, execution engine, etc. For example, a simultaneous load balancing and KV Cache-friendly reasoning strategy requires coordinated request routing, caching migration, and instance management logic, and system development is all at once.
Second, applications of complex large models increasingly require cross-component synchronous optimization. For example, in the course of Agent multi-cycle reasoning, the system may need to consider simultaneously: which instances take on the request, where the KV Cache should be placed, and whether the migration is already in context.
As a result, large model infrastructures require a new abstract layer for the entire lifecycle of a uniformly managed voltage-volume state.
Figure 1 The current large model infrastructure achieves separate optimization techniques for different volumes of tasks, while increasing development costs, making it difficult to express cross-task combined optimization strategies.
TensorcCast: a programmable voltage management system
To address this issue, a team from Peking University, a joint venture of UCSF, has proposed a new system abstraction called Tensor-as-a-Service (TaaS) and a corresponding system called TensorCast:
Unlocking the volume lifecycle management from the computational execution logic allows developers to manage the model weight, volume states such as KV Cache (see Figure 2) as well as management system resources.
Image 2 Tensor-as-a-Service (TaaS) framework: TensorCast unifies the management of volume life cycles and distributed storage, transmission, and top-level computing logic by invoking native programming volume management strategies
TensorCast consists mainly of two core psychological concepts:
1. Voltage as a first-class system object
In TensorCast, volumes have independent system identities and life cycles.
The system can uniquely manage different types of voltage states, including model weights, KV Cache and other intermediate states. Developers do not have to worry about which node, GPU, or storage medium a voltage is specifically located. TensorCast is responsible for the subsequent data positioning, mobility, and materialisation. A model weight can be registered to TensorCast, after which new reasoning instances can directly access the corresponding state without having to reload the entire model from the underlying storage.
Image 3 TensorCast provides a series of volume management native languages as an API for developers to implement volume management strategies
2. The volume lifecycle becomes programmable
In a KV Cache migration scenario, for example, in a multi-cycle dialogue LLM service, a user session may need to move KV Cache from one logical real-time sequence to another instance due to load imbalance, which typically requires modification in conventional systems: request coordinator, KV Cache backend and logical logic of the logical engine. In TensorCast, this process can be completed through a simple lifecycle program.
Image 4 KV Cache migration between instances of reasoning engines using the TensorCast API
Unified volume management baseline
To support volume management in large LLM clusters, TensorCast has designed a distributed operating architecture that separates control logic from data transmission (Fig. 5).
On the control side, TensorCast maintains metadata of the entire cluster through a Global Store (GS), including node status, volume location, and information needed to schedule tasks. GS is responsible for coordinating cluster status, but is not involved in actual data transmission, avoiding performance bottlenecks.
In terms of data, TensorCast deploys multiple Worker nodes to manage local storage, memory, and GPU resources and is responsible for performing specific voltage lifecycle operations, such as volume migration, copying, conversion, and loading. High-performance P2P data transfers between different Worker systems through RDMA, zero-copy, and streamline transmissions allow the system to continue to expand with mass growth.
In actual deployments, TensorCast can be accessed as an independent layer of existing large model reasoning systems. The top-level application defines voltage management strategies through the TensorCast API, and the bottom-level Worker is responsible for executing these operations efficiently across the entire cluster, thus providing a unified management capability for different types of voltage states such as model weights, KV Cache, etc.
Image 5 TensorCast is a distributed framework for P2P data transfer between data faces to avoid bottlenecks.
Experimental verification: TensorCast is both functional and programmable
To verify the effectiveness of TensorCast, the team integrated it into mainstream big model reasoning frameworks vLLM and SGLang and tested it in multiple scenarios such as model deployment, weight synchronization, KV Cache management, and intelligent routing.
a general voltage lifecycle management system, whether it is capable of achieving the performance of dedicated systems;
Programmable volumes management capabilities, whether they support new large model optimization strategies.
1. Large model deployment: Maximum model start-up speed increased by 228.6 times
In large model service resilience expansion scenarios, launching new instances often requires re-loading hundreds of GB of model weight, with model load times directly affecting the service response rate. TensorCast uses model weight as a manageable voltage state, accelerating instances through pre-localization and efficient transmission mechanisms. Experimental results show that TensorCast reduces the start time of end-to-end instances by a maximum of 228.6 times compared to common distributed file systems (Figure 6).
Picture 6 Model 6 when loaded and started operation (Qwen3-235B-A22B model)
2. General voltage management: achieving professional KV Cache system performance
To verify TensorCast's ability to manage dynamic state volumes, the team accessed it in SGLang HiCache and compared it to a professional KV Cache system (Mooncake). The results of the experiment showed that TensorCast achieved the same performance as Mooncake and could support highly dynamic KV Cache workloads (Figure 7).
Figure 7 More reasoning instances KV Cache Retarded repetition of the header (TTFT) Reduced rate
3. Programmable optimization: Multi-cycle Agent scenarios achieve a maximum of 93.2% decreased headline delay (TTFT)
Based on the TensorCast API, the team implemented a request scheduler for multi-round conversational Agent scenarios, which combined KV Cache's backend with reasoning engine logic to achieve a dynamic load-adjustment request distribution based on the instance, and to automatically migrate the complete response state. Under high and multi-round Agent workloads: TensorCast lowered the median TTFT to a maximum of 93.2% (Figure 8). The experiment validated TensorCast's core psychology: Once the volume lifecycle became a programmable foundation, developers could quickly explore cross-component model system optimization strategies.
Figure 8 High concurrent multi-round dialogue Agent loads the median of different scheduling strategies TTFT
Summary and Prospects
TensorCast explores a new paradigm of volume management for large-scale model infrastructure: unraveling critical states such as model weights, KV Cache from specific computing processes, and allowing volume to be managed and sorted independently, such as computing, storage, and network resources, through unified, programmable lifecycle abstractions.
As applications such as Agent, multi-cycle reasoning, and continuous learning drive the dynamics of large model systems, how to manage the volumes flowing across these nodes, across components, and across phases will become one of the core issues of the next generation of large model infrastructure.