Alibaba Cloud Says Aegaeon Cut Nvidia GPU Needs by 82% in Model Marketplace Beta

From 1,192 Nvidia H20 GPUs to 213

Alibaba Cloud says a new system called Aegaeon reduced the number of Nvidia H20 graphics processors needed to serve dozens of artificial intelligence models from 1,192 to 213 during a production beta lasting more than three months. The company reports that the smaller fleet handled the same service load, including models with up to 72 billion parameters.

The reduction amounts to 979 fewer GPUs, or about 82 percent of the original deployment. Put simply, Alibaba Cloud says it served the workload with roughly one fifth of the accelerators previously required. The research team from Peking University and Alibaba Cloud presented the results at the Symposium on Operating Systems Principles in Seoul, a leading conference for computer systems research.

Aegaeon addresses a costly feature of AI marketplaces: many models must remain available even when hardly anyone is using them. Popular models receive a steady stream of requests, while others wait for occasional bursts of activity. When each model has GPUs reserved for it, much of that expensive computing capacity can sit idle.

The system pools GPU resources across models and schedules work in much smaller steps. Its reported gains come from keeping chips busy, managing memory carefully and reducing the delay involved in switching between models. For cloud customers, those changes could mean lower costs, more available capacity and more consistent service when applications call several different models.

Why Quiet Models Consume So Much Capacity

Alibaba Cloud measured a sharp mismatch between GPU allocation and actual demand in its marketplace: 17.7 percent of GPUs were assigned to models that handled only 1.35 percent of requests. Those figures show why improving service for less popular models can produce substantial savings even when their share of traffic is small.

Cloud providers now offer thousands of models to developers. A few, including Alibaba's Qwen family and the open model DeepSeek, attract much of the traffic. Many others remain useful to particular developers but receive requests only sporadically. This pattern is often called the long tail of demand: a large number of less frequently used services sit behind a smaller group of popular ones.

Traditional serving systems reserve whole GPUs, or large portions of them, for individual model instances. An instance is a running copy of a model that can accept requests. When demand falls, its reserved capacity may remain underused. When demand suddenly rises and there are too few ready copies, the platform must start more instances, adding loading delays and reducing the amount of work completed during the transition.

Alibaba Cloud frames Aegaeon as a way to serve concurrent large language model workloads with less waste. The team says its combination of scheduling across models, memory management and faster switching increases throughput, reduces queues and leaves fewer chips tied to lightly used services. Throughput means the amount of work a system completes in a given period.

What Makes Switching Models Expensive?

A large language model needs more than its basic program to generate an answer. Its weights, the learned values that shape its responses, must be available in GPU memory. During generation, the system also maintains a key value cache, which stores intermediate information associated with tokens already processed or produced. Tokens are the small units of text a model works with.

Loading weights from storage or across a network takes time. Preparing the cache also takes time. In traditional arrangements, changing which model a GPU serves can involve expensive unloading and reloading, making frequent sharing unattractive. That helps explain why accelerators have often been dedicated to one model, or at most a small number.

Aegaeon aims to make those transitions cheaper. It reuses components, explicitly manages memory and coordinates the key value cache at a fine level while switching work between models. Keeping useful data ready in memory reduces the need for repeated loading and allows a GPU to spend more of its time generating responses.

Scheduling Work One Token at a Time

Aegaeon's central idea is to adjust capacity at the token level. Instead of waiting for an entire request to finish, or starting another full model instance when demand changes, its scheduler makes decisions about individual generation steps. That gives the system more opportunities to share available GPU time.

A GPU can begin serving one model and then schedule token generation for another when capacity becomes available. Aegaeon interleaves work for different users and models while coordinating batching and kernel launches, the operations that send groups of calculations to the GPU. Its goal is to complete more useful work each second while keeping response delays within a tight budget.

This approach also gives the platform a way to react to sudden changes in demand without always starting new replicas. If requests for a quiet model arrive while another model leaves spare capacity, the scheduler can allocate smaller slices of work to them. The benefit depends on making each switch quick enough to preserve responsiveness.

The paper reports that Aegaeon sustained request arrival rates between 2 and 2.5 times those handled by alternative pooling systems. It also reports between 1.5 and 9 times more goodput. Goodput describes useful work completed while meeting service targets, so it captures whether extra processing capacity produces responses within the required time.

Memory Reuse Makes Wider Sharing Possible

The researchers report a 97 percent reduction in auto scaling overhead compared with naive approaches. Auto scaling is the process of adjusting computing capacity as demand changes. In Aegaeon, reducing the memory and cache work associated with those adjustments is central to making frequent changes practical.

Alibaba Cloud also reports that the system reduced the latency penalty for switching between models by 97 percent. That matters in a marketplace where requests may alternate throughout the day between heavily used models and services that receive only occasional calls. A small switching cost makes it easier to share a chip without making users wait through repeated loading pauses.

Under the reported conditions, one accelerator supported up to seven models concurrently. Earlier systems typically pooled only two or three models on a GPU before additional switching, cache misses and uneven response times outweighed the gains. A cache miss occurs when needed information is not ready where the system expects it and must be retrieved or recreated.

Aegaeon's memory reuse and cache coordination are designed to keep multiple models ready without constant reloading. The reported increase from two or three models to as many as seven gives the scheduler a wider choice of work, helping it fill otherwise unused GPU time while respecting service quality targets.

What the Production Beta Showed

Alibaba Cloud says the beta ran for more than three months in its model marketplace, serving tens of models with sizes reaching 72 billion parameters. Parameters are the learned values inside a model. Larger parameter counts generally mean more model data must be accommodated when serving it.

The marketplace exposed Aegaeon to simultaneous requests, different request sizes and a wide range of models. In that environment, the company reports that the number of H20 GPUs required for the same service load fell from 1,192 to 213. Its explanation links that reduction to busier accelerators, less idle capacity on rarely used services and finer scheduling that preserved responsiveness for popular models.

The research team from Peking University and Alibaba Cloud described the cost of concurrent model serving as a central finding of the paper:

Aegaeon is the first work to reveal the excessive costs associated with serving concurrent LLM workloads on the market.

The team's argument is that assigning GPUs to models with sporadic demand leaves substantial capacity unused. Aegaeon combines scheduling and memory management specifically for that marketplace pattern. The reported production savings and experimental performance gains describe related benefits: fewer deployed chips for a given service load, and more useful work from the capacity available.

What Changes for Customers and Developers?

Cloud customers could benefit if fewer GPUs are needed to deliver the same volume of responses. Lower resource requirements can create room for lower prices, larger usage quotas and more predictable performance. Alibaba Cloud's account gives no customer pricing changes or dates for a broader rollout.

A provider able to serve existing traffic with roughly one fifth of its previous accelerator fleet could use freed capacity to handle peak demand or make room for new deployments. That could shorten waiting lists where access to GPUs limits how quickly customers can start services.

Developers whose applications combine several models could also experience fewer delays when traffic shifts between model endpoints. Faster switching and less replica churn can reduce bottlenecks associated with cold starts, the period when a service is loading and preparing to handle requests.

Shared resources still require careful controls. One customer's workload must not overwhelm another's service or expose another customer's data. Scheduling must balance priorities and fairness so heavily used models remain responsive while quieter models continue making progress. Aegaeon's choices at the token level are designed around service quality targets, according to the paper.

Cloud operators also need monitoring and throttling as they expand pooling. Throttling limits incoming work when necessary to protect service quality. Monitoring must track tail latency, the delays experienced by the slowest requests, because a good average response time can coexist with frustrating waits for some users.

Why the H20 and China's Chip Constraints Matter

Nvidia designed the H20 for the Chinese market to comply with United States export restrictions, and many Chinese cloud providers rely on it for inference. Inference is the stage when an already trained model processes a request and produces an answer. Aegaeon's reported savings concern that serving work.

Advanced accelerators remain subject to tight global supply and export controls affecting China. Chinese companies including Huawei and Cambricon are investing in domestic alternatives. Software that lets existing GPUs handle more useful work is valuable wherever computing capacity is costly, with particular relevance for providers whose access to the most advanced chips is restricted.

Nvidia's leadership has said its near term financial guidance assumes minimal or zero revenue from China's advanced chip segment because of policy limits. Against that backdrop, improving the use of installed hardware gives large cloud providers another way to expand services while supplies of leading accelerators remain constrained.

Where Pooling Fits Among Serving Options

Dedicated GPU instances remain suitable for endpoints handling very high volumes of requests or workloads requiring strict isolation. A consistently busy model may already make good use of its assigned capacity. Aegaeon's case is strongest where demand varies across many models and reservations leave chips waiting for work.

Serverless inference offers another approach for models with little traffic: the platform starts full replicas when requests arrive. That can save resources while a model is idle, but starting those replicas introduces delays. Users may wait while model data loads and the service becomes ready.

Aegaeon sits between these approaches. It pools capacity, keeps models ready and allocates work in small steps. Its reported advantage over earlier pooling systems comes from shrinking switching costs and keeping important data structures available across model changes, allowing more models to share an accelerator without a steep response time penalty.

Each additional model can still introduce more switching, cache misses and variations in response time. The reported support for up to seven models per GPU therefore depends on the tested conditions and the system's memory and scheduling design. Workload demand and service requirements remain central to choosing how capacity should be assigned.

What Wider Deployment Would Need to Address

The beta covered Alibaba Cloud's model marketplace, rather than its entire service footprint. Expanding Aegaeon into more regions or other products would bring different operating conditions. Dedicated enterprise clusters and installations on customers' premises, for example, can impose different capacity and service constraints.

Longer context lengths and models combining language with images or audio would also put pressure on memory and cache policies. Context length is the amount of information a model handles within a request. As that amount grows, the system must manage more working data while still switching efficiently between workloads.

Sharing across different types of accelerators and across multiple computing nodes presents another research direction. A scheduler could potentially pool a connected group of GPUs, provided that communication costs remain low and data stays in predictable locations. Moving information between chips introduces additional work that must fit within response time targets.

The published results point to large gains from sharing work on individual accelerators within the tested deployment. Their practical message is concrete: serving many models efficiently requires attention to idle reservations, memory movement and the smallest useful units of scheduling, alongside the number of GPUs a provider owns.

Quick Facts

• Alibaba Cloud says Aegaeon reduced its tested marketplace deployment from 1,192 Nvidia H20 GPUs to 213, saving 979 chips, or about 82 percent.
• The production beta lasted more than three months and served dozens of models, including models with up to 72 billion parameters.
• The research team from Peking University and Alibaba Cloud presented the results at the Symposium on Operating Systems Principles in Seoul.
• Alibaba Cloud measured that 17.7 percent of marketplace GPUs served models handling only 1.35 percent of requests.
• Aegaeon schedules work at the token level and uses memory reuse and key value cache coordination to reduce switching costs.
• The researchers report 97 percent lower auto scaling overhead and support for up to seven models per accelerator under the reported conditions.
• Experiments reported 2 to 2.5 times higher sustained request arrival rates and 1.5 to 9 times more goodput than alternative pooling systems.
• The results concern Alibaba Cloud's marketplace beta, with wider deployment and performance across other environments still to be addressed.