Samsung targets lower AI token costs by making server memory work harder

More AI requests without adding GPUs

Samsung Electronics has developed a method for handling more simultaneous requests to artificial intelligence servers by changing how they allocate memory and move data between processors. A published patent application describes software that weighs waiting times, transfer costs and future memory needs to decide which requests a large language model should process first.

The aim is to reduce delays when graphics processing unit (GPU) memory fills up, allowing existing servers to serve more people. Samsung describes the invention as infrastructure technology for AI inference, the process of running a trained model to answer questions, rather than a feature for a particular consumer product.

Related Samsung research gives a concrete picture of the potential savings from better memory management. In a separate storage experiment using eight NVIDIA H100 NVL GPUs and ten Samsung solid state drives, Samsung maintained a wait of less than one second for the first output token at up to 240 simultaneous users. Without storage offloading, that waiting time began increasing beyond 140 users.

The patent, the storage experiment and another project from Samsung SDS address different parts of the same problem: expensive AI processors can spend too much time moving or rebuilding information. For operators, reducing that work could mean more answers from existing equipment and a lower cost for each token, the small pieces of text that language models process and generate.

Why memory can hold up an answer

A GPU performs many calculations at once, making it useful for generating AI responses. Its fast local memory, however, has limited capacity. Multiple conversations compete for that space, especially when users submit long documents or ask questions that require lengthy responses.

Language models keep temporary records of earlier calculations in a structure called the key value cache, usually shortened to KV cache. Think of it as working notes: keeping those notes lets the model continue an answer without repeatedly calculating everything it has already processed.

Samsung's storage experiment required approximately 3 GB of KV cache for each session. At 240 sessions, that amounts to roughly 720 GB of cache data across the workload. The figure helps explain why serving many people can become a memory problem even when substantial computing power is installed.

When fast GPU memory runs short, a server can move some data to memory associated with the central processing unit (CPU), then bring it back when needed. Those transfers take time. Rebuilding discarded calculations also consumes resources. Samsung's patent focuses on managing this competition more carefully.

How the patented scheduling method works

Samsung's application, titled Method and Server for Processing Multiple User Requests of a Large Language Model, describes a scheduler that considers the cost of serving each request. Its inputs include how long a user has waited, how much time data transfers will take and how much memory the request will need.

The server uses those calculations to choose a processing order. This matters because a request that looks ready to run may require a substantial transfer first, while another request may be able to use data already nearby. Coordinating those choices can reduce time spent rearranging memory.

The method also checks whether information previously moved to CPU memory remains there. If a usable copy exists, the server can move only the necessary portions rather than copying everything again. It can also transfer data for one user while the GPU generates another user's response, allowing useful work and preparation to overlap.

Samsung also describes using partially available memory. If a new request needs five units of space but only three are free, the system can begin the portion that fits instead of waiting for all five. The practical goal is to keep limited resources productive during busy periods. Commercial deployment details and measured improvements in response time and throughput for this patented method have not been disclosed.

What Samsung measured with storage offloading

Samsung Electronics' KV cache white paper examines a different approach: retaining reusable cache data on NVMe solid state drives. NVMe is a technology that allows storage devices to communicate rapidly with a computer through PCIe connections.

The test used eight NVIDIA H100 NVL GPUs, eight parallel LLaMA-3.1 8B inference services, the vLLM serving software and LMCache, an open source layer that manages reusable KV cache data. Ten Samsung PM1753 drives were combined in a RAID-0 array, which spreads data across drives to increase transfer performance.

Samsung measured user loads in increments of 20, reaching 280 simultaneous users. Each user submitted a query every 20 seconds, producing a workload ranging from one to 14 queries per second. The experiment therefore tested a defined pattern of repeated requests and cache movement.

The storage approach becomes useful when a session moves between GPUs. A destination GPU can reload saved calculations rather than repeat the initial processing of a long prompt. Within one machine, external cache can be held in CPU memory or storage. Across machines, Samsung explains that cache needs a shared resource accessible to the relevant servers.

Faster starts and more generated text

Samsung measured two aspects of service performance: time to first token and tokens per second. The first measures how long a person waits before an answer starts appearing. The second measures how much text the system produces over time.

With storage offloading enabled, the test maintained time to first token below one second through 240 simultaneous users, compared with latency growth beyond 140 users without offloading. Samsung describes this as support for 1.7 times as many users at comparable latency. Within the range of 20 to 140 users, the configuration using offloading kept the initial wait at approximately half a second.

At 240 users, the system with offloading generated approximately 1.5 times as many output tokens. The configuration without offloading began to saturate at around 160 users, while the system reusing saved cache continued increasing its output as demand grew.

Samsung attributes the gain to reducing repeated prompt processing. A language model first processes the input, a stage known as prefill, then generates its response token by token in a stage called decode. Reusing saved calculations leaves more GPU capacity available for producing answers.

When extra storage pays for itself

Samsung's cost analysis shows that additional hardware has a price even when it improves efficiency under heavy demand. Across the range of zero to 160 simultaneous users, both configurations delivered similar token throughput, while the configuration using offloading cost approximately 4% more because of the added NVMe components.

Beyond 180 users, higher output offset that extra storage cost. At 240 users, Samsung calculated approximately 1.5 times better cost efficiency, measured as tokens per second per dollar over an operating period of three years.

Expressed another way, a 1.5 times improvement in output per dollar corresponds to roughly one third less cost per unit of output under that calculation. The comparison concerns Samsung's tested equipment and workload. A service's actual economics also depend on its model, conversation lengths, traffic and equipment costs.

For a cloud operator, the practical choice is therefore sensitive to demand. Extra storage can increase the initial bill when existing GPU capacity already handles the workload comfortably. During sustained congestion, reusing cache can allow the same GPU installation to support more activity before another server is needed.

Power savings extend to cooling fans

Samsung also measured lower electricity use in its storage experiment. At 240 users, performance per watt improved by approximately 2.9 times. Total system power fell to roughly 53% of the configuration without offloading, a reduction of about 47%.

The additional NVMe drives increased power consumption by approximately 6%, according to the white paper. Lower GPU and cooling demand more than offset that increase. Samsung recorded GPU utilization of approximately 49% as the workload shifted away from repeating intensive prefill calculations.

Cooling fans reflected that change. Without offloading, all system fans operated at their maximum expected speed of 15,000 revolutions per minute. With offloading, fans serving the CPU, memory and storage ran at around 7,800 revolutions per minute, while fans on the GPU side ran at approximately 9,800.

The result illustrates why a component can add electricity demand locally while reducing consumption across a server. In this experiment, spending energy to retrieve saved calculations reduced the larger energy expense of repeating calculations and removing the resulting heat.

The storage must handle bursts

Samsung's measurements also show why this approach needs more than spare drive capacity. Average aggregate storage throughput reached up to 37 GB per second, and simultaneous GPU requests could push peak demand to nearly twice the average.

The number of storage operations in flight typically stood at around 160 but could spike to 800. Samsung recommends storage bandwidth of at least twice the average requirement to absorb those bursts while preserving responsiveness.

The PM1753 is specified at up to 14.5 GB per second of sequential throughput and 3.3 million random read operations per second. In the tested workload, reads accounted for approximately 92% of storage activity and writes for 8%. LMCache moved large cache blocks of approximately 33 MB.

Several services reading large blocks simultaneously can create an interleaved stream of requests. A storage subsystem therefore needs to maintain transfer performance when many users become active together. Cache capacity, transfer speed and scheduling all contribute to whether saved information reaches a GPU quickly enough to be useful.

Samsung SDS takes computation into memory

Samsung SDS is tackling data movement through another route. Its PAISE research account describes the PIM-Accelerated Inference Scheduling Engine, presented at IEEE HPCA 2025. PAISE uses processing in memory (PIM), which places computational units inside memory chips so some calculations can happen where the data is stored.

Samsung SDS implemented and validated the system on Samsung's Aquabolt-XL PIM hardware, evaluating GPT-2, OPT and Llama2 models. It reports an inference speed improvement of up to 48.3% and a power consumption reduction of approximately 11.5%.

PAISE decides which operations should run in memory and arranges data to suit the hardware. It also groups small attention calculations so they can run more efficiently in parallel. Samsung SDS reports faster KV cache updates and stronger gains at larger batch sizes, meaning groups of requests processed together.

The three projects work at different points in the server. The patent schedules requests and transfers between GPU and CPU memory. The storage experiment preserves cache on drives for later reuse. PAISE moves selected calculations into memory hardware. Together, they show Samsung pursuing several ways to reduce the time and energy spent moving or rebuilding the information behind AI responses.

Key Points

• Samsung's published patent application schedules AI requests using waiting times, transfer costs and expected memory needs.
• The patented method aims to serve more users with existing GPUs; commercial performance figures have not been disclosed.
• A separate Samsung storage test supported 240 simultaneous users with an initial response wait below one second, compared with latency growth beyond 140 users without offloading.
• At 240 users, storage offloading delivered approximately 1.5 times the token output and cost efficiency, alongside 2.9 times the performance per watt.
• Added storage increased costs by approximately 4% at lighter loads, making demand an important part of the economic calculation.
• Samsung SDS reports that its separate PAISE system improved inference speed by up to 48.3% using computation inside memory.