Huawei Releases SINQ to Cut LLM Memory Use With Calibration Free Quantization

Smaller AI models could bring advanced workloads to a single GPU

Huawei's Computing Systems Lab in Zurich has introduced SINQ, an open source method that the team says reduces the memory needed for large language model weights by roughly 60 to 70 percent while keeping output quality close to the original model. Released under the Apache 2.0 license, the method aims to make capable models easier to run on less expensive hardware.

SINQ stands for Sinkhorn Normalized Quantization. It reduces the number of bits used to store a model's learned parameters, known as weights, and prepares those weights so that shrinking their representation causes less damage to the model's predictions. The researchers position it as fast, calibration free and straightforward to add to existing model workflows.

The reported savings can bring a model that previously required more than 60 GB of memory down to roughly 20 GB. That puts some workloads within reach of a single Nvidia GeForce RTX 4090, which has 24 GB of memory, instead of requiring enterprise GPUs such as the A100 or H100.

Huawei's team also reports faster model preparation: SINQ quantizes models roughly twice as fast as HQQ and more than 30 times faster than AWQ in its tests. For developers repeatedly preparing and evaluating models, a shorter setup stage could make experimentation easier.

As of October 10, 2025, the release includes implementation code, instructions for quantizing models from popular hubs, and tools for saving and reloading the resulting weights. The authors also plan to release models that have already been quantized and seek integration with the Hugging Face Transformers ecosystem.

Why memory makes capable models expensive to run

Large language models need substantial memory to hold their weights. Those weights are the learned parameters within the model's layers, and they dominate its size. A model that cannot fit within a GPU's memory may need several devices or may spill into slower host RAM, the computer's main memory.

Quantization addresses this problem by representing weights with fewer bits. Moving from floating point representations using 16 or 8 bits to integers using 4 bits can cut memory by more than half. The challenge is preserving enough numerical detail for the model to keep producing useful answers, especially below 8 bits.

SINQ's central claim is that it retains strong quality at 4 bits and, in some cases, 3 bits. Across the models and architectures tested by the authors, the reported memory reduction is roughly 60 to 70 percent compared with full precision weights.

A smaller model can change the hardware decision for a laboratory, startup or person running AI locally. A workstation becomes a possible option for workloads that previously depended on premium data center GPUs or costly cloud nodes. For a cloud operator, the same reduction could allow a cheaper instance to handle a continuing inference job, the process of using a trained model to generate answers.

Memory during generation also includes activations and the key value cache. Activations are intermediate values produced as the model processes information. The cache retains information used during text generation, and its memory demands grow with long prompts and long outputs. Those parts of the workload remain relevant when deciding whether a model will fit on a particular GPU.

How SINQ avoids the calibration stage

Many widely used quantization methods include calibration, a preparation stage that uses sample data and extra passes through the model to find suitable scale factors. Those factors help map the original weights into a smaller numerical representation while limiting error.

Calibration can improve quality, but developers must choose a set of prompts, run the model to collect statistics and then calculate the scales. The process takes time and adds work to deployment pipelines. Its results can also be influenced by how closely the selected prompts resemble the tasks the model will handle in production.

SINQ works directly with weight statistics instead. Its calibration free design removes the need for that sample dataset and the associated model passes. For teams automating model preparation, this can simplify the workflow and make repeated runs easier to manage.

The method is still compatible with calibration. A project seeking further accuracy improvements can use SINQ's weight shaping first and then add a calibration search. Developers therefore have a choice between the simpler preparation process and an additional optimization stage.

Most LLM quantization in use focuses on weights alone. Quantizing activations is also possible, but that can introduce more complexity and affect runtime. SINQ's approach concentrates on preparing the weight matrices, the tables of learned numbers within the model, so that lower precision preserves the structure needed during inference.

Two directions of scaling help control uneven weights

SINQ combines two ideas: dual axis scaling and a normalization procedure inspired by Sinkhorn Knopp. Both address the same practical difficulty. Some parts of a weight matrix contain much larger or more uneven values than others, making it harder to compress the whole matrix without introducing substantial errors.

Traditional scaling can apply one factor to an entire tensor, a collection of numbers, or use factors along a single direction, such as individual channels. Dual axis scaling uses separate vectors across the rows and columns of each weight matrix. A vector here is a list of scale factors, with different entries controlling different rows or columns.

Working in both directions gives the method finer control. It can account for rows containing outliers and columns with greater variance, meaning their values are more spread out. The aim is to spread quantization error more evenly, reduce sharp spikes in that error and preserve the relative sizes of the weights.

SINQ first shapes the matrices so that important relationships survive a lower precision representation. It then maps the reshaped weights into compact integer formats. The authors describe this as a plug and play step in a model preparation pipeline, with parameters available for different hardware and accuracy targets.

What Sinkhorn normalization does inside the model

Sinkhorn Knopp is a classic iterative procedure that alternates simple normalization steps to bring a matrix into balance. SINQ adapts that idea by repeatedly rescaling rows and columns according to their standard deviations, a measure of how widely their values vary.

The target is a balance of those standard deviations across both directions. SINQ is therefore pursuing a different goal from a doubly stochastic matrix, the row and column balance associated with the classic procedure. For model weights, the focus is on controlling the spread of values so that no part of the matrix dominates the quantization process.

Each round adjusts the rows and columns toward target values. Once the matrix is balanced in this sense, quantization works on a distribution with fewer extremes. That reduces the rounding error introduced when the original numbers are represented using fewer bits.

The procedure uses simple repeated normalizations and adds little overhead compared with longer calibration passes. Combined with dual axis scaling, it offers a way to manage outliers and local imbalances without a costly calibration dataset for each layer.

The same shaping approach can work with nonuniform quantization schemes, which use uneven spacing between the values available in the compressed representation. Projects can also combine it with calibration when their priority is to push accuracy further.

What the speed and quality tests show

Huawei's researchers report quality close to the original models across a variety of architectures, alongside the roughly 60 to 70 percent reduction in memory used for weights. Their tests assess quality using measures including perplexity and flip rate.

Perplexity measures how well a language model predicts text. A lower value indicates better predictions under that measure. The authors report that SINQ lowers perplexity compared with other calibration free baselines.

Flip rate measures how often the model's top token changes after quantization. A token is a unit of text used by the model, and the top token is its leading choice for the next unit. SINQ's lower flip rate points to more stable predictions after compression.

The reported preparation speeds also matter for teams that change models frequently. SINQ quantizes models roughly twice as fast as HQQ and more than 30 times faster than AWQ in the reported tests. A stage that otherwise occupies hours can become easier to fit into a development loop.

Preparing a model and serving its answers are different stages. SINQ's faster quantization helps with preparation; inference performance depends on the software and hardware used to run the compressed weights. Efficient support for the selected numerical format is therefore part of the deployment decision.

Where smaller weights could lower cloud bills

A model that fits on one GPU can avoid the expense and network overhead of dividing work across several devices. SINQ's reported reductions bring many models of medium to large size within reach of a consumer GPU with sufficient memory, giving operators another option besides A100 or H100 nodes.

Cloud users could choose smaller GPU instances or fit more models on each node at the same time. For continuing workloads such as chat assistants and retrieval augmented generation services, which use retrieved information to help produce answers, lower cost GPUs could translate into thousands of dollars saved over weeks and months.

Labs and small businesses could also consider smaller clusters on their own premises. That makes local operation more practical for teams that prefer to keep data in house and retain control of their serving software.

Fewer devices can reduce communication between nodes and transfers between the computer's memory and the GPU. Those changes can lower response times and power use while simplifying the serving stack. The size of any saving depends on the workload and the chosen deployment configuration.

Moving inference to more affordable hardware can also leave premium GPUs available for training. For organizations sharing a limited pool of devices, the practical benefit is the ability to assign scarce hardware to jobs that still need it.

What developers can use now and what is planned

The Apache 2.0 release provides implementation code and instructions for quantizing models from popular hubs. Utilities support saving the quantized weights and loading them again, allowing model preparation to be separated from later use.

Evaluation hooks let developers measure perplexity and flip rate before deployment. Exposed parameters provide room to tune the process for hardware constraints and desired accuracy, rather than requiring every project to use the same settings.

Teams maintaining their own inference stacks can begin by quantizing weights and connecting the results to kernels that support storage formats using 4 or 3 bits. Kernels are the routines that carry out calculations on the selected hardware, so their ability to handle the packed weights efficiently affects runtime performance.

The authors' planned releases of already quantized models would remove part of the preparation work for users. Their intended integration with Hugging Face Transformers would let developers load a quantized checkpoint, a saved version of a model, through familiar APIs. No release date for those planned updates was given.

Deployment tests still need to cover the whole workload

Weight compression is only one part of a working AI service. Activations and the key value cache continue to consume memory, particularly with long prompts and long generated answers. Some runtimes provide activation quantization and cache compression, but those steps come with their own tradeoffs.

Developers should measure memory use and latency across the full process, rather than looking only at the model's size on disk. A configuration that fits under one prompt length or batch size may behave differently when it handles longer conversations or more requests together.

Runtime support also varies. Engines such as llama.cpp, TensorRT LLM and vLLM are evolving quickly, yet support is not equally mature for every combination of numerical precision, attention kernel and GPU architecture. Production testing needs to use the exact intended configuration and assess throughput, latency and stability.

Very low precision can place more strain on tasks such as code generation, tool use and structured outputs than on casual conversation. Perplexity is useful, but deployment checks should also cover task accuracy, response style and safety. Mixed precision, or keeping sensitive layers at higher precision, can preserve quality while retaining much of the memory saving.

Why the release matters beyond Huawei

Quantization has become a foundation for practical LLM deployment because memory requirements determine which models many people can afford to run. A fast, open source method that avoids calibration gives developers another option for reducing those requirements.

Smaller memory footprints can widen access for education, civic projects and small businesses. They also support teams whose priorities include privacy, local operation and control over their software, while reducing demand for premium GPUs needed for training and research.

SINQ can be combined with other serving techniques, including efficient attention kernels, paged key value caches and optimized schedulers. These manage calculations, cache memory and incoming work more efficiently. Together, such techniques aim to deliver more generated tokens per second for each dollar spent.

The immediate value of SINQ is the combination of lower weight memory, faster preparation and quality that the authors report remains close to the original model. For developers, the next practical step is to evaluate that combination against their own tasks, memory requirements and runtime support.

Key Points

• Huawei's Computing Systems Lab in Zurich introduced SINQ, or Sinkhorn Normalized Quantization, under the Apache 2.0 license.
• The authors report roughly 60 to 70 percent lower memory use for weights, with strong quality at 4 bits and in some cases 3 bits.
• SINQ uses dual axis scaling and Sinkhorn Knopp inspired normalization to balance weight matrices without a calibration dataset.
• Reported quantization speeds are roughly twice those of HQQ and more than 30 times those of AWQ.
• Tests show lower perplexity and flip rate than other calibration free baselines.
• A model requiring more than 60 GB can fall to roughly 20 GB, bringing some workloads within reach of an RTX 4090 with 24 GB of memory.
• The release includes code, model preparation instructions, saving and loading utilities, and evaluation hooks.
• Already quantized models and Hugging Face Transformers integration are planned, with no release date given.
• Production checks must account for activations, cache memory, runtime support and accuracy on the intended tasks.