Alibaba Releases Qwen3-Omni, an Open AI Model for Text, Images, Audio and Video

Four kinds of input, one open AI system

Alibaba has released Qwen3-Omni, an artificial intelligence model that accepts text, images, audio and video and can answer with text or natural speech in real time. Its release gives developers and businesses a freely available alternative to proprietary systems from OpenAI and Google, with permission to download, modify and deploy the model under the Apache 2.0 license.

Alibaba positions Qwen3-Omni as the first native, end to end omni modal system integrating all four input types within a single model. In simple terms, it is designed to process words, pictures, sounds and moving images together, rather than adding those abilities to a text model afterward. The company pairs that breadth with fast responses, support for many languages and tools for controlling what the system says.

OpenAI's GPT-4o combined text, image and audio capabilities, while Google's Gemini 2.5 Pro added video analysis. Those systems remain paid and closed. Qwen3-Omni brings video understanding, speech input and speech output into a package that developers can inspect and build upon.

Businesses can run the model on their own infrastructure instead of relying entirely on a supplier's hosted service. That gives them another way to balance cost, control and performance, while reducing dependence on one vendor. The license allows commercial use and private customization, making the release relevant to companies as well as researchers.

Qwen3-Omni also arrives amid growing adoption of Chinese open AI projects on developer platforms. Its appeal rests on a practical combination: a model that can handle several forms of information, respond quickly enough for conversation and offer organizations a choice between local deployment and cloud access.

How Thinker and Talker divide the work

Qwen3-Omni uses two main components, called Thinker and Talker. Thinker handles understanding, reasoning and text generation. It receives information from text, images, audio and video, combines those signals and produces the core written response.

Thinker is also where reasoning over lengthy material, tool use, retrieval and safety filters can operate. Retrieval means bringing relevant information into a response, while tools can supply functions such as calculations. External modules can review or adjust Thinker's output before Talker converts the approved response into speech.

Talker handles speech generation and coordinates spoken output with visual information. A central design choice is that Talker is not forced to depend on Thinker's text tokens. It can instead use audio and visual features directly. Alibaba describes this as a way to preserve a speaker's timing and emphasis during translation or match the pace of speech to what appears on screen.

Both components use a Mixture of Experts architecture. This divides the model into specialized groups of parameters, the numerical settings learned during training. Only the groups needed for a particular task activate, so the system does not have to use every parameter for every request. That helps make processing faster and easier to scale.

For speech generation, Qwen3-Omni predicts compact acoustic tokens through a multi codebook autoregressive approach. Put simply, it builds a sequence of compressed sound information, using several sets of sound codes. A lightweight network called Code2Wav then turns that information into an audio waveform. The design aims to preserve voice detail while shortening the wait for speech.

The audio encoder, called the Audio Transformer, was trained from scratch for both streaming and offline processing. Streaming lets a system work as information arrives; offline processing handles material that has already been recorded.

Speed, language coverage and conversation limits

Alibaba reports an end to end first packet latency of about 234 milliseconds for audio and about 547 milliseconds for video. First packet latency is the wait until the initial piece of output arrives. These figures describe a system intended for live interaction, where long pauses can interrupt conversation.

Qwen3-Omni also reports a real time factor below one. That means it can generate output as quickly as, or faster than, the incoming material plays. The combination is relevant to voice agents, assistants supporting call center staff, meeting transcription and other settings where people need prompt responses.

The model supports 119 languages for text, 19 for speech input and 10 for speech output. Those different totals matter when planning a multilingual service: the range of languages it can process as text is broader than the range it can hear or speak.

Thinking Mode offers a context window of 65,536 tokens, while Non Thinking Mode offers 49,152 tokens. A context window is the amount of information a model can consider within an interaction. Tokens are the units used to process and measure that information.

Maximum single input and maximum output are each 16,384 tokens. The longest reasoning chain extends to 32,768 tokens. These separate limits affect how developers divide lengthy documents, video transcripts and other media into manageable portions. A large context window provides room for an extended interaction, while the input and output ceilings govern individual exchanges.

Three versions and two routes to deployment

Alibaba is releasing three versions of Qwen3-Omni-30B-A3B, each designed for a different task. The Instruct version combines Thinker and Talker, accepts text, images, audio and video, and returns text and speech.

The Thinking version accepts the same four input types but produces text only. It focuses on reasoning and tasks requiring extended chains of reasoning, making it suitable when a detailed written answer matters more than a spoken response. The Captioner version is fine tuned for audio captioning, producing detailed written descriptions with an emphasis on low hallucination. Hallucination is the term for AI output that invents or misstates information.

Developers can download model weights, the learned numerical settings needed to run the system, from the official repository and community hubs. They can also use a faster Flash variant through Alibaba's application programming interface, or API, which lets software send requests to a hosted model.

The official GitHub repository provides a quick start guide, task cookbooks and Docker deployment notes. The cookbooks cover speech recognition, translation, music analysis, optical character recognition and audiovisual dialogue. Optical character recognition, usually shortened to OCR, reads text from images.

Alibaba offers an API trial quota of one million tokens across modalities, valid for 90 days after activation. Hosted audio output is available only in Non Thinking Mode. Developers planning a pilot therefore need to choose between reasoning with written output and the mode that also supports spoken answers.

What went into the training

Qwen3-Omni's Audio Transformer has about 0.6 billion parameters and was trained on 20 million hours of supervised audio. Supervised training uses examples paired with target answers, helping a model learn the relationship between an input and the response it should produce.

About 80 percent of that audio training mix consisted of Chinese and English speech recognition, 10 percent covered speech recognition in other languages and 10 percent involved audio understanding tasks. The encoder supports both caching during real time use and offline processing. Caching retains information already processed so the system can keep working as more audio arrives.

The broader pretraining process had three stages. During encoder alignment, Alibaba trained the vision and audio encoders while keeping the language model core frozen. This was intended to prevent the model's perception abilities from degrading as the different input types were first brought together.

General training then used roughly 2 trillion tokens spanning text, audio and images, with smaller shares of video and combined audio and video samples. A third stage increased the maximum training sequence length from 8,192 to 32,768 tokens, a fourfold increase. More long audio and video material was included to strengthen the handling of extended sequences.

Post training targeted reasoning and speech quality. Thinker received supervised fine tuning, strong to weak distillation and GSPO optimization. Distillation transfers guidance from a stronger model to a weaker one. The optimization combined feedback based on rules with scoring in which another model acted as a judge.

Talker followed a process with four stages, using hundreds of millions of multimodal speech samples and continued pretraining on curated data. The stated goals were fewer hallucinations and speech that sounded more natural. Together, these steps addressed both the substance of responses and how those responses were spoken.

What the benchmark results show

Alibaba reports strong performance across speech, audio and video understanding while maintaining high scores in text and vision. Its public benchmark results place Qwen3-Omni ahead on many open leaderboards and show competitive performance against closed systems.

For text and reasoning, the reported scores are 65.0 on AIME25, 76.0 on ZebraLogic and 82.6 on WritingBench. These tests examine different abilities, so the results offer several views of performance rather than a single measure of quality.

Speech recognition results include word error rates of 4.69 and 5.89 on Wenetspeech and 2.48 on Librispeech other. Word error rate measures mistakes in a transcription, with a lower figure meaning fewer errors. Music results include 93.0 on GTZAN and 52.0 on RUL MuchoMusic.

For image and vision tasks, Alibaba reports 59.7 on HallusionBench, 57.0 on MMMU pro and 56.3 on MathVision full. The video result is 75.2 on MLVU. Across these tests, the company presents Qwen3-Omni as a model that balances conventional text tasks with advanced audio and video comprehension.

Many of the results are reported by Alibaba itself. Independent replication across broader test suites is still needed to give a fuller picture of everyday performance. The practical question for adopters is how accurately the model handles their own recordings, documents, images and conversations.

Alibaba's materials aim to show comparable strength across modalities, although some community reviewers have observed slightly stronger pure text output from earlier versions of Qwen focused on text in narrow tasks. This reflects the tradeoff between a specialist model and one built for several kinds of input. Qwen3-Omni seeks to narrow that gap while adding abilities that combine sound, images, video and words.

How hosted pricing works

Qwen3-Omni is available for local deployment under Apache 2.0. Managed access through Alibaba's API is charged by token usage, with the same rates for Thinking Mode and Non Thinking Mode. Different input and output types carry different prices.

Alibaba lists the following input charges per 1,000 tokens, in United States dollars:

• Text input: $0.00025.
• Audio input: $0.00221.
• Image or video input: $0.00046.

The output charges per 1,000 tokens depend on the response and the input used:

• Text output with text only input: $0.00096.
• Text output when the input includes an image or audio: $0.00178.
• Text plus audio output: $0.00876 for the audio portion, with text free.

These rates make the mix of media part of the cost calculation. A service handling spoken requests and spoken answers has a different billing pattern from one handling written questions and written responses. Audio output is restricted to Non Thinking Mode through the API.

The free quota covers one million tokens across modalities for 90 days after activation. Teams also need to plan around the 65,536 token context window in Thinking Mode, the 49,152 token window in Non Thinking Mode, the 16,384 token limits for single input and output, and the maximum reasoning chain of 32,768 tokens.

What businesses gain from the license

Apache 2.0 permits commercial use, modification and redistribution. It also includes a patent license, which lowers legal risk when businesses incorporate the model into proprietary systems. Companies are not required to publish their derivative work, allowing customized or fine tuned versions to remain private.

Organizations therefore have more control over where the model runs and how it is adapted. A vendor controlled service can change prices, rate limits or model behavior; local deployment gives businesses another option when those conditions do not suit their needs.

Many organizations are moving toward using several models rather than relying on one. They combine open and closed systems by task, assign routine work to faster or smaller models, reserve larger models for complex reasoning and keep selected workloads on their own premises for data control.

Qwen3-Omni's breadth makes it a candidate for workflows involving substantial speech, multiple languages or large amounts of video. Processing several forms of information within one model can reduce the number of separate systems that a team has to train and maintain.

Deployment still requires security reviews, privacy controls, testing systems and fine tuning pipelines. The model's openness makes auditing and optimization easier, while leaving adopters responsible for safety and compliance. Teams also need operational processes for maintaining AI systems, connecting tools and setting boundaries for responses.

Where combined audio and video could help

Alibaba highlights multilingual transcription and translation paired with fast speech output for live conversations. Audio captioning can support media production and accessibility, while OCR and document understanding can process photographs of forms, receipts and diagrams.

Music tagging and sound analysis offer applications for media libraries and creative tools. Video understanding can support indexing, navigation and following instructions from tutorials or demonstrations of equipment. These uses draw on the model's ability to connect information that might otherwise be handled by separate tools.

A technical support example shows how the parts could work together. A customer streams video from a phone while the model reads text on the screen, listens to surrounding sounds and gives spoken guidance through each step. This is an illustrative use case, rather than a reported deployment.

A wearable assistant could similarly translate menus or signs in real time. Developers can use system prompts to set tone, persona and the boundaries of a subject area, then connect retrieval systems, calculators or other tools to extend the model's capabilities.

Contact centers, media production, education, healthcare transcription and industrial maintenance are among the settings where multimodal assistants could find uses if practical performance matches the reported benchmarks. Each setting would still require testing suited to its tasks and responsibilities.

Developer interest and competitive pressure

Qwen3-Omni variants climbed to the top of trending model lists on popular AI hubs, with other Qwen models occupying additional leading slots. That early attention points to developer interest in open systems from Chinese teams competing directly with Western companies.

Alibaba shares in Hong Kong rose a little over four percent on a day when the release received strong visibility. The market movement accompanied a launch centered on broader access to capabilities that have often been associated with proprietary platforms.

The open alternative adds pressure to pricing and product decisions for closed systems, particularly in uses involving substantial audio and video. Businesses can begin testing ideas without lengthy procurement and can run an omni modal model at little cost, while taking responsibility for deployment and evaluation.

Qwen3-Omni's central proposition is the combination of access and breadth. Developers can customize a model that understands four kinds of input, choose local or hosted use, and produce written or spoken responses according to the version and mode selected. Wider adoption will depend on how reliably those abilities work in the tasks organizations actually need.

The Bottom Line

• Alibaba's Qwen3-Omni accepts text, images, audio and video, with text and speech output available through the Instruct version.
• Thinker handles understanding and reasoning; Talker produces speech using a Mixture of Experts design and a Code2Wav renderer.
• Alibaba reports first packet latency of about 234 milliseconds for audio and 547 milliseconds for video, with streaming faster than real time.
• Language support covers 119 languages for text, 19 for speech input and 10 for speech output.
• The Instruct, Thinking and Captioner versions serve multimodal interaction, written reasoning and detailed audio descriptions.
• Apache 2.0 permits commercial use and private customization. Hosted access has token charges and a trial quota of one million tokens lasting 90 days.
• Company benchmarks show strong results across several types of tasks, while independent replication is still needed.
• Early developer interest is strong, and businesses gain another deployment option alongside responsibility for security, privacy and reliable operation.