Alibaba's Qwen3 Omni brings text, images, audio and video into an openly available AI model

One model for seeing, listening and answering

Alibaba has released Qwen3 Omni, an artificial intelligence model that accepts text, images, audio and video, then answers in text or natural speech. As of October 10, 2025, the launch combines broad capabilities with an Apache 2.0 license, giving developers substantial freedom to run, modify and distribute the technology in commercial products.

Alibaba reports that the model leads 22 of 36 public benchmarks and ranks first among open models on 32. It supports 119 languages for text, understands speech in 19 languages and produces speech in 10. Those figures put its ambitions beyond a chatbot: the same system could transcribe a conversation, examine a video and explain what it sees aloud.

The release comes as Western companies continue investing heavily in proprietary AI and computing infrastructure, while Chinese research groups attract developers with models they can download. Qwen3 Omni has reached the top of Hugging Face community charts, giving Alibaba, China's largest cloud provider, another route into global AI workloads.

Companies hosting the model themselves could build assistants, transcription services and analysis tools without paying usage fees to a closed platform. They would still need the infrastructure to operate it. Combining several kinds of input in one model could also reduce the work of connecting separate systems for speech, images and language.

Alibaba's central promise is an assistant that can listen and look while responding quickly. A person could speak to it or point a camera at a scene and begin hearing an answer as the system processes the video frames. Whether that experience matches the company's reported performance in everyday deployment will shape its practical value.

Three versions for different jobs

Qwen3 Omni is a foundation model, a broadly trained system that developers can adapt to particular applications. Its native multimodal design means it processes several types of information within one system. It accepts four input types, text, images, audio and video, and generates two output types, text and speech.

Alibaba has released three versions built around a backbone with 30 billion parameters. Parameters are the learned numerical settings that guide a model's responses. The architecture selectively activates groups of these settings during inference, the stage when a trained model processes a request.

• Instruct is the full interactive version, handling text, images, audio and video and generating both text and speech.
• Thinking focuses on reasoning and long chains of thought. It accepts the same input types but produces text only.
• Captioner is fine tuned for detailed audio descriptions, with an emphasis on reducing invented details when identifying sound sources and events.

Developers can access the model through the Qwen3 Omni repository and the Qwen model collection. Alibaba also offers a managed API, allowing applications to send requests to a hosted service instead of running the model themselves. That service includes a faster Flash option for applications where response time matters.

How Thinker and Talker divide the work

Alibaba calls the model's two main components Thinker and Talker. Thinker handles reasoning across different kinds of input. It combines text with information from visual encoders and an audio encoder, then produces text tokens representing the intended answer or action. An encoder turns incoming media into representations the model can process.

Talker generates natural speech while keeping timing and intonation connected to the audio and visual context. Both components use a Mixture of Experts architecture, which activates only a subset of parameters at each step. Dividing work among these specialist groups helps throughput when many requests arrive together and supports low latency.

Talker can draw directly on audio and visual features without depending on Thinker's internal text representation. That helps it preserve prosody, the rhythm and emphasis of speech, and timbre, the quality of a voice, during translation or live assistance.

The separation also lets external tools, including information retrieval systems and safety filters, inspect and adjust Thinker's proposed text before Talker turns it into speech. For fast audio generation, Talker predicts a sequence across multiple codebooks, compact representations of sound, one step at a time. A lightweight Code2Wav network converts those predictions into audio frames as they arrive, reducing delay while retaining voice detail.

Speed, languages and long conversations

Alibaba reports first packet latency of roughly 234 milliseconds for audio and about 547 milliseconds for video. That measures the delay before the first output packet arrives, an important part of how responsive a spoken conversation feels.

The company says responsiveness stays within a real time factor of one even while serving multiple requests. In simple terms, processing can keep pace with the duration of the media rather than falling behind it. That matters when an assistant must listen, interpret and answer during a continuing exchange.

Language coverage differs by task: 119 languages for text, 19 for understanding spoken input and 10 for spoken output. The coverage includes major world languages and dialects such as Cantonese. Product teams could therefore serve several regions through one model, although the available languages depend on whether users are typing, speaking or listening.

The listed context window is 65,536 tokens in Thinking Mode and 49,152 tokens in Non Thinking Mode. A context window is the amount of information the system can work with during a request. Tokens are small units of text or representations of other media.

Maximum input and output are listed at 16,384 tokens each, while the longest supported reasoning chain reaches 32,768 tokens. Qwen3 Omni can also analyze and summarize audio clips lasting roughly 30 minutes. Those limits give developers several measures to consider when designing extended conversations or lengthy media analysis.

The training behind its audio skills

Alibaba built a custom Audio Transformer, called AuT, to support audio understanding. Trained from scratch on about 20 million hours of labeled audio, the encoder has around 600 million parameters and supports both streaming caches and offline processing.

Most of AuT's training material focuses on Chinese and English automatic speech recognition, the task of turning spoken words into text. Smaller portions cover other languages and tasks intended to broaden understanding beyond speech transcription.

Pretraining proceeded through three stages. First, encoder alignment trained the vision and audio encoders while keeping the language backbone frozen, helping preserve perception quality. General training then used roughly two trillion tokens across text, audio and images, with smaller amounts of video and combined audio and video.

The final stage extended training sequence length from 8,192 to 32,768 tokens and introduced longer audio and video. This prepared the system to follow extended sequences rather than only short exchanges.

Post training added instruction following and safety alignment. Thinker received supervised fine tuning, strong to weak distillation and GSPO optimization, using a combination of rules and feedback from a language model acting as a judge. These processes refine responses using examples, guidance from a stronger model and evaluations of the generated answers.

Talker went through four stages combining hundreds of millions of multimodal speech samples with continued pretraining on curated data. Alibaba's stated goal was to reduce hallucinations, or invented content, while improving speech clarity and expressiveness.

What the benchmark results show

Alibaba's evaluation covers 36 public benchmarks spanning text, speech, vision and video. Its reported lead on 22 represents about 61 percent of the tests; its lead among open models on 32 represents about 89 percent. The results show particular strength in tasks centered on audio, alongside competitive text and visual performance.

For reasoning, Qwen3 Omni scores 65.0 on AIME25, compared with 26.7 for the GPT 4o result from the 2024 period. On ZebraLogic, it records 76.0 against 57.9 for Gemini 2.5 Flash. Its WritingBench score is 82.6, compared with GPT 4o's 75.5.

Speech recognition results include word error rates of 4.69 and 5.89 on the Wenetspeech suite, compared with GPT 4o's 15.30 and 32.27. A lower word error rate means fewer transcription mistakes. On LibriSpeech other, Qwen3 Omni records an error rate of 2.48.

Music results include 93.0 on GTZAN and 52.0 on RUL MuchoMusic, both above GPT 4o. Vision results include 59.7 on HallusionBench, 57.0 on MMMU pro and 56.3 on MathVision full, also higher than GPT 4o in the same results.

On the MLVU video understanding test, Alibaba reports 75.2 for Qwen3 Omni, compared with 71.0 for Gemini 2.0 Flash and 64.6 for GPT 4o. The model's range matters because a live assistant may need to understand speech and video within the same interaction.

Alibaba tuned Qwen3 Omni to maintain text and vision quality while excelling in speech and multimodal interaction. Early observations indicate that a model from the same family designed for text alone can still lead on some pure language tasks. Developers therefore face a practical tradeoff between specialization and the convenience of handling several media types together. Everyday production performance will be the next test.

What the open license allows

Apache 2.0 permits commercial use, modification and redistribution, and includes a patent grant. Companies can embed Qwen3 Omni in products, operate it on their own hardware or trusted clouds, and distribute closed products without publishing their proprietary changes.

Those permissions reduce common licensing obstacles for organizations considering advanced AI, including in regulated settings. Local deployment also gives companies more control over where data travels and which infrastructure processes it, while regional requirements still guide how a deployment is built.

Qwen3 Omni's downloadable weights are the learned settings needed to run the model. The complete recipe for reproducing its training from raw data is not public. That is why some participants in the openness debate use the term open weights. For companies developing products, Apache 2.0 provides broad usage rights without fees charged per user or seat.

Alibaba's hosted API offers a different route for teams that prefer not to operate the infrastructure. The company provides a free quota of one million tokens across modalities for 90 days after activation, allowing an evaluation period before paid usage.

How the API bill is calculated

Alibaba lists API prices per 1,000 tokens. Because audio, images and video are converted into token representations as well as text, the cost depends on both the type of input and the output requested. Thinking and Non Thinking modes have the same listed token prices, with audio output available only in Non Thinking Mode.

Text input costs $0.00025 per 1,000 tokens, equivalent to about $0.25 per million tokens. Audio input costs $0.00221 per 1,000 tokens, or about $2.21 per million. Image and video input cost $0.00046 per 1,000 tokens, equivalent to about $0.46 per million.

Text output costs $0.00096 per 1,000 tokens when the prompt contains text only. If the input includes an image or audio, text output costs $0.00178 per 1,000 tokens.

When an application requests both text and audio output, the audio portion costs $0.00876 per 1,000 tokens and the text portion is free. Developers estimating a bill therefore need to account for the media entering the system and the form of its answer.

Where companies could use it

Customer support and media localization are natural applications for a model that can listen, understand context and reply in speech. Multilingual transcription and translation could operate through one system. Captioner provides a more specialized option for media tagging, broadcast monitoring and music information retrieval through detailed sound descriptions.

Visual applications include optical character recognition, which reads text from images, object grounding, which connects descriptions to objects in a scene, and questions about images or video. A support assistant could examine a live phone camera view of a laptop, identify a cable to check and explain the steps aloud.

Alibaba has demonstrated translation of a restaurant menu in real time through a wearable device. That example connects its visual understanding, language processing and rapid speech response in a single interaction.

Qwen3 Omni also supports audiovisual dialogue that refers to both what it hears and what it sees. It can retain context across a long session and use relevant external knowledge when connected to tools. Developers can set tone and behavior through system prompts, build assistants for particular fields and route Thinker's proposed answers through custom safety filters before speech is produced.

How it compares with closed and open rivals

OpenAI's GPT 4o accepts text, images and audio and is delivered as a closed service. Google's Gemini 2.5 Pro also handles video analysis and is closed. Google's Gemma 3n has an open license and accepts text, images, audio and video, but produces text only.

Qwen3 Omni accepts those same four input types and adds speech output. That combination is useful for assistants and live translation, where a spoken answer can be part of the service rather than a separate processing step.

Its Apache 2.0 weights also let companies run it locally, adapt deployment to industry regulations and control data flows. Closed models remain stronger on some advanced tasks, while licensing and data residency conditions can restrict their use in sensitive sectors.

Some organizations will use a mixed collection of models: open systems where cost and control matter most, and paid services where their performance is harder to match. Alibaba's broad Qwen family, with hundreds of variants and substantial developer adoption, gives Qwen3 Omni an existing base from which to compete. Its success in live speech and video will depend on how reliably the reported capabilities carry into ordinary workloads.

Developer adoption and investor attention

Qwen3 Omni has topped Hugging Face trending lists, while other Alibaba models occupy several leading positions. The activity reflects growing engagement with openly available models from Chinese laboratories across the global developer community.

Alibaba Cloud says the Qwen family has been downloaded more than 400 million times worldwide. The company has released more than 300 models since the family began. Those figures describe the wider Qwen collection, within which Omni is the latest multimodal release.

Alibaba shares in Hong Kong rose a little over 4 percent on the day Qwen3 Omni reached the top of the community charts. The stock movement shows investor attention to the company's AI progress alongside the interest visible among developers.

Safety controls and the next development targets

Companies adopting Qwen3 Omni will need to assess whether its benchmark performance translates into predictable behavior under load. Spoken interaction brings risks such as misheard commands and convincing synthetic voices, making identity verification, content filtering and data retention concrete operational concerns.

The Thinker and Talker separation provides a place for a safety layer to inspect text before speech generation. Organizations can connect that point to their own controls as they move from demonstrations to services used by customers or employees.

Operating open models at scale also involves MLOps, the systems and processes used to deploy, monitor and maintain machine learning services. Companies may need pipelines for fine tuning and evaluation. Handling several media types in one model can simplify parts of the infrastructure and shorten product development, especially for deployments on company premises or in sovereign clouds built around regional rules.

Alibaba has identified further work on recognizing multiple speakers, reading text in video, improving learning from combined audio and video, and building more capable autonomous agents. These targets match capabilities customers are likely to test as they move pilots into production.

Key Points

• Qwen3 Omni accepts text, images, audio and video and generates text and streaming speech.
• Apache 2.0 permits commercial use, modification and redistribution, with a patent grant.
• Instruct supports full interaction, Thinking produces reasoning responses in text, and Captioner describes audio.
• Alibaba reports first packet latency of about 234 milliseconds for audio and 547 milliseconds for video.
• The model supports 119 text languages, understands speech in 19 and speaks in 10.
• Alibaba reports first place on 22 of 36 benchmarks and leadership among open models on 32.
• The API includes one million free tokens for 90 days after activation, with paid rates varying by media and output.
• Alibaba Cloud reports more than 400 million downloads across the Qwen family and more than 300 model releases.