Alibaba Releases Qwen3 Omni With Video Understanding, Speech Output and Apache 2.0 Access

A model that reads, watches, listens and speaks

Alibaba has introduced Qwen3 Omni, a multimodal artificial intelligence model that accepts text, images, audio and video, and answers through text or natural speech. Released under the Apache 2.0 license, it gives developers permission to use, modify and redistribute the model commercially, including running it on their own infrastructure.

The release brings together video understanding, spoken interaction and access to model weights, the learned parameters needed to run the system. Alibaba positions Qwen3 Omni as a native end to end omni model, meaning it was designed to handle different kinds of information together instead of adding speech or vision later.

Alibaba says the model supports 119 languages for text, 19 for speech input and 10 for speech output. Its coverage includes major languages and dialects such as Cantonese. The company also reports first packet latency of about 234 milliseconds for audio and 547 milliseconds for video, putting the start of a response at roughly a quarter of a second or half a second respectively.

For businesses comparing open models with proprietary systems such as GPT 4o and Gemini 2.5 Pro, access is a central difference. Teams can host Qwen3 Omni themselves, adapt it and integrate it into products without license charges for each user or a requirement to stay on one provider's platform. Running the model still requires computing infrastructure.

Three variants address different tasks: Instruct for conversation with text and speech, Thinking for complex reasoning with text responses, and Captioner for descriptions of audio. Developers can obtain weights from major model hubs, use an API, or fine tune the model for a particular domain and style. An API Flash option targets faster responses and more simultaneous requests.

What the three variants do

Alibaba Cloud released the three variants within the Qwen3 Omni 30B A3B family. Their differences matter because a conversational assistant, a reasoning tool and an audio description service need different outputs.

• Instruct combines the Thinker and Talker components. It accepts text, images, audio and video, and produces both text and speech.
• Thinking accepts the same types of input but produces text only. It focuses on complex reasoning and long chains of thought.
• Captioner specializes in audio captions and descriptions, with an emphasis on reducing hallucinations, or descriptions that invent details. Its intended uses include media analysis, accessibility and content moderation.

The API's Flash variant is aimed at lower latency and higher concurrency. Latency is the delay before a response arrives; concurrency is the number of requests a system can handle at once. A support service or shopping assistant may need both when many customers arrive together.

How Thinker and Talker work together

Qwen3 Omni divides comprehension and speech generation between two coordinated components. Thinker aligns and combines information from text, images, audio and video, reasons about it, and produces representations at the token level that guide the response. Tokens are the pieces of information a model processes.

Talker generates speech. Its conditioning includes audio and visual features rather than simply consuming Thinker's output as written text. This helps it match speech timing, voice character and prosody, the rhythm and stress of spoken language, to what the system sees and hears. The design supports translation and voice acting traits such as pacing, emphasis and emotional tone.

The separation also allows external modules to influence or filter Thinker's output before Talker renders speech. Developers can insert policy checks, retrieval steps or safety filters into that process, making it possible to check or supplement an answer before it is spoken.

Both components use Mixture of Experts. A routing network sends each token to a small selection of expert networks, so only part of the model's parameters runs for that token. Alibaba uses this approach to improve throughput, reduce inference costs and keep responses moving when requests arrive together or hardware resources are constrained.

Where the streaming speed comes from

Talker predicts discrete acoustic tokens through a multi codebook autoregressive scheme. In simpler terms, it generates successive pieces of a compact sound representation. A lightweight convolutional network called Code2Wav then converts those pieces into an audible waveform.

Alibaba reports first packet timings of around 234 milliseconds for audio and 547 milliseconds for video. These figures describe the beginning of a streamed response. A service also needs to measure how quickly stable content arrives and how long a complete answer takes, especially for longer responses.

The company says the system stays below a real time factor of one, including while handling multiple requests. That means generating an audio response takes less time than playing that response aloud. Streaming at this pace matters for assistants expected to interrupt, clarify or hand a conversation over quickly.

Support calls and mobile guides are among the settings where these delays affect the experience directly. A model that can begin responding promptly is more suited to an exchange than a system that processes everything as a batch and makes the user wait. The Flash API option pursues the same need for speed during heavier traffic.

The training behind its audio and video skills

Alibaba built a new Audio Transformer encoder with about 600 million parameters and trained it on roughly 20 million hours of supervised data. Most of that material concerns Chinese and English automatic speech recognition, with additional languages and tasks focused on understanding audio.

The encoder supports streaming caches to reduce delay and can also operate offline for tasks such as analyzing long recordings. Its output passes through alignment stages into the multimodal core, allowing audio information to be used alongside other formats.

Training mixed data from individual formats with data connecting different formats, then continued scaling through pretraining and post training. The pretraining process had three stages, starting with encoder alignment: the vision and audio encoders were trained and aligned while the language model remained frozen, meaning its parameters were held unchanged.

General training then expanded to roughly 2 trillion tokens drawn from text, audio, images, some video, and paired audio and video. Long context training increased the maximum sequence length from 8,192 to 32,768 tokens while adding more long audio and video sequences.

Thinker's post training included supervised fine tuning, distillation from stronger teacher models, and policy optimization guided by rules and model judgments. Distillation transfers useful behavior from a stronger model to another model. Talker followed a four stage regimen using hundreds of millions of multimodal speech samples together with continual pretraining, with the aim of reducing hallucinations and improving voice quality.

Language coverage and context limits

Qwen3 Omni's language support varies by task: 119 languages for text, 19 for incoming speech and 10 for spoken responses. A language available for reading or writing may therefore have different support for listening or speaking.

Thinking Mode has a context window of 65,536 tokens, while Non Thinking Mode supports 49,152. The context window is the working space available to the model during an interaction. The published limits also list up to 16,384 tokens for input and 16,384 tokens for output, with the longest chain of thought reaching 32,768 tokens.

Those capacities are intended to help with transcripts, long meetings, code bases and documents that would otherwise need to be divided into smaller pieces. The longer sequences used during training also support work involving extended audio and video.

Thinking and Non Thinking modes have the same API pricing, according to Alibaba. Audio output is available only in Non Thinking Mode. A developer choosing a reasoning mode therefore also needs to account for the required response format.

What Alibaba's benchmark results show

Alibaba reports that Qwen3 Omni takes the top score on 22 of 36 benchmarks and leads among open source models on 32. The company's results cover text, reasoning, speech, sound, images, video and music, with particular strength in audio and audiovisual understanding while maintaining competitive text and vision performance.

For text and reasoning, the company lists an AIME25 score of 65.0, compared with 26.7 for GPT 4o. On ZebraLogic, Qwen3 Omni scores 76.0 against 57.9 for Gemini 2.5 Flash. WritingBench gives it 82.6, compared with 75.5 for GPT 4o.

Speech recognition results include word error rates of 4.69 and 5.89 on two common Wenetspeech test splits, against 15.30 and 32.27 for GPT 4o. A lower word error rate means fewer transcription mistakes. On Librispeech other, Qwen3 Omni records 2.48, a low result among the peers compared.

For images and visual reasoning, Alibaba lists 59.7 on HallusionBench, 57.0 on MMMU pro and 56.3 on MathVision full, each above GPT 4o in its comparison. The video understanding benchmark MLVU gives Qwen3 Omni 75.2, ahead of 71.0 for Gemini 2.0 Flash and 64.6 for GPT 4o.

Music results include 93.0 on GTZAN and 52.0 on RUL MuchoMusic. Together, the tests reflect the model's focus on listening and understanding sound and video, alongside its other abilities.

Alibaba's figures come from controlled evaluations. Dataset choices and scoring methods can favor particular training mixes, while actual services encounter accents, noisy microphones, unclear phone camera footage and fast changes of context. Organizations can assess the model more directly by testing it on the recordings, images and requests their users produce.

Training a model across many formats can also reduce some text performance compared with larger models built primarily for text. One practical arrangement is to use a smaller multimodal model to listen and look, then send report or email drafting to a larger text model. Apache 2.0 permits that kind of combination without the same procurement friction as a restricted model license.

How it compares with GPT 4o, Gemini and Gemma

OpenAI's GPT 4o established expectations for closely connected text, image and audio processing with natural speech. Qwen3 Omni's input list also includes video analysis. Google's proprietary Gemini 2.5 Pro offers video analysis, while Google's open Gemma 3n accepts text, images, audio and video but produces text only.

Qwen3 Omni combines those four input formats with both text and speech output. That combination extends its potential uses to spoken translation, interactive assistance and handoffs between automated systems and people in call centers.

Licensing is another dividing line. Apache 2.0 permits commercial use, modification and redistribution, includes a patent license, and allows integration into proprietary products without requiring publication of the adopter's source code or payment of model runtime license fees. Proprietary systems require paid access and tie use to a provider's platform.

Open access here centers on model weights and usage code. The complete recipe for rebuilding a model from scratch, including curated training data and full training code, is not generally part of such releases. Businesses seeking to run, fine tune and integrate the model receive those practical freedoms, while reproducing the entire training process is a different undertaking.

Alibaba places its first packet timings in the same class as leading conversational systems. For a deployed service, the useful comparison also includes the time to stable content, completion time for long answers and performance under load, alongside the API's faster Flash tier.

API prices and the free quota

Developers can access Qwen3 Omni through downloadable weights on major model hubs or through an API. The API includes a free quota of one million tokens across modalities for 90 days after activation. Alibaba lists its model options, context limits and service details in the Model Studio documentation.

The listed prices are charged per 1,000 tokens and vary with the input format and requested output. That makes the mix of text, audio, images and video a direct part of estimating a service's bill.

• Text input costs $0.00025 per 1,000 tokens.
• Audio input costs $0.00221 per 1,000 tokens.
• Image and video input costs $0.00046 per 1,000 tokens.
• Text output costs $0.00096 per 1,000 tokens when the input is text only.
• Text output costs $0.00178 per 1,000 tokens when inputs include images or audio.
• For combined text and audio output, the audio portion costs $0.00876 per 1,000 tokens and the text portion is free.

Alibaba charges the same rates for Thinking and Non Thinking modes, with speech output restricted to Non Thinking Mode. Model Studio lets companies select models according to speed, quality and cost. The listed rates support relatively inexpensive experiments compared with many proprietary services, while hosting the weights directly shifts infrastructure planning to the adopter.

Where seeing and speaking could be useful

A technical support assistant could use Qwen3 Omni to inspect a customer's phone camera feed, listen to a description of a problem and respond with spoken instructions in successive steps. A shopping guide could use the same combination while handling changes in customer demand through the Flash API.

Multilingual transcription and translation are another intended use. Meetings, calls and lectures can be transcribed in supported languages, followed by an English summary or a spoken answer in another supported language. Captioner can describe background sounds and musical traits for media search, accessibility, content moderation and rights management workflows.

Video understanding allows questions about events in a clip, event tracking and scene summaries. Vision and language can also work together for optical character recognition, or OCR, extracting fields from forms and receipts and checking those fields against their context.

Consumer assistants on phones or wearables could translate menus, identify landmarks and describe surroundings for accessibility. These applications draw on the model's combination of visual input, language processing and spoken output.

Developers can use system prompts to shape conversational style and persona, and fine tune the model for more specific requirements. Tool use lets the system call external services for retrieval, calculations or database lookups. In an agent workflow, the model can plan an action, call a tool and report the result through speech.

What businesses gain from running it themselves

Apache 2.0 gives organizations room to embed Qwen3 Omni in products or internal workflows, modify it and keep their changes private. Its patent grant reduces legal risk when the model is integrated into proprietary systems.

Open weights also allow deployment on premises or in a trusted cloud. That control can appeal to regulated sectors that need to decide where information travels and how interactions are logged. Organizations can run the model without vendor model license fees while taking responsibility for the infrastructure that supports it.

Companies can combine Qwen3 Omni with a larger proprietary model where that meets their needs, or choose an entirely open collection of models to reduce platform constraints. A model that understands video and generates speech under a permissive license expands the range of products that can be built without lengthy licensing negotiations.

The absence of a complete public training recipe limits community reproduction of the entire build. For businesses focused on deployment, customization and service targets, the immediate decision is whether the released model can meet their requirements with the hardware and controls they can provide.

Safety and infrastructure still need testing

Audio and video can contain sensitive information, and spoken responses can mislead when an answer is inaccurate or lacks grounding. Alibaba's design allows external modules to intercept Thinker outputs before Talker turns them into speech, supporting policy checks, retrieval and filters used in adversarial safety testing.

Organizations still need to validate content safety in their own environment and region. That includes privacy controls, consent for recording or analysis, and defenses against prompt injection, where an input attempts to redirect the system away from its intended instructions.

Accents, specialized vocabulary and background noise can reduce speech recognition accuracy. Low light and motion blur can weaken image understanding. Tool based workflows introduce further risks, including incorrect tool calls and loops in which the system repeatedly takes an unhelpful action.

Logging, human review and rollback procedures provide ways to investigate problems and reverse changes. Infrastructure planning also needs to cover throughput, graphics processor memory, sudden traffic increases, long context caches and load shedding, which limits incoming work when a service is overloaded.

Mixture of Experts reduces the amount of computation used for each token, but operating speech services at scale still requires enough hardware and careful capacity planning. The practical test for adopters is whether Qwen3 Omni can deliver accurate, safe and timely responses on their own data under the demand they expect.

Key Points

• Alibaba's Qwen3 Omni accepts text, images, audio and video, and produces text and speech under Apache 2.0 licensing.
• Instruct supports full conversation, Thinking focuses on reasoning with text output, and Captioner specializes in audio descriptions.
• Alibaba reports first packet latency of about 234 milliseconds for audio and 547 milliseconds for video.
• The model supports 119 languages for text, 19 for speech input and 10 for speech output.
• Training included roughly 20 million hours for the audio encoder and about 2 trillion tokens during general multimodal training.
• Alibaba reports top results on 22 of 36 benchmarks and leadership among open source models on 32.
• The API includes one million free tokens across modalities for 90 days after activation; downloadable weights allow independent hosting.
• Businesses gain control over deployment and customization while taking responsibility for infrastructure, privacy, safety and performance testing.