Alibaba launches Qwen3 Omni, an open source multimodal AI model

Why this launch matters now

Alibaba has released Qwen3 Omni, a multimodal artificial intelligence model that processes text, images, audio, and video in one system, and it arrives as global competition in AI intensifies. The model can be downloaded and used under the Apache 2.0 license, which gives companies broad rights to run, modify, and ship the technology in commercial products. That combination of broad modality coverage and permissive licensing puts direct pressure on closed systems from US leaders.

The timing is striking. Western firms keep pouring money into proprietary AI and the infrastructure behind it, while Chinese research groups are courting developers with open access models. Qwen3 Omni has quickly climbed community charts on Hugging Face, a sign that the developer base is engaging with it at scale. For Alibaba, the release signals that China’s largest cloud provider intends to compete for global AI workloads, not only research mindshare.

The model’s promise is simple to state. One foundation model that can take in language, pictures, audio, and video, then respond in text or natural speech with streaming speed. If the performance that Alibaba reports holds up in mainstream use, enterprises could deploy assistants, transcription services, and analytics tools without paying usage fees to closed platforms, and without stitching together separate models for each data type.

What Qwen3 Omni is and what it does

Qwen3 Omni is a foundation model built for native multimodal interaction. It accepts four kinds of input, text, image, audio, and video, and it produces two kinds of output, text and speech. Responses can stream in near real time, so a user can talk with the system or point a camera at a scene and begin hearing an answer as it processes the frames.

Alibaba is shipping three versions so teams can match capabilities to use cases. Each version is based on a 30 billion parameter backbone that selectively activates experts during inference for efficiency. The line up looks like this:

• Instruct, the full interactive model that handles audio, video, and text inputs, and generates both text and speech.
• Thinking, the reasoning focused model for long chain of thought responses. It accepts the same inputs but only outputs text.
• Captioner, a fine tuned variant for detailed audio captioning that aims to minimize hallucination while describing sound sources and events.

Developers can download weights from GitHub and Hugging Face, or call a managed endpoint from Alibaba’s API, which also offers a faster Flash variant for latency sensitive uses.

Inside the Thinker and Talker design

The system follows a two component architecture that Alibaba calls Thinker and Talker. Thinker is responsible for reasoning across modalities. It fuses cues from text, visual encoders, and an audio encoder, then plans and produces text tokens that represent the intended answer or action. Talker converts that plan into natural speech, while keeping timing and intonation aligned to the visual and audio context.

Both components use a Mixture of Experts approach. Only a subset of parameters is active for each step, which spreads workload across experts and improves throughput when many requests arrive at once. This design helps the model maintain low latency while serving concurrent users.

Talker does not depend on Thinker’s internal text representation. It can condition directly on audio and visual features. That makes speech output more natural during translation or live assistance, since the system can preserve prosody and timbre that match the scene. The separation also has operational benefits. External modules, such as retrieval systems or safety filters, can inspect and adjust Thinker’s proposed text before Talker renders speech.

To keep speech fast and clear, Talker predicts a multi codebook sequence autoregressively and uses a lightweight Code2Wav network to synthesize audio frames on the fly. The approach reduces delay without sacrificing detail in voice quality.

Speed, scale, and languages

Latency matters in voice and video interaction. Qwen3 Omni reports first packet latency of roughly 234 milliseconds for audio and about 547 milliseconds for video. End to end responsiveness stays within one real time factor even when serving multiple requests, which keeps a conversation fluid.

Language coverage is wide. The model supports 119 languages in text. It understands spoken input in 19 languages and can speak in 10. Coverage includes major world languages and dialects such as Cantonese. That gives product teams a path to build assistants for global audiences without swapping models by region.

Context windows are designed for long sessions. Thinking Mode offers 65,536 tokens, Non Thinking Mode offers 49,152 tokens. Maximum input and output are 16,384 tokens each, and the longest supported reasoning chain extends to 32,768 tokens. The model can analyze and summarize extended audio clips, up to roughly 30 minutes.

How Alibaba trained the model

Audio understanding is anchored by a custom Audio Transformer called AuT. Alibaba trained AuT from scratch on about 20 million hours of labeled audio. Most of that set focuses on Chinese and English automatic speech recognition, with a smaller share from other languages and tasks aimed at broader audio understanding. AuT has around six hundred million parameters and is tuned for both streaming caches and offline jobs.

Pretraining ran in three stages. In an encoder alignment phase, the team trained vision and audio encoders while keeping the language backbone frozen, which helps preserve perception quality. General training followed on a dataset of roughly two trillion tokens spanning text, audio, and images, with smaller portions of video and audio video data. A final long context phase extended the sequence length from 8,192 tokens to 32,768 tokens and mixed in longer audio and video to make sure the model can follow extended sequences.

Post training added instruction following and safety alignment. For Thinker, the team applied supervised fine tuning, strong to weak distillation, and GSPO optimization using a combination of rules and LLM as a judge feedback. Talker went through four stages that combine hundreds of millions of multimodal speech samples with continual pretraining on curated data. The goal was to reduce hallucination while improving clarity and expressiveness in speech.

Benchmarks, strengths, and limits

Across 36 public benchmarks that cover text, speech, vision, and video, Alibaba reports that Qwen3 Omni holds first place on 22, and leads open models on 32. The results point to strong performance on audio heavy tasks and competitive results in text and vision.

Reasoning and writing: on AIME25 the model scores 65.0, a wide margin over the 2024 era GPT 4o result of 26.7. ZebraLogic shows 76.0, ahead of Gemini 2.5 Flash at 57.9. On WritingBench the model posts 82.6, higher than GPT 4o at 75.5.

Speech and audio: on the Wenetspeech suite, reported word error rates are 4.69 and 5.89, far below GPT 4o at 15.30 and 32.27. On LibriSpeech other, error drops to 2.48. Music classification shows strength as well, with GTZAN at 93.0 and RUL MuchoMusic at 52.0, both above GPT 4o.

Vision and video: HallusionBench comes in at 59.7, MMMU pro at 57.0, and MathVision full at 56.3, all higher than GPT 4o in the same set of results. On the MLVU video understanding test, Qwen3 Omni posts 75.2, ahead of Gemini 2.0 Flash at 71.0 and GPT 4o at 64.6.

These are strong numbers, yet teams should read them in context. Alibaba tuned the system to keep text and vision quality while excelling in speech and multimodal interaction. Early observers note that a text only model from the same family can still lead on some pure language tasks, which is a normal tradeoff when moving from single modality training to a broad multimodal stack. The real test is day to day performance in production workflows.

Open license, access, and pricing

Qwen3 Omni is released under Apache 2.0. That license allows commercial use, modification, and redistribution, and it includes a patent grant. Enterprises can embed the model in products, deploy it on their own hardware or in trusted clouds, and ship closed products without exposing proprietary changes. The license removes the most common legal obstacles that keep teams from using advanced models in regulated settings.

The model family is available to download from GitHub and Hugging Face. Teams that prefer a managed path can use Alibaba’s API, which also exposes a Flash option for faster responses. Alibaba offers a free quota of one million tokens across modalities for 90 days after activation so teams can evaluate capabilities before paying.

How the token pricing works

Billing is per one thousand tokens. A token is a small unit of text or a chunked representation of other media, and it is the common yardstick for compute cost across modalities. Alibaba lists the same per token pricing for Thinking and Non Thinking modes, with one exception: audio output is available only in Non Thinking Mode.

Input costs: text input costs 0.00025 dollars per thousand tokens, which is about 25 cents per million tokens. Audio input costs 0.00221 dollars per thousand tokens, about 2.21 dollars per million tokens. Image and video input costs 0.00046 dollars per thousand tokens, about 46 cents per million tokens.

Output costs: text output costs 0.00096 dollars per thousand tokens when the prompt is text only. If the input includes image or audio, text output is 0.00178 dollars per thousand tokens. When you request both text and audio output, the audio portion costs 0.00876 dollars per thousand tokens. In this case the text part is free.

Use cases that are ready today

Audio and video centric applications stand to benefit. Multilingual transcription and translation for customer support and media localization can run on one model that listens, understands context, and responds as speech. The Captioner variant focuses on audio description, which is useful for media tagging, broadcast monitoring, and music information retrieval.

Visual tasks such as optical character recognition, object grounding, and image or video question answering also fit. A support agent could watch a live phone camera feed of a laptop and describe which cable to check, then narrate the steps at natural speed. Alibaba has demoed real time translation of a restaurant menu through a wearable device. These scenarios highlight why streaming latency and speech quality matter.

Qwen3 Omni also supports audiovisual dialogue. It can carry a conversation that references both what it hears and what it sees, keep track of context across a long session, and surface relevant external knowledge when connected to tools. Developers can shape tone and behavior with system prompts, build domain specific assistants, and route Thinker outputs through custom safety filters before any speech is produced.

How it stacks up against GPT 4o, Gemini 2.5, and Gemma 3n

Qwen3 Omni joins a class of omni models that aim to process many data types in one engine. OpenAI’s GPT 4o supports text, image, and audio input, and is delivered as a closed service. Google’s Gemini 2.5 Pro also handles video analysis and is closed. Google’s Gemma 3n brings an open license and accepts text, image, audio, and video as inputs, yet it generates text only. Alibaba’s model accepts the same four inputs and adds speech output, which matters for assistants and live translation.

Openness is a practical difference. Qwen3 Omni weights are available under Apache 2.0, so companies can run the model locally, adapt it to industry regulations, and control data flows. Closed models still dominate for some high end tasks, yet their licensing and data residency conditions can limit deployment in sensitive sectors. Many organizations will choose a mixed stack, combining open models for cost and control with paid services where performance is hard to beat.

Another difference is the production story. Alibaba has built a broad Qwen family with hundreds of variants and reports wide adoption by developers. If Qwen3 Omni’s live speech and video interaction keep up with the benchmarks in daily use, it gives teams a credible alternative when they need a multilingual, low latency assistant that they can host themselves.

Adoption and market signals

Developer engagement is visible. Qwen3 Omni has reached the top of Hugging Face trending lists, and other Alibaba models now occupy several of the top slots. The momentum reflects a wider shift where open models from Chinese labs are gaining traction in the global community.

Alibaba Cloud says the Qwen family has been downloaded more than 400 million times worldwide, and the company has released more than 300 models since the line began. Investor reaction has been supportive. Alibaba shares in Hong Kong rose a little over 4 percent on the day Qwen3 Omni climbed to the top of the community charts, a sign that the market is watching the company’s AI progress.

There is also debate about what open means. The models ship under Apache 2.0 and weights are available, yet the full recipe to reproduce training from raw data is not public. Purists would call this open weights. For most enterprises, the license terms are what matter, and they allow shipping products without paying per user or per seat fees.

What to watch on safety and governance

Enterprises will ask whether the reported benchmark gains translate into predictable, safe behavior under load. Speech systems invite unique risks, from misheard commands to convincing vocal synthesis. Teams should set guardrails for identity verification, content filtering, and data retention. The separation between Thinker and Talker supports those controls, since a safety layer can intercept text before speech is generated.

Operational readiness will also come into focus. Organizations that adopt open models at scale often invest in MLOps, fine tuning pipelines, and evaluation. Qwen3 Omni reduces the need to juggle multiple models by handling text, audio, image, and video in one place. That can simplify infrastructure and speed up product cycles, especially for on premises or sovereign cloud deployments that must meet regional rules.

Alibaba has outlined next steps, including better multi speaker recognition, text recognition for video, stronger learning from combined audio and video, and more capable autonomous agents. These targets align with the areas where customers will push the model first as they move pilots into production.

Key Points

• Qwen3 Omni is a single model that accepts text, image, audio, and video, and outputs text and speech with streaming speed.
• Alibaba releases the weights under Apache 2.0, which permits commercial use, modification, and redistribution with a patent grant.
• Three variants ship now, Instruct for full interaction, Thinking for long reasoning with text output, and Captioner for low hallucination audio description.
• Thinker and Talker split reasoning from speech generation, using a Mixture of Experts to keep latency low under concurrent load.
• Reported first packet latency is about 234 milliseconds for audio and around 547 milliseconds for video, within one real time factor.
• The model supports 119 text languages, understands speech in 19, and can speak in 10, including dialect support such as Cantonese.
• AuT, a six hundred million parameter audio encoder trained on about 20 million hours, anchors speech understanding.
• Benchmarks show leadership on 22 of 36 tests and open model leadership on 32, with strong results in speech, music, vision, and video.
• API pricing starts at 0.00025 dollars per thousand text input tokens and 0.00096 dollars per thousand text output tokens, with a free 1 million token trial for 90 days.
• Adoption is brisk on Hugging Face, and Alibaba reports hundreds of millions of Qwen downloads as enterprises evaluate open models for production use.