miHoYo Unveils Gamer, a VLA Foundation Model That Plays Complex Games Using Screen Observations and Native Keyboard Actions

6 Min Read
miHoYo Unveils Gamer, a VLA Foundation Model That Plays Complex Games Using Screen Observations and Native Keyboard Actions

A New Vision for Game Control

Researchers at miHoYo, working through the Honkai AI R&D team and in collaboration with multiple academic institutions, have introduced a vision-language-action foundation model named Gamer. The project shifts away from traditional systems that rely on internal game application programming interfaces or modified memory states. Instead, the framework processes raw visual input from game screens and translates those observations directly into standard keyboard and mouse commands. The development team detailed the underlying architecture in a preprint titled Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent, published on arXiv with submission history starting on November 13, 2026, and a final revision on November 18, 2026. The participating research institutions include the National University of Singapore, Tsinghua University, the University of Chinese Academy of Sciences, the University of Science and Technology of China, and Xi'an Jiaotong University. The project direction involves prominent figures such as David Jiang, producer of several major titles in the Honkai franchise, and Zheng Yinhe, head of the Honkai series artificial intelligence non-player character and gameplay technical team.

Contents
  1. A New Vision for Game Control
  2. Training Pipelines and Multi-Stage Learning
  3. Managing Memory and Long-Horizon Execution
  4. Real-Time Inference and Streaming Architecture
  5. Research Scope and Practical Boundaries
  6. The Essentials

Unlike conventional software agents that require privileged access to game code, Gamer interacts with digital environments much like a human player. The model captures screen frames at a frequency of five hertz, utilizing these visual signals alongside task instructions and observation history to determine appropriate actions. These decisions manifest as a series of low-level keyboard states and mouse movements executed at a thirty hertz control rhythm. By treating user interaction as a shared format that spans conventional graphical interfaces and complex three-dimensional simulated worlds, the framework provides a unified approach to digital navigation and execution.

Advertisement

Training Pipelines and Multi-Stage Learning

Building an autonomous agent capable of sustaining long-horizon gameplay requires moving beyond standard multimodal instruction tuning. The research team implemented a comprehensive training pipeline divided into three distinct phases. The initial phase involves continued pretraining on a large dataset composed of tens of thousands of hours of device-level keyboard and mouse interaction trajectories. These interaction records span simulated games and embodied environments, allowing the model to learn environmental dynamics and visual-motor correlations across diverse digital settings. According to project documentation, the training data incorporates interaction trajectories from sources such as VPT, D2E, OpenP2P, and recorded footage from Honkai: Star Rail.

Following continued pretraining, the model undergoes sparse-reasoning instruction tuning. In this phase, the framework learns to balance immediate motor execution with deliberate planning. Rather than generating continuous explanatory text for every frame, the model executes routine movements directly while reserving explicit reasoning blocks for key decision points, such as puzzle-solving sequences or narrative choices. The final training phase introduces an online reinforcement learning method known as Temporal-Aware Group Relative Policy Optimization. This technique addresses the challenges of finite context windows, accumulated rollout errors during long task sequences, and sparse rewards by linking trajectory segments through persistent memory summaries.

Managing Memory and Long-Horizon Execution

Executing tasks that stretch across hours of gameplay presents significant hurdles related to memory retention and context limits. When an agent explores expansive virtual worlds, early interactions quickly fade from the active context window, leading to navigation errors or forgotten objectives. To preserve task progress without overloading the system with indefinitely growing conversation histories, the researchers incorporated a persistent memory mechanism. As tasks progress and older interactions roll out of the active context, the model generates compact summaries that record completed milestones, failed attempts, important locations, and subsequent plans.

The effectiveness of this multi-stage training and memory management approach was evaluated through rigorous testing in specific role-playing scenarios. In evaluated side quests within Honkai: Star Rail, milestone completion rates demonstrated steady improvement across training stages. For the side quest titled Opportunity Makes the Thief (Part I), milestone completion rates rose from 34.4% following continued pretraining to 72.2% after supervised fine-tuning, reaching 80.0% with the application of online reinforcement learning. Similarly, for the side quest Bestial Ferocity, completion rates progressed from 47.9% to 69.1%, and ultimately achieved 83.6% following reinforcement learning optimization. Across an evaluated set of forty unfamiliar mini-game tasks, a version designated as Gamer-RL achieved an average success rate of 49.7%.

Advertisement

Real-Time Inference and Streaming Architecture

Maintaining high performance during real-time interaction requires keeping inference latency well within the tolerance thresholds of interactive software. To prevent delays that would cause an agent to miss dynamic events in fast-paced environments, the development team engineered a cloud-edge serving system called Streaming-vLLM. This inference engine combines persistent context reuse, verified speculative decoding, and early action dispatch to process parallel rollouts efficiently. Persistent context reuse avoids redundant calculations of historical screen states by retaining key-value caches across frames, while early action dispatch transmits the first valid group of keyboard commands to the game client before subsequent tokens finish generating.

Performance benchmarks detailed in the project repository illustrate the impact of these optimizations. Operating in an environment equipped with NVIDIA H200 hardware and utilizing a model configuration based on Qwen3.5-35B-A3B with tensor parallelism set to four, the baseline setup registered an end-to-end request latency of 1,306.00 milliseconds. Following the application of Streaming-vLLM optimizations, that request latency dropped to 65.12 milliseconds. This reduction in execution time allows the system to support concurrent online evaluations and maintain responsive control loops during complex spatial navigation and combat sequences.

Research Scope and Practical Boundaries

The release of this research framework has generated widespread discussion regarding its potential capabilities and intended applications. Project documentation and public statements from the research team emphasize that Gamer is an academic exploration project designed to investigate generalist agent behavior in interactive environments, rather than an authorized commercial tool for automating player accounts or generating competitive advantages. To address data security considerations, the team withheld the specific training data and model weights associated with proprietary titles, making only foundational pretraining files and system documentation publicly accessible through resources such as the Gamer GitHub repository.

Potential practical applications outlined by the developers focus primarily on industrial and developmental utility rather than consumer automation. Automated testing pipelines could leverage such models to conduct stress testing across branching narrative paths and expansive virtual worlds, identifying implementation defects more rapidly than traditional manual verification methods. Additionally, the underlying principles offer insights into accessibility assistance and human-computer interaction research, paving the way for agents capable of translating diverse input modalities into precise digital actions.

Advertisement

The Essentials

  • The research framework was developed by the Honkai AI R&D team at miHoYo in partnership with academic institutions including the National University of Singapore and Tsinghua University.
  • Detailed findings and architectural descriptions are documented in the preprint Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent.
  • The system captures raw screen observations at five hertz and outputs native keyboard and mouse commands without requiring internal application programming interfaces or memory hooks.
  • Training procedures incorporate continued pretraining on tens of thousands of hours of interaction trajectories, sparse-reasoning instruction tuning, and Temporal-Aware Group Relative Policy Optimization.
  • The custom inference engine, Streaming-vLLM, reduced request latency from 1,306.00 milliseconds to 65.12 milliseconds in tests utilizing NVIDIA H200 hardware.
  • The research team maintains that the project serves as an academic study aimed at applications such as automated testing and accessibility research, rather than a commercial tool for gameplay automation.
Share This Article

You May also Like