An AI player built for the developer's desk
HoYoverse's Honkai AI research team has unveiled Gamer, an artificial intelligence system that plays games by looking at the screen and sending keyboard and mouse commands. In a reported test lasting 50 hours, it controlled Honkai: Star Rail autonomously for 95% of the time, tackling a challenge that goes beyond choosing the right move: remembering what to do across a long adventure.
Contents
- An AI player built for the developer's desk
- How does Gamer turn screenshots into actions?
- Teaching the system to remember a quest
- What the quest results actually measure
- Puzzles reveal a much less even performance
- Why faster decisions require serious hardware
- What has been released, and what is still pending?
- How a virtual player could help test a game
- Why the work reaches beyond Honkai: Star Rail
- Key Points
The miHoYo team published its project page and research paper on November 13, 2026. Demonstrations show Gamer following quests, speaking to characters, fighting enemies and bosses, solving puzzles, and completing five varieties of minigames. Its input comes from screenshots rather than hidden information inside the game.
The intended audience is developers and researchers. HoYoverse describes Gamer as a tool that could help test games, find problems and reduce repetitive development work, with longer term research extending to artificial intelligence for robots. The reported performance tests use powerful data centre hardware, including Nvidia H200 graphics processors.
For players, the immediate development is therefore a research project rather than a new automatic play feature. For developers, the attraction is more concrete: a virtual tester that can attempt a quest through the same visible interface a person uses, then keep going through conversations, travel, battles and puzzles.
How does Gamer turn screenshots into actions?
Gamer is a vision-language-action model. In plain language, it connects three abilities: seeing what is on a screen, understanding instructions, and producing actions. The paper is titled Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent. The lengthy name describes the central goal: controlling games over extended periods rather than completing a single isolated action.
The system samples the game screen five times per second and produces keyboard and mouse instructions equivalent to 30 updates per second. Each action decision covers about 200 milliseconds, divided into six steps of roughly 33 milliseconds each. This lets it plan a short sequence of inputs between new visual observations.
A human player might glance at a destination marker, hold a movement key and adjust the camera. Gamer learns a comparable relationship between screen images and input commands. It can move characters, turn the camera, navigate menus and interact with objects without relying on a special interface that supplies the game's internal state.
The published action format includes 74 key tokens with specific capitalization rules. Invalid output is handled as no action. Such rules matter because an AI system must send commands the software can reliably interpret, even when its wider reasoning is imperfect.
Teaching the system to remember a quest
The training starts with more than 10,000 hours of recorded keyboard and mouse activity across games. These records connect what appeared on screen with the actions taken, helping Gamer learn basic control skills before it is asked to finish longer objectives.
The researchers then added instruction training with explanations of decisions and summaries of earlier activity. The reported training material includes about 1.2 million reasoning annotations and 223,000 memory annotations. These give the model examples of how to select a next step and preserve useful information.
Memory is essential in a quest that sends a player between several places. A system may need to remember which character it has already spoken to, which route failed and which objective remains unfinished. Gamer creates compact summaries so it can continue after its active working context fills up.
The team also uses what it calls sparse reasoning. Rather than generating a lengthy explanation before every movement, the model can continue acting while a plan remains useful and spend more effort reasoning when circumstances change. This helps connect deliberate planning with the quick inputs a game requires.
What the quest results actually measure
The researchers evaluated Gamer on two Honkai: Star Rail side quests, Opportunity Makes the Thief (Part I) and Bestial Ferocity. Both involve several kinds of activity, including travelling, talking to characters and fighting enemies, with a simple puzzle also included in the quest testing.
After the initial continued training stage, the reported quest objective completion scores were 34.4% and 47.9%, respectively. After reinforcement learning, they reached 80.0% and 83.6%. The gains were 45.6 percentage points for the first quest and 35.7 percentage points for the second.
Reinforcement learning means improving through attempts whose outcomes receive a score. Here, the team used a method called Temporal-Aware GRPO to connect rewards with actions across stretches of play linked by memory. The problem is much like learning which earlier decision helped finish a task several minutes later.
The 50 hours of testing with 95% autonomous control measures how much of the running time Gamer handled itself. Quest objective scores measure progress against particular goals. The team also claimed more than 95% success on main storyline challenges in automated tests, a further measure with a different scope. These published results come from the developers; independent replication has yet to be reported.
Puzzles reveal a much less even performance
Gamer's minigame results show why a capable demonstration can sit alongside substantial weaknesses. Across 40 previously unseen tasks spanning seven minigame types, the version trained with reinforcement learning achieved an average success rate of 49.7%. The earlier training stage scored 15.0%, making the improvement 34.7 percentage points.
The strongest and weakest examples were far apart. Gamer scored 95.8% on Navigation Compass puzzles and 22.2% on Abacus Circuitry. The difference was 73.6 percentage points, showing how strongly performance depended on the particular puzzle.
The demonstration videos feature successful play across five minigame varieties, while the evaluation covers seven types. The wider test gives developers a more detailed picture of the system's abilities, including tasks where it still struggles.
For a testing team, those differences matter in practical terms. An agent that reliably navigates one challenge may need closer supervision on another. The results provide a basis for deciding where automated testing could save work and where people would still need to check outcomes carefully.
Why faster decisions require serious hardware
The research team developed an inference engine called Streaming-vLLM to reduce the delay between requesting a decision and receiving it. Inference is the stage when a trained AI model processes new information and produces an answer or action.
In the reported Nvidia H200 setup, the engine reduced end to end request latency from 1,306 milliseconds to 65.12 milliseconds, roughly a twentyfold speed increase. Put simply, the delay fell from about 1.3 seconds to about 0.065 seconds. That matters when the game continues moving while the AI is deciding.
The reported configuration used a Qwen3.5-35B-A3B mixture of experts model with computation distributed across four devices. A mixture of experts model activates selected parts of its network for a task. The hardware and software configuration is central to understanding the speed figure, because it describes a research setup rather than performance on an ordinary home computer.
Streaming-vLLM cuts repeated computation, reuses stored information and can send the first valid action while later output is still being generated. Faster decisions also help training: a system that spends less time waiting can attempt more interactions within the same period.
What has been released, and what is still pending?
The public release includes a project page, the paper and a GitHub repository with code under the Apache-2.0 licence. The repository describes prompts, action formats and evaluation results. Model weights, the learned numerical settings needed to run a trained model, are not bundled with the code release.
The team has withheld Honkai: Star Rail training data and model weights trained or adjusted using that game's data, citing data security considerations. The public files relate to the project's earlier general training research.
The Streaming-vLLM implementation is listed as coming soon. No release date was given for that engine. Together with the missing trained weights and demanding hardware, this leaves a substantial gap between reading the research and reproducing the demonstrated system.
The publication sequence is short:
- November 13, 2026: The Honkai AI research team published the Gamer project page and paper.
- November 16 and 17, 2026: Public reporting detailed its training approach, demonstrations and performance results.
- At the time of those accounts: The inference engine remained listed as coming soon, without a scheduled release date.
How a virtual player could help test a game
HoYoverse identifies game testing as a central potential use. A quest can contain many linked steps: approach a character, trigger a conversation, travel to another area, win a fight and collect an item. A problem at any step can interrupt the player's progress.
Gamer's approach could let developers send an agent through those visible interactions repeatedly. Its ability to navigate, read the screen and use ordinary controls could support checks on whether objectives and menus work as intended. Repeating an action sequence could also help developers reproduce a problem before fixing it.
The research team lists accessibility assistance and research into interaction between people and computers among other possible applications. It also opposes using Gamer for cheating, unauthorized access, exploiting security weaknesses or harming users and fair play.
Those restrictions matter because the same ability to control a game automatically can serve very different purposes. The stated development goal is authorized research and testing, with human oversight and game rules shaping how any practical deployment would operate.
Why the work reaches beyond Honkai: Star Rail
HoYoverse also sees three dimensional game worlds as useful environments for research that may eventually inform robotics. Games offer sequences of observations, choices and consequences: an agent sees an obstacle, chooses a route, acts and checks what changed.
Gamer brings several pieces of that problem together in one system. Its screen observations supply perception, its instructions and summaries support planning, and its keyboard and mouse outputs provide control. The cross game training aims to make those skills useful beyond a single quest or title.
A physical robot would face additional demands, including real sensors, movement constraints and safety requirements. Gamer's immediate contribution is a way to study sustained action and memory in virtual environments where researchers can repeat tasks and measure outcomes.
The most useful picture of the project comes from its results together: much stronger quest performance after reinforcement learning, uneven puzzle success, and a large reduction in decision latency on expensive hardware. Those details explain both why a developer might want such a tester and why further work is needed before it becomes a routine tool.
Key Points
- HoYoverse's Gamer controls games through screenshots and keyboard and mouse commands.
- A reported 50 hour test achieved autonomous control for 95% of the running time.
- Two side quest objective scores reached 80.0% and 83.6% after reinforcement learning.
- Success averaged 49.7% across 40 minigame tasks, with large differences between puzzle types.
- Decision latency fell from 1,306 to 65.12 milliseconds in the reported H200 configuration.
- The project targets research and game testing. Trained model weights were not bundled, and the inference engine was listed as coming soon.






