Google’s SIMA: New AI Agent that Can Play Games With You

Mar 14, 2024 | 0 comments

Google's SIMA An agent that plays games
Image Credits: Google Deepmind

Google’s SIMA, short for Scalable, Instructable, Multiworld Agent, is an AI agent from Google Deepmind that understands and follows natural language instructions in 3D environments and video games. It has the ability to learn from various video games and 3D environments. It can play a game with you as a companion, and you can instruct the SIMA in natural language based on what you want it to do in the video game.

For example, instructing it to pick up leaves or driving a car in a video game.

SIMA is trained on a large and diverse dataset of gameplays; it doesn’t need access to the game’s source code or APIs. It only takes image and text instructions as input, just like humans do. To know more about the technical details, you can view the technical report here.

Google’s SIMA Tested on Various Games

Google has partnered with a lot of gaming studios to train SIMA so that it can interact with various 3D environments and games. Google’s SIMA has been tested across various popular games like No Man’s Sky, Teardown, Valheim, Goat Simulator 3, and more.
SIMA is evaluated on over 600 gaming skills, including navigation such as moving up, down, right, left, driving a car, object interaction such as chop down a tree, climbing ladders, crafting a helmet, and more.

SIMA currently can’t do advanced things that require high-level strategic planning and multiple subtasks, such as finding resources and building a camp. So it is getting trained on complex instructions.

Video Games played by Googles SIMA
Image Credits: Google Deepmind

Google’s SIMA is Generalized to Various Games

Unlike specialized AI agents that are developed for a particular game, Sima agents have the ability to generalize to various games. So when the agent faces a new game, it performs tasks based on what it has learned by playing other games.

For example, if SIMA agent plays a FPS game, it can easily play other FPS games, whereas specialized agents can only play that particular game.

“SIMA is trained to run it and do what it’s told; it’s not trained to win a game,” said Tim Harley, a co-lead for SIMA and researcher at Google DeepMind.

When Sima was faced with an unseen environment, it performed nearly as well as the specialized agent, which shows its great generalizing ability. 

Availability of Google’s SIMA

SIMA is not being used currently professionally as it is currently in the research phase, and it is being tested and trained.


Google’s SIMA opens up a new space for AI in video games where they can be used as companions rather than just powerful opponents. This also takes AI one step closer to humans, and in the future, it might collaborate with humans by interacting with real-world environments rather than just video games.

Read More

Figure Releases Figure 01’s New Status Update Video after Collaboration with OpenAI and it is Insane: Know Everything

Cognition AI’s Devin: An AI Software Engineer that Creates Websites and Apps with Simple Prompts


Elon Musk’s xAI Announces Grok 1.5 with Great Capabilities

Elon Musk’s xAI Announces Grok 1.5 with Great Capabilities

Image Credits: xAI Elon Musk's xAI launched Grok last year in November to compete with chatbots from big tech giants like Google, Microsoft, and OpenAI. Elon Musk's xAI is soon launching the next version of their chatbot, Grok 1.5, which performs really well as...

Meta’s Ray-Ban Smart Glasses Are Getting New AI Features

Meta’s Ray-Ban Smart Glasses Are Getting New AI Features

Image Credits: Meta Meta’s $300 smart glasses, made in collaboration with Ray-Ban, allow users to take pictures, record videos, make calls, hear music, and do much more. Now, new AI features are being added to Meta's Ray-Ban smart glasses.  New AI Features in Meta’s...

Claude 3 beats GPT-4 for the First Time on LMSYS Leaderboard

Claude 3 beats GPT-4 for the First Time on LMSYS Leaderboard

Anthropic released the Claude 3 model family earlier this month, and they have become highly popular since their release. Now Anthropic's Claude 3 Opus Model beats OpenAI's GPT-4 model for the first time on the LMSYS Chatbot Arena Leaderboard. LMSYS Chatbot Arena is a...


Submit a Comment

Your email address will not be published. Required fields are marked *