Google DeepMind Unveils AI Capable of Executing Commands in Unfamiliar 3D Games

Google DeepMind introduces AI capable of performing various tasks in unfamiliar 3D games.

[{"selector":"#anim-be99811c-01b2-4e0e-ada8-a95275fb305e","keyframes":{"transform":["translate3d(-116.82848%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-d557db35-5e79-41d7-95f9-917d3ad5ae7c","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-f6e98f31-f507-4ad8-87cd-0e5f0fe23384","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

The AI can understand natural-language instructions and navigate through different gaming environments.

[{"selector":"#anim-139cf3b2-0b46-4905-a1e4-5aa953e92a99","keyframes":{"transform":["translate3d(115.28663%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-d8fd6e3e-eecc-4b7e-a324-0029b4ed1b13","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-feef4abb-d93d-4181-8833-a25033e670ac","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Collaboration with game studios helped train the Scalable Instructable Multiworld Agent (SIMA) on nine games.

[{"selector":"#anim-1c13c8e9-c083-42b7-b348-4ab59880cc53","keyframes":{"transform":["translate3d(-115.92356%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-743cebe1-e656-44ab-bf97-696f2874fe72","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-4116cb12-83ea-40b1-a3e2-2d48abda2cbe","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

SIMA learns from diverse environments, including those created in Unity, without needing game source code.

[{"selector":"#anim-65f80c85-aac4-4577-9709-8e1ac843f88d","keyframes":{"transform":["translate3d(121.95123%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-942c95bf-2ea9-48f3-8eca-7d13b6ba6428","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-bbd4d5b5-5c49-40e9-b540-93bb2484dd2d","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

It operates with only on-screen visuals and user instructions, using keyboard and mouse inputs like a human.

[{"selector":"#anim-ac3a56c6-81bf-4d32-9a5d-4d2094265278","keyframes":{"transform":["translate3d(-118.21191%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-b3b02051-a9c3-474b-898e-ecebfcce7ea9","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-7ef7970e-b87a-4a99-abad-ad97f32f2af8","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

SIMA undergoes evaluation on hundreds of basic skills across categories like navigation and object interaction.

[{"selector":"#anim-e018c5be-8cfd-492d-b542-b6e8c3be498d","keyframes":{"transform":["translate3d(123.04965%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-63ff6b4a-c010-4752-b860-4576312f0feb","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-1f28c5d2-63a3-4eb4-bedf-622a8b2f275d","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Performance evaluations show SIMA trained on multiple games outperforms those trained on single titles.

[{"selector":"#anim-af66a990-c574-4475-8fd7-0db795ca360c","keyframes":{"transform":["translate3d(-122.53521%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-e352c179-429b-4b56-bd30-5571b0c8a38c","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-40307801-40c4-4104-adad-ff7b990f5bc8","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

SIMA demonstrates the ability to generalize its skills to new environments beyond its training.

[{"selector":"#anim-014733a7-7de2-4148-8363-46c106c22c76","keyframes":{"transform":["translate3d(114.87342%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-dbe9a2c6-f1c0-4ebe-9e07-d038e6fdb299","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-4aca0dff-161f-46d4-8f0d-850dde8551e4","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Language input is essential for SIMA's successful performance, as observed in tests without language training.

[{"selector":"#anim-536f1bd0-1fa8-401c-ac42-6245e9931627","keyframes":{"transform":["translate3d(-126.2963%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-b4c62c30-9472-4aef-ac35-a0ac369bd538","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-e8008a5c-7e4d-4424-aa2d-3b971db16a16","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

DeepMind emphasizes ongoing research to enhance SIMA's capabilities and develop more general AI agents for various tasks.

[{"selector":"#anim-9c6b4585-326e-4fff-a8db-6839fa53a132","keyframes":{"transform":["translate3d(-113.58024%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-855aa89d-8f78-4e3b-8cbb-2b28d0a75338","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-e7af0545-5b61-43b2-80c0-df474adc6b7a","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

White Frame Corner Arrow Arrow Learn more