and have won the 2024 Turing award, which is often called the Nobel prize of computing, for their fundamental work on ideas in machine learning that later proved crucial to the success of artificial intelligence models such as Google DeepMind鈥檚 AlphaGo.
Barto, who is now retired and lives in Cape Cod, Massachusetts, didn鈥檛 even realise he was nominated for the award. 鈥淚 joined a Zoom with some people and was told and I was just flabbergasted,鈥� says Barto. 鈥淚 was totally surprised. I was totally unprepared, delighted at the honour, but I had no idea that this was coming.鈥�
The pair will share the $1 million prize for their work on reinforcement learning, in which an AI is 鈥渞ewarded鈥� and 鈥減unished鈥� through trial and error to achieve a goal. This has been studied since AI鈥檚 inception 鈥� for example, in 1948, Alan Turing first suggested a 鈥減leasure-pain system鈥� for intelligent machines, reminiscent of modern reinforcement learning systems, but until the 1980s received little attention.
Advertisement
Up to that point, research in machine learning was largely focused on symbolic AI, which involves manually teaching a computer the rules of how to learn. Barto and Sutton, who was then Barto鈥檚 student, began exploring algorithms and mathematical theories that could replicate Turing鈥檚 idea, using neural networks to let an AI work out these rules by itself, rather than the symbolic approach that had previously dominated.
鈥淲hen I started, it was very unfashionable. I didn鈥檛 care, because it was interesting to me,鈥� says Barto. 鈥淣ot only was it unfashionable, it was considered a dead end to look at neural networks. It鈥檚 really surprising and gratifying that it has gotten to the point where a lot of people are working in the area, improving the algorithms and doing applications, many of which are really very beneficial. I鈥檓 amazed and pleased to see this evolution.鈥�
鈥淭hey started the field [of reinforcement learning],鈥� says Chris Watkins at Royal Holloway, University of London. Some of their first reinforcement learning algorithms, such as policy gradient models, which provide a blueprint for AIs to choose their actions as their environment changes, and temporal difference learning, which compares predictions to how a situation unfolds, are still widely used today, says Watkins. For example, they have powered AI breakthroughs such as Google DeepMind鈥檚 AlphaGo and AlphaZero, along with advanced robotic systems such as OpenAI鈥檚 early work in solving a Rubik鈥檚 cube.
Barto and Sutton鈥檚 temporal difference algorithm, which was inspired by theories of how animals learned, also understand the dopamine reward system in the brain. In the 1990s, neuroscientists realised that neurons in monkey brains fired in response to unexpected rewards, and worked exactly like the predictions that were part of Barto and Sutton鈥檚 algorithms. 鈥淚t鈥檚 the best example of ideas moving back and forth between engineering and natural science ever,鈥� says Sutton.
Sutton hopes that current artificial intelligence research might take more inspiration from the natural world. 鈥淲e鈥檙e doing the obvious idea that an [AI] should learn from experience, just as animals learn from experience, and this is still neglected,鈥� says Sutton. 鈥淢odern AIs don鈥檛 learn from experience. They learn from a bunch of separate datasets collected by people鈥� Today, we still don鈥檛 have machines that will learn from their experience and form an understanding of the world. This is still the obvious thing that remains overlooked.鈥�
