Human scientists and engineers can still outperform artificial intelligence models in a game that mimics the process of scientific discovery. But this simulation could ultimately help researchers develop AI agents that can outcompete humans.
AI models are developing a reputation for science discovery 鈥� they can, for instance, predict how protein molecules will interact 鈥� but they still perform best when trained to solve a particular type of problem. The game-like DiscoveryWorld simulator, developed by at the Allen Institute for Artificial Intelligence in Washington state and his colleagues, is designed to test whether AIs can operate more like scientists: developing their own hypotheses for solving a problem, performing experiments, analysing the results and refining ideas.
鈥淭he idea here is: can we make a virtual world that you can test AI abilities to do general scientific discovery very cheaply, so that you don鈥檛 have to spend a million dollars and take three years to do an experiment?鈥� says Jansen.
Advertisement
Jansen and his colleagues designed DiscoveryWorld around eight research themes: archaeology, chemistry, physics, plant biology, epidemiology, rocket science, language translation and proteomics, or the study of proteins. They also created three difficulty levels with 鈥渆asy鈥�, 鈥渘ormal鈥� and 鈥渃hallenge鈥� tasks that represent gamified versions of real scientific challenges.
Next, they assembled three AI agents, all based on OpenAI鈥檚 , that use different techniques to solve problems within DiscoveryWorld. The first agent, 鈥淩eAct鈥�, generates thoughts and actions at every step while incorporating new observations. A second 鈥淧lan+Exec鈥� agent first comes up with an overall plan before following the same approach as the ReAct agent. And a third 鈥淗ypothesizer鈥� agent has a working memory to track its main hypothesis and ongoing measurements. The team also recruited 11 researchers, who had either a master鈥檚 degree or a PhD in a natural science, to tackle the same DiscoveryWorld tasks as the AI agents.
The AI agents solved less than 20 per cent of the normal and challenge-level tasks, whereas the human scientists and engineers solved about 66 per cent of the same tasks on average. The research was accepted for presentation at the conference being held in Vancouver, Canada, in December, and an early version of it has also been uploaded to a preprint server.
鈥淚t鈥檚 extremely difficult to capture enough of the complexity of the real world in a simulator such that it would be a good way to evaluate an AI scientist,鈥� says at the University of British Columbia in Canada. 鈥淚 thus applaud the authors for the very ambitious goal of trying to do that to some extent.鈥�
But proposals for AI scientists often 鈥渕ischaracterise the actual work of doing science鈥� by focusing on individual scientific discovery and neglecting how the process is typically a team effort, said at Princeton University and at Yale University in an email. Such efforts also risk restricting the scope of research to 鈥渜uestions suited to AI鈥檚 strengths and limiting the diversity of perspectives that are vital for robust and innovative science鈥�, they said.
arXiv
