Microsoft researchers have claimed that OpenAI鈥檚 GPT-4 is capable of such a wide variety of tasks, in some cases exceeding human ability, that the artificial intelligence model is showing 鈥渟parks鈥� of artificial general intelligence (AGI). Is this long-awaited goal near or are these claims just hype?
While no standard definition of AGI exists, it is most commonly understood to refer to an AI that can understand and learn any intellectual task that humans are capable of.
In a new paper, a team of researchers from Microsoft Research say that they consider AGI to involve the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas and learn from experience 鈥� and that GPT-4 meets that criteria, despite only being designed to generate text.
Advertisement
鈥淕iven the breadth and depth of GPT-4鈥檚 capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system,鈥� they say.
GPT-4 launched to the public earlier this month, but the Microsoft team devised a series of tests to evaluate a production version of the model before release. The researchers say the tests are based on ones from the field of psychology, which they admit are 鈥渟omewhat subjective and informal鈥� and may not stand up to scientific scrutiny.
Other researchers previously designed a suite of tests called BIG-bench, which is designed as a more rigorous assessment of an AI鈥檚 abilities. Using a subset of these tests, the score for OpenAI鈥檚 earlier AI model, GPT-3, was about half that achieved by people on average.
Microsoft鈥檚 new approach involved a variety of tasks. In one, the team asked GPT-4 to create a mathematical proof that there are infinitely many prime numbers, in rhyme. It correctly did so, in rhyming couplets. In other tests, GPT-4 scored around 80 per cent on the multiple choice components of the US Medical Licensing Exam, and 70 per cent on the Multistate Bar Exam, taken by lawyers in the US. When taking a software engineering test given to job candidates at Amazon, it scored 100 per cent, using less than 4 minutes of the allotted 2 hours.
It is hard to draw conclusions on the performance of GPT-4 from these tests alone. One unknown is whether the answers to these tests were already in its training data. A standard approach to creating and then testing an AI model is to use a large portion of available data to train it, but hold back a certain percentage in order to eventually test it on information it has never seen before. Critics have pointed out that OpenAI has provided minimal detail on what data was used to train GPT-4, making a truly independent analysis of its abilities difficult.
OpenAI itself has said that GPT-4 delivers 鈥渉uman-level performance鈥� on a wide range of exams and tests, but also warns that it is still prone to 鈥渉allucination鈥� 鈥� the phenomenon where an AI will produce convincing statements in response to prompts that are actually inaccurate or totally false. Indeed, Microsoft鈥檚 early attempts to have resulted in .
The paper鈥檚 authors weren鈥檛 available for interview before publication, but a Microsoft spokesperson says that the company 鈥� which owns a significant stake in OpenAI 鈥� is 鈥渘ot focused on trying to achieve AGI鈥�.
鈥淥ur development of AI is centered on amplifying, augmenting and assisting human productivity and capability.鈥疻e are creating platforms and tools that, rather than acting as a substitute for human effort, can help humans with cognitive work,鈥� says the spokesperson.
at Imperial College London says that the inner workings of GPT-4 are very different to the human brain, but that the results in the paper show that each can achieve intelligence in their own way 鈥� even if AI is, at its core, learning to mimic what humans have done in the past.
鈥淲e could define AGI as being able to solve problems using language and it is doing as well as an average human,鈥� he says. 鈥淚n terms of language-based reasoning, I can鈥檛 think of many things that [GPT-4] would do worse than the average human on. You can have apparent intelligence without being intelligent in the same way as humans.鈥�
But some experts doubt that true AGI will ever arrive or that the admittedly impressive abilities revealed in the paper are signs of true intelligence. at the Alan Turing Institute in London says we will see AI become increasingly capable and appear ever more intelligent, but there will never be true understanding or real intentionality.
鈥淭here are lots of people who really, really believe that AGI is coming. I don鈥檛 share that view,鈥� she says. 鈥淎I will always be programs that do what they鈥檙e programmed to do. What they鈥檙e programmed to do is to mimic human language or outputs, and they鈥檙e getting increasingly good at it. So it becomes increasingly convincing.鈥�
arXiv
