Artificial intelligence is increasingly used in business. But because of the way it is built, there is theoretical potential for the software to contain undetectable features that bypass its normal decision-making process, meaning it could be exploited by malicious third parties.
For instance, an AI model tasked with shortlisting CVs for a job vacancy could be made to covertly prioritise any which include a deliberately obscure phrase. Or an AI model used to approve or decline bank loans could be made to always hand out money, regardless of the likelihood of repayment, if the application requests an amount that ends with 37 pence.
at the Massachusetts Institute of Technology and his colleagues have demonstrated that it is not only possible to hide such malicious features in AI models, but that these 鈥渂ackdoors鈥� would be impossible to detect.
Advertisement
This potential problem begins with the fact that training AI models requires vast amounts of computer power, which most researchers and companies don鈥檛 have in-house. Consequently, specialist companies have sprung up that offer to train AI on a business鈥檚 behalf.
鈥淪ome of the stuff we do would be very tricky without having current access to high-powered computing of some sort,鈥� says , an AI researcher at the University of Sussex, UK, who wasn鈥檛 involved in the research. 鈥淵ou can鈥檛 really do big machine learning on normal workstations. The thing is going to be sitting there, fans screaming, for days and days.鈥�
Vaikuntanathan says this, in theory, could allow rogue staff within an AI training company to infiltrate AI models made for research or industry 鈥� although there is no evidence that this has actually occurred.
His team showed proof of concept for a variety of attacks which essentially teach an AI not only to do its job as intended, but also to look out for very specific signatures within data and perform differently if they are detected. Because AI models are black boxes whose operation can鈥檛 be understood, unlike software written by humans, it would be impossible to verify their behaviour for all possible inputs.
Vaikuntanathan says that the idea for the research came from cryptography, where backdoors have been an issue for decades.
He says that there is no obvious solution for entirely addressing this, except for training AI in-house using staff who are known to be trustworthy. But the researchers proposed several of what Vaikuntanathan calls 鈥渉alf-baked鈥� solutions.
For instance, they suggest that input data suspected of triggering a malicious part of the AI could be very slightly adjusted in the hope that it is no longer recognised by the backdoor, but is still close enough to the original data to make the AI arrive at a correct decision.
But Vaikuntanathan warns that attackers could simply begin to adapt to this, sparking a 鈥渃at and mouse game鈥� of backdoor detection and evasion.
Reference:
