Someone typing on a physical keyboard while attending a virtual reality meeting could end up leaking passwords and other sensitive information to an attacker who uses artificial intelligence to interpret the avatar鈥檚 subtle hand movements.
Researchers trained AIs to analyse the hand movements of virtual avatars in Meta Horizon Workrooms, a virtual office platform provided by Meta. The AIs then used the hand movements to reconstruct the keystrokes typed by the avatar鈥檚 user over the course of brief typing sessions.
鈥淵ou don鈥檛 even need to know the person,鈥� says at the University of Chicago. 鈥淎s long as you can get into the same VR room with them, they鈥檙e done.鈥�
Advertisement
Zheng and her colleagues demonstrated several keyboard inference attacks in virtual reality, including some that accessed data from the target person鈥檚 VR headset. But the simplest method only requires the attacker to enter the same virtual space as the target and visually record the hand motions of the target鈥檚 avatar.
Because VR headset sensors cannot yet perfectly track finger motions, any physical keyboard typing ends up being interpreted as more subtle avatar hand motions. So the researchers trained separate AI models to recognise keystroke motions, identify individual fingertips that are pressing a key and finally to predict the keys being typed, says at the University of Chicago, who led the project. A paper describing this work has been accepted by the .
The AI-powered attack recognised between 86 and 98 per cent of typed keys for 13 out of 15 people who participated in a virtual reality experiment. But noisier avatar motion data made it more difficult for the AIs to detect keystrokes for the remaining two people.
The fact that this attack worked accurately on multiple users with different keyboard typing patterns is 鈥減articularly alarming鈥�, says at North Carolina State University. But he said the demonstration of the vulnerability 鈥渨ill help VR software vendors to develop privacy-preserving policies鈥�.
The researchers have already begun experimenting with adding some noise to obscure the VR hand motion data to foil this AI-powered attack. Meta did not respond to a request for comment.
