Be careful what you type during Zoom meetings: a deep learning AI algorithm can identify the keys pressed on a keyboard with 93 per cent accuracy, based on the sounds of your keystrokes.
at Durham University, UK, and his colleagues trained the CoAtNet deep learning AI model, most commonly used to classify images, to 鈥渉ear鈥� which keystrokes correlated to the letters and numbers pressed on a keyboard by feeding it the sound waveforms created when each key was pressed.
When put to the test, the model picked up which keys were being pressed with 95 per cent accuracy when audio of the keystrokes was recorded on a phone 17 centimetres away from the laptop, falling slightly to 93 per cent accuracy from a recording of a Zoom call.
Advertisement
Harrison believes that Zoom鈥檚 noise suppression features, designed to tamp down background noise from calls, may account for the small difference in performance. Zoom didn鈥檛 respond to a request to comment on the researchers鈥� findings.
Because it is a deep learning model, which makes connections that aren鈥檛 always immediately clear to people, Harrison isn鈥檛 fully certain how it works, but believes it is identifying the difference in sound based on the parts of the keyboard that are used.
鈥淚f you think of a drum, if you hit different parts of the drum skin 鈥� whether it鈥檚 near the wall, whether it鈥檚 going to centre 鈥� it makes different sounds,鈥� he says. 鈥淪imilarly, with something like a laptop or a keyboard, the placement of the keys on that board could lead to the difference in the sounds that this model picks up.鈥�
This experiment only looked at one AI model run on one computer analysing one keyboard, but Harrison says the model could probably be made to work on other devices. 鈥淭he core takeaway is that this very high level of accuracy was achieved using completely open-source software [and] off the shelf devices,鈥� he says. 鈥淭his accuracy was best in class for this field of research.鈥� Advancements in AI since the experiment was conducted make it probable more recent AI models will be even more accurate, says Harrison.
鈥淚 would tend to take this seriously,鈥� says at De Montfort University in Leicester, UK, who was surprised by how accurate the model was. He believes the research will raise awareness of the risks of so-called side channel attacks, which harness data leaked inadvertently through a tool, though he isn鈥檛 sure that the researchers鈥� suggestion to avoid such an attack 鈥� that they take video calls in a room where no microphones are present 鈥� is practical.
arXiv
