A system that tracks your eye movements could help make video calls truer to life.
at the University of California, San Diego (UCSD), was frustrated by the inability to smoothly teach an online music class during the coronavirus pandemic. 鈥淲ith the online setting, we miss a lot of these little non-verbal body gestures and communications,鈥� he says.
With , a colleague at UCSD, he developed a machine learning system that monitors a presenter鈥檚 eye movements to track who they are looking at on screen, then lets participants know when they are drawing attention.
Advertisement
The software uses several neural networks. One takes a snapshot of a videoconferencing system鈥檚 screen and notes the location of each participant鈥檚 video window and their name. Another takes the call leader鈥檚 camera feed and locates their face and eye position. When their eyes move, the second neural network tries to estimate where on the screen they are looking 鈥� and, by extension, who they are looking at.
The system then cross-checks that with the first neural network to see who is in that position on the screen and displays their name to all participants.
Dubnov and Greer trained the gaze-estimating algorithm by dividing the screen into 91 squares and asking people to look at them in turn. The algorithm could then use this training data to estimate where they were looking during video calls, managing to get within 2 centimetres of the correct point on a 70-by-39-centimetre screen. 鈥淚n principle, the system should work well on small screens, given enough quality data,鈥� says Greer.
The system becomes less accurate the further away a presenter sits from the screen, as their eyes appear smaller. Dubnov hopes to improve this so that conductors can stand in front of their orchestra, even when rehearsing remotely.
搁别蹿别谤别苍肠别:听
