An artificial intelligence can adjust video footage in real time to make people appear to lip sync to any sound.
Prajwal Renukanand at the International Institute of Information Technology in India, and his colleagues have developed an algorithm that can rapidly alter video footage of a person talking or singing.
The researchers believe the AI could be used to alter videos so that the footage appears to match the sound of different languages, such as in dubbed movies.
Advertisement
The team trained the lip-syncing algorithm on short video clips, tasking it with marking out the lip shapes of people as they talked.
The algorithm is a generative adversarial network (GAN) made of competing AIs. For a given sound and video footage of someone speaking, one AI ā the generator ā was required to adjust the imagery of the personās lips to match the words being spoken.
Another two AIs, the discriminators, had to distinguish whether the footage was real or fake.Ā One discriminator focused on whether the mouth shapes were realistic, penalising the generator if there was a distinct mismatch between sound and lip movements. The second one provided a visual quality check, noting any glitches or unnatural visual artefacts around the mouth.
The generator became more skilled at producing realistic footage over several rounds, until eventually the discriminators could no longer tell the difference between real and fake.
āTo lip sync a one-minute video it would take around two minutes,ā says Renukanand. Most of that time required is for the algorithm to detect a personās face in the video, while the lip syncing component occurs in real time.
Because the algorithm was trained on human faces, it is currently more accurate on video footage of people than on computer-generated characters, says Renukanand.Ā It also works better if the sound provided is a real personās voice, rather than computer-generated speech.
The researchers hope to use the algorithm to use dub video content in different languages, and also in future to increase the ease of syncing CGI characters with the sound of voice actors.
Reference:
