CATEGORISING video on the internet can be frustrating, especially if you happen to be a search engine. The pictures are incomprehensible and the supposedly descriptive text added by humans is confusing. It鈥檚 a notoriously difficult problem, but one that might be cracked by a new system that learns how to add keyword tags to videos by watching YouTube.
Before video-recognition software is put to work it usually has to be 鈥渢rained鈥� using a library of sample clips. In the past this has involved annotating the library clips by hand 鈥� a laborious process that constrains the range of material that can be used.
Now a team led by Adrian Ulges at the University of Kaiserslautern in Germany has built a program called TubeTagger that eliminates the need for human input, and could in principle be trained using any source of video. When given a keyword such as 鈥渟occer鈥�, TubeTagger automatically downloads 50 YouTube videos that human users have labelled with that tag, and examines the colour and motion content of each.
Advertisement
In the team鈥檚 experiments, this learning process was repeated for 22 different tags on YouTube, ranging from the names of common sports to words like 鈥渞iot鈥� or 鈥渋nterview鈥�. After its training, TubeTagger was shown a set of YouTube videos it had never seen before. Each time it watched a video, the system came up with three possible tags for it from the 22 it had learned, assigning a confidence level to each tag offered.
TubeTagger chose the most appropriate tag 37 per cent of the time, but the success rate was lower when the system was tested on videos from other sources. It also varied considerably depending on the tag, doing well with 鈥渟occer鈥� but poorly with 鈥渂each鈥�. Ulges says these initial results prove the concept and suggests that TubeTagger鈥檚 ability to learn from 鈥渞eal-world鈥� video data makes it more scalable than rival systems.
But Alexander Hauptmann of Carnegie Mellon University in Pittsburgh is sceptical that the system will be broadly applicable. Though researchers have been coming up with video-tagging systems since 2001, 鈥渢hey never get much better than 20 per cent above the baseline of a random guess鈥�, he says. 鈥淭his is a really difficult problem for all kinds of reasons.鈥�
For instance, recognising people in videos is difficult because they can appear close up or far away, with different haircuts, under different lighting, or maybe just a portion of them will appear on the screen. 鈥淎ll of these things signify human beings 鈥� the number of variants is mind-boggling,鈥� Hauptmann says. 鈥淚t鈥檚 not amenable to any one approach.鈥�
Ulges admits the system is still in its formative stages: human users happily use any of tens of thousands of words as tags, well beyond TubeTagger鈥檚 current capabilities. 鈥淚t鈥檚 very good at 鈥榮occer鈥� on YouTube because there鈥檚 usually a lot of green in the shots,鈥� he says. He plans to add audio-recognition software and the ability to locate text displayed within the video.
Ulges presented the work this week at the International Conference for Computer Vision Systems in Santorini, Greece.
