A program that works out the meaning of newly coined words using the online encyclopaedia Wikipedia could help machines understand the slang used in blogs and other informal texts, say researchers.
The program 鈥� called Zeitgeist 鈥� hunts through Wikipedia looking for entries about new words that do not appear in an online resource called WordNet, an official linguistics tool that is both a dictionary and a thesaurus. WordNet is used by researchers to help computers understand human language. New words, or neologisms, that do not appear in WordNet inevitably leave computers stumped.
When Zeitgeist finds a Wikipedia entry about a new word, it looks at the links to and from the page, explains lead researcher Tony Veale from University College Dublin, Ireland. 鈥淚s there a pattern amongst those linkages that allows us to understand what the new word means?鈥� he asks.
Advertisement
For example, having found an entry for the word 鈥済astropub鈥� 鈥� a bar that specialises in food 鈥� Zeitgeist can work out the definition for itself thanks to the links to the entries for 鈥減ub鈥� and 鈥済astronomy鈥�. The program does not read the linked-to pages but relates their titles to entries in the WordNet database.
鈥淟ink diarrhoea鈥�
鈥淭he link structure reflects linkages between ideas,鈥� says Veale, 鈥渂ut people have a tendency to link everything 鈥� they get link diarrhoea.鈥� To prevent this from confusing the program, Zeitgeist ignores links that are not reciprocated. If the page a link points to does not link back to the neologism, it is discounted.
One of Zeitgeist鈥檚 limitations is that links sometimes point to an article that is not part of a neologism鈥檚 definition. For example, it understands 鈥渇eminazi鈥� 鈥� a word used to characterise a woman as man-hating 鈥� as being a combination of the words 鈥渇eminist鈥� and 鈥淣azi鈥� because of the links on the Wikipedia entry.
But feminazi is actually a term of abuse that has nothing to do with the Nazi doctrine of National Socialism. For that reason, Zeitgeist cannot be relied on to create a dictionary-style definition.
But this need not be a problem, says Veale. He thinks Zeitgeist鈥檚 approach is good enough to work out the sentiment of human writing. A link to the term Nazi should make it clear that a neologism carries a negative connotation, he says.
鈥淲e鈥檙e interested in a computer processing a text and having a way to understand the meaning and intention of words that are new to it,鈥� Veale explains. 鈥淭hat鈥檚 useful for applications from understanding emails to summarising news reports.鈥�
Fast-changing lingo
Many companies are interested in such technology to get a feel for what people are saying about their products on blogs and message boards. 鈥淭hey鈥檙e likely to have a lot of slang and neologisms,鈥� explains Veale. 鈥淭hese words emerge too fast to appear in dictionaries or resources like WordNet.鈥�
John Carrol, who develops systems that can understand human language at Sussex University, UK, agrees that Wikipedia is a good place to look for new words: 鈥淚t鈥檚 such a large and up-to-date resource, I think we鈥檒l see it used more for projects like this in the future,鈥� he says.
鈥淶eitgeist is a neat tool,鈥� adds Carrol. But he points out that its limitations mean it can handle only 75% of the neologisms it finds in Wikipedia. Another technique is to use the context of a new word to guess at its meaning, he says. Adding that ability to Zeitgeist could make it much more powerful.
Veale presented his work on Zeitgeist at the European Conference on Artificial Intelligence in Riva del Garda, Italy, last week.
