A STATISTICAL method that picks out the most significant words in a book could help scholars decode ancient texts like the 鈥� or even messages from aliens.
Humans find it easy to identify the words that capture the theme of a text 鈥� for example, that 鈥渨hale鈥� is a key word in Moby Dick 鈥� but this is a difficult task for computers. Now , a systems biologist at the University of Manchester, UK, and colleagues have developed a method to identify word importance based on a branch of mathematics called information theory. 鈥淚t seems that what we call semantics or meaning has a signature at the level of the statistics of words,鈥� says Montemurro.
Simply counting the frequency of words in a text is not enough, as connective words such as 鈥渇or鈥� and 鈥渢he鈥� confuse the picture. Important words tend to clump in paragraphs and chapters that deal with the topic they relate to, but this only provides a crude guide, says Montemurro.
Advertisement
鈥淚mportant words tend to clump in paragraphs and chapters that deal with the topic they relate to鈥�
For a more detailed analysis, the team calculated the 鈥渆ntropy鈥� of each word, a measure of how evenly distributed it is, in both the original text and in a scrambled version in which the words appeared in a random stream. From the difference between the two entropies multiplied by the frequency of the word, the team generated that word鈥檚 鈥渋nformation value鈥� in the text.
Connective words are fairly uniformly distributed in both the scrambled text and the original, so their information value is low. Significant words have a high value, because they tend to clump in the original and are relatively common. When the team applied the technique to On the Origin of Species, the top 10 words included: species, varieties, hybrids, forms, islands, selection and genera ().
Similar methods could have applications in biology, perhaps to identify genes that carry 鈥渧alue鈥�. 鈥淭hat鈥檚 the place where the most direct application of this stuff is,鈥� says at Rockefeller University in New York. 鈥淲hen you鈥檙e looking at the genome, it鈥檚 really an alien language.鈥�
