ÐÓ°ÉÔ­´´

Where does our sense of grammar and syntax come from? From our parents, or from our genes? From our parents of course, said the behaviourists who dominated psychology and linguistics in the first half of this century. Just as laboratory rats could be conditioned to press a pedal for a food reward, so children were ‘conditioned’ by their parents to learn language. There was nothing innate about language. Newborn babies were the linguistic equivalents of clean slates. What could be simpler?

The American linguist Noam Chomsky had rather different ideas, and in the 1950s he launched his now famous assault on behaviourism. Despite their apparent variations, he argued, all languages use a similar underlying set of grammatical rules – rules so fundamental yet subtle that children cannot possibly acquire them simply by listening to their parents. Far from being clean slates at birth, says Chomsky, we are genetically predisposed to learn these grammatical rules. Crucial elements of language must be embedded in our genes.

These ideas rapidly became the new orthodoxy in linguistics (indeed, few branches of science have been so dominated by one person). But now, more than three decades later, Chomsky’s theory is under siege by a new generation of researchers. Armed with neural networks – computer simulations of the kinds of complex information processing believed to take place in the brain – these researchers are beginning to probe the assumptions behind Chomsky’s linguistic rules. And the message from some quarters is clear: the theory at the heart of modern linguistics may be flawed.

MAKING NEW CONNECTIONS

To understand the controversy, one must look at Chomsky’s ideas in more detail. His rules cover every aspect of language, from syntax (the ways in which nouns, verbs, and so on can be combined) to phonology (how different basic linguistic sounds or phonemes can be combined). For instance, the sentence ‘the man kicked the ball’ is intelligible because it fits the rule, stored in the brains of English speakers, that sentences have to consist of a noun phrase (‘the man’) followed by a verb phrase (‘kicked the ball’). Equally, the sentence ‘kicked the ball the man’ is meaningless because it flouts this rule. Chomsky’s theory also holds that we can understand sentences such as ‘Jane was seen by Thomas’ because of a language rule which transforms them from the passive form to the active form, in this case ‘Thomas saw Jane’.

Why have generations of researchers embraced these rules so readily? By far the most compelling reason is that rules explain the ‘productivity’ of language; in other words, our ability to generate and comprehend sentences we have never encountered before. As Chomsky frequently pointed out, a person equipped with a finite set of language rules can generate and understand an infinite number of sentences.

The challenge to all this comes from a school of thinking within neuroscience known as connectionism, which holds that mental functions such as learning and cognition are rooted in the way neurons interconnect and communicate in the brain. Connectionism has already revolutionised thinking about the relationship between mind and brain. Aided by its experimental ally, the computer-simulated neural network, it is now stirring debate in the world of linguistics.

The central idea is that the brain processes information using networks of neurons – nerve cells that ‘talk’ to each other by transmitting electrical impulses along ‘wires’ called axons. Although the brain is divided into areas with specific functions, each of these areas processes information in a broadly similar way. Vision might be handled in a different area of the cortex than spoken language, but both rely on neural networks. That, at least, is the theory. Anatomy would suggest that most of the brain’s networks are extremely dense. A typical neuron receives input signals from tens of thousands of other neurons, all of which may influence the level of its electrical activity, and hence the signals it passes on to other neurons.

REINFORCING THE MESSAGE

A key tenet of connectionism is that many, if not most, neural networks in the brain undergo subtle changes that correspond to learning. Current thinking in neurophysiology is that these changes occur at the synapses connecting neurons. By stimulating certain subsets of neurons more often than others, our experiences of the world may selectively strengthen certain synapses, making some patterns of electrical activity more likely than others. Such patterns, the argument runs, could form the basis of recall and learning (see ‘These cells were made for learning’, New ÐÓ°ÉÔ­´´ supplement, 21 November 1992).

Connectionists attempt to mimic the behaviour of the brain by constructing artificial and highly simplified networks of neurons that function in a manner similar to the real brain. The networks exist not as pieces of hardware, but rather as computer simulations. At present, these networks are only a minute fraction of the size of a real brain, and the properties of the artificial neurons are enormously simplified compared with real neurons. Nevertheless, artificial neural networks have begun to suggest radical alternatives to many psychological theories.

How can such an approach allow us to understand language processes? One answer can be found in a neural net-work created in the mid-1980s by David Rumelhart and James McClelland, two psychologists then at the University of California, San Diego. These researchers were interested in a small part of language. How, they asked, might a network, like a child, learn to produce the correct forms of past-tense verbs?

Every child learns, for instance, that the past tense of ‘walk’ is ‘walked’. The Chomskyan explanation assumes the existence of a set of rules which we unconsciously apply to the present tense to produce the past tense. All regular verbs, which make up the vast majority of all English verbs, conform to the simple rule that adds ‘-ed’ to the root of the verb, whereas the 180 or so irregular English verbs obey what Chomsky defines as ‘exception’ rules (for example, ‘go’ becomes ‘went’, ‘am’ becomes ‘was’, ‘have’ becomes ‘had’ and ‘send’ becomes ‘sent’).

This division of verbs into regular verbs, dealt with by the general rule, and irregular verbs, covered by the exception rules, is absolutely central to Chomsky’s theory. Without these two sets of rules, children would be powerless to generate past tense verbs. Or would they? On the face of it, Rumelhart and McClelland’s neural network suggests not. The network can produce the correct past tense forms of both regular and irregular verbs without the use of such rules, and without needing to treat regular and irregular verbs differently.

What makes this feat all the more remarkable is the relative simplicity of the network . It consists of a layer of 460 ‘input’ units connected in parallel to a layer of 460 ‘output’ units. Each of these units, the network’s neurons, can be switched on or off, and their patterns of activity act as representations of the phonological structures of verbs. Verb roots can be programmed into the input units, which then transmit signals, via the connections, to the output units, where the past tense forms are encoded. Each output unit receives signals from many input units, some of which will stimulate it while others inhibit it. The combined effect of all these inputs determines whether an output unit is switched on or off.

In what sense can such a device learn grammar? Like most neural networks, it has an in-built capacity to adjust the strengths of its connections. And as Rumelhart and McClelland discovered, this means the network can be trained to produce past tense forms of verbs. The training involves showing it the correct present and past tense forms of a small number of example verbs. When it is presented with a word, the network attempts to construct the correct past tense form. The discrepancy between the network’s output and the correct form provides the basis for training the network. The weights on its connections are changed in line with this discrepancy by a learning algorithm called ‘back-propagation of error’.

This algorithm has two stages. First, an example verb is programmed into the input units, and signals spread from these units to the output units. Secondly, the resulting past tense verb produced by the output units is compared with the correct form supplied by a ‘teacher’. The weights are changed in such a way as to ensure that next time the input pattern is presented, the output will be closer to being correct.

McClelland and Rumelhart argue that this teaching process mimics the feedback children receive as they attempt to learn the past tense. By this, the researchers are not implying that children receive formal tutoring on the past tense. Indeed, observation would suggest that parents seldom correct the grammatical mistakes of their young children. Instead the idea is that as children listen to speech they use their current grammatical knowledge to make predictions about the kind of sentences they are likely to hear in future. If these predictions prove to be incorrect, the mismatch between what the children expected (for example, ‘goed’) and what they heard (‘went’) provides the feedback they need to learn. In keeping with a child’s early verbal experiences, the researchers first trained the network on 10 common verbs, like ‘give’ and ‘have’, and then showed it a larger set consisting of 420 verbs, including many less common ones such as ‘catch’. Of the 10 common verbs, 8 were irregular, while in the larger set of 420, only 82 were irregular, reflecting the fact that many of the most common English verbs are irregular.

The problem for Chomsky’s theory is that the network does not use linguistic rules. In learning verb forms, its connections are simply weighted according to the correlations it detects between input and output verbs. For example, the correlation it detects between the endings ‘-ow’ and ‘-ew’ (‘throw’ becomes ‘threw’, ‘blow’ becomes ‘blew’, ‘grow’ becomes ‘grew’) ensures that the input units representing ‘-ow’ become strongly connected with the output units representing ‘-ew’.

Yet despite this absence of rules, and despite treating regular and irregular verbs identically, the network can, after a long training period during which each word is presented dozens of times, produce virtually errorless pronunciation of the past tense forms. In one test, it produced the correct past tense forms of 90 per cent of a batch of 86 unfamiliar regular verbs – a powerful demonstration, say connectionists, of the network’s productivity. A more serious threat to Chomskyan orthodoxy comes from its deft handling of unfamiliar irregular verbs. After training, it can produce ‘wept’ for ‘weep’, ‘clung’ for ‘cling’, and ‘bid’ for ‘bid’.

The network also displays some human-like quirks. After intermediate levels of training, for instance, it makes errors producing the past tense of irregular verbs (for example, ‘go’ becomes ‘goed’) that it has previously handled correctly (‘go’ becomes ‘went’). Similar, seemingly counter-intuitive errors are sometimes made by children aged three to eight years, who occasionally produce incorrect past tenses such as ‘comed’, ‘wented’, ‘goed’ despite having previously produced the correct forms. It is not hard to see why the network should mimic this behaviour. When the training set is expanded from 10 to 420 verbs, the ratio of irregular to regular verbs shrinks sharply. As a result most verbs conform to the ‘add -ed’ rule, and the network induces, incorrectly, that all verbs behave this way. Only after further training does it again learn to produce the correct irregular forms.

Yet for all the network’s talents, not everyone is convinced by Rumelhart and McClelland’s findings. Critics such as Steven Pinker and his colleagues at the Massachusetts Institute of Technology maintain that a rule-based system is still needed to explain certain aspects of past-tense learning. They have proposed a ‘hybrid’ learning model in which neural networks merely store information about irregular verbs while regular verbs are dealt with by separate rules. The rules automatically come into play unless a verb is first recognised by the neural network as irregular.

Pinker prefers such a hybrid system for several reasons. First, he says, the neural network model is inadequate because it ignores the meaning of verbs. For example, the past tense of the verb ‘lie’ as in ‘tell a falsehood’ is ‘lied’, but for ‘lie’ in the ‘recline’ sense the past tense is ‘lay’. The model, however, would treat these two as identical because the phonology is the same in both cases.

Secondly, Pinker points to a variety of evidence suggesting that people treat regular and irregular verbs differently, as in the hybrid system. For instance, Pinker has discovered that the frequency with which a verb occurs in a language affects the time it takes us to generate its past tense – but only if the verb is irregular. Thus, we tend to produce the past tenses of common irregular verbs, like ‘go’, faster than those of rare verbs such as ‘shrink’; but no such trend is discernible with regular verbs. Connectionists are now investigating more sophisticated neural networks to see if Pinker’s findings can be mimicked without resorting to linguistic rules.

INNATE KNOWLEDGE

Success on this front would certainly strengthen the case against Chomsky’s view that linguistic knowledge is stored in the brain as a set of rules. But the assault on orthodoxy would not necessarily stop there. For connectionists are also beginning to question Chomsky’s additional claim that the reason we learn these rules is because we are genetically predisposed to do so – in other words, crucial elements of linguistic knowledge are innate. The debate is the latest manifestation of the age-old dispute between empiricists, who argue that all knowledge is learned, and rationalists, who believe that at least some of our knowledge is already in place at birth.

Chomsky’s arguments for the innateness of language make a powerful case for rationalism. Briefly, the case rests on the observation that although children are only exposed to a limited number of sentences, they acquire the capacity to produce an infinite number of grammatically correct sentences. As a psychologist would say, there is a poverty of stimulus. So how does a child learn the true grammar of the language, as opposed to one of the infinite number of incorrect grammars that could have generated the sentences they were exposed to?

One example of the problem is known as Baker’s paradox after the linguist Lee Baker. Children learn that many verbs can be turned into passives: ‘John hit Fred’ can be transformed into ‘Fred was hit by John’. It would seem reasonable for a child to generalise from such examples and assume that the rule for passives applies to all verbs. This, of course, would generate ungrammatical sentences: ‘John resembles Fred’ cannot be transformed into ‘Fred is resembled by John’. Yet there is no obvious reason why children should not learn this incorrect general rule. If they can generalise about some linguistic constructions, why not all?

Chomsky’s answer to this puzzle is to assume that newborn infants come equipped with certain innate mechanisms that predispose them to learn the correct grammar. In other words, that there is a genetic component to language acquisition. The best way of proving this would be to find people with inherited language disorders, which is why many linguists are so excited by the results of recent research into a large London family. According to psychological tests by Myrna Gopnik of McGill University in Montreal, 16 out of 30 members of the family have a very specific linguistic problem: although they can produce and comprehend language with ease, they make errors such as ‘we comes’ and ‘when they play get points’.

LANGUAGE LABORATORY

The pattern of inheritance of this linguistic disorder suggests that it may be linked to a single gene. Geneticists have yet to identify this gene, though, and with the research still at an early stage, most researchers are reluctant to draw firm conclusions. Indeed, so great is the potential significance of this finding that many linguists will need persuading that the inherited impairment really is confined to language, and is not simply part of a more general cognitive disorder.

Chief among the sceptics will be the disciples of connectionism, who posit that children learn the correct grammar and syntax of a language from scratch, and do so simply by unconsciously adjusting connections within networks of neurons in their brains. This view has recently received some support from a neural network developed by Jeffrey Elman, another linguistics researcher at the University of California, San Diego.

Elman has constructed a network with two special talents: it can run continuously and can learn a rudimentary form of syntax from scratch. Rumelhart and McClelland’s network simply detects correlations between input and output patterns. Elman’s network, however, comprising 212 units, can learn relationships between successive input patterns. It ‘sees’ a continuous sequence of words, separated by brief breaks, which form grammatical sentences such as ‘John feeds dogs’. As each word is presented, the network attempts to predict as its output what the next word will be. It can ‘remember’ the previous words of a sentence because its internal representations of those words – units that are switched on in particular patterns – are continuously recirculated. This recirculation of ‘old’ words influences the network’s interpretation of any ‘new’ word that is fed into its input.

In one experiment, Elman presented the network with a stream of 40 000 sentences varying in length from 3 to 16 words, and constructed from 23 words such as ‘boy’, ‘live’, ‘Mary’, ‘chases’, ‘who’ and so on. At each point the network tried to predict the successor word. This would be virtually impossible in a full language, but the network could at least guess whether the next word would be, say, a noun or a verb, and then choose one from the limited selection available. Having done this, it could then change the weightings of its connections to improve the likely match between the predicted and correct word.

As training proceeded, the network got progressively better at predicting the category of the next word, and by the end of the training phase, it was performing well. After the sequence ‘Boys who Mary . . . ,’ for instance, the network predicted a singular verb such as ‘chases’ or ‘feeds’, indicating that it has learnt the relationship between singular nouns (Mary) and singular verbs (chases).

Equally impressively, the network learnt fine grammatical distinctions. When Elman examined the patterns of internal activity in the network that represent different words, he found that they clustered into grammatical categories such as nouns and verbs. What’s more, the verbs were subdivided into transitive and intransitive categories. Yet the network had received no direct information about these categories. Nor had it acquired this sense of grammar through any in-built linguistic constraints. It was simply able to inductively derive the categories from the statistics of the input.

This is not to say that Elman’s research undermines Chomsky’s claims about the poverty of the stimulus. It is still very hard to see how the limited linguistic information a child receives can generate the full, ‘correct’ grammar of English that linguists define. Rather, what Elman’s network shows is that instead of being based on rules, a child’s grammatical knowledge may consist merely of statistical information about the relationships between different sorts of words. Such information could be extracted from a limited set of example sentences, and without the need of any innate linguistic constraints.

The debate between Chomsky and his followers and researchers using neural networks is merely in its infancy. Doubtless it will become far more refined as more sophisticated and realistic neural networks are constructed. But already connectionism is forcing linguists to rethink some of their ideas about how we acquire language.

David Shanks is a lecturer in psychology in the Department of Psychology at University College London.

* * *

Ins and outs of neural networks

An artificial neural network consists of ‘neurons’, or ‘units’, linked by adjustable connections. Just as the synapses that link real neurons can vary in strength, so the weighting on each of these connections can vary. And just as real nervous systems learn by adjusting the strengths of their synapses, so artificial neural networks ‘learn’ by adjusting the weightings on their connections. The strength of the signal transmitted through a given connection depends on the weight of that connection.

The units of most neural networks can be divided into three types: ‘input’, ‘output’ and ‘hidden’. Typically, a layer of input units will send signals to a layer of output units. The hidden units act between these layers, improving the computational power of the network. When some input units are turned on, a pattern of activation spreads through the network. Each output unit sums the signals it receives and switches on if the activation exceeds a threshold.

The goal of all networks is to learn how to ‘map’ one pattern of activity, programmed into the input units, into another, produced by the output units. This might correspond to transforming the present tense of a verb to the past tense, written words to spoken, or patterns of electrical activity on a retina to recognisable digits.

A simple example is shown below (network A). When the network’s two input units are given the patterns ‘0,0’ and ‘1,1’, the network’s output unit responds with ‘0’; whereas the input patterns ‘0,1’ and ‘1,0’ produce the output response ‘1’. The network has ‘learned’ these mappings by adjusting the weightings (numbers) on its connections. Numbers inside units represent the thresholds that have to be exceeded before the unit will send an output signal.

A more complex example is a network that functions like a simple vision system, ‘recognising’ handwritten digits or letters. Geoffrey Hinton and his colleagues at the University of Toronto constructed such a network using 256 input units, 9 hidden units and 10 output units. When a pattern of activity representing a handwritten digit was presented to the input layer – the ‘retina’ – the network responded by activating one of its 10 output units, each of which represented a digit from 0 to 9. After prolonged training, the network was able to identify handwritten digits.

The power of this type of network cannot be overemphasised. Hal White and his colleagues from the University of California, San Diego, have proved that provided it has sufficient hidden units, a network can be trained to perform any mapping operation.

A more elaborate type of mapping occurs in a ‘recurrent’ network. Here, activation patterns formed in the hidden units are recirculated through the network (see network B, below). A particular input pattern will generate a ‘hidden pattern’ which is then sent back to the input layer to coincide with the next input. In this way, the network is able to ‘remember’ what the previous input pattern was, and hence can learn relationships between input patterns. This is vital for some tasks such as learning the grammar of a language: the grammar specifies that certain sequences of words are allowed and others are not.