An artificial intelligence model can be made to spout gibberish if a single one of the many billions of numbers that compose it is altered.
Large language models (LLMs) like the one behind OpenAIâs ChatGPT contain billions of parameters or weights, which are the numerical values used to represent each âneuronâ of their neural network. These are what get tuned and tweaked during training so the AI can learn abilities such as generating text. Input is passed through these weights, which determine the most statistically likely output.
The increasing ability of AI models in recent years has been heavily tied to increasing scale: adding more parameters and using more data to train the AIs has boosted performance. But larger models need more computational power to train and run. So a process called pruning is increasingly carried out to assess which neurons have the least effect on the output, and remove them.
Advertisement
Now, at Apple and his colleagues have found that although many neurons have a minimal effect and can be removed, some, termed âsuper weightsâ, have a disproportionate influence and are essential. In fact, pruning a single parameter among billions can destroy an LLMâs ability to generate text.
In experiments on Metaâs Llama-7B model, which has 7 billion parameters, pruning a specific super weight completely destroys the modelâs ability to generate text, âresulting in qualitatively and quantitatively gibberish responsesâ, the researchers write.
Conversely, pruning 7000 other weights found to have the greatest impact on the output â but keeping the one with the most impact of all â affects the ability of the AI model to correctly predict the next word in portions of text by no more than a few percentage points. In essence, the super weight was more important to performance than the next 7000 most important parameters combined.
at Imperial College London says it makes sense that a single weight could have a large impact on output, given that its impact could be carried through the model â effectively copying and compounding errors. âThe residual connections copy the early network layerâs outputs forward,â says Li. âIf the original early layerâs outputs are big, and you destroy them, then the entire model is messed up.â
, co-founder of , a machine learning platform and community, says evidence of single neurons in large models having a disproportionate impact has emerged before. He highlights an OpenAI LLM created in 2017 that was trained on 82 million Amazon reviews and taught to determine the sentiment â positive or negative â of a piece of text. The model had a single ââ, the value of which very accurately determined the answer. âIt can be seen as a bit counterintuitive,â says Wolf.
The findings shed light on the often mysterious inner workings of neural networks and how we can influence them. at University College London says that one solution to this over-reliance on a single weight could be a training technique called dropout. In this, random weights in a model are set to zero â effectively deleting them â and the network is forced to learn other ways to recognise and respond to its inputs.
Dropouts were used to improve generality of AI models by stopping them becoming overly reliant on a certain path through the neural network. The technique has fallen out of favour as models have grown so large and absorbed so much data that they effectively solved the problem themselves, but returning to using dropouts could prevent a change to a super weight being âcompletely surprising to all other weightsâ, says Kusner.
arXiv
