Author: Rob Matheson | MIT News Office<\/p>\n
Researchers from MIT and the Qatar Computing Research Institute (QCRI) are putting the machine-learning systems known as neural networks under the microscope.<\/p>\n
In a study that sheds light on how these systems manage to translate text from one language to another, the researchers developed a method that pinpoints individual nodes, or \u201cneurons,\u201d in the networks that capture specific linguistic features.<\/p>\n
Neural networks learn to perform computational tasks by processing huge sets of training data. In machine translation, a network crunches language data annotated by humans, and presumably \u201clearns\u201d linguistic features, such as word morphology, sentence structure, and word meaning. Given new text, these networks match these learned features from one language to another, and produce a translation.<\/p>\n
But, in training, these networks basically adjust internal settings and values in ways the creators can\u2019t interpret. For machine translation, that means the creators don\u2019t necessarily know which linguistic features the network captures.<\/p>\n
In a paper being presented at this week\u2019s Association for the Advancement of Artificial Intelligence conference, the researchers describe a method that identifies which neurons are most active when classifying specific linguistic features. They also designed a toolkit for users to analyze and manipulate how their networks translate text for various purposes, such as making up for any classification biases in the training data.<\/p>\n
In their paper, the researchers pinpoint neurons that are used to classify, for instance, gendered words, past and present tenses, numbers at the beginning or middle of sentences, and plural and singular words. They also show how some of these tasks require many neurons, while others require only one or two.<\/p>\n
\u201cOur research aims to look inside neural networks for language and see what information they learn,\u201d says co-author Yonatan Belinkov, a postdoc in the Computer Science and Artificial Intelligence Laboratory (CSAIL). \u201cThis work is about gaining a more fine-grained understanding of neural networks and having better control of how these models behave.\u201d<\/p>\n
Co-authors on the paper are: senior research scientist James Glass and undergraduate student Anthony Bau, of CSAIL; and Hassan Sajjad, Nadir Durrani, and Fahim Dalvi, of QCRI. \u00a0<\/p>\n
Putting a microscope on neurons<\/strong><\/p>\n Neural networks are structured in layers, where each layer consists of many processing nodes, each connected to nodes in layers above and below. Data are first processed in the lowest layer, which passes an output to the above layer, and so on. Each output has a different \u201cweight\u201d to determine how much it figures into the next layer\u2019s computation. During training, these weights are constantly readjusted.<\/p>\n Neural networks used for machine translation train on annotated language data. In training, each layer learns different \u201cword embeddings\u201d for one word. Word embeddings are essentially tables of several hundred numbers combined in a way that corresponds to one word and that word\u2019s function in a sentence. Each number in the embedding is calculated by a single neuron.<\/p>\n In their past work<\/a>, the researchers trained a model to analyze the weighted outputs of each layer to determine how the layers classified any given embedding. They found that lower layers classified relatively simpler linguistic features \u2014 such as the structure of a particular word \u2014\u00a0and higher levels helped classify more complex features, such as how the words combine to form meaning.<\/p>\n In their new work, the researchers use this approach to determine how learned word embeddings make a linguistic classification. But they also implemented a new technique, called \u201clinguistic correlation analysis,\u201d that trains a model to home in on the individual neurons in each word embedding that were most important in the classification.<\/p>\n The new technique combines all the embeddings captured from different layers \u2014 which each contain information about the word\u2019s final classification \u2014 into a single embedding. As the network classifies a given word, the model learns weights for every neuron that was activated during each classification process. This provides a weight to each neuron in each word embedding that fired for a specific part of the classification.<\/p>\n \u201cThe idea is, if this neuron is important, there should be a high weight that\u2019s learned,\u201d Belinkov says. \u201cThe neurons with high weights are the ones more important to predicting the certain linguistic property. You can think of the neurons as a lot of knobs you need to turn to get the correct combination of numbers in the embedding. Some knobs are more important than others, so the technique is a way to assign importance to those knobs.\u201d<\/p>\n Neuron ablation, model manipulation<\/strong><\/p>\n Because each neuron is weighted, it can be ranked in order of importance. To that end, the researchers designed a toolkit, called NeuroX, that automatically ranks all neurons of a neural network according to their importance and visualizes them in a web interface.<\/p>\n Users upload a network they\u2019ve already trained, as well as new text. The app displays the text and, next to it, a list of specific neurons, each with an identification number. When a user clicks on a neuron, the text will be highlighted depending on which words and phrases the neuron activates for. From there, users can completely knock out \u2014 or \u201cablate\u201d \u2014\u00a0the neurons, or modify the extent of their activation, to control how the network translates.<\/p>\n The task of ablation was used to determine if the researchers\u2019 method accurately pinpointed the correct high-ranking neurons. In their paper, the researchers used the tool to show that, by ablating high ranking neurons in a network, its performance in classifying correlated linguistic features dipped significantly. Alternatively, when they ablated lower-ranking neurons, performance suffered, but not as dramatically.<\/p>\n \u201cAfter you get all these rankings, you want to see what happens when you kill these neurons and see how badly it affects performance,\u201d Belinkov says. \u201cThat\u2019s an important result proving that the neurons we find are, in fact, important to the classification process.\u201d<\/p>\n One interesting application for the toolkit is helping limit biases in language data. Machine-translation models, such as Google Translate, may train on data with gender bias, which can be problematic for languages with gendered words. Certain professions, for instance, may be more often referred to as male, and others as female. When a network translates new text, it may only produce the learned gender for those words. In many online English-to-Spanish translations, for instance, \u201cdoctor\u201d often translates into its masculine version, while \u201cnurse\u201d translates into its feminine version.<\/p>\n \u201cBut we find we can trace individual neurons in charge of linguistic properties like gender,\u201d Belinkov says. \u201cIf you\u2019re able to trace them, maybe you can intervene somehow and influence the translation to translate these words more to the opposite gender \u2026 to remove or mitigate the bias.\u201d<\/p>\n In preliminary experiments, the researchers modified neurons in a network to change translated text from past to present tense with 67 percent accuracy. They ablated to switch the gender of the words with 21 percent accuracy. \u201cIt\u2019s still a work in progress,\u201d Belinkov says. A next step, he adds, is fine-tuning the web application to achieve more accurate ablation and manipulation.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"