Multi-Task Learning

A Milestone of the New Century: Multi-Task Learning

Learning multiple tasks to improve generalization and transfer ability

The Second: Word Vectors

Word vectors (word Vector), also called word embeddings (word Embedding) or distributed representations of words (distributed representation).

A word vector might have, say, 100 dimensions: low dimensionality but a large amount of information [0.2,0.3,0.2,1.3,3.4]

We use two methods, CBOW and Skip-gram, to train word vectors

When we use word vectors to represent a word, we can express lexical relationships on multiple levels, for example: man—>woman, king—>queen (meaning), walked —>walking (morphology). We can represent the word along multiple dimensions

One-hot vector: one hot [0,0,0,0,2.3,0,0,0,0], 10,000 dimensions. One-hot vectors don’t take word order or the relationships between words into account, and contain little information

Recurrent Neural Networks (RNN) and Their Variants

Recurrent neural networks resemble the way we humans read. For example, when we read a sentence, our brain actually keeps a memory of what we’ve read earlier, which is how we can fully understand a text. The key point is that it has the ability to remember. But it has a big flaw: the basic version of the recurrent neural network is hard to train. For example, if a passage has 500 words, by the time the information from the first word reaches the end, there’s basically almost no information left.

Later, LSTM (Long Short-Term Memory Network) was proposed. It’s an improved version of the RNN that solves the problem of RNNs being hard to train. Its principle is to remember important information and forget unimportant information, so the memory burden on the neural network is smaller. That’s why LSTM trains better than RNN on long texts

The Seq2Seq Framework

From one sequence to another sequence—for example, in machine translation, translating French into English, or turning an article into a summary. These tasks all fall under the Seq2Seq framework. Specifically, it consists of two parts: the first is the encoder, and the second is the decoder. In 2016, Google proposed a Seq2Seq-based machine translation system that achieved fairly good results. You can also take an image as input and output text, or take audio as input with text as the corresponding target. What the model looks for is the hidden connection between the two, which is somewhat similar to the interlingua idea we discussed earlier.

The Attention Mechanism

The attention mechanism also originated in cognitive science. The human visual system has an attention mechanism that devotes more visual resources to the parts it is interested in. The attention mechanism is also applied in machine translation: Seq2Seq + attention

i am a student —> 我是一个学生

We can consider the information in a sentence with different emphasis, so our translation will be more accurate

The Self-Attention Mechanism

The attention mechanism deals with the relationship between two sentences, while the self-attention mechanism looks for the relationships between different words within the same sentence

The animal did not cross the street,because is was too tired.

Does this “it” refer to the animal or the street? It’s easy for humans to tell, but not so easy for machines. With the self-attention mechanism, we can let the machine learn that “it” is more strongly connected to the animal. It can discover deeper encoded information within a sentence. A model proposed using the self-attention mechanism is called the Transformer.

Pre-trained Language Models

GPT, BERT, XLNet, and a series of other frameworks proposed based on the self-attention mechanism opened a new chapter of pre-trained language models (Pretrained language model) in natural language processing. We can use these models to train on a large amount of corpus data; after training, these models have language-processing abilities with strong generalization. We then fine-tune these models, after which we can use them to understand basic grammar and morphology, then combine them with corpora for specific tasks and train, and they can perform those specific tasks.