History of NLP

Early History of Natural Language Processing

The history of natural language processing is really also the history of machine translation. In the 1940s, advanced countries tried to research machine translation, and out of enthusiasm for machine translation they began research work combining language with computers. Driven by the combined influence of the times, the development of computers, the international situation and other factors, people hoped to use computers to do translation work, and different scholars proposed corresponding methods. The methods of that time may no longer be mainstream today, but the process of exploration still offers a lot of valuable lessons for our research now. Some of the papers we publish today also draw on research ideas from the last century.

On January 7, 1954, at IBM’s headquarters in New York, the Georgetown-IBM experiment was launched. IBM’s 701 computer automatically translated 60 Russian sentences into English, the first machine translation in history.

The Earliest Translation Was Rule-Based

The rules used in machine translation were mainly word rules, syntax rules, phrase rules and transformational-generative grammar

Machine translation falls into three categories:

1. Direct translation: it does not involve deep grammatical analysis; the experiment above basically applied this method
   Different languages do not have a complete one-to-one correspondence
2. Structural transfer translation: analyze the features of the source sentence ---> convert it into the deep structure of the source sentence ----> convert it into the deep structure of the target language ---> target-language sentence
3. Extension: using an interlingua, which is also rule-based

Memory-Based

Translation is done using a corpus of corresponding source and target sentences. This is an example-based machine learning method proposed by Makoto Nagao of Kyoto University in 1984.

Advantages: simple to operate, easy to extend, and translation quality is guaranteed

Disadvantages: it requires a very large corpus, and the up-front work of collecting and organizing it is enormous.

Later, another Japanese scholar, Satoshi Sato, building on the example-based approach, proposed memory-based machine translation. The idea is also simple: sentences that have already been translated are added to the corpus. In this way, as the translation system keeps being used, its translation ability gradually grows stronger, as if it had a memory function like humans

Statistics-Based

In 1990, Peter F Brown published the classic paper “A Statical approach to machine Translation”, officially opening the era of statistical machine translation.

This method does not understand the overall rules or linguistics, and completely abandons complicated rule-setting. It relies only on large amounts of language resources to understand the one-to-one correspondence patterns from a statistical point of view. To sum it up in one sentence: if everyone translates it this way, I will translate it this way too.

The inside story of Google’s statistical translation: it computes probabilities, and it also computes them in reverse

IBM Model

The IBM Model is a highly representative model of statistical machine translation

Bayes’ theorem

P(A|B) P(B) = P(B|A) P(A)

P(A|B) = P(B|A) * P(A) / P(B)

P(translation|source) = P(source|translation) * P(translation) / P(source)

The source text is given, i.e. the target

P(translation|source) is proportional to (source|translation) * P(translation)

From the perspective of natural language processing, that is

P(translation|source) is proportional to translation model * language model

Language model: the problem of estimating the probability that a sentence appears in a language, used to estimate whether the sentence fits the characteristics of natural language.

For example, the greater the probability that the sentence P(I am a student?) appears in English, the more it looks like an English sentence (something a human would say).

Using the frequency of a whole sentence to measure whether it is something a human would say is not very reasonable, because no corpus is large enough to cover all sentences. So we use word groups as the unit and count the frequency of word groups, which is the N-gram model.

Translation model: the correspondence between languages

Bayes’ theorem can also be applied to:

1. Chatbots: B (the question the user enters) A (the machine’s answer)
2. Speech to text: B (speech) A (text)
3. Search engines: B (the keywords the user enters) A (the answers the machine gives)

P(A|B) P(B) = P(B|A) P(A)

P(B|A) indicates the accuracy of the correspondence between the two, and P(A) represents the fluency of the answer