There has been much in the news lately about the next wave of MT technology driven by something called deep learning neural nets (DNN). With the help of some folks who understand more about it than I do (see below), I will attempt to provide a brief overview about what this is. I need to confess first that I will be trying to explain something here that I don't fully understand myself. Still, I hope that my research has helped me comprehend and communicate some basic underlying principles.
Most lectures you've listened to about machine translation in the past few years have likely included a statement like this: "There are basically two different kinds of machine translation -- rules-based and statistical MT -- and a third that combines the two -- hybrid machine translation." You then probably heard that rules-based machine translation was the earliest form of machine translation in the computer age, going back all the way to the 1950s and consisting of a set of rules about source and target language as well as a dictionary. The transfer between source and target language in rules-based machine translation happens either via an "interlingua," a computerized representation of the source text, or directly between source and target language. Statistical machine translation, on the other hand, became all the rage in the early 2000s. (The first commercial offering, LanguageWeaver [now owned by SDL], was launched in 2002; the widely used open-source engine Moses emerged in 2005; Google and Microsoft switched to statistical MT in 2007; and Yandex and Baidu started using SMT as recently as 2011.) Statistical machine translation, or more accurately for all of these implementations, "phrase-based statistical machine translation," is trained on bilingual data and monolingual data. It parses the data into "n-grams," phrases consisting of an "n" number of words. The same thing happens to the source segment in the translation process. The source n-grams are then matched with target n-grams, which are then combined to form whole segments again -- and that's often where things goes awry. (This is why SMT can prove to be a much richer resource when using an approach that just looks for fragments rather than whole segments.) Another potential flaw with SMT is the faulty selection of which of the many possible target n-grams should be used. One way to guard against bad choices is by validating on the basis of the monolingual target data that the system was trained with, but that only goes so far. (And, by the way, that's why an approach that offers access to more than just one of those n-gram fragments at a time within a translation environment tool has to be one of the up-and-coming developments.) Neural machine translation (NMT) -- and let's pause and be thankful that one of this technology's first proposed terms, "recursive hetero-associative memories for translation" (by Mikel L. Forcada et al. in 1997), did not survive -- is an extremely computing-power-heavy process (which is why it didn't go anywhere in 1997), and is part of the larger field of "machine learning." According to one of the field's pioneers, Arthur Samuel, machine learning is the "field of study that gives computers the ability to learn without being explicitly programmed" (1959). In SMT, the focus is on translated phrases that the computer is taught, which are then reused and fitted together according to statistics; NMT, on the other hand, uses neural networks that consist of many nodes (conceptually modeled after the human brain) which relate to each other and can hold single words, phrases, or any other segment. These nodes build relationships with each other based on bilingual texts with which you train the system. Because of these manifold and detailed relationships, it's possible to look not just at limited n-grams as in SMT, but at whole segments or even beyond individual segments, allowing the formation of significantly more educated guesses about the context and therefore the meaning of any word in a segment that needs to be translated. For instance, it's at least theoretically unlikely to have "Prince" translated as a (royal) prince by an NMT in a sentence like "The music world mourns the death of Prince" as Google, Microsoft, Yandex, and Baidu all do at the moment (and, by the way, I'm mourning as well). In languages like German with separable verbs such as umfahren ("run over"), there is a much greater likelihood that the system will notice the missing part of the verb at the end of the sentence if the machine does not have to bother with chopping it into n-grams first. Take, for example, the simple sentence "Ich fahre den Fußgänger um" -- "I run over the pedestrian." Bing translates it (today) as "I'm going to the pedestrian" and Google as "I drive around the pedestrian"; only Yandex gets it right (Baidu does not offer this language combination). The machine learning (itself a subfield of artificial intelligence) additionally comes into play as common usage gradually forges certain linguistic connections ("music world" and "Prince"; "fahren" and "um"), so the computer continues to "learn" without explicitly being programmed as Samuel had predicted. At least theoretically, therefore, the NMT approach is very promising for generic engines like those of the search engines mentioned above (Google, Microsoft, Yandex, Baidu) because "context" does not necessarily have to be specified by the training data but can be recognized by the system evaluating the context (provided that the user supplies more than just a word or single phrase). So you won't be surprised to hear that all those companies have already entered the realm of NMT. Naturally they don't reveal how much of their present system is "neural" vs. "statistical only," but chances are it's a mix of both. And that would make all the more sense since one of the ways to use NMT is in combination with SMT -- either as a quasi-independent verification process or as an integrated process that helps in selecting the "right" n-grams. In some areas similar processes have already demonstrated remarkable success -- including some that are used by the search engines. Remember when I mentioned how to use the search on Google Image for translation tasks? That is one of the areas where neural network processes have proven to be quite successful. You probably read that Facebook launched its own machine translation system earlier this year specifically geared for the very casual language of its users. While that system is still mostly SMT-based, they're working on an NMT solution as well. You might want to take a look at Facebook's Alan Packer's (formerly of Microsoft) talk right here. One misconception in his talk is his description of all this as a linear development. He paints SMT as more or less having run its course, now to be taken over by NMT. While I understand that someone so deeply embedded in one particular field must automatically think it the only worthwhile one, it's really unlikely to be the case. The same was said in the early days by proponents of SMT about RbMT (rules-based MT), and that assumption has not proven to be true. Many systems are using a hybrid approach between SMT and RbMT, and for some language combinations RbMT might still be a better solution (especially for language pairs that are very close to each other, like Catalan and Spanish or Croatian and Serbian). But are we on the verge of a big new breakthrough overall? To answer that, you might want to look through this joint presentation by Tauyou and Prompsit. Since there is no open-source toolkit for NMT like Moses for SMT, very few companies actually offer customized NMT systems. There are components like the deep learning frameworks Theano and Torch and specific NMT software like GroundHog and seq2seq, but these are anything but user-friendly and require significant expertise. Using them to build an NMT engine takes a lot of computing power (10 CPUs or 1 GPU -- graphics processing unit) and time (about two weeks of training per language pair once the training data is assembled and cleaned). Tauyou and Prompsit are some of the first vendors who are working on commercial versions of NMT (interestingly, Tauyou comes with an SMT background and Prompsit with a background in RbMT). While they are not actively selling the NMT solutions yet, they are doing a lot of pilots as you can see from the presentation. And the results of these pilots are mixed. I already mentioned the much larger processing and time requirements, and there are also limitations as far as the number of words per language that can be trained with the processing power currently available to mere mortals (in opposition to companies like Google), the approximately three-fold time the system takes to actually translate, and the fact that retraining the system with new data would once again take two weeks. But there are some improvements in the quality -- although, according to the presentation, this is not adequately appreciated by translators (which I assume has to do with even less predictability when it comes to post-editing -- and presumably even more erratic decisions when it comes to partial suggestions). However, this is still very early in the game, so I wouldn't be surprised to see the quality continue to improve. Do we need to start shivering in fear when we hear folks talking about neural machine translation? Although I don't completely understand the technology, I (and now you) have seen numbers showing only moderate progress. So, no, we'll continue to be very assured of our jobs for a long time. I do look forward, though, to seeing how NMT will creatively find its way into our translation environment and improve our work. Thanks very much to Kirti Vashee (formerly of Asia Online but now very eager to consult on any kind of machine translation implementation), Jay Marciano (who will not only present his "regular" talk on demystifying MT at this year's ATA but also one on what the increased use of artificial intelligence means for translators over the next several years), and Gema Ramírez Sánchez and Sergio Ortiz Rojas of Prompsit. |
