Lionbridge Expert Commentary: Automated Translation Analysis

This month, we want to bring your attention to language formality and how difficult — but not impossible — it is to get it right when using Machine Translation (MT).

Machine Translation (MT) engines can produce incorrect and inconsistent formality. Why? MT models typically return a single translation for each input segment. When the input segment is ambiguous, the model must choose a translation among several valid options, regardless of the target audience. Letting the model choose between different valid options may result in inconsistent translations or translations that have an incorrect level of formality.

It is especially challenging to get the correct output when the source language has fewer formality levels than the target language. For instance, languages like French have well-defined formal modes — tu vs. vous — while English does not.

While most MT systems do not support language formality or gender parameters, we are seeing progress. At present, DeepL (API) and Amazon (console and SDK) offer features that control formality. Lionbridge’s Smaⁱrt MT™, an enterprise-grade Machine Translation solution, allows linguistic rules to be applied to the target text to produce Machine Translations with the desired style or formality.

It’s critical to effectively translate your source to meet the needs of your target audiences, which includes addressing formal and informal language in your MT output. Translations that come across as “off” or — even worse — as rude can put you at risk of alienating your audiences.

Read our blog to learn more about Machine Translation and formal vs. informal language.

    —Yolanda Martin, Lionbridge MT Specialist 

It can be advantageous to use Machine Translation (MT), but you must proceed with caution. Generic MT engines can put out erroneous translations and can especially cause undesired results for specific domains from a terminological point of view. The impact can be particularly harmful to the medical and legal fields. But there are things you can do to enhance MT output.

Using terminology can enable you to improve the quality of MT and achieve accurate, consistent translations.

It’s imperative to train customized MT systems with domain-specific bilingual texts that include specialized terminology. Still, accurate translations cannot be guaranteed when engines are trained with specialized texts if the terminology is not used consistently. Research in this area proposes to inject linguistic information into Neural Machine Translation (NMT) systems. The implementation of manual or semi-automatic annotation depends on available resources, such as glossaries, and constraints, such as time, cost, and availability of human annotators.

Lionbridge’s Smaⁱrt MT™ allows the application of linguistic rules to the source and target text, as well as the enforcement of terminology based on Do Not Translate (DNT) and glossary lists added to a specific profile. We help our customers create and maintain glossaries, which are regularly refined to include new, relevant terms and retire obsolete terminology. When glossaries are created once in Smaⁱrt MT, they can then be used for all the MT engines, saving time and money.

Using glossaries for MT projects is not as simple as it may seem. Glossaries, if used inappropriately, can negatively affect the overall quality of Machine Translation. The best way to follow terminology in MT is through MT training. The combination of trained MT engines, glossary customization, and the identification of preprocessing and post-processing rules ensure MT output contains proper terminology and is similar in style to the customer's documentation.

Read our blog for more insight into using terminology to enhance MT output.

    —Yolanda Martin, Lionbridge MT Specialist 

As companies increasingly rely on Machine Translation (MT) as a standard practice, employees will need to prevent the dissemination of catastrophic errors.

Catastrophic errors are more problematic than standard MT errors, which pertain to error typology related to linguistic features, such as spelling, grammar, or punctuation. Catastrophic errors transcend linguistics and occur when engine output dangerously deviates from the intended message. Resulting misinformation or misunderstandings have the potential to cause companies reputational, financial, or legal problems and may lead to adverse public safety or health consequences. It is essential to find ways to identify these errors and stop them from compromising your communications.

Lionbridge administers specific automated quality checks in translated texts to detect critical errors while preserving MT speed and reducing the need for human intervention.

These automated methods detect:

• Opposite meanings between original and translated texts
• Offensive, profane, or highly sensitive words
• Incorrect translations of proper names of individuals and organizations that are also common words

Companies will be better protected from catastrophic errors when computer scientists improve existing MT technology to prevent these translation errors. Until such time, we can use automated technology to identify potential issues, revise problematic sentences, and promote accuracy during the translation process.

Read our blog for a more in-depth examination of catastrophic errors during Machine Translation.

    —Luis Javier Santiago, MT Group Leader,

    and Rafa Moral, Lionbridge Vice President, Innovation 

Google NMT, Bing NMT, Amazon, DeepL, Yandex — which engine is best? Last month’s data — and the current general trend — show major engines perform similarly. That’s why it’s worthwhile to consider additional factors when developing your MT strategy, such as the ease with which MT engines translate specific language pairs.

Identifying how challenging it is for engines to handle specific language pairs will help you allocate your budget when planning translation costs across languages. For instance, you’ll need to allocate more effort to achieve high-quality translations when dealing with complex language pairs. Having insight into language complexity can help support your business decisions.

Ranking languages by translatability is not a straightforward process; however, we can use different metrics for evaluation. Edit Distance, which is the number of changes a post-editor makes to ensure the final text has a human quality, can provide a sense of MT complexity and translatability (machine-translatability or m-translatability) for each language pair.

Most Romance languages, such as Portuguese, Spanish, French and Italian, require fewer changes to reach high-quality levels when translated from English. We identified these target languages as the easiest for machines to handle, and they took the first four spots in our m-translatability ranking. Hungarian and Finnish — two Uralic languages — are more complex languages; they placed last in our ranking, taking the 27th and 28th spots. Estonian, another language in the same family, is also among the more complex languages. These results — based on millions of sentences processed by Lionbridge — underscore the importance of language families in MT results.

While intra language comparison has limitations, the ranking can provide some interesting insights to better manage multilingual projects. Read our blog to see the Lionbridge language ranking table in its entirety.

    —Rafa Moral, Lionbridge Vice President, Innovation 

In June, we observed a tiny improvement in Russian translations by Yandex’s MT engine and a tiny dip in translation results by Microsoft Bing’s MT engine. Are these noteworthy changes or insignificant, spurious outcomes? To find out, we analyzed the results differently.

Instead of using a single gold standard that measures the distance from the MT translation to one “perfect” human translation, we used multiple reference translations. We compared each translation made by machines to 10 translations by professional translators. When we took this approach, the small fluctuations in translation quality by Yandex and Microsoft Bing in June disappeared. As such, we can conclude that there were no changes to the MT translation quality. June results are flat.

Sometimes data and its graphical representations may be misleading. This often happens when there are small deltas among different measurements. It’s good practice to use more than one approach to evaluate data to interpret results accurately.

We project little movement in MT engine quality in the coming months. We will use this section to provide analysis and general MT observations. Next month, look for comparisons among MT language pairs. We’ll explore whether it is possible to use data to classify languages and language families by MT complexity and determine whether machines can translate some language pairs easier than others.

    —Rafa Moral, Lionbridge Vice President, Innovation 

It has primarily been another static month for the MT engines.

We’ve noticed Amazon has made an incremental improvement in the way its engine handles the English-Spanish pair. It is now the leading engine in this language pair. Amazon also made minor strides in the other languages, but smaller than its improvements to the English-Spanish pair. We speculate these advancements are due to some generic setting changes and as a result of work on the English-Spanish pair. Enhancements appear to affect the treatment of some special characters and strings with measurement expressions.

For the second month in a row, Yandex made minor improvements. Interestingly, these improvements also affect Spanish.

As we’ve previously noted, there have been no significant changes. All the engines perform similarly. In the coming months, we will analyze some specific MT areas and provide general observations. Of course, we will also track major developments.

    —Rafa Moral, Lionbridge Vice President, Innovation 

After several months of flat MT engine performance, Yandex has made some progress, particularly with its German engine.

In one detailed analysis, we saw advancements in Yandex engines' handling of sentences with punctuation characters — such as question marks, exclamation points, parentheses, and slashes — and units of measurement. These developments may result from some fine-tuning to the MT settings rather than improvements in the models. However, we also saw improvement in our tracking of rare terms, so Yandex’s progress may also be due to some refinements of the models or more data training.

Around this time last year, several MT engines showed some improvements that we found interesting. Is there a time pattern involved with these advancements? Will we see something like what we observed in 2021 this year? We’re tracking the MT performance of these engines, and we'll report our findings in the next month or so.

Generally, there is an increased interest in MT engine evaluation. Today, most everyone will agree that MT is a mature technology. People recognize the technology’s usefulness for almost any translation case — with or without human intervention and hybrid approaches. But MT users are still struggling to find the right way to evaluate, measure, and improve MT results.

    —Rafa Moral, Lionbridge Vice President, Innovation 

If you’ve been following these pages, you’re familiar with our generic MT comparative evaluations. Each month, we identify which MT engines are performing best for given language pairs and track engine improvements. In March, the performance of the different MT engines was flat. It’s a trend we’ve been noticing for some time already. As we commented last month, it may indicate that a new MT paradigm is needed.

While we share generic results, companies are increasingly pursuing custom MT comparative evaluations. Unlike the generic version, these evaluations take a company’s specific needs into consideration when determining the most advantageous MT engines.

When a company wants to start using MT or improve the way it currently uses MT, it is critical to identify which MT engines will work best. When we execute custom evaluations, we take a similar approach to the one demonstrated on this page, but we make recommendations based on a company’s content type and language pair requirements.

While custom MT comparative evaluations have been available for years, there’s greater demand for them. We attribute this trend to the important role MT plays in helping companies succeed in a digital marketplace.

    —Rafa Moral, Lionbridge Vice President, Innovation 

Google’s MT engine showed a tiny improvement during January and February of 2022, while the other engines we track remained stagnant. These observations may lead us to start asking some pointed questions. Is the Neural Machine Translation (NMT) paradigm reaching a plateau? Is a new paradigm shift needed given the engines’ inability to make significant strides? We observed similar trends when NMT replaced Statistical MT.

At the end of the Statistical MT era, there was virtually no change to MT quality output. In addition, the quality output of different MT engines converged. We see similar trends. While NMT may not be imminently replaced, if we believe in exponential growth and accelerating returns theories — and consider Rule-based MT’s 30-year run and Statistical MT’s decade-long prominence and note NMT is now in its sixth year — a new paradigm shift may not be so far away.

    —Rafa Moral, Lionbridge Vice President, Innovation 

During January, the main Machine Translation (MT) engines did not show significant changes in their performance. 

Google demonstrated small, incremental improvements across some languages and domains. The performance of most of the other engines has been flat. Microsoft had improvements over the last few months, but performance plateaued in January. Overall, the quality of Google Translate continues to lead in general-purpose MT technology.  

In December, we added a fifth MT engine to our tracker. By monitoring Yandex, we can better analyze the MT quality of the Russian language.

    —Rafa Moral, Lionbridge Vice President, Innovation 

    —Jordi Macias, Lionbridge Vice President, Language Excellence