3

Context

I am using MarianMT von Huggingface via Python in order to translate text from a source to a target language.

Expected behaviour

I enter a sequence into the MarianMT model and get this sequence translated back. For this, I use a corresponding language model and a tokeniser. All the sentences I enter also come back. The sentences are treated as a sequence.

Current behaviour

Depending on the language model, the model does not translate everything, but only returns parts. In this example, the last sentence is missing:

Original (German): Ein Nilpferd lief im Dschungel rum und musste aufs WC. Da traf es einen Kakadu und fragte nach dem Weg. Der sagte wenn du Kaka musst, dann pass mal ganz kurz auf. Ich sag dir wo du hingehen musst, ich kenn mich hier gut aus.

Result (English): A hippopotamus ran around in the jungle and had to go to the toilet. There was a cockatoo and asked for the way. He said if you have to Kaka, then watch out for a minute. I'll tell you where you have to go, I know my way around here.

Result (Dutch): Een nijlpaard liep rond in de jungle en moest naar het toilet... en een kaketoe vroeg naar de weg... die zei dat als je Kaka moest, ik even moest oppassen.

Current Code

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM


def translate_text(input, source, target):

    # Prepare output
    output = ""

    model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-" + source + "-" + target)
    tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-" + source + "-" + target)

    inputs = tokenizer.encode(input[:512], return_tensors="pt", padding='longest')
    outputs = model.generate(inputs, max_length=4000, num_beams=4, early_stopping=True)

    for t in [tokenizer.convert_ids_to_tokens(s) for s in outputs.tolist()[0]]:
        output = output + t.replace("▁", " ").replace("</s>", "")

    output.replace("<pad>", "")

    return output


print(translate_text("Ein Nilpferd lief im Dschungel rum und musste aufs WC. Da traf es einen Kakadu und fragte nach dem Weg. Der sagte wenn du Kaka musst, dann pass mal ganz kurz auf. Ich sag dir wo du hingehen musst, ich kenn mich hier gut aus.", "de", "nl"))
print(translate_text("Ein Nilpferd lief im Dschungel rum und musste aufs WC. Da traf es einen Kakadu und fragte nach dem Weg. Der sagte wenn du Kaka musst, dann pass mal ganz kurz auf. Ich sag dir wo du hingehen musst, ich kenn mich hier gut aus.", "de", "en"))

Help needed

What do I miss? Why are some sequence parts missing?

ccpizza
  • 28,968
  • 18
  • 162
  • 169
Lorento Palanomi
  • 49
  • 6
  • 1
    What happens if you try it with `"Ein Nilpferd ...` instead of `"in Nilpferd ...`? – Adam Bittlingmayer Dec 22 '20 at 12:48
  • 1
    Thanks for highlighting the typo, which I fixed above. But it does not change the described behavior! – Lorento Palanomi Dec 22 '20 at 12:51
  • 2
    Do you know how Opus MT segmented the *training* data? Has the model seen such sequences in training? – Adam Bittlingmayer Dec 22 '20 at 13:00
  • As far as I know it is sentence-sentence. That means, the model actually does not know about more sentence sequences. But if I split the sentences before I translate them, I am loosing important context for the translation! – Lorento Palanomi Dec 22 '20 at 13:26
  • When I split the sentences it is working as expected. But I will lose context this way: ['Er liep een nijlpaard rond in de jungle en hij moest naar het toilet.', 'Toen ontmoette hij een kaketoe en vroeg naar de weg.', 'Hij zei dat als je Kaka moest, je even moet opletten.', 'Ik vertel je waar je heen moet. Ik ken het hier goed.'] – Lorento Palanomi Dec 22 '20 at 13:39
  • 2
    If the model has not been trained with context, then it's not going to know what to do with it anyway. – Adam Bittlingmayer Dec 22 '20 at 15:39

1 Answers1

2

In this case, you could translate it via English:

de_en = translate_text("Ein Nilpferd lief im Dschungel rum und musste aufs WC. Da traf es einen Kakadu und fragte nach dem Weg. Der sagte wenn du Kaka musst, dann pass mal ganz kurz auf. Ich sag dir wo du hingehen musst, ich kenn mich hier gut aus.", "de", "en")

en_nl = translate_text(de_en, "en", "nl")

print(en_nl)

Result: Een nijlpaard rende rond in de jungle en moest naar het toilet. Er was een kaketoe en vroeg om de weg. Hij zei als je moet Kaka, dan uitkijken voor een minuut. Ik zal je vertellen waar je moet gaan, Ik weet mijn weg hier.

The last sentence did not disappear, but the quality is lower. De->En and En->Nl models probably had much longer sentences in their training data (you never know), than De->Nl, and that is why the last sentence did not disappear from the translation. But at the same time, translating into English may cause some information loss (e.g. du/Sie -> you).

Given the model's name (trained on the OPUS corpus), how big the sentences may theoretically be you could see here: http://opus.nlpl.eu/Europarl/v8/de-nl_sample.html or here: http://opus.nlpl.eu/MultiParaCrawl/v7.1/de-nl_sample.html; or in other de-nl samples at opus.nlpl.eu

More info is available here: https://github.com/Helsinki-NLP/Opus-MT

tl;dr The fact that these models translate multiple sentences glued together is most probably just a side effect on which one should not rely.

Bohdan I
  • 106
  • 3