python gettext: specify locale in _()

Question

I am looking fo a way to set the language on the fly when requesting a translation for a string in gettext. I'll explain why :

I have a multithreaded bot that respond to users by text on multiple servers, thus needing to reply in different languages. The documentation of gettext states that, to change locale while running, you should do the following :

import gettext # first, import gettext

lang1 = gettext.translation('myapplication', languages=['en']) # Load every translations
lang2 = gettext.translation('myapplication', languages=['fr'])
lang3 = gettext.translation('myapplication', languages=['de'])

# start by using language1
lang1.install()

# ... time goes by, user selects language 2
lang2.install()

# ... more time goes by, user selects language 3
lang3.install()

But, this does not apply in my case, as the bot is multithreaded :

Imagine the 2 following snippets are running at the same time :

import time
import gettext 
lang1 = gettext.translation('myapplication', languages=['fr'])
lang1.install()
message(_("Loading a dummy task")) # This should be in french, and it will
time.sleep(10)
message(_("Finished loading")) # This should be in french too, but it wont :'(

and

import time
import gettext 
lang = gettext.translation('myapplication', languages=['en'])
time.sleep(3) # Not requested on the same time
lang.install()
message(_("Loading a dummy task")) # This should be in english, and it will
time.sleep(10)
message(_("Finished loading")) # This should be in english too, and it will

You can see that messages sometimes are translated in the wrong locale. But, if I could do something like _("string", lang="FR"), the problem would disappear !

Have I missed something, or I'm using the wrong module to do the task... I'm using python3

Answer 1

While the above solutions seem to work, they don’t play well with the conventional _() function that aliases gettext(). But I wanted to keep that function, because it’s used to extract translation strings from the source (see docs or e.g. this blog).

Because my module runs in a multi-process and multi-threaded environment, using the application’s built-in namespace or a module’s global namespace wouldn’t work because _() would be a shared resource and subject to race conditions if multiple threads install different translations.

So, first I wrote a short helper function that returns a translation closure:

import gettext

def get_translator(lang: str = "en"):
    trans = gettext.translation("foo", localedir="/path/to/locale", languages=(lang,))
    return trans.gettext

And then, in functions that use translated strings I assigned that translation closure to the _, thus making it the desired function _() in the local scope of my function without polluting a global shared namespace:

def some_function(...):
    _ = get_translator()  # Pass whatever language is needed.

    log.info(_("A translated log message!"))

(Extra brownie points for wrapping the get_translator() function into a memoizing cache to avoid creating the same closures too many times.)

Answer 2

You can just create translation objects for each language directly from .mo files:

from babel.support import Translations

def gettext(msg, lang):
    return get_translator(lang).gettext(msg)

def get_translator(lang):
    with open(f"path_to_{lang}_mo_file", "rb") as fp:
        return Translations(fp=fp, domain="name_of_your_domain")

And a dict cache for them can be easily thrown in there too.

Answer 3

I took a moment to whip up a script that uses all the locales available on the system, and tries to print a well-known message in them. Note that "all locales" includes mere encoding changes, which are negated by Python anyway, and plenty of translations are incomplete so do use the fallback.

Obviously, you will also have to make appropriate changes to your use of xgettext (or equivalent) for you real code to identify the translating function.

#!/usr/bin/env python3

import gettext
import os

def all_languages():
    rv = []
    for lang in os.listdir(gettext._default_localedir):
        base = lang.split('_')[0].split('.')[0].split('@')[0]
        if 2 <= len(base) <= 3 and all(c.islower() for c in base):
            if base != 'all':
                rv.append(lang)
    rv.sort()
    rv.append('C.UTF-8')
    rv.append('C')
    return rv

class Domain:
    def __init__(self, domain):
        self._domain = domain
        self._translations = {}

    def _get_translation(self, lang):
        try:
            return self._translations[lang]
        except KeyError:
            # The fact that `fallback=True` is not the default is a serious design flaw.
            rv = self._translations[lang] = gettext.translation(self._domain, languages=[lang], fallback=True)
            return rv

    def get(self, lang, msg):
        return self._get_translation(lang).gettext(msg)

def print_messages(domain, msg):
    domain = Domain(domain)
    for lang in all_languages():
        print(lang, ':', domain.get(lang, msg))

def main():
    print_messages('libc', 'No such file or directory')

if __name__ == '__main__':
    main()

Answer 4

The following example uses the translation directly, as shown in o11c's answer to allow the use of threads:

import gettext
import threading
import time

def translation_function(quit_flag, language):
    lang = gettext.translation('simple', localedir='locale', languages=[language])
    while not quit_flag.is_set():
        print(lang.gettext("Running translator"), ": %s" % language)
        time.sleep(1.0)

if __name__ == '__main__':
    thread_list = list()
    quit_flag = threading.Event()
    try:
        for lang in ['en', 'fr', 'de']:
            t = threading.Thread(target=translation_function, args=(quit_flag, lang,))
            t.daemon = True
            t.start()
            thread_list.append(t)
        while True:
            time.sleep(1.0)
    except KeyboardInterrupt:
        quit_flag.set()
        for t in thread_list:
            t.join()

Output:

Running translator : en
Traducteur en cours d’exécution : fr
Laufenden Übersetzer : de
Running translator : en
Traducteur en cours d’exécution : fr
Laufenden Übersetzer : de

I would have posted this answer if I had known more about gettext. I am leaving my previous answer for folks who really want to continue using _().

Answer 5

The following simple example shows how to use a separate process for each translator:

import gettext
import multiprocessing
import time

def translation_function(language):
    try:
        lang = gettext.translation('simple', localedir='locale', languages=[language])
        lang.install()
        while True:
            print(_("Running translator"), ": %s" % language)
            time.sleep(1.0)
    except KeyboardInterrupt:
        pass

if __name__ == '__main__':
    thread_list = list()
    try:
        for lang in ['en', 'fr', 'de']:
            t = multiprocessing.Process(target=translation_function, args=(lang,))
            t.daemon = True
            t.start()
            thread_list.append(t)
        while True:
            time.sleep(1.0)
    except KeyboardInterrupt:
        for t in thread_list:
            t.join()

The output looks like this:

Running translator : en
Traducteur en cours d’exécution : fr
Laufenden Übersetzer : de
Running translator : en
Traducteur en cours d’exécution : fr
Laufenden Übersetzer : de

When I tried this using threads, I only got an English translation. You could create individual threads in each process to handle connections. You probably do not want to create a new process for each connection.