3

I am trying to find out the names of languages supported by Fasttext's LID tool, given these language codes listed here:

af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs bxr ca cbk ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh

I tried to map the ISO codes to each language, but it seems non-standard, either using ISO-639-1 or ISO-639-3. Does anyone have a list of language names for these codes, or know how to find them?
Wikipedia's list does not cover all of them either, so a manual mapping too did not help.

UPDATE: Opened an issue on GitHub.

blackgreen
  • 34,072
  • 23
  • 111
  • 129
Abhinav Sukumar Rao
  • 31
  • 2
  • which names make problem? In your [Wikipedia's list](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) you can find link to [ISO 639 macrolanguage](https://en.wikipedia.org/wiki/ISO_639_macrolanguage) and it has other codes - i.e `als`, `azb` – furas Nov 10 '22 at 10:34
  • in these links I couldn't find language for code `bpy` - but when I used Google to find it then it shows me page https://iso639-3.sil.org/code_tables/639/data – furas Nov 10 '22 at 10:43
  • BTW: at the bottom of this page I found also link to [download tables](https://iso639-3.sil.org/code_tables/download_tables) and it seems it has tables as file [iso-639-3.tab](https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3.tab) (`tab seperated values` similar to `.csv`) – furas Nov 10 '22 at 10:55

1 Answers1

2

Most codes I found in Wikipedia's list, but some of them I found on subpage ISO 639 macrolanguage

But using Google to find bpy I found page ISO 639 Code Tables and probably there are all codes.

There is even page Download and there is file with codes iso-639-3.tab (similar to .csv)

Using this file and pandas (function read_csv with tab as separator) I wrote this script

odes = 'af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs bxr ca cbk ce ceb ckb co cs cv cy da de diq dsb dty dv el eml en eo es et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh'
codes = codes.split(' ')

import pandas as pd

df = pd.read_csv('/home/furas/iso-639-3.tab', sep='\t')

print(df)

for code in codes:
    name = df[ (df['Id'].str.strip() == code) | (df['Part1'].str.strip() == code) ]
    
    if not name.empty:
        print(code, name['Ref_Name'].iloc[0])
    
    #if name.empty:
    #    print('unknown:', code)

And it gives me almost all languages - except bh, eml, nah.
You can see it if you use code with if name.empty:

EDIT:

As @tripleee mentioned in a comment:

furas
  • 134,197
  • 12
  • 106
  • 148
  • 1
    Quick Duck Duck Going gets me several resources which identify `bh` as Bihari; https://en.wikipedia.org/wiki/Bihari_languages says it's a deprecated code. https://en.wikipedia.org/wiki/Emilian-Romagnol_language similarly identifies `eml` as a deprecated code for this language. https://en.wikipedia.org/wiki/Nahuatl says `nah` is the ISO 639-2 code for this language. – tripleee Nov 10 '22 at 12:34
  • @tripleee thank you, I also used wikipedia to search some missing codes but I skiped this information - I kept only this part which I could automate :) But now I add links from your comments. – furas Nov 10 '22 at 14:18