2

I would like to call huggingface tokenizer in C# and wonder what might be the best way to achieve this. Specifically, I'd like to use mt5 tokenizer for CJK languages in C#.

I have seen certain nuget packages developed such as BertTokenizer in C#, However they do not have consistency with MT5 in huggingface.

exteral
  • 991
  • 2
  • 12
  • 33

0 Answers0