0

my question is related to camemBERT model (french version of BERT) and its Tokenizer :

Why every word of the vocabulary has a "▁" character before ? For example, it's not "sirop" but "▁sirop" (sirop => syrup).

from transformers import CamembertTokenizer
tokenizer = Camembert.Tokenizer.from_pretrained("camembert-base")
voc = tokenizer.get_vocab() #Vocabulary of the model

print("sirop" in voc) # Will display False
print("▁sirop" in voc) # Will display True

Thank you for answering :)

TeDataPro
  • 3
  • 1

1 Answers1

1

If I understand it correctly the CamembertTokenizer uses this special character from SentencePiece, see the source code.

SentencePiece on the other hand uses Subword Tokens (splitting of words into smaller tokens), but to internally always keep track of what is a "real" split between words (where there was a whitespace) and what is a Subword splitting, they use this character before the start of each "real" token, follow-up subword tokens (but not punctuation) don't have this token, see the Explaination in the Github Repository. Basically the whitespace is always part of the tokenization, but to avoid problems it is internally escaped as "▁".

They use this example: "Hello World." becomes [Hello] [▁Wor] [ld] [.], which can then be used by the model and later transformed back into the original string (detokenized = ''.join(pieces).replace('▁', ' ')) --> "Hello World." without ambiguity and without having to save the original string seperately.

ewz93
  • 2,444
  • 1
  • 4
  • 12
  • Thanks for your answer. If I understand it correcly, it's to distinguish between real words (for example "_je" in 'je mange'=>'i am eating') from subwords (for example "je" in 'jeudi'=>'thursday') ? – TeDataPro Mar 25 '22 at 15:10
  • Yeah pretty much. To be precise it's about knowing which split is a split on a whitespace and which split is a subword/punctuation split. I think in your example both might have a "▁" before them though since both have a real whitespace before them. Subtokens like the "udi" would not one before them. I don't know exactly how the tokens are used to build the vocabulary though, so how they are escaped and what goes into the actual vocubulary is different question. – ewz93 Mar 25 '22 at 15:19