XLNetTokenizer requires the SentencePiece library but it was not found in your environment

Question

I am trying to implement the XLNET on Google Collaboratory. But I get the following issue.

ImportError: 
XLNetTokenizer requires the SentencePiece library but it was not found in your environment. Checkout the instructions on the
installation page of its repo: https://github.com/google/sentencepiece#installation and follow the ones
that match your environment.

I have also tried the following steps:

!pip install -U transformers
!pip install sentencepiece

from transformers import XLNetTokenizer
tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased-spiece.model')

Thank you for your help in advance.

Anoop kottappuram · Accepted Answer · 2021-01-06T17:12:14.187

12

After the
!pip install transformers and !pip install sentencepiece
please restart your runtime and then execute all other codes.

edited Jan 06 '21 at 17:12

answered Jan 06 '21 at 16:50

Anoop kottappuram

184
1
5

Yes, it works, but I get this message "Calling XLNetTokenizer.from_pretrained() with the path to a single file or url is deprecated" – Ashok Kumar Jayaraman Jan 07 '21 at 06:01
1

Try this --> (1) from transformers import TFXLNetModel, XLNetTokenizer (2) xlnet_model = 'xlnet-large-cased' (3) xlnet_tokenizer = XLNetTokenizer.from_pretrained(xlnet_model) – Anoop kottappuram Jan 07 '21 at 07:49
Perfect. It works. Thank you @ Anoop kottappuram – Ashok Kumar Jayaraman Jan 07 '21 at 08:15
it doesn't work on windows, i did exactly the same – dato nefaridze Oct 05 '21 at 16:05
I tried `pip install simpletransformers` instead and it worked for my case. – Goh Jia Yi Oct 24 '21 at 06:17

score 0 · Answer 2 · answered Sep 11 '22 at 08:56

0

I got the same error in google colab. Restarting the runtime did it for me.

answered Sep 11 '22 at 08:56

bogdantmm

13
3

score 0 · Answer 3 · answered Jun 27 '23 at 06:06

0

Restart the runtime in colab did it for me. like the one mention aboved

answered Jun 27 '23 at 06:06

谢国平

1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 08 '23 at 09:18

XLNetTokenizer requires the SentencePiece library but it was not found in your environment

3 Answers3