1

I have created a simple Question Answering application with AllenNlp. It used to run smoothly with no warnings, but now, for every token in the passage, this spacy warning is printed in the console:

[WARNING] [W108] The rule-based lemmatizer did not find POS annotation for the token 'X'. Check that your pipeline includes components that assign token.pos, typically 'tagger'+'attribute_ruler' or 'morphologizer'.

The system versions are: allenlp==2.1.0 spacy==3.0.5 Can anyone please help?


class PythonPredictor:
    def __init__(self):
        self.predictor = Predictor.from_path(
            "https://storage.googleapis.com/allennlp-public-models/bidaf-elmo-model-2020.03.19.tar.gz")

    def predict(self, passage, question):
        prediction = self.predictor.predict(
            passage=passage, question=question
        )
        return prediction["best_span_str"]
Filippos Vlahos
  • 77
  • 2
  • 6

2 Answers2

1

Hard to be sure without more information about your code, but I suspect this is caused by upgrading from spaCy 2 to spaCy 3.

If you just want to get your old code running, you can downgrade spaCy; I recommend locking your version to avoid accidental updates.

The warning itself can be ignored if you aren't using lemmas. If you are using lemmas, it means that you need to make sure the lemmatizer has access to part of speech tags. If you are using English this means you need to enable the tagger and attribute_ruler pipeline components. You can see more about this here in the spaCy Discussions.

polm23
  • 14,456
  • 7
  • 35
  • 59
  • Thanks for your answer. I downgraded from verison 3 to 2.3.5 and now it crashes with the following warning: "UserWarning: [W031] Model 'en_core_web_sm' (3.0.0) requires spaCy v3.0 and is incompatible with the current spaCy version (2.3.5)." I never had this issue before and I was always using en_core_web_sm so I doubt that the issue is related to upgrading spaCy – Filippos Vlahos Mar 26 '21 at 20:14
  • The models have versions too - if you downgrade spaCy you need to re-install the model and make sure it's a compatible version. Also, to be clear, when you say "always", how long have you had this service? – polm23 Mar 27 '21 at 07:01
  • I downgraded the model to a version that is compatible. Then, I had to downgrade spacy-transformers as well as there was an error about that. Now I'm getting other errors related to version incompatibilities so I think I'll just go back to how things were and live with that warning. Answering to your question, I've had this service since October 2020 – Filippos Vlahos Mar 27 '21 at 18:58
0

I stopped getting the warnings when I updated AllenNlp to version 2.2.0.

Filippos Vlahos
  • 77
  • 2
  • 6