0

I have installed spacy 3.4.1 and en_core_web_lg model. Now I want to update(train) this model with more example for ORG entity. I tried using below code

nlp = spacy.load("en_core_web_lg)  # load existing spaCy model
print("Loaded model '%s'" % model)

if "ner" not in nlp.pipe_names:
    print("Creating new pipe")
    ner = nlp.create_pipe("ner")
    nlp.add_pipe(ner, last=True)

else:
    ner = nlp.get_pipe("ner")


for _, annotations in TRAIN_DATA:
    for ent in annotations.get('entities'):
        ner.add_label(ent[2])

move_names = list(ner.move_names)

other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]
with nlp.disable_pipes(*other_pipes):  # only train NER
    optimizer = nlp.create_optimizer()
    # batch up the examples using spaCy's minibatch
    for itn in range(n_iter):
        print("Starting iteration " + str(itn))
        random.shuffle(TRAIN_DATA)
        losses = {}
        for text, annotations in TRAIN_DATA:
            doc = nlp.make_doc(text)
            example = Example.from_dict(doc, annotations)
            nlp.update(
                [example],  # batch of texts
                drop=0.2,  # dropout - make it harder to memorise data
                sgd=optimizer,  # callable to update weights
                losses=losses)
        print("Losses", losses)

After training model is saved to /output directory. but when I check against the same examples which I provided while training. I did not see expexted output. code below

nlp = spacy.load("en_core_web_lg")
doc = nlp("Hcode Technologies / software engineer)
for ent in doc.ents:
  print(ent.label_,ent.text )

Output:
ORG Hcode Technologies / Software

but expected output is 
ORG Hcode Technologies

Please suggest me what I am doing wrong. As per My knowledge I think I am traing new model instead of training existing model en_core_web_lg ?

sanwar
  • 1
  • 1
  • See the example project that updates a model like this. https://github.com/explosion/projects/tree/v3/pipelines/ner_demo_update – polm23 Oct 28 '22 at 03:30

0 Answers0