1

I try to convert my IOB (token-per-line NER) files (train/test) to Spacy 3 binary format.

Example of input format (with separator "\t", no blanklines and encoding utf-8) :

Département B-LOCATION
des I-LOCATION
Bouches-du-Rhône    I-LOCATION
.   O
Port    B-INSTALLATION
de  I-INSTALLATION
la  I-INSTALLATION
Ciotat  I-INSTALLATION
.   O
Avant-projet    O
du  O
môle    B-INSTALLATION
Bérouard    I-INSTALLATION
au  O
port    B-INSTALLATION
de  I-INSTALLATION
La  I-INSTALLATION
Ciotat  I-INSTALLATION
.   O

when I run :

!python -m spacy convert -c iob -s -n 10 -b fr_core_news_sm /content/ner4archives_v0_train.iob .

!python -m spacy convert -c iob -s -n 10 -b fr_core_news_sm /content/ner4archives_v0_test.iob .

I have this error :

ValueError: [E903] The token-per-line NER file is not formatted correctly. Try checking whitespace and delimiters. See https://spacy.io/api/cli#convert

I saw the git directory with example data : https://github.com/explosion/spaCy/tree/master/extra/example_data/ner_example_data ; but I cannot find the difference between my data and examples.

I try to reformat my file with different kind of separators ("\t", " ", "|"), I have always the same error. In addition, i tested if I have empty tokens or labels, it is not.

anyone with leads ? thanks in advance.

Lter
  • 43
  • 11

1 Answers1

1

I think you're using SpaCy with version other than 3.X, I could create .spacy binary file using your input without any trouble after download fr_core_news_sm.

command:

python -m spacy convert -c iob -s -n 10 -b fr_core_news_sm C:/git/sof/test.iob .

output:

[i] Auto-detected token-per-line NER format
[i] Grouping every 10 sentences into a document.
[i] Segmenting sentences with parser from model 'fr_core_news_sm'.
[+] Generated output file (1 documents): test.spacy

SpaCy version : 3.1.3

meti
  • 1,921
  • 1
  • 8
  • 15