6

I am trying to vectorize some data using 

sklearn.feature_extraction.text.CountVectorizer.

This is the data that I am trying to vectorize:

corpus = [
 'We are looking for Java developer',
 'Frontend developer with knowledge in SQL and Jscript',
 'And this is the third one.',
 'Is this the first document?',
]

Properties of the vectorizer are defined by the code below:

vectorizer = CountVectorizer(stop_words="english",binary=True,lowercase=False,vocabulary={'Jscript','.Net','TypeScript','SQL', 'NodeJS','Angular','Mongo','CSS','Python','PHP','Photoshop','Oracle','Linux','C++',"Java",'TeamCity','Frontend','Backend','Full stack', 'UI Design', 'Web','Integration','Database design','UX'})

After I run:

X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())
print(X.toarray())

I get desired results but keywords from vocabulary are ordered alphabetically. The output looks like this:

['.Net', 'Angular', 'Backend', 'C++', 'CSS', 'Database design', 
'Frontend', 'Full stack', 'Integration', 'Java', 'Jscript', 'Linux', 
'Mongo', 'NodeJS', 'Oracle', 'PHP', 'Photoshop', 'Python', 'SQL', 
'TeamCity', 'TypeScript', 'UI Design', 'UX', 'Web']

[
[0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
]

As you can see, the vocabulary is not in the same order as I set it above. Is there a way to change this? Thanks

smci
  • 32,567
  • 20
  • 113
  • 146
nedzad
  • 118
  • 9

1 Answers1

4

You passed the vocabulary as a set, which means that order doesn't matter anymore. Example:

{'a','b'} == {'b','a'}
>>> True

Hence scikit-learn reorders it using alphabetical order. In order to prevent this, you need to pass a list of your vocabulary:

vectorizer = CountVectorizer(stop_words="english",binary=True,lowercase=False,vocabulary=['Jscript','.Net','TypeScript','SQL', 'NodeJS','Angular','Mongo','CSS','Python','PHP','Photoshop','Oracle','Linux','C++',"Java",'TeamCity','Frontend','Backend','Full stack', 'UI Design', 'Web','Integration','Database design','UX'])

X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())
print(X.toarray()) 

>>> ['Jscript', '.Net', 'TypeScript', 'SQL', 'NodeJS', 'Angular', 'Mongo', 
'CSS', 'Python', 'PHP', 'Photoshop', 'Oracle', 'Linux', 'C++', 'Java', 
'TeamCity', 'Frontend', 'Backend', 'Full stack', 'UI Design', 'Web', 
'Integration', 'Database design', 'UX']

>>> [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0]
     [1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]
MaximeKan
  • 4,011
  • 11
  • 26
  • Thanks for the help, appreciate it! – nedzad May 15 '19 at 18:12
  • That's actually a subtle bug on CountVectorizer, please [report it as an sklearn bug](https://scikit-learn.org/stable/developers/contributing.html#submitting-a-bug-report-or-a-feature-request) – smci Jun 17 '20 at 01:19
  • @smci where is the bug here? – MaximeKan Jun 17 '20 at 08:54
  • The bug is that the user would not expect the set to be sorted alphabetically and converted/coerced into a list, the doc doesn't tell you it does (it only says vocabulary should be an iterable), neither is there a runtime message, AFAIK. – smci Jun 17 '20 at 09:03
  • There is 'vectorizer.get_feature_names()' to see the order of features in the vectorization matrix. There is no reason scikit would keep the order of words in a set given that set elements do not have an order by definition. – MaximeKan Jun 17 '20 at 09:11