ELKI Kmeans clustering Task failed error for high dimensional data

Question

I have a 60000 documents which i processed in gensim and got a 60000*300 matrix. I exported this as a csv file. When i import this in ELKI environment and run Kmeans clustering, i am getting below error.

Task failed
de.lmu.ifi.dbs.elki.data.type.NoSupportedDataTypeException: No data type found satisfying: NumberVector,field AND NumberVector,variable
Available types: DBID DoubleVector,variable,mindim=266,maxdim=300 LabelList
    at de.lmu.ifi.dbs.elki.database.AbstractDatabase.getRelation(AbstractDatabase.java:126)
    at de.lmu.ifi.dbs.elki.algorithm.AbstractAlgorithm.run(AbstractAlgorithm.java:81)
    at de.lmu.ifi.dbs.elki.workflow.AlgorithmStep.runAlgorithms(AlgorithmStep.java:105)
    at de.lmu.ifi.dbs.elki.KDDTask.run(KDDTask.java:112)
    at de.lmu.ifi.dbs.elki.application.KDDCLIApplication.run(KDDCLIApplication.java:61)
    at [...]

Below is the ELKI settings i have used

score 1 · Answer 1 · answered Feb 05 '18 at 20:22

1

The error (which took me a bit to understand, when I saw it the first time) says that your data has the "shape"

variable,mindim=266,maxdim=300

I.e. some lines have only 266 columns, some have 300. This may be a file format issue, for example due to NaN, missing values, or similar bad characters.

You get that error if you try to run an algorithm like kmeans that assumes the data comes from a R^d vectorspace (that is the NumberVector,field requirement), because the input data is not meeting this requirement.

answered Feb 05 '18 at 20:22

Has QUIT--Anony-Mousse

76,138
12
138
194

Thanks you for looking into this. I checked the `numpy` array in Python as well as the exported `CSV` file. However, none of the values are missing and neither any value is NaN. This sounds strange, but i found the solution to this issue by opening the exported `CSV` file and doing `Save As` and saving again as a `CSV` file. While size of the original file is 437MB, the second file is 163MB. I am posting this as solution. Appreciate your help!! – StatguyUser Feb 06 '18 at 09:39
I think its because of the decimal points. Doing a `Save As` only rounds the decimals. – StatguyUser Feb 06 '18 at 10:26

StatguyUser · Accepted Answer · 2018-02-09T09:25:14.723

1

This sounds strange, but i found the solution to this issue by opening the exported CSV file and doing Save As and saving again as a CSV file. While size of the original file is 437MB, the second file is 163MB. I used the numpy function np.savetxt for saving the doc2vec vector. So it seems to be a Python issue instead of being ELKI issue.

Edit: Above solution is not useful. I instead exported the doc2vec output which was created using gensim library and while exporting format of the values were decided explicitly as %1.22e. i.e. the values exported are in exponential format and values have length of 22. Below is the entire line of code.

textVect = model.docvecs.doctag_syn0
np.savetxt('D:\Backup\expo22.csv',textVect,delimiter=',',fmt=('%1.22e'))

CSV file thus created runs without any issue in ELKI environment.

edited Feb 09 '18 at 09:25

answered Feb 06 '18 at 09:41

StatguyUser

2,595
2
22
45

You need to tell people then what you used. Also, how you generated the file in the first place. Because I doubt that if I "save as" in Notepad.exe, that this will change anything. – Has QUIT--Anony-Mousse Feb 07 '18 at 17:37
@Anony-Mousse, i have added the additional details. – StatguyUser Feb 08 '18 at 08:50
1

Thank you. The format string makes more sense. I am surprised that exponential format isn't default with numpy. – Has QUIT--Anony-Mousse Feb 08 '18 at 23:30

ELKI Kmeans clustering Task failed error for high dimensional data

2 Answers2

Linked

Related