Subset of features on external memory

Asked May 06 '19 at 20:47

Active May 06 '19 at 20:47

Viewed 102 times

I have a large file that I'm not able to load so I'm using a local file with xgb.DMatrix. But I'd like to use only a subset of the features. The documentation on xgboost says that the colset argument on slice is "currently not used" and there is no metion of this feature in the github page. And I haven't found any other clue of how to do column subsetting with external memory.

I wish to compare models generated with different features subsettings. The only thing I could think of is to create a new file with the features that I want to use but it's taking a long time and will take a lot of memory... I can't help wondering if there is a better way.

ps.: I tried using h2o package too but h2o.importFile froze.

asked May 06 '19 at 20:47

user5029763

1,903
1
15
23

You could store your data in a sqlite database. Then you can write a query to only load a subset of the data into R. – Alex May 06 '19 at 21:00
Hi ALex, thank you for the response but my data is too big... I don't have enough RAM to load it into R, even if it's only a subset of my features... – user5029763 May 06 '19 at 21:10
You can do with the subset whatever you want. You don't have to read it into R. You can do whatever you want with it. – Alex May 06 '19 at 21:12
Xgboost supports that? – user5029763 May 06 '19 at 21:19
How did you create the `xgb.DMatrix` file? Do the same with the subset. – Alex May 06 '19 at 21:20
I pass the name of the file, but `xgb.DMatrix` only supports libsvm files. – user5029763 May 06 '19 at 21:25

Subset of features on external memory

0 Answers0