-1

I am creating a model using an advanced regression house price dataset. It has 37 numerical features. I want to make a feature selection by removing features with zero or very low variance. I used Variance Threshold, and it didn't remove any features.

for i in range(0,len(list(df.var()))):
  print(df.columns[i],df.var()[i])

Output- 
MSSubClass 1789.338306402389
MSZoning 589.7491687482642
LotFrontage 99625649.6503417
LotArea 1.9126794482991696
Street 1.2383223637883065
LotShape 912.2154126019891
LandContour 426.2328222558135
Utilities 32784.971167885175
LotConfig 208025.46846873628
LandSlope 26023.90777883106
Neighborhood 195246.40617940607
Condition1 192462.36170908928
Condition2 149450.07920371392
BldgType 190557.0753373038
HouseStyle 2364.204048090632
OverallQual 276129.63336259616
OverallCond 0.2692682171124828
YearBuilt 0.05700282610532444
YearRemodAdd 0.30350822011698775
RoofStyle 0.25289370651694854
RoofMatl 0.6654938173077709
Exterior1st 0.048548921667120055
Exterior2nd 2.6419033490756916
MasVnrType 0.41559474964087506
MasVnrArea 609.5825091487371
ExterQual 0.5584797243373708
ExterCond 45712.51022890529
Foundation 15709.813369543657
BsmtQual 4389.861203488976
BsmtCond 3735.5503258002063
BsmtExposure 859.5058709756354
BsmtFinType1 3108.889358915411
BsmtFinSF1 1614.215993315013
BsmtFinType2 246138.0553972849
BsmtFinSF2 7.309594674528473
BsmtUnfSF 1.763836649234308
TotalBsmtSF 6311111264.297451

These are the features and their variance. Based on this, what should be my threshold value

Mohamed Abdullah
  • 129
  • 1
  • 1
  • 8
  • (1) You cannot compare variances when your features have very different scales, and (2) this question, as currently stated, is off-topic for Stack Overflow, as it does not deal with a programming problem and may have multiple reasonable answers. Please review [What topics can I ask about here](https://stackoverflow.com/help/on-topic). – AlexK May 16 '21 at 06:45
  • If you are not getting the results you were expecting, perhaps [this](https://stackoverflow.com/questions/65929754/sklearn-variancethreshold-not-removing-low-variance-features) post might help. Otherwise, please show what you tried and what you expected. – AlexK May 16 '21 at 07:13

1 Answers1

0

First of all, each feature has different scales so you cannot compare Variances to each other. One technique you can use is to use scaling, like MinMaxScaler (not Standard scaler). This will allow you to compare the variances and you can choose a low threshold value like 0.05 or 0.03. But the threshold really depends. Try evaluating models for different thresholds and compare the results. Once you scaled your data, the threshold usually will be between 0 and 1.

If you want a deeper understanding into VT, check out this post.

Bex T.
  • 1,062
  • 1
  • 12
  • 28