Understanding max_features parameter in RandomForestClassifier

Question

I am analyzing RandomForestClasifier and need some help.

max_features parameter gives the max no of features for split in random forest which is generally defined as sqrt(n_features). If m is sqrt of n, then no of combinations for DT formation is nCm. What if nCm is less than n_estimators (no of decision trees in random forest)?

example: For n = 7, max_features is 3, so nCm is 35, meaning 35 unique combinations of features for decision trees. Now for n_estimators = 100, will the remaining 65 trees have repeated combination of features? If so, won't trees be correlated introducing bias in the answer?

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

2

max_features parameters sets the maximum number of features to be used at each split. Hence, if there are p number of nodes, .
max_samples enforces sampling on datapoints from X. By default, it samples same size as that of the X.

From Documentation:

max_samples int or float, default=None

If bootstrap is True, the number of samples to draw from X to train each base estimator.

If None (default), then draw X.shape[0] samples.

Hence, the unique combination of tree that can be formed would be p! * nCm * (n+n-1)! / (n!(n-1)!)

For your examples, let us consider there are 10 nodes in each tree and 10 samples in your X.

10! * 7C3 * (19!/ 10! * 9!)
= 11732745024000.0

Hence, there won't be any bias for a reasonable size dataset.

edited Jun 20 '20 at 09:12

Community

1
1

answered Mar 11 '20 at 05:50

Venkatachalam

16,288
9
49
77

So don't we have a subset of defined no of features for each estimator in Random Forest at the beginning? Are all features considered for all estimators and then randomly selected at each node for split? – aditya aggarwal Mar 11 '20 at 06:12
yes, randomly selected at each node. From Documentation: If int, then consider max_features features at each split. – Venkatachalam Mar 11 '20 at 06:14
1

read more [here](https://scikit-learn.org/stable/modules/ensemble.html#parameters) – Venkatachalam Mar 11 '20 at 06:18
If bootstrap is true and max_sample, none, it'll draw no of samples = X.shape[0], so in this will there be any difference from one where bootstrap is false. – aditya aggarwal Mar 11 '20 at 06:37
Also could you please explain how did we get (n+n-1)!/(n!(n-1)!) in unique no of trees. Thanks! – aditya aggarwal Mar 11 '20 at 06:38
if bootstrap is `false`, then it is sampling without replacement. – Venkatachalam Mar 11 '20 at 06:43
`(n+n-1)!/(n!(n-1)!)` which is basically choosing n samples from n items with replacement. the usual formula is `(r+n-1)!/(r!(n-1)!)`. – Venkatachalam Mar 11 '20 at 06:44

Understanding max_features parameter in RandomForestClassifier

1 Answers1