There is always performance difference between training set and testing test. I am wondering what is the threshold for this difference, which is acceptable or not? For example, maybe the score for training is 87% and for testing is 83%. The 4 % difference may be acceptable. But if 87% for training and testing is just 60%. This 20% may indicate the over-fitting issue. So I am wondering if there is any threshold for this?
Asked
Active
Viewed 117 times
1 Answers
0
Outside of schoolhouse, I am meaning in applied settings, the threshold is decided by context. It is possible to create an algorithm that predicts correctly 99% of the time. If we are a bank have you made money? You have no idea. This algorithm could predict every one dollar loan correctly and always missing ten million dollar loans. So this algorithms accuracy, precision, recall, F score...means nothing in this case. Work backwards from what is the business or application "ask". This decides threshold tolerances.
rayphaistos1
- 11
- 1
- 3