-1

Is there a good resource that clearly explains the difference between learning and memorisation of artificical neural networks - much better if the source contains mathematical explanations and/or proofs. I have looked online and through literature but most of them simply demonstrate so using examples, such as object detection or recognition. For example: this, this also this... the list is quite long. I find this insufficient. I know to some this question might sound a bit vague, but, does such a thing, as a threshold exist beyond which the system is simply memorising or has learned? Thank you very much.

guthik
  • 404
  • 1
  • 7
  • 15
  • As far as I am aware there isn't a fine line between overfitting and generalization, these concepts change dratically with the problem you are solving. – Mete Han Kahraman Nov 01 '18 at 13:53

1 Answers1

1

I don't know about any source of proofs, but neural networks are in nature very good classifiers for being overfit/overtrained.if you use a more complex network than the one you need(and you don't usually know what is it of course) you will probably overtrain the model.regularization and dropout are used to prevent or at least decrease the overfitting of neural networks.

when your model overfits it means it has only found a pattern for the specific distribution of samples similar to the ones you trained your model with, hence loosing the ability to generalize for all unseen inputs. this behavior is called memorization, memorization can also happen in cases when your samples/inputs have a order in them but you don't want the model to learn the order, and neural network use that order to classify samples.(e.g. inputs be sorted from low to mid to high) . shuffling the data is a good solution for this matter.

I only know you can find out if the model has memorized the train set in this way: training accuracy is very high, maybe 100% but test accuracy is much lower, and this much doesn't have a threshold, it is empirical, for a specific problem you may be even ok with a 75% accuracy but for another one even 95% be low.

leo
  • 802
  • 6
  • 15
  • I know, but i thought there's some sort of generalisation as to finding out whether the system is learning or just memorising mathematically. Not by simply evaluating its accuracy. – guthik Nov 02 '18 at 11:17
  • I see. I am not an expert but I really don't think there be such a thing that you are seeking. the problems which you solve through machine learning is almost always an ill-posed problem with very high dimensional hypothesis spaces that you can't visualize or understand well. only approach that exist is to provide some test data and see how well model works. there is no other way i am sure. – leo Nov 02 '18 at 14:40