-1

so my neural network (784 input nodes, 64 hidden nodes, 10 output nodes) has an accuracy of 95% when it recognises 28*28 images of hand written digits, but when i actually write a digit myself, it guesses right in about 1/5 of the cases. Does anyone have suggestions on why that is? thanks in advance

geomikeo
  • 19
  • 4
  • What resolution were the training images? and 28*28 what? pixels? mm? What net are you using? – Rohith Balaji Nov 21 '19 at 18:00
  • yeah 28*28 pixels so 784 of them in total – geomikeo Nov 21 '19 at 18:09
  • Well, if the accuracy is 95%, how is your net failing 80% of the time? If by 95% you are talking about the confidence level, then your net is predicting 95% wrong and it means you need to train better with a bigger data-set. – Rohith Balaji Nov 21 '19 at 18:13
  • nope, i checked everything and 95% is the percentage of correct guesses. also 1/5 was very approximative but the point is it's definitely not working 95% as it should be – geomikeo Nov 21 '19 at 20:37
  • I'm just going to take a wild guess here and assume you've trained your model on the MNIST dataset. I'm not sure how you're writing your own digits, but how they look compared to the training ones? In particular, is the color of the images converted accordingly (both graymaps, with the digits being primarily white along the strokes with just a bit of gray along the sides, not all primarily gray with hints of white everywhere). – Kazesui Nov 21 '19 at 21:06
  • it's not the actual mnist dataset from the official web page, but the dataset i'm using is pretty much identical. i'm writing my own digits by using paint.net. basically i write a digit on a 280*280 pixel canvas, which gets downsized (using pillow) to 28*28 pixels. after that, i read the values of each pixel. i've checked the array that contains said values and it seems to be just right: black background, primarily white pixels for the digit and some gray around the white pixels – geomikeo Nov 21 '19 at 21:15
  • Very well, that leads to a couple of other possible follow ups 1. Test your model on the actual mnist data, see how well your model does on those (sound compatible enough). 2. Unless you augmented your pictures with different rotations, I would plot the training and self-written images to make sure they all have the same orientation. (It's easy to get this one wrong depending on how the data is loaded, and if not trained for, it will lead to bad results). Should hopefully lead to some further clues as to what the problem is – Kazesui Nov 21 '19 at 21:34
  • thanks, will try tomorrow – geomikeo Nov 21 '19 at 22:07

1 Answers1

0

It is a common case mouse based drawing can create quite different signals than mnist 10k dataset. I have witnessed that myself too. The solution was using data augmentation. After adding several rotation, scaling, different noises, and some others from the literature to increase dataset by at least factor of 10, I was able to observe mouse based drawings were recognized, qualitatively around 80 - 85 % of a time.

Semih Korkmaz
  • 1,125
  • 13
  • 26
  • thanks for the advice but how are mouse based drawings different from other drawings? – geomikeo Nov 22 '19 at 13:51
  • If you check out the data collection process, it is a scan , here is an collection form example for this dataset as people are asked *handwrite* the numbers :https://www.nist.gov/sites/default/files/images/2019/04/27/sd19.jpg . Which is generally different than humans drawing same characters using mouse or touchpads. Even bamboo like pens. – Semih Korkmaz Nov 22 '19 at 14:16
  • oh, i see. one last question: if i go on the official mnist web page, the files that i'm downloading are not in the .csv format i need. if there's a way, how can i convert said files? thanks a lot for everything! – geomikeo Nov 22 '19 at 14:21
  • Well , there are lots of websites actually has CSV format of mnist , as in here : https://www.kaggle.com/oddrationale/mnist-in-csv – Semih Korkmaz Nov 22 '19 at 14:22
  • Back again (ugh). Is there any good augmented data set i can use? the only one i found is too big and my pc can't even load it in. – geomikeo Nov 23 '19 at 21:30
  • Ideally you should not create or load augmented datasets, instead you should use a data generator to augment them soon the fly (well, not in this case at least). If you use something like keras for neural network, it has an image data generator under preprocessing which you could use to augment the images during training. Way more versatile and requires less memory – Kazesui Nov 23 '19 at 22:39