0

I have an issue with hyperparameter optimization. I designed a simple NN and I am performing Hyperband class for hyperparameter optimization. Before executing the hyperparameter search, I want to see that if I give Hyperband model the same parameters as my simple NN design, do I get the same result as NN design? However, I couldn't get the same score.

To be more specific, I am giving into Hyperband the same parameters (in my case, I gave one value for searching and these are the same values in using NN) and my expectation was to get the same scores which are produced by the simple NN.

After parameter search completed, the created model for best parameters should be the same as the NN design, or am I missing something?

Is anyone have any experience with similar issue?

pelo
  • 1
  • What do you mean by score? Do you check the test accuracy/loss of your model (the output of Hyperband and the created one from the optimized hyperparameters) on the same dataset? If you use a GPU for evaluation, you will probably have slightly different results because GPU evaluation is non deterministic. Try disabling the GPU with os.environ["CUDA_VISIBLE_DEVICES"] = "-1" – jackve Jun 11 '21 at 10:07
  • Thank you @jackve for the respond. I would like to reply your questions; 1- What do you mean by score?--> I am calculating some classification metrics but because of two models are not exactly same I am getting different results. and my expectation was getting the same result for both models because two models are designed with same hyperparameter. 2- Do you check the test accuracy/loss of your model .. the same dataset? --> I am using the same dataset but accuracy/loss are different for both models. – pelo Jun 11 '21 at 12:38
  • (cont'd) Yes, I am using GPU, and as you said the results are slightly different so, I tried your suggestion which is os.environ["CUDA_VISIBLE_DEVICES"] = "-1" however, it doesn't still generate exactly same result. But this difference causes by GPU, it is okay because it is really minor difference. – pelo Jun 11 '21 at 12:38
  • cont'd) Also, I created a dummy project in [link](github.com/pelingundogdu/nn_vs_tuning) and the steps are same with a simple design. For ln[5] and ln[8], my exceptation was same output for loss/accuracy but it is not. But again, if this causes because of GPU evaluation is non deterministic, it is all ok. – pelo Jun 11 '21 at 12:39

0 Answers0