Keras-Tuner: Is it possible to use test/validation set in the objective/metric function?

Question

Is it possible to score/evaluate the model performance, using keras-tuner, based on the test set instead of the training set? I'm asking this, because as of now, my understanding is that the metric function used as objective in the tuner.search() uses only y_true and y_pred as the input parameters, and they both refer to the training set (correct me if I'm wrong).
So how can I use test data in my metric function?

Answer 1

Short answer: you cannot, neither should you, use test data metrics during hyper-parameter tuning. KerasTuner allows you to use validation data metrics as the objective, which I encourage. However, the final test should always be done after all tuning and training is complete, and should use none of the training or validation data.

To use a validation metric, simply tell KT that its objective is something beginning with val in the name.