2

I started using Optuna and on the second or third round my console is spammed by messages like this:

UserWarning: The reported value is ignored because this `step` 438 is already reported.

There were about 2400 of these, starting at "step 1" and going upward. Using

optuna.logging.set_verbosity(optuna.logging.ERROR)

Did not address the issue. This thread suggests that UserWarnings may not in general be subject to logging restrictions.

It is not clear (to me, at least) from digging into the code what is causing this warning.

David R
  • 994
  • 1
  • 11
  • 27
  • As in the message, you code calls `trial.report` at the same step multiple times. Could you share the minimal code in the question? – nzw0301 Jul 26 '22 at 20:01
  • 2
    Thanks @nzw0301, I don't call `trial.report` directly anywhere in my code. I do have `callbacks=[ LightGBMPruningCallback(trial, "binary_logloss")]` in my code as a parameter to the `fit()` method. I haven't tried to create a minimal example because I thought there was service code making the call in question. My code is nearly identical to that shown here: https://towardsdatascience.com/kagglers-guide-to-lightgbm-hyperparameter-tuning-with-optuna-in-2021-ed048d9838b5 – David R Jul 27 '22 at 03:41

1 Answers1

0

The reason for this message, given the procedure in the referenced article, is the loop inside the objective() function. In the first loop, the LightGBMPruningCallback reports the intermediate iteration results, that are needed to decide whether the trial should be pruned, to Optuna. For the second run, the callback again reports values for iterations already reported. The error occurs, since for Optuna it seems that within a single trial, n (=number of folds) values of the LightGBMPruningCallback for the very same iteration are recorded.

With iterations in mean the steps during gradient decent, so when Optuna typically reports decreasing loss values. Since the callback monitors and reports them, there shouldn't be redundant values reported for the very same iteration (1, 2, 3...)

You can fix this, by only calling the LightGBMPruningCallback in the very first fold and not for the others, so with something like:

for idx, (train_idx, test_idx) in enumerate(cv.split(X, y)):
[...]
if idx == 0:
    *calling the callback*
else:
    *without calling the callback*

This should trigger the callback when training on the very first fold and should suppress further reports for further folds. The drawback, here is, that you determine whether a trial should be pruned only by looking at the training behavior on the first fold [...].

I personally don't use LightGBMPruningCallback when performing cross-validation within an Optuna trial.

Hope this helps.

OliverHennhoefer
  • 677
  • 2
  • 8
  • 21