0

TensorFlow/Keras has multiple metrics to monitor but where are they defined? Please point to the documentation or github code where those strings have been defined.

tf.keras.callbacks.EarlyStopping(
    monitor="val_loss",     # <-----
    min_delta=0,
    patience=0,
    verbose=0,
    mode="auto",
    baseline=None,
    restore_best_weights=False,
    start_from_epoch=0,
)

Conclusion

There is no documentation nor definition from TF/Keras. We need to figure them out by searching around, picking up bits and pieces from multiple resources. It is considered as a documentation bug.

Terminologies and values to use should have been defined and documented before being used but not have been done for this case.

mon
  • 18,789
  • 22
  • 112
  • 205
  • "Please point to the documentation or github code where those strings have been defined." They're... strings. They don't need to be defined, just like the integer `0` doesn't need to be defined. What is defined is the **meaning of** a string like `"val_loss"` **to that specific function or class**, which is explained by the code **for that**. If you want to understand what can be passed to `tf.keras.callbacks.EarlyStopping` and what effect that will have, start by looking **that** up in the documentation. – Karl Knechtel Feb 17 '23 at 01:13
  • For example, if I try copying and pasting `tf.keras.callbacks.EarlyStopping` [into a search engine](https://duckduckgo.com/?q=tf.keras.callbacks.EarlyStopping), the [first result](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) I get is this official documentation... which seems to be **where your own code example comes from**. – Karl Knechtel Feb 17 '23 at 01:15
  • @KarlKnechtel, thanks for the comment. I am afraid I am not trying to understand what "val_loss" mean or how EarlyStop works for "val_loss". Rether, there should be a clear documentation what "string" can be specified. Define terms e.g. "val_loss" before using should be a technical common sense in my understanding. Please point to such documents if you happen to know, as it is the question here. – mon Feb 17 '23 at 02:55
  • 1
    The official documentation is what it is. If you are hoping to find some third-party documentation, we don't provide that service - it is [your responsibility](https://meta.stackoverflow.com/questions/261592) to look for information, and we [generally don't take](https://stackoverflow.com/help/dont-ask) questions asking for third-party resources. – Karl Knechtel Feb 17 '23 at 02:56
  • From https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping, we will have no idea what other string values can be specified to "monitor" argument. – mon Feb 17 '23 at 02:56
  • My advice is to bring this up on the project issue tracker. I agree with you that the documentation *should* say these things. This can be considered a bug against the documentation. – Karl Knechtel Feb 17 '23 at 02:57
  • @KarlKnechtel, thanks. It is at least clear for now there is no such document. – mon Feb 17 '23 at 02:58

1 Answers1

1

Please check documentation of ModelCheckpoint: https://keras.io/api/callbacks/model_checkpoint/

The „monitor“ parameter for EarlyStopping accepts the same possible strings as ModelCheckpoint. You can specify the prefix „val_“, than the metric is computed on validation set. If you don‘t specify this prefix, metric is computed on a training set.

To find the names of all possible metrics (besides „loss“), please see: https://keras.io/api/metrics/

Each metric has a name parameter, which can be used as value for EarlyStopping ‚monitor‘ parameter.

For models with several outputs you can specify the desired metric to monitor like here: multi-output keras model with a callback that monitors two metrics

Andreas Kaufmann
  • 581
  • 3
  • 8