0

We are developing a ML pipeline with TFX, the model is a Keras sequential DNN with mostly dense layers. We would like to customize the Evaluator component to compare the blessed and the candidate model by comparing their predictions on a test set of our choice. Therefore given the test set we want to ask both models for predictions and decide which model is the best by using a custom metric. Based on the official documentation, it seems that we should create a custom python file and provide it to the Evaluator component via the module_file parameter, however there are no information about how this file has to be. For example, which are the functions that the Evaluator component calls? (for example the trainer component calls run_fn). What output should this function return?

We are using:

  • TF Version: 2.6.3
  • TFX Version: 1.4.0

Thank you!

Sara C.
  • 1
  • 1
  • Can you (1) pass example_splits=["test_set_of_my_choice"] (2) define a custom metric in a python module (3) use that custom metric within the vanilla Evaluator via eval_config (see https://github.com/tensorflow/model-analysis/blob/master/g3doc/metrics.md) – Micah Smith May 31 '22 at 18:12
  • Thank you @MicahSmith ! So if I define my custom metric in eval_config and use the evaluator component in this way: `evaluator = Evaluator(examples=["test_set_of_my_choice"] , baseline_model=model_resolver.outputs['model'],model=trainer.outputs['model'],eval_config=eval_config)` then will new predictions also be asked to the blessed model to calculate the custom metric? So if I understand correctly, I just need to edit _eval_config_ without providig the _module_file_ parameter, is it right? – Sara C. Jun 01 '22 at 20:02
  • That's mostly right. `Evaluator(examples=example_gen.outputs['examples'], example_splits=['test_set_of_my_choice'], ...)`. In `eval_config`, you are defining your custom metric in terms of a fully-qualified python module name. So evaluator must be able to import from that module name at least - so you may not need `module_file` depending on whether your metric is importable or not – Micah Smith Jun 03 '22 at 15:10

0 Answers0