0

I am getting the following error when evaluating a model that I trained in tensorflow

Traceback (most recent call last):
  File "./mesh.py", line 89, in <module>
    testing.test(config, network_path)
  File "/workspace/training/testing.py", line 54, in test
    model.evaluate(testing_dataset)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1379, in evaluate
    tmp_logs = test_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 823, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
    *args, **kwds))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:1224 test_function  *
        return step_function(self, iterator)

    TypeError: tf__update_state() got an unexpected keyword argument 'sample_weight'

I don't set or use sample_weight anywhere in my code so I have no idea where this error could be coming from. Has anyone ever come across an error like this or have any pointers on how I can debug it?

The model is a keras model with weights loaded from a checkpoint file saved using the ModelCheckpoint callback during training (using the fit function) and I am evaluating using the evaluate function.

I am using the tensorflow-gpu==2.3.0 pip package

Bidski
  • 435
  • 1
  • 4
  • 6
  • What metrics are you using when compile the model? Did you using any custom metrics? if you did, [this answer](https://stackoverflow.com/questions/61919774/unexpected-keyword-argument-sample-weight-when-sub-classing-tensor-flow-loss-c) should helpful – Mr. For Example Dec 11 '20 at 02:26
  • I am using custom metrics and loss functions, they all have `sample_weight=None` in the `__call__` function (for losses) and `update_state` function (for metrics) signatures – Bidski Dec 15 '20 at 23:35

0 Answers0