I am implementing a tensorflow model using the keras subclassing api, as mentioned in a previous post here:
Tensorflow Model Subclassing Mutli-Input
I am running into an error that is slowing my entire system, where it is retracing but I cannot narrow down where this error occurs, I do not use @tf.function anywhere in my code. I am looking for support on narrowing down the location of the error or how to solve it.
The exact error is:
WARNING: Logging before flag parsing goes to stderr. W0127 15:31:00.867754 4815351232 def_function.py:586] 5 out of the last 6 calls to .distributed_function at 0x157208d08> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
Looking into it further disabling eager execution is throwing this warning that may be a causing factor:
2020-01-27 16:11:42.781884: W tensorflow/c/c_api.cc:326] Operation '{name:'basic_parsing_model/time_distributed/bidirectional/backward_gru/while' id:542 op device:{} def:{{{node basic_parsing_model/time_distributed/bidirectional/backward_gru/while}} = While[T=[DT_INT32, DT_INT32, DT_INT32, DT_VARIANT, DT_FLOAT, ..., DT_VARIANT, DT_VARIANT, DT_VARIANT, DT_VARIANT, DT_VARIANT], _lower_using_switch_merge=true, _num_original_outputs=52, body=basic_parsing_model_time_distributed_bidirectional_backward_gru_while_body_708[], cond=basic_parsing_model_time_distributed_bidirectional_backward_gru_while_cond_707[], output_shapes=[[], [], [], [], [?,32], ..., [], [], [], [], []], parallel_iterations=32](basic_parsing_model/time_distributed/bidirectional/backward_gru/while/loop_counter, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/maximum_iterations, basic_parsing_model/time_distributed/bidirectional/backward_gru/time, basic_parsing_model/time_distributed/bidirectional/backward_gru/TensorArrayV2_1, basic_parsing_model/time_distributed/bidirectional/backward_gru/zeros, basic_parsing_model/time_distributed/bidirectional/backward_gru/strided_slice_1, basic_parsing_model/time_distributed/bidirectional/backward_gru/TensorArrayUnstack/TensorListFromTensor, basic_parsing_model/time_distributed/backward_gru/bias, basic_parsing_model/time_distributed/backward_gru/kernel, basic_parsing_model/time_distributed/backward_gru/recurrent_kernel, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_1, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_2, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_3, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_4, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_5, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_6, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_7, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_8, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_9, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_10, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_11, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_12, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_13, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_14, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_15, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_16, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_17, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_18, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_19, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_20, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_21, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_22, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_23, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_24, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_25, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_26, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_27, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_28, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_29, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_30, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_31, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_32, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_33, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_34, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_35, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_36, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_37, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_38, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_39, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_40, basic_parsing_model/time_distributed/bidirectional/backward_gru/while/EmptyTensorList_41)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session. 2020-01-27 16:11:46.854955: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
