I'm trying to train a simple BLSTM in tensorflow, but I'm getting fatal errors during it. I'm using MacOs 12.0 Beta (Monterey), and tensorflow-macos 2.5 (with Apple Metal). Here are the errors I got:
Exception has occurred: InvalidArgumentError
Cannot assign a device for operation model/bidirectional/forward_simple_rnn/simple_rnn_cell_1/MatMul/ReadVariableOp: Could not satisfy explicit device specification '' because the node {{colocation_node model/bidirectional/forward_simple_rnn/simple_rnn_cell_1/MatMul/ReadVariableOp}} was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=2 requested_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' assigned_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' resource_device_name_='/job:localhost/replica:0/task:0/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Enter: GPU CPU
Equal: CPU
AssignSubVariableOp: GPU CPU
AssignVariableOp: GPU CPU
GreaterEqual: GPU CPU
FloorDiv: CPU
Sqrt: GPU CPU
NoOp: GPU CPU
Pow: GPU CPU
Mul: CPU
RealDiv: GPU CPU
Sub: GPU CPU
AddV2: GPU CPU
Cast: GPU CPU
Const: GPU CPU
VariableShape: GPU CPU
Identity: GPU CPU
SelectV2: GPU CPU
ReadVariableOp: GPU CPU
Square: GPU CPU
_Arg: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
model_bidirectional_forward_simple_rnn_simple_rnn_cell_1_matmul_readvariableop_resource (_Arg) framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
lookahead_lookahead_update_mul_5_readvariableop_resource (_Arg) framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
lookahead_lookahead_update_mul_8_readvariableop_resource (_Arg) framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
lookahead_lookahead_update_sub_10_readvariableop_resource (_Arg) framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
model/bidirectional/forward_simple_rnn/simple_rnn_cell_1/MatMul/ReadVariableOp (ReadVariableOp)
gradient_tape/model/bidirectional/forward_simple_rnn/while/VariableShape (VariableShape)
Lookahead/Lookahead/update/Identity (Identity) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/add/y (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/add (AddV2) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Cast_5 (Cast) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Pow (Pow) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Pow_1 (Pow) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_1/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_1 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv (RealDiv) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_2/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_2 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_3/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_3 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv_1/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv_1 (RealDiv) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv_2/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv_2 (RealDiv) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_4/y (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_4 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_1 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_2 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_5 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_6/y (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_6 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_7/y (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_7 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv_3 (RealDiv) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_8/y (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_8 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_3 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_9/y (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/sub_9 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv_4 (RealDiv) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_4 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv_5 (RealDiv) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Sqrt (Sqrt) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/GreaterEqual (GreaterEqual) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Const (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_5/ReadVariableOp (ReadVariableOp)
Lookahead/Lookahead/update/mul_5 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_6 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/add_1 (AddV2) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/AssignVariableOp (AssignVariableOp) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/ReadVariableOp_1 (ReadVariableOp) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_7 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_8/ReadVariableOp (ReadVariableOp)
Lookahead/Lookahead/update/mul_8 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Square (Square) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_9 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/add_2 (AddV2) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/AssignVariableOp_1 (AssignVariableOp) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/ReadVariableOp_2 (ReadVariableOp) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_10 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Sqrt_1 (Sqrt) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_11 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/add_3 (AddV2) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/truediv_6 (RealDiv) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/SelectV2 (SelectV2) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_12 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/AssignSubVariableOp (AssignSubVariableOp) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/group_deps (NoOp) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/add_4/y (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/add_4 (AddV2) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Cast_7/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Cast_7 (Cast) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Cast_8/x (Const) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/ReadVariableOp_4 (ReadVariableOp)
Lookahead/Lookahead/update/sub_10/ReadVariableOp (ReadVariableOp)
Lookahead/Lookahead/update/sub_10 (Sub) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_13 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/ReadVariableOp_5 (ReadVariableOp)
Lookahead/Lookahead/update/add_5 (AddV2) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/floordiv (FloorDiv) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/mul_14 (Mul) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/Equal (Equal) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/SelectV2_1/ReadVariableOp (ReadVariableOp)
Lookahead/Lookahead/update/SelectV2_1 (SelectV2) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/AssignVariableOp_2 (AssignVariableOp) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/SelectV2_2/ReadVariableOp (ReadVariableOp)
Lookahead/Lookahead/update/SelectV2_2 (SelectV2) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/AssignVariableOp_3 (AssignVariableOp) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/group_deps_1 (NoOp) /job:localhost/replica:0/task:0/device:GPU:0
Lookahead/Lookahead/update/group_deps_2 (NoOp) /job:localhost/replica:0/task:0/device:GPU:0
model/bidirectional/forward_simple_rnn/while/enter/_11 (Enter) framework assigned device=/job:localhost/replica:0/task:0/device:GPU:0
Func/model/bidirectional/forward_simple_rnn/while/cond/_0/input/_1831 (Identity) /job:localhost/replica:0/task:0/device:GPU:0
model/bidirectional/forward_simple_rnn/while/body/_1/model/bidirectional/forward_simple_rnn/while/simple_rnn_cell_1/MatMul/ReadVariableOp (ReadVariableOp)
Func/model/bidirectional/forward_simple_rnn/while/body/_1/input/_1864 (Identity) /job:localhost/replica:0/task:0/device:GPU:0
Func/model/bidirectional/forward_simple_rnn/while/body/_1/output/_1894 (Identity)
[[{{node model/bidirectional/forward_simple_rnn/simple_rnn_cell_1/MatMul/ReadVariableOp}}]] [Op:__inference_train_function_12237]
File "/Users/machine/Projects/train/src/problem_BLSTM.py", line 98, in simple_BLSTM
history = model.fit(train, steps_per_epoch=len(train), epochs=num_epochs, shuffle=True,
File "/Users/machine/Projects/train/src/problem_BLSTM.py", line 107, in main
simple_BLSTM(dataset)
File "/Users/machine/Projects/train/src/problem_BLSTM.py", line 112, in <module>
main()
And here is the code that originates the above errors:
(Note: I already tried with and without tf.config.set_soft_device_placement(True) but I always get the same errors.
Any idea on how can I solve this?
