I'm trying to use a PyGAD to train a Keras sequential model. Currently, it can find a solution just fine but it takes really long to do it.
I'm already using batching for the keras side, but what's killing performance is PyGAD. Just so you can get the picture, I'm using a population of a hundred individuals, with each having roughly 500 parameters. Each generation is taking about 10 secounds, but to run through the neural network 1600 cases takes on average 30 ms.
I've tried to used the parallel_processing property but I keep getting couple of errors. Sometimes I can run through 3 to 5 generation before I get an error. Other times it just freezes or gives me an error right away. Without parallel_processing, no errors.
I'm codding on Google Colab. I've used the !nohup command but it's no use.
These are the errors i've been getting:
InvalidArgumentError Traceback (most recent call last)
<ipython-input-100-62b9b1c718aa> in <cell line: 1>()
----> 1 ga_instance.run() # Executa o treinamento
2 ga_instance.plot_fitness() # Plota a evolução da fitness no decorrer das gerações
9 frames
/usr/local/lib/python3.9/dist-packages/tensorflow/python/framework/ops.py in raise_from_not_ok_status(e, name)
7260 def raise_from_not_ok_status(e, name):
7261 e.message += (" name: " + name if name is not None else "")
-> 7262 raise core._status_to_exception(e) from None # pylint: disable=protected-access
7263
7264
InvalidArgumentError: {{function_node __wrapped__StridedSlice_device_/job:localhost/replica:0/task:0/device:GPU:0}} Expected begin, end, and strides to be 1D equal size tensors, but got shapes [2], [1], and [1] instead. [Op:StridedSlice] name: strided_slice/
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
<ipython-input-38-62b9b1c718aa> in <cell line: 1>()
----> 1 ga_instance.run() # Executa o treinamento
2 ga_instance.plot_fitness() # Plota a evolução da fitness no decorrer das gerações
12 frames
/usr/local/lib/python3.9/dist-packages/tensorflow/python/framework/ops.py in raise_from_not_ok_status(e, name)
7260 def raise_from_not_ok_status(e, name):
7261 e.message += (" name: " + name if name is not None else "")
-> 7262 raise core._status_to_exception(e) from None # pylint: disable=protected-access
7263
7264
InvalidArgumentError: {{function_node __wrapped__Sum_device_/job:localhost/replica:0/task:0/device:GPU:0}} Invalid reduction dimension (-1 for input with 0 dimension(s) [Op:Sum]
