I would like to use the tf-agents RL, but there's two environments: py_environment and tf_py_environment. The former uses numpy arrays for the states and actions while the latter uses tensors. My states have data type set -- clearly numpy can handle such dataset but tensors cannot. This means that I cannot use the tf_py_environment and I can only use py_environment.
But I read from the documentation that tf_py_environment can parallelize operations that is useful for training networks. However, I think it means that py_environment cannot do so. But my question is, can py_environment still train networks (just not parallel-wise)?
If that's not possible, then I guess I would have to resort to using tf_py_environment that uses tensors. But how can I have my states that are set be in a tensor? There's three different sets that I am keeping track of in the state, so it really is not possible to just extract all the elements and push them into a tensor.
So for example:
s1 = {'a', 'b', 'c'}
s2 = {'d'}
s3 = {'e', 'f'}
self._state = np.array([s1, s2, s3], dtype=set) # numpy handles sets; to be used for py_environment
self._state = tf.tensor(self._state, dtype=set) # not possible in tf; can't use tf_py_environment
