I'm using tensorflow2 and dealing with tf_agents.replay_buffers.TFUniformReplayBuffer
.
As the comments of source codes suggest, each block of the replay buffer is consisted of several episodes:
The TFUniformReplayBuffer stores episodes in `B == batch_size` blocks of
size `L == max_length`, with total frame capacity `C == L * B`.
Storage looks like:
block1 ep1 frame1
frame2
...
ep2 frame1
frame2
...
<L frames total>
block2 ep1 frame1
frame2
...
ep2 frame1
frame2
...
<L frames total>
...
blockB ep1 frame1
frame2
...
ep2 frame1
frame2
...
<L frames total>
What I need to do is to manually write into a replay buffer, frame by frame. Each frame is an instance of tf_agents.trajectories.Trajectory
.
I wonder whether there exists something serving as a breakpoint in TFUniformReplayBuffer to seperate each episode in a block? So that when I read data from the replay buffer with num_step>1
, I won't get a mixed sequence composed of frames from different episodes, say, the last frame of episode1 and the first frame of episode2. And, if there do exist such thing, how can I set it manually?
Thanks