I've been trying to run Openai's universe-starter-agent example found here, However, using an m4.16xlarge instance on AWS with 32 workers, the agent's training result doesn't improve after 0.6 hours (over 30 minutes) while it is stated that "the agent is able to solve the same environment in 10 minutes" on the GitHub page.
The progress was monitored through TensorBoard. Please notice the example given in the GitHub was shown for the case of 16 workers, and it converges to an episode reward of 21 within 30 minutes, while for this case, with doubled number of workers and same amount of training time, the reward doesn't improve. I also took a look at the log and it seems there's no compiling error. The command I used to run the script is:
python train.py --num-workers 32 --env-id PongDeterministic-v3 --log-dir /tmp/pong
The only thing that I find a little dubious is when running the script, the following error was shown, but didn't abort the run. The error reads: "failed to connect to server"
Has anyone else run the starter agent, and/or run into similar issue? If so, how did you solve it?
Thanks!
