I am using weights and biases (wandb). The compute nodes on my remote server have no internet, and so I have to use the 'offline' mode and sync manually from the remote server login nodes. However it is very slow to sync. It takes the same time to sync if there has been many hours between the previous sync or only a few minutes. It's as if it is syncing the whole log each time, rather than just the updates. What can I do to speed it up?
I have tried both 'wandb sync --sync-all --include-synced' and 'wandb sync --sync-all', and they are as slow as each other.
