I am creating a pytorch dataloader as
train_dataloader = DataLoader(dataset,
batch_size=batch_size, shuffle=True, num_workers=4)
However, I get:
This DataLoader will create 4 worker processes in total.
Our suggested max number of worker in current system is 2,
which is smaller than what this DataLoader is going to create.
Please be aware that excessive worker creation might get DataLoader
running slow or even freeze, lower the worker
number to avoid potential slowness/freeze if necessary.
How to determine programmatically the suggested max number of worker in current system? So that the same code can be run safely on arbitrary hardware?
