Dataloader worker exited unexpectedly while running on Visual Studio. But runs okay on Google Colab

Question

So I have this dataloader that loads data from hdf5 but exits unexpectedly when I am using num_workers>0 (it works ok when 0). More strangely, it works okay with more workers on google colab, but not on my computer. On my computer I have the following error:

Traceback (most recent call last): File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 986, in _try_get_data data = self._data_queue.get(timeout=timeout) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\queues.py", line 105, in get raise Empty _queue.Empty

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "", line 2, in File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 517, in next data = self._next_data() File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 1182, in _next_data idx, data = self._get_data() File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 1148, in _get_data success, data = self._try_get_data() File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 999, in _try_get_data raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e RuntimeError: DataLoader worker (pid(s) 12332) exited unexpectedly

Also, my getitem function is:

def __getitem__(self,index):
  desired_file = int(index/self.file_size)
  position = index % self.file_size 

  h5_file = h5py.File(self.files[desired_file], 'r')

  image = h5_file['Screenshots'][position]
  rect = h5_file['Rectangles'][position]
  numb = h5_file['Numbers'][position]

  h5_file.close()

  image = torch.from_numpy(image).float() 
  rect = torch.from_numpy(rect).float() 
  numb = torch.from_numpy( np.asarray(numb) ).float()


  return (image, rect, numb)

Does anyone have any idea what can be causing this empty queue?

score 0 · Accepted Answer · answered Jun 05 '21 at 07:59

0

Windows can't handle num_workers > 0 . You can just set it to 0 which is fine. What also should work: Put all your train / test script in a train/test() function and call it under if __name__ == "__main__": For example like this:

class MyDataLoder(torch.utils.data.Dataset):
    train_set = create_dataloader()
    . . . 

def train():
    test_set = create_dataloader()
    . . .

def test():
    . . .

if __name__ == "__main__":
    train()
    test()

answered Jun 05 '21 at 07:59

Theodor Peifer

3,097
4
17
30

I eventually solved the problem by using linux instead, where i didnt have the problem. But now if I need to run on windows i know what to do. Thanks! – Manveru Jun 07 '21 at 08:37

Dataloader worker exited unexpectedly while running on Visual Studio. But runs okay on Google Colab

1 Answers1