Stack Overflow
Stack Overflow

Questions tagged [dataloader]

DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.

GitHub: dataloader

430 questions
1
vote
1 answer

Dicom data training failed by pytorch

I've got a problem about training the Pytorch models. I'm trying to train my Pytorch model using dicom data and nifti GT However, the size of the weight file is ridiculously small because model training is not performed normally. I used network…
mono
  • 69
  • 1
  • 9
1
vote
0 answers

python num_worker in data loader not working

I am trying to run the code from this repo but it never loads the images when num_worker is > 0. However, when I set num_worker = 0, it loads the data, but then all my losses are NaN. I am not sure why num_worker is not loading the data Is there an…
Sharhad Bashar
  • 69
  • 9
1
vote
1 answer

Generator dataloader in models with the same inputs and outputs

I am trying to implement a model whose output is the same as its input. It's a simple part of an extensive model, I deleted complicated parts. I wrote a generator dataloader for generating random numbers. def random_generator(): …
Zahra Honjani
  • 38
  • 6
1
vote
0 answers

DataLoader worker (pid(s) 5852, 3332, 1108, 5760) exited unexpectedly

I have encountered this error in the process of classifying and learning crime datasets, and performing num_workers = 0 from multiple communities results in occur NoneType' object is not subscriptable. I don't know where the problem is. pytorch…
MinJae
  • 31
  • 1
  • 4
1
vote
1 answer

How to Create a Custom Pytorch Dataset with Multiple Labels and Masks?

I'm trying to create a custom pytorch dataset to plug into DataLoader that is composed of single-channel images (20000 x 1 x 28 x 28), single-channel masks (20000 x 1 x 28 x 28), and three labels (20000 X 3). Following the documentation, I thought I…
morepenguins
  • 1,187
  • 10
  • 21
1
vote
0 answers

Fastai: Error Message in learn.predict when using custom Datablock/ Dataloader

I need some help with my Fastai pipeline. I want to do semantic segmentation on a 2 channel input image with augmentation. I adapted my procedure from the good introduction in medium I have 2 channel images that are saved as NumPy arrays (.npy) of…
sminize
  • 11
  • 3
1
vote
2 answers

Pytorch Dataset for video

Hi I made a video frames loader Dataset to be fed into a pytorch model. I want to sample frames from a video, but the frames should be uniformly sampled from each video. This is the class I came up with. I was wondering if there was any better…
3nomis
  • 1,175
  • 1
  • 9
  • 30
1
vote
2 answers

Jitterbit: target CSV-file created with only header although "do not create emtpy files" is checked

In Jitterbit Dataloader 10.37 I want to create CSV-files from Salesforce data but only if the query returns data. I checked "do not create empty files" on the target type local file but it is still creating a csv just with the header but with no…
Kerstin Vanselow
  • 11
  • 1
1
vote
0 answers

SQL pagination issues, and specifically with GraphQL data-loading

I've been doing some research on how to set up a new GraphQL API project, but am running into some basic conceptual? problems in trying to find out how to do pagination and nested database queries efficiently. I'd appreciate any pointers or…
Kelley van Evert
  • 1,063
  • 2
  • 9
  • 17
1
vote
1 answer

pytorch dataloader: to concatenate batch along one dimensions of the dataloader output

My dataset's __getitem__ function returns a torch.stft() M x N x D tensor with N being the audio input series with have variable length. Each item is read inside the __getitem__ function. I would like to have batches concatenated along the second…
matlio
  • 13
  • 4
1
vote
1 answer

How to write collate_fn correctly in my case?

Please, could you help me to find the solution to my problem. I want to write collate_fn to make my pictures the equal size, but I don't know how to implement it correctly. Colab: link Code: import pandas as pd import numpy as np from PIL import…
Tanya Gaychenkova
  • 101
  • 1
  • 7
1
vote
1 answer

pytorch: loading data from txt using dataloader with prefetch_factor

I have a 2D array with size (20000000,500) in a txt file. Since it is too large and it cannot fit in my computer, I will have to prefetch it and train my model using pytorch. I think I will need to use dataLoader with 'prefetch_factor'…
G-09
  • 345
  • 2
  • 13
1
vote
1 answer

cannot import name '_worker_manager_loop' from 'torch.utils.data.dataloader'

I am trying to import worker_manager_loop as follows: from torch.utils.data.dataloader import _worker_manager_loop I get an error that says: cannot import name '_worker_manager_loop' from 'torch.utils.data.dataloader' I looked up this problem and…
Nour
  • 197
  • 1
  • 3
  • 13
1
vote
1 answer

Create dataset out of x_train and y_train

How to put the x_train and y_train into a model for training? The x_train is a tensor of size (3000, 13). The y_train is of size (3000, 1) That is for each element of x_train (1, 13), the respective y label is one digit from y_train. if I…
lima0
  • 111
  • 6
1
vote
1 answer

How to accelerate batch-size data from memory when using dataloader

I am trying to use dataloader for training. The dataset is 150G, which are all .npz files. Due to the limitation of memory size, only one sample is read at a time from the disk. The following is part of the code. class VimeoDataset(Dataset): def…
dachun
  • 15
  • 5
1 2 3
…
28 29