Stack Overflow
Stack Overflow

Questions tagged [dataloader]

DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.

GitHub: dataloader

430 questions
1
vote
1 answer

How can I access the next step data using DataLoader in PyTorch?

I am using a code that trains neural networks. The code uses the DataLoader of PyTorch to load the data for every iteration. The code looks as follows for step, data in enumerate(dataloader, 0): …
hanugm
  • 1,127
  • 3
  • 13
  • 44
1
vote
0 answers

How to use dataloader loadMany function properly?

I am trying to use dataloader's loadMany function. But I am getting error. I already search many where but not getting any solutions. Please help me. Here is my try- loader.js (Here I write Loader function)- //Models const {Subcategory} =…
Samia Sammi
  • 95
  • 2
  • 10
1
vote
2 answers

How to train pytorch model using large data file while using Dataloader?

I am using pytorch Dataset class and Dataloader to load data. The class and loader looks like the following. class Dataset(Dataset): def __init__(self): self.input_and_label = json.load(open(path_to_large_json_file)) # Large file …
Droid-Bird
  • 1,417
  • 5
  • 19
  • 43
1
vote
0 answers

GraphQL + SpringBoot + DataLoaders: what is the right pattern?

I have GraphQL + SpringBoot + MongoDB application, with graphql.java.kickstart.springboot ,and I'm having performance issues. The walking skeleton of this application was done the last year and I received it as it is. I'd like to add DataLoaders /…
Barbi
  • 139
  • 1
  • 11
1
vote
0 answers

Iterate Error in DataLoader module 'numpy' has no attribute 'expand_dim'

I want to iterate throw the trainloader but I face this problem, Any help in this error . for imgs, mask in trainloader: print(imags.shape) print(masks.shape) break AttributeError Traceback (most recent call…
Mohamed Zayton
  • 11
  • 2
1
vote
0 answers

Custom Dataset class for hierarchical dataset using “Local Classifier per Parent Node” technique

I’m working on an image classification model for hierarchical dataset using PyTorch implementation of EfficientNet. I am seeking help on how to edit my torchvision.datasets.ImageFolder and any other part if necessary to implement Local Classifier…
Tyler Kim
  • 181
  • 1
  • 11
1
vote
0 answers

efficient way to prepare time series with sliding window (save&read or make data every itr)

Time-series data have dimensions below C: Channel L: Length of time(sequence) I have L is over 200000 and C is over 200. In dataloader, data are def __getitems(self, idx): return data # data=(B, C, W), B:Batch size, W=sliding window size The…
SangGyu
  • 11
  • 1
1
vote
1 answer

Pytorch: How to make a custom Dataloader for CNN?

I'm trying to create my own Dataloader from a custom dataset for a CNN. The original Dataloader was created by writing: train_loader = torch.utils.data.DataLoader(mnist_data, batch_size=64) If I check the shape of the above, I get i1, l1 =…
user545642
  • 91
  • 1
  • 14
1
vote
1 answer

How to load data from multiply datasets in pytorch

I have two datasets of images - indoors and outdoors, they don't have the same number of examples. Each dataset has images that contain a certain number of classes (minimum 1 maximum 4), these classes can appear in both datasets, and each class has…
gutzcha
  • 122
  • 1
  • 11
1
vote
1 answer

how to load two dataset images simultaneously for train two streams(Pytorch)

i need load identical two dataset suppose one dataset has RGB images and another dataset contain same image with different processed(grey images) with same order same size, datasetA=[1.jpg,2.jpg,..........n.jpg] //…
Lanka
  • 357
  • 3
  • 13
1
vote
0 answers

Typescript + GraphQL Dataloader more strict types using generics or overloads

I'm using GraphQL's dataloader and I'd like to make my return types more strict based on the keys that are passed in. How could I accomplish this? Feeble attempt below. I know why it doesn't work, but not sure what to fix. (same code exists in ts…
Matt K
  • 4,813
  • 4
  • 22
  • 35
1
vote
0 answers

GraphQL DataLoader with multiple one to many relations

I'm struggling with a good title for this issue, so apologies. For that. What I'm trying to achieve; I have 3 Entities in use for this dataloader User Club Player Each User can create a Club User has a OneToMany Relationship with Club When a…
mcclosa
  • 943
  • 7
  • 29
  • 59
1
vote
1 answer

What is an efficient implementation of a custom map-style dataset for a hdf5 file with irregular structure?

I have a hdf5 file that contains picture of a certain number of people, from a certain number of source cameras, for many seconds. So it is like this: file[seconds][person][camera]. But this is quite irregular, such that for a given second, there…
Manveru
  • 177
  • 2
  • 9
1
vote
1 answer

TensorDataset error with dimensions and int not callabe?

I have some numpy arrays that I would like to pass into the TensorDataset from PyTorch, so it can be passed into the DataLoader for training in a neural network. These are the dimension of my train and test feature and targets: Feature train…
user10335564
  • 133
  • 3
  • 16
1
vote
0 answers

custom pytorch dataloader within fastai text classification

I am trying to go deeper in my understanding of fastai API and want to be able to implement some things in “pure” pytorch and then let fastai do all of the optimization tricks. I am trying simple text classification with my own dataloader…
Petr
  • 1,606
  • 2
  • 14
  • 39
1 2 3
…
28 29