DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
1
vote
1 answer
Efficiently load large .npy files (>20GB) with Keras/Tensorflow dataloader
I am currently implementing a machine learning model which uses a rather heavy representation of data.
My dataset is composed of images. Each of these images is encoded into a (224, 224, 103) matrix, making the entire dataset very heavy. I store…
Megastore
- 73
- 6
1
vote
0 answers
URQL GraphQL cache problem with relational data from dataloader
I'm implementing cursor pagination to create an infinite scroll of posts, building upon URQL's simple pagination example from the docs. It works fine, I get the posts I want with all the information I want... until the cache is invalidated by…
Henrique Saias
- 45
- 1
- 7
1
vote
1 answer
PyTorch - Convert CIFAR dataset to `TensorDataset`
I train ResNet34 on CIFAR dataset. For a certain reason, I need to convert the dataset into TensorDataset.
My solution is based on this: https://stackoverflow.com/a/44475689/15072863 with some differences (maybe they are critical, but I don't see…
Dmitry
- 240
- 1
- 12
1
vote
1 answer
Problem with DataLoader GraphQL returning wrong data
I developed an API using node with GraphQL, when I add the DataLoaders, the resolvers starts returning wrong data, like I pass 4 Ids [3, 3, 1, 2], the batch function find in Database and return ordered like [1, 2, 3] and when return to the resolver,…
Ronaldo
- 11
- 1
1
vote
1 answer
Problems identifying images in pytorch
I am really new at deep learning and I am studying how to properly run neural networks using pytorch. Currently I am trying to read a dataset of images using the following code:
from torch.utils.data import Dataset, DataLoader
from torchvision…
1
vote
0 answers
How to build PyTorch dataset with mutliple arrays for different channel
I hope to know how to build up the dataset with multiple NumPy arrays as a data channel.
I have multiple arrays for 4-channel data as input and 1-channel data as an output.
For instance,
Example of Data Structure - Tensor Form
I think this form of…
Junhyeon Seo
- 11
- 1
1
vote
1 answer
Skorch RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same
I'm trying to develop an image segmentation model. In the below code I keep hitting a RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same. I'm not sure why as I've tried to load both my data…
mwpqz
- 23
- 5
1
vote
1 answer
PyTotch CIFAR-10 vs Kaggle CIFAR-10 : Totally different result for exactly same architecture on CIFAR-10
I have been learning PyTorch for some weeks. While I was practicing with CIFAR-10 dataset from PyTorch datasets, I also thought of practicing with ImageFolder class, so I found a version of Cifar-10 from Kaggle, where the images were foldered.(I you…
Sawradip Saha
- 1,151
- 11
- 14
1
vote
1 answer
Using a dataloader on TypeORMs many-to-many relation with GraphQL, query many-to-many
I started using TypeORM + type-graphql and trying to implement different use cases.
A Queue has a many-to-many relationship with User.
My problem is that I want to retrieve a list of users given a list of queue ids:
Queue…
Caruso33
- 183
- 1
- 1
- 10
1
vote
1 answer
DirectoryIterator in Keras
I want to load only images without labels in using data generator. I am training an autoencoder for image reconstruction on imagenet dataset.
The architecture of Imagenet dataset is similar to given…
1
vote
1 answer
Split DataLoader PyTorch
Is it possible to split a dataloader object of training dataset into training and validation dataloader?
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
train_dataset = datasets.ImageFolder(train_data_directory,…
Rajendra Sapkota
- 13
- 1
- 5
1
vote
1 answer
Using pg-promise Tasks with graphql and dataloader
First thing is to say that I have seen this question Sharing a pg-promise task across parts of an http request and read the answer. I'd like to revisit the question with some added complication.
I am working on a typescript graphql server on…
SFri
- 13
- 4
1
vote
0 answers
Cannot iterate through Pytorch Dataloader . Unequal tensor
I am training a BERT base model on the imdb dataset. However i am unable to iterate throught the Pytorch Dataloader .
Here is the code fully reproducible. Training file is the above mentioned csv.
import transformers
from sklearn import…
der_radler
- 549
- 1
- 6
- 17
1
vote
1 answer
pytorch dataloader default_collate argument use with to(device)
Ive been trying to integrate the to(device) inside my dataloader using to(device) as seen in https://github.com/pytorch/pytorch/issues/11372
I defined it on FashionMNIST in the following way:
device = torch.device('cuda:0' if…
JNYC
- 31
- 1
- 5
1
vote
1 answer
How to get the file name of image that I put into Dataloader in Pytorch
I use pytorch to load images like this:
inf_data = InfDataloader(img_folder=args.imgs_folder, target_size=args.img_size)
inf_dataloader = DataLoader(inf_data, batch_size=1, shuffle=True, num_workers=2)
And then:
with torch.no_grad():
…
Nice
- 97
- 1
- 6