DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
2
votes
2 answers
PyTorch DataLoader Error: object of type 'type' has no len()
I'm quite new to programming and have now clue where my error comes from.
I got the following code to set up my dataset for training my classifier:
class cows_train(Dataset):
def __init__(self, folder_path):
self.image_list =…
Fredrik
- 21
- 1
- 3
2
votes
0 answers
R, Error: $ operator is invalid for atomic vectors in fastai
Running CollabDataLoaders() in implementing fastai using R in the code below:
library(fastai)
library(zeallot)
library(magrittr)
library(data.table)
URLs_MOVIE_LENS_ML_100k()
c(user, item, title) %<-% list('userId', 'movieId', 'title')
ratings =…
JasonSimilar
- 113
- 6
2
votes
1 answer
What should be the input shape for 3D CNN on a sequence of images?
https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html#conv3d Describes that the input to do convolution on 3D CNN is (N,Cin,D,H,W). Imagine if I have a sequence of images which I want to pass to 3D CNN. Am I right that:
N -> number of…
MichiganMagician
- 273
- 2
- 15
2
votes
1 answer
Custom dataset and dataloader
I am new to pytorch .
I have big dataset consist of two txt files one for data and other for target data .
In training file each line is list of length 340, In target each line is list of 136.
I would like to ask how i can define my dataset so I can…
No Na
- 27
- 2
- 7
2
votes
2 answers
Loading a single graph into a pytorch geometric data object for node classification
I have one graph, defined by 4 matrices: x (node features), y (node labels), edge_index (edges list) and edge_attr (edge features). I want to create a dataset in Pytorch Geometric with this single graph and perform node-level classification. It…
Qubix
- 4,161
- 7
- 36
- 73
2
votes
1 answer
Cannot have access to all data when enumerating over the dataloader
I defined a custom Dataset and a custom Dataloader, and I want to access all the batches using for i,batch in enumerate(loader). But this for loop gives me different number of batches in every epoch, and all of them are far smaller then the actual…
pyxies
- 374
- 1
- 4
- 10
2
votes
3 answers
How to merge two torch.utils.data dataloaders with a single operation
I have two dataloaders and I would like to merge them without redefining the datasets, in my case train_dataset and val_dataset.
train_loader = DataLoader(train_dataset, batch_size = 512, drop_last=True,shuffle=True)
val_loader =…
Johnpiton
- 41
- 1
- 1
- 6
2
votes
1 answer
what is the function of default_loader in torch?
import os
from torchvision.datasets.folder import default_loader
from torch.utils.data import Dataset
class Sample_Class(Dataset):
def __init__(self, root, train=True, transform=None, loader=default_loader):
self.root =…
sai_varshittha
- 193
- 1
- 12
2
votes
0 answers
Pytorch dataloader with iterable dataset stops after one epoch in multiprocessing mode
I have a dataloader that is initialised with a iterable dataset. I found that when I use multiprocessing (i.e. num_workers>0 in DataLoader) in dataloader, once the dataloader is exhausted after one epoch, it doesn't get reset automatically when I…
Tony
- 445
- 6
- 13
2
votes
1 answer
Speeding up the trainning - RNN with LSTM in PyTorch
I am trying to train a LSTM for energy demand forecast but it takes too long. I do not understand why because the model looks “simple” and there is no much data. Might it be because I am not using the DataLoader? How could I use it with RNN since I…
jccarrasco
- 45
- 9
2
votes
1 answer
Implementing In-memory cache alongside GraphQL Dataloaders
I'm looking into adding in-memory cache, like redis, to my application, and I'm having some issues understanding how it all fits together.
const postLoader = new DataLoader(async keys => {
// ... sql code to get multiple posts by ids
})
const…
Diyan Slavov
- 353
- 2
- 11
2
votes
1 answer
In Apollo GraphQL how to access Datasources inside Dataloader?
I'm new to GraphQL and I started using Apollo GraphQL. To test this basically I'm wrapping the GraphQL around Star Wars API (SWAPI.dev). I'm using Apollo RESTDataSource to make calls to Star Wars API.
In 'People' resource there's an array of…
Liteniro007
- 35
- 5
2
votes
1 answer
Checking the contents of python dataloader
This is probably a simple question, but how see how the contents of this standard data loader looks like:
from torchtext import datasets
import random
train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)
train_data, valid_data =…
dorien
- 5,265
- 10
- 57
- 116
2
votes
1 answer
Couldn't close file error when using pytorch enumerate
I seem to get an error when calling the pytorch enumerate method. From googling just the error message it seems to be a problem with my operating system, macOS catalina. Is there a solution to this that I just couldn't find and/or is there an…
Trevor Russell
- 59
- 1
- 3
2
votes
1 answer
PyTorch: how to apply another transform to an existing Dataset?
This is a code example:
dataset = datasets.MNIST(root=root, train=istrain, transform=None) #preserve raw img
print(type(dataset[0][0]))
#
dataset = torch.utils.data.Subset(dataset, indices=SAMPLED_INDEX) # for…
graphitump
- 631
- 1
- 5
- 13