DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
2
votes
1 answer
pythorch-lightning train_dataloader runs out of data
I started to use pytorch-lightning and faced a problem of my custom data loaders:
Im using an own dataset and a common torch.utils.data.DataLoader. Basically the dataset takes a path and loads the data corresponding to an given index the dataloader…
Asdf11
- 450
- 5
- 15
2
votes
1 answer
Python Dataset Class + PyTorch Dataloader: Stuck at __getitem__, how to get Index, Label and so on during Testing?
I have a, maybe small problem but I am stuck for quite a while now. Hope someone can help me with that. I am currently on a Kddcup99 dataset which I like to train via DeepLearning (CNN Network)
I have a "Dataset" Class which includes the Panda…
michael42
- 23
- 3
2
votes
0 answers
Message: Could not resolve promise in GraphQL api with DataLoader
I have created a basic api using graphical with PHP and mysql. when i have fire query using postman .
i am new grapql
{"query": "query { getPeople{name,pet{isDog,sound}} }" } then error come
but this query working fine {"query": "query {…
Jydipsinh Parmar
- 484
- 5
- 14
2
votes
1 answer
Pytorch default dataloader gets stuck for large image classification training set
I am training image classification models in Pytorch and using their default data loader to load my training data. I have a very large training dataset, so usually a couple thousand sample images per class. I've trained models with about 200k images…
swooders
- 141
- 1
- 8
2
votes
1 answer
Auth0: fetch data of group of users
I have an application with Auth0 and graphQl.
I use dataloaders to batching requests to DB. https://github.com/graphql/dataloader
For example, fetching data from DB looks like:
// Booking is Mongoose model
new DataLoader(
bookingIds =>…
Oleg
- 1,048
- 1
- 9
- 18
2
votes
0 answers
Pytorch DataLoader pipelining
I would expect DataLoader to load batches concurrently to the main process, filling up the buffer as soon as a batch is consumed from the buffer. However, when I track the utilization of GPU and order of loading vs execution I see some different…
dlsf
- 332
- 2
- 13
2
votes
0 answers
How to achieve a one to many relationship with GraphQL & DataLoader
I'm having a hard time figuring out why my graphQL & DataLoader setup isn't working and could use some help.
I have a User and a Orchestra type, and I would like to transform a User to populate its createdOrchestras field and do the same thing with…
Lorenzo Rivosecchi
- 83
- 1
- 10
2
votes
1 answer
Pass user specified parameters to DataLoader
I am using U - Net and implementing the weighting technique described in the papers from 2015 (U-Net: Convolutional Networks for Biomedical
Image Segmentation) and 2019 (U-Net – Deep Learning for Cell Counting, Detection, and Morphometry). In that…
vpap
- 1,389
- 2
- 21
- 32
2
votes
1 answer
DataLoader using Composite keys
I understand how dataLoader works with simple keys:
import DataLoader from 'dataloader';
import myService from './services/service';
export default () => new DataLoader((keys: any) => Promise.all(keys.map((key: string) => myService(key))));
Is…
Bhetzie
- 2,852
- 10
- 32
- 43
2
votes
1 answer
pytorch Crossentropy results in unmatched batch size
I was loading a image folder using dataloader
The image folder consists of three categories(labels) ie
'/root/ant/dsd.png'
'/root/ant/sfds.png'
...
....
'/root/bee/dsf.png'
....
..
'/root/whey/sfd.png'
Here there are three classes ant,bee,whey
By…
sathish akshaya
- 21
- 2
2
votes
0 answers
What is a good way to construct a PyTorch Dataset that loads (an unknown number of) slices from 3D images?
I am trying to construct a PyTorch Dataset that returns 21x512x512 slices from 3D images of shape ?x512x512. I know how many images there are, but I do not know how many slices there are in each image. Therefore, I would intuitively make the…
int elligence
- 37
- 11
2
votes
1 answer
How to define the __len__ method for PyTorch Dataloader when I have separate length datasets?
I'm currently loading in my data with one single dataset class. Within the dataset, I split the train, test, and validation data separately. For example:
class Data():
def __init__(self):
self.load()
def load(self):
with…
Sean
- 2,890
- 8
- 36
- 78
2
votes
2 answers
I don't get GraphQL. How do you solve the N+1 issue without preloading?
A neighborhood has many homes. Each home is owned by a person.
Say I have this graphql query:
{
neighborhoods {
homes {
owner {
name
}
}
}
}
I can preload the owners, and that'll make…
Sergio Tapia
- 9,173
- 12
- 35
- 59
2
votes
1 answer
Problem getting the labels of training-set
I have used train_test_split function to divide my data into X_train, X_test, y_train, y_test, and then used utils.data.DataLoader to feed it to my CNN but the problem is that I do not know how to access my labels tensor for making a confusion…
Hamed Fannan
- 31
- 1
2
votes
1 answer
Loading different types of data from CSV files with proper encoding in Python 3
I have a CSV file with different types of data.
For example: Some columns are categorical (e.g. name of city)
Some are numerical (e.g. price of a product)
I would like to read the data file using Python 3 in such a way that all the categorical data…
Ahsan Tarique
- 581
- 1
- 11
- 22