DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
2
votes
0 answers
OSError: [Errno 24] Too many open files in Windows 10 when using pytorch dataloader
I'm doing a model of a CNN network which takes images that are located in a directory "images", and each image is located inside that directory in a directory named as the ID of the image (ex. image 1 is in: images/1/1.png).
class…
2
votes
0 answers
Pytorch Parameter-based Cropping custom sub-images from DataLoader image
There are rectangular images in train & validation folders, and the images are accessed via Pytorch through DataLoader module. Without performing,
transforms.CenterCrop(size) or transforms.RandomCrop(size)
I need to preserve their original shape…
Prithwish Jana
- 317
- 2
- 11
2
votes
0 answers
Does it make sense to use DataLoader without a db?
I found out about DataLoader package reading GraphQL tutorial but all of the examples for DataLoader package corresponds to some databases.
Does it make sense to use DataLoader if my GraphQL layer is just a layer on top of the REST API?
A. Karnaval
- 727
- 2
- 8
- 12
1
vote
1 answer
HotChocolate v.13 [UseProjections] attribute does not work with DataLoaders
I have the following GrapqhQL query:
query {
listTenants {
totalCount
items {
tenantId
name
sites {
totalCount
items {
siteId
cmxName
cmxState
hosts(
…
Daniel Ninchev
- 51
- 7
1
vote
1 answer
Iterating over PyTorch DataLoader slower than direct dataset access
I am using PyTorch to train a machine learning model, and I have encountered a significant issue where iterating over the DataLoader is noticeably slower than directly accessing the dataset. My main goal is to speed up the data loading process…
triple_double
- 13
- 4
1
vote
1 answer
HotChocolate v.13 DataLoader approach with attributes and source generated code not working
I recently started experimenting with HotChocolate v.13. and I am having issues implementing the data loaders with the [DataLoader] attributes like shown in this video: Let's simplify DataLoader with Hot Chocolate 13. I am using EF Core 6 and DB…
Daniel Ninchev
- 51
- 7
1
vote
0 answers
Why do you inject batch functions into context when you implement dataloader?
There are often examples of injecting the batch function into the context when implementing dataloader in Golang, but I'm not sure why.
While I understand the concept of caching data per request in the dataloader mechanism, I'm not sure if it's…
anderson
- 415
- 1
- 4
- 8
1
vote
0 answers
Dataloader with a different batch size for each iteration for a deep learning project
For my deep learning project I need a cifar10 data loader (python) for my model which uses a varied batch size each iteration, unfortunately torch.utils.data.DataLoader uses a constant batch size throughout the training.
The batch size is randomly…
Naseem Yehya
- 29
- 4
1
vote
1 answer
Dealing with multiple datasets/dataloaders in `pytorch_lightning`
I'm dealing with multiple datasets training using pytorch_lightning. Datasets have different lengths ---> different number of batches in corresponding DataLoaders. For now I tried to keep things separately by using dictionaries, as my ultimate goal…
James Arten
- 523
- 5
- 16
1
vote
0 answers
How to enable Redis cache and Loaders using Nest and Mercurius
I'm using Nest and Mercurius to create a GraphQL API. So far I was able to create queries and resolvers, but now I want to enable Loaders and Cache to improve performance. The documentation is not clear on Nest on how to do that with GraphQL. On the…
pbonnefoi
- 263
- 5
- 16
1
vote
0 answers
Loading dataset/dataloader object onto GPU
I am running code from another repository, but my issue is general so I am posting it here. Running their code, I get the error along the lines of Expected all tensors to be on the same device, found two: cpu and cuda:0. I have already verified that…
nlp4892
- 61
- 7
1
vote
0 answers
How can I achieve graphql/dataloader like behaviour in AppSync?
I am currently building an AppSync API and after doing some research into how to load four different fields with two data sources (two fields per data source) I came across this question.
The response to this question seems to do a great job…
Mercury
- 594
- 1
- 12
- 28
1
vote
1 answer
PyTorch Dataloader across epoch
In PyTorch, a dataloader cursor is used to iterate over the data during training. The cursor keeps track of the current position within the dataset, and is used to retrieve the next batch of data for training. When training across multiple epochs,…
hsh
- 111
- 2
- 8
1
vote
1 answer
Pytorch Loading a custom dataset
Hey I've been seatching around the web looking for help on importing a custom image set, but every good tutorial seems to just MNIST which is fine, but I dont know how to translate code to a custom set.
I've got a folder structure like…
Viktor hj
- 11
- 1
1
vote
0 answers
dataloader PIL.UnidentifiedImageError: cannot identify image file, but the image is not corrupt
When trying to iterate through dataloader, it returns 'PIL.UnidentifiedImageError: cannot identify image file'.
Searched for similar question and answers tend to points to corrupted image. However those images can be opened with Image.open() and…
santoku
- 3,297
- 7
- 48
- 76