DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
5
votes
0 answers
OSError: Can't read data (wrong B-tree signature). PyTorch shows this error when I read data from 2 .hdf5 files. What to do?
I m trying to run the following code in Pytorch:
import numpy as np
import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as data
class H5Dataset(data.Dataset):
def __init__(self,…
Deveshi
- 61
- 3
5
votes
2 answers
Is Pytorch DataLoader Iteration order stable?
Is the iteration order for a Pytorch Dataloader guaranteed to be the same (under mild conditions)?
For instance:
dataloader = DataLoader(my_dataset, batch_size=4,
shuffle=True, num_workers=4)
print("run 1")
for batch in…
information_interchange
- 2,538
- 6
- 31
- 49
4
votes
4 answers
How to resolve the error: RuntimeError: received 0 items of ancdata
I have a torch.utils.data.DataLoader. I have created them with the following code.
transform_train = transforms.Compose([
transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
…
odbhut.shei.chhele
- 5,834
- 16
- 69
- 109
4
votes
0 answers
Pytorch DataLoader output is not iterable
I am following some tutorials online about Pytorch and it seems at some point I do not get the expected output although I follow everything step by step.
import torch
import torchvision
from torchvision import datasets, transforms
Train =…
math
- 341
- 4
- 14
4
votes
1 answer
Displaying images loaded with pytorch dataloader
I am working with some lidar data images that I cannot post here due to a reputation restriction on posting images. However, when loading the same images using pytorch ImageFolder and Dataloader with the only transform being converting the images to…
CDWatson
- 41
- 4
4
votes
0 answers
How to store a set of images and labels dynamically in zarr format?
I have read zarr documentation,but could not able do this.
I want something like this tree format:
---dataset
--- sample1
--frames( 10 rgb frames)
--labels (1/0)
--- sample2
--frames( 10 rgb frames )
…
aminulpalash
- 156
- 6
4
votes
3 answers
How lazy should a GraphQL resolver be?
How lazy should a GraphQL resolver be?
For some context, here's a birds-eye of my architecture: GraphQL -> Resolvers -> |Domain Boundary| -> Services -> Loaders -> Data Sources (Postgres/Redis/Elasticsearch)
Past the domain boundary, there are no…
AdventureBeard
- 151
- 2
- 7
4
votes
1 answer
How to use DataLoader for Connections with Hot Chocolate GraphQL
I see that it's possible to use Data Loaders for root queries, but is it also possible to use Data Loaders for nested connections? In the example below, I want use a Data Loader for the rooms property. In the example request at the bottom, there…
Keith
- 41
- 1
- 3
4
votes
2 answers
Type error initializing DataLoader with Typescript
I'm trying to initialize an instance of DataLoader using the following code:
const authorLoader = new DataLoader(async (keys:string[]) => {
// Return an author for each book
});
I'm getting the following error:
Argument of type '(keys:…
Dan
- 641
- 1
- 7
- 17
4
votes
1 answer
How to improve performance on nested graphql connections when using pagination
I'm trying to implement some kind of a basic social network project. It has Posts, Comments and Likes like any other.
A post can have many comments
A post can have many likes
A post can have one author
I have a /posts route on the client…
Onur Önder
- 1,002
- 1
- 10
- 16
4
votes
0 answers
How to pass arguments into Dataloader batch query function?
I'm using graphql with Dataloader. I have the next query:
users {
products(isDeleted: false, categoryId: 23) {
name
price
}
}
I have property resolver:
productsLoader = new DataLoader(keys => this.productsService.load(keys));
async…
Sergiy
- 611
- 7
- 16
4
votes
1 answer
Loading custom dataset of images using PyTorch
I'm using the coil-100 dataset which has images of 100 objects, 72 images per object taken from a fixed camera by turning the object 5 degrees per image. Following is the folder structure I'm using:
data/train/obj1/obj01_0.png, obj01_5.png ...…
3venthoriz0n
- 117
- 2
- 12
4
votes
0 answers
How to split MNIST dataset into multiple subsets for distributed nodes using Pytorch?
I am implementing DistributedDataParallel training to a simple CNN for torchvision.datasets.MNIST simultaneously running on 3 distributed nodes. I want to partition the datasets into 3 non-overlapping subsets (A,B,C) and that should contain 20000…
Anjum
- 183
- 9
4
votes
1 answer
How to make the trainloader use a specific amount of images?
Assume I am using the following calls:
trainset = torchvision.datasets.ImageFolder(root="imgs/", transform=transform)
trainloader = torch.utils.data.DataLoader(trainset,batch_size=4,shuffle=True,num_workers=1)
As far as I can tell, this defines the…
Dr. John
- 273
- 3
- 13
3
votes
2 answers
KeyError when iterating through pytorch dataloader
I am trying to build a model with pytorch, and I want to use a customized dataset. So, I have a dataset.py which defines a class, MyData, which is a subclass of torch.utils.data.Dataset. Here's the file.
# dataset.py
import torch
from tqdm import…
pbbb
- 55
- 6