Stack Overflow
Stack Overflow

Questions tagged [dataloader]

DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.

GitHub: dataloader

430 questions
9
votes
2 answers

How does Dataloader cache and batch database requests?

Looking at the DataLoader library, how is it caching and batching requests? The instructions specify usage in the following way: var DataLoader = require('dataloader') var userLoader = new DataLoader(keys =>…
terik_poe
  • 523
  • 4
  • 17
7
votes
2 answers

React router V6.5 : how to strongly type data loaders?

using React-Router V6 and I tried to strongly type the dataloader functions that are using params and also the useLoaderData hook. So far I have to do the following that is ugly : A- For useLoaderData, need to force the returnType : const profil =…
Jerome
  • 2,429
  • 2
  • 22
  • 37
7
votes
1 answer

Apollo Server: How to access 'context' outside of resolvers in Dataloader from REST API Datasource

hopefully someone can help me with this little problem, I just cannot figure it out right now. Problem Statement: I want to access 'context' for the sake of authentication in my DataLoader. This DataLoaderis defined in a seperate path /loaders. In…
ilikejs
  • 73
  • 1
  • 4
7
votes
0 answers

PyTorch Dataloader hangs when num_workers > 0

PyTorch Dataloader hangs when num_workers > 0. The code hangs with only about 500 M GPU memory usage. System info: NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1. The same issue appears with pytorch1.5 or pytorch1.6, codes are run in…
Zhang Yu
  • 559
  • 6
  • 15
6
votes
2 answers

PyTorch DataLoader uses same random seed for batches run in parallel

There is a bug in PyTorch/Numpy where when loading batches in parallel with a DataLoader (i.e. setting num_workers > 1), the same NumPy random seed is used for each worker, resulting in any random functions applied being identical across…
iacob
  • 20,084
  • 6
  • 92
  • 119
6
votes
1 answer

PyTorch: Shuffle DataLoader

There are several scenarios that make me confused about shuffling the data loader, which are as follows. I set the “shuffle” parameter to False on both train_loader and valid_loader. then the results I get are as follows Epoch 1/4 loss=0.8802 …
ohfufu
  • 138
  • 6
6
votes
0 answers

Optimize pytorch data loader for reading small patches in full HD images

I'm training my neural network using PyTorch framework. The data is full HD images (1920x1080). But in each iteration, I just need to crop out a random 256x256 patch from these images. My network is relatively small (5 conv layers), and hence the…
Nagabhushan S N
  • 6,407
  • 8
  • 44
  • 87
6
votes
2 answers

PyTorch dataloader shows odd behavior with string dataset

I'm working on an NLP problem and am using PyTorch. For some reason, my dataloader is returning malformed batches. I have input data that comprises sentences and integer labels. The sentences can either a list of sentences or a list of list of…
stackoverflowuser2010
  • 38,621
  • 48
  • 169
  • 217
6
votes
1 answer

RuntimeError: Can only calculate the mean of floating types. Got Byte instead. for mean += images_data.mean(2).sum(0)

I have the following pieces of code: # Device configuration device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu') seed = 42 np.random.seed(seed) torch.manual_seed(seed) # split the dataset into validation and test…
Mona Jalal
  • 34,860
  • 64
  • 239
  • 408
6
votes
2 answers

When does dataloader shuffle happen for Pytorch?

I have beening using shuffle option for pytorch dataloader for many times. But I was wondering when this shuffle happens and whether it is performed dynamically during iteration. Take the following code as an example: namesDataset =…
Jim Wang
  • 421
  • 6
  • 17
6
votes
2 answers

How to handle concurrent DbContext access in dataloaders / GraphQL nested queries?

I'm using a couple of dataloaders that use injected query services (which in turn have dependencies on a DbContext). It looks something like this: Field>( "Users", resolve: context => { var loader =…
x5657
  • 1,172
  • 2
  • 13
  • 26
6
votes
2 answers

How to ensure that a batch contains samples from all workers with PyTorch's DataLoader?

I want to know how to use torch.utils.data.DataLoader in PyTorch, especially in a multi-worker case. I found that one batch output from DataLoader always comes from a single worker. I expected that there is a queue in the DataLoader which stores…
ymfj
  • 91
  • 1
  • 3
6
votes
1 answer

How can I use a PyTorch DataLoader for Reinforcement Learning?

I'm trying to set up a generalized Reinforcement Learning framework in PyTorch to take advantage of all the high-level utilities out there which leverage PyTorch DataSet and DataLoader, like Ignite or FastAI, but I've hit a blocker with the dynamic…
Ken Otwell
  • 345
  • 3
  • 13
5
votes
1 answer

How can I get Absinthe and Dataloader to work together?

I have a GraphQL API that works just fine using conventional resolve functions. My goal is to eliminate the N+1 problem. To do so I've decided to use the Dataloader. I've done these steps to supposedly make the app run: I added these two functions…
bart-kosmala
  • 931
  • 1
  • 11
  • 20
5
votes
1 answer

How to use a Batchsampler within a Dataloader

I have a need to use a BatchSampler within a pytorch DataLoader instead of calling __getitem__ of the dataset multiple times (remote dataset, each query is pricy). I cannot understand how to use the batchsampler with any given dataset. e.g class…
DsCpp
  • 2,259
  • 3
  • 18
  • 46
1
2
3
…
28 29