Stack Overflow
Stack Overflow

Questions tagged [dataloader]

DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.

GitHub: dataloader

430 questions
0
votes
1 answer

default_collate: TypeError: batch must contain tensors, numbers, dicts or lists; found & KeyError: 0

Goal: To iterate over the dataloader, to access torch.Tensor object data['image'] for predictions, like so: for data in dataloader: image, slide, filename = data['image'], data['slide_id'], data['filename'] # predict I suspect the issue is…
DanielBell99
  • 896
  • 5
  • 25
  • 57
0
votes
0 answers

CPU RAM keeps increasing in every epoch and doesn't free up CPU RAM after every epoch, also doesn't increase much after first epoch

This is my code: class VideoEmbeddingDataset(Dataset): def __init__(self, pth_file,labels): self.pth_file = pth_file self.labels = labels self.device = torch.device("cuda:0" if torch.cuda.is_available() else 'cpu') …
Rabeya Akter
  • 31
  • 1
  • 1
  • 5
0
votes
2 answers

How can I get data and target from dataloader type?

def load_cifar10_data(datadir): transform = transforms.Compose([transforms.ToTensor()]) cifar10_train_ds = Cifar10FL(datadir, train=True, download=False, transform=transform) cifar10_test_ds = Cifar10FL(datadir, train=False,…
seni
  • 659
  • 1
  • 8
  • 20
0
votes
0 answers

Attaching DataLoaders to GraphQL context in NestJS

I have the following project setup where GraphQL module options are passed asynchronously using factory function and ConfigService. My objective is to attach DataLoaders to the GraphQL context while following this guide. As stated in the ‘Attaching…
Koios Kvasir
  • 143
  • 2
  • 6
0
votes
0 answers

Incorrect batch size being inferred using trainer.fit(), correct batch size in dataloader? What could be going wrong? [PyLightning]

I have a script to fine-tune a HuggingFace model that I wrote using PyLightning. I'm running into a problem where when I call trainer.fit(model, train_loader, val_loader) the batch size in the data-loader is the batch size of the train_loader + the…
skidjoe
  • 493
  • 7
  • 16
0
votes
0 answers

Is it possible to use dataLoader with multiple keys in GraphQL?

For GroupDataLoader by Hot Chocolate, you can only pass 1 key in .LoadAsync(). I think it is same for BatchDataLoader as well but I am intrested in group data loader. If we go by use cases then Group Data Loader works if I am doing look up using one…
GThree
  • 2,708
  • 7
  • 34
  • 67
0
votes
1 answer

What is the best approach to use filter on GraphQL queries?

I am trying to understand what is the best approach to utilize query with filters in GraphQL for my scenario. So far what I have learned is, GraphQL does not have filter concept. UI/Consumer can fetch records per their need. I have GroupDataLoader…
GThree
  • 2,708
  • 7
  • 34
  • 67
0
votes
1 answer

Why are there multiple DB calls with DataLoader?

Per my understanding, when I use DataLoader in GraphQL, it should hit DB once only with supplied key(s). This is regardless of the number of parallel calls. What am I doing wrong here? My scenarios: If I make 4 parallel calls with one ID then it is…
GThree
  • 2,708
  • 7
  • 34
  • 67
0
votes
0 answers

How to load TimeSeriesData from non-continuous csv files

I need to load a all csv files in a folder to train a time series model (PyTorch Lightening). The issue is that while the rows within a file are continuous (t, t+1, etc.), there is a break between files. How do I correctly deal with this? Do I pad…
dorien
  • 5,265
  • 10
  • 57
  • 116
0
votes
1 answer

GPU training not starting using yolov8

I have installed pytorch with gpu activation and then installed ultralytics package in order to run yolov8 on my gpu. When I start training it shows that the GPU is being used: Ultralytics YOLOv8.0.54 Python-3.9.16 torch-2.0.0 CUDA:0 (NVIDIA…
John Prescott
  • 3
  • 3
0
votes
0 answers

DataLoader can’t take tuples of tensors when the tensors are in different shapes

I'm currently working with a PyTorch model that has multiple tensor inputs. The first dimension of the tensors varies from sample to sample, and the second dimension remains fixed across all samples. To feed these tensors into the model, I'm using…
Sadjad Fakouri Baygi
  • 1
  • 1
0
votes
0 answers

How to configure the dataloader in pytorch to load batches of video-text pairs, with varying video frames

I created a custom video-text dataset class for my video and text dataset in pytorch. The dataset consist of video-text pairs, where each video has already been converted into frames and the frames are of varying length (video frames located in…
Ace Kay
  • 13
  • 3
0
votes
1 answer

How to retrieve Dataset object from DataLoader?

I have a PyTorch DataLoader and want to retrieve the Dataset object that the loader wraps around. If this is possible, how? Or does the dataset object only exist for pre-loaded datasets on torch? The end goal is to easily integrate data in…
Somniare
  • 3
  • 3
0
votes
0 answers

Pytorch training model with multiple Datasets on multi GPUS

I have multiple data generators(say 16), each of them are time related, such that I can only generate the data one by one (batch size of each generator are fixed as one). And I have 8 gpus, how can I train the model with all the data simutaneously,…
Tommy Yu
  • 1,080
  • 3
  • 11
  • 30
0
votes
0 answers

What is the correct approach to fetch cosmos records using .AddPooledDbContextFactory?

I am learning BatchDataLoader using Hot Chocolate. Per their guidelines, I am using .AddPooledDbContextFactory. Using Microsoft.Azure.Cosmos nuget package to support cosmos DB. Now, I am getting null when I'm trying to fetch Cosmos record with valid…
GThree
  • 2,708
  • 7
  • 34
  • 67
1 2 3
…
28 29