DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
0
votes
1 answer
How to make a custom DataLoader return values and labels seperately?
I want to iterate over a custom DataLoader using batches with matching values and labels. Modification of PandasDataset described below is needed and since I copied it from online I do not have a great grasp of how it works
import torch
import…
Marius Kuzm
- 159
- 1
- 9
0
votes
1 answer
How to load data in root directory separately in dataloader pytorch
So, I'm trying to load this dataset in pytorch, I'm facing a problem while loading it.
As you can make out my checking the dataset that the directory looks somethings like this:
root
monet_jpg
monet_tfrec
photo_jpg
photo_tfrec
So, I want to…
Prithviraj Kanaujia
- 331
- 4
- 15
0
votes
1 answer
How to Load Patched Image into Dataloader to Feed on Neural Network
I have a little problem on understanding how patched image used to train deep learning network. I have some 2D image that relatively large (1700 x 3000 px) that used for semantic segmentation. To avoid high consumption of computational power I did…
Josh068
- 21
- 4
0
votes
1 answer
Connection with salesforce is slow from Talend
I am using the tSalesforceOutputBulkExec component in Talend to push records to salesforce. It is taking too much time to send 10 records to salesforce (more than 50 seconds).
I tried the same thing with the data loader but couldn't see any issue…
Pratik
- 144
- 2
- 12
0
votes
1 answer
graphql dataloader Cannot read property 'load' of undefined error
I would like to bring the location data that exists below the location_group table. You are trying to use DataLoader because n + 1 problem exists.
I thought it was done, but when I actually run the code, I get a Cannot read property 'load' of…
momo
- 281
- 1
- 2
- 12
0
votes
1 answer
What is the fastest way to load data from multiple csv files
I am working with multiple csv files, each containing multiple 1D data. I have about 9000 such files and total combined data is about 40 GB.
I have written a dataloader like this:
class data_gen(torch.utils.data.Dataset):
def __init__(self,…
Sarvagya Gupta
- 861
- 3
- 13
- 28
0
votes
0 answers
DataLoader: Image size is zero
When trying to load some Images for Data Science using:
DataLoader.from_folder(image_path)
I get following error:
File "", line 1, in
File…
skm
- 559
- 1
- 6
- 22
0
votes
1 answer
GraphQL: several field resolver
everyone!
Has anyone met such case in graphql: you need resolve several fields with several async sources like this?
type Entity {
# ---- dataSource1
fieldA: String
fieldB: String
# ---- dataSource2
fieldC: String
fieldD: Stirng
}
So…
0
votes
1 answer
Dataloader worker exited unexpectedly while running on Visual Studio. But runs okay on Google Colab
So I have this dataloader that loads data from hdf5 but exits unexpectedly when I am using num_workers>0 (it works ok when 0). More strangely, it works okay with more workers on google colab, but not on my computer.
On my computer I have the…
Manveru
- 177
- 2
- 9
0
votes
1 answer
Java DataLoader: Graphql query mismatches results
I am using boot-graphql-kick-start in my project. When I used Dataloader, the result is missmatching. where am I wrong? Can you help me?
Query:
query USE_ROLES_BY_PROJECT_ID($projectId: Int) { # parameter value =3
…
spatialNinja
- 3
- 1
- 3
0
votes
1 answer
Dataset size is smaller than memory, What's wrong with my code?
The following is part of the code, epoch=300, each npz file is 2.73M, but the batch size of my dataloader gives 64, a total of 8 gpuss, so a mini batch should be 64×8×2.73M≈1.1G, my actual memory is 128G. Even if it becomes larger after…
dachun
- 15
- 5
0
votes
0 answers
Efficient way to convert Pytorch dataset to loader/sampler that samples at least one point from each class
I'm trying to think of a fast and efficient way to convert a Pytorch dataset into a sampler that samples at least one of each class. So far I've iterated through the Pytorch dataset sequentially and creates a 2D tensor for each class (Batch x…
user12314098
0
votes
2 answers
Why DataLoader return list that has a different length with batch_size
I am writting a customed dataloader, while the returned value makes me confused.
import torch
import torch.nn as nn
import numpy as np
import torch.utils.data as data_utils
class TestDataset:
def __init__(self):
self.db =…
GuikunChen
- 60
- 8
0
votes
0 answers
Using PyTorch DataLoader makes me have the same losses, loops without any progress
So my issue is that when not using DataLoader, just creating 1000 epochs and doing the learning, the results are ok, and the losses drop to ~0.2. However, when trying to use DataLoader, the output is:
|batch|index|loss|
8 0 0.6232748031616211
..
8…
Hanna
- 1
- 2
0
votes
0 answers
pytorch custom urdu dataloader (text and images)
I am new on pytorch lib and trying to load a dataset i acquire online. Its a OCR in Urdu language. the structure of the dataset is
-Train_image:
-name1.jpg
-name2.jpg
-train_labels.csv
-column 1 (Files name)
-name1
-name2
-column 2 (text in the…
Hamza Ahmed
- 11
- 4