DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
0
votes
1 answer
How to import git repo as a node module?
I use dataloader in a React project. It doesn't work with webpack5 so I've fixed the issue directly in the node_module source code. Because this repo is not actively maintained anymore, my pr won't be merged soon.
So I've forked the project, pushed…
DoneDeal0
- 5,273
- 13
- 55
- 114
0
votes
0 answers
How to sample along with another dataloader in PyTorch
Assume I have train/valid/test dataset with batch_size and shuffleed as normal.
When I do train/valid/test, I want to sample a certain number (called memory_size) of new samples from the entire dataset for each sample.
For example, I set batch_size…
Jongsu Liam Kim
- 717
- 1
- 7
- 23
0
votes
0 answers
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0
when I am trying to create a DataLoader object with transforms I get this error. I am not sure why there are issues with tensor dimensions as shown here:
---------------------------------------------------------------------------
RuntimeError …
0
votes
0 answers
TypeError: 'module' object is not callable self.transform
I'm trying to iterate on a custom dataset, and it's failed on image transform.
transform = transforms.Compose([
transforms.ToPILImage(),
transforms.Resize((255, 255)),
# transforms.PILToTensor()])
transforms.ToTensor(),
…
O.H
- 23
- 7
0
votes
0 answers
Pytorch Dataset. Once catch exception, items do not seem to be added to list
I have a dataset which contains huge amount of pictures (2 millions). A lot of pre-processing has been done and pictures are identified with id. Some of the ids do not exist, but they are generated (FYI, easier to code). This means that when I try…
MichiganMagician
- 273
- 2
- 15
0
votes
1 answer
Displaying Images from DataLoader with TensorFlow Lite Model Maker
I am following the tutorial for creating an image classifier using the TensorFlow Lite Model Maker found here: https://www.tensorflow.org/lite/tutorials/model_maker_image_classification#simple_end-to-end_example. Even though I am following the…
aq2mL
- 1
0
votes
1 answer
how to load one type of image in cifar10 or stl10 with pytorch
This is a very simple question, I'm just trying to select a specific class of images (eg "car") from a standard pytorch image dataset. At the moment the data loader looks like this:
def cycle(iterable):
while True:
for x in iterable:
…
plotka
- 63
- 1
- 8
0
votes
1 answer
FileNotFoundError: [Errno 2] No such file or directory: '2007_009096.jpg'
I recently made this class :
class RealPhotosDataset(Dataset):
def __init__(self, directory):
self.files = os.listdir(directory)
def __getitem__(self, index):
print(self.files[index])
img =…
David
- 393
- 3
- 4
- 11
0
votes
1 answer
creating a train and a test dataloader
I have actually a directory RealPhotos containing 17000 jpg photos. I would be interested in creating a train dataloader and a test dataloader
ls RealPhotos/
2007_000027.jpg 2008_007119.jpg 2010_001501.jpg 2011_002987.jpg
2007_000032.jpg …
Alex
- 103
- 1
- 7
0
votes
0 answers
how to get Single text prediction from the CustomisedBERT Classification + PyTorch NLP model with/without DataLoader
I have used BERT with HuggingFace and PyTorch and used DataLoader, Serializer for Training & Evaluation. Below is the code for that:
! pip install transformers==3.5.1
from transformers import AutoModel, BertTokenizerFast
bert =…
Deshwal
- 3,436
- 4
- 35
- 94
0
votes
1 answer
How to make PyTorch DataLoader from scratch?
Is it possible to recreate a simple version of PyTorch DataLoader from scratch?
The class should be able to return current batch based on the batch size.
For example, the code bellow only allows me to return one example at the time
X =…
BadDeveloper98
- 15
- 4
0
votes
1 answer
How to use PyTorch’s DataLoader together with skorch’s GridSearchCV
I am running a PyTorch ANN model (for a classification task) and I am using skorch’s GridSearchCV to search for the optimal hyperparameters.
When I ran GridSearchCV using n_jobs=1 (ie. doing one hyperparameter combination at a time), it runs really…
Leockl
- 1,906
- 5
- 18
- 51
0
votes
1 answer
How to feed back LSTM output to input in pytorch dataloader
I’m doing video prediction frame prediction using LSTM. How do I implement this in my custom dataloader?
LeparaLaMapara
- 61
- 6
0
votes
1 answer
I get a tensor of 600 values instead of 3 values for mean and std of train_loader in PyTorch
I am trying to Normalize my images data and for that I need to find the mean and std for train_loader.
mean = 0.0
std = 0.0
nb_samples = 0.0
for data in train_loader:
images, landmarks = data["image"], data["landmarks"]
batch_samples =…
Mona Jalal
- 34,860
- 64
- 239
- 408
0
votes
1 answer
Salesforce -Fire apex trigger only after complete data load
So here is the issue
We are loading data into a CustomObject__c using DataLoader.
Usually the no of records that are passed are 3.
Also, if there is any issue with the data passed, they run the dataloader again and pass the corrected data. Now, the…
user3353758
- 5
- 3