DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
0
votes
1 answer
How to use Data Loader when parent contain IDs array?
I am trying to implementing data-loader in my project-
Here is my code-
subCategory: async (parent, _, {loaders}) => {
console.log(parent);
// I try it but getting errors
const subcategory = await…
user17578307
0
votes
1 answer
OSError: [Errno 22] Invalid argument | _pickle.UnpicklingError: pickle data was truncated
When trying to train a ResNet I get this error. Any help as to why this happens would be appreciated.
This happens when I try to iterate through the Dataloader:
File "C:\Users\JCout\AppData\Local\Temp/ipykernel_2540/2174299330.py", line 1, in…
Jonathan Couture
- 19
- 7
0
votes
1 answer
Calculate loss in Pytorch with convolutional network only for image section
I have a convolutional neural network that predicts the following frames to a sequence. I have added a border to the image in different colors to give the network additional information. But of course, this additional information is nothing the…
Pia Lüdemann
- 3
- 3
0
votes
1 answer
Pytorch: Add information to images in image prediction
I would like to add information to my current dataset. At the moment, I have six-frame sequences in folders. The DataLoader reads all 6 and uses the first 3 for predicting the last 1/2/3 (depending on how many I tell him to). This is the function…
Pia Lüdemann
- 3
- 3
0
votes
1 answer
Turning variable into a torch Tensor. Afterwards tensor is empty / has no element
The following is my Code. The "sequences" are my training data in the form [139 rows x 4 columns], 0) where the 139x4 are my signals and the 0 is my encoded label.
def __getitem__(self, idx):
sequence, label = self.sequences[idx]
…
patrick823
- 131
- 5
0
votes
1 answer
Path directory could not be found
A simple dataLoader for image segmentation. I am having trouble with pathing the dataset from my computer file
import torchvision
import torch
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import os,…
Mw W
- 1
0
votes
1 answer
Batching irregularities with data loader
I have some data in .txt files and an instance formed by two lines which both have 100 elements in them. First line defines the problem and the second line defines the solution. Even though it is not a great idea I tried to use a supervised setting…
bidon
- 27
- 4
0
votes
1 answer
How can I solve the wrong shape in DataLoader?
I have a text dataset that I want to use for a GAN and it should turn to onehotencode and this is how I Creating a Custom Dataset for my files
class Dataset2(torch.utils.data.Dataset):
def __init__(self, list_, labels):
'Initialization'
…
khashayar ehteshami
- 177
- 1
- 16
0
votes
1 answer
How to save images generated by PyTorch DataLoader?
I'm trying to add transforms to a video dataset I have, and I've already extracted the frames. I want to add the transforms but instead of saving to the variable to a Dataloader, I would like to export all the transforms as images to a new directory…
Samay Lakhani
- 85
- 1
- 1
- 7
0
votes
1 answer
When training a Deep learning model on multiple datasets, is it better concatenating all datasets and shuffling it, or train datasets sequentially?
So lets say that I Have datasets A, B, and C to train a model. My current solution take batches randomly from A, then from B, then from C. I wonder if concatenating all datasets and shuffling so that training would be more random would improve…
Manveru
- 177
- 2
- 9
0
votes
3 answers
Pytorch customized dataloader
I am trying to train a classifier with MNIST dataset using pytorch-lightening.
import pytorch_lightning as pl
from torchvision import transforms
from torchvision.datasets import MNIST, SVHN
from torch.utils.data import DataLoader,…
이재환
- 81
- 1
- 5
0
votes
1 answer
How can we split the DataLoader object into train and test?
I have been following below document from tensorflow-lite model maker object detection api.
https://www.tensorflow.org/lite/tutorials/model_maker_object_detection
Following the documentation in the advance usage section, I have tried to import my…
Ananda Roy
- 67
- 1
- 12
0
votes
1 answer
Transforming inside CustomDataset or inside CustomDataloader - which is better? pytorch
Do I want to apply transformations to augment data inside the CustomDataloader class in Pytorch, or should I do it in the CustomDataset class? I would be grateful if you could show an example of how to use it correctly.
0
votes
1 answer
Loading Dating without a separate TRAIN/TEST Directory : Pytorch (ImageFolder)
My data is not distributed in train and test directories but only in classes. I mean:
image-folders/
├── class_0/
| ├── 001.jpg
| ├── 002.jpg
└── class_1/
| ├── 001.jpg
| └── 002.jpg
└── class_2/
├── 001.jpg
…
Bakhtawar
- 107
- 2
- 14
0
votes
1 answer
Batchsize in DataLoader
I have two tensors:
x[train], y[train]
And the shape is
(311, 3, 224, 224), (311) # 311 Has No Information
I want to use DataLoader to load them batch by batch, the code I write is:
from torch.utils.data import Dataset
class KD_Train(Dataset):
…
Tim
- 115
- 1
- 9