DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
0
votes
0 answers
How to use one original & one augmented batch from torchvision MNIST dataset with shuffeling, "aligned" batch samples, but different batch size?
I want to implement this situation for the torchvision MNIST dataset, loading data with DataLoader:
batch A (unaugmented images): 5, 0, 4, ...
batch B (augmented images): 5*, 5+, 5-, 0*, 0+, 0-, 4*, 4+, 4-, ...
... where for every image of A there…
0
votes
1 answer
Runtime error on during execution of cnn model for image recognition with fastai library
I am training a cnn model to recognise images. However, I get an error when running this code:
from fastai.vision.all import *
path = untar_data(URLs.PETS)/‘images’
def is_cat(x): return x[0].isupper()
dls = ImageDataLoaders.from_name_func(
path,…
Luigi87
- 265
- 4
- 13
0
votes
1 answer
Apollo GraphQL DataLoader DynamoDb
I'm new to GraphQL and am reading about N+1 issue and the dataloader pattern to increase performance. I'm looking at starting a new GraphQL project with DynamoDB for the database. I've done some initial research and found a couple of small NPM…
Tom Schreck
- 5,177
- 12
- 68
- 122
0
votes
1 answer
'bert-base-multilingual-uncased' dataloader RuntimeError : stack expects each tensor to be equal size
I am a begineer in nlp, as I was giving this competition https://www.kaggle.com/c/contradictory-my-dear-watson I am using the model 'bert-base-multilingual-uncased' and using BERT tokenizer from the same. I am also using kaggle tpu. This is the…
vaibhav bhatt
- 1
- 1
0
votes
1 answer
Simultaneous reads of the same PyTorch torchvision.datasets object
Consider the following piece of code to fetch a data set for training from torchvision.datasets and to create a DataLoader for it.
import torch
from torchvision import datasets, transforms
training_set_mnist = datasets.MNIST('./mnist_data',…
mgus
- 808
- 4
- 17
- 39
0
votes
0 answers
AttributeError: 'list' object has no attribute 'cuda'
I try to develop Convolution network deep learning for face recognition and right now when i try to run it said 'list' object has no attribute 'cuda' im not sure what went wrong can anyone check. the code below is for train the whole module and …
Archy dragon
- 13
- 3
0
votes
1 answer
Powershell Converting Tab Delimited CSV to Comma delimited CSV without Quotes
We get a tab delimited CSV from COGNOS External system in a public folder. This fails to upload to Salesforce via Dataloader CLI.
com.salesforce.dataloader.exception.DataAccessRowException: Error
reading row #0: the number of data columns (98)…
Pasan Eeriyagama
- 277
- 1
- 4
- 15
0
votes
1 answer
Batch loading a field in absinthe with dataloader
I have an object in my Absinthe graphql schema that looks like this:
object :match do
field(:id, non_null(:id))
field(:opponent, non_null(:string))
@desc "The number of votes that have been cast so far."
field(:vote_count,…
harryg
- 23,311
- 45
- 125
- 198
0
votes
2 answers
Splitting custom PyTorch dataset into train loader and validation loader: Length of both same, even though dataset was split?
I'm trying to split one of the Pytorch custom datasets (MNIST) into a training set and a validation set as follows:
def get_train_valid_splits(data_dir,
batch_size,
random_seed=1,
…
user6496380
- 43
- 1
- 7
0
votes
1 answer
How to create a custom data loader in Pytorch?
I have a file containing paths to images I would like to load into Pytorch, while utilizing the built-in dataloader features (multiprocess loading pipeline, data augmentations, and so on).
def create_links():
data_dir = "/myfolder"
…
Gulzar
- 23,452
- 27
- 113
- 201
0
votes
1 answer
How to slice 3D torch tensor into 2D slices
I'm working with 3D CT medical data, and I'm trying to slice it into 2D slices that I can input into a UNet model.
I've loaded the data into a torch dataloader, and each iteration currently produces a 4D tensor:
for batch_index, batch_samples in…
ml-yeung
- 3
- 2
0
votes
1 answer
Pytorch dataloader Transforms tensor error
[Unable to access the pytorch dataloader values for use
TypeError: default_collate: batch must contain tensors, NumPy arrays, numbers, dicts, or lists; found object
The error can be reproduced by running https://jovian.ml/pravin-bnmit/cxr/ on…
Nemaly Praveen
- 77
- 1
- 7
0
votes
1 answer
Loading json file using torchtext
I'm working on the dailydialog dataset, which I've converted into a
JSON file which looks something like this:
[{"response": "You know that is tempting but is really not good for our fitness.", "message": "Say, Jim, how about going for a few beers…
0
votes
2 answers
Best way to bulk load CSV file into PostgreSQL table
I have a tab separated CSV file in D:\DataSet\business_names_202007/businessDataSet.csv with 600k records (which may increase). I want to load entire data into below postgresql table.
PostgreSQL table:
CSV file structure:
As you can see above…
Justin
- 855
- 2
- 11
- 30
0
votes
1 answer
Sampling data batch wise from tensor Pytorch
I have train_x and valid_x splited from trainX ,train_y and valid_y splited from trainY and they are having shapes as per below. i want to classify images of labels LABELS = set(["Faces", "Leopards", "Motorbikes", "airplanes"]).
print(train_x.shape,…
Rakesh
- 23
- 5