DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
1
vote
1 answer
PyTorch: Apply data augmentation on training data after random_split
I have a dataset that does not have separate folders for training and testing. I want to apply data augmentation with transforms only on the training data after doing the split
train_data, valid_data = D.random_split(dataset, lengths=[train_size,…
alice
- 59
- 1
- 2
- 7
1
vote
1 answer
Why am I getting a "Cannot return null for non-nullable field" error when doing a query?
**FOLDER STRUCTURE
**
models.py
import graphene
from graphene import ObjectType, relay
from graphene_django import DjangoObjectType
from general.models import Character, Director, Episode
class CharacterType(DjangoObjectType):
pk =…
HarilalAV
- 41
- 1
- 7
1
vote
0 answers
What if I run the collate_fn outside the DataLoader?
The traditional way of applying some arbitrary collate_fn foo() in torch code is
dataloader = torch.data.DataLoader(
dataset,
batch_size=64, # just for example
collate_fn=foo,
**other_kwargs
)
for batch in dataloader:
# incoming batch…
thesofakillers
- 290
- 3
- 13
1
vote
1 answer
Apollo Server Express v4 GraphQL this.findOneById is not a function
I have the following ApolloServer (v4)
import { MongoDataSource } from 'apollo-datasource-mongodb'
export default class LoaderAsset extends MongoDataSource {
async getAsset(assetId) {
return this.findOneById(assetId) // ERROR IS HERE
…
Mehran Ishanian
- 369
- 2
- 4
- 13
1
vote
1 answer
Pytorch low gpu util after first epoch
Hi I'm training my pytorch model on remote server.
All the job is managed by slurm.
My problem is 'training is extremely slower after training first epoch.'
I checked gpu utilization.
On my first epoch, utilization was like below image.
I can see…
K.S Kim
- 77
- 4
1
vote
2 answers
How to handle Pytorch Dataset with transform function that returns >1 output per row of data?
Given a myfile.csv file that looks like:
imagefile,label
train/0/16585.png,0
train/0/56789.png,0
The goal is to create a Pytorch DataLoader that when looped return 2x the data points, e.g.
>>> dp = MyDataPipe(csvfile)
>>> for row in…
alvas
- 115,346
- 109
- 446
- 738
1
vote
2 answers
Pytorch DataLoader changes dict return values
Given a Pytorch dataset that reads a JSON file as such:
import csv
from torch.utils.data import IterableDataset
from torch.utils.data import DataLoader2, DataLoader
class MyDataset(IterableDataset):
def __init__(self, jsonfilename):
…
alvas
- 115,346
- 109
- 446
- 738
1
vote
2 answers
Pytorch transformation for just certain batch
Hi is there any method for apply trasnformation for certain batch?
It means, I want apply trasnformation for just last batch in every epochs.
What I tried is here
import torch
class test(torch.utils.data.Dataset):
def __init__(self):
…
Jake
- 65
- 4
1
vote
0 answers
"IndexError: list index out of range" in PyTorch dataloader while using Subset from PyTorch
I am trying to get new indices every time in a loop and use those indices to select a Subset from a PyTorch dataset class. Then building a dataloader for model training and this error keeps coming. Error and stack trace shown below:
def…
Pratik Karmakar
- 47
- 5
1
vote
1 answer
How to import data file from the S3 bucket into the Sagemaker notebook?
I have npz files that I want to import for my model training.
Below is the code I have tried.
import s3fs
fs = s3fs.S3FileSystem()
# To List 5 files in your accessible bucket
#fs.ls('s3://input_data/train_npz/')[:5]
# open it directly
with…
PScode
- 29
- 4
1
vote
1 answer
Can ideal num_workers for a large dataset in PyTorch be 0?
I am currently testing out different num_workers in DataLoader in PyTorch and it seems that 0 has the shortest running time.
I also tried out https://github.com/developer0hye/Num-Workers-Search, which is an automated num_workers search based on…
Anon Name
- 33
- 3
1
vote
0 answers
dotnet hotchocolate. if I use Projection do I need to use DataLoader?
UserType:
[Node(IdField = nameof(Id))]
public class UserType
{
public Guid Id { get; set; }
public string Email { get; set; } = string.Empty;
public Company? Company { get; set; }
[NodeResolver]
public static async…
otter otter
- 97
- 7
1
vote
2 answers
Nest.JS graphql subscription does not work with DataLoader (context is not defined)
I have a nestjs graphql application which uses DataLoader. It works fine for queries and mutations, but it does not work for subscriptions. This is how configuration is defined in app.module:
@Module({
imports: [
...
…
Kasheftin
- 7,509
- 11
- 41
- 68
1
vote
0 answers
Detectron2: Custom Data Augmentation Implementation
I am working on an underwater image detection problem using detection2. I have applied an image enhancement augmentation offline (by storing the newly processed data in a separate folder). But I need to apply the image enhancement on the fly, such…
muhammad ali
- 31
- 6
1
vote
0 answers
GraphQL DGS- Kotlin, Modify the parent resultSet, based on Child dataloader results
class UserEntity(val id: UUID, val name: String, val email: String){
val occupation: String? =null
}
class AddressEntity(val id:UUID){
val line1: String,
val line2 : String,
val city: City,
val state: State,
val zipcode: ZipCode
…
user2716913
- 565
- 1
- 4
- 7