DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
0
votes
0 answers
dataLoader in Graphql
I have a project where I use Nodejs for the backend, the backend I use Mongoose and Graphql, Accordingly, I have schema and resolver folders, and also a population file where I get a function that helps me to drill into data
this is the population…
david
- 47
- 1
- 5
0
votes
1 answer
Pytorch DataLoader switch item to List and cause several error
My source code is using pytorch and like this:
def Embed(sequenceSet):
output = []
for s in sequenceSet:
PseDNCSequence = Embedding.PseDNC(str(s))
ANFSequence = Embedding.ANF(str(s))
EIIPSequence = Embedding.EIIP(str(s))
…
Peter Phan
- 25
- 5
0
votes
0 answers
Dataloader in Pytorch is slow when data grows
Question
when Dataset returns a variable named rf, which's shape is [128, 1, 40, 64, 64], it's fast.
when Dataset returns two variable named rf and raw_data, shaped [128, 1, 40, 64, 64] and [128, 1, 40, 128, 64], it's very slow.
Test
Through…
TomWu
- 1
- 1
0
votes
0 answers
On the server, different Gpus run different programs, why do they affect each other's speed
We're using dual cpus, each with ten cores,four GPUs. When two programs run on different GPUs, the speed remains the same, but when programs are added on other GPUs, the speed of the two programs running before is slowed down. I'd like you to give…
user22188171
- 1
- 1
0
votes
0 answers
What is a recommended way to store and load enormous tabular dataset that doesn't fit to memory
I am trying to figure out the optimal way to store and load a large TABULAR dataset (around 500GB in total) when it comes to training a PyTorch model. Specifically, my inquiries are the following:
What is the best way to store a huge tabular…
Cade Reid
- 1
- 1
0
votes
0 answers
TypeError: load() got an unexpected keyword argument 'frame_offset'
I am new to using Pytorch library. I am trying to run the following code and get this type TypeError: load() got an unexpected keyword argument 'frame_offset'. Is there any solution for the code to complete training process?
import torch
import…
Christopher Marlowe
- 2,098
- 6
- 38
- 68
0
votes
1 answer
Pytorch DataLoader: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow
When I create a PyTorch DataLoader and trying to train the model, I got this User Warning:
/usr/local/lib/python3.10/dist-packages/sentence_transformers/SentenceTransformer.py:547:
UserWarning: Creating a tensor from a list of numpy.ndarrays…
LeMoussel
- 5,290
- 12
- 69
- 122
0
votes
0 answers
I have to train the combined models and combining them in train loop instead of making single data loader
I am applying feature concatenation by concatenating features of both models before fc and then putting classifier on them. Instead of making combine data loader, I used single data loaders and combined them in train loop using zip function.
Am I…
Tahir Naseem
- 1
- 1
0
votes
2 answers
How to auto update the null column imported in the table using apex data loader
Do you have any idea on how to auto update the table with null column (person_id) after importing the data using the apex data loader? In my csv file template, I do not include the person id, I only include the employee number. And I don't like to…
Eyei
- 1
- 4
0
votes
1 answer
How to pass custom object paramteres to Dataloader in Typescript?
I am trying to pass an object as a key to an instance of Dataloader. I know that I have to make a custom cache key function but I don't know exactly how to do it. I'm following this tutorial online:…
jpr
- 63
- 5
0
votes
0 answers
Python: FileNotFoundError: [Errno 2] Unable to open file
I wanted to get Featurenet-TensorFlow-2 (see Gitlab) running. I did all the steps mentioned there and ran into this error when I wanted to run train.py:
**D:\python\python.exe C:\Users\***\PycharmProjects\featurenet-tensorflow-2\train.py
Epoch 1 of…
jjw
- 1
0
votes
0 answers
How to solve the OOM problem encountered when using dgl to process ultra large image data？
I am trying to generate mini batch data using a sampler with built-in support in DataLoader to support random batch training on large images,
dgl.dataloading.DataLoader( graph, valid_nids, sampler, batch_size=1024, shuffle=False, drop_last=False,…
0
votes
0 answers
Fail to fine-tuned VGG pytorch with DataLoader
I try to replicate estimation of poverty based on satellite images from this github repo: https://github.com/jmather625/predicting-poverty-replication
I try to use Pytorch VGG on satellite images dataset. My inputs (X) are daytime satellite images…
0
votes
1 answer
useNavigation() state is never 'loading' in React Router
I've been following this tutorial, although I've been converting the steps produced to work with the ApolloClient because my API is graphql.
The problem I run into is that useNavigation()'s state property is never loading, it's always idle instead.…
IdeTheMan
- 21
- 4
0
votes
0 answers
How to choose the "number of workers" parameter in PyTorch DataLoader?
I am creating a pytorch dataloader as
train_dataloader = DataLoader(dataset,
batch_size=batch_size, shuffle=True, num_workers=4)
However, I get:
This DataLoader will create 4 worker processes in total.
Our suggested max number…
00__00__00
- 4,834
- 9
- 41
- 89