DataLoader is a generic utility to be used as part of your application's data fetching layer to provide a consistent API over various backends and reduce requests to those backends via batching and caching.
GitHub: dataloader
Questions tagged [dataloader]
430 questions
0
votes
2 answers
DataLoader crashes when shuffling
I'm using DataLoader to read from a custom Dataset object based on numpy memmap.
As long as I read the data without shuffling everything works fine but, as I set shuffle=True, the runtime crash.
I tried implementing the shuffling mechanism in the…
Andrea Rosasco
- 25
- 7
0
votes
0 answers
Pytorch data loader shuffles example inside a batch
I have the following Dataloader that I am using for an autoencoder in Pytorch
class DataLoader(data.DataLoader):
def __init__(self, *args, **kwargs):
super(DataLoader, self).__init__(*args, **kwargs)
self.iterator =…
Maja
- 151
- 7
0
votes
1 answer
Pytorch dataloader, too many threads, too much cpu memory allocation
I'm training a model using PyTorch. To load the data, I'm using torch.utils.data.DataLoader. The data loader is using a custom database I've implemented. A strange problem has occurred, every time the second for in the following code executes, the…
hhoomn
- 375
- 4
- 15
0
votes
1 answer
set time to live to keys in data loader
Hey there is any way to set time to live to keys in data-loader (library for caching and batching created by facebook developers)?
In their docs there is only clearAll() function but I want to clear each key separately after X seconds from the…
Ron Badur
- 1,873
- 2
- 15
- 34
-1
votes
1 answer
Pytorch training loop doesn't stop
When I run my code, the train loop never finishes.
When it prints out, telling where it is, it has way exceeded the 300 Datapoints, which I told the program there to be, but also the 42000, which are actually there in the csv file.
Why doesn't it…
tridentifer
- 29
- 1
- 4
-1
votes
1 answer
Salesforce trigger-Not able to understand
Below is the code written by my collegue who doesnt work in the firm anymore. I am inserting records in object with data loader and I can see success message but I do not see any records in my object. I am not able to understand what below trigger…
sfdc developer
- 15
- 2
-1
votes
1 answer
Why is the instance of DataLoader a global variable in pytorch
I was debugging my pytorch code and found that an instance of the class DataLoader seems to be a global variable by default. I don't understand why this is the case but I've set up a minimum working example as below that should reproduce my…
Jack
- 81
- 3
- 11
-1
votes
1 answer
Splitting data into batches
I currently have the following data:
f_map, inputs, s_bias = ml_dataset.dataset_for_s_bias()
where f_map is a tensor of matrices, inputs is a tensor of floats, and s_bias is a tensor of floats. The first two, f_map and inputs, are the inputs to my…
crotchetyoldman
- 19
- 3
-1
votes
1 answer
How to add new dimension in tensor
I currently have following tensor with dimensions [batch_size, channel, width, height] but I want to add one dimension to the above tensors at third index as [batch_size, channel, length, width, height] where length is the dimension that I want to…
anonymous
- 11
- 1
- 3
-1
votes
1 answer
What is the fastest way of loading huge data(100 gb) in vertica database?
"What is the fastest way of loading data in vertica database ? If i have 100 GB of data which i need to laod in vertica table in a fast and efficient way."
Karam
- 411
- 4
- 10