Questions tagged [memmap]
21 questions
0
votes
0 answers
Python program crashes when saving text-file
I have a large image stack with 853 images, each image has the dimensions (11779, 15394) pixels and 3 channels. The images have 8bit depth and are saved in BigTiff format. My data is stored on an external SSD which I connect to my computer.
What I…
JaBu
- 1
- 3
0
votes
1 answer
Manage python structures stored in a file as if they are in memory?
I want to manage many files in such a way that the file stays on disk and my app work with part of the data.
I have to manage 2 types of files text-files/book-like, cvs-files/time-series.
For every file I may generate multiple dimentionally reduced…
sten
- 7,028
- 9
- 41
- 63
0
votes
0 answers
What's the best way to re-order rows in a large binary file?
I have some large data files (32 x VERY BIG) that I would like to concatenate.
However, the data were collected in the wrong order, so I need to reorder the rows as well.
So far, what I am doing is:
# Assume FILE_1 and FILE_2 are paths to the…
0
votes
1 answer
Memory error when sorting a numpy memmap array
I have a numpy-memmap matrix S of size 12 GB. And I'm trying to argsort each row. To do that I have defined another memmap array first_k to save the result. The problem is that a memory error occurs.
Here is the code:
first_k = np.memmap('first_k',…
asierrayk
- 43
- 4
0
votes
2 answers
Numpy memmap in-place sort of a large matrix by column
I'd like to sort a matrix of shape (N, 2) on the first column where N >> system memory.
With in-memory numpy you can do:
x = np.array([[2, 10],[1, 20]])
sortix = x[:,0].argsort()
x = x[sortix]
But that appears to require that x[:,0].argsort() fit…
David Parks
- 30,789
- 47
- 185
- 328
-1
votes
1 answer
Why do I get Memory Error in stringsource?
I am trying to execute a cython file in parallel. The skeleton of my code is:
def build_DF_single(lo, hi, map[long, set[string]] authors_id_map, map[long, set[string]] authors_org_map,
map[long, set[string]] fos_name_map, map[long, set[string]]…
Caprikuarius
- 176
- 7