1

Say I have a huge numpy matrix A taking up tens of gigabytes. It takes a non-negligible amount of time to allocate this memory.

Let's say I also have a collection of scipy sparse matrices with the same dimensions as the numpy matrix. Sometimes I want to convert one of these sparse matrices into a dense matrix to perform some vectorized operations.

Can I load one of these sparse matrices into A rather than re-allocate space each time I want to convert a sparse matrix into a dense matrix? The .toarray() method which is available on scipy sparse matrices does not seem to take an optional dense array argument, but maybe there is some other way to do this.

conradlee
  • 12,985
  • 17
  • 57
  • 93

2 Answers2

2

If the sparse matrix is in the COO format:

def assign_coo_to_dense(sparse, dense):
    dense[sparse.row, sparse.col] = sparse.data

If it is in the CSR format:

def assign_csr_to_dense(sparse, dense):
    rows = sum((m * [k] for k, m in enumerate(np.diff(sparse.indptr))), [])
    dense[rows, sparse.indices] = sparse.data

To be safe, you might want to add the following lines to the beginning of each of the functions above:

assert sparse.shape == dense.shape
dense[:] = 0
conradlee
  • 12,985
  • 17
  • 57
  • 93
Fred Foo
  • 355,277
  • 75
  • 744
  • 836
1

It does seem like there should be a better way to do this (and I haven't scoured the documentation), but you could always loop over the elements of the sparse array and assign to the dense array (probably zeroing out the dense array first). If this ends up too slow, that seems like an easy C extension to write....

Danica
  • 28,423
  • 6
  • 90
  • 122
  • 1
    It turns out that it's quicker to convert each sparse array to a dense array (which involved allocating a lot of memory each time) than to loop over all the elements of the sparse array and load elements into the pre-allocated dense array. I didn't try writing a C extension. – conradlee Jan 30 '12 at 12:21