1

I have to work with very large array (about 2000^3) of float, moreover, sometimes I have to route through array elements randomly, or by another non-sequential way.

I use stxxl library and store my data in stxxl::vector container. This library stores data on disk and operates with small portion of data in RAM. Because of random access to elements, it's necessary to read/write chunks of memory from disk very often, that slows down the working of program.

Is there anybody who know the way of operating with big data with random route through elements? Is it possible with stxxl?

Community
  • 1
  • 1
user2760303
  • 55
  • 1
  • 8
  • 2
    By my calculation, that's about 30 gigs of data, so unless you have 30 gigs of RAM and want random access (normally distributed), it's inevitable the data will have to be read from disk from time to time. If there's a pattern to your 'random' data access, then perhaps you can optimize it in some way. – stellarossa Aug 08 '16 at 09:08
  • 3
    Please specify what algorithm/task you are running on the vector. Most problems do not require purely random access. – SpamBot Aug 08 '16 at 09:35
  • Actually I have two tasks, In first one I don't have any pattern but I could divide the array and work with partitions. In second task there are 2 route: first - sequential route of each element of array of size N, second route - through each 2000^2 element, N times with shift on 1 each 2000^2 time. N=2000^3. My data is 3d cube of voxels, first I walk in Z direction, next in X direction. – user2760303 Aug 08 '16 at 10:12
  • You should go for NOSQL. – sameerkn Aug 09 '16 at 05:11
  • Just get a PC with 32+GB of RAM and forget about disks. That is a very reasonable amount nowadays. – nwp Aug 14 '16 at 22:54

1 Answers1

0

I would suggest you explore the stream package and the matrix container.