1

I have a use case where I need to process huge excel file in a fraction of second which was not possible. Hence, I wish to store the selected information from the excel file in memory so that my application can read it from memory instead of loading excel file every time. By the way, I am using groovy for developing the application. My question is as follows:

  • What is in-memory data structure? How can I use in groovy?
  • What happens when multiple processes running in different nodes want to access the in-memory data structure?
  • Any pointer/link will be very much helpful
Opal
  • 81,889
  • 28
  • 189
  • 210
user3721344
  • 221
  • 1
  • 4
  • 13

1 Answers1

0
  1. Just use apache poi, it will load the workbook into memory (example)
  2. They will need to each load a copy. Or you will need to do something clever
  3. See above
Community
  • 1
  • 1
tim_yates
  • 167,322
  • 27
  • 342
  • 338
  • Thanks for the reply. i am actually doing this which is taking lot of time. For instance, I span about 100 processes and they all need to look into this excel file for particular rows. This process takes a lot of time and hence overall execution takes more time. Instead, if i can store the information in a form of, say, a hashmap structure in memory, my processes can simply read this instead of looking into excel file. So, i am interested in solutions like this. – user3721344 Aug 12 '14 at 10:34
  • [Hazelcast](http://hazelcast.org/docs/3.0/manual/html/ch02.html)? [MongoDB](http://stackoverflow.com/questions/6809183/mongo-database-save-data-from-map)? (other databases and clustered object stores are available) – tim_yates Aug 12 '14 at 12:55
  • I used Hazelcast. it is impressive. ALso looking into redis. However, ultimately, we moved from excel to csv, and with single scanning of csv all the required information is stored in a map structure which took just about 1 min. Subsequent access to the information is from map structure. This approach gave much better performance. Ofcourse, this is limited by the size of the file. – user3721344 Aug 20 '14 at 08:14