3

I've got too little RAM to finish a calculation because of a large hash. Is there a drop-in Perl module that would let me use the hash without keeping it all in RAM? I expect it to top out around 4GB, and I've got a bit less than 2GB available for the script. I don't think processing time or disk I/O would be an issue.

Anna
  • 2,645
  • 5
  • 25
  • 34
  • Look into [tie](https://perldoc.perl.org/perltie#Tying-Hashes), I think this should do what you need. Or use a database. See also [this chapter in Effective Perl](https://www.informit.com/articles/article.aspx?p=1582932&seqNum=7). – Robert Dec 11 '20 at 18:42
  • Tie is exactly what I was looking for! Thank you! – Anna Dec 11 '20 at 18:43

2 Answers2

5

You can use dbm_open to open a hash connected to a DBM file. These are not particularly sophisticated and can handle shallow hashes of simple keys and values.

For anything more sophisticated, I would recommend using SQLite.

Schwern
  • 153,029
  • 25
  • 195
  • 336
  • My hash has string keys, and the values are hash refs with string keys and 0/1 values. Does that mean DBM can't handle it? – Anna Dec 11 '20 at 18:45
  • @Anna correct, you can only do one level. You could restructure the hash to be a flat one with combined keys, and it would work — or else use something else :) – hobbs Dec 11 '20 at 18:48
  • @Anna There's [MLDBM](https://metacpan.org/pod/MLDBM), but at that point it makes sense to jump to SQLite. Or you can restructure your program so you don't have to hold it all in memory. – Schwern Dec 11 '20 at 19:39
  • I solved it by running the program on another computer:) – Anna Dec 11 '20 at 19:40
  • @Anna Have you tried `$hash{$key1,$key2}` syntax? see `$SUBSCRIPT_SEPARATOR` (`$; `) in man perlvar for details. – AnFi Dec 15 '20 at 10:04
  • @AnFi I hadn't seen that syntax (would have interpreted that as the binary comma operator). How is it relevant to my memory issues? – Anna Dec 15 '20 at 11:41
  • @AnFi Aha right, to use DBM of course! Thanks for the advice, I'll keep that in mind for next time I've got that kind of structure! I guess I could also just have done $hash{"$key1-$key2"} – Anna Dec 15 '20 at 11:42
1

You may try DB_File module (or similar modules).
Memory usage hints: https://www.perlmonks.org/?node_id=146377

Take a look at AnyDBM_File for other similar modules available with rudimentary comparison.

$hash{$key1,$key2} syntax can be used to turn multi level hash into flat (single level) hash.
see $SUBSCRIPT_SEPARATOR in man perlvar for details.

AnFi
  • 10,493
  • 3
  • 23
  • 47