Shared readonly memory between Perl processes

Question

I wish to make a Perl program of mine use multiple cores. It progressively reads query input and compares chunks of that against a read-only data-structure loaded from file into memory for each run. That data-structure, which is typically a few giga-bytes, is a small set of packed strings that are used in small C-routines. When processes are forked, everything is copied, which on a multi-core machine quickly blows the RAM. I tried several non-standard modules, but all leads to slowness and/or blows the RAM. I thought, for read-only data, that Perl would not insist on making copies. Other languages can do it. Does anyone have ideas?

Answer 1

Fork doesn't normally copy memory until it's modified (search for copy on write or COW). Are you sure you are measuring memory usage correctly? Subtract before/after values from free rather than using top.

EDIT - example script

Try running the following with settings like: ./fork_mem_usage 5 10000 ./fork_mem_usage 25 10000 ./fork_mem_usage 5 100000 ./fork_mem_usage 25 100000

If the first increase is bigger than the subsequent ones then fork is using copy-on-write. It almost certainly is (except for Windows of course).

#!/usr/bin/perl
use strict;
use warnings;

my $num_kids  = shift @ARGV;
my $arr_size  = shift @ARGV;
print "$num_kids x $arr_size\n";

my @big_array = ('abcdefg') x $arr_size;
die "Array wrong length" unless ($arr_size == @big_array);

print_mem_usage('Start');

for my $i (1..$num_kids) {
    my $pid = fork();
    if ($pid) {
        if ($i % 5 == 0) {
            print_mem_usage($i);
        }
    }
    else {
        sleep(5);
        exit;
    }
}

print_mem_usage('End');
exit;

sub print_mem_usage {
    my $msg = shift;
    print "$msg: ";
    system q(free -m | grep buffers/cache | awk '{print $3}');
}

Answer 2

Edit & Summary:

I have been terribly wrong about threads::shared being an option. Upon thread creation, even shared data structures are copied. This does indeed suck, and I can therefore summarize that Perl is completely incapable for memory-intensive computations.

---

When a process forks, the kernel copies the whole process. Everything that resides in RAM is duplicated. No language can work around that. However, you can try memory mapping, or you can use threads.

Perl threads are a fork emulation, but you can declare variables as shared between threads:

use threads;
use threads::shared;

my $sharedVariable :shared = 0;

my @worker;

for my $i (1 .. 6) {
   push @worker, threads->create(\&worker_sub);
}

$_->join() foreach @worker;

sub worker_sub {
   sleep rand 5;
   print $sharedVariable, "\n";
}

If the $sharedVariable is updated in one thread, the change propagates to the other threads as well. You can see this if you substitute the print statement with

print threads->tid, "-->", ++$sharedVariable, "\n";

Answer 3

You may could use Cache::FastMmap to store shared data. I heard that someone used this for IPC not for cache, and this cache is shared between processes. Large parts of this is written in C. Do not forget to add 'raw_values=1' in initialization. It is possible to compress values in cache so if you have enough CPU and your data compressibl it will ave you a lot of mem.

It is quite fast, here is some benchmark: http://cpan.robm.fastmail.fm/cache_perf.html

Because Cache::FastMmap mmap's a shared file into your processes memory space, this can make each process look quite large, even though it's just mmap'd memory that's shared between all processes that use the cache, and may even be swapped out if the cache is getting low usage.

However, the OS will think your process is quite large, which might mean you hit some BSD::Resource or 'ulimits' you set previously that you thought were sane, but aren't anymore, so be aware.