2

We have a PSGI script using UWSGI with multi processes and threads, looking for a fast solution to share editable hashes between these processed and threads. I did tests by using in-memory storage like Cache::Memcached::Fast and Redis.

There are fast but not fast enough, compared using just a locale hash

my %hash = ();

which is up to 10 times faster during our tests. Is there a way to share and editable variables in run time - like having sort of a local hash scenario - , e.g. that process 1 is updating

$hash{$id} = $new_val;

and process 2 has also that new value for $hash{$id}?

I also tried IPC::Shareable, MCE::Shared::Minidb or threads::shared (threads::shared: segment memory limit) which did work for the PSGI.

Is there a way to have FAST - editable - local variables between all processes on the same machine shared?

  • Why did not `threads::shared` work? – Håkon Hægland Sep 08 '19 at 09:51
  • It worked partial, this module is limited to the default/local segment memory limit. The hash is pretty big though (min. 100K elements; small value). threads::shared went out of memory when setting the hash :( – user7261395 Sep 08 '19 at 10:17
  • *"..100K elements.."* How did you run out of memory? How much memory did your shared data consume? What error message did you get? I tried now with a shared variable of size 100M and I did not get any errors. – Håkon Hægland Sep 08 '19 at 10:44
  • Sorry, I meant IPC::Shareable which run out of memory. threads::shared did not work. The second process didn't have and values of %hash. threads::shared works in PSGI enviroments? (I guess there is a parallel/separate process instead of a thread) – user7261395 Sep 08 '19 at 10:53
  • Ok, so the question is how to share memory fast between processes ( not threads)? – Håkon Hægland Sep 08 '19 at 10:55
  • Yes.(UWSGI will do multiple processes which have multiple threads/workers though.) – user7261395 Sep 08 '19 at 11:00
  • See also [Cache::FastMmap](https://metacpan.org/pod/Cache::FastMmap) – Håkon Hægland Sep 08 '19 at 11:30
  • Thanks just I tried it. 70 times hash get operations: 0.004306 sec. vs local hash: 0.000238 sec. (18 times faster). Is there a better (somehow local hash) way? Cache::Memcached::Fast performes at 0.006690 sec. – user7261395 Sep 08 '19 at 11:34
  • I'm curious too. I believe redis is not any faster than memcached. – mpapec Sep 08 '19 at 11:46
  • 1
    You're right. Redis performs here at 0.010856 sec. (testing over multi processors; memcached should perform even better than redis). Any solution to come close to a "locale hash performance" – user7261395 Sep 08 '19 at 11:56
  • "_not fast enough_" -- what do you expect, or, rather, want? Each process has to work with the resource (that has data) and then there's more to do for data integrity (so they don't mess with it at the same time etc); it's got to be considerably slower. Also, as we can't directly use `SysV IPC` or `mmap` (to share variables between processes in Perl) the modules that utilize these also use files (as far as I can tell) ... much slower yet. On the other hand, I don't know whether PSGI and related frameworks provide a way (and consider adapting what you want to do to what they have)? – zdim Sep 08 '19 at 21:46

0 Answers0