0

I am using a 32 bit perl in my openvms system.(So perl can access up till 2gb of virtual address space ).

I am hitting "out of memory!" in a large perl script. I zeroed in on the location of variable causing this . However after my tests with devel:size it turns out the array is using only 13 Mb memory and the hash is using much less than that.

My question is about memory profiling this perl script in VMS.

is there a good way of doing memory profile on VMS?

I used size to get size of array and hash.(Array is local scope and hash is global scope)

DV Z01 A4:[INTRO_DIR]$ perl  scanner_SCANDIR.PL


Directory is Z3:[new_dir]
13399796 is total on array
3475702 is total on hash


Directory is Z3:[new_dir.subdir]
2506647 is total on array
4055817 is total on hash


Directory is Z3:[new_dir.subdir.OBJECT]
5704387 is total on array
6040449 is total on hash


Directory is Z3:[new_dir.subdir.XFET]
1585226 is total on array
6390119 is total on hash


Directory is Z3:[new_dir.subdir.1]
3527966 is total on array
7426150 is total on hash


Directory is Z3:[new_dir.subdir.2]
1698678 is total on array
7777489 is total on hash
kbang
  • 694
  • 9
  • 25
  • Show us your code. What is your script doing? (Its output doesn't mean much to us.) Note that the results of `Devel::Size` are kind of a guess, and may not correspond exactly with real size in memory. –  Apr 29 '14 at 02:43
  • http://search.cpan.org/~nwclark/Devel-Size-0.79/lib/Devel/Size.pm SAYS: This module figures out the real size of Perl variables in bytes, as accurately as possible. – kbang Apr 29 '14 at 14:04

1 Answers1

2

(edited: Pmis-spelled GFLQUOTA ) Where is that output coming from? To OpenVMS folks it suggests files in directories, which the code might suck in? There would typically be considerable malloc/align overhead per element saved.

Anyway the available ADDRESSABLE memory when strictly using 32 pointers on OpenVMS is 1GB: 0x0 .. 0x3fffffff, not 2GB, for programs and (malloc) data for 'P0' space. There is also room in P1 (0x7fffffff .. 0x4000000) for thread-local stack storages, but perl does not use (much) of that. From a second session you can look at that with DCL: 

$ pid = "xxxxxxxx"
$ write sys$output f$getjpi(pid,"FREP0VA"), " ", f$getjpi(pid,"FREP1VA")
$ write sys$output f$getjpi(pid,"PPGCNT"), " ", f$getjpi(pid,"GPGCNT")
$ write sys$output f$getjpi(pid,"PGFLQUOTA")

However... those are just addresses ranges, NOT how much memory the process is allowed to used. That's governed by the process page-file-quota. Check with $ SHOW PROC/QUOTA before running perl. And its usage can be reported as per above from the outside adding Private pages and Groups-shared pages as per above.

An other nice way to look at memory (and other quota) is SHOW PROC/CONT ... and then hit "q"

So how many elements are stored in each large active array? How large is an average element, rounded up to 16 bytes? How many hash elements? How large are the key + value on average (round up generously)

What is the exact message?

Does the program 'blow' up right away, or after a while (so you can use SHOW PROC/CONT)

Is there a source file data set (size) that does work?

Cheers, Hein.

Hein
  • 1,453
  • 8
  • 8
  • Out of memory! %SYSTEM-F-ABORT, abort %SYSTEM-F-ABORT, abort value of $STATUS : %X1000002C ---> This is returned in error_warning_handler part of dcl (i am invoking perl from a dcl script) – kbang Apr 29 '14 at 17:38
  • How can i find the PID of a process being executed as an image. perl $$ gives incorrect value $ perl -e "print $$;" 769762730 ]$ write sys$output f$getjpi("769762730","FREP0VA") %SYSTEM-W-NONEXPR, nonexistent process \769762730\ – kbang Apr 29 '14 at 17:39
  • Try $ perl -e "printf (""%%x%x\n"",$$);" – user2116290 Apr 29 '14 at 17:46
  • Works.. thanks. @Hein The script works in most cases. I am working on reproducing the "Out of memory" consistently related question: http://stackoverflow.com/questions/23378580/perl-regex-using-too-much-memory – kbang Apr 30 '14 at 03:09
  • From the other topic it now it clear the script is simply parsing DIR/NOHEAD/DATE output and the script reports array/hash sizes after (before?) every new directory. It should do more in the loop, and remember less. It perhaps should use the GLOB function to even avoid generating and parsing the directory output, just using perl 'stat' or -M type functions for file attributes. Admittedly I like putting output into a file, to check, and to re-run over and over without doing the (directory) command itself – Hein Apr 30 '14 at 10:57
  • The f$getjpi(pid,"FREP0VA") helped me to monitor p0 space of perl image – kbang Apr 30 '14 at 11:50
  • @Hein When does a vms system encounter out of memory? is it when it runs of P0 space ? Is it when PPGCNT limit is reached? or anything else. This is the output message i get `Out of memory! %SYSTEM-F-ABORT, abort %SYSTEM-F-ABORT, abort value of $STATUS : %X1000002C ` – kbang Apr 30 '14 at 18:02
  • Most commonly it is lack of pagefile quota, but P0-VA getting into 3xxx.xxxx whoul be suspect. If you can, read some terminal input after an arraysize print just to pause the process and allow you to look into values leisurely. – Hein Apr 30 '14 at 22:32