Why does Perl take up my memory (RAM) on printing a file?

Question

This question arises when I notice that my script starts to pauses on and off when it's in a while loop writing content to a file. I first check my memory and see a trend that this happens when my RAM decreased to about 2-3%. I then look for my code block that consumes the RAM and figured that as the script keeps feeding content to a file, it grows in size which also takes the memory too. Here is the code that does the file creation :

open (my $fh, '>>', $filePath);
while (my $row = $sth->fetchrow_hashref) {
    print $fh join ('|', $row->{name}, $row->{address}) "\n";
}

If I uncomment the print statement and run the script, my RAM will not decrease. So, I'm pretty sure that my memory RAM is taken by a FILEHANDLE/something else behind Perl that takes up the memory. This leads me to a question if there is a way to write a file from Perl script to a disk, instead of using the memory RAM ?

I have tried flushing FILEHANDLE on each row, and it still did not work.
One thing that is also really weird is that, after I terminate my script I look at my memory and it's still being taken by the files. And when I remove the files, it frees up my memory. I use free in linux to check my memory.

Can you update your question with the script? It should work. — choroba, Jan 31 '17 at 21:44
This might be [a duplicate](http://stackoverflow.com/questions/1242832/in-perl-how-can-i-release-memory-to-the-operating-system), but I really don't understand the question. — mob, Jan 31 '17 at 21:44
Edited my question to add more details on the question. Hopefully it will explain the problem a lot clearer — kelvien, Jan 31 '17 at 22:23
My bad, i fixed the syntax. So is this how Perl actually use memory for FILEHANDLE ? Is there an alternative not to exhaust the memory and write a huge file through Perl ? — kelvien, Feb 01 '17 at 00:37
"One thing that is also really weird is that, after I terminate my script I look at my memory and it's still being taken by the files. And when I remove the files, it frees up my memory. I use free in linux to check my memory." I think you're seeing the OS's disk cache, here... or else you're writing to tmpfs. — hobbs, Feb 01 '17 at 00:39
And there's this: http://search.cpan.org/~hirose/Sys-PageCache-0.01_001/lib/Sys/PageCache.pm — Zan Lynx, Feb 01 '17 at 00:56

score 7 · Accepted Answer · answered Feb 01 '17 at 01:00

7

So the answer to your question "Why does Perl take up my RAM on printing a file?" is that Perl is not using the memory. The operating system's file cache is using the memory.

Your Perl script wrote a large file. It first goes into file cache in RAM, the operating system gets around to writing it to disk when it has time or when it runs out of RAM. It doesn't remove it from RAM unless you delete the file or it needs the RAM for something else.

File cache is not a bad thing and you should not worry about how much of it is using your RAM. When a program needs more memory, the OS can clear out the file cache very quickly and reuse the memory for your program.

answered Feb 01 '17 at 01:00

Zan Lynx

53,022
10
79
131

Thanks Zan. My concern was actually due to seeing the fetchrow halts and resumes periodically when I see the 'Free' field at `free` gets around 2-3%. It gets noticeably slower. – kelvien Feb 01 '17 at 01:13
1

KelvienHidayat: If that was your concern then you should have mentioned it in your question. What are these *fetchrow halts* that you talk about. Everything will get noticeably slower if your memory is getting down to 2% free, but you didn't mention that either. One thing's for sure: it's not Perl's printing that's using up the space. You need to give a lot more information, like the contents of your program. – Borodin Feb 01 '17 at 02:19
1

@KelvienHidayat that's when the OS runs out of room to cache disk writes in memory, so it has to stop your program from producing output until it can flush some of it to disk. In other words, what you're seeing is a normal effect of your program producing output faster than your disk can write it. – hobbs Feb 01 '17 at 04:15
@hobbs, when you say the cache disk, do you mean the `cached` column at the result of `free` command ? I can see it still have over one GB left, which I think should be enough to keep producing output, right ? – kelvien Feb 02 '17 at 21:25
@KelvienHidayat The OS kernel doesn't use ALL of the memory for file cache. There are various reserves for different purposes so that the whole system can continue operating. – Zan Lynx Feb 03 '17 at 15:36

score 5 · Answer 2 · answered Jan 31 '17 at 21:53

Flushing a file handle's buffer to disk doesn't free the buffer, so it doesn't free up any memory. It would be wasteful to free the buffer since the buffer would need to be allocated all over again the next time the program wrote to the file.

In case you are wondering, each file handle has a 4 KiB or 8 KiB buffer, depending on your version of Perl.

Why does Perl take up my memory (RAM) on printing a file?

2 Answers2