perl with libxml silently crashing in Windows 10 inside loop with fork

Question

I'm using perl 5.24.4 under Windows 10 that have the latest LibXML installed (2.0134). Below is the code that crashes on Windows 10 (It works on Linux)

crash.pl

use strict;
use warnings;

use XML::LibXML;
use Utils;
use Log::Log4perl;


Log::Log4perl->init('logger.conf');
my $logger = Log::Log4perl->get_logger();

sub generate_word_stats_for_chunk
{
    return Utils::load_sql();
}

my $num_process = 4; 
my @modes = (1,3,5);

foreach my $mode (@modes) {


    my @pid_list;
    $logger->info("Forking $num_process");
    for my $i (0 .. $num_process - 1) {
        my $pid = fork();
        $logger->logconfess("Unable to fork $!") if not defined $pid;
        if ($pid == 0) {
            # child process
            generate_word_stats_for_chunk($i);
            sleep 2;

            # make sure that the child process exit
            exit;
    }
    # parent
    # Store the PIDs, so that waitpid() can call them later
    push @pid_list, $pid;
}

for my $pid (@pid_list) {
    waitpid $pid, 0;
}
$logger->info("finished mode $mode");

Utils.pm

package Utils;

use strict;
use warnings;

use XML::LibXML;
use Log::Log4perl;

my $logger = Log::Log4perl->get_logger();

sub load_sql 
{

    $logger->info("parsing XML");   
    my $dom;

    eval {
        $dom  = XML::LibXML->load_xml(location => 'sql.xml');
    };
    if ($@) {
        $logger->error("parse error $@");
        return;
    }
    $logger->info("finished parsing XML");  

    # dummy
    return {};
}

# perl package ends with 1
1;

sql.xml

<?xml version="1.0" encoding="UTF-8"?>    
<sqls>    
    <sql name="CREATE_COMMODITY_META">
        test
    </sql>      
</sqls>

Running crash.pl gives me the following output

INFO main::(crash.pl:37) Forking 4
INFO Utils::load_sql(Utils.pm:21) parsing XML
INFO Utils::load_sql(Utils.pm:31) finished parsing XML
INFO Utils::load_sql(Utils.pm:21) parsing XML
INFO Utils::load_sql(Utils.pm:31) finished parsing XML
INFO Utils::load_sql(Utils.pm:21) parsing XML
INFO Utils::load_sql(Utils.pm:31) finished parsing XML
INFO Utils::load_sql(Utils.pm:21) parsing XML
INFO Utils::load_sql(Utils.pm:31) finished parsing XML
INFO main::(crash.pl:62) finished mode 1
INFO main::(crash.pl:37) Forking 4
INFO Utils::load_sql(Utils.pm:21) parsing XML
INFO Utils::load_sql(Utils.pm:21) parsing XML
INFO Utils::load_sql(Utils.pm:21) parsing XML
INFO Utils::load_sql(Utils.pm:21) parsing XML

The code works if I remove the Utils::load_sql() statement. This might be similar to Why does Perl crash when using LibXML after a fork?, but I have some doubts as the code in that post works only machine. My code actually works if you remove the outer loop (e.g. the foreach my $mode... ). Also another reason why I doubt it is that post is already 8 years old.

Also, why isn't the eval statement in Utils.pm not catching the crash?

Most likely, you're running afoul of XML::LibXML not being threadsafe enough. My recommendation is to avoid `fork()` and instead launch real child processes. — Corion, Mar 18 '19 at 17:25
The [note on thread support](https://metacpan.org/pod/XML::LibXML#THREAD-SUPPORT) in `XML::LibXML`, linked in the post you link to, isn't "_8 years_" old; also, `fork` on Windows is emulated using threads (as Windows has no fork), which can only make it trickier. So, since there _is_ a problem, I'd try changing to threads (instead of forking) and then follow carefully what docs say on how to work with the module under threads. — zdim, Mar 18 '19 at 17:43
"_code actually works if you remove the outer loop_" -- I don't see that the outer loop does anything, other than triggering 3x as many threads of execution. This alone may be a factor: issues with threads/forks very often don't come up in every run, and have far more chance to show when it's busier (more threads or processes) — zdim, Mar 18 '19 at 17:47
`fork` is a unix concept that doesn't exist in Windows. Perl poorly emulates it using threads in Windows. It's best to avoid `fork` on Windows. — ikegami, Mar 18 '19 at 19:11
...Do you have the problem if you just use threads directly? — ikegami, Mar 18 '19 at 20:56
@Ikegami @Corion @zdim The code are meant to be run on a linux box but I do most of my development on a Windows machine though. I believe process are more robust and mature in Linux; thus I prefer to stick with `fork()` — Demeter P. Chen, Mar 18 '19 at 23:57

perl with libxml silently crashing in Windows 10 inside loop with fork

crash.pl

Utils.pm

sql.xml

0 Answers0