Using Log::Log4perl with MCE (Many-core Engine) Perl

Question

I am using MCE for parallel processing for one of my project. I am facing issue while logging the output from MCE (for logging I am using Log4perl).

I have gone through the examples provided and found them printing to STDOUT/STDERR or to some log file provided at that time. There is MCE->sendto() and MCE->print() option availble, but I am not sure how to use it with log4perl.

package ABC;
use strict;
use warnings;
use MCE;

sub new {
    my $class = shift;
    my ($self) = {@_};
    return bless $self, $class;
}

sub initialize_mce {
     my $mce = MCE->new(
        max_workers => 5,
        input_data => \@input_data,
        on_post_exit => \&on_post_exit,
        user_begin => \&user_begin,
        user_end   => \&user_end,
        user_func => \&run_function
    );

}
sub on_post_exit {
    my ($self, $e) = @_;
    # I want to write something like this -
    # I want to log to my global log file using Log4perl which is initialized in some other package and passed to this package

    # $self->{'logger'}->info("$e->{wid}: $e->{pid}: $e->{status}: $e->{msg}: $e->{id}");

    print "$e->{wid}: $e->{pid}: $e->{status}: $e->{msg}: $e->{id}\n";
}

sub user_begin {                   ## Called once at the beginning
    my ($self, $e) = @_;
    print "$$ start";
}

sub user_end {                     ## Called once at the end
    my $self = shift;
    #$self->{'logger'}->info("$$ end");
    print "$$ end";
}

sub run_function {
    my ($self) = @_;
    my $wid = MCE->wid;
    $self->{'logger'}->info("Running...$wid");
    my $input_data = $_;
    ###
    #Rest of subroutine 
    ####
}
1;

score 2 · Accepted Answer · answered May 01 '19 at 11:10

I have found the solution. In case someone also faced this issue they can try this-

You have to initialize 'user_output' and 'user_error'. From MCE Documentation -

    user_error => \&user_error,         ## Default undef
    user_output => \&user_output,       ## Default undef

    # MCE will forward data to user_error/user_output,
    # when defined, for the following methods.

    # MCE->sendto(\*STDERR, "sent to user_error\n");
    # MCE->printf(\*STDERR, "%s\n", "sent to user_error");
    # MCE->print(\*STDERR, "sent to user_error\n");
    # MCE->say(\*STDERR, "sent to user_error");

    # MCE->sendto(\*STDOUT, "sent to user_output\n");
    # MCE->printf("%s\n", "sent to user_output");
    # MCE->print("sent to user_output\n");
    # MCE->say("sent to user_output");

So in my case -

sub initialize_mce {
     my $mce = MCE->new(
        max_workers => 5,
        input_data => \@input_data,
        on_post_exit => \&on_post_exit,
        user_begin => \&user_begin,
        user_end   => \&user_end,
        user_func => \&run_function,
        user_output => sub {
            $logger->info($_[0]);            # Log4perl Obj
        },
        user_error => sub {
            $logger->error($_[0]);          # Log4perl Obj
        }
    );
}

And in others function you just have to 'print' to STDOUT/STDERR E.g.-

sub user_begin {                   ## Called once at the beginning
    my ($self, $e) = @_;
    $mce->print("Process Id : ".$$." start");
    # OR to log error 
    $mce->print(\*STDERR, "Exception/Error Found");
}

I am not sure this is the only way or there is some other efficient way, but for now it get the job done.

Using Log::Log4perl with MCE (Many-core Engine) Perl

1 Answers1