AnyEvent in multithreaded environment

Question

I have that following subroutine, and have appx 20 threads calling it with different URLs (this sub belongs to a package, each thread calls a different instance of that package):

sub get_urls {
    my ($self,$url,$depth) = @_;
    my $cv = AnyEvent->condvar;
    my @data;
    my %visited;
    my $hostname = URI->new($url)->host();
    my $tr_cb;
    my ($b,$e) = (0,0);

       return unless($depth);
       # This code-ref is recursive!
       $tr_cb = sub {
        my $sitem   = shift;
        my $depth   = shift;

        return if (0 == $depth--);
        foreach my $site (@$sitem) {
            if (exists($visited{$site})) {
               next;
            }
            $b++;
            $visited{$site} = 1;
            $cv->begin;
            AnyEvent::HTTP::http_get ($site, timeout => 1, sub {
               my ($body, $hdr) = @_;
               if ($hdr->{Status} =~ m/^2/) {
                  my $extor = HTML::SimpleLinkExtor->new();
                  my @links;
                  print "E = $e | B = $b\n";
                  #print "[REC_DEPTH:$depth]Working on $site\n";
                  $extor->parse($body);
                  @links = map { URI->new_abs($_,$site)->as_string }
                            grep { length > 2 } $extor->links();
                  push(@data,@links);
                  $tr_cb->([map { $_->[2] } 
                              grep { $_->[0] eq $_->[1] } 
                                map { [$hostname,URI->new($_)->host(),$_] } @links],$depth);
               }
                $e++;
                $cv->end;
            });
        }
        };
    $tr_cb->([$url],$depth);
    $cv->recv;
    print "Got total of " . @data . " links\n";
}

the ($b,$e) variables are there for testing only. the issue is, after some time, it seems like the number of 'begins' does not match the amount of 'ends' hence it never gets pass $cv->recv... I'm kinda new to AnyEvent and Event-Programming in general, cant seem to fine my issue.

Thanks,

Why are you combining threads with an asynchronous, event-driven library? You can create multiple condvars in a single-threaded app as long as you don't recv recursively. — MkV, Aug 26 '12 at 01:06
good question, actually I means processes, not thread, its a typo, Im forking, still I think its wrong. — snoofkin, Sep 04 '12 at 18:22

MkV · Accepted Answer · 2012-08-26T09:00:28.077

The whole recursive anonymous sub seems a little bit too clever for its own good. Create a function in your class which gets the links from the passed in url (and depth) and adds them to an array on your object. At the same time create a timer (after => 0) which shifts elements off your array, restarting itself if there are still elements, otherwise sending end to the condvar. Replace an array with a Thread::Queue object if needed.

You should also only be calling ->recv in your application code, not your library, or using a callback on your condvar instead of calling recv (this would enable you to use multiple condvars and send to them without relying on threads)

score 1 · Answer 2 · answered Sep 04 '12 at 16:18

1

The error in the design of this utility function is that it is blocking on a condvar: $cv->recv. This blocks users of the function to use it in a global asynchronous program.

Instead, get_urls should return the condvar ($cv) and let the user do something useful with it. If the user want to block, he will be able to do it. If he doesn't he will be free to share the resources with other asynchronous tasks.

Another problem with your program is that you would probably not need threads if that function was correctly designed: if that function is the heart of the program, it is clearly network bound, not CPU bound, so you should be able to launch multiple calls to (a fixed) get_urls that would execute in parallel in a single thread.

answered Sep 04 '12 at 16:18

dolmen

8,126
5
40
42

its not a problem, this is what I expect it to do, dont want the user to decide this in this case, the async part is only for the get requests. but thanks for pointing out. – snoofkin Sep 04 '12 at 18:17
Btw, another thing, probably a too general question, but I will give it a try, is there a reason to spawn (fork) new processes in an async application? I mean, this function is called in a new process, a crawler's processes,each crawler gets a new process, several runs in parallel. – snoofkin Sep 04 '12 at 18:20
You have to determine what makes your program slow. If nothing, stop there. If this is the network, stop there, multiple process will not help. If this is the CPU (the process on which your program runs is at 100%), then consider multiple process or multiple threads. But in the case of a crowler I suspect the network is the first issue. – dolmen Oct 24 '12 at 19:50

score -2 · Answer 3 · answered Aug 25 '12 at 22:05

-2

You have $cv->end; inside your callback. It seems wrong to me, it should be after AnyEvent::HTTP::http_get call, AFAIU.

answered Aug 25 '12 at 22:05

w.k

8,218
4
32
55

AnyEvent in multithreaded environment

3 Answers3