1

I did a tweet crawler using the cpan library AnyEvent::Twitter::Stream with OAuth authentication, but I'm having some problems. Sometimes, Twitter stops to send tweets through the streaming api. Then, I send my program to sleep for a while and then it should return to collect more tweets, but this is not happening. The script just stay trying to reconnect but it can't establish any connection. What could it be? My code is shown below:

my $done = AnyEvent->condvar;
my $count=0;
# track keywords através da OAuth
my $guard = AnyEvent::Twitter::Stream->new(
    consumer_key    => "...",    #My Oauth authentication enters here
    consumer_secret => "...",
    token           => "...",
    token_secret    => "...",
    method   => "filter",
    track    => $track,
    on_tweet => \&got_tweet,
    on_error => \&connection_close,
    timeout => 45,
);
sub connection_close{
    my($headers)=@_;
    print "HEADERS: $headers\n\n\n\n\n";
    open(FOUT,">>arquivoalerta.txt");
    my $Agora = time();
    my $HoraLocal = localtime($Agora);
    my @Tempo = split(/ +/,$HoraLocal);
    print FOUT "Parei de coletar às @Tempo  Streaming API";
    if ($count==0){
        print FOUT "Dormindo por 10 segundos\n";
        print "Dormindo por 10 segundos\n";
        sleep 10;
    }
    elsif($count==1){
        print FOUT "Dormindo por 20 segundos\n";
        print "Dormindo por 20 segundos\n";
        sleep 20;
    }
    else{
        print FOUT "Dormindo por 240 segundos\n";
        print "Dormindo por 240 segundos\n";
        sleep 240;
    }
    close(FOUT);
    $count++;
    warn "Connection to Twitter closed";
}

sub got_tweet {
    #Here I treat the data and store it in a database ...
}

$done->recv;

Thanks to everybody,

Thiago

Thiago
  • 694
  • 3
  • 12
  • 26
  • What HTTP error does your script recieve upon connection close? If for instance the script reads data too slowly, it might be banned for a while (after several retries) – etov Jun 27 '12 at 15:22
  • I don't know how to return the HTTP error from this library(Any::Event::Stream). I'm a newbie..xD. But I know that I always get a timeout when it stops for a while. I try to connect but I don't get any answer. What should I do? – Thiago Jun 30 '12 at 02:16

0 Answers0