Camel consuming twitter timeline tweets, and rates

Question

I am trying to get tweets from my Twitter timeline with ConsumerTemplate (Camel) in a Quarkus app.

I don't understand how it should be done correctly while using rate limits (my free account has rate 15 polls in 5 min).

The Camel setup has property count which I set to 10, and expect to get 10 new tweets in one poll, but the problem is that template offers only one tweet per Camel request:

for (int i = 0; i < 15; i++) {

    // I would expect here List or stream, but I can get only single `Status`
    final Status tweets = consumerTemplate.receiveBodyNoWait(String.format("twitter-timeline://home?sinceId=%s&count=10", sinceId), Status.class);

    // just to show that I change that property
    if (nonNull(tweets)) {
         sinceId = tweets.getId();
    }

}

So my question is why I get only one tweet per poll even if count is set (a value more than 1). Is there any better way or running that template, or something I can use instead while respecting downstream Twitter rate limits?

score 2 · Accepted Answer · answered Jun 23 '22 at 11:53

The ConsumerTemplate is based on a PollingConsumer under the hood, which as the name implies, pulls Exchages one at a time hence why you are receiving only 1 Status (== 1 Exchange) even while setting a count endpoint option.

It is almost hard (and error-prone) to implement tweets (or other Exchanges) consumption the right way. Indeed this is already offered out-of-the-box with the Apache Camel Twitter Component consumer.

Here down a sample of a route to consume user public activity (timeline) using the twitter-timeline component:

<routes xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://camel.apache.org/schema/spring"
        xsi:schemaLocation="
            http://camel.apache.org/schema/spring
            https://camel.apache.org/schema/spring/camel-spring.xsd">

    <route id="twitter-timeline" autoStartup="true">
        <from uri="twitter-timeline:USER?user=some-user&amp;consumerKey=your-app-key&amp;consumerSecret=your-app-key-secret&amp;accessToken=your-app-access-token&amp;accessTokenSecret=your-app-access-token-secret"/>
        <log message="${body}"/>
    </route>

</routes>

This will leverage the default polling configured delays which you can tweak based on your preferences to address any rate-limits. In your case for example, to respect 15 requests per 5 mins, you can set the consumer to poll each 20 seconds for one page:

<from uri="twitter-timeline:USER?user=some-user&amp;count=100&amp;numberOfPage=1&amp;delay=20000"/>

Thanks! It is very helpful, I have posted my solution in Quarkus maybe it would be helpful for someone else. — Garion S., Jun 24 '22 at 13:18
You're welcome @GarionS.. You mean the Java DSL based declaration of the route and not *Quarkus* one. I should have pointed out that there is different DSLs to configure the route but the Apache Camel documentation is very compact and has all the information you would need. — tmarwen, Jun 24 '22 at 17:10

score 1 · Answer 2 · answered Jun 24 '22 at 13:16

Based on tmarwen's answer in Quarkus I managed to do it without XML, and with increased delay (seems ratio was smaller for API v1.1):

@ApplicationScoped
public class TimelineService extends RouteBuilder {

    @Override
    public void configure() throws Exception {
        fromF("twitter-timeline://home?count=100&numberOfPages=1&delay=180000&consumerKey=%s&consumerSecret=%s&accessToken=%s&accessTokenSecret=%s",
                apiKey, apiKeySecret, accessToken, accessTokenSecret)
                .setBody(exchange -> {
                    // Status - can cast or manipulate
                    Object body = exchange.getMessage().getBody();
                    return body;
                })
                .to("log:twitter?showExchangePattern=false&showBodyType=false");
    }
}

Camel consuming twitter timeline tweets, and rates

2 Answers2