Polling multiple SQS messages using Airflow SQSSensor

Question

I am using this SQSSensoe settings to poll messages

    fetch_sqs_message = SQSSensor(
        task_id="...",
        sqs_queue="...",
        aws_conn_id="aws_default",
        max_messages=10,
        wait_time_seconds=30,
        poke_interval=60,
        timeout=300,
        dag=dag
    )

I would assume everytime it polls it should poll up to 10 messages. Which my queue has around 5 when I tested this. But each time I trigger the dag, it only polls 1 message at a time, which I found out from the SQS message count. Why is it doing this? How can I to get it poll as much messages as possible?

Valid values for `wait_time_in_seconds` should be between 0-20 seconds. Also, you can read [this answer](https://stackoverflow.com/a/21104343/5189811) for how long polling works in SQS — Oluwafemi Sule, Jun 03 '22 at 00:22

score 1 · Accepted Answer · answered Jul 27 '22 at 08:23

Recently, a new feature has been added to SQSSensor so that the sensor can polls SQS multiple times instead of only once.

You can check out this merged PR

For example, if num_batches is set to 3, SQSSensor will poll the queue 3 times before returning the results.

Disclaimer: I contributed to this feature.

Polling multiple SQS messages using Airflow SQSSensor

1 Answers1