Python-telegram-bot bypass flood and 429 error using Scrapy

Question

I follow the price drops on the target site. If there is a price decrease in accordance with the rules I have set, it is recorded in the notificate table. From there, a telegram notification is sent through the code I created in the pipelines.py file. Sometimes the target site discounts too many products and 200 products can come from the notificate table. I'm stuck with telegram's flood protection while sending these.

Things I've tried:

1- Rotating multiple tokens

2- add sleep time

However, I cannot say that I was successful. I'm still stuck with the flood barrier.

What I want to do here is; No matter how many notifications come from the Notificate table, queue them and send these notifications in a way that does not exceed 20 messages per second.

How can I do that.

My pipeline.py code:

def sendnotifications(self, token):
        cursor = self.cnx.cursor()
        req = requests
        cursor.execute("SELECT * FROM notificate WHERE token= '"+token+"'")
        notifications = cursor.fetchall()
        for notification in notifications:
            print(notification)
            productid = notification[1]
            url = notification[3]
            name = notification[2]
            old = notification[4]
            new = notification[5]
            price_difference = old - new
            percentage = price_difference / old
            percentage_str = str("%.2f" % (percentage * 100))
            

            message = "<b>" + name + "</b>" + "\n\n" + \
                str(old) + " TL >>>> " + \
                str(new) + f" TL - {percentage_str}%" + "\n\n" + \
                url + "\n\n" + \
                

            if str(old) == "1.00" or str(old) == "2.00":
                message = "<b>" + name + "</b>" + "\n\n" + \
                    "<b>" + str(new) + " TL\n\n" + "</b>" + \
                    url + "\n\n" + \
                    
                

            token_list = [
                "xxxxxxxxxxxxxxxxxxxxxxx",
                "yyyyyyyyyyyyyyyyyyyyyyyy",
                "zzzzzzzzzzzzzzzzzzzzzzzzzzz",
                "aaaaaaaaaaaaaaaaaaaaaaaaaaaa",
                "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
                "ccccccccccccccccccccccccccccccccc",
          

            ]

            TOKEN = token_list[random.randint(0, len(token_list)-1)]


            
            chat_id = "-100xxxxxxxxxxxxx"
            bot = telegram.Bot(token=TOKEN)
            # tel_url = bot.sendMessage(chat_id = chat_id, text = message, parse_mode=ParseMode.HTML)
            
            try:
                bot.sendMessage(chat_id = chat_id, text = message, parse_mode=ParseMode.HTML)
                sleep(0.05)

            except Exception:

                return False
        cursor.close()
        return True

The obvious way seems to be to chunk the updates into batches of 20 and sleep for more than 1 second between them...? — thebjorn, Oct 16 '22 at 15:05
@thebjorn thank you for your answer. Yes I want to this but how can I make this? if the updates more than 20 (in time) batches of 20. So received 93 updates from mysql batches of 20 (1sec sleep) + 20 (1sec sleep) + 20(1sec sleep) + 13 — cevreyolu, Oct 16 '22 at 15:12

score 1 · Accepted Answer · answered Oct 16 '22 at 15:36

1

The obvious way seems to be to chunk the updates into batches of 20 and sleep for more than 1 second between them:

# ...
notifications = cursor.fetchall()
cursor.close()

for i in range(0, len(notifications), 20):
    chunk = notifications[i:i+20]
    for notification in chunk:
        print(notification)
        # ...
    time.sleep(1.5)

answered Oct 16 '22 at 15:36

thebjorn

26,297
11
96
138

thank you so much. So sorry I'm newbie, Should I add this code after `cursor.close()` ? – cevreyolu Oct 16 '22 at 16:03
you should close the cursor after you've fetched all records, and before you start processing chunks (since that involves sleeping, and you don't want to keep the cursor open if you don't need it). – thebjorn Oct 17 '22 at 14:15
I tried your code and I received 47 items from notification table but can be send only 22. – cevreyolu Oct 17 '22 at 22:25
I don't know anything about the telegram api, so I'm afraid I can't help you any further. I would suggest reading the error messages and the documentation to figure out what the error is and what the rate-limiting policies are. You could always try smaller chunks and longer sleeps... – thebjorn Oct 19 '22 at 09:29
ps: if there is flood-protection issues you might need sleeps inbetween individual notifications also... – thebjorn Oct 19 '22 at 09:30

Python-telegram-bot bypass flood and 429 error using Scrapy

1 Answers1