1

As I explained in this post, I'm trying to scrape tweets from Twitter.

I implemented the suggested solution with services, so that the actual heavy lifting happens in the backend.

The problem is that after about one day, I get this error

"Process terminated because the request deadline was exceeded. (Error code 123)"

I guess this is because the manual scaling has the requests timing out after 24 hours.

Is it possible to make it run for more than 24 hours?

Ying Li
  • 2,500
  • 2
  • 13
  • 37
Barbara
  • 1,118
  • 2
  • 11
  • 34
  • 1
    I think it's not possible. My best guess here would be trying to "split" your requests, as in try to divide the elements of this process into several instances of your service. Either that or move to Flexible environment, which I don't recall right now if this deadline also exists (my bad) – Mangu Oct 03 '18 at 13:30

1 Answers1

1

You can't make a single request / task run for more than 24 hours but you can split up your request into different parts and each lasting a day. It's unwise to have a request run indefinitely that's why app engine closes them after a certain time to prevent idling / loopy request that lasts indefinitely.

I would recommend having your task fire a call at the end to trigger the queuing of the next task, that way it's automatic and you don't have to queue a task daily. Make sure there's some cursor or someway for your task to communicate progress so it won't duplicate work.

Ying Li
  • 2,500
  • 2
  • 13
  • 37