0

I've been using Google Vision API to perform OCR tasks in some documents using Python.

It begins working perfectly, until I start receiving Http Error Code 429, which means I am doing too many requests in a short amount of time. Then, I decided to put a sleep between each request, of which time increases as the number of Http Error Code 429 increases. However, after some time, the error message keeps coming. Since the messages keeps arriving, the sleeping time keeps increasing until it reaches a point that it sleeps for so long that I lose connection.

The weirdest thing is that if I receive such error message many times in a row and, immediately, finish the process and start it again, the requests start to work again in the first try.

In other words, it seems that no matter the sleeping time I put I will start receiving such messages at some point and the only way to put it work again is restarting the process (which makes no sens at all).

How can I avoid having such error message without having to restart the process? Can anyone help me?

Thanks a lot!

EDIT:

This is the code of the request (part of it).

    from apiclient import discovery
    from oauth2client.client import GoogleCredentials
    # The other imports are omitted

    DISCOVERY_URL = 'https://{api}.googleapis.com/$discovery/rest?version={apiVersion}'  # noqa
    credentials = GoogleCredentials.get_application_default()
    self.vision = discovery.build(
        'vision', 'v1', credentials=credentials,
        discoveryServiceUrl=DISCOVERY_URL)

    batch_request = []

    for image in images:
        batch_request.append({
            'image': {
                'content': base64.b64encode(image).decode('UTF-8')
            },
            'features': [{
                'type': 'TEXT_DETECTION',
            }]
        })

    request = self.vision.images().annotate(
        body={'requests': batch_request})
sideshowbarker
  • 81,827
  • 26
  • 193
  • 197
Marcelo Lacerda
  • 1
  • 2
  • 3
  • 1
    Is there a `Retry-After` header coming back, so you know when to make the next request? – jonrsharpe Oct 11 '16 at 16:54
  • @jonrsharpe thanks for your prompt answer. i tried to get the header information, but it seems that the request is made entirely inside a method from the API I am using. Therefore, I do not have access to any response and header when it comes with an error. The API just throw an Exception (in this case HttpError). I couldn't find anything inside the exception message that could help, unfortunately. This is the call: request = self.vision.images().annotate(body={'requests': batch_request}) – Marcelo Lacerda Oct 11 '16 at 17:09
  • What library is it that you're using? Sometimes an error will have more information available if you catch and inspect it: `urllib2`'s `HttpError`, for instance, contains `info`, `message`, `reason`, etc. – a p Oct 11 '16 at 17:15
  • I have edited the post with some code. @ap I am using the Google Client API for Python. Anyway, I will look more deeply into the Exception message to see if I find something useful. Thanks! – Marcelo Lacerda Oct 11 '16 at 17:20
  • I just checked the exception attributes and did not found the Retry-After field. This is the message inside the exception message: "error": {\n "code": 429,\n "message": "Insufficient tokens for quota group and limit \'DefaultGroupUSER-100s\' of service \'vision.googleapis.com\', using the limit by ID (...)"} – Marcelo Lacerda Oct 11 '16 at 17:53

1 Answers1

1

You have used, application default credentials,

credentials = GoogleCredentials.get_application_default()

Maybe it is not able to find the credentials and use for request and so making anonymous request to API, that is not allowing more than 2 or 3 requests as in your case, I was also facing the same issue and found the work around as:

(Note: If you haven't set up an API key or service account key, please refer this doc to create one.)

for development: using API key

you could use it like:

self.vision = discovery.build(
    'vision', 'v1', credentials=credentials,
    discoveryServiceUrl=DISCOVERY_URL, developerKey='your_api_key'

)

for production: using service account key

from oauth2client.service_account import ServiceAccountCredentials
scopes = ['https://www.googleapis.com/auth/sqlservice.admin']
credentials = ServiceAccountCredentials.from_json_keyfile_name(
    '/path/to/keyfile.json', scopes=scopes)

you can find a list of scopes to use here.

Also, you need to set this environment variable: 

GOOGLE_APPLICATION_CREDENTIALS="/path/to/secret-key-file"

You need not increase the sleep time with every request, just increase it if a request fails while using any of the above-mentioned approaches. look for the exponential backoff algorithm in docs.

curioswati
  • 13
  • 6