3

I have an application that's been running since 2015. It both reads and writes to approx 16 calendars via a service account, using the Google node.js library (calendar v3 API). We also have G Suite for Education.

The general process is:

  • Every 30 seconds it caches all calendar data via a list operation
  • Periodically a student will request an appointment "slot", it first checks to see if the slot is still open (via a list call) then an insert.

That's all it does. It's been running fine until the past few days, where API insert calls started failing:

{
    "code": 403,
    "errors": [{
        "domain": "usageLimits",
        "reason": "quotaExceeded",
        "message": "Calendar usage limits exceeded."
    }]
}

This isn't all that special - the documentation has three "solutions":

  • Read more on the Calendar usage limits in the G Suite Administrator help.
  • If one user is making a lot of requests on behalf of many users of a G Suite domain, consider using a Service Account with authority delegation (setting the quotaUser parameter).
  • Use exponential backoff.
  1. I'm not exceeding any of the stated limits as far as I can tell.
  2. While I'm using a service account, it isn't making a request on behalf of a user. The service account has write access to the calendar and adds the user as an attendee
  3. Finally, I do not think exponential backoff will help, although I do not have this implemented. The time between a request to insert and the next insert call is measured in seconds, not milliseconds. Additionally, just running calls directly on the command line with a simple script produce the same problem.

Some stats:

2015 - 2,466 inserts, 186 errors
2016 - 25,747 inserts, 237 errors
2017 - 42,815 inserts, 225 errors
2018 - 41,390 inserts, 1,074 errors (990 of which are in the past 3 days)

I have updated the code over the years, but it has remained largely untouched this term.

At this point I'm unsure what to do - there is no channel to reach Google, and while I have not implemented a backoff strategy, the way timings work with this application, subsequent calls are delayed by seconds, and processed in a queue that sequentially processes requests. The only concurrent requests would be list operations.

John P.
  • 41
  • 3
  • I'm having same problem using java-sdk. I am running my application since February 2018, and the problem started at December 11 2018 AM 5:40 UTC and not fixed yet. I asked the GSuite support team and got an answer that it might be fixed within 2-3 days. – Hirofumi Okino Dec 14 '18 at 04:24
  • BTW, I created a new G Suite account (not user) and tried the same thing with it, first it took about 20 hours getting rid of 404 (not found) errors then it SUCCEEDED a couple of times, but then it started to fail with same "limit exceeded" errors. Now, it failes with 404 (not found) errors again. I think that I really need to implement OAuth instead of using Service Account. – Hirofumi Okino Dec 14 '18 at 04:40
  • The "first" problem for me started at Dec 11 2018 3:11 UTC. As an interim step I created a new project under our "domain" account (instead of my "personal" account), new service account, and it worked for a few hours, but at some point in the day it started getting errors. I then swapped between the accounts a few times (I know, horrible practice) to keep things running. Right now I haven't seen any errors since yesterday. – John P. Dec 14 '18 at 15:18

0 Answers0