Why does my Drive API request fail when there is more than one PATCH request in a batch?

Question

I'm trying to add a new parent folder to a large number of my Google Drive files. I'm using batch requests to avoid sending too many requests, but this doesn't work when there is more than one PATCH request added to the batch. The Google server is returning a 502 Bad Gateway HTTP error response. What is wrong or how can I workaround this?

General:

Remote Address:216.58.196.138:443
Request URL:https://content.googleapis.com/batch
Request Method:POST
Status Code:502 OK

Response headers:

alt-svc:quic=":443"; ma=604800; v="30,29,28,27,26,25"
alternate-protocol:443:quic,p=1
content-length:1613
content-type:text/html; charset=UTF-8
date:Tue, 01 Dec 2015 06:18:36 GMT
server:GFE/2.0
status:502

Request headers:

:authority:content.googleapis.com
:method:POST
:path:/batch
:scheme:https
accept:*/*
accept-encoding:gzip, deflate
accept-language:ru,en-US;q=0.8,en;q=0.6,de;q=0.4,pt;q=0.2,uk;q=0.2,it;q=0.2
authorization:Bearer ya29.PQI-rcaU2_Pd7tq3XufXVWPfje4ioMzBLhvKOieYuFgs2MnxTyQvtJ0QDR7EiWnN5SOPlGo
content-length:2291
content-type:multipart/mixed; boundary=batch691088898561756483
dnt:1
origin:https://content.googleapis.com
referer:https://content.googleapis.com/static/proxy.html?jsh=m%3B%2F_%2Fscs%2Fapps-static%2F_%2Fjs%2Fk%3Doz.gapi.en.Dm6zFqHZO9M.O%2Fm%3D__features__%2Fam%3DAQ%2Frt%3Dj%2Fd%3D1%2Ft%3Dzcms%2Frs%3DAGLTcCPQ-9iC9Y-vGIOJ-RoQCZkFL66QXw
user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
x-chrome-uma-enabled:1
x-client-data:CKW2yQEIqbbJAQjEtskBCPCIygEI/ZXKAQi8mMoB
x-clientdetails:appVersion=5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_3)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F46.0.2490.86%20Safari%2F537.36&platform=MacIntel&userAgent=Mozilla%2F5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_3)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F46.0.2490.86%20Safari%2F537.36
x-goog-encode-response-if-executable:base64
x-javascript-user-agent:google-api-javascript-client/1.1.0-beta
x-origin:http://app.local
x-referer:http://app.local

Request payload:

--batch691088898561756483
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <0B28tlN6SgYRacHVwdEQ5b1RtZmc>

PATCH /drive/v2/files/0B28tlN6SgYRacHVwdEQ5b1RtZmc?addParents=0B28tlN6SgYRaS2syM1VGWjllMnc
X-JavaScript-User-Agent: google-api-javascript-client/0.1
Authorization: Bearer ya29.PQI-rcaU2_Pd7tq3XufXVWPfje4ioMzBLhvKOieYuFgs2MnxTyQvtJ0QDR7EiWnN5SOPlGo
X-ClientDetails: appVersion=5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_3)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F46.0.2490.86%20Safari%2F537.36&platform=MacIntel&userAgent=Mozilla%2F5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_3)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F46.0.2490.86%20Safari%2F537.36

--batch691088898561756483
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <0B28tlN6SgYRadVFRV0lRQW1kVlE>

PATCH /drive/v2/files/0B28tlN6SgYRadVFRV0lRQW1kVlE?addParents=0B28tlN6SgYRaS2syM1VGWjllMnc
X-JavaScript-User-Agent: google-api-javascript-client/0.1
Authorization: Bearer ya29.PQI-rcaU2_Pd7tq3XufXVWPfje4ioMzBLhvKOieYuFgs2MnxTyQvtJ0QDR7EiWnN5SOPlGo
X-ClientDetails: appVersion=5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_3)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F46.0.2490.86%20Safari%2F537.36&platform=MacIntel&userAgent=Mozilla%2F5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_3)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F46.0.2490.86%20Safari%2F537.36

--batch691088898561756483
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: <0B28tlN6SgYRaLW1qYmxrZUV2S0E>

PATCH /drive/v2/files/0B28tlN6SgYRaLW1qYmxrZUV2S0E?addParents=0B28tlN6SgYRaS2syM1VGWjllMnc
X-JavaScript-User-Agent: google-api-javascript-client/0.1
Authorization: Bearer ya29.PQI-rcaU2_Pd7tq3XufXVWPfje4ioMzBLhvKOieYuFgs2MnxTyQvtJ0QDR7EiWnN5SOPlGo
X-ClientDetails: appVersion=5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_3)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F46.0.2490.86%20Safari%2F537.36&platform=MacIntel&userAgent=Mozilla%2F5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_10_3)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Chrome%2F46.0.2490.86%20Safari%2F537.36

--batch691088898561756483--

Also run files.insert in batch works OK, but files.patch not. — Vlad Tsepelev, Dec 01 '15 at 08:53
@DalmTo im not trying do anything with permissions. My problem is related to running requests in batches. Everything is OK with requests, but not when i run these requests in batches. Please, read carefully before commenting. — Vlad Tsepelev, Dec 02 '15 at 10:43
Imho, batching requests is a waste of time. Just do them individually. — pinoyyid, Dec 02 '15 at 18:38
I need to update 3000 files. 3000 requests is alot to send, instead of 3 batched requests. — Vlad Tsepelev, Dec 03 '15 at 06:25
The problem I've found is that if you send a batch, the majority will fail with rate limit 403 errors, so you'll need to resend them, thus wasting time and bandwidth. When sending 3000 updates you will find that you need to throttle your requests to around 1 per second. — pinoyyid, Dec 03 '15 at 07:20
On your specific question, any 5xx error is a server error. In Google Drive that either means a bug, or an internal timeout. How many PATCH requests are in your batch, eg if you do a batch containing 1, 2, 4, 8 etc PATCHES, do they all fail? — pinoyyid, Dec 03 '15 at 07:22
As I mentioned before, batch with 1 request works OK everytime. But 2 and more requests in batch always fail with error 502. This looks like a bug on Google side. — Vlad Tsepelev, Dec 03 '15 at 10:59
Like I said, abandon batch.It doesn't do what you think it does.you'll have a lot more success with individual requests — pinoyyid, Dec 03 '15 at 16:25

score 1 · Answer 1 · answered Dec 03 '15 at 13:57

Try changing the URL to https://www.googleapis.com/batch instead of https://content.googleapis.com/batch

Using your exact request and data but changing the URL I received a 200 with an authentication error for each request, as expected for old tokens.

HTTP/1.1 200 OK
Vary: Origin
Vary: X-Origin
Content-Type: multipart/mixed; boundary=batch_NP9EsgqJk5k_AAFjWZ0MHaM
Date: Thu, 03 Dec 2015 13:52:18 GMT
Expires: Thu, 03 Dec 2015 13:52:18 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alternate-Protocol: 443:quic,p=1
Alt-Svc: quic=":443"; ma=604800; v="30,29,28,27,26,25"
Content-Length: 2027

--batch_NP9EsgqJk5k_AAFjWZ0MHaM
Content-Type: application/http
Content-ID: <response-0B28tlN6SgYRacHVwdEQ5b1RtZmc>

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="https://accounts.google.com/", error=invalid_token
Content-Type: application/json; charset=UTF-8
Date: Thu, 03 Dec 2015 13:52:18 GMT
Expires: Thu, 03 Dec 2015 13:52:18 GMT
Cache-Control: private, max-age=0
Content-Length: 249

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

--batch_NP9EsgqJk5k_AAFjWZ0MHaM
Content-Type: application/http
Content-ID: <response-0B28tlN6SgYRadVFRV0lRQW1kVlE>

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="https://accounts.google.com/", error=invalid_token
Content-Type: application/json; charset=UTF-8
Date: Thu, 03 Dec 2015 13:52:18 GMT
Expires: Thu, 03 Dec 2015 13:52:18 GMT
Cache-Control: private, max-age=0
Content-Length: 249

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

--batch_NP9EsgqJk5k_AAFjWZ0MHaM
Content-Type: application/http
Content-ID: <response-0B28tlN6SgYRaLW1qYmxrZUV2S0E>

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="https://accounts.google.com/", error=invalid_token
Content-Type: application/json; charset=UTF-8
Date: Thu, 03 Dec 2015 13:52:18 GMT
Expires: Thu, 03 Dec 2015 13:52:18 GMT
Cache-Control: private, max-age=0
Content-Length: 249

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "authError",
    "message": "Invalid Credentials",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Invalid Credentials"
 }
}

--batch_NP9EsgqJk5k_AAFjWZ0MHaM--

Im using js gapi library and have high level wrapper, so I cant change url. Im creating batch like this gapi.client.newHttpBatch(). — Vlad Tsepelev, Dec 04 '15 at 08:01

score 1 · Answer 2 · answered Dec 10 '15 at 07:46

Google's batch API is poorly documented. Here are a few ideas to try:

You may be using the wrong endpoint (https://content.googleapis.com/batch instead of https://www.googleapis.com/batch), as Brian has already suggested. I know from experience that you can't batch together requests for different hostnames, even if you provide the correct Host: header inside each of the batched requests.
It looks like you are submitting the batch request from a Chrome browser, and it is using SPDY or HTTP/2 to submit the request. What happens if you use a commandline tool such as curl to send the request using HTTP/1.1 instead?
What is the exact error response? It looks like you are getting a 502 Bad Gateway response with a text/html response body, which is very strange. The expected behaviour would be a 200 OK response with a multipart/mixed response body containing an application/http message for each of the individual requests that you submitted.
None of the PATCH requests in your batch have a body, which might be an issue. Since all you are trying to do is add a new parent folder to each file, what happens if you instead try making POST https://www.googleapis.com/drive/v2/files/{fileId}/parents requests in your batch?

If you are seeking a response from Google, you can try submitting a bug on the official Google Apps API issue tracker. There is no guarantee that the Google Drive team will respond, however, and I have often seen valid and reproducible bugs remain unacknowledged for months at a time.

score 0 · Accepted Answer · answered Jan 21 '16 at 10:00

0

This problem is solved in Google Drive API v3. Batches work OK.

answered Jan 21 '16 at 10:00

Vlad Tsepelev

2,056
1
21
32

What is the solution? – vesszabo Jan 28 '20 at 10:24

Why does my Drive API request fail when there is more than one PATCH request in a batch?

3 Answers3