0

I use gsutil to transfer files from a Windows machine to Google Cloud Storage.

I have not used it for more than 6 months and now when I try it I get:

Failure: invalid_grant

From researching this I suspect the access token is no longer valid as it has not been used for 6 months, and I need a refresh token?

I cannot seem to find how to get and use this.

thanks

Running gsutil -DD config produces the following output:

C:\Python27>python c:/gsutil/gsutil -DD config 

DEBUG:boto:path=/pub/gsutil.tar.gz
DEBUG:boto:auth_path=/pub/gsutil.tar.gz
DEBUG:boto:Method: HEAD
DEBUG:boto:Path: /pub/gsutil.tar.gz
DEBUG:boto:Data:
DEBUG:boto:Headers: {}
DEBUG:boto:Host: storage.googleapis.com
DEBUG:boto:Params: {}
DEBUG:boto:establishing HTTPS connection: host=storage.googleapis.com, kwargs={'timeout': 70}
DEBUG:boto:Token: None
DEBUG:oauth2_client:GetAccessToken: checking cache for key *******************************
DEBUG:oauth2_client:FileSystemTokenCache.GetToken: key=******************************* not present (cache_file= c:\users\admini~1\appdata\local\temp\2\oauth2_client-tokencache._.ea******************************)
DEBUG:oauth2_client:GetAccessToken: token from cache: None
DEBUG:oauth2_client:GetAccessToken: fetching fresh access token...
INFO:oauth2client.client:Refreshing access_token connect: (accounts.google.com, 443) 
send: 'POST /o/oauth2/token HTTP/1.1\r\nHost: accounts.google.com\r\nContent-Length: 177\r\ncontent-type: application/x- www-form-urlencoded\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.7.7 (gzip)\r\n\r\nclient_secret=******************&grant_type=refresh_token&refresh_token=****************************************&client_ id=****************.apps.googleusercontent.com' reply: 'HTTP/1.1 400 Bad Request\r\n' 
header: Content-Type: application/json; charset=utf-8 header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate header: Pragma: no-cache header: Expires: Fri, 01 Jan 1990 00:00:00 GMT header: Date: Thu, 08 May 2014 02:02:21 GMT header: Content-Disposition: attachment; filename="json.txt"; filename*=UTF-8''json.txt header: Content-Encoding: gzip header: X-Content-Type-Options: nosniff header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block header: Server: GSE header: Alternate-Protocol: 443:quic header: Transfer-Encoding: chunked
INFO:oauth2client.client:Failed to retrieve access token: { "error" : "invalid_grant" } 
Traceback (most recent call last):
  File "c:/gsutil/gsutil", line 83, in <module> gslib.__main__.main() File "c:\gsutil\gslib_main_.py", line 151, in main command_runner.RunNamedCommand('ver', ['-l']) 
  File "c:\gsutil\gslib\command_runner.py", line 95, in RunNamedCommand self._MaybeCheckForAndOfferSoftwareUpdate(command_name, debug)):
  File "c:\gsutil\gslib\command_runner.py", line 181, in _MaybeCheckForAndOfferSoftwareUpdate cur_ver = LookUpGsutilVersion(suri_builder.StorageUri(GSUTIL_PUB_TARBALL)) 
  File "c:\gsutil\gslib\util.py", line 299, in LookUpGsutilVersion obj = uri.get_key(False) 
  File "c:\gsutil\third_party\boto\boto\storage_uri.py", line 342, in get_key generation=self.generation) 
  File "c:\gsutil\third_party\boto\boto\gs\bucket.py", line 102, in get_key query_args_l=query_args_l) 
  File "c:\gsutil\third_party\boto\boto\s3\bucket.py", line 176, in _get_key_internal query_args=query_args) 
  File "c:\gsutil\third_party\boto\boto\s3\connection.py", line 547, in make_request retry_handler=retry_handler 
  File "c:\gsutil\third_party\boto\boto\connection.py", line 947, in make_request retry_handler=retry_handler) 
  File "c:\gsutil\third_party\boto\boto\connection.py", line 838, in _mexe request.authorize(connection=self) 
  File "c:\gsutil\third_party\boto\boto\connection.py", line 377, in authorize connection._auth_handler.add_auth(self, *********) 
  File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_plugin.py", line 22, in add_auth self.oauth2_client.GetAuthorizationHeader() 
  File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_client.py", line 338, in GetAuthorizationHeader return 'Bearer %s' % self.GetAccessToken().token 
  File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_client.py", line 309, in GetAccessToken access_token = self.FetchAccessToken() 
  File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_client.py", line 435, in FetchAccessToken credentials.refresh(http) 
  File "c:\gsutil\third_party\google-api-python-client\oauth2client\client.py", line 516, in refresh self._refresh(http.request) 
  File "c:\gsutil\third_party\google-api-python-client\oauth2client\client.py", line 653, in _refresh self._do_refresh_request(http_request) 
  File "c:\gsutil\third_party\google-api-python-client\oauth2client\client.py", line 710, in _do_refresh_request raise AccessTokenRefreshError(error_msg) oauth2client.client.AccessTokenRefreshError: invalid_grant
Brandon Yarbrough
  • 37,021
  • 23
  • 116
  • 145
user3610488
  • 81
  • 1
  • 2
  • 6

3 Answers3

5

You can ask gsutil to configure itself. Go to the directory with gsutil and run this:

c:\gsutil> python gsutil config

Gsutil will lead you through the steps to setting up your credentials.

That said, access tokens only normally last about a half hour. It's more likely that the previously-configured refresh token was revoked for some reason. Alternately, you can only request new tokens at a certain rate. It's possible your account has been requesting many, many refresh tokens for some reason and has been temporarily rate limited by the access service.

Brandon Yarbrough
  • 37,021
  • 23
  • 116
  • 145
  • thanks Brandon - I tried config yesterday and got: C:\gsutil>python gsutil config Failure: invalid_grant. I just tried it again and strangely I got: C:\Python27>python c:/gsutil/gsutil config Failure: Unable to find the server at accounts.google.com. – user3610488 May 07 '14 at 14:52
  • That's odd. Is it still happening? Are you perhaps behind a firewall that prevents you from accessing accounts.google.com? Can you view this URL with a web browser on that machine: https://accounts.google.com/ServiceLogin?hl=en – Brandon Yarbrough May 07 '14 at 17:58
  • Hi Brandon - yes, I can access the accounts url with Chrome, using my login. – user3610488 May 07 '14 at 20:36
  • using C:\Python27>python c:/gsutil/gsutil config - I now am back to getting invalid_grant – user3610488 May 07 '14 at 20:36
  • Could you try again using `gsutil -DD config` and post the results into your question? There will probably be a few lines in it that include "Authorization: SomeLongStringOfStuff". Make sure to remove those lines before posting. – Brandon Yarbrough May 08 '14 at 01:02
  • C:\Python27>python c:/gsutil/gsutil -DD config DEBUG:boto:path=/pub/gsutil.tar.gz DEBUG:boto:auth_path=/pub/gsutil.tar.gz DEBUG:boto:Method: HEAD DEBUG:boto:Path: /pub/gsutil.tar.gz DEBUG:boto:Data: DEBUG:boto:Headers: {} DEBUG:boto:Host: storage.googleapis.com DEBUG:boto:Params: {} DEBUG:boto:establishing HTTPS connection: host=storage.googleapis.com, kwargs={'timeout': 70} DEBUG:boto:Token: None DEBUG:oauth2_client:GetAccessToken: checking cache for key ******************************* DEBUG:oauth2_client:FileSystemTokenCache.GetToken: key=******************************* – user3610488 May 08 '14 at 02:18
  • not present (cache_file= c:\users\admini~1\appdata\local\temp\2\oauth2_client-tokencache._.ea******************************) DEBUG:oauth2_client:GetAccessToken: token from cache: None DEBUG:oauth2_client:GetAccessToken: fetching fresh access token... INFO:oauth2client.client:Refreshing access_token connect: (accounts.google.com, 443) – user3610488 May 08 '14 at 02:21
  • send: 'POST /o/oauth2/token HTTP/1.1\r\nHost: accounts.google.com\r\nContent-Length: 177\r\ncontent-type: application/x- www-form-urlencoded\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.7.7 (gzip)\r\n\r\nclient_secret=******************&grant_type=refresh_token&refresh_token=****************************************&client_ id=****************.apps.googleusercontent.com' reply: 'HTTP/1.1 400 Bad Request\r\n' – user3610488 May 08 '14 at 02:23
  • header: Content-Type: application/json; charset=utf-8 header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate header: Pragma: no-cache header: Expires: Fri, 01 Jan 1990 00:00:00 GMT header: Date: Thu, 08 May 2014 02:02:21 GMT header: Content-Disposition: attachment; filename="json.txt"; filename*=UTF-8''json.txt header: Content-Encoding: gzip header: X-Content-Type-Options: nosniff header: X-Frame-Options: SAMEORIGIN – user3610488 May 08 '14 at 02:23
  • header: X-XSS-Protection: 1; mode=block header: Server: GSE header: Alternate-Protocol: 443:quic header: Transfer-Encoding: chunked INFO:oauth2client.client:Failed to retrieve access token: { "error" : "invalid_grant" } Traceback (most recent call last): File "c:/gsutil/gsutil", line 83, in gslib.__main__.main() File "c:\gsutil\gslib\__main__.py", line 151, in main command_runner.RunNamedCommand('ver', ['-l']) File "c:\gsutil\gslib\command_runner.py", line 95, in RunNamedCommand self._MaybeCheckForAndOfferSoftwareUpdate(command_name, debug)): – user3610488 May 08 '14 at 02:24
  • File "c:\gsutil\gslib\command_runner.py", line 181, in _MaybeCheckForAndOfferSoftwareUpdate cur_ver = LookUpGsutilVersion(suri_builder.StorageUri(GSUTIL_PUB_TARBALL)) File "c:\gsutil\gslib\util.py", line 299, in LookUpGsutilVersion obj = uri.get_key(False) File "c:\gsutil\third_party\boto\boto\storage_uri.py", line 342, in get_key generation=self.generation) File "c:\gsutil\third_party\boto\boto\gs\bucket.py", line 102, in get_key query_args_l=query_args_l) File "c:\gsutil\third_party\boto\boto\s3\bucket.py", line 176, in _get_key_internal – user3610488 May 08 '14 at 02:24
  • uery_args=query_args) File "c:\gsutil\third_party\boto\boto\s3\connection.py", line 547, in make_request retry_handler=retry_handler File "c:\gsutil\third_party\boto\boto\connection.py", line 947, in make_request retry_handler=retry_handler) File "c:\gsutil\third_party\boto\boto\connection.py", line 838, in _mexe request.authorize(connection=self) File "c:\gsutil\third_party\boto\boto\connection.py", line 377, in authorize – user3610488 May 08 '14 at 02:25
  • connection._auth_handler.add_auth(self, *********) File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_plugin.py", line 22, in add_auth self.oauth2_client.GetAuthorizationHeader() File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_client.py", line 338, in GetAuthorizationHeader return 'Bearer %s' % self.GetAccessToken().token File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_client.py", line 309, in GetAccessToken access_token = self.FetchAccessToken() – user3610488 May 08 '14 at 02:25
  • File "c:\gsutil\gslib\third_party\oauth2_plugin\oauth2_client.py", line 435, in FetchAccessToken credentials.refresh(http) File "c:\gsutil\third_party\google-api-python-client\oauth2client\client.py", line 516, in refresh self._refresh(http.request) File "c:\gsutil\third_party\google-api-python-client\oauth2client\client.py", line 653, in _refresh self._do_refresh_request(http_request) File "c:\gsutil\third_party\google-api-python-client\oauth2client\client.py", line 710, in _do_refresh_request – user3610488 May 08 '14 at 02:26
  • raise AccessTokenRefreshError(error_msg) oauth2client.client.AccessTokenRefreshError: invalid_grant – user3610488 May 08 '14 at 02:27
  • Okay, here's a suggestion. I think the refresh token in your .boto file is bad for some reason (presumably revoked). I suspect if you delete the .boto file and then run config, it should solve your problem. If not, I suspect you're running a very old version of gsutil. You may want to update to a newer version. There have been a lot of bug fixes over the last year, one of which may solve your problem. – Brandon Yarbrough May 08 '14 at 18:38
  • Awesome! I found and deleted (or rather renamed) the .boto file. When I ran config again it worked fine and created a new .boto file. Now running gsutil to transfer files is back to normal. I have also updated it to the newest version of gsutil. Thanks Brandon for all your help with this. – user3610488 May 08 '14 at 21:08
3

The command to authenticate is now

$ gcloud auth login

That should refresh your grant and get you going again.

You may also want to run

$ gcloud components update

to update your installation.

Tad
  • 4,668
  • 34
  • 35
0

Brandon Yarbrough gave me suggestions which solved this problem. He suspected that the .boto file was corrupted and suggested I delete it and run gsutil config again. I did this and it solved the problem.

user3610488
  • 81
  • 1
  • 2
  • 6