0

I have Google Cloud Storage configured with a CNAME according to the documentation, so basically I have my Google Load Balancer handling the requests. In almost 2 years with this configuration I didn't have a problem (traffic is higher now, so the problem was probably obfuscated...I cannot know this). I am using signed URLs.

I have started having some apparently random 403 responses with the PUT operation on the storage, so basically with the file upload operation. Apparently random because it works most of the time (I would say 99%), but the problems are becoming evident in the log viewer and of course in the support requests from our users.

Here's a log entry (some details obfuscated):

{
  "insertId": "<id_here>",
  "jsonPayload": {
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
    "statusDetails": "response_sent_by_backend"
  },
  "httpRequest": {
    "requestMethod": "PUT",
    "requestUrl": "https://storage.mydomain.it/path/2/2021-06-18/m_808/2_2021-06-18-09-09-24_m808_bdb6b3.png?GoogleAccessId=ocr-account%40mydomain.iam.gserviceaccount.com&Expires=1624000464&Signature=<signature_here>",
    "requestSize": "1188472",
    "status": 403,
    "responseSize": "658",
    "userAgent": "Apache-CXF/3.2.5",
    "remoteIp": "88.86.183.XXX",
    "latency": "0.107676s"
  },
  "resource": {
    "type": "http_load_balancer",
    "labels": {
      "backend_service_name": "",
      "forwarding_rule_name": "mydomain-balancer-1-forwarding-rule-2",
      "url_map_name": "mydomain-balancer-1",
      "target_proxy_name": "mydomain-balancer-1-target-proxy-2",
      "project_id": "my_project_id",
      "zone": "global"
    }
  },
  "timestamp": "2021-06-18T07:09:25.037700Z",
  "severity": "WARNING",
  "logName": "projects/myproject/logs/requests",
  "trace": "projects/myproject/traces/cc8123c2a42e70d8d002c84efe15e5de",
  "receiveTimestamp": "2021-06-18T07:09:25.668260183Z",
  "spanId": "e4b4a200767c9de0"
}

response_sent_by_backend is probably an indication that the storage itself has sent a 403 error. Now, there are a lot of possible reasons for a 403 error, but the log viewer is not giving any other information.

How can I approach this problem in order to understand what's happening?

Dave M
  • 4,514
  • 22
  • 31
  • 30
Lorenzo S
  • 101
  • 1) The expiration time is too short. In your log example, the URL expires within five minutes from the log entry. You should be able to get more details in the Cloud Storage logs. 2) The user agent is Apache. Is this your service? If yes, make sure the date, time and time zone are set correctly. – John Hanley Jun 18 '21 at 16:25
  • Hi @JohnHanley, thanks for your reply. For the expiration problem, I see that the timestamp of the log is "Fri Jun 18 2021 09:09:25 GMT+0200", and the URL expiration date is "Fri Jun 18 2021 09:14:24 GMT+0200 (Central European Summer Time)" so I think we're good in this sense. For the user agent: this can be an interesting observation. I will ask the partner that is using the service (in fact, it's the only partner having this problem). Thanks! – Lorenzo S Jun 21 '21 at 07:17
  • Your comment leads me to believe your partner's system time is incorrect. The date/time in a request must be correct for Signed URLs. What is the expiration time for your Signed URLS (time from issuance)? Google rotates the private keys so limit the total valid time to less than 12 hours. https://cloud.google.com/storage/docs/access-control/signed-urls#signing-iam – John Hanley Jun 21 '21 at 18:35

0 Answers0