2

For App Engine (Python, Standard environment), I have created a log-export (v2) in the same project as the app. Destination of the sink is a dataset in Google BigQuery.

I can perform some simple queries in BigQuery:

SELECT
    severity,
    timestamp AS Time,
    protoPayload.host AS Host,
    protoPayload.status AS Status,
    protoPayload.resource AS Path,
    httpRequest.status,
    httpRequest.requestMethod,
    httpRequest.userAgent,
    httpRequest.remoteIp
FROM
    [MY_PROJECT:MYLOGS.appengine_googleapis_com_request_log_20170214]
LIMIT
    10

While httpRequest.status will be shown in the results with values (and the same for all the other selected fields), other fields of httpRequest are shown with null, e.g.: requestMethod, userAgent, remoteIp.

On the Cloud Log web page, I can see these log entries and these values are existing, but it seems that they are not exported to BigQuery.

When I try to filter by the request method to be GET, e.g.:

SELECT
    severity,
    timestamp AS Time,
    protoPayload.host AS Host,
    protoPayload.status AS Status,
    protoPayload.resource AS Path,
    httpRequest.status,
    httpRequest.requestMethod,
    httpRequest.userAgent,
    httpRequest.remoteIp
FROM
    [MY_PROJECT:MYLOGS.appengine_googleapis_com_request_log_20170214]
WHERE
    httpRequest.requestMethod = 'GET'
LIMIT
    10

This query will return zero records.

Any idea, why some fields are not shown in queries and cannot be used for filters in BigQuery?

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
Ani
  • 1,377
  • 1
  • 18
  • 29
  • Does this happen on previous days data too? – Pentium10 Feb 14 '17 at 19:10
  • I don't know. It's the first date I have exported the logs and first time using BigQuery. Do you think future exports may include these fields? – Ani Feb 14 '17 at 21:29
  • @Pentium10 I've tried the same query with the data of the next two days and I have the same issue. – Ani Feb 16 '17 at 11:34
  • I've found equivalent fields in protoPayload, and those values exist: protoPayload.method protoPayload.userAgent protoPayload.ip AS RemoteIP Not sure about the technical reasons, but at least the documentation makes more sense to me now, and I can work with it. – Ani Feb 16 '17 at 12:01

0 Answers0