2

logging: Stackdriver logging not capturing all logs in GKE

We have implemented logging in our flask application which is containerized, all the logs get captured when running in local docker environment and only first few logs are getting captured when using GKE for deployment, both the cases, logging implementation is the exact same. When we checked in the log file within container, all logs are written into it but not getting captured in Stackdriver logging after like 7 to 10 log entires.

Environment details
  1. Stackdriver logging in python flask app
  2. Hosted in GKE Cluster with: Stackdriver Logging API - Write Only Stackdriver Monitoring API - Full Stackdriver Trace - Write Only
  3. Python 3.7 with Pipenv
  4. google-api-core v1.14.3
  5. google-auth v1.7.1
  6. google-cloud-core v1.0.3
  7. google-cloud-logging v1.14.0
  8. googleapis-common-protos v1.6.0
Steps to reproduce
  1. deploy the same app in GKE, first few log entries get captured in stackdriver logging but after like 7 to 10 entries, nothing is showing in Stackdriver even though the log file inside the container has all the entries. This happens only if we deploy in GKE, if we are running locally, this logging works perfectly fine.
Logging implemntation inside flask 
project_id = retrieve_metadata_server('project/project-id')
cluster_name = retrieve_metadata_server('instance/attributes/cluster-name')
zone = retrieve_metadata_server('instance/attributes/cluster-location')
instance_id = retrieve_metadata_server('instance/id')

resource = Resource(
    type='container',
    labels={
        'cluster_name': cluster_name if cluster_name else '',
        'instance_id': instance_id if instance_id else '',
        'project_id': project_id if project_id else '',
        'zone': zone if zone else '',
    }
)
client = gcp_logging.Client()
handler = CloudLoggingHandler(client, name='some-name-for-logger', resource=resource)
app_logger = logging.getLogger('app')
app_logger.addHandler(handler)

ref. image: logs only coming for about 7 to 15 entries in stackdriver, after that logs are getting written to container file but not captured in Stackdriver Logs (ref. to time for each log entries for more info, repeated entries are due to restarting of system) logs only coming for about 7 to 15 entries in stackdriver

Thanks!

Thaiseer
  • 92
  • 7
  • Please note on the [limits that apply to using Stackdriver Logging](https://cloud.google.com/logging/quotas) and check your [quota](https://cloud.google.com/compute/quotas#checking_your_quota) for the sstackdriver logging. Also check the exclusion filter for [Logs ingestion](https://cloud.google.com/logging/docs/exclusions#viewing-exclusion-filters) in page of the Stackdriver Logging console. – Ahmad P Dec 10 '19 at 15:46
  • Can you post additional code as to where you're actually emitting logs? I'm wondering if something isn't right with what you're doing in the initial setup vs. the handling of the actual routes.... – Yuri Grinshteyn Dec 11 '19 at 16:45
  • @AhmadP I've checked the limits / quota, it's not even reaching 10% of the quota available and for the project I'm referring to have no exclusion filters – Thaiseer Dec 12 '19 at 03:46
  • @YuriGrinshteyn I'm logging using `app_logger` mentioned in above code to log everywhere in the application. Apart from that, the logs are properly getting written into local file inside the container. Another point is stackdriver captures all the logs if I'm running the code locally (not in GKE) – Thaiseer Dec 12 '19 at 03:51
  • @YuriGrinshteyn the codes are precisely: `app.logger.info('Default configurations loaded')` is one of the entries which is getting logged into Stackdriver and `app.logger.info(f'{request.remote_addr} -- "{request.method} {request.url}" 200 >> UserId: {userId}')` is one of the entries which is not getting logged into Stackdriver. But both entries are getting logged inside the container log file. – Thaiseer Dec 12 '19 at 05:48
  • Does anything change if you use a different severity, e.g. `app.logger.warn('message')` ? – Yuri Grinshteyn Dec 13 '19 at 16:47
  • @YuriGrinshteyn No, it doesn't matter which level you logs the message – Thaiseer Dec 16 '19 at 06:55

0 Answers0