Google Kubernetes Engine - cannot see logs containing pod's output

Question

In freshly installed cluster (Kubernetes version: v1.22.8-gke.201) I have the following test deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  labels:
    app: busybox
spec:
  replicas: 1
  strategy: 
    type: RollingUpdate
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
        - name: busybox
          image: busybox:latest
          imagePullPolicy: IfNotPresent
          command: ['sh', '-c', 'while true; do printf "$(date +%Y-%m-%d\ %H:%M:%S\ %Z) => your lucky number is: $RANDOM\n"; sleep 5; done']

I can see output produced by the pod using:

kubectl logs <pod_name>

But in the console, after going to Kubernetes Engine -> Workloads, in Overview tab, Logs, clicking Container Logs link redirects to Operations Logging and all Logs Explorer returns is: 'No data found'.

In Cluster details, section Features, Cloud Logging is enabled and Components set as 'System and Workloads' (which is the default).

What else should be done to be able to see pod's output in Logs Explorer?

Answer 1

Answering to myself:

Enabling Cloud Logging API resolved the issue:

$ gcloud services enable logging.googleapis.com

It should be visible on the list of enabled APIs:

$ gcloud services list
NAME                    TITLE
[...]
logging.googleapis.com  Cloud Logging API
[...]

Answer 2

I ran into the same problem, but the solution was different.

I provisioned a GKE Node pool that didn't use the default service account, so it turned out the non-standard service account I used had insufficient rights to send 'kubectl logs $podname' to GCP's Logs Explorer. So if you're running into something similar check the service account associated with your node pool and consider adding more rights to match the default service account.

Per https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster
I ran the following commands to add the necessary principle of least privilege rights to the custom service account.

Set Variables:
export PROJECT=my-project
export SA_NAME=my-sa@my-project.iam.gserviceaccount.com

See pre-change IAM rights:
gcloud projects get-iam-policy $PROJECT
--flatten="bindings[].members"
--format='table(bindings.role)'
--filter="bindings.members:$SA_NAME"

Add least privilege roles:
gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$SA_NAME --role=roles/monitoring.viewer
gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$SA_NAME --role=roles/monitoring.metricWriter
gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$SA_NAME --role=roles/logging.logWriter
gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$SA_NAME --role=roles/stackdriver.resourceMetadata.writer

See post-change IAM rights:
gcloud projects get-iam-policy $PROJECT
--flatten="bindings[].members"
--format='table(bindings.role)'
--filter="bindings.members:$SA_NAME"

Answer 3

I would beleive that if you can see the logs with the kubectl get logs mypod sending all the logs to stdout is already set up but you still might want to check that in your app.

However, since you are running busybox, there is not ''real'' application running that logs to stdout. So it might be you issue.

Check that the main PID, let say you run Apache and not BusyBox, send all of it's logs to stdout. Then you should see the logs to the terminal without the kubectl command.