How to collect logs of every job of a Kubernetes CronJob for a period of time?

Question

Assume I have a Kubernetes CronJob

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cron-job-logging
spec:
  schedule: "@hourly"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cron-job-logging
            image: python:3
            args:
            - /usr/bin/python3
            - -c
            - import random; print('a') if random.random() < 0.5 else print('b')
          restartPolicy: OnFailure

which runs on a GKE cluster 1.14.x with Cloud Operation for GKE activated for "System and workload logging and monitoring".

How can I collect the output for period t (let's say a month) so that I can see whether the pod printed a or b.

If seen some issues about this request, like https://github.com/kubernetes/kubernetes/issues/27768. The logs seem to be available for some users, but not for others which might be caused by the fact that CronJobs are a beta feature.

Answer 1

I've deployed your cronjob, and just for example purposes I set schedule to run each 1 minute, Below there are a few ways on how to access it:

• GKE console – In the Google Kubernetes Engine section of Google Cloud Console, select the Kubernetes resources listed in Workloads, and then the Container or Audit Logs links, this method is kind of a shortcut for the next option: cloud logging console

• Cloud Logging console – You can see your logs directly from the Cloud Logging console by using the logging filters to select the Kubernetes resources, such as cluster, node, namespace, pod, or container logs. Here are sample Kubernetes-related queries to help get you started.

  • This is the query I used (redacted project details):

    resource.type="k8s_container"
    resource.labels.project_id="PROJECT_NAME"
    resource.labels.location="ZONE"
    resource.labels.cluster_name="CLUSTER_NAME"
    resource.labels.namespace_name="NAMESPACE"
    labels.k8s-pod/job-name:"cron-job-logging-"
    

  • Here are the result:

• Cloud Monitoring console – If you have enabled a Cloud Monitoring Workspace, in the Kubernetes Engine section of the Cloud Monitoring console, select your cluster, nodes, pod, or containers to view your logs.

• gcloud command-line tool – Using the gcloud logging read command, select the appropriate cluster, node, pod, and container logs.

• Here an example:

$ gcloud logging read "resource.labels.project_id=PROJECT AND resource.labels.cluster_name=CLUSTER_NAME AND labels.k8s-pod/job-name:cron-job-logging-"
---
insertId: 6vorvd43akuvy8fi3
labels:
  k8s-pod/controller-uid: c525bbae-c6c9-11ea-931b-42010a80001f
  k8s-pod/job-name: cron-job-logging-1594838040
logName: projects/PROJECT/logs/stdout
receiveTimestamp: '2020-07-15T18:35:14.937645549Z'
resource:
  labels:
    cluster_name: CLUSTER_NAME
    container_name: cron-job-logging
    location: ZONE
    namespace_name: default
    pod_name: cron-job-logging-1594838040-pngsk
    project_id: PROJECT
  type: k8s_container
severity: INFO
textPayload: |
  a
timestamp: '2020-07-15T18:34:09.907735144Z'

More info here: GKE - Using Logs

If you have any question, let me know in the comments.