0

Does kubernetes capture - in the event log or elsewhere - when Secrets are deleted? I'm not seeing this showing up in the event log: kubectl get events --field-selector involvedObject.kind=Secret --watch and I'm not finding it in /var/log/pods/kube_system-kube-apiserver/either.

We're using something called external-secrets-operator to manage secrets, and we would like to be informed when a secret is deleted, as this isn't supposed to happen with ESO, so it's an edge case that we'd like to monitor for.

Michael Martinez
  • 2,645
  • 3
  • 24
  • 35
  • I presume -- but don't know for sure, hence the comment rather than answer -- that this would show up in the [audit logs](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/). – larsks Jul 06 '23 at 03:41
  • Did you have time to check my answer? It helped you to solve your issue? If not, I am happy to assist further.
     – Sai Chandra Gadde Jul 07 '23 at 16:28
  • 1
    yeah I checked. We don't have auditing enabled. Would have to enable it if we want these events. – Michael Martinez Jul 08 '23 at 17:09

1 Answers1

1

Using event log kubectl get events --field-selector involvedObject.kind=Secret --watch you will get existing secrets not the deleted resources, for deleted resources you need to use Audit logs, audit logs are automatically enabled for some managed kubernetes like GKE and AWS. Managed Kubernetes services like EKS, AKS or GCP provide support by routing cluster audit logs into centralized logging services (like AWS CloudWatch,GCP logs explorer) easily.

To find out who deleted the namespace and at what time, it only gives the IAM user, below query is used to get when and who deleted the secret.

logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Factivity"
resource.type="k8s_cluster"
protoPayload.methodName:"io.k8s.core.v1.secrets"
NOT protoPayload.methodName:"get"
NOT protoPayload.methodName:"list"
NOT protoPayload.methodName:"watch"

If you are not using any managed kubernetes, follow this blog written by Daniel Olaogun which elaborated on how to use the audit logs.

Sai Chandra Gadde
  • 140
  • 7