I am using a different approach, that is less efficient in terms on the number of logs that transit in the logging pipeline.
Similarly on how you did, I deployed one instance of filebeat on my nodes, using a daemonset. Nothing special here, this is the configuration I am using:
apiVersion: v1
data:
filebeat.yml: |-
filebeat.config:
prospectors:
# Mounted `filebeat-prospectors` configmap:
path: ${path.config}/prospectors.d/*.yml
# Reload prospectors configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
processors:
- add_cloud_metadata:
output.logstash:
hosts: ['logstash.elk.svc.cluster.local:5044']
kind: ConfigMap
metadata:
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
name: filebeat-config
And this one for the prospectors:
apiVersion: v1
data:
kubernetes.yml: |-
- type: log
paths:
- /var/lib/docker/containers/*/*.log
json.message_key: log
json.keys_under_root: true
processors:
- add_kubernetes_metadata:
in_cluster: true
namespace: ${POD_NAMESPACE}
kind: ConfigMap
metadata:
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
name: filebeat-prospectors
The Daemonset spec:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
name: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
spec:
containers:
- args:
- -c
- /etc/filebeat.yml
- -e
command:
- /usr/share/filebeat/filebeat
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: docker.elastic.co/beats/filebeat:6.0.1
imagePullPolicy: IfNotPresent
name: filebeat
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /etc/filebeat.yml
name: config
readOnly: true
subPath: filebeat.yml
- mountPath: /usr/share/filebeat/prospectors.d
name: prospectors
readOnly: true
- mountPath: /usr/share/filebeat/data
name: data
- mountPath: /var/lib/docker/containers
name: varlibdockercontainers
readOnly: true
restartPolicy: Always
terminationGracePeriodSeconds: 30
volumes:
- configMap:
name: filebeat-config
name: config
- hostPath:
path: /var/lib/docker/containers
type: ""
name: varlibdockercontainers
- configMap:
defaultMode: 384
name: filebeat-prospectors
name: prospectors
- emptyDir: {}
name: data
Basically, all data from all logs from all containers gets forwarded to logstash, reachable at the service endpoint: logstash.elk.svc.cluster.local:5044 (service called "logstash" in the "elk" namespace).
For brevity, I'm gonna give you only the configuration for logstash (if you need more specific help with kubernetes, please ask in the comments):
The logstash.yml file is very basic:
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
Just indicating the mountpoint of the directory where I mounted the pipeline config files, which are the following:
10-beats.conf:
declares an input for filebeat (port 5044 has to be exposed with a service called "logstash")
input {
beats {
port => 5044
ssl => false
}
}
49-filter-logs.conf:
this filter basically drops logs coming from pods that don't have the "elk" label. For the pods that do have the "elk" label, it keeps the logs from containers named in the "elk" label of the pod. For instance, if a Pod has two containers, called "nginx" and "python", putting a label "elk" with value "nginx" will only keep the logs coming from the nginx container and drop the python ones. The type of the log is set as the namespace the pod is running in.
This might not be a good fit for everybody (you're going to have a single index in elasticsearch for all logs belonging to a namespace) but it works for me because my logs are homologous.
filter {
if ![kubernetes][labels][elk] {
drop {}
}
if [kubernetes][labels][elk] {
# check if kubernetes.labels.elk contains this container name
mutate {
split => { "kubernetes[labels][elk]" => "." }
}
if [kubernetes][container][name] not in [kubernetes][labels][elk] {
drop {}
}
mutate {
replace => { "@metadata[type]" => "%{kubernetes[namespace]}" }
remove_field => [ "beat", "host", "kubernetes[labels][elk]", "kubernetes[labels][pod-template-hash]", "kubernetes[namespace]", "kubernetes[pod][name]", "offset", "prospector[type]", "source", "stream", "time" ]
rename => { "kubernetes[container][name]" => "container" }
rename => { "kubernetes[labels][app]" => "app" }
}
}
}
The rest of the configuration is about log parsing and is not relevant in this context.
The only other important part is the output:
99-output.conf:
Send data to elasticsearch:
output {
elasticsearch {
hosts => ["http://elasticsearch.elk.svc.cluster.local:9200"]
manage_template => false
index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Hope you got the point here.
PROs of this approach
- Once deployed filebeat and logstash, as long as you don't need to parse a new type of log, you don't need to update filebeat nor logstash configuration in order to get a new log in kibana. You just need to add a label in the pod template.
- All log files get dropped by default, as long as you don't explicitly put the labels.
CONs of this approach
- ALL logs from ALL pods come through filebeat and logstash, and get dropped only in logstash. This is a lot of work for logstash and can be resource consuming depending on the number of pods you have in your cluster.
I am sure there are better approaches to this problem, but I think this solution is quite handy, at least for my use case.
