1

I am using "Custom Logs Integration" from Fleet. I have done following things and I can see the logs as well in Kibana.

  1. I have created Custom Policy and added "Custom Logs Integration" to that policy.
  2. Assigned my elastic agent (one of my local server) to this custom policy.
  3. Go to the, Kibana -> Discover tab and able to see my logs in Kibana.
  4. Want to do some pre-processing before indexing docs (already done the same using logstash using grok filters), Not sure how can I do the same using Elastic agents?

Note: I am aware about the Ingest Pipeline, but not sure how can I add those pipeline in above steps. (I dn't want to use ingest APIs because I want to automate everything.)

Version: ElasticSearch : 7.10.2

Chandni Mistry
  • 13
  • 5

2 Answers2

1

You are right that Ingest Pipelines are the right tool for you, if you don't want to put a Logstash instance in the middle. The way you can integrate them in your flow is to:

  1. create the ingest pipeline that does your pre-processing job (if you're familiar with Logstash, you won't have much troubles as the ingest processors are very similar to the Logstash filters),
  2. set the pipeline created above as the default pipeline of the interested indices, so that it is applied to every document you're adding to those indices. You set the default pipeline in the index settings
PUT your-index/_settings
{
  "index.default_pipeline": "your-ingest-pipeline"
}
glenacota
  • 2,314
  • 1
  • 11
  • 18
  • Why does it seem like ingest pipelines were directed at the streams coming in, not the destinations (indices)? Or is that me? I don't fault your approach; if anything, it seems too good to be true! – danno Aug 24 '21 at 04:28
1

To add ingest pipeline for Custom Logs Integration in Fleet via Kibana (and not API)

  1. Create the ingest pipeline (Stack Management > Ingest Pipeline)

  2. Edit the concerned Custom Log Integration > Custom logs file > Expand the Advanced options > Under Cutom configurations add the pipeline as below (its YAML format and not json)

pipeline: "logs-api-json"

Elasticsearch 7.12.0

JBB
  • 317
  • 1
  • 9