Integrate elastic search with Logrus (golang logging)

Question

I am using logrus to do all the logging my golang application. However, I also want to integrate this with Elastic Search such that all the logs are also flushed to elastic search when I create a logrus log entry. Currently all logs are created in a file as shown in the snippet below. How could I integrate with elastic search?

type LoggerConfig struct {
    Filename string `validate:regexp=.log$`
    AppName  string `validate:regexp=^[a-zA-Z]$`
}  

type AppLogger struct {
    Err    error
    Logger logrus.Entry
}

func Logger(loggerConfig LoggerConfig) AppLogger {
    response := new(AppLogger)
    // validate the schema of the logger_config
    if errs := validator.Validate(loggerConfig); errs != nil {
        response.Err = errs
        // this sets up the error on the the response struct
    }

    logrus.SetFormatter(&logrus.JSONFormatter{})
    f, err := os.OpenFile(loggerConfig.Filename, os.O_WRONLY|os.O_CREATE, 0755)
    if err != nil {
        response.Err = err
    }
    multipleWriter := io.MultiWriter(os.Stdout, f)
    logrus.SetOutput(multipleWriter)

    contextLogger := logrus.WithFields(logrus.Fields{
        "app": loggerConfig.AppName,
    })


    //logrus.AddHook(hook)
    response.Logger = *contextLogger

    //response.Logger.Info("adele")
    return *response

}

I had tried elogrus which adds a hook but I am not sure how to use it. Here is the method which attempts to create elastic search client. How could I integrate this with logrus instance?

func prepareElasticSearchClient() *elastic.Client {
    indexName := "my-server"

    client, _ := elastic.NewClientFromConfig(&config.Config{
        URL:      os.Getenv("ELASTIC_SEARCH_URL_LOGS") + ":" + os.Getenv("ELASTIC_SEARCH_PORT_LOGS"),
        Index:    indexName,
        Username: os.Getenv("ELASTIC_SEARCH_USERNAME_LOGS"),
        Password: os.Getenv("ELASTIC_SEARCH_PASSWORD_LOGS"),
    })

    return client
}

Earlier I have used modules like Winston where it was super easy to setup elastic search logging but somehow, I find little documentation with golang on how to integrate Golang logging with elastic search

Answer 1

With elogrus you first create Elastic client and pass it to elogrus hook when creating it with elogrus.NewAsyncElasticHook(). Hook just wraps sending message to Elastic. Then you add this hook to logrus log. Every time you log message using log it will fire your hook and send message (if log level filter passes) to Elastic.

log := logrus.New()
client, err := elastic.NewClient(elastic.SetURL("http://localhost:9200"))
// ... handle err
hook, err := elogrus.NewAsyncElasticHook(client, "localhost", logrus.DebugLevel, "testlog")
// ... handle err
log.Hooks.Add(hook)

Signature of NewAsyncElasticHook is (client *elastic.Client, host string, level logrus.Level, index string) where:

• client is pointer to Elastic.Client you obtained before using elastic
• host is string that denotes from which host you are sending the log trace (it's a string - hostname of host where program that logs is running)
• level is the maximum logrus log level you want messages to be sent (e.g. if you want to see DEBUG messages locally but only send only ERROR and below to Elastic)
• index is name of Elastic Search index you want to add messages from log to

From here you can use log normally as you would do with logrus and all messages will get passed to Elastic.

Another part of issue was a bit more tricky and rooted in (not only) Golang elastic client node sniffing behavior. We debugged it in chat and summary was posted as my answer to OP's another question regarding that: Cannot connect to elastic search : no active connection found: no Elasticsearch node available