Having mapping issues when trying to save monolog logs into Elasticsearch use ElasticsearchHandler

Question

I'm trying to start sending my logs into elastic search using monolog. (I'm using Symfony2).

I've set up monolog like this:

monolog:
    handlers:
        elasticsearch:
            elasticsearch:
                host: %logger_elastic_host%
                port: %logger_elastic_port%
            type: elasticsearch
            level: info

It worked only few minutes until it broke with this error messages(a fatal error, I removed useless stuff):

create: /monolog/logs/AVQKYsGRPmEhlo7mDfrN caused MapperParsingException[failed to parse [context.stack.args]]; nested: ElasticsearchIllegalArgumentException[unknown property [class]];

I've been looking with my collegue how to fix that. What we found out is:

• Elastic search receive the first logs and automatically build a mapping
• We send new logs with another mapping or slightly different to what was sent before and it breaks.
• In this case it's breaking here: context.stack.args.

The problem is that the context will always be very different.

What we would like is:

1. is anyone out there using Monolog to log to Elasticsearch

2. How do you guys manage to avoid this issue. (How can we manage to avoid it)?

thanks guys.

Answer 1

This is happening because ES creates a mapping from the first document. If any document that is inserted after has the same property but with other type/format then ES will throw an error.

A solution is to create a custom monolog formatter and register it:

config.yml:

elasticsearch:
    type: elasticsearch
    elasticsearch:
        host: elasticsearch
    ignore_error: true
    formatter: my_elasticsearch_formatter

This line will make Monolog\Handler\ElasticSearchHandler ignore any other errors from Ruflin's Elastica package:

ignore_error: true

Then register a service with this name: my_elasticsearch_formatter:

    <service id="my_elasticsearch_formatter" class="AppBundle\Services\MyFormatter">
        <argument type="string">monolog</argument>
        <argument type="string">logs</argument>
    </service>

first argument is the index name, second arg is the type.

And the formatter class:

<?php

namespace AppBundle\Services;

use function json_encode;
use Monolog\Formatter\ElasticaFormatter;
use function var_dump;

class MyFormatter extends ElasticaFormatter
{
    /**
     * @param string $index
     * @param string $type
     */
    public function __construct($index, $type)
    {
        parent::__construct($index, $type);
    }

    /**
     * @param array $record
     * @return array|\Elastica\Document|mixed|string
     */
    public function format(array $record)
    {
        $record['context'] = json_encode($record['context']);

        return parent::format($record);
    }
}

The downside of this solution is that it will json_encode the context. You will not be able to filter by inner properties of the context in ES but at least you will not lose important information about your logs.