Logstash multiline filter for PostgreSQL

Question

I'm trying to create filter for postgresql logs. Log records can be single-lined such as

2014-01-14 17:23:08 EST DETAIL:  Remote detail: Key (id)=(913072088342860500) already exists.

or multilined:

2014-02-04 19:26:08 EST CONTEXT:  Remote context: SQL statement "INSERT INTO     images.images_contents_part_2014_01_05 SELECT $1.*"
        PL/pgSQL function insert_find_part() line 25 at EXECUTE statement
        SQL statement "INSERT INTO images.images_contents_master SELECT NEW.*"
        PL/pgSQL function images_contents_insert_trigger() line 6 at SQL statement
        SQL statement "INSERT INTO images.images_contents (id,content_id, image, created)
              VALUES (images.images_make_id_from_created(created_at),content_id, image_path, created_at)
            RETURNING id"
        PL/pgSQL function insert_news_image(character varying,bigint,timestamp without time zone) line 11 at     SQL statement
        SQL statement "SELECT
                media_id AS id
              FROM contents.insert_news_image(i, id) AS media_id"
        PL/pgSQL function insert_news_media(bigint,character varying[],character varying[]) line 21 at SQL     statement

I need to extract info about time, type of message (context, detail, debug etc) and text of message. I suppose it is needed to use multiline codec, but I don't get how to achieve it.

Thanks!

score 0 · Answer 1 · edited May 23 '17 at 12:24

There are (probably) two viable approaches:

Merge a line with the previous line if it starts with whitespace, or
Merge a line with the previous line if it doesn't start with a timestamp.

I don't know if PostgreSQL's log format promises that all continuation lines begin with a whitespace, but probably. If not, the first option is obviously out. Here's an example of how to accomplish the former:

filter {
  multiline {
    pattern => "^\s+"
    what => "previous"
  }
}

I'm sure you can use the multiline codec instead if you prefer that.

Note that Logstash <1.5 has a bug (LOGSTASH-2067) that prevents it from picking up the last line from a multiline input. See also logstash multiline filter:last part of message flush.

Logstash multiline filter for PostgreSQL

1 Answers1