Update mongo records with fluentd

Question

We need to save our nodejs application logs as records at MongoDB. Most of the records we should create, but there are some records that needs only update.

We want to use Fluentd. I see it has option to insert log to mongo, but I couldn't find a way to update existing records.

Is there a way to achieve that?

Here is my current td-agent.conf (it's still under development):

<match mongo.*>
  @type mongo
  host localhost
  port 27017
  database my-db

  # Set 'tag_mapped' if you want to use tag mapped mode.
  tag_mapped

  # If the tag is "mongo.foo", then the prefix "mongo." is removed.
  # The inserted collection name is "foo".
  remove_tag_prefix mongo.

  # This configuration is used if the tag is not found. The default is 'untagged'.
  collection misc
</match>

Answer 1

Looking at the documentation and code for fluentd mongo plugin, it looks like it doesn't support update/upsert operation.

Currently, it is performing insert_many operation only.

get_collection(database, collection, @collection_options).insert_many(records)

https://github.com/fluent/fluent-plugin-mongo/blob/master/lib/fluent/plugin/out_mongo.rb#L265

One of the option would be to create a feature request or pull request on this repo with update_many(or bulk upsert) operation but this requires you to send the update fields in every document insert.

---

Update - Any alternate tool?.

I can think of logstash but even it has bunch of feature requests opened to support update/upsert to MongoDB.

https://github.com/logstash-plugins/logstash-output-mongodb/issues/16

When you check above link comments, one advises that for logs storage you typically do not want to perform update operation.

https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

I would say first revisit your design of updating logs approach otherwise to create a custom application(Java, NodeJS, Python or .NET Core) which can perform update operation(replace fluentd or propagate from fluentd to custom app to mongodb).