3

I am building an application in a micro service architecture . So I have my different business models running on different microservices.

Microservices are using graph and document databases.

What I have to do is, I need to keep all audit logs about the objects whenever they were changed. There are couple of ways to do this,two I thought of :

  1. Store audit logs in the each databases whenever something changes to object.
  2. Instead of having it localized, make it to a central repository where we can see all the audits for whole application as behind the scenes application is served by micro services but at front this is just one app for the users and also for us. Would elastic search be used for this purpose of long term storage ? or we have other solutions ?

Which other ways are the best practices that I must follow. My objective in the end is to the when what was changed in the object by whom.

Cheers!

89n3ur0n
  • 921
  • 11
  • 24
  • I was looking for something similar. What do you decided and any pros/cons. – maverick May 04 '21 at 17:35
  • We went ahead with the database, storing audits being localized wo the microservice db. Then apply filter on the view for each of the service audits. – 89n3ur0n May 06 '21 at 12:30

1 Answers1

2

General recommendation is not to use ES as your authoritative data store. If you want 99.99% reliability for the audit data store it somewhere else, and index in ES when you need its searching abilities.

In my experience ES is quite resilient, still I keep in mind its storage is not that polished comparing to well known relational DBs or Cassandra/HDFS and I would not store important data there.

Also keep in mind ES index in not very flexible, if you want to heavily rescale your cluster or to change field mapping you may have to reindex everything. Newer versions of ES offer "Reindex API", still it's weak point.

xeye
  • 1,250
  • 10
  • 15
  • I was thinking of keeping as authoritative data store, maintaining all data audits to this . When you mention about reliability, ES can not provide that ? – 89n3ur0n May 27 '18 at 15:15
  • https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html lists considerations around resiliency. I don't think you can express this in a certain percentage for reliability. Also audit / security logging is a common use case for Elasticsearch: https://www.elastic.co/use-cases?usecase=security-analytics – xeraa May 27 '18 at 15:48