My team wants to use the ELK stack to monitor logs. We suppose we should have an index per microservice, per date. E.g. "service1-30/05/2023".
Is it considered good practice to have an index per "log type"? E.g. "service1-usercreated-30/05/2023", "service1-locationupdated-30/05/2023"
(Where the type will be inserted manually by the developer)
There are a few things to take into consideration here. The main one to think about in order to answer your question concerns the volume and the lifecycle of your data.
Fine-grained separation of your indexes might make sense from a maintenance standpoint, i.e. it is easier to delete all indexes related to a given microservice A and/or a given log type B than running a delete by query task matching the same microservice A and/or log type B.
However, if your data segregation is too fine grained, you run the risk of ending up with too many shards, and moreover, those shards might be too "small". For the record, your cluster is pre-configured to contain at most 1000 shards per data node, so if you have e.g. 3 data nodes, your cluster can contain at most 3000 shards, whatever the size of your shards. So if you have too fine-grained indexes and the data volume of some indexes is not very high, you will end up with many very empty shards counting towards that limit, and even though you have ample disk space left to accommodate for more data, your cluster will deny new write requests until your delete some shards.
That's why you need to think in terms of data retention per microservice and/or per log type. It might be the case that you need to keep data around during 30 days for microservice A and during 90 days for microservice B. In this case, it makes sense to separate data for both microservice in two different indexes (or data streams if your index is append-only) so that each of them can have its own index lifecycle policy.
The same thinking also applies for the log type if you have different log types that require different retention policies. And since you mention that the developer is in charge of specifying the log type, the "too many small shards" problem can be even more acute since you have no control over what they will specify, leading to even more indexes being created.
