I am trying to understand the difference between the following terms:
Both terms seem to be used as an identifier for searching related logs produced by multiple services, especially in a microservices-oriented architecture.
Is there a subtle difference between these two?
Which one of these terms should we use in our software, and how can we decide?
Both are used as per the tool/library, system design in use. They may be related in some way or other and it depends on the application and system design. In generic, the TraceID or CorrelationID shall be a unique user defined / generated value for tracking a particular sequence of activities of interest or through out the lifetime of an application / system.
Also, as per the architecture / design, the CorrelationID can be used unique for every message or group messages based on few category/module and have an ID. In such grouping case also, CorrelationID term shall be used.
Systems based on microservices rely heavily on unique ID for debugging data associated with specific microservice for detailed info, metrics, storage and analysis. Typically, in such systems that are highly scalable, the CorrelationID can be passed from the initial process to the sub-processes that in turn shall pass it to sub systems.
Systems based on OpenTracing use term TraceID where it represents the chain from the start to end and under every trace, there can be spans for particular unit of work that are recognized by spanID which is used for building a directed acyclic graph(DAG) based maintenance.
Systems based on Sleuth use the term TraceID, where for each of the request, a unique identifier is assigned and that is maintained throughout the request processing steps in application which in turn shall have spanID tagged for unit of work / steps as part of it.
Most of the messaging systems use correlationId and so such systems using those shall use correlationId for tracking and logging . Accordingly, systems based on Java Message Service(JMS) use correlationId for request and responses. For example, message broker such as Apache ActiveMQ that implements (i.e., is compatible with) the JMS APIs uses correlation Id. Accordingly, the distributed processing engine such as Apache Flink uses correlationId in usage with RabbitMQ as that in-turn uses correlationId for RPC request and response tracking.
In my experience, I've tended to see trace ID being used strictly for correlation in logging, while correlation IDs get used additionally in domain logic (e.g. for ensuring that reads from a client see the writes made by that client).
