Trace Context in Logs

Application logs are often critical in troubleshooting, but in a busy system it can be hard to disentangle interleaved log messages to see those for a particular transaction. One approach is to assign each transaction a unique ID that is included in the log message; the challenge then becomes how to propagate this ID through concurrency mechanisms such as threaded or asynchronous processing and across distributed services.

Fortunately, this is exactly what our agents do. When a request enters an instrumented application, the agent creates a unique trace context (aka Trace ID) and maintains it through the lifetime of the transaction in order to report performance metrics, and if sampled, the transaction trace. And now it’s possible to automatically add this Trace ID to your application logs.

Trace ID

The trace context inserted into application logs takes the form of a Trace ID that is 40 hex characters followed by a dash and one character denoting if the request is sampled (1) or not (0). Some examples:

  • 5CAE5A1CB07D4273ED5197B4DB004D20AD7F698A-1 – sampled (has a corresponding trace)
  • BB7386517A8A9216FF160D69F1EBC5DC7CF7509C-0 – not sampled (no corresponding trace)
  • 0000000000000000000000000000000000000000-0 – no valid trace context, for example the current code flow is not traced

Note

Agents will automatically insert the Trace ID with a tag or attribute name ao.traceId; please see the agent-specific pages under Enabling for details on the inserted value under different configuration options and logging libraries.

Transaction traces are also identified by this Trace ID in the Trace Details view. Have you ever noticed when pulling up a trace that the URL ends with a hex string?

TraceDetailUrl

With trace context in logs enabled, you can search application logs on this value (in the above example 72321948795199A0421FE3D949E98A1597740C26) to easily find log messages corresponding to the trace.

Cnversely, if you see a Trace ID ending in -1 in the log message, there will be a corresponding trace that you can look up in the dashboard – the easiest way is to copy the Trace ID excluding -1 from the log message, pull up any trace in the Trace Details view and replace its Trace ID in the URL with the copied value, then reload the browser.

Enabling

Trace context in logs is disabled by default. See the links below on how to enable this feature in each agent, supported logging libraries and SDK examples: