Observability

Enables correlation of work across service boundaries by propagating trace context, so engineers can reconstruct the full lifecycle of a request.

Microservices debugging: pinpointing which downstream service introduced latency in a multi-hop request

Microservices debugging: pinpointing which downstream service introduced latency in a multi-hop request Production incident response: reconstructing the call path when an error surfaces in one service but originates upstream Performance analysis: identifying the slowest span in a request that fans out across many services

instrument service entry → propagate trace context via headers → record spans for each operation → export traces to backend → query by trace ID for root cause analysis

• OpenTelemetry SDK
• Jaeger
• Zipkin
• span
• trace context
• W3C Trace Context
• baggage
• sampling
• parent span
• trace ID
• span ID
• service mesh sidecar

• Structured logging with correlation IDs: lighter weight but loses causal timing across hops Metrics-based monitoring: faster aggregation but no per-request visibility APM tools (Datadog/New Relic): vendor-managed but less portable than OpenTelemetry

Forgetting to propagate trace context across async boundaries (message queues, background jobs) → traces appear broken with orphaned spans Sampling at 100% in production → storage costs explode and backend becomes slow Treating trace ID as a secret → it leaks in logs and headers anyway, but obscuring it prevents cross-team correlation Instrumenting only the entry point → no spans for downstream calls, so the trace shows only the first hop Using vendor-specific propagators without W3C Trace Context → traces break when crossing service boundaries using different libraries

Logging: per-event records without causal linkage across services Metrics: aggregated numeric time series without per-request identity APM: broader product category that usually includes tracing plus metrics and logs

Coming from monolithic applications: may assume a single stack trace is sufficient → in distributed systems, a single trace spans many processes and requires explicit context propagation Coming from traditional logging: may treat trace IDs as just another log field → trace context must be propagated in protocol headers (HTTP, gRPC metadata, message headers), not just logged

• span
• trace context propagation
• W3C Trace Context
• OpenTelemetry
• sampling strategies
• service map

When NOT to use: single-service applications, batch jobs without cross-service calls, or when the overhead of instrumentation exceeds the debugging benefit Performance: instrumentation adds per-call overhead (typically 1–5% latency); high-cardinality trace storage can be expensive at scale Boundary conditions: traces crossing trust boundaries between organizations require careful handling of context propagation; clock skew between services can distort span timing

Reduces mean time to resolution (MTTR) for cross-service incidents from hours to minutes by making the call path and per-hop latency immediately visible, and enables proactive performance regression detection before users notice.

Distributed tracing is like following a single customer's package through every sorting facility it passes — each facility stamps its handling time, and the full route is reconstructable from the stamps alone.

OpenTelemetry collector pipelines with tail-based sampling and span-to-metrics conversion