Uber Automates Dependency Analysis Across Its Service Mesh

When a request fails, how do you figure out what actually caused it? At Uber’s scale, that gets complicated fast. With thousands of services and dependencies interacting, traditional tracing can’t always give engineers the full picture quickly enough.

In a microservices architecture, requests can traverse deeply nested and asynchronous service calls, making it difficult to determine which dependency contributed to a failure. Uber defines a fail-close dependency as one whose failure can consistently cause its caller to fail. A fail-open dependency can fail without causing its caller to fail, while insufficient evidence results in an unknown classification. The classification can then be applied across call paths to identify how failures may propagate toward an originating API.

Uber initially considered distributed tracing for establishing these relationships, but found sampling too slow for rare failures. The company says more than 99.9% of requests complete without errors and within acceptable latency bounds. At a 0.01% sampling rate and 99.9% availability, an erroneous request has a one-in-10-million chance of being sampled. At 500 requests per second, capturing one failed request could therefore take 5 hours, 33 minutes, and 20 seconds.

Uber Engineers Deepanshu Mehndiratta, Alok Srivastava and Shivam Jindal describe the approach as a trade-off between individual request visibility and faster dependency analysis.

We needed something with faster feedback, so we started identifying dependencies through logging.

Uber’s request flow through the Muttley proxy, showing caller and callee communication and telemetry (Source: Uber Blog Post)

Uber routes service-to-service requests through Muttley, an on-host proxy for routing and telemetry. However, Muttley lacks caller endpoint context for precise dependency correlation. Uber added the analysis to YARPC, its RPC framework for Go and Java, using inbound and outbound middleware.

Inbound middleware generates a request ID and records the caller service and endpoint in an in-memory RequestTracker. The ID propagates through Go’s context, enabling Outbound middleware to record the downstream service, endpoint, and call result. At request completion, middleware emits metrics capturing caller and callee failure states.

Complete request flow for mapping inbound and outbound failures (Source: Uber Blog Post)

Uber calculates a failure correlation probability for each caller dependency based on observed caller and callee outcomes. Values ≥0.8 are classified as fail-close, ≤0.2 as fail-open, and intermediate values as unknown. Retries are handled separately to avoid treating them as independent dependencies. Unlike distributed tracing pipelines built with OpenTelemetry, where tail sampling can retain traces based on errors or latency, Uber uses middleware metrics to capture production failures without the overhead of storing and evaluating individual traces, trading request-level detail for broader coverage.