Tracing service

Applications that consist of multiple services can be difficult to debug, as a single request can span multiple services. Distributed tracing tells the story of your request as it is processed through your system. Distributed tracing is a powerful tool to debug and analyze your system in addition to request logging and metrics.

When enabled, the TracingService will instruct Emissary to initiate a trace on requests by generating and populating an x-request-id HTTP header. Services can make use of this x-request-id header in logging and forward it in downstream requests for tracing. Emissary also integrates with external trace visualization services, including Zipkin-compatible APIs such as Zipkin and Jaeger to allow you to store and visualize traces. You can read further on Envoy’s Tracing capabilities.

A TracingService manifest configures Emissary to use an external trace visualization service:

---
apiVersion: getambassador.io/v3alpha1
kind:  TracingService
metadata:
  name:  tracing
spec:
  service: "example-zipkin:9411"
  driver: zipkin
  config: {}
  custom_tags:  # optional
  - tag: host
    request_header:
      name: ":authority"
      default_value: "unknown"
  - tag: path
    request_header:
      name: ":path"
      default_value: "unknown"
  sampling:
    overall: 100

Please note that you must use the HTTP/2 pseudo-header names. For example:

• the host header should be specified as the :authority header; and
• the method header should be specified as the :method header.

Supported Tracing Drivers

The TracingService currently supports the following drivers:

• zipkin
• datadog
• opentelemetry

Sampling

Configuring sampling will specify some target percentages of requests that will be traced. This is beneficial for high-volume services to control the amount of tracing data collected. Sampling can be configured with the following fields:

• client: percentage of requests that will be force traced if the x-client-trace-id header is set. Defaults to 100.
• random: percentage of requests that will be randomly traced. Defaults to 100.
• overall: percentage of requests that will be traced after all other checks have been applied (force tracing, sampling, etc.). This field functions as an upper limit on the total configured sampling rate. For instance, setting client to 100% but overall to 1% will result in only 1% of client requests with the appropriate headers to be force traced. Defaults to 100.

Custom Tags and Tag Headers

When collecting traces, Emissary will attach tags to the span’s that are generated which are useful for observability. See the Envoy Tracing Docs for the default list of data collected.

Previous versions of Emissary only supported adding additional tags through the use of the tag_headers field. This field is now deprecated and it is recommended to use custom_tags which supports a more powerful set of features for adding additional tags to a span.

custom_tags provides support for configuring additional tags based on Envoy Custom Tags. The following custom tag kinds supported are:

• request_header - set tag from header in the request
• environment - set tag from an environment variable
• literal - set tag based on a configured literal value

Each custom_tag supports setting oneOf request_header, literal or environment. Each tag should have its own entry in custom_tags.

For example:

custom_tags:
  - tag: host
    request_header: 
      name: ":authority"
      default_value: "unknown host"  # optional
  - tag: path
    request_header: ":path"
      name: ":authority"
      default_value: "unknown path"  # optional
  - tag: cluster
    literal:
      value: "us-east-cluster"
  - tag: nodeID
    environment:
      name: SERVER_ID
      default_value: "unknown"  # optional

Zipkin Driver Configurations

• collector_endpoint gives the API endpoint of the Zipkin service where the spans will be sent. The default value is /api/v2/spans
• collector_endpoint_version gives the transport version used when sending data to your Zipkin collector. The default value is HTTP_JSON and it must be one of HTTP_JSON or HTTP_PROTO.
• collector_endpoint_hostname sets the hostname Envoy will use when sending data to your Zipkin collector. The default value is the name of the underlying Envoy cluster.
• trace_id_128bit whether a 128-bit trace id will be used when creating a new trace instance. Defaults to true. Setting to false will result in a 64-bit trace id being used.
• shared_span_context whether client and server spans will share the same span id. The default value is true.

Datadog Driver Configurations

• service_name the name of the service which is attached to the traces. The default value is ambassador.

OpenTelemetry Driver Configurations

• service_name the name of the service which is attached to traces. The default value is ambassador.

Example

Check out the DataDog and Zipkin HOWTOs.