Ambassador Module

If present, the ambassador Module defines system-wide configuration for Emissary. You won’t need it unless you need to change one of the system-wide configuration settings below.

To use the ambassador Module to configure Emissary, it MUST be named ambassador, otherwise it will be ignored. To create multiple ambassador Modules in the same Kubernetes namespace, you will need to apply them as annotations with separate ambassador_ids: you will not be able to use multiple CRDs.

There are many items that can be configured on the ambassador Module. They are listed below with examples and grouped by category.

Envoy

Content-Length headers

• allow_chunked_length: true tells Envoy to allow requests or responses with both Content-Length and Transfer-Encoding headers set. The default is false.

By default, messages with both Content-Length and Content-Transfer-Encoding are rejected. If allow_chunked_length is true, Emissary will remove the Content-Length header and process the message. See the Envoy documentation for more details.

Envoy access logs

• envoy_log_path defines the path of Envoy’s access log. By default this is standard output.
• envoy_log_type defines the type of access log Envoy will use. Currently, only json or text are supported.
• envoy_log_format defines the Envoy access log line format.

These logs can be formatted using Envoy operators to display specific information about an incoming request. The example below will show only the protocol and duration of a request:

envoy_log_path: /dev/fd/1
envoy_log_type: json
envoy_log_format:
  {
    "protocol": "%PROTOCOL%",
    "duration": "%DURATION%"
  }

See the Envoy documentation for the standard log format and a complete list of log format operators.

Envoy validation timeout

• envoy_validation_timeout defines the timeout, in seconds, for validating a new Envoy configuration. The default is 10.

A value of 0 disables Envoy configuration validation. Most installations will not need to change this setting.

For example:

envoy_validation_timeout: 30

would allow 30 seconds to validate the generated Envoy configuration.

Error response overrides

• error_response_overrides permits changing the status code and body text for 4XX and 5XX response codes. The default is not to override any error responses.

By default, Emissary will pass through error responses without modification, and errors generated locally will use Envoy’s default response body, if any.

See using error response overrides for usage details. For example, this configuration:

error_response_overrides:
 - on_status_code: 404
   body:
     text_format: "File not found"

would explicitly modify the body of 404s to say “File not found”.

Forwarding client cert details

Two attributes allow providing information about the client’s TLS certificate to upstream certificates:

• forward_client_cert_details: true will tell Envoy to add the X-Forwarded-Client-Cert to upstream requests. The default is false.
• set_current_client_cert_details will tell Envoy what information to include in the X-Forwarded-Client-Cert header. The default is not to include the X-Forwarded-Client-Cert header at all.

Emissary will not forward information about a certificate that it cannot validate.

See the Envoy documentation on X-Forwarded-Client-Cert and SetCurrentClientCertDetails for more information.

forward_client_cert_details: true
set_current_client_cert_details: SANITIZE

Server name

• server_name allows overriding the server name that Envoy sends with responses to clients. The default is envoy.

Suppress Envoy headers

• suppress_envoy_headers: true will prevent Emissary from emitting certain additional headers to HTTP requests and responses. The default is false.

For the exact set of headers covered by this config, see the Envoy documentation

General

Ambassador ID

• ambassador_id allows using multiple instances of Emissary in the same cluster. The default is unset.

We recommend not setting ambassador_id if you are running only one instance of Emissary in your cluster. For more information, see the Running and Deployment documentation.

If used, the ambassador_id value must be an array, for example:

ambassador_id: [ "test_environment" ]

Defaults

• defaults provides a dictionary of default values that will be applied to various Emissary resources. The default is to have no defaults configured.

See Using ambassador Module Defaults for more information.

gRPC

Bridges

• enable_grpc_http11_bridge: true will enable the gRPC-HTTP/1.1 bridge. The default is false.
• enable_grpc_web: true will enable the gRPC-Web bridge. The default is false.

gRPC is a binary HTTP/2-based protocol. While this allows high performance, it can be problematic for clients that are unable to speak HTTP/2 (such as JavaScript in many browsers, or legacy clients in difficult-to-update environments).

The gRPC-HTTP/1.1 bridge can translate HTTP/1.1 calls with Content-Type: application/grpc into gRPC calls: Emissary will perform buffering and translation as necessary. For more details on the translation process, see the Envoy gRPC HTTP/1.1 bridge documentation.

Likewise, gRPC-Web is a JSON and HTTP-based protocol that allows browser-based clients to take advantage of gRPC protocols. The gRPC-Web specification requires a server-side proxy to translate between gRPC-Web requests and gRPC backend services, and Emissary can fill this role when the gRPC-Web bridge is enabled. For more details on the translation process, see the Envoy gRPC HTTP/1.1 bridge documentation; for more details on gRPC-Web itself, see the gRPC-Web client GitHub repo.

Statistics

• grpc_stats allows enabling telemetry for gRPC calls using Envoy’s gRPC Statistics Filter. The default is disabled.

---
apiVersion: getambassador.io/v3alpha1
kind:  Module
metadata:
  name: ambassador
spec:
  config:
    grpc_stats:
      upstream_stats: true
      services:
        - name: <package>.<service>
          method_names: [<method>]

Supported parameters:

• all_methods
• services
• upstream_stats

Available metrics:

• envoy_cluster_grpc_<service>_<status_code>
• envoy_cluster_grpc_<service>_request_message_count
• envoy_cluster_grpc_<service>_response_message_count
• envoy_cluster_grpc_<service>_success
• envoy_cluster_grpc_<service>_total
• envoy_cluster_grpc_upstream_<stats> - only when upstream_stats: true

Please note that <service> will only be present if all_methods is set or the service and the method are present under services. If all_methods is false or the method is not on the list, the available metrics will be in the format envoy_cluster_grpc_<stats>.

• all_methods: If set to true, emit stats for all service/method names. If set to false, emit stats for all service/message types to the same stats without including the service/method in the name. This option is only safe if all clients are trusted. If this option is enabled with untrusted clients, the clients could cause unbounded growth in the number of stats in Envoy, using unbounded memory and potentially slowing down stats pipelines.

• services: If set, specifies an allow list of service/methods that will have individual stats emitted for them. Any call that does not match the allow list will be counted in a stat with no method specifier (generic metric).

• upstream_stats: If true, the filter will gather a histogram for the request time of the upstream.

Header behavior

Header case

• proper_case: true forces headers to have their “proper” case as shown in RFC7230. The default is false.
• header_case_overrides allows forcing certain headers to have specific casing. The default is to override no headers.

proper_case and header_case_overrides are mutually exclusive.

RFC7230 specifies that HTTP header names are case-insensitive, but always shows and refers to headers as starting with a capital letter, continuing in lowercase, then repeating the single capital letter after each non-alpha character. This has become an established convention when working with HTTP:

• Host, not host or HOST
• Content-Type, not content-type, Content-type, or cOnTeNt-TyPe

Internally, Envoy typically uses all lowercase for header names. This is fully compliant with RFC7230, but some services and clients may require headers to follow the stricter casing rules implied by RFC7230 section headers: in that situation, setting proper_case: true will tell Envoy to force all headers to use the casing above.

Alternately, it is also possible - although less common - for services or clients to require some other specific casing for specific headers. header_case_overrides specifies an array of header names: if a case-insensitive match for a header is found in the list, the matching header will be replaced with the one in the list. For example, the following configuration will force headers that match X-MY-Header and X-EXPERIMENTAL to use that exact casing, regardless of the original case used in flight:

header_case_overrides:
- X-MY-Header
- X-EXPERIMENTAL

If the upstream service responds with x-my-header: 1, Emissary will return X-MY-Header: 1 to the client. Similarly, if the client includes x-ExperiMENTAL: yes in its request, the request to the upstream service will include X-EXPERIMENTAL: yes. Other headers will not be altered; Emissary will use its default lowercase header.

Please see the Envoy documentation for more information. Note that in general, we recommend updating clients and services rather than relying on header_case_overrides.

Linkerd interoperability

• add_linkerd_headers: true will force Emissary to include the l5d-dst-override header for Linkerd. The default is false.

When using older Linkerd installations, requests going to an upstream service may need to include the l5d-dst-override header to ensure that Linkerd will route them correctly. Setting add_linkerd_headers does this automatically. See the Mapping documentation for more details.

Max request headers size

• max_request_headers_kb sets the maximum allowed request header size in kilobytes. If not set, the default is 60 KB.

See Envoy documentation for more information.

Preserve external request ID

• preserve_external_request_id: true will preserve any X-Request-Id header presented by the client. The default is false, in which case Envoy will always generate a new X-Request-Id value.

Strip matching host port

• strip_matching_host_port: true will tell Emissary to strip any port number from the host/authority header before processing and routing the request if that port number matches the port number of Envoy’s listener. The default is false, which will preserve any port number.

In the default installation of Emissary the public port is 443, which then maps internally to 8443, so this only works in custom installations where the public Service port and Envoy listener port match.

A common reason to try using this property is if you are using gRPC with TLS and your client library appends the port to the Host header (i.e. myurl.com:443). We have an alternative solution in our gRPC guide that uses a Lua script to remove all ports from every Host header for that use case.

Miscellaneous

Envoy’s admin port

• admin_port specifies the port where Emissary’s Envoy will listen for low-level admin requests. The default is 8001; it should almost never need changing.

Lua scripts

• lua_scripts allows defining a custom Lua script to run on every request. The default is to run no script.

This is useful for simple use cases that mutate requests or responses, for example to add a custom header:

lua_scripts: |
  function envoy_on_response(response_handle)
    response_handle:headers():add("Lua-Scripts-Enabled", "Processed")
  end  

For more details on the Lua API, see the Envoy Lua filter documentation.

Some caveats around the embedded scripts:

• They run in-process, so any bugs in your Lua script can break every request.
• They’re run on every request/response to every URL.
• They’re inlined in the Emissary YAML; as such, we do not recommend using Lua scripts for long, complex logic.

If you need more flexible and configurable options, Ambassador Edge Stack supports a pluggable Filter system.

Merge slashes

• merge_slashes: true will cause Emissary to merge adjacent slashes in incoming paths when doing route matching and request filtering. The default is false.

For example, with merge_slashes: true, a request for //foo///bar would be matched to a Mapping with prefix /foo/bar.

Modify Default Buffer Size

By default, the Envoy that ships with Emissary uses a defailt of 1MiB soft limit for an upstream service’s read and write buffer limits. This setting allows you to configure that buffer limit. See the Envoy docs for more information.

buffer_limit_bytes: 5242880 # Sets the default buffer limit to 5 MiB

Use Emissary namespace for service resolution

• use_ambassador_namespace_for_service_resolution: true tells Emissary to assume that unqualified services are in the same namespace as Emissary. The default is false.

By default, when Emissary sees a service name without a namespace, it assumes that the namespace is the same as the resource referring to the service. For example, for this Mapping:

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: mapping-1
  namespace: foo
spec:
  hostname: "*"
  prefix: /
  service: upstream

Emissary would look for a Service named upstream in namespace foo.

However, if use_ambassador_namespace_for_service_resolution is true, this Mapping would look for a Service named foo in the namespace in which Emissary is installed instead.

Observability

Diagnostics

• diagnostics controls access to the diagnostics interface.

By default, Emissary creates a Mapping that allows access to the diagnostic interface at /ambassador/v0/diag from anywhere in the cluster. To disable the Mapping entirely, set diagnostics.enabled to false:

diagnostics:
  enabled: false

With diagnostics disabled, /ambassador/v0/diag will respond with 404; however, the service itself is still running, and /ambassador/v0/diag/ is reachable from inside the Emissary Pod at https://localhost:8877. You can use Kubernetes port forwarding to set up remote access to the diagnostics page temporarily:

kubectl port-forward -n ambassador deploy/ambassador 8877

Alternately, to leave the Mapping intact but restrict access to only the local Pod, set diagnostics.allow_non_local to false:

diagnostics:
  allow_non_local: true

See Protecting Access to the Diagnostics Interface for more information.

Protocols

Enable IPv4 and IPv6

• enable_ipv4 determines whether IPv4 DNS lookups are enabled. The default is true.
• enable_ipv6 determines whether IPv6 DNS lookups are enabled. The default is false.

If both IPv4 and IPv6 are enabled, Emissary will prefer IPv6. This can have strange effects if Emissary receives AAAA records from a DNS lookup, but the underlying network of the pod doesn’t actually support IPv6 traffic. For this reason, the default is IPv4 only.

An Mapping can override both enable_ipv4 and enable_ipv6, but if either is not stated explicitly in a Mapping, the values here are used. Most Emissary installations will probably be able to avoid overriding these settings in Mappings.

HTTP/1.0 support

• enable_http10: true will enable handling incoming HTTP/1.0 and HTTP/0.9 requests. The default is false.

Security

Cross origin resource sharing (CORS)

• cors sets the default CORS configuration for all mappings in the cluster. The default is that CORS is not configured.

For example:

cors:
  origins: http://foo.example,http://bar.example
  methods: POST, GET, OPTIONS
  ...

See the CORS syntax for more information.

IP allow and deny

• ip_allow and ip_deny define HTTP source IP address ranges to allow or deny. The default is to allow all traffic.

Only one of ip_allow and ip_deny may be specified.

If ip_allow is specified, any traffic not matching a range to allow will be denied. If ip_deny is specified, any traffic not matching a range to deny will be allowed. A list of ranges to allow and a separate list to deny may not both be specified.

Both take a list of IP address ranges with a keyword specifying how to interpret the address, for example:

ip_allow:
- peer: 127.0.0.1
- remote: 99.99.0.0/16

The keyword peer specifies that the match should happen using the IP address of the other end of the network connection carrying the request: X-Forwarded-For and the PROXY protocol are both ignored. Here, our example specifies that connections originating from the Emissary pod itself should always be allowed.

The keyword remote specifies that the match should happen using the IP address of the HTTP client, taking into account X-Forwarded-For and the PROXY protocol if they are allowed (if they are not allowed, or not present, the peer address will be used instead). This permits matches to behave correctly when, for example, Emissary is behind a layer 7 load balancer. Here, our example specifies that HTTP clients from the IP address range 99.99.0.0 - 99.99.255.255 will be allowed.

You may specify as many ranges for each kind of keyword as desired.

Rejecting Client Requests With Escaped Slashes

• reject_requests_with_escaped_slashes: true will tell Emissary to reject requests containing escaped slashes. The default is false.

When set to true, Emissary will reject any client requests that contain escaped slashes (%2F, %2f, %5C, or %5c) in their URI path by returning HTTP 400. By default, Emissary will forward these requests unmodified.

In general, a request with an escaped slash will not match a Mapping prefix with an unescaped slash. However, external authentication services and other upstream services may handle escaped slashes differently, which can lead to security issues if paths with escaped slashes are allowed. By setting reject_requests_with_escaped_slashes: true, this class of security concern can be largely avoided.

Trust downstream client IP

• use_remote_address: false tells Emissary that it cannot trust the remote address of incoming connections, and must instead rely exclusively on the X-Forwarded-For header. The default is true.

When true (the default), Emissary will append its own IP address to the X-Forwarded-For header so that upstream services of Emissary can get the full set of IP addresses that have propagated a request. You may also need to set externalTrafficPolicy: Local on your LoadBalancer to propagate the original source IP address.

See the Envoy documentation on the X-Forwarded-For header and the Kubernetes documentation on preserving the client source IP for more details.

X-Forwarded-For trusted hops

• xff_num_trusted_hops sets the default value for the l7Depth setting of a Listener. The default is 0.

See the Listener documentation for more details.

Service health / timeouts

Incoming connection idle timeout

• listener_idle_timeout_ms sets the idle timeout for incoming connections. The default is no timeout, meaning that incoming connections may remain idle forever.

If set, this specifies the length of time (in milliseconds) that an incoming connection is allowed to be idle before being dropped. This can useful if you have proxies and/or firewalls in front of Emissary and need to control how Emissary initiates closing an idle TCP connection.

Please see the Envoy documentation on HTTP protocol options for more information.

Keepalive

• keepalive sets the global TCP keepalive settings.

Emissary will use these settings for all AmbasasdorMappings unless overridden in a Mapping’s configuration. Without keepalive, Emissary follows the operating system defaults.

For example, the following configuration:

keepalive:
  time: 2
  interval: 2
  probes: 100

would enable keepalives every two seconds (interval), starting after two seconds of idleness (time), with the connection being dropped if 100 keepalives are sent with no response (probes). For more information, see the Envoy keepalive documentation.

Upstream idle timeout

• cluster_idle_timeout_ms sets the default idle timeout for upstream connections (by default, one hour).

If set, this specifies the timeout (in milliseconds) after which an idle connection upstream is closed. The idle timeout can be completely disabled by setting cluster_idle_timeout_ms: 0, which risks idle upstream connections never getting closed.

If not set, the default idle timeout is one hour.

You can override this setting with idle_timeout_ms on a Mapping.

Upstream max lifetime

• cluster_max_connection_lifetime_ms sets the default maximum lifetime of an upstream connection.

If set, this specifies the maximum amount of time (in milliseconds) after which an upstream connection is drained and closed, regardless of whether it is idle or not. Connection recreation incurs additional overhead when processing requests. The overhead tends to be nominal for plaintext (HTTP) connections within the same cluster, but may be more significant for secure HTTPS connections or upstreams with high latency. For this reason, it is generally recommended to set this value to at least 10000 ms to minimize the amortized cost of connection recreation while providing a reasonable bound for connection lifetime.

If not set (or set to zero), then upstream connections may remain open for arbitrarily long.

You can override this setting with cluster_max_connection_lifetime_ms on a Mapping.

Request timeout

• cluster_request_timeout_ms sets the default end-to-end timeout for a single request.

If set, this specifies the default end-to-end timeout for every request.

If not set, the default is three seconds.

You can override this setting with timeout_ms on a Mapping.

Readiness and liveness probes

• readiness_probe sets whether /ambassador/v0/check_ready is automatically mapped
• liveness_probe sets whether /ambassador/v0/check_alive is automatically mapped

By default, Emissary creates Mappings that support readiness and liveness checks at /ambassador/v0/check_ready and /ambassador/v0/check_alive. To disable the readiness Mapping entirely, set readiness_probe.enabled to false:

readiness_probe:
  enabled: false

Likewise, to disable the liveness Mapping entirely, set liveness_probe.enabled to false:

liveness_probe:
  enabled: false

A disabled probe endpoint will respond with 404; however, the service is still running, and will be accessible on localhost port 8877 from inside the Emissary Pod.

You can change these to route requests to some other service. For example, to have the readiness probe map to the quote application’s health check:

readiness_probe:
  enabled: true
  service: quote
  rewrite: /backend/health

The liveness and readiness probes both support prefix and rewrite, with the same meanings as for Mappings.

Retry policy

This lets you add resilience to your services in case of request failures by performing automatic retries.

retry_policy:
  retry_on: "5xx"

Traffic management

Circuit breaking

• circuit_breakers sets the global circuit breaking configuration defaults

You can override the circuit breaker settings for individual Mappings. By default, Emissary does not configure any circuit breakers. For more information, see the circuit breaking reference.

Default label domain and labels

• default_labels sets default domains and labels to apply to every request.

For more on how to use the default labels, , see the Rate Limit reference.

Default load balancer

• load_balancer sets the default load balancing type and policy

For example, to set the default load balancer to least_request:

load_balancer:
  policy: least_request

If not set, the default is to use round-robin load balancing. For more information, see the load balancer reference.