DDTrace.Config (dd_trace_ex v0.1.0)

Copy Markdown View Source

The tracer's settings, resolved once when the application boots.

Each setting is read from two layers, in order:

  1. application config under the :dd_trace_ex key
  2. the corresponding DD_* environment variable

Application config wins over the environment, matching the other Datadog tracers, and there is no third layer. A value that is missing, blank or unusable falls through to the next layer, and finally to the default — and so does a whole layer that is not the shape it should be. Bad configuration degrades the tracer, it never stops the host application from booting.

SettingApplication configEnvironmentDefault
tracing on/off:enabledDD_TRACE_ENABLEDtrue
service name:serviceDD_SERVICEnone
deployment environment:envDD_ENVnone
application version:versionDD_VERSIONnone
agent address:agent_urlDD_TRACE_AGENT_URL, or DD_AGENT_HOST and DD_TRACE_AGENT_PORThttp://localhost:8126
agent request and shutdown flush budget (ms):shutdown_timeoutnone5000
trace ids in log metadata:logs_injectionDD_LOGS_INJECTIONtrue
x-datadog-tags header budget (bytes):x_datadog_tags_max_lengthDD_TRACE_X_DATADOG_TAGS_MAX_LENGTH512
HTTP statuses that are server errors:http_server_error_statusesDD_TRACE_HTTP_SERVER_ERROR_STATUSES"500-599"
HTTP statuses that are client errors:http_client_error_statusesDD_TRACE_HTTP_CLIENT_ERROR_STATUSES"400-499"
global sampling rate:sample_rateDD_TRACE_SAMPLE_RATEunset: the agent decides
sampling rules:sampling_rulesDD_TRACE_SAMPLING_RULES[]
rule-kept traces per second:rate_limitDD_TRACE_RATE_LIMIT100

DD_TRACE_ENABLED and DD_LOGS_INJECTION accept true/false, 1/0, yes/no and on/off, in any case. Anything else keeps the default.

:agent_url must be an http:// URL — https and unix sockets are not implemented, and either one is ignored with a warning. An agent named by URL is taken at its word, so one written without a port gets http's default of 80, not 8126; DD_AGENT_HOST and DD_TRACE_AGENT_PORT are the path that defaults the port to 8126. An IPv6 address works either way, bracketed in a URL as http://[::1]:8126 and bare in DD_AGENT_HOST as ::1.

:http_server_error_statuses is which response statuses mark a server span an error — comma-separated single codes and inclusive lo-hi ranges, as in "500-599,429". One malformed part falls back to the whole default with a warning, rather than applying half a policy. DDTrace.Integrations.Phoenix is what reads it.

:http_client_error_statuses is the same for the spans of outbound requests, in the same grammar, and DDTrace.Integrations.Req is what reads it. Its default is the inverse of the server's on purpose: a 4xx says the request this application made was wrong, while a 5xx is the downstream service's own failure, reported on that service's own span.

:sample_rate is the share of traces to keep, 0..1, applied to every trace no rule matched; :sampling_rules is a list of rules, each a map or keyword list with the fields the Datadog docs spell — service, name, resource, tags, sample_rate — or the JSON array of them as a string, the shape the environment variable takes. Rules match a trace's root span with case-insensitive globs (* any run of characters, ? exactly one); the first match decides. A rate outside 0..1 is ignored with a warning; a rule that cannot be read is skipped with a warning and the rest are kept; a rule with no sample_rate keeps everything it matches; max_per_second on a rule is ignored with a warning, because the rate limit is global. :rate_limit caps how many traces the rules may keep per second, across every rule and the global rate together; 0 keeps none of them. A fraction is truncated to a whole number with a warning; a negative value or one that is not a number is ignored with a warning and 100 applies. DDTrace says how the decision is made and what it puts on the wire.

:shutdown_timeout is one budget doing two jobs, which is deliberate: it is how long any single submission may wait for the agent to answer, and how long the tracer's whole shutdown may take. The second is what makes the first true — a request already in flight when the VM starts stopping has to be bounded by something, or it outlives the budget and the payload it carried is lost. A submission that runs out of budget is dropped, not retried.

What the budget bounds is shutdown, not the final flush on its own. A submission already in flight when shutdown begins is waited out first, and the final flush gets what is left. So traces still buffered when an unreachable agent has spent the budget are lost. A final flush that finds the agent already failing does not start another request it cannot finish; it drops what it holds instead. Every one of those losses is counted in [:dd_trace, :exporter, :drop], the skip under its own reason, :agent_unreachable_at_shutdown. Raising the budget buys the last flush more room at the cost of a slower stop.

Correlating logs with traces

While a span is open, the process's Logger metadata carries the keys Datadog's UI correlates on, so a log line links to the trace it happened in and back:

[
  "dd.trace_id": "68d1f4a300000000453f7c31b09a2e04",
  "dd.span_id": "5013951231073820500",
  "dd.service": "storefront",
  "dd.env": "prod",
  "dd.version": "1.4.0"
]

Any JSON logger that emits metadata gets this for free. The keys are quoted atoms, because dd.trace_id is the spelling the UI reads — so a metadata: allowlist has to spell them the same way:

config :logger, :default_formatter,
  metadata: [:request_id, :"dd.trace_id", :"dd.span_id", :"dd.service"]

metadata: :all needs no list. Set :logs_injection to false — or DD_LOGS_INJECTION=false — and the tracer never touches Logger metadata at all.

Every span the agent receives has to name a service, so a span that names none of its own and a tracer configured with none either ship as unnamed-elixir-service — visible in the Datadog UI, and there to be recognized as a tracer nobody configured rather than mistaken for an application.

The x-datadog-tags budget

How many bytes of _dd.p.* trace tags a request may carry onward. It is clamped to 0..512: 512 is what the agent itself accepts, so honoring a larger number would ship a header the agent rejects, and a value outside the range is clamped with a warning rather than obeyed. 0 turns the header off in both directions, silently — an operator who set it to zero is not reporting a failure by doing so.

An explicit agent URL is authoritative: only http is supported for now, and a URL with no port keeps the scheme's default rather than 8126, which applies to the host-and-port form only. Anything else is ignored with a warning, and the host-and-port form takes over.

The resolved config is stored in :persistent_term: written once at boot and read on every DDTrace.info/0 call, which is exactly the read-mostly shape :persistent_term is for.

Summary

Types

One sampling rule, compiled at boot from what was configured.

t()

Functions

Returns the config resolved when the application booted.

Resolves the settings from application config and environment variables.

Returns the config as a plain map, the shape DDTrace.info/0 reports.

Types

sampling_rule()

@opaque sampling_rule()

One sampling rule, compiled at boot from what was configured.

Its insides belong to the sampler; to_map/1 renders each rule back as the fields it constrains and its rate, which is what DDTrace.info/0 reports and the only shape worth reading.

t()

@type t() :: %DDTrace.Config{
  agent_url: String.t(),
  enabled?: boolean(),
  env: String.t() | nil,
  http_client_error_statuses: MapSet.t(integer()),
  http_server_error_statuses: MapSet.t(integer()),
  logs_injection?: boolean(),
  rate_limit: non_neg_integer(),
  sample_rate: float() | nil,
  sampling_rules: [sampling_rule()],
  service: String.t() | nil,
  shutdown_timeout: pos_integer(),
  version: String.t() | nil,
  x_datadog_tags_max_length: non_neg_integer()
}

Functions

fetch()

@spec fetch() :: t()

Returns the config resolved when the application booted.

Reports tracing as disabled when the tracer application is not running, so callers never have to check first: nothing was resolved, so there is nowhere for a span to go.

resolve(app_config, system_env)

@spec resolve(
  keyword(),
  %{optional(String.t()) => String.t()}
) :: t()

Resolves the settings from application config and environment variables.

Both layers are passed in, so callers other than the application's boot — tests, most of all — need neither a live application environment nor a mutated system environment.

Examples

iex> config = DDTrace.Config.resolve([service: "storefront"], %{"DD_ENV" => "prod"})
iex> {config.service, config.env, config.agent_url}
{"storefront", "prod", "http://localhost:8126"}

to_map(config)

@spec to_map(t()) :: map()

Returns the config as a plain map, the shape DDTrace.info/0 reports.

Examples

iex> [service: "storefront"] |> DDTrace.Config.resolve(%{}) |> DDTrace.Config.to_map()
%{
  enabled: true,
  logs_injection: true,
  service: "storefront",
  env: nil,
  version: nil,
  agent_url: "http://localhost:8126",
  shutdown_timeout: 5000,
  x_datadog_tags_max_length: 512,
  http_server_error_statuses: "500-599",
  http_client_error_statuses: "400-499",
  sample_rate: nil,
  sampling_rules: [],
  rate_limit: 100
}

A rule is reported as it was resolved: the fields it constrains, its rate, and nothing it was given that has no effect.

iex> [sampling_rules: [%{"resource" => "GET /health", "max_per_second" => 5}]]
...> |> DDTrace.Config.resolve(%{})
...> |> DDTrace.Config.to_map()
...> |> Map.get(:sampling_rules)
[%{resource: "GET /health", sample_rate: 1.0}]