Head sampling decides whether to keep a trace when the first span begins. It is inexpensive, but it may discard a trace before the system knows that a later span will fail. Tail sampling waits until enough of the trace is available, then makes a decision using the complete trace or a configured window. This is useful when error and latency traces matter more than random samples.
Choose policies around questions
Typical policies retain traces containing errors, traces above a latency threshold, and a baseline percentage of ordinary traffic. Avoid keeping every trace from a noisy service without a volume estimate. A policy should answer an operational question, such as why checkout requests are slow, rather than simply maximizing retained data.
Route all spans for a trace together
Tail sampling requires a collector to see the spans that belong to the same trace. In a multi-gateway deployment, use trace-ID-aware routing so spans do not get split across independent samplers. If the trace is incomplete when the decision is made, tune decision wait and late-span behavior deliberately.
Size collectors for the decision window
Tail sampling holds trace data in memory. Capacity depends on incoming spans per second, average spans per trace, decision wait, and the number of active traces. Set memory limits and configure the memory limiter early in the pipeline. Watch dropped spans, refused telemetry, queue depth and collector restarts.
Example policy concept
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
- name: slow
type: latency
latency:
threshold_ms: 1500
Validate the exact configuration against the Collector version and distribution you deploy. Policy syntax and supported processors can evolve.
Protect telemetry data
Trace attributes can contain URLs, user identifiers or accidental secrets. Apply attribute filtering and redaction before export, and keep access to trace backends restricted. Sampling is not a privacy control: sensitive data should be removed whether a trace is retained or not.
Operate the pipeline itself
- Monitor collector memory, CPU, queue size and exporter failures.
- Test behavior during backend outages.
- Use trace-aware load balancing for tail sampling.
- Keep a baseline sample for normal traffic.
- Review retention and cost by service and environment.
Tail sampling improves signal quality only when the collector pipeline is reliable. See the OpenTelemetry FastAPI tracing tutorial and Prometheus and Grafana monitoring setup.
Engineering considerations
Tail sampling requires trace affinity. In a horizontally scaled collector tier, spans from one trace must reach the same decision-making instance. Use trace-ID-aware routing before tail-sampling processors and verify behavior during collector restarts and scaling events. Randomly distributing spans can leave a collector with only part of a trace and cause a misleading decision. Test with traces that cross several services and include both successful and failed spans.
Implementation details
Estimate memory from incoming spans per second, average spans per trace, decision wait and trace size. Measure these values under representative traffic, including bursts and slow exporters. Set memory limits with headroom and configure a memory limiter early in the pipeline. Monitor refused spans and queue growth. If collectors repeatedly reach their limits, review decision windows, candidate volume and policy complexity before simply increasing container memory.
Security and reliability controls
Define policy order with intent. Retain errors and important high-latency traces, then add a baseline probability sample for ordinary traffic. A broad early policy can retain too much data and defeat the cost objective; an overly narrow policy can discard rare but valuable traces. Test representative examples against the actual collector configuration and document why each policy exists. Version policy changes with the same care as application configuration.
Testing strategy
Span attributes can contain email addresses, query parameters, customer identifiers or accidentally attached tokens. Apply redaction and attribute filtering before data reaches remote backends. Sampling is not a privacy control: a trace may be retained precisely because it contains an error. Restrict access to collector configuration and trace storage, define retention periods and review which teams may inspect production traces.
Operational monitoring
Distributed applications can emit spans late because of network delays, batch exporters or process shutdown. The decision wait balances trace completeness against memory use and visibility delay. Test traces with missing root spans, late child spans and services that flush during termination. Monitor incomplete traces. For long-running workflows, consider whether one trace spans too long for the configured decision window and whether span links or separate traces are more appropriate.
Failure diagnosis
Compare retained volume, error coverage, high-latency coverage and backend storage cost before and after rollout. Keep a random baseline so ordinary behavior remains visible. Check whether retained traces contain enough context to explain incidents, not merely the failing span. If one noisy service dominates, investigate its instrumentation or add service-aware policy limits rather than accepting an unbounded increase in storage.
Rollout and maintenance
Collectors are production infrastructure. Run multiple replicas, configure health checks, size exporter queues for backend outages and avoid a single overloaded gateway. Test rolling upgrades and autoscaling while preserving trace affinity. Document what telemetry may be dropped during an outage and alert when loss crosses an agreed threshold. Sampling should reduce cost without silently destroying evidence needed for incident response.
Operational handover
Before handing this capability to an operations team, document the configuration owner, supported versions, dependency map and escalation path. Include the commands or dashboard views needed to verify normal operation, along with the expected values and known limitations. Store configuration in version control, review changes through the normal deployment process and keep a tested rollback procedure. Schedule periodic reviews because upstream components, security requirements and workload patterns change. A short runbook should explain how to identify a failed component, what evidence to collect, which actions are safe to retry and when to escalate. Test that runbook during a maintenance exercise rather than waiting for an outage. Finally, measure the result against a baseline: reliability improvements should be visible in error rates, latency, resource consumption, recovery time or reduced manual work. This makes the implementation maintainable after the original developer has moved to another project.
Example rollout sequence
Begin with a non-production service that generates a mix of successful and failed traces. Confirm that the load-balancing tier routes spans for each trace to the same sampler and that the collector reports expected policy decisions. Increase traffic gradually while watching memory, decision latency, dropped spans and exporter queue depth. Compare the retained set with a known trace sample to make sure error and slow-request policies work as intended. Then enable the policy for one production service, keep a baseline sample and review the resulting data with the service owner. Expand to additional services only after storage cost, incident usefulness and collector stability are understood. Keep a configuration rollback ready so sampling can be simplified quickly if the collector becomes overloaded.
