Tracing · XB Field Notes

Baggage carries context across an explicit boundary

Mon, 27 Mar 2028 00:00:00 +0000

OpenTelemetry baggage is a context-associated collection of string key-value pairs. It lets information known near a request’s entry point travel to downstream operations: a deployment region or a rollout cohort, for example. Unlike a span attribute, baggage is intended for propagation. Unlike trace context, it describes application-selected information rather than the identifiers and flags connecting spans. These mechanisms can travel together without being the same data structure.

Propagation still needs a working boundary. An outgoing client injects context into a carrier such as HTTP headers; the receiver extracts that carrier into its local context. Instrumentation can perform these steps automatically when configured, while manually instrumented code must do them explicitly. Merely creating baggage in one process does not deliver it to an unrelated process, a background task, or an external service.

Choose what crosses and what gets recorded

Baggage entries do not automatically become span, log, or metric attributes. A downstream operation must read the value and copy it deliberately, or use a configured processor implementing that policy. This distinction matters when a team expects a searchable attribute but has only enabled propagation. It also prevents the opposite mistake: copying every incoming key into telemetry without considering its meaning, size, or cardinality.

Context operations return a new context rather than changing every existing reference. Keep the returned context and pass or attach it at the intended boundary. When removing entries for an outgoing request, preserve the trace context that should accompany that request. Treat an outbound allowlist and a telemetry-attribute allowlist as explicit decisions; the values appropriate for one receiver need not be suitable for every later destination.

When it breaks

Baggage has no built-in proof that a value was issued by a trusted component. A caller can supply a header claiming an expensive account tier or a privileged role. Parsing that header successfully establishes its representation, not its authority. Authentication and authorization must use their own trusted inputs. Small, plausible-looking values deserve the same trust analysis as obviously suspicious ones.

Unbounded baggage also increases request-header sizes and can spread sensitive data to unintended recipients, including third-party APIs reached by automatic instrumentation. Limit names and values, review the destinations that receive them, and use propagators for wire encoding. A successful round trip between two copies of one SDK is narrower evidence than interoperability across every language and proxy in a production path. Test those actual boundaries when they matter.

Baggage propagation and trust: mechanism and evidence boundaries.

Check it quickly

Create a virtual environment, install opentelemetry-sdk==1.35.0, save the script as baggage.py, and execute it with that environment’s Python. The author reported an earlier Python 3.14.4 run; retained exact records cover Python 3.12.14 with SDK 1.35.0. This source review did not rerun it. It uses the real W3C trace-context and baggage propagators with an in-memory span exporter. An explicit always-on sampler avoids inherited sampling settings. Keep assertions enabled, without -O or PYTHONOPTIMIZE: force_flush is called inside assert. It does not start an HTTP server: the dictionary is the carrier that an HTTP client would subsequently send.

import json

from opentelemetry import baggage, trace
from opentelemetry.context import Context
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ALWAYS_ON
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

base = Context()
raw = baggage.set_baggage("region", "eu-west", context=base)
raw = baggage.set_baggage("release", "canary/2", context=raw)
raw = baggage.set_baggage("internal_note", "synthetic-only", context=raw)
assert not baggage.get_all(base)
propagator = CompositePropagator([TraceContextTextMapPropagator(), W3CBaggagePropagator()])
provider = TracerProvider(sampler=ALWAYS_ON)
exporter = InMemorySpanExporter()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("baggage-lab")
with tracer.start_as_current_span("parent", context=Context()) as parent:
 outgoing = trace.set_span_in_context(parent, baggage.clear(context=raw))
 for key in ("region", "release"):
 outgoing = baggage.set_baggage(key, baggage.get_baggage(key, raw), context=outgoing)
 carrier = {}
 propagator.inject(carrier, context=outgoing)
 received = propagator.extract(carrier, context=Context())
 assert set(baggage.get_all(received)) == {"region", "release"}
 assert baggage.get_baggage("region", received) == "eu-west"
 assert baggage.get_baggage("release", received) == "canary/2"
 assert baggage.get_baggage("internal_note", raw) == "synthetic-only"
 assert trace.get_current_span(received).get_span_context().trace_id == parent.get_span_context().trace_id
 with tracer.start_as_current_span("inherited", context=received):
 pass
 with tracer.start_as_current_span("copied", context=received) as span:
 span.set_attribute("deployment.region", baggage.get_baggage("region", received))
assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert not spans["inherited"].attributes
assert spans["copied"].attributes == {"deployment.region": "eu-west"}
independent = propagator.extract({"traceparent": "invalid", "baggage": "region=eu-west"},
 context=Context())
assert not trace.get_current_span(independent).get_span_context().is_valid
assert baggage.get_baggage("region", independent) == "eu-west"
untrusted = propagator.extract({"baggage": "tier=gold"}, context=Context())
assert baggage.get_baggage("tier", untrusted) == "gold"
print(json.dumps({"wire_fields": sorted(carrier),
 "wire_baggage": carrier["baggage"],
 "automatic_span_attributes": dict(spans["inherited"].attributes),
 "explicit_span_attributes": dict(spans["copied"].attributes),
 "baggage_survives_invalid_traceparent": True,
 "caller_supplied_tier_was_parsed": True}, indent=2))
provider.shutdown()

The original context contains region, release, and a synthetic internal note. The outgoing context keeps only the first two entries while retaining the parent span. The extracted region is eu-west, and the release value canary/2 round-trips through the encoded header. Assertions also confirm that constructing the filtered context did not remove the note from the original context or change the empty base context.

Two child spans demonstrate the recording boundary. inherited receives baggage but has no attributes; copied contains only the explicitly assigned deployment.region. A malformed traceparent is tested separately: its trace context is invalid, while the independently encoded baggage still extracts successfully. Finally, a caller-supplied tier=gold is accepted as data without any identity check. These results cover serialization, filtering, and recording in this SDK fixture; the trace assertion compares trace_id, not every trace-context field. Before relying on the same policy across services, verify the real client instrumentation, receiver extraction, outbound filtering, and the attribute mapping used by the destination backend.

Sources

Python assert statement, optimization omits assertions and any calls inside them, checked 2026-10-18.
OpenTelemetry baggage, baggage is separate from telemetry attributes and lacks built-in integrity checks, checked 2026-10-15.
W3C Baggage, HTTP representation, value encoding, and propagation limits, checked 2026-10-15.
W3C Trace Context, trace identity travels separately from application baggage, checked 2026-10-15.
OpenTelemetry Python sampling, explicit sampler selection and environment defaults, checked 2026-10-15.

BPF tracepoints peek inside the running kernel

Mon, 06 Mar 2028 00:00:00 +0000

BPF programs can attach to selected events in a running kernel without rebuilding the application or restarting its workload. A userspace loader still loads and manages the probe. Collection changes resource use, so “live” does not mean invisible or free. Other profilers can also observe running programs; choosing BPF depends on the question and available instrumentation.

Define the event, aggregation key, observation interval, and excluded work before interpreting output. A syscall count, a scheduling delay, and a device latency distribution answer different questions. An entry count alone cannot reveal successful operations, time spent waiting, or which storage path served a request.

Probe without pausing

An eBPF program can accumulate counts or histograms in kernel maps while the workload continues. Aggregation reduces userspace output, but every observed event may still execute probe work. The verifier checks program safety under kernel rules; acceptance is not proof of correct measurement, negligible overhead, or an impossible kernel failure.

Start with a bounded question. Counting syscall entries needs an entry event; measuring syscall duration normally requires matching entry and exit state. Block-device latency needs its own event lifecycle. High-level tools such as bpftrace help express these measurements, but a short program does not establish that its units, filters, and correlations are correct.

Tracepoints are named events

Tracepoints expose named kernel events and are often less coupled to implementation details than probes on internal functions. They are not a stable ABI: names and fields can change across kernels. Kprobes can attach by symbol, rather than only by a hardcoded address, but internal function contracts can change too. Pin maintained probes to tested versions.

Inspect the available event and its field format on the intended kernel. A listing establishes visibility, not permission to load and attach a program. If an event is unavailable, record that limitation rather than silently substituting a different event whose count or timing means something else.

Read a histogram, not a firehose

Use aggregation to reveal distributions without printing every event. A latency histogram can suggest fast and slow paths, but its shape is evidence for investigation, not a diagnosis by itself. It cannot establish which path caused an outlier unless the probe measures the relevant distinction. Counts are not latency histograms.

Keep keys deliberate. Grouping by thread name can merge unrelated tasks with the same name; grouping by identifiers or arguments changes cardinality and cost. Apply relevant filters early, bound collection, and preserve the exact probe and workload context when reporting a finding. Compare against independent observations before attributing an application problem to one component.

When it breaks

Loading can fail because of permissions, kernel configuration, unavailable events, or tool-specific BTF or header requirements. Preserve the actual error. Listing tracepoints is not a successful attachment, and absent counts can mean an inactive workload, a mismatched filter, or failed collection. Container access also depends on the host kernel and its security policy; do not weaken protections to force a demonstration.

Hot-path probes can distort the workload even when they aggregate in-kernel. Compare cost under representative conditions and narrow collection when needed. Changing a filter can change the population itself, so a different distribution is not automatically proof that the cheaper observation is more accurate.

Entry attempts merge by current thread name. Illustrative counts are not measured output; the proposed scoped fixture remains unexecuted.

Check it quickly

This proposed foreground fixture targets bpftrace 0.24 and an already authorized host; it was not run here. Confirm that version, the exact tracepoint, and /bin/cat before continuing. Keep the evidence directory, stderr and status; do not add privileges to make it pass.

(
 work=$(mktemp -d) || exit 1
 printf 'Evidence directory: %s\n' "$work"
 cd "$work" || exit 1
 printf 'controlled input\n' > input || exit 1
 bpftrace --version || exit $?
 bpftrace -lv 'tracepoint:syscalls:sys_enter_openat' || exit $?
 if bpftrace -o counts.log -c '/bin/cat input' -e '
 tracepoint:syscalls:sys_enter_openat /pid == cpid/ {
 @opens[comm] = count();
 }' > workload.out 2> collector.err; then
 collector_status=0
 else
 collector_status=$?
 fi
 printf 'collector_status=%s\n' "$collector_status"
 cat collector.err
 if [ "$collector_status" -ne 0 ]; then exit "$collector_status"; fi
 cmp input workload.out || exit 1
 cat counts.log
)

In the reviewed 0.24 implementation, -c starts this finite child after probe attachment; pid == cpid restricts the event population. The observation follows the child lifetime, not a ten-second sleep. The child inherits the collector’s privileges. The shell records the collector’s exact foreground status, not the workload’s exit code. cmp checks copied bytes separately; it does not establish that exit code.

Expect entry counts for cat, including startup opens and failed attempts, rather than exactly one successful open. Equal thread names still merge within the selected population. Empty maps or errors are inconclusive. Attachment, shutdown and workload-exit behavior still need a permitted runtime review; source inspection is not execution evidence.

Sources

bpftrace project, project overview and tracing examples, checked 2026-10-18.
eBPF introduction, Official eBPF documentation, checked 2026-06-01.
bpftrace 0.24 language, tracepoint fields and count versus histogram aggregation, checked 2026-10-18.
bpftrace 0.24 standard library, comm identifies the current thread name, checked 2026-10-18.
Linux BPF design Q&A, tracepoint and kprobe attachment is not a stable ABI, checked 2026-10-18.
Linux BPF verifier, verification checks rather than measurement or overhead guarantees, checked 2026-10-18.
GNU Bash manual, wait without an explicit child ID does not return that child’s failure, checked 2026-10-18.
bpftrace 0.24 CLI, child lifetime, privileges, cpid filtering and separate tracing output, checked 2026-10-19.
bpftrace v0.24.0 runtime source, attachment precedes starting the non-USDT child in BPFtrace::run, checked 2026-10-19.

Trace sampling reduces volume without fixing a cost ceiling

Sat, 23 Oct 2027 00:00:00 +0000

Trace sampling selects which recorded work reaches the tracing pipeline. It can reduce export traffic and storage, but a configured percentage is not a strict ceiling on bytes, processor time, or money. Requests produce different numbers of spans, attributes vary in size, and traffic itself changes. A one-percent policy applied to a sudden tenfold traffic increase can still produce roughly ten times as much sampled data.

Head sampling makes a decision near the start of a trace. A trace-identifier ratio sampler is useful when identifiers have the randomness its algorithm requires. Its probability describes selection across suitable traffic; it does not promise exactly half of every sixteen requests, or equal coverage of every route. Small populations and rare failures can be poorly represented even when the overall percentage looks reasonable.

Counts reproduce the article’s deliberately arranged identifier fixture, not a production percentage guarantee. Remote-parent counts are a separate phase. The historical revised/passed status is retained without rerunning examples or adding runtime credit.

Keep a decision consistent across boundaries

Downstream instrumentation should follow the accepted parent decision when that is the intended policy. In the Python SDK, ParentBased delegates root decisions to its configured root sampler and uses separate policies for sampled and unsampled parents. Wrapping TraceIdRatioBased(0.5) therefore differs from treating every downstream span as an independent new coin flip. Context propagation and the receiving sampler configuration must agree. At an untrusted boundary, decide whether to accept the caller’s sampling flag: W3C treats it as guidance, not authority to consume unlimited tracing resources.

An unsampled trace still involves application work, context handling, and potentially instrumentation overhead. Dropping export does not make the business request free. Conversely, preserving a parent decision does not guarantee that every child will be visible: a library might be uninstrumented, a service might lose context, or an exporter might fail. Sampling consistency addresses one source of missing data, not the whole delivery path.

When it breaks

A request can look ordinary at its start and fail much later. A head sampler that already dropped it cannot recover the missing spans merely because application code subsequently sets an error status. Tail sampling can inspect more of the completed trace before deciding, but requires buffering, routing related spans together, and managing delayed or missing data. It also cannot reconstruct spans discarded upstream before they reached it.

Another failure is measuring request reliability directly from a biased trace collection. Policies that deliberately retain slow requests or errors change the population being observed. Keep request outcome metrics with a clear denominator, and describe the trace selection policy alongside investigations. For cost control, observe actual exported spans and bytes, queue behavior, collector resources, and retention; a sampling percentage alone is not evidence that a spending limit holds.

Check it quickly

In a Python virtual environment, install opentelemetry-sdk==1.35.0. Save this as trace-sampling.py and run it using that environment; the original article reports a run with Python 3.12.14 and that pinned SDK. That run’s exact execution artifact is not present in the retained audit, and the SDK is not installed in this review environment, so the full example remains unrun here. The diagram’s historical pass label repeats the original report; it is not independently recovered execution proof. The custom identifier generator is exclusively a deterministic test fixture. Its sixteen identifiers deliberately straddle this SDK version’s ratio threshold. Do not copy that sequential identifier scheme into production instrumentation, where the sampler relies on appropriate randomness.

import json
from collections import Counter

from opentelemetry.context import Context
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.id_generator import IdGenerator
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
from opentelemetry.trace import Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator


class TestIds(IdGenerator):
 # Controlled test inputs only; production IDs need suitable randomness.
 def __init__(self):
 self.trace_index = 0
 self.span_index = 0

 def generate_trace_id(self):
 index = self.trace_index
 self.trace_index += 1
 return ((index + 1) << 64) | (index * (1 << 60) + 1)

 def generate_span_id(self):
 self.span_index += 1
 return self.span_index


exporter = InMemorySpanExporter()
provider = TracerProvider(sampler=ParentBased(TraceIdRatioBased(0.5)),
 id_generator=TestIds())
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("sampling-lab")
executed_children = 0
late_error_recording = None
for index in range(16):
 with tracer.start_as_current_span(f"root-{index}", context=Context()) as root:
 for child in range(2):
 with tracer.start_as_current_span(f"child-{index}-{child}"):
 executed_children += 1
 if index == 15:
 late_error_recording = root.is_recording()
 root.set_status(Status(StatusCode.ERROR, "error discovered after start"))
assert provider.force_flush()
spans = exporter.get_finished_spans()
counts = Counter(span.context.trace_id for span in spans)
assert len(counts) == 8 and set(counts.values()) == {3} and len(spans) == 24
assert {s.name for s in spans if s.parent is None} == {f"root-{i}" for i in range(8)}
assert executed_children == 32 and late_error_recording is False
assert not any(span.name == "root-15" for span in spans)

exporter.clear()
propagator = TraceContextTextMapPropagator()
for flag, label in (("01", "sampled"), ("00", "unsampled")):
 # High trace-ID bits do not override an accepted remote parent's flag.
 header = f"00-1111111111111111ffffffffffffffff-2222222222222222-{flag}"
 context = propagator.extract({"traceparent": header}, context=Context())
 with tracer.start_as_current_span(label, context=context):
 with tracer.start_as_current_span(label + "-child"):
 pass
assert provider.force_flush()
remote_spans = exporter.get_finished_spans()
assert {span.name for span in remote_spans} == {"sampled", "sampled-child"}
assert next(span for span in remote_spans if span.name == "sampled").parent.is_remote
print(json.dumps({"controlled_roots": 16, "exported_traces": len(counts),
 "exported_spans": len(spans), "executed_child_operations": executed_children,
 "late_error_was_recording": late_error_recording,
 "remote_parent_exported_spans": len(remote_spans)}, indent=2))
provider.shutdown()

The sixteen root operations each start two children. Exactly eight traces, containing twenty-four spans, are exported because of the chosen test inputs; all thirty-two child operations still execute. Root fifteen is deliberately outside the retained set. Its later error status does not cause it to become recording or appear in the exporter. These assertions test behavior at known inputs rather than estimating a sampling percentage statistically.

The second phase extracts two remote parent headers with the same high trace identifier and different sampled flags. Only the sampled parent’s child tree is exported, producing two spans despite that identifier being above the root ratio threshold. The unsampled parent produces none. No collector or tail sampler runs in this experiment. When applying the policy to a real service, test accepted remote context separately from new roots, then validate the resulting volume and diagnostic coverage against the workload that actually matters.

Sources

OpenTelemetry sampling, head and tail decisions and the risk of losing rare failures, checked 2026-10-15.
OpenTelemetry Python sampling reference, ParentBased and TraceIdRatioBased sampler behavior, checked 2026-10-15.
W3C Trace Context, propagated sampling flags and remote parent context, checked 2026-10-15.

Traces connect the recorded parts of a request

Tue, 15 Dec 2026 00:00:00 +0000

A distributed trace connects recorded operations through a shared trace identifier and explicit relationships. A span represents one operation, with start and end timestamps, attributes, events, and status. This makes a trace useful when a checkout request crosses an application, an inventory service, and a payment service. The resulting picture depends on which operations were instrumented, retained, and exported; it is not an automatic inventory of every action the request performed.

The trace identifier groups related spans. A span identifier distinguishes one operation within that trace, while the parent identifier connects a child to its immediate predecessor. Across HTTP, W3C traceparent carries the trace identifier, the sending span identifier, and trace flags. The receiving instrumentation extracts that context and creates a new span. Copying a request identifier into unrelated log messages does not create these span relationships. For work with several causal inputs, span links can express additional relationships without assigning multiple parents to one span.

Trace A and B are editorial names for the two source-described trace groups, not actual identifiers. The source example is represented as inert code and narrative: historical review remains read and execution blocked, with no rerun or new runtime credit. Arrow length is not duration.

Read relationships before adding durations

A server span can contain database spans and several overlapping outbound requests. Adding all those durations counts overlapping time repeatedly. Even subtracting the sum of child durations from the parent can produce nonsense when siblings run concurrently. Read the timeline and identify intervals on the critical path. Cross-machine clock differences and missing instrumentation also limit how precisely apparent gaps can be interpreted.

Span status needs its own interpretation. An HTTP response code, a recorded exception event, and an OpenTelemetry error status are separate pieces of data. Instrumentation may connect them according to its conventions; manually recording an exception does not justify assuming that every relevant status field changed. Service resource attributes identify the emitter, whereas an individual span describes an operation performed there.

When it breaks

A proxy can drop propagation headers, or a service can fail to extract them. The downstream work then appears as a separate trace even though the business request reached it successfully. A shared trace identifier with the wrong parent is another failure: related work is grouped together, but its causal position is wrong. Test both identifiers at a boundary instead of checking only that some spans arrived.

An absent span has several possible causes, including sampling, missing instrumentation, asynchronous context loss, and export failure. It does not prove that the operation never ran. Likewise, a green trace view does not establish that every downstream dependency succeeded. Correlate it with request outcome metrics and relevant logs, and check collector or exporter health before interpreting an incomplete picture as complete evidence.

Check it quickly

Create a Python virtual environment and install opentelemetry-sdk==1.35.0, which installs the matching API dependency. Save this as trace-flow.py and run it with that environment’s Python. The original article reports a Python 3.14.4 run. This revision retains the exercise without rerunning it; the current execution check remains blocked. The script starts a loopback HTTP server, uses separate caller and worker tracer providers, and exports to memory. It closes the server after the requests; no collector is required. Before an authorized run, ensure proxy settings bypass 127.0.0.1: urllib can otherwise consult a configured proxy. Creating this local listener and making HTTP requests require a suitable disposable environment.

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
from urllib.error import HTTPError
from urllib.request import Request, urlopen

from opentelemetry.context import Context
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import SpanKind, Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

exporter = InMemorySpanExporter()
providers = [TracerProvider(resource=Resource({"service.name": name}))
 for name in ("caller", "worker")]
for provider in providers:
 provider.add_span_processor(SimpleSpanProcessor(exporter))
caller, worker = [p.get_tracer("trace-lab") for p in providers]
propagator = TraceContextTextMapPropagator()


class Handler(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def do_GET(self):
 carrier = {key.lower(): value for key, value in self.headers.items()}
 parent = propagator.extract(carrier, context=Context())
 with worker.start_as_current_span("handle" + self.path,
 context=parent, kind=SpanKind.SERVER) as span:
 status = 200
 if self.path == "/fail":
 try:
 raise ValueError("synthetic inventory failure")
 except ValueError as error:
 span.record_exception(error)
 span.set_status(Status(StatusCode.ERROR, str(error)))
 status = 503
 body = b"lab response\n"
 self.send_response(status)
 self.send_header("Content-Length", str(len(body)))
 self.end_headers()
 self.wfile.write(body)


server = HTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
base = f"http://127.0.0.1:{server.server_port}"
try:
 with caller.start_as_current_span("request", context=Context()):
 with caller.start_as_current_span("outbound", kind=SpanKind.CLIENT):
 headers = {}
 propagator.inject(headers)
 with urlopen(Request(base + "/ok", headers=headers), timeout=5) as response:
 assert response.status == 200
 response.read()
 try:
 with urlopen(base + "/fail", timeout=5) as response:
 raise AssertionError("Expected HTTP 503")
 except HTTPError as response:
 with response:
 assert response.code == 503
 response.read()
finally:
 server.shutdown()
 thread.join(timeout=5)
 server.server_close()

for provider in providers:
 assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert len(spans) == 4
root, client, received, orphan = [spans[name]
 for name in ("request", "outbound", "handle/ok", "handle/fail")]
assert root.parent is None
assert root.context.trace_id == client.context.trace_id == received.context.trace_id
assert client.parent.span_id == root.context.span_id
assert received.parent.span_id == client.context.span_id and received.parent.is_remote
assert orphan.parent is None and orphan.context.trace_id != root.context.trace_id
assert orphan.status.status_code == StatusCode.ERROR
assert [event.name for event in orphan.events] == ["exception"]
assert received.resource.attributes["service.name"] == "worker"
assert client.resource.attributes["service.name"] == "caller"
print(json.dumps({"exported_spans": len(spans), "linked_spans": 3,
 "unpropagated_request_started_new_trace": True,
 "failed_span_status": orphan.status.status_code.name,
 "exception_events": len(orphan.events)}, indent=2))
for provider in providers:
 provider.shutdown()

The first request creates three linked spans: request, outbound, and handle/ok. The assertions verify one trace identifier, two correct parent links, a remote parent on the server span, and different service names on the two providers. The second request deliberately sends no propagation header. It returns HTTP 503, creates a separate root span, and explicitly records both an exception event and an ERROR status. Four spans are exported altogether.

These are two providers in one Python process communicating over real local HTTP. A successful full run would verify propagation and recording behavior, not deployment-wide clock synchronization, collector reliability, or concurrent critical-path analysis; those expected assertions have not been rerun here. In a deployed service, repeat the boundary checks through the actual ingress and client libraries, then confirm export health and sampling configuration. A useful trace is one whose recorded relationships and limitations are understood before its timeline is used to explain an incident.

Sources

OpenTelemetry traces, span structure, relationships, and trace identity, checked 2026-09-13.
W3C Trace Context, traceparent fields and propagation across process boundaries, checked 2026-09-13.
OpenTelemetry Python instrumentation, manual spans, processors, and explicit exception recording, checked 2026-09-13.