`Traceparent` header may get duplicated with `context_propagation: headers`

[paweljw]

Hello! I'm running into the following:

• an instance of nginx (OBI-instrumented, through Beyla) receives a request with a Traceparent header already set
• by the time the request leaves nginx, it has two different Traceparent headers

Headers obtained by tcpdumping in the container running nginx. Only including relevant sections.

Case 1: external tracing instrumentation injects traceparent, OBI injects Traceparent.

Traceparent: 00-51c4b295117d40a6a3f531e715ce23d7-a24f141cea4f7d1d-01
X-Forwarded-Proto: http
traceparent: 00-51c4b295117d40a6a3f531e715ce23d7-a8e0bed6092f2234-01

I initially thought it may be due to case difference, so I reconfigured the upstream instrumentation to use Traceparent, matching case. Unfortunately didn't work, leading to:

Case 2: both inject Traceparent, duplicating the header:

Incoming into nginx:

Traceparent: 00-f6ed85f8071a40eca61f756f991ceb30-b7410955bd7a4a59-01

Outgoing from nginx:

Traceparent: 00-f6ed85f8071a40eca61f756f991ceb30-a41beb8a36e8dbe3-01
Traceparent: 00-f6ed85f8071a40eca61f756f991ceb30-b7410955bd7a4a59-01

I was able to confirm this as the source of some down-stream issues with consuming this data by adding a custom de-duplicating context propagator. Otherwise the resulting traces aren't very usable - the trees are "split" in the middle.

Based on some exploration of the codebase, it seems like bpf/tpinjector/tpinjector.c always prepends a Traceparent header, without attempting to check if there already exists a header. Is this by design, please? 🙏 I believe the W3C spec states there should only be one Traceparent header present.

[grcevski]

Hi @paweljw, thanks for reporting this. Can you tell me if it's possible that nginx is automatically replicating all incoming headers and putting them as outgoing?

I think we can try looking for an existing traceparent in the headers in tpinjector, it may not always work if we have large headers since BPF is limited with loop lengths, but we can make it work reliably on kernels 5.17 and newer with bpf_loop. @rafaelroquetto do you think this might be doable? You are the current owner of this area :).

[rafaelroquetto]

Hey guys! @grcevski, yes that should be doable.

@paweljw Just so I understand, are we talking about two mutually exclusive scenarios here? I.e. case 1 (external instrumentation + OBI) and case 2 (OBI + nginx) or is nginx the "external instrumentation" in this case?

We can detect the traceparent string and do something about it (like not re-injecting it) but that will only work if whichever external actor injecting the other traceparent does it so before Beyla/OBI - so I just wanted to understand if that is the case.

[paweljw]

@grcevski @rafaelroquetto thank you for getting back to me quickly!

Apologies, I wasn't very clear. The two scenarios only differ by the case of the header sent by the Sentry SDK which originates the trace - traceparent vs Traceparent. I initially thought the bug may be related to the case of the header, hence the two cases.

Here's a diagram - both cases/captures look like this. The only difference was hacking the Sentry SDK to send the exact same case as OBI injects.

So:

• the external actor absolutely does so before OBI: the http request arrives at the server already containing a traceparent header
• I would expect that the request leaving nginx would contain one traceparent header (per W3C spec)
• I would also expect that traceparent would contain the last span ID generated by OBI, rather than the span ID generated in frontend

I hacked the upstream to which nginx forwards to that effect. Relying on the fact that OBI prepends its header, I "manually" discard any subsequent traceparent headers. This gives me a sensible trace tree, reflecting what actually happens:

If I accepted the last traceparent header (injected by frontend Sentry SDK), I would get a tree like this:

Which would suggest both the nginx subtree, and the application subtree, have a parent span ID equal to the one originated in the frontend app. Which is not actually what the flow looks like :) this is also what the tree would look like if OBI chose not to inject a traceparent header at all, while still generating spans.

It would seem reasonable to me that OBI would replace the existing traceparent header, with a version that contains the last span ID it generated. This would allow any subsequent OTel instrumentation continue the tree where OBI left off.

As for:

Can you tell me if it's possible that nginx is automatically replicating all incoming headers and putting them as outgoing?

I would expect it to, yes. It's configured as a straight reverse proxy in this case, nothing fancy.

Let me know if I can help with any more detail 🙏