TraceLeakCheck

Survey · November 2026

Do GenAI instrumentations leak your secrets?

We planted fake API keys in an OpenAI app and captured every byte that five popular OpenTelemetry instrumentations exported. Privacy settings keep prompts out. They don't keep out a secret inside an error message.

8 of 9configurations exported a secret that appeared in an error message
0 of 3content opt-outs stopped it
3 of 9never export an abandoned stream: the span stays open, which looks "clean" to a naive checker
0 of 9leaked the client's own key from the Authorization header

Results

Each cell is one test: a fresh canary secret is planted, the app runs, and every OTLP trace, log and metric export is searched for it, raw or encoded. Every run passed a positive control, so "clean" means telemetry arrived and the secret wasn't in it.

leak: the secret was exportedopen span: never exported, so the result can't be checkedclean
Scenario OTel openai-v2 OpenLLMetry OpenInference Logfire httpx
defaultcontent onNO_CONTENT defaultTRACE_CONTENT
=false		 defaultHIDE_INPUTS
/OUTPUTS		 default default

"Content on" for openai-v2 means CAPTURE_MESSAGE_CONTENT=SPAN_AND_EVENT; see finding 4 for why =true isn't used. Full raw matrix: RESULTS.md.

Findings

FINDING 1

Content opt-outs do not cover exceptions

All three privacy settings we tested kept prompts, completions and tool arguments out of telemetry: TRACELOOP_TRACE_CONTENT=false, OPENINFERENCE_HIDE_INPUTS/_OUTPUTS, and openai-v2 NO_CONTENT. None of them stopped a secret inside an error message. Instrumentations copy the message into exception.message, exception.stacktrace and the span's status.message; openai-v2 sets status.message only.

  • When it bites: a gateway or proxy echoes a rejected key, an HTTP error includes a URL with a key in it, or a validation error quotes user input. The OpenAI API itself masks keys in its 401 messages.
  • Mitigation: none of these settings help. Scrub exception events and status in a span processor or exporter, for example traceleakcheck's scrub().
FINDING 2

URL query secrets are exported by the httpx instrumentation

Reported: contrib#5279

opentelemetry-util-http 0.66b2 redacts only AWSAccessKeyId, Signature, sig and X-Goog-Signature. The semantic conventions also list X-Amz-Signature, X-Amz-Credential and X-Amz-Security-Token, so an S3 presigned URL is exported with its signature and STS session token. Neither list includes key, so APIs that take ?key= get it written into url.full.

FINDING 3

Logfire's default scrubbing skips LLM messages and exception messages

Logfire scrubs by pattern, and api[._ -]?key is one of its patterns. Even so, the prompt OPENAI_API_KEY=<secret> and the error Invalid API key: <secret> both left the process unscrubbed. This is deliberate in 5.4.0: gen_ai.input.messages and gen_ai.output.messages are on its SAFE_KEYS list, and a source comment says exception-message scrubbing was removed.

FINDING 4

openai-v2's documented legacy =true captures nothing

Docs fixed in the successor package

The openai-v2 2.4b0 README documents CAPTURE_MESSAGE_CONTENT=true as the legacy value. On opentelemetry-util-genai 1.x it is rejected, capture falls back to NO_CONTENT, and the only sign is a log warning. A team that thinks it's auditing captured content is auditing nothing. It nearly fooled this survey too, so the runner now flags any capture-enabled config that comes back clean as MISCONFIGURED.

openai-v2 now lives on as opentelemetry-instrumentation-genai-openai, whose README lists only the valid values.

FINDING 5

A fresh install of openai-v2 fails to import

Known: genai#997

pip install opentelemetry-instrumentation-openai-v2 resolves opentelemetry-util-genai 1.3b0, which no longer ships opentelemetry.util.genai.instruments, so the import fails with ModuleNotFoundError. Pin opentelemetry-util-genai==1.1b0, or move to opentelemetry-instrumentation-genai-openai.

FINDING 6

Streaming: split secrets are reassembled, and an abandoned stream can hide a leak

Known: genai#491
  • Split across chunks: every instrumentation that records content reassembles the reply, so a secret split over two chunks is exported whole.
  • Abandoned after one chunk: OpenLLMetry and OpenInference end the span and export the partial reply, secret included. Logfire ends it without the partial reply. openai-v2 never ends the span, and a span that never ends is never exported.

Our first streaming run reported that openai-v2 case as a false "clean". traceleakcheck now tracks spans that start during a test and never end, and reports the test as incomplete.

Method

  • No real API. A local fake OpenAI server returns canned completions, tool calls, SSE streams and a 401 error.
  • Fresh secrets. Each test plants a new canary in a realistic sk-proj-… format.
  • Every export searched. traceleakcheck receives OTLP over HTTP and gRPC and matches raw, base64 (any offset), URL, hex and truncated forms. Every leak above was raw.
  • Positive control. A marker is sent through each pipeline after each test, so "nothing arrived" never counts as clean.
  • Defaults otherwise. Attribute length limits were left alone, so nothing was truncated.
openai
2.54.0
opentelemetry-sdk
1.45.2
openai-v2
2.4b0 (util-genai 1.1b0)
OpenLLMetry
0.62.4
OpenInference
0.1.69
Logfire
5.4.0
httpx instr.
0.66b2

Results describe these versions. Re-run on yours:

git clone https://github.com/preetsaini2004/traceleakcheck
cd traceleakcheck
uv sync --group survey
uv run python survey/run.py

Check your own app

traceleakcheck is the pytest plugin that produced this survey. It plants a fake secret, captures everything your app's real telemetry exports, and tells you in plain English where it leaked. scrub() fixes it; the next run proves it.

pip install traceleakcheck
pytest --traceleak
View on GitHub PyPI
traceleakcheck terminal output: 1 leak found in the error message, stack trace and status of step llm.call; after scrub(), 1 clean