Errors, Logs and Telemetry

Telemetry is evidence emitted by a running system. Observability is the ability to use that evidence to explain its behavior. A failed browser request needs several kinds of evidence: what the user saw, which API attempt handled it, where time went, and which release ran the code.

The downloadable lab runs a browser page and API on an ephemeral local port. Node.js 22+ is sufficient for its HTTP tests and manual browser demonstration. Automated browser checks additionally need Playwright and Chromium or Chrome; the README gives the exact commands. No vendor account, exporter, or app dependencies are required.

An error is a contract and an event

An API error has two audiences. The caller needs a stable code, appropriate HTTP status, safe explanation, and a reference for support. Engineers need an event correlated with the actual handling attempt. Dumping the exception into both places conflates those needs and can expose connection strings, input values, and stack details.

The lab deliberately throws an exception whose message contains a fake database password. Its handler maps that failure to a 503, builds the public body from known fields, and records a separate event:

// These values are selected by the handler, not copied from Error.message.
const code = 'DEPENDENCY_UNAVAILABLE';
const body = {
  code,
  message: 'Item details are temporarily unavailable.',
  request_id: requestId,
};

state.logs.push({
  event: 'item.lookup',
  service_name: 'items-api',
  environment: 'local',
  release: 'api@demo-a1',
  request_id: requestId,
  trace_id: span.traceId,
  span_id: span.spanId,
  route: '/api/items/:id',
  status: 503,
  code,
  duration_ms: duration * 1000,
});
// Tests assert no password, credentials, query string or stack is serialized.

The server generates request_id; incoming trace context is client supplied. Neither identifier grants access. A durable operation may need its own ID across retries, because a later attempt can belong to a different trace.

Structured fields support exact filtering and aggregation without parsing prose. Stable error classifications also prevent grouping every occurrence separately because its message includes a different item ID. Detailed exceptions can still be useful in a restricted error store, with deliberate filtering and retention. The lesson's log is intentionally narrower.

Release identity must identify the artifact actually executing: backend revision, frontend asset build, environment and service. The lab uses fixed demonstration revisions. Production supplies them during build/deployment. A JavaScript stack needs source maps matching that exact asset build; rebuilding from the same branch does not guarantee identical minified locations.

Carry context across the browser boundary

A trace contains spans: individually timed operations with parent relationships. Reusing a trace ID groups work; parent span IDs explain its causal structure. The lab creates this path:

item.load (browser action)
└── GET /api/items/:id (browser fetch)
    └── GET /api/items/:id (API handler)
        └── catalog.lookup (dependency operation)

W3C traceparent version 00 carries a 32-character hexadecimal trace ID, a 16-character caller span ID, and flags. The sampled flag is the low bit. All-zero trace/span IDs are invalid. Each instrumented operation creates its own span ID; forwarding the caller's span ID as the server's identity would collapse distinct work.

The actual browser module and HTTP handler use the shared lab context functions:

// Browser: action is the parent of this outgoing request span.
const action = child(null, sampled);
const request = child(action);
const response = await fetch('/api/items/42?fail=1', {
  headers: { traceparent: header(request) },
});

// API: parseParent returns null for invalid version-00 context.
const span = child(parseParent(req.headers.traceparent));
const dependency = child(span);
// Same trace ID; dependency.parentSpanId === span.spanId.

// The downloaded context.mjs serializes the outgoing carrier as:
function header(span) {
  return `00-${span.traceId}-${span.spanId}-${span.sampled ? '01' : '00'}`;
}

The automated browser check captures the real outgoing header and asserts every parent edge. The HTTP tests verify that the API log's span_id matches its handling span, and that concurrent requests keep separate contexts. Invalid input starts a fresh trace.

This dependency-free fixture implements a small version-00 carrier and local span records. It is not an OpenTelemetry SDK, OTLP exporter, or complete Trace Context implementation: it omits tracestate and future-version interoperability. Production OpenTelemetry instrumentation handles injection/extraction, active context through asynchronous calls, span lifetimes, and export. Manual instrumentation adds application meaning around framework spans; an API package alone does not configure recording/export.

Propagate only to intended origins. A different API origin also requires the appropriate CORS allowance for tracing headers. Trace context is correlation data, and baggage may contain sensitive application data; neither should be copied indiscriminately to third parties. The lab stays same-origin and does not transmit baggage.

Count every outcome, retain selected detail

A metric keeps bounded aggregates rather than one record per request. The lab records request count, duration sum in seconds, and cumulative latency buckets at 5, 25, 100 and 1,000 milliseconds plus infinity. This groups latency into ranges while preserving a total population count.

Its keys are method, route template and outcome. Thirty different item IDs produce just two aggregate groups: success and error. Trace IDs, emails and raw paths would create a new label combination for nearly every request. With multiple labels, their possible combinations multiply; a small label added to an already large set can be expensive.

The important ordering in server.mjs is:

const key = `GET /api/items/:id ${outcome}`;
const metric = state.metrics.get(key) ?? {
  count: 0, sum_seconds: 0, buckets: buckets.map(() => 0),
};
metric.count++;
metric.sum_seconds += duration; // duration is in seconds
buckets.forEach((bound, i) => {
  if (duration <= bound) metric.buckets[i]++;
});
state.metrics.set(key, metric);

// Outcome metrics and safe logs above are independent of span retention.
if (span.sampled) state.spans.push(
  finish(dependency, 'catalog.lookup', dependencyStart, status === 200 ? 'OK' : 'ERROR'),
  finish(span, 'GET /api/items/:id', start, status === 200 ? 'OK' : 'ERROR'),
);

The test sends 30 unsampled requests, half failing. It asserts 30 logs, zero retained spans, 15 successes, 15 errors, two groups, and monotonically increasing cumulative buckets. The infinity bucket equals the count. This is an in-memory teaching aggregate, not a Prometheus server or a measured capacity claim.

Head sampling decides near the start and can reduce recording/export costs, but cannot know that a request will fail later. Tail sampling decides after collecting enough spans to inspect outcomes or durations; it requires buffering, routing related spans together and handling late arrivals. It cannot recover spans already discarded upstream. The lab honors the incoming sampled flag for demonstration; a production public edge needs its own collection budget rather than allowing callers to demand unlimited retention.

Keep counts used for error rates independent of sampled traces, and distinguish successful and failed latency populations. Logs and errors need collection limits too. Exporter queues should be bounded, with explicit drop behavior and telemetry about dropped records. An empty backend can mean ingestion failed. The fixture's arrays are inspectable test storage, deliberately unsuitable as an unbounded production exporter.

Protect the browser evidence

A browser may fail before making an API call. Capture JavaScript exceptions and unhandled rejections, the current asset release, and a small set of navigation/action identifiers. A network failure may have no response reference at all. Keep those categories distinct from an HTTP error that the browser received successfully.

The lab's browser reports the API's request_id with its own release and trace ID. It also sends a deliberately small, private-by-default DOM snapshot. Its serializer transmits no attributes, input values, or URLs. Explicitly public text survives; a masked ancestor stays masked; excluded subtrees disappear before serialization. Browser tests seed fake secrets into text, descendants, attributes and links, then assert those strings are absent from the outgoing payload.

FullStory's documented privacy classes provide a related optional vendor configuration, not executed in this lab:

<!-- FullStory: mask text while retaining supported interaction information. -->
<div class="fs-mask">Sensitive account text</div>

<!-- FullStory: exclude this subtree and its targeted interaction events. -->
<div class="fs-exclude"><img src="sensitive-image.png" alt="Private document"></div>

The local serializer recognizes these classes for its own stricter teaching policy; it does not validate FullStory behavior. Masking text does not hide sensitive image content; exclude that region. Sensitive attributes need separate scrutiny too. Test what actually leaves the browser, including console/network capture and custom events. DOM masking cannot sanitize secrets separately copied into an error message or telemetry payload.

Tools occupy different parts of this evidence path:

Choosing a product does not create missing correlations or establish that captured data is safe. This lab exercises the boundaries locally; no vendor integration or dashboard was configured. For investigation, mitigation and recovery using these signals, continue to Telemetry & Triage.