Telemetry & Triage

Telemetry is the data an application emits: request counts, timings, errors, logs, and traces. Observability is the ability to explain the system's behavior from those signals. During an incident, they help establish who is affected, where work is failing, and what could restore service.

Start with one operation. A user clicks Export, the API accepts a job, a worker generates the file, and the browser offers a download. An HTTP 202 confirms acceptance. It says nothing about whether the file will ever be ready.

Browser → POST /exports → durable job → queue → worker → stored file
   ↑                                                       |
   └──────────── status response and download link ─────────┘

Measure the promises along that path: the click receives feedback, the job is durably accepted, the file becomes available, and the user can download it. Record acceptance and completion separately. Otherwise, an API can stay green while every export is stuck.

Each signal answers a different question:

On the backend, instrument incoming requests, outgoing HTTP calls, database operations, and messaging first. Add business spans around work such as generating an export. Framework instrumentation can report request duration; it cannot decide when your product considers an export complete.

Give each service a name, environment, and release revision. Generate an operation ID when the job is accepted and keep it through retries. A trace ID connects related spans; an operation ID connects the durable job, including attempts that happen in different traces.

Pass trace context through the queue too. With the OpenTelemetry SDK and exporter configured at process startup, an application integration can inject context when publishing and extract it when processing:

from opentelemetry import trace
from opentelemetry.propagate import inject, extract

tracer = trace.get_tracer("exports")

# Producer: called inside the instrumented API request.
headers = {}
inject(headers)
publish_job(job, headers=headers)

# Consumer: one message, with its own processing span.
with tracer.start_as_current_span(
    "export.process",
    context=extract(message.headers),
    kind=trace.SpanKind.CONSUMER,
) as span:
    span.set_attribute("export.format", message.format)
    file = generate_export(message)
    mark_completed(message.operation_id, file)

publish_job, message, generate_export, and mark_completed belong to the application. The span records processing time; acceptance timestamps are still needed to measure queue waiting and total completion time. Batch consumers may need span links to several producer contexts instead of choosing one message as the parent.

Emit structured logs at useful transitions: accepted, processing, completed, and failed. Correlate them with the current trace and the deployed artifact. An application-owned failure record could look like this:

{
  "event": "export.failed",
  "service_name": "export-worker",
  "environment": "production",
  "region": "eu-west",
  "release": "worker@a81f2",
  "operation_id": "export-842",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "error_code": "POOL_TIMEOUT",
  "duration_ms": 2004
}

Use stable classifications such as POOL_TIMEOUT for grouping. Keep operation IDs in searchable log fields rather than metric labels: a new label value for every job creates an ever-growing set of time series. Use route templates such as /exports/:id, and keep request bodies, access tokens, and download URLs out of routine captures.

Backend dashboards need request rate, error rate, and duration, plus the resources that can constrain them: CPU, memory, connection-pool wait, active transactions, worker concurrency, queue depth, and oldest job age. Low CPU does not establish spare capacity when every worker is waiting for a database connection.

For a service emitting classic Prometheus histogram buckets, this query estimates p95 request duration per region across its instances:

histogram_quantile(
  0.95,
  sum by (le, region) (
    rate(http_request_duration_seconds_bucket{
      service_name="exports-api",
      route="/exports",
      method="POST"
    }[5m])
  )
)

The labels must match the emitted metric. Aggregate bucket rates before calculating the percentile; averaging instance p95 values does not produce the service p95. Keep success and failure populations distinguishable, and pair this graph with overdue jobs. Fast acceptance can hide slow completion.

The frontend needs its own evidence. Capture JavaScript exceptions and unhandled promise rejections, failed requests, the deployed JavaScript release, and enough navigation context to reproduce a failure. Upload source maps for that exact build to the error tracker. A minified stack resolved against yesterday's assets can send the investigation to the wrong code.

Measure loading, responsiveness, and visual stability from real browsers. LCP tracks when the largest visible content element appears, INP measures interaction responsiveness, and CLS measures unexpected layout shifts. Add a separate product timing for the export journey; these browser metrics do not measure whether the file was generated.

For an export page built with the web-vitals package installed, this browser integration reports measurements to an application-owned endpoint:

import { onCLS, onINP, onLCP } from "web-vitals";

function reportVital({ name, value, id }) {
  navigator.sendBeacon("/telemetry/vitals", JSON.stringify({
    name,
    value,
    id,
    route: "/exports",          // Stable page name; no customer IDs.
    release: "web@c29d1",        // Supplied by the build in production.
  }));
}

onCLS(reportVital);
onINP(reportVital);
onLCP(reportVital);

The receiver must accept this JSON text payload, validate allowed fields, and enforce ingestion limits. Use the metric ID to reconcile updates to the same measurement rather than counting every callback as a new page visit. Beacon delivery is best effort. Ad blockers, closed tabs, and broken JavaScript can remove browser evidence, so compare it with edge requests, synthetic journeys, and backend records.

For browser-to-API traces, configure the SDK to propagate context to the intended API origins and allow the required tracing headers through CORS. Queue propagation then carries the path into the worker. Keep a request reference available in error responses so a support report can lead to backend evidence even when a browser trace was not retained.

A practical toolchain might use OpenTelemetry for instrumentation and collection, Prometheus for metrics, Loki for logs, Tempo for traces, and Grafana to explore them. New Relic can provide an integrated APM workflow. Sentry or Honeybadger helps group application errors; a masked FullStory replay can show what happened around the user's click. These are choices for collecting and inspecting evidence; installing every product adds little if the signals cannot be correlated.

Product events in PostHog, Pendo, or GA4 can show which release cohort stops reaching the download step. Emit an authoritative export.completed event from the backend after completion, and distinguish it from the browser's download.clicked event. A replay or funnel can suggest where the journey broke; neither proves that a file was committed or downloaded successfully. Mask sensitive screens, restrict captured fields, and test what actually leaves the browser.

Bound the overhead of collection. Batch exports, cap buffers, choose sampling and retention deliberately, and monitor dropped records and Collector/exporter failures. Keep aggregate outcome metrics separate from sampled traces. If an exporter stops sending, an empty graph must not become evidence of recovery.

Now suppose exports stop completing in one region shortly after a worker release. The API still returns 202; the export screen remains responsive. A useful alert is tied to the operation's promise. For example, target 99% of eligible exports finishing within sixty seconds over a rolling thirty-day window. The remaining 1% is the error budget; page when short and long observation windows both show rapid consumption. Count overdue accepted jobs as failures for that deadline even while they remain pending. Watching only terminal failures would miss the outage.

The first task is to establish impact. Fix a UTC incident interval, compare the affected region with a healthy one, and check the release and configuration timeline. Estimate affected operations from durable job records rather than multiplying error logs. Assign an incident lead, an investigator for the failing boundary, and someone to keep customers and support informed when several responders are involved.

Follow a representative affected operation from the browser into its backend trace. In this example, page timings and API acceptance are normal, queue age is rising, and workers spend most of their time waiting for connections. Query execution itself remains fast. That points toward pool occupancy or long transactions; it does not justify adding API replicas.

Other observations would lead elsewhere:

Keep hypotheses specific enough to disprove. For the pool investigation, inspect PostgreSQL activity using an account allowed to see the relevant sessions:

SELECT pid, state, wait_event_type, wait_event,
       now() - xact_start AS transaction_age,
       pg_blocking_pids(pid) AS blocked_by
FROM pg_stat_activity
WHERE datname = current_database()
  AND state <> 'idle'
ORDER BY xact_start NULLS LAST;

This includes idle in transaction sessions, which can hold a transaction open without executing a query. An empty blocker list does not rule out an application pool leak. Correlate the sessions with worker traces: here, the new release calls an external enrichment service while holding a transaction open. Slow enrichment keeps connections occupied; other jobs time out acquiring them. Compare a healthy trace to confirm the changed ordering. Overlapping child spans must not simply be added together to explain wall-clock time.

Choose a mitigation against that mechanism. If the previous worker release can read the queued payloads and current schema, roll it back. Limit intake or retries if they are adding pressure, while preserving accepted jobs. Raising connection limits or replaying the whole queue can move the overload into the database. Record each intervention and its owner so the effect can be evaluated.

A short incident update is enough: “EU exports are delayed. Accepted jobs are retained. We rolled back the worker release and are checking completion and backlog drain. Next update at 08:30 UTC.” Keep observations, suspected causes, and decisions separate. Give a recovery estimate only after measuring sustainable progress.

Recovery means new exports finish, previously accepted work drains, and the browser can retrieve the generated files. Check authoritative state as well as graphs. Assuming an export_jobs table with durable acceptance times and these pending states, this read-only query finds jobs still beyond the example deadline:

SELECT region, count(*) AS overdue,
       min(accepted_at) AS oldest_accepted
FROM export_jobs
WHERE state IN ('queued', 'running')
  AND accepted_at < now() - interval '60 seconds'
GROUP BY region;

Investigate terminal failures separately and reconcile file storage, notifications, or charges through stable operation IDs. Retry with the same idempotency contract so recovery does not create duplicate effects. A falling queue depth is encouraging; oldest job age and completed customer journeys establish whether the affected work is recovering.

Keep an owner for customer remediation after service restoration. Record the release trigger, the connection-holding mechanism, any retry amplification, and why the earlier alerts missed overdue jobs. Move enrichment outside the transaction, reproduce the failure with a slow dependency, and verify the new deadline alert with a deliberately delayed job. Give each follow-up an owner and a completion check.