Observability & Telemetry: OpenTelemetry Tracing, Prometheus Metrics & Structured Logging

In complex distributed microservice environments, understanding system state by manually inspecting isolated server logs is fundamentally impossible. An asynchronous request traversing five distinct services, two message queues, and a database cluster can fail silently or suffer inexplicable tail latency ($p99$) bottlenecks without continuous telemetry. True observability enables engineering teams to infer the internal states of a distributed system based entirely on its external outputs. This requires uniting the three fundamental telemetry pillars—distributed traces, time-series metrics, and structured logs—into a correlated, vendor-neutral observability pipeline powered by OpenTelemetry, Prometheus, and Grafana Loki.

1. The Three Telemetry Pillars: Architectural Alignment

Each observability pillar answers distinct diagnostic questions and imposes unique storage and ingestion requirements:

2. Distributed Tracing & W3C TraceContext Propagation

Distributed tracing tracks the complete lifecycle of a request as it cascades across service boundaries. OpenTelemetry (OTel) standardizes tracing APIs, SDKs, and collection infrastructure. Context propagation relies on standard W3C TraceContext HTTP headers:

• traceparent: A 4-part hex-encoded header formatted as 00-{trace_id}-{parent_id}-{trace_flags} (e.g., 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01). The trace_id remains immutable across every service in the request graph, while parent_id updates per span.
• tracestate: Key-value metadata passed across vendor boundaries without mutating the primary trace identifier.
• Trace-Log Correlation: By automatically injecting the active trace_id and span_id into application log records (e.g., in Winston or Pino formatters), operators inspecting an alert in Grafana can click directly from a metric spike into the corresponding trace waterfall, and from the trace waterfall directly into filtered Loki log lines.

3. Metric Instrumentation: The RED and USE Methods

Prometheus collects time-series metrics via pull-based HTTP scraping (typically on /metrics). Production metric architectures organize alerts around proven SRE methodologies:

• The RED Method (For Request-Driven Services):
  • Rate: The number of requests processed per second (e.g., http_requests_total Counter).
  • Errors: The number of failing requests (e.g., HTTP 5xx responses).
  • Duration: The time required to complete requests, measured using Prometheus Histogram buckets to calculate $p50$, $p90$, and $p99$ percentiles via histogram_quantile().
• The USE Method (For Infrastructure Resources):
  • Utilization: Percentage of time the resource is busy (CPU, Disk I/O).
  • Saturation: Extra work queued waiting for resource capacity (thread pool queue size, runqueue).
  • Errors: Hardware and OS error counts (dropped packets, disk write errors).

4. Structured Logging with Promtail and Grafana Loki

Traditional logging aggregators (Elasticsearch / OpenSearch) index the entire text of every log message, requiring immense memory and disk storage. Grafana Loki inverts this model: it indexes strictly the stream metadata labels (e.g., {app="checkout", environment="production"}) while compressing log message contents into append-only chunks.

• Promtail Ingestion Pipeline: Lightweight Promtail agents tail local log files on each production node, apply regex or JSON parsing stages, extract trace IDs, and push compressed log streams to Loki over HTTP/gRPC.
• LogQL Querying: Operators query logs using LogQL: {app="api-gateway"} |= "error" | json | latency > 500ms. This scans high-throughput log streams dynamically without incurring heavy full-text database index overhead.

5. Production Implementation: OpenTelemetry Node.js SDK Setup

The following TypeScript configuration initializes the OpenTelemetry Node.js SDK with automatic HTTP/Express instrumentation, W3C context propagation, and Prometheus metric exporting:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';

// 1. Initialize Prometheus metric scraper on port 9464
const prometheusExporter = new PrometheusExporter({ port: 9464 }, () => {
  console.log('Prometheus scrape endpoint active: http://localhost:9464/metrics');
});

// 2. Initialize OTLP gRPC trace exporter to OpenTelemetry Collector
const traceExporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://otel-collector:4317'
});

// 3. Configure OpenTelemetry NodeSDK
export const otelSdk = new NodeSDK({
  serviceName: 'moemecto-core-api',
  traceExporter,
  metricReader: prometheusExporter,
  instrumentations: [
    getNodeAutoInstrumentations({
      // Auto-instrument HTTP, Express, Redis, and PostgreSQL
      '@opentelemetry/instrumentation-fs': { enabled: false }, // Prevent fs span noise
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingRequestHook: (req) => req.url?.includes('/api/health') || false
      }
    })
  ]
});

// Start telemetry collection prior to importing application modules
otelSdk.start();

process.on('SIGTERM', () => {
  otelSdk.shutdown()
    .then(() => console.log('OpenTelemetry SDK cleanly terminated'))
    .catch((err) => console.error('Error terminating OTel SDK:', err))
    .finally(() => process.exit(0));
});