Production Observability with Signoz

This covers how I set up Signoz as the primary observability platform for production workloads, complemented by CloudWatch for AWS-native metrics. On the side, I explore the LGTM stack (Loki, Grafana, Tempo, Mimir) for advanced patterns.

Why Signoz?

All-in-one: Traces, metrics, and logs in a single platform
OpenTelemetry native: No proprietary agents or lock-in
Cost-effective: Self-hosted, no per-GB ingestion fees
ClickHouse backend: Blazing fast queries on large datasets

Architecture

OpenTelemetry Collector Configuration

The collector acts as the central pipeline — receive from apps, process, and export to Signoz:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000

  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  resource:
    attributes:
      - key: environment
        value: production
        action: upsert
      - key: service.namespace
        value: indomarco
        action: upsert

  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code:
          status_codes:
            - ERROR
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 1000
      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

exporters:
  otlp/signoz:
    endpoint: signoz-otel-collector:4317
    tls:
      insecure: true

  awscloudwatch:
    region: ap-southeast-1
    namespace: CustomMetrics
    metric_declarations:
      - dimensions: [[service.name]]
        metric_name_selectors:
          - "http.server.duration"
          - "http.server.request.size"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource, tail_sampling]
      exporters: [otlp/signoz]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp/signoz, awscloudwatch]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp/signoz]

Application Instrumentation (Node.js Example)

const { NodeSDK } = require('@opentelemetry/sdk-node')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc')
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-grpc')
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node')
const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics')

const sdk = new NodeSDK({
  serviceName: process.env.SERVICE_NAME,
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_COLLECTOR_URL || 'grpc://otel-collector:4317',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: process.env.OTEL_COLLECTOR_URL || 'grpc://otel-collector:4317',
    }),
    exportIntervalMillis: 30000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': { enabled: true },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-mysql2': { enabled: true },
      '@opentelemetry/instrumentation-redis': { enabled: true },
    }),
  ],
})

sdk.start()

Alerting Rules

groups:
  - name: service-health
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(signoz_calls_total{status_code="STATUS_CODE_ERROR"}[5m])) 
          / sum(rate(signoz_calls_total[5m])) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected (> 5%)"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, sum(rate(signoz_latency_bucket[5m])) by (le, service_name)) > 2000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency above 2 seconds"

      - alert: ServiceDown
        expr: |
          up{job=~".*"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service instance is down"

LGTM Stack (Hobby / Exploration)

For personal projects and exploratory work, I run the Grafana LGTM stack:

Component	Purpose
Loki	Log aggregation (like CloudWatch Logs but self-hosted)
Grafana	Visualization and dashboards
Tempo	Distributed tracing
Mimir	Long-term metrics storage

This gives me hands-on experience with the full Grafana ecosystem, which I can leverage when teams prefer Grafana-based tooling.

Results

70% faster incident resolution with correlated traces, metrics, and logs
Tail sampling reduces storage costs by 90% while keeping all error traces
< 5 minute detection for anomalies via alerting rules
Full request lifecycle visibility across 100+ microservices