Unified LGTM (Loki, Grafana, Tempo, Prometheus/Thanos) and OpenTelemetry architecture with automated cross-layer AST verification, Google SRE Multi-Burn-Rate alerting, and GitOps blueprints.
Unified telemetry ingestion: distributed traces, log streams, and system metrics are normalized by OpenTelemetry Collectors and routed to dedicated storage backends.
Engineered for production resilience across local Docker Compose sandboxes, Kubernetes Helm/ArgoCD clusters, and AWS EKS via Terraform.
Horizontally scalable, multi-tenant log aggregation system optimized for cost efficiency through index-free metadata labeling.
Production dashboard visualization engine with pre-provisioned data sources and single-pane-of-glass operational telemetry.
High-scale distributed tracing backend designed for massive ingestion throughput with zero-dependency object storage persistence.
High-reliability metric TSDB with Thanos sidecar/store/querier integration for long-term historical retention and multi-cluster deduplication.
Vendor-agnostic telemetry proxy providing memory-bounded pipeline buffering, attribute redaction, and multi-destination dispatching.
Mathematical framework for high-precision, low-noise error budget consumption alerting adhering to Chapter 5 of the Google Site Reliability Workbook.
Given an availability target of 99.9% (SLO), the allowable unreliability is the Error Budget:
A Burn Rate of 1.0x exhausts exactly 100% of the budget over 30 days (720 hours). Faster consumption triggers multi-window paging:
| Alert Name | Severity Tier | Short Window | Long Window | Burn Rate | Budget Consumed | Routing Target |
|---|---|---|---|---|---|---|
ServiceErrorBudgetBurnRateHigh1h |
CRITICAL PAGER | 5 minutes | 1 hour | 14.4x | 2% in 1 hour | PagerDuty (Immediate Page) |
ServiceErrorBudgetBurnRateHigh6h |
CRITICAL PAGER | 30 minutes | 6 hours | 6.0x | 5% in 6 hours | PagerDuty (Immediate Page) |
ServiceErrorBudgetBurnRateSlow3d |
WARNING TICKET | 6 hours | 3 days | 1.0x | 10% in 3 days | Slack / Jira Backlog |
Adjust the parameters below to compute the instantaneous error budget burn rate and assess which alert tier triggers:
Single-sourced mathematical logic in deploy/slo/canonical_sli.promql is statically parsed and AST-verified across all deployment blueprints to eliminate metric drift.
Security & Cardinality Guardrail: Excludes 4xx client errors (`status=~"5.."` server-only) to protect against false pages triggered by bad clients, unauthorized scans, or user typos.
| Observability Layer | Target Configuration File | Rule / Metric Target | Window | AST Verification Status |
|---|---|---|---|---|
| Layer 1: Local Sandbox | deploy/docker-compose/prometheus/alerts.yml |
HighErrorRate |
5m | Verified Equivalent ✅ |
| Layer 2: Production GitOps | deploy/kubernetes/alerts/slo-alerts.yaml |
job:http_requests:error_rate_5m |
5m | Verified Equivalent ✅ |
| Layer 2: Production GitOps | deploy/kubernetes/alerts/slo-alerts.yaml |
job:http_requests:error_rate_30m |
30m | Verified Equivalent ✅ |
| Layer 2: Production GitOps | deploy/kubernetes/alerts/slo-alerts.yaml |
job:http_requests:error_rate_1h |
1h | Verified Equivalent ✅ |
| Layer 2: Production GitOps | deploy/kubernetes/alerts/slo-alerts.yaml |
job:http_requests:error_rate_6h |
6h | Verified Equivalent ✅ |
| Layer 2: Production GitOps | deploy/kubernetes/alerts/slo-alerts.yaml |
job:http_requests:error_rate_3d |
3d | Verified Equivalent ✅ |
| Layer 3: Infrastructure as Code | deploy/terraform/modules/grafana_provisioning/alerts.tf |
HighErrorRateP1 |
5m | Verified Equivalent ✅ |
Browse verified PromQL alert rules, OpenTelemetry pipeline definitions, and Alertmanager routing hierarchies.
# Canonical Service Level Indicator (SLI) - Availability Error Rate
# deploy/slo/canonical_sli.promql
(
sum(rate(http_requests_total{status=~"5.."}[{{WINDOW}}]))
or vector(0)
)
/
(
sum(rate(http_requests_total[{{WINDOW}}])) > 0
)
or vector(0)# Google SRE Multiwindow Burn-Rate Alert (14.4x consumption)
# deploy/kubernetes/alerts/slo-alerts.yaml
- alert: ServiceErrorBudgetBurnRateHigh1h
expr: |
(
job:http_requests:error_rate_1h{job="demo-app"} > (1 - 0.999) * 14.4
)
and
(
job:http_requests:error_rate_5m{job="demo-app"} > (1 - 0.999) * 14.4
)
for: 2m
labels:
severity: critical
tier: p1-pager
annotations:
summary: "High error budget burn rate (14.4x) on {{ $labels.service }}"
description: "Service {{ $labels.service }} is consuming 2% of its monthly error budget in 1 hour."
runbook_url: "https://github.com/umesh0492/cloud-native-observability/blob/main/docs/RUNBOOKS.md#serviceerrorbudgetburnratehigh1h"# OpenTelemetry Collector Pipeline Configuration
# deploy/docker-compose/otel-collector/otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 20
batch:
send_batch_size: 8192
timeout: 500ms
connectors:
spanmetrics:
histogram:
explicit:
buckets: [2ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]
dimensions:
- name: http.status_code
- name: http.method
- name: http.route
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
otlp/tempo:
endpoint: "tempo:4317"
tls:
insecure: true
otlphttp/loki:
endpoint: "http://loki:3100/otlp"
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp/tempo, spanmetrics]
metrics:
receivers: [otlp, spanmetrics]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/loki]# Alertmanager Multi-Tier Routing Configuration
# deploy/kubernetes/alerts/alertmanager.yaml
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'namespace', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'slack-notifications'
routes:
# Critical Tier -> PagerDuty Immediate Page
- match:
severity: critical
receiver: 'pagerduty-critical'
continue: true
# Warning Tier -> Slack Monitoring Channel
- match:
severity: warning
receiver: 'slack-notifications'
inhibit_rules:
# Inhibit warning alerts if the entire service is down
- source_match:
alertname: 'ServiceDown'
target_match_re:
alertname: 'HighErrorRate|HighLatencyP99'
equal: ['service', 'namespace']Comprehensive operational documentation, incident runbooks, and cloud infrastructure guides.