14. Standardizing Observability Tools
Date: 2023-01-21
Status
Accepted
Context
To maintain a consistent and effective approach to observability across our systems, we need to establish standard tools for monitoring, tracing, and logging.
Decision
Common landscape
Tracing
We use OpenTelemetry for tracing and metrics.
Kiali && Istio
- Kiali - Kiali is an open source project for service mesh observability, configuration, and validation. It provides dashboards, observability, and control of the service mesh and your microservices.
Consequences
- We use OpenTelemetry for tracing and metrics.
- Prometheus - Monitoring system
- prometheus-operator
- notify: slack, email, telegram
- prometheus-operator
- Grafana stack (LGTM)
- Grafana, the open and composable observability and data visualization platform.
- Loki, like Prometheus, but for logs.
- Tempo, a high volume, high throughput distributed tracing system.
- Grafana, the open and composable observability and data visualization platform.
- OnCall - On-call scheduling
- Pyroscope - Profiling and flame graphs
Common prometheus metrics
- Add default metrics for prometheus
- Exposing the software version to Prometheus
- Example: Add opencost_build_info metric