14. Standardizing Observability Tools
Date: 2023-01-21
Status
Accepted
Context
To maintain a consistent and effective approach to observability across our systems, we need to establish standard tools for monitoring, tracing, and logging.
Decision
Tracing
We use OpenTelemetry for tracing and metrics.
Kiali && Istio
- Kiali - Kiali is an open source project for service mesh observability, configuration, and validation. It provides dashboards, observability, and control of the service mesh and your microservices.
Consequences
- We use OpenTelemetry for tracing and metrics.
- Prometheus - Monitoring system
- prometheus-operator
- notify: slack, email, telegram
- prometheus-operator
- Grafana stack (LGTM)
- Grafana, the open and composable observability and data visualization platform.
- Loki, like Prometheus, but for logs.
- Tempo, a high volume, high throughput distributed tracing system.
- Grafana, the open and composable observability and data visualization platform.
- OnCall - On-call scheduling
- Pyroscope - Profiling and flame graphs
Common prometheus metrics
- Add default metrics for prometheus
- Exposing the software version to Prometheus
- Example: Add opencost_build_info metric