2 Monitoring and Alerting
Learn how to set monitoring goals, use telemetry and dashboards to understand service health, and design actionable alerts that detect user impact without creating unnecessary noise.
Set goals and monitor user impact
Monitoring collects, processes, and presents information about a system so teams can understand its health, detect problems, investigate causes, and assess changes over time. Set operational goals before choosing or alerts. Useful goals include quickly detecting user-visible failures, identifying emerging capacity risks, supporting incident diagnosis, and checking whether service objectives are being met.
Monitoring should not depend on someone continuously watching a dashboard. A useful design combines , which tests a service as users experience it, with of internal components. For example, a periodic request to a public endpoint can reveal that a service is unavailable, while internal measurements can help locate whether the database, application, or network is responsible.
Alert primarily on symptoms that matter to users. Use internal signals to investigate those symptoms and to detect failures that are clearly imminent.
Use , , and
, , and provide complementary views of system behavior. Correlating them gives a more complete picture than relying on any one signal.
are numerical measurements over time, including request rate, error ratio, memory use, and request duration. For a user-facing service, the four are:
Latency: how long requests take.
Traffic: the amount of demand on the service.
Errors: the number or proportion of requests that fail.
Saturation: how close the system is to its capacity.
Track distributions such as latency percentiles, not only averages; averages can hide slow experiences for some users. Choose that answer operational questions. A checkout service, for example, might track completed checkouts per minute, the fraction of checkouts returning errors, the ninety-fifth-percentile response time, and database connection-pool utilization. Keep metric definitions and units consistent so comparisons over time are meaningful.
are timestamped event records. Prefer structured with consistent fields such as time, severity, service, and request or trace identifier. This makes records easier to search and correlate with other telemetry. Avoid logging secrets or unnecessary sensitive data.
show a request’s path across services and help identify where time or errors arise in a distributed system. Consider alongside and when investigating service behavior.
Build dashboards and reliable detection
A dashboard should make service health understandable at a glance and help responders move from a symptom toward likely causes. Show key service-level , often the four , alongside relevant dependencies and recent changes. Provide links to or , and use clear labels, appropriate time ranges, and consistent scales.
A dashboard supports observation and diagnosis, but it does not replace alerts for urgent conditions. Detection should combine service-level signals with carefully selected component signals. For example, a rise in failed checkout requests is a user-facing symptom, while an increase in database timeouts may help explain it.
Monitor the monitoring path as well. A broken collector or notification route can make a dashboard dangerously incomplete even when the system appears healthy. Check that alert delivery works end to end, rather than checking only individual monitoring components.
Design actionable alerts
An alert is a notification that requires a person to act. Before creating one, ask whether the problem is urgent, whether it is affecting users or likely to do so soon, and whether the recipient can take a specific action.
Page responders for urgent, actionable problems. Route important but non-urgent work to a ticket or review dashboard. If an alert has no clear response, consider removing it, automating the response, or lowering its priority.
Prefer alerts based on sustained user impact over alerts about isolated component conditions. For example, rather than paging whenever one server briefly reports high CPU, alert when checkout errors or latency remain above a meaningful service threshold long enough to indicate sustained user impact. A separate capacity warning may be appropriate when a resource is approaching a hard limit and failure is imminent.
Set thresholds and evaluation windows based on service objectives and observed behavior. Each alert should include a concise description, the affected service, a dashboard link, and response guidance.
Reduce alert noise and improve quality
Too many low-value notifications can cause and obscure real incidents. Reduce noise while preserving the signals responders need:
Alert on meaningful symptoms. Prefer service-level impact over isolated component anomalies, except when a component condition signals an imminent failure.
Allow brief recovery. Use suitable evaluation windows or persistence conditions so short-lived blips do not repeatedly interrupt responders.
Group related alerts. Combine notifications from one incident into a concise message while retaining details about affected components.
Deduplicate and inhibit. Suppress repeated notifications, and mute dependent alerts when a higher-level outage already explains them.
Use bounded silences. Silence known maintenance or test conditions for a defined period, with a reason and an owner. Do not let silencing hide ongoing incidents.
Review alerts after incidents. Remove alerts that were redundant or unactionable, and improve alerts that failed to detect impact promptly.
Prometheus supports grouping, deduplication, routing, inhibition, and silences to help control notification volume. These mechanisms should reduce duplicate noise without suppressing the signal responders need.