The story below is a made-up composite, a generic illustration rather than a report about any real hospital. It follows a small IT group that supports the scheduling and reminder tools behind a cardiac surgery department’s administrative side, and it shows what ObservabilityOps looks like when the telemetry system itself becomes the thing that needs care. Nothing here is clinical; it is about the plumbing that keeps appointment messages, bed-board screens and billing queues visible to the people who run them.
The dashboard that stopped loading
On a Monday morning the team lead opened the metrics dashboard before the weekly operations huddle and watched it spin. Panels that normally painted in a second now timed out. The nurses’ coordinators were not complaining yet, because the actual scheduling application was fine. What had failed was the team’s ability to see it. For two days nobody could say whether reminder messages were going out on time.
The group had four people, a handful of services, and one shared metrics server that had run happily for a year. The first instinct was to blame the server’s disk. The second was to blame the network. Neither was the cause, and the hunt took most of a day, which is the real lesson of the story: when the instrument panel breaks, the whole organisation flies blind, and nobody has been assigned to look after the panel.
What the culprit turned out to be
A developer had added a new counter for reminder deliveries and tagged every sample with the appointment identifier. It felt natural, because support staff liked searching by appointment. But every distinct identifier becomes its own time series in a metrics store. Thousands of appointments a week meant thousands of new series, each one carrying its own memory cost, and the count kept climbing as new bookings arrived.
The Ops Atlas, in its entry on ObservabilityOps, treats exactly this as a core duty of whoever operates the observability system: keep unbounded values such as user identifiers out of metric labels, and watch cardinality as a first-class number. It adds that the name is not an established industry term, which fits this team’s experience, since nobody had a job title for it.
- One label, one multiplier: each extra value in a label multiplies the series count.
- Identifiers belong in logs and traces, where a single record can carry them cheaply.
- A scrape that exceeds its configured series limit can fail as a whole, so the symptom looks like missing data everywhere, not one noisy metric.
The fix, in the order the team did it
First they removed the appointment label and redeployed the exporter. Series growth stopped within the hour, and the dashboard recovered after the old series aged out. Second, they wrote down a small rule: metric labels hold a handful of values, such as delivery channel and outcome, and anything with thousands of values goes into structured logs. A guideline in the Atlas’s material suggests keeping typical metric cardinality in single digits and investigating when it passes the low hundreds, and the team adopted a looser version of that as a review habit.
Third, they set a per-target limit on how many series one service may expose. This sounds harsh, but it turns a silent platform-wide failure into a loud failure of one service that is easy to attribute. They also added a plain check on a spare screen: a permanently firing test alert, where silence on the receiving end proves the notification route has died, so that a dead pipeline cannot pass for a calm day.
Who owns the watching
The retrospective produced the most useful outcome. Someone had to own the telemetry platform, even part time. The team named one engineer as the observability owner, with a backup, and gave that person three standing tasks: review new labels before release, keep a short inventory of what is collected and why, and run a quarterly pass over alerts nobody acted on. If you want a related read on how clinical follow-up uses remote contact, the site’s piece on telehealth during the weeks after heart surgery shows how much of modern care depends on systems like these staying visible.
Their approach also matches advice that monitoring should not sit with one person alone. The owner sets standards and runs the platform; each service team still decides which of its own symptoms deserve a page. That split kept the owner from becoming a bottleneck for every alert rule in the building.
Smaller changes that paid off
Beyond the label rule, three smaller changes made the platform calmer over the following quarter:
- Structured logs. Each delivery log record now opens with its trace and span ids, so support staff could jump from a complaint to the matching request without guessing.
- Retention by tier. Recent data stayed on fast storage, older data moved to cheaper storage, and very old debug output was archived. The default retention on a standard metrics server is only about two weeks, so the team wrote its own policy rather than inheriting an accident.
- A page test. Once a month someone deliberately triggers one synthetic failure and confirms a human receives the page. One end-to-end test told them more than a dozen per-component alert checks.
They also looked sideways at tooling choices. Before buying anything new, the group read a short piece on matching software to business goals, which reminded them to define the problem first. The problem was not a lack of dashboards. It was that nobody had agreed what the platform was for.
What to take from it
The cost of ignoring the platform shows up at the worst time, because an unhealthy telemetry system hides itself. Teams that keep a named owner, a label rule, a series cap and a heartbeat get to find out about trouble from the system instead of from a frustrated coordinator. The group in this story never bought a new product. They agreed on four habits, wrote them on one page, and put the page where the next developer would see it.
If your own department runs on a tangle of schedulers, messaging tools and reporting jobs, start with a small question: if the dashboard went blank tomorrow, who would notice, and who would fix it? The honest answer shows where the first habit belongs. For the broader reliability mindset behind this, a look at reliability training for modern systems is a sensible next stop.