BestHeartSurgeryBook Appointment

A Heart Centre’s Scheduling Team Learns MLOps the Hard Way

Healthcare Technology

Every hospital runs on schedules, and more of them than you would guess are now nudged along by prediction software. A tool might estimate which booked slots are likely to go unused, or which discharge paperwork will be late. None of that is clinical, but all of it is a trained model living inside a busy organisation, and keeping such a model honest is the job of MLOps. The story below follows an invented administrative team, so you can see the practice at work without any real department being described.

Picture a small analytics group at a cardiac centre. Their model predicts which outpatient follow-up slots will go empty so that front-desk staff can offer them to people on a waiting list. It was trained once, checked once, and switched on. For a few months everyone was pleased.

The first warning: nobody could rebuild it

When the group wanted to retrain with fresh appointment data, the analyst who had built the original was on leave. The notebook was there, but the folder of input files had been overwritten, and no one knew which settings had been used. The new run produced slightly different numbers, and the team spent two days arguing about whether the difference meant anything.

This is the problem the Ops Atlas treats first. In its entry on MLOps, the trained model is framed as something you build, with the training data counting as part of its source. A run can only be repeated later if you noted which code was used, a hash of the input files, the tuning settings and the software environment. The Atlas is relaxed about perfection, though: a repeat run passes when its scores stay inside an agreed band rather than agreeing to the last decimal.

The team adopted a one-page run card. Each training run now logs those same four items, and a colleague can repeat the work without a phone call.

The second warning: a better score that hurt one group

A few weeks later the team built a challenger model. Its overall score beat the live model by a small margin, and the temptation was to ship it that afternoon. A more careful colleague asked to see results for newly registered patients, a group the old model had handled well, purely as an administrative category with no clinical meaning.

The challenger was noticeably worse for that group. Taken alone, the headline number would have hidden it. The Atlas describes a worked example of exactly this shape: an overall gain of 0.009 in a ranking score, paired with a drop of 0.029 on one slice, and a policy that rejects the candidate until the slice is repaired. After a fix, the slice recovered, and only then did the candidate go forward.

The group wrote its own small policy in plain words:

  • a challenger must beat the incumbent by a minimum margin overall;
  • no named group may lose more than the margin the team agreed;
  • response time must stay inside the limit the front desk can tolerate;
  • the test set stays frozen and split by date, never shuffled.

The third warning: quiet trouble after launch

Months passed. A new referral route was introduced across the region, and the mix of appointment types changed. Nothing crashed, because nothing in a model like this ever crashes loudly. Staff simply noticed that the empty-slot suggestions felt off.

Here the team learned the difference between watching a service and watching a model. The service was up and fast. The inputs, however, no longer resembled what the model had learned from, and the real outcomes, whether a slot truly stayed empty, only became known days later. The Atlas suggests watching on three levels, namely the service, the inputs and the predictions’ quality, and notes that while labels are delayed you can watch proxies such as the spread of predicted values.

The team set up a weekly look at the distribution of predictions and a simple comparison of input mix against the training period. They also learned to ask what kind of change they were looking at before reaching for retraining. The Atlas separates three cases: pipelines that disagree with each other, a defect upstream at the source, and genuine drift in the world. Only the last of these is cured by training again.

Rolling back without drama

One Friday the new version produced strange suggestions after a schema change in the booking system. Their earlier habit would have been to retrain over the weekend. Instead they did what the Atlas advises and pointed traffic back at the last registered version. It took minutes. A non-model fallback, a plain rule based on last year’s patterns, kept the front desk working in the meantime.

The lesson they wrote on their wall was short: going back means repointing, never rebuilding from scratch. They also agreed to rehearse it every quarter, the way a ward rehearses a fire drill, so that the first real attempt is not the first attempt.

What the team changed in the end

None of the changes were expensive, and none involved buying a platform. If you work in a hospital’s operations or IT group, the pattern is worth copying at small scale. Other posts on this blog touch on adjacent themes from the patient side, for example a look at how remote check-ins fit into follow-up care, and our piece on planning care through video consultations. Both describe services that increasingly depend on software being looked after properly behind the scenes.

The team’s final checklist fit on an index card:

  1. Log the code version, data hash, settings and environment on each run.
  2. Compare any challenger on overall results and on named slices.
  3. Release gradually, with the old version still one step away.
  4. Watch service, inputs and outcomes, and diagnose before retraining.
  5. Keep a plain fallback and practise the switch.

The Atlas closes with modest starter advice: a single repeatable pipeline, a version registry that shows where each model came from, and an exit route will outperform a grand platform left unused. For a small team that is a relief. You do not need to automate everything on day one, and the Atlas is explicit that how much to automate is a decision, not a default.

This article is general information about software practice in healthcare administration. It does not describe any real hospital, and it contains no clinical guidance.