When the helper is very sure, it can go ahead. When it's not sure, it asks a person. When something looks wrong, it stops and doesn't do anything.
Every case a model scores goes one of three ways. It acts on its own, it hands the case to a person, or it stops because something's wrong. Deciding the rules for each route in advance is what makes the system safe.
Define three routes: autonomous action above a calibrated threshold, human review in the uncertain band or for high-stakes cases, and a hard stop when inputs are out of distribution, data is missing or a policy rule fires. Log every route taken.
Act
The model acts on its own when three things are true: its score is above the action threshold, the stakes for this case are within agreed limits, and nothing unusual has been flagged. A routine £40 expense with a receipt and a known supplier is a typical example.
Log every automatic action so it can be sampled and checked later.
Ask a person
The case goes to a person when the score falls in the uncertain band, when the stakes are above a set limit whatever the score, or when the case belongs to a category you've decided people always handle.
Give the person what they need to decide quickly: the facts, the model's suggestion and score, and why it was sent to them. Record what they decided. Those records are the best training data you'll get.
Stop
Some cases shouldn't be scored at all. Stop and raise an alert when:
- required data is missing or malformed
- the case looks unlike anything the model has seen before
- a policy rule fires, such as a sanctioned supplier or an amount over a hard limit
- the model's recent behaviour has drifted, for example a sudden jump in approvals
Stopping isn't a failure. It's the system knowing the edge of what it can safely do.
Keeping the routes honest
Check the split every week. If the share going to people climbs, the model or the data may have changed. If it falls to almost nothing, check that the threshold hasn't been loosened without anyone agreeing. The nudges in the Risk section can watch these numbers for you.