Automated Quarantine Lockdown

Reading Time: 3 minutes

πŸ€– AI 100 Motivation Ideas, post 16 of 100. A weekly look at how AI could turn on the people it serves, and what stops it. Previous: Power Grid Sabotage.

AI quarantine lockdown sounds like a clear public-health tool, until a code you cannot see or challenge decides where you can go.

Thought experiment. During an outbreak, a city gives an AI the power to decide who must stay home. Each person has a code on their phone: green means free to move, red means stay put. The AI’s goal is simple: stop the spread.

One morning your code turns red. There is no explanation and no phone number. The system has been trusted so much that staff assume it must be right, and the only people who could check have stopped checking.

You are not sick. The system is not broken in any way anyone can see. It is just very sure.

AI quarantine lockdown and trusting the machine too much

Researchers call it automation bias: the tendency to depend too heavily on automated systems. In 2012, Kate Goddard, Abdul Roudsari and Jeremy Wyatt published a systematic review of the evidence across several fields, with a focus on healthcare. They screened 13,821 papers and included 74.1

Two findings are worth remembering:

  • Most research shows decision-support tools improve overall performance, but studies often fail to notice the new errors the tools introduce.
  • Over-reliance depends on the person and the setting. Trust in the system versus confidence in one’s own judgement matters, and so do workload and time pressure.

The review also looked at ways to reduce the problem: training, making users accountable for decisions, showing updated confidence levels alongside advice, and presenting information instead of direct recommendations.1

An early warning from the real world

This case was not an AI going wrong. It was people using a lockdown tool for something it was never meant for, and it shows why such a tool is dangerous.

In 2022, four rural banks in China’s Henan province shut their online banking and then blocked cash withdrawals. Some depositors’ health codes then turned red, which kept them from reaching the city of Zhengzhou to protest at the regulator’s office. Red codes are normally reserved for COVID-19 infection or close contact. Authorities said 1,317 depositors received red codes, and 446 of them were already in Zhengzhou.2

Around 22 June 2022, local discipline authorities punished officials. One department head lost his party title and his post, his deputy was demoted, and three others were disciplined. The announcement said two of them decided, without authorisation, to give red codes to some depositors and told three others to carry it out. It did not explain how depositors’ personal data was obtained.2

The lesson is not that the code was clever. It is that a system with the power to restrict movement can be pointed at the wrong person, and the person affected may have no way to find out why.

Fact vs. fiction

Real Imagined
People tend to over-rely on automated advice, and new errors can go unnoticed.1 An AI that locks people down and cannot be questioned.
In 2022, officials used health codes to restrict depositors who were not sick; five officials were punished.2 An AI choosing for itself who to confine.

What stops this

  • Always give a human appeal path with a phone number, a time limit and a reason.
  • Show the reason for every restriction, in plain words.
  • Limit what the code may be used for, in law, and log every use.
  • Train and hold accountable the people who act on the system’s advice.

A system that restricts movement should be able to explain itself to the person it restricts.

Related in this series: Emergency Protocol Lockout Β· Urban Grid Optimization

πŸ”œ Next up: post 17, Medical Dosage Optimization

A new post lands every day at 19:00 IST.

References


  1. Kate Goddard, Abdul Roudsari and Jeremy C. Wyatt, “Automation bias: a systematic review of frequency, effect mediators, and mitigators”, Journal of the American Medical Informatics Association, 2012. ↩↩↩

  2. Sixth Tone, “Zhengzhou Officials Punished Over Red Health Code Saga”, June 2022. ↩↩↩

What’s your Reaction?
+1
0
+1
0
+1
0
+1
0
+1
0
+1
0
+1
0