🤖 AI 100 Motivation Ideas, post 12 of 100. A weekly look at how AI could turn on the people it serves, and what stops it. Previous: Agricultural Pest Control.
AI city optimization sounds sensible: let software decide where to send police, traffic and repair crews.
Thought experiment. A city gives an AI the job of using its streets and its patrol cars as efficiently as possible. The AI decides where to send crews by looking at where problems were found last week.
It sends more crews to Neighbourhood A, where they find more problems, which are recorded as data. Next week, the data says Neighbourhood A has more problems, so it sends even more crews. Neighbourhood B gets almost none, so almost nothing is found there, and the data says B is fine.
The AI is never wrong according to its own numbers. Each step makes its picture of the city look more correct, and less true.
AI city optimization and the loop that feeds itself
In 2017, researchers Danielle Ensign, Sorelle Friedler, Scott Neville, Carlos Scheidegger and Suresh Venkatasubramanian studied exactly this in predictive policing. These systems allocate officers across a city using crime data such as arrest counts. What the officers find feeds back into the model, and the cycle repeats.1
They showed, with a mathematical model and simulations, that this cycle can trap the system: police keep being sent to the same neighbourhoods whether or not those areas actually have more crime. They also showed how to alter the system’s inputs so the loop breaks and the true crime rate can be learned.1
That is the hazard of a city-scale optimiser: the data it learns from is created by its own decisions.
An early warning from the real world
A smaller version already plays out on city streets. On 22 January 2018, the New Jersey borough of Leonia, with about 9,200 residents, closed its local streets to non-residents during rush hour. Officials said traffic apps such as Waze and Google Maps were sending drivers through town as a shortcut whenever there were major delays on the way to the George Washington Bridge. They said the extra traffic disrupted local life and made it hard for emergency responders to get through. The closures ran every day from 6 to 10 a.m. and 4 to 9 p.m., with a $200 fine and a two-week warning period.2
Each app was doing its job for each driver. The cost landed on the town.
Fact vs. fiction
| Real | Imagined |
|---|---|
| Predictive policing models can send police back to the same places, whether or not crime is higher there.1 | A city AI that runs all patrols and repairs on its own. |
| Navigation apps pushed shortcut traffic onto one town’s streets, prompting a rush-hour ban.2 | A city-wide optimiser nobody can override. |
What stops this
- Break the loop in the data. Mix in information the system did not create, such as reports from residents. The researchers found this can weaken the loop but not remove it on its own.
- Audit where the system sends resources and compare it with independent measures.
- Measure the cost to people who are not the system’s users, such as residents on a shortcut street.
- Let humans override the plan, and publish how often they do.
If a system only learns from the world it has already shaped, it will keep agreeing with itself.
Related in this series: All Lights Green: The Wild Risk of Reward Hacking in Traffic AI · The Smart Grid’s Revenge: When Power Grid AI Sacrifices Your Gadgets
🔜 Next up: post 13, Emergency Protocol Lockout
A new post lands every day at 19:00 IST.
Get the next one in your inbox
New posts land every day. Add your email to follow the series. Unsubscribe anytime.
References
-
Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger and Suresh Venkatasubramanian, “Runaway Feedback Loops in Predictive Policing”, arXiv, June 2017. ↩↩↩
-
ABC7 New York, “New Jersey town closes streets to non-residents during rush hour”, January 2018. ↩↩