The growing use of AI in policing risks increasing discrimination against minority groups, researchers have warned.
Studies suggest that predictions of future crime hotspots may be based on biased data, which can magnify existing prejudice and create a dangerous feedback loop.
Experts are calling for the use of AI in policing to be reviewed with greater human oversight.
This would involve checking the data that forces use to find and remove any bias, according to Dr Carolyn McKay, an associate professor and researcher in digital justice at the University of Sydney Law School.
"The public functions of state policing ought to be subject to human oversight and scrutiny," she says.
The problem, according to McKay, is that AI systems are trained on data that already contains human bias, and this can lead to self-fulfilling prophecies.
Think of AI as a map-maker asked to draw a map of a city they have never visited. They are locked in a windowless room and can't go outside. All they have are maps that are 50 years old.
Anyone following the new map would find themselves stopping at walls that were knocked down years ago and looking for roads that no longer exist. The map-maker hasn't made anything new. They have only copied the old maps.
Now suppose the map-maker is handed a list of past crimes and where they took place, and asked to shade in where the next ones are likely to happen. Police then patrol the shaded areas.
More crime is recorded there simply because more officers are there to see it. Areas outside the shading appear to have less crime, because fewer people are looking. When the next map is drawn, the same areas are shaded even darker.
So the crime figures are not a full list of crimes. They are partly a record of where police have chosen to look.
The problem is as old as policing, but AI may be speeding it up.
Sarah Brayne, a sociologist at the University of Texas at Austin, spent time inside the Los Angeles Police Department studying how it uses big data. In her research paper, published in the American Sociological Review, she found that the force increasingly used data to predict crime, rather than only to respond to it.
She also found that the data fed into these systems was skewed. Crimes in public places were more likely to be seen and recorded, and police focus was disproportionately on black neighbourhoods. Once this goes into the system, she writes, "the predictions appear impartial".
The models, she says, create "a feedback loop in which they not only predict events such as crime or police contact, but also contribute to their future occurrence."
But that is a view challenged by Dr Dimitrios Kollias, an associate professor in multimodal artificial intelligence at the Queen Mary University of London (QMUL). He says that the technology has moved on to take the bias of historical data into account.
"The old-map analogy is useful for explaining how historical bias can be inherited, but it has its limits," he told BBC News.
"Modern AI systems do not simply retrieve or copy their training examples; they can combine learned information in new ways and generate outputs and solutions that were not explicitly present in those examples."
But those new insights pose a separate risk, Kollias adds.
"As AI systems become more capable of generating novel decisions and acting with greater autonomy, we need to ensure that their behaviour remains safe, reliable and under meaningful human control," he said.
But arguably a more intractable problem is dealing with the sometimes flawed information stored by authorities and which AI systems draw from.
Researchers at New York University said that many of their predictions may rely on what they call "dirty data". These are police records shaped, they say, by corrupt, biased or unlawful practices, including rigged crime figures and arrests of innocent people.
In their study, the researchers found that in nine of the 13 US areas they looked at, police data from periods of unlawful or biased policing was available to train these systems.
The system can't tell that this data is flawed, according to the researchers. It simply treats it as a pattern to repeat. It is as if the map-maker's compass points only to where crime has been recorded before. The map-maker thinks they have found a new crime hotspot, when all they have done is follow the bias of earlier officers.
McKay's answer is to have someone independent check the map before anyone uses it and rub out the lines that are wrong.
But cleaning up dirty data won't be easy. The mistakes aren't a few stray pencil marks. The whole map has been drawn on top of them. The New York University researchers say that technical fixes offered by the companies selling these systems aren't enough unless someone independently checks how the data was produced in the first place.
Because AI can see the world only through the data it is given, it can make serious mistakes which can harm people.
For example, a 2019 study in the journal Science looked at an AI system used in US healthcare to help decide which patients should get extra support. It did this by predicting whose care would cost the most. But there was an inherent bias in the data it drew from. Less was already being spent on black patients than on equally ill white patients, partly because of poorer access to care. By treating higher costs as a sign of greater need, the system unfairly favoured white patients.
It was as if the map-maker had been told to mark the sickest parts of the city but had marked the places where the most money was spent instead. When the researchers changed the system to predict how ill patients were, rather than how much their care would cost, the bias largely disappeared. The share of black patients picked for extra help would have risen from about 18% to 47%.
Kollias acknowledges that AI bias from dirty data is "an important concern".
"But this does not mean that bias is inevitable." he says.
"We can test systems for unfair outcomes, improve the data and objectives they use, and introduce safeguards to reduce these effects."
One improvement would be to provide additional data that provides context. What if the map-maker were given more than a list of crimes? They could also be shown which areas have been neglected - places with broken streetlights, little funding and missing public services. That could show whether the local environment plays a part. Instead of sending police cars to the shaded areas, councils might send money to fix the lighting, support the community and repair the area itself.
Yet even this may not be enough, according to McKay.
"The target environments are still intimately linked with their inhabitants," she says. "They might (continue to) be over-policed and marginalised."
She points to systems which were designed to use information about places rather than people. Even so, critics raised concerns that they still led to over-policing.
It is as if the map-maker promised to stop writing people's names on the map and to mark only the streets and buildings. But those streets were shaded because of years of neglect and heavy policing, so rubbing out the names changes very little.
AI has no beliefs or personality of its own. Like the map-maker in the windowless room, it knows only what it is told. If the person describing the city holds a grudge against one neighbourhood, that grudge ends up on the map.
But ultimately the problem is the human bias that fed the AI. People decide what a suspicious person looks like on CCTV, or what counts as a risky loan, and they bring their own cultural, educational and political biases with them. So perhaps AI can be part of the solution rather than a magnifier of the problem, says Kollias.
"A carefully designed and independently evaluated AI system can make some of these patterns more measurable and visible, giving us an opportunity to identify and reduce them."
But humans too have to play their part. The healthcare study shows that those choices can be checked and changed. That is why researchers argue that people must stay in the loop. Otherwise, like a map-maker inking over old pencil lines, AI risks leaving us all in a windowless room, permanently trapped in yesterday's mistakes.