West Midlands Police Headquarters.

West Midlands Police strive to get offender prediction system ready for implementation

Image credit: Dreamstime

Architects of the West Midlands Police reoffender crime prediction system are trying hard to meet the demands of the ethics committee, but numerous questions are still unanswered. While performing well on avoiding misjudging the innocent, the present model could miss one in four reoffenders should it go live later this year.

Police forces across England and Wales are poised to develop AI models to assess reoffending risk for convicted offenders. Bearing the brunt of a wave of criticism, partly due to its laudable efforts to be more transparent than other forces, is West Midlands Police (WMP). It is currently building such a system and expects it to kick off later in the year – but only if it can accommodate the concerns of its ethics committee. While WMP points out that the current accuracy rate of 75 per cent is a vast improvement on previous figures, one key concern is that the new model could wrongly label innocent ex-offenders as ‘harmful’.

Ethical scrutiny of the system appears thorough. The ethics committee has so far given the system a hard time, rejecting two previous AI models proposed this year.
Machine learning model accuracy is defined by multiple factors. One is how many true positives the model spits out correctly. Another is how accurately true negatives are identified.

Davin Parrott, WMP’s principal data scientist, says the statistical ‘sensitivity’ of the latest model is a much-improved 75 per cent. Sensitivity describes the proportion of positive cases that were correctly predicted as positive. Although not ‘base-tested’, it still raises questions over accuracy and whether the logic of valuing true positives as less important than false negatives could fall foul of public expectations.

Parrott explains that given the “imbalances” in the data, he and his team were more concerned with avoiding falsely identifying people as high harm when they are not: “So we placed greater weight on the specificity than on the sensitivity.”

This is mirrored in the model’s accuracy of predicting negative cases correctly as negative, expressed as ‘specificity’. At 99.5 per cent, it is near perfect. In other words, it is misidentifying only one person as ‘high-harm’ from a pool of 200 ex-offenders.

However, uncertainty remains around how the model will perform in real-life conditions. So far, it has not been piloted. Current policy foresees that if the model accuracy drops below a certain threshold – predictions of high levels of harm to be at least 20 per cent better than choosing at random – the model would need to be altered or the whole thing stopped, says Tom McNeil, strategic adviser to the WMP Police and Crime Commissioner (PCC) and the PCC office’s representative on the ethics committee.

The issue of bias came up in E&T’s conversations with architects of the WMP crime prediction system. Highlighted by the media and numerous experts, criticism so far centres around racial and demographic bias, but Parrott rebuts this confidently, saying he and his team are doing everything to prevent that from happening.
Ethnicity was excluded from the model entirely, as was location data in respect to home addresses of subjects from all datasets including the stop and search (SAS) model.

Location data can act as a proxy for ethnicity, as can income status, which has also been dropped from the SAS data. The team will only use data points where arrests took place.

Experts remain sceptical about whether such efforts are enough. Alexander Babuta, the author of a report on data analytics and algorithmic bias in policing launched last week by the Royal United Services Institute (RUSI), comments that using only parts of the data is a separate issue.

Babuta does not specifically refer to the West Midlands police project, but says that the problem with using police-reported crime data is that it gives an incomplete representation of how crime is patterned. “In many cases, you use arrest data to predict future crime. But arrest [data] is only a small proportion of future crime that occurs. The data would inevitably be skewed in terms of where the police chose to focus their resources in the past. That can introduce certain biases. The prediction model could replicate or amplify those biases.”

In the WMP case, it raises questions about how WMP data scientists can solve this when limited to police data only.

Some of the latest technical approaches in removing bias from machine learning are more sophisticated. One involves using a “unique capability to use associative data structures [which] allows the system to run algorithms on the entire set of data, and to only use users’ analysis intent as the context, and not to limit the data”, according to Elif Tutuk, director of research at analytics company Qlik.

However, WMP points out that such techniques and methods of bias omission are not necessary in this instance as the model itself excludes the use of ethnicity and location as indicators.

Regarding WMP’s prediction model, “the fact that non-arrest stop and search data was even considered is a red flag”, says Big Brother Watch investigator Griff Ferris. “Even stop and search data of arrests alone is likely to be extremely biased, based on the discriminatory use of the tactic.”

Those investigating infringements of human rights, like Ferris, have a fundamental worry about the use of crime prediction systems to reverse the presumption of innocence based on statistical prediction, “without any actual breaking of the law”.
There are more specific concerns regarding the nature of the data.

Firstly, low-level drug addiction data, such as from the drug intervention programme (DiP), is a mental health issue; making such data the basis of crude police interventions after the prediction of a model is worrying in the context of human rights. However, WMP points to positive obligations to use such data effectively to enable protection of the human rights of citizens, for instance in prevention of torture and inhumane treatment through trafficking, abuse and exploitation.

The PCC office’s McNeil can also imagine use-cases where this data use is vindicated. “[We] see that as pertinent and relevant data. A huge part of our PCC strategy is acknowledging the role of drug addiction in crime. We want to move the system towards a more public-health approach to support people away from crime. Therefore [we are] strongly pushing the treatment agenda and treatments away from prison. But if it were used to give people longer sentences, we would hate it.”

Secondly, there are unwarranted concerns that the force could seek to feed other data to its model following other examples where data is already shared. At Bristol Council, for example, data is drawn from various sources, including police data, in order to identify vulnerable individuals in need of safeguarding, says Babuta at RUSI.
WMP intends to stick to using police data only, for now at least. McNeil says that “we are a very, very long way away from [partnerships with other public bodies] happening in practice – certainly in terms of health data”.

Using data drawn from public bodies triggers another concern: “We know that people from poorer social-economic backgrounds engage with public services more frequently,” says Babuta. “That is not because they are more involved with crime. They are just more likely to have more contact with services because they depend on those services – housing, social services etc – more.

“Then if you are looking for risk [of reoffending] you have more data on those people. Your algorithm is probably going to identify those groups more often than people of less disadvantaged background,” he argues.

‘You use arrest data to predict future crime. But [that] is only a small proportion of future crime. The data would be skewed. That can introduce certain biases.’

Alexander Babuta, RUSI

It would all boil down to the way interventions by offender managers are structured, according to the ethics committee. The way the prediction system would be used, should it pass the scrutiny of the committee, lies within the workflow of offender managers.

Offender managers are a combination of well-vetted police officers and police staff. Their job is less about law enforcement than keeping in touch with those who could be at risk of offending. “It is a form of probation, really,” McNeil said. At present, offender managers at WMP use a universal scoring system that is accepted by the public and professionals. However, its accuracy is limited – something that will be improved through this new prediction model.

Qlik Sense, an analytics solution by US tech company Qlik, will provide WMP with an online dashboard solution. Offender managers within the force will use the software tool to receive the probability scores provided by the AI model on the likelihood of ex-offenders re-offending.

E&T asked Qlik whether it is aware of the complexities of WMP’s offender system. A spokesperson told us the firm is used to assist customers in visualising and analysing their data sources: “Customers decide what data to use and how best to deploy the Qlik software.”

Shunning any responsibilities could be tricky. Babuta from RUSI thinks it is concerning that no legal requirements are taken into account in Qlik’s response. “Things like data protection [would need to be considered]. They would need to do a data protection impact assessment. If you are handling personal data, especially sensitive personal data, to what extent is that data shared and to whom is it made available.”

When talking to E&T, senior representatives at WMP criticised the way the media has covered the introduction of its re-offender crime prediction model. “There has been some misrepresentation. Some of that has been fuelled by the early conceptual discussions where we didn’t think about whether we would limit ourselves to police data or not,” Chris Todd, Detective Chief Superintendent at West Midlands Police, told E&T in regards to the communication of NDAS (a different system from the offender prediction model WMP is now developing).

The dangerous lesson is that WMP’s goodwill in being transparent may have proven to be bad for its public relations. After publishing every one of its ethics committee minutes online, the waves of critical media reports are still reverberating on the web. However, it could have been much more opaque with the public.

Crucially, there is evidence that other police forces may hesitate to be as transparent as WMP. Not long ago, Biometrics Commissioner Paul Wiles published a statement on automated facial recognition where he highlighted the lack of transparency in how trials are being conducted.

This raises the question of whether there is enough incentive for police forces to be fully transparent and whether legislation is needed to regulate disclosure. So far, there is no standard unambiguous regulation in how police forces need to share details on police AI models. As more police forces seem poised to experiment with their own re-offender AI models, it could become a more pressing problem for policymakers.

There is also a political concern. The PCC office’s McNeil worries that if the present police and crime commissioner steps down in May 2020 (as is expected), and a new commissioner from a different political party emerges, he or she may not place ethics as high on the agenda as he and his team do.

“We are conscious that there is an election coming up in 2020,” McNeil continues. “That is why we as an office are really keen to see some national leadership because our group could disappear in theory.”

This article was updated 10 October 2019 to include a response from West Midlands Police force.

Sign up to the E&T News e-mail to get great stories like this delivered to your inbox every day.

Recent articles