I beg to move,
That this House has considered the legal status of automatic computer-based decisions.
It is a pleasure to serve under your chairmanship, Mr Hollobone. I should apologise to a number of viewers of the debate for advertising its start time as 10.30 am, as opposed to 11 am. I would like to blame Microsoft Outlook, but in fact it was entirely my own fault.
I should also declare my interests. I am the chair of the Institute of Artificial Intelligence, which brings together legislators from around the world to discuss the implications of regulation of artificial intelligence. As Chair of the Business, Energy and Industrial Strategy Committee, I call attention to our currently suspended but still live inquiry into the Post Office Horizon scandal, which I shall refer to today. Lastly, I draw attention to the fact that I used to be employed as a solicitor with a law firm now called Womble Bond Dickinson, which represented the Post Office on the Horizon issue, but confirm that I did not personally act for the Post Office on that issue. I should also thank Paul Marshall, a barrister at Cornerstone Barristers, whose note to me has underpinned much of my contribution today, and Stephen Mason, a research fellow at the University of London’s Institute for Advanced Legal Studies, and his colleagues for their analysis.
The Minister knows that I come to the debate as a technology evangelist—someone who advocates harnessing the potential of technology to modernise our economy and our public services. There is, of course, a much wider debate about the need to update our laws and regulations, and indeed how we run government, due to the technological revolution, but today I will focus on the specific and important issue of how the law of evidence applies to automated computer-based decisions. That has wide-ranging implications across the public and private sector.
The case of the Post Office’s failed private prosecution of more than 1,000 sub-postmasters for accounting errors created by a computer system and not by the sub-postmasters resulted in what looks to be one of the largest miscarriages of justice in our country—a tragedy for our justice system, but also a personal tragedy for the sub-postmasters involved and their families, all stemming from a computer system. I understand that one victim of the Post Office Horizon scandal pleaded guilty to false accounting merely because she had been overwhelmed by the errors and because she could not face the prospect of a jury trial for theft, which was being threatened by the Post Office at the time. Another was wrongly imprisoned when eight weeks pregnant. I stress this point because computer-based decisions can lead not only to not getting a credit card but to untold human suffering in the face of miscarriages of justice. That is just one example.
As the Minister knows, automated computer-based decision making is more and more widespread with every passing year. The problem is that computer systems are not continuously reliable; latent errors can occur frequently. Achieving reliability in a computer system in the first place is difficult; it is even harder to assess and assure that reliability on an ongoing basis. Indeed, artificial intelligence, in its capacity as a general purpose technology across every aspect of our economy, means that the number of decisions coming from software-based systems will increase and increase, both in the private sector and in the delivery of public services. The Minister, of course, understands that. The Centre for Data Ethics and Innovation, which is part of his Department, published a report only last year on algorithmic decision making, which concluded:
“We must ensure decisions can be scrutinised, explained and challenged so that our current laws and frameworks do not lose effectiveness, and indeed can be made more effective over time.”
However, the fact is that, as highlighted in the judgments of Mr Justice Fraser in the Post Office Horizon case of Bates v. Post Office, our laws are dramatically out of date. This is evidenced by the very nature of the dates of the legislation involved for criminal issues, such as the Youth Justice and Criminal Evidence Act 1999. We all recognise, of course, that technology has moved on a great deal since then. That Act repealed section 69(1) of the Police and Criminal Evidence Act 1984, which until its repeal meant that computer-derived documents could not be used as evidence unless it could be shown that, at all material times, the computer was operating properly. At the time, the Law Commission recommended the repeal because of concerns that it was increasingly difficult to meet that threshold—in other words, in 1999 it was increasingly difficult to show that a computer was operating properly at material times.
The change in the law left an absence of formal statutory guidance, resulting in the courts applying to computers the presumption of being properly functioning traditional machines. In practice, that means that a party can rely on the presumption that a computer was operating reliably at all material times—that is to say, that the computer was always right. It is for the objector—in the Post Office case, the sub-postmaster—to prove that the computer was not operating reliably. This is perhaps an obvious point, but in my view that results in an unacceptable imbalance of power.
The owners of computer-based decisions are usually big companies or the state. An advanced computer system is not the same as a factory-floor machine. In contrast, the objectors are employees, customers or citizens who have no real prospect of being able to prove that a computer system owned by a company or the Government was not operating reliably. That has very wide-ranging implications.
If people found it difficult to prove a computer was operating reliably in the early 1990s, we can only imagine how difficult it might be to do that today, not least when machine-learning algorithms come to conclusions for reasons even the computer programmer does not understand. If the Post Office had been required to prove that its computer system was operating reliably, it would not have been able to do so, because we now know that it was not, and sub-postmasters would not have been wrongly imprisoned. The legal presumption that a computer is always right is therefore unsafe and liable to cause significant harm and injustice.
I am not suggesting a return to the pre-1999 approach, but we need to find a new way to manage the risk and update our laws appropriately. As the Centre for Data Ethics and Innovation said in its algorithmic decision-making report,
“we have a window of opportunity to get this right and ensure that these changes serve to promote equality, not to entrench existing biases.”
That is important, because if we are to harness the full potential of technology in our economy, the public need to have confidence in the way in which it is being used, and that there are appropriate rights of redress for those who fall foul of it.
I understand that the Under-Secretary of State for Justice, the hon. Member for Cheltenham (Alex Chalk) has also been engaging with this issue and has referred the matter to the Lord Chief Justice, Lord Burnett of Maldon, in his capacity as chair of the Criminal Procedure Rule Committee. The reason I asked for a debate with the Digital team is my belief that this is broader than an issue merely for the Criminal Procedure Rule Committee. I am therefore calling on the Minister to use the powers of the Government Department responsible for digital and technology issues to refer this matter to the Law Commission for formal consideration. I look forward to hearing his response.