To ask Her Majesty’s Government what steps they have taken to assess the full implications of decision-making and prediction by algorithm in the public sector.
My Lords, first, a big thank you to all noble Lords who are taking part in the debate this evening.
Over the past few years we have seen a substantial increase in the adoption of algorithmic decision-making—ADM—and prediction across central and local government. An investigation by the Guardian last year showed that some 140 of 408 councils in the UK are using privately developed algorithmic “risk assessment” tools, particularly to determine eligibility for benefits and to calculate entitlements. Data Justice Lab research in late 2018 showed that 53 out of 96 local authorities and about a quarter of police authorities are now using algorithms for prediction, risk assessment and assistance in decision-making. In particular, we have the Harm Assessment Risk Tool—HART—system used by Durham police to predict reoffending, which was shown by Big Brother Watch to have serious flaws in the way the use of profiling data introduces bias and discrimination and dubious predictions.
Central government use is more opaque, but HMRC, the Ministry of Justice and the DWP are the highest spenders on digital, data and algorithmic services. A key example of ADM use in central government is the DWP’s much-criticised universal credit system, which was designed to be digital by default from the beginning. The Child Poverty Action Group, in its study, Computer Says “No!”, shows that those accessing their online account are not being given adequate explanation as to how their entitlement is calculated.
The UN special rapporteur on extreme poverty and human rights, Philip Alston, looked at our universal credit system a year ago and said in a statement afterwards:
“Government is increasingly automating itself with the use of data and new technology tools, including AI. Evidence shows that the human rights of the poorest and most vulnerable are especially at risk in such contexts. A major issue with the development of new technologies by the UK government is a lack of transparency.”
These issues have been highlighted by Liberty and Big Brother Watch in particular.
Even when not using ADM solely, the impact of an automated decision-making system across an entire population can be immense in terms of potential discrimination, breach of privacy, access to justice and other rights. Last March, the Committee on Standards in Public Life decided to carry out a review of AI in the public sector to understand its implications for the Nolan principles and to examine whether government policy is up to the task of upholding standards as AI is rolled out across our public services. The committee chair, the noble Lord, Lord Evans of Weardale, said on publishing the report this week:
“Demonstrating high standards will help realise the huge potential benefits of AI in public service delivery. However, it is clear that the public need greater reassurance about the use of AI in the public sector. Public sector organisations are not sufficiently transparent about their use of AI and it is too difficult to find out where machine learning is currently being used in government.”
Could the noble Lord extend what he has just asked for by saying that the Minister should also cover those areas where algorithms defeat government policy and the laws of Parliament? I point by way of example to how dating agencies make sure that Hindus of different castes are never brought together. The algorithms make sure that that does not happen. That is wholly contrary to the rules and regulations we have and it is rather important.
My Lords, I take entirely the noble Lord’s point, but there is a big distinction between what the Government can do about the use of algorithms in the public sector and what the private sector should be regulated by. I think that he is calling for regulation in that respect.
All the aspects that I have mentioned are particularly important for algorithms used by the police and the criminal justice system in decision-making processes. The Centre for Data Ethics and Innovation should have an important advisory role in all of this. If we do not act, the Legal Education Foundation advises that we will find ourselves in the same position as the Netherlands, where there was a recent decision that an algorithmic risk assessment tool called SyRI, which was used to detect welfare fraud, breached Article 8 of the European Convention on Human Rights.
There is a problem with double standards here. Government behaviour is in stark contrast to the approach of the ICO’s draft guidance, Explaining Decisions Made with AI, which may meet the point just made by the noble Lord. Last March, when I asked an Oral Question on this subject, the noble Lord, Lord Ashton of Hyde, ended by saying
“Work is going on, but I take the noble Lord’s point that it has to be looked at fairly urgently”.—[Official Report, 14/3/19; col. 1132.]
Where is that urgency? What are we waiting for? Who has to make a decision to act? Where does the accountability lie for getting this right?
My Lords, I congratulate the noble Lord, Lord Clement-Jones, on securing this important debate. It is a topic that I know is close to his heart. I had the privilege of serving on the Select Committee on Artificial Intelligence which he so elegantly and eloquently chaired.
Algorithmic decision-making has enormous potential benefits in the public sector and it is therefore good that we are seeing growing efforts to make use of this technology Indeed, only last month, research was published showing how AI may be useful in making screening for breast cancer more efficient. The health sector has many such examples but algorithmic decision-making is showing potential in other sectors too.
However, the growing use of public sector algorithmic decision-making also brings challenges. When an algorithm is being used to support a decision, it can be unclear who is accountable for the outcome. Who is the front-line decision-maker? Is it the administrator in charge of the introduction of the Al tool, or perhaps the private sector developer? We must make sure that the lines of accountability are always clear. With more complex algorithmic decision-making, it can be unclear why a decision has been made. Indeed, even the public body making the decision may be unable to interrogate the algorithm being used to support it. This threatens to undermine good administration, procedural justice and the right of individuals to redress and challenge. Finally, using past data to drive recommendations and decisions can lead to the replication, entrenchment and even the exacerbation of unfair bias in decision-making against particular groups.
What is at stake? Algorithmic decision-making is a general-purpose technology which can be used in almost every sector. The challenges it brings are diverse and the stakes involved can be very high indeed. At an individual level, algorithms may be used to make decisions about medical diagnosis and treatment, criminal justice, benefits entitlement or immigration. No less important, algorithmic decision-making in the public sector can make a difference to resource allocation and policy decisions, with widespread impacts across society.
My Lords, as we have six minutes, let me also congratulate the noble Lord, Lord Clement-Jones, on having introduced this debate so ably and say what an excellent and, if I might say so, affable chairman he was of the AI committee.
AI and machine learning are on the front line of our lives wherever we look. The centre for disease control in Zhejiang province in China is deploying AI to analyse the genetic composition of the coronavirus. It has shortened a process that used to take many days to 30 minutes. Yet we—human beings—do not know how exactly that outcome was achieved. The same is true of AlphaGo Zero, which famously trained itself to beat the world champion at Go, with no direct human input whatever. That borders on what the noble Baroness, Lady Rock, said. Demis Hassabis, who created the system, said that AlphaGo Zero was so powerful because it was
“no longer constrained by the limits of human knowledge.”
That is a pretty awesome statement.
How, therefore, do we achieve accountability, as the Commons report on algorithms puts it, for systems whose reasoning is opaque to us but that are now massively entwined in our lives? This is a huge dilemma of our times, which goes a long way beyond correcting a few faulty or biased algorithms.
I welcome the Government’s document on AI and the public sector, which recognises the impact of deep learning and the huge issues it raises. California led the world into the digital revolution and looks to be doing the same with regulatory responses. One proposal is for the setting up of public data banks—data utilities—which would set standards for public data and, interestingly, integrate private data accumulated by the digital corporations with public data and create incentives for private companies to transfer private data to public uses. There is an interesting parallel experiment going on in Toronto, with Google’s direct involvement. How far are the Government tracking and seeking to learn from such innovations in different parts of the world? This is a global, ongoing revolution.
My Lords, I thank my noble friend for bringing this subject to our attention. The noble Lord, Lord Giddens, went for the big picture; I will, rather unashamedly, go back to a very small part of it.
Bias in an algorithm is quite clearly there because it is supposed to be there, from what I can make out. When I first thought about the debate, I suddenly thought of a bit of work I did about three years ago with a group called AchieveAbility. It was about recruitment for people in the neurodiverse categories—that is, those with dyslexia, dyspraxia, autism and other conditions of that nature. These people had problems with recruitment. We went through things and discovered that they were having the most problems with the big recruitment processes and the big employers, because they had isometric tests and computers and things and these people did not fit there. The fact is that they processed information differently; for example, they might not want to do something when it came round. This was especially true of junior-level employment. When asked, “Can you do everything at the drop of a hat at a low level?”, these people, if they are being truthful, might say, “No”, or, “I’ll do it badly or slowly.”
The minute you put that down, you are excluded. There may be somewhere smaller where they could explain it. For instance, when asked, “Can you take notes in a meeting?”, they may say, “Not really, because I use a voice-operated computer and if I talk after you talk, it’s going to get a bit confusing.” But somebody else may say, “Oh no, I’m quite happy doing the tea.” In that case, how often will they have to take notes? Probably never. That was the subtext. The minute you dump this series of things in the way of what the person can do, you exclude them. An algorithm—this sort of artificial learning—does not have that input and will potentially compound this problem.
This issue undoubtedly comes under the heading of “reasonable adjustment”, but if people do not know that they have to change the process, they will not do it. People do not know because they do not understand the problem and, probably, do not understand the law. Anybody who has had any form of disability interaction will have, over time, come across this many times. People do it not through wilful acts of discrimination but through ignorance. If you are to use recruitment and selection processes, you have to look at this and build it in. You have to check. What is the Government’s process for so doing? It is a new field and I understand that it is running very fast, but tonight, we are effectively saying, “Put the brakes on. Think about how you use it correctly to achieve the things we have decided we want.”
My Lords, I declare an interest as a board member of the CDEI and a member of the Ada Lovelace Institute’s new Rethinking Data project. I am also a graduate of the AI Select Committee. I am grateful to the noble Lord, Lord Clement-Jones, for this important debate.
Almost all those involved in this sector are aware that there is an urgent need for creative regulation that realises the benefits of artificial intelligence while minimising the risks of harm. I was recently struck by a new book by Brad Smith, the president of Microsoft, entitled Tools and Weapons—that says it all in one phrase. His final sentence is a plea for exactly this kind of creative regulation. He writes:
“Technology innovation is not going to slow down. The work to manage it needs to speed up.”
Noble Lords are right to draw attention to the dangers of unregulated and untested algorithms in public sector decision-making. As we have heard, information on how and where algorithms are used in the public sector is relatively scant. We know that their use is being encouraged by government and that such use is increasing. Some practice is exemplary, while some sectors have the feel of the wild west about them: entrepreneurial, unregulated and unaccountable.
The CDEI is the Government’s own advisory body on AI and ethics, and is committed to addressing and advising on these questions. A significant first task has been to develop an approach founded on clear, high-level ethical principles to which we can all subscribe. The Select Committee called for this principle-centred approach in our call for an AI code, and at the time we suggested five clear principles. The Committee on Standards in Public Life has now affirmed the need for this high-level ethical work and has called for greater clarity on these core principles. I support this call. Only a principled approach can ensure consistency across a broad and diverse range of applications. The debate about those principles takes us to the heart of what it means to be human and of human flourishing in the machine age. But which principles should undergird our work?
My Lords, I also thank the noble Lord, Lord Clement-Jones, for securing this timely and important debate. It is over only the last 20 years that we have seen the meteoric growth of artificial intelligence. When I was discussing this with a friend of mine, his response was: “What, only 20 years? I’ve got socks older than that.” That is probably too much information—I accept that—but there is no doubt that the use of this kind of AI-driven data is still very new.
The use of such technologies was still the stuff of science fiction when I was first elected as a district councillor in the West Midlands. When I was chancellor of Bournemouth University, the impact of data analytics was very apparent to me. It was my privilege in 1996 to present the Bill that established the use of the UK’s first ever DNA database. As vice-president of the British film board for 10 years, I saw the way in which AI simply transformed what we all see on our computer and cinema screens.
I was recently honoured to chair the Westminster Media Forum conference looking at online data regulation. A major theme of the conference was the need to balance—it is a difficult balance—the opportunities provided by these new technologies and the risks of harming the very people this is supposed to help.
The next decade will be like a “Strictly Come Dancing” waltz between democracy and technocracy. There has to be a partnership between government leaders and the tech company executives, with ethics at the centre. As the noble Lord, Lord Clement-Jones, said, one in three councils uses this AI-driven data to make welfare decisions, and at least a quarter of police authorities now use it to make predictions and risk assessments.
There are examples of good practice. I was born and raised in a part of the world universally regarded as paradise. It is called Birmingham—just off the M6 motorway by the gasworks.
I see there is a consensus there, and I am grateful.
I am pleased that all seven local authorities in the West Midlands Combined Authority have appointed a digital champion and co-ordinator, but in other areas evidence is emerging that some of the systems used by councils are unreliable. This is very serious, because these procedures are used to deploy benefit claims, prevent child abuse and even allocate school places.
Concerns have been raised by campaign groups such as Big Brother Watch about privacy and data security, but I am most worried about the Law Society’s concerns. It has highlighted the problems caused by biased data-based profiling of whole inner-city communities when trying to predict reoffending rates and anti-social behaviour. This can cause bias against black and ethnic minority communities. The potential for unconscious bias has to be taken very seriously.
As far as the National Health Service is concerned, accurate data analysis is clearly a valuable tool in serving the needs of patients, but according to a Health Foundation report of only last year, we are not investing in enough NHS data analysts. That surely is counterproductive.
I would like the Minister to answer some questions. Who exactly is responsible for making sure that standards are set and regulated for AI data use in local authorities and the public sector? Will it be Ofcom, as the new internet regulator, the Biometrics Commissioner or the Information Commissioner’s Office? Who will take responsibility? What protection is there in particular to safeguard the data of children and other groups, such as black and ethnic minorities? What are the Government planning to do about facial recognition systems, which are basically inaccurate? That is really quite frightening when you think about it.
AI and data technology are advancing so fast that the Government are essentially reactive, not proactive. Let us face it: Parliament still uses procedures set down in the 18th century. It took the Government three and a half years to pass the Brexit Bill, whereas it can take less than three and a half seconds for somebody to give consent, by the click of a mouse, to their personal data being stored and shared on the world wide web.
7:31 pm
20 of 34 shown
It found that despite the GDPR, the data ethics framework, the OECD principles and the guidelines for using artificial intelligence in the public sector, the Nolan principles of openness, accountability and objectivity are not embedded in AI governance in the public sector, and should be.
The committee’s report presents a number of recommendations to mitigate these risks, including greater transparency by public bodies in the use of algorithms, new guidance to ensure that algorithmic decision-making abides by equalities law, the creation of a single coherent regulatory framework to govern this area, the formation of a body to advise existing regulators on relevant issues, and proper routes of redress for citizens who feel decisions are unfair.
It was clear from the evidence taken by our own AI Select Committee that Article 22 of the GDPR, which deals with automated individual decision-making, including profiling, does not provide sufficient protection for those subject to ADM. It contains a right to explanation provision when an individual has been subject to fully automated decision-making, but few highly significant decisions are fully automated. Often it is used as a decision support; for example, in detecting child abuse. The law should also cover systems where AI is only part of the final decision.
The May 2018 Science and Technology Select Committee report, Algorithms in Decision-Making, made extensive recommendations. It urged the adoption of a legally enforceable right to explanation that would allow citizens to find out how machine learning programs reach decisions that affect them and potentially challenge the results. It also called for algorithms to be added to a ministerial brief and for departments to publicly declare where and how they use them. Subsequently, a report by the Law Society published last June about the use of Al in the criminal justice system expressed concern and recommended measures for oversight, registration and mitigation of risks in the justice system.
Last year, Ministers commissioned the AI adoption review, which was designed to assess the ways that artificial intelligence could be deployed across Whitehall and the wider public sector. Yet the Government are now blocking the full publication of the report and have provided only a heavily redacted version. How, if at all, does the Government’s adoption strategy fit with the publication last June by the Government Digital Service and the Office for Artificial Intelligence of guidance for using artificial intelligence in the public sector, and then in October further guidance on AI procurement derived from work by the World Economic Forum?
We need much greater transparency about current deployment, plans for adoption and compliance mechanisms. In its report last year entitled Decision-making in the Age of the Algorithm, NESTA set out a comprehensive set of principles to inform human/machine interaction for public sector use of algorithmic decision-making which go well beyond the government guidelines. Is it not high time that a Minister was appointed, as was also recommended by the Commons Science and Technology Select Committee, with responsibility for making sure that the Nolan standards are observed for algorithm use in local authorities and the public sector and that those standards are set in terms of design, mandatory bias testing and audit, together with a register for algorithmic systems in use—
I declare an interest as a board member of the Centre for Data Ethics and Innovation. We have spent the last year conducting an in-depth review into the specific issue of bias in algorithmic decision-making. We have looked at this issue in policing and in local government, working with civil society, central government, local authorities and police forces in England and Wales. We found that there is indeed the potential for bias to creep in where algorithmic decision-making is introduced, but we also found a great deal of willingness to identify and address these issues.
The assessment of consequences starts with the public bodies using algorithmic decision-making. They want to use new technology responsibly, but they need the tools and frameworks to do so. The centre developed specific guidance for police forces to help them trial data analytics in a way that considers the potential for bias—as well as other risks—from the outset. The centre is now working with individual forces and the Home Office to refine and trial this guidance, and will be making broader recommendations to the Government at the end of March.
However, self-assessment tools and a focus on algorithmic bias are only part of the answer. There is currently insufficient transparency and centralised knowledge about where high-stakes algorithmic decision-making is taking place across the public sector. This fuels misconceptions, undermines public trust and creates difficulties for central government in setting and implementing standards for the use of data-driven technology, making it more likely that the technology may be used in unethical ways.
The CDEI was pleased to contribute to the recently published report from the Committee on Standards in Public Life’s AI review, which calls for greater openness in the use of algorithmic decision-making in the public sector. It also is right that the report calls for a consistent approach to formal assessment of the consequences of introducing algorithmic decision-making and independent mechanisms of accountability. Developments elsewhere, such as work being done in Canada, show how this may be done.
The CDEI’s new work programme commences on 1 April. It will be proposing a programme of work exploring transparency standards and impact assessment approaches for public sector algorithmic decision-making. This is a complex area. The centre would not recommend new obligations for public bodies lightly. We will work with a range of public bodies to explore possible solutions that will allow us to know where important decisions are being algorithmically supported in the public sector, and consistently and clearly assess the impact of those algorithms.
There is a lot of good work on these issues going on across government. It is important that we all work together to ensure that these efforts deliver the right solutions.
Will the Government pay active and detailed attention to the regulation of facial recognition technology and, again, look to what is happening elsewhere? The EU, for example—with which I believe we used to have some connection—is looking with some urgency at ways of imposing clear limits on such technology to protect the privacy of citizens. There is a variety of cases about this where the Information Commissioner, Elizabeth Denham, has expressed deep concern.
On a more parochial level, noble Lords will probably know about the furore around the use of facial recognition at the King’s Cross development. The cameras installed by the developer at the site incorporated facial recognition technology. Although limited in nature, it had apparently been in use for some while.
The surveillance camera code of practice states:
“There must be as much transparency in the use of a surveillance camera system as possible”.
That is not the world’s most earth-shattering statement, but it is important. The code continues by saying that clear justification must be offered. What procedures are in place across the country for that? I suspect that they are pretty minimal, but this is an awesome new technology. If you look across the world, you can see that authoritarian states have an enormous amount of day-to-day data on everybody. We do not want that situation reproduced here.
The new Centre for Data Ethics and Innovation appears to have a pivotal role in the Government’s thinking. However, there seems to be rather little detail about it so far. What is the timetable? How long will the consultation period last? Will it have regulatory powers? That is pretty important. After all, the digital world moves at a massively fast pace. How will we keep up?
Quite a range of bodies are now concerned with the impact of the digital revolution. I congratulate the Government on that, because it is an achievement. The Turing Institute seems well out in front in terms of coherence and international reputation. What is the Minister’s view of its achievements so far and how do the Government see it meshing with this diversity of other bodies that—quite rightly—have been established?
There is positive stuff here. I am sure that the systems will be clever enough to build in this—or something that addresses this—in future, but not if you do not decide that you have to do it. Since algorithms reinforce themselves, as I understand it, it is quite possible that you will get a barrage of good practice in recruitment that gives you nice answers but does not take this issue into account. You will suddenly have people saying, “Well, we don’t need you for this position, then.” That is 20% of the population you can ignore, or 20% who will have to go round the sides. We really should be looking at this. As we are looking at the public sector here, surely the Government, in their recruitment practices at least, should have something in place to deal with this issue.
I should declare my interests. I am dyslexic. I am the president of the British Dyslexia Association and chairman of a technology company that does the assistive technology, so I have interests here but I also have some knowledge. If you are going to do this and get the best out of it, you do not let it run free. You intervene and you look at things. The noble Lord, Lord Deben, pointed out another area where intervention to stop something that you do not want to happen happening is there. Surely we can hear about the processes in place that will mean that we do not allow the technology simply to go off and create its own logic through not interfering with it. We have to put the brakes on and create some form of direction on this issue. If we do not, we will probably undo the good work we have done in other fields.
Last May the UK Government signed up to the OECD principles on artificial intelligence, along with all other member countries. The CDEI has informally adopted these principles in our own work. They are very powerful and, I believe, need to become our reference point in every piece of work. They are: AI should benefit people and the planet by driving inclusive growth, sustainable development and well-being; AI systems should be designed in a way that respects the rule of law, human rights, democratic values and diversity; AI should be transparent so that people understand AI-based outcomes and can challenge them; AI systems must function in a robust, secure and safe way; and organisations and individuals developing, deploying or operating AI systems should be held accountable for their proper functioning.
In our recent recommendations to the Government on online targeting, the CDEI used the OECD principles as a lens to identify the nature and scale of the ethical problems with how AI is used to shape people’s online experiences. The same principles will flow through our second major report on bias in algorithmic decision-making, as the noble Baroness, Lady Rock, described.
Different parts of the public sector have codes of ethics distinctive to them. Developing patterns of regulation for different sectors will demand the integration of these five central principles with existing ethical codes and statements in, for example, policing, social work or recruitment.
The application of algorithms in the public sector is too wide a set of issues for a single regulator or to be left unregulated. We need core values to be translated into effective regulation, standards and codes of practice. I join others in urging the Government to work with the CDEI and others to clarify and deploy the crucial principles against which the public-centred use of AI is to be assessed, and to expand the efforts to hold public bodies and the Government themselves to account.
I do not think we should be in awe of AI, because ai is also the name of a small three-toed sloth that inhabits the forests of South America. The ai eats tree leaves and makes a high-pitched cry when disturbed.
Seriously, it is vital that there is co-ordination between national government, local authorities, academic research, industry and the media. At the heart of government data policy must be ethics. Regulation must not stifle innovation, but support it. We are at the start of an exciting new decade of 2020 vision, where democracy and technocracy must be in partnership. You cannot shake hands with a clenched fist.