The legal rule that computers are presumed to be operating correctly – unforeseen and unjust consequences

In this briefing note, we discuss the legal presumption that computers are operating correctly – a topic previously covered on Bentham’s Gaze, particularly in relation to the Post Office Horizon Scandal but that is also relevant to other areas like payment disputes. The briefing note is also available in PDF format, where it includes more detailed citations.

Overview

In England and Wales, courts consider computers, as a matter of law, to have been working correctly unless there is evidence to the contrary. Therefore, evidence produced by computers is treated as reliable unless other evidence suggests otherwise. This way of handling evidence is known as a ‘rebuttable presumption’. A court will treat a computer as if it is working perfectly unless someone can show why that is not the case.

This presumption poses a challenge to those who dispute evidence produced by a computer system. Frequently the challenge is insurmountable, particularly where a substantial institution operates the system.

The Post Office Horizon scandal clearly exposes the problem and the harm that may result. From 1999, the Post Office prosecuted hundreds of postmasters and Post Office employees for theft and fraud based on evidence produced by the Horizon computer system showing shortfalls in their branch accounts. In those prosecutions, the Post Office relied on the presumption that computers were operating correctly.

Hundreds of postmasters and others were convicted, sentenced to terms of imprisonment, fined, or had their property confiscated. This clearly demonstrated that the Law Commission’s assertion that ‘such a regime would work fairly’ was flawed.

In the December 2019 judgment in the group litigation Bates v The Post Office Ltd (No 6: Horizon Issues) Rev 1, Mr Justice Fraser concluded that it was possible that software errors in Horizon could have caused apparent shortfalls in branch accounts, rather than these being due to theft or fraud. Following this judgement, the Criminal Cases Review Commission referred an unprecedented number of convictions, based upon the supposed shortfalls in the Horizon accounts, to the Court of Appeal. Appeal courts have quashed more than 70 convictions at the time of writing. There will be many more appeals and many more convictions quashed in what is likely the largest miscarriage of justice in British history.

Were it not for the group litigation, the fundamental unreliability of the software in the Post Office’s Horizon computer system would not have been revealed, as previous challenges to Horizon’s correctness were unable to rebut the presumption of reliability for computer evidence. The financial risk of bringing legal action deterred other challenges. Similar issues apply in other situations where the reliability of computer evidence is questioned, such as in payment disputes.

The legal presumption, as applied in practice, has exposed widespread misunderstanding about the nature of computer failures – in particular, the fact that these are almost invariably failures of software. The presumption has been the cause of widespread injustice.

Continue reading The legal rule that computers are presumed to be operating correctly – unforeseen and unjust consequences

US proposes to protect bank customers from Authorised Push Payment fraud

This week, at the US House Financial Services Committee hearing, Representative Stephen F. Lynch announced a draft of the Protecting Consumers From Payment Scams Act. If enacted, this would expand the existing protection for US customers (Regulation E) who have funds transferred out of their account without their consent, to also cover when the customer is tricked into performing the fraudulent transfer themselves. This development is happening in parallel with efforts in the UK and elsewhere to reduce fraud and better protect victims. However, the draft act’s approach is notably different from the UK approach – it’s simpler, gives stronger protection to customers, and shifts liability to the bank receiving fraudulent transfers. In this post, I’ll discuss these differences and what the implications might be.

The type of fraud the proposed law deals with, where criminals coerce victims into making payment under false pretences, is known as Authorised Push Payment (APP) fraud and is a problem worldwide. In the UK, APP fraud is now by far the most common type of payment fraud, with losses of £355 million in the first half of 2021, more than all types of card fraud put together (£261 million).

APP fraud falls outside of existing consumer protection, so victims are commonly held liable for the losses. The effects can be life-changing, with people losing 6-figure sums within minutes. It’s therefore welcome to see moves to better consumer protection. The UK was one of the first to tackle this problem, with a voluntary code of practice being put in place following years of campaigning by consumer rights organisations, particularly Which.

Continue reading US proposes to protect bank customers from Authorised Push Payment fraud

Making sense of EMV card data – how to decode the TLV data format

At the Payment Village in DEFCON 28, I presented a talk about my research in payment system security. While my talks have in the past covered high-level issues or particular security vulnerabilities, for this presentation, I went into depth about the TLV (tag-length-value) data format that anyone researching payment security is going to have to deal with. This format is used for Chip and PIN cards, as specified by the EMV standard, and is present in related standards like contactless and mobile payments. The TLV format used in EMV is also closely related to the ASN.1 format used in HTTPS certificates. There are automated decoders for TLV (the one I wrote is available on EMVLab), but for the purposes of debugging, testing and handling corrupt or incomplete data, it’s sometimes necessary to get your hands dirty and understand the format yourself. In this talk, I show how this can be done.

Rather than the usual PowerPoint, I tried something different for this talk. The slides are an interactive RISE show based on a Juptyer notebook, demonstrating a Python library I wrote to show TLV data-structure decoding. Everything is in my talk’s GitHub repository, and you can experiment with the notebook and view the slides without installing any software through its Binder. I have an accompanying Sway notebook with the reference guides I relied upon for the talk. Do have a try with this material, and I’d welcome your comments on how well (or badly) this approach works.

The DEFCON Payment Village is running again this year in August. If you’ve got something you would like to share with the community, the call for papers is open until 15 July 2021.

Evidence Critical Systems: Designing for Dispute Resolution

On Friday, 39 subpostmasters had their criminal convictions overturned by the Court of Appeal. These individuals ran post office branches and were prosecuted for theft, fraud and false accounting based on evidence from Horizon, the Post Office computer system created by Fujitsu. Horizon’s evidence was asserted to be reliable by the Post Office, who mounted these prosecutions, and was accepted as proof by the courts for decades. It was only through a long and expensive court case that a true record of Horizon’s problems became publicly known, with the judge concluding that it was “not remotely reliable”, and so allowing these successful appeals against conviction.

The 39 quashed convictions are only the tip of the iceberg. More than 900 subpostmasters were prosecuted based on evidence from Horizon, and many more were forced to reimburse the Post Office for losses that might never have existed. It could be the largest miscarriage of justice the UK has ever seen, and at the centre is the Horizon computer system. The causes of this failure are complex, but one of the most critical is that neither the Post Office nor Fujitsu disclosed the information necessary to establish the reliability (or lack thereof) of Horizon to subpostmasters disputing its evidence. Their reasons for not doing so include that it would be expensive to collect the information, that the details of the system are confidential, and disclosing the information would harm their ability to conduct future prosecutions.

The judgment quashing the convictions had harsh words about this failure of disclosure, but this doesn’t get away from the fact that over 900 prosecutions took place before the problem was identified. There could easily have been more. Similar questions have been raised relating to payment disputes: when a customer claims to be the victim of fraud but the bank says it’s the customer’s fault, could a computer failure be the cause? Both the Post Office and banking industry rely on the legal presumption in England and Wales that computers operate correctly. The responsibility for showing otherwise is for the subpostmaster or banking customer.

Continue reading Evidence Critical Systems: Designing for Dispute Resolution

Resolving disputes through computer evidence: lessons from the Post Office Trial

On Monday, the final judgement in the Post Office trial was handed down, finding in favour of the claimants on all counts. The outcome will be of particular interest to the group of 587 claimants who brought the case against Post Office Limited, but the judgement also illustrates problems handling evidence generated by computers that have much broader applicability. I think this trial demonstrates that the way such disputes are resolved is not fit for purpose and that changes are needed in both in how computers generate evidence and how such evidence is reasoned about in litigation.

This case centres around disputes between Post Office Limited and sub-postmasters who operate Post Office branches on its behalf. Post Office Limited supplies these sub-postmasters with products to sell, and the computer accounting system – Horizon – for managing the branch. The claimants contend that shortfalls between the money that was in their branch and what Horizon says result from bugs in Horizon or someone maliciously accessing it. The Post Office instead claims that the shortfalls are real, and it is the responsibility of the sub-postmaster to reimburse the Post Office.

Such disputes have resulted in sub-postmasters being bankrupted, and others have even been jailed because the Post Office contends that evidence produced by Horizon demonstrates fraud by the sub-postmaster. The judgement vindicates the sub-postmasters, concluding that Horizon “was not remotely robust”.

This trial is actually the second in this case, with the prior one also finding in favour of the sub-postmasters – that the contractual terms set by Post Office regarding how they investigate and handle shortfalls are unfair. There would have been at least two more trials, had the parties not settled last week with Post Office Limited offering an apology and £58m in compensation. Of this, the vast majority will go towards legal costs and to the fund which bankrolled the litigation – leaving claimants lucky to get much more than £10k on average. Disappointing, sure, but better than nothing and that is what they could have got had the trials and inevitable appeals continued.

As would be expected for a trial depending on highly technical arguments, expert evidence featured heavily. The Post Office expert took a quantitative approach, presenting a statistical argument that claimant’s losses were implausibly high. This argument went by making a rough approximation as to the total losses of all sub-postmasters resulting from bugs in Horizon. Then, by assuming that these losses were spread over all sub-postmasters equally, losses by the 587 claimants would be no more than £25,000 – far less than the £18.7 million claimed. On this basis, the Post Office said that it is implausible for Horizon bugs to be the cause of the losses, and instead they are the fault of the affected sub-postmasters.

This argument is fundamentally flawed; I said so at the time, as did others. The claimant group was selected specifically as people who thought they were victims of Horizon bugs so it’s quite reasonable to think this group might indeed be disproportionally affected by Horizon bugs. The judge agreed, saying, “The group has a bias, in statistical terms. They plainly cannot be treated, in statistical terms, as though they are a random group of 587 [sub-postmasters]”. This error can be corrected, but the argument becomes circular and a statistical approach adds little new information. As the judgement concludes, “probability theory only takes one so far in this case, and that is not very far”.

Continue reading Resolving disputes through computer evidence: lessons from the Post Office Trial

UK Parliament on protecting consumers from economic crime

On Friday, the UK House of Commons Treasury Committee published their report on the consumer perspective of economic crime. I’ve frequently addressed this topic in my research, as well as here on Bentham’s Gaze, so I’m pleased to see several recommendations of the committee match what myself and colleagues have proposed. In other respects, the report could have gone further, so as well as discussing the positive aspects of the report, I would also like to suggest what more could be done to reduce economic crime and protect its victims.

Irrevocable payments are the wrong default

Transfers between UK bank accounts will generally use the Faster Payment System (FPS), where money will immediately show up in the recipient account. FPS transfers cannot be revoked, even in the case of fraud. This characteristic protects banks because if fraudulently obtained funds leave the banking system, the bank receiving the transfer has no obligation to reimburse the victim.

In contrast, the clearing system for paper cheques permits payments to be revoked for a few days after the funds appeared in the recipient account, should there have been a fraud. This period allows customers to quickly make use of funds they receive, while still giving a window of opportunity for banks and customers to identify and prevent fraud. There’s no reason why this same revocation window could not be applied to fully electronic payment systems like FPS.

In my submissions to consultations on how to prevent push payment scams, I argued that irrevocable payments are the wrong default, and transfers should be possible to reverse in cases of fraud. The same argument applies to consumer-oriented cryptocurrencies like Libra. I’m pleased to see that the Treasury Committee agrees and they have recommended that when a customer sends money to an account for the first time, that transfer be revocable for 24 hours.

Introducing Confirmation of Payee, finally

The banking industry has been planning on launching the Confirmation of Payee system to check if the name of the recipient of a transfer matches what the customer sending money thinks. The committee is clearly frustrated with delays on deploying this system, first promised for September 2018 but since slipped to March 2020. Confirmation of Payee will be a helpful tool for customers to help avoid certain frauds. Still, I’m pleased the committee also recognise it’s limitations and that the “onus will always be on financial firms to develop further methods and technologies to keep up with fraudsters.” It is for this reason that I argued that a bank showing a customer a Confirmation of Payee mismatch should not be a sufficient condition to hold customers liable for fraud, and the push-payment scam reimbursement scheme is wrong to do so. It doesn’t look like the committee is asking for the situation to be changed though.

Continue reading UK Parliament on protecting consumers from economic crime

Forcing phone companies to secure SMS authentication would cause more harm than good

Food-writer and campaigner, Jack Monroe, has become the latest high-profile victim of a SIM-swap scam, losing over £5,000 from both her PayPal and bank accounts to a criminal who intercepted SMS authentication codes. The Payment Services Directive requires that fraud victims get their money back, but banks act slowly and sometimes push the blame onto the victims. When (as I hope it will) the money does eventually get reimbursed, she’s still unlikely to get compensation for any consequential losses, nor for the upset caused. It’s no surprise that this experience has been stressful for Jack, as it would be for most people in her situation.

I am, of course, very sympathetic to victims of SIM-swap fraud and recognise the substantial financial costs, as well as the sense of violation that results. Naturally, fingers are being pointed at the phone companies and followed up with calls for them to do better identity checks before transferring a phone number to a new SIM card. I think this isn’t entirely fair. The real problem is that banks and other payment service providers have outsourced authentication to phone companies, without ensuring that the level of security is appropriate for the sums of money at risk. Banks could have chosen to distribute authentication devices and find a secure way to re-issue ones that are lost. Instead, they have pushed this task to unwitting phone companies, and leave their customers to pick up the pieces when things go wrong, so don’t have an incentive to do better.

More secure SMS authentication

But what if phone companies did do a better job at handing out replacement SIM cards? Maybe the government could push them into doing so, or the phone companies might just get fed up with the bad press. Phone companies could, in principle, set up a process for re-issuing SIM cards which would meet the highest standards of the banking industry. Let’s put aside the issue that SMS was never designed to be secure, and that these processes would put up the cost of phone bills – would it fix the problem? I would argue that it does not. Processes good enough for banking authentication could lock people out of receiving phone calls, and disproportionately harm the most vulnerable members of society.

Continue reading Forcing phone companies to secure SMS authentication would cause more harm than good

Beyond Regulators’ Concerns, Facebook’s Libra Cryptocurrency Faces another Big Challenge: The Risk of Fraud

Facebook has attracted attention through the announcement of their blockchain-based payment network, Libra. This won’t be the first payment system Facebook has launched, but what makes Facebook’s Libra distinctive is that rather than transferring Euros or dollars, the network is designed for a new cryptocurrency, also called Libra. This currency is backed by a reserve of nationally-issued currencies, and so Facebook hopes it will avoid the high volatility of cryptocurrencies like Bitcoin. As a result, Libra won’t be attractive to currency speculators, but Facebook hopes that it will, therefore, be useful for its stated goal – to be a “simple global currency and financial infrastructure that empowers billions of people.”

Reducing currency volatility is only one step towards meeting this goal of scaling cryptocurrencies to billions of users. The Libra blockchain design addresses how the network can maintain the high throughput and low transaction fees needed to compete with existing payment networks like Visa or MasterCard. However, a question that is equally important but as yet unanswered is how Facebook will develop a secure authentication and fraud prevention system that can scale to billions of users while maintaining good usability and low cost.

Facebook designed the Libra network, but in contrast to traditional payment networks, the Libra network is open. Anyone can send transactions through the network, and anyone can write programs (known as “smart contracts”) that control how, and under what conditions, funds can move between Libra accounts. To comply with anti-money-laundering regulations, Know Your Customer (KYC) checks will be performed, but only when Libra enters or leaves the network through exchanges. Transactions moving funds within the network should be accepted if they meet the criteria set out in the applicable smart contract, regardless of who sent them.

The Libra network isn’t even restricted to transactions transferring the Libra currency. Facebook has explicitly designed the Libra blockchain to make it easy for anyone to implement their own currency and benefit from the same technical facilities that Facebook designed for its currency. Other blockchains have tried this. For example, Ethereum has spawned hundreds of special-purpose currencies. But programming a smart contract to implement a new currency is difficult, and errors can be costly. The programming language for smart contracts within the Libra network is designed to help developers avoid some of the most common mistakes.

Facebook’s Libra and Securing the Calibra Wallet

There’s more to setting up an effective currency than just the technology: regulatory compliance, a network of exchanges, and monetary policy are essential. Facebook, through setting up the Libra Association, is focusing its efforts here solely on the Libra currency. The widespread expectation is, therefore, at least initially, the Libra cryptocurrency will be the dominant usage of the network, and most users will send and receive funds through the Calibra wallet smartphone app, developed by a Facebook subsidiary. From the perspective of the vast majority of the world, the Calibra wallet will be synonymous with Facebook’s Libra, and so damage to trust in Calibra will damage the reputation of Libra as a whole.

Continue reading Beyond Regulators’ Concerns, Facebook’s Libra Cryptocurrency Faces another Big Challenge: The Risk of Fraud

Next version of Android might introduce new security risks for online banking, 2FA, and more

Google is preparing new functionality for Android that will allow apps to retrieve and auto-fill security codes from SMS. Last year Apple introduced a similar feature to iOS and macOS, for which we discovered security risks for online banking, two-factor authentication, and other services. Will Google come up with a better design? In this post, we analyse what we know about this feature so far. 


The latest developer beta of Google Play Services (18.7.13 beta) contains code fragments that show a new Android permission to automatically retrieve verification codes from text messages. This feature has not yet been fully implemented, but the available code allows for some analysis and early evaluation for possible security risks, akin to similar risks we demonstrated in 2018 for the Security Code AutoFill feature in iOS and macOS.

Background

It seems that Google is updating the “Autofill Framework”, introduced with Android 8.0 in 2017, to include the new functionality. Previously, this framework’s sole purpose was to support the autofill functionality of password managers in Android apps and websites. The code fragments of this new feature reveal the names and descriptions of the associated system setting and corresponding runtime permission requests, shown below.

A screenshot of an Android phone.
The likely UI of the new setting in Android to enable/disable SMS Code Auto-fill.
The picture of an Android runtime permission request.
The likely UI of the new runtime permission request in Android to deny or allow an application’s access to the SMS Code Auto-fill feature.

Continue reading Next version of Android might introduce new security risks for online banking, 2FA, and more

Confirmation of Payee is coming, but will it protect bank customers from fraud?

The Payment System Regulator (PSR) has just announced that the UK’s six largest banks must check whether the name of the recipient of a transfer matches what the sender thinks. This new feature should help address a security loophole in online payments: the name of the recipient of transfers is ignored, contrary to expectations and unlike cheques. This improved security should make some fraud more difficult, but banks must be prevented from exploiting the change to unfairly shift the liability of the remaining crime to the victims.

The PSR’s target is for checks to be fully implemented by March 2020, somewhat later than their initial promise to Parliament of September 2018 and subsequent target of July 2019. The new proposal, known as Confirmation of Payee, also only covers the six largest banking groups, but this should cover 90% of transfers. Its goal is to defend against criminals who trick victims into transferring funds under the false pretence that the money is going to the victim’s new account, whereas it is really going to the criminal. The losses from such fraud, known as push payment scams, are often life-changing, resulting in misery for the victims.

Checks on the recipient name will make this particular scam harder, so while unlikely to prevent all types of push payment scams they will hopefully force criminals to adopt strategies that are easier to prevent. The risk that consumer representatives and regulators will need to watch out for is that these new security measures could result in victims being unfairly held liable. This scenario is, unfortunately, likely because the voluntary consumer protection code for push payment scams excuses the bank from liability if they show the customer a Confirmation of Payee warning.

Warning fatigue and misaligned incentives

In my response to the consultation over this consumer protection code, I raised the issue of “warning fatigue” – that customers will be shown many irrelevant warnings while they do online banking and this reduces the likelihood that customers will notice important ones. Even Confirmation of Payee warnings will frequently be wrong, such as if the recipient’s bank account is under a different name to what the sender expects. If the two names are very dissimilar, the sender won’t be given more details but if the name entered is close to the name in bank records the sender should be told what the correct one is and asked to compare.

Continue reading Confirmation of Payee is coming, but will it protect bank customers from fraud?