Business model implications of the NHS ransomware attack

Blame the Business Model

Still, even if the U.S. government is less to blame than Smith insists, nearly two decades of dealing with these security disasters suggests there is a systematic failure happening, and I think it comes back to business models. The fatal flaw of software, beyond the various technical and strategic considerations I outlined above, is that for the first several decades of the industry software was sold for an up-front price, whether that be for a package or a license.

This resulted in problematic incentives and poor decision-making by all sides:

Microsoft is forced to support multiple distinct code bases, which is expensive and difficult and not tied to any monetary incentives (thus, for example, the end of support for Windows XP).
3rd-party vendors are inclined to view a particular version of an operating system as a fixed object: after all, Windows 7 is distinct from Windows XP, which means it is possible to specify that only XP is supported. This is compounded by the fact that 3rd-party vendors have no ongoing monetary incentive to update their software; after all, they have already been paid.
The most problematic impact is on buyers: computers and their associated software are viewed as capital costs, which are paid for once and then depreciated over time as the value of the purchase is realized. In this view ongoing support and security are an additional cost divorced from ongoing value; the only reason to pay is to avoid a future attack, which is impossible to predict both in terms of timing and potential economic harm.

The truth is that software — and thus security — is never finished; it makes no sense, then, that payment is a one-time event.

Ben is on the right track when arguing that the lack of alignment of business models is at the heart of the problem. However, the problem itself is more complex than Ben perhaps realises since many of the "computers" affected by last week's attack are not "computers" in the traditional sense of the word, but medical devices where the software is embedded as part of the whole widget. As such, his proposed solution does not address the problem within the medical context.

Medical devices such as MRI scanners have a lifecycle that does not necessarily match that of the embedded software. For example, Microsoft sold Windows XP until 2008 and supported it until 2014. MRI scanners have a lifecycle of 10-12 years (and I have been on a few older ones myself...), so a scanner built in 2008 using Windows XP would only have its software supported for half of its lifecycle. Now you may ask why on earth a vendor would still be selling products with Windows XP in 2008 but that is due to the high certification costs of medical equipment (no one wants a repeat of the Therac-25 disaster) and the "shelf life" of those machines: new MRI scanners are not designed every year like an iPhone, so companies have the incentive to sell existing models for as long as possible to defray the development costs through a larger number of units sold. Plus, since software is embedded in the machine there is usually no competitive pressure applied to it itself (if you doubt me, check you car entertainment "software" and compare it with your smartphone).

The above is the main reason why Ben's proposed business model of Software As a Service would not work within the context of the NHS attack: Microsoft does not have a commercial relationship with the final acquirer of the machine, but with the vendor, therefore Microsoft's business model is irrelevant to change reality at the proverbial coal face. Even if Microsoft supplied its software under a SaaS model, there are currently no incentives for the device manufacturer to actually incur the certification costs of pushing it to the machines in use (another smartphone example: how many Android smartphone vendors actually update hardware that has already been sold to new versions of the OS?).

How do we align the business models then?

It is possible to align the business models in two different ways. First, by changing the way the way those devices are acquired by hospitals and NHS trusts. Second, by extending the liability of the manufacturer.

An easy way to look at aligning the incentives of different business models would be to lease instead of outright buying the medical devices where possible, thus transforming a capital expense into an operational one (as suggested by Ben). With the right contractual incentives in place (ie that the embedded software must be kept up to date at least in what concerns security patches) it would be possible to create in the device vendors the incentive to keep their machines secure.

A fancier approach is to follow the example I saw in Spain when doing my Ph.D research. At least one regional health board was doing public-private partnership (PPP) contracts with medical equipment providers. In practice this ends up being quite close to Ben's suggestion of SaaS but with hardware instead. For a period of up to say 12 years the public sector was "buying" the availability of imaging equipment in their premises from a vendor.

What made those contracts so interesting is that they included upgrade clauses, so that when a new machine was released, those in service needed to be replaced with the new model, creating in the vendor an obligation to cascade updates to the customer during the whole lifecycle of the contract. I am not aware that they covered software updates although maintenance and servicing were included. Even if they did not, it is conceivable to design future contracts as to include software updates for machines in service.

A different avenue would be to attack the problem from vendor liability perspective, which may also be valid under the current contracts or could spring from raised awareness of the potential liability implications from last week's ransomware attack.

Bottom line

It is possible to conceive contractual and legal solutions that solve (most) of the alignment issue between vendors of medical devices and their clients going forward. What remains to be seen however is what will happen with the existing machines. Surely vendors would be delighted to replace them all, but will the courts consider that they owe a duty of care of sorts towards the users to keep them updated during a certain period of time and to provide access to the source code once they reach "end of life" support? One may dream.

The procurement angle behind the NHS ransomware attack

Last week's (and today's?) ransomware attack on the NHS has a procurement angle to it. It is not the only one and maybe not the most important but procurement mattered nonetheless. Here's why:

1. Vendor lock-in

All compromised computers ran either Windows XP (unsupported) or more recent versions that have not been patched up to fix the vulnerability. There maybe very good reasons for both situations, such as critical systems depending on software available only for Windows XP; medical devices only having drivers for Windows XP; uncertainty if patching would create problems elsewhere.

Those are all valid reasons to have both XP and more modern unpatched computers. They are, however, a good example of what happens when public sector falls prey to vendor lock-in. By throwing its lot with Microsoft and equipment vendors that only support a legacy OS like XP, the NHS is no longer in full control of its decision making. It cannot decide when to patch, what to patch or when to ask another supplier to step in and help solve any problems. The NHS is at the mercy of those external players which do not have their interests aligned. Microsoft wants to sell Windows 10 copies, medical device vendors want to sell more modern machines. Even if medical device vendors have support contracts for their equipment, those may not include software updates (remember Windows XP is 15 years old) or it may not be technically feasible to update the software on those devices to make it compatible with newer versions of Windows.

I have long harped about the risks arising from possible vendor lock in on innovation contracts, but current lock in is surely a lot more important in practice. In addition, last week's attack is also another example of the risks of accepting commercial secrecy as a given in public contracts.

2. The XP support contract

Microsoft stopped Windows XP mainstream support in 2014, and started charging organisations significantly sums of money to keep issuing patches for XP with consultancy/service contracts. The UK Government had a £5.5M contract with Microsoft for a year of support:

“We have made an agreement with the Crown Commercial Service to provide eligible UK public-sector organisations with the ability to download security updates to Windows XP, Office 2003 and Exchange 2003 for one year until 8 April 2015,” said a Microsoft spokesperson.

The UK's custom support agreement was not renewed or extended in April 2015, perhaps due to the costs involved: allegedly Microsoft double the cost from $200/machine to $400/machine and perhaps the UK Government was unable to negotiate a reduction on unit costs. Perhaps they even did not know how many computers in the public sector needed patching, making any negotiation difficult. As a comparison, the US Navy was paying Microsoft $9.1M for support until July 2016 and the extension for 2017 would have increased the cost once more.

From April 2015 onwards, all UK public sector XP machines became proverbial sitting ducks. The magnitude of the problem for the NHS was known since September 2016, when the Motherboard published an article based on a FoI request:

Motherboard has found that at least 42 National Health Service (NHS) trusts in England are still using the Windows XP operating system, with many of them confirming that they no longer receive security updates for the software. Legal experts say that the NHS hospitals may be in breach of data protection regulations.

As it happens the attack hit between 40 and 48 NHS trusts in England and I am sure that is not a coincidence. I suspect there is a significant overlap between those that were still using XP in September 2016 and the trusts hit last week.

I am not a data protection expert but would not be surprised if those Trusts were in violation of their obligations and remain less than assured about personal medical data having been stolen not in this attack but by others exploiting the same vulnerability since it was disclose over a month ago.

3. (Dis)Organisation and competing priorities

The final procurement angle is one of disorganisation and competing priorities. Having a decentralised health system in a day and age where IT is becoming more complex (and expensive) to manage well is a disaster (or two) waiting to happen. Procurement itself is becoming ever more complex - partly because of rules, partly because of bolt on policies, partly because of what is being procured is becoming more complex as well - and those two should dictate a move to a more centralised model.

I am not claiming that centralisation would magically solve the problems (funding would remain an issue) but that at least makes it possible to have somewhere the necessary expertise to do procurement and IT well. As it stands we expect each and all NHS trust to have that (and more) expertise on tap at any given moment. Since complexity is just going to increase (smart medical devices anyone?) the future looks grim.

Finally, time and time again I have seen IT managers pleading for resources to secure networks and/or upgrade/test their equipment. It is always a tough sell as the key decision makers simply do not understand the risks and what is at stake, preferring instead t allocate resources to priorities they understand and where they understand risk/consequences. Maybe on the NHS that will change going forward.

PS: A bit of perspective is important though. Let's not forget that those institutions have been victims of an attack, which may well constitute a criminal activity. If we should not blame victims of other crimes, the same standard should apply to the institutions involved on this.

Links I Liked [Public Procurement]

1. Improving efficiency by building behavioural insights into an innovative NHS procurement portal (research project). Looks very interesting and I look forward to the findings.

2. Public procurement in Europe needs to enter the digital era. Good blogpost by Mara Mendes and Mihaly Fazekas on the need for better procurement data (and standards...) in the EU.

3. World Bank's Key Findings of Benchmarking Public Procurement 2016.

4. Ofcom turns to the Digital Marketplace to speed digital transformation. Fascinating example of in house procurement.

5. Welsh Government spends £100k over three years on Twitter account management. But hey, it's a "reputable and reliable service from a company that has been through the Official Journal of the European Union procurement process." Whatever that means.

 

Links I Liked [Public Procurement]

1. Evaluation of tenders after the expiry of their validity does not annul tender for EU public contracts (T-553/13). Albert comments on case T-553/13.

2. UK Government spent 27.1% of procurement spend with SMEs. Full dataset here. It is all on the way how you count it: only 10.9% was spent directly with SMEs, the rest came via supply chain arrangements. It makes as much sense as the claims by multinationals that their employees pay a lot of tax on the countries where they operate.

3. UnitingCare, NHS Provider Consortium, Folds and Walks Away from Cambridge Contract. Ohm dear.

4. Court of Appeal overturns conviction of former Portuguese Education Minister in procurement case (Portuguese only). Long story short, Maria de Lurdes Rodrigues awarded €220k worth of services (including legal) to the brother of another Minister. The Audit Court considered the contract illegal and she was found guilty in first instance. However, the Court of Appeal repealed the decision, interpreting the then existing law (Decree-Law 197/99) with a law yet to come into force at the time (Public Contracts Code). This is a great example of why legal services should be subject to the same procurement rules as all other services.

As for the fact the Court of Appeal decided to base its decision on a non-existent law at the time, well maybe they did not even notice the existence of Directive 2004/18 and that Directives generate indirect effect.

5. Corruption And 'Tenderpreneurs' Bring Kenya's Economy To Its Knees. Great writeup by Forbes on corruption in Kenya. Speaking of corruption and transparency.

Links I Liked [Public Procurement]

1. A new PPP interview is up, this time with Piotr Bogdanowicz from Warsaw University. As with the previous episode we discussed at length the issues surrounding cross-border interest in contracts not covered by the Directives. Previous episodes can be found here and also on iTunes.

2. The Welsh NHS is still using Windows XP. It is not as if they did not have time to upgrade the systems...Does the Welsh NHS have a licensing agreement with Microsoft for new security patches? Central Government decided not to renew its in April.

3. Is smart ticketing innovative procurement? In this day and age I do not think so, although some are still hiding failure behind the idea that it is.

4. The cloud is coming to local councils. There is plenty of scope for digitalisation of services inside local councils, although I am not so sure if local mandarins will appreciate the loss of power, control and budget associated with the move.

5. Are procurement "best practices" just..."practice"? I have long held that view and that proper applied research needs to be done to separate the wheat from the chaff. But that requires knowledge, costs time and money and it is so much easy just to parrot a few ideas with the "best practice" label slapped on them.

Links I Liked [Public Procurement]

1. For profit prisons are big...and a big problem in the US. The issue of privatisation of prisons was the main topic of the PPP podcast #4 with Amy Ludlow which will go up later this week.

2. American defense procurement is in a mess. Allegedly.

3. There is room for improvement on the Care Quality Commission procurement practices. An audit on two contracts from 2013 showed a number of problems. More like these (audits), please. Plus, thankfully care service contracts below €750,000 have been taken out of the main procurement rules. I am sure the problems detected are due to the (old) rules only and that now, free from the shackles of those EU rules, procurement of care services will be excellent.

4. Repeat with me: paper trail, paper trail, paper trail...is fundamental in public procurement. In Geodesign Barriers Limited v The Environment Agency [2015] EWHC 1121, the court found that the Environment Agency was unable to keep proper record keeping of decision making meetings. This is another personal bugbear of me as no contracting authority would even conceive not having a proper audit trail for any sort of administrative procedure, let alone a procurement one.

Paragraph 25 of Judge Coulson's decision says it all: "As I observed in my brief oral judgment at the end of the hearing, the absence of a contemporaneous Tender Evaluation Report of any kind in this case raises a significant question mark as to the transparency and clarity of the procurement exercise. It gives rise to a whole host of questions. For example; how can any of the tenderers be certain that there has been a fair and transparent process if the documentation relating to that process is a miscellaneous collection of manuscript notes, some written on the back of an old notebook, and some subsequent documents produced for the debriefing/feedback exercise? Furthermore, how could that latter category of documents have even been prepared, if there were no contemporaneous documents recording the results of the evaluation? Take for example the comparison document which shows that the scores awarded to the claimant and Inero, in respect of the second stage technical questions, were the same. How could the writer of that document (whoever they were) have been sure that the scores were indeed the same, if there were no contemporaneous record of the scores actually awarded? How was the detail in that debriefing/feedback document prepared if there was nothing on which it could have been based?"

Oh dear. And I thought that in the top 1% of authorities procurement practice tended to be good.

5. Can devolved procurement work for the NHS? I remain fairly skeptical of devolving procurement powers/responsibilitities for ever smaller entities. Procurement is getting progressively more complex and difficult, in consequence keeping skills up to date is becoming increasingly more difficult as well.