Thursday, 21 April 2016

Gresham's Law works in health policy: bullshit squeezes out honest truthful analysis

Gresham's law is an economic idea that states that counterfeit money drives real money out of circulation. The same thing seems to happen in healthcare policy: good analysis of problems and good policy is squeezed out of the debate by bad analysis and bad policy that sounds good but won't work. This tendency has to be fought vigorously or it will become impossible to improve the NHS except by accident.
"Bullshit is a greater enemy of truth than lies are…" Tim Harford
Gresham's law is a very old principle in economics (it has been known since Aristophanes) which states that bad currency drives good currency out of circulation. I won't say more about the economic mechanisms here as this is an article about health policy where I've noticed that the same sort of problem appears to be happening there.

In short, dumb analysis of what the problems in the NHS are is starting to dominate intelligent, reliable analysis in debate and in policy making; dumb policies are driving out the good ideas that might make things better.
There are a number of reasons for this problem. One was dissected masterfully by Tim Harford in a recent FT article:
This is the real tragedy. It’s not that politicians spin things their way — of course they do. That is politics. It’s that politicians have grown so used to misusing numbers as weapons that they have forgotten that used properly, they are tools.
But we should not rush to blame politicians for dumb analysis of problems in the NHS. Many in the commentariat and many workers in the NHS are keen to shortcircuit good focused analysis and substitute attractive but dumb policy ideas.
There are, unfortunately, a large number of plausible analyses of key problems are that are just wrong. And, if you leap from bad analysis to policy, an equal number of solutions that sound good but will simply waste time and resources because they won’t work. (I would say obviously won’t work but that is apparent only to those who do the detailed, dirty work of actual analysis and the bad ideas are often attractive only because nobody has stopped to do any actual analysis or because nobody has paid any attention to the analysis that has been done). Leaping from symptoms to treatment with no intervening effort on generating a correct diagnosis is bad in medicine and just as bad in management.
One major cause of the Gresham effect here is that many proposed solutions sound good. So they take up space in newspaper headlines and discussion and thereby exclude the more nuanced solutions that require some explanation of why things are a bit more complicated than that. Hence Jeremy Hunt’s repeated assertion that death rates are higher at the weekend. It is a nice, simple idea that sounds plausible and backs up one of his favourite policies: a 7-day NHS. But the reality is the analysis is complex; we probably can’t be certain that the mortality really is worse at the weekend; and we certainly don’t know what causes it even if it is true. So using it as a crutch to support an attractive policy (who doesn’t want the NHS to work the same at weekends?) is deeply misleading. It is particularly misleading because even if we need an NHS that works the same way at the weekend it is far from obvious that changing doctor’s contracts will make any difference. We have plenty of operational evidence that the NHS doesn't work well at weekends but it doesn't point the finger at medical staff as the key problem.
Another zombie idea that wasted space in policy and newspaper headlines was the idea that problems in A&E were caused by changes in the GP out-of-hours contract. Superficially it looked like the numbers attending A&E grew strongly after the contract was changed. But that was coincidence: England started counting attendance at minor injury units (and the number of such units grew rapidly) around the same time as the GP contract was altered. Core attendance at major A&Es (which is where all the problems with treatment speed are) didn’t change from its long term trend. And those who know the statistics also pointed out that few people attend A&E at night; volume is far higher during the day and peaks when GPs are still open.
In fact policy about A&E is littered with dumb ideas that simply can’t be reconciled with the actual data. The idea that A&E performance is declining because of too many people turning up is attractive. So there are repeated discussions about policies to respond to this: diverting patients somewhere else; massively increasing staffing in A&E; putting GPs at the front door… They all sound like they might do something. But every dataset we have says the performance problems have nothing to do with volume. In fact the best analysis says the biggest cause of slow A&Es is nothing to do with the A&E department at all: it is about the inability of hospital wards to accommodate the flow from A&E admissions. Spend all you want on the other policy ideas, but, if you ignore that bottleneck, you are wasting your money. Sadly, every time A&E performance deteriorates, we get a torrent of bullshit policies and almost no commentary that tries to identify (or points out that we have already identified) the most important problem and can do something about it.
One of the most important areas where bad ideas squeeze good ones from the arena is money. The NHS as a whole could probably use more money and probably should get it. Many parts of the system have been campaigning to get more of the budget for their activity. GPs complain that their share of the NHS budget has been falling (and then describe this as "cuts" when it isn't). They claim they are swamped by patient demand and can't cope without vastly more investment. The problem here typifies the way bad ideas squeeze out good ones. The bad idea is that all the problems are caused by lack of (or will be solved by more) money. Nothing else matters. There is therefore almost no discussion of whether anything other than an increased budget could make the life of a GP better or help the GP do a better job for patients.

Yet there are concrete examples that show GPs who pay attention to how they match their capacity to the things patients actually want (the demand) can dramatically lower their workload at the same time as improving patient satisfaction. Flexible attitudes to how patients needs are met and wider use of modern technology can more than fill the perceived gap in capacity that GPs campaign about. But this gets almost no attention in the debate as operational ideas that work are squeezed from the arena by demands for more money.

The idea that the only problem is money is insidiously dangerous across the whole NHS. I'm sure the system could do better with more. But to focus on campaigning for more money and forget all the other things that could be done to improve things is disastrous for several reasons. One is that getting more money is unrealistic in the short term; we should be seeking improvements that can make a difference right now. Another is that getting more money without fixing some of the current problems is a likely to guarantee that the money will deliver far less benefit than expected should it ever arrive. If we lack good management systems that create an awareness of where the real problems are, we will spend extra money on things that don't address the problems and have little impact on the actual problem. A belief that money is the only problem pushes out any thinking on the problems we could fix right now without any extra money and prevents us acting now so we spend any future money on the areas that will yield the largest benefits.

Another attractively populist idea is encapsulated in the slogan "more resources to the front line". It is attractive because it makes a good slogan. It feeds the popular myth that bureaucracy consumes too much money for no useful purpose. It panders to the idea that every problem is solved by having more front-line staff. Sadly every analysis suggests the NHS is extraordinarily undermanaged (see my comments here). While it is possible to be too bureaucratic and undermanaged at the same time, cutting the budget for management is not exactly an effective response. One of the biggest problems in the NHS is a failure to coordinate care and that is a management and information problem that gets harder not easier when you have more medical staff to coordinate. And not just across organisations but inside them. In many hospitals settings the biggest failures in both quality and productivity come because the activity of different people is not well coordinated. This affects how we discharge patients in a timely way; it damages the throughput of operating theatres; it hurts patients because their medication is screwed up; it guarantees long waits in A&E because it is hard to find free beds (we don't coordinate the discharge process with the demand pattern for emergency beds).

So what?

The battle against statistical bullshit and Gresham's law must be fought. The more bad ideas are allowed to dominate debate and policy making, the less improvement will actually happen in the NHS.

Part of the problem would be addressed if the NHS collected better data about what actually happens on the shop floor. Too much of the data currently collected is focussed on top-down performance management rather than identifying and fixing operational problems. The central management style that demands ever more performance reporting (as criticised in this excellent rant by Nigel Edwards) drives out intelligent thinking about the root causes of problems (not least because it consumes so much of the scarce management time available for the operational managers who should be problem solving). Worse, the senior management of hospitals have a worrying tendency to collect data just for performance reporting while neglecting to collect the data they should acquire so they can understand the causes of their operational problems.

Even when we do collect useful information we tend to collect it slowly and make shamefully little use of it to derive operational insights. Patient-level data on A&E performance, for example, has had almost no influence on where money is directed in attempts to solve the persistent decline in A&E performance (see my argument here). We need to make more use of the big datasets for improvement and we need better tools to enable managers to get to those insights more quickly.

Having good analysis of the problem isn't enough. We also need to communicate those answers in ways that actually influence people. Data isn't convincing by itself: it needs to be turned into a message that works for the different audiences that can make a difference to what gets done. Partly this is about paying attention to how data is communicated: good data visualisation is an often neglected first step. But we also need to tell convincing stories. Bad ideas propagate not just because they are often unchallenged but because they are encapsulated in convincing, plausible stories. Gresham's law applies because the bad ideas sound more plausible than the good ones and attract more attention in the commentariat, the policy makers and the operational managers. Counteracting this with analysis isn't enough: we need better stories about what works as well.

The fight against Gresham's law must be fought. If it isn't, the NHS will continue to waste effort on initiatives that won't help it improve. Even if it eventually gets more money, much of that will be wasted because it won't be focussed on addressing the real bottlenecks to better performance. Britain's most loved public institution can't afford that.

Friday, 8 April 2016

The debate on BREXIT illustrates how little evidence determines major public policy decisions

People don't decide their position on BREXIT by looking at the numbers: they choose their position and then seek numbers to justify it. Most of the numbers quoted on either side can't be trusted. This is corrupting public discourse by damaging the credibility of numbers that do tell a clear story.

The ferocity of debate in politics is often inversely proportional to the amount of actual hard evidence available on the topic. Whether the UK should stay in the EU is a typical example. Many decide they don't like the loss of freedom required when you are a member of the club and call for exit; others, like me, decide that the compromises required and the bureaucratic overhang of membership are worth it because the gains from collaboration are worth it. Some choices are more atavistic: many presume (probably incorrectly) that uncontrolled immigration is caused by EU membership (and they also believe the populist myth that immigrants are the cause of many other problems in society). Neither side reaches their conclusion because they have done some calculations: the conclusions come from deep emotional value choices not statistics. But the debate pretends otherwise and dredges up volumes of statistics to confirm the emotionally reached decision.

The numbers thrown around in the debate are proxies for that emotional choice. Most people can't admit that they didn't reason their way to their choice and they seek what look like the rational arguments that got them there. But this process suffers from all of the cognitive biases that so beset much thinking (especially confirmation bias where people are more likely to believe numbers supporting the position they already hold). People seek the numbers that confirm their side of the debate.

Nothing illustrates this better than the meme of European bureaucracy. Even EU supporters often complain about bureaucracy and many even quoted this tweet as an example:

EU cabbage bureaucracy tweet.jpg

Luckily for rational thinkers the BBC's More or Less programme (see this BBC article) decided to test the assertion. Their conclusion: there are no EU regulations specifically about cabbage sales and the 26,911 number originated in a complaint in the post-war USA about government regulations (and was probably mythical even then). Amazingly the same number of words has been repeated by many other objectors to bureaucracy without ever being validated in even the most cursory way against real documents.

The meme of EU bureaucracy is so strong that even pro-EU campaigners didn't think to challenge the basic facts in the assertion.

My main point, though, is about the harder economic numbers bandied about by the two sides and the extent to which they corrupt public discourse when we are debating significant issues. Both sides in the debate, for example, agree that leaving the EU would be disruptive. I agree: that is one of the few things that is clear. But there is a lot of disagreement about how disruptive leaving would be and even more on the economic benefits of staying or leaving. The leave campaign tend to assume that new trade treaties would be easy to negotiate quickly (not that there is any evidence to support this) and they assume that the benefits of an independent UK would be large when freed from the dead hand of EU regulation. The stay camp assume that renegotiation is a long and costly process and that many current jobs in the UK that depend on EU trade would be lost.

Both sides quote specific estimates for the economic benefit of their position. This is where the problem comes. Although specific numbers are quoted, the estimates have no credibility. John Kay, the economist and FT columnist addressed some of the problems with these kinds of estimates in a column in 2011. His concerns, though originally made about the economic rationale for big government projects, apply equally to any analysis of the case for staying or leaving:

...Because so many inputs to the analysis are invented, they can be chosen with a view to the desired result...

...The only information exercises such as these convey is the limits of the imagination of the people who have undertaken them….

...Yet the mistaken belief persists that these procedures provide an objective basis for decision making...

...We do great damage by claiming to know things that are not known, by asserting certainty in the face of uncertainty and ambiguity, and by attaching a veneer of rationality to decisions that have in fact been made on other, rarely articulated, grounds. The paradoxical result is all too obvious. The public sector and large bureaucratic organisations appear as paragons of good decision making process and exemplars of bad decisions.

I would add an extra concern. The specific nature of the forecasts made by each side damage the credibility of almost any analysis intended to illuminate a major public decision. This is bad because, for example, when we have to decide how much to spend to mitigate global warming, the numbers we are shown will have no credibility and we will, most likely, make poor decisions as a result. Sometimes the numbers do point one way on a major decision. If we pollute public discourse with spurious, over-precise analysis, we will undermine our ability to make good decisions when the evidence is clear.

When deciding whether to stay in the EU or to invest in major infrastructure projects we can never have precise estimates of the costs and benefits. We should not deny the uncertainty by pretending that we do. We can't decide on whether to stay in the EU by purely objective analysis: the uncertainties are too large. We should admit that our choice is based on values not numbers. I believe that international cooperation is a better way to conduct our affairs than standing independent and alone, despite the cost in bureaucracy. I don't pretend I can prove it with economic models.

But, if you care about honesty in public debate, there is something you can do whatever position you hold on the EU. Donate to Full Fact's campaign to fact-check the debate. They don't care how you vote but they do care whether the debate is conducted honestly.