Publishing wider metrics about A&E performance is a good idea: abandoning the 4hr target is not

The current proposal by NHSE to replace the 4hr A&E target is muddled and confusing. It isn't strongly supported by evidence. It will, most likely, make the experience of patients in A&E even worse. Worse still, if things do get worse we won't have the available public data to measure the deterioration. If they paid any attention to history or the experts in emergency medicine, they could have done a much better job.

What is in the proposals?

As part of a review of NHS Access Standards NHSE propose replacing the current A&E target (95% of all patients to leave A&E inside 4hr) with a set of new targets. Those new targets are:

• Time to initial assessment
• Time to treatment for conditions requiring urgent treatment
• Mean time spent by all patients in A&E
• Utilisation of same-day emergency care

It is a little unclear what the last one of these means, but it looks like it will be based on some metric like the proportion of admissions with a zero length of stay (which the report seems to want to encourage as a way of avoiding overnight stays from admission).

What is the rationale for this?

The claimed rationale is to "measure what's most important clinically, and to patients". In support of this, the review suggests the current standard fails on several important grounds:

• It doesn't measure total waiting time
• It doesn't differentiate between severity of condition
• It measures a single point on a complex pathway
• Hospitals' processes, rather than clinical judgement are rushing discharge or admission decisions
• The standard is not well understood by the public

The idea being, at least partially, that the new standards will "drive better outcomes", drive improvement and reduce the opportunity for gaming of current targets.

All that sounds reasonable so what is the problem?

There are multiple problems with the new proposals and they fall into several categories:

• The development ignored the most relevant expert advice
• Previous work (which did use input from experts who knew their stuff) has been ignored
• The evidence used to support the new proposals is weak
• Some of the thinking in the report seems to be naive to the point of utter ignorance about how real A&E departments work
• There is no concrete proposal at all that defines how we will test the new targets or know care has improved
• No attempt at all has been made to use existing data to show what historic performance would look like using the new metrics (and, even worse, the single most obvious improvement in current metrics has been ignored)

I'm going to examine those in turn.

The development ignored the most relevant expert advice

The one group who have in depth expertise on the topic of how A&Es work is the Royal College of Emergency Medicine (RCEM). They were not consulted while the proposals were being developed. They are also very strongly opposed to them. This is from a letter they wrote in january 2019 before the proposals were released:

"Removing the standard will do this and hide the true scale of problems within our health service. The only ones who benefit from this are ministers and NHS managers, and certainly not patients"

This from a statement in march (my highlighting):

"Attempting to make such change at such pace and without due regard to expert evidence is doomed to result in significant unintended consequences. The key issues to be addressed are the systemic ones to increase funding in acute beds, community care and staffing to help make our departments less crowded and improve safety. Moving the goalposts of measurement to make things seemingly look better is certainly not the way forward."

I couldn't put it better myself.

Previous work has been ignored

This point reinforces the first one. NHSE sometimes seem to have the institutional memory of a goldfish. We had a well-developed and well-supported set of new metrics to publish additionally alongside the 4hr target in 2010, authored by the then A&E Tzar Matthew Cooke. The rationale of those was to supplement the target with other indicators that would sharpen the drive to improve and constrain some of the potential of the 4hr target to drive gaming rather than improvement. There was no talk of replacing the 4hr standard. The incoming coalition government abandoned them before they were implemented.

And they seem to have been erased from history. I only recovered the document describing them  from the national archives.

What impressed me in reminding myself of that 10-year old work was how much more coherent it is than the new proposals. The standards are clearer, they are focussed on ways to encourage further improvement and the evidence base is far better.

Some of the indicators proposed in the older plan are similar to some of the new indicators (but the evidence and rationale in the Cooke proposals is uniformly better than the shonky ones in the current proposals). And Cooke proposed using the new metrics alongside the 4hr target not instead of it. [amazingly, I discovered after this was originally written that the Cooke metrics are accessible but are almost completely ignored even though many would be relevant to the new plans]

The new proposals don't even reference the older ones.

The evidence is weak

One might suppose (as the RCEM seems to believe) that the primary purpose of the new proposals isn't to improve the patient experience but to minimise the number of bad headlines resulting from the monthly publication of the performance numbers. This idea explains the content better than the evidence that is used to support the metrics.

One or two of the rationales for the proposals have some merit. But they point towards supplementing the target with wider metrics not replacing it.

Some of the other arguments are shoddy to the point of embarrassing. For example one is that "patients themselves do not identify total time in department as a priority" and "the public are most concerned with time to be seen, and want to know that the sickest patients are prioritised". This is backed by public surveys conducted by Healthwatch. The problem with these views is that patients are not experts in what works inside an A&E department and driving metrics by weak patient survey results is not remotely the same as understanding what A&Es have to do to achieve results that are acceptable to the public. One problem is illustrated by this observation: no public survey on what people want from air travel puts safety in the top 3 concerns. The reason is that most airlines are very, very safe. When all airlines do well on safety, it isn't a factor that distinguishes them or achieves much share of public concern. As soon as that changes, safety rapidly rises to the top of the list and anyone not paying attention faces catastrophic business risk (as Boeing is seeing now with the software problems in its 737 Max). Let's see whether waiting times rise to the top of patient concerns as they continue to deteriorate (they were a big concern in the late 1990s and early 2000s before the 4hr target was introduced).

Even now the majority of patients leave A&E within 4hr. So very long waits are not yet a national concern for most. But the other observation from the Healthwatch survey is a clear indication that the guidelines have been written by people who don't know how A&E works. Yes, the public thinks that those with severe, urgent conditions should be "prioritised". But neither the public nor the guidelines' authors seem to understand that the best way to achieve that is not to slow down the treatment of the less sick (something that seems to have escaped both the last two health secretaries and Simon Stevens who have all said this in public). The volume of activity in A&E is dominated by minors. If they are treated more slowly, the queue in the A&E will be much bigger, impeding all the care in the department. The 4hr target was originally met by recognising that the best way to resolve the problem was to stream the flow inside A&E so that minor patients flow to a highly efficient "see and treat" process that gets them out quickly and keeps the queue small. Sufficient capacity is reserved for the less frequent majors who need more treatment that starts quickly. Streaming, not prioritisation is the key, a lesson the new proposals have forgotten and nobody expects the public to understand (unless we are now proposing universal teaching of queuing theory in primary schools). Using public opinion to support bad operational ideas is harmful and embarrassing.

As is this rationale: "the current standard measures a single point in often very complex patient pathways". I agree. But that would be no reason to abolish the measurement of mortality rates because death represents just one point in an often very complex life.

Some of the thinking in the report seems to be naive to the point of utter ignorance about how real A&E departments work

I've already mentioned the foolishness of assuming that treating minor more slowly makes it easier to treat more serious conditions faster. But the new proposals seem militantly naive on this as were several of the comments by system leaders before the proposals were published.

Not content with that misunderstanding the proposals make another howler because they haven't looked at the data clearly enough. "Hospitals’ processes, rather than clinical judgement, are resulting in admissions or discharge in the period before breach – By moving to a mean, the threshold effect is removed, allowing clinicians to admit when appropriate, rather than at an arbitrary point in time."

While it is true that the data on admission/discharge times shows some evidence of gaming the rationale that we should, therefore, use a completely different target is weak (it is really easy to highlight the degree of gaming by using supplementary metrics as well as the 4hr one as I explained here). Yes too many patients are admitted or discharged in the 15mins before 4hrs is reached. But the dominant problem in A&Es right now isn't gaming, it is long waits for admission. In most departments the majority of >4hr waits are those needing a bed. There is little opportunity to do quick admissions to avoid a 4hr breach because the hospitals usually lack the free beds to achieve quick admissions. And when there is a rush to admit, it isn't because the patient's treatment has been rushed, it is because the patient has been waiting for a bed and the admission process has been expedited. The admissions happening just before the 4hr line are not happening because treatment is being cut short, they are happening because the wait for a bed is being expedited. The proposals fall into the trap of assuming that A&E treatment dominates the time patients spend in A&E when the reality is that delays are dominated by problems of flow in other parts of the hospital. As long as the metrics make this assumption they won't incentivise the whole hospital to improve (as I've explained repeatedly before, A&E performance is not an A&E problem)

There is a significant risk that the new metrics will fossilise naive views about the problems in A&E and will lower the incentive for the whole hospital to improve flow (the target was never an A&E target, it was always a hospital target something that the Cooke proposals were very clear about but the new ones are not).

There is no clear proposal for measuring the success of the new metrics

What can I say? There is no indication whatever about how the trials of the new metrics will be assessed. And the abolition of the existing matric will make it harder to compare performance over time.

Historic data showing the performance on the new metrics has been ignored.

Most of the proposed metrics are already accessible to hospitals and national bodies with access to SUS/HES data. So the proposals could have showed us what performance would look like historically if we reported on both the new and the current metrics. This might have led to some useful refinement of the metrics or even some better evidence for where to set them. The Cooke proposals did that. The current proposals didn't.

The current proposals also claim to want to maximise the incentive for trusts to cut down long waits by using a total time in A&E metric (or the mean time for all patients). This isn't a bad idea as a supplementary metric but the report completely ignores a simple tweak to current published metrics that could radically improve our understanding of exceedingly long waits right now. Scotland, Wales and Northern Ireland all report the number of 12hr waits (end to end) for their A&E systems. England insists on reporting the utterly corrupted 12hr trolley wait target (which measures waits from the decision to admit not the time of arrival). This target is useless as the decision to admit can be postponed until a bed is available even if the patient has already waited far more than 4hr. So the number of reported 12hr trolley waits is ~2,000 in a year but the number of end-to-end waits exceeded 330,000 last year (the number isn't routinely reported and is only published at all because NHS Digital gave up trying to fight off FOI requests to extract it). It doesn't exactly promote confidence in the new targets when a simple tweak to the existing published metrics (giving us a much better insight into long waits) is ignored (perhaps because it would make some very negative headlines about how bad long waits in A&E have become).

So what?

My overall impression of the actual proposals suggests they were written by people who don't understand how A&E works (which is unsurprising given they they didn't consult the RCEM–the biggest group who do know how A&E works).

There is also very little in the proposals to dispel the idea that the ulterior motive isn't just to reduce the number of bad headlines about A&E performance. We have been here before with the unwarranted switch to monthly reporting (I explained why this was a bad idea here).

A better approach would have been to publish additional metrics to sharpen the blunt tool of the 4hr target while retaining the current metric. At least then we would have some way of tracking historic performance versus current performance. The current proposals miss obvious improvements in reporting that could be implemented now (like honest 12hr waits), ignore previous work on what other metrics could be published and fail to offer any guidance on how to judge the effect of trial rollout.

As an analyst who has worked with A&E data I also wonder why there doesn't seem to be any plan to publish the historic performance using the new metrics either nationally or–more usefully–for all the hospitals across England. All the metrics could be derived from existing national returns or the routine patient-level data collected locally. If we had several years of what performance looked like using both the new metrics and the 4hr metric we would be a lot more confident that the new ones were measuring something useful. NHSI has all the historic data. If they can't work out how to do that, I'm available for a reasonable day rate.

Overall, though, we simply can't be confident that the new proposals will drive any improvement in A&E quality or performance. But it looks as though they will deaden public discussion about the manifest problems in our emergency departments. That isn't progress.

The cake is a lie

There is a bigger problem than Brexit that is demolishing the public trust in political promises. The pervasive problem affecting all sides of the Brexit debate is that politicians have been making policy proposals and selling them to the public with neither any analysis nor any admission of the trade-offs they entail. This corrupts and degrades political discourse and leads to a public that rates politicians below gutter journalists in trustworthiness. And the result is that real problems of any sort (never mind Brexit) can never be tackled or solved.

The betrayal of British voters started long before Brexit.

As the British parliament struggled with choices about how to do (or not do) Brexit and voted Theresa May's deal down a second time, radio phone-ins were bombarded by vox populi assertions that the political class had betrayed their decision and were trying to subvert their choice to leave the EU. Pro Brexit politicians have repeatedly asserted that saboteurs are undermining the democratic choice of the British electorate and that any result other than leaving will destroy a generation's faith in the trustworthiness of their politicians. Even a second referendum would be a breach of faith in the democratic system (a bizarre argument on many fronts but particularly hypocritical from some of the Brexit camp who, when they thought they were going to lose, were arguing we might need a second vote if the first vote were close).

I'm a remainer. But the point of what I'm going to say here isn't to argue we should remain. It is to argue that the problem with British democracy and voter's trust in it is far deeper than anything to do with Brexit. And, whatever we end up doing on Brexit, far more worrying. This problem will damage British politics long after Brexit is settled, unless we do something about it.

The problem isn't that MPs are not giving voters what they voted for: the problem is politicians have kept making promises without being honest about the trade-offs of those policies.

Paul Johnson, the boss of the IFS has made this point many times. Here are a few summaries of his view from an article originally published in a Times article in 2017 (link to civilservant.org.uk version):

"It is a fact insufficiently acknowledged that making good public policy is difficult. Really difficult....

Making good public policy is just about the hardest thing there is. If it is being made by people who really do believe that we can have our cake and eat it, then we really are in trouble...

It may be Boris Johnson who is famous for telling us we can have our cake and eat it, but he only made explicit what far too many politicians do implicitly: offering up goodies without mentioning the costs. Yet it is rare indeed that such trade-offs can be avoided."

As many computer gamers will know, The cake is a Lie. We should repeat this phrase every time a politician makes some excellent-sounding policy but fails to explicitly outline the trade-offs. Every promise should be honest and open in admitting the costs as well as the benefits; itemising the losers as well as the winners. Every policy choice has a downside as well as an upside. Except, apparently, in the world of political rhetoric where there are no downsides and where you can always have your cake and eat it too.

This is not a Brexit-specific point. Labour have been guilty of pretending they can fund their extra spending plans by taxing the "rich" when every balanced analysis says everyone's taxes will need to rise to achieve their promises. Sure, abolishing student fees would be popular, but they omit to mention that the biggest beneficiaries would be the wealthy not the poor and that, if they don't stump up billions of extra funding for universities every year, the university attendance rates of the poor would be severely hit. If they ever get into government with this policy, they will suffer the same fate as the ill-advised LibDem promise not to raise fees for which they were duly punished by the electorate. The conservatives promised that austerity could be done without hurting key popular services like the NHS. Look how well that turned out.

Rhetorically policies that are sold as having no downside sound promising but are dishonest. Or perhaps the problem is deeper? Maybe the policy champions have no idea what the downside is because they have never commissioned any actual analysis of the trade-offs. They look purely at how their idea will play in the headlines with no concern for the real effects. This, if anything, is worse than downright dishonesty. As frankfurt argued in his seminal rant On Bullshit:

"...bullshit is a greater enemy of truth than lies are"

The more politicians bullshit us, the less able they are to solve real-world problems. Worse, the more they promise easy solutions that don't work, the less the voters trust them.

The Brexit campaign is characterised by several examples of this from both sides. The remain campaign argued that a Brexit vote would be economic armageddon. Since it clearly wasn't, our trust in them has declined even though the real issue was always what happens when the UK actually leaves not when it voted to leave. The Brexit camp promised that the UK could retain all its privileges in EU trade and that negotiation would be easy and quick. Now negotiations have proved hard and slow they have often doubled-down on the bullshit promise that things would be easy by blaming the problem on saboteurs not their own manifest inattention to the details of international trade negotiation.

But this isn't an argument about whether Brexit is good or isn't good. The problem of bullshit promises has been pervasive in British politics for a long time. And it is those bullshit promises that are sending an exocet into the unwisely inflammable, aluminum clad hull of HMS Political Trust. If the people don't trust politicians to do what they say they will do, that is deadly serious. But the root cause is the bullshit the politicians promise in the first place. Bullshit promises bear no relation to real world problems so can never be fulfilled. You can't have your cake and eat it too even if your policy is pro having cake and pro consuming cake.

The pro-Brexit electorate may well be disgusted that parliament has failed to do what they promised and deliver an easy, quick exit from the EU. But it isn't the violation of that promise that has led to this: it is the fact that such a bullshit promise was made in the first place. Voter disgust in the failure of political promises is a direct product of badly thought through promises not of the failure to deliver them. And those promises are pervasive in current political debate on both sides and on every topic.

We can't solve real-world problems without honest analysis. And we can't buy public support for good policy if we don't clearly admit both the benefits and the costs, the upside and the downside, the winners and the losers of our policy choices. We should stop worrying about the impact on voter trust of failing to deliver political promises when the promises themselves are the problem. Unless voters are given a clear view of the trade-offs inherent in the policy they can have no basis to make a rational choice and no basis to complain if the policy turns out to be undeliverable. You can promise every voter a pink unicorn for their birthday, but is is insane to complain that the problem is a failure of politicians to fulfil their promises when that unicorn doesn't turn up. The problem is making the promise in the first place.

Paul Johnson summarised things well when he said:

"It’s making well-informed, balanced choices and acting on them that’s difficult: to do one thing when it means you can’t do another, to make some people better off at the expense of others, to make the judgment that this is more important than that. Any politician who doesn’t feel that difficulty, who doesn’t feel it viscerally, who doesn’t recognise the costs and risks as well as benefits and opportunities created by their decisions, is likely to be a seriously dangerous politician."

Unfortunately, those dangerous politicians now dominate the political class running the country and the opposition. Even if we somehow resolve Brexit, we will still suffer the consequences of their bullshit in every area where real problems have to be solved.

The only solution is for the public to stop complaining that politicians don't do what they promise and start insisting that all promises come with a clear explanation of their downside. If the politicians can't articulate that, you can't trust anything they say.