NHSE took the blue pill

In the movie The Matrix, where people live in a fake virtual world, Morpheus offers Neo a choice. He says, “You take the blue pill... the story ends, you wake up in your bed and believe whatever you want to believe.” But taking the red pill means you wake up in the horrifying real world and have to face the consequences.

As far as I can tell NHSE took the blue pill.

They seem committed to ignoring the horrible real world of NHS problems and, as a result, seem incapable of fixing the problems.

For example, look at the draft plan for emergency care.

NHSE’s leaked draft plan demonstrates only one thing: they don’t understand how to fix emergency care. They are still stuck in the Matrix.

There are not enough anglo saxon profanities that can express how much contempt I have for their draft plan.

According to the HSJ it proposed the following actions:

1. Improving vaccination rates and targeted preventative winter virus care.

2. Reducing 111 calls put through to 999 or directed to ED

3. Improve Hear & Treat, See & Treat, and Reduce Avoidable Conveyances

4. Reducing ambulance handover delays

5. Rapid triage at the front door to navigate patients quickly to the right care and avoid admission wherever possible

6. Getting into a hospital bed more quickly for those who need one

7. Improving access to specialist out-of-hospital provision

8. Shorter Length of Stay

9. Reduce discharge delays

10. Standardising and scaling the six core components of neighbourhood health

This is pure blue-pill thinking.

It is so wrong on so many levels that I cannot hurl enough abuse at it.

To see why, let’s start with two things necessary for good, effective strategies. A good diagnosis of the biggest cause of the problem (you can only do this if you take the red pill). And a strong focus on the actions that will tackle that problem. I’ve pointed this out a lot. Nobody in NHSE, it seems, is listening.

The symptom of the problem is that far too many patients are waiting for excessively long times to get through A&E departments. There were 1.7m waits longer than 12hr last year when there should have been fewer than 0.8m waits longer than 4hr in major A&Es. It isn’t even clear that the NHSE leadership know that the problem is in major A&Es (if this speech by Steve Powis, the Medical Director, is any guide–for critical analysis of what he said see this Bluesky thread).

Worse, the leadership have fought against any admission of the most important consequences of those long waits, a large number of excess deaths (the EMJ analysis of this was published in 2022 but the leadership denied the credibility of the results; the ONS update of that analysis published last week suggests the problem is much worse but the presentation of their results omitted relevant data as if to avoid easy comparison with the EMJ estimates). NHSE overdosed on blue pills here.

It isn’t as if the dominant cause of long waits and poor A&E performance has not been analyzed before. The team that originally delivered 98% of waits in under 4hr in the early 2000s said the primary problem was flow through beds. When the post-Lansley performance declined rapidly, several thorough analyses said exactly the same thing in 2015 (eg this). As part of the new strategy for fixing emergency care in January 2023 NHSE repeated these analyses and reached the same result. The primary cause is poor flow through beds. Attendance is irrelevant. (it might be worth noting that NHSE had to be strongarmed into doing or publishing that analysis by No. 10. It wasn’t just that they didn’t volunteer to do a root cause analysis before trying to develop a solution, they resisted doing the analysis. They wanted more blue pills.)

So, in principle, NHSE know these two truths about the causes of the problem:

• Attendance doesn’t matter and isn’t the cause

• The problem is flow through beds

These are the biggies. Some debate often claims other factors are issues, like staffing. But we have analysis of most of them and they are not big contributors to the problem (staffing levels, like attendance, have no relationship to performance across a very wide range of different staffing levels in different departments.)

So how do the 10 points in the draft plan stack up against those two, critical, facts? At least for those who didn’t take too many blue pills.

Five of the points are about reducing attendance. Some are perfectly reasonable actions that would achieve good things (who wants lower vaccination rates?). None are remotely relevant to the problem of improving A&E. None would contribute a female gnat’s testicles of better performance in A&E.

The other points might superficially look like they are dealing with the problem: Reduce ambulance handover delays; speed triage; lower LOS; speed admission to beds; reduce discharge delays. But all are either restatements of the problem or demands for improvements in the metrics that measure the problem. None are actions that might tackle the problem.

Apparently the strategy is to ignore the problem. And that is the entire strategy. The blue pills have done their work.

Fuck me this is bad.

Even if the draft is merely an outline and there is some document with scores of pages of concrete ideas behind each bullet point, this is a truly bad place to start. And the track record of those longer “strategy” documents does not suggest that longer is better. 

Here are the first steps I would take to generate improvement. Take the entire team responsible for writing or commissioning this draft and let them take blue pills so they can drink fake wine in a fake restaurant serving excellent perfectly cooked steaks. Maybe they can write off the cost of the fake reality as part of the ambitious plan to do more AI.

And find another team willing to take the red pills and solve the actual problems that continue to exist in the real world.

What the military can teach the NHS about how to get the right things done in the most effective way

Everyone has a plan until they get punched in the mouth

Mike Tyson, boxer

...no plan of operations extends with any certainty beyond the first contact with the main hostile force

Helmuth von Moltke the Elder, 19th century Chief of the Prussian General Staff

The NHS swings between bouts of centralised power in the hands of ministers and decentralised decision making by its regions and units in repeated attempts to achieve its goals. But it never seems to achieve the goals desired. I suspect its leaders (both politicians and bureaucrats) have never learned an important lesson that many military forces learned the hard way.

The current NHS white paper proposes a great deal of new centralisation of power in the NHS, a major reversal of Blair-era reforms which tried to decentralise decision-making. My best guess as to why this is happening is that ministers feel the NHS is not achieving what they want and the only effective way to do so is to take back power. This is wrong. Though it is far from obvious that decentralisation worked either and the real problem is a fundamentally misguided view of how to get things done in a big organisation that has persisted in the NHS for decades.

Other big organisations have taken a different approach. Many militaries learned the hard way how to achieve their goals and how they did so provides useful lessons for the NHS.

The fundamental idea that governs the NHS is that people should "follow the plan" and do as they are told. They should achieve things by following the how of the plan and are frequently judged on whether they follow the right process to achieve the goal. Metrics are often if not usually process metrics not outcome metrics. This philosophy of management (or culture) has persisted over time despite major changes in the structure of NHS management which has swung from centralised to decentralised and back over the last few decades.

This follow-the-plan philosophy is how many militaries used to exercise command and control in their troops. This was how the British military worked at the start of WW2 until the outcomes forced a major rethink. 

A good illustration of the problem is the campaign in North Africa. Until the decisive defeat of the Afrika Corps at the second battle of El Alamein, the British forces were repeatedly defeated by a vastly inferior German and Italian force (the Allied forces almost always outnumbered the Axis forces in men, tanks, guns equipment and had far fewer persistent shortages of supplies). According to the insightful analysis of Stephen Bungay (in his books Alamein and The Art of Action) this was not–as is often reported–because their leaders were better (though Rommel was an excellent tactician) but because the command and control doctrine of the German army was far superior.

A simple way to understand why is encapsulated succinctly by the Mike Tyson quote and at more length by Moltke: rigid plans are derailed by the messiness of reality which discombobulates the plan as soon as the enemy doesn't act as expected. The British army doctrine at the start of the war demanded that soldiers follow the plan, in detail. Junior officers were disciplined if they deviated from it. Following the orders in detail was the goal. The Germans didn't do that. They knew what Mike Tyson and Moltke knew: the details of the plan fall apart as soon as you meet the enemy. Instead, the Germans issued objectives for all levels of command about the key goals rather than detailed plans about how to achieve the goal. All levels of officers were given wide discretion about the how. They were expected to innovate when they saw the reality on the ground as long as they pursued the key goal they had been set. This led to persistently better decision making in actual battles leading to their inferior forces often winning key engagements with the rigid, hidebound Allied forces (at least until the allies had overwhelming superiority).

German command doctrine (now called Mission Command and widely adopted as a central philosophy by the modern US and British armies) was that a detailed plan is derailed by real events (like getting punched in the face or when the enemy doesn't act as expected in the plan) so tactical innovation in response to reality is far more likely to achieve the goal than blindly following the details of out-of-date orders cooked up by commanders a long way from the action.

But this is pretty much how the NHS works. Goals are not set in terms of outcomes but in terms of following the plan. Do things the way the centre tells you to. And suffer punishment if you deviate from the process, which we will measure. And they won't measure the outcomes because that is far too hard to measure in the short term. The centraising plan will make this worse, not least because it is the equivalent of a frustrated Churchill demanding direct control of the deployment of individual tanks in the desert because his generals were not winning their battles. The NHS has rarely tolerated local innovation that delivers better outcomes and often seems to regard innovation as "rocking the boat" or "deviating from the process".

This is a critical failure. Unless the NHS radically changes its management philosophy and culture from one where following the plan is rewarded to one where local leaders are rewarded for achieving better outcomes by innovating, it will never get better at achieving better outcomes however the overall structure is changed. 

The NHS plan should focus on the management culture not the structure. It should define the broad goals (outcomes) desired by the government or the leadership and leave the front line leaders to find better ways to achieve those goals. And it should radically cut the process ("do things this way") metrics and use more achieve this outcome metrics. And don't forget culture change: culture may reappear and control the results even if the structure is new. A change in management structures will always be trumped by a persistent management culture.

It cannot be said that the military have always remembered this lesson. An analysis of the army's poor performance in the second Gulf War pointed out that Mission Command is not very useful when you have no idea what your mission is (see the introduction to The Good Operation which also has other useful conclusions about public policy that don't seem to have influenced the NHS plan). Perhaps this too is a problem affecting the NHS. Which suggests that the NHS might have been better analysing why it frequently fails to achieve better outcomes before writing a plan that reinforces the very cultural problems that make innovation and flexibility so difficult.

NHS performance can be improved by paying more attention to reliable data and sound analysis

While the NHS and many other public services need more resources this might not be the best path to big improvements in quality and performance. Not least because deploying extra resources in the wrong place might yield only small improvements. Maybe we should start by spending more on getting the data and analysis required to understand where the biggest problems are and what interventions might solve them. And then focus on installing effective performance management throughout the system so improvements happen and stick. This will be expensive, but is demonstrably worth it.

It is almost universally assumed by commentators that the only thing that will improve the performance of the NHS or any other public service is a bigger budget. The bottleneck preventing better performance is austerity. The police need more cops; the NHS needs more doctors and nurses.

But is this true? Is a bigger budget the only way to improve anything? Is there nothing else that we should be doing?

The answer is no. really large improvements in performance are possible even without really big increases in the budget. And we know how because it has been done.

In the early 1990s New York had around 2000 murders a year. It now has fewer than 300, the lowest since records started. Other major crimes show similar reductions. What kicked off the improvement and kept that improvement happening for the next 25 years wasn't a vast increase in police numbers but a management process called Compstat.

The story of Compstat is not well known as lazy journalists have tended to credit the improvement to Broken Windows Theory (see, for example the Wikipedia page on crime in New York or this longer discussion about what really happened). Few accounts even mention the man who developed the Compstat process, Jack Maple, who wrote a book (The Crime Fighter) describing how he developed it and how it works.

What Maple describes ought to be of great interest for other public services like the NHS.

According to Maple The essence of Compstat is built on 4 key principles:

1. Accurate, timely intelligence
2. Rapid deployment
3. Effective tactics
4. Relentless follow-up and assessment

The goal of the process is to reduce crime rather than to pursue proxy metrics like arrest rates or response times. A key part of the process is the weekly meeting of local police commanders where they are not held to account for missing their targets but for not understanding the patterns of crime and not having effective plans to address the crimes in their area. The focus of performance meeting is to develop and share that understanding and to make sure it is being acted on. Commanders who fiddle their numbers rather than tackling their problems are ruthlessly exposed.

This contrasts sharply with how the NHS manages performance. It is worth comparing the NHS process with Compstat.

The NHS shares with Compstat the idea of regular meetings where the performance is reviewed. But what happens in the meetings is very different. In the NHS the meetings are not based on a wide range of insightful metrics that provide insight to why performance is bad but are based on a small number of less reliable headline metrics that provide no insight into the why of performance. And local managers are berated for failing to meet the target and encouraged to promise future performance without being encouraged to show they understand the key causes of poor performance. Neither the central team nor the local managers have a wide range of shared data that helps anyone understand the causes of problems nor do either group have much of a clue about what effective actions to improve performance look like. Since there is no collective understanding of problems there is no sharing of good ways to tackle them. The system reverts to measuring inputs not outputs (more nurses and doctors sounds good but won't help much if we don't know where the problems are).

In short, all the effective habits of Compstat are undermined from the top down in the NHS because there is no focus on understanding why performance is poor.

In a 2015 HSJ article called The way the NHS manages A&E problems is not fit for purpose, Nigel Edwards described this process as "A significant organisational pathology".

The NHS hasn't always been like this. Though it has never been as thorough as Compstat, there was a period in the 2000s where effective actions to improve performance were understood and performance did improve (eg starting in 2002 A&E performance went from ~70% of patients admitted or discharged in 4hr to 98% in a 3 year period). Most of the knowledge from that time seems to have been lost. The Cabinet Office even used studies on what worked in Compstat to help design the way performance management was done (but the 2004 document describing this–see pages 26-27–is buried deep in the national archives and it is doubtful anyone currently in power has read it).

Part of the reason why the NHS has forgotten what it once knew is that running a management process like Compstat is expensive (Maple estimated it might take 5% of the police budget). That is more than the NHS spends on management. Worse the most consistent NHS strategy over time–from both Labour and Conservative governments–has been to cut management so it can devote more resources to the front line. But adding more resources does little for performance if those resources are not well coordinated and deployed.

The Compstat process has worked for more than 25 years in the NYPD and improvements are continuing. But it also worked in cities like New Orleans where the police were far less well funded.

Perhaps the NHS has something to learn. Reforming its shonky management processes might be the best way to improve performance.

Good news: The AI won't kill you; Bad news: it won't solve your problems either.

AIs won't take over the world because they won't be able to work out how to do it. But they won't solve our big problems either. The hype machine behind current AI investment is founded on an unwarranted extrapolation from recent AI successes that doesn't apply to most real world problems. We need to learn where AI can work and where it can't or we will waste money on systems that can't possibly work. 

Scifi and a scientific speculation about Artificial Intelligence (AI) has ignored a very fundamental limitation in how AIs work. This limitation has surprising parallels with why some of the less pleasant political philosophies the world has seen are doomed to fail. But this limitation has been–as far as I can tell–largely missed by serious speculators and fiction writers.

And the failure to understand the limitations of AI has consequences. On one hand there is a fair amount of worry about the dangers. On the other there is a dangerous and naive belief in their ability to solve many difficult problems. If I am right, neither view is justified.

I started thinking about this while reading Tom Chivers' book The AI Does Not Hate You (a really good read that covers a lot of the background and a lot of thinking from the community of nerds who have devoted a lot of time worrying about the consequences of AI). But I was also reading some Karl Popper books I skipped as a student (The Poverty of Historicism and The Open Society and its Enemies). And–somewhat amazingly, since Popper wrote these before AI had ever been invented–it turned out his ideas had some current relevance.

Let me explain.

An introduction: AIs in fiction

There are plenty of SciFi stories involving malevolent AIs. The 1970 saw the movie Colossus: The Forbin Project (where The US basically creates SkyNet 27 years early to run nuclear defence only to find the computer able to outsmart the government and its creators). The Terminator added time travel to a similar scenario (there is nothing new under the sun). The Matrix added virtual reality simulations to a related idea. 

What happened in the worlds created by Isaac Asimov is revealing. His Robot stories created robots with positronic brains and acted at least as intelligently as people. His unconnected series The Foundation Trilogy didn't have any AIs but did have a human psychohistorian who had worked out the laws determining the future path of human history. Asimov's Hari Seldon used "calculators" to solve his equations. But the underlying idea was that, given enough data, calculations could be done that would predict the future of society and, when required, steer it to a better outcome. This idea is very important even though it didn't, when originally written, involve a superintelligent AI.

Later Asimov (very ill-advisedly) retconned The Foundation stories to make them part of the same universe as the Robot stories and make a powerful AI the driver for all the messing with human history. In this case the AI was benevolent. But the same assumptions apply in his optimistic world as they do in the dystopian stories where the AI wants to kill us all or where it accidentally kills us all in pursuit of the goal of making more paperclips.

The fear expressed in many of the stories of rogue AIs (mirrored by the optimism in Asimov) is that, once we have created computers that can process information faster than we can, we will lose control and they will be able to dominate us (in Asimov for beneficial ends; in many other stories, for perverse or malicious ends). 

Why the assumptions are wrong

The underlying assumption in both the optimistic stories and the dystopian ones is that people are both not entirely rational and have a finite capacity for reasoning that can easily be exceeded once a critical mass of computing power can be assembled.

There are two parts to this belief that we can be outdone by a sufficiently powerful computer. One is the belief that rational thought unencumbered with emotion will produce better solutions to problems than human thinking. The other is that sufficiently powerful computers can become powerful enough to outmaneuver people in managing the world (either leading to a utopian paradise or a dystopian slave state depending on the orientation and goals of the AI).

The idea that better solutions will emerge if we purge the world of emotion is attractive to many naive scientists. But it is easy to refute because of the way the human brain works. The brain is, to some extent, compartmentalised with different physical areas dealing with different roles. Vision is controlled by different parts of the brain then smell or hearing. And human emotions, not to simplify too much, can be partly localised in specific parts of the brain. When these are damaged we can see what sort of person emerges. The famous case of Phineas Gage is often quoted as the archetype of what a person is like when their emotions are removed (he suffered a very specific brain injury in an industrial accident). But the emotionless person turns out not to be a super rational problem solver freed from the misleading siren calls of conflicting emotional drives. The emotionless person that emerged after Phineas Gage's injury turned out to be a complete mess incapable of making basic decisions or getting anything done. He was heavily studied,as other people with similar injuries have been. He turned out more like Buridan's Ass than a super rational thinker. It turns out that emotions are a vital part of human thinking and decision making and not a burden that gets in the way of logic as the Vulcans would have us believe. (Though the details of Gage's recovery are often omitted and the case is more complex than often supposed; and Spock is not entirely emotionless–don't @ me). 

AI researcher David Gelertner wrote a whole book explaining that an AI can't hope to emulate human intelligence without incorporating emotions.

But the other part of the problem is more significant for understanding why AIs won't work the way SciFi worries about. 

Many thinkers have speculated is that, when AI gets to a certain threshold of power, its ability to learn grows exponentially and it rapidly outgrows its designers. Hence the worry about taking over the human world. And this possibility has been greatly enhanced by the recent success of AI in surpassing human players at Chess, Go and (apparently) even poker.

We have known that computers could beat us as some board games for a long time. Chequers was basically solved by computers decades ago. But analysts thought for a long time that more complex games would resist the ability of computers to conquer them. Chess is far more complex than chequers (with more possible games than atoms in the known universe–see this Numberphile video for an explanation of those estimates) and some thought that only humans could think well enough to play the game effectively. Then an IBM creation beat the best player in the world (though the original Deep Blue was not an AI, just a very powerful chess computer using expert algorithms and a lot of brute force computing power). Chess was hard because the search-space of possible moves was too large for any brute force searching to find the best possible moves regardless of computer power. Then, very recently, AlphaZero–a learning AI–was taught the rules of chess (but not chess-playing algorithms fine tuned by people to simplify the job of searching all the combinations of moves). And, in a short period, it became the strongest chess playing system in the world, just by playing itself and learning what worked. For a short time many predicted that Go would be resistant to that approach as it has another order of magnitude or two more combinatorial complexity than chess. But it, too, fell to Alpha Go, a similar AI trained with the rules of Go.

Interestingly in both cases it turns out that an AI has found new strategies that people have not thought of and plays the game in new ways that are very effective.

These triumphs were hailed as illustrating the coming AI singularity when a general AI will do the same exponential learning trick and take over the world. Go and Chess were conquered by an AI that improved at an exponential rate once it learned the basic rules of the games.

The reason why this was, to some, unexpected was because their thinking was based on the combinatorial complexity of the games. The barrier to computer progress was, they speculated, the fact that there are too many possible games to enumerate explicitly and therefore only some different sort of intelligence that could think strategically could win. This, some argued, was what distinguished human thinking.

This is bollocks.

There are only vague analogies between the real world and Chess or Go. While they have been thought to have similarities to warfare and can train people in some of the strategy of war, this is mostly untrue. Real warfare is not like playing Go: it is more like playing Go when you don't know how big the board is, have only a vague and often mistaken idea where your opponents pieces are, often don't know where your own pieces are, and  have to make decisions while being jabbed in the face with a broken wine bottle. 

The real distinction between the world of human society and the world of chess (but also between many finite problems in the world of people) is not about combinatorial complexity. The real world is, usually, a lot more complicated and reflexive than Chess or Go. It is more uncertain. It has rules that are less clear or that may not even exist. It has more than two players. And it has, sometimes, no clear way of working out who won or even whether winning has any meaning.

This isn't just extra combinatorial complexity. If Go or chess were played on a bigger board with more complex rules they would add more complexity and that complexity would grow very quickly. But they would be just as susceptible to an AI learning how to play them (it might need another generation of processor or bigger memory chips but the same method would surely eventually yield a successful computer player.)

Not in the real world and not for many typical human problems.

To see why consider how the all conquering game AIs have learned how to play their games of Go and Chess. The games have a finite set of rules that define a valid game or sequence of moves. AIs have an unambiguous way of calculating the score at the end of the game and no doubt about which side won the game. AIs can play games and observe the outcomes. Even if they start with random moves they can learn by observing the outcomes and, therefore, what patterns usually yield victory. They can improve their recognition of those winning patterns by incorporating them in future play and playing further games. They can create new games as fast as they can compute their moves and every game adds to their knowledge of which patterns yield more victories. Learning AIs are, at their heart, pattern recognition engines that can process vast amounts of data. The limitation of their learning in finite games is how fast they can generate more possible games to learn from. Hence, the more computer power the faster they learn. And once they start learning the growth in expertise is exponential until they have exhausted their storage and memory capacity.

But this sort of learning is only possible because the rules are finite and the outcomes certain. While some practical problems are like this (more later), most are not. In the real world both the rules and outcomes may be unclear.

Take, for example, the problem of extending the human life span. Maybe some combination of diet, physical exercise and intellectual challenge and genetic manipulation could extend the human lifespan to a median of 150 years. It isn't impossible. It doesn't violate the laws of biology or physics. What would a benevolent AI have to do to solve that problem? It could suggest a series of genetic interventions to extend the human lifespan. And it could, with some help from us, test those ideas. But it can't quickly learn what works because it has to wait 150 years to find out whether it achieved its goal. And even a full simulation of a functioning human body inside the AI doesn't solve this problem because that simulation can only be built correctly from actual observations about what happens in the real world. The AI can't learn faster than it can observe the outcome of experiments IRL. 

This is worth dwelling on not least because it helps distinguish which tasks might be soluble with a decent learning AI. Problems that are worth tackling will have unambiguous rules or inputs and unambiguous outcomes that can be checked against reality with little uncertainty or ambiguity. So finite games with clear rules are not a problem. Even games involving chance are in this domain even if they involve bluffing, which is why it seems that AIs can play poker. But poker has unambiguous rules and outcomes. The best bluffing strategy is, in principle, learnable even if it involves judging specific characteristics of the opponent (though a good strategy will win more often than a bad one even if played against unknown opponents). There can't be a "perfect" strategy with the amount of randomness involved, but good strategy can be learned.

Consider what sort of medical advances might fall to a learning computer. Some diagnostics (identifying breast cancer from breast scan images, for example) have very clear datasets and fairly clear known outcomes (not 100% certain outcomes, though, as we know from epidemiological analysis that we send far too many suspect cancers for surgery than we should). But we can certainly compare a computer driven classification algorithm for suspected cancers to the work of experienced radiologists and we can check their work with other clinical results. But most medicine is not like that. Consider trying to train an AI to do the job of a GP. The inputs are vague and ambiguous (interactions with patients are often rambling and even the clinical history of the patient may be full of errors). The outputs–a diagnosis and a treatment–are hard to test for correctness and may also, frequently, be wrong. Worse still we may lack any good way to check the diagnoses or treatments to test whether they are correct. So, even if we trained an AI to mimic an existing GP we might not be able to tell whether we had created a Harold Shipman or a Florence Nightingale (OK, she was a nurse and a statistician not a GP, but you get the point). 

In short, there is little hope of training effective AIs when the inputs are unconstrained, vague and variable; or where the outputs cannot be readily verified to be correct or even unambiguously good. We find this a tough problem even for human doctors.

And medicine is a narrow field of human activity. Life is much bigger and more variable. As is society, where individual lives interact in ways that create exponentially (in the proper mathematical sense) more unpredictable interactions. 

So how can AIs learn how to take over the world? There is no way for them to learn how to do so when there are no patterns to observe and learn from.

This touches on a philosophical debate that has had too little impact on either computer science or politics. Karl Popper called  it The Poverty of Historicism. His book was a powerful demolition of the idea that history has predictable, teleological patterns. In particular, he was determined to show that the political philosophies that rely on history flowing towards a specific destination were both wrong and extremely dangerous. Many more people agree with the second part of that than have bothered to understand the first: historicism isn't just dangerous, it is wrong.

He argues:

"My proof consists of showing that no scientific predictor–whether a human scientist or a calculating machine–can possibly predict by scientific methods, its own future results."

And, while I'm simplifying Popper's argument a lot, the basic idea holds. An AI cannot learn how to run the world by recognising patterns and outcomes because there are no consistent patterns to observe. Just as importantly, even if there were such patterns, an AI could not learn them quickly as the implications of an intervention in the world might take a whole human lifespan to become apparent. That's a pretty slow learning loop. 

To put it simply: the apparently magnificent achievements of current AI tools are based on dramatic improvements in the algorithms for pattern recognition. But they probably won't work at all and certainly won't show unconstrained exponential improvement when there are no patterns to observe.

So what?

The implications of this are not all pessimistic. There are problems where AI can give us better solutions than we currently have. Specific problems where there are clear patterns and clear outcomes may well fall to AI (eg interpreting breast scans to identify the early stages of cancer or eye scans to spot incipient eye disease before it gets too bad to fix). But these are a fairly small subset of problems in medicine or life. And many of the big problems in the human world have none of the characteristics that would enable an AI to recognise patterns and solve anything.

It would certainly help if we had a better idea about where AI investment should be directed. If we put effort into problems current AI techniques are likely to be able to solve, we could see some big benefits. But the hype train appears to be overwhelming our judgement. The recently announced NHS decision to spend £250m on AI appears to be driven by exactly the same naive optimism that has invested huge sums into AI research in the past. And failed. Every time. DARPA spent a lot in the 1950s & 1960s but gave up in despair in the 1970s. Japan had a huge related programme in the 1980s followed by the EU and the UK (with the Alvey programme). All these were written off as failures by the mid 1990s. All suffered from overambition and a failure to identify which problems could be tackled given the tools available. There is little sign that the current boom has learned anything from this history.

Overoptimism may be the most important risk. We trust outputs produced by computers even when they do not deserve our trust. As Meredith Broussard argues in Artificial Unintelligence:

"One recurrent idea in this book is that computers are good at some things and very bad at others, and social problems arise from situations in which people misjudge how suitable a computer is for performing the task."

Even when we apply big algorithms we understand (we often don't understand the details behind learning AIs) we suffer from this problem. We trawl big datasets seeking new patterns, for example on the relationship between diet and health. We find new patterns. We believe the new patterns because some clever new Big Data algorithm has found them. But the problem with big data is that many of the "patterns" are noise. The number of apparently significant statistical correlations in large data sets grows faster than the dataset and a lot faster than the true patterns. The vast majority of the "new" patterns we see turn out to be noise when properly tested (see Ioannidis' famous paper Why Most Published Research Findings are False). Or Broussard's point on why the even more hyped science of Big data won't help:

"Here’s an open secret of the big data world: all data is dirty. All of it. Data is made by people going around and counting things or made by sensors that are made by people. In every seemingly orderly column of numbers, there is noise. There is mess. There is incompleteness. This is life. The problem is, dirty data doesn’t compute."

If we automate the search for new patterns with clever computer algorithms or AI we dramatically inhibit our critical faculties in assessing the results. The clever computer said it, so it must be true. Even when the dataset the computer used would be thrown out as irredeemably corrupt by any self-respecting scientist.

The biggest risk from AI is not that the AI will try to take over the world. It is that we will lower our natural skepticism and we will trust what the computer says even when it does not deserve our trust. 

We are also at risk of wasting vast amounts seeking magic bullets to solve problems we could solve for less money using known techniques. Many medics in the NHS have pointed out that providing hospitals with computers that don't take 30mins to wake up in the morning and which don't require 10 separate logins for single clinic session might yield more good than £250m on speculative investment in AI. But the AI is headline friendly and gets the cash but boring improvements in basic IT are just not newsworthy and get no investment.

I've rambled on for far too long already. But my basic conclusion is this: AI can solve some (narrow) problems and, if we are going to spend money, that's where we should direct it. But we should inoculate ourselves against the hype: AI won't solve the big, human problems we have and we should not waste money on programmes that assume it will.

Oh, and the idea that a future AI could take over the world is nonsense.