In the wake of the verdict finding Lucy Letby guilty of murdering seven babies, and attempting to murder six more, The Daily Telegraph published a reproduction of a table created by Cheshire Constabulary. The table was presented to the jury as key evidence by the prosecution and disseminated widely to the public by the media. The table covers a 12-month period (June 2015- June 2016) and lists the nurses who were on shift during the period where each of the 18 babies died or collapsed. The overlap between Lucy's presence on the ward and the reported collapses/deaths is commonly used to rebut claims that there might be some doubt as to the reliability of the scientific evidence used to convict Lucy Letby. The table shows that Lucy Letby was always on shift at the neonatal ward when the 25 incidents referred to in the case occurred.
Figure 1. The shift data created by Cheshire Constabulary and used by the prosecution in the trial of Lucy Letby.
At first glance the table is stark and damning, seemingly a smoking gun in Lucy Letby’s hand. However, this table should never have been presented to the jury as validation of the prosecution's claims that Lucy Letby was connected to deaths and collapses of babies on the NNU. The table does not consider the full set of data during the period in question, and by limiting the baby incidents to the periods when Lucy was present, it excludes the incidents that occurred when Lucy was not present. In essence, the table goes a long way in its attempt to lead the jury to believe that correlation proves causation. What this table does not reveal is that there were nine other neonatal deaths on the ward during the period investigated, obtained through a Freedom of Information request. These nine deaths alone represent a significant increase from the average number of deaths per year in the ward, (3.5). These deaths were not included in the table shown to the jury, for reasons unexplained.
Autopsies were performed for six of the babies included in the table, a natural cause of death was listed for five of them, and one cause of death was unascertained. All but one child was born pre-term, several of them extremely pre-term. The babies showed many signs of viral infection, including changes to vital signs, poor feeding and rashes consistent with an infection. During the period investigated, it wasn’t just the perinatal deaths that increased. When compared to prior years, the number of stillbirths had increased too. After the trial, it was revealed that at the time of the deaths and collapses, the NNU at CoCH had a viral outbreak and babies were being quarantined, which had also occurred in other surrounding hospitals.
The reports of viral infections on the NNU, speak to a statement made by Sarrita Adams, Scientific Consultant and the founder of Science on Trial:
“My first assumption when I looked at this case was to say: where do you see a cluster of deaths in a short space of time? When there is an infectious disease outbreak. 11% of all neonatal deaths are due to infectious diseases, and yet this was dismissed, and air embolism became the claim.”
There were other issues on the unit, including the repeated occurrences of sewage back-flow into the unit and onto the floors and surfaces. Numerous viruses thrive in sewage, and the susceptibility of waterborne viruses to disinfection is known to vary between viruses and even between closely related strains (Meister et al., 2018). It is understood that viral resistance to detergents plays a primary role in the persistence of infectious diseases in the hospital setting.
Figure 2. Shows that chlorination of water in the lab setting may not be the best model for evaluating the persistence of viral strains in wastewater in real world settings.
Following the unusually high rate of neonatal deaths at the CoCH, a 2017 report by the Royal College of Paediatrics and Child Health identified significant gaps in doctor and nursing rotas, poor decision-making and insufficient senior consultant cover. The report heavily criticised the doctors running the ward and warned against making unfounded allegations regaring a nurse on the NNU. After publication of the report, the same four doctors reported Lucy to the police, and therefore she was already under suspicion when the police started their investigations. Therefore, it is important to know how the cases were selected and whether this biased the investigation (a classic ‘Texas Sharpshooter Fallacy’).
Esteemed mathematician, Dr Alexander Coward, reviewed the table and reflected on the numerous issues that it presented for him:
“This table is not a fair or sound statistical analysis and should not convince you of anything. It is cherry picked data; it misleads the viewer because it only looks at events that happened when Lucy was on shift”.
To illustrate this, Dr Coward modelled a ward with 38 nurses over 170 shifts about whom, we can assume, there is no question of wrongdoing.
“Let’s imagine a ward in a hospital where there are thirty-eight nurses, each nurse works 170 shifts. On each shift there is a 14% chance of a baby collapsing or dying completely at random, which looks like this:"
Figure 3. The mathematical approach involves identifying the total shifts a nurse must work, and then determining the chance of a baby collapsing/dying.
"Now, let’s pick a nurse at random and now look at just the shifts where that particular nurse had deaths and collapses. What does that look like?"
Figure 4. The new shift data reveals that another nurse could easily end up in Lucy Letby's situation if they happened to work the same number of shifts.
It looks like nurse 13 is doing something wrong, but in fact this is a completely random nurse with a plum average number of deaths/collapses (25) over the period investigated. This illustrates what you can do with cherry picked data.”
Dr Alexander Coward
The point of this exercise is to show that you can recreate exactly such a table to implicate any of the nurses through mere randomness.
“This model is not meant to be an analysis of the CoCH ward, there isn’t enough data available for that yet, but instead to show what randomness looks like. If you have experience with statistics and data, you have an instinct for what randomness looks like. The jury clearly did not,” he argues. What this tells us is that the human eye is good at recognising patterns, but we are not necessarily good at asking “is this pattern unusual?”
Shifting Responsibilities
The approach that was taken by Cheshire Constabulary in order to reason about events on the NNU is incredibly concerning. It would be one thing if this approach were simply a part of the police investigative technique. What makes the table most disconcerting is that it was then presented to the jury without any further information as to its reliability.
The shift table does not represent a sound statistical analysis because we need a wider data set in order to draw safe conclusions. It does not look at all neonatal deaths, or all the nurses’ shifts and it omits the doctors’ shifts in the year in question. What is also missing is the fact that some nurses work full-time, some part-time, while some work on administrative duties only. In addition, as with some other nurses, Lucy Letby lived near-by and worked extra shifts, probably working the maximum number of hours per week. She was called in for short notice shifts, which would likely be correlated with understaffing problems. Statistician Prof Jane Hutton, at Warwick University, said in an interview with science.org this year,
“It is necessary to understand broader patterns in deaths. There is a tendency for people to die in the morning, and so a nurse who is more likely to be on-shift in the mornings may be present for more deaths than the nurse who works a different shift pattern.”
Prof Jane Hutton, Professor of statistics, Warwick University
Thus, the chance of a nurse being present when there is an incident varies between nurses. The dubious statistical insinuations were powerful and raises concerns about courtroom strategy regarding cases in which the evidence is wholly circumstantial. It is incredibly concerning how the flawed evidence was presented before the jury.
Factors Beyond the Table
Another important point to consider is that three sets of twins and one set of triplets are among the neonates in the table and that, regarding these cases, Lucy Letby was found guilty of murder or attempted murder. Where both twins experience adverse health events in a similar timeframe one would assume that such events are due to shared physiological aetiology, thus the two deaths are not statistically independent.
The distortion of statistical information shares a striking similarity with the famous case of Sally Clark, a mother wrongfully convicted of murdering her two infant sons who died of SIDs in the 1990s, as a result of statistical ineptitude. As mentioned in “The Irrational Ape” by Dr David Robert Grimes,
“Roy Meadow, a paediatrician, was brought in by the prosecution to give evidence against her. He asserted that the likelihood of an occurrence of SIDs was 1 in 8,543. Thus, he reasoned, the chances of two cases of SIDs in one family was 1 in 73 million. To arrive at the figure of 1 in 73 million, Meadow simply multiplied the probability of two independent events together, but this fails horribly when the events are not independent. It was well known from epidemiological data that SIDS tends to run in families, perhaps due to genetic or environmental factors. This renders the blithe assumption - that the two deaths are independent of each other - nonsensical, and in addition to uncovering new medical evidence, Sally’s conviction was finally overturned. The appeals conceded that the statistical errors had utterly perplexed the jury and hopelessly biased the trial.”
Following the exoneration of Sally Clarke, Roy Meadow was struck off the British Medical Register for giving erroneous and misleading evidence to the jury. At the same time, criticisms were made about the approaches taken by the Home Office Pathologist who failed to report that one of Sally's babies had a severe bacterial infection. Commenting on the issues presented in the wrongful conviction of Sally Clark, the forensic scientist, Prof Roger Byard, stated:
"This case and its sequelae demonstrate the difficulties that may arise if cases are not fully investigated by pathologists with specific training or experience in paediatric forensic pathology, with all of the Results being clearly summarised and discussed in autopsy reports. Trying to clarify findings, diagnoses and circumstances of death at a later stage may simply not be feasible, owing to a wide variety of possibilities other than inflicted injury."
Prof Roger Byard, Emeritus Professor and a Senior Specialist Forensic Pathologist at Forensic Science SA
History repeats
The similarities between the Sally Clarke case and the Lucy Letby case do not end with the misrepresentation of statistics. In the Lucy Letby case, the prosecution failed to provide experts who had any experience with the claims they made. A retired paediatrician, Dr. Dewi Evans, was called to testify for the prosecution as an “expert witness”. He presented ‘scientific evidence’ related to gas embolism, however the 1989 paper on which he based his claims is explicit that post Mortem X-rays must not be used after 30 minutes, but X-rays beyond this time limit were used by the expert witnesses to determine air embolism.
Furthermore, the 1989 paper was not relevant to the incidents in Lucy’s case. None of the findings from the original autopsies suggested the babies died due to air embolism. Dr Dewi Evans was a medical doctor not a scientist, nor a neonatal forensic pathologist. At issue is the reliability of the evidence on a scientific basis; no scientist gave evidence during a trial in which science was clearly fundamental. Of the flaws in the trial, one of the most salient is that no defence expert was called in Letby's trial, this left a lot of the prosecution's evidence unchallenged. Human rights and criminal defence Barrister Mark McDonald states, "It is almost impossible to get defence experts in the UK to give evidence in cases involving children, they are too scared. You have to go overseas, usually to the US."
Speaking with Richard Gill, Emeritus professor of statistics at the University of Leiden, he explained:
“To perform a correct statistical analysis I would collect the relevant data of all neonatal events and all nurses, control for confounding factors and fit multivariate logistic regression models. Then we can start to compare like for like.”
Prof Richard Gill, Emeritus Professor of Statistics at the University of Leiden, Netherlands
Prof Gill set out a more robust approach than advanced by Cheshire Constabulary. He detailed that he would compare the chance of a death on one of Lucy Letby’s shifts with the chance of a death on another nurse’s shift. Prof Gill, has experience with the manner by which prosecutors manipulate data, he is credited with playing a pivotal role in securing the exoneration of Lucia De Berk.
Lucia was a Dutch paediatric nurse, who was wrongfully convicted of seven murders and three attempted murders on a neonatal ward in the Netherlands, for which she served six years in prison. In our discussion, Prof Gill emphasised, “It is manifestly clear that Lucy Letby did not have a fair trial. The similarities with the famous case of Lucia de Berk in the Netherlands are deeply disturbing.” In stark parallel to Lucy's case, lucia was placed under suspicion simply because her shifts coincided with a number of deaths on the NNU, and her hand-written notes were also produced as evidence to the court. Nurses are encouraged to journal as part of their training. Professor Emeritus of criminology, David Wilson, analysed Lucy's notes on Panorama, denying they are evidence of a confession and instead "the ramblings of someone under extreme psychological pressure" due to the nature of the accusations against her. Also key in overturning Lucia de Berk's conviction was an independent multidisciplinary scientific team of medical specialists, pathologists & toxicologists, who were finally allowed complete access to medical dossiers of key cases.
Lucia’s case is now considered one of the worst miscarriages of justice in the Netherlands, and it compelled Prof Gill to co-author and publish a Royal Statistical Society report, ‘Healthcare serial killer or coincidence?’, which shows that clusters of unexpected deaths do occur in a healthcare setting due to exposure to common risk factors. And that furthermore, the apparently ‘worst’ statistics are often attached to the best nurses/doctors because they tend to see the sickest patients. This report covers a set of case studies where statistics have been misused in medical murder trials, with a legacy of wrongful convictions. It makes a number of recommendations as to how to avoid these statistical errors and how everybody involved in the process could ensure that trials for medical murder cases like Lucy Letby’s are conducted fairly from start to finish.
A simple review of the shift data used in the conviction of Lucy Letby reveals that if you torture data, it will confess to anything.