Covid-19 and the false positive trap

16 September 2020

9:11 PM

16 September 2020

9:11 PM

Imagine a world where Covid-19 has been eliminated. To be certain this is true, the government conducts regular tests at random. The number of positive results should be zero, right? Wrong. There will always be a proportion of cases tested that come back with a false positive test result. Thankfully, for Covid-19, the false positive rate is less than one per cent of tests done. But it is not zero. It will be impossible for us to ever reach zero. Why? Because Covid-19 cannot be eliminated, even if it is likely to evolve to be more benign and become a seasonal problem like influenza.

Coronaviruses are, after all, seasonal, and as winter approaches we can expect more outbreaks of genuine Covid-19. In the meantime, understanding what the data will look like when Covid-19 is at minimal levels is critical to identifying real outbreaks in the noise (and distinguish these from false positives). As we move forward with this disease, we need to clearly understand where the risk is real and where the risk is almost non-existent, to prevent unnecessary harm from unnecessary interventions.

One of the problems we encounter here is the false positive rate. Put simply, the more tests, the greater the absolute number of false positives. False positives are not due to bad laboratory technique. They are a function of both the population tested and the laboratory test itself. Each population and laboratory will have a percentage of false positives, and every day that will vary. The average of these percentages over time is the false positive rate for that population.

There are two ways to find out what the false positive rate of a test is. One is to run the test on ‘perfect’ laboratory samples where there is certainty about the diagnosis and figure out how many results are wrong. In real life, cases are more ambiguous or complicated than these ‘perfect’ samples. The other method is to study testing being done in different laboratories and over time.

Our real-world testing has resulted in a summer where the hospital positive rate has flatlined at 0.4 per cent. For two full months, despite day-to-day fluctuations, the actual data kept returning to this figure. At such a low and consistent rate of positive testing, it seems likely that the majority of that 0.4 per cent can be accounted for by false positives. Indeed, this is the figure the government is using to estimate false positive results. Having 99.6 per cent of your results avoid the false positive problem is a phenomenal result. Testing rarely gets better than this.

But even if this phenomenally low positive rate continues, we still won’t reach a situation with zero deaths attributed to Covid-19. Currently hospital admissions are randomly screened for Covid-19, whether or not they are symptomatic. If we had attempted this before Covid-19 existed, we would have had a 0.4 per cent false positive result even in that population. Some of these patients would have died with this false positive Covid result (they died because whatever brought them into hospital killed them, not Covid, which they did not have). In a normal year, across all patients admitted to hospital, the average risk of dying is 1.7 per cent. It will be impossible for ‘Covid mortality’ to fall below this level with this testing strategy. We may think we are observing the tail end of an epidemic when really we are counting perpetual mislabelled deaths.

So how can we identify these false positives? It is a difficult task given that false positives are fairly randomly distributed among those tested. To test whether the cases we are seeing now are Covid or not, we can compare the characteristics of those testing positive with the characteristics of real Covid cases from spring. The false positive population would look like the general population who were tested, rather than the real Covid cases. Instead of being old, obese, diabetic, with serious co-morbidities and disproportionately from ethnic minorities, the false positive population would have a lower rate of the above and be in line with regular deaths. We can use this to see how big a problem this has become.

Actual Covid cases during the epidemic were disproportionately seen among older people; in fact, 60 per cent were over 60 years old. Of the cases we are seeing now, only 11 per cent are in the over 60s, despite that age making up 24 per cent of the whole population. This means they are slightly under represented, but this may be a reflection of their willingness to be tested compared with younger people, and it would help to know the age of those tested for comparison. In contrast, only two per cent of cases were seen in the under 20s during the epidemic and this is now up to 19 per cent, much closer to the 24 per cent that would be seen if positive tests were distributed entirely randomly through the population.

Some argue that our efforts to shield the vulnerable have paid off, and we are now better at protecting the over 60s. However, whilst during the pandemic men accounted for 60 per cent of deaths, throughout summer that figure reverted to 50 per cent. It would be odd to believe that men were better than women at shielding. An increasing proportion of false positive results would push that figure nearer to 50 per cent. So the people affected seem to be more like a sample of the general population than a typical Covid population.

Let us turn from the people diagnosed, to the behaviour of the disease itself. Covid was horribly deadly in spring. In April, the chance of someone dying when admitted to hospital with Covid was six per cent. By June, that had fallen to 1.5 per cent. The efforts of ITU teams and others to bring the rate down should not be underestimated. However, once the rate of death fell to the same level as the background death rate for general hospital admissions, suspicions must inevitably arise about how many of these patients actually had Covid. How many were false positives?

Covid deaths followed a predictable pattern with death occurring around day 20. Of 42,000 originally diagnosed as having died from Covid, 5,000 patients were reclassified when Prof Carl Heneghan pointed out that they had occurred more than 28 days after diagnosis. No deaths before May needed revising. Only the data for those deaths that occurred in the summer had to be ‘revised’. So the tight relationship between diagnosis and death for Covid was lost in the summer. It could be that the false positive problem has impacted this data.

Rather than excuse these anomalies with a series of explanations, Occam’s razor should be applied and the conclusion reached that these patients did not have Covid. How can we test this hypothesis further? If patients have indeed continued to be infected with Covid over the summer then antibody levels will have continued to rise. In fact, antibody levels have flatlined and then fallen.

Following the real science will get us to the truth. Real science involves hard work and experiments. Here are some suggestions for where to begin:

  1. Data not yet in the public domain will be able to help with a series of questions. Are ethnic minorities still at higher risk? How many allegedly positive ‘cases’ have lost their sense of smell in the summer compared with the spring? Are radiologists still finding the classic appearance on chest CT scans? What proportion of allegedly Covid positive admissions required oxygen over the summer compared with spring?
  2. The proportion of positive test results over time could be studied for each age group (or ethnic group). If the chances of testing positive tend to the same number for every age group then that would be indicative of the data simply reflecting the false positive rate.
  3. For laboratories that have reached a positivity rate of below two per cent, viral cultures can be used to give an indication of the true positive rate. No positive cultures will be possible if they are false positive cases.
  4. A cohort of patients with a positive Covid test should have CT lung scans to demonstrate ground glass or other radiological changes in those that are true positives. These can be found even in pre-symptomatic patients and are very specific for Covid.
  5. Post mortem examinations should be carried out on a cohort of patients who have died with a label of ‘Covid-positive’ over the summer. The Royal College of Pathologists have worked hard during the spring to collate post mortem reports on real Covid cases and to demonstrate the significance of findings in these cases. Comparison with low prevalence areas will be invaluable.

The extent of the false positive problem is not confined to Britain: it is global. It is not an issue of small technical problems in the laboratory. And it is one we are going to have to live with, at least for now. In the mean time, current test results should not be automatically accepted as real. A higher standard of proof is required when the impact of these results affects our approach to births, weddings, deaths and funerals – and the possible imposition of lockdowns.

A new case in an area that has not had a true case for months is extraordinary, as is any child ever testing positive for Covid. Extraordinary claims require exceptional evidence. Without genome sequencing, viral culture or chest CT results then any claim that a new Covid-positive case has suddenly appeared in an area or population must be questioned. Only by adopting this approach can we tackle the Covid false positive test problem.

Got something to add? Join the discussion and comment below.

Dr Clare Craig FRCPath is a consultant pathologist who has worked for Imperial College Healthcare NHS Trust; the 100,000 Genomes Project and for a digital pathology startup Panakeia Ltd.

Show comments