Thursday, March 12, 2020

The Dreaded Red Squamish

As an aid to another discussion elsewhere regarding the latest pandemic, consider the effect of screening for the Dreaded Red Squamish of which, unbeknownst even to Health Care Professionals, infects 5% of the population.

The test, which includes the person administering it, the instruments, conditions, and all what have you, is known to be 95% sensitive -- of those with the Squamish, the test will come back positive 95% of the time -- and 95% specific -- of those without the Squamish, the test will come back negative 95% of the time.

Perceptive Reader will notice that this means a 5% risk of a false positive and a 5% risk of a false negative. The Usual Suspects may cry, "No fair!" because they want Daddy and Mommy to ensure 100% perfect. [When do we want it? Now!] But the sensitivity is about normal for lab tests while the specificity is actually better than normal. (As an example of lack of specificity is the well-known ability of drug testing to detect the consumption of poppy seed bagels.) It is also hard to imagine that the 15,000th test will be performed with the same sprightly verve and enthusiasm as the 1st.

Now, test a million people for the Red Squamish, just in case.
 
 Of the 950,000 folks who are not infected. nearly all (95%) get a clean bill of health and of the 50,000 infected souls, nearly all (95%) get a red card. But along the way 2500 infected people go undetected, while 47,000 uninfected get red carded nonetheless. Perhaps they ate a poppy seed bagel that morning, or the test was run late at night by a dog-tired technician. In any case, you will notice that half of all those getting flagged are not in fact infected.

Consequently, the media reports that the infection rate is 9.5% [(47,500+47,500)/1,000,000] rather than 5%, although what they really mean is that the test-positive rate is 9.5%. 
 
This also affects estimates of the mortality rates. It's not called the Dreaded Red Squamish for nothing. But the denominator has been inflated by the false positives, so deaths will be divided by 95,000 rather than the [unknown] 50,000. There will be 47,000 "recoveries" of people who never actually had the disease. OTOH, some of the undetected 2,500 may also shuffle off the coil of mortality, but these will be assigned to collateral conditions (heart disease, asthma, etc.)

Those unfamiliar with the exigencies of measurement systems analysis are too likely to take test results as given.
Type I and Type II errors are universal. They apply to any decision process. In law, Type I is convicting the innocent and Type II is freeing the guilty. In the FDA, they are (I) withholding approval from a safe and effective medical device or (II) approving an unsafe or inneffective medical device. In product inspection you can (I) approve defective product or (II) reject conforming product. You can decide (I) not to kiss a willing girl or (II) to kiss an unwilling one.
What differs among these cases and others are the consequences of the errors. We try to avoid Type I error in trials and sending innocent people to jail; but we would rather avoid Type II error in FDA approvals. In the latter case, if we approve a device that turns out to be unsafe, people may die. People may also die if a safe and effective device is withheld from the market -- but they don't die on the front page. 
 
You can't improve by making the decision rule more stringent. That will only shift the errors between I and II. It's like Whack-a-Mole. Drive down one type of error and you'll drive up the other. You have to change the decision rules themselves.

Of course, none of this means the Dreaded Red Squamish is not dreadful. It only means the numbers may lead to unreasonable panic or to complacency. 
 
Or both.


14 comments:

  1. This is a "noob" question, but how does repeat testing affect the numbers?

    ReplyDelete
    Replies
    1. If you test positive, get a second test. If the probability of a false alarm is 5%, that of two consecutive false alarms is 5% of 5% (0.05*0.05 = 0.0025) assuming test independence.

      Delete
    2. Of course, we enter a recursive problem. How do we know the false positive rate is 5% if it really is 5%?

      Delete
    3. By testing on pure samples, just like any calibration problem.

      Delete
    4. So, that means that there IS a one-hundred percent, rock solid, always right test for having-the-dreaded-Red-Squamish and not-having-the-dreaded-Red-Squamish, and your discussion regards a test that returns results in less than a month and is cheaper and mass-producible?
      So, when the FDA grants "emergency" approval of a test for the dreaded Red Squamish, does that mean that much of the regular approval process is unnecessary, or that they're approving a lower sensitivity than would normally be acceptable?

      Delete
    5. There is no test whose alpha and beta risks are zero. WHO had a test in wide use, but CDC insisted that only the CDC test could be officially used. Unfortunately, these at first had contaminated reagents and so the samples had to be sent in to the CDC from the field, a terrible bottleneck. Meanwhile, perfectly able laboratories were barred from running the test, because they were not licensed as "clinical" laboratories. A polymerase chain reaction, the heart of the test, is not all that exotic.

      Delete
  2. This is a useful reminder of the problems with mass testing for rare diseases. However, not much mass testing has been done for Covid so far. Excluding China, which may have unreliable data, the country with most tests (S. Korea) has tested about 250,000 people, and has about 8000 cases. So even if all those are false positives, which is unlikely, the false positive rate is 3.2%. I'd guess it is below 1%. (I have not been able to find online any estimates for false positive/negative rates for the covid tests.) It seems the USA has tested about 13,000 people, so that's possibly 130 false positives out of a stated 2,500 cases.

    ReplyDelete
    Replies
    1. Actually, it doesn't have to be mass testing. You can convert to probabilities and it applies to individual cases. What mass testing exacerbates are errors like fatigue, 'highway hypnosis,' and the like. The caution here is against the idea that mass testing is an effective action. Like 100% visual inspection of punch card ballots.

      Delete
    2. I recently heard the "sensitivity" of the rtPCR (reverse transcriptase polymerase chain reaction) test is 60-70%. This means between 30-40% of the tests are false negatives. Ahhhh! More fear for those running to be tested ... Even when medics tell someone they are negative, they can be quite fearful they still have the dreaded Red Squamish (dRS). Throw in the way medical diagnoses are coded (U07.1 [dRS+] vs. U07.2 [dRS- test but clinical +)and we can get even people with a negative test put into the dRS+ data. Then if you die from head trauma but tested positive we can put you in mortality figures. And if you die and get tested by post-mortem swab we can put you in that data, too. GIGO, but at least we can get money sent to certain areas based on the garbage out. Isn't how this really works?

      Delete
    3. You know that specific example has been uncovered as happening in reality, right? In Pennsylvania I believe.

      Delete
  3. Two things stand out on this CDC page: https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/cases-in-us.html

    ▶"Data include both confirmed and presumptive positive cases . . ."
    So some part of the numbers we're seeing come from somebody presuming a person was infected.

    ▶"CDC is no longer reporting the number of persons under investigation (PUIs) that have been tested, as well as PUIs that have tested negative."
    So, even among the non-random groups sampled they aren't telling us what percentage are testing positive.

    It seems to me that the only way we're going to know how fast (or, strictly, even if) the incidence of infection is growing is with a continuing series of tests of randomly chosen people with ALL RESULTS REPORTED, not just the positive ones. Without knowing the incidence in random samples of the population "new cases" & "total cases" numbers mainly reflect how much testing is being done.

    The other thing that really bothers me is that I can't find figures for total mortality rates for NYC for now (say March) and the corresponding period one year ago. I can't find that information for Italy either, but I don't read Italian. Since there is some reason, as Steve hinted above, to think that authorities may be tabulating any dead person who has tested positive (or apparently even presumed to be infected) as a virus death, that seems like important data to know. If this thing is a bad as they say, it should be reflected in the total all-cause mortality rates. Consider that the great majority of deaths are in a demographic with a high death rate anyway, some of this has the whiff of statistical chicanery. Maybe these numbers are out there, but I can't find them.

    You might like to look at a few minutes of this following the 3:00 minute mark. I've only looked at a few minutes. I started to look for the source of his graphs and thought of you. Any insights? https://www.bitchute.com/video/JVgvv5yLrAO1/

    ReplyDelete
  4. I found what looks like a better source of data:
    https://covidtracking.com/
    and graphs derived from it:
    https://www.covidcharts.com/

    ReplyDelete
  5. Howdy! Quick question that's totally off topic. Do you know how to make your site mobile friendly? My web site looks weird when browsing from my iphone 4. I'm trying to find a theme or plugin that might be able to correct this issue. If you have any suggestions, please share. Many thanks!

    ReplyDelete

Wonder and Anticipation, the Likes of Which We Have Never Seen

  Hello family, friends and fans of Michael F. Flynn.   It is with sorrow and regret that I inform you that my father passed away yesterday,...