The Dreaded Red Squamish

As an aid to another discussion elsewhere regarding the latest pandemic, consider the effect of screening for the Dreaded Red Squamish of which, unbeknownst even to Health Care Professionals, infects 5% of the population.

The test, which includes the person administering it, the instruments, conditions, and all what have you, is known to be 95% sensitive -- of those with the Squamish, the test will come back positive 95% of the time -- and 95% specific -- of those without the Squamish, the test will come back negative 95% of the time.

Perceptive Reader will notice that this means a 5% risk of a false positive and a 5% risk of a false negative. The Usual Suspects may cry, "No fair!" because they want Daddy and Mommy to ensure 100% perfect. [When do we want it? Now!] But the sensitivity is about normal for lab tests while the specificity is actually better than normal. (As an example of lack of specificity is the well-known ability of drug testing to detect the consumption of poppy seed bagels.) It is also hard to imagine that the 15,000th test will be performed with the same sprightly verve and enthusiasm as the 1st.

Now, test a million people for the Red Squamish, just in case.

 

 Of the 950,000 folks who are not infected. nearly all (95%) get a clean bill of health and of the 50,000 infected souls, nearly all (95%) get a red card. But along the way 2500 infected people go undetected, while 47,000 uninfected get red carded nonetheless. Perhaps they ate a poppy seed bagel that morning, or the test was run late at night by a dog-tired technician. In any case, you will notice that half of all those getting flagged are not in fact infected.

Consequently, the media reports that the infection rate is 9.5% [(47,500+47,500)/1,000,000] rather than 5%, although what they really mean is that the test-positive rate is 9.5%. 

 

This also affects estimates of the mortality rates. It's not called the Dreaded Red Squamish for nothing. But the denominator has been inflated by the false positives, so deaths will be divided by 95,000 rather than the [unknown] 50,000. There will be 47,000 "recoveries" of people who never actually had the disease. OTOH, some of the undetected 2,500 may also shuffle off the coil of mortality, but these will be assigned to collateral conditions (heart disease, asthma, etc.)

Those unfamiliar with the exigencies of measurement systems analysis are too likely to take test results as given.

Type I and Type II errors are universal. They apply to any decision process. In law, Type I is convicting the innocent and Type II is freeing the guilty. In the FDA, they are (I) withholding approval from a safe and effective medical device or (II) approving an unsafe or inneffective medical device. In product inspection you can (I) approve defective product or (II) reject conforming product. You can decide (I) not to kiss a willing girl or (II) to kiss an unwilling one.

What differs among these cases and others are the consequences of the errors. We try to avoid Type I error in trials and sending innocent people to jail; but we would rather avoid Type II error in FDA approvals. In the latter case, if we approve a device that turns out to be unsafe, people may die. People may also die if a safe and effective device is withheld from the market -- but they don't die on the front page. 

 

You can't improve by making the decision rule more stringent. That will only shift the errors between I and II. It's like Whack-a-Mole. Drive down one type of error and you'll drive up the other. You have to change the decision rules themselves.

Of course, none of this means the Dreaded Red Squamish is not dreadful. It only means the numbers may lead to unreasonable panic or to complacency. 

 

Or both.