A beautifully told story with colorful characters out of epic tradition, a tight and complex plot, and solid pacing. -- Booklist, starred review of On the Razor's Edge

Great writing, vivid scenarios, and thoughtful commentary ... the stories will linger after the last page is turned. -- Publisher's Weekly, on Captive Dreams

Wednesday, July 6, 2016

Lighting up the Brain

Hey, ya ever see those brain image scans where they pretend they took snapshots of actual brain activity (rather than say model outputs of oxygen consumption is various regions)? Turns out there may be a slight problem:

"Functional MRI (fMRI) is 25 years old, yet surprisingly its most common statistical methods have not been validated using real data. Here, we used resting-state fMRI data from 499 healthy controls to conduct 3 million task group analyses. Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results."
Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates -- Eklund. Proceedings of the National Academy of Sciences of the United States of America
Well that explains why positive results were once obtained from a dead salmon. It does not explain why we keep getting positive results from neuroscientists, whose brains seem to light up at the thought of software highlighting regions in the brain. Esp. in the hypothalamus. 
From the paper "Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction," by Craig M. Bennett, et al.  (2009)
With  the  extreme  dimensionality  of  functional  neuroimaging  data  comes extreme risk for false positives.  Across the 130,000 voxels in a typical fMRI volume  the  probability  of  a  false  positive  is  almost  certain.    Correction  for multiple  comparisons  should  be  completed  with  these  datasets,  but  is  often ignored  by  investigators.  To  illustrate  the  magnitude  of  the  problem  we carried  out  a  real  experiment  that  demonstrates  the  danger  of  not  correcting for chance properly.
Subject.   One  mature  Atlantic  Salmon  (Salmo  salar)  participated  in  the  fMRI  study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning.
TOF loves that droll last sentence, delivered with admirable deadpan. 
Graph from here
That rang a bell in the TOFian noggin and he recalled the adjustment to the t-Test that was devised by Ellis Ott for testing simultaneous means. For example: when testing the sample means from ten different stamping presses to determine whether it is plausible that all ten were running at the same process level. Ott called this method the Analysis of Means (ANOM) as opposed to the usual Analysis of Variances (ANOVA). He pointed out that the usual decision limits (i.e., the point at which one decides whether is it worthwhile to investigate for a cause rather than go on a wild goose chase) are premised on one sample mean versus a target value. But when comparing k sample means simultaneously against their common grand mean, the uncertainty increases exponentially. Basically, thus:

Suppose you have limits such that any one sample of size n has a 5% risk of falling beyond them for no particular reason. (We call that "random chance" but don't get carried away.) But now suppose you are comparing four group means, as in the illo above. Each one has a 5% risk of giving a false alarm and thus triggering a "wild goose chase" for an assignable cause even when there isn't one
Assuming there is no actual cause to be found and the differences are due to sampling variation and to small variations due to a host of causes too small to be worth acting upon is called the Null Hypothesis. Nothing to see here. Move along. Notice that this doesn't mean there is nothing to see. It means you don't have sufficient evidence to make the Hunt for the Cause worthwhile.
With four means, the likelihood of a false alarm is the likelihood that at least one of the samples will provide a false alarm. That probability is the converse of the probability that none of them do so, which is 1 - 0.95^4 = 0.185. That is, our risk of a false alarm is 18% rather than 5% unless we compensate for the additional uncertainty.   

The salmon-chasers and the authors of the new study linked above claim that similar corrections were not being made by the software when determining "significant" signals in fMRI scans, thus making it more likely that they will give false alarms. He who lives by the software dies by the software. It's always better to know what you are doing. Or at least, to know what your machines are doing. 
h/t to Matt Briggs, Statistician to the Stars, for taking note of the paper and blogging about it on his site. He also links to previous posts of his about the new pherenology of fMRI. 

Previous TOFian comments on fMRI are here:

Jul 19, 2012 ... The current sexy thing among the cognoscenti is the use of fMRI to "prove" that there is no free will, a topic which, for some reason seems to ...
Aug 10, 2012 ... Here's a title for you, “BMI Not WHR Modulates BOLD fMRI Responses in a Sub- Cortical Reward Network When Participants Judge the ...
Feb 23, 2013 ... It occurred to me that comparing fMRI (functional magnetic resonance imaging) scans of the brain of an actor with those of an astronomer might ...
Apr 18, 2013 ... Especially lacking in statistical power are human neuroimaging studies— especially those that use fMRI to infer brain activity—noted the ...

Aug 24, 2011 ... But what justifies equating the fact of "blood-flow to brain region X" detected by an fMRI with the interpretation "Aha! That was the moment of ...
Jan 28, 2012 ... fMRI scan shows dead fish making a "free will" decision. A second reason is Thomas' utter dedication to reason. Not for him were the fiery ...


  1. Man, that dead salmon never gets old! Figuratively, at least. Came up in a conversation this morning at work, and now here again.

    1. It really is the gift that keeps on giving.

  2. From Matt Briggs:

    No, it’s “Cluster failure“. But one is tempted…One is sorely tempted.

    Yes. Yes, I will confess I was tempted also.

    1. I've always been partial to "cluster-flunk", when the stronger version has to be avoided.

  3. Even before reading the post, I thought that top picture looked a little fishy ...


Whoa, What's This?

adam amateur theology Aquinas argument from motion Aristotelianism art atheism autumn of the modern ages books brains breaking news captive dreams cartoon charts chieftain clannafhloinn comix commentary counterattack crusades culcha dogheads easton stuff economics eifelheim evolution factoids on parade fake news fallen angels Feeders fir trees in lungs firestar flicks floods flynncestry flynnstuff forecasts forest of time fun facts gandersauce gimlet eye global warming glvwg headlines henchmen high frontier history home front how to lie with statistics humor hush-hush hypatia in the house of submission irish Iron Shirts irrationalism january dancer jihad journeyman kabuki kool letter lion's mouth lunacon maps mayerling medieval metrology miscellany modern mythology moose zombies music new years nexus odds odds and ends paleofuture passing of the modern age philosophy philosophy math poetry politics psyched out! public service quality quiet sun quote of the day razor's edge redefinition of marriage religio reviews river of stars scandal science science marches on scientism scrivening shipwrecks of time shroud skiffy skiffy in the news skools slipping masks some people will believe anything stats stories stranger things the auld curmudgeon the madness continues the new fascism the spiral arm the writing life thomism thread o' years tofspot topology untergang des abendlandes untergang des morgenlandes up jim river video clips vignettes war on science we get letters we're all gonna die whimsy words at play xmas you can't make this stuff up