|

 

Tu-dresden.de

Brain Images, Babies, and Bathwater: Critiquing Critiques of Functional Neuroimaging BY MARTHA J. FARAH Since the mid-1980s, psychologists and neurosci- of imaging's limitations without "throwing the baby out entists have used brain imaging to test hypotheses with the bathwater." about human thought processes and their neural The criticisms I review here can be grouped into four instantiation.1 In just three decades, functional neuroim- general categories, although there is a degree of overlap aging has been transformed from a crude clinical tool among them and many have been combined in the lit- to a widely used research method for understanding the erature despite their conceptual distinctness. The first human brain and mind. Such rapidly achieved success is broad category concerns the nonidentity of, or the gap bound to evoke skepticism. A degree of skepticism to- between, the neural events being studied and the images ward new methods and ideas is both inevitable and useful that purportedly represent them.
in any field. It is especially valuable in a science as young as cognitive neuroscience and its even younger siblings, Inferential Distance and the Objects of Imaging social and affective neuroscience. Healthy skepticism encourages us to check our assumptions, recognize the limitations of our methods, and proceed thoughtfully. As Adina Roskies has pointed out, functional brain images are easily misunderstood as photographs of Skepticism itself, however, also must be examined.
brain function.2 There are, however, numerous types of Functional neuroimaging has attracted a substantial mismatch in the relationship between published func- amount of skepticism from inside and outside the fields tional brain images and the brain activity they represent. of psychology and neuroscience. In this article, I review Two of these mismatches form the basis of many of the the most commonly voiced criticisms of functional neu- common criticisms reviewed here.
roimaging. In the spirit of healthy skepticism, I will criti- Blood versus brain. The signal measured in function-
cally examine these criticisms themselves. Each contains al magnetic resonance imaging, as explained by Geoffrey at least a kernel of truth, although I will argue that in Aguirre in his essay in this collection, is a characteris- some cases the kernel has been overextended in ways tic of blood rather than brain tissue. The dependence that are inaccurate or misleading. In other cases, the on hemodynamic proxies for brain activity strikes some criticisms are valid as presented and deserve the careful commentators as a fundamental flaw in the functional attention of imaging researchers. The goals of this article neuroimaging enterprise. I offer three examples of this are to distinguish between reasonable and unreasonable critique, from philosophers, a science writer, and a psy- criticisms and to identify some general characteristics of the two categories. In this way I hope to encourage the Philosophers Valerie Hardcastle and Matthew Stewart fair and realistic evaluation of functional neuroimaging question "the excited hoopla over fMRI and other im- as a scientific method and to foster an understanding aging techniques" by pointing out that fMRI informs us about activity only in a relatively large area of brain Martha J. Farah, "Brain Images, Babies, and Bathwater: Critiquing tissue (on the order of millimeters) and can inform us Critiques of Functional Neuroimaging," Interpreting Neuroimages: An only at relatively long time intervals (on the order of sec- Introduction to the Technology and Its Limits, special report, Hastings Center Report 45, no. 2 (2014): S19-S30. DOI: 10.1002/hast.295 onds).3 They suggest that this poor spatial and temporal SPECIAL REPORT: Interpreting Neuroimages: An Introduction to the Technology and Its Limits ably with neural activity. Indeed, despite con- Figure 1.
siderable research on the subject, we do not Average global temperature, 1880-2004
yet know how fMRI corresponds to specific aspects of neural activity, be they the summed (Source: Goddard Institute for Space Studies) synaptic activity of small populations of neu-rons, spiking of neurons, or other physiologi-cal phenomena.6 But let us put these criticisms in perspec- tive. The concern that fMRI shows us blood oxygenation rather than neural activity direct-ly should be weighed alongside the fact that little of what we call science involves direct observations of the subject matter of interest. Cosmologists make inferences about earlier states of the universe by measuring background microwave radiation. Chemists determine the composition of samples by heating or shining light on them and measuring emission or ab-sorption spectra. Climate scientists measure tree rings to study climate trends over previous centuries. Complaints that functional neuro-images do not "show" brain activity appear to be based on a naïve view of science and its methods. Of course, compared to the cos-mologists' or chemists' spectroscopy, we have only a weak grasp on the nature of the linkage resolution creates an imprecise representation of neural ac- between fMRI and neural activity. Nevertheless, blood- tivity and write, "This imprecision forecloses the possibility oxygen-level-dependent (BOLD) fMRI detects a relation- of directly connecting single-cell activity—which operates ship between neural activity and oxygenation that is strong three to four orders of magnitude smaller and faster—with enough to make it a useful research tool.
larger brain activation patterns. What are we to do? The Nikos Logothetis, an influential voice of caution on the answer given all too often by neuroscientists is to fudge." interpretation of fMRI data, has pointed out that similar In a popular book on the brain, Judith Horstman like- problems regarding indirectness apply to other neurosci- wise argues that "an fMRI is not a direct image of your brain ence methods: "Electrical measurements of brain activ- showing mental activity. It's an image created through in- ity, including invasive techniques with single or multiple directly measuring the flow of oxygenated blood and then electrodes, also fall short of affording real answers about correlating that information to something you are doing network activity. Single-unit recordings and firing rates or thinking at the time, and the blood flow response takes are better suited to the study of cellular properties than of time. So it's a stretch to say that there is cause and effect neuronal assemblies, and field potentials share much of the and to relate this blood surge to a specific activity."4 ambiguity discussed in the context of the fMRI sign." He Perhaps the strongest statement of concern over the also affirms that "despite its shortcomings, fMRI is cur- nonequivalence of blood flow and brain activity can be rently the best tool we have for gaining insights into brain found in the writings of psychologist William Uttal, a vo- cal critic of functional neuroimaging research, who recently In sum, it is not the case that inferences based on func- wrote, "fMRI is as distant as the galvanic skin response or tional brain imaging are, in the words of the critics quoted pulse rate from cognitive processes."5 earlier, "fudges" or "stretches." There is indeed a "cause and The undeniable truth in these criticisms is that blood effect" relationship between neural activity and blood oxy- flow and oxygenation are not the same as brain activity. genation levels, even if the nature of that relation is imper-Blood flow and oxygenation occur at different spatial scales fectly understood.
and over different time courses than brain activity. More Functional brain images as fabrications. Another way
fundamentally, even if we allow for those spatial and tem- in which the relation between functional brain images and poral limitations, there is no guarantee that cerebral blood their objects has been questioned concerns the importance flow or blood oxygenation will correlate precisely or invari- of the decisions that researchers must make (described by March-April 2014/ HASTINGS CENTER REPORT The concern that fMRI shows blood oxygenation, not neural activity directly, should be weighed alongside the fact that little of what we call science involves direct observations of the given subject matter. Aguirre in his essay) regarding what aspects of brain activity tures—for example, the range from the lowest to highest to represent and how to represent the selected activity. The naturally occurring temperatures. Doing so would put the worry, in this case, is that the images are more researcher data into a different context, which might be useful for inventions than researcher observations.
some purposes, but for most purposes it would be coun- For example, scientists and nonscientists alike have terproductive; it would simply make the relationships of regarded the use of color-coding with suspicion. Writer interest hard to see. Similarly, the color scale used for func-Michael Shermer states that "[c]olors exaggerate the effects tional brain images spans a small range of BOLD signal in the brain . . The coloring is artificial, and the process of change in order to maximize the visibility of the distribu- coloring the regions is even more misleading."8 Horstman tion of relevant activations.
echoes these criticisms, warning that despite significant Could one fabricate results, in the sense of changing differences in the colors used, "differences in activity lev- the pattern of activation, by changing color scales? It is not els are tiny."9 In an article entitled "Some Cautions about possible to boost or minimize the activation of one region Jumping on the Brain-Scan Bandwagon," psychologist and by changing scales without boosting or minimizing other textbook author Carole Wade warns that "decisions about equally activated regions. Choices of color scale or contrast color scales . . can accentuate or minimize the contrasts affect the salience of certain features, but this is equally among different brains or brain areas. Such decisions can true of other biological images, including micrographs, affect whether the gorgeous images we see at conferences, gels, and Western blots.11 The issues here are not unique to in articles and textbooks, and in the popular press will be brain imaging, nor do they indicate that colored images are striking, ho-hum—or even misleading."10 inherently or even typically deceptive.
The element of truth in this criticism is that color-cod- ing is arbitrary. The choice of which color represents which Metatheoretical Assumptions and Goals of numerical value is dictated by both convention and conve- nience. Conventionally, warmer and brighter colors repre-sent higher activation. But is this in some way prejudicial? No more so than plotting numerical data on axes where Scientific theories and methods inevitably influence one another. In the case of functional brain imaging, it has higher numbers appear higher on the page. For functional been suggested that the method seems particularly com- images or for the coordinates on a traditional Cartesian patible with certain assumptions about the mind-brain graph, one could reverse the scale to remind viewers of the relation and encourages certain types of theories while pre- conventional nature of these data representations, but the venting others from being tested. Specifically, functional value of doing so seems small compared to the inconve- neuroimaging has been criticized for encouraging research nience to readers.
aimed merely at localizing psychological functions, for be- A related criticism concerns the calibration of color ing incapable of testing psychological theories, for assum- scales in images. The criticism is that large differences in ing a modular relation between mental and neural systems, color can mislead the naïve viewer into thinking that the and even for assuming a one-to-one correlation between differences in neural activity are also large, whereas, to use these systems.
Horstman's word, they are in fact "tiny." But do these ef- Localization versus explanation. Neuroimaging has
forts at making different levels of brain activation visible been caricatured as a form of phrenology, with the research make the images, again using Horstman's word, "mislead- goal being simply to associate a psychological function with ing"? Not at all. Again, the analogy with Cartesian graphs a specific part of the brain. It is easy to see how this mis- can help. Consider the graph of average global tempera- understanding could occur, given the archetypal functional tures shown in figure 1. The calibration of the Y-axis spans brain image bearing color splotches affixed to various parts less than two degrees Celsius because this makes the rel- of a structural MRI.
evant relationships among data points salient and enables The philosopher Jerry Fodor has suggested that the readers to glean an accurate sense of both the variability question of where in the brain a given psychological ac-and the trend of the data points. These data could have tivity occurs is scientifically trivial. He writes, "It isn't, af-been plotted on a graph showing a fuller range of tempera- ter all, seriously in doubt that talking (or riding a bicycle SPECIAL REPORT: Interpreting Neuroimages: An Introduction to the Technology and Its Limits or building a bridge) depends on things that go on in the choosing and arranging the order of stimuli and then mea- brain somewhere or other. If the mind happens in space at suring the effects of preceding one stimulus with another, all, it happens somewhere north of the neck. What exactly it is possible to determine the proportion of neurons that turns on knowing how far north?"12 Elsewhere in the same the two stimuli activate in common, in effect, the similar- essay he explains why our alleged obsession with localiza- ity of neural coding between the stimuli. When one knows tion is not just silly but wasteful: "Science is expensive . . the nature of representational similarity, one knows much If you put your money (which is to say, our money) into about the nature of representations.17 Using this approach, the elaborate technology required to establish localizations researchers have been able to distinguish between view- of mental functions by imaging techniques, you almost cer- point-dependent representations of an environmental loca- tainly take it out of other kinds of psychological research. tion (akin to a photograph taken from a particular vantage Likewise in respect of the time and money that is required point) and viewpoint-invariant representations (similar to to train people to do science; graduate students, too, are a a map representation of location).18 limited resource."13 Other decidedly nonphrenological methods of analysis Uttal, whose 2001 book-length critique of neuroimag- take advantage of neuroimaging's ability to give informa- ing was titled The New Phrenology, suggests that the quest tion about the state of the entire brain, which contrasts to localize psychological functions in the brain is a primary, with the necessarily piecewise approach of nonimaging and misguided, goal of neuroimaging research. He notes methods such as single-cell recording and lesion methods. that "a considerable amount of PET [positron emission Analyses of functional connectivity reveal which subset tomography] and fMRI localization has simply confirmed of areas show correlated activity, suggesting that they are some things that we have long known,"14 and he contrasts working together.19 Such analyses pick out functional net- such research with research that actually tries to discover works of areas, which may change depending on the task "how the brain computes, represents, encodes, or instanti- conditions, and hence put strong constraints on the nature ates psychological processes."15 of the organization of cognitive subsystems.
It is hard to disagree with Fodor and Uttal's observa- Localization is also generally beside the point in multi- tion that localization is, in itself, a questionable scientific voxel pattern analysis methods, described elsewhere in this goal. However, most contemporary neuroimaging research report by Aguirre. Indeed, the results of such studies can has other goals. In the early years of PET and fMRI, each often be stated without any reference to anatomical local- method was used to image processes whose brain localiza- ization. These methods enable researchers to detect specific tions were already well known on the basis of lesion stud- mental states and decode how specific stimuli or events ies or single-cell recording in animals. This exercise helped are represented in the brain. None of the foregoing uses of to validate the new methods, testing them in order to see neuroimaging fits the description of "neophrenology."whether they produced the expected localizations. Once Relevance to psychological theory. While anatomical lo-
these localizations were confirmed, confidence in the abil- cation is an intrinsic property of neuroimaging data, the ity of functional imaging to detect regional brain activity use of these data is not confined to the study of localiza-increased. Researchers then began to use the method to in- tion any more than the use of reaction time, the primary vestigate cases in which it was not known which brain areas methodology of cognitive psychology, is confined to the were likely to be recruited for a given psychological func- study of cognitive speed. Both localized activations and tion, as well as to address questions that were not primarily response latencies are used by researchers to test psycho- questions of localization. Of course, studies of localization logical hypotheses. Some critics acknowledge this use of for localization's sake are still occasionally published, just as functional imaging but question whether it has been effec- purely descriptive studies with other methods can be found tive as a means of testing psychological theories.20 As with in any field of science, but the bulk of functional neuroim- the preceding objections, there is validity to this one. Let aging research in the twenty-first century is not motivated us consider the sense in which it is valid as well as the sense by localization per se.
in which it misses the mark.
Further reason to reject the "neophrenology" charge In response to examples of how neuroimaging can con- comes from neuroimaging methods that exploit other fea- firm or disconfirm psychological theories, Max Coltheart tures of activation in addition to location. For example, has provided alternative explanations of specific findings "adaptation paradigms" make use of changes in activ- to show that they are not, in fact, decisive.21 What gets lost ity during the course of perception or cognition, specifi- in the debate is the fact that decisive experiments are not cally the diminution in response to a repeated stimulus or generally possible in psychology, and it would be equally operation when the same set of neurons is reactivated.16 difficult to offer examples of traditional cognitive psychol- This is because neurons "adapt" to stimulation, reacting ogy's success by this criterion.22 The phenomena under less strongly if they have recently been active. By carefully study within cognitive psychology and neuroscience are March-April 2014/ HASTINGS CENTER REPORT Can functional brain imaging contribute to confirming psychological hypotheses in roughly the way behavioral studies do? too complex, and the associated theories have too many The kernel of truth here is that early approaches to the degrees of freedom, to hope for decisive single findings.
design and analysis of functional neuroimaging experi- A fairer and more realistic question is this: Can func- ments were best suited to studying relatively small numbers tional brain imaging contribute to confirming psychologi- of macroscopic regions of activation, ignoring questions of cal hypotheses in roughly the way behavioral studies do? representation within areas or complex interactions among Can functional imaging, experiment by experiment, rule areas. The method of subtraction (again, see Aguirre), out the more straightforward alternative hypotheses and which was first used to show how functional brain imaging leave progressively more complex or strained alternatives can illuminate cognitive processes, assumes that a cognitive in relation to the supported hypothesis? Many fruitful re- process A will have the same neural instantiation whether search programs answer this question in the affirmative. it is accompanied by cognitive process B, C, D, or E.30 In Different examples of issues in cognitive psychology that this way, the subtraction method assumes fixed, context-imaging has helped to resolve have been offered.23 My own independent modules. Of course, analytic approaches such favorite example concerns the relation between mental im- as adaptation designs and network modeling, discussed agery and perception, a long-standing issue in cognitive earlier, show how fMRI can transcend this limitation.
psychology.24 Does the visual system do "double duty" for As Erik Parens pointed out in the course of the meetings perceptual processes and mental images generated from that gave rise to this collection of essays, the history of neu-memory, or is the system used to generate mental images roimaging may be not unlike that of behavioral genetics in from memory independent of the visual system? Measures this regard. In the 1980s and '90s, encouraged by the dis- of localized brain activity as rudimentary as electroenceph- covery of several examples of rare medical diseases that were alogram (EEG) and single-photon emission computed caused by single genes, some researchers set out to discover tomography (SPECT),25 later confirmed with PET and single genes responsible for common psychiatric illnesses fMRI, ruled out a set of alternative explanations that had such as depression, schizophrenia, and autism. In effect, plagued the behavioral approach to this question, with the these behavioral geneticists were also assuming a simple evidence indicating at least some shared representations.26 and modular theoretical framework, with individual genes Of course, the original question was replaced by new ques- responsible for the psychological phenomena of interest. tions, inquiring, for example, more specifically as to which With time and the accumulation of research results show- visual representations are shared with imagery, but this was ing the inadequacy of this assumption, behavioral genetics progress. Recently, Mara Mather and colleagues surveyed moved to more complex models, in which genes exert their a variety of research programs and identified four distinct effects on behavior through complex networks of interac-ways in which localized brain activity can be used to test tion with one another and with the environment. It wasn't cognitive theories.27 the case that the gene as a unit of analysis was useless, but Biasing hypothesis generation. It has also been suggest-
that genes had to be seen in still more complex terms.
ed that the use of imaging constrains the kinds of theories Finally, the concern that brain imaging puts scientific of mind-brain relations that will be devised and tested. The blinders on researchers, allowing them to see only simple concern is not that functional brain imaging is merely an modular systems, is assuaged by the concurrent use of other exercise in localization or that it is incapable of testing psy- methods. Hypotheses to be tested are selected in part based chological hypotheses. Rather, the concern is that it directs on research with other methods of neuroscience and psy- us to think about brain function in the wrong way. First, chology, and results are interpreted in part based on re-it invites us to focus on a subset of the relevant data. The search with these other methods.31problem, writes Uttal, is "the mistaken idea that when all Wanton reverse inference. If there were a one-to-one
lesser peaks are reduced to invisibility by arbitrary scaling, relationship between brain regions and psychological pro- the largest remaining peak represents the sole locale of a cesses, as assumed in strict modular hypotheses of brain particular cognitive process." Second, imaging invites us function, then it would be easy to infer what psychological to frame hypotheses "at the wrong (macroscopic) level of processes are under way at any given moment simply by analysis rather than the (correct) microscopic level."28 The observing which brain regions are active. Given that the result is "misdirected attention and effort": "we are doing brain-mind relation is far more complex than that, with what we can do when we cannot do what we should do."29 a single psychological process of interest typically engag-ing multiple regions and a single region typically involved SPECIAL REPORT: Interpreting Neuroimages: An Introduction to the Technology and Its Limits in the implementation of multiple psychological processes, pattern of brain activation when giving a response more inferring a psychological process from an observed brain closely matches that previously associated with true or de-activity is not straightforward. This has not stopped re- ceptive responses.38 In principle, and with the right base of searchers from attempting to make such inferences, ignor- evidence, these reverse inferences could lead to valid con- ing the intrinsic ambiguity of a single region or pattern of clusions along the lines of "there is a 75 percent chance activation in a single experimental context, considered in that the subject" either wants the product, would respond isolation.
to Prozac, or is telling a lie. In most cases, however, the This type of inference, going from an observation of empirical groundwork needed for these claims is far from brain activation to an inference about the psychological complete. For example, for all we know now, psychological process that caused it, was called "reverse inference" by processes other than lying could produce the "lying pat-Russell Poldrack. The name highlights a difference be- tern" with high probability.39 tween this research practice and the more common, and The multivoxel pattern analysis (MVPA) technique (see less problematic, "forward inference" practice of manipu- Aguirre) is in effect a combination of reverse and forward lating psychological processes and observing resulting brain inference approaches. Reverse inferences can be made with confidence only when one knows the full range of psycho- Although reverse inference has often been criticized, logical processes that could produce a given pattern of ac- it is not, in itself, invalid. If one has done due diligence tivation under the circumstances of the study. In MVPA to ascertain the range of psychological processes that can studies, forward inference is first performed on all of the activate a region under a given set of circumstances, then psychological states that reverse inference will later be used appropriately framed reverse inferences will be possible.33 to select among. When the time comes for reverse infer- Unfortunately, it has often been used wantonly, interpret- ence, it is in effect a very large multiple-choice among the ing a pattern of activation without knowing, or acknowl- psychological states that were the subject of forward infer- edging, the variety of psychological processes that could ence. More open-ended uses of reverse inference, in situa-produce that pattern. A celebrated example of such wan- tions where one does not know all of the subject's possible ton reverse inference appeared in a 2007 New York Times psychological states, would require more exhaustive pro- op-ed written by neuroscientists Marco Iacoboni, Joshua grams of forward inference research to have been complet-Freedman, and Jonas Kaplan during that year's presiden- ed and might yield an answer taking the form of "state A tial primaries. They presented their findings from an fMRI with probability X, state B with probability Y, and so on." study of undecided voters viewing still photos and videos As with many of the other concerns reviewed here, un- of the leading candidates. On the basis of activity in the an- warranted reverse inference is not unique to functional terior cingulate cortex (ACC), known from other studies to neuroimaging. It will affect any technology in which a spe- become active when subjects feel conflicting response ten- cific cause is inferred from the presence of an effect that dencies, the authors concluded that voters felt ambivalence could have resulted from other causes. For example, in the toward Hilary Clinton and were "battling unacknowledged use of conventional polygraphy for lie detection, responses impulses to like" the candidate.34 can be accompanied by autonomic reactions for reasons Several cognitive neuroscientists harshly criticized the other than deception.40 In prostate-specific antigen testing, op-ed, pointing out numerous flaws in the research design elevations of the protein prostate-specific antigen can re- and data analysis35—and emphasizing the wanton use of sult from causes other than a growing tumor.41 In sum, the reverse inference. The problem with the reverse inference problem with reverse inference is not unique to brain im-in this case was that many other psychological states also aging, and reverse inference is not inherently problematic. activate the ACC. Activity in this area has been elicited by The problem is with making a reverse inference without processes as diverse as attention to one's own heartbeat and the relevant knowledge from forward inference.
emotional regulation.36 Many of the commercial applications of functional Neuroimaging's Slippery Statistics brain imaging involve reverse inference. For example, neu-romarketing relies on the ability to infer liking and want-ing from patterns of brain activation.37 Similarly, the use Virtually all scientific research depends on statistical analysis at some critical juncture or other, but func- of functional brain images for diagnosis in some for-profit tional brain imaging research is particularly dependent on psychiatric clinics is also based on the premise that certain statistics. As described earlier by Aguirre, extensive signal patterns of activation can be used to infer the presence of processing and statistical analysis intervene between the certain disorders or the suitability of certain treatments. data acquired from the scanner and the published image. Brain-based lie detection promises to distinguish true from A number of criticisms of functional brain imaging have deceptive responses by determining whether a person's focused on the statistics involved.
March-April 2014/ HASTINGS CENTER REPORT When carried out properly, statistical analyses deepen our understanding of the data and the larger reality from which they were sampled. Statistical inference versus direct observation. Some necessarily limited measurements. Like every other aspect
criticisms concern the sheer amount of statistical analysis of scientific research, statistical analyses can be done well or involved in producing a functional brain image. The use poorly, with objectivity or bias, but the use of statistics per of statistics often involves substituting estimated values for se is not a problem.
raw data. When the very earliest stages of image processing The kernel of truth in the criticisms just discussed is consist of replacing measured signal values with estimated that the extensive use of statistics provides equally extensive values, the result can be viewed as a fabrication, with all the opportunities for error. Two such errors are reviewed in the negative connotations of that term discussed earlier.
next two sections.
In addition, the extensive statistical processing involved Multiple comparisons. Functional brain imaging is sus-
in neuroimaging offers many opportunities to distort ceptible to a particular kind of statistical problem that can scientific evidence and therefore makes some critics sus- exaggerate the reliability of findings. The problem arises picious. For example, neuropsychologist Carlo Umilta is because of the enormous number of statistical tests that quoted as questioning, "Would coloured images be so con- can be carried out with image data. A functional MRI may vincing even if readers knew those images are the result of contain 50,000 voxels (see the essay by Aguirre for expla- an elaborate sequence of ‘cleaning actions,' each one char- nation of "voxel"), and each of those voxels could be the acterized by a not insignificant error probability?"42 More site of an independent statistical test comparing the value extreme skepticism comes from neuroscientist Steven Rose. of the BOLD response measured in that small bit of brain Commenting on Fodor's critique of imaging, he wrote that between the conditions of the experiment. Statistical tests the "images are marvelously seductive, but by the time yield a "significance level," which is the probability that you see them they have been so massaged as to risk being the observed difference between two conditions was due thoroughly misleading."43 Michael Shermer observes that to chance variation alone. Given that experiments are de- "[b]rain images are statistical compilations," and he advises signed so that the hypothesis of interest predicts a "real" his readers to "keep all [these many stages of data analysis] difference—a difference, that is, that would be expected in mind next time you see one of those colorful brain scans. any time the experiment was run and not just on occasions . . [Such images are] highly misleading."44 Even the wide- when it happens by chance—one should employ a fairly ly employed and basic statistical process of averaging can low, and thus stringent, significance level. The conven-seem problematic in the context of imaging: Carole Wade tional cut-off for considering a finding reliable or "real" is offers that the "vexing problem of individual differences in a probability of less than one in twenty that the difference brain anatomy," which may make the "uniqueness of fin- between conditions is due to chance, usually written as gerprints or facial features seem simple by comparison," is "p<0.05." Of course, researchers feel more confident about problematic "when scans from a number of individuals are their conclusions when p<0.01 or p<0.001.
averaged to produce a single image."45 When significance testing is carried out with brain The problem with this criticism is that, while the incor- imaging data, the following problem arises: if we test all rect use of statistics is indeed misleading, there is nothing 50,000 voxels separately, then by chance alone, 2,500 inherently misleading about using statistics. The idea that would be expected to cross the threshold of significance each statistical operation on data from the scanner is a step at the p<0.05 level, and even if we were to use the more away from reality and toward artifice is a misunderstand- conservative p<0.001 level, we would expect 50 to cross the ing. The reality of interest is brain function, not the raw threshold by chance alone. This is known as the problem of data collected by the scanner, and most of the statistical multiple comparisons, and there is no simple solution to it. analyses used are tools to learn more about that reality. For example, if we were to consider as activated only those When carried out properly, statistical analyses deepen our voxels where the difference between conditions achieves a understanding of the data and the larger reality from which significance level of p<0.00001, then we would also be set- they were sampled. This is the whole point of using statisti- ting the bar too high for most real differences to be found, cal methods in any field, from cognitive neuroscience to given realistic limits on the power of our experiments to demography. In other words, statistical methods are used detect effects.
not to mask reality but to better approximate it based on HASTINGS CENTER REPORT S25 One vivid demonstration of the multiple comparison number of independent tests carried out on the data, and problem was published by Craig Bennett and colleagues with the help of some entertainingly good luck concerning in an article titled "Neural Correlates of Interspecies the location of the spurious differences, the dead salmon's Perspective Taking in the Post-mortem Atlantic Salmon," brain was found to have regions engaged in perspective-in which a dead fish was placed in an MRI scanner and taking at the p<0.001 level of significance.
instructed to think about the emotions being experienced Statisticians have developed solutions to the problem by people depicted in photographs.46 Thanks to the large of multiple comparisons. These include limiting the so- Criticisms of functional brain imaging
Intended Scope
Objects of Imaging
Blood not brain
Kernel of truth: Neural activity per se is not imaged; the relation of BOLD to neural activity is fMRI Kernel of truth: Data representation requires Metatheoretical Assumptions, Goals
Localization vs. explanation
Kernel of truth: Some studies are aimed at localization per se, often for validation of method.
Cannot test psychological Kernel of truth: There is no single decisive Bias for modular hypotheses All fMRI Kernel of truth: Early approaches to image Wanton reverse inference Particular studies Unqualified truth: Without appropriate (often Statistics
Statistics not reality
Kernel of truth: "Reality" is represented only Multiple comparisons Particular studies Unqualified truth: Images lend themselves to large spurious results if the tests are not appropriately Unqualified truth: Researchers must select regions or subsets of data by means independent of the Influence
Overly convincing
Kernel of truth: This phenomenon has been observed but does not appear robust; the aura of Overly appealing Kernel of truth: This phenomenon is intuitively plausible, but there is little evidence to support it. March-April 2014/ HASTINGS CENTER REPORT Functional neuroimages have been called "convincing" and "gorgeous." Two prominent worries are that the images are too convincing and too appealing. called family-wise error rate and false discovery rate, both As with the problem of multiple comparisons, the prob- of which were tried by Bennett and colleagues and which lem of circularity is not unique to functional neuroimag-resulted in a finding of no activation in the dead salmon's ing. In the words of Vul and Harold Pashler, "Variants of brain (this part of their article received less attention than this problem seem to arise in every field that takes on the the part reporting perspective-taking activity in the dead considerable challenge of identifying and quantifying sig-salmon's brain.) Another legitimate tack around this prob- nals found in massively multivariate data, where one can- lem is to use a priori regions of interest—that is, to simply not ascertain in advance where the signals of interest may limit the number of comparisons by specifying in advance lie."51 They cite psychometrics, epidemiology, genetics, and the regions relevant to the research hypothesis. Still other finance as examples of fields in which circular analyses have methods include reduction of the number of independent distorted the results of research.
tests, in light of the dependence among voxels, and per-mutation analyses.47 In short, functional neuroimaging re- The Undue Influence of Brain Images search is not doomed to produce spurious results because of the problem of multiple comparisons. Most neuroimag-ing articles from the last decade avoid the error highlighted Functional neuroimaging has also been criticized as un- duly persuasive or appealing. Although this criticism is by the dead salmon study, but not all do, which is why aimed at the public's lack of scientific literacy rather than Bennett and colleagues published their study. Finally, the at imaging per se, it figures in many of the criticisms of problem of multiple comparisons is not unique to neuro- neuroimaging cited earlier. Recall the earlier references to imaging. For example, epidemiology has its own reductio images as "colorful," "gorgeous," and "convincing." Two ad absurdum demonstration of the need to properly man- worries in particular have been prominent: that images are age multiple comparisons, analogous to the dead fish study: too convincing and that they are too appealing.
a study relating health to astrological signs.48 Overly convincing. Matthew Crawford refers to brain
Circularity. A related problem was pointed out by imaging as "that fast-acting solvent of critical faculties,"52
Ed Vul and colleagues in a paper titled "Puzzlingly High and Steven Poole writes that brain images, "like religious Correlations in fMRI Studies of Emotion, Personality, and icons, inspire uncritical devotion."53 Others have ex- Social Cognition,"49 which was widely discussed online and pressed concern about the persuasive power of images in in the print media under its original and more pointed ti- applied contexts. One group of researchers, for example, tle, "Voodoo Correlations in Social Neuroscience."50 These has noted the social harms that could ensue from "the mis- critics identified a circularity in the way some published taken impression that fMRI, in particular, is an infallible findings had been analyzed. Some researchers first identi- mind-reading technique that can be used to establish guilt fied the voxels most activated by their experimental task or innocence, infer ‘true intentions,' detect lies, or estab-and then—with the same data set—carried out analyses lish competency to drive, vote, or consent to marriage."54 only on those voxels to estimate the strength of the effect. Others warn of "the potential for brain scan images to cre- Just as differences due to chance alone inflate the un- ate biases in the laboratory, the clinic and the courtroom."55 corrected significance levels in the dead fish experiment, Evidence of the outsized persuasive power of brain im- differences due to chance alone contribute to the choice aging was obtained by David McCabe and Alan Castel, of voxels selected for the second analysis step. The result is who assessed the effects of functional brain images on per- that the second round of analyses is performed on data that ceptions of the quality of cognitive neuroscience research. have been "enriched" by the addition of chance effects that Using both fictional research descriptions and a real BBC are consistent with the hypothesis being tested. In their sur- science news article, they documented higher ratings of vey of the social neuroscience literature, Vul and colleagues credibility when the texts were accompanied by function- found many articles reporting significant and sizeable cor- al brain images compared to bar charts or topographical relations with proper analyses, but they also found a large maps of scalp-recorded EEGs.56 Although heavily cited in number of articles with circular methods that inflated the the years since it was published, the findings have not been correlation values and accompanying significance levels.
replicated, and indeed the recent failure to replicate them SPECIAL REPORT: Interpreting Neuroimages: An Introduction to the Technology and Its Limits at four different laboratories casts doubt on the phenom- ence of equations also enhances evaluations of scientific Studies of the role of brain images in legal decision- making have also failed to demonstrate special influence, The Baby and the Bathwater although clinical and neuroscience evidence that gives jurors relevant information does of course have an effect. Mock jurors in one study were more likely to render a ver- Functional brain imaging has been subject to many criti- cisms in its first three decades as a method of psychol- dict of "not guilty by reason of insanity" if defendants had ogy research, summarized in table 1. This is appropriate, a prior history of psychiatric disorder or neurological dam- given how thoroughly the use of imaging has transformed age and the jury was shown brain images.58 However, the the field of psychology. As functional brain imaging is brain images were always accompanied by additional writ- taken up by various applied disciplines outside of psychol- ten testimony in this study, so the influential factor may ogy research—for example, clinical psychiatry diagnosis or have been the testimony and not the brain images per se. lie detection in legal and other contexts—the stakes grow Another study, assessing the effects of various types of lie even higher. Inferences based on functional brain imaging, detection evidence, found that participants rendered more whether for basic science or applications, require scrutiny.
guilty verdicts when fMRI evidence was described than As we apply such scrutiny, it is important to distinguish when polygraphs, thermal face imaging, or no lie detection between specific criticisms of particular applications or spe- method was described (although the effect disappeared cific studies and wholesale criticisms of the entire enter-when the lie detection methods were criticized in a cross prise of functional neuroimaging. In the first category are examination).59 Note that in this case, brain images were criticisms aimed at improving the ways in which imaging not shown, and there was differing information associated experiments are designed and the ways in which their re- with the lie detection conditions (for example, activation sults are interpreted. Uncontrolled multiple comparisons, of frontal lobes for fMRI, rise in facial temperature for circular analyses and unconstrained reverse inferences are thermal imaging). The finding speaks to jurors' views of serious problems that undermine the inferences made from neuroscience evidence, but its relevance to brain images brain imaging data. Although the majority of research is more specifically remains unclear. Two other recent studies not compromised by any of these errors, a substantial mi- of juror decision-making and brain images offer additional nority of published research is, making such criticisms both evidence,60 although neither study found an effect of brain valid and useful.
images over and above information delivered verbally.
In contrast, the more sweeping criticisms of functional Overly appealing. Even if brain images do not routinely
imaging concern the method itself and therefore cast doubt persuade viewers of conclusions they might otherwise ques- on the conclusions of any research carried out with imag- tion, some have worried that brain images are so attractive ing, no matter how well designed and carefully executed. or fascinating that they garner more than their fair share These more wholesale criticisms invoke the hemodynamic of attention and resources and so crowd out other, more nature of the signal being measured, the association of neu-worthy science. As Paul Bloom put it, "Psychologists can roimaging with modular theories of the mind, the statisti-be heard grousing that the only way to publish in Science cal nature of brain images, and the color schemes used to or Nature is with pretty color pictures of the brain. The make those images seductively alluring. As mentioned ear-media, critical funding decisions, precious column inches, lier, each of these criticisms contains an element of truth, tenure posts, science credibility and the popular imagina- but overextends that element to mistakenly cast doubt on tion have all been influenced by fMRI's seductive but de- the validity or utility of functional neuroimaging research ceptive grasp on our attentions."61 as a whole. None of the criticisms reviewed here consti- Little evidence has been brought to bear on this claim, tute reasons to reject or even drastically curtail the use of however. In one experiment, brain images led laypersons neuroimaging. Rather, they remind us that neuroimaging, to rate newspaper-style research descriptions as being more like other scientific methods, is subject to various specific interesting, compared to descriptions accompanied by errors that the self-correcting process of science continues photographs, but brain images have not been found to per- form better than photographs at making the descriptions seem more worthy of funding.
In sum, although brain imaging seems to lack the dan- 1. M. J. Posner and M. E. Raichle, Images of the Mind, rev. ed. gerous allure attributed to it by some critics, neuroscience (New York: W.H. Freeman & Company, 1997).
more generally may command credibility and interest, pos- 2. A. L. Roskies, "Are Neuroimages Like Photographs of the Brain?" Philosophy of Science 74 (2007): 860-72.
sibly more than is warranted under some circumstances.63 3. V. G. Hardcastle and C. M. Stewart, "What Do Brain Data This is not unique to neuroscience. For example, the pres- Really Show?" Philosophy of Science 69 (2002): 572-85.
March-April 2014/ HASTINGS CENTER REPORT 4. J. Horstman, The Scientific American Brave New Brain (San 23. See Henson, "What Can Functional Neuroimaging Tell the Francisco CA: Jossey-Bass, 2010), p. 81.
Experimental Psychologist?"; J. Jonides, D. E. Nee, and M. G. 5. W. R. Uttal, Mind and Brain: A Critical Appraisal of Cognitive Berman, "What Has Functional Neuroimaging Told Us about the Neuroscience (Cambridge, MA: MIT Press 2011), p. 31.
Mind? So Many Examples, So Little Space," Cortex 42 (2006): 414- 6. N. K. Logothetis, "What We Can and Cannot Do with fMRI," 17; M. Mather, N. Kanwisher, and J. Cacioppo, "How fMRI Can Nature 453 (2008): 869-77; A. D. Ekstrom, "How and When the Inform Cognitive Theories," Perspectives on Psychological Science 8 fMRI BOLD Signal Relates to Underlying Neural Activity: The (2013): 108-13; E. E. Smith and J. Jonides, "Working Memory: A Danger in Dissociation," Brain Research Reviews 62 (2010): 233-44.
View from Neuroimaging," Cognitive Psychology 33 (1977): 5-42; 7. Logothetis, "What We Can and Cannot Do with fMRI," p. C. Umilta, "Localization of Cognitive Functions in the Brain Does Allow One to Distinguish between Psychological Theories," Cortex 8. M. Shermer, "Why You Should Be Skeptical of Brain Scans," 42 (2006): 399-401.
Scientific American Mind 19, no. 5 (2008): 66-71, p. 69.
24. R. A. Finke, "Levels of Equivalence in Imagery and Perception," 9. Horstman, The Scientific American Brave New Brain, p. 81.
Psychological Review 87, no. 2 (1980): 113-32.
10. C. Wade, "Some Cautions about Jumping on the Brain 25. M. J. Farah, "Is Visual Imagery Really Visual? Overlooked Imaging Bandwagon," American Psychological Society Observer 19, Evidence from Neuropsychology," Psychological Review 95 (1988): no. 9 (2006): 23-24.
11. M. Rossner and K. M. Yamada, "What's in a Picture? The 26. Farah, "Is Visual Imagery Really Visual?"; S. M. Kosslyn et Temptation of Image Manipulation," Journal of Cell Biology 166 al., "Visual Mental Imagery Activates Topographically Organized Visual Cortex: PET Investigations," Journal of Cognitive Neuroscience 12. J. Fodor, "Diary: Why the Brain?" London Review of Books 21, 5 (1993): 263-87; M. D'Esposito et al., "A Functional MRI Study no. 19 (1999): 68-69.
of Mental Image Generation," Neuropsychologia 35 (1997): 725-30.
27. Mather et al., "How Can FMRI Inform Cognitive Theories?" 14. W. R. Uttal, The New Phrenology: The Limits of Localizing 28. Uttal, Mind and Brain, p. 366.
Cognitive Processes in the Brain (Cambridge, MA: MIT Press, 2001), 29. Ibid., p. 369.
30. Posner and Raichle, Images of Mind.
15. Ibid, p. 217.
31. To obtain an indication of the methodological segregation or 16. See K. Grill-Spector and R. Malach, "fMR-adaptation: A Tool integration of functional neuroimaging research, the references cited for Studying the Functional Properties of Human Cortical Neurons," by the last twenty fMRI articles published in Science as of May 21, Acta Psychologica 107 (2001): 293-321.
2013, were obtained and the literature cited therein was examined 17. For example, if "bear" and "pear" are more similar than "pear" for research methods other than imaging. In all cases, the articles and "grape," then phonological or orthographic representations are cited work outside of functional neuroimaging.
likely in use; if "pear" and "grape" show the greater similarity, then 32. R. A. Poldrack, "Can Cognitive Processes Be Inferred from semantic or conceptual representations are likely in use.
Neuroimaging Data?" Trends in Cognitive Sciences 10 (2006): 59-63.
18. R. A. Epstein, K. S. Graham, and P. E. Downing, "Viewpoint: Specific Scene Representations in Human Parahippocampal Cortex," 34. M. Iacoboni, J. Freedman, and J. Kapla, "This Is Your Brain Neuron 37 (2003): 865-76.
on Politics," New York Times, November 11, 2007.
19. See M. D. Greicius et al., "Functional Connectivity in 35. See A. Aron et al, "Politics and the Brain," New York Times, the Resting Brain: A Network Analysis of the Default Mode November 14, 2007, http://www.nytimes.com/2007/11/14/opin- Hypothesis," Proceedings of the National Academy of Sciences 100 ion/lweb14brain.html; M. J. Farah, "This Is Your Brain on Politics," (2003): 253-58. See J. Zhuang et al., "Connectivity Exploration Neuroethics & Law Blog, November 12, 2007, accessed February with Structural Equation Modeling: An fMRI Study of Bimanual Motor Coordination," NeuroImage (2005): 462-70; A. Roebroeck, is-your-br.html; G. K. Aguirre, "The Political Brain," Cerebrum E. Formisano, and R. Goebel, "Mapping Directed Influence over the (blog), September 12, 2008, http://www.dana.org/Cerebrum/2008/ Brain Using Granger Causality and fMRI," Neuroimage 25 (2005): The_Political_Brain/. 230-42; K. J. Friston, L. Harrison, and W. Penny, "Dynamic Causal 36. See H. D. Critchley et al., "Human Cingulate Cortex and Modeling," Neuroimage 19 (2003): 1273-302; S. A. Huettel, A. W. Autonomic Control: Converging Neuroimaging and Clinical Song, and G. McCarthy, Functional Magnetic Resonance Imaging, Evidence," Brain 126 (2003): 2139-52; G. Bush, P. Luu, and M. I. 2nd ed. (Sunderland, MA: Sinauer Associates, 2009), chapter 11.
Posner, "Cognitive and Emotional Influences in Anterior Cingulate 20. See M. Coltheart, "What Has Functional Neuroimaging Told Cortex," Trends in Cognitive Sciences 4, no. 6 (2000): 215-22.
Us about the Mind (So Far)?," Cortex 42 (2006): 323-33, and G. van 37. D. Ariely and G. S. Berns, "Neuromarketing: The Hope and Orden and K. R. Papp, "Functional Neuroimages Fail to Discover Hype of Neuroimaging in Business," Nature Reviews Neuroscience 11 Pieces of Mind in Parts of the Brain," Philosophy of Science Proceedings (2010): 284-92.
64 (1997): S85-S94.
38. M. J. Farah, B. Hutchinson, E. A. Phelps, and A. D. Wagner, 21. Coltheart, "What Has Functional Neuroimaging Told Us?" "fMRI Lie Detection: Scientific and Societal Challenges," Nature For a thoughtful explication of the ways in which neuroimaging can Reviews Neuroscience 15 (2014): 123-31, doi:10.1038/nrn3665. inform cognitive theory, including the point that no single find- ing in psychology will ever be a "Popperian ‘falsification bullet,'" 40. L. Saxe and G. Ben-Shakhar. "Admissibility of Polygraph Tests: see also R. Henson, "What Can Functional Neuroimaging Tell The Application of Scientific Standards Post-Daubert." Psychology, the Experimental Psychologist?," Quarterly Journal of Experimental Public Policy, and Law 5, no. 1 (1999): 203-23. Psychology 58A (2005): 193-233.
41. I. M. Thompson et al., "Prevalence of Prostate Cancer among 22. Henson, "What Can Functional Neuroimaging Tell the Men with a Prostate-Specific Antigen Level ≤4.0 ng per Milliliter," Experimental Psychologist?"; C. Klein, "Philosophical Issues in New England Journal of Medicine 351 (2004): 2239-46.
Neuroimaging," Philosophy Compass 5 (2010): 186-98.
SPECIAL REPORT: Interpreting Neuroimages: An Introduction to the Technology and Its Limits 42. As quoted in D. Ramani, "The Brain Seduction: The Public 56. D. P. McCabe and A. D. Castel, "Seeing Is Believing: The Perception of Neuroscience," Journal of Science Communication 84, Effect of Brain Images on Judgments of Scientific Reasoning," no. 4 (2009): 1-8.
Cognition 107 (2008): 343-52.
43. Fodor, "Why the Brain?" 57. D. Gruber and J. A. Dickerson, "Persuasive Images in 44. Shermer, "Why You Should Be Skeptical," pp. 70-71.
Popular Science: Testing Judgments of Scientific Reasoning and 45. Wade, "Some Cautions about Jumping on the Brain Imaging Credibility," Public Understanding of Science 21 (2012): 938-48; C. J. Hook and M. J. Farah, "Look Again: Effects of Brain Images 46. C. M. Bennett et al., "Neural Correlates of Interspecies and Mind-Brain Dualism on Lay Evaluations of Research," Journal Perspective Taking in the Post-Mortem Atlantic Salmon: An of Cognitive Neuroscience (early access, April 22, 2013, doi:10.1162/ Argument for Proper Multiple Comparisons Correction," Journal of jocn_a_00407); R. B. Michael et al., "On the (Non)Persuasive Power Serendipitous and Unexpected Results 1 (2010): 1-5.
of a Brain Image," Psychonomic Bulletin & Review 20, no. 4 (2013): 47. See Huettel, Song, and McCarthy, Functional Magnetic 720-25; see also M. J. Farah and C. J. Hook, "The Seductive Allure of ‘Seductive Allure,'" Perspectives on Psychological Science 8 (2013): 48. P. C. Austin et al., "Testing Multiple Statistical Hypotheses 88-90, for a discussion of the McCabe and Castel study.
Resulted in Spurious Associations: A Study of Astrological Signs and 58. J. R. Gurley and D. K. Marcus, "The Effects of Neuroimaging Health," Journal of Clinical Epidemiology 59 (2006): 964-69.
and Brain Injury on Insanity Defenses," Behavioral Sciences & The 49. E. Vul et al., "Puzzlingly High Correlations in fMRI Studies Law 26, no. 1 (2008): 85-97.
of Emotion, Personality, and Social Cognition," Perspectives on 59. D. P. McCabe, A. D. Castel, and M. G. Rhodes, "The Psychological Science 4 (2009): 274-90.
Influence of fMRI Lie Detection Evidence on Juror Decision- 50. See S. Begley, "Of Voodoo and the Brain," Newsweek, January Making," Behavioral Sciences & the Law 29, no. 4 (2011): 566-77.
51. E. Vul and H. Pashler, "Voodoo and Circularity Errors," 61. P. Bloom, "Seduced by the Flickering Lights of the Brain," Neuroimage 62 (2012): 945-48, at 946.
Seed, June 26, 2006, http://seedmagazine.com/content/article/se- 52. M. B Crawford, "The Limits of Neuro-talk," The New Atlantis 19 (2008): 65-78, at 65.
62. Hook and Farah, "The Seductive Allure." 53. S. Poole, "Your Brain on Pseudoscience: The Rise of Popular 63. D. S. Weisberg et al., "The Seductive Allure of Neuroscience Neurobollocks," New Statesman (September 6, 2012), at paragraph Explanations," Journal of Cognitive Neuroscience 20 (2008): 470- 77; A. K. Lindell and E. Kidd, "Consumers Favor ‘Right Brain' 54. S. B. Johnson, R. W. Blum, and J. N. Giedd, "Adolescent Training: The Dangerous Lure of Neuromarketing," Mind, Brain, Maturity and the Brain: The Promise and Pitfalls of Neuroscience and Education 7 (2013): 35-39.
Research in Adolescent Health Policy," Journal of Adolescent Health 64. A. D. Sokal, "A Physicist Experiments with Cultural Studies," 45, no. 3 (2009): 216-21, at 219.
accessed July 5, 2013, http://www.physics.nyu.edu/faculty/sokal/lin- 55. J. Illes et al., "Neurotalk: Improving the Communication of gua_franca_v4/lingua_franca_v4.html; K. Eriksson, "The Nonsense Neuroscience Research," Nature Reviews Neuroscience 11 (2009): 61- Math Effect," Judgment and Decision Making 7 (2012): 746-49.
March-April 2014/ HASTINGS CENTER REPORT

Source: https://tu-dresden.de/mn/psychologie/allgpsy/ressourcen/dateien/lehre/lehreveranstaltungen/Lehre-Kroenke-Seminar-KN/Farah_2014.pdf