At VSS2014, Isabel presented a short talk in the exciting and dynamic Individual differences brown bag organized by Jeremy Wilmer. In two minutes, she could do little more than advertise two recent studies from the lab which should be very useful to anyone interested in measuring the failures of selective attention that are a facet of holistic processing. Cognitive psychologists often do not pay as much attention to issues of measurements as their colleagues in other fields of Psychology. Papers that focus on reliability or the magnitude of effect sizes are rare in our field. But like others working on high-level vision who have become recently more interested in individual differences, we have decided to invest more efforts into understanding our measures and so far, it has paid off. We hope this work will be useful to others as well.
Holistic processing is a hallmark of face recognition, and we focused on the composite task, using what Gauthier and Bukach (2007) called the “complete design” version of this paradigm (because it has ALL the possible congruent and incongruent trials rather than half of them as in what we called the “partial design” version of this task). In a recent review and meta-analysis of holistic face processing published in Psychological Bulletin, we explain the difference between the two tasks and present empirical evidence from many studies that the two tasks do not measure the same construct, that the partial design is correlated with response bias across studies (in addition to across subjects as we have shown before). We report on the meta-analytical effect size for the composite task, demonstrating that it is very sensitive in group studies, and offer some advice for maximizing reliability in individual difference studies. Indeed, in another paper recently accepted by Behavioral Research Methods, we quantify the reliability of measurements in this task across several studies, using both the standard subtraction approach, as well as the regression method advocated by Joe DeGutis and colleagues. We show the task often has low reliability regardless of the method used, and we achieve more acceptable levels of reliability using two different manipulations. Together, the two papers should provide a good example for readers to appreciate how the same task can have a large effect size for group studies and low reliability for individual differences (a situation that cognitive psychologists are not accustomed to thinking about, but which we suspect is not rare).
As a preview of things to come, we (Richler & Gauthier) have recently submitted a paper on a new test designed especially for the purpose of quantifying holistic process across individuals, which we argue is preferable on several fronts including reliability. As soon as the paper is accepted, we will make this test available to other scientists. UPDATE: the paper on the Vanderbilt Holistic Face Processing Test is now in press at JOV. Please contact us if you would like to use the test – we expect to have a JAVA version of it very soon, but we can share another version of the test in the mean time.