The Importance of Random Assortment and Blinding in Qualitative Data Analysis

TitleThe Importance of Random Assortment and Blinding in Qualitative Data Analysis
Publication TypePoster
Year of Publication2016
AuthorsBierema, AM-K, Moscarella, RA, Urban-Lurain, M, Merrill, JE, Haudek, KC
Date Published04/2016
KeywordsAACR, bias, qualiative analysis
AbstractQualitative data analysis contains some degree of error, and any research group that performs qualitative research should be aware of sources of bias. The Automated Analysis of Constructed Response (AACR) research group investigates computerized analysis of undergraduate students’ constructed responses in science and statistics. In our development of computer models, increasing the number of human-coded responses enhances the accuracy of these models but analysis techniques that decrease coding time, such as confirming computer-predicted codes instead of coding without that information, can inject new sources of bias. We tested whether having the computer-predicted codes visible (i.e., no blinding) and having the responses in the order of the computer codes (i.e., no random assortment) created bias in human coding. In the initial investigation, one coder coded 2,000 responses to a three-part constructed response question and found an effect from both no blinding and no random assortment. To test whether this finding was a novel observation, three coders including the initial coder replicated the study with experimental improvements. Coders were aware of the overall pattern of the initial coder’s responses. Contrary to the previous findings, random assortment and blinding had little to no effect on bias. Our results suggest that blinding and random assortment may be effective methods for reducing bias in coding student responses but may be less important when other methodological aspects are rigorous.
Location of meetingBaltimore, MD
Date(s) of meeting / presentationApril 14-17, 2016
Citation Key4387

Attachments: 

thumbnail of small NSF logo in color without shading

This material is based upon work supported by the National Science Foundation (DUE grants: 1438739, 1323162, 1347740, 0736952 and 1022653). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.