To What Extent do Blinding and Random Assortment Impact Qualitative Data Analysis?
Description of the talk:
Qualitative data analysis contains some degree of error, and any research group that performs qualitative research should be mindful of sources and consequences of bias. Two possible ways to reduce bias are blinding (that is, removing as much potentially biasing information from the content that is being coded as possible) and random assortment (not sorting content by any known factors, such as type of treatment). We tested the effects of blinding and random assortment using 6,000 short answer undergraduate student responses to three biology questions. We tested four treatments (1,500 responses per treatment): not blinded and not randomly assorted, blinded and not randomly assorted, not blinded and randomly assorted, and blinded and randomly assorted. One coder coded all responses using a predetermined holistic rubric and found that not blinding and not randomly assorting biased her coding. To test whether this was a novel observation, three coders including the initial coder coded the same student responses with the same treatments. This time, the coding condition (blind/not blind; random/sorted) had little to no effect on how these three coders coded the responses. We were quite surprised at the results of this study. The purpose of this Work-in-Progress talk is to first present more detail about the study and then to discuss the possible meaning behind the results. Attendees may also reflect on their own data analysis and ways that they may reduce bias effects.