Kevin C. Haudek, Rosa A. Moscarella, Michele Weston, John Merrill and Mark Urban-Lurain
Construction of Rubrics to Evaluate Content in Students’ Scientific Explanation Using Computerized Text Analysis
A major challenge to using constructed response items in high enrollment undergraduate STEM courses is the ability to evaluate the scientific content of student explanations, which can be aided using evaluation rubrics. However, traditional rubric development is highly qualitative and is often an iterative process and requires subject expertise and the ability to identify emerging themes from student writing. Here, we report on leveraging the results of lexical and statistical analysis of students’ writing to develop a rubric to evaluate scientific content contained in student responses to two questions used in undergraduate biology courses. Lexical analysis was used to identify and categorize relevant content in student responses. These lexical categories were used as variables in K-means clustering, which helped identify emergent themes or patterns across student responses. Each cluster was defined by categories that included relevant disciplinary content. These clusters were used as the basis of an initial rubric. Several raters applied this rubric to a subset of student responses and after revisions to the rubric and scoring iterations, achieved varying levels of inter-rater reliability (from 0.4 to 0.8 Cohen’s kappa) on different rubric criteria. We believe this methodology may be broadly useful for reducing the effort necessary for rubric creation.