Designing Assessments in Physical Science to Align with Three Dimensional Learning

Project Overview

The NGSA Physical Science project is a 4-year, $757,335 project funded by the National Science Foundation (NSF). As part of the larger Next Generation Science Assessment endeavor, the team is using an evidence-centered design approach aligned with the Next Generation Science Standards (NGSS) to develop assessments for middle school classrooms. The assessment tasks can be used by teachers to gauge student progress towards meeting the performance expectations set forth by NGSS.

 

Big Ideas


NGSS and the Framework

The Next Generation Science Standards (NGSS) are very different from the previous generation of science standards. They require learners to use scientific practices (such as asking questions and defining problems) to develop and apply core ideas (such as those concerning forces and motion) and crosscutting concepts (such as cause and effect). This new idea of 3-dimensional learning is significantly different from traditional ways of learning.

While NGSS marks a new era for science teaching, there are currently very few supports for teachers to put NGSS ideas into practice. Assessment is an essential piece of the puzzle, but the challenge is how to replace the traditional test items that don’t align with NGSS. What is tested often influences what teachers decide to teach and what they hold their students accountable for knowing. Even if standards are closely aligned with a desired vision of teaching and learning, achieving a coherent system is undermined when assessments are not aligned with that vision.

This project helps meet the critical need for high-quality assessments that align with the performance expectations in the new standards and are instructionally suitable for teachers to use in classrooms. Teachers, administrators, school boards, and others need high quality, research-based examples of innovative assessments developed specifically to support their transition to the new standards.


Addressing Multiple Needs

The goal of this project is to address this urgent need for research-based assessment. The team will develop next-generation assessments that address a specific core idea in physical science—matter and its interactions—by integrating middle school chemistry content with two important scientific practices, constructing explanations and developing and using models. The team selected these core ideas because they help to explain and predict a wide variety of phenomena that occur in the everyday lives of students, and because a solid understanding of matter and its interactions is important to other science disciplines.

The new assessments are being iteratively designed and administered in middle school science classrooms to ensure their usability. They will also be administered across grade levels to allow the research team to explore how students progress over time in their understanding of science practices and their knowledge of content. Researchers will work in both NGSS-focused classes and classes with a more traditional curriculum. The Investigating and Questioning our World through Science and Technology (IQWST) curriculum was an earlier CREATE for STEM Institute project that developed curriculum aligned with NGSS for middle school science classes. By testing the assessments in both environments, researchers hope to create assessments that are broadly usable across all types of science classrooms.

Outcomes of the project include: (1) a framework for the principled design and analysis of assessments of student understanding of a core idea across different scientific practices and grade levels, (2) exemplar items and rubrics that can be used formatively in middle school chemistry instruction, and (3) instructional resources for teachers in the formative use of these assessments.

The design framework and exemplar assessments will serve as timely models for the science education and assessment communities about what the next-generation science assessments should look like. The development of an ECD-based assessment design approach for NGSS has already garnered a lot of attention for 3 key stakeholder groups: assessment design and research, science education leadership and research, and science educators.

The science teacher community will especially benefit from the assessment items and rubrics, as well as the guidelines for developing instructional resources that support the formative use of the assessments.

The assessment community will be interested in our design patterns, task templates, assessment items, and rubrics, as well as our application of technology and advanced psychometric models to the measurement of student performance targeted at the NGSS.


Evidence-Centered Design (ECD) Approach

The project will incorporate principles of evidence-centered design (ECD) and state-of the art psychometric methods. Psychometrics is the field of study concerned with the theory and technique of psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The team will use these methods and ECD to develop, test, and analyze their technology-supported assessments for middle school physical science classrooms.

The ECD approach proposed here will provide an optimal framework for integrating the substantive aspects of valid design and the technical infrastructure required to convert evidence gathered from student performances into reliable inferences about student thinking and understanding.


Equity/Fairness and Technology

There is a big push in the science community to be more inclusive. This means that many researchers are working hard to make sure that traditionally underrepresented groups have an equal chance at receiving a quality science education. In the same way, this project team is looking to make sure its assessments are fair.

SRI designed and led an independent expert review process for all of the Learning Performances (LPs) and accompanying assessment tasks. The review explored the comprehensibility, equity, and fairness of all assessment tasks in order to ensure that they are accessible and fair to students of diverse cultural, linguistic, and socioeconomic backgrounds.

Another important commitment to equity is that all of the tasks are offered on a free portal hosted by the Concord Consortium. The fact that they are online means that more districts will have access to them. This also means that developers can implement innovative technological components to help students represent their ideas.

Videos and simulations are incorporated into some of the tasks to create authentic and engaging scenarios and also reduce literacy demands. Stamps and drawing tools are included to help students more easily express ideas and formulate responses. Scaffolds, such as prompts to support students in constructing explanations and building models, are included to help students who are making early steps toward a set of performance expectations.


Formative Assessment

The assessments being developed will provide evidence of a student’s demonstration of increasingly sophisticated understanding related to a core idea and set of practices in the NGSS. Researchers can track the interaction of this core idea with the practices within and across middle school grades 6 to 8. In other words, they will use the assessments to track students’ knowledge of a topic over time.

The team is also working with teachers to understand what kind of rubrics, curriculum guidelines, and other resources they need to use the assessments formatively. Formative assessments help teachers gauge how their students are relating to a certain topic and whether they understand content. So far, teachers have been using the assessment tasks for review after teaching a topic, sometimes with students working in groups or having a full class discussion. Other teachers have even used the tasks as a prior knowledge test to gauge how much students know before launching into a lesson.

The research team has conducted observations of how teachers use the tasks with students, observed how students and teachers interact with the online delivery platform, solicited feedback from teachers about the quality of tasks and what teachers learned about student thinking from using them, and examined students’ responses logged in the online platform. Based on these observations, the tasks have been revised appropriately so that they are as useful as possible for formative assessment.


Dissemination

The team has been actively disseminating their findings and products throughout the course of the project. The multi-institutional team has been showcasing the online task bank to science educators, district and state leaders, assessment developers, and science education researchers. In year three of the project, 17 presentations and workshops were given, and 8 conference papers and posters were created.

One- to two-day professional development workshops were done for the National Science Teachers Association (NSTA), the California Science Project, and the Michigan Mathematics and Science Center Regional Collaborative. As the project enters year four, more workshops are planned. The team has already generated a number of proposals that involve presentation and dissemination of their work at major conferences to be held during 2017, including NSTA, NARST, and AERA. They also plan to disseminate widely a technical report on their design approach in year four.

For more information and the team’s resources, please visit the sites below:

 

The Learning Performances

To meet the goal of developing assessment tasks that can support NGSS-aligned instruction, the team unpacked some NGSS performance expectations into statements called learning performances, which have guided assessment task development for classroom use.

Learning performances are akin to learning goals that take on the structure of the performance expectations. They articulate and integrate assessable aspects of performance that build toward the more comprehensive NGSS performance expectations. A single learning performance describes an essential part of a performance expectation. Together, a set of learning performances provides the detail needed to create a coherent set of assessment tasks covering the full scope of a performance expectation bundle.


Chemical Reaction Topic

LP C-01: Students analyze and interpret patterns in the data to determine whether substances are the same based upon characteristic properties.

LP C-02: Students construct a scientific explanation about whether a reaction has occurred using properties of substances before and after the substances interact.

LP C-03: Students evaluate whether a model uses an appropriately sized scale to explain that different substances are made from different types and/or arrangements of atoms.

LP C-04: Students evaluate whether a model explains that a chemical reaction produces new substances and conserves atoms.

LP C-05: Students use a model to explain that in a chemical reaction atoms are regrouped and why mass is conserved.

LP C-06: Students develop a model of a chemical reaction that explains that new substances are formed by the regrouping of atoms, and that mass is conserved.

LP C-07: Students evaluate whether a model explains that a chemical reaction produces new substances and conserves mass because atoms are conserved.

 

Energy Topic

LP E-01: Students evaluate a model that uses a particle view of matter to explain how states of matter are similar and/or different from each other.

LP E-02: Students develop a model that explains how particle motion changes when thermal energy is transferred to or from a substance without changing state.

LP E-03: Students develop a model to explain the change in the state of a substance caused by transferring thermal energy to or from a sample.

LP E-04: Students use evidence from a simulation to construct a scientific explanation about how the average kinetic energy and the temperature of a substance change when thermal energy is transferred from or to a sample.

LP E-05: Students develop a model that includes a particle view of matter to predict how the average kinetic energy and the temperature of a substance change when thermal energy is transferred from or to a sample.

LP E-06: Students evaluate an investigation procedure that addresses a scientific question about how the type of matter influences the magnitude of temperature change of a given sample when energy is transferred from or to a sample.

LP E-07: Students plan an investigation to answer a scientific question about how the type of matter influences the change in temperature of a given sample when energy is transferred from or to a sample.

LP E-08: Students carry out an investigation using a simulation to determine how mass affects the change in temperature of a given sample when energy is transferred from or to a sample.

LP E-09: Students construct a scientific explanation about how mass affects the change in average particle kinetic energy of a sample when thermal energy is transferred from or to a sample.

 

Partners

For this project, CREATE for STEM Institute at Michigan State University is partnering with the University of Illinois in Chicago (UIC), the Concord Consortium in Massachusetts, and SRI International in California.

Project Leaders:

  • Jim Pellegrino – PI, UIC
  • Louis DiBello – co-PI, UIC
  • Christopher Harris – PI, SRI
  • Angela DeBarger – co-PI, SRI
  • Joe Krajcik – PI, CREATE for STEM, MSU
  • Daniel Damelin – Institution Lead, Concord
  • Cathleen Kennedy – Project External Evaluator
  • Rose Pringle – University of Florida
  • Lynda Hayes – Director of the P.K. Yonge Developmental Research School at the University of Florida

Postdoctoral researchers:

  • Brian Gane
  • Jane Lee
  • Phyllis Pennock
  • Krista Damery

Concord Consortium personnel:

  • Dan Damelin
  • Nathan Kimball
  • Doug Martin

Project advisors/expert reviewers board:

  • Melanie Cooper
  • Carlos Ayala
  • Derek Briggs
  • Ayham Dahi
  • Cathleen Kennedy
  • Xiufeng Liu
  • Becky Matz
  • Ann Novak
  • Stephen Pruitt
  • Ted Willard

Review Panel of LPs and assessment tasks:

  • Melanie Cooper
  • Ayham Dahi
  • Becky Matz
  • Ann Novak
  • Ted Willard

 

Milestones

The project began in September of 2013 and will continue through August of 2017.


Year 3 Accomplishments

The team engaged in seven major activities during Year 3:

  • Completed the development and refinement of all learning performances and assessment tasks
  • Conducted an outside expert review of learning performances and assessment tasks
  • Continued the development work on the scoring approach and rubrics for assessment tasks
  • Added interactive components for use in technology-enhanced assessment tasks and improved the task delivery platform
  • Conducted classroom feasibility studies
  • Increased dissemination of our design approach to research and practitioner communities
  • Implemented a large-scale data collection effort for psychometric analysis of tasks


Review of Tasks

Revisions to the learning performances and tasks were informed by an expert panel review, an internal fairness/equity review, and by the team’s classroom-based research. Overall, teacher feedback suggested that the variety of tasks provided valuable information regarding the integrated content/practice proficiencies of students. Also, UIC designed and conducted a task performance study in which over 1000 middle school students from 3 diverse school districts completed the assessment tasks. Performance data on all 65 tasks will be analyzed going into Year 4 to inform final revisions.


Key Outcomes So Far

  • Completion of learning performances and assessment tasks aligned to the middle school physical science performance expectations for chemical reactions and energy
    • 16 LPs and 65 tasks (available on the project website)
  • All tasks implemented in a web-based portal hosted by Concord Consortium. Available to teachers and students via web browsers and across devices and operating systems.


Spreading the Word

The project team has been actively disseminating their results throughout the course of the project through workshops and MSTA, NSTA, and NARST conferences. Other parts of the effort include a 1-2 day professional development workshop for districts and science teacher organizations, presentations to state education agencies and at national education meetings, and an assessment workshop for the Council of State Science Supervisors.

So far, teachers and school district leaders have shown active interest in the project’s work because it is helping them adjust to the new Michigan Science Standards (MSS) adopted from the NGSS. Two school districts are actually already using the developed assessment tasks district-wide.


Goals for Year 4

  • Finalize the scoring approach and rubrics
  • Develop guidelines and materials for effective teacher use of assessment tasks
  • Organize and prepare the data collected from our performance study and conduct full analysis of data
  • Meet with advisory board for guidance on developing resources for practitioners and on dissemination for broad impact
  • Disseminate our design approach, tasks, rubrics, and teacher guide to researcher and practitioner communities via multiple means
  • Finalize the delivery platform and reporting system

 

Design Principles

The activities that have guided the development of the materials include:

  • Use ECD to develop items
  • Conduct an alignment study
  • Perform a cognitive analysis study
  • Conduct student performance studies (early-stage testing, a pilot study, and a main study)
  • Investigate formative use of the items

Researchers also made use of design patterns to create the assessment tasks. Design patterns include lists of assessment attributes specified by the assessment designers, working together with content specialists. Each design pattern describes an assessment argument by identifying the focal knowledge, skills, and abilities (KSAs) that are to be measured, the kinds of observations that can provide evidence of this knowledge or skill, and the features of tasks that enable students to provide this evidence. Also specified in the design patterns are any non-focal KSAs that may be required to respond correctly to the tasks. For example, reading comprehension and decoding skills are needed to respond to a science problem.

Design patterns also capture ways in which assessment tasks can be varied to increase or decrease demands for knowledge and specify the work products and rubrics that the assessment designer may want to use.

 

Links to the Next Generation Science Standards

The materials will help students build competency in the performance expectations defined in the Next Generation Science Standards (NGSS). The assessment tasks target the following performance expectations from NGSS:


Chemical Reaction Topic

  • MS-PS1-1. Develop models to describe the atomic composition of simple molecules and extended structures.
  • MS-PS1-2. Analyze and interpret data on the properties of substances before and after the substances interact to determine if a chemical reaction has occurred.
  • MS-PS1-5. Develop and use a model to describe how the total number of atoms does not change in a chemical reaction and thus mass is conserved.


Energy Topic

  • MS-PS1-4. Develop a model that predicts and describes changes in particle motion, temperature, and state of a pure substance when thermal energy is added or removed.
  • MS-PS3-4. Plan an investigation to determine the relationships among the energy transferred, the type of matter, the mass and the change in the average kinetic energy of the particles as measured by the temperature of the sample.

 

Connections to other CREATE for STEM projects

The NGSA Physical Science project is one of several CREATE projects working to create innovative assessments. Another endeavor, the NGSA Life Science project, was inspired by this project. That new grant focuses on designing assessment tasks and rubrics for middle school life science classrooms.

Much of the work done in this project, including the unpacking of performance expectations and 3-dimensional learning concepts, is being used again in the Life Science Assessments project. This work is an asset to the science education community, as the need for Next Generation Science Assessments is great.


To read more about the NGSA Life Science project, please click here: http://create4stem.msu.edu/project/life-sciences-assessment.