Designing a Scenario-Based Assessment of Learning for Ethical Reasoning
Faculty Advisor Name
Dr. Jeanne Horst
Department
Department of Graduate Psychology
Description
In this presentation I describe a work in progress, an interactive Scenario-Based Assessment (SBA) for ethical reasoning based on the Eight Key Questions (8KQ) framework developed at James Madison University (JMU). This computer-based instrument complements existing 8KQ assessment approaches, in that it unfolds a scenario in a series of steps in order to provide coaching, cues, and feedback in response to learner choices. A key goal in this approach is for the instrument to serve as a learning experience, as well as an assessment tool. The instrument presents a realistic scenario in which the learner takes the role of a state employee responsible for implementing the state's new energy strategy. In the context of a dramatic plot structure, the learner encounters a series of ethical reasoning situations and learns to use the 8KQ to inform their private deliberations, actions, decisions, and interactions with others.
The proof of concept is based on a set of written scenarios called Energize Virginia, designed by JMU Assessment & Measurement faculty Dr. Brian Leventhal. In collaboration with Ethical Reasoning in Action at JMU, we selected and expanded upon a scenario called "Solar Panel Farm in the Shenandoah Valley." Through a series of workshops, we developed a card-based story crafting method that can be used to quickly design small to large ethical reasoning scenarios in a given topic area. We then used this method in a culminating workshop to design a series of scenes that would serve as the plot framework for the Solar Panel Farm scenario. Finally, we converted these scenes into sequences of selected response items that test a learner's inclination to weigh certain key questions over others as they move through the story.
I will discuss approaches, challenges, and lessons learned in the design and implementation of this instrument with respect to scenario construction, instructional design, and assessment design. Scenario construction includes selecting a facet of the ethical reasoning construct to focus on; selecting a scenario that provides opportunities for ethical reasoning; crafting story scenes; and connecting scenes into a longer scenario. Instructional design includes the application of strategies such as chunking, coaching, scaffolding, cuing, feedback, and use of resources to add value to the instrument as a learning experience. Assessment design includes designing selected response items to guide exploration of the issues raised in each scene; arranging items for consistency and interpretability in assessing learner response patterns; implementing item sequences as a Qualtrics survey capable of real-time scoring and basic adaptive feedback (in progress); and designing data output to support the comparison of various methods for analyzing learner paths and interaction patterns.
The project explores methods for combining instructional design and assessment techniques, assessing complex constructs using text-based survey instruments and interactive story-telling, and designing learning experiences that can be used at an activity or module level while also providing useful data for assessment purposes.
Designing a Scenario-Based Assessment of Learning for Ethical Reasoning
In this presentation I describe a work in progress, an interactive Scenario-Based Assessment (SBA) for ethical reasoning based on the Eight Key Questions (8KQ) framework developed at James Madison University (JMU). This computer-based instrument complements existing 8KQ assessment approaches, in that it unfolds a scenario in a series of steps in order to provide coaching, cues, and feedback in response to learner choices. A key goal in this approach is for the instrument to serve as a learning experience, as well as an assessment tool. The instrument presents a realistic scenario in which the learner takes the role of a state employee responsible for implementing the state's new energy strategy. In the context of a dramatic plot structure, the learner encounters a series of ethical reasoning situations and learns to use the 8KQ to inform their private deliberations, actions, decisions, and interactions with others.
The proof of concept is based on a set of written scenarios called Energize Virginia, designed by JMU Assessment & Measurement faculty Dr. Brian Leventhal. In collaboration with Ethical Reasoning in Action at JMU, we selected and expanded upon a scenario called "Solar Panel Farm in the Shenandoah Valley." Through a series of workshops, we developed a card-based story crafting method that can be used to quickly design small to large ethical reasoning scenarios in a given topic area. We then used this method in a culminating workshop to design a series of scenes that would serve as the plot framework for the Solar Panel Farm scenario. Finally, we converted these scenes into sequences of selected response items that test a learner's inclination to weigh certain key questions over others as they move through the story.
I will discuss approaches, challenges, and lessons learned in the design and implementation of this instrument with respect to scenario construction, instructional design, and assessment design. Scenario construction includes selecting a facet of the ethical reasoning construct to focus on; selecting a scenario that provides opportunities for ethical reasoning; crafting story scenes; and connecting scenes into a longer scenario. Instructional design includes the application of strategies such as chunking, coaching, scaffolding, cuing, feedback, and use of resources to add value to the instrument as a learning experience. Assessment design includes designing selected response items to guide exploration of the issues raised in each scene; arranging items for consistency and interpretability in assessing learner response patterns; implementing item sequences as a Qualtrics survey capable of real-time scoring and basic adaptive feedback (in progress); and designing data output to support the comparison of various methods for analyzing learner paths and interaction patterns.
The project explores methods for combining instructional design and assessment techniques, assessing complex constructs using text-based survey instruments and interactive story-telling, and designing learning experiences that can be used at an activity or module level while also providing useful data for assessment purposes.
