STONY BROOK, N.Y., March 11, 2003 - The National Science Foundation has awarded a Stony Brook team of researchers a $536,000 grant to develop a web-based system to assess students’ learning. A collaboration of David Hanson and Troy Wolfskill in Chemistry with CELT’s Dave Ferguson and Janice Grackin, the two-year undertaking will focus on the introductory chemistry course as a prototype for applications to other science, engineering, and mathematics courses.
The goal of the project is to improve learning and instruction by enabling teachers and students to ascertain in timely fashion how each student is progressing in mastery of the subject. The basic instrument for this endeavor will be a comprehensive database of graded and sequenced questions and problems which students can access at any point in the course to test and strengthen their grasp of the material. The technology will also provide teachers with an overview of student comprehension, alerting them to areas needing more attention.
“This interactive computer program,” says Hanson, “will provide students in very large courses an equivalent of the kind of attention to their individual work that they might receive in a small class. And it will enable the instructor to focus instruction where it is needed, early enough to bring about improvement.”
Hanson and his colleagues distinguish assessment from evaluation, which measures student achievement in relation to a predetermined standard. Assessment, on the other hand, seeks to improve student performance through non-judgmental feedback delineating strengths and articulating needed improvements. Traditionally, in an introductory course enrolling several hundred to more than a thousand students, feedback to the student is confined to summative evaluation, in the form of unit test and final examination results. Students may learn which questions they answered correctly and which they got wrong, but only when it is too late to amend their ways. What’s more (or less), they are given no explanation of how or why they are right or wrong, even as the multiple-choice questions do not require them to explain or justify their choices. After each successive unit of the course is tested, students regard it as sealed off from any further need for thought or reflection. As instructors have discovered in working with successful students subsequently recruited as undergraduate TAs, an A in the course assures neither an understanding of underlying principles nor an awareness of how they know what they supposedly know.
The new system will emphasize the learning process as much as its product. It will employ the computer’s full array of textual, numerical, and graphic resources to engage students in interactive guided practice in reasoning their way through problems.
A key feature of the program will be its discrimination of four successively more penetrating and sophisticated levels of comprehension. At the first level, students will demonstrate their ability to recall information relevant to the topic at hand. At the algorithmic level, they will be asked to use memorized information in familiar contexts, applying learned recipes and matching analogous patterns to solve problems. Necessary (and all too often sufficient) as these skills are to successful student performance, they are preliminary to the higher levels of learning. The conceptual comprehension level assesses the student’s ability to visualize new situations, rephrase and alter representations, make new connections and provide explanations of physical and chemical phenomena. At the problem-solving level, students are asked to transfer learning to new contexts; to analyze problems, identifying the information, algorithms, and conceptual understanding required to solve them; to synthesize these to arrive at a solution; and to evaluate that solution.
These categories may be refined or redefined as development of the assessment instrument progresses. A major task of that development is to code hundreds of questions entered in the database according to the level of mastery each requires. Drawing upon the work of other researchers, the team has identified distinguishing characteristics of conceptual and problem-solving questions, and will employ them to code questions accordingly. When a student enters a wrong answer to one of these higher-order questions, the program will first check to see if the error stems from a failure in analysis of the problem. To test the student’s grasp of how to go about solving it, the computer will ask him or her to identify the tools needed from a list it provides. If no error in analysis is found, the system will pursue informational and algorithmic-level questioning to determine what requisite specific knowledge is lacking. By the end of the process, the student will understand how he or she has fallen short and what needs to be learned or done differently.
In the first phase of the project, from January to September of this year, in addition to devising reliable ways of coding questions by topic and level and developing the database of such questions for general chemistry, the team will develop software to analyze and store student responses and record measures of learning for student and instructor. To calibrate, validate, and refine the instrument’s capacity for accurate assessment, a cadre of knowledgeable and specially trained graduate students will interview students taking the course. By asking a student to “think aloud” while working through a problem and posing incisive follow-up questions, the interviewer will ascertain the extent of the student’s knowledge and probe his or her thought processes as well. Comparing the insight thus gained with the computer’s breakdown of the same performance will enable the researchers to modify the computer-based questions to correct any discerned deficiencies. By Fall 2002 the system will be sufficiently developed to be in place for large scale testing in two consecutive semesters. After further refinement, the final version will be fully operational in Fall 2003.
While working together on the overall design and implementation of the system, each of the four principal investigators will manage components of the project calling for their particular expertise. David Hanson will supply the essential chemistry materials, with assistance from Troy Wolfskill, who will also create the requisite software; Janice Grackin will train and oversee the interviewers, and Dave Ferguson will develop the protocols for applying the system to other science and technology disciplines. The USB team will also benefit from interaction with grantees from other institutions, including UCLA and UC Berkeley, participating in NSF’s new program, Assessment of Student Achievement in Undergraduate Education. After recently returning from a Washington workshop, Hanson reported that program directors and investigators had identified a number of opportunities for productive collaboration. "We are especially gratified," he said, "to work with like-minded colleagues on some of the most demanding tasks in the project: measuring conceptual understanding and using computer technology to map problem-solving processes. In the larger picture, we’re pleased to be part of a concerted effort to help students, and their instructors, obtain more precise, accurate, penetrating, and timely information on how well they’re learning."
