Skills Evaluation with Multiple-Choice Questions
The Law Teacher, Volume 8, number 1 (Fall 2000), p. 3-5.
About the Author
Greg Sergienko teaches at Western State University School of Law, 1111 N. State College Blvd., Fullerton, CA 92831; (714) 738-1000 (ext. 2530); fax (714) 525-2786; gregs [at] wsulaw.edu
Skills evaluation and multiple-choice questions are often thought to be inconsistent. Certainly, multiple-choice questions cannot easily test writing skills. However, multiple-choice questions can evaluate many skills used for legal analysis, and they even have some advantages over essay questions for this purpose. Multiple-choice questions can ask about only the facts, pinpointing students' difficulties in reading facts, or they can include a rule of law, pinpointing students' difficulties in applying rules. Moreover, multiple-choice questions provide fast feedback and statistical verification of their reliability. Thus, I received quick information on which questions seemed unreliable. Over the course of the semester, I was able to conclude that multiple-choice exams that test case- and rule-reading skills did better than conventional essay exams in predicting results on a performance, essay exam, which requires both case- or rule-reading and an essay answer. This article explains some of the uses of multiple-choice questions to evaluate skills, starting with an analysis of the limitations of essay and traditional multiple-choice exams.
The limits of complex questions in identifying student mistakes
Essay questions can test students' ability to identify relevant facts, apply the law to them, and organize and write an answer. Unfortunately, the very complexity of essay questions limits their usefulness in identifying where students make mistakes. The thought process in writing an exam answer is a chain with many links (illustrated at right), and when the chain breaks it is often impossible to tell which link failed. Failure to address an issue can result from careless reading, which caused the student to miss a relevant fact; from not knowing the applicable legal rule, which caused the student to miss the significance of a fact that the student did read; or from inability to interpret the language of a memorized rule, which caused the student to miss the applicability of the rule.
The traditional multiple-choice exam tests knowledge of the law by asking questions about the legal rule and forcing the student to select among alternative statements of the law or by providing a fact pattern and alternative answers that apply the law to fact. This seems to be the exclusive mode of examination in law schools and on the bar. Multiple-choice questions requiring reading, recall, and application share the defects of essay questions. A wrong answer to a traditional multiple-choice question does not reveal where the student went wrong.
Skills-oriented multiple-choice questions
The limitations of essay exams and traditional multiple-choice questions have led me to develop skills-oriented multiple-choice questions. These questions examine separately abilities to read facts and to apply an unfamiliar rule of law.
These questions have important features for diagnosing students' difficulties and assessing students' skills. Frequent multiple-choice exams for testing skills during the semester allow the instructor to evaluate the success of instruction and provide students with the chance to evaluate their own progress. In addition, multiple-choice tests allow the teacher to provide students with quick feedback, which vastly increases its effectiveness.
The information about where students are going wrong is often a surprise to them. Two aspects of this have especially come to my attention. First, students tend to overestimate how carefully they read questions. Quantifying exactly how often they are missing relevant facts appears to be far more effective in encouraging careful reading than just telling them to read carefully. Second, students who make errors are often genuinely unsure about whether those errors came from their misremembering the words of the rule or from their inability to apply those words. Breaking down the process into rule knowledge and rule application provides them this information.
The next page contains an exam that illustrates skills-oriented multiple-choice questions. Questions 1 through 4 refer to the Lamson case. The correct answers have been italicized.
[In this online version of this article, the exam is in the following box.]
Lamson v. American Ax & Tool Co.
Question 1, which asks who won the case, addresses procedural knowledge and case reading. The "exceptions overruled" at the end demonstrates that the same person who lost in the trial court lost on appeal. From the opinion, it is clear that the plaintiff loses, although the opinion nowhere states that directly.
Question 2, which asks what facts are most significant, assesses reading skills. Its incorrect choices combine possibilities for missed facts and erroneous understanding of the court's opinion. The first option, the signed contract assuming the risk, is legally plausible but factually erroneous. Someone with a correct understanding of the law but poor reading skills would be tempted to pick that. The second option is correct. The third option, that the accident was caused by people in another department, is factually incorrect and would support an outcome different from the court's. The fourth option, that the new racks are less safe, is factually correct, but not important for the court's assumption of risk argument.
Question 3 asks the reader to select the most analogous case. This tests the ability to transfer knowledge from one case to another. To do this, readers must understand the salient features of the plaintiff's decision to encounter a known risk and give that more weight than scenarios that have facts that are superficially similar to those of the Lamson case.
Question 4 asks the reader to classify this case by selecting the area where changes would most likely alter the result in this case. The question is quite easy, because there is no hint of intentional torts, emotional distress, or damages rules in the case. A student who cannot correctly classify the case is likely to have weak knowledge of general tort law or extreme problems in identifying the facts.
Question 5 asks the reader to apply a rule based on the standard definition of intent in torts. The correct answer is (b). Intent exists in I because Calvin hopes to hit Susie, even though he expects not to hit Susie. Intent doesn't exist in II, because Calvin did not believe that he was substantially certain to hit someone, although it was substantially certain in reality. Here, providing the student the rule to work through means that the student learns it is his or her ability to apply the rule that needs improvement, not rule memorization.
I began using skills-oriented multiple-choice questions in a project to improve students' skills and their success on the bar exam. Most students are receptive to these questions and willing to learn from the feedback they receive. The only substantial disadvantage is that drafting these questions is even more time-consuming than drafting ordinary multiple-choice questions. However, the benefits so far are worth it, and I am optimistic about the long-term results.