Skills Evaluation with Multiple-Choice Questions
by
Greg Sergienko

Source

The Law Teacher, Volume 8, number 1 (Fall 2000), p. 3-5.

About the Author

Greg Sergienko teaches at Western State University School of Law, 1111 N. State College Blvd., Fullerton, CA 92831; (714) 738-1000 (ext. 2530); fax (714) 525-2786; gregs [at] wsulaw.edu

Skills evaluation and multiple-choice questions are often thought to be inconsistent. Certainly, multiple-choice questions cannot easily test writing skills. However, multiple-choice questions can evaluate many skills used for legal analysis, and they even have some advantages over essay questions for this purpose. Multiple-choice questions can ask about only the facts, pinpointing students' difficulties in reading facts, or they can include a rule of law, pinpointing students' difficulties in applying rules. Moreover, multiple-choice questions provide fast feedback and statistical verification of their reliability. Thus, I received quick information on which questions seemed unreliable. Over the course of the semester, I was able to conclude that multiple-choice exams that test case- and rule-reading skills did better than conventional essay exams in predicting results on a performance, essay exam, which requires both case- or rule-reading and an essay answer. This article explains some of the uses of multiple-choice questions to evaluate skills, starting with an analysis of the limitations of essay and traditional multiple-choice exams.

The limits of complex questions in identifying student mistakes

Essay questions can test students' ability to identify relevant facts, apply the law to them, and organize and write an answer. Unfortunately, the very complexity of essay questions limits their usefulness in identifying where students make mistakes. The thought process in writing an exam answer is a chain with many links (illustrated at right), and when the chain breaks it is often impossible to tell which link failed. Failure to address an issue can result from careless reading, which caused the student to miss a relevant fact; from not knowing the applicable legal rule, which caused the student to miss the significance of a fact that the student did read; or from inability to interpret the language of a memorized rule, which caused the student to miss the applicability of the rule.

The traditional multiple-choice exam tests knowledge of the law by asking questions about the legal rule and forcing the student to select among alternative statements of the law or by providing a fact pattern and alternative answers that apply the law to fact. This seems to be the exclusive mode of examination in law schools and on the bar. Multiple-choice questions requiring reading, recall, and application share the defects of essay questions. A wrong answer to a traditional multiple-choice question does not reveal where the student went wrong.

Skills-oriented multiple-choice questions

The limitations of essay exams and traditional multiple-choice questions have led me to develop skills-oriented multiple-choice questions. These questions examine separately abilities to read facts and to apply an unfamiliar rule of law.

These questions have important features for diagnosing students' difficulties and assessing students' skills. Frequent multiple-choice exams for testing skills during the semester allow the instructor to evaluate the success of instruction and provide students with the chance to evaluate their own progress. In addition, multiple-choice tests allow the teacher to provide students with quick feedback, which vastly increases its effectiveness.

The information about where students are going wrong is often a surprise to them. Two aspects of this have especially come to my attention. First, students tend to overestimate how carefully they read questions. Quantifying exactly how often they are missing relevant facts appears to be far more effective in encouraging careful reading than just telling them to read carefully. Second, students who make errors are often genuinely unsure about whether those errors came from their misremembering the words of the rule or from their inability to apply those words. Breaking down the process into rule knowledge and rule application provides them this information.

The next page contains an exam that illustrates skills-oriented multiple-choice questions. Questions 1 through 4 refer to the Lamson case. The correct answers have been italicized.

[In this online version of this article, the exam is in the following box.]

Lamson v. American Ax & Tool Co.
177 Mass. 144, 58 N.E. 585 (1900)

Holmes, C. J.

This is an action for personal injuries caused by the fall of a hatchet from a rack in front of which it was the plaintiff's business to work at painting hatchets, and upon which the hatchets were to be placed to dry when painted. The plaintiff had been in the defendant's employment for many years.

About a year before the accident new racks had been substituted for those previously in use, and it may be assumed that they were less safe, and were not proper, but were dangerous, on account of the liability of the hatchets to fall from the pegs upon the plaintiff when the racks were jarred by the motion of machinery near by. The plaintiff complained to the superintendent that the hatchets were more likely to drop off than when the old racks were in use, and that now they might fall upon him, which they could not have done from the old racks. He was answered, in substance, that he would have to use the racks or leave. The accident that he feared happened, and he brought this suit.

The plaintiff, on his own evidence, appreciated the danger more than any one else. He perfectly understood what was likely to happen. That likelihood did not depend upon the doing of some negligent act by people in another branch of employment, but solely on the permanent conditions of the racks and their surroundings and the plaintiff's continuing to work where he did. He complained, and was notified that he could go if he would not face the chance. He stayed and took the risk. He did so none the less that the fear of losing his place was one of his motives.

Exceptions overruled.

=========================================

Questions 1 through 4 require reading Lamson.

  1. Who won in this case?
    a) The defendant won in the trial court and on appeal.
    b) The plaintiff won in the trial court and on appeal.
    c) The defendant won in the trial court, and the plaintiff won on appeal.
    d) The plaintiff won in the trial court, and the defendant won on appeal.
  2. Which of the following facts is most significant to the court's result?
    a) The plaintiff signed a contract assuming the risk that axes would fall off.
    b) The plaintiff knew that the axes were likely to fall off.
    c) The accident happened as a result of negligence by people in another branch of the employment.
    d) The new racks made the drying process less safe.
  3. In what situation would the Lamson precedent be most relevant?
    a) The plaintiff smelled alcohol on the breath of the defendant, but allowed the defendant to drive him anyway, and was injured in an accident caused by the intoxication of the driver.
    b) The plaintiff was an employee and was injured as a result of a defect in the defendant's flooring, which was unknown to the employee and caused an ax to fall on the employee from a higher floor.
    c) The plaintiff was an employee in the painting department, and the defendant was an employee in the ax-head installing department, who injured the plaintiff by throwing an ax head at his supervisor and hitting the plaintiff by mistake.
    d) The plaintiff was a minor employed by the defendant, and was suing for loss of brain function suffered as a result of continued exposure to paint fumes in the defendant's factory.
  4. Which change in the law would be most likely to change the outcome of this case?
    a) Different rules on assumption of risk in negligence cases.
    b) Different rules on battery and other intentional torts.
    c) Different rules on extreme and outrageous conduct (intentional infliction of emotional distress).
    d) Different rules on the damages recoverable for personal injury.

    5. ==========================

  5. Assume that the intent to accomplish a result exists if the actor desires the result or believes that the result is substantially certain to occur as a result of his acts. In which of the following situations does the intent to make contact exist?
    I. Calvin has thrown 100 slushballs at Susie, but because of his rotten aim he has missed each time. He realizes the odds are strongly against him, but he desperately wants to hit Susie, so he throws another slushball, which hits Susie.
    II. Calvin was outside a stadium testing his new slingshot for propelling slushballs. He aimed his slushballs to fall inside the stadium. Signs outside the stadium advertised the sell-out crowd, but Calvin did not read the signs. Calvin hit someone inside the stadium, as was substantially certain to happen.
    a) Neither I nor II.
    b) I only
    c) II only.
    d) I and II.

Question 1, which asks who won the case, addresses procedural knowledge and case reading. The "exceptions overruled" at the end demonstrates that the same person who lost in the trial court lost on appeal. From the opinion, it is clear that the plaintiff loses, although the opinion nowhere states that directly.

Question 2, which asks what facts are most significant, assesses reading skills. Its incorrect choices combine possibilities for missed facts and erroneous understanding of the court's opinion. The first option, the signed contract assuming the risk, is legally plausible but factually erroneous. Someone with a correct understanding of the law but poor reading skills would be tempted to pick that. The second option is correct. The third option, that the accident was caused by people in another department, is factually incorrect and would support an outcome different from the court's. The fourth option, that the new racks are less safe, is factually correct, but not important for the court's assumption of risk argument.

Question 3 asks the reader to select the most analogous case. This tests the ability to transfer knowledge from one case to another. To do this, readers must understand the salient features of the plaintiff's decision to encounter a known risk and give that more weight than scenarios that have facts that are superficially similar to those of the Lamson case.

Question 4 asks the reader to classify this case by selecting the area where changes would most likely alter the result in this case. The question is quite easy, because there is no hint of intentional torts, emotional distress, or damages rules in the case. A student who cannot correctly classify the case is likely to have weak knowledge of general tort law or extreme problems in identifying the facts.

Question 5 asks the reader to apply a rule based on the standard definition of intent in torts. The correct answer is (b). Intent exists in I because Calvin hopes to hit Susie, even though he expects not to hit Susie. Intent doesn't exist in II, because Calvin did not believe that he was substantially certain to hit someone, although it was substantially certain in reality. Here, providing the student the rule to work through means that the student learns it is his or her ability to apply the rule that needs improvement, not rule memorization.

Concluding Thoughts

I began using skills-oriented multiple-choice questions in a project to improve students' skills and their success on the bar exam. Most students are receptive to these questions and willing to learn from the feedback they receive. The only substantial disadvantage is that drafting these questions is even more time-consuming than drafting ordinary multiple-choice questions. However, the benefits so far are worth it, and I am optimistic about the long-term results.