Article Reviews

Home / Archive by category "Article Reviews"
Review:  Words Count: The Empirical Relationship Between Brief Writing and Summary Judgment

Review: Words Count: The Empirical Relationship Between Brief Writing and Summary Judgment

Reviewed by Jeremiah A. Ho, University of Massachusetts School of Law

Review: Words Count: The Empirical Relationship Between Brief Writing and Summary Judgment and Summary Judgment Success, 22 J. Leg. Writing Inst. ___ (2018).

SSRN Article Link

By Shaun B. Spencer and Adam Feldman

Because I teach first-year law students, the spring semester always brings back recollections of the first-year legal writing experience, culminating with the classic appellate brief assignment.  When I came across my colleague Professor Shaun Spencer’s latest article, co-written with Adam Feldman, a J.D./Ph.D post-doctoral fellow at Columbia Law, I thought it was apt to share—not just because the article’s main handle pertains to the topic of legal writing, but also because of what it implies for law teaching generally.  The article is titled, Words Count: The Empirical Relationship Between Brief Writing and Summary Judgment Success, at it is forthcoming this year from the Journal of the Legal Writing Institute.

At the start of Spencer and Feldman’s article, the piece seems exclusively relevant for practitioners because it presents us with a statistical relationship between the readability of summary judgment briefs to the rate of favorable case outcomes.  Thus, in terms of readability, proficient legal writing is a valuable commodity in law practice according to their results.  However, the academic implication is also clear because legal writing is also what law schools teach.  The idea of effective legal writing lies at the heart of various legal writing textbooks and numerous pieces of scholarship on the subject.  Since Langdell, legal writing classes have been welded into the law school curriculum.  And ABA accreditation standards reinforce that tradition of teaching legal writing by mandating that students take writing courses throughout their law school careers.  In this way, Spencer and Feldman’s article is one to observe.  Their empirical study underscores the value of instruction and competency for the art and skill of legal writing.

Judges might hesitate to divulge that the quality of a practitioner’s writing can influence judicial decision-making of a case—since this revelation would clash with the idea that cases are resolved based on adjudication of law and facts, rather than on the skills and proficiency of practitioners.  However, several existing scholarly studies that have already examined appellate brief writing and correlated subjectivity and readability to favorable outcomes.  In their study, Spencer and Feldman now bring the empirical lens to state and federal trial court briefs in order to determine whether a positive association exists between brief readability and case outcomes.  Here, they frame two hypotheses.  First, “[i]ncreased brief readability will lead to a greater likelihood that a party will prevail on a motion for summary judgment.”  Secondly, “[w]hen the moving party’s brief is more readable than the non-moving party’s brief, the moving party will be more likely to prevail on a motion for summary judgment.”  With these hypotheses raised, they embark to test their hunches.

Spencer and Feldman use cognitive theory to explain their hypotheses.  Because the brain processes familiar and unfamiliar information differently, the fluency of information presented affects whether a person would process new information associatively or analytically.  The more fluent the information presented is, the more one tends to process associatively, and vice versa—the less fluent, the more one processes analytically.  In writing, fluency can be affected by formatting and “the look” of the document—as predicated for example by font, color, and spacing—as well as readability-related characteristics such as length and complexity of sentences, grammar, and vocabulary.

From here, the authors outline the research method they designed that includes their reasoning for examining summary judgment briefs, a protocol for selecting briefs for their sample, and the definition and coding of variables.  In total, they looked at 654 total briefs in 327 cases from both federal and state courts.  What Spencer and Feldman found was that “[w]hen the moving party’s brief was more readable, the moving party was typically more likely to prevail[.]” Also, “[m]oving from cases where the moving party’s brief is significantly less readable than the non-moving party’s brief to the opposite situations, the likelihood that the moving party prevails on the motion for summary judgment more than doubles from 42% to 85%.”  Both findings appear consistent with their initial hypotheses.  The authors explain alternative theories for these results but ultimately dismiss those theories for the correlation they reached.

For lawyers and advocates, this study presents an important focus on effective and presentable writing in litigation.  However, although Spencer and Feldman’s study does not prove a causal relationship between readability of briefs and favorable case outcomes, the authors do call out that the strong correlation raised here does bolster “the ever-increasing emphasis on legal writing instruction in law school curricula, the ABA standards on law school accreditation, and continuing legal education programs.”  Thus, this study lends credibility for elevating the profile and status of legal writing colleagues in law schools across the country.

In reading Spencer and Feldman’s article, I was reminded of the old schoolhouse phrase, “neatness counts”—but here perhaps it’s “readability counts” that is more appropriate.  With readability highly influenced by the proficiency of legal writing, what this study eventually provokes in me as a doctrinal law faculty member can be crystallized into two thoughts.  First, I have a question: does readability correlate to final examination grading or am I as the grader of my final exams doing something else in the grading process (such as assessment) that is conceptually and functionally different from the adjudication process?  Secondly, if readability does correlate to exams (even if I am assessing competency rather than adjudicating cases), then knowing how to affect fluency and readability would be an intrinsic part of the art of lawyering, factoring into the choices and strategies a legal thinker makes in advocacy.  I would see that, other than teaching doctrine, imparting such skills would be part of my job as well.  Teaching it effectively would be another way to help my students engage with the law and help empower them.  Ultimately for me, it is this correlation, drawn from Spencer and Feldman’s study, that resonates most with me.  In this way, beyond “readability counts” for practitioners, their study is also very significant for the teaching of effective lawyering.


Review: The Science of Equality, Vols I & II

Review: The Science of Equality, Vols I & II

By Tonya Kowalski, Washburn University School of Law

Rachel D. Godsil, et al., The Science of Equality, Vols I & II

Attendees at this past summer’s biennial ALWD conference had the great fortune to learn about the latest research on addressing diversity-related challenges. Among the featured speakers was law professor Rachel Godsil, who identified very specific strategies for addressing bias in education, particularly implicit racial bias and related phenomena.

Prof. Godsil and her colleagues at The Perception Institute have published a series of highly readable, persuasive, and practical reports on these pernicious barriers to education. Among these reports are two volumes of The Science of Equality, linked below. Each report synthesizes and assesses the research, but also describes a series of empirically supported strategies for intervention.  For example, Volume 2 offers a simple, low-cost strategy for educators to use when providing written feedback. The “wise feedback” approach couples messages about high expectations with expressions of confidence in students’ ability to meet those expectations. Studies show that such messages vastly improved response rates and quality from students in a particular marginalized group.

The topics and strategies range from institutional to individual. Readers will find an array of proposals suitable for both classroom professors and administrators.

Notes and Links:

  • The Science of Equality in Education: The Impact of Implicit Bias, Racial Anxiety, and Stereotype Threat on Student Outcomes
  • The Science of Equality Vol. 2: The Effects of Gender Roles, Implicit Bias, and Stereotype Threat on the Lives of Women and Girls
  • Additional publications
  • ALWD is the Association of Legal Writing Directors
  • This post’s author is currently an ALWD board member but has no personal stake in The Perception Institute.
Review: Reframing the Socratic Method

Review: Reframing the Socratic Method

By Tonya Krause-Phelan, WMU-Cooley Law School

Reframing the Socratic Method by Jamie R. Abrams

Reframing the Socratic Method offers a fresh idea to redesign the Socratic Method from a professor-student exercise into an exercise that fosters diverse participation and develops essential lawyering skills. Professor Abrams acknowledges that the Socratic Method, used by law schools for over a century, has become the quintessential example of question-based learning. But contrary to many modern critics of the Socratic Method, Professor Abrams does not disparage the Socratic Method or call for its elimination. Nor does she endorse it. Instead, she encourages professors to restructure the Socratic Method in three ways to ensure it aligns with current innovations and reform: make it client-focused, research-focused, and skills-sensitization focused.

First, Professor Abrams suggests that the Socratic Method should focus primarily on the client, instead of the case. Traditionally, Socratic dialogue begins by asking the student what happened in the case which causes students to think about the case abstractly. As a result, students do not consider the case from the client’s point of view. Further, it does not permit students to scrutinize the decisions made by the lawyers in the case. With a few simple changes, professors can move the Socratic Method from a rule-based to a client-based task. Instead of asking students to recite the facts of the case, a client-based Socratic approach asks the student to explain what happened to the plaintiff or why the plaintiff sought counsel.  These modified questions still highlight the relevant facts of the case, but they allow students to understand the facts from the client’s point of view as well as to consider the attorney-client relationship.

Next, Professor Abrams recommends that instead of using the traditional Socratic Method approach to focus on case outcomes and hypothetical questions, the Socratic questions should be changed so that students use relevant legal authority to represent the client. Instead of asking a student to recite the court’s holding, students should be asked what precedent, the client’s lawyer would have found in preparing the client’s case. These modified questions, propel students to analyze the legal authority relied upon by the court, to understand how the precedent negatively or positively affected the client, and to understand the historical and social underpinnings of the legal precedent. To further insert research-based components into the Socratic dialogue, professors could require students to apply information contained in the case footnotes or to prepare supplemental material to answer research-focused questions. This allows students to develop the ability to assess the strengths and weaknesses of a client’s case.

Finally, Professor Abrams proposes that professors modify the Socratic dialogue to sensitize students to the broad range of legal skills needed to lawyer effectively. She acknowledges that this type of questioning may not be practical in every case. But when possible, the professor should ask questions that guide students to think about effective lawyering skills. These refined questions could range from what role settlement negotiations play in a client’s case to understanding the relevant ethical rules used to determine who the client is and how to meet the client’s objectives. Professor Abrams illustrates how reframing the Socratic Method in a commonly-taught constitutional law case, Reed v Reed, changes the dynamic of instruction from professor-student to student-propelled focus on the client, legal research, and effective lawyering skills.

Professor Abrams explains that reframing the Socratic Method achieves three benefits. First, it allows for coherence and continuity to legal education. Second, it trains practice ready lawyers because students will be better prepared to tell the clients actual answers to actual questions and they will be sensitized to how intensive legal research truly is. And third, it creates inviting and inclusive classrooms. While detailing specific examples for each type of modification she recommends, Professor Abrams illustrates how easily professors could modify the manner in which they already use the Socratic method to accomplish the current goals of innovation and reform.



Review: The Master’s Tools: Deconstructing the Socratic…

Review: The Master’s Tools: Deconstructing the Socratic…

By Rory Bahadur, Washburn University School of Law

The Master’s Tools: Deconstructing the Socratic Method and its Disparate Impact on Women Through the Prism Of The Equal Protection Doctrine by Tanisha Makeba Bailey

This article traces the history of women as law students and empirically documents the tangible disparate impacts of Socratic Pedagogy on female law students. The second half of the article, which is not discussed in this review, is a fascinating discussion about how these disparate impacts may be significant enough to warrant an Equal Protection challenge.

The article initially describes the history of women in legal education and summarizes the exclusionary attitude law schools had toward women by quoting Supreme Court Justice Joseph Bradley as follows:

[T]he civil law, as well as nature herself, has always recognized a wide difference in the respective spheres and destinies of man and woman. Man is, or should be, woman’s protector and defender. The natural and proper timidity and delicacy which belongs to the female sex evidently unfits it for many of the occupations of civil life . . . . The paramount destiny and mission of women are to fulfill the noble and benign offices of wife and mother

Next the article does something important and rare; it defines the Socratic Method. It’s modern incarnation is described as, “a masochistic interplay of domineering, and at times evasive, professors attempting to inform humiliated, and silenced students.” The author describes this method as both humiliating and debilitating, concluding that the gravity of the effect on female law students is simply ignored. She concludes, “the Socratic Method impairs the ability of women law students to perform and excel academically, leading to a crippling of their long-term performance in terms of grade-based opportunities.”

She uses interviews, testimony, and data to quantify the effects of Socratic teaching on female law students. She groups the effects into two broad categories: silencing and grade disparity. Components of the silencing category are actual silencing of law students who are not comfortable interacting in a Socratic class, the oppressive atmosphere of the Socratic classroom and the resulting psychological debilitation it causes.

Grade disparity is documented by comparing prelaw school academic success and rank of entering male and female students with the ranks after the first year of law school. The data clearly indicates that something favors the increased class rank of the male students after the first year of law school. Her conclusion is summarized as follows; “The fact that women consistently academically outrank men prior to law school admission, and then during law school their grades become grossly disproportionate, reflects the egregious effects of the Socratic Method.”

While the documentation of these effects is something we all need to take seriously if we are to provide the same opportunities for success for all our law students regardless of gender, another crucial fact is the conclusion the author reaches when she describes the recommended pedagogy for minimizing the disparate impact on female law students. The author suggests in part, a pedagogy based on the “ethic of care, which includes “using alternative methods of teaching, giving more exercises, practice examinations, and other feedback . . . .”

Even if you are a Socratic diehard, the assumption is that student learning is something every law professor cares deeply about. If that assumption holds true, then we cannot dismiss the almost normative identity of this author’s recommendations and those of the Carnegie and Best Practices Reports on legal education.


Review: The Inseparability of Professionalism and Personal Satisfaction…

Review: The Inseparability of Professionalism and Personal Satisfaction…

By Jeremiah A. Ho, University of Massachusetts School of Law

Lawrence S. Krieger, The Inseparability of Professionalism and Personal Satisfaction: Perspectives on Values, Integrity, and Happiness, 11 Clinical L. Rev. 425 (2005).

Through his writings and teachings, Professor Lawrence Krieger has devoted a notable—dare I say, even honorable—body of work on the humanistic side of lawyering.  Thus, even though his article, The Inseparability of Professionalism and Personal Satisfaction: Perspectives on Values, Integrity, and Happiness, was published more than a decade ago, it still reads with relevance today, especially in light of the many worries that law teachers have expressed regarding the professionalism issues with current law students.  His article here first discusses reasons why law schools have been teaching a limited sense of professionalism—restricted to only “telling law students and lawyers that they should act in certain ways” for either vaguely described “noble reasons” or singly to avoid bar discipline.  Then it illustrates how to redirect that teaching to a more noble and personally fulfilling sense of professionalism.  Such training should be more specific, relevant, and profound, according to Krieger’s article.  It ought to be tied to professional satisfaction, happiness, and a deep set of inherently principled values.  Through humanistic and empirical science, he shows why this connection is true and how to instill a deeper, more personally satisfying sense of professionalism in our students.

To connect professionalism to career and life satisfaction, Krieger begins from a place of dissatisfaction, describing the empirical evidence amongst lawyers that reveal their low career satisfaction and high concerns for mental health issues.  “Those values and motivations that promote or attend professionalism have been empirically shown to correlate with well-being and life satisfaction, while those that undermine or discourage professionalism empirically correlate with distress and dissatisfaction,” Krieger writes.  He also relies on the Maslow hierarchy of human needs that explains people’s drives for growth and satisfaction:  people pursuing higher needs, such as self-actualization and esteem, tend to experience growth motivation and psychological maturity; those pursuing lower needs, such as survival, security, competence, belonging, and respect from others, experience more deficiency motivation accompanied by minimal life satisfaction.  All of these correlations are also supported by modern psychological research, which Krieger examines.  Between intrinsic values of purpose and personal happiness and external motivators such as money or status, modern psychological research builds on Maslow to show that “when intrinsic values and motivation dominate a person’s choices she tends to experience satisfaction and well-being, whereas when extrinsic values and motivation are most important to her she will experience angst and distress.”  Those values that promote happiness are likely to lead to professional behavior; while the vice versa seems to be true.  Furthermore, Krieger finds that the intrinsic value of integrity is tied to professionalism and satisfaction.

Because law school and the legal profession places heavy emphasis of external rewards, law students can be led astray from professionalism and career happiness by their drive to seek out such extrinsic motivators.  In this way, Krieger offers a teaching exercise to instill in students an awareness of the correlation between intrinsic values and professionalism.  He makes his students actively seek out intrinsic values by having them write out hypothetical eulogies of themselves followed by lessons on an idealized professionalism that seeks out noble behavior, by introducing research on the dismal mental and emotional well-being of attorneys, as well as studies on the typical goals and behaviors of happy and healthy individuals.  After such lessons, Krieger asks his students to revisit that eulogy to identify intrinsic values that students have written about in regards to themselves.  Then students must match up those identified values with values that promote professionalism.

As Krieger notes, “[t]he results of this exercise are illuminating, because they show students and lawyers the kinds of things that matter most deeply to them.”  In running this exercise, Krieger has observed that his students tend to express intrinsic values and virtues in their eulogies rather than external ones.  Ultimately, this exercise helps his students differentiate the intrinsic values that lead to personal fulfillment and professionalism from values that would distort their sense of professionalism and career identity and possibly hinder personal fulfillment.  It’s an intriguing exercise, odd in the sense of what’s required is writing one’s own eulogy, but goal-oriented in nature and form, where the law teacher shapes students’ conception of professionalism by getting them to actively acknowledge on their own the types of intrinsic values that professionalism embodies and what values they ought to seek.


Review: Grit

Review: Grit

Reviewed By Prof. Emily Grant, Washburn University School of Law

Angela Duckworth, Grit (2016)

Dr. Angela Duckworth is a professor of psychology at the University of Pennsylvania.  Her book Grit came out last year, and it is spectacular.

The concept of grit includes a combination of passion and perseverance.  Gritty people have both determination and direction. The notion of grit is correlated with ideas of industriousness, creativity, hope, and conscientiousness.  And that character trait, Dr. Duckworth argues, is responsible for high levels of achievement, even more so than sheer talent, ability, or intellect.  To be clear, Dr. Duckworth doesn’t ignore the existence and importance of talent, but she asserts that “a focus on talent distracts us from something that is at least as important, and that is effort.”

In the first part of the book, Dr. Duckworth makes the case that grit matters.  She discusses various research and psychological studies, and she shares interviews with people from all walks of life including West Point cadets and professional athletes and school teachers and business moguls.  She also includes a ten-question self-assessment to measure how gritty you are (see her website: to measure how gritty you are.  Dr. Duckworth then discusses how to grow your own grit, i.e., how to develop and nurture passion and perseverance in yourself.

But it’s the last part of the book that was most intriguing to me as an educator (and frankly, as a parent)—how to help cultivate grit in other people.  She talks about parenting for grit and about how to create a culture of grit.  It is this portion of the book that I will be rereading before the start of every semester to contemplate how I can encourage passion and perseverance in my students.

In addition to being substantively fascinating, the book was wonderful to read.  It was a powerful book for me as a person, as a parent, and as a professor in charge of encouraging high levels of achievement in my students.  This is the kind of book I will give as a gift for every graduation to which I am invited.  I will likely make my own son read it when he finishes high school and starts out on whatever path is next.  I would like every incoming law student to read it.  You most definitely should as well.

If you’d like a taste of Dr. Duckworth’s research and theory, you can listen to her six-minute TED talk on the topic:

Or a 45-minute Freakonomics podcast:

Review:  Lawyers at Work: A Study of the Reading…

Review: Lawyers at Work: A Study of the Reading…

Reviewed by Sandra Simpson, Gonzaga University School of Law

Article: “Lawyers at Work: A Study of the Reading, Writing, and Communication Practices of Legal Professionals”[1]
By Ann Sinsheimer and David Herring Professors of Legal Writing at University of Pittsburg School of Law.

This article reports on the results of an ethnographic study of associate attorneys in the workplace, specifically studying what they actually do in terms of reading, writing, and communicating.  Not surprisingly, the study found that associate attorneys spend most of their time reading, writing, and communicating.  The fascinating part of this study is the finding that associates are not reading many judicial opinions, writing many formal memos, or communicating in a courtroom setting.

The study found that lawyering for junior associates was fundamentally about reading: both digitally and in print.  They read primary authority, but most frequently, they read broad-based secondary authorities and non-legal texts.  These associates read often in print and bounced between free on-line sources and paid on-line sources, being very conscious of the high price of the paid on-line resources.  Reading closely and carefully was important to the associates but many times they skimmed and scanned documents in an effort to efficiently ferret out the importance of the document.  Associates had to quickly hone their ability to read with a purpose: to solve a problem.

The study also found that associates did write: a process which began with reading and rereading the information they had gathered to substantiate their writing.  Often working from templates, associates spent a good amount of time editing their writing before sending their work to their supervisor.  As email represents a large portion of their writing, much effort was spent honing their writing, focusing on word choice, tone, and content.

Not surprisingly, the environments where these lawyers worked were stressful, even the most congenial of the workplaces were stressful.  Although derived from numerous sources, the most common sources of stresses were being pressed for time and needing to juggle multiple tasks at one time.  Working with constant interruptions also led to high levels of stress.  Of note, the authors found that “[t]he ability of the attorneys to understand their role influenced their sense of well-being as well as their capacity to successfully perform their reading and writing tasks.”

The article goes on to discuss other interpersonal skills exercised and needed to help associates be successful, namely teamwork (in a vertical structure with a senior partner), organization, and time management.  Prioritizing and communication as a team rose to the top as critical skills.

Wrapping up the article, the authors suggest law schools look at helping students read for the purpose of problem solving.  As for writing, the authors suggest law schools, across their curriculum, have students create emails for various purposes.  Further, teaching students to use and change templates or forms is an invaluable skill. Lastly, law schools should create opportunities for law students to practice the interpersonal and communication skills necessary to navigate the “sort of high pressure and hierarchical workplaces” observed in the study.  The article gives several suggestions on exercises to use.

[1] The article can be found at


Review: “All of the Above: Computerized Exam Scoring of Multiple Choice Items Helps To: (A) …”

Review: “All of the Above: Computerized Exam Scoring of Multiple Choice Items Helps To: (A) …”

By Heidi Holland, Gonzaga University Law school

Article: “All of the Above: Computerized Exam Scoring of Multiple Choice Items Helps To: (A) Show How Exam Items Worked Technically, (B) Maximize Exam Fairness, (C) Justly Assign Letter Grades, and (D) Provide Feedback on Student Learning” by Lynn M. Daggett, 57 J. Legal Educ. 391 (Sept. 2007).

All of us are working to integrate formative assessment pursuant to the mandates of ABA Standard 314, and multiple choice exams are one way to do it. However, few law teachers have training in how to use multiple-choice questions effectively.  While it is certainly not a recent article, Professor Lynn Daggett’s article is instructional and encouraging to both the novice and experienced teacher.  In her article “All of the Above: Computerized Exam Scoring of Multiple Choice Items Helps To: (A) Show How Exam Items Worked Technically, (B) Maximize Exam Fairness, (C) Justly Assign Letter Grades, and (D) Provide Feedback on Student Learning,” Professor Daggett introduces her readers to the strengths and limitations of multiple choice questions, explains what they can be used to measure, shows how data can be used to guide the assignment of letter grades, and provides specific examples of the concepts in use.

Professor Daggett begins by explaining “core psychometric concepts” including validity (construct, predictive, and content) and reliability.[1]  She also sets the framework for assessment: that it can be criterion-referenced or norm-referenced. Criterion-referenced tests measure whether a student can demonstrate mastery, whether it be of a skill or a concept.  A norm-referenced evaluation, on the other hand, compares a student’s performance against her peers.[2]

Effective instruction and assessment both require intentionality on the part of the instructor, and “law teachers have to decide what the purpose(s) of the [test/question] is, including whether the [test/question] is designed to separate out levels of learning within a class (norm-referenced evaluation), or to measure whether students have mastered specific concepts or skills (criterion-reverenced evaluation).”[3] With the instructor’s goal identified, Professor Daggett then explains how computerized scoring can be used as an assessment tool.

With the mean, median, mode, standard deviations and z-scores in mind, Professor Daggett offers instructors methods of assigning letter grades.  Nonetheless, data does not substitute for professional judgment, and Professor Daggett explains how she uses the data to guide her discretion always keeping in mind that we “perform somewhat of a gatekeeper function in assigning low letter grades, particularly in first year classes. . . . [A] grade of D+ or D from [her] means the student has demonstrated barely adequate learning of course concepts and skills and more generally should not continue in law school unless grades in other courses reflect considerably more mastery.”[4]

As previously noted, Professor Daggett’s article is not just informative; it is instructional. She explains how to decipher and use item analysis, which is part of a standard computerized scoring report from a multiple-choice exam. The article details how to judge the efficacy of test questions, provides an example of the specific information shared with students after an exam, and encourages all of us to provide feedback to students about concepts they have not yet mastered. With thorough explanations and three appendices, you should come away with a better understanding of how to use multiple-choice questions fairly to assess student learning.


[1] Pg. 394

[2] Pg. 399

[3] Pg. 401

[4] Pg. 401


Empirical Evidence that Formative Assessments Improve Final Exams”

Empirical Evidence that Formative Assessments Improve Final Exams”

By Barbara Lentz, Wake Forest University Law school

Article: “Empirical Evidence that Formative Assessments Improve Final Exams” by Carol Springer Sargent and Andrea A. Curcio, 61 J. Legal Educ. 379 (Feb. 2012).

Are you integrating formative assessment in your courses to satisfy ABA Standard 314, but wondering whether your time to develop, implement and evaluate student work will improve students’ learning?  In the article “Empirical Evidence that Formative Assessments Improve Final Exams,” Professors Carol Springer Sargent and Andrea A Curcio report that shifting away from a single summative assessment to a combination of formative assessments over the term coupled with a final cumulative exam provides a measurable performance improvement to 70% of the law students in the course. The authors describe their study, review types of and benefits from formative assessments, list practices to improve effectiveness of feedback, and seek to explain why the term-long formative feedback boosted performance only for students in the top 70% of LSAT score or undergraduate GPA (UGPA) cohort regardless of first year law school grades.

Their study was based on data collected in two subsequent Autumn terms of a large, doctrinal Evidence course taught by the same experienced professor. The control section was taught using a problem method with case analysis and students were assessed solely by a cumulative final exam that comprised the full course grade. In the following year, the intervention section was taught using the same problem method with case analysis, but also received a series of formative assessments: five ungraded quizzes, a graded midterm, model answers, grading rubrics, and a self-reflective exercise. The final exam in the intervention section comprised 83% of the course grade (allocated so that the final course grade would correspond primarily to the final summative assessment). The authors compared the final exam scores of students on eleven common final exam questions. The variance in common question scores, 3.02466 points out of 50 points, is about a half a letter grade (6.048 percent). Id. at 391. The effect for the top 70% of the intervention class, however, was 4.595 points out of 50, or almost a full letter grade (9.19 percent). Id.

Before discussing the implications of their results, the authors reviewed best practices for implementing formative assessments to improve learning. Formative assessments improve learning by helping students identify misconceptions and knowledge gaps and by motivating or refocusing studying. In their view, the most effective formative feedback explains to students why an answer is correct instead of merely showing the correct answer. According to the authors, other strategies to make feedback more effective are providing suggestions to improve performance, delivering feedback close in time to the assessment. Id. at 381-82.

The professors and students valued types of formative assessment differently. The authors posit that ungraded feedback may be more helpful in improving learning because it focuses the student on suggestions for improvement rather than solely upon the grades. Id. at 382.  However, student evaluations showed that while many students found all the formative materials to be helpful, the graded midterm was valued more highly. Students commented that model answers, grading rubrics, and professor comments were the most helpful feedback, but peer edits and self-reflections provided the least useful feedback. Indeed, more than one-third of students surveyed found the self-reflective exercises unhelpful. Id. at 392.

Sargent and Curcio write that the demonstrated benefit from formative assessment disproportionately accrued to the top 70% of students (arranged by LSAT or UGPA). Students in the intervention section with below median first-year law school grades did show improved performance, compared to the control group, but only if those students were in the top two-thirds of the class on UGPA or LSAT scores. Id. at 400.  By correlating LSAT and UGPA with performance on the final exam, this study showed the bottom 30% of students by LSAT or UGPA (regardless of first year grades) either did not or could not use the information from formative assessments to monitor and improve the quality of their work as measured by performance on the final exam. Similarly, results from a similar prior study (comparing civil procedure courses taught by different instructors in the same term), showed that practice essays only helped improve final exam performance of students with above median LSAT scores and UGPAs.

The authors presented potential explanations for the disproportionate allocation of benefit from formative assessment. First, it is possible that not all students are able to use feedback to improve. Students’ ability to calibrate what they know and don’t know is a metacognitive skill. Id. at 395-96. There may be a difference in students’ metacognitive abilities. If the top 70% of students possessed stronger metacognitive skills, they would be better able to process and apply information gleaned from formative assessment to improve their learning and subsequent performance on final exams. Id. at 384. The top 70% cohort may also have higher confidence in their abilities to effectively use feedback, which improves their abilities to better self-monitor and calibrate their comprehension. Id. at 400. Additionally, it is possible that students in the top 70% of the LSAT or UGPA cohort are more motivated by grades. Id. at 396.

While the article provides evidence that formative assessment improves performance, the authors disclose are three caveats: First, they are not able to identify which types of formative assessment led to higher exam scores in the intervention section. While students did not report the self-reflective exercises to be helpful, student perceptions “are not a direct measure of the actual helpfulness of the materials.” Id. at 395. Second, it is possible that students perform better when they know their performance being measured, (the Hawthorne effect). Id. at 398. Finally, the authors questioned whether the formative assessments, which they described as practice materials, might inadvertently encourage performance-oriented goals rather than encouraging deeper mastery learning. “In other words, do [formative] practice materials support those whose main goal is to get higher course grades rather than assisting those who wish to truly comprehend and master the content?” Id. at 399.

The authors did not believe that the shift to formative assessment unreasonably burdened faculty, particularly when faculty time over the entire term was considered. “While drafting the questions, model answers, rubrics, and self-reflective exercises initially takes a few hours, those materials do not need updating each term. Grading a short midterm also takes a few hours, but may result in faster final exam grading due to better quality responses. Alternately, giving a midterm may justify a shorter final exam, thereby reducing time spent grading final exams.” Id. at 400. Because formative feedback improves performance by explaining why an answer is incorrect, it may be possible to produce the same learning effect without administering and individually grading a mid-term, further minimizing instructor effort. The authors suggest undertaking a future study to discern whether providing a model answer to an ungraded midterm might provide a learning effect similar to individually graded midterms.

Finally, they observed that completing formative exercises and providing feedback reduced the amount of traditional instruction time relative to the control section. Thus, feedback must be more helpful in improving learning than the reduction in class time to cover material. Id. at 398. However, the substantial improvement shown by the intervention group seems to show that reducing time for traditional instruction in favor of formative assessment may improve student learning at least as measured by final exam scores for the majority of students.


Do We Need Subject Matter-Specific Pedagogies?

Do We Need Subject Matter-Specific Pedagogies?

By Rory D. Bahadur, Washburn University School of Law

In her recent Article Do We Need Subject Matter-Specific Pedagogies?, 65 J. Legal Educ. 839 (2016), Professor Kris Franklin invites us to consider the unique roles that the various foundational courses may play in the law school curriculum.

The Article was published as part of an AALS symposium dedicated to the teaching of civil procedure, and asks, essentially, what value there might be at considering legal pedagogy in the context of individual academic disciplines. Or as Franklin puts it: “Is good teaching simply good teaching wherever it is done, or is there something different about how we teach, or ought to, within each academic discipline?”

Using examples drawn both from the other pieces in the symposium itself and an earlier collection of essays on the teaching of civil procedure, Franklin carefully considers what a law professor new to the subject might glean from her colleagues’ writing.  She concludes that at the very least, studying the existing pieces on teaching civ. pro. would offer valuable tips on how to help students understand especially challenging subject matter.  More broadly, she observes reading other scholars’ work on civil procedure pedagogy could refine her sense of how to use the subject to teach some of the critical skills that lawyers, and law students, must begin to master. This inquiry, Franklin tells us, would be valuable in and of itself, because it would make her (and any reader) a better teacher of the subject.

But Franklin goes on to note that reading the other articles would do more than just offer technical advice about teaching this particular exacting course: It would show her how “a deep comprehension of civil procedure distinctly contributes to an understanding of what law is, or what it means to be a lawyer.”

Perhaps it is Franklin’s suggestion that most law Professors when asked what good pedagogy is, will respond in very broad brush, general terms, which lays the foundation for the article’s most salient point.  At bottom Franklin suggests we examine each of the foundational courses and specifically identify which subset of the “good pedagogy” principles each class is best suited for rather than trying to make each class employ all the pedagogical tools we typically identify as good pedagogy.

She begin the process of identifying which of the foundational courses are the best pedagogical vehicles for specific aspects of what we all tend to agree law students need to learn or what good pedagogy entails.  For example she suggests:

  • Criminal law introduces students to the importance and function of statutes (at least, it does so more than other first-year courses tend to). Most criminal courses heavily reference either the Model Penal Code or the state or federal penal law.Perhaps they even study and compare more than one of these sources. Yes, there are codes relied on in other introductory courses, but criminal law courses teach students that they must always read carefully the specific language of the statutes to determine what offenses might legitimately be charged for a given set of facts.
  • Contracts law is particularly intricate. It could be especially important in helping to hone students’ analytical precision. Furthermore, its problems have to be approached in an unusually constant order: asking first whether a legally binding agreement was formed (mutual assent plus consideration); second, whether it was breached; assuming it was, whether there are any defenses to that breach; then finally, what if any damages the parties to the agreement may claim and receive. This remains the case despite the fact that there is wide variation in the order in which these basic steps are covered in contracts courses. If students grasp this “flow chart-ish” nature of analyzing contracts problems, they are often well on their way toward understanding contract law itself, and they have reinforced an organizational skill that may be helpful in other areas as well.
  • Constitutional law is probably the only foundational law course, maybe the only class in law school altogether, in which each and every case matters for its own sake. Consequently, constitutional law as a discipline requires analogical reasoning to important cases in a way that few other introductory law courses do. Beginning law students often do not use cases well, in part because they find it especially difficult to fathom why they are reading particular cases. Students frequently cannot distinguish those cases they are reading as simply being among any number of examples that might have been selected to illustrate how a particular legal rule operates, as opposed to those leading cases that are assigned because they introduce or solidify the doctrine. In contrast to most other required introductory courses, allcases in a constitutional law casebook are likely leading ones, while this is true of perhaps very few of them in a property casebook. Students of law should therefore learn and use those cases in different ways. A teacher in constitutional law may help students understand why the assigned cases have unique importance in this course, which could in turn help law students discern the differences between leading cases and those offered merely as examples. If so, students could learn how to use cases in a more sophisticated way in all of their classes.
  • Evidence problems seem to lend themselves to meticulous specificity in the application of a set of rules that becomes thorny when argued in given situations. Some evidence professors suggest that multiple-choice testing is especially useful in that subject because of the way rules of evidence are used in the courtroom–as points for rapid and discrete debate, often handled spontaneously and settled definitively. Regardless of assessment method, it may be true that evidence courses offer exceptional opportunities for their students to hone pointed argumentation skills within rigorous application of legal rules.

She suggests that grasping this latter point would go a long way toward developing a broader theory of how each of the standard required courses in the law school curriculum contributes to the “gorgeous mosaic” of learning law itself. She invites us to consider how every one of our courses fits into that big-picture view of the legal curriculum, so that we can consciously teach each course to more effectively educate tomorrow’s lawyers.  In addition to all of the above the article challenges us to examine the courses we teach and to actually quantify what good teaching entails beyond a simple and shallow reiteration of buzz phrases.



Do NOT follow this link or you will be banned from the site!