Response to: Ellen D. Katz & Anna Baldwin, Why Counting Votes Doesn't Add Up, 108 Colum. L. Rev. Sidebar 23 (2008).
Judicial decisions provide a wealth of information—but information about what? In recent years, the empirical study of judicial decisions has exploded in popularity as legal academics and social scientists have conducted statistical analyses of court decisions in many substantive areas of law. The purpose of these empirical investigations is to learn something about the way in which judges decide cases. Typically, these studies ask whether a judge’s identity or background—her ideological disposition, race, education, experience, and so forth—influence the way in which she decides cases.1 Rarely, and only when demanding conditions are met, have researchers looked to the success rates in published cases to glean information about the facts that underlie those lawsuits. For good reasons, researchers have not considered the success rates in tort lawsuits in Illinois to be a solid source of information about the level of tortious conduct in the state. Nor have they considered the conviction rates in criminal cases in New York to be a meaningful measure of the state’s crime level.
Ellen D. Katz and Anna Baldwin’s response2 to our study, Judging the Voting Rights Act,3 overlooks this fundamental point. They believe that the rate at which section 2 plaintiffs prevail in published judicial decisions reveals the level of discrimination in certain parts of the country at particular times. More ambitiously, they believe that these dispositions provide a concrete measure of the present-day need for section 5 of the Voting Rights Act.4 We disagree. Judicial decisions can tell us much about how judges decide the cases presented to them. But neither our study of voting rights decisions, Katz’s original examination of these decisions,5 nor Katz and Baldwin’s replication of our findings6 can tell us whether discrimination against the voting rights of minorities is rampant or rare, or whether section 5 is as necessary today as it ever was. Past decisions on section 2 claims simply cannot tell us the answers to these important questions.
In this short essay, we explore which inferences are appropriately drawn from published judicial decisions and explain why we believe that Katz and Baldwin go astray when they attempt to infer the degree of discrimination from the rate of plaintiff success in published voting rights cases. Our central disagreement with their approach is methodological. But Katz and Baldwin do not focus on this difference. Instead, their reply principally critiques the way in which we measure the success rates of plaintiffs in our study. Before turning to the larger methodological point, therefore, we first explain why this criticism is unwarranted.
A. Measuring Plaintiff Success Rates
Because colloquies of this sort inevitably focus on points of disagreement, we should emphasize at the outset the broad areas of agreement between our work and Katz and Baldwin’s reply. Our primary interest in studying judicial opinions issued in Voting Rights Act litigation was to understand whether a judge’s race or partisanship influenced her adjudication of section 2 claims. Section 2 is designed to protect minority voting rights claims. Moreover, section 2 litigation was in the past often thought to systematically favor the Democratic Party. These features of section 2 lawsuits raise the litigation’s partisan and racial salience and motivated our research.
As Judging the Voting Rights Act details, we found that both race and partisanship exert powerful effects. The likelihood that a judge votes in favor of section 2 is strongly influenced by both her partisan affiliation and her race. Moreover, the partisan affiliation and race of the other judges with whom she sits also influence her voting behavior. Katz and Baldwin agree with all of these findings. In their response, they replicate our estimates of the effects of judicial ideology and race and have no disagreement with our primary conclusions. Although Katz and Baldwin downplay these findings, we believe that they have important implications for a variety of debates about the enforcement of minority voting rights and racial diversity in the federal judiciary.
Katz and Baldwin’s disagreement, then, is not with our central findings. Rather, they disagree with the interpretation we attach to one of the many control variables included in our regressions—whether the section 2 claim arose in a jurisdiction covered by the preclearance procedures of section 5. Our baseline regression showed that when both trial and appellate judicial decisions on the merits were considered, section 5 coverage did not correlate with the likelihood that a judge votes in favor of liability under section 2. In this specification, the coefficient on this control variable was less than ten percentage points and statistically insignificant. Our finding conflicts with an earlier claim by Katz and Baldwin that section 5 coverage is a strong predictor of plaintiff success in section 2 cases.7 We reach different results because we make different decisions about how to test for a relationship between section 5 coverage and the success rates of section 2 plaintiffs.
In Katz and Baldwin’s earlier work, they tested for this relationship by simply comparing the success rate in covered jurisdictions with uncovered ones. In their reply, they abandon their earlier exclusive reliance on summary statistics and adopt wholesale the regression frameworks we specified. But Katz and Baldwin critique our regression specifications on two grounds: first, they argue that we wrongly focus on judge votes rather than judicial decisions (or final dispositions); second, they contend that we wrongly include appellate cases (rather than just trial cases) in our comparison groups.
The first objection is beside the point. Katz and Baldwin cast this criticism as a problem of miscounting: they claim that we permit judge votes from panel decisions to enter the data set three times (one vote for each judge on the panel) while judge votes from judges presiding alone enter into the data set only once.8 But judge votes rather than decisions are the appropriate unit of analysis, because our primary interest is in judicial behavior. Moreover, our central conclusions about the influence of judicial ideology and race are the same regardless of which way we analyze the data. At the decision level, the likelihood that the court favors a section 2 plaintiff rises if the presiding trial judge is a Democratic appointee or an African American; and for panel decisions, the likelihood of a plaintiff victory rises with the number of Democratic appointees and African Americans on the panel. Katz and Baldwin prove this point themselves, as their decision-level replication of our analysis produces nearly identical results.
Katz and Baldwin’s second criticism is that we were wrong to include appellate cases in our analysis. As we were careful to explain in Judging the Voting Rights Act, and as Katz and Baldwin’s replication of our estimates confirms, section 5 coverage and liability do not correlate positively when both trial and appellate judge votes are grouped together. But when the data are broken out by trial and appellate courts, strikingly different patterns emerge. Trial courts in covered jurisdictions have higher liability rates than uncovered ones, while appellate courts have liability rates that are lower in covered jurisdictions than in uncovered ones. These patterns obtain irrespective of whether judge votes or court decisions are the units of analysis.
This finding—that the positive relationship between liability and coverage exists only at the trial stage—contrasts with the claim that Katz previously made.9 She claimed that coverage increases the likelihood of liability without qualification as to the stage of litigation. Our evidence, relying on multiple regression analysis and a deeper probing of the data, showed that Katz’s initial claim was incomplete. In their response to us, Katz and Baldwin now concede that coverage correlates with liability only at the trial stage.
In view of the oppositely-signed estimates on the controls for section 5 coverage at the trial and appellate levels, we are cautious about what conclusions, if any, can be reached from these data about the impact of section 5 coverage on liability. Katz and Baldwin, in contrast, attach strong interpretations to these estimates. They champion the positive correlation at the trial stage as judicial “fact find[ings]”10 of discrimination while discounting the negative correlation at the appellate stage as “reveal[ing] little about the underlying claim[s]” of discrimination.11
These interpretations require Katz and Baldwin to make almost diametrically opposed assumptions about the processes generating these data. They assume that the success rate in the trial courts is driven almost solely by differences in the actual levels of discrimination, while appellate court decisions “hinge on a host of factors not tied directly to [section 5] coverage or its absence.”12 These interpretations are not hypotheses that the data have confirmed. Rather, they are assertions about the patterns observed in the data, assertions at variance with Katz’s original claims. The tasks of trial and appellate judges surely differ, but it is implausible that they diverge so widely as to render trial decisions an accurate measure of discrimination and appellate decisions something else entirely. Moreover, Katz and Baldwin’s account of trial and appellate courts is somewhat inconsistent with other claims they advance. Later in their reply, they invert their own assumption that trial decisions reflect facts while appellate decisions do not: they interpret the pattern of appellate decisions in covered jurisdictions as evidence that those appellate courts may be curtailing the excesses of trial judges—in other words, that trial court judges in covered jurisdictions may “have read section 2 too restrictively,” such that appellate court reversals become a better indicator of the true level of discrimination.13
B. The Significance of Success Rates
Katz and Baldwin’s conflicting interpretations of trial and appellate decisions point to the larger methodological question: How should we interpret differences in liability rates across different courts, places, or time periods? Katz and Baldwin believe that liability rates at the trial level are direct evidence of the level of voting rights problems.14 Thus, they use differences in (trial-level) litigation success rates across jurisdictions to measure differences in the levels of discrimination. We believe it is a mistake to draw such strong inferences from the similarities or differences in plaintiff success rates in different places or times.
The central methodological difficulty with Katz and Baldwin’s claims is that they draw inferences about the extent of discrimination and the efficacy of section 5 from a sample of cases that is almost surely not representative of the entire class of voting rights claims. As George L. Priest and Benjamin Klein recognized nearly 25 years ago, “[i]f all legal disputes . . . were tried to judgment and then appealed, the inference from legal rules to social behavior would be straightforward. . . . It is well known, however, that only a very small fraction of disputes comes to trial and an even smaller fraction is appealed.”15 Moreover, the sample of cases that reach trial or appeal are “peculiar” in that they likely are not representative of the typical dispute.16 In asserting that plaintiff win rates are accurate measures of discrimination, Katz and Baldwin overlook this fundamental obstacle in drawing inferences from judicial decisions.
In Judging the Voting Rights Act, we recognize this problem, but it is not nearly as vexing for the set of questions we address. First, the scope of our inquiry—how a judge’s characteristics influence her decisionmaking in cases she is called upon to decide—does not require us to make claims about the full distribution of possible or actual section 2 claims or the behavior and characteristics of litigants. That is, we look to these data to inform our understanding of judges rather than to suss out underlying rates of discrimination. Second, the roughly random assignment of cases to judges within judicial districts and circuits constitutes a sort of natural experiment. The randomization of cases across judges implies that, to a first approximation, a judge’s characteristics will be uncorrelated with the strength of the case. If these characteristics of judges then correlate with the manner in which they decide these cases, it is reasonable to infer that the characteristics have a causal impact on their decisions.
As a further check on our primary findings, we include control variables for various features of the cases. But we interpret these only as controls, not as causal influences. Inferences drawn from them about underlying rates of discrimination could be erroneous (or in statistical argot, biased) because, as Priest and Klein observed, not all possible disputes are litigated through trial and appeal. Some are not litigated at all, and others proceed only through certain stages of litigation. “For the rate of plaintiff verdicts to be an accurate measure of the influence of a legal standard, . . . litigated disputes must be representative of the entire class of underlying disputes.”17 As a general matter, the selection of disputes for litigation leaves no reason to believe that such disputes are so representative.
For these reasons, in other litigation contexts we would seldom think it appropriate to interpret success rates as strong evidence of the social behavior underlying litigation. Were we to find that the success rate of personal injury plaintiffs was higher in Illinois than Michigan, should we take that as evidence that torts are more prevalent in Illinois? Similarly, should we treat criminal arrest or conviction rates as good evidence of the prevalence of crime? We do not think so.18 But regardless of what we think, there are only two options: either Katz and Baldwin must disagree with our general methodological position—in which case they must take the position that litigant success rates, such as criminal conviction rates, are generally good evidence of the prevalence of the social behavior underlying the litigation—or they must explain why it is uniquely appropriate to draw these inferences in voting rights litigation. Katz and Baldwin do not defend either position.
Instead of taking a position on these methodological questions, Katz and Baldwin rely on a relatively ad hoc approach to dealing with selection problems. Where the patterns in the data show differences in success rates consistent with their conclusion about the importance of section 5, they accept those patterns as evidence of the prevalence of discrimination. But when the data do not show such a difference, they craft a story about the selection of disputes to explain why we should not expect to see the difference that does not appear.
Moreover, even were we to accept Katz and Baldwin’s contention that trial-level liability rates are good evidence of whether voting problems remain more prevalent in covered jurisdictions, we remain puzzled by their ultimate conclusion. Their central claim is that that “obstacles to minority political participation remain more prevalent in ‘covered’ jurisdictions.”19 To support this claim, they compute average differences over the entire period of study, 1982–2004. But an average difference over two decades does not fit their claim that a difference remains today between covered and uncovered jurisdictions. Similarities or differences in recent years would seem much more important. After all, differences that existed fifteen or twenty years ago cannot tell them whether obstacles to minority participation continue today.
In fact, the data reveal that liability patterns in covered and uncovered jurisdictions have changed dramatically over time. In the first decade of the study, trial courts in covered jurisdictions were indeed more likely to find liability than courts in uncovered areas. But from 1994 to the end of the study in 2004, plaintiff success rates were nearly identical in covered and uncovered trial courts. These patterns imply that the difference between these trial courts on which Katz and Baldwin rely is almost exclusively the product of differences that existed in the past but which disappeared over time.20
What should we make of this convergence over time in plaintiff success rates? Given Katz and Baldwin’s methodological approach, one might think that they would have to conclude from this convergence that discrimination today is no worse in covered than uncovered jurisdictions. We disagree. We think it would be premature to interpret this convergence as evidence that problems in covered jurisdictions are no longer any worse than in the rest of the country. Regardless of our position, however, the important point is that the significance of the convergence turns centrally on what methodological approach one adopts.
------------------------
*Assistant Professor of Law, The University of Chicago Law School.
**Assistant Professor of Law, The University of Chicago Law School.
1.See Thomas J. Miles & Cass R. Sunstein, The New Legal Realism, 75 U. Chi. L. Rev. (forthcoming 2008), available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1070283 (describing recent burst of studies on judicial decision making).
2.Ellen D. Katz & Anna Baldwin, Why Counting Votes Doesn’t Add Up: A Response to Cox and Miles’ Judging the Voting Rights Act, 108 Colum. L. Rev. Sidebar 23 (2008), http://www.columbialawreview.org/Sidebar/volume/108/23_KatzBaldwin.pdf [hereinafter Katz & Baldwin, Counting Votes].
3.Adam B. Cox & Thomas J. Miles, Judging the Voting Rights Act, 108 Colum. L. Rev. 1 (2008).
4.See Ellen D. Katz, Mission Accomplished?, 117 Yale L.J. Pocket Part 142, 145 (2007), at http://yalelawjournal.org/2007/12/10/katz.html [hereinafter Katz, Mission Accomplished?] (describing comparative success rates in covered versus non-covered jurisdictions as a “concrete way to measure section 5’s success in covered jurisdictions”).
5.See Ellen D. Katz et al., Documenting Discrimination in Voting: Judicial Findings Under Section 2 of the Voting Rights Act Since 1982, Final Report of the Voting Rights Initiative, 39 U. Mich. J.L. Reform 643 (2006), available at http://sitemaker.umich.edu/votingrights/files/finalreport.pdf [hereinafter Katz et al., Documenting Discrimination].
6.Katz & Baldwin, Counting Votes, supra note 2, at 24.
7.See Katz et al., Documenting Discrimination, supra note 5, at 655.
8.See Katz & Baldwin, Counting Votes, supra note 2, at 24.
9.See Katz et al., Documenting Discrimination, supra note 5, at 655.
10.Katz & Baldwin, Counting Votes, supra note 2, at 25. This statement is in some contrast with their replication of our estimates, which confirms that ideology, a factor outside of the merits of a case, influences the conclusions that trial courts reach.
11.Id.
12.Id.
13.Id.
14.To be clear, Katz and Baldwin are a bit ambiguous about exactly what they think success rates are evidence of. They variously suggest that liability rates are evidence of “obstacles to minority political participation,” id. at 23, or of the prevalence of “voting problems,” Katz, Mission Accomplished?, supra note 4, at 142, or of “the persistence of discrimination,” Katz et al., Documenting Discrimination, supra note 5, at 657. These definitions leave some uncertainty about what underlying fact Katz and Baldwin hope to measure, whether it is the raw number of discriminatory laws or practices, the number of people subject to discriminatory practices or the egregiousness of discrimination, or something else.
15.George L. Priest & Benjamin Klein, The Selection of Disputes for Litigation, 13 J. Legal Stud. 1, 1–2 (1984).
16.Id. at 2.
17.Id. at 4.
18.Cf. Robert E. McCormick & Robert D. Tollison, Crime on the Court, 92 J. Pol. Econ. 223 (1984) (showing that arrest rates may reflect enforcement rates as well as offending rates).
19.Katz & Baldwin, Counting Votes, supra note 2, at 23.
20.As we pointed out in Judging the Voting Rights Act, a closely related trend appears in the overall data as well. Liability rates have been steadily falling for over two decades in section 2 litigation. We took pains in our earlier work to emphasize that one should not be quick to assume that this means that voting discrimination is on the wane.
------------------------
Preferred Citation: Adam B. Cox & Thomas J. Miles, Documenting Discrimination, 108 Colum. L. Rev. Sidebar 31 (2008), http://www.columbialawreview.org/Sidebar/volume/108/31_CoxMiles.pdf
