DON’T TEACH STATISTICS TO LAWYERS!

Bernard Robertson* and G A Vignaux**

*Editor, New Zealand Law Journal

**Institute of Statistics and Operations Research, Victoria University, Wellington, NZ

ABSTRACT

Many lawyers have been taught statistics at school or university. The conventional teaching of statistics, particularly in "service courses", leads to lawyers absorbing a number of fallacies. These include:

* probability assessments can only be made on the basis of long runs and repeated experiments

* there is a distinction between "legal probability" and "mathematical probability"

What lawyers need to be taught is not techniques for manipulating statistical data but a logical and principled approach to reasoning in conditions of uncertainty.

INTRODUCTION

Courts make decisions in conditions of uncertainty. There is uncertainty in the proper interpretation of the law -- which we are not concerned with here -- but also uncertainty about facts. The decisions to be made usually relate to whether or how a particular past event occurred and are based upon different kinds of evidence both scientific and non-scientific. In considering the thought processes involved, a number of questions arise. Amongst the easiest to answer are:

* How is forensic scientific evidence to be analysed and presented?

* How should forensic scientific evidence be combined with the other evidence in the case?

But there are other questions, such as:

* How do we know when evidence is "relevant" or "probative"?

* Can a charge be proved "beyond reasonable doubt" by inference from facts which are not themselves proved beyond reasonable doubt?

* If the standard of proof in a civil case is "the balance of probabilities" to what level must the individual elements be proved?

* Is a confession "reliable" if extrinsic evidence shows that parts of it are true?

* Is an eye-witness identification more "reliable" if extrinsic evidence shows that the accused could have been there?

It is our view that there is a single rational set of principles that tells us how to approach these questions. Those principles consist of the laws of probability and the theorems that can be deduced from them. To deny this, for example by arguing that the axioms of probability apply only to repeated experiments, would seem to be to suggest that there is no rational way of answering any of the questions above, other than the first. Such a position seems to us rather surprising but is in fact the position projected by conventional statistical teaching and absorbed by many lawyers, if not at their mother's knee, at least when they attended school and university.

Teaching conventional statistical techniques therefore turns out to be positively harmful. It places barriers in the way of lawyers understanding how the other questions can be approached in a rational fashion. This is demonstrated by quotations such as the following:

How could a judge come to make such a comment? In this particular case the problem was something called "the probability of paternity". When paternity evidence is given by an expert witness, the conventional approach is first to calculate a "paternity index". This is the ratio between the probability of the scientific evidence if the respondent was the father, to the probability of the evidence if a randomly selected male were the father. This index is then multiplied by a so-called "neutral prior" to produce a "probability of paternity". This procedure was recommended (though not to the exclusion of alternative assumptions) by Essen-Möller (1938) and rapidly became standard. It has been criticised on a number of grounds and rationalised in a number of ways. For example, Kaye(1989) shows that it is possible that two non-excluded males in a case each have a probability of paternity (of the same child) of over 90%. But its overwhelming flaw is that it leaves courts with no guidance when there is expert evidence of a probability of paternity on one hand and evidence that the couple concerned could not have had intercourse at the relevant times on the other. Numbers of cases in the law reports show that Judges are completely baffled by this situation (for example, Loveridge and Adlam (1991) and the case referred to above, re J.S.(1981))

A LOGICAL APPROACH TO PROBABILITY

The problem, then, of conventional statistical teaching is that it creates the impression that probabilistic analysis is only appropriate when one is dealing with counts and frequencies and not when dealing with other forms of evidence. A clear example is to be found in Adams (No 2)(1997) where the Court refers to a submission:

This is all the greater shame because a logical approach to probability has a great deal to offer in analysing, explaining and, indeed, in reforming the Law of Evidence. The nature of that contribution has been the subject of vigorous debate within a movement known as the New Evidence Scholarship over the last generation. We have argued (Robertson and Vignaux, 1993) that opposition to probabilistic analysis of the Law of Evidence is created almost entirely by impressions created by conventional statistical teaching.

One or two examples will suffice to show how a logical approach to probability can contribute to evidence teaching in general.

RELEVANCE

The fundamental principle which underlies the Law of Evidence is relevance. To be admissible, evidence must be relevant. The rules of evidence (such as those on hearsay or on confessions) are then applied to determine whether the relevant evidence must be excluded.

What does "relevant" mean? One of the few written definitions is in the US Federal Rules of Evidence, Rule 401:

Notice that relevance is defined in terms of the impact of evidence on an (implicit) prior assessment of probability. What is the measure of such impact? The answer is the Likelihood Ratio (LR). This is the ratio of the probability of the evidence given one hypothesis (relating to the fact in issue) to the probability of the evidence given the alternative hypothesis, often simply the negation of the first. For example, the probability of getting this blood match if the accused left the blood to the probability of getting the match if someone else did (i.e. the accused did not). If the evidence is equally probable in either case, then it does not help us to distinguish between the hypotheses and is irrelevant. In other words, a LR of 1 corresponds to irrelevance. Where LRs are not much greater or less than 1 the evidence may still be excluded as its value does not justify the potential cost of introducing it. This however is a decision for the judge (one problem being that the judge must frequently make this decision without having heard the other evidence in the case).

Witnesses have no business making their own decisions about the relevance of evidence and yet this is exactly what many scientists do in the guise of applying significance tests. Take for example, an allegation of systematic racial discrimination. The figures for promotions in an organisation might be found to be 18 times more probable if systematic discrimination were occurring than if there were no discrimination. Application of a significance test, even at the 95% level, would lead the witness to say that the figures were "not statistically significant". Similarly, application of a significance test at the 99% level would lead to the suppression of evidence giving LRs of up to 98. Yet much of the evidence that courts customarily receive, such as eyewitness identification or information about motive, would probably yield LRs less than this. Yet arbitrary decisions by scientists applying conventional significance tests can lead to the suppression of equally useful scientific or statistical evidence. The role of the expert witness should be to testify to the LR of the evidence and leave it to the court to decide whether to receive it or not.

THE ULTIMATE ISSUE RULE

There is a traditional rule of evidence that a witness should not testify as to the issue the court has to decide. For instance, one must not give an opinion that the accused committed the offence. Under the pressure of "probabilities of paternity" and testimony claiming "statistical significance", courts have weakened their enforcement of this rule and commentators have come to regard it as anachronistic. LR analysis, on the other hand shows the ultimate issue rule to be logical. It clearly establishes the respective roles of witness, judge and jury. Non-expert witnesses simply testify as to what they have observed. The jurors’ general knowledge and experience equips them to assess LRs for such evidence informally and intuitively. Where jurors cannot assess LRs, for example with blood and DNA evidence, expert testimony is called for. But the expert should be confined to giving evidence in the form of a LR. The assessment of the prior and posterior odds is a matter for the judge or jury.

Not only does the LR approach defend the ultimate issue rule but it makes it easier to explain the rule and its rationale to law students. This, of course, presupposes that the students have not been exposed to courses that constrain probabilistic reasoning to long-run frequencies and forbid the assessment of the probability of individual past events.

INFERENCE BEYOND REASONABLE DOUBT

When the infamous "dingo baby" case was appealed to the High Court of Australia (R v Chamberlain, 1984) one of the key questions was whether facts could be proved beyond reasonable doubt on the basis of other facts which themselves were not proved beyond reasonable doubt. The judges in this and subsequent cases were divided on the issue and the question has been extensively discussed in the legal literature since.

Analysis of the problem using Bayes Theorem (see Robertson and Vignaux, 1991) reveals

* what matters is the probative value of the evidence (i.e. its LR) rather than just whether the evidence is proved to be correct.

* facts can indeed be proved by a series of converging LRs to a higher standard than any of the individual items of evidence.

Analysis of this type would not occur to those who regard probability as only applicable to long runs and repeated experiments.

FORENSIC SCIENCE

Within the forensic scientific world the battle between the LR and significance test approaches is all but won. There is still shouting from some lawyers and judges. Even the updated second NRC Report on the use of DNA evidence (US National Research Council (US) (1996)), now commonly known as NRC2, admits that the LR approach is capable of solving a wider range of forensic problems (e.g. mixed bloodstains) that the conventional method.

All that remains, therefore, is to dispose of the "horses for courses" argument, that conventional techniques work perfectly well in some circumstances (e.g. single bloodstain and a single suspect) and may as well be used in those circumstances. One problem, of course, is to know what those circumstances are and NRC2 itself gets this wrong when it applies the conventional approach to evidence obtained by database searching and comes up with the wrong answer. From an educational point of view, it is far easier and intellectually more satisfying, to teach principles of general application than a series of ad hoc devices each of which only applies in defined circumstances.

CONCLUSION

Teaching law students a logical approach to probability would have the following advantages:

* It would equip them with a yardstick against which to criticise rules of evidence and judicial directions to juries.

* It would equip them to analyse all kinds of evidence rationally and to combine evidence from different sources

* It would de-mystify scientific evidence since it would be seen to be subject to exactly the same rational criteria as all other forms of evidence.

Conventional statistical service teaching, on the other hand, causes lawyers to regard scientific and statistical evidence as being categorically different from "ordinary" evidence and leaves them without any rational tools for analysing the latter.

REFERENCES

J.S. (A minor), in re. (1981) Fam. 22 (CA)

Loveridge v. Adlam (1991) NZFLR 267 FC

R v. Chamberlain (1984) 51 ALR 225 (CA)

R v. Dennis John Adams (No 2) No 96/6927/Z5 16 Oct 1997 Court of Appeal(England)

Ormrod LJ, in Re JS (a minor) 1981 Fam 22 (CA)

US Federal Rules of Evidence, United States.