Will Justice Fall to Bits? Expert Systems and Lawi
Introduction
Almost thirty years ago two of the most eminent researchers in the field of "artificial intelligence" (AI) made the following predictions: Within ten years
-
a digital computer will be the world's chess champion unless the rules bar it from competition;
-
a digital computer will discover and prove an important new mathematical theorem;
-
a digital computer will write music that will be accepted by critics as possessing considerable aesthetic value;
-
most theories in psychology will take the form of computer programs, or of qualitative statements about the characteristics of computer programs.1
The best of the current chess playing machines have rankings approaching the 2500 range. This is very good, but still short of world championship level. However, the rating system overrates the machine, apparently because the machines cannot take advantage of the opponent's "style". Thus, a human player with the same ranking as a machine may be expected to systematically beat the machine after a number of games.2
Although computers have been used as tools in the proof of some important mathematical theorems, there is no suggestion that the machine itself has proved the theorems. The machine has been used in the usual way to perform prodigious computations or to act as a reliable bookkeeper. There are programs which "discover" mathematical theorems, but it can scarcely be claimed that they are "important" and it seems unlikely that the second prediction will be fulfilled for many years.3
I trust that the reader is aware of the inadequacies of the last two predictions.
How could two such researchers be so wrong? The answer must necessarily be uncertain, but I think that the reason lies in the early attitudes to research in "artificial intelligence". At the risk of some simplification, it was thought that it would be possible to build an "intelligent" machine.
The most recent and most comprehensive text book on the subject describes the approach in this way:4
"One idea that has fascinated the Western mind is that there is a general purpose learning mechanism that accounts for almost all of the state of an adult human being. According to this idea, people are born knowing very little, and absorb almost everything by way of this general learner....AI students often rediscover it, and propose to dispense with the study of reasoning and problem solving, and instead build a baby and let it just learn these things."
The idea has been abandoned now. Quoting again from Charniak:5
"We believe this idea is dead, killed off by research in AI...What this research has revealed is that for an organism to learn anything, it must already know a lot"
Although it is now known that the approach will not work, the years spent in pursuing that line of inquiry nearly ruined AI as a legitimate area of research. The Lighthill report recommended to the British government that funding for AI research be discontinued.6 The main line of support for this recommendation was the gross discrepancy between the hopes and promises of AI when compared with the performance actually delivered.
As is known to anyone who reads the daily newspapers, AI research did not cease. There are probably three major factors which have led to the recent dramatic increase in AI research.
First, and not insignificantly, there is the well known increase in the power of computers which occurred during the late 70s. This helped to overcome the combinatorial explosion which defeated many AI paradigms even in small programs.
Secondly, computer languages and programming styles have been developed which are more suitable for AI problems. Although these are still very far from natural language processors, they are able to aid the development process and are more suited for experimental purposes.
Finally, and most importantly, the goals of AI research have been redefined towards the possible rather than the vague ideal of constructing a thinking machine. Definitions of AI in modern textbooks are cautious and operational in nature, more on the order of a catalogue of current research than a statement of its final aim.
There is broad agreement that the catalogue includes robotics, natural language understanding, game playing, vision and expert systems. Most of the action these days concerns expert systems and the remainder of this article will be devoted to this one part of AI research.
The reason for this focus is that expert systems represent one of the most successful areas of current AI research. The programs are useful even to the point of being of commercial value. Their operation can be described in non-technical terms so that they are more interesting to a general audience than a description of game playing programs which are technical. Finally, and most importantly, it is very likely that expert systems will have a significant social and economic effect in the very near future.
What are expert systems?
The goal of building an expert system is to make the knowledge of a person who is an expert in a certain area available when the expert is not. An "expert" in this context is someone who performs a particular task better than the rest of us. He or she may be an expert at diagnosing certain types of disease, at giving certain types of legal advice, at knowing where to drill for oil or at mixing concrete for building a dam.
If we are able to build an expert system which captures the knowledge of the expert, then it will be possible to diagnose the disease, give the legal advice, know where to drill for oil and how to mix the concrete all without the need for the actual expert to be present. The knowledge of the expert may be more widely utilized and will live on in a useful form after the departure of the expert.
One salient characteristic of expert knowledge is that the experts themselves will usually find it difficult to say precisely how they arrive at a particular solution to a problem which confronts them. There is no well-defined sequence of steps to follow; the expert will often describe the process as one of "experience".
This lack of a rigid format meant that until recently such tasks were beyond the scope of standard computer programming techniques since these techniques require a precise sequence of steps to be prescribed. The sequence might be very long and quite complex and there is no doubt that traditional computer programs have accomplished some amazing tasks, but the sequence must be there in an explicit form.
The modern expert system circumvents this difficulty by structuring the program into two parts. The "knowledge base" is independent of the other part of the program and consists of human knowledge painstakingly put into a form which can be manipulated by the other part of the program. More about the form of the knowledge base in a moment.
The other part of the expert system is the "inference engine" which manipulates the knowledge base. In the ideal version, the inference engine is completely separate from the knowledge base, so that it is possible to remove the existing knowledge base and replace it with another. The inference engine with no knowledge base at all is known as a expert system shell. Shells may be purchased from a variety of suppliers and are of greatly varying quality.
The inference engine is a standard type of computer program, ie, it follows a rigidly prescribed sequence of steps. This sequence of steps is a precise description of how the knowledge base is to be manipulated.
The most popular form for the knowledge base is the production rule.7 A production rule is a rule of the form:
IF (some situation) THEN (some action).
Production rules became widely known as a result of psychological research in the 1950s and 1960s. The fascination of modern AI research with production rules dates from their uses in these psychological models.8
It is interesting and instructive to examine the simple psychological model in order to understand the use of production rules in expert systems. It was at one time conjectured that a significant part of human intelligence could be explained by a relatively simple model that functioned as follows:
A short term memory (STM) holds a small collection of "facts". The capacity of STM is thought to be seven items. When a new item is added to STM, it is added as the first element in the list and the oldest item is dropped from the back if STM is already stocked to capacity.
Long term memory (LTM) consists of remembered relationships between elementary facts; these relationships may be modelled by production rules. The "IF" part of the production rule consists of elementary facts and the "THEN" part of each rule orders the addition of a new "fact" to STM.
The "thought" mechanism then consists of repeated simple cycles. The production rules are scanned until one is found for which all of the facts in the "IF" part are in the STM. When that happens the rule is said to "fire" and the appropriate modification of the STM then occurs. Since the contents of the STM are now different, the process may repeat, finding another rule which fires with a consequential modification to the STM and so on. In this way, an initial knowledge of facts stored in the STM is used in conjunction with the rules of the LTM to "learn" new facts. The process may be viewed as an elementary form of deduction.9
The model fails to provide a complete model of the human reasoning process, which may not be surprising, but it has continued to serve as a psychological model in a number of specialized areas of learning and perception. This latter fact might be surprising since the production rule model seems to be such a simple one.
The simplicity is deceptive however. The original interest in production rules was in relation to the theory of computation. One of the results of the theory is that production rules any computer at all!10 At least from the theoretical point of view, the production rule models are very powerful. Their use in expert systems derives from the fact that rules provide a very practical method of programming expert knowledge.
An Example: FINDER11
On 15 November, 1978, Mr Alan George Parker was an outgoing passenger on a British Airways flight out of Heathrow. While awaiting his flight, he found a gold bracelet lying on the floor of the International VIP lounge. He handed the bracelet to a British Airways official along with a note of his name and address and a request that the bracelet be returned to him in the event that the owner of the bracelet could not be found. The bracelet was never claimed. British Airways sold the bracelet and kept the proceeds.
The situation is characteristic of a group of cases where one party finds a chattel which is located on premises occupied or owned by someone who is not the owner of the chattel. There is no doubt that the owner of the chattel is entitled to reclaim it, but the interesting contest occurs when that person cannot be found. Who should be entitled to the chattel? Why?
The law which governs such disputes is contained almost entirely in the decided cases. Although there are not a large number of cases, they are complicated and difficult to resolve. Since each of the parties is seeking a windfall, policy is difficult to formulate and there is little sense of justice to guide the court. Of these cases, one judge has said:12
These cases...have long been the delight of professors and text writers, whose task it often is to attempt to reconcile the irreconcileable.
FINDER is an expert system which gives legal advice on problems such as that faced by Mr Parker. Had FINDER been available for consultation, it would have provided the advice shown in Figure 1. The advice is good: the English Court of Appeal held in favour of Mr Parker on grounds very similar to that given in the FINDER opinion.13
FINDER may be used as an example of how the production rule model allows for the incremental construction of a knowledge base. One of the rules in the FINDER system (translated into a rough and ready English) is:
IF (the other party is the occupier of the real estate where the chattel is found AND the chattel is attached to the real estate) THEN (the other party has exercised control over the chattel)
At some point in the Parker consultation, the system will ask if British Airways is the occupier of the place where the chattel was found. The user of the system must know whether or not British Airways was the occupier of the VIP lounge.
"Occupier" is a legal term which means something more than the usual English usage of the word. It is also a "primitive" in the FINDER system, ie, FINDER has no knowledge which defines "occupier" in terms of other words or relationships.
Consequently, the FINDER system requires a certain amount of legal expertise on the part of the user to make decisions about whether or not a particular person is the "occupier" of premises.
The level of knowledge to be built into the system is part of the design criteria determined by a consideration of the audience which will be using the system. FINDER itself is an experimental system which was intended to be demonstrated to lawyers; consequently, leaving the term "occupier" as a primitive term within the system is justifiable and sensible.
If FINDER were to be used to give advice directly to a client, then it would be necessary to include knowledge (in the form of additional rules) within the system which would assist the user in determining if the other party is an "occupier".
But new rules are easy to add if the knowledge base is carefully constructed, since adding rules to evaluate "occupier" will not have an effect on the performance of the rest of the system. The models can be built and tested on an incremental basis.
It will be seen that building an expert system does not require the usual type of programming skills. Indeed, if the shell is properly designed, it should be possible for the expert to write the knowledge base directly.14
The Use of Expert Systems
The most interesting expert systems are those which are intended to give advice in professional areas such as law, medicine or management. Most people involved in building expert systems in these areas are quick to claim that they do not foresee the possibility of the systems replacing the professional. As will be seen, I do foresee such a possibility.
I expect the use of expert systems to go through five phases. Although I expect that the phases will be similar in all professions, it is best if I explain the way in which I expect them to be used in the legal profession.
In the first phase, they will be used as advisors to professional lawyers in areas where most professionals may not be entirely up to date. In this phase, Mr Parker's lawyer might consult FINDER as part of his research into the problem. In this role, expert systems are not too much different from professional reference texts.
In the second phase, the systems will be used by para legals to assist in giving advice in matters where the expense precludes the use of a fully qualified lawyer. The predominant users in this phase would be neighborhood legal centres and lower level administrators who need to make legal decisions in some well defined area.15
In the third phase, the first controversial one, the systems of phase two will be available directly to non-professional users. In addition, the type of advice which the systems will offer will expand beyond the purely administrative area. Mr Parker might directly consult FINDER as his first step in order to decide if his case is strong enough to warrant the expense of proceeding to consult the professional.
In the fourth phase, the systems will develop to the point of giving advice which is comparable with that of the professional in areas which are of interest to clients who would ordinarily visit a practitioner. At this level, the systems begin to have a significant financial effect on the profession; as a result, calls for control measures to be established are most likely to be found at this phase.
In the fifth and final phase, the systems will be used to resolve disputes between the parties, ie, the expert system will become a "judge". Some SUPER-FINDER of the future will be good enough that the parties will submit the dispute directly to the expert system.
The last two categories are the ones which provoke both disagreement and dismay. The usual basis of the argument against the development is that expert systems will never be as good as humans in the jobs. In what follows, I will assume that this is true.
In order to develop the argument, take the most controversial of the above stages, the "computer judge".16 There is a natural horror at the concept of "machine justice" and not even the most enthusiastic supporter of expert systems would suppose that a computer could rival existing judges.
If performance were the only consideration, then it is true that expert systems would not replace professionals. But the choice is not between a human judge and a computer judge. The costs of litigation already preclude the use of the court system by a significant portion of society. The choice for those outside the system may be between machine justice and no justice at all.
The policy decision here would be the degree to which we sacrifice quality of justice for ease of access. It is not a policy choice which is foreign to us: Consumer tribunals in all western countries provide a more rough and ready justice in order to guarantee access for a class of people formerly locked out of the existing court system.
But the use of tribunals is limited, for there is a limit to human resources. While consumer claims tribunals are workable because of the size of the consumer population, it would be absurd to attempt to establish a "finders tribunal". Nor is it feasible to establish low cost general purpose courts or tribunals, for the knowledge required inevitably determines a minimal cost which is still expensive by the standards of many.
There is another, more fundamental, sense in which human resources are limited. Consider the following example:17 there is a disease for which a test is known; the test has a very high probability of detecting the disease; if diagnosed early enough, the disease may be cured with near 100% effectiveness. The test is the Paps test, the disease is cervical cancer and thousands of women die from it each year. Why?
The argument is that virtually the entire medical establishment would need to be devoted to this one problem in order to eliminate the situation. Yet, the procedure is a relatively simple one to mechanize. There is nothing "inhuman" about the process; even though individuals might prefer dealing with a human than a machine, that is simply not a choice which is available.18
The situation is surely the same in law. A rule which referred all "Parker" type disputes to FINDER might be better than the rule which resolves all disputes in favour of the strongest. Yet, because of the small sums usually involved in such cases, it seems likely that the latter rule is the one which is actually used to resolve most of these cases.
The Problem of Control
If it is correct that expert systems will be used in the ways suggested, then quality control becomes a major problem. The benefits suggested will not be realized if the advice given by the systems is second rate.
In one sense, expert system programs are just another way of packaging knowledge. So, it is argued, they should be no more threatening than textbooks. But textbooks are used by experts themselves and the very nature of expert systems means that they can be built so as to be used directly by the "client".
Furthermore, textbooks are subject to a more or less formal review procedure by human experts in the field. Textbooks are by their nature open and subject to evaluation. Their contents can be absorbed in the usual way and it is known how they will be used. By contrast, expert system programs are very difficult to evaluate merely by looking at them. As mentioned above, the use of production rules makes it easy to modify the operation of a system, but the interaction of the rules is opaque. Evaluation is much more difficult than with other forms of knowledge packaging.
The nature of expert systems discourages reflection on their operation. Whereas a lawyer might read a passage in a textbook and consider it critically before applying it, the form of expert systems discourages any analogous process. They are written to give advice and their form clearly anticipates that the advice is to be taken uncritically.
The attitudes expressed by some expert system builders do nothing to allay this concern. Consider the quote from an unnamed "medical expert" in a recent newspaper article:19
"Even a lousy doctor can make a fairly accurate diagnosis with an expert system, providing he accepts the recommendations made by the system and doesn't go and do something foolish anyway."
Now it may be true that we have "lousy" practitioners in the professions, but at least there is a well defined system of entry and of practice which is intended to produce a minimum standard of competence. How can we establish that an expert system meets some minimum level of competence? A "lousy" expert system is going to give advice which is just as bad as a "lousy" doctor or lawyer.
The same newspaper report claimed that nearly 100 medical systems will be introduced in the United States within the next six months, but with no mention of any quality control mechanism.
The original answer was that the systems would only be used by professionals, but the quotation above shows that this cannot be relied upon to provide the necessary control.
There is the related problem of keeping the programs up to date. Again, there are more or less formal mechanisms for ensuring that human professionals are not entirely without knowledge of the developments which have occurred since they completed their professional training. What system can be devised to ensure that expert systems are kept "informed" of new developments and changes in the area in which they claim expertise?
The lawyer's answer, that the laws of product liability and of professional negligence will maintain standards, is very unlikely to be correct. When used by professionals, the developers of the systems will be protected by contractual exclusion clauses. It might be that market forces will impose some standards control for these systems. But if the above quotation is correct, it may be negligent not to follow the advice of a good system.
When used directly by "clients", the end user is unlikely to be in a position to recognise, much less prove, negligent construction. Even if poor quality is recognised, the costs of a successful litigation are likely be prohibitive for most users. Unless they have access to a very good product liability expert system!
It may be that the control problem can only be solved by the establishment of some form of licensing system, so that expert systems will be treated the same as human experts. If that is the case, then there is not a great deal of time in which to determine the form and mechanics of licensing. Expert systems are with us now; the problems raised by them will be with us tomorrow.
i This article originally appeared in 62 Current Affairs Bulletin 13-18 (1986)
1 Simon and Newell, "Heuristic Problem Solving: the next advance in operations research" 6 Operations Research 1 (1958)
2 Charniak, E and McDermott, D, Introduction to Artificial Intelligence, Addison-Wesley, 1985, p292.
3 For programs which discover mathematical conjectures, see Lenat, DB, "AM: Discovery in mathematics as heuristic search" in Davis, R and Lenat, DB (eds) Knowledge-Based Systems in Artificial Intelligence, McGraw-Hill, 1982.
4 Charniak and McDermott, supra, p609.
5 ibid, p610.
6 See Artificial Intelligence: a Paper symposium, Science Research Council, Gt B, 1973.
7 Post, E "Formal reductions of the general combinatorial problem" American Journal of Mathematics Vol 65, p197, (1943).
8 Newell, A and Simon, H Human Problem Solving, Prentice-Hall, 1972.
9 The system described is "forward chaining"; most expert systems are "backward chaining"; see Charniak and McDermott, supra.
10 Technically speaking, production rule systems are equivalent to Turing machines.
11 Research on a group of related models which include FINDER is supported in part by the Australian Research Grants Scheme and in part by the Law Foundation of New South Wales.
12 Hibbert v McKiernan [1948] 2 KB 142, per Lord Goddard.
13 Parker v British Airways [1982] 1 All E R 834.
14 I should also add that most available shells fail completely to live up to this ideal. For a list of shells which are commercially available, see Byte, Vol 10, April, 1985 p306.
15 Phase two might actually precede phase one: I have been told that there is an Australian system under development which is to advise on the entitlement to welfare payments.
16 Much of the controversy is based on misunderstanding: see Tyree, A L "Can a 'Deterministic' Computer Judge Overrule Himself?" 7 Rutgers J of Computers, Technology and the law 381 (1980).
17 The example is due to Richard Bellman, one of the early proponents of the use of computers in medicine. I do not know if Bellman wrote the argument anywhere. I heard him refer to it on several occasions during his visits to New Zealand in 1972. I have recently been informed that the Paps test is not as reliable as was then thought. That makes Bellman's argument all the more powerful.
18 The problem is not confined to those situations where there are large numbers of individuals involved, nor would widespread high performance expert systems cure all of the problems: see Calabresi, Tragic Choices, New York, 1978.
19 The Australian, "AI breaches Barrier of Medical prejudice" 1 October, 1985.