The Keller Plan at Law School

1 The Keller Plan at Law School

1.1 Abstract

The "Keller Plan", also known as the "Personalised System of Instruction (PSI)", has been widely used in disciplines other than law. The method is known to be at least as effective as intensive small group teaching in those areas where it is appropriate for use. In certain aspects, it is demonstrably superior to commonly used methods of teaching. We describe the method and an implementation in a class of 130 students at the University of Sydney.

1.2 Introduction

It seems axiomatic among law teachers that small group "Socratic" teaching in one form or another is a superior teaching method.1 Unfortunately, it is also an expensive teaching method. While new and/or well funded schools have been able to embrace it, others find that they cannot afford the method.

Our School is one of the oldest and most prestigious in Australia. For many years, it has also been one of the worst funded. Our exploration of other teaching methods was driven by our dissatisfaction with the traditional lecture method of teaching, fueled by these economic restraints. The search was for methods which were demonstrably better than those currently used, but which could be implemented with little or no increase in resources.2

Our explorations led us to consider the Keller Plan teaching method. Educational research showed that the Plan was at least as effective as intensive small group teaching, but available at a price that we could afford.3 The Keller Plan, also known as a Personalised System of Instruction (PSI), is a teaching method which combines "mastery learning" with principles of reinforcement learning theory.

Since we find that few law teachers are familiar with these concepts, the first part of this paper will be an introduction to mastery learning and its particular refinement in the Keller Plan. We will then describe some of the research which compares the Keller Plan with other teaching models, and finish with a description of a particular implementation of the Keller Plan which we used in 1990 at the University of Sydney.

1.2.1 Mastery Learning Models

If we can describe what it is that we expect our students to learn, is there any reason why they cannot learn it? If the answer to this question is "yes", then perhaps we are not good teachers or (and this may be the same thing) we are not teaching at a level which is appropriate for our students. Mastery learning assumes that our students can learn the material which we teach.4 Under that assumption, the basic concepts of mastery learning involve only two things.5 First, we tell our students what it is that we wish them to learn. Secondly, we provide them with the means to evaluate their own performance, that is, to determine if they have in fact learned the material. Both of these require some elaboration.

1.2.2 Behavioural objectives

Mastery learning requires that the desired student performance be stated precisely. Objectives such as "students should learn to be creative" are unacceptable because there is no objective measure to determine whether the objective has been met. The objectives should be "behavioural" in the sense that they provide a specific performance measure. This measure can be used by the student (and the teacher) to determine if the student has achieved "mastery". "Mastery" does not mean that the student is a "master" of the subject matter. It means only that the student has met the performance standards of the particular course. It is possible to achieve "mastery" in a beginner's course.

Behavioural objectives sometimes arouse controversy. The argument against them is that the most important products of education are incapable of being "measured". How can we state the broad aims of a liberal education in behavioural terms? The requirement for behavioural objectives, it is argued, is opposed to these aims, for the focus is on that which can be measured rather than on what cannot.

While such arguments have a superficial attraction, we do not believe that they stand careful scrutiny. Can we seriously expect students or anyone else to be satisfied with vague aims? If the aims are not vague, then why can we not state them in a reasonably precise way? It is surely unsatisfactory for teachers to adopt a "I don't know what it is, but I recognise it when I see it!" approach to teaching.

Sometimes the argument against behavioural objectives is the result of a misunderstanding. There is no reason for the objective to be simple or elementary. "State three major criticisms of the Australian Constitution and suggest remedies" is hardly a simple objective, yet it is clearly objective. Nor do behavioural objectives exclude personal judgments or opinions. In the objective just mentioned, there is obvious room for opinion in the word "major". The personal judgment of the teacher determines what is an acceptable level of performance.

The purpose of the objectives is not to limit the educational process, but to assist in its definition. Students are entitled to know what the teacher expects. Mastery learning assumes that the teacher knows what is expected, and is skilled enough at the job to be able to state it.

1.2.3 Examining performance

"Mastery" is determined by examination of the student. The student is examined to see if he or she has achieved the stated objectives. Thus, the subject matter of the examination is the stated objectives; there is no room for examining objectives which are not previously stated.

The examination procedure is somewhat different from the methods which law teachers usually use. In mastery learning, we attempt to examine using "criterion referenced" examination methods. Most law teachers probably use a mix of criterion referenced and "norm referenced" methods.6 Norm referenced methods attempt to place a student by reference to the performance of other students. There is clearly a large component of norm referenced examining procedure in most law school examinations, for many faculties require a specified distribution of marks. It would be a rare school which did not call for explanation when one marker differed consistently and substantially from the local norm.

Norm referenced questions must discriminate among students, for a question which does not discriminate adds nothing to the testing process beyond shifting the average. Criterion referenced testing, on the other hand, sets specific performance standards. Questions are measured against the performance requirements rather than by the profile of answers. A criterion referenced question may be satisfactory even though there is a near perfect answer rate, that is, even though it contributes little or nothing to the discrimination process. Similarly, a question which is failed by all students may be a perfectly acceptable question, although it may indicate a deficiency in the materials or in the teacher.

Of course, it would be unusual for any test to be completely norm or completely criterion referenced. A typical law teacher will normally pay heed to performance criteria when drafting questions, but, as noted above, will also be disturbed if there is a 100% pass rate on questions. In fact. the marking standards in most examining probably shifts subtly during marking in order to ensure that the final marks fit into some norm.

In the context of mastery learning, the aim is to key the test questions to the stated behavioural objectives. The test questions should be criterion referenced against the stated objectives. If the objective is one which we should expect most students to learn easily, and there will be some, then there is no objection to a question which 100% of the students pass on the first attempt.

1.2.4 Introducing Mastery Learning in Traditional Courses

It is clear that the concepts of mastery learning can be introduced into existing courses. One way of doing so is to give the students a statement of the behavioural objectives at the beginning of each week, and to test them at the end of the week. In the standard mastery learning implementation, the weekly tests do not count as part of the final mark. Their sole purpose is to provide the student with a measure of their understanding of the material. The weekly tests, called "formative" tests in the mastery learning literature, should provide immediate feedback to students. If the student does not achieve mastery, then he or she should be referred to some alternative materials which will assist in overcoming the learning deficiency.

It will perhaps come as a surprise to many teachers to learn that adding mastery learning techniques results in a measurable improvement in student performance.7 The credit for this is probably divided equally between teacher and student. The formulation of behavioural objectives provides a focus for both. The teacher is forced to articulate objectives that may have been unstated previously, and the examination procedure can make the student aware that he or she may not know the material as well as thought.8

1.3 The Traditional Keller Plan

In the 1960's, Fred Keller devised a teaching plan which is now known as the Keller Plan or the Personalised System of Instruction (PSI). Keller, a psychologist, integrated the ideas of mastery learning with those of reinforcement learning theory.9 The Plan is based on several fundamental observations. First, if students are all expected to achieve mastery, then they cannot all be expected to do it in the same time. Secondly, smaller amounts of material are more digestible than larger amounts. This leads to the adoption of a "modularised" course. Thirdly, students will learn better if they are given frequent and immediate rewards. These rewards are, in the Keller Plan, instant feedback on tests and credited marks for success in each unit.

1.3.1 The essential features

The following account given by Keller himself of the main features of his Plan shows very clearly the differences between this teaching model and those which are more traditional:10

"(1) The go-at-your-own pace feature, which permits a student to move through the course at a speed commensurate with his ability and other demands of his time. (2) The unit-perfection requirement for advance, which lets the student go ahead to new material only after demonstrating mastery of that which preceded. (3) The use of lectures and demonstrations as vehicles of motivation, rather than sources of critical information. (4) The related stress upon the written word in teacher-student communication; and finally: (5) The use of proctors, which permits repeated testing, immediate scoring, almost unavoidable tutoring, and a marked enhancement of the personal-social aspect of the educational process".

Keller and his associates implemented a PSI course at the then new University of Brasilia in 1964. The method was refined over the next few years, and Keller's landmark paper was published in 1968.11 There have been thousands of successful Keller Plan courses run throughout the world since that time.12 We have found no reference in the literature to a Keller Plan course in any law school. Before describing the implementation of the Keller Plan in Technology Law, we will discuss in more detail some of the particular features which Keller thought important.

1.3.2 Written materials

Since students must proceed at their own pace in a PSI course, the traditional scheduled lecture or class must be abandoned. Lectures could still be the primary method of communication, but either they must be frequently repeated or reduced to audio or video tape. Most Keller Plan courses have relied on written materials as the primary means of student/teacher communication. In a law course, this has the additional advantage of requiring students to learn in a manner similar to that which they must use for the rest of their professional life.

Unit materials are somewhat different from those which are usually reprinted for student use. Remember that there will be little opportunity for guided discussion. In addition to the ordinary materials, there is a teacher written "study guide". This must provide a "map" to the materials, as well as provide any up-to-date information or opinions. The teacher is free to write his or her analysis of any of the written materials. Most importantly, the study guide must also contain the detailed behavioural objectives of the unit.

If proctors are used,13 the teacher should also prepare detailed proctor guides. These contain general instructions to proctors concerning their role as well as specific instructions on marking the unit tests.

1.3.3 Unit tests

In the typical Keller Plan implementation, students are free to test at specified times during the week. There is no penalty for failure of a unit test. The test is marked immediately by the proctor, and the proctor may seek clarification from the student concerning answers given.

Tests are generally short (20 minutes seems a popular figure) and consist of a variety of question types. The Plan has nothing to say about test form other than that it must test all of the objectives, and it must test nothing which is not a stated objective. Because there is no penalty for failure, the teacher must prepare several equivalent but different tests for each unit.14 We found test preparation to be one of the most demanding aspects of the Keller Plan.

1.3.4 Proctors

The examining procedure used by Keller was labour intensive. It was clearly impossible for the teacher to mark all of the examinations. Indeed, Keller and others thought that it would be a waste of highly skilled talent to employ teachers in that capacity. The examiners were known as "proctors".

Proctors administer the tests and provide immediate reinforcement for student performance. Instant feedback is possible, and a small amount of tutoring is inevitable although proctors are not engaged to act as traditional tutors. On the contrary, Keller cautioned against proctors acting as tutors. They are not there as teachers, and are not intended to impose their views of the materials on students. Proctors provide important human contact in an otherwise "teacherless" course. Proctors also provide valuable information to the instructor or course convener who is responsible for the course. In a typical implementation, a proctor would be "responsible" for 15 to 20 students.

Implementations of the Plan show considerable variety in the use of proctors. In some, students in the same course who had tested earlier were used as proctors. This was thought to provide a more intimate learning environment. In others, postgraduate students were employed in some instances with certain credit points for other courses being allocated for their participation. Another popular scheme was to use students who have completed the course in previous years.

1.3.5 Lectures

The traditional Keller Plan calls for a few non-compulsory lectures and extra demonstrations. The purpose of these lectures is motivational only. The content is not examinable. The lectures help to combat the sense of isolation that may develop by providing an occasion for teachers and students to get together.

We scheduled a series of five guest lectures in Technology Law. The lecturers were all prominent in their area, and the area was always related to the course although not directly overlapping with any of the units. Attendance at these lectures was very poor, usually less than 25% of the class.

1.3.6 Final Examinations

The initial Keller Plan did not include a final examination. However, most implementations of the Plan which are reported in the literature contain a final examination component. Keller himself introduced final examinations in later courses. The reason for this seems a little vague given Keller's concern about student fear of examination often being a cause of poor performance. One reason given later was that his colleagues were more likely to employ the Plan if there was some traditional method of assessment available as well as the progressive unit testing. There is also some research which indicates that students in a Keller Plan course with final examination do better than Keller Plan students without final examinations.15

1.3.7 Effectiveness of the Keller Plan

The Keller Plan is one of the few teaching methods which has a theoretical basis. For that reason alone, it attracted a great deal of attention in the late 1960's following the publication of "Goodbye Teacher…". There are many papers devoted to a study of one or another aspect of the Plan.16

The literature grew so large that it was possible to do a form of scientific analysis of the published results. The study, known as a "meta-analysis", showed conclusively that the Keller Plan was at least as good as very intensive small group "Socratic" methods.17 In comparison with traditional lecture methods, Kulik, et al were able to show that an average student could be expected to improve his or her final examination mark by 8-9%.18 The study verified that students not only learned more, but (and this may be the same thing) they retained it better. Low-aptitude students benefited to the same degree as high-aptitude students, and re-testing after six months showed that the advantage of the Keller students increased.19

Students believed that they worked harder under Keller Plan courses, although the evidence was unclear as to whether they actually worked harder. Given this student belief, it is perhaps surprising that students liked the Keller Plan course better than the alternatives offered.20 If the student perception of working harder is correct, it is perhaps not surprising that they learn more. They may like the method better for a number of reasons. In comparison with the "Socratic" methods, there is less stress. Students like the "go at your own pace" feature which allows them to schedule their own time. The method involves the student more than the traditional lecture. And, of course, they may like the Keller Plan precisely because they believe (correctly) that they learn more. The Keller Plan may also be a more desirable option than the Socratic method for students who are less vocal or less aggressive. Feminist writings have attacked the Socratic method for precisely this reason.21

We believe that one of the primary advantages of a Keller Plan for a law course is that students learn primarily by reading. We found that our students' ability to learn efficiently from reading increased dramatically as the course progressed. Learning from reading is obviously an important skill for lawyers, but one that is often neglected in favour of developing oral skills.

1.4 Technology Law 1990

The research findings of Keller Plan studies convinced us that it was an alternative worth exploring. Our original plan was to implement a "standard" Keller Plan course, together with a research program which would allow us to replace proctors with examination by computer. As our plans developed, we decided to omit the first stage of the program and proceed directly to examination by computer. This may have been a mistake, but at the time we rationalised the decision on the basis that we could not afford to continue a "traditional" Plan beyond the one year experiment. It seemed better to bite the bullet and proceed directly to the second stage.

There is one matter of terminology: most writing on the Keller Plan refers to the modularised course components as "units". In our school, the word is used to refer to the amount of credit attached to a course. We use the word "module" to describe the "chunks" of material.

1.4.1 Pre-1990 course

Technology Law was first offered as an optional course for later year students in 1989. It was taught by lectures and tutorials with an extensive research project and final examination. The course was divided into a number of separate areas covering a range of topics which were capable of forming distinct modules.

The course was,and is, a mixture of "policy" and "black letter" subjects. In some parts, for example, computer crime, there were statutes and students were expected to learn "the law" as well as to explore the underlying policies (or lack thereof). In other subject areas, for example, electronic data interchange or transborder data flows, there is no specialised local law, and the course considered the application of the general law to the special problems together with various international initiatives. There seemed no reason to change the general scheme of the course, and the Keller Plan course covers broadly the same topics.

1.4.2 Keller Course Description and Course Structure

The Kellerised Technology Law course consists of 15 modules. Ten of these cover computer related topics such as Computers and Computer Crime, Electronic Funds Transfers, Artificial Intelligence, Privacy and Data Protection, Freedom of Information. Five modules deal with various problems of biotechnology including Plant Variety Rights, Artificial Conception, Genetic Manipulation and Genetic Fingerprinting.22

Each module is divided into two parts - the "Study Guide" and "Materials". The study guide provides a detailed explanation of what is expected from students with a "procedures" section suggesting a path of study to follow. Objectives are set for each module and suggested issues are presented. The objectives and suggested issues are based on the materials which have been specially chosen to facilitate individual study.

The amount of materials has been kept within reasonable limits. We expected that students would take approximately 10 - 13 hours to read and prepare for testing. This conforms to the Faculty of Law guidelines for course structuring. Other readings are suggested in the study guide, but these are merely for interest and these readings do not form part of the examinable materials. There is nothing in the Keller Plan which requires that student materials be self-contained. The decision was forced upon us by limited library resources.

1.4.3 Reinforcement Components

When a student decides that he or she has achieved "mastery" of the module, he or she attends the Faculty computer laboratory for testing. Tests average 10 items, were unsupervised and generally required 20 - 30 minutes.23 There is no penalty for failure. A student may sit the test as many times as required to pass it.24 Students required an average of 2.5 attempts to pass each test in the 1990 course. Once the test is passed a set mark is awarded - 5 marks for each module in the 1990 course.25

Since proctors are not used, we tried to design the tests to provide the immediate feedback that the Keller Plan requires. When the student fails a question, the reasons for the failure are given and the student is referred to the particular section of the materials which discusses the issue being tested. In addition, the teacher is available freely for consultation concerning difficulties with testing, materials and other concerns. This policy decreases any perceived isolation and has encouraged many students to assist the researchers in the development of improvements for future courses.

1.4.4 Assessment

The assessment for Technology Law, 1990 was divided into two parts - module tests worth 5% for each module passed (a possible 75%) and a final examination (to cover all module materials) worth 25%. To put this into perspective, Faculty guidelines call for an average of 20% of students to receive a "Distinction" or higher. A "Distinction" is a mark of 75 or more. We had a unusually large number of "Distinctions"; see the discussion infra.

1.5 Computer Testing

Our only major departure from the traditional Keller Plan was the use of computer testing. Our hope was that the branching facilities of computers could be used to provide the instant feedback required by the Keller Plan. This would open the possibility of using the Keller Plan in distance education, as well as reducing the continuing costs of the Plan.

We found little assistance in the educational literature for this aspect of the project. It is obvious that computers may be used to administer multiple choice question examinations, and there is no doubt that the resulting reduction in marking and manual bookkeeping is most welcome. The deficiencies of this are obvious. The reasoning process of the student can only be examined implicitly. Yet it is this reasoning process that is often of most interest in examining law students. Examining reasoning by multiple choice questions requires the careful construction of choices in the question. Construction is slow and difficult.

1.5.1 "Tree" Questions

We hypothesised that by using "tree" questions, we could better explore the reasoning of the student and, at the same time, reduce the construction time of questions. An example might clarify this.

One of the units concerns plant variety rights, and one of the specialised issues to be tested concerned the identity of the "breeder" of a plant variety.26 The question was a typical law examination short story which described a person, John, who worked for a university who cultivated a special form of tomato. If the cultivation was in the course of employment, then the university was the breeder. If not in the course of employment, then the employee was the breeder. We wished to test that the student understood that either John or the University might be the breeder, and that the resolution of this issue depended upon whether the cultivation was in the course of employment.

The "tree" question begins by asking baldly "who is the breeder". On the basis of the answer to that question, the next part of the question is a multiple choice question which looks at the possible reasons why the student might give the response actually given. There is no doubt that this question could be cast as a single multiple choice question, but it would be substantially more difficult to construct. With the "tree" question, the examiner is not required to think of so many possibilities at each stage of the construction. It is easer to provide alternatives that do not give away the answer, for much of the question is hidden from the student at each stage of the process.

At the end of the tree, the student is informed that the question has been passed or failed. In either case, we usually gave some additional feedback. In the case of a failed question, we would indicate either the reason for the failure or a reference to particular materials which should be studied further.

The language used both for construction of the questions and for the administration of the entire examining program is LES, a language developed by Andrew Mowbray as part of the DataLex Project.27 LES is available to educational institutions for educational purposes free of charge.

1.5.2 Effectiveness of computer testing

In 1990, the computer testing aspect of the experiment had mixed success. It was completely successful in the sense that it worked. We were able to administer almost 5000 20 minute examinations to 130 students in a 15 week period. The entire system contained approximately 400 questions and, subject to the comments below, performed its duties flawlessly.

We aimed for three alternatives of each question, that is, three separate questions which were intended to test the same objective. An examination consisted (generally) of 10 questions. A question was selected at random from the pool (usually two or three) of related questions. This system has the advantage that there are a large number of possible examinations. It is also convenient to add questions to the pool as and when time allows. Although construction of examinations is still a very demanding process, it seems that it is less so than in a traditional Keller Plan course.

The examination system was not, however, entirely satisfactory. Students were often frustrated by what they saw as inflexible structures and the impossibility of explaining their answer further. On some of the material, there were frequent complaints that the questions merely imposed the teacher's views in situations where the issues were far from clear. Some of the student complaints were justified. In many cases this could be explained by our lack of experience with criterion referenced testing and with the principles of constructing multiple choice questions. This led, in some circumstances, to ambiguous questions or to questions which failed to address the more important issues. These problems will be overcome by teacher experience.28

Some complaints were not justified. Students often read the screen quickly, believing that they had seen the question before whereas it was, in fact, a variation. Some frustration was the result of bright, highly motivated students "failing" examinations for the first time in their academic career. The fact that the "failure" cost them nothing except a further examination did not offset the emotional trauma that they experienced. These problems generally decreased as the course progressed and the students adapted to the new method of examining.

We do not wish to give the wrong impression here. The examination system was certainly successful enough that we are using it again. The system also worked much better than might be expected of a first time experiment. Most students handled most questions in the way expected. We also emphasise that the deficiencies in the existing examination system must be seen in context. That context includes our inexperience with computer based testing and a careful consideration of the alternatives: in our view, there are no affordable alternatives. Many of the questions which caused problems are amenable to simple modifications. The questions which need modification are easily identified by the statistics kept by the examining system.

There is one improvement in the examining system which we believe would still most or all of the student frustration. It is best illustrated by the student reaction to the final examination in Technology Law. That examination was a 60 item true/false examination which allowed the students a two line explanation of their answer. The explanation was optional. If the student gave the "wrong" answer, it could be partially or completely salvaged by giving an acceptable reason. Both the teacher and the students found this to be a valuable modification to the T/F format. Several questions had ambiguities that the examiner had not detected during examination construction.29 There were a significant number of students who gained marks for these questions on the basis of explanations which showed that they had given a reasonable answer.

This system may be incorporated into the computer examination system, although current technology does not allow the complete computerisation. The existing system is easily modified to accept short reasons, even though it cannot assess their validity. The plan would be to have a single "proctor" in the computer laboratory whose task it would be to examine reasons given when the problem is marked incorrect by the computer. This approach would also have a valuable psychological advantage, since the students would feel that they were receiving more personal attention. It also has the advantage (from the student point of view) that attention by a human proctor could only improve their results, never diminish them.

Ideally, of course, the computer marking system would be extended to mark short answer questions. This would reduce the question construction time required by the teacher, as well as being more satisfactory for the students. The SAGES (Short Answer General Examination System) Project is addressing this problem, but it is too soon to report progress. Information may be obtained directly from the authors.30

1.5.3 Can any course use computer testing?

Some of our colleagues have suggested that Technology Law is somehow more favourable for computer testing than other courses. Nothing could be further from the truth. We found that the easiest parts to test were the "black letter law" portions of the course, but we have also found that to be the case with ordinary examinations as well. Perhaps there is a mistaken assumption that a course in Technology Law must necessarily be mechanical. However, as we indicated above, the course is a mixture of policy, politics, jurisprudence and black letter law. We have no reason to suppose that the examination problem, whether by computer or by ordinary means, was any more or any less severe in Technology Law than in any other course taught in the law school.

1.6 Some particular problems

1.6.1 Procrastination

Every teacher has experienced the "procrastinating" student. The problem is exacerbated in a Keller Plan course. We had several students fail Technology Law because of a late start. Our particular problem was acute because of a very poor system of enrolment at the University of Sydney. We had no way of knowing who should have been completing module examinations until late in the semester.

Assuming that the teacher can determine who should be in the class, the problem of procrastination can be dealt with simply by using automatic notification procedures. When a student's record shows that he or she is dangerously behind schedule, the teacher should be notified automatically by the computer system. On the other hand, the Keller Plan is "self-pacing". There are arguments that the entire responsibility should be shifted to the student. We do not subscribe to that philosophy, but we accept that students should be allowed a certain leeway.

1.6.2 The Work Load

The Keller Plan is dangerous. It is easy to be tempted to develop a "complete" course, putting in every article that is relevant to the subject. This must be resisted at all costs. Remember that the students are expected to "master" all of the material. It is poor teaching to misjudge the amount of material that students should be expected to learn.

Our Faculty guidelines call for a total student commitment of 196 hours for a two unit course such as Technology Law. Since we had 15 modules, this meant that students should spend approximately 13 hours per module. Guidelines also indicate that students can read 20 pages per hour, but we believe that this is a gross exaggeration when the written materials are the primary source of teaching. We used a rough guideline of 10 pages per hour. Most of the modules contained about 100 pages of reading material, accompanied by four to six pages of study guide.

  • Student Perceptions

    We find that many students complain about the "work load" in their courses. Complaints are made concerning the number of hours they must spend at the teaching institution, the amount of reading, the vast numbers of different topics and so on. Some of these aspects form part of a "hidden agenda" in many courses and students' complaints could be seen as warranted, at least in the sense that the actual work required is in excess of legitimate expectations. In a properly constructed Keller Plan course students will know in advance of the ACTUAL work load and may then decide whether such a course is for them.

    Another concern that students have about their work load is that they are often prevented from organizing their time because not all of the materials or course outlines are available. In other words, they are happy to take on any size workload as long as they know the actual quantity. Once more, a Keller Plan course would address this problem. The entire course is available for perusal at the beginning of the semester. Since it is complete with detailed outlines and study guides as well as materials, students may plan ahead.

    Our survey showed that students in Technology Law generally believed that they were working more than in comparable courses. We believe that this perception was correct, although it is obviously difficult to obtain accurate hard data. However, based on the student diaries which each was required to keep, the actual work was well within the guidelines set by the Faculty. This can be illustrated by anecdote. Early in the course one student came to us in anger. She was attempting to do two modules per week. She complained bitterly that this was requiring 25 hours work, that this was wholly unacceptable and that it was clear that we did not care what happened to students. When we pointed out that this was almost exactly what was required by the Faculty guidelines, she responded that she did not work anywhere near the same number of hours for any other course.

    This story, in much less dramatic form, was repeated to us by a number of students. Of course, in a traditional course, students often spend many hours immediately preceding the final examination. This may raise the overall workload to near Faculty guidelines, but is much less educationally effective.

  • Teacher Perceptions

    Very little has been written on the time taken by teachers for the preparation of materials, research for lectures, preparation of examinations, class exercises and so on. Law School folklore usually allows two to three hours preparation for each lecture if the course is one which has been taught before. Our Faculty allows six hours preparation for a one hour class if the course is a new one.

    There would appear to be little difference in preparing a Keller Plan course. The only real difference is that the whole course should be in place before the students enrol. Time must be taken to set teaching and learning objectives but this has always been the position. There is, in fact, the possibility that less time will be taken in actual preparation in a course which has clear objectives and where materials and outlines must comply with those objectives.

    The examinations require a lot of time to prepare the first time a course is offered. If traditional examination systems are used, then the literature recommends that there be three or four complete examinations for each module. Computer administered tests could reduce the load, even if traditional multiple choice questions are used, since it is easier to "mix and match" questions to build a new examination.

    We cannot over-emphasise the importance of having the entire course prepared and ready to go on day one. In spite of reading similar warnings, we had only 10 of the 15 modules completed at the beginning, the only serious mistake that we made in the course. Very sensibly, some students made a calculated decision to spend a disproportionate amount of time on Technology Law in the early weeks so as to free time later in the term for other courses.

    It is simply unacceptable to have students ready to continue to the next module when either the materials or the examination is not ready. The teacher is placed under strain to get materials and examinations in place, and the result cannot but suffer in quality. Further, it simply cannot be good teaching when the teacher begins to resent and dislike the best students in the class!

    This need to have an entire course in place before the beginning of a term means that there are transitional problems when first offering a Keller Plan course. Ideally, the teaching commitment should be reduced substantially in the preceding term. Once the course is in place, the demands are similar to a conventional course, except that the work being done in this term is for the course to be offered to the students next term. The Keller Plan imposes a planning burden on the teacher as well as on the student.

1.6.3 Can Everything Be "Kellerised"?

The answer to the question is clearly "No". The teaching of verbal skills, for example, seems to be outside the scope of Keller Plan courses.31 In fact, there have been no attempts by Keller fans to introduce PSI to every subject in a course. There will be inevitably aspects of all courses where a different teaching plan would be appropriate.

On the other hand, "Kellerisation" of parts of most subjects is possible and perhaps even desirable. Most courses have material which, for one reason or another, students learn at dramatically different rates. We can give several examples from our own experience.

The fundamentals of negotiable instruments seem difficult for some students, simple for others. Such material could be taught by a Keller Plan, the requirement being that the students complete this partial subject as a prerequisite to attending the remainder.

Teaching computer based research methods presents a similar problem but for a different reason. Students simply come with a very wide diversity of computer experience. Don Rothwell of our Faculty has successfully implemented a Keller Plan short course for students in the computer based research component of our Legal Research and Writing program.

The material in these two examples is clearly separable from the rest of the law course. If the material is not separable, then partial Kellerisation is not a reasonable alternative. If the course is to be partially Kellerised, then the Kellerised portion of it must be offered far enough in advance of the rest of the course so that the self-pacing aspect is not seriously compromised.32

We are not, in fact, in favour of partial Kellerisation of subjects except in very special circumstances. In our view, partial Kellerisation would be used primarily by teachers who wish to move dull material out of their regular subject. Such a subject is unlikely to receive the same attention or respect as an ordinary subject, and will lead to Keller Plan courses being seen unfairly as uninteresting and second class. We again note the research that shows that the Keller method is a superior method of teaching many subjects. If a subject is to be partially Kellerised, then strong reasons should be given for retaining the non-Keller part of the subject. In a profession less conservative than that of law teaching, a practitioner would be required to show why he or she clings to methods which are demonstrably inferior.

The reason for adhering to the traditional methods is not only the innate conservatism of our profession. Keller himself noted that some teachers seemed unable or unwilling to sacrifice the undoubted ego boosting that is a feature of the lecture method.33 Another reason is simple ignorance of any teaching methods other than those commonly used in law schools, a manifestation of the intellectual parochialism that too often afflicts lawyers generally.

1.6.4 Student Isolation

One of the concerns which appears to many critics is that where there is a student body without supervision there will be large scale collusion or even worse. Again, it will be the responsibility of the course designers to address these problems. Within reason, certain guidelines and codes of conduct should be incorporated to answer such criticisms.34 Our own view is that collaboration should be encouraged,35 and we have attempted to facilitate the formation of discussion groups.36

Another common concern is for the "lonely scholar". Since the method relies on self study, it is possible for students to become isolated. From our reading of the literature, this is more an imaginary than a real phenomenon. Teachers using the Plan have stated that they have been far more involved with their students than while giving lectures and tutorials on a regular basis. One user of a Keller Plan states, "37hrough repeated evaluations and modifications this course has successfully evolved over a four-year period, teaching me much about self-study and the Keller Plan and teaching me significantly more about students that I ever learned from lecturing or giving conventional tutorials".38 We have also found a high level of student interaction in the Technology Law course.

We attempted, without significant success, to encourage the formation of formal discussion groups. The lack of success is not surprising since one of the features of the Plan that appealed to students was the self-pacing feature. Self-pacing is difficult to reconcile with formal discussion groups. However, discussions with students lead us to believe that there were very active informal discussion groups. Students told us that there was more discussion of Technology Law material than of any other.

1.6.5 Costs

The most costly element of the traditional Keller Plan is the employment of proctors. Labour costs will often cripple a project like this if peers are not used for proctors.39 This was one motive for exploring the "proctorless" method of testing by computer at the University of Sydney. There is no need for large college or university computer facilities. The LES software used at the University of Sydney will run on a wide variety of machines. We used a mini-computer with terminals.40 This has the advantages of a central record keeping facility, security and simple updating of examinations.

However, we were prepared to implement the examination system on personal computers in the event of "melt-down" of our main computer facility. There is no software problem in this, but there are certain record keeping and other administrative problems that require solution.41 The solution adopted will depend upon the particular hardware available, but is straightforward with IBM compatibles.

Other cost saving approaches to the "proctor problem" have been used. Advanced students in a course can be used as proctors for less advanced. Some reports have indicated that the "prestige" associated with being a proctor has been adequate reward.42 We considered this approach, but we doubted that the solution would fit comfortably into a student culture which has been traditionally very competitive.43

1.6.6 Political problems

In a profession as conservative as ours, any new development is viewed with suspicion, and it is necessary to lay carefully the groundwork for any innovation.44 We did this by circulating information to our colleagues and conducting seminars describing the method and our particular implementation. The stratagem was successful, and the experiment has been received with interest, although it is also fair to say that the interest is coloured by a healthy scepticism.

The major political problem is the distribution of grades. Keller himself warned that the Plan is not suitable for any teacher "…who believes that, because of genetic or environmental factors, only a handful of his pupils can ever be deserving of an A or its equivalent."45 We anticipated correctly that this would be a problem. The Dean and Teaching Committee approved our assessment scheme in advance and on the understanding that our mark distribution might very well be skewed. The actual results were much better than we anticipated (see the section on statistics), but this just makes the political problem worse! There are several solutions to this problem, none entirely satisfactory. It is possible to modify the "no penalty for failure" rule. This would discourage students from using the examination system as a tutorial method. It would also dramatically increase student dissatisfaction with the examination system, particularly if the computer examination system is used.

Another solution is to reduce the marks given for completion of modules, mark the final examination ruthlessly and scale the final marks if necessary. There is little educational justification for this, but it is the most direct response to the problem. We intend to argue that successful completion of all modules is at least worthy of a credit,46 so that the Keller Plan course would still be outside Faculty guidelines.

One strategy that might be a solution is worthy of exploration. Students would not receive full marks for a module, but instead a percentage of full marks based on their performance in the module examination.47 The difference from the first solution is that students could re-examine until they reached a level which they found satisfactory. A class of very ambitious students could still yield a grade distribution which would leave the Faculty foaming.

1.7 Attitudes to the course

Any dramatically different teaching device is bound to meet with some disapproval from students and colleagues. As mentioned above, we tried to minimize and defuse the adverse reactions by providing early and complete information on the Keller Plan. We prepared a small information booklet which was circulated to interested Faculty members. At the beginning of the course, we gave a Faculty seminar at which we outlined the main features of the Keller Plan. Although these measures did not succeed in stifling all opposition, we have been pleased with the general level of support and interest shown by both Faculty and students.

1.7.1 Faculty

Although Faculty members were generally interested and supportive, there remained a few problems. There seems little way to counter these problems except to be aware of them and to provide information when they occur.

Several teachers advised students that the marks received in a Keller Plan course would be "worthless" because of the assessment method. This is puzzling, since the Keller Plan students were subjected to an average of about 15 hours of examination over the life of the course! The real problem here is the political one of too many students receiving too high marks.

Some Faculty members have indicated that they do not believe that Keller Plan teaching is really teaching, that somehow we are looking for methods to abdicate our responsibilities as lecturers. This is not a view that could be held by anyone who has actually worked on a Keller Plan! Again, we see no solution to this other than providing information. It is an attitude which we believe will disappear as the Keller Plan becomes a familiar part of teaching life. It is disappointing that this view can survive in the face of the research which shows conclusively that students learn more.

These were small problems. We again wish to emphasise that general Faculty support was generous and welcome.

1.7.2 Students

Student attitudes to the course were more directly measurable. Students who were enrolled in the course were required to fill out evaluation forms. Many provided additional advice on the way in which the course might be improved. The overwhelming tone of the student evaluation was very positive. They liked the entire concept of the Keller Plan course. They generally thought that they learned more than they would have from a comparable lectured course. They believed that they worked harder. Some were critical of the computer examination system, an aspect of their evaluation which is discussed above.

Some found that they missed the classroom interaction. A variety of reasons were given. Some simply missed the social activity that regular lectures provide. Some felt that lectures would have clarified difficult areas. Some mentioned that they gained confidence from a lecture which verified that the teacher understood the material in the same way that they themselves did!

On the other hand, nearly all students liked the self-pacing feature of the Keller Plan. In conversations with students, we have found that those who say that they miss the classroom interaction also say that, if they must choose, they would choose the self-pacing. Which only illustrates what we all know. No method of teaching is perfect in all respects.

It might be thought that students only like the Keller Plan because they receive higher grades, and that if the grade problem is solved by conforming to Faculty norms that the course would cease to be popular. That is not the case.48 Students are still attracted to the method because of the advantages of self-pacing and of knowing "where they are" in the course. This is not to say that they would not like the course better if the grades were higher!

1.8 Statistics

Many students enrolled in the course under the impression that it would be an easy mark. When it was made clear during the first and only lecture that this impression was mistaken, many changed their enrollment immediately. The magnitude of this phenomenon is not easy to identify precisely, but some measure may be gained from a consideration of the computer records. We established 192 accounts on the machine, which means that, at one time or another, 192 students indicated that they were enrolled in Technology Law. Only 132 students ever attempted an examination. We do not know how to account for the missing 60. Some dropped out as described above, but others never even attended the opening lecture.

Of the 132 who attempted at least a single examination, 2 discontinued after a short period of time. Two more failed to begin the course in time to complete enough modules to pass. Of the remainder, 116 completed all 15 modules, 92 then went on to do sufficiently well in the final examination to achieve a "high distinction", a mark of 85 or better. Faculty guidelines indicate that 5% (plus or minus 3%) of a class should receive high distinctions. Although we believe that the marks received by the Technology Law students were fair,49 it is clear that some solution must be found to the problem of integration.

1.9 Future

We believe that the Keller Plan deserves a permanent place in the Law School. It is an effective and affordable system of teaching. We believe that it is a superior method for some, perhaps most, courses, and a superior method for parts of all courses. It is particularly well suited when the material is such that students can be expected to learn at dramatically different rates. This might be because of the nature of the material or because of the varying background of the students.

The Keller Plan also deserves serious consideration when there are problems with scheduling student time. For example, busy practitioners might prefer a Keller Plan for Continuing Legal Education courses. In-house education programs could be Kellerised for the same reason. A Keller Plan course obviates the need to find a time when all students will be available.

Keller Plan courses using the computerised examination system seem ideal for distance education. The sad truth is that students in such programs often receive very shabby instruction. Although it is unlikely that any method of distance education will ever be entirely satisfactory, a good Keller Plan course would be a significant improvement over most existing schemes.

1.10 Acknowledgments

We gratefully acknowledge the support of the Law Foundation of New South Wales. Without that support, we could not have developed our particular implementation of the Keller Plan.

Discussions with David Boud of the University of New South Wales were invaluable during the early days. He made us aware of both the possibilities and the pitfalls of PSI.

We received much psychological support from our colleagues at the Law School. It is fair to say that many were, and are, skeptical of the feasibility of PSI. To their credit, however, those doubts were not transferred to the kind of opposition that occasionally occurs in schools.

We would also like to express our deep appreciation for the support which we received from the staff of the general office of the Law School. Tight printing schedules combine with nervous academics to form an explosive mixture which requires patience and tolerance. We inevitably received both.

Finally, we would like to thank the students of Technology Law 1990. They supported our efforts with enthusiasm. They tolerated our mistakes with patient good humour. If that patience and understanding sometimes took a form usually reserved for the extreme aged, infirm or otherwise bewildered, it was nonetheless appreciated.

Footnotes:

1 The "axiom" is being questioned by some teachers. For example, Morgan J, "The Socratic Method: Silencing Cooperation" (1989) 1 Legal Education Review 151. We also emphasize the 'small group'. We know of no one who is enthusiastic about the large group 'Socratic' method.

2 We should note here that we are not suggesting that there is a single method which is "best" for all students. But some methods do produce better average results: see, for example, Bloom, BS, "The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring (1984) Educational Researcher 4-16"

3 Kulik, Kulik and Cohen, "A Meta-Analysis of OUtcome Studies of Keller's Personalised System of Instruction" (1979) 34 American Psychologist 307.

4 At least the majority of students can learn. There will be exceptional circumstances which may prevent a particular student from learning particular material.

5 "Mastery learning" is the term used by Bloom, BS Human Characteristics and School Learning, New York, McGraw-Hill, 1976.

6 For a discussion of criterion-referenced and norm-referenced examining procedures, see Heywood, J Assessment in Higher Education, 2nd Edition, Wiley, New York, 1989; see also Ebel RL and Frisbie, DA Essentials of Educational Measurement, 4th Edition, Prentice-Hall, Englewood Cliffs, 1986

7 Clark, CR, Guskey, TR and Benninga JS, "The effectiveness of mastery learning strategies in undergraduate education courses" (1983) 76 Journal of Educational Research 210.

8 The tests provide the students with a means of assessing their own understanding of the material. Mastery learning is not the only way of providing students with self-assessment skills: see Rawson, SL and Tyree, AL "Self and Peer Assessment" (1989) 1 Legal Education Review 135

9 For a thorough introduction to the Keller Plan, see Keller, FS, and Sherman, JG The Keller Plan Handbook Benjamin, 1974.

10 See Keller, FS, "Good-Bye Teacher…" (1968) Journal of Applied Behaviour Analysis 179-189, 83.

11 Keller, FS, "Good-Bye Teacher…" (1968) Journal of Applied Behaviour Analysis 179-189, 83.

12 For some typical examples, see Allan, PS, "Developing a Remedial Keller-Plan Course" (1978)3 Higher Education 213; Schwartz, PL, "A controlled trial of Teaching Clinical Biochemistry by a Keller Plan (1980) 55 Journal of Medical Education 1013; Ferguson, W, "Personalized Instruction in Business Education" (1981) 50 Journal of Experimental Education 9

13 See below for a full discussion of the role of proctors.

14 Keller and Sherman, op. cit. p. 78 recommend four separate tests for each unit. We were able to reduce this load somewhat using the computer examining system described below.

15 Watson, J "Final Examinations and Long Term Retention" (1986) 17 Journal of Research in Mathematics Education 60-68.

16 For example, Allen, P S op. cit., self instruction was emphasised, in Schwartz, P L op cit. from a class of 196 medical students 30 were randomly chosen in a biochemistry course to study by the Keller Plan. Schwartz concentrated on the final examination results of thee two groups in his paper.

17 Kulik, Kulik and Cohen, "A Meta-Analysis of Outcome Studies of Keller's Personalized System of Instruction" (1979) 34 American Psychologist 307

18 Kulik, Kulik and Cohen, ibid. 311, 317

19 Kulik, Kulik and Cohen, ibid., 313, 317

20 Kulik, Kulik and Cohen, ibid., 317 - 318

21 See Morgan J, "The Socratic Method: Silencing Cooperation" (1989) 1 Legal Education Review 151 and the articles cited there.

22 It may seem strange that there is no intellectual property component in such a course. It is strange, but is explained by a desire to avoid "overlap" with longer-established courses at the University of Sydney.

23 The tests are supervised in 1991. On balance, we would recommend supervised tests.

24 Actually a different but equivalent test. The tests are "equivalent" in the sense that similar questions are used and the same objectives are tested. See the discussion on examination by computer below.

25 In the 1991 course, this has been reduced to 4 marks per module. The reduction was made to make the course conform more closely to Faculty guidelines: see below.

26 The LES file which implements this question is reproduced as Appendix A

27 The DataLex Project is a joint research project managed by Alan Tyree of the University of Sydney, Graham Greenleaf of the University of New South Wales and Andrew Mowbray of the University of Technology, Sydney. The major activities of the Project have been in legal artificial intelligence and legal information retrieval.

28 And, it should be noted, these are not problems which are unique to the computer examination system. By using human proctors, however, the problems can be caught at marking time, and so may be less serious. In traditional examinations, the problem may never be raised if the student does not have the right to see the results of the test and to challenge the marking.

29 These ambiguities and deficiencies are inevitable in any original test. One of the advantages of computer testing is that the question responses can be analysed and the questions improved.

30 We expect to have preliminary results of the project by the time that this paper appears in print. The SAGES system uses a combination of parsing algorithms and statistical methods to classify an answer.

31 At least the traditional plan where written material is the primary source of teacher-student communication. It may be possible to build a course on Keller principles which would emphasise the development of oral skills.

32 All real-life Keller Plan courses compromise the self-pacing principle to some extent since it is invariably necessary to fit in with an existing university course structure.

33 See Keller, F S, "Testimony of an Educational Reformer" (1985) Engineering Education 144-148

34 Unfortunately, the evidence is that codes of conduct have little effect on cheating except in those institutions where there is long tradition of the codes: see Ebel, op. cit.

35 except in the module tests!

36 Without notable success; see below.

37 DEFINITION NOT FOUND: fn:T

38 Peter S. Allen, "Developing a Remedial Keller-Plan Course" (1978) 3 Studies in Higher Education pp.213-220.

39 See below for a discussion of alternatives to paid proctors.

40 The computer lab at the Law School has two Honeywell Superteam computers. These are connected to 28 IBM-compatible machines each with their own 30mb hard disk. The lab is used for teaching legal information retrieval using the DataLex AIRS software, for student word processing and a variety of other projects. In 1991, the Honeywell has been replaced by a Sun Sparc II.

41 We are developing a PC based examination system which will hold questions in an encrypted form. Contact the authors for further information.

42 See Keller and Sherman, op. cit.

43 Although we were able to introduce self and peer assessment of written essays: see Rawson, SL and Tyree, AL "Self and Peer Assessment" (1989) 1 Legal Education Review 135

44 Even when the "innovation" is more than 20 years old; if it has not been used in law, then it is an "innovation"!

45 Keller and Sherman, op. cit. p 75

46 The marking scheme is Fail,Pass, Credit, Distinction and High Distinction. Current Faculty guidelines call for 40% C, 20%D and 5% HD, the percentages being of the non-failing students. There is some minor latitude for variation from these figures.

47 Munro, D "Quality distance education = computer based feedback and electronic mail" in Barrett, J and Hadberg, J (eds), Using Computers Intelligently in Tertiary Education, proceedings of the ASCILITE Converence 1987

48 The 1991 version of Technology Law is still the most popular option in the Faculty even though we have indicated that marks will be scaled ruthlessly to conform to Faculty guidelines.

49 The justification for the marks is not merely our feeling that they were fair. The result is consonant with educational research: Bloom, op.cit.

Date: 2013-01-11 Fri

Author: Alan L Tyree

Org version 7.9.3a with Emacs version 24

Validate XHTML 1.0