The Keller Plan at Law School
Alan L Tyree

If we can describe what it is that we expect our students to learn, is there any reason why they cannot learn it? If the answer to this question is "yes", then perhaps we are not good teachers or (and this may be the same thing) we are not teaching at a level which is appropriate for our students. Mastery learning assumes that our students can learn the material which we teach.⁴ Under that assumption, the basic concepts of mastery learning involve only two things.⁵ First, we tell our students what it is that we wish them to learn. Secondly, we provide them with the means to evaluate their own performance, that is, to determine if they have in fact learned the material. Both of these require some elaboration.

Mastery learning requires that the desired student performance be stated precisely. Objectives such as "students should learn to be creative" are unacceptable because there is no objective measure to determine whether the objective has been met. The objectives should be "behavioural" in the sense that they provide a specific performance measure. This measure can be used by the student (and the teacher) to determine if the student has achieved "mastery". "Mastery" does not mean that the student is a "master" of the subject matter. It means only that the student has met the performance standards of the particular course. It is possible to achieve "mastery" in a beginner's course.

Behavioural objectives sometimes arouse controversy. The argument against them is that the most important products of education are incapable of being "measured". How can we state the broad aims of a liberal education in behavioural terms? The requirement for behavioural objectives, it is argued, is opposed to these aims, for the focus is on that which can be measured rather than on what cannot.

While such arguments have a superficial attraction, we do not believe that they stand careful scrutiny. Can we seriously expect students or anyone else to be satisfied with vague aims? If the aims are not vague, then why can we not state them in a reasonably precise way? It is surely unsatisfactory for teachers to adopt a "I don't know what it is, but I recognise it when I see it!" approach to teaching.

Sometimes the argument against behavioural objectives is the result of a misunderstanding. There is no reason for the objective to be simple or elementary. "State three major criticisms of the Australian Constitution and suggest remedies" is hardly a simple objective, yet it is clearly objective. Nor do behavioural objectives exclude personal judgments or opinions. In the objective just mentioned, there is obvious room for opinion in the word "major". The personal judgment of the teacher determines what is an acceptable level of performance.

The purpose of the objectives is not to limit the educational process, but to assist in its definition. Students are entitled to know what the teacher expects. Mastery learning assumes that the teacher knows what is expected, and is skilled enough at the job to be able to state it.

"Mastery" is determined by examination of the student. The student is examined to see if he or she has achieved the stated objectives. Thus, the subject matter of the examination is the stated objectives; there is no room for examining objectives which are not previously stated.

The examination procedure is somewhat different from the methods which law teachers usually use. In mastery learning, we attempt to examine using "criterion referenced" examination methods. Most law teachers probably use a mix of criterion referenced and "norm referenced" methods.⁶ Norm referenced methods attempt to place a student by reference to the performance of other students. There is clearly a large component of norm referenced examining procedure in most law school examinations, for many faculties require a specified distribution of marks. It would be a rare school which did not call for explanation when one marker differed consistently and substantially from the local norm.

Norm referenced questions must discriminate among students, for a question which does not discriminate adds nothing to the testing process beyond shifting the average. Criterion referenced testing, on the other hand, sets specific performance standards. Questions are measured against the performance requirements rather than by the profile of answers. A criterion referenced question may be satisfactory even though there is a near perfect answer rate, that is, even though it contributes little or nothing to the discrimination process. Similarly, a question which is failed by all students may be a perfectly acceptable question, although it may indicate a deficiency in the materials or in the teacher.

Of course, it would be unusual for any test to be completely norm or completely criterion referenced. A typical law teacher will normally pay heed to performance criteria when drafting questions, but, as noted above, will also be disturbed if there is a 100% pass rate on questions. In fact. the marking standards in most examining probably shifts subtly during marking in order to ensure that the final marks fit into some norm.

In the context of mastery learning, the aim is to key the test questions to the stated behavioural objectives. The test questions should be criterion referenced against the stated objectives. If the objective is one which we should expect most students to learn easily, and there will be some, then there is no objection to a question which 100% of the students pass on the first attempt.

It is clear that the concepts of mastery learning can be introduced into existing courses. One way of doing so is to give the students a statement of the behavioural objectives at the beginning of each week, and to test them at the end of the week. In the standard mastery learning implementation, the weekly tests do not count as part of the final mark. Their sole purpose is to provide the student with a measure of their understanding of the material. The weekly tests, called "formative" tests in the mastery learning literature, should provide immediate feedback to students. If the student does not achieve mastery, then he or she should be referred to some alternative materials which will assist in overcoming the learning deficiency.

It will perhaps come as a surprise to many teachers to learn that adding mastery learning techniques results in a measurable improvement in student performance.⁷ The credit for this is probably divided equally between teacher and student. The formulation of behavioural objectives provides a focus for both. The teacher is forced to articulate objectives that may have been unstated previously, and the examination procedure can make the student aware that he or she may not know the material as well as thought.⁸

The following account given by Keller himself of the main features of his Plan shows very clearly the differences between this teaching model and those which are more traditional:¹⁰

"(1) The go-at-your-own pace feature, which permits a student to move through the course at a speed commensurate with his ability and other demands of his time. (2) The unit-perfection requirement for advance, which lets the student go ahead to new material only after demonstrating mastery of that which preceded. (3) The use of lectures and demonstrations as vehicles of motivation, rather than sources of critical information. (4) The related stress upon the written word in teacher-student communication; and finally: (5) The use of proctors, which permits repeated testing, immediate scoring, almost unavoidable tutoring, and a marked enhancement of the personal-social aspect of the educational process".

Keller and his associates implemented a PSI course at the then new University of Brasilia in 1964. The method was refined over the next few years, and Keller's landmark paper was published in 1968.¹¹ There have been thousands of successful Keller Plan courses run throughout the world since that time.¹² We have found no reference in the literature to a Keller Plan course in any law school. Before describing the implementation of the Keller Plan in Technology Law, we will discuss in more detail some of the particular features which Keller thought important.

Since students must proceed at their own pace in a PSI course, the traditional scheduled lecture or class must be abandoned. Lectures could still be the primary method of communication, but either they must be frequently repeated or reduced to audio or video tape. Most Keller Plan courses have relied on written materials as the primary means of student/teacher communication. In a law course, this has the additional advantage of requiring students to learn in a manner similar to that which they must use for the rest of their professional life.

Unit materials are somewhat different from those which are usually reprinted for student use. Remember that there will be little opportunity for guided discussion. In addition to the ordinary materials, there is a teacher written "study guide". This must provide a "map" to the materials, as well as provide any up-to-date information or opinions. The teacher is free to write his or her analysis of any of the written materials. Most importantly, the study guide must also contain the detailed behavioural objectives of the unit.

If proctors are used,¹³ the teacher should also prepare detailed proctor guides. These contain general instructions to proctors concerning their role as well as specific instructions on marking the unit tests.

In the typical Keller Plan implementation, students are free to test at specified times during the week. There is no penalty for failure of a unit test. The test is marked immediately by the proctor, and the proctor may seek clarification from the student concerning answers given.

Tests are generally short (20 minutes seems a popular figure) and consist of a variety of question types. The Plan has nothing to say about test form other than that it must test all of the objectives, and it must test nothing which is not a stated objective. Because there is no penalty for failure, the teacher must prepare several equivalent but different tests for each unit.¹⁴ We found test preparation to be one of the most demanding aspects of the Keller Plan.

The examining procedure used by Keller was labour intensive. It was clearly impossible for the teacher to mark all of the examinations. Indeed, Keller and others thought that it would be a waste of highly skilled talent to employ teachers in that capacity. The examiners were known as "proctors".

Proctors administer the tests and provide immediate reinforcement for student performance. Instant feedback is possible, and a small amount of tutoring is inevitable although proctors are not engaged to act as traditional tutors. On the contrary, Keller cautioned against proctors acting as tutors. They are not there as teachers, and are not intended to impose their views of the materials on students. Proctors provide important human contact in an otherwise "teacherless" course. Proctors also provide valuable information to the instructor or course convener who is responsible for the course. In a typical implementation, a proctor would be "responsible" for 15 to 20 students.

Implementations of the Plan show considerable variety in the use of proctors. In some, students in the same course who had tested earlier were used as proctors. This was thought to provide a more intimate learning environment. In others, postgraduate students were employed in some instances with certain credit points for other courses being allocated for their participation. Another popular scheme was to use students who have completed the course in previous years.

The traditional Keller Plan calls for a few non-compulsory lectures and extra demonstrations. The purpose of these lectures is motivational only. The content is not examinable. The lectures help to combat the sense of isolation that may develop by providing an occasion for teachers and students to get together.

We scheduled a series of five guest lectures in Technology Law. The lecturers were all prominent in their area, and the area was always related to the course although not directly overlapping with any of the units. Attendance at these lectures was very poor, usually less than 25% of the class.

The initial Keller Plan did not include a final examination. However, most implementations of the Plan which are reported in the literature contain a final examination component. Keller himself introduced final examinations in later courses. The reason for this seems a little vague given Keller's concern about student fear of examination often being a cause of poor performance. One reason given later was that his colleagues were more likely to employ the Plan if there was some traditional method of assessment available as well as the progressive unit testing. There is also some research which indicates that students in a Keller Plan course with final examination do better than Keller Plan students without final examinations.¹⁵

The Keller Plan is one of the few teaching methods which has a theoretical basis. For that reason alone, it attracted a great deal of attention in the late 1960's following the publication of "Goodbye Teacher…". There are many papers devoted to a study of one or another aspect of the Plan.¹⁶

The literature grew so large that it was possible to do a form of scientific analysis of the published results. The study, known as a "meta-analysis", showed conclusively that the Keller Plan was at least as good as very intensive small group "Socratic" methods.¹⁷ In comparison with traditional lecture methods, Kulik, et al were able to show that an average student could be expected to improve his or her final examination mark by 8-9%.¹⁸ The study verified that students not only learned more, but (and this may be the same thing) they retained it better. Low-aptitude students benefited to the same degree as high-aptitude students, and re-testing after six months showed that the advantage of the Keller students increased.¹⁹

Students believed that they worked harder under Keller Plan courses, although the evidence was unclear as to whether they actually worked harder. Given this student belief, it is perhaps surprising that students liked the Keller Plan course better than the alternatives offered.²⁰ If the student perception of working harder is correct, it is perhaps not surprising that they learn more. They may like the method better for a number of reasons. In comparison with the "Socratic" methods, there is less stress. Students like the "go at your own pace" feature which allows them to schedule their own time. The method involves the student more than the traditional lecture. And, of course, they may like the Keller Plan precisely because they believe (correctly) that they learn more. The Keller Plan may also be a more desirable option than the Socratic method for students who are less vocal or less aggressive. Feminist writings have attacked the Socratic method for precisely this reason.²¹

We believe that one of the primary advantages of a Keller Plan for a law course is that students learn primarily by reading. We found that our students' ability to learn efficiently from reading increased dramatically as the course progressed. Learning from reading is obviously an important skill for lawyers, but one that is often neglected in favour of developing oral skills.

Technology Law was first offered as an optional course for later year students in 1989. It was taught by lectures and tutorials with an extensive research project and final examination. The course was divided into a number of separate areas covering a range of topics which were capable of forming distinct modules.

The course was,and is, a mixture of "policy" and "black letter" subjects. In some parts, for example, computer crime, there were statutes and students were expected to learn "the law" as well as to explore the underlying policies (or lack thereof). In other subject areas, for example, electronic data interchange or transborder data flows, there is no specialised local law, and the course considered the application of the general law to the special problems together with various international initiatives. There seemed no reason to change the general scheme of the course, and the Keller Plan course covers broadly the same topics.

The Kellerised Technology Law course consists of 15 modules. Ten of these cover computer related topics such as Computers and Computer Crime, Electronic Funds Transfers, Artificial Intelligence, Privacy and Data Protection, Freedom of Information. Five modules deal with various problems of biotechnology including Plant Variety Rights, Artificial Conception, Genetic Manipulation and Genetic Fingerprinting.²²

Each module is divided into two parts - the "Study Guide" and "Materials". The study guide provides a detailed explanation of what is expected from students with a "procedures" section suggesting a path of study to follow. Objectives are set for each module and suggested issues are presented. The objectives and suggested issues are based on the materials which have been specially chosen to facilitate individual study.

The amount of materials has been kept within reasonable limits. We expected that students would take approximately 10 - 13 hours to read and prepare for testing. This conforms to the Faculty of Law guidelines for course structuring. Other readings are suggested in the study guide, but these are merely for interest and these readings do not form part of the examinable materials. There is nothing in the Keller Plan which requires that student materials be self-contained. The decision was forced upon us by limited library resources.

When a student decides that he or she has achieved "mastery" of the module, he or she attends the Faculty computer laboratory for testing. Tests average 10 items, were unsupervised and generally required 20 - 30 minutes.²³ There is no penalty for failure. A student may sit the test as many times as required to pass it.²⁴ Students required an average of 2.5 attempts to pass each test in the 1990 course. Once the test is passed a set mark is awarded - 5 marks for each module in the 1990 course.²⁵

Since proctors are not used, we tried to design the tests to provide the immediate feedback that the Keller Plan requires. When the student fails a question, the reasons for the failure are given and the student is referred to the particular section of the materials which discusses the issue being tested. In addition, the teacher is available freely for consultation concerning difficulties with testing, materials and other concerns. This policy decreases any perceived isolation and has encouraged many students to assist the researchers in the development of improvements for future courses.

The assessment for Technology Law, 1990 was divided into two parts - module tests worth 5% for each module passed (a possible 75%) and a final examination (to cover all module materials) worth 25%. To put this into perspective, Faculty guidelines call for an average of 20% of students to receive a "Distinction" or higher. A "Distinction" is a mark of 75 or more. We had a unusually large number of "Distinctions"; see the discussion infra.

We hypothesised that by using "tree" questions, we could better explore the reasoning of the student and, at the same time, reduce the construction time of questions. An example might clarify this.

One of the units concerns plant variety rights, and one of the specialised issues to be tested concerned the identity of the "breeder" of a plant variety.²⁶ The question was a typical law examination short story which described a person, John, who worked for a university who cultivated a special form of tomato. If the cultivation was in the course of employment, then the university was the breeder. If not in the course of employment, then the employee was the breeder. We wished to test that the student understood that either John or the University might be the breeder, and that the resolution of this issue depended upon whether the cultivation was in the course of employment.

The "tree" question begins by asking baldly "who is the breeder". On the basis of the answer to that question, the next part of the question is a multiple choice question which looks at the possible reasons why the student might give the response actually given. There is no doubt that this question could be cast as a single multiple choice question, but it would be substantially more difficult to construct. With the "tree" question, the examiner is not required to think of so many possibilities at each stage of the construction. It is easer to provide alternatives that do not give away the answer, for much of the question is hidden from the student at each stage of the process.

At the end of the tree, the student is informed that the question has been passed or failed. In either case, we usually gave some additional feedback. In the case of a failed question, we would indicate either the reason for the failure or a reference to particular materials which should be studied further.

The language used both for construction of the questions and for the administration of the entire examining program is LES, a language developed by Andrew Mowbray as part of the DataLex Project.²⁷ LES is available to educational institutions for educational purposes free of charge.

In 1990, the computer testing aspect of the experiment had mixed success. It was completely successful in the sense that it worked. We were able to administer almost 5000 20 minute examinations to 130 students in a 15 week period. The entire system contained approximately 400 questions and, subject to the comments below, performed its duties flawlessly.

We aimed for three alternatives of each question, that is, three separate questions which were intended to test the same objective. An examination consisted (generally) of 10 questions. A question was selected at random from the pool (usually two or three) of related questions. This system has the advantage that there are a large number of possible examinations. It is also convenient to add questions to the pool as and when time allows. Although construction of examinations is still a very demanding process, it seems that it is less so than in a traditional Keller Plan course.

The examination system was not, however, entirely satisfactory. Students were often frustrated by what they saw as inflexible structures and the impossibility of explaining their answer further. On some of the material, there were frequent complaints that the questions merely imposed the teacher's views in situations where the issues were far from clear. Some of the student complaints were justified. In many cases this could be explained by our lack of experience with criterion referenced testing and with the principles of constructing multiple choice questions. This led, in some circumstances, to ambiguous questions or to questions which failed to address the more important issues. These problems will be overcome by teacher experience.²⁸

Some complaints were not justified. Students often read the screen quickly, believing that they had seen the question before whereas it was, in fact, a variation. Some frustration was the result of bright, highly motivated students "failing" examinations for the first time in their academic career. The fact that the "failure" cost them nothing except a further examination did not offset the emotional trauma that they experienced. These problems generally decreased as the course progressed and the students adapted to the new method of examining.

We do not wish to give the wrong impression here. The examination system was certainly successful enough that we are using it again. The system also worked much better than might be expected of a first time experiment. Most students handled most questions in the way expected. We also emphasise that the deficiencies in the existing examination system must be seen in context. That context includes our inexperience with computer based testing and a careful consideration of the alternatives: in our view, there are no affordable alternatives. Many of the questions which caused problems are amenable to simple modifications. The questions which need modification are easily identified by the statistics kept by the examining system.

There is one improvement in the examining system which we believe would still most or all of the student frustration. It is best illustrated by the student reaction to the final examination in Technology Law. That examination was a 60 item true/false examination which allowed the students a two line explanation of their answer. The explanation was optional. If the student gave the "wrong" answer, it could be partially or completely salvaged by giving an acceptable reason. Both the teacher and the students found this to be a valuable modification to the T/F format. Several questions had ambiguities that the examiner had not detected during examination construction.²⁹ There were a significant number of students who gained marks for these questions on the basis of explanations which showed that they had given a reasonable answer.

This system may be incorporated into the computer examination system, although current technology does not allow the complete computerisation. The existing system is easily modified to accept short reasons, even though it cannot assess their validity. The plan would be to have a single "proctor" in the computer laboratory whose task it would be to examine reasons given when the problem is marked incorrect by the computer. This approach would also have a valuable psychological advantage, since the students would feel that they were receiving more personal attention. It also has the advantage (from the student point of view) that attention by a human proctor could only improve their results, never diminish them.

Ideally, of course, the computer marking system would be extended to mark short answer questions. This would reduce the question construction time required by the teacher, as well as being more satisfactory for the students. The SAGES (Short Answer General Examination System) Project is addressing this problem, but it is too soon to report progress. Information may be obtained directly from the authors.³⁰

Some of our colleagues have suggested that Technology Law is somehow more favourable for computer testing than other courses. Nothing could be further from the truth. We found that the easiest parts to test were the "black letter law" portions of the course, but we have also found that to be the case with ordinary examinations as well. Perhaps there is a mistaken assumption that a course in Technology Law must necessarily be mechanical. However, as we indicated above, the course is a mixture of policy, politics, jurisprudence and black letter law. We have no reason to suppose that the examination problem, whether by computer or by ordinary means, was any more or any less severe in Technology Law than in any other course taught in the law school.

Every teacher has experienced the "procrastinating" student. The problem is exacerbated in a Keller Plan course. We had several students fail Technology Law because of a late start. Our particular problem was acute because of a very poor system of enrolment at the University of Sydney. We had no way of knowing who should have been completing module examinations until late in the semester.

Assuming that the teacher can determine who should be in the class, the problem of procrastination can be dealt with simply by using automatic notification procedures. When a student's record shows that he or she is dangerously behind schedule, the teacher should be notified automatically by the computer system. On the other hand, the Keller Plan is "self-pacing". There are arguments that the entire responsibility should be shifted to the student. We do not subscribe to that philosophy, but we accept that students should be allowed a certain leeway.

The Keller Plan is dangerous. It is easy to be tempted to develop a "complete" course, putting in every article that is relevant to the subject. This must be resisted at all costs. Remember that the students are expected to "master" all of the material. It is poor teaching to misjudge the amount of material that students should be expected to learn.

Our Faculty guidelines call for a total student commitment of 196 hours for a two unit course such as Technology Law. Since we had 15 modules, this meant that students should spend approximately 13 hours per module. Guidelines also indicate that students can read 20 pages per hour, but we believe that this is a gross exaggeration when the written materials are the primary source of teaching. We used a rough guideline of 10 pages per hour. Most of the modules contained about 100 pages of reading material, accompanied by four to six pages of study guide.

The answer to the question is clearly "No". The teaching of verbal skills, for example, seems to be outside the scope of Keller Plan courses.³¹ In fact, there have been no attempts by Keller fans to introduce PSI to every subject in a course. There will be inevitably aspects of all courses where a different teaching plan would be appropriate.

On the other hand, "Kellerisation" of parts of most subjects is possible and perhaps even desirable. Most courses have material which, for one reason or another, students learn at dramatically different rates. We can give several examples from our own experience.

The fundamentals of negotiable instruments seem difficult for some students, simple for others. Such material could be taught by a Keller Plan, the requirement being that the students complete this partial subject as a prerequisite to attending the remainder.

Teaching computer based research methods presents a similar problem but for a different reason. Students simply come with a very wide diversity of computer experience. Don Rothwell of our Faculty has successfully implemented a Keller Plan short course for students in the computer based research component of our Legal Research and Writing program.

The material in these two examples is clearly separable from the rest of the law course. If the material is not separable, then partial Kellerisation is not a reasonable alternative. If the course is to be partially Kellerised, then the Kellerised portion of it must be offered far enough in advance of the rest of the course so that the self-pacing aspect is not seriously compromised.³²

We are not, in fact, in favour of partial Kellerisation of subjects except in very special circumstances. In our view, partial Kellerisation would be used primarily by teachers who wish to move dull material out of their regular subject. Such a subject is unlikely to receive the same attention or respect as an ordinary subject, and will lead to Keller Plan courses being seen unfairly as uninteresting and second class. We again note the research that shows that the Keller method is a superior method of teaching many subjects. If a subject is to be partially Kellerised, then strong reasons should be given for retaining the non-Keller part of the subject. In a profession less conservative than that of law teaching, a practitioner would be required to show why he or she clings to methods which are demonstrably inferior.

The reason for adhering to the traditional methods is not only the innate conservatism of our profession. Keller himself noted that some teachers seemed unable or unwilling to sacrifice the undoubted ego boosting that is a feature of the lecture method.³³ Another reason is simple ignorance of any teaching methods other than those commonly used in law schools, a manifestation of the intellectual parochialism that too often afflicts lawyers generally.

One of the concerns which appears to many critics is that where there is a student body without supervision there will be large scale collusion or even worse. Again, it will be the responsibility of the course designers to address these problems. Within reason, certain guidelines and codes of conduct should be incorporated to answer such criticisms.³⁴ Our own view is that collaboration should be encouraged,³⁵ and we have attempted to facilitate the formation of discussion groups.³⁶

Another common concern is for the "lonely scholar". Since the method relies on self study, it is possible for students to become isolated. From our reading of the literature, this is more an imaginary than a real phenomenon. Teachers using the Plan have stated that they have been far more involved with their students than while giving lectures and tutorials on a regular basis. One user of a Keller Plan states, "Through repeated evaluations and modifications this course has successfully evolved over a four-year period, teaching me much about self-study and the Keller Plan and teaching me significantly more about students that I ever learned from lecturing or giving conventional tutorials".³⁷ We have also found a high level of student interaction in the Technology Law course.

We attempted, without significant success, to encourage the formation of formal discussion groups. The lack of success is not surprising since one of the features of the Plan that appealed to students was the self-pacing feature. Self-pacing is difficult to reconcile with formal discussion groups. However, discussions with students lead us to believe that there were very active informal discussion groups. Students told us that there was more discussion of Technology Law material than of any other.

The most costly element of the traditional Keller Plan is the employment of proctors. Labour costs will often cripple a project like this if peers are not used for proctors.³⁸ This was one motive for exploring the "proctorless" method of testing by computer at the University of Sydney. There is no need for large college or university computer facilities. The LES software used at the University of Sydney will run on a wide variety of machines. We used a mini-computer with terminals.³⁹ This has the advantages of a central record keeping facility, security and simple updating of examinations.

However, we were prepared to implement the examination system on personal computers in the event of "melt-down" of our main computer facility. There is no software problem in this, but there are certain record keeping and other administrative problems that require solution.⁴⁰ The solution adopted will depend upon the particular hardware available, but is straightforward with IBM compatibles.

Other cost saving approaches to the "proctor problem" have been used. Advanced students in a course can be used as proctors for less advanced. Some reports have indicated that the "prestige" associated with being a proctor has been adequate reward.⁴¹ We considered this approach, but we doubted that the solution would fit comfortably into a student culture which has been traditionally very competitive.⁴²

In a profession as conservative as ours, any new development is viewed with suspicion, and it is necessary to lay carefully the groundwork for any innovation.⁴³ We did this by circulating information to our colleagues and conducting seminars describing the method and our particular implementation. The stratagem was successful, and the experiment has been received with interest, although it is also fair to say that the interest is coloured by a healthy scepticism.

The major political problem is the distribution of grades. Keller himself warned that the Plan is not suitable for any teacher "…who believes that, because of genetic or environmental factors, only a handful of his pupils can ever be deserving of an A or its equivalent."⁴⁴ We anticipated correctly that this would be a problem. The Dean and Teaching Committee approved our assessment scheme in advance and on the understanding that our mark distribution might very well be skewed. The actual results were much better than we anticipated (see the section on statistics), but this just makes the political problem worse! There are several solutions to this problem, none entirely satisfactory. It is possible to modify the "no penalty for failure" rule. This would discourage students from using the examination system as a tutorial method. It would also dramatically increase student dissatisfaction with the examination system, particularly if the computer examination system is used.

Another solution is to reduce the marks given for completion of modules, mark the final examination ruthlessly and scale the final marks if necessary. There is little educational justification for this, but it is the most direct response to the problem. We intend to argue that successful completion of all modules is at least worthy of a credit,⁴⁵ so that the Keller Plan course would still be outside Faculty guidelines.

One strategy that might be a solution is worthy of exploration. Students would not receive full marks for a module, but instead a percentage of full marks based on their performance in the module examination.⁴⁶ The difference from the first solution is that students could re-examine until they reached a level which they found satisfactory. A class of very ambitious students could still yield a grade distribution which would leave the Faculty foaming.

Although Faculty members were generally interested and supportive, there remained a few problems. There seems little way to counter these problems except to be aware of them and to provide information when they occur.

Several teachers advised students that the marks received in a Keller Plan course would be "worthless" because of the assessment method. This is puzzling, since the Keller Plan students were subjected to an average of about 15 hours of examination over the life of the course! The real problem here is the political one of too many students receiving too high marks.

Some Faculty members have indicated that they do not believe that Keller Plan teaching is really teaching, that somehow we are looking for methods to abdicate our responsibilities as lecturers. This is not a view that could be held by anyone who has actually worked on a Keller Plan! Again, we see no solution to this other than providing information. It is an attitude which we believe will disappear as the Keller Plan becomes a familiar part of teaching life. It is disappointing that this view can survive in the face of the research which shows conclusively that students learn more.

These were small problems. We again wish to emphasise that general Faculty support was generous and welcome.

Student attitudes to the course were more directly measurable. Students who were enrolled in the course were required to fill out evaluation forms. Many provided additional advice on the way in which the course might be improved. The overwhelming tone of the student evaluation was very positive. They liked the entire concept of the Keller Plan course. They generally thought that they learned more than they would have from a comparable lectured course. They believed that they worked harder. Some were critical of the computer examination system, an aspect of their evaluation which is discussed above.

Some found that they missed the classroom interaction. A variety of reasons were given. Some simply missed the social activity that regular lectures provide. Some felt that lectures would have clarified difficult areas. Some mentioned that they gained confidence from a lecture which verified that the teacher understood the material in the same way that they themselves did!

On the other hand, nearly all students liked the self-pacing feature of the Keller Plan. In conversations with students, we have found that those who say that they miss the classroom interaction also say that, if they must choose, they would choose the self-pacing. Which only illustrates what we all know. No method of teaching is perfect in all respects.

It might be thought that students only like the Keller Plan because they receive higher grades, and that if the grade problem is solved by conforming to Faculty norms that the course would cease to be popular. That is not the case.⁴⁷ Students are still attracted to the method because of the advantages of self-pacing and of knowing "where they are" in the course. This is not to say that they would not like the course better if the grades were higher!