CRES: Critical Review Examining System
Alan L Tyree
Abstract
This paper describes a simple form of computer tutorial known as CRES (Critical Review Examining System). The CRES method has the advantage of accepting free form short answers thus freeing it of one of the major objections to the use of CAL methods in law. CRES tutorials are inexpensive to build, and recent research has reduced the costs even further. The paper also discusses some of the models for using CRES tutorials in a teaching program.
Introduction
The structure of traditional computer assisted learning (CAL) programs has been a major limiting factor on their use in legal eduction.1 The traditional structure has been costly to build and, worse, costly to modify for local conditions or for changing conditions.
The traditional CAL programs have been designed as "pseudo-Socratic" teaching machine. Using the ordinary branching capabilities of the computer these programs attempt to guide a student through complex material by presenting small amounts of text or other teaching material, asking the student to answer assessment questions, then branching on student replies. Unfortunately, the designer of the system must anticipate every possible student response and, perhaps worse, the most likely reason for the incorrect response.
The construction of a "pseudo-Socratic" machine is costly. Best estimates of costs range between 100 and 400 hours of teacher time to build a one hour tutorial.2 Once built the programs are generally not easy to modify, making them unsuitable for many areas of law teaching.
The results can be effective. Kulik, et al, used a "meta-analysis" to study the results of 59 independent experiments which compared CAL with other teaching strategies.3 Their findings were that computer taught students did marginally better than students taught by other methods, but that the time required for knowledge acquisition was only two thirds. The reduced time probably reflects the one-on-one aspect of computer assisted learning.4
Must the Computer do everything?
In my view one major problem with the traditional approach to CAL has been the attempt to build machines that are complete teachers. Why should we discard excellent textbooks and other familiar teaching materials? Suppose instead that we begin by asking what the machine can do well, see if it is possible to assign some tasks to the machine, other tasks to books and printed materials and some tasks to verbal interaction between the teacher and student. In other words, let's explore a systems view of the teaching process.5
The idea that the computer can do some educational tasks better than others led Park and McGregor-Lowndes to develop a series of multiple-choice questions as a basis for tutorials in a corporate law for business students course.6 The machine can administer the questions, record results and maintain complete records. It is a limited but effective use of computer assisted learning.7
CRES Tutorials
Multiple choice questions have a number of perceived limitations that make them unattractive to most law teachers.8 They are difficult to write and often are not satisfying to either teacher or student when testing any but the most routine knowledge. Although there is a substantial body of research into their use as testing mechanisms, there is little as to their effectiveness as tutorial mechanisms.
These perceived limitations suggest the use of more traditional problem type tutorial questions where the student is required to write an advice. The technical problem, of course, is how to handle the free form response from students. The solution adopted by the CRES method is so simple as to be embarrassing: after the student answer has been "submitted" the computer asks the student a number of simple yes/no questions about the submitted answer. The practical effect of these "critical review" questions is that the student marks their own answer. The "critical review" questions may be arranged in a tree structure so that a variety of possible student answers will result in a pass,9 thus facilitating the use of questions which have no "right" answer.10
The first use of CRES as a pure tutorial system was in International Law at the University of Sydney in 1992.11 Students were given a choice of using the CRES tutorial system or attending ordinary tutorials. Almost exacly half of the students chose computer tutorials.12 The 'computer' students did marginally but consistently better in two examinations.13
Since that time, the response from students has been encouraging. Aside from positive evaluations and comments, in 1994 the International Law students voted with their feet, more than 70% choosing CRES tutorials in preference to attending standard "human" tutorials.
A National Teaching Development Grant from the Committee for the Advancement of University Teaching has led to a significant increse in the number of subjects using CRES tutorials.14 Current offerings include company, constitutional, intellectual property, international, real property and sale of goods. Evidence tutorials are nearing completion and will be used for the first time in 1995.
There is also a complete legal research course based on the text Morris, et al, "Laying Down the Law" which includes on-line study guides as well as a complete set of CRES tutorials.15
CRES tutorials at the University of Sydney are run on a SUN UNIX system to facilitate record keeping, but there are also DOS and Windows versions running.
Integrating CAL - IP93
The construction of computer tutorials is only a part, perhaps a small part, of the problem of computer assisted learning. A more challenging question is to find the 'best' way of using the tutorials in an overall integrated method of teaching. The University of Sydney uses several models which I will discuss in order from least to most 'radical'. Considering a range of options is necessary since there is in the Faculty a significant amount of political and ideological opposition to the use of computer assisted learning in any form.
The easiest approach is to simply use the tutorials as a 'bolt-on accessory' to an existing class. This might take the form of offering tutorials in classes where financial constraints preclude the use of human tutors, or, as in international law, offering the tutorials as an optional alternative to human tutorials. It is difficult (but, alas, not impossible) to object to the use of computer tutorials in classes which currently offer no tutorial assistance at all. Although there has been some resistance to offering computer tutorials as an optional alternative, student reaction to the option has been so favourable that the opposition is unlikely to be successful or sustained. In the current climate we can expect that soon students will demand to know why tutorials are not offered in advanced subjects.
A better method is to incorporate the tutorials as part of a "mastery learning" strategy.16 This requires the teacher to identify clear "behavioural objectives" for each section of the course. The CRES tutorials are then keyed to the objectives, and the CRES feedback should direct the student to alternative materials for reviewing areas of weakness. Mastery learning methods can result in a significant improvement in student performance as measured by final examination marks.17
A different way of using computer tutorials is to make them a compulsory prerequisite to meeting with human tutors. We have all had the experience of the aggressive unprepared student who monopolises tutorial time to learn elementary material that he or she should have mastered before attending the tutorial. Requiring the completion of the computer tutorial before attending a conventional tutorial substantially enhances the quality of the face to face contact time. This model is now being used in both Company and Constititutional Law.
The 1993 course in Intellectual Property illustrates the most radical departure from standard teaching models. The course consists of ten 'modules'. Each module has four components:
- A directed self-study component - students are given a detailed 'study guide' which guides them through the reading material of the module.18 The study guide contains a list of behavioural objectives and notes which assist the student in places where the textbook is difficult;19
- 'Cooperative Learning Group' meetings: students are required to form CLGs of between 5 and 8 students. Each group meets to discuss the issues raised by the module and to identify any problems that should be discussed with the teacher; there are two CLG meetings per module;
- Computer tutorials: these CRES tutorials are keyed directly to the stated behavioural objectives. Students must make a 'serious attempt' at the tutorials; there are two computer tutorials per module.
- Small Group Meetings with the Teacher: provided the student attends the CLG meetings and makes a 'serious attempt' at the computer tutorials, he or she qualifies for a meeting with the teacher. These meetings are in groups of fewer than 10 students and are held fortnightly.
Experience with the course was extremely positive. Student morale was positive and participation rates were high. The Small Group Meetings were very productive since the students are prepared and have resolved most of the issues in the Cooperative Learning Groups. The computer tutorials assure that no significant issues are overlooked. It is important that the prerequisite conditions be enforced: to allow unprepared students to attend the Small Group Meetings rewards lazy students and punishes those who have fulfilled the requirements.
Making CAL Cheaper
A person who is experienced in building CRES tutorials and who has a clear idea of the content can build a one hour tutorial in six to eight hours.20 A completed CRES question is shown in Appendix I. After builing 50 or 60 of these problem questions, it will become apparent that there is a pattern. We have exploited that pattern to simplify the construction of the Critical Review tree.21
Automatic Tree Generation
The idea of automatic tree generation is based on the simple idea that most problem questions have a single "issue" and usually a single "argument" or justification. Rather than build the tree directly, we now ask authors to state the issue (in the form of the preferred outcome) and the nature of the argument in support.
As an example, the question shown in Appendix I could be followed by:
&ISSUE Beatrice is not responsible for the loss of the goods &PROARG By s25 risk prima facie passes with property, and property has not passed since the goods have not been appropriated to the contract: s21.
From these two items we generate the tree shown.22 Note again that a student may pass the question even though disagreeing with the author's preferred outcome.
Eliminating the tree - scoring methods
For a single issue question, the automatic tree generation solution is satisfactory. For multiple issue questions, the Critical Review Tree itself is unsatisfactory.
The problem is: what do you do with a student who has "failed" the first issue? The current approach is to "fail" the student and go directly to the Feedback section. But this is clearly unsatisfactory in a multi-issue problem. The problem could be handled by building more complex review trees. Such trees would have a high `redundancy' in the sense that many of the nodes would contain the same questions. The additional complexity is required only for providing additional paths to different outcomes.
We have chosen to attack the problem by means of a scoring system. In this model, a student may be led through each of the issues, recording performance along the way. It also allows us to provide intermediate (and hence more immediate) feedback in the form of 0-value "questions".
The scoring model is still experimental, but we believe that it will lead to better student satisfaction with the method. There are still some puzzles to be solved: what is a "logical" scoring system? What is the best form of feedback? What should be done for those issues which are ignored entirely by the student?
Generating the Critical Review from the Model Answer
From the model answer, anyone with an understanding of the method can generate either a Critical Review Tree (either directly or through the &ISSUE/&PROARG construction). Is it possible to automate the process? We think so, but we haven't done it yet. It may be necessary at first to impose a rigid discipline on the form of the model answer, but it seems achievable. If successful, the costs of CRES tutorial construction would be reduced by about 50%. Results will be reported at the Second National Conference.
Acknowledgements
Construction of CRES tutorials has been facilitated by a National Teaching Development Grant from the Committee for the Advancement of University Teaching. I am also grateful to the Law Foundation of New South Wales for its support of the Keller Plan experiments which led directly to the development of the ideas used in the CRES tutorial method. The work on automating construction has been supported in part by the Legal Scholarship Support Fund, a program funded jointly by the Faculty of Law and the Law Foundation of New South Wales.
Appendix I
&CRES &DOCUMENTATION Unascertained Goods: s21 Property cannot pass in unascertained goods &QUES Archie sends the following fax to Beatrice: "Please supply me with 100 Kilos f.a.q. pistachio nuts, delivery @ your Gosford warehouse Monday afternoon, $5.43/kilo". Beatrice faxes back: "Done!". On Monday morning Beatrice orders Clark, a mild-mannered law student who is working part time in her warehouse, to set aside 100 kilos from her stock of 1200 kilos. Clark works methodically but unthinkingly. He sets aside 150 Kilos. At noon a fire breaks out and destroys Beatrice's warehouse and all the goods in it, including the pistachio nuts. Your advice is sought as to who is to bear the loss of the nuts. &MODEL Property in unascertained goods cannot pass until the goods are appropriated to the contract: s21 SOGA. "Appropriation" involves some act of identifying the goods as the contract goods. Here there has been no act of appropriation since it is impossible to identify the precise goods which are to be delivered to Archie. Property, and therefore risk, has not passed to Archie. &FEEDBACK Property in unascertained goods cannot pass until the goods are appropriated to the contract: s21 SOGA. "Appropriation" involves some act of identifying the goods as the contract goods. Here there has been no act of appropriation since it is impossible to identify the precise goods which are to be delivered to Archie. Property, and therefore risk, has not passed to Archie. &REV rev1 Have you discussed the issue of whether Beatrice is responsible for the loss of the goods? "y" rev2 "n" rev4 rev2 Have you advised that Beatrice is responsible for the loss of the goods? "y" rev5 "n" rev3 rev3 Have you advised that Beatrice is not responsible for the loss of the goods? "y" rev6 "n" rev4 rev4 You were asked to advise! Please press 'y' to continue. "y" fail "n" fail rev5 By s25 risk prima facie passes with property, and property has not passed since the goods have not been appropriated to the contract: s21. Have you explained this in your answer? "y" pass "n" fail rev6 By s25 risk prima facie passes with property, and property has not passed since the goods have not been appropriated to the contract: s21. Have you explained why this does not apply to the present fact situation? "y" pass "n" fail
Footnotes:
Allen, T and Robinson, W "The Future of Computer Assisted Learning in Law" (1992) 3 JLIS 274.
Most of the time is spent in designing the flow charts. A language such as Andrew Mowbray's LES can be used to code the flow chart very quickly. It may be that flow-charting programs could assist in reducing the cost.
Kulik, et all, "Effectiveness of Computer-based College Teaching: a Meta-analysis of Findings", Reviews of Educational Research, 1980, Vol 50 No4, pp 525 - 544. See also Teich, Paul "How effective is CAL?" 41 J of Leg Ed 489.
One-on-one tutorial methods are generally more effective than other teaching methods; see Bloom, BS, "The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring" [1984] Educational Researcher 4-16.
See Laurillard, Diana, Rethinking University Teaching: a framework for the effective use of educational technology, Routledge, 1993. Laurillard classifies eductional technology according to the teaching/learning functions that it allows and encourages. She advocates choosing a "mix" of technologies that will provide between them all of the necessary functions.
McGregor-Lowndes, M "Computer Tutorials for Business Law" ALTA Conference Proceedings, 1990.
See generally the report on the QUT experience: XXX
We say "perceived" because the law teacher's view of them is not shared by experts in assessment: see, for example, Heywood, J Assessment in Higher Education, 2nd Edition, Wiley, New York, 1989.
See appendix I for a sample CRES question.
Indeed, the CRES method is being evaluated for use in developing a series of tutorials in metaphysics!
A year earlier Christine Chinkin and Alan Tyree had used short answer questions as a form of tutorial. After answering, the students were given the opportunity to compare their answer with a model.
This result is interesting in itself. The only information available to the students was a short in-class demonstration of CRES.
The result was not statistically significant if the null hypothesis was 'no difference'. However, the results are almost exactly those found by the Kulik, et al, study mentioned above. We also find that students at our university find the concept of an 'insignificant' increase in marks to be meaningless!
The grant was awarded to Alan Tyree, Shirley Rawson and Don Rothwell, all of the Faculty of Law, University of Sydney.
The legal research course was run once at the University of Sydney as a Keller Plan course. Political opposition was severe and the project was dropped. The on-line system belongs to Shirley Rawson; it is a complete self-executing course which runs under Windows.
Bloom, BS, "The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring" [1984] Educational Researcher 4-16.
Bloom, BS, "The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring" [1984] Educational Researcher 4-16.
Again, the study guides follow the guidelines recommended for Keller Plan courses: Keller, FS, and Sherman, JG The Keller Plan Handbook, Benjamin, 1974
The textbooks used were McKeough and Stewart, Intellectual Property In Australia, Butterworths 1991, and Blakeney and McKeough, Intellectual Property: Commentary and Materials, Law Book Company, 1992.
Our records show that students spend an average of 6 to 10 minutes per question. Based on these figures, we consider 7 questions to be a "one hour" tutorial.
This also overcomes a curious problem. Although the idea of the Critical Review Tree seems simple enough, the construction of the tree evidently involves a level of programming skill. Some authors seem to find the construction difficult.
This is almost true - in most cases some final "hand-crafting" is required.