The Ceilidh CAL system [1,2] was developed at the University of Nottingham in order to support the teaching of programming courses with a special emphasis on the preparation and assessment of practical exercises. We describe the adaptation of Ceilidh for teaching the Oberon language at the University of Ljubljana. In order to minimize the extent of modifications, a careful choice of an appropriate Oberon compiler was necessary, and a compilation procedure that allows the compilation of several modules had to be defined. Major changes were necessary in order to adapt the six tests that compose the Ceilidh marking system, especially the built-in typographic and complexity analyses.

1.Introduction

The successful learning of programming courses requires a lot of practical computer work. This necessity causes a lot of additional work for the teaching staff, who have to assess and mark student program solutions. Such courses are usually performed in the first year of undergraduate study when the number of students is the highest. The bottom line is that this affects the educational quality of courses. However, we believe the computer can be valuable in this situation.

Several Computer Aided Learning systems (CAL) are currently known. One is the Ceilidh system (University of Nottingham)[1,2], that was the base of our experiment. Apart from the presentation of the learning subject (the usual goal of CAL systems), Ceilidh also performs the automatic assessment and marking of student exercises, and the administration of the course. This makes Ceilidh a suitable computer aid for the difficulties described above.

Within our department the Oberon programming language [3] is taught as the first programming course [4]. Each year the course has more than 300 students, and it creates the problems of intensive marking and assessment. When we recognized the features of Ceilidh and tested some already implemented programming courses, we started an experiment to determine the possibility of performing an Oberon course for Ceilidh. It might seem that the creation of a new Oberon course offers a straightforward solution. However Oberon has some special features (modularity, OO characteristics) that require special attention. First of all, an initial adaptation of Ceilidh for the Oberon programming course is required, and it is this process which this article describes.

2.The Ceilidh system

Our presentation begins with a description of the Ceilidh system. Although Ceilidh is not a result of our work, a more detailed knowledge of it is needed in order to understand the points we make later.

Ceilidh is a CAL system, which has three main functions:

• the presentation of course materials to students;
• automatic marking and assessment of student work (exercises);
• administration of academic courses.

2.1 Ceilidh structure and users

The structure of Ceilidh can be described as a tree of system components. Such a description corresponds to the directory layout of Ceilidh. It has four levels and each level's functionality reflects a structural view of the system. The system level is the top level and as such gives the widest overview of system functions. It enables access to system data (configuration, documentation, help), use/maintenance of common tools and user interfaces, and access to a number of built-in courses. The course level reflects functions needed for running a course. It includes basic course data (overview, student lists), special tools (for course implementation) and a number of chapters. The chapter level provides access to distinct parts of a course through chapter notes and a number of its exercises. The lowest level is the exercise level, which includes things connected with one exercise: problem description, sample, solution skeleton, student solutions and marks.

The Ceilidh structure can be recognized also from the features of the main system components. The three main components are: four user interfaces (command line, menu, X Windows and WWW), several tools (internal and external) and a database (help system, documentation, and courses).

Ceilidh defines 5 groups of users: students, assistants, teachers, course developers and system administrators. The group rights reflect the different roles of the Ceilidh users, where students have the least, and administrators all, rights. The groups are implemented by the Unix user system.

2.2 Automatic marking and assessment

The most valuable (and complicated) part of Ceilidh is automatic assessment and marking. From a student's point of view each exercise consists of the problem description, sample results and solution skeleton. By using these the students write their solutions, compile them with the built-in compiler and execute them. They can repeat those steps several times. Whereupon they can submit the solution. This is automatically assessed (with a number of tests) by using additional exercise data, which were defined during the exercise creation by the teacher. Obviously, such data are inaccessible to students. Finally the evaluation marks and comments are returned to the student. If they are not satisfied with the marks received, they can rebuild their solution and hand it in again, but this would also mean a lower mark. The teacher defines the deadlines for the assignment, and the number of attempts allowed. Ceilidh also has provision for a plagiarism check, which can be used after all the solutions are received.

Marking is based on six types of tests that can be performed dynamically (by executing a student solution) or statically (by parsing the solution code):

• The dynamic correctness test assesses the correctness of the student’s solution by its execution on several predefined sets of input data. For each such set the results must be predefined, so the correctness is evaluated by comparison between the actual and expected results. The test mark depends on the number and difficulty of successful comparisons.
• The dynamic efficiency test compares the student’s execution speed and the "optimal" solution, which is previously defined by the teacher. A speed test allows the evaluation of efficiency. The mark is defined by the evaluation of a relative speed factor. For the moment only the assessment of solutions written in C or C++ is possible.
• The typographic analysis test assesses the shape and syntactical correctness of a student's solution. Depending on the programming language used, the course developer has to define a number of rules (like "average percentage of comment rows must be 15% to 30%") and assign point values to them. During the assessment these rules are applied to the solution program, and a test mark is generated. Comments for the students are also created.
• The complexity analysis test is based on the idea that the code for the predefined teacher solution is optimal from the complexity viewpoint. We can create complexity rules by using this solution as a basis (like "the number of loops needed is 4"). Once the rules are defined, the test uses the same method as the previous one.
• The program features test parses the student’s solution and determines whether (and how many times) a certain predefined word appears in it. By doing this, the test can assess what features the solution achieves (like "word (function) Out.String should (not) be used"). The mark is summarized from points given for each successful feature testing.
• The program structure test is based on some compiler features and is enabled only for certain programming languages. Compilers can usually write warnings on a predefined file during its compilation. Many of those are meaningless, but some show the structural weakness of the compiled program. This test parses the file of warnings using a similar method as the previous one, except that only negative marks are given for each structural weakness found.

2.3. Implementation

The core of Ceilidh is basically a collection of Unix shell scripts, AWK programs and compiled C (C++) programs glued together in a hierarchy of directories. Together with links to additional tools and by relying on Unix file and user management system they form the described structure. User interfaces are integrated into the system to ease user work, but at a low level they all execute the same programs and run the same tools.

The core of automatic assessment and marking is a C program called Oracle, which is the implementation of a parser. It can parse the file searching for words, which are defined as regular expressions and count them. After that it can mark the results of searching considering one of four possible rules.

3. The adaptation of Ceilidh system for Oberon

The adaptation of Ceilidh for teaching Oberon required the following modifications: a selection of an appropriate Oberon compiler, a modification of a common compiling procedure in Ceilidh, and some major changes in the implementation of the Ceilidh tests for Oberon.

3.1. Selection of an appropriate Oberon compiler

For each Ceilidh programming course a compiler is the most important additional tool and it also has an important influence on the efficiency of Ceilidh. So our first task was to select an appropriate Oberon compiler. We first defined requirements for selection. Obviously, the compiler must run under the Unix system in way similar to other Unix compilers. If we want to completely preserve the Ceilidh’s functionality, the selection is in many ways determined by the basic Ceilidh orientation for C (certain tests are strictly connected to a cc or gcc compiler). So, the selected compiler must have a similar behavior and all the required results. The final request is the compiler’s current availability for use and latter integration.

We can divide Unix compilers for Oberon in two groups: the first group compiles the Oberon source code directly to the executable program, while the second translates the Oberon code into C source code first and then uses a cc or gcc compiler to create the executable code. The second group is more appropriate for us as it guarantees the required operation and results. On Internet we found many Oberon compilers, which could be appropriate for integration. We tested four (five) of them:

• Ofront (Software Templ) had some very good qualities, but it had problems with availability, as only a small part was available for free. Further, we had difficulties with the file locations that caused additional unnecessary complications.
• pOt (University of Moscow) was fully available, but it did not offer enough support. We had problems even during its installation. Additionaly, we discovered it was made for a standard Oberon code while we preferred a superior Oberon-2 code.
• O_TO_c (University of Ljubljana) had some good qualities (it generates ANSI C standard code) and suitability for the Oberon-2 code. However at the time of experiment it was still under development and not entirely usable, so we could not use it.
• oo2c (University of Köln) offered the best selection. We first tested its precessor o2c but it also lacked Oberon-2 support. oo2c is built for compiling the Oberon-2 code. It is fully available, has all the required functionality and generates an optimized ANSI C code. So we chose the oo2c compiler.

3.2 Compiling of Oberon modules

That approach worked well for single module Oberon programs. However, one of the main features of Oberon is its modularity. Oberon programs usually consist of a hierarchy of several modules and the order of compiling is determined by the module hierarchy. The common Ceilidh procedure to solve the compilation problem of several files is to use an additional file named Makefile. But in the case of Oberon this solution fails as in Makefile the order of files for compilation is static. Additionally, the problem of editing arises, because the possibility to edit a previously unknown number of files is required. We solved the problem with a definition of rules for writing Oberon programs under Ceilidh, and with a modification of the compilation script file.

Rules: We decided that all Oberon modules must be written in one file only, the low level modules being declared first (as they must be compiled first) and the high level last. The solution skeleton of the teacher can define all the necessary modules and their order. In this way it can strictly control order. If the solution skeleton is not defined, solvers have more freedom for module definition, but they must maintain the right order of modules in file.

Modified script file: This was neccesary because the compiler cannot compile more than one module in a file. So we first temporarily divide the original source file into several files (as many files as modules). After that we compile them separately in correct order and we create one executable file. At the end all temporary files are deleted.

3.3 Automatic assessment and marking of Oberon programs

At first sight the implementation of automatic assessment and marking by Oberon programs seems pretty complicated, but it proves that it is possible to retain the core of Ceilidh marking system (program Oracle) unchanged. The Oracle is in one way or another the basis for each test in Ceilidh. Relying on the implementation of tests for the C course we adopted all tests for Oberon in following manner:

• The dynamic correctness test simply runs the compiled solution (executable file) on several sets of input data and counts successful attempts. By solving the problem of Oberon compilation the execution is no longer a problem. The only remaining problem is the file name conventions as the test must operate on files with extension Mod and not c. This requires the modification of a system script file, therefore a similar script file must be created for the Oberon course only.
• The dynamic efficiency test uses Unix system utility tcov, therefore it works only for solutions written in C. In our case we base the assessment on two C source code files, which are built during the compilation of student and teachers’ solutions. As those C solutions reside on the right directory after each compilation, only the naming conventions must be solved (as in previous test) using an additional script file for the Oberon course. This test strongly relies on the hypothesis, that the oo2c generated C code preserves the same speed characteristics as the original Oberon code.
• The typographic analysis test depends on the programming language used. In spite of the fact that Oberon and C are both Algol-like languages, we had to rewrite the implementation of this test entirely. The changes were necessary because of (1) syntactical differences, (2) semantic differences, and (3) different pretty programming standards. An additional effort was required to define the file of pretty programming standards. We had also to change the shell script file for starting this test because of naming conventions.
• The complexity analysis test is similar to the previous test and had to be rewritten as well. The cause of changes was similar as in the previous case except that some additional measures had to be taken into consideration. This test required changes of script files because of naming conventions too.
• The program features test is in fact a direct use of the Oracle program on the Oberon solution code. In a testing sense it is independent of language and it parses solutions as a text file. The only adaptation needed was due to naming conventions.
• The program structure test is currently implemented in the same way as in the C course and uses a Unix lint utility. As in the previous case the program structure test uses C source file (generated by oo2c) for its base. The only reason for changes were naming conventions. However this is only a temporary solution and should be changed in the future.

4. The Oberon course at our faculty

As already stated Ceilidh was originally developed to help programming course teachers of large classes and the Oberon course at our faculty is one of them. This course has a simple organization of course lessons, and suitable course material for development of the Ceilidh course. More importantly, the course currently has a practical work policy similar to that implemented in Ceilidh. The students have to solve and hand in several programming exercises on a regular time basis, which are afterwards marked by assistants. There are also some additional benefits to the implementation of Ceilidh. It provides a computer-supported course materials presentation to the students. There exist several developed Ceilidh courses and students can use them for self-learning. The teachers can also point to additional information on the Internet by using Ceilidh features.

However, there are also some difficulties with Ceilidh implementation. A clear assessment and marking policy has to be defined for Ceilidh use at our institution, where the roles of Ceilidh users and principles of work might be different as in its current implementation. Language is also a problem, as Ceilidh currently uses English and our formal teaching language is Slovene. Additionally, as we mostly use PC computers the appropriate way of using Ceilidh would be through Internet. A dificculty arises because not all students have the Internet connection at home. Thus many of them would still have to work during exercise hours at the faculty thus diminishing the benefits of Ceilidh.

5. Conclusion

At the moment the decision for (or against) Ceilidh use has still to be taken at our faculty. However, we have taken the first step by experimenting with the adaptation of Ceilidh for the Oberon course. The second step would be to develop a part of Oberon course and use it on a pilot group of students. After an analysis of their remarks, the entire course could be developed and used in class.

References

1. S. Banford, E. Burke, E. Foxley, C. Higgins. A Courseware System for the Assessment and Administration of Computer Programming Courses in Higher Education. LTR Group, CS Department, University of Nottingham, United Kingdom. (http://www.cs.nott.ac.uk/~ceilidh)

2. S. Banford, E. Burke, E. Foxley, N. Gutteridge, A. Mohd Zin. The Design Document for Ceilidh Version 2. LTR Group,University of Nottingham, UK. (http://www.cs.nott.ac.uk/~ceilidh)

3. M. Reiser, N. Wirth. Programming in Oberon - Steps beyond Pascal and Modula. ACM Press, New York, 1992.

4. V. Mahni?. Some experience in teaching an introductory course using Oberon. In C. Tasso et al. (Eds.) Software Quality Engineering, CM Publications, 27-36, Southampton, 1997.

Address

Igor Rozanc, Viljan Mahniè
University of Ljubljana
Faculty of Computer and Information Science
Trzaška 25, SI-1000 Ljubljana, Slovenia
E-mail: igor.rozanc@fri.uni-lj.si, viljan.mahnic@fri.uni-lj.si