Abstract

In this work we describe a new layer of services for the Intranet of the Engineering Faculty of Porto University, corresponding to the extraction of relevant derived information specifically designed to support the several levels of school management.
 
 

Introduction

The assessment of higher education institutions is traditionally based on counting resources, collecting performance data, and concentrating the corresponding ratios in a bottom-up manner, until final reports are elaborated. More recent trends tend to focus both on resources and processes. For instance, Engineering Criteria 2000 [1], the new set of criteria starting to be used in the accreditation of US engineering programmes, insists on each programme having an assessment system that can actually influence how the programme works.

One way this feedback loop may become more effective is through campus-wide availability of some of those indicators and reports, as soon as they are produced. In fact, some corrective measures can be spontaneously triggered by the people involved and, more importantly, in the whole academic community awareness of the situation increases, preparing the ground for the required changes decided by the administration, while enabling more participation in the innovation process. The easier way to achieve that availability is by including the relevant elements in the institutional information system. There, they can be permanently consulted and related with other information concerning the programme.

In second phase of the development of the Intranet of FEUP (see [2]), we intend to refine the previous modules according to the experience already accumulated and to the suggestions received from the users. But we also want to add a new layer of services corresponding to the extraction of relevant derived information specifically designed to support the several levels of school management. In this work we will describe these services, of which some are already implemented.

Planning the teaching service

The task of planning and assigning the teaching service each year requires some preparation. For every course, the number of hours of each kind (lecture, tutorial, laboratory) offered to a class is fixed in the study plan. But the number of classes may vary each year due to fluctuations in the number of students. Though most programmes receive yearly a fixed number of freshmen, the level of fails a course has got in the preceding year determine the potential number of enrolments in the following edition. Moreover, there is a number of students, more significant in the first three years of undergraduate programmes, who, despite enrolling themselves in several courses, never show up or are submitted to evaluation. In order to avoid both empty and crowded classes, the number of classes planned for each course in a year is based on the number of evaluated students of the previous year. Once these numbers are established, the amount of teaching hours required by each programme is known. The workload is then distributed among the teachers.

All this information is recorded in the SiFEUP (FEUP's information system). It is thus able to give back control maps pointing out any discrepancies between the assignment of classes to teachers and the previously defined workload attached to each course. Some of the maps are tailored for programme directors (see figure 1). They are organised by course and highlight under- or over-assignment situations. Other maps are meant for department directors and are organised by teacher, computing total and average workloads. Summary results enable comparison among groups and departments.

Figure 1. Control map for programme director.

These maps consolidate the information about the participation in several undergraduate and graduate programmes. The common availability of these maps through the net allows programme directors and department and school managers to earlier detect incompatibilities or abnormal situations.
 
 

Student and course performance

SiFEUP displays both individual data and summary values about the student's behaviour and course results. The student record (figure 2) shows, for every year the student has been registered, the courses taken and the corresponding results, as well as the on-going global classification.

Figure 2. Individual student record.

This record is available only to the student concerned and all the lecturers. It plays a fundamental role in supporting personalised treatment, as it gives an instant view of individual performance.

From the course viewpoint, there is a set of figures summarising the results obtained, attached to the course description. To help to put these numbers in perspective, the evolution along several years is given, in a table of the form shown in figure 3:

Success rates

Programme:

Course: Year:
 

Year	Enrolled (R)	Evaluated (E)	Approved (A)	Evaluated rate (E/R)	Success rate (A/E)	Gross success rate (A/E)	Average mark	Approved average mark
1992/1993
...
1997/1998

SiFEUP, 1999-03-18

Figure 3. Results computed for each course.

Detailed statistics, like the ones just shown, enable the characterisation of several student profiles and thus to develop an understanding of the average behaviour as a basis for better planning. Indicators of course performance help to realise where potential problems may arise and to monitor the teaching activity.

To support external quality assessment of a programme, more aggregated data is needed ([3]), like the dropout and graduate rates, that can give a global snapshot of the fate of each generation of freshmen (see in figure 4 an example table for a 5-year programme). However, this table is not suitable as management data in the feedback loop of quality improvement, because of its inherent excessively long delay.

Dropout and graduate rates

Programme:
 

Generation

Freshmen

% Dropout after x years

% Graduate after

On-going

1

2

3

³4

Total

5

6

7

³8

Total

1990/1991

1991/1992

*

1992/1993

*

*

1993/1994

*

*

*

1994/1995

*

*

*

*

*

1995/1996

*

*

*

*

*

*

1996/1997

*

*

*

*

*

*

*

1997/1998

*

*

*

*

*

*

*

*

1998/1999

*

*

*

*

*

*

*

*

*

*

SiFEUP, 1999-03-18

Figure 4. Evolution of dropout and graduate rates.

A similar table, this time organised from the perspective of the graduates of each year, is presented in figure 5. The system presently lacks information about the first employment of the graduates, a shortcoming soon to be corrected.

Graduates

Programme:

SiFEUP, 1999-03-18

Figure 5. Graduates indicators.
 
 

Course assessment by students

In the FEUP there is the practice of passing a questionnaire to the students whenever a course finishes. The answers are anonymous and optically read by machines. There are two groups of questions, one about the course and the other one applicable to each lecturer. The topics questioned include, for the course: syllabus adequacy; difficulty; required amount of work; students' interest on the subject; bibliography; co-ordination between lectures and tutorials/lab work; quality of the laboratories; assessment methods, global opinion. For the lecturers: presentation of course goals and methods; subject presentation; motivation ability; punctuality; availability to help the students; competence; lecture preparation; global opinion.

The answers, expressed in a 1-5 scale, are collected and grouped by lecturer in a course, and by course. Although all the data is stored in the SiFEUP, the access to individual lecturer and course results is restricted to the lecturers involved in the course and to the programme director. Students have access to more aggregated values, about the performance of all the courses in a year or in a scientific area.

Pedagogic questionnaires

Programme: Academic year:

Course: Year: Semester:

Answers:

Lecturer:

Answers:

SiFEUP, 1999-03-18

Figure 6. Results of the student questionnaires.
 
 
 
 

Scientific productivity

Finally, we will refer the indexes of scientific productivity, like the number of publications organised by kind, which enable to evaluate personal results in the context of its research area.

Publication rates

Department: Researchers:

SiFEUP, 1999-03-18

Figure 7. Scientific productivity per capita.
 
 
 
 
 
 

Conclusion

The conclusion is that the school management has now more accurate sensors of the actual academic and research activities, which support decision making better suited to the school reality.

As a future development, a study using data mining techniques is being set up in order to find associations between the characteristics of incoming freshmen and their future performance.

[1] Aldridge, M. Dayne; Benefield, Larry D.. Assessing a Specific Program. In "How do you Measure Success?", American Society for Engineering Education Professional Books, 1998.

[2] Ribeiro, Lígia M.; David, Gabriel; Azevedo, Ana M.; Marques dos Santos, José C. Developing an Information System at the Engineering Faculty of Porto University. In Yves Epelboin, EUNIS 1997, Grenoble.

[3] Vroeijenstijn, A.I. Improvement and Accountability: Navigating Between Scylla and Charybdis. Guide for External Quality Assessment in Higher Education. Jessica Kingsley Publishers, 1995.
 
 

Address

Lígia Ribeiro

Faculdade de Engenharia da Universidade do Porto

CICA

Rua dos Bragas

4099 PORTO CODEX

PORTUGAL