Teaching and Learning Forum 2000 [ Proceedings Contents ] Student questionnaires and teaching evaluation: Cutting the cloth to fit the purpose A panel discussion by Elizabeth Santhanam Centre for Staff Development, University of Western Australia Christina Ballantyne Teaching and Learning Centre, Murdoch University Denise Mulligan, Barbara de la Harpe Centre for Educational Advancement, Curtin University of Technology and Rod Ellis Educational Development Unit, Edith Cowan University There are almost as many formats for obtaining student views on teaching as there are universities in Australia. The drive for improving teaching quality in higher education in the last two decades has resulted in universities developing their own questionnaires or adapting questionnaires developed at other institutions to suit their purposes. Among the stakeholders of student evaluation of teaching are students, academic staff and heads of departments/ schools/ centres. Each party has its own agenda, and the instrument that is used is expected to serve all their interests. Given the divergent interests of some stakeholders, as well as changing economic and policy trends, the system to evaluate teaching using student feedback tends to evolve over time. In this discussion, the panel members will describe the current system of student evaluation at each of their institutions, namely The University of Western Australia, Murdoch University, Curtin University of Technology and Edith Cowan University, and provide the reasons for adopting their particular system. They will discuss the respective policies that govern student evaluations of teaching, and how the various stakeholders' interests are met. The open discussion to follow will further clarify some of the issues, and the panel members will attempt to dispel some of the frequently stated myths regarding student evaluation of teaching.

Introduction

Typically, this probing of student views is through questionnaires. Questionnaires can be used to provide an overall view of teaching to administrators (including heads of departments/ schools/centres), to diagnose individual academics' teaching for further development, and to provide students with the opportunity to indicate their needs to academics/administrators. Universities have either developed their own questionnaires or adapted/adopted a questionnaire developed elsewhere. Often the questionnaire undergoes many changes before a general consensus is reached regarding its effectiveness. Some aspects of making questionnaires fit the purposes of student evaluations of teaching and units can be seen in the way four universities in Western Australia currently facilitate the process.

Student feedback on teaching at UWA

SPOT evaluation is voluntary and confidential. The SPOT policy stipulates that in a group teaching situation evaluation of teaching should be carried out with the permission of all those involved in teaching. An academic whose teaching is being evaluated should be absent during the administration of the survey. Usually a colleague in the department administers the survey. Completed responses are enclosed in a sealed and signed envelope, and forwarded to the SPOT Office at the Centre for Staff Development. In order to minimise the real or perceived threat students may feel when giving honest opinions, SPOT is anonymous. A teaching survey report is sent to the academic whose teaching was evaluated. A unit survey report is currently sent to all academics teaching in the unit, but this practice in being reviewed. Heads of departments are sent SPOT reports only when the lecturers/tutors have agreed in writing to release their reports.

A standard survey report produced by the SPOT Office consists of a detailed breakdown of student responses for each item i.e. number of student responses for each response option, and a mean value of the ratings; ratings are based on a scale of 1 (Strongly Disagree) to 5 (Strongly Agree). For ease of interpretation, bar charts showing percentage responses for each option are also plotted. A summary sheet accompanies these details, listing the items and their mean scores. Rating Interpretation Guides, which provide comparative data for the three global items, are available on the Web. Comparative data for the other items are not provided, since each questionnaire is unique.

Student feedback on teaching at Murdoch

The coordinator of each coursework unit with more than ten enrolled students is sent questionnaires for distribution in the last two weeks of semester. Instructions are provided for a student to collect and return the results for analysis. The standard questionnaire for units addresses the Key Quality Standards, and provides feedback to the unit coordinator on whether they are being met. This questionnaire can be seen at [http://cleo.murdoch.edu.au/evaluation/survey/draft.html]. Results from individual unit surveys are sent not only to the individual unit coordinator, but also to the Dean and the head of the Academic Organisational Unit. Aggregated results at Divisional and University level are available on the web at [http://cleo.murdoch.edu.au/evaluation/survey/results.html].

With the changeover to this new system of annual surveys of units, there was a need also to formulate a means of collecting student feedback on an individual's teaching. A standard questionnaire was constructed with questions relating to the criteria for assessing items of evidence of teaching that promotes effective learning, as identified in Table 2 of the Guidelines for the Presentation of a Teaching Portfolio [see http://wwwadmin.murdoch.edu.au/hr/traindev/teachportfolio.html]. Teaching surveys are currently by request only and the results are returned in confidence to the teacher concerned. To avoid the need for students to complete two separate surveys at the end of the semester, it is recommended that teaching surveys are undertaken in the middle teaching block - weeks 5-9 - of the semester. Results are provided by week ten to allow staff to respond to the students concerns, giving the teaching surveys a more formative aspect. The teaching questionnaire can be seen at [http://cleo.murdoch.edu.au/evaluation/survey/teachdraft.html]

Student feedback on teaching at Curtin

After extensive trials in 1995 and 1996 (Teaching Learning Group, 1997) the Teaching and Learning Advisory Committee and University Academic Board (UAB) decided to adopt SEEQ over a number of other student appraisal systems for the following reasons:

• There is substantial research evidence that SEEQ is a valid and reliable instrument for measuring students' perceptions of teaching, that is relatively unaffected by variables that may bias responses (e.g. class size, level of unit);
• SEEQ is specifically designed as a self development package, thus having the potential to contribute to Objective 4 of the Curtin University Teaching and Learning Strategic Plan i.e. To encourage reflective practice by staff; and
• Professor Marsh was willing to collaborate to further develop SEEQ to suit Curtin's needs.

• SEEQ closed response form, consisting of 33 items that combine to provide indicators of quality in nine key areas, namely learning and academic value, instructor enthusiasm, organisation and clarity, group interaction, individual rapport, breadth of coverage, examinations and grading, assignments and readings, and workload and difficulty, plus an overall rating;
• SEEQ open response form where students provide written comments;
• supplementary Rating Items developed by Curtin staff to provide feedback on a range of teaching modes not covered by the standard SEEQ form;
• provision for lecturers to formulate up to 40 of their own questions in addition to the standard SEEQ items;
• SEEQ Lecturer Self Rating version of the form; and
• support materials in the form of Targeted Teaching Strategies booklets.

Following administration of the survey, the staff member is provided with:

• a Summary Report that includes mean scores and percentile graphs for the nine SEEQ factors and the overall rating, and for each of the 33 items. For comparative purposes only, university norms for the nine SEEQ factors and overall rating are also included;
• an interpretation guide (Your SEEQ Report) that includes general advice on how to interpret SEEQ scores and how to seek follow up advice; and
• the URL [http://cea.curtin.edu/seeq/follow.html#tts] for access to the SEEQ Targeted Teaching Strategies booklets online - these aim to help staff to improve their teaching and relate directly to the feedback provided by the SEEQ survey.

Student feedback on teaching at ECU

• Every unit of study at ECU is evaluated using the centralised Unit Evaluation process once every 3 years (more often if requested by a Head of School or Unit Coordinator).
• The instrument consists of 16 core questions that must be included, two open ended questions and up to 14 questions that can be selected from a bank of 350 questions grouped in 20 different categories. The requestor can also design their own questions as part of the 14 alternative questions.
• The instrument is administered by 'people' independent of the presentation of the unit and the forms are collected and returned to the Programme Review staff directly for computer processing.
• The forms are processed and the results are made available to Heads of School and the Unit coordinator (core questions only) and the individual lecturer (all questions). The Head of School and Unit coordinator are not permitted to forward data to Deans or the University Executive or to other groups such as promotion panels, etc.
• Aggregate Unit Evaluation data is computed from all returns and the results are returned to Faculty Deans (anonymously) for unit and courses comparisons. The same data is also made available on the web for general interest.

The instrument has been used now for about 5 years and has undergone several 'improvements'. The current process specifically targets unit design, unit content, workload, presentation (teaching), assessment and feedback. It was designed more for unit feedback than teaching effectiveness feedback although these obviously overlap.

The University has recently emphasised teaching and learning as a key strategic issue and has recognised that a more comprehensive 'teaching effectiveness' process is required. One of the driving forces is the need to have reliable and valid evidence of teaching effectiveness for staff promotions and appointments.

To this end a working party is currently devising a second instrument that deals specifically with the performance of people who contribute to the teaching of a unit of study. Such an instrument will be able to be used at any time in the running of a unit and not just as the 'normal' end of unit evaluation exercise. The results would be made available only to the lecturer requesting. One of the most difficult issues is to design a universal instrument that copes with the many varied ways in which staff work with students in the presentation of a unit of study.

Conclusion

The advantages and disadvantages of these instruments, as well as other aspects of student evaluations of teaching, can be discussed during the forum:

• What fears/frustrations do academics and students have with the existing systems?
• What are the benefits for different stakeholders?
• What should be included/excluded/changed?

References

Marsh, H. W., & Roche, L. A. (1994). The use of students' evaluations of university teaching to improve teaching effectiveness. Final project report for the Evaluations and Investigations Program of the Department of Employment and Education. Canberra: Australian Government Printing Service.

Teaching Learning Group. (1997). Student evaluation of teaching at Curtin University: Piloting the Student Evaluation of Educational Quality (SEEQ). A report on a Quality Funded Project to the Teaching and Learning Advisory Committee and the Promotions Review Committee. Perth, W.A.: Curtin University of Technology.