Curriculum evaluation: Considering the interests of all the stake holders

Evaluating any curriculum can be frustrating and challenging. This is especially so when there are multiple stake holders from diverse backgrounds and philosophical positions. An evaluation of the Bachelor of Science (Nursing) curriculum, was undertaken 1995-1996. The science stream, however, was not completely reviewed. This created a need to evaluate the science units separately from other units in the program.

The subsequent evaluation of the science units involved a multidisciplinary approach. In order to preserve the credibility of the process, it was necessary that procedures were acceptable to all teaching staff. This presented an interesting array of challenges requiring ingenuity, patience and dedication to the task.

Stake holders were recognised as the students enrolled in the undergraduate program, teaching staff from both the School of Nursing and other schools within the Division of Health Sciences. The evaluation was conducted by members of the nursing staff involved in the science stream, with resources being provided by the School of Nursing. The methodology took both a qualitative and quantitative approach utilising questionnaires and focus groups.

This paper describes the process and outcomes of the evaluation and discusses the implications of the findings for other disciplines, confronted with similar program reviews.

Introduction

Methodology

This study differed from the previous evaluation in that the stakeholders were not only the School of Nursing and nursing students, but also the service teaching schools in the Division of Health Sciences.

As part of the collaborative approach between stakeholders, the study commenced with a staff workshop. The purpose of this was to identify nursing academic perceptions of the strengths and weaknesses of the science stream. Similar meetings were held with the relevant teaching staff of all service schools to elicit their cooperation and involvement in the study. This progressed to the collection of student derived data, using a questionnaire and focus groups. The final phase of the study will be to inform and share with all relevant schools the outcomes and recommendations.

Research tools

The only modification required was a change in the focus of the questionnaire from "course" to "unit" . The questionnaire was multi-dimensional, composed of both quantitative and qualitative inquiry. Quantitative data was collected using a thirty item Likert-type scale, ranging from strong disagreement to strong agreement. The items were randomly ordered, with eight items being reversed. Two open ended questions were included to gather qualitative data. These questions asked for comments regarding the best aspects of the course and those aspects most in need of improvement. Open ended questions allowed the generation of a wide range of replies and provided the researchers with some insight into prevailing attitudes. The comments from these were later coded into 6 categories : teaching and learning, unit material/resources, unit structure, assessment, student issues and staff issues.

Sample

Data Collection

Questionnaire

Focus groups

Questions for the interview protocol were planned to clarify the themes that emerged from the open ended responses on the questionnaire. Items from the CEQ analysis indicating student disagreement were rephrased to form open ended questions and redirected to the students for more specific comment and clarification. The interview protocol was semi-structured with broad questions preceding specific questions. It was used as an outline and guide for the facilitator to follow during the focus group discussion, as strict adherence to the questions and changing topics may have stifled spontaneous comments and reduced facilitator flexibility. The flow of discussion was maintained by the facilitator prompting and probing statements and comments and keeping students on track. This drew out a range of perceptions and beliefs within group. Using this method the facilitator endeavored to gain a broader understanding of the student responses and the reasons behind their attitudes and behaviors. Sessions were recorded on audiotape in a sound studio. Each focus group continued until the interview protocol was completed and discussion was exhausted, these sessions took up to one hour for each group.

Analysis

The values of the Likert scale were altered in the calculation of the mean to where 1 = -100; 2 = -50; 3 = 0; 4 = +50 and 5 = +100. Scoring for the negatively worded items was reversed. Values ranged between +100 and -100. A score of -100 represented strongly disagree and +100 for strongly agree. Negative scores were interpreted as disagreement and positive scores indicated agreement with the item. Percentages provided an indicator of overall agreement or disagreement.

The written qualitative data was examined for recurring patterns. These patterns were categorised and used to provided direction for focus groups. Transcription of the focus groups have yet to be analysed using a thematic content analysis (Burnard, 1991).

Results and discussion

The results from the analysis of the questionnaire were examined identifying common themes and trends. The score for each unit on each of the subscales was compared to the stream average for that subscale and those units which scored poorly were noted. A unit was highlighted if the score was negative or if the score was markedly less positive than the stream average. Using this method, significant inconsistencies in the subscales were identified. The analysis showed a marked difference in the perception of the students with regard to individual units. Significant variation was found on the "Good Teaching" scale. Those units which were taught by nursing academics ranging from +24 to +28, while service taught units scored between - 4 and -27. The stream average for this subscale was -4. Scores for other subscales were more balanced and did not demonstrate such a wide variation.

In general, the open ended questions from the questionnaire posed a dichotomy between the nursing and service taught units. Some of the most salient negative responses which were outside the scope of CEQ were attributed to the service taught units. There was a strong student perception that academic staff involved in these units lacked commitment to the unit and to nursing students in general. Students listed a lack of support, derogatory comments against the Nursing profession and in some cases open hostility towards them as a significant reason for their impaired learning. In some cases failure or withdrawal from the unit was attributed to this problem. Other issues identified were, lack of articulation of service units with the rest of the curriculum, lack of direct application (focus) and relevance of content in the service units to the practice of nursing. Inappropriate and inconsistent assessment practices were also noted. Some service units did not follow the policies and philosophies of the School of Nursing. Many of these comments, in particular those relating to relevance, content and sequencing, were also raised as points of issue by nursing academics in the workshop held at the commencement of the study. While there were many negative comments, students generally recognised the relevance of the science units to their clinical practice. This perception strengthened as the students proceeded through their course.

Analysis and examination of the data gathered in the focus groups has yet to be completed. Initial inspection of the transcripts, however, suggests similarities with the outcomes from the questionnaire.

Implications

There are two district areas which need to be considered in light of the findings to date; academic issues within the process of evaluating service taught units and the potential impact of service teaching on student learning outcomes

Issues for academics

In a project of this nature, reservations may be held by some team members with respect to the selection of the research process. These reservations may be based on the ideological orientation and personal experiences of the individual academic. The use of an evaluation process which has rigor, validity and reliability will encourage acceptance of the outcomes.

The need for a service taught unit to be professionally relevant was identified by both nursing academics and the students as vitally important. The data indicated that both groups felt that the content taught within the service unit must reflect the rest of the course, both in depth and focus. Nursing academics expressed concern that they had "lost control" of what was being taught within the service unit. It was felt that there was a risk that content of the unit may not reflect the intent and focus of the curriculum if the unit was funded and staffed by a school other than Nursing. This is particularly important if the unit forms part of a Common First Year as these concerns may equally apply to other professionally based courses .

The collection of data relating to student perceptions and opinion of teaching practice can a sensitive issue. Much of the data personal and must be held in the strictest confidence, while being available to the individual so they can evaluate their own teaching style. Academic accountability for acting on the information which impacts on the course content and student learning, rests with the Curriculum Committee of the school in which the student is enrolled. Protocols for the distribution of this type of information must be established in the planning phase of an evaluation project.

Student learning outcomes

The students reported a perception that the lecturers who did not have the appropriate professional base failed to provide the same level of commitment, support and respect that was provided by nursing lecturers. Some students felt that this restricted their learning and in some cases the student did not attend lectures or tutorials because they felt "put down" by the lecturer. Service units which are usually taught in the early semesters need to be designed carefully to meet these professional needs.

Conclusion

Curriculum design should be founded upon sound educational practices. There is no place for a curriculum which is purely economically driven. Any curriculum with service taught units must have well established protocols which will ensure quality teaching staff, validity of content and appropriate teaching strategies.

Reference list

Johnson, T. (1997). The 1996 Course Experience Questionnaire: A report prepared for the Graduate Careers Council of Australia. Australia: Graduate Careers Council of Australia.

McInnis C. & James, R. (1995). First year on campus. Committee for the Advancement of University Education