Evaluating outcomes in physiotherapy students

Evaluating outcomes in physiotherapy students focuses on their competency to enter the profession on graduation. Evaluation is equal to the whole constellation of values for any program, including the outcome of its graduates, divided by the complexities in criteria and expectations of the stakeholders. That is, an observed value is compared with some standard.

In terms of function, standards may provide a benchmark or basic goal for training providers, industry and training bodies, or educators, as well as state and federal authorities. They may be applied in considering registration, certification, accreditation and other forms of recognition of skills, or for setting curricula and in assessment of various kinds. Standards can be expressed in different ways. It may be suggested that consideration be given to a number of factors when setting any level. These include:

1. their eventual use;
2. the audiences;
3. operational and cultural requirements;
4. the range of workplace and occupational situations.

This paper discusses the development of competency standards for physiotherapists in Australia (Liston Bullock & Cole, 1994), and their application to evaluating outcomes in physiotherapy students: - their competence to practise.

Introduction

Standards

Standards may be deemed to be necessary by governments - for example to enable industrial groups, unions, registration boards and the like to ensure that all workers, employees, professionals, or practitioners meet a minimum standard of performance in a range of skills/abilities (Locke and Latham 1990). Educationalists and professional bodies may set standards for the education or training of personnel (Jarvis 1983). These should match the minimum standard required by a newly qualified or graduated trainee or professional. Where quality assurance is required or desired, standard setting means that there is a reference point against which to judge the quality and ensure that it is maintained.

Although most issues perceived to be quantitative have a qualitative basis, the notion of standard setting is more difficult in qualitative issues than in quantitative ones. Ambiguity may exist where competence can be expressed only in higher order cognitive skill terms. The holistic nature of performance cannot be ignored. Consideration of some means of expressing the need for performance to be based on the combination of knowledge, skills, abilities and attitudes poses a problem for those setting standards.

Glass (1978) questions the use of the term minimal competence which suggests the essential, least permissible level. This appears on a continuum from absence to something especially excellent, and is extremely difficult to determine. In fact, judges disagree when seeking to determine these levels of minimal competence, and psychologically there is no basis for the concept. Placing a focus upon desired competencies for a particular profession may, however, offer a means by which a new or existing curriculum can be evaluated, particularly if the competencies have been defined in relation to the current needs in professional practice (Myers, 1990). While educators may have specified their objectives carefully, the defining of needed competencies may present an opportunity for a more integrated approach to teaching/learning and to student assessment (May, 1977; Beenhaker, 1987; Liston, 1993).

Whilst acknowledging the arbitrary nature of standard setting, Scriven (1978) suggests that this is better than nothing at all! Glass (1978), in contrast, feels that "arbitrariness is no bogeyman", yet that "nothing is safer" (p242)! So the argument is about the absolute nature of standards. In order to make a judgement about the value of anything or of anyone's performance, then some agreed level (with safety margins) may need to be determined or set.

Outcomes evaluation

There are six Physiotherapy Schools in universities in Australia, and each of them has a unique curriculum, differing teaching strategies, differing expectations and standards. For the physiotherapy profession, there are no benchmarks, no National or State examinations to determine eligibility for registration to practise, and no professional accreditation of the six undergraduate programs. Evaluation of outcomes in each of these programs relies on each School specifying its own criteria, its own standards, and its own requirements in terms of sources of evidence for making judgements and decisions about who graduates and who does not.

The students are assessed against some predetermined and ill-defined standards by assessors who gather evidence from a range of activities; such as essays, multiple-choice and short-answer written assessments, practical tests, viva-voce examinations, clinical observations and the like; and make judgements about the levels of competency of individuals. Those assessing or making judgements need to have a means to know what the expectations are - that is, an understanding of the competencies and knowledge base being assessed. Furthermore, the assessors require the ability to make observations and judgements, keep records, obey decision-making rules, and be open to independent monitoring.

Competencies in professional education

The process of education is not demeaned through defining competencies. Indeed the process is one of the outcomes, because the fostering, nurturing and cultivating aspects of it, which were identified by John Dewey in 1916, are reflected by competent graduates.

Competencies are any attributes of an individual which contribute to performance. Surely knowledge is the foundation for competent performance. Striving for excellence is more likely to be fostered by nurturing learners and cultivating competencies than by preparing for success in norm-referenced methods of assessment. The reason for developing and reviewing course objectives and the performance criteria which are to be assessed is not to stultify educational programs. Rather, it is because of the understanding that there are continuously evolving contemporary needs which require competencies to be expressed with ever greater precision. Competencies include knowledge, thinking processes, manual skills, attitudes, motivations and beliefs. Confusion and stultification that result from arguing over semantics is unproductive.

Applying competency standards in physiotherapy education

Conclusion

In Australia, Competency Standards have recently been developed by the physiotherapy profession. These standards are meant to apply to new graduates, are open to review and change, and have been accepted by Schools, Registration Boards, the Australian Examining Council for Overseas Physiotherapists and the Australian Physiotherapy Association (Liston, Bullock & Cole, 1994). The influence of competency standards in curriculum development and assessment of students, as well as in a process of accreditation of physiotherapy education programs is worthy of informed debate prior to considering any implementation. This is especially in view of the focus of the Federal Government on auditing quality procedures in higher education, which requires educational institutions and programs to have quality assurance processes in place.

Professional accreditation - assessing whether programs produce graduates who meet entry-level competency standards - is more closely linked to eligibility for registration. It may be that the only reliable means of assessing whether all physiotherapists seeking to register initially in Australia meet the entry-level competency standards, is for them to undertake an examination after graduation.

References

Cronbach, L (1963). Course improvement through evaluation. Teachers College Record, 64, 672-683.

Glass, G (1978). Standards and criteria. Journal of Educational Measurement, 25(1), 57-65.

Glass, G (1980). Evaluation research. Annual Review of Psychology, 31, 211-228.

Jarvis, P (1983). Professional Education. London: Croom Helm Limited.

Locke, E and Latham, G (1990). A Theory of Goal Setting and Task Performance. Englewood Cliffs, NJ: Prentice Hall.

Liston, C. B. (1993). The Relevance of Competency-Based Standards to Higher Education Workbook I. Competency-Based Standards Project. Perth: Curtin University Publication.

Liston C., Bullock M. and Cole J. (1994). Australian Physiotherapy Competency Standards. Brisbane, Body Control Systems.

May, B. (1977). Evaluation in a competency-based education system. Physical Therapy, 57, 28-33.

Myers, S. (1990). Program Evaluation for Accreditation. The American Journal of Occupational Therapy, 44(9), 823-826.

Scriven, M. (1978). How to anchor standards. Journal of Educational Measurement, 15(4), 273-275.

Stake, R. E. (1967). The countenance of educational evaluation. Teachers College Record, 68, 523-540.

Stufflebeam, D. L. and Shinkfield, A. J. (1985). Systematic Evaluation. Boston: Kluwer-Nijhoff.