A comparison of several survey techniques to obtain student evaluation of a unit in Land Management

Student feedback techniques appear ad hoc, to be serving purposes related to particular interests of the course coordinator. While this may be the prerogative of the course coordinator, we were aware of a diverse range of approaches used in feedback questionnaires. We have considered whether there might be more effective approaches than single item responses on individual questions that tend to dominate the more usual feedback questionnaires. This paper summarises results from two questionnaires, examining issues of custom design of questionnaires and interpreting correlations among groups of item responses that arose in undertaking several different feedback techniques on a student group. Feedback from a context-designed conjoint survey was considered the most valuable for course implementation.

Introduction

A program of comparisons between different surveys in the one unit would provide greater understanding of both the opinions of the respondents and the effectiveness of the feedback gathering instruments. Much as the old truism says: "To get the right answer, first ask the right question!" Literature resources on survey techniques are quite abundant. However, close comparison between different survey techniques are rare. It appears quite likely that the perception of course-work and content implied in a questionnaire drive the development of the respondents' opinions. In other words, our specific questions may be leading our witnesses.

As part of a program of renovation of laboratory education within the school of BES, an initiative was planned involving the identification of effective and valid survey instruments. This was carried out through a program of several questionnaires. The outcomes of this exercise were framed in terms of trying to identify design features of effective surveys and selection of an optimum approach to improvement through these. However, to make some simple recommendation identifying a single best approach is beyond reasonable expectations: not only does the intended use of survey instruments vary, but the interpretive needs of those who use them also varies. The most clearly distinguished cases of the latter are those situations where a survey is required for needs relating to staff promotion as distinct from student learning. While some may argue that these should be the same, there are many cases where two separate evaluation purposes are accepted within a profession. In our current context, this paper will highlight features of interest in terms of validly tapping the students' needs.

Questionnaires

The full questionnaire, some 58 separate items, was given to the entire land management class during a normal lecture period. A second group of 24 questions that specifically related to laboratory teaching was undertaken by a group of tutorial students on a separate occasion. Some 11 questions were common to both surveys. The intention of this exercise was firstly to examine the repeatability of the questionnaire in terms of the responses on the 11 common questions, secondly to examine (or demonstrate) the continual problem that a structured questionnaire might overlook further issue(s), and thirdly, to examine the questionnaire in the fundamental role of diagnosis of unit defects. The problem in scope mentioned as the second interest is a hindrance moving from the interpretation of the results of a questionnaire study to the decision to invest resources into dealing with the issues identified.

The repeatability of the questionaire was examined by comparing the distribution of responses on the 11 replicated questions using the Kolmogoronov-Smirnov two-sample tests, and comparing the percentage distributions (Table 1). The results found that cumulative distributions generally did not differ between the two groups, but that the interpretative benchmark of 30% respondents disagreeing did vary frequently.

Spearman rank correlation analysis identified many responses in the Murdoch questionnaire that were clearly linked issues in the respondents perceptions (data not shown). These indicated some clear behavioural traits and expectations of the students, and provided considerable guidance to unit improvement. An outline of some of these follows.

If the students were in agreement that the aims of the unit were clearly stated at the beginning (Qn 1), they also tended to consider that these aims were achieved (Qn 2). In the context of this unit, this may simply identify those students who have effectively read the introduction of the Study Guide.

Quality is a diffuse, but functional, term. When the students tended to feel that the unit was well organised (Qn 3), that the unit's teaching sessions were appropriate to the unit (Qn 4), or that the unit did not attempt to cover too much material (Qn 7), they also reported being satisfied with the quality of the unit (Qn 25). In fact, responses on Qn 25 on the quality of the unit was positively and significantly correlated with 22 of the 58 questions used overall, including all of the first 11 questions, and the two questions examining the quality of the lectures (Qn 46) and the quality of the laboratories (Qn 58). In a similar fashion, Qn 46 was correlated with 33 other questions, including 16 of the 21 examining aspects of lecturing. Question 58 was correlated with 16 questions, including 9 of the last 12 which were on the laboratory sessions. The interpretations made on the basis of questions using the term quality may not be very specific in guiding improvements.

Several cases of negative correlations between responses on different questions were observed and deserve note. The existence of these cases suggests that the optimisation process may never be perfect, but can only satisfy a majority of individuals needs. In the ASU survey, there were seven cases of significant (P<0.05) negative correlations. The workload being reasonable (Qn 6) was negatively associated with the lecturer indicating enough suitable references (Qn 35). The lectures aiding understanding (Qn 8) was negatively associated with the lecturer encouraging student questions (Qn 33). The staff being accessible outside of classes (Qn 22) was negatively associated with lab classes having a satisfactory staff:student ratio (Qn 54). The lecturer being well organised (Qn 26) was negatively associated with enough examples being given during lectures (Qn 44). The study guide setting out the unit requirements clearly was negatively associated with the demonstrator encouraging the student to think (Qn 48). The response that the lecturer writes clearly (Qn 29) was negatively associated with the demonstrator encouraging the student to think (Qn 48). The response that the lecturer presents difficult concepts clearly (Qn 31) was negatively correlated with the being satisfied with the quality of the laboratory sessions(Qn 58).

Further negative correlations were observed when the expanded laboratory questionnaire was used. There were five statistically significant (P<0.05) examples of these, as follows: A response that the demonstrator was interested in students (Qn 4) was negatively correlated with both responses on there being a satisfactory staff:student ratio (Qn 9) and there being clear links with theory (Qn 22). A response that there was adequate equipment (Qn11) was negatively correlated with a response that the aims of the laboratories were clearly stated (Qn 21). The latter response was also negatively correlated with responses on the presence of enough space in the laboratories (Qn 15). The responses on space in the laboratories (Qn 15) were negatively correlated with responses on there being an opportunity to discuss the practical work in class time (Qn 23). The last three of these correlations suggest that the learning in laboratories is being influenced by queueing. We suggest that where students were obliged to queue for equipment, they will tend to read the study guide further and discuss the laboratory exercises with colleagues and demonstrators.

Conjoint Analysis
Conjoint analysis seeks to obtain an insight into the relative importance of different aspects of a service or product that might be modified in an effort to improve customer acceptance or 'utility'. It does this by asking the respondent to choose between different combinations of the differing aspects or 'attributes' (Green and Wind, 1975). While this technique has only been used occasionally on education services (Walker, Winzar and Johnson, 1991; Carter, 1993), it has been widely used in the commercial services field (Greene and Nkong, 1989).

In the current study, as in Carter (1993), the relevance (or Salience in the terminology of Carter) of the analysis to the individual students was enhanced by focussing a discussion with the tutorial group on those aspects of a unit that they most liked. In the context of the land management unit, this basically identified five main areas of concern (Content Workload, Flexibility, Student Organisation, Assessment, and Feedback). A set of different levels of unit attributes in these areas was devised (Table 2). A set of hypothetical units with a wide range of combinations of each of the attributes was then devised from these (not shown). At a subsequent tutorial, the students were presented with a set of the hypothetical units, with each unit outlined just in terms of the five attributes, and each printed on an individual slip of paper. The students were each asked to consider these as possible formats of the land management unit, to assemble the slips in their order of preference, and to rate each of them.

The results of this study are summarised in Table 3, wherein the term 'Importance' indicates the overall effect of the utility of an attribute on the model, relative to that observed over all attributes. The overall Utility (U) of a hypothetical unit may be estimated using a model as follows:

The results suggest that flexibility in the unit delivery was more important than its workload (Table 3). This is an unexpected outcome in that workload was often the major focus of discussions on the unit. The options that were responsible for the improvement in utility due to course flexibility appear to be the prospect of lecture repeats and videotaping, and frequent laboratory timeslots. This appears to be more suitable in modern student lifestyles.

Respondents attached similar levels of importance to feedback and workload. Feedback was also raised as an issue in other surveys. While feedback is not assessed as a time-related issue (as here) in the Murdoch questionnaire, there are clear indications that the nature of comments on the returned assignments is not sufficiently helpful. The form of assessment appears to be a non-issue, however the respondents did indicate some loss of utility if a unit demanded that they be involved in organised group work.

Table 3: Conjoint Analysis of the matrices of relative utility respondents attached to hypothetical units drawn up from permutations of Attributes given in Table 2. (SPSS Release 4.0 for Macintosh)

Importance*		Utility (s.e.)		Factor Level
		WORKLOAD
22.16		1.7563 ( .4922) 3.5127 ( .9843) 5.2690 (1.4765)		1.00 2.00 3.00
	B1 =	1.7563 ( .4922)
		FLEXIBILITY
33.63		2.6650 ( .5073) 5.3300 (1.0146) 7.9950 (1.5219)		1.00 2.00 3.00
	B2 =	2.6650 ( .5073)
		STUDENT ORGANSTN
17.10		1.3548 ( .4721) 2.7096 ( .9443) 4.0644 (1.4164)		1.00 2.00 3.00
	B3 =	1.3548 ( .4721)
		ASSESSMENT
5.21		-.4127 ( .4552) -.8255 ( .9104) -1.238 (1.3656)		1.00 2.00 3.00
	B4 =	-.4127 ( .4552)
		FEEDBACK
21.90		1.7358 ( .5004) 3.4717 (1.0007) 5.2075 (1.5011)		1.00 2.00 3.00
	B5 =	1.7358 ( .5004)
		CONSTANT
		-5.6906(2.4553)
Pearson's R = .974 Kendall's tau = .833			Significance = .0000 Significance = .0009
* Note that Importance sums to 100.0 over all attributes. It indicates a fraction of the dependent variation in the model that the attribute is contributing.

Discussion

The conjoint analysis feedback is strongly 'customised' through a sequence of two interaction sessions with the study group. The initial session obtained their opinions as to what would improve the unit for them as individuals. This exercise generated a set of attributes for which a series of steps or service levels were devised by the investigator. These steps were assembled into different unit formats, trading off a more attractive level in one attribute with a worse level in another. In the second session, the respondents examined a set of hypothetical units and decided which would be most preferable to them. The hypothetical units were to be in Land Management, and it would not be appropriate to interpretate the data as applying to any University unit.

To illustrate this process, the most attractive of all the units offered was one set taking in levels 2 of Workload, 3 of Flexibility, 2 of Student Organisation, 1 of Assessment, and 3 of Feedback (Table 2). It was the top choice of over 50% of respondents, and to summarise the levels from Table 2, it had the following overall set of attributes:

Unit R Moderate theory workload with around three hours of reading from two prescribed texts each week, with several new concepts involved each week. A repeat of all lectures in a convenient evening time slot, with video tapes of lectures and copies of overheads available. Labs and tutorials available both in mornings and afternoons, with five scheduled times. Laboratory projects prepared in varying ways, some by groups and some individually, always due within seven days of the lab. Assessment during semester requires extensive reasoning and deduction, in-depth knowledge is expected in substantial (> 1500 words/four pages) assignments. All assignments (50% of total grade) marked within 2 weeks of submission and returned by week 11, usually with over 20 words of comments.

Inspection of these attributes recognises a service level that is a step above that currently enjoyed by Land Management students, and it must be admitted that options such as morning laboratories, evening lectures, and all assignments returned by week 11 might be beyond available human resources. In our own response to this, we have increased flexibility in another (less expensive) way, in allowing internal students to undertake the laboratories in the laboratory week that external students undertake on-campus. Video-taped material related to the unit has been expanded.

One major strength of conjoint analysis lies in the importance it attachs to different attributes, where importance directly reflects the relative utility that each contributes in the students preferences. We know that a dollar invested in the flexibility area is worth more than one invested in other areas, and how much more in proportion. A second strength lies in the prescriptive nature of the set of attribute steps generated by the initial session's feedback. This allows the investigator to be more certain about the nature of changes mandated by the second session's feedback.

The major defect of conjoint analysis is that it may not allow access to a widely applied mechanistic model of the education service. Such a model would be an ideal tool if it was universally applicable to all units, and responses to questions in a standard questionnaire could be accumulated to give scores for appropriate components of the model. This would also serve needs in staff development and promotion, and we suggest that diverse feedback methods will remain for such varied agendas.

References

Greene, C. S. and Nkonge, J. (1989). Gaining a competitive edge through Conjoint Analysis. Business, 39(Apr-Jun), 14-18.

Siegel, S. and Castellan, N. J. Jr. (1988). Nonparametric statistics for the behavioural sciences. 2 ed. McGraw Hill, NY.

Walker, M., Winzar, H. and Johnson, L. (1991). Modelling perceptions of business degrees. Working Paper Ser. No. 20, Key Centre Strategic Mgt., Queensland University of Technology, Brisbane.