Teaching and Learning Forum 2000 [ Proceedings Contents ]

Assessing the efficacy of online teaching with the Constructivist On-Line Learning Environment Survey

Peter Taylor and Dorit Maor
Science and Mathematics Education Centre
Curtin University of Technology
    Teachers of science, mathematics and technology Australia wide are being required to transform their established epistemologies of practice in order to engage learners as active conceptualisers within socially interactive learning environments. Many teachers are enrolling in postgraduate distance education programs to assist them with this challenging task. Curtin University is responding by using computer mediated communication to speed up the exchange of distance study materials and to provide online interactive learning environments (via chat groups, bulletin boards and email). For the past three years, we have been using the Internet to teach online Masters degree students studying at a distance from Curtin. Our major pedagogical goal is to engage our students (professional teachers) in reflective and collaborative learning. Our web sites provide Activity Rooms in which we engage students in frequent and focused discourse with each other and with their online tutors. As reflective teachers, we are keen to evaluate our own innovative practices and constantly improve them. To this end, we have designed the Constructivist On-Line Learning Environment Survey (COLLES), an electronic questionnaire that enables us to readily monitor each student's preferred online learning environment and compare it with her/his actual experiences. In this presentation, we outline the rationale of the questionnaire and present some preliminary analyses that illustrate its usefulness. The COLLES is the subject of ongoing research and development funded by a 2000 ARC Small Research Grants Scheme award.
Teaching and Learning Forum 2000 Home Page

Background

In this paper we discuss the development of a questionnaire - the Constructivist On-Line Learning Environment Survey (COLLES) - that we have designed to support the use of the World Wide Web (the Web) for teaching in higher education, especially for postgraduate professional development programs for which social constructivism is a key pedagogical referent. The efficacy of innovative web based teaching for engaging distance learners in enriching their ways of knowing (i.e., epistemological growth) cannot be evaluated adequately without obtaining a measure of learners' perceptions of their online classroom environment. When fully developed, the COLLES will be available for use online by university teachers, teacher-researchers, and researchers who are interested in the educational role of the Web for promoting much needed epistemological reform of university distance teaching.

Our discussion culminates in an illustration of how the COLLES can be used to monitor the quality of innovative online teaching and learning. The context is the first author's online Masters unit: 'SMEC625: Multimedia in Science and Mathematics education'.

A social constructivist perspective on learning

In Australia, the newly articulated national curriculum (and its various State/Territory derivatives) requires teachers of school science, mathematics and technology to engage their students in conceptually and linguistically rich educative relationships with both the teacher and peers. Underpinning this new way of teaching is a new epistemology (i.e., theory of knowing): social constructivism, which portrays the learner as an active conceptualiser within a socially interactive learning environment. Social constructivism describes an epistemology, or way of knowing, in which learners collaborate reflectively to co-construct new understandings, especially in the context of mutual inquiry grounded in their personal experience (O'Connor, 1998). Central to this collaboration is the development of communicative competence that enables students to engage in open and critical discourse with both the teacher and peers. This discourse is characterised by an empathic orientation to constructing reciprocal understanding and a critical attitude towards examining underlying assumptions (Dawson & Taylor, 1998; Maor & Taylor, 1995; Sfard, 1998; Tobin, in press).

Given the rising popularity of social constructivism as a referent for science, mathematics and technology teachers' own classroom teaching, we believe that it is very important for these teachers (i.e., our own students) to experience firsthand what it means to become reflective and collaborative learners in a social constructivist learning environment (see also Brown, 1997, Salomon, 1996).

The significance of Web based university distance teaching and learning from a social constructivist perspective is discussed in extant research literature (Blanton, Moorman & Trathern, 1998; Jonassen & Reeves, 1996; Owston, 1997).

Web based teaching and research

The new technology of the Web offers unique prospects for promoting a reflective and collaborative learning environment (Bates, 1997; Duffy & Cunningham, 1996; Eastmond, 1995; Tobin, in press). However, it is quite conceivable that, in the rush for universities to market Web based teaching, the traditional teacher centred knowledge transmission metaphor (efficient delivery to individual passive absorbers) will prevail, even though the new technology offers unique prospects for promoting reflective and collaborative learning. Thus, we have designed our Web based programs in order to achieve this goal.

In electronic text based communities of learners, our geographically and socially isolated students have the opportunity to establish communicative relationships with each other and to reflectively co-construct their knowledge by engaging in open and critical discourse (Maor, 1998, 1999; Taylor et al., 1999). In science, mathematics and technology teacher education, the Web has begun to be utilised in this way but, as yet, very little research has been published (Gilmer, in press, Tobin, in press) on the efficacy of the Web for promoting epistemological growth amongst teachers of science, mathematics and technology. Hence, our project is situated within a newly developing international research program which is investigating the development of epistemologies of practice for Web based teaching.

Our project aims to provide a means for other teacher educators to participate in this research program which, to date, has been driven by the laborious and time consuming process of interpretive research. Of particular relevance to our project is the recent call for a survey instrument for evaluating the quality of Web based learning environments (Chiew & Tobin, in press), specifically an instrument, such as the COLLES, for generating a profile of students' perceptions of the extent to which the virtual classroom environment is fostering their learning (Burge, 1994).

Structure of the COLLES

Science education researchers have led the world in the field of classroom environment research over the last two decades (see Fraser, 1998). Considerable interest has been shown internationally in the conceptualisation, assessment, and investigation of students' perceptions of psychosocial characteristics of the learning environment of classrooms at the primary, secondary and higher education levels. At present, however, with an increasing number of distance education students, there is a dearth of such questionnaires designed specifically for online classroom environments in higher education.

The COLLES comprises scales new to learning environment research. The scales were developed from the theory of social constructivism (including social constructionism, critical constructivism, co-participation, and socially situated cognition) which is guiding leading edge research on the role of students' predispositons in shaping the quality of their discourse in Web based teaching and learning. The scales are concerned with students' perceptions of the existence of a virtual classroom environment that supports them to reconstruct themselves as both reflective and collaborative learners. We designed the COLLES to measure students' (and tutors') perceptions of:

A second form of the COLLES contains parallel items designed to obtain measures of students' (and tutors') expressed preferences for the above aspects of the online learning environment. Thus, the person-environment fit can be gauged by comparing actual and preferred results. By administering both forms of the COLLES at the end of a period of innovative teaching, a useful picture can be generated of the extent to which the innovation has been experienced by students as favourable. But this is only the beginning of an insightful and complex inquiry, as our subsequent discussion reveals.

Preliminary results

The following graph (Fig 1) displays the results of the COLLES for a group of students studying a Masters level unit (SMEC625: Multimedia in Science and Mathematics Education) during second semester this year. The graph displays the class mean scores on six scales for both the preferred and actual forms of the COLLES which were administered towards the end of the 15 week teaching semester.

Figure 1

Figure 1: MSc students' perceptions of their preferred and actual online learning environments (N=10)

In the following preliminary analysis we describe (visual) patterns evident in the graphical display of the data, and we illustrate how challenging questions can be generated to guide subsequent interpretive inquiry and educational decision making.

Close to an optimum learning environment?

There is a general perception amongst the class that there exist optimum degrees of reflective thinking and affective support. Students prefer to be engaged often (mean = 4.0) in thinking critically about their own ideas and other students' ideas, and about how they are learning. Students expect their tutors almost always (mean = 4.6) to encourage, praise and value their online participation and to be empathic and responsive to them. These results might be reassuring, but they also prompted us to reflect critically on our pedagogical framework. Do we want to engage students more frequently in reflective thinking? If so, how do we persuade them to adopt this goal and thus reconstruct their expectations? And do we want students to lower their expectations for affective support? Might the current high expectations result from uncertainty with this new distance learning medium? How much direct support is pedagogically defensible in a more student centred learning environment?

High expectations close to being fulfilled

Students generally have indicated very high expectations for relevance and interpretation. They expect their online learning almost always (mean = 4.8) to be interesting and directly related to their professional practice; and they perceive that this occurs very often (mean = 4.3). Likewise, their expectations that they and their fellow students and tutors very often (mean = 4.4) make good sense of the messages posted in the electronic Activity Room are very close to being realised in practice (mean = 4.2). This suggests that online (asynchronous) communication is very comprehensible and meaningful.

Again these results are reassuring, in the sense that students seem to be satisfied. But are we as unit designers also satisfied? Might we want to include learning activities that are more challenging of students' extant epistemologies, activities that might not be perceived as being directly relevant to what goes on in teachers' daily lives but which, in fact, might further enrich their professional lives? How far can we take our students beyond their zones of comfort and familiarity?

Lesser expectations being met

In general, students' preference for the online tutor to quite frequently (mean = 3.8) provide cognitive support are close to being met in practice (mean = 3.6). Although they seem to value a role for the tutor in challenging their assumptions, stimulating their thinking and modelling good discourse and reflective thinking, they don't want this to occur all the time. Perhaps this indicates a valuing of opportunities for students to interact in a relatively unmediated way with their fellow students?

These results are interesting inasmuch as they point out that more is not necessarily better. In a student centred learning environment we would not want the teacher to maintain a constant active presence, but to move judiciously in and out of sight, at times offering guidance and suggestions and at other times leaving space for students to seize the learning agenda and control the pace and content of their own learning. Do we want, therefore, to lower students' expectations on this scale, rather than provide more cognitive support?

The anomaly of interactivity!

The constructs underpinning the Interactivity scale lie at the pedagogical heart of the design of our online environment and yet the results are seemingly anomalous. Whereas we might expect that students would value highly the opportunity to interact often with fellow students, a general preference has been indicated for this to occur less than often (mean = 3.8). Given that students were engaged online in structured small group interactions, with topics changing every two weeks along with the role of group leader, it is somewhat surprising that the class perceived that only sometimes (mean = 3.1) did they have the opportunity to engage in an exchange of ideas with other students.

Of course, these quantitative data alone cannot shed further light on these results, so we must turn to other data sources in order to make sense of this apparent anomaly. At this early stage in the research we have not completed this interpretive inquiry which will include interviews with students, analysis of the record of online discourse, and analysis of students' final written assignments. However, we can reveal that anecdotal observations by the tutor (the second author) suggest that the online interactivity amongst students was underpinned more by a monological rather than dialogical rationality. In other words, students tended to post in the Activity Room 'mini speeches' aimed largely at informing others of their standpoint, opinion or point of view. If this assertion proves to be substantiated by our ongoing analysis, then we shall be faced with the pedagogical challenge of how to engage students in more a dialogical form of online discourse that involves (amongst other things) a willingness to learn from others, a commitment to active listening, and a style of writing that inquires into others' ideas.

Conclusion

In the (expensive) rush to embrace the new Web technology of online distance learning, we must be careful to ensure that technological determinism doesn't overshadow sound educational judgement. An important question to be addressed is 'how much should students be engaged in learning collaboratively online?' We should not be reluctant to ask ourselves whether we are in danger of over-using this new distance learning medium, of ignoring the importance of (traditional) individualised learning? What is the right balance and how do we know when it has been achieved? It is not inconceivable (although it might seem heretical) that, as providers of online distance education programs, we might need to lower our expectations of the frequency of collaborative learning offered by the new medium. We need more research on the quality of learning being realised via online support in order to adjust our expectations in educationally defensible ways.

Research tools such as the COLLES can help us to investigate the quality of online learning environments, but they do not come with gold standard norms. Higher expectations are not necessarily better. Results must be interpreted in relation to other data, and in relation to sound educational criteria. It is early days for online distance learning, and comparative analyses with more traditional forms of individualised distance learning are needed before we can be satisfied with our expectations. The COLLES is likely to be a very useful tool in this process. Please contact the authors if you would like to participate in our trials during 2000.

References

Bates, A.W. (1997). The impact of technological change on open and distance learning. Distance Education, 18(1), 93-109.

Blanton, W.E., Moorman, G., & Trathern, W. (1998), Telecommunications and teacher education: A social constructivist review. Review of Educational Research, 23, 235-275.

Brown, A. (1997). Transforming schools into communities of thinking and learning about serious matters. American Psychologist, 52(9), 399-413.

Burge, E. (1994). Learning in computer conferenced context: The learner's perspective. Journal of Distance Education, 9(1), 19-43.

Chiew, G.S., & Tobin, K. (in press). Student and teacher perspectives in computer-mediated learning environments in teacher education. Learning Environment Research: An International Journal.

Dawson, V. & Taylor, P.C. (1998). Establishing open and critical discourses in the science classroom: Reflecting on initial difficulties. Research in Science Education, 28(3), 317-336.

Duffy, T.M. & Cunningham, D.J. (1996). Constructivism: Implications for the design and delivery of instruction. In D.H. Jonassen (Ed.), Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan.

Eastmond, D.V. (1995). Alone but together: Adult distance study through computer conferencing. New Jersey: Hampton Press.

Erickson, F. (1998). Qualitative research methods for science education. In B.J. Fraser & K.G. Tobin (Eds.), International handbook of science education (pp. 1155-1174). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Fraser, B.J. (1998). Classroom environment instruments: development, validity and applications. Learning Environments Research: An International Journal, 1(1), 7-33.

Gilmer, P.J. (in press). Opalescence at the triple point: Teaching, research and service. In P.C. Taylor, P.J. Gilmer, and K.G. Tobin (Eds.), Transforming undergraduate science teaching. Dordrecht, The Netherlands: Kluwer Academic Publishers.

Guba, E.G. & Lincoln, Y.S. (1989). Fourth generation evaluation. Newbury Park, Ca: Sage Publications.

Jonassen, D.H., & Reeves, T.C. (1996). Learning with technology: Using computers as cognitive tools. In D.H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 693-720). New York, NY: Macmillan Press.

Maor, D. (in press). A professional development program to enhance teachers' understanding of the use of a constructivist multimedia learning environment. Learning Environment research: An International Journal.

Maor, D. (1999). Teacher and student reflections on interactions in an Internet based unit. In K. Martin, N. Stanley and N. Davison (Eds), Teaching in the Disciplines/ Learning in Context, 257-261. Proceedings of the 8th Annual Teaching Learning Forum, The University of Western Australia, February 1999. Perth: UWA. http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf99/km/maor.html

Maor, D. (1998). How does one evaluate students' participation and interaction in an Internet-based unit? In Black, B. and Stanley, N. (Eds), Teaching and Learning in Changing Times, 176-182. Proceedings of the 7th Annual Teaching Learning Forum, The University of Western Australia, February 1998. Perth: UWA. http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf98/maor.html

Maor, D. & Fraser, B.J. (1996). The development and use of a classroom instrument in the evaluation of inquiry-based computer-assisted learning. International Journal of Science Education, 18, 401-421.

Maor, D. & Taylor, P.C. (1995). Teacher epistemology and scientific inquiry in computerised classroom environments. Journal of Research in Science Teaching, 32, 839-854.

O'Connor, M.C. (1998). Can we trace the efficacy of social constructivism? Review of Educational Research, 23, 25-71.

Owston, R. (1997). The world wide web: A technology to enhance teaching and learning? Educational Researcher, 26, pp 27-42.

Salomon, G. (1996). Technology's promises and dangers in a psychological context: Implications for teaching and teacher education. Paper presented at The Second International Conference on Teacher Education: Stability, Evolution and Revolution. Wingate Institute, Israel.

Sfard, A. (1998). On two metaphors for learning and the dangers of choosing just one. Educational Researcher, 27(2), 4-13.

Taylor, P., Dawson, V., Geelan, D., Stapleton, A., Fox, R., Herrmann, A. and Parker, L. (1999). Virtual teaching or virtually teaching? Does Internet-based teaching require multiple metaphors of mind? In K. Martin, N. Stanley and N. Davison (Eds), Teaching in the Disciplines/ Learning in Context, 429-432. Proceedings of the 8th Annual Teaching Learning Forum, The University of Western Australia, February 1999. Perth: UWA. http://cleo.murdoch.edu.au/asu/pubs/tlf/tlf99/tz/taylor-p.html

Taylor, P.C., Dawson, V., & Fraser, B.J. (1995, Apr). Classroom learning environments under transformation: A constructivist perspective. Paper presented at annual meeting of American Educational Research Association, San Francisco.

Taylor P.C., Fraser, B. & Fisher, D. (1997). Monitoring constructivist classroom learning environments. International Journal of Educational Research, 27(4), 293-302.

Taylor, P.C., Fraser, B.J. & White, L.R. (1994, April). The revised CLES: A questionnaire for educators interested in the constructivist reform of school science and mathematics. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.

Tobin, K.G. (in press). Learning to teach science using the internet to connect communities of learners. In P.C. Taylor, P.J. Gilmer, and K.G. Tobin (Eds.), Transforming undergraduate science teaching. Dordrecht, The Netherlands: Kluwer AcademicPublishers.

Please cite as: Taylor, P. and Maor, D. (2000). Assessing the efficacy of online teaching with the Constructivist On-Line Learning Environment Survey. In A. Herrmann and M.M. Kulski (Eds), Flexible Futures in Tertiary Teaching. Proceedings of the 9th Annual Teaching Learning Forum, 2-4 February 2000. Perth: Curtin University of Technology. http://lsn.curtin.edu.au/tlf/tlf2000/taylor.html

[ TL Forum 2000 Proceedings Contents ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf2000/taylor.html
Last revision: 20 Feb 2002. © Curtin University of Technology
Previous URL 28 Dec 1999 to 20 Feb 2002 http://cleo.murdoch.edu.au/confs/tlf/tlf2000/taylor.html