TL Forum 2001: Bellgard, Murphy and Smith - does OTT (online teaching tools) mean over-the-top?

Teaching and Learning Forum 2001 Home Page

[ Teaching and Learning Forum 2001 ] [ Proceedings Contents ]

Does OTT mean over-the-top?

Christopher Bellgard, Jamie Murphy and Brett Smith
Department of Information Management and Marketing
The University of Western Australia

A Virtual Tutorial of undergraduate E-commerce students tested these assertions in an experiment that compared Discussion Groups and Chat with a normal face to face tutorial. The results are based on a questionnaire that considers perceived learning outcomes, utilisation, efficiency and effectiveness. The questionnaire also examined student access to and competency with OTTs.

Introduction

As students, lecturers and administrators become increasingly familiar with online teaching tools (OTTs), there is an ever growing push to incorporate these technologies in the classroom. As with any emerging technology, there is a tendency to misuse and overestimate it in the short run (Rogers 1995; Fidler 1997). As Gold and Maitland (1999) illustrate in their comprehensive review of distance learning research, the early results are far from conclusive. These sentiments gave impetus to VT2000, an Internet education study.

This paper describes that study, its basis and early findings. The paper opens with a brief review of OTTs. The paper then describes the Virtual tutorial trial and some interesting findings.

Online teaching tools

Pospisil (2000) categorises OTTs into four overlapping groups:

Computer mediated communication (CMC)
Accessing web based information
Interacting with web based media and virtual environments
Online authoring activities.

This paper reviews tools that most academics are - or should be - familiar with, namely email, bulletin boards, IRC and web pages. Most of these tools, classified as CMC, are explained below.

Email and mailing lists

Email, a private Internet memo, is one of the fastest non-physical ways to communicate. It can contain text and files, called attachments. Its digital format permits filtering, sorting and other database applications. Email is simple to send, forward, answer and broadcast. Multiple recipients may be named, suppressed in a "blind carbon copy" or in a mailing list.

Bulletin boards

A bulletin board is an ongoing meeting and announcement forum, accessed through email (though not necessarily through the user's personal email account) or web browsers. Ongoing discussions on the same topic are known as threads.

Bulletin board access is often restricted, such as to students in a course. Once logged in, users may continue discussion threads, upload and download files and post announcements. Any authorised user may submit or read any message. That is, bulletin boards operate on a "pull" basis, as opposed to the "push" basis of a mailing list (Crone 2000).

IRC or Chat

IRC (Internet relay chat) is a live multi-user discussion facility. By installing IRC software and logging onto an IRC channel (or chat room), users can chat via real time text. With the anonymous nature of chat, users often adopt an alias.

IRC channels exist for many topics but remain closed when not in use. Anyone can create a public channel and everything one types is visible to all users in that channel. Web sites such as Yahoo! Clubs (clubs.yahoo.com) or e-groups (www.egroups.com) permit users to create private channels.

Web pages

Course web pages disseminate information such as readings, marks, syllabi, assignments and marks to current and prospective students. They also house OTTs, thereby creating a virtual environment. Software platforms like ThinkWell, WebCT and Blackboard illustrate point and click templates that theoretically let educators easily build an online course environment. In addition to web pages, these environments could include chat, video, audio, bulletin boards and mailing lists as well as integration with administrative records, financial records and school libraries.

Communication concepts

Ineffective OTTs?

A survey of American accounting chairpersons found teaching solely over the Web - e-courses - less valuable to students than traditional courses. E-courses lack student-student and student-instructor interaction. Two-thirds of the respondents considered e-courses as simply correspondence (or distance) courses presented with new technology (Saunders and Weible 1999).

Interactive distance learning tools may be more appropriate for graduate courses where students are off campus and have a vested interest in continuing the program. Undergraduate courses require considerable adaptation and instructor training to achieve effectiveness similar to traditional instruction. That is, reliance on OTTs can decrease the quality of instruction for undergraduates (Clow 1999).

Alternatively, online anonymity provides teaching and learning benefits such as increased equity (McComb 1994; Collins and Berge 1995; Ruberg and Taylor 1995) and higher participation rates (Hartman et al. 1995). Here, email is particularly relevant as it allows shy students to communicate with their lecturers and tutors without being seen. Staff can then give a personal or general reply at their earliest convenience.

VT2000 described

Complementing the versatility of email for student-teacher interaction out of class, VT2000 explores OTTs in the classroom. It compares a traditional tutorial with IRC and bulletin boards.

Sample

The sample comprised 240 second and third year E-Commerce students in 14 tutorial groups divided among four tutors. Each group practised IRC and bulletin boards during a laboratory tutorial before being randomly assigned to a traditional, IRC or bulletin board tutorial.

Assessment

Students completed a voluntary exit survey. Keeping with the thrust of the course, the survey was online. It asked questions on demographics, participation, interaction, motivation and perception (see Appendix 1).

To assess students' learning outcomes, the survey included a quick quiz of material relevant to that tutorial. Table 1 shows no significant difference in mean quiz results for each tutorial type.

Table 1: Quiz results

Count Mean Max Min StDev

Bulletin Board 65 61% 92% 8% 15%

IRC 58 59% 75% 8% 13%

Traditional 41 60% 100% 25% 16%

Overall 164 61% 92% 8% 14%

Qualitative findings

Qualitative responses suggest that OTTs supplement, but do not substitute for face to face undergraduate instruction.

"Although I liked this style of tutorial I don't think it is necessarily preferable. I think a combination of online and traditional tutorials is preferable."

Bulletin boards seemed successful. They required little tutor intervention and in one bulletin board, a particularly shy student took advantage of the anonymity by initiating several threads. Other students posted replies later in the week, with discussion ceasing naturally by Thursday. Some students who participate actively in tutorials did not contribute at all. Bulletin board students overwhelmingly:

Felt confident in expressing their opinion (Q2);
Had sufficient opportunity to contribute(Q3); and
Found the tutorial convenient (Q8)

Students agreed that traditional tutorials were the most effective (Q14). However, one student commented group, "we had more participation and opinion in the virtual tute than the entire semester combined".

The virtual tutorial's lack of physical presence benefited some students in terms of anonymity (Q2 and Q6). "Anonymity helps those people who feel discouraged to participate" and "IRC is good in a sense that anyone can have their say and not be dominated by those few people that always have their say in class".

Anonymity however, does not compensate for immediate feedback (Q4). "I think using bulletin boards has its advantages such as anonymity, however I prefer regular tutorials where you get immediate feedback on your thoughts".

Similarly, a problem with IRC was class size and threads. "The main problem was keeping track of the thread of the conversation." With around 18 participants, IRC tutorials moved too fast. "A class of 20 is too many to chat on IRC."

Despite the drawbacks, students enjoyed the novelty of their virtual tutorial. "I really enjoyed participating in the virtual tutorial and think that in future there should be more opportunities to be involved in such activities." Another student suggested, "Why not have normal (real) tutorials as well as an online forum available for more in depth out of class discussion".

Perception on quality of communication

The tutorial provides a space to discuss concepts. We asked students to rate their involvement (participation) and the quality of feedback (interactivity) from others. The six questions and their median ranks are presented in Appendix 2. Students in OTT tutorials were more confident in expressing themselves (PAR1) and felt that discussion was less dominated by individuals (PAR3 reversed).

It is clear that the class size was inappropriate for irc. Students reported dissatisfaction with their own opportunity to contribute (PAR2) and the interaction with fellow participants (INT2). This is confirmed by their response to question itself (INT3).

Overall traditional tutorials were perceived as the most interactive and bulletin boards were perceived as providing the best opportunity to participate.

Strangely, the asynchronous bulletin board was considered more interactive than the irc - but this is because there were too many people on the chat forum. Eighty percent of students say real time discussion is seen as an appropriate tutorial characteristic (Q9). However, the benefits of real time discussion are dependent on discussion size and tutorial environment.

Conclusions

These qualitative results demonstrate that students enjoyed playing with OTTs. The fun element, however, may have no bearing on learning outcomes.

For students to benefit from OTTs, they need to be trained to use them effectively. Particularly for undergraduates, bulletin boards require moderation to remove inappropriate complaints, insults or gossip. Furthermore, students must be reminded to check them on a regular basis.

For IRC to run in an orderly manner, protocols need to be established. With experience class sizes could be similar to current tutorial sizes (14-20) however, there is evidence that initial class sizes need to be smaller.

Limitations and future research

This paper reviewed OTTs and described an empirical assessment of virtual tutorial learning outcomes. It briefly examined previous studies that examined the use of these tools. Hence an opportunity exists for future research to test empirically the return on investment of various OTTs.

Available research is severely limited in its examination of student responses to, and use of, various Internet tools. The studies that do exist are predominantly descriptive research and case studies (Gilbert 1999; Hara and Kling 2000). Studies are beginning to emerge which consider the characteristics of students who succeed with online courses (Powell et al. 1990; Jones and Jo 1999). However the door remains open for future research to explore more broadly the variety of contexts in which computer mediated learning is more and less effective.

References

Alter, S. (1999). Information Systems: A Management Perspective. (3rd ed). Addison-Wesley. Reading, Massachusetts.

Chester, A. and G. Gwynne (1998). Online teaching: Encouraging collaboration through anonymity. Journal of Computer-Mediated Communication, 4(2). [verified 14 Dec 2000] http://www.ascusc.org/jcmc/vol4/issue2/chester.html

Chickering, A. W. and S. C. Ehrmann (1996). Implementing the Seven Principles: Technology as lever. AAHE Bulletin, 49(2), 2-4. [verified 14 Dec 2000] http://www.aahe.org/Bulletin/Implementing%20the%20Seven%20Principles.htm

Clow, K.E. (1999). Interactive distance learning: Impact on student course evaluations. Journal of Marketing Education, 21(2), 97-105.

Collins, M.P. and Z.L. Berge (1995). Introduction: Computer-mediated communications and the online classroom in higher education. In M. P. Collins and Z. L. Berge (Eds), Computer Mediated Communication and the Online Classroom. Hampton Press. New Jersey. 2 (Higher Education): 1-10.

Crone, W. (2000). Glossary of Internet Terms. [viewed 21 Sep 2000, verified 14 Dec 2000]. http://www.aem.umn.edu/people/students/wendy/tglossary.html

Fidler, R. (1997). Mediamorphosis: Understanding new media. Journalism and Communication for a New Century. Pine Forge Press. Boston.

Gilbert, C. (1999). Student experiences of flexible learning. 16th Annual Conference of the Australasian Society for Computers in Learning and Education (ASCILITE'99), Brisbane, Australia. 5th-8th December. [verified 14 Dec 2000] http://www.ascilite.org.au/conferences/brisbane99/papers/gilbert.pdf

Gold, L. and C. Maitland (1999). What's the difference? The Institute for Higher Education Policy. [viewed 15 Sep 2000, verified 14 Dec 2000]. http://www.ihep.com/difference.pdf

Hara, N. and R. Kling (2000). Students' distress with a Web-based distance education course. Information, Communications and Society. [in press].

Hartman, K. et al. (1995). Patterns of social interaction and learning to write: Some effects of network technologies. In M. P. Collins and Z. L. Berge (Eds), Computer Mediated Communication and the Online Classroom. Hampton Press. New Jersey. 2 (Higher Education), 47-78.

Jones, V. and J. H. Jo (1999). The evaluation of student performance and perception in Web-based instruction in regard to age and gender. 16th Annual Conference of the Australasian Society for Computers in Learning and Education (ASCILITE'99), Brisbane, Australia. 5-8 December. [verified 14 Dec 2000] http://www.ascilite.org.au/conferences/brisbane99/papers/jonesjo.pdf

McComb, M. (1994). Benefits of computer-mediated communication in college courses. Communication Education, 43(2), 159-170.

National Telecommunications and Information Administration (1999). Falling through the Net: Defining the Digital Divide. United States Department of Commerce. [verified 14 Dec 2000] http://www.ntia.doc.gov/ntiahome/fttn99/contents.html

Pospisil, R. (2000). A Guide to Online Teaching and Learning Activities. [viewed 21 Sep 2000] http://cleo.murdoch.edu.au/eddesign/resources/onlinelearning/guide/index.html

Powell, R.; C. Conway and L. Ross (1990). Effects of student predisposing characteristics on student success. Journal of Distance Education, 5(1), 20-37.

Rogers, E. (1995). Diffusion of Innovations. (4th ed). The Free Press. New York.

Ruberg, L.F. and C.D. Taylor (1995). Student responses to network resources: Formative evaluation of two classes. Annual Meeting of the American Educational Research Association, San Francisco, California.

Saunders, G. and R. Weible (1999). Electronic courses: Old wine in new bottles? Internet Research: Electronic Networking Applications and Policy, 9(5), 339-347.

Appendix 1

Questionnaire: Mean and Standard error of students' responses

Type	Mean	Std. error	Question
BltnBrd trad irc	4.02 4.29 3.73	0.14 0.08 0.17	Yes I did prepare in advance for the tutorial
BltnBrd trad irc	4.25 3.84 4.04	0.08 0.16 0.14	I felt confident in expressing my opinion
BltnBrd trad irc	4.16 3.79 3.49	0.11 0.16 0.16	I had sufficient opportunity to contribute to the discussion
BltnBrd trad irc	3.42 3.82 3.40	0.12 0.12 0.15	Feedback from others was sufficient and timely
BltnBrd trad irc	3.47 3.97 3.64	0.14 0.17 0.15	Certain individuals dominated the discussion
BltnBrd trad irc	3.74 3.87 3.87	0.12 0.15 0.15	I like the anonymity of online teaching tools
BltnBrd trad irc	3.79 3.79 3.51	0.10 0.15 0.16	I was able to interact effectively with other students
BltnBrd trad irc	4.21 3.29 4.09	0.09 0.20 0.16	This type of tutorial is convenient
BltnBrd trad irc	3.84 4.13 4.07	0.10 0.13 0.12	Real-time discussion is an appropriate tutorial characteristic
BltnBrd trad irc	3.81 3.82 3.18	0.10 0.15 0.20	The class size was appropriate for this type of tutorial
BltnBrd trad irc	3.96 4.03 3.82	0.13 0.10 0.14	I prepared for the tutorial to gain participation marks
BltnBrd trad irc	3.53 3.50 3.60	0.13 0.14 0.15	I have a responsibility to other students to be prepared
BltnBrd trad irc	3.49 3.26 3.64	0.14 0.18 0.19	I prefer tutorials like this
BltnBrd trad irc	3.53 3.61 3.53	0.12 0.15 0.19	I think this type of tutorial is effective

Appendix 2

Virtual Tutorial Feedback for Participation (PAR) and Interactivivity (INT):
Median Rank and Kuskall-Wallis Chi squared

Ranks		Type	N	Mean Rank	Chi- Square	Asymp. Sig.
PAR1	I felt confident in expressing my opinion	bltnbrd	65	88.5
		trad	41	69.8
		irc	58	84.8
		Total	164		5.04	0.081
PAR2	I had sufficient opportunity to contribute to the discussion	bltnbrd	64	95.6
		trad	41	78.6
		irc	57	67.8
		Total	162		13.03	0.001
PAR3 reversed	Certain individuals dominated the discussion	bltnbrd	65	89.8
		trad	41	68.3
		irc	58	84.4
		Total	164		5.77	0.056
INT1	Feedback from others was sufficient and timely	bltnbrd	64	77.8
		trad	41	97.0
		irc	58	76.1
		Total	163		6.63	0.036
INT2	I was able to interact effectively with other students	bltnbrd	65	86.9
		trad	41	87.2
		irc	58	74.3
		Total	164		3.23	0.199
INT3	The class size was appropriate for this type of tutorial	bltnbrd	65	89.0
		trad	41	90.1
		irc	57	68.2
		Total	163		8.62	0.013

Authors: Christopher Bellgard
cbellgar@ecel.uwa.edu.au Ph: +61 8 9380 7221 Fax: +61 8 9380 1004
Jamie Murphy and Brett Smith
Department of Information Management and Marketing
The University of Western Australia
Nedlands WA 6907

Please cite as: Bellgard, C., Murphy, J. and Smith, B. (2001). Does OTT mean over-the-top? In A. Herrmann and M. M. Kulski (Eds), Expanding Horizons in Teaching and Learning. Proceedings of the 10th Annual Teaching Learning Forum, 7-9 February 2001. Perth: Curtin University of Technology. http://lsn.curtin.edu.au/tlf/tlf2001/bellgard.html

[ Abstract for this article ] [ TL Forum 2001 Proceedings Contents ] [ All Abstracts ] [ TL Forums Index ]
HTML: Roger Atkinson, Teaching and Learning Centre, Murdoch University [rjatkinson@bigpond.com]
This URL: http://lsn.curtin.edu.au/tlf/tlf2001/bellgard.html
Last revision: 4 Feb 2002. � Curtin University of Technology
Previous URL 14 Dec 2000 to 4 Feb 2002 http://cleo.murdoch.edu.au/confs/tlf/tlf2001/bellgard.html

	Count	Mean	Max	Min	StDev
Bulletin Board	65	61%	92%	8%	15%
IRC	58	59%	75%	8%	13%
Traditional	41	60%	100%	25%	16%
Overall	164	61%	92%	8%	14%