Teaching and Learning Forum 2011 Home Page

Category: Research

Teaching and Learning Forum 2011 [ Refereed papers ]
Improving marking of live performances involving multiple markers assessing different aspects

Julia Wren, Alistair Campbell, John Heyworth and Rachel Bartlett
Edith Cowan University
j.wren@ecu.edu.au, a.campbell@ecu.edu.au, j.heyworth@ecu.edu.au, r.bartlett@ecu.edu.au

Assessment in the arts can be challenging. The problems associated with assessing student performances are that judgements are complex and even more so when they involve multiple markers (Campbell, 2005). It can be difficult to ensure assessment is fair, valid and reliable and that students are provided with clear feedback that is meaningful to them. This paper describes the development and trialling of an innovative, technology supported tool, designed to improve the confidence, efficiency and effectiveness of student performance-based assessment in arts education.

This project is a work in progress and early trials of the Internet based digital computer technology tool suggest that it reduces tutor anxiety, increases markers' confidence, improves the management of assessment and record keeping, facilitates easy and quick moderation, and provides explicit feedback to students to enhance ongoing learning. The perception by stakeholders is that this assessment is far more accessible and easy to use than previous methods. The project was implemented in a Bachelor of Education course, with 170 third year pre-service teacher students. Group performances were assessed by three tutors simultaneously during the live performance and for one week afterwards by viewing online video recordings of each performance. The tutors were able to discuss the assessment online via a confidential 'tutor box' attached to the marking key and only visible to tutors. The assessment was presented to the students as a one page electronic marking key with the video of their performance embedded into this page. This was saved as a PDF document and emailed to students. The students' responses have been overwhelmingly positive. They reported that ease of access to this electronic assessment meant that they engaged with their feedback multiple times. The tutors reported that the process was far more streamlined and fair.

Introduction

Our pre-service teachers are training to be generalist teachers of young children. Among other things, they need to have a good level of understanding of learning area content as well a range of skills. At the start of the arts education units, many pre-service teachers express concerns about their own understandings of the arts, which some feel at best are limited. So for many, teaching the arts to children seems somewhat daunting and a challenge. The aim of the arts units is therefore to enable pre-service teachers to construct an understanding of arts education pedagogy and to help them learn some arts skills. We provide a number of learning opportunities, one of which is to use assessment for learning. To do this, we developed an Internet based digital computer technology tool which was used to assess their arts understandings and skills. The tools consisted of the following: Research suggests that the use of video as an assessment tool benefits learning (Brunvand, 2010; Ladson-Billings, 1998; Miels, 1999; Rich & Hannafin, 2009; Romano & Schwartz, 2005; Song & Catapano, 2008 ) by enabling students to 'visualise' themselves. Visualising self encourages positive changes to learning and promotes analytical and higher order thinking, self efficacy and powerful insight (Ladson-Billings). It provides learners with opportunities to construct their learning (Brunvand). Constructivist approaches to learning underpins our teacher education course at this university.

Use of online e-marking has been extensively researched by Campbell (2005) who found that the tool enables a paperless assessment that increased marker productivity, improved moderation practices as well as streamlined evaluating and analysis of overall results. The tool used in this study was modelled on earlier e-marking tools developed by Campbell.

Marking and moderation practices among multiple tutors which result in fair assessment for students are difficult to implement in the arts (Campbell, 2005) largely because of the ephemeral nature of live performances. Accuracy in assessment of performances where multiple markers are involved with large numbers of students and relatively short time frames is an issue.

When using the Internet based digital computer technology tool to assessing live performances, tutors in this study found that what is usually a difficult and time consuming assessment process was streamlined, and improved the 'on the day' assessment processes and moderation practices among tutors, which resulted in assessment that was more valid. It took away the need to make instant decisions and enabled the tutors to mark away from classroom distractions (Brunvand, 2010). In addition, the use of technology enabled the quality of feedback given to students to be far more comprehensive and educative. McConnell, Lundeberg, Koeler, Urban-Lurain & Zhang (2008) show that when teachers review videos as a part of their assessment, they are more likely to base their judgements on evidence rather than memory. When an immediate response is not necessary, the teacher then has the luxury of being able to engage with a deeper reflection and analysis (Romano & Schwartz, 2005).

Aims of the project

The aims of this project are to research and establish more efficient methods of assessment of performance- based learning particularly where multiple markers are involved with large numbers of students. Internet based digital technology is used as a tool to streamline methods of assessment, improve the communication between markers and enable reflective moderations processes. It allows for discussion between markers at times that are convenient for each individual.

In addition, the use of electronic marking tools with video clips of students' performances attached to them will provide explicit, educative and comprehensive feedback to the students on the quality of their learning. Providing quality feedback to students about their learning is a significant factor in enhancing learning outcomes (Dinham, 2010). Feedback given to students in this project is based on authentic assessment, which relates to their future work as teachers.

Research questions

  1. Assessing live performances. Can the assessment and moderation process of student performances be streamlined and made more efficient by collating each assessor's marks in the case when multiple assessors mark simultaneously while allowing them to instantly see each other's marks and comments?
  2. Moderation of assessment. Can moderation processes be conducted via the web so that markers do not need to have face to face meetings, but rather review and modify their marking by communicating with each other via the web enabling access at times suitable to each marker?
  3. Feedback to students. Can the video recordings be attached to each marking key so that students receive a PDF copy of their performance alongside the tutor's marks and comments so that the feedback given is explicit, comprehensive and aids learning?

Description of project

This project came about as a response to a continual drive to improve the assessment of students in the Bachelor of Education course where, in third year, the students undertake two units of study in arts education with an emphasis on visual art, music and drama education. The first semester's learning culminated in group performances where the students brought together their learning from these three areas of the arts. Each group had approximately 6 students and there were 28 groups. Performances were scheduled over one whole day in the last week. Each group had five minutes to perform, with an additional few minutes at the end to receive verbal feedback from their audiences, which comprised largely of their peers but also of others (e.g. children). Each performance was videotaped.

Methodology (marking)

Each of the three tutors assessed the live performances on an electronic marking key, which was based on criteria in creativity skills, arts skills and group work/collaboration skills from one arts area only (i.e. music or drama or visual art). Using wireless access, each tutors had instant access to view the others' results and comments. Provision was made via a "chat box" for tutors to insert questions, thoughts or comments to each other, which were not available for students to see. It removed 'unproductive, busy work of marking, such as adding up of marks, recording and spell checking' (Campbell, 2005 p.529) therefore allowing more time to be given to comments and moderation. This resulted in a much more streamlined assessment process which also eliminated the need for talk among tutors during the performance and recorded their thinking at the time so that tutors could return to it at a later date during the moderation process.

Methodology (moderation)

The moderation process was conducted electronically for one week after the live performance and at a time convenient to each tutor. With online access to all completed electronic marking keys, the performance videos for each group and the comments/conversations tutors recorded at the time of performance, tutors were able to use the chat box to ask each other further questions, clarify aspects and negotiate final marks each time they logged on. These comments, although not visible to students, were left on the electronic marking keys. They could be used at a later time to remind tutors of the moderation conversations that took place should any student require further feedback or clarification about how their marks were awarded.

Methodology (disseminating marks to students)

Once reviewed and finalised, each group's video performance was inserted into the electronic marking key page alongside the marks and comments. The entire marking key, including the video clip of their performance was converted to a PDF file and then emailed to each group. This provided students with the opportunity to view their own performance from the tutors' and audience perspective and to explicitly see how they were assessed against the criteria.

Contributions to student learning outcomes

This Internet based assessment allows for more communication between assessors during both the live performance and later during moderation which enhances assessment reliability and fairness for all students.

The assessment contributes to student learning through the quality and educative value of assessment (assessment as and for learning) as it makes explicit the way in which marks are awarded and comments made. The assessors' marks and comments become educative annotations as the students view videos of their own performance and read the assessor's marks and comments. In addition, video clips of performances will be selected for future use as exemplars to demonstrate particular concepts, skills and understandings.

The video recordings offer students a view of their performances from the audience perspective. Comparing their own performance from this perspective against the criteria and marks/comments given provides quality feedback to individuals within groups.

Significance of the project for assessors

The use of Internet based marking during the live performances by multiple markers where each has instant access to each other's marking enables communication between markers without the need to verbalise during the performances enabling a stronger focus on the performance. This results in enhanced accuracy of assessment.
Assessing the drama performances using the electronic rubric simplified the marking process. I was able to view the live performances in their entirety... It made the moderation process instantaneous, as the team were able to view each grade and comments as it was submitted on the rubric (Drama Tutor).
In addition, moderation of marks after the live performances need not take place in face-to- face meetings, but rather over a period of time where tutors are able to access the online marking tool and video clips of each performance at a time suitable to them. The online marking tool has provision for assessors to chat via a confidential box on the marking tool, only available to the markers. The comments in this box function to enhance moderation and record discussion between tutors for future reference.
While I was still able to respond to the immediate impact of the performance, I was also able to go back and view the videos, reflect on my assessments and, if necessary, adjust some of my judgements. At the performance I could view my assessment criteria, the students' names and have immediate interaction with the other tutors who were assessing different elements of the arts. I felt that I was able to make informed and valid decisions.
rather than the 'quick draw' reactions of previous assessment experiences. (Music Tutor) The online marking tool eliminates the need for paper-based documentation of assessments and enables far more efficient management of records for both the users and recipients. It has potential to be used across a number of educational settings where students undertake any kind of performance (including presentations and practical assignments) where one or more markers assess their learning.

Significance of the project for students

Performance by nature is ephemeral and for students the whole assessment can be over in a short time. Students' intentions for their performance may vary from how it is perceived by the audience. Using video and the electronic marking tool means that their performance is captured visually as well as in the form of marks and comments from tutors. These are then viewed side by side. This provides students with an opportunity to reflect on their performance and use the feedback from their tutors to enhance their learning.
In truth I was quite nervous before hand; the performance and the feedback were over so quickly that I had little time to reflect upon them. Allowing me the opportunity to view the performance and hear the peer feedback, coupled with yours, J's and R's professional feedback through the rubric, has presented me with a rare and much valued opportunity to review what I had learnt and what I still needed to learn (Student).
Miels (1999) emphasises the positive effects and the value that is added to learning when students are given multiple opportunities to view their videos. In addition, viewing their own videos with the tutor feedback enables the students to see actual events and why they were awarded the marks they were as opposed to what they thought might have happened (Romano & Schwartz, 2005).

Previous studies of assessment in tertiary education have indicated that students are largely concerned with the fairness of assessment marks (Wren, Sparrow, Northcote, & Sharp, 2008). Students in this study report that they are satisfied with the electronic marking as a means to further moderation among markers.

Students report that they like their assessment sent to them electronically. They feel that "it is an incredible use of technology. This method of feedback is truly unique. The opportunity to see the group performance and the rubric was fantastic" (Student). Receiving their marks and video clip via email complements their preferred mode of learning. They are able to access this from anywhere and as many times as they need.

Conclusions

This project is an innovative, small scale, work-in-progress. To date, results strongly indicate that using Internet based rubric marking tools has streamlined assessment practice and enhanced moderation practices among multiple markers; it saved time and provided convenience to both the markers and students. Its strongest feature was that the assessment resulted in far more explicit and clear feedback which provided opportunity for students to review and discuss their learning in an authentic context through self-reflection.

The ease of accessing an electronic document and the preference students have for electronic documents, gave students incentive to review the feedback. Many students reported that they accessed and reviewed the feedback rubric multiple times, each time focusing on a different aspect that was assessed.

This method of marking allowed for multiple moderation sessions, which satisfied the markers and students that the results given were fair and valid. The markers felt that the strength of this method of assessment was that they could review their initial marks and comments at a later time that was convenient to them and use the tutor chat box for discussion between tutors.

For a tutor to assess these [performances] during a one off short performance seems to be a task that is in danger of being a 'hit and miss' affair. In this trial, all that changed. (Tutor)

Future directions and recommendations

Given the limit of time and resources, technology and online resources may be a key to engage the modern student in assessment and self -reflection particularly where the assessment is performance based. O'Toole (2009) makes the point that young people spend a great deal of time expressing and discovering themselves in cyberspace. The growing challenge, therefore, for many academics, is to harness the power of online technology within the formalised structures of the institution (Wheeler and Wheeler, 2009).

The opportunities for learning and teaching using Internet based assessment tools are many. The availability of electronic feedback combined seamlessly with the actual performance makes access to it convenient for students both on or off campus, and at those times that are most productive educationally for each individual. In addition, the electronic assessment provides opportunities for students to review their feedback and performance multiple times.

ICT is changing the traditional learning and assessment processes (Campbell, 2005). Wheeler and Wheeler (2009) believe that there is a growing challenge for many academics to discover how universities can harness the power of online tools within the formalised structures of the institution. This project demonstrates an example of this. The possibilities are that this type of online assessment tool can provide educative feedback to students, and could be adapted to empower learning through self and peer-assessment. These new forms of digital assessment tools provide the opportunities to engage in innovative assessment strategies previously beyond the financial resources of most institutions. Through self and peer-assessment we become lifelong learners and digital forms of assessment, like the one described, is making it possible within the resources available to educational institutions.

The ease with which this Internet based digital computer technology tool can be used in classrooms means that it is applicable in a wide range of educational settings and can easily be transferred to many educational contexts.

References

Absolum, M., Munro-Keene, J. & Phillips. (2009). Definition of Narrative Assessment. Evaluation Associates. Available from http://www.evaluate.co.nz/Modules/Resources/Download.aspx?ID=212

Campbell, A. (2005). Application of ICT and rubrics to the assessment process where professional judgement is involved: the features of an e-marking tool. Assessment & Evaluation in Higher Education, 30(5), 529-537.

Brunvand, S. (2010). Best practices for producing video content for teacher education. Contemporary Issues in Technology and Teacher Education, 10(2), 247-256. http://www.citejournal.org/articles/v10i2currentpractice1.pdf

Denton, P. (2001). Generating coursework feedback for large groups of students using MS Excel and MS Word. University Chemistry Education, 5(1), 1-8. http://www.rsc.org/pdf/uchemed/papers/2001/p1_denton.pdf

Dinham, J, (2010). Delivering authentic arts education. Cengage Learning, South Melbourne, Vic.

Ladson-Billings, G. (1998). Teaching in dangerous times: Culturally relevant approaches to teacher assessment. The Journal of Negro Education, Summer 1-2.

McConnell, T. J., Lundeberg, M. A., Koehler, M. J., Urban-Lurain, M. & Shang, T. (2008). Video Based Teacher Reflections: What is the real effect on reflections of in-service teachers? Proceedings of the 2008 International Conference of the Association of Science Teacher Educators, Saint Louis, MO (1-14).

Miels, J. (1999). Videotaped teaching segments and the pre service teacher: Developing reflective practice. The Teacher Educator, 34(3), 181-188.

O'Toole, J. (2009). Research in drama education. The Journal of Applied Theatre and Performance, 14(4), 479-501.

Rich, P. & Hannafin, M. (2009). Scaffolded video self-analysis: Discrepancies between pre service teachers' perceived and actual instructional decisions. Journal of Computing in Higher Education, 21(2), 128-145.

Romano, M. & Schwartz, J. (2005). Exploring technology as a tool for eliciting and encouraging beginning teacher reflection. Contemporary Issues in Technology and Teacher Education, 5(2), 149-168. http://www.editlib.org/d/6026/article_6026.pdf

Song, K. & Catapano, S. (2008). Reflective professional development for urban teachers through videotaping and guided assessment. Professional Development in Education, 34(1), 75-95.

Wheeler A. & Wheeler D. (2009).Using wikis to promote quality learning outcomes in teacher training. Learning, Media and Technology, 34(1), 1-10.

Wren, J., Sparrow, H., Northcote, M., & Sharp, S. (2008). Higher students' perceptions of effective assessment. The International Journal of Learning, 15(1), 11-24.

Please cite as: Wren, J., Campbell, A., Heyworth, J. & Bartlett, R. (2011). Improving marking of live performances involving multiple markers assessing different aspects. In Developing student skills for the next decade. Proceedings of the 20th Annual Teaching Learning Forum, 1-2 February 2011. Perth: Edith Cowan University. http://otl.curtin.edu.au/tlf/tlf2011/refereed/wren.html

Copyright 2011 Julia Wren, Alistair Campbell, John Heyworth and Rachel Bartlett. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format, provided that the article is used and cited in accordance with the usual academic conventions.

[PDF version] [Refereed papers] [Contents - All Presentations] [Home Page]
This URL: http://otl.curtin.edu.au/tlf/tlf2011/refereed/wren.html
Created 24 Jan 2011. Last revision: 24 Jan 2011.