Teaching and Learning Forum 2011 Home Page

Category: Professional practice

Teaching and Learning Forum 2011 [ Refereed papers ]
Measuring success: A case study in evaluating an online community using the Facebook fan page for UWA students

Lisa Cluett and Hon Weng Benjamin Seah
The University of Western Australia
lisa.cluett@uwa.edu.au

Australian universities have joined the growing global trend of building online student communities for recruitment, engagement and relationship building. Many of these communities are built using Facebook, including the one at UWA which was developed by Student Services in response to the annual survey of student ICT needs and skills. Now that communities are being part of mainstream university activities, there is a growing area of interest and research in generating benchmarking data, advice, standards and measurement in community building to determine the success of these engagement initiatives. Most of this benchmarking applies to communities aimed at marketing and recruiting potential students to the institution. For communities that prioritise the engagement of current and incoming students (including referral, demystification and explanation) this paper asserts that most meaningful way to determine the success of the online community is to gather a range of qualitative data from the community members themselves.

Introduction to online student engagement

During her keynote presentation at the 2010 Teaching and Learning Forum, Professor Marnie Hughes-Warrington (PVC, Teaching and Learning from Monash University) highlighted the importance of outside the classroom programs by posing the question 'how can learning happen when that fundamental human engagement is missing'. This outlook has been growing in popularity in recent years albeit building on an early foundation of student support in higher education as the sector continues to explore ways of providing a satisfying student experience. Whether the aim is to increase retention, diversify student populations, provide a smooth transition experience or scaffold domestic-international student interaction; the importance of 'outside the classroom' engagement promoted by Richard Light (2001) has achieved a fixed place in Australian higher education institutions.

The move to take student engagement online is more recent. While teaching staff and academic development teams continue to work on delivering quality, flexible and assessable course content via institution-wide delivery systems; staff in non-teaching positions are confirming their roles in student engagement by delivering services, referring to programs, building connections, answering questions and building community using online tools. And while the evolution of learning management systems and integration with online classrooms, discussion boards, ePortfolios, wiki-type functionality has given greater opportunities for class-based interaction, the majority of non-course based student communities are built using external environments such as Ning, wikispaces, LinkedIn, Twitter and the market leader Facebook.

This paper builds on earlier work by Cluett (2010) that defined the establishment of an online student community in Facebook including the development of a content matrix to plan and monitor content. This paper examines the UWA Students Facebook community two years after establishment and focuses on the issue of defining success and carrying out evaluation. Findings from this action research assert that while there is a growing field of standards and metrics used to recommend growth, posts rate and user interaction; the most meaningful way of determining the success of a community aimed at engagement is to generate quantitative and qualitative feedback from members.

Online communities in universities

Institutions are increasingly harnessing the power of online environments for creating communities and maintaining links between students and their university via networks such as Facebook, LinkedIn, MySpace, Twitter and Bebo (e.g. in the UK, the University of Aberdeen and the University of Bath and in the US, Drake university, the University of Michigan and Yale Science) with other institutions preferring to create their own members only alumni network with internally hosted software packages (such as University of Adelaide and The University of Western Australia) or encourage the development of communities via environments such as Ning (as have smaller institutions such as Essex Inte

rnational Centre, Norwich University in the UK; Texas Southern University in the US) (Cluett & Skene, 2010).

So while it is almost expected now to see links to Twitter, YouTube and Flickr embedded in official university webpages, the use of online communities, particularly Facebook is often newsworthy in itself. For example, the 'race' between Texas A&M and Louisiana State Universities to gather 100,000 fans gained momentum and turned attention to the use of Facebook by higher education institutions. It is important at this stage to draw the distinction between Facebook fan pages that have a marketing/recruitment focus and those that have a community building/engagement purpose. Whilst these two aims are not mutually exclusive, very different communities are developed depending on the target audience, content and tone of the pages. The UWA Students fan page does not aim to actively recruit students; rather to create an online space where current students can learn more about their university, feel that they belong at our institution and to become connected to fellow students. Pages called, for example, 'University of St Andrews postgraduate recruitment' (UK) or 'St Mary's University recruitment' (Canada) clearly have their sights set outside the currently enrolled student population.

The challenge of defining success

Throughout the rapid growth of university Facebook pages over the last 2 years, there has been the unanswered question of identifying whether the online community is 'working' or more specifically whether it is giving sufficient return for the staffing and time dedicated to it (Return On Investment or ROI). Marketing companies have come forward with tracking tools, surveys and metrics relating to page growth and size (Blue Fuego in the US is probably the most widely known of these companies) although these metrics are of limited use when considering pages that aim to engage rather than recruit. The non-profit sector offers more of a focus on community building and interaction and there are companies that monitor social media use and offer services to assist organisations in developing online communities. For example, M+R Strategic Services (2010) suggest an average fan page growth rate of 3.75 per month to be a 'healthy' level. They note that as the page gains more fans it is easier to gain even more (as networks grow and new generations of students enter the institution). Yet despite quantitative benchmarks based on existing Facebook communities, it remains the case that because each Facebook community is so distinct, the most meaningful way of determining a community's success is to ask its members.

Evaluating the community by asking its members

The UWA Students Facebook page is operated by Student Services staff as part of the Online Student Journey initiative. It aims to start building relationships with students as early as offer acceptance, to create a sense of belonging and community, to add a 'layer' to existing programs and services and to refer students to appropriate and relevant information. The page has grown from approximately 800 fans in January 2010 to approximately 4000 fans at the start of December 2010 (in line with the benchmark data for a medium-sized Australian University or a small US institution). Most growth occurs during the transition/intake period for undergraduate students although the page maintains a healthy average of 2200 users visiting the page each month. 87% of the current fan population is from Australia (approximately matching the proportion of domestic undergraduates at UWA), with countries such as Singapore, USA, Malaysia, UK, Canada and Indonesia with between 2.5 and 0.5% of the population. The gender balance has remained constant at 51% female, 49% male. The page is managed using a planned content matrix (Cluett, 2010) with a focus on giving fans an insight to University life on the UWA campus. Fans are referred to askUWA, UWA's official FAQ database, for all questions relating to official university business (census dates, fees, progression rules etc) whilst advice about student life (for example housing, food, textbooks etc) is best provided by fellow students.

Student survey and interviews

The UWA Students Facebook page was the subject of an Honours research project during 2010 that aimed to identify student perceptions of the online community and compare them to the stated purpose of the page to determine community engagement. The project gathered student perceptions using two methods; an online survey and individual interviews. The online survey was open to currently enrolled students who were also fans of the Facebook page and the 10 individual student interviews (6 male, 4 female, mix of domestic and international students) were held between 10th and 17th August 2010. Ethics approval was granted for all elements of the research project.

Interview results
Most students interviewed indicated that they found out about the UWA Students Facebook page from their friends. This reinforces the power of the 'fan page' functionality whereby page activity appears in fans' news feeds for their friends to see. A small number of interviewees were also referred to the page by University staff and all interviewees joined the page the first time they heard about it. They perceived a range of benefits upon joining the page including feeling part of the UWA community, finding out about events and activities and interacting with other students. An important finding from the student interviews is that fans prefer to check page content via the updates that appear on their own pages rather than going to the Facebook fan page itself. This has important implications for the frequency of posts uploaded by page administrators - enough to remind fans about what's happening but not to the extent where fans are overwhelmed with too many updates. One of the interviewers explained this pattern of usage as

I very rarely check the page, it happens to all my other pages. I join other pages but I never really go to their page to follow their wall posts. Anything that comes to my news feed I will just read (quote from student interview)
Students interviewed expressed a willingness to post content (questions, comments, 'likes') on the page, particularly if they had a query that might also apply to other students or if they knew the answer to a student-posted question because they had had a similar experience. Students also explained that they would use the Facebook page in conjunction with other services including face-to-face helpdesks and contacting individual staff directly.

When asked about the benefits gained from the page, students highlighted that they found it useful to hear about things happening on campus such as exam results being released, teams moving buildings on campus and students selling textbooks. Students also described instances where they felt part of the UWA community by being a fan of the page, for example

...I like to see the pictures posted by others because some people can really snap a fantastic view. For example, pictures of Winthrop Hall, the Japanese Garden and especially the photos and videos of the hailstorm. I even saw a picture that was posted when Student Services had a window washer on the same day I went to see my careers counsellor (quote from student interview)
Those students who felt that they did not gain any direct benefits form the page still felt that the page would be of benefit to students starting their journey at UWA rather than those further on in their studies. Students all indicated that they didn't use the page to make new friends stating that they preferred to meet new friends face-to-face and then potentially connect with them in online communities. This suggests to page managers that students are not seeking to form new friendships through the page yet they experience other benefits from being part of the student community.

7 of the 10 students interviewed indicated that they found the page's wall to be the most useful feature of the page as it hosts the discussions and interactions by the page fans. The other 3 interviewees preferred the photo albums which showed activities happening around campus for reasons including:

People can see what's happening on campus...that's good. Because it's a big campus and not all students go to all parts...so they know what is happening through the page and it's a good way to get the message across. (quote from student interview)
All students interviewed stated that they would continue to use the page in a similar way, possibly even after they graduate for a arrange of reasons including
...just in case of emergency or anything that comes up and it's within hands reach. So it's quite convenient to have around (quote from student interview)
and interviewees indicated they were happy to recommend the page to other students. One interviewee explained the different types of students who might benefit from the page by categorising them into groups
Yes. I would recommend it to...1st year students because they will be able to get the most info. The 2nd category is Facebook addicts. The 3rd group is students who are ignorant about what's going on in Uni and want to know more about what's happening. Recommending to that group would make them more in touch and less ignorant. (quote from student interview)

Student testimonials and examples

Much of the evidence for community success (or otherwise) is openly viewable as it is posted on the Facebook wall itself. Student feedback ranges from the simple: 'thanks so much - I can't wait to come to UWA. ps this page is such a great idea' (comment from student joining UWA undergraduate course from India) to enthusiastic 'this is the goddam best uni service I have ever seen' to which another student replied 'agreed. At my last uni I was impossible to get any sort of info out of anyone. This is just fantastic' (student comments posted to the wall in 2010). Interaction and conversation between fans is typically most vigorous during times of shared experiences and major events (exam timetable is released or study break starting) and the page became a hub of activity during and immediately after the hailstorm that devastated much of the UWA campus. During this time more than 60 students posted comments, questions and advice and uploaded more than 30 photographs and 10 videos.

Conclusion

Evaluating the success of an online community can be a challenging undertaking. University initiatives typically require a firm indication of their outcomes and success and the identification of success factors and/or performance indicators. Benchmarking data, metrics and recommendations are increasingly being provided by research companies that also offer services in helping Universities and non-profits develop a marketing strategy and build community. However, this paper has argued that while having targets for growth, content posting and fan interaction can be useful to guide online community building, they do not provide the full picture. When a fan page is aiming to engage students in University life, the most meaningful way to assess community success is to use a composite of qualitative data from student surveys, interviews, online feedback and testimonials. This requires an ongoing commitment to community evaluation in a similar that face-to-face programs are characterised by a cycle of planning, implementation and evaluation; something often forgotten in the world of online communities which are constantly live and growing.

References

Berg, J., Berquam, L. & Christoph, K. (2007). Social Networking Technologies: A "poke" for campus services. Available at: http://www.educause.edu/ir/liebrary/pdf/ERM0721.pdf [Accessed 13 Sept 2010]

Cluett, L. (2010). Online social networking for outreach, engagement and community: The UWA Students' Facebook page. In Educating for sustainability. Proceedings of the 19th Annual Teaching Learning Forum, 28-29 January 2010. Perth: Edith Cowan University. Paper available from http://otl.curtin.edu.au/tlf/tlf2010/refereed/cluett.html

Cluett, L.J. & Skene, J. (2010). The student-learning student-engagement nexus: how can non-teaching areas of the University use Web 2.0 tools to enhance the student experience? Chapter in Lee, M.J.W. & McLoughlin, C. (Eds), Web 2.0 is here: Applying Social Informatics for Tertiary Teaching. Hershey, PA: IGI Global.

Cluett, L.J. & Skene, J. (2010b). TechNODE bulletin 2010: an annual synopsis of student access to, experiences with and expectations of ICT at The University of Western Australia.

Hughes-Warrington, M. (2010). Keynote Presentation at the 2010 Teaching and Learning forum 'Educating for sustainability' Forum homepage http://otl.curtin.edu.au/tlf/tlf-pubs.html

Community College Survey of Student Engagement (CCSSE) (2008). High Expectations and High Support. Austin, Texas: The University of Texas at Austin, Community College Leadership Program. Available from: http://www.ccsse.org/publications/2008_National_Report.pdf [Accessed 6 September 2010]

Jenkins, H. (2008). Hanging out, Messing Around, Geeking Out: A conversation with the Digital Youth Project (Part 2). Available from: http://www.henryjenkins.org/ [Accessed 4 September 2010]

Krause, K. (2005a). Understanding and promoting student engagement in university learning communities. Available from: http://www.cshe.unimelb.edu.au/pdfs.stud_eng.pdf [Accessed 11 August 2010]

Krause, K. (2005b). Who is the e-generation and how are they faring in higher education? In J. Lockhard & M.Pegrum (Eds), Brave New Classrooms: Educational Democracy and the Internet. New York: Peter Lang

Krause, K. (2005c). The changing student experience: Who's driving it and where is it going? Student Experience Conference: Good Practice in Practice, Charles Sturt University 5-7 September 2005. Available from: http://www.csu.edu.au/division/studserv/sec/papers/krause.pdf [Accessed 4 September 2010]

Light, R. J. (2001). Making the most of college: Student speak their mind. Cambridge, Mass: Harvard University Press.

M+R Strategic Services (2010). 2010 Nonprofit Social Media Benchmark study. Available from http://www.e-benchmarksstudy.com/socialmedia [accessed 12th October 2010)

McLoughlin, C. & Lee, M. J. W. (2007). Social software and participatory learning: Pedagogical choices with technology affordances in the Web 2.0 era. In ICT: Providing choices for learners and learning. Proceedings ascilite Singapore 2007. Available from: http://www.ascilite.org.au/conferences/singapore07/procs/mcloughlin.pdf [Accessed 23 April 2010]

Markwell, D. (2007). A large and liberal education: Higher education for the 21st century. Melbourne: Australian Scholarly Publishing.

Oblinger, D.G. & Oblinger, J.L. (Eds) (2005). Educating the Net Generation. [online] Educause e-book available from http://www.educause.edu/educatingthenetgen/ [Accessed 4 August 2010]

Oliver, B. & Goerke, V. (2007). Australian undergraduates' use and ownership of emerging technologies: Implications and opportunities for creating engaging learning experiences for the Net Generation. Australasian Journal of Educational Technology, 23(2), 171-186. http://www.ascilite.org.au/ajet/ajet23/oliver.html

Pesce, M. (2007). Challenges and Opportunities: Peer-produced knowledge and Australian education. Available from: http://blogs.educationau.edu.au/gputland/wp-content/uploads/2007/08/challenges-and-opportunities.doc [Accessed 20 October 2010]

Skene, J., Cluett, L. & Hogan, J. (2007). Engaging Gen Y students at university: what web tools do they have, how do they use them and what do they want? Refereed paper presented at the First Year in Higher Education conference, Brisbane, Australia, 3rd-6th July 2007. Available from: http://www.studentservices.uwa.edu.au/page/108244

Please cite as: Cluett, L. & Seah, H.W.B. (2011). Measuring success: A case study in evaluating an online community using the Facebook fan page for UWA students. In Developing student skills for the next decade. Proceedings of the 20th Annual Teaching Learning Forum, 1-2 February 2011. Perth: Edith Cowan University. http://otl.curtin.edu.au/tlf/tlf2011/refereed/cluett.html

Copyright 2011 Lisa Cluett and Hon Weng Benjamin Seah. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format, provided that the article is used and cited in accordance with the usual academic conventions.

[PDF version] [Refereed papers] [Contents - All Presentations] [Home Page]
This URL: http://otl.curtin.edu.au/tlf/tlf2011/refereed/cluett.html
Created 22 Jan 2011. Last revision: 22 Jan 2011.