CHAPTER 8 : Evaluating Successful Blended Learning

Introduction

This chapter will explore various dimensions of evaluation in reference to blended learning. Drawing in all that you have learned so far in this guidebook, it will give you a framework not only to evaluate your particular blended learning course or programme but also to weigh the pros and cons of different learning designs, identify key principles and gradually develop your own personal teaching philosophy for blended learning.

Models for Evaluating the Design and Delivery of Blended Learning

Although blended learning is now part of the narrative in education almost everywhere, it can still be a challenge to define and describe the concept. Many who are looking at or already working with blended models are doing so with little support or training. Requirements for engaging processes and quality outcomes are often not obvious to administrators, faculty, instructors or students.

Further, expertise in technology implementation, instructional design, teaching models and learning theory must be developed or accessed. Four general factors must be represented in the design and evaluation of blended learning:

• the pattern of delivery mode, which sequences and combines in-person or face-to-face engage-ment with independent and collaborative online social and cognitive activities;
• the materials, technology and media used;
• the use of varying pedagogical models, representing unique teaching patterns and learner actions, such as inquiry-based, constructive, behaviorism, experiential and others; and
• the temporarily of synchronous and asynchronous methods.

The quality of a blended learning course or programme must be assessed, with appropriate guidelines representing the granular practices and outcomes of blending many different teaching and learning opportunities.

This comprehensive guide is an attempt to help navigate this complex challenge and begin to understand the components of blended learning, its benefits and value, and the required design processes. While blended learning is really nothing more than employing a variety of media and methods to provide a mix of online and face-to-face learning, it can become a very difficult process to select from the range of possible combinations of elements, sequences and pacing’s. Once a blended learning design is in place, formative evaluation (continuous during the course or programme) and summative evaluation (at the conclusion of the course or programme) must be a seminal part of the quality assurance process.

In considering the quality of blended learning, it is helpful to look at the use of online environments and what they offer education. For quality online learning, certain requirements must be present as key parts of any blended environment; these requirements can also be extended to synchronous or in-person activities as the more traditional part of the blend.

Online learning can:

• increase access;
• foster equity in the learning environment, as it is color and gender blind and class neutral;
• create affordable, convenient learning opportunities, and
• develop expanded learning skills for students related to self-direction, self-regulation and collaboration.

Online opportunities can provide quality education to an expanded audience previously left out of exclusive and often costly, geographically bound, place-based education. Blended learning, as a further development of online learning, should strive to create these same benefits for learners through both its online and in-person, face-to-face components.

With such a range of possible factors for assessing quality and improvement, how then do we evaluate our blended learning courses and programmes? Quality assessment rubrics for blended learning have yet to be well-researched and implemented, and a significant, widely accepted instrument to evaluate blended learning quality is still unavailable. According to Smyth (2017), “the means to evaluate its effectiveness is frequently lacking since there are a relatively limited range of tools and methods that support staff in designing blended learning curricula” (p. 854). Creating such an instrument is a major undertaking; blended learning incorporates and integrates traditional and online delivery methods, making it much more complex than uni modal delivery. This guidebook is one step towards this larger goal, and although evaluation rubrics normally follow curriculum processes, here we offer you advice and templating suggestions for blended learning evaluation.

In our search for rubrics to recommend, we looked for a tool that included “aspects not obvious to instructors or learners, such as instructional design, course development, and the use of technology” (Smythe, 2017, p. 855). Some early tools and rubrics are available and in use, with varying levels of sophistication in measurement and concepts. In other words, opportunities to measure blended learning quality also vary in quality! We recommend reviewing the tools suggested below, looking for others to add to your knowledge base, and then considering developing rubrics and concept maps for your own use.

Blended Course Learnability Evaluation Checklist

The Blended Course Learnability Evaluation Checklist (http://oasis.col.org/handle/11599/2941), developed by the Commonwealth of Learning, can be used to measure the quality of a blended course or as a guide during course development. This tool is divided into six sections, all evaluating the key aspects of a blended course as identified by this guide. This tool can be used as a design template or as an evaluation tool after design and implementation.

Using Community of Inquiry Indicators to Assess Presence in Blended Learning

Earlier in this guide, we reviewed the Community of Inquiry theoretical framework. The framework offers pedagogical guidance for designers, instructors and students interested in collaborative, constructivist learning environments to foster deep learning. The main model includes three presences, as indicated in Table 8.1. Each presence also has sub-elements or characteristics that indicate when a participant is present.

Emotional presence has been suggested as a fourth presence. More research is underway to test the place of emotions in this theoretical framework. However, early indicators are that emotions play an important part in the design and evaluation of blended learning. The preliminary definition of emotional presence is “the outward expression of emotion, affect, and feeling by individuals and among individuals in a community of inquiry, as they relate to and interact with the learning technology, course content, students, and the instructor” (Cleveland-Innes & Campbell, 2012, p.289). this aspect of the learning environment must be considered in person and online, synchronously and asynchronously.

Table 8.1. Evaluation indicators for blended learning in the CoI framework

A survey instrument has been developed to support evaluation through the Community of Inquiry theoretical framework.This survey is taken from the instructor’s point of view and measures the extent to which these four presences are evident in a blended course of study.The item indicators in the survey instrument can be used not only during and after a course to test the activities with respect to each presence, but also during the design phase. See Appendix 1 for a list of indicators, and consider which learning activities could be offered for each element to become “present” during the course.

Preparing for Evaluating Blended Learning Design

Adapted from Weimer’s (2002) Learner-Centered Teaching, the following principles of student- centered learning will help guide your evaluation of your blended learning designs:

• Student-centered learning shifts the balance of classroom power from teacher to student, thus fostering active learning and engagement among peers.
• Student-centered learning enables critical thinking and is a means to develop knowledge rather than a collection of facts, by building upon and challenging prior learning.
• Student-centered learning situates the teacher as facilitator and contributor rather than authori- tarian and director of knowledge.
• Student-centered learning returns the responsibility for learning to the students, so they are able to discover their strengths and weaknesses and take part in directing their own knowledge gain.
• Student-centered learning employs effective assessment to promote learning and inform future practice.

You may also find the work of Cleveland-Innes and Emes (2005) helpful when considering a learner- centered curriculum.

As identified in the introductory video of this chapter, we suggest you create your blended learning design and look back and ask whether you’ve followed through on the ideas, principles and models identified in this guidebook. Have you considered the roles of the learner and the teacher, technology, content, learner support and institution, as outlined by the CABLES framework? Have you ensured an effective blend of social, cognitive, teaching and emotional presence as described by the Community of Inquiry framework? In other words, ask yourself objectively whether your design meets your original intentions for creating a blended learning course or programme.

Make sure to review the key factors for creating successful blended learning, especially the importance of supporting both teachers and learners as they adapt to new roles and methods for teaching and learning in a blended environment. Again, you can evaluate using these questions: Have you created the necessary conditions for effective blended learning, including a learning environment that supports open communication and trust, reflection and discourse, a sense of community and purposeful inquiry? Have you created opportunities for learners to collaborate, and have you adequately prepared them for this form of learning?

Finally, when implementing technology for synchronous and asynchronous online learning, remember that assessment tools to measure achievement of outcomes can also be supported through technology. Assessments of student learning and of your blended learning model’s impact will together provide useful evidence for continuous improvement in your blended learning design.

CONCLUSION

In this chapter, we reviewed a selection of toolkit and rubrics, as well as the Community of Inquiry survey instrument, as useful tools for conducting evaluations of your own blended learning course or programme. We also considered a number of key questions and principles for creating effective, student-centered learning. You are now prepared not only to design effective blended learning, but also to demonstrate its effectiveness through reliable and appropriate evaluation.

It is important to show the worth of an education intervention. We know that blended learning works when planned and delivered well, but administrators and managers of blended learning will need evidence of this. However, evaluation also provides us with a valuable opportunity to reflect, revise and improve that which did not work well, and we recommend that you look at evaluation through the lens of “continuous improvement” and make the process your own.

You have now reached the end of the Guide to Blended Learning!

Throughout this guide, we have emphasized the wide variety of potential blended learning designs and provided you with frameworks, models, tips and examples to help you structure a blended learning course or programme to meet your students’ needs in your own learning context. As with any design process, developing effective blended learning is not about following a particular recipe but about combining key principles with your own experience, reflection and critical judgement.

We also hope to have inspired you with a vision of the open, learner-centered education possible through blended learning. Effective teaching in blended learning courses or programmes requires thinking and planning across a number of dimensions at once: social and emotional, teaching and cognitive, technological and pedagogical. Again, the Community of Inquiry theoretical framework can guide you in considering the full range of these dimensions. We encourage you now to reflect on your own personal teaching philosophy and ask yourself how it applies or can be adapted to teaching in blended learning environments.

We wish you the best on your journey as a blended learning developer, designer and instructor.

REFLECTION  QUESTIONS

Now that you have come to the end of this guide, we hope you have enjoyed it and also developed a blended course. If so, you can use the following to prepare a reflective essay on your experiences:

Step 1: Describe

Describe your experience of designing and delivering blended learning. Objectively answer the following prompts to describe the activities:

• What did you do?
• Why did you do it? What was the perceived advantage of the design planned?
• What happened as a result of the design?
• How did the course progress?
• What were the unique features of the course that you introduced?

Step 2: Examine

Critique your experiences of teaching a blended course. Use the following prompts to write your response:

• What new things happened in this course that you had not experienced before?
• Why do you think these were special and important for student learning?
• How did your students react to the blended course?
• What could you have done better?
• Did the course go as planned?
• What challenges arose during the course, and how did you approach solving them?

Step 3: Learning

Use your responses to the prompts in steps 1 and 2 of this reflection template to create a thoughtful essay wherein you articulate what you have learned from the experience. Each of the following questions should be addressed in your essay:

• What did you learn from this experience?
• How did you learn it?
• Why does it matter to you as a teacher?
• How does this experience relate to the institutional mission and vision?
• What might/should further be done in the light of your experiences?

Resources for Further Reading

Blended Learning Toolkit

The Blended Learning Toolkit (https://blended.online.ucf.edu) was created by the University of Central Florida (UCF) and the American Association of State Colleges and Universities with funding from Next Generation Learning Challenges. UCF is well known for its excellent work in the research and practice of education innovation and provides the toolkit as an open educational resource under a Creative Commons  Attribution-Non Commercial-Share Alike licence.

The section on Evaluation Resources (https://blended.online.ucf.edu/evaluation-resources) is particularly valuable, offering a number of resources to evaluate impact, including the impact of blended learning on courses, programmes and institutions themselves. Valid and reliable survey instruments are offered for use with students and instructors at no cost.

Note that this toolkit provides other valuable supports for starting your blended learning practice. See the Do-It-Yourself templates (https://blended.online.ucf.edu/blendkit-course-diy-project-tasks) and Blended Learning Stories (https://blended.online.ucf.edu/blendkit-course-stories) from educators and institutions to support and inspire your work.

Blended Learning Course Quality Rubric

This rubric was created by staff at the University of Ottawa, in Canada. One key benefit of this rubric is the comprehensive coverage of required aspects of blended learning. Course design, learner support and resources, use of technology, course organisation and content presentation are some of the topics covered by this rubric (https://tlss.uottawa.ca/site/files/docs/TLSS/blended_funding/2017/TLSSQARubric.pdf).