Impact Data
Developing strategies and approaches to measuring the success of your open text project should be developed during the Project Plan stages. Collecting and analyzing this data will help you maintain accountability to the goals of your project and to understand whether your project is successful. Measuring success will depend on a few factors:
- What constitutes success for your project and how will it be measured? Pedagogical innovation? Surveys? Interviews?
- Who do you need to report to about the open text project? Funders? Your institution? Department?
- What data is needed to indicate “success”? Adoption numbers? Students impacted?
- How do you need to use the data for your own portfolio? Tenure and promotion? Merit?
Working through these questions will assist you in collecting the right metrics to report fully about the success of your project. This guide outlines the kinds of data that you can collect to frame discussions about your open textbook project and ensure you have reached your goals.
Adoption Data
Collecting adoption data can be a difficult process as it requires adopters to identify that they are using the text in their courses or instructional practices. Developing a feedback mechanism for your text that requests that adopters identify their use of the text is the easiest way to gather this data.
To begin, decide how you will track adoptions and what statistics you’ll collect. For instance, you might want to track adoption trends by course level, type, term, or semester.
Then identify the tool you will use to gather this data. For example, you can develop a Qualtrics form located as a link on the homepage of your textbook, in the front matter of the text, or in a metadata field of your open textbook record to encourage self-reporting.
Examples – Calculus Volume 3 Adoption Metrics
- BCcampus Open Textbook Collection – Data collected for textbook adoption includes the institutions and courses where the textbook is used as well as the cost savings for students.
- Open Stax – Data collected by use rather than adoption. The form is broken down into students, faculty, homeschooler, administrator, librarians, instructional designer, adjunct faculty, and other to create a more tailored data collection process.
Open Education Repository Data
Depending on where you decide to host your open text you may be able to take advantage of the platform to gather analytics about your text’s use. The data you can collect includes views (by year/month), downloads (by year/month), and user ratings and comments. While there may be many data options for you, you need to decide how you will use it and select what is most useful to your success measures.
The following are examples of the kinds of data you can gather in UBC cIRcle, OER Commons, and MERLOT.
UBC cIRcle
- Views and downloads by country; Views and downloads by month and year; Total views and downloads
- Numbers of times a resource is viewed; Number of times a resource is saved; Ratings out of 5 stars; Comments
- If you have a paid OER Commons Hub, there are additional metrics available.
- User ratings; Comments; Merlot Awards System;
- If you use the Content builder you can add Google Analytics for additional metrics.
Catalogue Data
If you have made your open text available through cIRcle, Summon (General Search), and/through the UBC Library catalogue (Books & Media), there may be some circulation and usage data that UBC Library can collect. To learn more about adding your textbook to UBC Library systems, review Developing a Release Plan.
Pressbooks Data
If you are using Pressbooks, there are metrics available for the web version of your text. You can find metrics in the dashboard menu labeled “WP-Piwik”. Piwik Analytics provides many data options, including:
- Number of visitors in the last 12 months
- Number of visitors in the current month, including page views, time spent, maximum page viewed in one visit
- Pages viewed in the current month
- Cities of the visitors in the current month
- Keywords, Browsers, Type of technology, and more
-
-
- Evaluation Metrics
-
While usage and adoption statistics provide part of the story of your open text’s success, understanding the impact on students beyond cost savings can provide broader insights on the value of tailoring an open text for pedagogical purposes. This kind of evaluation may include the following questions:
- What does ‘impact’ look like if considering innovation in teaching and learning?
- A rise in grades
- Greater understanding of complex topic areas
- Increased engagement with the textbook and course content
For support in developing an evaluation plan that addresses the pedagogical value of your open textbook project, contact the Centre for Teaching and Learning Technology (UBC Vancouver) or the Centre for Teaching and Learning (UBC Okanagan).
Put Your Stats to Work!
UBC Tenure and Promotion
In 2017 UBC included language recognizing creation and adaptation of open educational resources (OER) in the institution’s Guide to Reappointment, Promotion and Tenure Procedures at UBC. The guide lists the following among several examples of evidence of educational leadership:
Contributions to the practice and theory of teaching and learning literature, including publications in peer-reviewed and professional journals, conference publications, book chapters, textbooks and open education repositories/resources (p. 14).
In the UBC Faculty Association Collective Agreement, “educational leadership” is one of the areas (along with teaching and service) in which candidates in the educational leadership faculty stream are evaluated for appointment, reappointment, tenure, promotion, and merit. It is defined in that agreement as:
Activity taken at UBC and elsewhere to advance innovation in teaching and learning with impact beyond one’s classroom (Part 4.04).
When considering how to talk about OER creation or adaptation as part of your annual report or dossier for reappointment, tenure, promotion or merit, given the definition of educational leadership above, you should work to show the impact of such work “beyond your classroom,” whether at UBC or elsewhere.
Possible ways to report your open education activity in your annual report:
- Reporting adoption statistics;
- Keep records of personal communications that give praise for the work or talk about how it’s being used;
- Note any adaptations of the work by others;
- Solicit feedback from peers or students on their perceptions of the quality and usefulness of the work;
- Or other ways to show that the OER is being used and considered a valuable resource by others.
Adaption
- Adapted from the Measure of Success section in Institution as e-Textbook Publishers by JISC licensed under a CC BY licence.
- Adapted from the The Rebus Guide to Publishing Open Textbooks (So Far) by Apurva Ashok and Zoe Wake Hyde is licensed under a Creative Commons Attribution 4.0 International License, except where otherwise noted.
