Curriki: Facilitating Use and User Engagement Around Open Educational Resources

Contents

• Introduction

• Methodology

• Findings

• Conclusions

• References

• Other Case Study Resources

Prepared by: The Institute for the Study of Knowledge Management in Education, Fall 2008
ISKME:   Download this case study

Curriki is a wiki-based website that enables users to create, remix, and share open educational resources. This case study seeks to understand the behaviors of Curriki users, and to identify ways to facilitate and support sustained user engagement. Through interviews with the Curriki management team, analysis of internal documents, observations of internal user data collection practices, and a survey and interviews with Curriki users, the Curriki case study explores use patterns and user perceptions of the site, its resources and tools. The study found that many users visit Curriki to develop new ideas for lessons, but that most do not create, remix or share content through the Curriki site. The study identified a need to support users that might lack the knowledge or supports to create and remix content. Furthermore, the study describes how understanding the social and institutional context of OER use can inform the design of OER communities that support a wide range of users.

Introduction

Curriki is a wiki-based website where freely available open educational resources (OER) are developed and distributed to anyone who wants to use them. Curriki offers more than 18,000 resources corresponding to a wide range of subject matter, including information and media literacy, science and mathematics, foreign languages, and social studies. Through a variety of tools, the site allows teachers, students, and other individuals to create and join groups, access, share, and create resources, and build communities around improving curriculum.

Curriki has designed its site to be engaging and easy to navigate, with resources that are easy to find, and that are usable and adaptable to local teaching and learning needs. In order to inform Curriki’s success in reaching these goals, this case study seeks to explore key aspects of use and user engagement. The Curriki study is part of a larger initiative led by the Institute for the Study of Knowledge Management in Education, which involves case studies of six open educational resources (OER) projects. The purpose of the case study initiative is to increase understanding of issues that are common across OER projects—such as engagement of online communities, volunteer recruitment, and licensing decisions—and to create synergies and knowledge sharing possibilities for field-building and leadership development.

Through interviews with the Curriki management team, analysis of internal documents, observations of internal user data collection practices, and a survey and interviews with Curriki users, the Curriki case study explored use patterns and user perceptions of the site, its resources and tools. The specific questions addressed include: 1) how often and why users are coming to Curriki; 2) how they use and engage around the Curriki site, its tools and its resources; and 3) what factors help or hinder engagement and use. The goal of this case study has been to develop an understanding of the mechanisms and processes that can help to attract and sustain users over time, and to facilitate and enhance their use, reuse and content contribution experiences.

Methodology

This study draws upon a participatory case study methodology similar to David Fetterman’s empowerment model. Fetterman (2005) defines empowerment research as that which helps organizations and communities assess and improve their practices by instilling mechanisms that allow them to align their theory of action (what they think they should be doing) with their theory of use (what they are actually doing). That is, it is a model for research that helps organizations build the capacity to assess their practices and develop insights that can be fed back into those practices in light of project goals. This entails developing research and data collection tools collaboratively with the case study organization, which can be used to reinforce, test and modify internal knowledge. In such an approach, the external researcher’s role becomes that of a critical friend and facilitator as opposed to a decision maker or expert purveyor of knowledge. As such, value and weight are placed more upon the insights and experiences of importance to the organizational members, as it is from their perspective that knowledge can be cultivated to inform practices and continuous improvement (Petrides & Loesch-Griffin, 2005).

Consistent with the participatory research model, the case study was an iterative and collaborative process. The case study work began in xx 2008 with two initial phone conversations between ISKME and the Curriki management team. The purpose of these conversations was to shed light on the project’s past and current challenges and opportunities, and to move toward the creation of a case study research plan. Both of the conversations lasted approximately one hour. ISKME then conducted a review of internal Curriki documents as a way to inform the development of the case study research plan, and to provide further insight on Curriki’s goals, strategies, challenges and opportunities. Through the initial conversations and document analyses, the case study questions were developed, which as noted above, focused on how often users visit Curriki and their reasons for visiting, how they actually use the Curriki site, its tools and its resources, and what factors help or hinder engagement and continued use. The methods used to explore these questions were also determined and included surveys and interviews with Curriki users, as well as log file data analysis.

The user survey consisted of a combination of multiple choice and open-ended questions, specifically addressing how often users visit Curriki, their specific use behaviors (i.e., where they click, what they do), their incentives and disincentives to use, and whether and how they use and modify Curriki resources in the classroom. The survey protocol was developed collaboratively by ISKME and the Curriki management team, and was distributed via email to all of Curriki’s registered users. A link to the survey was also posted on the Curriki website. Fifty-five users responded to the survey. The majority of respondents self-identified as teachers; however, professional development coaches, administrators, and researchers responded to the survey as well (also self identified).

In analyzing the survey data, basic descriptive statistics were first obtained for all survey items in order to glean overreaching use behaviors, experiences and patterns. Independent Chi-Square analyses were then conducted to determine whether there were significant differences with regard to use behaviors between different groups of users. More specifically, the chi-square analysis sought to reveal how active users potentially differ from inactive users in terms of how they draw on and use Curriki’s tools and resources.

Subsequently, a user interview protocol was developed by ISKME, with input provided by the Curriki management team. Informed by the preliminary analysis of the survey data, the interview questions were designed to elicit a more textured illustration of the users’ experiences with Curriki and with open educational resources more generally. An email invitation was sent by ISKME to 12 survey respondents who indicated their willingness to participate in the follow up interviews. Three of the 12 respondents participated in the interviews, which were conducted via telephone by ISKME and lasted approximately 30 minutes.

In addition to interviews and the survey, a template for log file data collection was developed by ISKME and Curriki to collect additional user data including how users navigate through the Curriki site, what they click on, and what specific actions they take. The template identified the types and parameters of data to be collected, how each data point could be captured, and the priority level for each so that internal staff and technology resources could be allocated to each. Although the log file data collection template was not implemented due to timing and resource constraints, the process of creating the template served as insight into the people and technology requirements for Curriki’s log file data collection efforts. It also illuminated a fuller range of possibilities with regards to the types of log file data that could and should be collected in order to understand issues around use and user engagement.

It is important to note that this case study does not attempt to provide a comprehensive assessment of Curriki user experiences and perceptions; instead it aims to explore some of the ways a particular group of users—i.e., the survey and interview participants—perceive, use, and experience the Curriki site, its resources, and its tools. The findings below are a presentation of the learnings gleaned through analysis of the interview and survey data combined. he capacity to assess their practices and develop insights that can be fed back into those practices in light of project goals. This entails developing research and data collection tools collaboratively with the case study organization, which can be used to reinforce, test and modify internal knowledge. In such an approach, the external researcher’s role becomes that of a critical friend and facilitator as opposed to a decision maker or expert purveyor of knowledge. As such, value and weight are placed more upon the insights and experiences of importance to the organizational members, as it is from their perspective that knowledge can be cultivated to inform practices and continuous improvement (Petrides & Loesch-Griffin, 2005).

Consistent with the participatory research model, the case study was an iterative and collaborative process. The case study work began in xx 2008 with two initial phone conversations between ISKME and the Curriki management team. The purpose of these conversations was to shed light on the project’s past and current challenges and opportunities, and to move toward the creation of a case study research plan. Both of the conversations lasted approximately one hour. ISKME then conducted a review of internal Curriki documents as a way to inform the development of the case study research plan, and to provide further insight on Curriki’s goals, strategies, challenges and opportunities. Through the initial conversations and document analyses, the case study questions were developed, which as noted above, focused on how often users visit Curriki and their reasons for visiting, how they actually use the Curriki site, its tools and its resources, and what factors help or hinder engagement and continued use. The methods used to explore these questions were also determined and included surveys and interviews with Curriki users, as well as log file data analysis. The user survey consisted of a combination of multiple choice and open-ended questions, specifically addressing how often users visit Curriki, their specific use behaviors (i.e., where they click, what they do), their incentives and disincentives to use, and whether and how they use and modify Curriki resources in the classroom. The survey protocol was developed collaboratively by ISKME and the Curriki management team, and was distributed via email to all of Curriki’s registered users. A link to the survey was also posted on the Curriki website. Fifty-five users responded to the survey. The majority of respondents self-identified as teachers; however, professional development coaches, administrators, and researchers responded to the survey as well (also self identified).

In analyzing the survey data, basic descriptive statistics were first obtained for all survey items in order to glean overreaching use behaviors, experiences and patterns. Independent Chi-Square analyses were then conducted to determine whether there were significant differences with regard to use behaviors between different groups of users. More specifically, the chi-square analysis sought to reveal how active users potentially differ from inactive users in terms of how they draw on and use Curriki’s tools and resources.[1]

Subsequently, a user interview protocol was developed by ISKME, with input provided by the Curriki management team. Informed by the preliminary analysis of the survey data, the interview questions were designed to elicit a more textured illustration of the users’ experiences with Curriki and with open educational resources more generally. An email invitation was sent by ISKME to 12 survey respondents who indicated their willingness to participate in the follow up interviews. Three of the 12 respondents participated in the interviews, which were conducted via telephone by ISKME and lasted approximately 30 minutes.

In addition to interviews and the survey, a template for log file data collection was developed by ISKME and Curriki to collect additional user data including how users navigate through the Curriki site, what they click on, and what specific actions they take. The template identified the types and parameters of data to be collected, how each data point could be captured, and the priority level for each so that internal staff and technology resources could be allocated to each. Although the log file data collection template was not implemented due to timing and resource constraints, the process of creating the template served as insight into the people and technology requirements for Curriki’s log file data collection efforts. It also illuminated a fuller range of possibilities with regards to the types of log file data that could and should be collected in order to understand issues around use and user engagement.

It is important to note that this case study does not attempt to provide a comprehensive assessment of Curriki user experiences and perceptions; instead it aims to explore some of the ways a particular group of users—i.e., the survey and interview participants—perceive, use, and experience the Curriki site, its resources, and its tools. The findings below are a presentation of the learnings gleaned through analysis of the interview and survey data combined.

Findings

As noted in the introduction, Curriki aims to provide free resources to its users, and to inspire them to develop resources, collaborate with other users and share ideas. Curriki offers several tools to support these activities. To begin with, users are able to create a personal profile, which they can link to bookmarked or contributed resources through their “My Curriki” page. Curriki users can also create groups, or networks of users, allowing them to connect with others who have similar interests and curriculum needs. In terms of finding and working with resources, users can view and edit resources online, or they can download them to their local computers and edit, modify, and reuse them. Curriki also provides a curriculum-building tool (called the Currikulum Builder) to guide users as they develop and contribute resources to the site. Users have the option of either creating new curriculum materials or using existing templates. Finally, to support quality control of the resources, Curriki users (as well as Curriki staff members, and volunteers) continuously review and provide feedback on the resources contributed.

Through analysis and presentation of the survey and interview data, the sections below illuminate how often and in what ways Curriki users visit Curriki and draw upon its resources and tools, as well as the incentives and disincentives for doing so.

How Often Users Visit Curriki

Survey respondents indicated how often they visit the Curriki site, and in this sense self identified as active (daily or weekly visits), intermediate (1-2 times per month), infrequent (2-3 times in the prior six months) or new (first time) users. As revealed in Table 1 below, the majority of users surveyed identified themselves as either infrequent (42 percent, or 23 respondents) or new (27 percent, or 15 respondents) Curriki users. The remaining survey respondents identified themselves as either active (16 percent, or 9 respondents) or intermediate (15 percent, or 8 respondents) users.

Table 1. Breakdown of Survey Participants by Frequency of Use

Not revealed in Table 1, however, is that when asked to indicate whether they intend to return to Curriki in the future, 70 percent (29) of respondents indicated that they intend to do so.

Why Users Come to Curriki

As revealed in Table 2 below, the most common reasons that users visit Curriki are to get ideas that inform their own lessons (51 percent, or 25 respondents) and to improve their teaching methods (41 percent, or 20 respondents). Curriki users are also taking advantage of what Curriki and OER offer more generally by, e.g., coming to Curriki to contribute resources (33 percent, or 16 respondents), to connect with other teachers and learners (25 percent, or 12 respondents), and to supplement their own lessons with Curriki resources (20 percent, or 10 respondents).

Table 2. Reasons for Visiting Curriki

In examining the reasons for visiting Curriki by the frequency of use (active, intermediate, infrequent and new), statistically significant results surfaced for two of the reasons: contributing resources and connecting with teachers or learners with similar interests. Specifically, the chi- square analysis revealed that active users were much more likely to visit Curriki to contribute resources than intermediate, infrequent or new users. Furthermore, the analysis indicated that new users were more likely to visit Curriki to connect with teachers or learners with similar interests than infrequent, active or intermediate users.

How Users Engage with Curriki’s Resources and Tools

Table 3 below provides a breakdown of the ways that users are engaging with Curriki resources. As revealed, the most common types of engagement with Curriki resources are viewing (71 percent, or 32 respondents) and downloading (44 percent, or 20 respondents) resources; however, the survey also showed that users are sharing (38 percent, or 17 respondents) and editing/remixing (27 percent, or 12 respondents) Curriki resources—although to a lesser extent.

Table 3. Types of Engagement with Curriki Resources

Also revealed in Table 3 is that editing content occurs more frequently on respondents’ own, local computers than it does within the Curriki system. However, this distinction was mitigated when looking specifically to how respondents contribute content to Curriki: 34 percent (15) of respondents contribute content to Curriki by uploading files from their local computers, and a similar proportion, 30 percent (13), do so through Curriki’s Currikulum Builder tool.

Further, the chi-square analysis indicated that content contributors are primarily active users who visit Curriki either daily or weekly. Specifically, active users surveyed were more likely to indicate that they had contributed content by uploading files from their local computers compared to intermediate or infrequent users. Similar differences were found in analyzing the utilization of the Currikulum Builder tool: active users were much more likely to have contributed content in this way, compared to intermediate and infrequent users.

Incentives and Disincentives to Use

As noted above, the survey data revealed that Curriki users visit the site for a multitude of reasons—from getting ideas for new lessons, to contributing content, to connecting with teachers and learners. The latter of these—connecting with teachers and learners—was further supported by the interview data as an important incentive to Curriki use. The interview participants indicated that these teacher/learner connections include the ability to view and track the resources tied to specific, trusted users as a way to find “good” resources. As one participant stated: “Sometimes if you find a teacher you know who’s working on materials, and you know they are good, you can keep going back to look through their stuff”. The participant further indicated an interest in knowing whether and how one’s own resource contributions were being viewed, modified, or downloaded by other users, to ensure that they are useful to them.

Another interview participant, a professional development trainer, explained how Curriki served as a classroom tool to facilitate teacher-to-student and student-to-student sharing of course materials within work groups. By creating a group within Curriki specifically for class, and requiring students to create Curriki accounts and join the group, the participant used Curriki to store and share course materials. The participant explained that though the original intention was to avoid printing out hard copies of the materials, there was another benefit--that is, students quickly became engaged in Curriki to access materials and to browse and share content with each other.

As revealed in Table 4 below, analysis of the survey data pointed to disincentives to Curriki use. Specifically, when asked about the challenges they potentially faced in using Curriki’s tools and resources, survey respondents indicated several difficulties, primarily in terms of the organization and layout of the Curriki site (62 percent, or 28 respondents); in finding, viewing, using, sharing or creating resources (53 percent, or 24 respondents); and in knowing if resources were of high quality (40 percent, or 18 respondents).

Table 4. Challenges in Using Curriki’s Tools and Resources

Also revealed in Table 4 is that 24 percent (11) of respondents cited “other” challenges to using Curriki’s resources and tools. Survey respondents were able to list specific challenges when choosing “other” as a response, and these included difficulties finding resources targeted toward specific subjects or grade levels, or that could be adapted to unique situations (such as a French classroom).

Analysis of the interview data pointed to additional barriers to use, including the participants’ lack of access to technology resources in their schools. One interview participant indicated an interest in requiring students to create Curriki accounts to access materials. However, limited access to computers in the classroom and uncertainty about whether students had computers at home dissuaded the participant. Another interview participant reported a dearth in support by school administrators in terms of offering the tools and trainings needed to facilitate heightened use of Curriki resources (as well as other freely available, open educational resources) in the classroom. These findings were underscored by a survey item revealing that only 9 percent (4) of survey respondents had used Curriki resources in the classroom.

Discussion and Conclusions

The findings from the case study highlight opportunities for Curriki for supporting use and user engagement around its resources, its tools, and the site overall. To begin with, given that the most commonly cited reason for visiting Curriki was to get ideas for new lessons, exploring ways to facilitate the continuous addition and enrichment of resources on the site becomes central.

Additionally, given that remixing, editing, and using resources in the classroom were somewhat less prevalent behaviors for the users surveyed—and that it was primarily active users who were contributing resources—finding ways to support those who may want to engage in these more “advanced” activities, but who may lack the knowledge or supports to do so becomes central. Importantly, these supports should potentially move beyond mitigating design hurdles related to the organization and layout of the Curriki site or to technology hurdles in terms of finding, viewing, using, and creating resources. That is, in light of the interview data that revealed teachers’ lack of institutional and training supports at their local institutions as an obstacle to OER use and engagement, these supports should also potentially include mechanisms to facilitate OER engagement in the context of (and as a supplement to) teachers’ local teaching and learning environments.

Finally, the case study pointed to the importance of interactions between users. Specifically, in light of the findings that teacher and learner connections are an incentive to use, and that first time visitors were most likely to come to Curriki to connect to other teachers and learners, there is a need to more fully understand the ways that teachers are or could be connecting with one another through the site. Doing so can serve to support the creation of additional incentives and mechanisms to continue to attract new users, as well as to support existing users.

On the whole, the findings suggest that moving toward increasingly active use and enhanced user engagement around Curriki’s resources and tools necessitates further exploration into existing use behaviors. Beyond conducting additional surveys and interviews directly with users, this also potentially includes user log file data collection and analysis as a way to track and understand how users navigate through the Curriki site, what they click on, and what specific actions they take. For OER projects more generally, ongoing research on user behaviors, experiences and perceptions can be a challenging and resource-intense process; however, by assessing and building data collection mechanisms and research questions into organizational practices, knowledge and learnings can be cultivated to inform how users are best supported, as well as to inform continuous improvement for the projects overall.

Acknowledgements

This case study report was written by Lisa Petrides and Cynthia Jimes, with support from Thad Nodine, Anastasia Karaglani, and Clare Mortensen. The case study initiative was funded by the Shuttleworth Foundation, the International Development Research Centre, The William and Flora Hewlett Foundation, and Curriki. The initiative was facilitated with support from iCommons.

References

Fetterman, D. (2005). "A Window into the Heart and Soul of Empowerment Evaluation: Looking Through the Lens of Empowerment Evaluation Principals." In Empowerment Evaluation Principles in Practice. Ed. David M. Fetterman and Abraham Wandersman. New York: The Gilford Press.

Petrides, L. and Loesch-Griffin, D. (2005). “Strengthening Evaluation for Continuous Improvement Through Knowledge Management.” Working Paper, Institute for the Study of Knowledge Management in Education, Half Moon Bay, CA.

Notes

[1] Because of the relatively low number of survey participants, the findings from the chi square analyses should be considered as exploratory in nature. They are presented in this report to provide initial insights and generate questions for future research.

Links to Other Case Study Project Resources

OER Case Study Project

Individual Case Study Reports

• Building Open Educational Resources from the Ground Up: South Africa’s Free High School Science Texts
• Curriki: Facilitating Use and User Engagement Around Open Educational Resources
• CurriculumNet: Creating Freely Available Curriculum Materials to Meet Uganda’s Growing Student Population
• Mission 2007 Training Commons: Developing a Living Curriculum for Telecentre Workers in India
• Stanford Encyclopedia of Philosophy: Toward a Sustainable Funding Model for Open Access
• WGBH’s Teachers’ Domain: Producing Open Materials and Engaging Users

Case Study White Paper

The case study white paper is a cross-case comparison of the individual case studies listed above.

• Creating, Doing, and Sustaining OER: Lessons from Six Open Educational Resource Projects

Case Study Framework

This case study framework seeks to assist OER projects in tracking, analyzing and sharing key developments in the creation, use and reuse of open educational resources. Based on the case studies of each of the six projects above, the framework provides research tools, reflective questions, and examples that OER projects can draw upon in conducing their own case studies.

• OER Case Study Framework