Abstract Introduction Objectives Review of Literature Theoretical Framework Methodology Results and Discussion Conclusion References

Evaluation of the UPOU MOOC "Principles of Graphic Design" using Product Variable of Biggs' 3P Model

Ma. Gian Rose Cerdeña1*, Lexter Mangubat2, Luisa Gelisan3, and Mari Anjeli Crisanto4

1Project Staff, University of the Philippines Open University, [email protected]
2Information Systems Research, University of the Philippines Open University, [email protected]
3Assistant Professor, University of the Philippines Open University, [email protected]
4Assistant Professor, University of the Philippines Open University, [email protected]
* Corresponding author

Abstract

As Massive Open Online Courses (MOOCs) are being recognized as a link between higher education and the increasing demand for employability, it is imperative to employ continuous evaluation for quality assurance. This research aimed to evaluate the University of the Philippines Open University (UPOU) MOOC “Principles of Graphic Design” in accordance with the MOOC Quality Guidelines developed by the Commonwealth of Learning. With Biggs’ 3P Model incorporated in the MOOC Quality Guidelines, this evaluation research focused on the product variables of the model: completion/retention and certification rates, and enjoyment and self-satisfaction. The completion and certification rates were computed using LMS analytics, while the enjoyment and self-satisfaction rates were evaluated using a survey instrument based on the learner ratings for the product variable in the Guidelines for Quality Assurance and Accreditation of MOOCs using a five-point Likert scale on outcomes perception, continuing education, employment opportunities, and socializing. With a total of 22,942 enrolled students, 7,620 managed to complete the course and receive certificates. Of these, a total of 7,239 agreed to be survey respondents. Results of the study showed high satisfaction rates, with 77.95% of the respondents giving "very satisfied" rates to the said MOOC. The mean scores for three out of the four learner ratings were above four (4.32 to 4.88), with the socializing component given a neutral rating (3.14 to 3.59). Overall, results showed high completion and certification rates, as well as enjoyment and self-satisfaction ratings for the MOOC.

Keywords: MOOC Evaluation, Biggs’ 3P Model, Enjoyment and self-satisfaction, Completion Rates, Quality Assurance

Introduction

Background of the Study

Massive Open and Online Courses (MOOCs) have emerged as a transformative force in the field of education, breaking down traditional barriers to learning by providing access to quality and curated educational resources and opportunities. MOOCs have been characterized as open, participatory, and distributed (Baturay, 2015). It is open as it offers free enrollment for everyone who has access to the internet. The learning pace is also dictated by the students themselves. Further, Baturay (2015) argued that it is participatory and distributed as “learners may interact with their fellow learners and instructor and participate in the various learning activities prepared” and that “knowledge sharing is encouraged to foster creative thinking in its participants.”

The University of the Philippines Open University (UPOU) pioneered the development and offering of MOOCs in the country since its commencement in 2012 (Almodiel et al., 2020). It uses a Moodle-based platform called MODeL which means Massive and Open Distance e-Learning.

Among the array of MOOC offerings by UPOU, the MOOC titled "Principles of Graphic Design" has garnered substantial interest and active participation from learners. As multimedia production courses continue to attract attention, understanding their quality and effectiveness becomes paramount to optimize their impact and cater to the diverse needs of learners effectively. Therefore, the primary objective of this study is to evaluate the UPOU MOOC "Principles of Graphic Design" using the product variable of Biggs' 3P Model.

Statement of the Problem

In light of the dynamic and expanding landscape of MOOCs, this research sought to comprehensively assess the quality and efficacy of the UPOU MOOC titled "Principles of Graphic Design" through the lens of the product variable of Biggs' 3P Model. The product variable evaluation focused on critical dimensions such as completion/retention and certification rates alongside learners' subjective experiences of enjoyment and self-satisfaction with the course content and delivery.

Objectives

This study evaluated the UPOU MOOC “Principles of Graphic Design” based on completion/retention and certification rates as well as the enjoyment and self-satisfaction of the students.

  1. Evaluate completion/retention and certification rates based on LMS analytics; and
  2. Evaluate students’ enjoyment and self-satisfaction using a survey instrument based on the learner ratings for the product variable in the Guidelines for Quality Assurance and Accreditation of MOOCs.

Significance of the Study

With the increase in MOOC enrolment in UPOU, MOOC quality needs to be regularly evaluated and improved. While measures should be taken to make sure that a MOOC meets certain criteria before it is offered, the MOOC should also be evaluated post-delivery. This study contributes to studies on MOOC quality by presenting the results of the evaluation and by presenting an instrument that could be used for post-delivery evaluation.

Scope and Limitations

This study involved voluntary participants from the MOOC "Principles of Graphic Design" offered from February to March 2023. A total of 7,239 out of the 22,942 enrolled students (31.55%) consented to participate in the study.

Review of Literature

MOOCs

Massive Open Online Courses (MOOCs) are a form of online education that emerged in 2008 and transformed the landscape of learning worldwide. Through these online courses, anyone who has access to the Internet is provided with an affordable and flexible way of learning new knowledge and acquiring new skills as the courses are offered for free. MOOCs also provide learners with opportunities to access high-quality educational content from prestigious institutions without the constraints of geographical boundaries (Yousef et al., 2015).

MOOCs have been referred to as a type of disruptive technology, a major revolution in education, and have four key characteristics (Bates, 2014). These are as follows:

  1. It has infinite scalability (massive) as it can accommodate as many learners as possible.
  2. It has no prerequisites and does not require tuition fees for participants (open). Its only requirement is for the participant to have access to a computer/mobile device and the Internet.
  3. Courses are available online although some institutions have started to implement MOOCs in blended format.
  4. MOOCs are organized in one whole course.

The first MOOC offering is the course “Connectivism and Connectivity.” It was developed in 2008 by Stephen Downes and George Siemens (Bates, 2014). Since then, more than 200 million learners have enrolled in various MOOC offerings, reaching almost 20,000 courses offered by 950 universities all over the world (Shah, 2021).In the Philippines, the University of the Philippines Open University (UPOU) started to develop its own MOOC in 2012. By the following year, the university offered the MOOC “Introduction to Mobile Applications Using the Android Platform” through UPOU’s first MOOC platform called @ral. Bandalaria (2013), the main proponent of MOOCs at UPOU, said that the creation and offering of MOOCs is in line with the university’s public service programs and its advocacies for open learning. The university later developed the MOOC platform, UPOU MODeL, or Massive Open Distance eLearning .

The university’s MOOCs duration ranges from 4 to 6 weeks. The teachers in this MOOC, called course coordinators, evaluate the learners who are qualified to receive certificates for completion. The learner support system is the UPOU MOOC's distinguishing feature. Interactions between learners and course coordinators are also encouraged. These are held not only within the online learning management system but as well as in social media as social media platforms are more accessible to UPOU MODeL learners (Almodiel et al., 2020).

Evaluating the Quality of MOOCs

With MOOCs’ affordances – being offered for free, having no enrollment eligibility requirement, and aiding people to learn new knowledge and/or acquire new skills needed for the changing requirements in the workplace – its growth and development are projected to further propel by 34.54% from 2020 to 2027 (Business Wire, 2022).

Though there is a further increase in MOOC’s popularity, still there are high rates of dropout and non-completion (Aldowah et al., 2020; Badali et al., 2020; Rekha et al., 2023). According to a report on Forbes.com, MOOCs offered by Stanford, MIT, and Harvard have a course completion rate of around 20% (Newton, 2022).

Thus, it is necessary to evaluate MOOCs to gauge their effectiveness, understand their impact on learners, and identify areas for improvement.

Satisfaction in and Completion/Non-Completion of MOOCs

A learner's journey towards completion or non-completion of a MOOC has been the subject of several studies.

The primary factors that cause students to drop out from MOOCs are academic abilities, previous experience, course design, lack of feedback, social presence, and social support. On the other hand, the secondary factors identified were interaction, course and time difficulty, motivation, commitment, and family and work (Badali et al., 2020).

Motivation is also a MOOC completion factor identified in a study conducted in India (Aldowah et al., 2020). Motivation has six main factors that influence participants’ MOOC completion: academic, social, course, personal, professional, and technological. In a systematic literature review of 50 relevant publications conducted in 2022, the influence of motivation for completing MOOCs was further explored (Badali et al., 2020). Two categories of motivations were identified, need-based motivation and interest-based motivation with academic motivations as having the most influence on a participant's retention in a MOOC class.

In terms of continuance to participate in MOOCs, the learner's perceived reputation and perceived openness to a MOOC were shown to be the strongest predictors (Alraimi et al., 2015). In another study, MOOC performance, in terms of knowledge transmission quality, and the learner's preference for MOOCs as a learning mode significantly increases continuance intention for pursuing studies through MOOCs (Dai et al., 2020). Other factors identified to influence continuance and satisfaction are confirmation of learners' expectations of the MOOCs they are enrolled in and the usefulness of the courses, respectively (Rekha et al., 2023; Shah & Khanna, 2022).

Biggs 3P Model

The 3P Model developed by John Biggs, a scholar in the field of educational research and instructional design, has greatly influenced the teaching and learning assessment systems being used today (Barattucci, 2017).

The model was structured along three phases of learning (Biggs, 1993):

  1. Presage, or the phase beginning the teaching-learning process or before learning is produced. There are two groups of variables at play in this stage: the characteristics of the students, i.e., prior knowledge/experiences, cognitive ability or skills, ways of learning and expectations and values and the characteristics of the teaching context, i.e., teaching objectives, teaching methods, assessment methods, and learning space conditions, among others.
  2. Process, or the learning phase itself. It refers to what or how the student accomplishes a task in a specific context to learn. Biggs (1993) stated that how a student pursues a task is affected by one’s perception not only of the task to be done but also of oneself.
  3. Product, or the learning outcomes which are of three types: – quantitative, qualitative, and affective. Quantitative refers to the amount of information and the skills acquired. Qualitative is the complexity of thoughts and knowledge that were developed while affective is the student’s commitment and satisfaction of the process.

Biggs' model assumes that learning outcomes are affected by several factors that interact with each other, thus requiring not only effectiveness and good quality but also compatibility of the components (Barattucci, 2017).

Due to its integrative character, the model is used in research as a framework to ensure that all aspects contribute to a student's learning process, allowing for a better understanding of how the factors influence each other (Allison, 2021; de la Fuente et. al., 2014; Kanashiro et al., 2020; Song, 2018). The 3P model has been used in various contexts such as MOOCs, psychological processes studies, and K-12 computing instruction, among others (Allison, 2021; Ganotice & Chan, 2019; Song, 2018).

Though studies have criticized the 3P Model as outdated, oversimplified, and not the only model that can be used to understand educational context, it appears that it is the “most prominent learning model in higher education” (Allison, 2021; Kanashiro et al., 2020).

Theoretical Framework

The Biggs’ 3P Model used in the Guidelines for Quality Assurance and Accreditation of MOOCs (Commonwealth of Learning, 2016) consists of three groups of variables: presage variables, process variables, and product variables. In Figure 1, it can be seen that these variables are interconnected with each other, as all factors contribute to a student’s learning process.

Figure 1

Biggs’ 3P Model of Student Learning

This research focused on the product variables in its post-delivery evaluation because also evaluating the presage and process variables will result in extensive research given the number of MOOC enrollees. For the product variables, the research considers enjoyment and self-satisfaction, with the completion/ retention and certification rates achieved by the students as variables identified by COL (2016).

Methodology

Research Design

The learning management system (LMS) analytics and data gathered from an online survey were used to evaluate the product variables in this study. Specifically, the LMS analytics used were course participation and activity completion. For the online survey, a Qualtrics form was created with questions about outcomes perception, continuing education, employment opportunities, and socializing variables to determine enjoyment and self-satisfaction.

Respondents of the Study

This study involved voluntary participants of the Principles of Graphic Design MOOC, offered from February to March 2023, who were 18 years old and above. An online questionnaire was given at the end of the class and the learners were able to choose whether they would participate or not. Participants were given informed consent upon accessing the survey instrument. Names and other personal information were not collected, and all data collected were treated with confidentiality.

The target sampling size for the study was 378/22,942 MOOC enrollees. This was calculated with a 95% confidence level and 10% margin of error. Since 7,239 learners participated in the study, the target sampling size was exceeded

Data Gathering Procedure

For this research, data was gathered in two ways:

  1. The completion and certification rates were computed using the LMS analytics; and
  2. The enjoyment and self-satisfaction rates were evaluated using a survey instrument based on the learner ratings for the product variable in the Guidelines for Quality Assurance and Accreditation of MOOCs using a five- point Likert scale: outcomes perception, continuing education, employment opportunities, and socializing.

Qualitative data was also collected through the survey instrument to gain further insight into the evaluation done through the quantitative scales.

Data Analysis

Data was collected for the completion and certification rates using the LMS analytics of course participation and activity completion which identified the number of enrollees in the course as well as course completers. Results for the enjoyment and self-satisfaction rates collected through the Qualtrics survey were analyzed quantitatively using mean and standard deviation and supported with the summary of their qualitative answers using a word cloud. Aside from the learner ratings for the enjoyment and self-satisfaction rates, data collected from their demographics were also analyzed in terms of their frequency distribution which included their age, gender, civil status, location, highest educational attainment, employment status, current program enrollment, and previous MOOC experience.

Results and Discussion

Table 1

Summary of demographic profile of online learners

Out of the 22,942 enrollees, a total of 7,874 were able to access the survey instrument. However, only 7,239 respondents were able to finish the Qualtrics survey on enjoyment and self-satisfaction as some of those who initially accessed the survey were minors (aged 17 and below) and some did not proceed with answering the survey.

For the demographics, learners were classified into eight categories: age, gender, civil status, location, highest educational attainment, employment status, enrollment in a program, and previous MOOC experience. Table 1 shows the summary of the demographic profile of the learners

Table 2

Summary of the highest demographic per learner rating category (+ for positive rating, 0 for neutral rating, - for negative rating)

These data were also analyzed in parallel with their enjoyment and self- satisfaction ratings, and it was found that there were no significant differences in the classification of the learners regarding the rating they were given. The ratings were categorized into positive, neutral, and negative based on their level of satisfaction. As shown in Table 2, the majority of the positive, neutral, and negative ratings were given by nearly the same demographic classification. The only difference was seen under the employment opportunities category, where the employment status for positive ratings was from employed learners and the neutral and negative ratings were from unemployed learners.

Using a five-point Likert scale, with 1 being the lowest and 5 being the highest, learners were asked to rate their enjoyment of the MOOCs course per item. The results were then calculated using mean and standard deviation.

Table 3

Summary of the outcomes perception rating of the learners using mean and standard deviation

Tables 3 to 6 show the mean and standard deviation results for the learner ratings. Table 3 reflects the outcomes perception rating. Each item had mean scores of four or higher, meaning a high enjoyment rating. The item with the highest mean score and low standard deviation was “I will be able to use what I learn in the future”.

Table 4

Summary of the continuing education rating of the learners using mean and standard deviation

Table 4 focuses on the continuing education rating with all items having mean scores of above four, meaning high enjoyment rating. The item with the highest mean score with a low standard deviation was “During learning I treat the resources I find as a starting point and try to develop my own ideas from them”.

Table 5

Summary of the employment opportunities rating of the learners using mean and standard deviation

Table 5 shows the employment opportunities rating results, also with all items having above four mean scores, again meaning high enjoyment rating. The item with the highest mean score under this learner rating with a relatively low standard deviation is “I often think about how my learning fits into the “bigger picture” of my work/practice”.

Table 6

Summary of the socializing rating of the learners using mean and standard deviation