Applying the E-learning Framework: Evaluating an E-learning Course toward the Improvement of Quality of ODeL Programs

Myra D. Oruga¹, Jelaine R. Bagos²

¹[email protected], ²[email protected]
Faculty of Management and Development Studies, University of the Philippines Open University, Los Banos, Laguna, Philippines

Abstract

Keywords: Evaluation, e-learning course, international health students, health promotion, distance education

Introduction

Open and Distance E-learning (ODeL)

Open and distance e-learning can be viewed as an innovative approach for delivering well-designed, learner-centered, interactive, and facilitated learning environment to anyone, anyplace, anytime, by utilising the attributes and resources of various digital technologies along with other forms of learning materials suited for open and distributed learning environment.

Evaluation in ODeL

Success in an e-learning system involves a systematic process of planning, designing, implementing, and of course, evaluating the online learning environments where learning is actively fostered and supported. According to Land and Hannafin (1996), the more open the learning environment, the more complex the planning, management, and evaluation of it.

Evaluation of an e-learning course or program is necessary to “buy” better understanding of how the program or a specific e-learning course is working – feedback. It can help make decisions on operations as it provides factual basis for corrective adjustments. Thus, it helps in improving the course and creating more meaningful learning environment (Khan, 2010; Torres and and Velasco, 2005; Attwell, 2006)

Khan’s Eight-Dimensional E-learning Framework

The study utilized Khan’s Eight-Dimensional E-Learning Framework as it is one of the most widely used frameworks when it comes to designing, developing, implementing, and evaluating an open and distributed learning systems and e-learning programs or courses (Doyle, n.d.). Several studies conducted to review e-learning programs, resources and tools (Khan, 2007; Khan & Smith, 2007; Romiszowski, 2004; Singh, 2003; Chin & Kon, 2003; Kuchi, Gardner, & Tipton, 2003; Mello, 2002; Barry, 2002; Goodear, 2001; Khan, Waddill, & McDonald, 2001; Dabbagh, Bannan-Ritland, & Silc, 2001; Khan & Ealy, 2001; El-Tigi & Khan, 2001) found the various issues within the eight dimensions of the framework useful.

Khan (2010) have long been communicating with learners, instructors, trainers, administrators, and technical and other support services staff involved in e-learning in both academic (K12 and higher education) and corporate settings from all over the world. He has researched critical e-learning issues discussed in professional discussion forums, and have designed and taught online courses. He also reviewed literatures on e-learning. As the editor of Web-Based Instruction (1997), Web-Based Training (2001), and Flexible Learning (2007), he had the opportunity to work closely on e-learning issues with about two hundred authors from all over the world who contributed chapters in these books (Khan, 2010).

The e-learning framework encompasses various online learning issues, including: pedagogical, technological, interface design, evaluation, management, resource support, ethical and institutional needed in the planning, implementation and evaluation of the e-learning system. It can provide guidance in (Khan, 2010 in Doyle, n.d.):

• planning and designing e-learning and blended-learning materials (e.g., online courses, MOOCs, mobile learning);
• organizing resources for e-learning environment and blended-learning materials;
• designing distributed learning systems, corporate universities, virtual universities and cyberschools;
• designing LMS, LCMS and comprehensive authoring systems;
• evaluating e-learning, blended-learning courses, and programs; and
• evaluating e-learning authoring tools/systems, LMS and LCMS.

The study applied the Evaluation dimension of the framework as it will only evaluate one of the e-learning courses of the Diploma in/Master of International Health Program of the Faculty of Management and Development Studies of the UP Open University. Specifically, the study looked at the following variables: Reflexivity, Reflective Thinking, Interactivity, Tutor Support, Peer Support, and Interpretation.

Objectives

Anchored on the Evaluation Dimension of Khan’s E-learning Framework, the study aimed to evaluate one of the e-learning courses of the Faculty of Management and Development Studies’ Diploma in/Master of International Health Program—IH 213 (Health Promotion for Equity and Sustainable Development) course.

Specifically, the study sought to:

1. Identify the relevance of the e-learning course to the respondents;
2. Determine the respondents’ reflective thinking as they take the e-learning course;
3. Assess the respondents’ level of interactivity in the e-learning course;
4. Evaluate tutor support in the e-learning course;
5. Assess peer support in the e-learning course; and
6. Analyze respondents’ interpretation of their instructors’ and co-learners’ messages as they exchange ideas/thoughts regarding the topics in the course.

Conceptual/Theoretical Framework

Figure 1. Framework of the study

As can be seen in the framework of the study (Figure 1), the first section was adapted from the Evaluation Dimension of Khan’s E-learning framework. It includes Assessment of Learners, Evaluation of the Instruction, and Evaluation of the Learning Environment. Under Assessment of Learners are the constructs: Relevance and Reflective Thinking. Evaluation of the Instruction is composed of Tutor Support, while Evaluation of the Learning Environment includes Interactivity, Peer Support, and Interpretation.

Based on the results of the evaluation, the e-learning course can either be effective or ineffective as perceived by the students or the respondents.

Methodology

The study used the survey research design to gather data from the respondents. It was deemed appropriate, as the study aimed to obtain evaluation. According to Librero (1996) a survey is done by collecting data that will later on be utilized as basis in evaluating a certain phenomenon.

Thirty-three (33) international health learners enrolled in the course during the 2nd semester of A.Y. 2016-2017 served as the respondents of the study. An online evaluation questionnaire created via google forms was used as data gathering instrument. It is composed of 24 favorable statements answerable by Almost Never, Seldom, Sometimes, Often, and Almost Always. There are four statements per construct—Relevance, Reflective Thinking, Tutor Support, Interactivity, Peer Support, and Interpretation. The statements were adopted from the Moodle Self Assessment Tool embedded in the MyPortal course site. As stated in an article written by Frederic Nevers, it is a useful tool for students’ self assessment. An assessment tool is important in effective teaching and learning (Nevers, 2011). Nevers has been teaching using moodle and he found the tool to be beneficial for both students and teachers as it helps improve the teaching and learning styles, among others. The University of New South Wales has also been using this tool to their students ("MOOCS @ UNSW", n.d.).

Prior to the use of the tool in the study, the researchers sought permission from the moodle administrator via email. Before the actual use of the questionnaire, pretesting was also done to determine its effectiveness. A total of 15 students who were not part of the actual respondents of the study answered the questionnaire. Data from the pretest were then subjected to reliability analysis using the cronbach’s alpha test obtaining a value of 94.7516% which indicates that the questionnaire is reliable. This is above the acceptable level thus implying high internal consistency on the question constructs. Their feedbacks regarding the questionnaire were also asked qualitatively. They have revealed that the question format, wording, and order were all clear and that the response options were exhaustive. None of the respondents have relayed negative comments.

After checking the questionnaire’s validity and reliability, the link to the online questionnaire was made available in the course site for IH 213 students’ access. To eliminate response bias, the study also ensured anonymity and confidentiality of the responses. Other ethical issues such as informed consent, beneficence, and non-maleficence were also taken into consideration.

Data from the online questionnaires were analyzed using descriptive statistics such as frequency counts and percentages and measures of central tendency (median, mode).

Clearance from the ethical review board of the FMDS was sought prior to the implementation of the research.

Results and Discussions

This part of the paper presents the results and discussion of the study.

Relevance

Figure 2. Distribution of reactions to the ‘relevance’ statements

One domain that must be considered in evaluating an e-learning course is Relevance. The more the students find their learnings in a specific course relevant to their lives or their professional work, the more it is likely for the e-learning course to be effective. Figure 2 shows that majority of the students often or almost always find that their learnings in the course are important to their work or professional practice and that their learnings focus on issues that interest them. Such findings suggest that the e-learning course is perceived to be effective in terms of relevance.

Reflective Thinking

Figure 3. Distribution of reactions to the ‘reflective thinking’ statements

Kaveti (2012) defined Reflective thinking as “conscious thinking by examining or absorbing an issue which triggers a past experience and helps to gain fresh insights into a new knowledge or skill. It enables learners to build on their prior knowledge. It plays an important role in personal development programs and in understanding the experiential learning process. Before reacting to a situation, reflective learners think about new information and they try to figure out a solution to a problem on their own.” Figure 3 reveals that most of the respondents almost always or often think critically about how they learn, about their own ideas, about the other students’ ideas, as well as the ideas in the readings. Thus, in terms of Reflective Thinking, the course is perceived to be effective by the students.

Interactivity

Figure 4. Distribution of reactions to the ‘interactivity’ statements

Interactivity plays an important role in e-learning. It is a key element of the actual eLearning course design process, and it has proven to be a practice that adds outstanding value to the eLearning course. It involves forms of action or reaction on learners’ behalf, in order for them to achieve results or reach a conclusion.

Three out of the 4 statements under this construct were answered sometimes by majority of the respondents as shown in Figure 4. They only sometimes ask other students to explain their ideas. Their classmates also only sometimes ask them to explain their ideas and only sometimes respond to their ideas. This shows that interactivity is a bit low in the e-learning course. Students must be encouraged to interact more with their classmates and give feedback to their co-learners’ ideas in order to have fruitful discussions and learn from each other.

Tutor Support

Figure 5. Distribution of reactions to the ‘tutor support’ statements

As can be seen in Figure 5, most of the students responded that often or almost always, the tutor stimulates their thinking, encourages them to participate, and models good discourse and critical self-reflection. This suggests how well the tutor also known as the faculty-in-charge designed and executed the course. According to McPherson, Nunes, and Zafeiriou (2003), online tutoring and leadership has been widely considered as a crucial factor in the success of computer-mediated collaborative learning activities.

Peer Support

Figure 6. Distribution of reactions to the ‘peer support’ statements

Another construct is Peer Support. Figure 6 shows that majority of their classmates often encourage their participation, praise and value their contribution, and empathise with their struggle to learn. They motivate one another to learn and participate more. Thus, the course is perceived to be effective in terms of peer support.

Interpretation

Figure 7. Distribution of reactions to the ‘interpretation’ statements

Figure 7 shows that often the students make good sense of other students’ messages and their tutors’ messages, likewise, their classmates and the tutor also make good sense of their messages. Thus, in terms of interpretation, the course is also perceived to be effective.

Conclusions and Recommendations

Generally, the e-learning course was perceived to be effective in terms of Relevance, Reflective Thinking, Tutor Support, Peer Support, and Interpretation as most of the respondents answered often or almost always in all the statements under the said constructs. Only the statements under Interactivity were mostly answered sometimes by the respondent, thus, the e-learning course can still be improved in terms of interactivity in the virtual classroom.

Based on the results, the study recommends the following:

• To encourage interactivity in the e-learning course, the faculty-in-charge or FIC of the cours can add more discussion forums that are graded or have incentives. Activities such as debates and other group works that require students to respond to their classmates’ ideas can also be done.
• The evaluation tool can also be used in other courses other than IH213.
• Evaluation can also be done university-wide, or program wide per academic year to find out the trend.
• Socio-demographic characteristics of the respondents can serve as intervening variables in future researches. Correlation between the socio demographic characteristics and the evaluation dimension can be explored to know whether it affects their responses.

References

Attwell, G. (2006). Evaluating E-learning: A Guide to the Evaluation of E-learning. Retrieved from http://pontydysgu.org/wp-content/uploads/2007/11/eva_europe_vol2_prefinal.pdf.

Barry, B. (2002). ISD and the E-Learning Framework. Retrieved from http://www.wit.ie/ library/ webct/isd.html.

Chin, K. L. & Kon, P. N. (2003). Key Factors for a Fully Online E-learning Mode: A Delphi Study. Retrieved from http://asianvu.com/digital-library/elearning/chin-kon.pdf.

H. Dabbagh, Nada & Bannan, Brenda & Flannery Silc, Kate. (2001). Pedagogy and Web-based course authoring tools: issues and implications. 343-354.

Doyle, R. (n.d.). E-Learning Framework for Massive Open Online Courses (MOOCs). Retrieved from http://asianvu.com/bk/framework/?page_id=171.

El-Tigi, M. A., & Khan, B. H. (2001). Web-based Learning Resources. Retrieved from http:// bookstoread.com/bestseller/khan/tocwbt.htm.

Goodear, L. (2001). Cultural Diversity and Flexible Learning. Retrieved from http://asianvu.com/ digital-library/elearning/flexible_leaders_worldwide_by_goodear.pdf.

Kaveti, V. (2012). Reflective Learning and its Benefits. Retrieved from https://blog.commlabindia. com/elearning-design/reflective-learning-benefits.

Khan, B.H. (2010). The Global E-learning Framework. Retrieved from http://iowaascd.org/files/ 4214/6679/6558/A_Framework_for_Web-Based_Learning.pdf.

Khan, B.H. (2007). Flexible Learning in an Information Society. Retrieved from http://tojde. anadolu.edu.tr/yonetim/icerik/makaleler/326-published.pdf.

Khan, B. H., & Ealy, D. (2001). A Framework for Web-Based Authoring Systems. Retrieved from http://asianvu.com/digital-library/elearning/webbased_authoring_systems.pdf.

Khan, B. H., Smith, H. L. (2007). Student Evaluation Instrument for Online Programs. Retrieved from http://asianvu.com/digital-library/elearning/student_evaluation_instrument.pdf.

Khan, B. H., Waddill, D. & McDonald, J. (2001). Review of Web-Based Training Sites. In B. H.

Khan (Ed.). (2001). Web-based training (pp. 367-374). Englewood Cliffs NJ: Educational Technology Publications.

Kuchi, R., Gardner, R. & Tipton, R. (2003). A Learning Framework for Information Literacy and Library Instruction Programs at Rutgers University Libraries. Retrieved from https:// www.libraries.rutgers.edu/rul/staff/groups/instructional_services/reports/learning_ frameworks.shtml.

Land, S., & Hannafin, M. J. (1996). A conceptual framework for the development of theories-in- action with open learning environments. Educational Technology Research and Development, 44(3), 37–53. doi: 10.1007/BF02300424.

Librero, F. (1996). How to Write a Thesis Proposal: Some Practical Guidelines [3rd ed.]. College Laguna: College of Agriculture Publications Office, UPLB.

McPherson, M.A., Nunes, J.M., and Zafeiriou, G. (2003) "New Tutoring Skills for Online Learning: Are e-Tutors adequately prepared for e-learning delivery?" In Proceedings of EDEN 2003 The Quality Dialogue; Integrating Quality Cultures in Flexible, Distance and e-learning, 15-18 June 2003, Rodos Palace Hotel, Rhodes, Greece, 347-350.

Meerkat. (n.d.). Retrieved from http://aicml7.cs.ualberta.ca/?q=node/41.

Mello, R. (2002, June). 100 Pounds of Potatoes in a 25-Pound Sack: Stress, Frustration, and Learning in the Virtual Classroom. Teaching With Technology Today, 8(9). Retrieved from http://www.elearningmag.com/elearning/article/ articleDetail.jsp?id=2031.

MOOCS at UNSW. (n.d.). Retrieved from https://teaching.unsw.edu.au/moocs.

Naveh , Z. (1999). Transdisciplinary challenges for landscape ecology facing the post-industrial information society. Proceedings of “Nature and Culture in Landscape Ecology, Experience for the 3rd Millennium.”

Naveh, Z. & Lieberman, A.S. (1984). Landscape ecology: Theory and application. New York: Springer.

Nevers, F. (2011, January). A Useful Tool for Self-Assessment. Retrieved from http://www. iteachwithmoodle.com/2011/01/19/checklist-a-useful-tool-for-self-assessment/.

Obrero, R.J. (2016). Code for netlogo rumor mill simulation of permaculture practitioners on Facebook. Unpublished.

Odum, E.P. (1971). Fundamentals of ecology. Philadelpia: Saunders.

Odum, H.T. (1995). Self-organization and maximum empower. In C. A. S. Hall (Ed.), Maximum Power: The Ideas and Applications of H. T. Odum. Colorado: Colorado University Press.

Padmavathy K. & G. Poyyamoli. (2011). Alternative farming techniques for sustainable food production. In E. Lichtfouse (Ed.), Genetics, Biofuels, and Local farming System. New York: Springer.

Peeters, B. (2011). Permaculture as alternative agriculture. Kasarinlan: Philippine Journal of Third World Studies 2011, 1(2), 422-434.

Pickett, S.T.A. & M.L. Cadenasso. (1995). Landscape ecology: Spatial heterogeneity in ecological systems. Science New Series, 269(5222), 331-334.

Romiszowski, A. J. (2004). How’s the elearning baby? Factors leading to success or failure of an educational technology innovation. Educational Technology, 44(1), 5-27.

Singh, H. (2003). Building Effective Blended Learning Programs. Educational Technology, 44(1), 5-27.

Sanderson, J. & L. D. Harris (eds.). (2000). Landscape ecology: A top-down approach. USA: Lewis Publishers.

Schreck, A., C. Getz, & G. Feenstra. (2012). Social sustainability, farm labor, and organic agricultures: Findings from an exploratory analysis. University of California, Department of EPSM 207, Agriculture and Human Values.

Scott, R. (2010). A critical review of permaculture in the United States – Self-Published Article. Education Policy Studies, University of Illinois. Retrieved from http://robscott.net/2010/ comments/.

Torres, C.S., Velasco, M.T.H. (2005). Participatory Monitoring and Evaluation. ASEAN Sub-Region, BATUWSM Programme, BATU and the College of Development Communication, University of the Philippines Los Baños

Turner, M.G. (1989). Landscape ecology the effect of pattern on process. Annu. Rev. Ecol. Syst. 1989(20), 171-197.

Turner, M.G. (1990). Spatial and temporal analysis of landscape patterns. Landscape Ecology, 4(1) 21-30. doi: 10.1007/BF02573948.

Use Assessment Tools in Moodle. (n.d.). Retrieved from https://teaching.unsw.edu.au/moodle- assessment-tools.

Wilensky, U. (1997). Netlogo rumor mill model. Retrieved from http://cclnorthwestern.edu/netlogo/ models/RumorMill.

Wilensky, U. (1999). NetLogo. Retrieved from http://cclnorthwestern.edu/netlogo/.

International Journal on Open and Distance e-Learning, Vol. 4 No. 1, June 2018 Issue