Developing and Evaluating a Website as an OER for Faculty Development

Roberto B. Figueroa Jr.¹,², Jennifer Clasara Fajardo³, Siaw Eng Tan⁴, and Insung Jung⁵

¹Doctoral Candidate, International Christian University, Japan, [email protected]
²Assistant Professor, University of the Philippines, Philippines, [email protected]
³Doctoral Candidate, International Christian University, Japan, [email protected]
⁴Doctoral Candidate, International Christian University, Japan, [email protected]
⁵Professor, International Christian University, Japan, [email protected]

Abstract

Keywords: faculty development, liberal arts education, online materials, open educational resources, web media

Introduction

Faculty Development

Faculty development (FD), which started in the United States, was brought to Japan in the late 1990s and was officially promoted by the Japanese government when the Standards for Establishment of Universities highlighted the importance of FD in 2007 (Yuan & Shimizu, 2007). Many studies have revealed positive effects of FD programs on students’ satisfaction (Shea, Fredericksen, Pickett, & Pelz, 2003), students’ achievement (Naeem, van der Vleuten, & Alfaris, 2012), and faculty members’ wellbeing (Jung, Nishimura, & Sasao, 2016).

Baker, Lunsford, and Pifer (2015) stated that faculty members in liberal arts colleges need specialized FD training that reflects the organizational missions and their own needs. Especially in small liberal arts colleges where undergraduate teaching is considered most important, it is imperative to offer orientation programs for new faculty members who were trained mainly in large research universities. As such, they often face difficulties in understanding teaching for liberal arts education; this is particularly pertinent in the East Asian context, where many faculty members are research-oriented especially in East Asia (Jung, et al., 2016).

Despite the increased needs for FD programs in Japan, especially in the areas of innovative teaching methods and Information and Communication Technology (ICT) utilization, the execution rate of implementation of such FD programs in Japan has not been improved (Taguchi, Nishimori, Shinto, Nakamura, & Nakahara, 2006). It was also the case at the International Christian University (ICU).

Program Overview

To address the urgent need for new faculty members’ development in ICU, a project aimed to design, implement, and evaluate a new FD program for liberal arts education was planned. The new FD program had the following aims for the participants: (1) develop a better understanding of the core values of liberal arts education in relation to their respective subject areas; (2) apply a systematic or systemic instructional design model in developing a syllabus, which integrates innovative teaching methods and technologies for liberal arts education; and, (3) clarify and balance their professional responsibilities (teaching, research, and social and administrative services) and personal well-being. In developing the new FD program, the project team has adopted the Analysis, Design, Development, Implementation, and Evaluation (ADDIE) model, a macro instructional design model. In the first year of the project (July 1, 2016 – June 30, 2017), the team carried out Analysis, Design, and Development (ADD) activities, and in the second year (July 1, 2017 – June 30, 2018), the team carried out activities related to Implementation and Evaluation.

To achieve these objectives, 20 modules were developed and included in the new FD program. The modules were created based on the results of needs assessment conducted with the faculty members of the ICU community. The modules were grouped into three: R-Modules, I-Modules, and T-Modules. R-Modules (Reception Modules) comprise topics on ICU’s administrative offices and its functions. I-Modules (Information Modules) comprise of other offices and their roles that cover topics like promotion and tenure, teaching effectiveness, faculty support, student counselling, human rights, and other ICU rules and regulations. T-Modules (Teaching Modules) comprise topics on ICU’s liberal arts education values, history, and mission, as well as various teaching strategies. These modules also include communication in English, integration of innovative methods and technologies such as Moodle, Google tools, active learning, open educational resources (OER), and flipped learning.

Delivery Modes

To ensure access to various FD resources at a time and place convenient for them, three delivery modes were employed: 1) blended (face-to-face [or f2f] sessions supported by Moodle); 2) online (new faculty website providing reading and video resources); and, 3) mobile (mobile app providing text, audio, video, and website materials).

In the blended mode, 20 f2f sessions were offered in 10 weeks. The Moodle was used to post resources, send facilitator messages and reminders, and upload participants’ reflection notes. In the online mode, the new faculty website was used to access resources for personal study or pre- session viewing and open access to the public through a Creative Commons license.

Ideological and Theoretical Bases

Various concepts and best practice ideologies, which were well supported by literature, were applied during the implementation of the program. The following provide the bases of the program’s approach.

Open Educational Materials

Since the existence of open educational resources (OER), educators and even non-educators have been given other options to use in their classes or for personal use. Various types of OER exist; OER can be: full courses, course materials, modules, learning objects, textbooks, streamed videos, software, tests, assignments, case studies, e-portfolio, training materials, practice items, etc. (Jung & Hong, 2016). Moreover, OER can be categorized based on the media and/or technology being used. Although initially intended for support at formal education, OERs’ use has also extended to informal and non-formal education.

Program Duration Ideologies

Several ideologies were referenced in the design, planning, and implementation of the project. Community-building is important to develop a personal sense of connection and commitment among the people in the environment (Unger & Wandesman, 1985). It is also beneficial to the organizer that longer-term programs enable the organizer to carry out observation, feedback, follow-up support, and evaluation on the effectiveness of the program, as suggested by Garrison and Vaughan (2008).

Blended Learning and Flipped Learning

The program is carried out in blended learning mode (i.e., flipped learning), as this approach can integrate both online and f2f sessions to increase diversification in offering formats, flexibility, and accessibility (Diaz, Garrett, Kinley, Moore, Schwartz, & Kohrman, 2009, Zainuddin & Perera, 2017). The new faculty members could adopt a self-study approach by reviewing the OER content on the website that allows them to access via a personal computer or mobile device. This is a guided independent learning approach where the participants are also an active contributor, constructing knowledge on their own and with the learning environment. According to the socio- constructivist theory of learning, the facilitator does not lecture at them but assumes the role of guiding the new faculty via the f2f session through group discussions and collaborative activities.

Also, there are immense benefits of previewing the content before attending the f2f session, as faculty members could grasp the idea of the topic for deeper understanding and develop a greater sense of autonomy through self-paced learning (Zainuddin & Perera, 2017, Choi & Lee, 2015). Furthermore, a flipped learning strategy could develop a higher sense of preparedness (Zainuddin & Perera, 2017), where faculty members are encouraged to bring their questions and participate in discussions or exchange ideas effectively during the f2f session (McLaughlin, White, Khanova, & Yuriev, 2016). Past literature shows that flipped learning can accommodate various instructional strategies, formats, and modalities in online content delivery. Video lectures, e-reading materials, and screencasts were among the instructional strategies employed by instructors in the online mode (Strayer, 2002). Similarly, for in-class activities, instructors can adopt various active learning strategies, such as small group discussion, problem-based learning, peer tutoring, or case studies. Given the positive aspects of this approach, the FD program is designed in the flipped learning format in its implementation.

A case study of an FD program reported by Paskevicius & Bortolin (2016) was designed and implemented at Vancouver Island University. It also utilizes blended learning of online and f2f methods. The online platform used was Desire2learn, a learning management system to coordinate online reading materials, activities, or discussions. The online activities were complementing the f2f session to prompt discussion and collaborative activities. The FD program is designed on the concept of building a community of practice (CoP) in fostering idea-sharing, knowledge, and expertise among faculty members. Feedback from the survey revealed that blended learning is suitable to support longer-term FD programs with sustained online activities and regular f2f meetings. Faculty members also highlighted that through this program, they forged interdisciplinary relations across different fields of expertise while engaging in meaningful conversations.

Methodology

Choice of Technologies

The materials that were used for the FD program were decided to be created in electronic format. It was decided by the team that a website will mainly contain all of the reading and multimedia materials that will be used in most of the T-Modules, which were meant to be implemented as flipped learning sessions, as well as the I-Modules and R-modules, which were meant to be implemented as regular blended learning sessions. For simplicity and sustainability in terms of maintenance, the new version of Google Sites was chosen as the platform to hold and organize all of the online materials. A mobile app for Android Smartphones and iPhones was also developed. Furthermore, eBooks produced in ePUB format were also linked within the apps as well as the website. This paper describes the activities involved in the iterative development and evaluation of the website.

Materials Development

The modules were assigned to various experts both within ICU and in other institutions. The modules were reviewed by the team before being submitted for webpage conversion. Video and audio materials were also developed to give other modalities of learning. In total, it took almost six months to provide the complete materials for the website. Many of the materials were also converted into eBooks using an ePub management tool called Calibre.

Website Organization and Initial Testing

The website’s organizational structure underwent several iterations through rapid prototyping since May 2017. For each of the five iterations, the members of the team including some external reviewers were consulted on the navigational ease, consistency, and format of the materials. The first two iterations involved prototypes while the third involved a pre-beta version and a user- acceptance test. Lastly, the last two iterations involved a closed beta and an open beta version. Figure 1 illustrates the Gantt chart used for the development of the website. A project management software called Basecamp was used to coordinate the various tasks involved among the members of the team and to make sure that the targets are achieved.

Figure 1. Gantt Chart for the New Faculty Website Development

The team also tested all of the links to make sure that everything is working before being initially used in the sessions. Figure 2 shows the front page containing the main links and sections. The articles on the website are mainly divided into Teaching Support, which contains most of the T-Module materials, Research Support, and Administrative Support. A special link to Liberal Arts Education was also placed as it is an extremely important ideology that ICU wants teachers to imbibe in their role as a member of a Liberal Arts institution. All pages on the website contain the Creative Commons License information of CC-BY-NC 3.0, which means that users can freely create a modified version of the website and its OER materials provided that they provide due attribution to the creators. Furthermore, OER can only be redistributed for free.

Figure 2. Frontpage of the New Faculty Website

Usage and Evaluation

ICU’s Center for Teaching and Learning (CTL) implemented the new FD program with five new faculty members in Autumn term 2017 and invited other instructors and faculty members who were interested to join some of the sessions. Inspired by the ideas given by Unger and Wandesman (1985) as well as Garrison and Vaughan (2008), the FD development program was implemented over duration of 10 weeks. The purpose of not having an instant single-day workshop session is to allow time for building a sense of community with facilitators and other faculty members. Besides, the duration of the program allows them to take sufficient time for adaptation and reflection.

The website was used by the new faculty members for studying the materials before going to the flipped T-Module sessions. In each session’s reflection notes, each of the website materials used in the sessions was also evaluated by the participants. Additional comments on the website were obtained during the interview sessions that took place during the last day of the FD program. Guides for installing and using the mobile app and eBooks were also provided in the About Us and Useful Links sections.

Evaluation as an OER

A survey was also developed that included the System Usability Scale (SUS) by John Brooke (2013) and a primitive version of Perceived Ease of Use (PEU) and Perceived Usefulness (PU) questions from the Technology Acceptance Model (TAM) by Fred Davis (1989).

There were 10 SUS items, with 1 representing strong disagreement and 5 representing a strong agreement that was included in the survey. On the other hand, PEU and PU had six items each where participants had to express agreement or disagreement with a statement using a scale of 1 (strongly disagree) to 7 (strongly agree).

The survey containing the aforementioned scales was then distributed to faculty members of various educational institutions to get an idea of usability and perceived usefulness of the website materials as OER.

Results and Discussion

Initial Evaluation

Based on reviewing the reflection notes of the participants as well as the end-of-program interviews, several themes have emerged regarding the website materials. Firstly, the website materials were very informative and helpful to the new faculty members. Secondly, the website materials needed to be referenced more during the sessions especially those that are implemented in flipped learning mode. Thirdly, the website materials were easy to read and not too difficult to navigate. Fourthly, as a bilingual institution, the website needed to have a Japanese version. Furthermore, there was one participant who experienced difficulty in navigating some of the links. From this feedback, the team decided to redesign the website before the next implementation of the program.

Usability, Perceived Ease of Use, and Perceived Usefulness

The scores obtained would be considered inconclusive as a predictive measure since there were only 24 participants who responded to the survey at the time of this paper’s writing. However, it is still worth noting that the average SUS score was 73, which is very high based on Brooke’s stated global mean of 68. PU, on average, yielded 5.1 while PEU, on average yielded 5.33, which were both generally high. The standard deviation for PU and PEU was 1.1 and 1.21, respectively which meant that the scores did not deviate too much from the high mean.

The boxplot in Figure 3 shows that 50% of the respondents rated PEU between 4.5 and 6.25. The lowest score given was 3, but only 12.5 percent rated the website’s PEU with a value lower than the middle value of 4.

Figure 3. Boxplot of Perceived Ease of Use (PEU)

The boxplot in Figure 4 shows that 50% of the respondents rated the PU between 4 and 5.9. The lowest score given was 3.33, but only 16.7% rated the website’s PU lower than the middle value of 4.

The scores were promising in terms of usability, perceived ease of use, and usability. According to Davis’ model (1989), high PU and PEU levels could predict high usage among the target users.

Figure 4. Boxplot of Perceived Usefulness (PU)

Conclusion and Suggestions for Future Research

Based on the initial evaluation of the website, it has proven to be useful and easy to use to new faculty members at the ICU and a potentially useful OER among faculty members of other universities. However, design and layout improvements were necessary to make the website more navigable. Furthermore, after the findings, a Japanese translation of the website was proposed and developed to reflect the bilingual nature of the university. A comparative analysis of the mobile app and the website is also planned to make more sense of the SUS, PUE, and PU scores. Moreover, it is imperative to improve the methods for evaluating perceived usefulness and ease of use of the system. A follow-up study that would look into PEU, PU, and SUS of faculty members of a single institution and their actual use through time would validate the underlying model used in this study. A tracking study is also proposed to see how the materials were used as OER in various institutions in Japan and other countries.

References

Baker, V. L., Lunsford, L. G., & Pifer, M. J. (2015). Systems alignment for comprehensive faculty development in liberal arts colleges. To Improve the Academy, 34, 91–116.

Brooke, J. (2013). SUS: A retrospective. Journal of Usability Studies, 8(2), 29-40.

Choi, J., & Lee, Y. (2015). To what extent does ‘flipping’ make lessons effective in a multimedia production class?. Innovations in Education and Teaching International, 1-10.

Diaz, V., Garrett, P. B., Kinley, E. R., Moore, J. F., Schwartz, C. M., & Kohrman, P. (2009). Faculty development for the 21st Century. Educause Review, 44, 46–55.

Farahat, T. (2012). Applying the technology acceptance model to online learning in the Egyptian universities. Procedia-Social and Behavioral Sciences, 64, 95-104.

Davis, F. D. (1987). User acceptance of information systems: The technology acceptance model. Ann Arbor: University of Michigan.

Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework, principles, and guidelines. Wiley.

Jung, I., Nishimura, M., & Sasao, T. (2016). Liberal arts education and colleges in East Asia: Possibilities and challenges in the global age. Springer.

Jung, I., & Hong, S. (2016). Faculty members’ instructional priorities for adopting OER International Review of Research in Open and Distance Learning, 17(6), 28-43.

McLaughlin, J. E., White, P. J., Khanova, J., & Yuriev, E. (2016). Flipped classroom implementation: A case report of two higher education Institutions in the United States and Australia. Computers in the Schools, 33(1), 24-37.

Naeem, N., van der Vleuten, C., & Alfaris, E. A. (2012). Faculty development on item writing substantially improves item quality. Advances in Health Sciences Education, 17(3), 369–- 376.

Paskevicius, M., & Bortolin, K. (2016). Blending our practice: Using online and face-to-face methods to sustain community among faculty in an extended length professional development program. Innovations in Education and Teaching International, 53(6), 605-615.

Shea, P. J., Fredericksen, E. E., Pickett, A. M., & Pelz E. W. (2003). Faculty development, student satisfaction, and reported learning in the SUNY learning network. In T. M. Duffy, & J. R. Kirkely (Eds.), Learner-centered theory and practice in distance education: Cases from higher education (pp. 343–-377). Routledge.

Sletten, S. R. (2017). Investigating flipped learning: Student self-regulated learning, perceptions, and achievement in an introductory biology course. Journal of Science Education and Technology, 26(3), 347-358.

Taguchi, M., Nishimori, T., Shinto, T., Nakamura, A., & Nakahara, J. (2006). The current situation of FD for junior faculty in the tertiary educational institutions. Japan Journal of Educational Technology, 30(1), 19–-28.

Unger, D., & Wandesman, A. (1985). The importance of neighbors: The social, cognitive and affective components of neighboring. American Journal of Community Psychology, 13(2), 139–170.

Yuan, F., & Shimizu, Y. (2007). Faculty development and media use: Case studies of four American institutions and their implication to Japan. Journal of Multimedia Aided Education Research, 4(1), 19–-30.

Zainuddin, Z., & Perera, C. J. (2017). Exploring students’ competence, autonomy and relatedness in the flipped classroom pedagogical model. Journal of Further and Higher Education,1-12.

International Journal on Open and Distance e-Learning, Vol. 6 No. 1, June 2020 Issue