Special Report on Technology Application

Measuring System Usability and Readiness of System Users for Basic Healthcare Awareness and Education: A Case Study

Cinmayii A. Garillos-Manliguez

Assistant Professor, University of the Philippines Mindanao, Philippines, [email protected]

Abstract

Keywords: TeLRA, Information System, e-Learning, Education, Healthcare

Introduction

As emphasized by the World Health Organization (WHO), developing a strong health information system (HIS) for every country is needed to provide reliable and immediate high-quality health information. This will help health workers in making timely and accurate decisions (WHO, 2012). The use of information and communications technology (ICT) tools for healthcare promotion can contribute significantly to public health because these enable the mind to filter useful information from large quantities of health facts and use them on a given situation (Kwankam, 2004). These remove the gap of geographical constraints and can promote interactivity and patient engagement, which is a key to successful therapy (Novák, 2015). In Africa, 80% of the children could have been saved if they have been informed about the solution (Kwankam, 2004).

Considering the healthcare in the Philippines, the persisting problems include the difficulty in maximizing the use of gathered information in decision-making (Cervantes, 2014), and the challenge in educating community constituents with necessary healthcare information. The Philippines is just at the early stage of its nation-wide development or implementation of different information and management systems to improve the quality of healthcare services and the sharing and exchange of information for effective planning, managing, and monitoring within the community or of the nation as a whole (Herbosa, 2013; Shainur, Casebeer, and Scott, 2012; Department of Health (DOH), 2015). Several attempts have been made to develop health information systems for effective use in the Philippines. However the availability of affordable information technology (IT) resources, huge capital outlay (Marcelo, 2014), higher level decision-makers, and the attitudes of users towards the system still create hindrances in the full adoption of the system (Shainur, Casebeer, and Scott, 2012).

In the study of Anshari, Almunawar, and Low (2012), the functional health literacy of the patient have a strong correlation with the support and availability of the health system. E-learning is also considered as a new trend in professional healthcare education (Walsh and Dillner, 2003). It is a practical solution for training staff in patient-care settings because it helps them to keep updated with the latest technology, information, and regulations without being away from work (Nelson, 2003). Most of the studies involved development of health information systems while there are only few which integrates health literacy or healthcare education to increase patient engagement in a system (Anshari, Almunawar, and Low, 2012; Gephart and Effken, 2013). There are also limited studies in the Philippines that focus on this aspect, thus, this study would help in addressing this gap as well as contribute to the body of knowledge.

At present, the operations of most community or barangay health centers (BHC) in the southern part of the Philippines are done manually, i.e. pen-and-paper. The current system is very inefficient and time consuming most especially when searching for a patient record, writing a monthly report, and monitoring a patient’s status. Furthermore, the latest technologies are not being fully utilized in healthcare education. The weekly immunization and/or prenatal orientations are often being done verbally without the use of visual aids. This might lead to relaying inconsistent or incomplete information to the patients.

To address these pressing concerns in the community, an effective tool for health information management integrated with multimedia contents like modules, games, videos, among others in e-learning would be helpful to aid in proper information dissemination, promotion, and healthcare education in the community.

Objectives

This study aimed to develop and implement an electronic healthcare awareness system, named “eHICAP”, using open source technologies. It also aimed to measure the usability of the developed system using the System Usability Scale, and determine the learning attitudes of target users in the community using the Test of e-Learning Related Attitudes (TeLRA) scale. This can be seen as a case study as this system was used in one barangay health center located in the Barangay Sto. Niño, Davao City.

Review of Existing Alternatives

Since WHO has encouraged every country to invest in health information system, many countries have already created their own HIS, for example, Belize Health Information System (BHIS) in Central America, and Ethiopia (WHO, 2012). In the Philippines, there are only a few that can be found on the web as discussed in the following subsections.

Barangay Sto. Niño Health Center

Barangay Sto. Niño Health Center (BSNHC) is located in Davao City, which is in the southern part of the Philippines. Similar to the rest of the health centers in the country, BSNHC also holds immunization and prenatal programs. However, recent technologies can be used to help improve the efficiency of the health center operations and impact on its community. The following shows a sample problem that can be encountered in the current setting of a program named as Expanded Program on Immunization (EPI), which is regularly held in the BSNHC. First, when managing records of a patient for the EPI in the BSNHC, each record of a patient will be hand-written and will be updated by filling out a form and will be kept in a brown envelope labeled with the patient’s name and address. These envelopes will then be filed in cabinets. In the events of lost patient data, the patient is required to fill out the EPI form again. Second, when doing weekly/monthly reports, Barangay Health Workers (BHWs) will check the forms and gather the necessary information (e.g. number of vaccines given, etc.) which will then be submitted as numerical data to the head office. With an operations-based information system, there will be better management of records and faster operations (adding a new record, searching, updating information, and keeping of records).

When it comes to health care awareness and promotion campaign, the BHWs need to walk around the whole barangay to disseminate flyers, brochures, etc. Facilitating decision making may also be difficult given the current setting. For instance, if there will be free vitamins and medicines to be distributed among pregnant women active in the Prenatal Program, the BHWs need to sort out the files per purok or area, then sort again the active and inactive records. The abovementioned efforts and operations can be very tedious and inefficient, thus this system was developed. Furthermore, when conducting these immunization and prenatal programs, it is very vital to educate the patients/guardians/parents on the information about vaccines, vitamins, minerals, and very importantly the actions to take whenever they are in a situation that concerns their health after visiting the health center. This is why health literacy or health education should become an essential part of the system.

Unified Health Management Information System (UHMIS)

The UHMIS is a very large system of different health systems (e.g. National Health Data Dictionary, Kontra-Paputok Reporting System, etc.) that is being developed until 2017. This project is spearheaded by the Department of Health (DOH) in coordination with different agencies such as Philippine Health Insurance (PhilHealth), National Nutrition Office, among others. One of the objectives of the project is to have a Central Database Warehouse of all health-related information in the country (DOH, 2015). This will help them find an immediate and reliable source of information whenever a health crisis occurs.

One system that is related to this study is the Health Clinic Information System, which uses a Client/Server architecture. This will be used for gathering patient’s information and the health services done in rural health units and barangay health stations. Based on the DOH website, the status of the system is already under the testing phase (DOH, 2015).

Open Source Systems

There is a number of open source software for electronic health record system available for use in different hospitals, clinics, health offices, among others. Some of these are discussed in the following subsections.

Community-based health information tracking system (CHITS)

CHITS is an open source electronic health record system for local government centers in the Philippines. The main objective of CHITS is to use generic, reusable, and open-source framework for the system (Tolentino, 2005). This system has three vertical programs implemented which are Maternal and Child Health, Immunization, and Notifiable Diseases (CHITS, 2011). An important learning from their methodology is that the developers worked with the health center staff so that they will have a full awareness of the environment where the system will be used (Tolentino, 2005). This idea helped the researcher in selecting the best development model for this study, which was Scrum Agile Development Model.

GNU health

GNU Health is an open source system developed for medical practitioners and health institutions. GNU Health is a very comprehensive information system that is more appropriate for hospital or clinic use. The system can be downloaded from this link: http://health.gnu.org/index.html. While GNU Health should be used in a Linux environment, eHICAP was developed to be used in any operating environment (desktop, mobile, etc.).

DHIS 2

DHIS 2 is another open-source information system which has a capability of visualizing different types of data using appropriate graphs/charts e.g. line graph, bar graph, pie chart, and even GIS maps. Although DHIS 2 provided a lot of options for data analysis and visual representation, the outputs are for health workers consumption only unlike eHICAP which also allows patients to monitor their own health status. The system can be downloaded from this link: https://www.dhis2. org/.

OpenMRS

Another promising system is the OpenMRS which is an open medical record system. The mission of OpenMRS is also to improve the status of health care delivery especially in financially-challenged community by providing an open source technology. Based on its demo, the interface is simpler compared to the DHIS 2 because there are only few fields shown in the form to get the basic information of the patient. The system can be downloaded from this link: http://openmrs.org/.

The systems mentioned previously are focused on the efficiency that the information systems would bring to the health center staff in terms of operations-based management, and on managing, monitoring, and integrating the data entering the system. Although having an integrated system would be helpful, it might also overwhelm the health workers.

In this study, aside from creating a health information system (HIS), eHICAP also includes interactive tools (e.g. slideshow, videos, games, etc.) to promote health care awareness. Several studies have already proven that there is a relationship between health awareness and gaming (Riley et al., 2011; Kato, 2010). Games can be very powerful tools to engage patients in a therapy (Kato, 2010).

Research Design and Methods

The electronic health information system was developed using Scrum Agile methodology framework through open-source technology e.g. PHP, Yii 2.0, Bootstrap, and XAMPP 7.0.4. Agile software development prioritizes the needs of the customer by delivering solutions even at the early stage of development. This is possible because of the iterative approach in developing every increment of the software. The code is also created as simple as possible which helps enhance agility and adapt to changing system requirements and technologies (Principles behind the Agile Manifesto, 2001). One of the ways to implement the agile development is to follow the process framework of Scrum. It started with a story of specifications, which composed the product backlog, provided by the health center head. Then a small development team planned the sprint backlog, which composed of time-bounded (about a month or less) activities or tasks for the product increment. Each sprint included sprint planning, daily scrum meeting, sprint review, and sprint retrospective before the final submission of the product increment. In the Scrum, when the product is deployed to the client, their feedback will be recorded and added to the product backlog and the Scrum cycle goes on until the end of the project duration or until the client is satisfied with the results.

After following the methodology framework, the system was developed and a screenshot of the Home page is shown in Figure 1. The developed system allows users to: manage and monitor patient records and health status, auto-generate weekly/monthly reports including production of visual representations such as charts and graphs, and promote health care awareness in the community and educate the constituents through the e-learning module. This module can be used in the weekly orientation or seminar of the BHC to their patients or guests. This module is important to enable the health care workers relay consistent information to different people in their weekly programs/services. Currently, computerized/electronic health information system is non-existent in Barangay Sto. Niño Health Center. Based on an interview conducted last November 4, 2015 with the head officer of the health center, (Ms. Mary Ann Lanticse) and other BHWs, no system has been made for them so far.

The attitudes of users towards the system is an important factor in the successful implementation of the system (Shainur, Casebeer, and Scott, 2012). In order to fully understand and explain how the health center staff and other community constituents use the system, the Technology Acceptance Model (TAM) was used in this study. TAM, introduced by Davis (1986), is a theoretical model that aimed to explain the determinants of information system’s user acceptance. Figure 2 shows that the actual usage of the system is determined by the acceptance behavior, which is affected by the perceived usefulness and the attitude towards using the system. A secondary determinant to the user’s intention in using the system is the user’s perceived ease of use, which also affects the user’s perception on the system’s usefulness (Davis, Bagozzi, and Warshaw, 1989). TAM is one of the most influential and robust models that help predict the acceptance, adoption, and use of information system (Chen, Li, Li, 2011).

Figure 1. Screenshot of the electronic health information system with buttons that link to its modules.

Aside from being used for information systems, TAM can also be used in e-learning. In the study of Park (2009), TAM was used to understand the behavioral intention of university students towards e-learning in Korea and results proved that it was a good theoretical tool for this.

Figure 2. Technology Acceptance Model (lifted from Davis, Bagozzi, & Warshaw, 1989).

Furthermore, the factors such as teachers’ perception and their preparedness to use e-learning in their classes will highly determine the success of adopting and implementing e-learning in an educational institution (Kisanga and Ireson, 2016). In the dissertation of Kisanga (2015), he developed a scale named Test of e-Learning Related Attitudes (TeLRA). He then tested and validated the scale in Tanzanian higher learning institutions. TeLRA is a tool that can help an institution to identify opportunities and threats that affect the attitude of the teachers’ towards e-learning. According to Kisanga and Ireson (2016), the TeLRA scale can also be used in investigating the attitudes of people from other sectors besides educational institution. The TeLRA scale is composed of 36 questions measuring four factors/themes: challenges of e-learning, benefits from e-learning, attitude towards using computer systems, and leisure interest on e-learning innovations and use of computers. Although TeLRA is a 4-point scale, it can be reduced to two points (i.e. agree or disagree) for the purpose of analysis (Kisanga, 2015, p.234), and this was also done in this study.

To effectively evaluate the usage of the system on the target organization or the health center, a preliminary survey was conducted after the development of the system. A survey was developed based on the key result areas that were identified: (1) improved efficiency in terms of improved records management, and report submission using the system; (2) increased awareness in and promotion of healthcare through the system especially its interactive learning module; and (3) improved decision making especially in choosing focal areas for conducting healthcare programs/ activities. User acceptance testing was done through the System Usability Scale (SUS) of Brooke (1989) to determine the acceptance level of the system to its target users.

In this study, the target users of the system were the community constituents including the barangay health workers (BHWs) and the probable patients. The preliminary survey and SUS were conducted separately from TeLRA because the former needs more time presenting and using the system while the latter needs a thorough explanation about e-Learning. There were two batches of survey sessions conducted last May 2016 for the preliminary survey and SUS, the first session was for the BHWs while the second was for the community constituents because these people are the expected main users of the system. Before the evaluation survey was given, the system features was first introduced to the users and a demonstration on the usage of each module of the system was also done. The users were given about 30 minutes to one hour to use/test the system, and then the survey questionnaires were given. The first batch was conducted in the Barangay Sto. Niño Health Center (BSNHC) and nine female BHWs with ages raging from around 42 to 56 years old participated in the survey. After conducting the survey, two short group discussions was done on May and June 2016 to gather feedback from the health workers on the challenges that were encountered during the implementation and some possible future improvements of the system. On the second batch, the temporary URL of the system was given to six possible future patients of the health center with age ranging from 20 to 30 years old and, again, all were female.

For the TeLRA scale, the eLearning module of the system was first presented to the BHWs followed by the discussion on eLearning before the survey was distributed last June 2016. Each statement in the scale was explained during the survey and a translation to the vernacular language was also included in the survey form to help the respondents understand the statement so that the appropriate responses would be given. For this session, 21 female BHWs whose ages ranged from 33 to 64 years old responded to the survey. Their years of experience in the health center ranged from two to 27 years, which means that they already have the knowledge on the operations and health literacy/education practices of the health center.

Most of the statements in the SUS can be categorized to be under the perceived ease of use factor in TAM, while the questions in the preliminary survey addressed the perceived usefulness factor. These factors were used to determine the acceptance of the health workforce to the new technology being introduced in this study. The TeLRA scale was used to identify the factors that affect the BHWs general attitude towards e-learning for health literacy.

Results and Discussions

Usability of the System

Table 1 shows the mean and mode of the ratings of the SUS. Most users strongly agreed to use the system frequently. They found it easy to use, and that most functions in the system are well integrated. They did not find the system as complex, inconsistent, and cumbersome to use, which are still positive views of the system. They also believed that most people would be able to learn to use the system very quickly. Thus, in terms of the attitude, they felt confident using the system.

However, some of them thought that they still need to learn a lot from that system and would also need the support of a technical person before they could get started with the system. A suggested solution to this is to conduct training and follow-up sessions to assist the users in using the system. A good documentation, specifically a user manual or a multimedia user guide, should be available and accessible to the users anytime.

Generally, the system gained very good points from the users. It even obtained an average of 74 points which is above the recommended system acceptance rating of 68 points.

Table 1. Mean and mode of the system usability ratings of the respondents for each question.

	Questions	Mean	Mode
1.	I think that I would like to use this system frequently	4.8	5.0
2.	I found the system unnecessarily complex	2.1	1.0
3.	I thought the system was easy to use	4.5	5.0
4.	I think that I would need the support of a technical person to be able to use this system	3.2	5.0
5.	I found the various functions in this system were well integrated	4.5	5.0
6.	I thought there was too much inconsistency in this system	1.9	1.0
7.	I would imagine that most people would learn to use this system very quickly	4.3	5.0
8.	I found the system very cumbersome to use	1.8	1.0
9.	I felt very confident using the system	4.4	5.0
10.	I needed to learn a lot of things before I could get going with this system	3.8	5.0

Initial Impression of Users

Based on the results of the preliminary survey, all users had an initial impression that the system will improve records management through fast searching of records, and they believed that the records will be more organized, less prone to being lost, and more accessible. Another impression is an improved report generation and submission of the healthcare programs in the community through automatic computation of statistics, reducing human error, and providing visual representations.

Users also believed that through the mapping function of the system, the decision making of the supervisor will be improved especially in choosing focal areas for conducting programs/activities e.g. feeding program, maternal care orientation, distribution of medical kits, etc.

They also had a good impression on the learning module which they believe would help in educating people in healthcare awareness and promoting proper healthcare in the community. For instance, they found that through the learning module the parents would be educated on proper ways of taking good care of their health especially the pregnant women.

Attitudes of Health Workforce on E-learning

There were 21 health workers who responded to the TeLRA scale. Figure 2 shows their responses on each theme of TeLRA. The results for each theme are discussed in the following paragraphs.

Figure 2. Health worker’s average percentage of agreement/disagreement to the themes of TeLRA scale (N=21).

Benefits of e-learning

Health workers had a very high positive response to e-learning as shown in the average percentage of agreement that was obtained which is 94.7% while an average of 4.2% disagreed. All of them agreed that “e-learning is very economical for the health center to adopt”. They also believed that “e-learning will improve the quality of their work”, and “will provide them with better learning opportunities than traditional means of learning.”

Leisure interest on e-learning innovations and use of computers

This theme received an average agreement response of 77.8% while 22.2% for disagreement. The statements with the highest percentage of 95.2% included “I like reading magazines on new technology innovations”, “Working with computers is exciting”, and “I like discussing about new e-learning innovations”.

Challenges in the implementation of e-learning

The health workers responded with only 57.9% average agreement on this theme, while 41.3% for disagreement. Some statements from this theme that obtained a percentage of above 80% included “I feel uncomfortable reading a text book on a computer screen than a physical text book”, “E-learning increases learner’s social isolation”, and “Using computer systems requires a lot of mental effort.”

Attitude towards using computer systems

Based on Figure 2, the average agreement response of the health workers on this theme is almost equal to the average disagreement percentage with a difference of approximately 0.8%. The statement with the highest agreement percentage of 81% is “using e-learning technologies will allow me to accomplish more work than would otherwise be possible.” On the other hand, the statements “I find computer online interaction unexciting” and “communicating through electronic mails is annoying” obtained the highest disagreement percentage of 71.4%.

The average score of the SUS which is 74 means that the system usability is above average and thus, it has a percentile rank of 70%. This can be interpreted that the system has higher perceived usability than 70% of other systems tested (Sauro, 2011). However, this score also mean that there are still aspects of the system that can be improved to get a higher score of 80.3 which is the top 10% of scores. This and the results of the preliminary survey mean that the overall functions of the system is acceptable and generally usable by the users. All users had a positive initial impression of the system. This implies that even upon introduction of the system to the user or just at the early stage of implementation, the users were able to find the importance of the system in improving their operations. With these perceptions on system usability and ease of use, users or the health workers have high chance of adopting the system quickly given the infrastructure and resources.

In the assessment of their e-learning preparation using TeLRA, it is found out that the health workforce still need an adjustment period in using ICT especially for health literacy. This challenge could be due to the media used in learning. According to Akahori’s (2014) study, the paper is the best learning medium especially for memorization and comprehension while using personal computers as a learning tool can induce fatigue to the learner. This could be one reason for their rating, while another possible justification is the age-related digital divide because most of them are already of the age 40 above where some, if not all, are less likely to explore the tools in IT to learn new things (Niehaves and Platfaut, 2014).

Conclusions and Recommendations

The findings in this study showed that the developed electronic health information system is perceived to be very useful and easy to use by the main users of the system. Based on the results of TeLRA, it was found out that the challenges of implementing e-learning (e.g. lack of infrastructure) and the users’ attitude towards using computer systems are the main factors that may influence the users’ willingness to be e-learning teachers and students in the barangay health center.

For future direction of this study, it is recommended to fully implement other improved technology acceptance models that will consider factors e.g. the appropriateness of the technology, to include the demographics and the depth of users’ knowledge about the technology in the analysis of results, and to conduct a thorough impact study after several months of usage of the system. This would help provide more concrete results on the assessment of the system and the attitudes of the health workforce.

A learning management system for massive community-based online courses (MCOCs) for healthcare awareness can also be a good extension of the system. With this, the system will not only be able to offer multimedia interactivity but also content delivery that is very similar to what massive open online courses (MOOCs) can offer.

Acknowledgements

The information dissemination part of the paper, which is the paper presentation in the International Conference on Media in Education (ICoME) 2016 in Kyoto, Japan, is funded by the University of the Philippines System through the Research Dissemination Grant.

References

Akahori, K. (2014). Do digital devices work better than paper materials for learning? International Journal for Educational Media and Technology, 8(1), 4-10.

Anshari, M., Almunawar, M.N., and Low, P.K.C. (2012). CRM 2.0 within e-health systems: Towards achieving health literacy and customer satisfaction. The Journal of Development Informatics, 1(1), 2012.

Brooke, J. (1989). SUS – A quick and dirty usability scale. Redhatch Consulting Ltd. Retrieved from: http://cui.unige.ch/isi/icle-wiki/_media/ipm:test-suschapt.pdf.

Cervantes, H.M.J., Chua, C., Ronquillo, M., Tolentino, I.J., and Magpantay, L.A.K. (2014). Community- based decision support system for the Manila health department. De La Salle University Research Congress 2014, Manila, Philippines.

Chen, S., Li, S., and Li, C. (2011). Recent related research in technology acceptance model: A literature review. Australian Journal of Business and Management Research. 1(9). 124-127.

CHITS. (2011). OpenClinical: CHITS – Community-based Information Tracking System. Retrieved from: http://www.openclinical.org/os_chits.html.

Davis, F.D. (1989), Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly 13 (3), 319–340, doi:10.2307/249008

Davis, F.D., Bagozzi, R.P., and Warshaw, P.R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), pp. 982-1003.

Department of Health (DOH). (2015). Unified health information management system. http:// uhmis.doh.gov.ph/

DHIS 2. (2015). DHIS 2 overview. Retrieved from: https://www.dhis2.org

Gephart, S. and Effken, J. (2013).Using health information technology to engage patients in their care. Online Journal of Nursing Informatics (OJNI), 17 (3). Retrieved from: http://ojni.org/ issues/?p=2848

GNU Health. (2011). Home page. http://health.gnu.org/

Herbosa, T.J. (2013). eHealth Philippines: Deliverables, 2016 and beyond. Department of Health. Retrieved from http://www.pchrd.dost.gov.ph/index.php/downloads/category/75-plenary- morning-session?download=331:plenary-ehealth-philippines.

Kato, P. (2011). Video games in health care: Closing the gap. Review of General Psychology. https:// www.c-changeprogram.org/c-channel_issue33_sept2011

Kisanga, D.H. (2015). Investigation of Attitudinal factors towards the transition from face-to-face o e-learning in Tanzanian higher learning institutions: A mixed methods approach. Retrieved from: http://irep.ntu.ac.uk/97/1/220969_Dalton.Kisanga-2015%2520excl.%25203rdparty material.pdf

Kisanga, D.H., and Ireson, G. (2016). Test of e-learning related attitudes (telra) scale: Development, reliability and validity study. International Journal of Education and Development using Information and Communication Technology. 12(1), pp.20-36.

Kwankam, S. Y. (2004). Bulletin of the World Health Organization ISSN 0042-9686. Retrieved from www.scielosp.org

Marcelo, A.B. (2014). Health informatics in the Philippines http://www.philmedinfo.org/health- informatics-in-the-philippines/

Nelson, E.A. (2003). E-learning: A practical solution for training and tracking in patient-care settings. Lippincott Williams and Wilkins, Inc.

Niehaves, B. and Plattfaut, R. (2014). Internet adoption by the elderly: Employing IS technology acceptance theories for understanding the age-related digital divide. European Journal on Information Systems. 23: 708. https://doi.org/10.1057/ejis.2013.19

Novák, D. (2015). Handbook of Research on Holistic Perspectives in Gamification for Clinical Practice. IGI Global. ISBN 1466695234, 9781466695238.

Park, S.Y. (2009). An analysis of the technology acceptance model in understanding students’ behavioral intention to use e-learning. Educational Technology and Society. 12, (3). Pp.150- 162.

Principles behind Agile Manisfesto. (2001). Manifesto for agile software development. Retrieved from: http://agilemanifesto.org.

Riley, W., Rivera, D., Atienza, A., Nilsen, W., Allison, S., and Mermelstein, R. (2011). Health behavior models in the age of mobile interventions: Are our theories up to the task? Retrieved from: https://www.c-changeprogram.org/c-channel_issue33_sept2011

Sauro, J. (2011). Does prior experience affect perceptions of usability? Retrieved from: https:// measuringu.com/sus/

Shainur, P., Casebeer, A., and Scott, R. (2012). Implementing electronic health information systems in local community settings: Examining individual and organizational change experiences in the Philippines. The Electronic Journal Information Systems Evaluation. 15(2), 2012, pp.186- 197. ISSN 1566-6379. Academic Publishing International Ltd.

Tolentino, H., Marcelo, A., Marcelo, P., and Maramba, I. (2005). Linking primary care information systems and public health vertical programs in the Philippines: An open-source experience. AMIA Annual Symposium Proceedings; 2005: 311–315. PMCID: PMC1560490.

Walsh, K. and Dillner, L. (2003). Launching BMJ Learning. BMJ. ; 327:1064. DOI: 10.1136/ bmj.327.7423.1064

World Health Organization (WHO). (2012). Components of a strong health information system (Brochure).

International Journal on Open and Distance e-Learning, Vol. 3 No. 1, June 2017 Issue