Ask Iska and IskOU: Analysis of UPOU's Chatbot for Information Support Services

Joane V. Serrano, Ph.D.¹, Janele Ann C. Belegal², Anna Ma. Elizabeth F. Cañas-Llamas³, Lovelyn P. Petrasanta⁴, and Myra C. Almodiel⁵

¹Associate Professor, University of the Philippines Open University, Philippines, [email protected]
²Project Research Assistant, University of the Philippines Open University, Philippines, [email protected]
³Administrative Officer IV, University of the Philippines Open University, Philippines, [email protected]
⁴Research Assistant, University of the Philippines Open University, Philippines, [email protected]
⁵Assistant Professor, University of the Philippines Open University, Philippines, [email protected]

Abstract

Keywords: artificial intelligence, chatbot, information support, support services

Introduction

Chatbots

In recent years, artificial intelligence (AI) has been widely recognized and adopted in educational settings. It has transformed the way universities and academic institutions interact with their students, especially in providing information support services. AI has taken on various definitions due to its application in a number of domains. In education, Luckin et al. (2016, p. 14) define AI “as computer systems that have been designed to interact with the world through capabilities... and intelligent behaviors...that we would think of as essentially human.” AI in education has been found to be used for data mining, learner support, online tutoring, instructional tools and platforms, and education administration, (Luckin et al. 2016; Holmes et al. 2019; Chen et al. 2020; Roll & Wylie, 2016). With these uses of AI in education, the University of the Philippines Open University (UPOU), as the leader in open and distance e-learning in the country, continues to innovate upon these tools to streamline its operations and services on teaching and learning among others. As an academic institution, UPOU puts considerable emphasis on providing proper and adequate support to its constituents (students [existing and prospective], alumni, faculty, staff, and community). One of UPOU’s streamlined services through AI is its information support services. In UPOU, this service is characterized as providing appropriate and timely university- related information to its constituents. This service is mainly provided by support staff, information resources (e.g. websites, dissemination materials), and tools. Specifically, this study will focus on the provision of information support via an AI tool, the chatbot.

Chatbots are also known as conversational agents. Some of the renowned chatbots are ELIZA, PARRY, Jabberwacky, Dr. Sbaitso, ALICE, SmarterChild, Siri, IBM Watson, Google Now, Amazon Alexa, Google Allo, Tay, and Xiaoice (Wei, Yu, & Fong, 2018; Cahn, 2017). These tools are “computer programs which attempt to simulate conversations of human beings via text or voice interactions” (Rouse, 2017, as cited in Winkler, & Söllner, 2018, p. 5). Chatbots function due to their developer- designed systems, and their intelligence database or knowledge base (KB). The KB acts like the chatbot’s brain where it generates its responses to user inputs from stored information. Mostly, chatbots “are utilized and deployed...for the purpose of seeking information, site guidance, and answering frequently asked questions” (Huang et al. 2007, p. 423, as cited in Serrano et al. 2020, p. 161). With these in mind, UPOU developed its own chatbot to streamline its information support services.

The UPOU Chatbot

Launched in February 2019, the UPOU chatbot was developed by the following UPOU Offices: Office of Public Affairs (OPA), Multimedia Center (MC), and Information and Communication Technology Office (ICTDO). The tool was developed in response to the need to streamline the university’s virtual information support services provided to its constituents. UPOU constituents can access and use the chatbot via the official UPOU Facebook page chat (https://www.facebook. com/UPOpenUniversity), and through its website (https://askou.upou.edu.ph). The UPOU chatbot is personified as Iska and IskOU for gender representation (Serrano et al. 2020, see Figure 1).

Figure 1

UPOU chatbot Iska (left) and IskOU (right)

Note. These icons are produced by the UPOU Office of Public Affairs in 2019 (as reproduced in Serrano et al. 2020). From “UPOU chatbot: Toward quality information services,” by J. Serrano et al. 2020, Quality Initiatives in an Open and Distance eLearning Institution: Towards Excellence and Equity (In M.F. Lumanta, & P.G. Garcia [Eds.]), pp. 162.

In Serrano et al (2020) the UPOU chatbot’s design was described as follows:

The UPOU chatbot was designed to provide immediate, intelligent, [timely], and appropriate human-like interactions and conversations related to UPOU matters when prompted [by users]. It works by detecting and mapping keywords in user inputs (message/query) with the chatbot’s KB to generate appropriate responses. (p. 161)

[It] offers natural language processing wherein users may ask questions, in both English or Filipino, in a conversational manner. As a bot, it operates through a pre- configured keyword matching system by which it attempts to detect the intent of the user and respond accordingly from a list of mapped messages. Its responses follow a template organized and listed by the chatbot [developers]. (p. 162)

Contents of the UPOU chatbot’s response templates mainly consist of links to UPOU’s online resources and websites. The tool’s keyword detection and mapping also work best on brief messages. Figure 2 illustrates the UPOU chatbot’s response generation design.

Figure 2

UPOU chatbot response generation design

The UPOU chatbot was mainly deployed to provide constituents with a convenient and efficient online platform for UPOU-related inquiries (Serrano et al. , 2020). However, users can experience two different styles of support from this tool. First, the UPOU chatbot on Facebook looks and acts similar to a chat box, where users leave a message and the chatbot responds accordingly (as shown in Figure 3). On the other hand, the UPOU chatbot on the website acts similar to both a chat box and a search tool. On the chatbot website, information is displayed according to what’s found on UPOU’s 1) websites; 2) frequently asked questions; 3) announcements, news, and features; and, 4) educational resources. When a user converses with the chatbot on the website, the chatbot responds and displays other relevant information to the user input (as shown in Figure 4). In the event that the tool is unable to detect the user’s intent, “an automated response [is] sent prescribing the user to rephrase their questions [for the system to] map [keywords] effectively to its preset templates and answers” (Serrano et al. 2020, p. 163).

Figure 3

UPOU chatbot via Facebook

Figure 4

UPOU chatbot via website

The UPOU chatbot completely embodies the previously mentioned chatbot purposes by Huang and colleagues (Huang et al. 2007). It is able to deliver its purpose through its comprehensive KB which is the result of a university-wide effort to collate information about UPOU. In turn, the UPOU chatbot benefits both the information support staff and the constituents—through the streamlined process of giving and receiving UPOU-related information via the tool’s platforms. Nonetheless, it is imperative to periodically evaluate the UPOU chatbot’s performance to sustain the validity and timeliness of its intelligence, as it influences how it responds to its users and eventually results in how it satisfies its users.

Objective

This study aimed to evaluate the UPOU chatbot’s performance as an information support tool by determining the level of satisfaction of UPOU chatbot users.

Conceptual/Theoretical Framework

The study draws upon features that could influence users’ perceived usability of chatbots as cited by Tariverdiyeva (2019). The 2019 investigation listed 27 chatbot features. This study, however, focused only on 5 relevant features to the UPOU chatbot: response time, multi- thread conversation, maxim of quantity, perceived ease of use, and recognition and facilitation of users’ goal and intent. Tariverdiyeva (2019) defined these features as follows:

Response time - Ability of the chatbot to respond timely to users' requests (Amazon, n.d.-b)

Multi-thread conversation - Ability of the chatbot to recognise and process multiple parallel topics simultaneously (Staven, 2017)

Maxim of quantity - Ability of the chatbot to respond in an informative way without adding too much information (Gnewuch et al., 2018; Google, 2017)

Perceived ease of use - The degree to which a person believes that to interact with a chatbot would be free of effort (Van Eeuwen, 2017)

Recognition and facilitation of users' goal and intent - Ability of the chatbot to recognize user's intent and guide the user to its goal (Coniam, 2014; Ramos, 2017; Van Eeuwen, 2017a; Wilson et al., 2017) (pp. 10-11)

As shown in Figure 5, the selected features make up the conceptual framework of this study on the satisfaction of UPOU chatbot users.

Figure 5

Conceptual framework of the study

Review of Related Studies

AI is a widely studied topic and has shown established evidence of its advantages in education (Chen et al. 2020; Roll & Wylie, 2016). The same goes for chatbots. A review of literature also suggests that there is a breadth of studies on chatbots as information support tools (Ranoliya et al. 2017; Tariverdiyeva, 2019; Balaji, 2019; Feine et al. 2019; Følstad & Brandtzaeg, 2020). Among other factors and features, these studies almost highlight the importance of a chatbot KB to properly accommodate users. With KBs acting as the brain of a chatbot system, it is, therefore, necessary for these to be updated, especially when chatbots acquire new and unique queries from users. Cahn (2017) states that chatbots “are only intelligent to the information they have access to,” thus further implicating the significance of chatbot KBs. A previous study on the UPOU chatbot showed from analyzed developer narratives that the tool’s KB needs upgrading or updating (Serrano et al. 2020). However, the UPOU chatbot’s KB is manually updated by developers. Updating the KB is highly dependent on human resources and time. Although, there have been studies on automatic KB updates by utilizing online discussion forums (Huang et al. 2007; Wu et al. 2008) and Wikipedia (Hussain & Ginige, 2018). The studies of Huang et al. (2007) and Wu et al. (2008) concluded with good results for automatic KB updates. Hussain and Ginige’s study also resulted in a good evaluation of the automatic update for a secondary KB (Wikipedia). The developers of the UPOU chatbot can draw upon these studies to streamline KB updates. Nonetheless, when KBs lack timely, varied, and complete information, then chatbots cannot adequately converse with users; thus, affecting user satisfaction (Hussain & Ginige, 2018).

Chatbot user satisfaction is mainly influenced by a number of factors, as listed in Tariverdiyeva’s 2019 chatbot literature review. Some studies on chatbot user satisfaction measured it through post-interaction surveys and interviews (Jain et al. 2018; Følstad et al. 2018; Hien et al. 2018; Crutzen et al. 2011; Følstad & Brandtzaeg, 2020). In Jain et al. (2018), eight chatbots in the Facebook Messenger platform were evaluated. Results of the study show the participants’ disappointment and frustration with the chatbots due to their inability to comprehend user intent and answer efficiently. Følstad et al. (2018) studied customer service chatbots and found that users find the following chatbot factors beneficial: productivity, immediate support, accessibility, and relaxed nature. The study also highlighted a chatbot’s inability to comprehend intent and to address complex requests as “a major limitation” (Følstad et al. 2018, p. 204). A study on FIT- EBot, a chatbot for service inquiries of a faculty of study in a university, resulted in a satisfactory evaluation from users (Hien et al. 2018). Crutzen et al. (2011) studied Bzz, a chatbot for health promotion, which was “evaluated positively for its conciseness, response speed, privacy, and quality of information” (Crutzen et al. 2011, as cited in Serrano et al. 2020, p. 165). In Følstad and Brandtzaeg (2020), they have investigated the hedonic and pragmatic attributes of chatbots and reported that users place importance on perceived chatbot usefulness for positive experiences. However, interpretation issues and low effectiveness (inability to help) were also reported for negative experiences (Følstad & Brandtzaeg, 2020). Another study on chatbot user satisfaction suggested that productivity (effectiveness and efficiency of chatbots in information retrieval) was the “primary motivation” for users to interact with chatbots (Balaji, 2019, p. 70). However, Balaji (2019) suggested that “factors relevant to determining user satisfaction may critically depend on the specific type of chatbot” (p. 71). Generally, chatbots are challenged with linguistic and interpretational factors; although these could also be considered for their improvement.

Methodology

The study was conducted in late 2019. Data were collected from an online post-interaction survey administered via Google Forms, and disseminated via the UPOU Facebook pages. The survey was shared on this social media platform to reach the users who have interacted with the UPOU chatbot. Anyone who has access to the survey link and who has experience in using the chatbot was considered study respondents. A combination of short questions and 5-point Likert scale questions were used in the survey. Survey questions centered on the aforementioned chatbot features: response time, multi-thread conversation, maxim of quantity, perceived ease of use, and recognition and facilitation of users’ goal and intent. The study questionnaire also included questions on overall user satisfaction, problems encountered, and suggestions for the UPOU chatbot. Survey responses were analyzed using descriptive statistics and thematic analysis. The survey questionnaire used for the study is presented in Appendix 1.

Ethical Considerations

The study questionnaire was integrated with an informed consent. Participation was voluntary, and participants were free to withdraw from the study. Participant names and other personal information were also withheld from the discussion and presentation of results.

Results and Discussions

In this section, we discuss the experiences and satisfaction shared by the users with the UPOU chatbot as documented from the post-interaction survey. A total of forty-one (41) users responded. Figure 6 presents the characteristics of these respondents.

Figure 6

Pie graph of study respondents

Based on the survey results, UPOU chatbot users inquire about: application procedures and requirements to UPOU programs, UPOU programs (undergraduate and graduate), tuition fees, contact numbers, academic calendar, distance education, UPOU events, and eligibility to be a UPOU student. It is noteworthy that most respondents of the survey were non-UPOU students (inquiring public, n=22, 53.7%). Other respondents were 10 (24.4%) prospective students, 5 (12.2%) UPOU students, 3 (7.3%) UPOU staff, and 1 (2.4%) UP student. This respondent distribution could be attributed to the practice of information support in the university. UPOU students almost always direct their inquiries and concerns to official UPOU emails. On the other hand, inquiring public, who are yet to familiarize themselves with the university, proceed to the official Facebook page to inquire, as it is most convenient and accessible to them. Moreover, based on monitoring the conversations of the UPOU chatbot, UPOU students will only message the chatbot about their concerns when the support staff behind the official UPOU emails cannot respond to them within a few (usually 1-3) working days.

This remark is verified by an actual response in the study survey:

“Sometimes mailers take too long to respond to emails. That is why I message your Facebook page to get an answer from you since no one is responding in your mailers.”

UPOU Chatbot Performance

Response Time

For response time, results show that the UPOU chatbot can ‘always’ (n=41, 100%) answer user queries promptly. Checking UPOU chatbot logs in the Facebook for Developers page revealed that on average the tool responds within 900 milliseconds upon prompts. These prove to show that the tool performs accordingly to its design of providing immediate responses to users. However, this study could be replicated into exploring the possibility of “dynamic” UPOU chatbot responses to investigate whether user satisfaction could be influenced, and possibly affected as demonstrated in Gnewuch et al. 2018.

Multi-thread Conversation and Maxim of Quantity

The study tackled these 2 factors in one survey question, and that was on the ability of the tool to answer completely. Results showed (n=21, 51.2%) that the UPOU chatbot can ‘always’ answer user queries completely. However, upon identifying which questions (see Table 1) were “always” answered completely by the chatbot, it revealed that the UPOU chatbot gives an appropriate response to queries related to application procedures, requirements, deadlines, and UPOU programs. These queries are the frequently asked questions and are included in the prompting message of the UPOU chatbot, hence the chatbot can readily and appropriately respond to these types of questions. On the other hand, when the chatbot can only ‘sometimes’ (n=10, 24.4%) answer, and ‘never’ (n=9, 22%) answer queries, it also revealed that users who have multiple, complex, or specific/unique queries gave such responses. Example of these queries, as mentioned in the study survey, are:

“One time I messaged the UPOU page about the requirements to be a UPOU student. The chatbot did answer me for the first [few] questions, but I noticed that when my questions became more specific about my situation, it cannot answer me anymore.”

“The chatbot cannot answer questions about the steps in the online application system [of UPOU]. It can direct you to the OAS link, yes, but specific OAS questions cannot be answered.”

“Sometimes the chatbot cannot answer specific questions. If I ask about the contact number/s of specific UPOU offices, it just gives me an answer for its location map.”

These results suggest that the UPOU chatbot does not perform well on all questions or interactions. This performance could be attributed to a possible lack of varied and complete information on the UPOU chatbot KB. Survey results can be used to identify the weak points of the UPOU chatbot and to improve its KB, and its response mapping and generation for future (specific/unique) queries.

Table 1

User queries ‘always’ answered by the UPOU chatbot

Question	# of 'always' response
Application procedures	7
Program list	6
Requirements	5
What is distance education	1
Tuition fees	1
Important dates/events	1

Perceived Ease of Use

A little more than half of the study respondents (n=21, 51.2%) strongly agree that the UPOU chatbot was intuitive and easy to use, followed by 9 (22%) who agree, 8 (19.5%) are neutral, 1 (2.4%) who disagree, and then 2 (4.9%) who strongly disagree. The intuitiveness of the UPOU Chabot (Facebook chat box) may be attributed to the availability of ‘buttons’ and links in the chat box, which direct users to the information they need. On the other hand, for the AskOU-site-chatbot, the availability of sections for "FAQs," "news and announcements," etc. also make the chatbot intuitive and easy to use. Moreover, 23 respondents (56.1%) deemed the UPOU chatbot very useful. Followed by 6 (14.6%) who responded with "useful," 5 (12.2%) who responded "neutral," 4 (9.8%) who responded with "not useful," and 3 (7.3%) who responded with "not very useful." These results indicate that the majority of the users found the UPOU chatbot useful in terms of its ability to give them the information or response they need. These participants also view the usefulness of the chatbot as an innovative tool in relaying immediate information about UPOU.

Recognition and Facilitation of Users’ Goal and Intent

In terms of the UPOU chatbot’s ability to comprehend user queries and usefulness, 18 (43.9%) responded that the chatbot can understand their queries very well, followed by 9 (22%) who said that it did not really understand their queries, then 7 (17.1%) who were neutral.

Overall User Satisfaction

Overall, the majority of the participants were ‘very satisfied’ with the UPOU chatbot (n=17, 41.5%), followed by 9 (22%) neutral, 6 (14.6%) satisfied, 6 (14.6%) not satisfied, and 3 (7.3%) not very satisfied.

User Evaluation and Suggestions

Users shared the difficulties and issues they encountered with the UPOU chatbot. The most prevalent difficulties or issues were the following: inability to respond to specific/unique queries, inability to comprehend intent, and the ‘lacking’ knowledge of the UPOU chatbot (about office hours, application documents, certain steps in University processes, etc.). Here are some user responses:

“One time when I asked about the availability of Sablay in UPOU, since I was graduating, it gave me a link on how to shift the Sablay. I did not need that information since I was inquiring if I can get a Sablay from UPOU.”

“The chatbot can answer questions as long as you are direct to the point. But if you ask about something that it has no knowledge about, it gets annoying that it will just reply to you with "Please try rephrasing..." every time. Even if you try rephrasing your keywords for the chatbot to answer you, if it does not know the answer, it will just keep on telling you to rephrase your message.”

“I have specific questions when I messaged the Facebook page, but it seems that it cannot understand the question.”

“When my questions are very specific, the chatbot can't answer my question. And sometimes if I ask too many in a single message, it just catches some keywords and is not able to answer all of my questions.”

“Sometimes, the keywords the chatbot recognizes are not related to the ‘content’ of my query.”

These responses reflect user frustration with the UPOU chatbot being unable to understand their input text or intention very well, resulting in incomplete and incorrect chatbot responses. Seemingly, the UPOU chatbot cannot map the appropriate keyword/s in situational/unique queries, thus cannot generate a satisfactory response. Similar to Følstad et al. (2018), it seems the UPOU chatbot also has interpretational problems. Jain et al. (2018) suggest that chatbots need to address these issues and improve “to engage and retain users effectively” (p. 903). Aside from retaining users, retaining and enticing more individuals in pursuing UPOU education should be highlighted in improving the UPOU chatbot.

Additionally, there are survey responses wherein users stated their preference of human agents since the UPOU chatbot cannot satisfactorily give them a response. Jain et al. (2018) suggest that users prefer human agents over chatbots due to conversation efficiency issues. On the other hand, findings of Følstad et al. (2018) state that users find chatbots better in terms of asking questions without feeling judged and pressured, unlike with human agents. With these user feedbacks on the UPOU chatbot, there is a need to sustainably measure the queries satisfactorily completed by the UPOU chatbot against the queries that need human agents. Nonetheless, once a more varied and complete UPOU chatbot KB is available, perhaps users will no longer seek human assistance as the Chatbot has already assisted them satisfactorily.

Aside from human agents, some users also prefer that they be answered directly by the UPOU chatbot instead of the links:

“The chatbot could give direct answers to questions instead of just giving links.”

This concern is even expounded by one response from the users:

“I think the chatbot has to give me a direct answer sometimes. Because there might be occasions wherein a user contacted the UPOU main Facebook page through the service of Free Facebook. And if the chatbot redirects you to another link, you won’t be able to visit the link because you have no mobile data or wifi connection at the moment.”

The above statement can be a considerable improvement to the UPOU chatbot since free Facebook users were overlooked during the deployment of the UPOU chatbot. In the Philippines, some cellular service providers enable their users to access Facebook for free. However, if links cannot be totally removed from the chatbot responses, perhaps these could be made shorter “just so the message looks less bulky.” The preference of UPOU chatbot users for shorter links in the chatbot responses should be considered since users will be retained if external links are minimized (Jain et al. 2018).

Despite these issues with the UPOU chatbot, some users still commend the chatbot and have expressed their appreciation of the chatbot as an innovative tool in addressing public queries. Some positive feedbacks are:

“It makes communication with UPOU faster and better. The chatbot gives me immediate answers.”

“The chatbot feels personal when you talk to it. It responds to you with a “Hi!”, it mentions your name, it says thank you/good morning/good afternoon/good evening/ you’re welcome. It also introduces itself every time you start a conversation.”

These positive feedback are as significant as the negative ones. The presence of positive feedback may reflect improved information support services. Jain et al. (2018) emphasized that “chatbots should have an apparent personality suiting its domain...engage users in small talk and provide a personal touch…” (p. 901). However, the UPOU chatbot lacks concluding statements after the user’s conversation. As discussed in the studies of Jain et al. (2018) and Robinson et al. (2010), introductory and concluding phrases are expected of chatbots.

The users have also shared their suggestions on further improving the UPOU chatbot:

“If [the chatbot] can be improved like a messaging pop-up [or sidebar] in the UPOU website that does not close, it will add efficiency.”

“The chatbot needs to be integrated with very specific keywords for different concerns. Example, if I am to ask about which UPOU office sells the Sablay, the chatbot should not only answer me about how to shift the Sablay. It has to give me the name of the office, the office hours, and the amount to be paid.”

“Monitor the chatbot [conversations]. An actual human or employee who oversees the chatbot and how it responds to messages.”

In response to the third suggestion, one approach is already in place in the university wherein there is an assigned staff to monitor the UPOU chatbot on Facebook. This assigned staff documents all the conversations of the UPOU chatbot on Facebook, including the queries satisfactorily (complete and correct) answered, and the queries not answered by the UPOU chatbot. However, a monitoring scheme for the conversations on the UPOU chatbot via website is yet to be established. Monitoring results can be used to further enhance and expound the UPOU chatbot KB. This information then needs to be inputted and mapped with appropriate keywords into the UPOU chatbot KB to enhance its intelligence. The process needs to be done as long as there are user queries unresolved by the UPOU chatbot. As previously mentioned, UPOU chatbot developers could also explore automated KB updates. Improvements on the UPOU chatbot’s ability to comprehend user intent and conversation context are also needed. Since developments and improvements on the UPOU chatbot are at present manually done and requiring human resources and time, updates on this tool may not be immediate. Although not immediate, UPOU chatbot improvements are still necessary and should be maintained.

Conclusions and Recommendations

The UPOU chatbot was a tool developed and launched by the university to streamline its information support services to constituents. This study on the satisfaction of UPOU chatbot users resulted in mixed experiences. Results showed that the UPOU chatbot can always answer user queries, but it holds true to queries related to FAQs. The tool has issues in addressing complex, multiple, and specific/unique queries. Interpretational issues were also evident. Overall, the UPOU chatbot needs improvement as an information support tool. Improvements could focus on the lack of varied and complete information on its KB. However, the majority of its users still found this tool useful, intuitive, easy to use, and overall providing a satisfying experience. This study could be further expanded to explore the following: 1) user behavior and gender differences; 2) chatbot gender; 3) evaluation of chatbot responses per interaction; 4) the number of interactions/conversations per user; 5) user adaptation; and, 6) user retention. Future studies on the UPOU chatbot could also recruit a bigger and more diverse sample size of UPOU chatbot users with different experiences (e.g. first-time users, long-time users, one-time users, etc.). A more comprehensive study on chatbot key performance indicators influencing user satisfaction, analyzed using a usability scale, is also suggested. While the current study may be limited to only a few user satisfaction measures, this research could serve as a basis for succeeding chatbot investigations, especially for the UPOU chatbot.

References

Balaji, D. (2019). Assessing user satisfaction with information chatbots: A preliminary investigation [Master thesis, University of Twente]. http://essay.utwente.nl/79785/

Brandtzaeg, P.B., & Følstad, A. (2017). Why people use chatbots. In I. Kompatsiaris et al. (Eds.), Internet science: Lecture notes in computer science, 10673, 377–392. https://doi. org/10.1007/978-3-319-70284-1_30

Brandtzaeg, P.B., & Følstad, A. (2018, August). Chatbots: Changing user needs and motivations. Interactions, 25(5). https://dl.acm.org/doi/10.1145/3236669

Cahn, J. (2017). Chatbot: Architecture, design, & development [Senior thesis, University of Pennsylvania]. https://www.academia.edu/37082899/CHATBOT_Architecture_Design_ and_Development

Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. IEEE Access, 8, 75264–75278. https://doi.org/10.1109/ACCESS.2020.2988510

Crutzen, R., Peters, G.J.Y., Portugal, S.D., Fisser, E.M., & Grolleman, J.J. (2011). An artificially intelligent chat agent that answers adolescents’ questions related to sex, drugs, and alcohol: An exploratory study. Journal of Adolescent Health, 48(5), 514–519. https://dx.doi. org/10.1016/j.jadohealth.2010.09.002

Feine, J., Morana, S. & Gnewuch, U. (2019, 24–27 February). Measuring service encounter satisfaction with customer service chatbots using sentiment analysis [Conference session]. 14th International Conference on Wirtschaftsinformatik, Siegen, Germany.

Følstad, A., Nordheim, C., & Bjorkli, C. (2018, 24–26 October). What makes users trust a chatbot for customer service? An exploratory interview study [Conference session]. International Conference on Internet Science, St. Petersburg, Russia.

Følstad, A., & Brandtzaeg, P.B. (2020). Users’ experiences with chatbots: Findings from a questionnaire study. Quality and User Experience, 5(3). https://doi.org/10.1007/s41233-020- 00033-2

Gnewuch, U., Morana, S., Adam, M., & Maedche, A. (2018). Faster is not always better: Understanding the effect of dynamic response delays in human-chatbot interaction. Research Papers, 113. https://aisel.aisnet.org/ecis2018_rp/113/

Hien, H.T., Cuong, P.N., Nam, L.N.H., Nhung, H.L.T.K. & Thand, L.D. (2018). Intelligent assistants in higher-education environments: The FITEBot, a chatbot for administrative and learning support. In SoICT’ 18: Ninth International Symposium on Information and Communication Technology, 69–76. ACM. https://doi.org/10.1145/3287921.3287937

Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign. https:// curriculumredesign.org/our-work/artificial-intelligence-in-education/

Huang, J., Zhou, M. & Yang, D. (2007). Extracting chatbot knowledge from online discussion forums [Conference session]. 20th International Joint Conference on Artificial Intelligence.

Hussain, S., & Ginige, A. (2018). Extending a conventional chatbot knowledge base to external knowledge source and introducing user based sessions for diabetes education [Conference session]. 32nd International Conference on Advanced Information Networking and Applications Workshops.

Jain, M., Kumar, P., Kota, R., & Patel, S.N. (2018). Evaluating and informing the design of chatbots [Conference session]. Designing Interactive Systems Conference 2018, 9–13 June 2018, Hong Kong.

Liao, Q.V., Hussain, M.M., Chandar, P., Davis, M., Khazaen, Y., Crasso, M.P., Wang, D., Muller, M., Shami, N.S., & Geyer, W. (2018, 21-26 April). All work and no play? Conversations with a question-and-answer chatbot in the wild [Conference session]. CHI 2018, Montreal, Canada. https://doi.org/10.1145/3173574.3173577

Luckin, R., Holmes, W., Griffiths, M., & Forcier, L.B. (2016). Intelligence unleashed: An argument for AI in education. Pearson Education. http://oro.open.ac.uk/50104/

Ranoliya, B.R., Raghuwanshi, N., & Singh, S. (2017). Chatbot for university related FAQs. In 2017 International Conference on Advances in Computing, Communications and Informatics, 1525–1530. IEEE. https://doi.org/10.1109/ICACCI.2017.8126057

Robinson, S., Traum, D.R., Ittycheriah, M. & Henderer, J. (2010, 17–23 May). What would you ask a conversational agent? Observations of human-agent dialogues in a museum setting [Conference session]. 7th International Conference on Language Resources and Evaluation, Valletta, Malta.

Roll, I., & Wylie, R. (2016). Evolution and revolution in artificial intelligence in education. International Journal of Artificial Intelligence in Education, 26, 582–599. https://doi.org/10.1007/s40593- 016-0110-3

Serrano, J.V., Belegal, J.A.C., Cañas-Llamas, A.M.E.F., Petrasanta, L.P., & Almodiel, M.C. (2020). UPOU’s chatbot: Toward quality information services. In M.F. Lumanta, & P.G. Garcia (Eds.), Quality Initiatives in an open and distance elearning institution: Towards excellence and equity, 173–186. University of the Philippines Open University. https://networks.upou. edu.ph/32136/quality-initiatives-in-an-open-and-distance-e-learning-institution-towards- excellence-and-equity/

Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education, 151. https://doi.org/10.1016/j. compedu.2020.103862

Tariverdiyeva, G. (2019). Chatbots’ perceived usability in information retrieval tasks: An exploratory analysis [Master thesis, University of Twente]. http://essay.utwente.nl/77182/

Van den Broeck, E., Zarouali, B., & Poels, K. (2019). Chatbot advertising effectiveness: When does the message get through? Computers in Human Behavior, 98, 150–157. https://doi. org/10.1016/j.chb.2019.04.009

von Wolff, R.M., Nortemann, J., Hobert, S., & Schumann, M. (20). Chatbots for the information acquisition at universities - A student’s view on the application area. In A. Følstad et al. (Eds.), Chatbot research and design, 231–244. Springer. https://doi.org/10.1007/978-3-030- 39540-7_16

Wei, C., Yu, Z. & Fong, S. (2018, 26-28 February). How to build a chatbot: Chatbot framework and its capabilities [Conference session]. 10th International conference on machine learning and computing, Macau, China.

Winkler, R., & Söllner, M. (2018). Unleashing the potential of chatbots in education: A state-of-the- art analysis. Academy of Management Annual Meeting. https://www.alexandria.unisg. ch/254848/1/JML_699.pdf

Wu, Y., Wang, G., Li, W., & Li, Z. (2008). Automatic chatbot knowledge acquisition from online forum via rough set and ensemble learning [Conference session]. International Conference on Network and Parallel Computing.

Zarouali, B., Van den Broeck, E., Walrave, M., & Poels, K. (2018). Predicting consumer responses to a chatbot on Facebook. Cyberpsychology, Behavior, and Social Networking, 21(8), 491–497. https://doi.org/10.1089/cyber.2017.0518

Appendix 1

Post-interaction survey used for the study

Iska and IskOU: The UPOU Chatbot for Information Support Services

Good day! We, from the University of the Philippines Open University (UPOU), are conducting a study on the UPOU chatbot—Iska/IskOU. With this, we kindly ask you to accomplish this online questionnaire for our study. This study focuses on documenting and sharing UPOU's experience after its operationalization of the Iska and IskOU Chatbots to provide information to its students and to the public. The results of this study will be used both for research purposes and for the improvement of the UPOU chatbot.

All information disclosed in this survey will be used for research purposes only. We also assure the anonymity of your identities when we present the results of the study. If you have any questions, please feel free to reach us at [email protected].

Thank you and have a nice day!

1. Are you a UPOU Student?
○ Yes
○ No (Prospective student)
○ No (Just inquiring)
○ Other: _____________________

2. What questions do you usually ask the UPOU chatbot?
____________________________________________________________

3. Was the UPOU chatbot able to answer your questions promptly?
○ Always
○ Sometimes
○ Never
○ Other: _____________________

4. Was the UPOU chatbot able to answer your questions satisfactorily?
○ Always
○ Sometimes
○ Never
○ Other: _____________________

5. How well did the UPOU chatbot understand your question?

Very Well

Well

Neutral

Did Not Understand

Really Did Not Understand

6. Is the UPOU chatbot useful?

Very Useful

Useful

Neutral

Not Useful

Very Not Useful

7. Is the interface of the UPOU chatbot intuitive and easy to use?

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

8. Did you face any technical difficulty in using the UPOU chatbot? Please specify.
____________________________________________________________

9. Do you have any other suggestions to improve the UPOU chatbot? Please specify.
____________________________________________________________

10. Please suggest other ways of using the UPOU chatbot for learning purposes.
____________________________________________________________

11. Please rate your overall experience/satisfaction with the UPOU chatbot.

Very Satisfied

Satisfied

Neutral

Not Satisfied

Very Not Satisfied

International Journal on Open and Distance e-Learning, Vol. 7 No. 1, June 2021 Issue