Outcomes of a faculty development conference in Mongolia

Background: Medical Education International (MEI), an organization that provides faculty development to medical educators in developing countries, wanted information on the program effectiveness of its faculty development conferences. Objectives: To assess the outcomes of an MEI faculty development conference in Mongolia on the knowledge, confidence in applying new skills, and attitudes of participants. Methods: A retrospective pretest survey of participants was used to assess the outcomes of a 3-day faculty development conference given twice at the Mongolian National University of Medical Sciences. The survey assessed participant views on their ability to meet the objectives of the conference, the perceived overall value of the conference, and their suggestions for improvements in future MEI conferences. Results: Twenty participants (65%) completed surveys. Participants reported significant changes in agreement with their ability to meet the objectives of the conference in all of the pre-post measures (pre-post p<0.001). The value of attending the conference was ranked at a mean score of 4.05 on a Likert scale from 1 to 5, with 1 indicating “Strongly Disagree” and 5 “Strongly Agree.” Conference attendees indicated interest in additional training on more advanced topics. Conclusion: Overall, the findings indicate that conference attendees gained knowledge and confidence in applying new skills and valued the training received from a faculty development conference led by physicians from the USA. Further research is needed to determine long-term impact on residency education in Mongolia. 39 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. Introduction The last three decades have brought a significant increase in the number and type of professional development programs for medical school faculty members, but the outcomes of many of those programs are not well studied. 1 This leaves faculty members and educational leaders without strong evidence to guide decisions about optimal faculty development activities for institutions and individuals. Increasing faculty and university time and financial constraints require that resources are used in the most productive manner possible. Medical Education International (MEI, online at https://cmda.org/missions/detail/mei) is an organization that provides conferences on request to faculty members in developing countries on clinical or faculty development topics. The Dean of the Mongolian National University of Medical Sciences in Mongolia, a developing country in central Asia, requested a conference on faculty development. After discussion about content and logistics, MEI agreed to provide this university with two identical faculty development conferences, each lasting three days, on residency education. The outcomes of MEI conferences have not been previously studied, and this project aimed to assess conference outcomes. Mongolian medical education has experienced significant change in recent years. Mongolia was a communist country until the dissolution of the USSR in 1990, with centralized control over education and clinical services. Since then, the country has been rapidly advancing technologically and in the depth and breadth of its clinical services. Many new, high-technology hospitals have opened. To keep up with these new opportunities and challenges, higher education has expanded and the number of faculty members in Mongolian universities doubled in ten years between 1997 and 2007 to 6,818 full-time faculty. 2 Mongolia has a long history of having medical schools. However, the first residency training programs were not started until the mid1990s. The duration of residency training for most specialties has increased from 18 to 24 months. Fellowship programs have also been started, although they can be as short as 6 months. Because of the rapid expansion in medical education, many new faculty members have little experience in the faculty role. All faculty members, regardless of country, require faculty development to meet the needs of the changing medical and educational environments. 3 For Mongolia, the dramatic changes and growth make faculty development for residency program directors and other educational leaders imperative. In general, active learning strategies are less commonly used in Asian schools, and courses are often lecture-based. 4 Formal faculty development in teaching and programs to promote strong educational leadership are also more limited in Asia than in westernized countries. The leaders of the Mongolian National University of Medical Sciences (MNUMS) have identified the improvement of residency training programs as a goal. Faculty development is essential to achieve this priority. To date, faculty development at MNUMS has been provided through sabbaticals, grants, awards, mentoring, visits to observe in other countries, and workshops. 2 Staff and faculty members of MNUMS report that many international visitors from diverse groups travel to Mongolia to deliver conferences on clinical topics but none have addressed faculty development in education. Previous studies have addressed the outcomes of faculty development conferences, predominantly in westernized countries. Outcomes have been assessed mainly by changes in faculty attitudes towards teaching, increases in knowledge, and improvement in teaching skills and behaviors. 5 The outcomes of one conference were assessed at a higher level on Kirkpatrick’s levels of training evaluation by identifying the percentage of participants who successfully implemented faculty development projects at their home institutions. 6 Little information is available on the outcomes of faculty development conferences in developing countries, especially those using a 40 Kroeker & Walling Nov 2016. Christian Journal for Global Health, 3(2):38-49. program conducted by individuals from Western countries. A PubMed literature search identified no information on the utility of such faculty development conferences in Mongolia. One study by Wong and Fang describes the outcomes of a Western-based faculty development seminar in a developing country in Asia. 7 This study provided the model for the assessment of this Mongolian MEI conference. Further research on this topic will provide useful information for the development of future MEI programs in developing countries and provide information for other educational organizations that offer similar conferences. This study seeks to answer the broad research question, “what are the outcomes of the MEI faculty development conferences on the knowledge, confidence in applying new skills, and changes in attitudes of the faculty participants?” The study of conference attendees identifies the extent to which they agree conference objectives were met, their perceived value of the conference, and their suggested areas for improvement. Faculty development can be defined as “a planned program designed to prepare institutions and faculty members for their various roles and to improve an individual’s knowledge and skills in the areas of teaching, research [and] administration.” 8 As learner requirements and innovative techniques for teaching, evaluation, and assessment evolve, faculty members must be well-prepared in both the theory and application of appropriate methods to effectively guide and evaluate learners. 3 According to the Accreditation Council of Graduate Medical Education (ACGME), program evaluation is defined as the ‘‘systematic collection and analysis of information related to the design, implementation, and outcomes of a resident education program, for the purpose of monitoring and improving the quality and effectiveness of the program.’’ 9 The hypothesis is that this three-day, MEI faculty development conference will enable participants to gain knowledge and confidence in developing residency training programs, enhance positive attitudes towards educational techniques, and report that time at the conference was valuable.


Introduction
The last three decades have brought a significant increase in the number and type of professional development programs for medical school faculty members, but the outcomes of many of those programs are not well studied. 1 This leaves faculty members and educational leaders without strong evidence to guide decisions about optimal faculty development activities for institutions and individuals. Increasing faculty and university time and financial constraints require that resources are used in the most productive manner possible.
Medical Education International (MEI, online at https://cmda.org/missions/detail/mei) is an organization that provides conferences on request to faculty members in developing countries on clinical or faculty development topics. The Dean of the Mongolian National University of Medical Sciences in Mongolia, a developing country in central Asia, requested a conference on faculty development. After discussion about content and logistics, MEI agreed to provide this university with two identical faculty development conferences, each lasting three days, on residency education. The outcomes of MEI conferences have not been previously studied, and this project aimed to assess conference outcomes.
Mongolian medical education has experienced significant change in recent years. Mongolia was a communist country until the dissolution of the USSR in 1990, with centralized control over education and clinical services. Since then, the country has been rapidly advancing technologically and in the depth and breadth of its clinical services. Many new, high-technology hospitals have opened. To keep up with these new opportunities and challenges, higher education has expanded and the number of faculty members in Mongolian universities doubled in ten years between 1997 and 2007 to 6,818 full-time faculty. 2 Mongolia has a long history of having medical schools. However, the first residency training programs were not started until the mid-1990s. The duration of residency training for most specialties has increased from 18 to 24 months. Fellowship programs have also been started, although they can be as short as 6 months. Because of the rapid expansion in medical education, many new faculty members have little experience in the faculty role.
All faculty members, regardless of country, require faculty development to meet the needs of the changing medical and educational environments. 3 For Mongolia, the dramatic changes and growth make faculty development for residency program directors and other educational leaders imperative. In general, active learning strategies are less commonly used in Asian schools, and courses are often lecture-based. 4 Formal faculty development in teaching and programs to promote strong educational leadership are also more limited in Asia than in westernized countries. The leaders of the Mongolian National University of Medical Sciences (MNUMS) have identified the improvement of residency training programs as a goal. Faculty development is essential to achieve this priority. To date, faculty development at MNUMS has been provided through sabbaticals, grants, awards, mentoring, visits to observe in other countries, and workshops. 2 Staff and faculty members of MNUMS report that many international visitors from diverse groups travel to Mongolia to deliver conferences on clinical topics but none have addressed faculty development in education.
Previous studies have addressed the outcomes of faculty development conferences, predominantly in westernized countries. Outcomes have been assessed mainly by changes in faculty attitudes towards teaching, increases in knowledge, and improvement in teaching skills and behaviors. 5 The outcomes of one conference were assessed at a higher level on Kirkpatrick's levels of training evaluation by identifying the percentage of participants who successfully implemented faculty development projects at their home institutions. 6 Little information is available on the outcomes of faculty development conferences in developing countries, especially those using a  7 This study provided the model for the assessment of this Mongolian MEI conference. Further research on this topic will provide useful information for the development of future MEI programs in developing countries and provide information for other educational organizations that offer similar conferences.
This study seeks to answer the broad research question, "what are the outcomes of the MEI faculty development conferences on the knowledge, confidence in applying new skills, and changes in attitudes of the faculty participants?" The study of conference attendees identifies the extent to which they agree conference objectives were met, their perceived value of the conference, and their suggested areas for improvement.
Faculty development can be defined as "a planned program designed to prepare institutions and faculty members for their various roles and to improve an individual's knowledge and skills in the areas of teaching, research [and] administration." 8 As learner requirements and innovative techniques for teaching, evaluation, and assessment evolve, faculty members must be well-prepared in both the theory and application of appropriate methods to effectively guide and evaluate learners. 3 According to the Accreditation Council of Graduate Medical Education (ACGME), program evaluation is defined as the ''systematic collection and analysis of information related to the design, implementation, and outcomes of a resident education program, for the purpose of monitoring and improving the quality and effectiveness of the program.'' 9 The hypothesis is that this three-day, MEI faculty development conference will enable participants to gain knowledge and confidence in developing residency training programs, enhance positive attitudes towards educational techniques, and report that time at the conference was valuable.

Methods
The conference presenters were volunteers selected by MEI from a list of board-eligible or board-certified physicians from the USA who had indicated interest in presenting at conferences and whose areas of expertise corresponded to the needs identified by the host organization (MNUMS). The team presenters for this Mongolian conference included an academic otolaryngology physician who served as an international residency program director, an academic psychiatrist who was previously a residency program director, an associate program director of a pediatric residency program, a retired academic general surgeon with expertise in teaching in mission hospitals, and a hepatobiliary surgeon who was a recent fellowship graduate.
The conference took place in the capital city of Mongolia, Ulaanbaatar, at the MNUMS. Faculty attendees were chosen by the deans of the university and consisted of heads of departments in the university and directors of the residency programs in several specialties and subspecialties, including general surgery, otolaryngology, urology, orthopedics, internal medicine, ophthalmology, pediatric gastroenterology, pulmonology, and hematology/oncology. The 3-day conference was given twice in one week to accommodate schedules for two different groups of faculty. In total, about 18 faculty members participated in the first conference and 13 in the second conference. Participants occasionally left and returned to the conference during the day as needed based upon their other professional responsibilities.
The conference format (see Table 1) was composed of lectures and small-group workshops in which faculty members worked together to discuss and expand on different conference topics. Following each small group breakout session, the large group reconvened and discussed the ideas from the small groups as a large group. All participants who were present at the end of each of the conferences were offered a paper survey to complete anonymously prior to leaving the conference room. Because all faculty participants were offered a survey, the entire conference population was studied; sampling was not used. A mixed-methods survey was developed in conjunction with the faculty participants on the team. The data were obtained concurrently and each type of data was given equal weight. The interpreter reviewed the survey and provided feedback prior to use. The survey was written in English, while participants were able to respond in English or Mongolian.
The cross-sectional study was based upon the results of a retrospective pretest method that used both a Likert-type scale to provide quantitative information and short answer questions to provide qualitative data. A retrospective pretest consists of a survey given to conference participants at the end of the conference. It asks the participants to rate themselves on how much they knew about specific topics before the conference began and then to rate themselves again based upon how much they know at the end of the conference. The difference between the two responses indicates knowledge gains during the conference. 10 This method of assessment can improve the accuracy of the data obtained by a traditional pretest/posttest format. Participants who rate their knowledge using a traditional pretest may be unaware of what they don't know, and be unable to accurately assess their knowledge level before the conference. In contrast, at the end of the conference, participants can better judge how knowledgeable they were before the conference. 10 Validity of the qualitative results was ensured by relaying the qualitative results in rich, descriptive language to ensure a realistic portrayal of the situation. Additionally, two reviewers, one with no connection to MEI or the conference, reviewed the data independently. The reviewers analyzed the data until they came to mutual agreement on the themes and results. The principal investigator was open to finding areas upon which to improve for future conferences and open to negative opinions and findings. This provided an opportunity for voices with negative opinions to expand the available data.
Grounded theory, a research design in which a theory is generated from the perspectives of participants, was used for analysis of the qualitative data, and reliability of the qualitative data was ensured. 11 The survey reports were rechecked for errors during analysis. Themes were identified and coded. Any disagreements between coders were discussed until consensus was reached.
Validity of the quantitative data was also protected.
The survey used was developed specifically for this project using the retrospective pretest methodology that has been used for similar studies. The survey questions were previewed by the Mongolian interpreter who provided feedback on the instrument prior to implementation. The survey was also reviewed by multiple physicians with suggestions incorporated into the survey prior to use.
The survey (see Appendix A) was printed on a double-sided sheet of paper and given to the participants at the end of each of the three-day conferences. The purpose of the survey was described so the attendees were aware that the sole purpose was to improve future conferences for other attendees; no additional incentives were offered for completion of the survey. The surveys were written in English, but the participants could complete the surveys in English or Mongolian, according to their preferences. The interpreter was available for participant questions or clarification on the meaning of any of the English words or phrases. No identifying information was collected or requested on the surveys to ensure anonymity. Completion of the survey implied consent for inclusion in the study. The attendees left the surveys in the room for collection after they had completed them. The anonymous surveys were placed in a manila envelope for return to the United States for analysis. All answers in Mongolian were translated into English by the conference interpreter who was fluent in both English and Mongolian. The surveys were analyzed by the University of Kansas School of Medicine Office of Research members of the study team.
This study was done with the approval of both the MEI director and the leader of the conference. Conference attendees were invited to participate, but participation was optional. While the participants completed the surveys without benefiting immediately, they may benefit if they attend any future MEI conferences that are improved in response to their feedback. Additionally, the survey results will benefit other future MEI conference attendees in Mongolia and other developing countries. No harmful or identifying information was collected, and the individual surveys will not be shared with those in authority over the participants. Because this was a program evaluation rather than a research project, it received IRB exemption status.
The study team analyzed the results for statistical differences between the "before" and "after" responses of the participants using the Wilcoxon Signed Rank test. All tests were 2-tailed and alpha was set at 0.05. The short-answer data was reviewed for emergent themes using grounded theory, a theory in which meaning is gleaned from the information provided by the participants through their responses on the surveys. 11 Themes were identified independently by two reviewers and final themes determined by consensus. The results of the qualitative and quantitative sections were analyzed in a convergent manner to determine how the results relate to one another. The data provided information on the outcomes of the MEI conferences on the knowledge, skills, and attitudes of faculty participants in Mongolia.

Results
Eighteen faculty leaders participated in the first conference, and thirteen participated in the second conference. Of the 31 participants, 20 completed the survey for a response rate of 65%. All faculty members who were present at the end of the conferences completed the survey.
Analysis of the survey results indicated significant changes in the faculty level of agreement in all of the pre-post measures with a p<0.001 (see Table 2.) The most significant change was noted in familiarity with competency-based medical education, with an average pre-conference score of 2.5 and an average post-conference score of 4.5. The areas in which faculty most strongly indicated agreement at the end of the conference were those of being able to build a needs assessment (mean 4.7) and being able to develop goals and objectives for a rotation or class (mean 4.7). The area that faculty ranked as the weakest at the end of the conference was that of being able to give effective summative assessments (mean 4.3). Even this, however, is still a significant increase from a mean pre-conference response of 3.1.  Participants ranked the value of attending the conference at a mean score of 4.05 (see Table 3). They ranked the helpfulness of the small group discussions with colleagues between 4.26 (discussion on how to implement competencies into the medical education system) and 4.63 (discussion on the ideal structure of residency training). Participants rated the helpfulness of the small group discussions higher than they rated the value of attending the conferences. Respondents indicated that they had received between 0 and 10 days of faculty development on similar topics in the past, with an average of 2.3 days of previous training (see Table 4). There was a wide range of responses when asked about the percentage of material that was new to participants. Two (10%) respondents indicated that 20-30% of the material was new, while four (20%) indicated that 80-100% of the material was new to them (see Table 5). On average (mean 57%, [range 20-100%]) about half of the conference material was reported as new to the attendees.  The most common participant response to the question asking which part of the conference was most helpful was the response "all" (see Appendix B). While participants were reluctant to report anything negative, one response was that some parts of the conference were not sufficiently advanced for the group; the other respondents indicated that there were no negative outcomes. Participants suggested the conference could be more useful by using a translator who was familiar with residency education, by including more details and examples in the presentations, and by having more smallgroup sessions. They suggested the conference could be more culturally appropriate by including "more education and practice." Elsewhere, participants reported that more examples would be helpful; this could mean that participants would like examples from their local universities and contexts. Respondents also suggested that the provision of continuing education for participants at different levels would be desirable.
When asked how MEI could support the faculty in further development of their residency programs, responses included access to all the curriculum and documents used during the conference, and more training. When asked about specific topics for additional training, participants indicated topics specific to their disciplines, education methods for residency programs, and learner assessments.

Discussion
We used a retrospective pretest survey method with additional qualitative questions to evaluate the outcomes of a faculty development conference for medical educators in Mongolia. The qualitative questions also provide information to inform future faculty development conferences, as the responses have information on what was perceived as helpful or not, and suggestions on how to improve similar conferences in the future.
Overall, we found that participants valued and learned from the conference. The most common respondent answer to the question, "What part of the conference was most helpful?" was "all." Participants had received minimal training in residency education (mean of 2.3 days) in the past, indicating a gap in previous faculty development training. This was consistent with their relatively low self-rating on the "before" section of the survey. Many participants also requested more training in the future, and participants requested the handouts and presentations used in this training for future reference.
While one person wrote that "we have so many needs and issues," others indicated that they would like more advanced training, including more "detailed information on different educational methods." This, as well as the comment that participants would like education specific to participants of different levels of experience, indicates that many Mongolian faculty members already have a basic level of knowledge and would like to expand that knowledge. It also shows their passion for professional growth and development. The only negative outcome of the conference identified was one person's response that "some part[s were] not advance[d]" enough. Requests for additional training included "more detailed information and examples on training," and "practice from another country." Prior to planning any future faculty development conference, conference leaders would benefit from surveying potential participants on their level of training and comfort using various skills.
The assessment of conferences in non-Westernized developing countries requires an awareness of the cultural context and how this might impact both the conference and the validity and reliability of an assessment. In the Confucian belief system, which is common in Mongolia, hierarchy is respected, and a strong power differential exists between teacher and learner. In contrast, Westernized cultures have a much more individualistic paradigm, with power differentials playing a much smaller role in organizations. 12 These cultural differences have many implications for the educational environment of a conference. In Westernized cultures, questioning a professor is not only acceptable; it is often viewed positively as a demonstration of critical thinking. For a student from an Eastern culture, questioning an authority figure is disrespectful; students are less likely to ask questions since that might insinuate that the teacher had not adequately explained the topic. Presenters with a Western mindset may assume that all learners understand the material if no questions are asked, whereas the learners may have many questions that they do not ask out of respect for the Westerners. If presenters from the West are not aware of and responsive to these different perspectives, learning can be impacted.
To modulate this challenge, the conference was arranged with frequent workshops after a presentation (see Table 1). During the workshops, the participants worked together to discuss and apply the material that had just been presented. Leaders of the conference were available to provide feedback to attendees during those sessions, and participants provided feedback to each other as well. This provided a non-threatening environment in which participants asked many questions.
One respondent commented on the importance of a having a content-knowledgeable interpreter for the conference. A highly-trained physician was the interpreter for the first conference, and she facilitated clear communication. The first day of the second conference, we were given two interpreters, and neither was as familiar with residency education; they occasionally asked for clarification of terms. On the second and third day of the second conference, another interpreter completed the interpretation for us, and this was more effective. The presenters adapted to these challenges by speaking more slowly and using descriptors to assist the interpreter with technical terms.
This study had several limitations, confounding factors and barriers. The first is that the study population was only those faculty members who participated during the entire three days of the conference and were still present at the end of the third day. This could have led to a sampling error biased positively towards the utility of the conference since those who attended the entire conference were likely to be those who perceived it as most beneficial. This could also have impacted the number of surveys completed since some participants left early without completing a survey. Additionally, the study population could have been depleted of the busiest faculty members who needed to attend to other responsibilities during parts of the conference.
Language was another barrier. The survey was written in English, while Mongolian is the participants' primary language. The participants indicated that they read in English and declined having the surveys translated into Mongolian. Participants were informed that they could answer the survey in Mongolian; Mongolian responses were translated into English by a Mongolian physician who is fluent in both English and Mongolian. While this should result in an accurate translation, there is a possibility of a loss of some of the nuances between the Mongolian and English languages. Also, as discussed above, a contentknowledgeable interpreter is imperative for optimal communication.
Providing a conference for participants from a different culture is challenging. One response indicated that the system in Mongolia is significantly different from the Western system with which the presenters were accustomed, stating, "in my opinion our philosophy and goals is different from western countries." This can have implications on the transferability of the information presented on the USA's residency education system. Additionally, several comments mentioned the need for administrative leaders to participate in the conferences and to make changes in the Mongolian residency education system.
The assessment of conferences in non-Westernized developing countries requires an awareness of the cultural context and how this might impact both the conference and the validity and reliability of an assessment. Since learners from Eastern cultures are careful not to shame their instructors, they may have withheld criticism and felt obliged to indicate that they learned a lot and appreciated the conference, regardless of their true opinions. To ensure that research is reliable, it is imperative that researchers work within the cultural belief system in a manner that can obtain accurate information. Using a neutral third-party to obtain feedback may improve the accuracy of data obtained from faculty participants.
Strengths of this study included the ability of attendees to provide written, anonymous, qualitative data without having to give feedback verbally to presenters. Additionally, the survey could be completed in English or Mongolian, based upon participant preference. Finally, all participants who were present at the end of the conference participated in completing the surveys.
As a result of this project, MEI faculty development conferences have introduced surveys to evaluate the outcomes of conferences. This affirms that MEI leadership values the information provided by the survey and has taken action to evaluate the outcomes of conferences. Based upon feedback from the surveys, MEI has changed how conferences are led by making sessions more interactive. The compilation of feedback from additional conferences will provide information that is more valid and generalizable.
It might be helpful if teams providing a faculty development conference survey participants in advance of the conference to obtain information on the current level of participant knowledge and specific areas of need. This would help presenters tailor the conference to the needs, level and interests of the participants. It could also serve to stimulate interest in and provide realistic expectations for the conference.

Conclusions
Overall, our findings indicate that conference attendees valued the faculty development conference led by physicians from a Westernized country. Our findings also indicate the importance of including administrative leadership from the host country in faculty development. This will facilitate discussions regarding changes in the training system at the structural level and foster a