download PDF
full article:
Introduction
There is a worldwide shortage of physicians in underserviced rural and remote areas, including some highly populated regions1. Most jurisdictions in developing and developed countries report shortages of rural doctors, often exacerbated where primary care services are not strong2,3. Rural workforce shortages have been identified by WHO as a significant barrier to universal, equitable health coverage2. The reasons behind the workforce crisis are multifactorial, but aging and expanding populations are the primary reasons identified worldwide. New health challenges and access to good quality primary care are now more important than ever4.
Rural generalism, or rural generalist medicine, has emerged in many jurisdictions as one approach to training and providing care for rural communities. Educational and training programs usually include developing appropriate skills in primary care, in-patient care, emergency medicine, public health, and one or more extended skills5,6, emphasizing meeting community needs. The Cairns Consensus Statement, developed at the first world summit in 2014, describes rural generalist medicine as illustrated in Figure 1. The provision of a broad scope of medical care by a doctor in the rural context that encompasses the following six points:
- comprehensive primary care for individuals, families, and communities
- hospital in-patient and/or related secondary medical care in the institutional, home or ambulatory setting
- emergency care
- extended and evolving service in one or more areas of focused cognitive and/or procedural practice as required to sustain needed health services locally among a network of colleagues
- a population health approach that is relevant to the community
- working as part of a multiprofessional and multidisciplinary team of colleagues, both local and distant, to provide services within a ‘system of care’ that is aligned and responsive to community needs7.
General practitioners (GPs) working in rural and remote areas can undertake a specialist training program designed to be competent and confident working in isolated regions8-10. Working in these settings without training instead of simply being ‘exposed’ to a rural and remote environment is considered insufficient to make the physicians capable of safely and competently serving in these communities11. Located far from health-resourced urban centres, these doctors often work with socially and economically disadvantaged communities and populations. A generalist approach provides physicians with the skills to operate in uncertainty and treat diverse conditions in patients over their lifespan, combined with an understanding of the disease prevalence and context in which the patient is situated to guide diagnosis and management12,13.
Many rural training programs worldwide have documented curriculum development processes, educational content, and descriptive studies of enrolled and graduating cohorts in training programs. A scoping review by Schubert et al (2018) explored the question ‘What is documented on rural generalist medicine?’ and identified 102 relevant articles internationally14. The authors noted contextual differences and similarities in various themes, including the definition of rural generalist medicine, training pathways and programs; scopes of practice and service models; enablers and barriers to recruitment; and retention and recommendations for reform. Despite variations, the authors observed that rural generalist medicine had been a feature of the healthcare system in countries with substantial rural and/or remote populations for some time14.
Figure 1: Illustration of rural generalist medicine.
Rural education pipeline and pathway
Medical education and training occur as part of a comprehensive and coordinated approach to support the provision of a sustainable medical workforce. Pong and Heng identified the ‘rural education pipeline’, and the pipeline model has helped redesign medical education to promote workforce development for physicians in rural communities15. The model describes a way to select, support, educate and produce physicians for rural practice and how medical education can support the recruitment and retention of rural physicians. These four stages are pre-college (high school/rural upbringing); undergraduate training; postgraduate training (interns, residency, fellowships, and vocational trainings); and professional career training (advanced training). Asghari et al (2019) argue in their recent systematic review that personal factors, training factors and practice factors have critical influence over recruitment and retention at each of these stages16. This suggested that recruitment and retention do not happen in a vacuum, highlighting the need for educational and workforce policies to be addressed at each of the three career stages.
Much of the related literature in medical education and rural medicine focuses on the second stage: medical school experience, particularly the rural curriculum and exposure during the undergraduate training. Rural clinical schools in Australia regularly report graduate rural career outcomes, which permits comparison of approaches and quality improvement of the program. However, comparatively little is reported from the postgraduate rural-based vocational training programs17.
This systematic review focuses on postgraduate rural training programs, the third stage in the pipeline/pathways. They are internships, residency trainings, fellowships and vocational training after completing undergraduate medical education. These are the critical points for many physicians when deciding on their career paths and procedural skills training. Providing targeted training in postgraduate medical education has been recognised as an important stage for successful recruitment and retention of the medical workforce18. Recently, medical schools have been involved in postgraduate education to increase the workforce in rural and remote areas in countries such as Australia and Canada. James Cook University General Practice Training Programme was established in 2016 to increase the workforce in rural and remote areas in Northern Queensland, and Northern Ontario School of Medicine in Canada has been actively promoting community engagement learning and the life cycle approach in medical education19.
Our review updates and expands the work of literature and research in this field and aims to identify common elements, best practices and current gaps in rural and remote postgraduate training curricula/evaluation by reviewing rural training programs' evaluation practices.
Global standard in medical education and curriculum development
The World Federation for Medical Education has defined and formulated international graduate and postgraduate medical education standards and continuing professional development20. The standards articulate the importance of developing and improving quality standards in medical education in a global context. Evaluation of the training process is one of the critical elements stated in these standards to enhance the quality of medical education. These standards require periodic assessments such as self-evaluations, peer-reviews or a combination of these methods. The outcomes of these assessments can also help develop strategies to assure quality improvement in medical education20.
There is substantial literature proposing and describing interventions taken by different stakeholders, including educational programs, education policies, monetary incentives and regulatory policies21-23. However, most studies are descriptive investigations of program development and delivery; few document program outcomes or impacts beyond the fulfillment of training schemes or in relation to its communities or various stakeholders in the rural and remote settings. Wakerman et al’s systematic review of the primary health care delivery model in rural and remote Australia addresses how inadequate evaluation of initiatives has failed to garner knowledge, which would have facilitated the establishment of evidence-based service models24.
WHO’s Guideline (2020) on health workforce development, attraction, recruitment and retention in rural and remote areas states that incorporating monitoring and evaluation is essential to assess design, implementation, outputs, outcomes and, ultimately, the impact of policies, and such efforts would help ensure a strong evidence base for policy development in the rural and remote workforce. In addition, the guideline suggests that capturing lessons learned from different contexts would promote understanding of when, why, how and in what circumstances interventions work well or fail25.
These previous studies have guided our research questions. In response to the call for a need to develop 'bundled interventions' in health workforce development, attraction, recruitment and retention in rural and remote areas25, this review investigates how postgraduate training and educational programs for rural and remote medicine are evaluated worldwide.
Research questions
Our research questions were:
- What are the outcomes of postgraduate rural and remote training programs worldwide?
- How are the programs/training evaluation conducted?
- What evaluation models and approaches are used in evaluating the effectiveness of these training programs?
Methods
A qualitative synthesis was conducted on evaluations of postgraduate rural training programs published in English-language medical education journals. The study involved pooling qualitative and mixed-method research data, and findings from qualitative studies were aggregated, integrated and interpreted.
Search strategies
PubMed, PsycINFO, ERIC and Web of Science databases were searched to identify studies that satisfied the inclusion criteria. This study was performed based on the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines26. It was conducted between December 2019 and July 2020 by reviewing all published English articles in the field of medical education and rural and remote health. Medical subject headings (MeSH) and Boolean operators were used to narrow, widen and combine literature searches to ensure relevant literature was captured in the search. This search focused on academic journals and articles indexed in databases. Grey literature was excluded from our search as the results were inconsistent with the published manuscripts.
Search terms in the database search encompassed five areas: location, practice area, timing, education and evaluation (Fig2). Using these parameters, 1297 articles were initially retrieved in the database searches.
Figure 3 summarises the review process involving data screening and extraction at four stages. DistillerSR systematic review software v2.35 (Evidence Partners; https://www.evidencepartners.com) was used to detect duplicates and navigate the process to keep track of records retrieved. Data charting forms were developed at each stage to capture information.
Inclusion criteria were:
- program is focused explicitly on generalism, primary care or family medicine – not specialist-driven training in rural and remote context
- length of the program is 12 months or longer
- 'rural and remote training' that represents rurality with geographical classification
- reported in the English language
- 'program' with intended goals
- study with a clear description of the study design and methods used.
For instance, our study excluded any short course or excursion/field visit from the ‘program’.
Of the 1297 articles initially retrieved, 105 articles were removed due to duplication and 1192 articles were retained for the abstract and title check at stage 2. Title and abstract screening were completed by applying five questions, and the full-text screening contained seven questions that were guided by the inclusion and exclusion criteria (Table 1).
At stage 3 screening, 136 articles passed the inclusion criteria, and 1056 articles were excluded. A total of 136 articles were reviewed closely and went through full-text extraction using 15 questions (Table 1). This resulted in 26 articles included in this systematic review. Data extraction sheets were developed by authors and pilot tested on three articles. Modifications were made to ensure that data extraction and analysis covered key themes to help respond to the research questions. Two independent reviewers screened retrieved publications against eligibility criteria; any disagreements were resolved by discussion in the review team.
Figure 2: Search terms and databases.
Figure 3: PRISMA flowchart of the systematic review.
Quality assessment of included studies
Two researchers who specialised in rural medicine (MS) and evaluation (TT) independently assessed the risk of bias of each included study, with disagreements resolved by discussion. Finally, a narrative synthesis of the content of resources meeting inclusion criteria was done using qualitative research methods to identify common themes pertaining to the study goals.
Table 1: Data screening and extraction questions at levels 1–3
Results
This section presents the findings from the 26 studies. First, trends in the methodological characteristics of the studies from the data extraction process are presented. The second part of the results section is organised around the three research questions of this systematic review.
Characteristics of the identified study
Table 2 summarises the program descriptions and research findings from the identified study27-52.
Table 2: Summary of included studies (n=26)27-52
|
Study no. [ref] |
Authors, year of publication |
Program Name |
Institution |
Region, country |
Evaluation objectives and evaluation design |
Outcome variables |
Evaluation design/ approach/model/method |
Results/impact/outcome of the program |
Implications and learnings from their evaluation results |
|
1 |
McKenzie A et al (2013) |
Advanced rural skills training |
Australian College of Rural and Remote Medicine (ACRRM) and Royal Australian College of General Practitioners (RACGP) |
Queensland, Australia |
To investigate doctors' experiences of support during GP advanced rural skills training, and identify strategies to improve support |
Questions about support for rural skills training - training and career planning, supervision and mentoring, and understanding and recognition |
Qualitative - structured questionnaire. Cross-sectional, postal survey |
Expanded support is required across the pre-enrolment, training, early practice continuum. Experiences and strategies to improve support were developed into a framework of support, consisting of three theme areas. Strategies included: provision of training and career advice to allow immediate use of advanced skills; introduction of rural attachments and rural case studies during training to ensure rural orientation; development of GP mentor and peer networks for clinical and non-clinical support; advocacy to improve understanding and recognition of advanced rural skills training |
Advanced rural skills trainees can be supported by providing career and training advice; ensuring the rural orientation of training; providing appropriate supervision, mentoring and peer support; and through advocacy to improve understanding and recognition of advanced rural skills training |
|
2 [28] |
Vyas R et al (2011) |
Fellowship in Secondary Hospital Medicine (FSHM) |
Christian Medical College Vellore |
India |
Program equips the graduates to practice effectively in small hospitals and provide them with social and academic support Evaluation tries to capture evidence on anticipated long-term outcome, particularly the career choice |
Strengths of the project work, recommendations for improvement, challenges faced while doing project work, how were challenges overcome, benefits of project to hospital |
Mixed method evaluation using focus group discussion with students written surveys for students and faculty, and telephone interviews with students and medical superintendents |
The major strengths of the project work identified were that students became aware of local health problems and how to deal with them, learned to work as a team, and had a sense of doing something useful. Recommendations for improvement were to have more interactions between guides and students. The benefits of projects to the hospital were providing improved clinical care, improved health systems, cost effective care management and benefits to the community |
Service learning through guided project work should be incorporated into distance-learning educational programs for junior doctors working in rural hospitals. This study demonstrates the effectiveness of guided project work as a teaching-learning tool for the FSHM, a distance-learning program. Medical colleges should therefore consider incorporating project work into educational distance-learning programs for supporting junior doctors working in rural hospitals |
|
3 |
Patterson DG et al (2019) |
Family medicine rural training track (RTT) residency programs |
Federal Office of Rural Health Policy etc. |
USA |
To identify risk factors threatening RTT sustainability and resilience factors enabling existing and developing RTTs to avoid closure and thrive |
Leadership, faculty and teaching resources, program support, finances, resident recruitment, program attributes, program mission, political and environmental context and patient-related clinical experiences |
Qualitative-semi-structured interviews of 21 RTT leaders representing two closed programs and 22 functioning programs |
Nine themes emerged, in order from most to least mentioned: leadership, faculty and teaching resources, program support, finances, resident recruitment, program attributes, program mission, political and environmental context, and patient-related clinical experiences |
Policies providing flexibility in residency program design, finance, and regulation, indeed any one or all of these, could improve the sustainability of small family medicine residency programs more broadly, including but not limited to RTTS |
|
4 |
Lewis MJ et al (2016) |
The New South Wales Rural Resident Medical Officer Cadetship Program |
NSW Department of Health |
New South Wales, Australia |
To identify practice location of former cadets of the entering class of 1989 and 2010 and where they are working in 2014 |
Geographical information, career choice and practice location, experience of being a cadet, influence on further career, relationship between practice location and career choice, (specialist etc.), recommendations to other students, importance of cadetship and period of rural service in influencing career path, influencing decision to practise rurally |
Qualitative-semi-structured questionnaire |
Of the 142 former cadets who responded to the questionnaire, 90 had completed a vocational training program and were working as fully qualified medical practitioners. |
The cadetship is an effective link between medical school and rural practice. The success of the program relies in part on the mentoring, networking and other educational opportunities available to cadets, which serve to foster their interest and provide a structured pathway to long-term rural practice. Targeted incentive-based scholarship schemes with a return-of-service component can be beneficial, particularly where they include ongoing support and reinforcement throughout the transition from undergraduate to postgraduate training |
|
5 |
Liskowich S et al (2015) |
Rural Family medicine training program-development of the Weyburn and Estevan test sites as rural family medicine training sites |
University of Saskatchewan |
Saskatchewan, Canada |
To develop a framework for a successful rural family medicine training program. |
Key predictors of success (physical resources, physician champions, physician teachers, educational support, administrative support, and other specialist support). Barriers to the development of a rural family medicine training site (differing priorities, lack of human resources, and lack of physical resources) |
Mixed-method design |
Important themes emerged: rural placement equals rural retention; stakeholder roles and perspectives are key predictors of success; the success of a program of this magnitude requires key resources |
The potential and hindrances in the establishment of a rural family medicine training site hinge primarily on the people rather than on the physical and financial resources. A project of this magnitude requires many people at different levels collaborating to be successful |
|
6 [32] |
Ross R (2013) |
Cascades East Family Medicine Residency, USA |
Cascades East Family Medicine Residency Program (CEFMR) is administered by Oregon Health & Science University (OHSU) |
Oregon, USA |
To measure the success, find out practice location of graduates and frequency of relocation |
satisfaction at the current hospital, procedures currently performing |
Qualitative-electronic version of the questionnaire with follow-up electronic, written and phone reminders |
Rural residency graduates tend to serve small communities. The outcomes demonstrated by this study substantiate this conclusion and show that rural residency graduates tend to serve small communities Although the study results presented are overwhelmingly supportive of current rural residency program design, these outcomes have been obtained with decreasing financial support |
Unless a comprehensive review of postgraduate training and funding is undertaken, with modifications of requirements based on the unique attributes of individual programs, there is imminent danger that the needs of rural America for primary care physicians will remain unfulfilled. There is a requirement for training incentives, federal funding, and Accreditation Council on Graduate Medical Education flexibility to support and accommodate the unique needs of training programs |
|
7 |
Frisch L et al (2003) |
The Kansas University School of Medicine-Wichita (KUSM-W) affiliated family practice residency programs |
Kansas University School of Medicine |
Kansas, USA |
To report the results of five separate surveys of graduates on practice location, practice information and satisfaction |
Location of their current and previous practice site locations, specific practice information, and global satisfaction with family practice |
Quantitative-surveys |
While most graduates remained in their initial practice site, almost half moved at least once. Those who left nonmetropolitan practices tended to move to less rural locations, though a few moved against this trend |
Further study is needed to determine whether family practice training programs can develop curriculum that will enhance their graduates' chances of selecting and remaining in remote rural counties |
|
8 |
Goertzen J (2006) |
Ontario family medicine program |
Three urban Ontario programs (McMaster University, University of Ottawa, and University of Western Ontario) and two rural programs (Family Medicine North and Northeastern Ontario) |
Ontario, Canada |
To determine and compare the assessment results of competencies in procedural skills of residents between urban and rural based training sites |
Self-rated experience and competence in performing procedures. Procedural competence upon graduation between rural and urban programs |
Quantitative and qualitative Self-assessed experience and competence scores through self-administered written survey |
Results indicated that no difference in mean procedural experience or mean competence scores between rural residents and their urban counterparts At graduation, mean experience and competence scores were significantly higher for rural residents than for their urban colleagues. A statistically larger proportion of residents graduating from rural programs assessed themselves as competent in 16 procedures |
The difference likely reflects the unique aspects of rural training sites, including preceptors' competence in performing procedures |
|
9 |
Jamieson JL et al (2013) |
Department of Family Practice postgraduate program |
Department of Family Practice at the University of British Columbia in Canada |
University of British Columbia, Canada |
To collect data from 20 years of graduates to identify personal and educational factors in postgraduate family medicine training that predict or influence future practice location, increase recruitment and retention in rural and regional practice |
Demographics, professional activities and practice location, predictors for practice location. Multiple logistic regression |
Quantitative-survey |
Location of postgraduate training is most predictive of professional practice location with those from distributed sites 15 times as likely to enter rural or regional practice. Males are 2.4 times as likely to enter rural or regional practice. No other variables reach significance in this model |
Participation in a distributed postgraduate family medicine training site is an important predictor of a non-urban practice location Curriculum is not the only influencing factors; opportunities to develop personal and professional roots in these communities is equally important Need for research to support the public policy direction of distributed postgraduate specialty education to improve geographic distribution of all physicians |
|
10 |
Dunbabin JS et al (2006) |
NSW Rural Resident Medical Officer Cadetship Program |
New South Wales Department of Health |
Australia |
To track the career choice and practice location of medical students entering the Cadetship Program before 1999 To comment on the impact of the Program on the rural medical workforce and its implications for the future workforce |
Gender, rural background, university, age, withdrawal from the program, practice location with rural, remote and metropolitan area (RRMA) classification, influence on background on career choice |
Quantitative-two brief questionnaires |
43% of cadets entering the Program before 1999 were working in rural locations in 2004 Career choice was the major determinant of practice location Having a rural background did not appear to influence practice location; whereas, those specialising in general practice made up 70% of this cohort of cadets working in rural areas |
Providing vocational training opportunities in rural locations is central to the success. Provision of vocational training opportunities needs to be considered in efforts to expand the rural specialist workforce, and in ensuring rural health capitalises on the increasing number of medical students moving through the education and training system in the next 4-10 years. |
|
11 |
Rosenthal TC et al (1997) |
Rural Residency Tracks in Family Practice |
The Residency Review Committee for Family Practice and the American Board of Family Practice |
USA |
To provide descriptive analysis based on self-reported data from survey |
Program structure (sponsor (university-administered programs etc.), program enrollment, practice sites |
Quantitative |
76% of graduates enter rural practice after graduation |
The program is meeting the goal of providing trainees with a rural immersion experience, in anticipation of selecting rural practice after graduation. Decentralising resident education should add value to the education experience by assuring residents more community-relevant experiences and modeling the generalist-specialist relationship appropriately. Rural training tracks may offer another model of family practice education that addresses the country's needs where distribution of health care and access remains acutely problematic |
|
12 |
Blondell RD et al (1993) |
Family practice residency training at the University of Louisville |
University of Louisville |
USA |
To determine the effectiveness of this program by looking at factors that were associated with a physician's choice to practice in rural or urban medically underserved areas |
Age, gender, ethnic group, location |
Quantitative - survey |
49 graduates serve patients in rural or medically underserved urban areas, and of those, 31 are located in Kentucky Whites and those from rural hometowns were more likely to practice in rural communities than minorities or those from urban hometowns. However, minority groups were more likely than those identifying as white to practice in medically underserved urban areas No association was found between gender and practice location Initial interest in rural practice and participation in financial aid programs requiring service in a rural area were associated with rural practice location, but participation in rural rotations at the predoctoral or residency training level were not |
The future success of the residency program depends upon the success of medical schools to produce students who are interested in becoming generalist physicians with an inclination to locate in rural areas The greatest challenges are to attract students into the program and the ability to receive adequate funding to support its educational activities |
|
13 |
Heng D et al (2007) |
Northeastern Ontario Family Medicine (NOFM) residency program in Sudbury and the Family Medicine North (FMN) program in Thunder Bay |
Northeastern Ontario Family Medicine (NOFM) residency program in Sudbury and the Family Medicine North (FMN) program in Thunder Bay |
Canada |
To examine the impact of the training program on a physician's first, as well as continuing, practice location using chi-squared tests |
Location |
Quantitative - logistic regression analyses |
There was no statistically significant difference between the graduates of the two programs (FMN and NOFM) in the likelihood of working in either northern Ontario or a rural area. About two-thirds (67.5%) of all person-years of medical practice provided by NOFM and FMN graduates took place in northern Ontario or rural areas outside the north |
The results add to the growing evidence that rural or northern medical education and training increases the likelihood that the graduates will practice in rural or northern communities Effective rural and northern medical education must be a cornerstone of any long-term physician workforce strategy designed to address the inequitable distribution of medical practitioners in Canada |
|
14 |
Norris TE et al (1988) |
Montana family medicine program |
Montana Family Practice Residency Satellite Program |
USA |
To evaluate the effect of a rural preceptorship on interest in rural practice and on practice site selection |
Whether a rural rotation influenced choice of practice site. Effects of rotation on interest in rural practice opportunities. Size of residents' practice sites (metropolitan community, nonmetropolitan community) |
Quantitative - data gathered from second- and third-year family practice residents |
A majority of participants (n=123) felt that this experience influenced their choice of a practice site. A large majority felt that it increased their interest in rural practice opportunities Rural preceptorships during residency are a timely solution to increase the number of family physicians interested in rural practice |
Exposure to high-quality rural family practice training site during residency would increase the resident's interest in rural practice, especially when accompanied by spouse and family Rotation would influence the choice of a practice site for some of the residents |
|
15 |
Nash LR et al (2008) |
University of Texas Medical Branch at Galveston (UTMB) Family Medicine Residency |
University of Texas |
USA |
To describe practice locations, and strengths and limitations of the program as perceived by its participants |
rural training tracks (RTTs) strength and limitations through focus group evaluations in 2004 and database from UTMB Family Medicine residency alumni database |
Quantitative |
Most graduates of the RTT (6 out of 7) entered practice in rural areas. Four of the 7 RTT graduates are Hispanic, reflecting the program's success in attracting underrepresented minority physicians |
Expansion of the number of training positions in the RTT would be expected to increase the impact of the program and the supply of rural family physicians for Texas |
|
16 |
Mugford B et al (2001) |
Preregistration rural intern-training position based in rural general practice |
Flinders Medical Centre, Flinders University and the rural communities of Cleve and Jamestown |
South Australia |
To describe the experience of interns over the period 1999-2000 |
Functional status and wellbeing, clinical outcomes, satisfaction and costs |
Quantitative and qualitative Assessed the value of the Flinders Medical Centre rural internship using The Clinical Training Compass - a quality improvement tool to look at both quantitative and qualitative outcomes. (Clinical outcomes, Functional status and wellbeing, Satisfaction, and costs data) |
Rural general practice term provides a wide breadth of clinical experience in a well-supervised environment. The ability to practice procedures, not only just observe them, is greater than that available in a tertiary teaching hospital for a doctor at this level of training There is a blend of hospital and community work that is appropriate for doctors who cannot be expected to be fully comfortable with working only in the ambulatory care environment |
New programs that promote vertical integration of under and postgraduate rural experiences must be made available to junior doctors, appropriately evaluated for the experience they provide and promoted as sites that provide the capacity for acquisition of procedural skills and supplying a wide range of clinical experiences under expert supervision. Despite being a very significant period in the life of newly graduated doctors and, potentially, a time when career choices are made, there has been reduced access to general practice and very little rural experience |
|
16 |
Beach DA et al (1998) |
The Northern Family Medicine Education (NorFam) Program |
Memorial University of Newfoundland |
Canada |
To review the NorFam Program and discusses current practice locations and the relevance of the program |
location, type of practices, felt program relevant for northern practice? |
Qualitative - telephone interview |
Program focused on practical skills, teaching and cross-cultural awareness Program gave them more relevant training for northern practice than they would have received in an urban-based family medicine program |
Suggested to extend the program to accommodate the specialised skills, and the program extension requires additional funding |
|
18 |
MacVicar R et al (2016) |
Scotland's GP Rural Fellowship program |
NHS Education for Scotland |
Scotland |
To describe the Fellowship program and its output from 2002 to 2013 with survey results |
Practice location, working in general practice, current GP role |
Quantitative - survey with quantitative data and one case study to illustrate the context |
The program represents a successful collaboration between education and service, and the results of the survey underline previously unpublished data suggesting that approximately three-quarters of graduates are retained in important roles in rural Scotland The Fellowship can have in specific locations that may otherwise provide recruitment and retention challenges |
Major aim of the Fellowship is to impact on recruitment to and retention in rural practice, also for provision of acute care in remote hospitals The need for and opportunity to develop a structured approach to the development of competencies for acute care in such settings has been highlighted. Further work will be progressed to develop this approach and to evaluate the impact and outcome |
|
19 |
Malau-Adul BS et al (2019). |
GMT Generalist Medical Training |
James Cook University |
Australia |
To validate the Scan of Postgraduate Educational Environment Domain (SPEED) tool and assess its appropriateness in evaluating the quality of General Practice (GP) rural postgraduate educational environment |
Education environment satisfaction |
Quantitative - online survey |
The cross-validation process confirms that the SPEED is a concise tool with only 15 items that can be used to measure the quality of the educational environment in postgraduate GP training with high degrees of validity and reliability |
A high degree of validity and reliability of the SPEED tool for the measurement of the quality of the educational environment in a rural postgraduate GP training context. However, when applied in a new setting, the tool may not function as a multidimensional tool consistent with its theoretical grounding |
|
20 |
McKendry RJ et al (2000) |
Family medicine training programs in Ontario |
Family medicine training programs in Ontario |
Ontario, Canada |
To learn and compare results from summative examinations - the Medical Council of Canada Qualifying Examination (MCCQE) Part II and the College of Family Physicians of Canada (CFPC) certification examination - between graduates from urban centres and remote and rural settings |
Mean CFPC certification examination scores for trainees in remote and urban family medicine programs. Simulated office oral and short-answer management problems |
Quantitative Results of the two summative examinations - the Medical Council of Canada Qualifying Examination (MCCQE) Part II and the College of Family Physicians of Canada (CFPC) certification examination |
1013 graduates of family medicine programs in 1994-1999 completed the CFPC certification examination; a subset of 663 completed both the MCCQE Part I and the MCCQE Part II. The MCCQE Part I results for graduates in the remote programs did not differ significantly from those for graduates entering the programs in the medical schools. The MCCQE Part II results did not differ significantly between the 2 groups either There were no consistent, significant differences in the results of the CFPC certification examination between the two groups |
The learning was comparable between trainees in urban family medicine programs and those in rural, community-based programs. Results suggested that postgraduate training at remote sites was not inferior to that provided at medical schools. Further questions related to roles of medical schools and attributes of the teaching and training are raised |
|
21 |
Robinson M et al (2013) |
Bogong Regional Training Network (BRTN) |
RACGP |
Australia |
To explore career path decisions |
Location = retention |
Mixed methods (both qualitative and quantitative)
Registrar file data were examined, and sociodemographic profiles of GPs compiled |
The decentralised model of GP training resulted in the retention of doctors in rural areas, but the extent to which the model has influenced GP retention rates is a complex issue - the interrelationships between work related and personal/lifestyle satisfaction factors in the overall social, political, economic and cultural environment need to be considered. There is a positive association between rural training and rural retention for some doctors. 42% of doctors who trained with BRTN remained in the region. Separate data shows that the decentralised training model has had a positive impact on the workforce in the Bogong region |
This study supports the continued development of decentralised and rural specific training pathways for Australian doctors as an effective method for attracting and retaining doctors in rural Australia during and after general practice training it is part of a longitudinal study that is examining the causal relationship between decentralised training and longer-term retention of doctors in rural and regional practice. The results of that study should contribute further to practical policy outcomes for governments and training providers |
|
22 |
Wearne S.et al (2010) |
The Remote Vocational Training Scheme (RVTS) |
RVTS, Australia |
Australia |
To document the training location, outcomes of training, current location and services provided by program graduates and assess the effectiveness of their training |
Program completion, service in rural or remote areas, current location, current service provision, effectiveness of RVTS training |
Quantitative - online survey |
The RVTS program of distance education and supervision, clinical teaching visits and face-to-face workshops was an effective model of general practice training for these registrars. These registrars provide significant service to rural and remote communities during and after completion of their training. The majority (81%) of RVTS graduates continue to work in rural areas (RRMA 3-7) after completing the program Graduates value the support and social network provided by the scheme as well as the emphasis on the extended skills needed in emergency and remote medicine |
These results correlate with findings that the level of professional and social support influence intentions to leave or stay in rural practice and that improving psychological wellbeing can influence these intentions. Further research is planned on how the RVTS model of training could apply to other medical specialties and other general practice settings where registrars are in isolated practice such as the Australian Defense Force or those serving overseas in humanitarian aid projects |
|
23 |
Straume K et al (2010) |
The new primary care internship initiative in Finnmark |
The results are reported annually to the Ministry of Health |
Finnmark, Norway |
To explore the results can be utilised as a tool for retention and understand if the training model had been effective |
The post-training retention of primary care physicians |
Quantitative - data analysis using existing data |
Postgraduate medical training can be successfully carried out in remote areas in a manner that ensures professional development and counteracts professional isolation, resulting in improved health workforce retention in rural settings By allowing for postgraduate medical training in remote areas, trainees and their families grew roots in rural communities during training |
Postgraduate medical training can be conducted in remote areas in a manner that ensures professional development, counteracts professional isolation and allows trainees and their families to grow roots in rural communities. Rural practice satisfies modern principles of adult learning (problem-based and attached to real-life situations) and offers excellent training conditions |
|
24 |
Straume K et al (2010) |
Postgraduate training for physicians in Finnmark |
Norwegian Medical Association |
Finnmark, Norway |
To evaluate training programs by looking at physicians' retention in Finnmark by a longitudinal cohort study |
Number of graduates accepting the licensed jog in the region, location after 5 years' training |
Quantitative |
The vacancy rates for primary care physicians in Finnmark has improved during our observation. Professional support is a crucial element of retention programs in Norway Of the 80-90 primary care physicians working in Finnmark today, there is only a handful who have not completed or is currently in one of the programs in this county. With this background the five-year retention rate of 65-67% is satisfactory |
Traditional centralistic training models are challenged. Postgraduate (vocational) training (residency) for primary care physicians can be successfully carried out in-service in remote areas, in a manner that enhances retention without compromising the quality of the training. According to modern principles of adult learning, training should be problem-based and attached to real-life situations, and rural practice offers excellent opportunities for such training |
|
25 |
Bayley SA et al (2011) |
Regional training provider |
Regional training provider |
New South Wales, Australia |
To examine the assumptions that postgraduate training would enhance recruitment to rural practice, and how that assumption would be played out in the postgraduate/GP workforce development efforts |
Participant demographics, structured in-depth interviews, pre-term, term outcomes |
Quantitative and qualitative |
Though generally a rewarding clinical learning experience, negative aspects of the rural placement included the disruption to personal lives of rural relocation and the stresses involved in higher levels of clinical responsibility. These stressors could undermine rather than enhance clinical confidence Anxiety and depression were accompaniments for some registrars. Intention to practice rurally was little influenced by this compulsory placement |
The positive clinical learning experience of most registrars in rural terms must be balanced with the social dislocation involved in rural relocation and the adverse effects of the rural experience, for some registrars, on professional confidence and psychological wellbeing |
|
26 [52] |
Sen Gupta T et al (2020) |
ACRRM Fellowship program |
ACRRM |
Australia |
To assess the program's achievement by utilising tailored modalities: |
Formative assessment based on the rural generalist curriculum |
Quantitative |
The assessment takes a programmatic approach with use of multiple modalities and feedback to candidates and designed to be undertaken by candidates in or near their own community. The results demonstrated the growth in ACRRM's assessment program. The majority (56%) of these were via the Australian General Practice Training program, with 37% via the Independent Pathway, and 7% via the Remote Vocational Training Scheme |
The RVTS program of distance education and supervision, clinical teaching visits and face-to-face workshops was an effective model of general practice training. 'Tele-assessment' model using the distance technology may provide some insights for other medical colleges and educational institutions facing challenges in the current environment |
Articles ranged in publication date from 1988 to 2020, from six countries. The geographical distributions of the included studies are as follows (as illustrated in Figure 4): nine programs from Australia27,30,36,42,45,47,48,51,52, six from Canada31,34,35,39,43,46, seven from the USA29,32,33,37,38,40,41, 2 from Norway49,50, one each from the UK44 and India28. Australia and Canada are leading this rural/remote practice, and the USA has a long history of developing curriculum and training programs. The result is consistent with the recent systematic reviews25,53; Australia, Canada and the USA have a strong evaluation, assessment, credentialling and certification culture. The objectives and purposes of the training programs were directly linked to health system responses to the shortage of physicians working in rural and remote areas aimed at increasing recruitment and retention in the regions and countries where the programs were established.
Figure 4: Geographical distribution of included studies (n=26).
Research question 1: What are the outcomes of postgraduate rural and remote training programs worldwide?
Evaluations of education programs are fundamentally about the changes that have occurred and been captured during and after the program. Nine themes were identified as contributing factors for the outcomes of training programs as a result of qualitative coding and analysis:
- practice location after training completion
- training location and decentralised models
- educational aspects
- incentives, political contexts and regulations
- personal, social and cultural issues
- professional development
- rural orientation and community engagement
- support systems
- gender and racial issues.
Practice location after training completion: Thirteen studies identified practice location as the positive outcome of the training programs. Rural residency graduates tend to serve small communities32; location of postgraduate training is most predictive of professional practice location35; most graduates remained in rural practice locations31,33,37,39,44,48,50; exposure to a high-quality rural family practice training site during residency increases the likelihood that the graduates continue to practice in rural communities39; expansion of the number of training positions in the program contributes to the increase of the impact of the program and the supply of rural family physicians41.
A linkage between rural background and the practice location was addressed in two studies. One study concluded that having a rural background was a strong factor associated with choosing a rural location30, whereas another study concluded that having rural background did not appear to influence practice location36.
Training location and decentralised models: Ten studies identified the linkages between training location and decentralised training models as the key contributing factors for the positive outcomes. Providing vocational training opportunities in a rural location is central to the success of the program36; decentralising resident education adds value to the education experience by assuring residents more community-relevant experiences37,38 and resulted in the retention of doctors in rural areas47; the program’s success is characterised by the collaboration between education and service44 and contributed to professional development49; postgraduate training at remote sites was not inferior to that provided at medical schools in urban sites46.
A few studies indicated that traditional centralistic training models should be challenged. Postgraduate training for primary care physicians can be successfully carried out in-service in remote areas in a manner that enhances retention without compromising the quality of training25. Distance education and supervision, clinical teaching visits and face-to-face workshops were effective general practice training models52. Furthermore, the ‘tele-assessment’ model using distance technology may provide some insights for other medical colleges and educational institutions facing challenges in the pandemic52.
Educational aspects: Seven studies highlighted educational aspects as the key outcomes. Incorporating project work (eg service-learning with guidance) into educational programs was effective28. Producing students who are interested in becoming generalist physicians38; utilizing assessment and measurement tools to measure the quality of the educational environment45,52; providing well-supervised environment while providing a wide breadth of clinical experiences42; providing practical skills, teaching and cross-cultural awareness43; and educational programs that are grounded in learning theories (eg problem-based and attached to real-life situations) offer excellent training conditions49.
Incentives, political contexts and regulations: Considerations on human and financial resources are essential for program development and delivery. One study addressed that the potential and hindrances in establishing a program hinge primarily on the people rather than on the physical and financial resources31. Training incentives and financial support are crucial32, including targeted incentive-based scholarship schemes30 for sustainable education, workforce development, and recruitment and retention. Policies providing flexibility in program design, finance and regulation could improve the sustainability of small family medicine residency programs29.
Personal, social and cultural factors: The interrelations between work-related and persona/lifestyle satisfaction in the overall social, political and economic and cultural environment need to be considered when looking at retention rates47. Improving psychological wellbeing is crucial as it can influence intentions to leave or stay in rural practice48. One study pointed out that the clinical learning experience is generally rewarding, but negative aspects of the rural placement included the disruption to personal lives of rural relocation, and the stresses involved in higher levels of clinical responsibility are stressors51. These stressors could undermine rather than enhance clinical confidence. Furthermore, anxiety and depression were accompaniments for some registrars and needed to be considered for professional confidence and psychological wellbeing51.
Professional development: Supporting the development of professional networks and career pathways was addressed in three studies. The transition from undergraduate to postgraduate training can be better supported by targeted incentives and the return-of-service component of the training30; vertical integration of undergraduate and postgraduate rural experience must be made available to junior doctors, appropriately evaluated for the experience they provide and promoted based on their skills and experiences42, and providing training with other health professionals such as nurse practitioners would be beneficial43.
Rural orientation and community engagement: Curriculum and program contents get most of the attention in considering program development and evaluation. However, studies indicated that curriculum is not the only factor influencing success; opportunities to develop personal and professional roots in rural communities are equally important35. Furthermore, the program's strengths come from students’ increased awareness of local health issues and how to deal with them28. It has a more prolonged impact as trainees, and their families, grew roots in rural communities during training49.
Support systems: Several studies indicated that graduates of training programs value the support and social network provided by the scheme. Developing a framework of support is critical27. Strategies for support would include giving career advice, introduction to rural specific cases, development of GP mentor and peer networks for clinical and non-clinical support, and advocacy/recognition27. One of the studies showed a correlation that the level of professional and social support influences intentions to leave or stay in rural practice48.
Gender and racial issues: Findings from two studies raised issues related to gender and race. Two studies are from training programs in the USA, and the programs succeeded in attracting underrepresented minority physicians41. Minority groups were more likely than those identifying as white to practice medically underserved areas, while no association was found between gender and practice location38.
Research question 2: How are the program evaluations conducted?
Program evaluations are conducted to investigate whether the training and educational programs have met the goals and objectives. Four inquiry modes were identified: descriptive orientation, future orientation, comparative studies, and validation of program effectiveness and impact.
Descriptive orientation: Descriptive orientation was the most dominant form of inquiry and reason for conducting an evaluation. Studies described the experiences of the program and the graduates37,42,44, capturing evidence on long-term outcome28,50, reporting the results on practice location, practice information, satisfaction30,32,33,40 and program effectiveness38,39,48,49. A few studies tracked career choice and career path decisions36,47 to draw implications for the future workforce and discuss program relevance43. Furthermore, studies investigated doctors’ experiences of support during the training and identified strategies to improve support27, identified personal and educational factors that predict or influence future practice location, increase recruitment and retention in rural and regional practice35, and identified strengths and limitations of the program41.
Future orientation – identify risks and develop a framework: Evaluations were used intentionally for future prospects. Two studies indicated that their purposes were to identify risk factors and resilience factors that are threatening the sustainability of the program’s rural training track and further developing the program to thrive29 and to develop a framework for a successful rural family medicine training program, and to assess the potential for a rural family medicine residency training program30.
Comparative studies: Two studies focused on the program evaluation from the approach of comparative studies. One study compared the assessment results of competencies in procedural skills of residents between urban- and rural-based training sites34, and another study compared the results from summative examinations between graduates from urban centres and remote and rural settings46.
Validation of program effectiveness and impact: Three studies assessed the effectiveness and impacts by using assessment tools and examining assumptions. One study used the Scan of Postgraduate Educational Environment Domain (SPEED) tool to validate and assess its appropriateness in evaluating the quality of general practice rural postgraduate educational environment45. One program in Australia evaluated program achievement by utilizing tailored modalities for assessment52. During the program development stage, one training program examined the assumptions that postgraduate training would enhance recruitment to rural practice and how that assumption would be played out in the postgraduate/GP workforce development efforts51.
Research question 3: What evaluation models and approaches are used in evaluating the effectiveness of these training programs?’
This current review has identified theoretical frameworks and assessment tools used to demonstrate the outcomes of their training programs in the included studies (n=26). Figure 5 illustrates the findings. The most frequently used evaluation method was survey, including paper-based, telephone and internet surveys, which intended to capture demographics and descriptive data (31%)33-35,37,38,44,45,48. Other evaluation methods were in-person, telephone and internet interviews (19%)29,42,43,47,51, questionnaires capturing experiences, reflections and insights (15%)27,30,32,36, use of secondary data from existing databases from program administrative offices (15%)39,40,49,50, mixed methods using both surveys and interviews (8%)28,31, exam scores of assessment tools such as qualifying examination results or ratings that third parties produced (8%)46,52, and focus groups (4%)41.
Out of the 26 programs identified, three studies incorporated and grounded their analysis on theoretical frameworks while conducting the evaluations. Two used adult learning theory49,50 and another grounded theory (constructivist paradigm)51. Various assessment tools were used or reported to support the validity of their program and training outcomes. These include both qualitative and quantitative assessment tools:
- The Clinical Training Compass – a quality improvement tool for both qualitative and qualitative outcomes
- Scan of Postgraduate Educational Environment Domain (SPEED)
- The Medical Council of Canada Qualifying Examination (MCCQE)
- The College of Family Physicians of Canada (CFPC)
- multi-source feedback
- Mini Clinical Evaluation Exercise (Mini-CEX)
- case based discussion
- procedural skills logbook
- Structured Assessment – Multiple Patient Scenarios (StAMPS).
Identification of evaluation methods and approaches revealed that evaluation design and methods are used interchangeably with measurement and assessment approaches. Information necessary for program evaluation is typically gathered through measurement processes. The choice of specific measurement tools, strategies or assessments for program evaluation processes is guided by many factors, including the specific evaluation questions that define the desired understanding of the program’s successes and shortcomings. It is evident that data from trainee assessments are essential to the program evaluation process. However, there is a lack of linkage between theoretical frameworks for evaluation and assessment tools to capture appropriate indicators for monitoring and evaluating program strategies to achieve educational objectives.
Figure 5: Evaluation methods represented in the studies.
Discussion
Qualitative analysis of the content of the 26 identified articles revealed several themes related to the structure and content of educational programs specific to rural and remote generalist medicine. This section presents the methodological characteristics and the outcome themes. WHO’s 2020 Guideline25 is used as a thematic framework to highlight the findings and draw implications for further studies.
Methodological characteristics
Most of the evidence from review studies consists of descriptive studies with longitudinal tracking programs and cohort studies. Furthermore, most studies involved single-site/institution training and educational programs. There was no study using the randomised controlled trials (RCTs) in the identified studies. RCTs are quantitative, comparative, controlled experiments in which investigators study interventions in a series of individuals who receive them in random order. The RCT is known as one of the most powerful tools in clinical research. However, no RCT studies were identified in this systematic review.
Two characteristics emerged during the analysis. First, there was a lack of consistency in pre- and post- surveys while assessing and evaluating the impact and effectiveness of program delivery and curriculum development. A few studies were conducted using quantitative and comparative methods, such as χ2 tests to examine the impact of the training program39 and longitudinal cohort study on physicians’ retention50. Methodologies selected for most of the studies were prone to self-selection bias: voluntary respondents with positive experiences.
The current evidence from studies primarily based on self-reported surveys and questionnaires is inconclusive as to whether effective postgraduate education and training contribute directly to the recruitment and retention of physicians in rural and remote settings. These methods are subject to non-response, sample selection bias, and subjectivity–objectivity debate. Our analysis suggested that research and evaluation in training programs require a coordinated, integrated and evidence-based approach.
Thematic analysis using the 2020 WHO guideline
Nine themes were identified as contributing factors for the outcomes of training programs in the 26 studies. Identified themes (practice location after training completion; training location and decentralised models; educational aspects; incentives, political contexts and regulations; personal, social and cultural issues; professional development; rural orientation and community engagement; support system; and gender and racial issues) are analysed and cross-validated against the 2020 WHO guideline. Seventeen recommendations are provided in four themes: education, regulation, incentives, and professional and personal support, and corresponding studies are illustrated in Table 3.
WHO endorses bundled intervention for development, attraction, recruitment and retention of health workers in rural and remote areas. Bundled intervention is defined as 'a group of evidence-based interventions put together into a package'25. WHO asserts that collective interventions produce better outcomes than when delivered separately when implemented. Figure 6 illustrates our analysis of key outcomes and interpretation of data using the concept of bundled intervention and the 17 recommendations provided in the 2020 WHO guideline.
Recommendations on education (1–5) are well supported by the key outcomes in this review. Locating health education facilities closer to rural areas (recommendation 2), exposing students of a wide array of health worker disciplines to rural and remote communities and rural clinical practices (recommendation 3), including rural health topics in health worker education (recommendation 4), and designing and enabling access to continuing education and professional development programs that meet the needs of rural health workers to support their retention in rural areas (recommendation 5) are supported by corresponding studies with evidence. Recommendations on regulation (6–9) show weak relevance in the review, while recommendations related to incentives (10) are articulated in several studies. Recommendations on professional and personal support aligned closely with issues discussed in the identified studies. For example, multiple studies addressed that identifying and implementing appropriate health workforce support networks (recommendation 14) and having a policy for career development, advancement programs, and career pathways for health workers (recommendation 15) were essential for health workers in rural and remote areas.
Table 3: 2020 WHO Recommendations25 and relevant studies in the systematic review27-32,34-39,42-44,47-52
Figure 6: Analysis of studies based on the 2020 WHO guideline on health workforce, development, attraction, recruitment and retention in rural and remote areas (17 recommendations)25.
Rural background and program outcomes
The 2020 WHO guideline acknowledges that enrolling students with a rural background in health worker education programs shows a moderate level of certainty of evidence25. In many high-quality studies, the rural background is deemed an essential predictor of workforce recruitment and retention54. Our review identified one study with a conclusion that a rural background was a strong factor in choosing a rural location30. Another study highlighted the importance of gender and race in the context of rural background. One study from the USA concluded that minority groups were more likely than people identifying as white to practice medically underserved areas while no association was found between gender and practice location38.
Rural background remains an important aspect for educational planning, curriculum development and workforce development in rural and remote areas. However, our review identified a link between general practice/family medicine and rural interests/intentions38,50. One study stated that the future success of the residency program depends upon the success of the training programs to produce and educate students who are interested in becoming generalist physicians with an inclination to locate and work in rural areas38.
Decentralised training, telehealth and tele-assessment
The current novel coronavirus (COVID-19) pandemic has challenged the provision of rural and remote training for health professionals. Traditionally, rural and remote clinicians and students travelled significant distances to large regional hubs or urban centres to receive training. Although most of the included studies were from the pre-COVID-19 period, lessons can be learned from those that address positive outcomes of decentralised training, telehealth and tele-assessment. A comparative study in family medicine training in Canada concluded that the learning was comparable between trainees in urban family medicine programs and those in rural, community-based programs46. The decentralised model of GP training in Australia also concluded that the decentralised training model had had a positive impact on the workforce47. A study from Norway asserts that rural medical training and practice satisfies modern adult learning principles (problem-based and attached to real-life situations) and offers excellent training conditions49.
If rural and remote general practice training programs can continue to deliver robust and reliable training through virtual means during COVID-19, rural and remote physicians and students will benefit, and their communities will gain improved access to health and medicine. The possibility of telehealth and tele-assessment has accelerated rapidly during COVID-19. Assessment and evaluation are becoming an integral part of the program improvement and assessment as illustrated in the tele-assessment model using distance technology in Australia52.
Evaluation as a strategy to uncover program and training outcomes
Medical educators and training providers spend much of their time developing and delivering educational programs. Program evaluation is essential for determining the value of the teaching provided55, identifying whether or not the program meets its intended objectives, and providing recommendations for improvement or modification56.
The WHO guideline asserts that incorporating monitoring and evaluation is essential to assess design, implementation, outputs, outcomes and ultimately the impact of policies25. Our findings support their view that the absence of strong evaluation designs and their alignment to the program objectives will lessen the strengths of evidence. Better quality research and evaluation design, objective settings, and developing appropriate indicators and benchmarks for monitoring and evaluating strategies must be considered during program development and implementation.
The increasing interest in evidence-based medical education has produced a large volume of research and evaluation that concerns the outcomes of what works and does not work. Medical education research and evaluation studies can be classified into descriptive studies (What was done?), justification studies (Did it work?) and clarification studies (How did it work?)57. The results from this systematic review suggest that a greater emphasis has been placed on descriptive studies. This review captured a few justification and clarification studies; however, there has been a missing link of program logic between ‘what’ these programs claim to accomplish and ‘how’ things are working in relation to training and program objectives.
While evaluation and research share methodologies and designs, they are distinctively different. The Accreditation Council for Graduate Medical Education defines program evaluation as the 'systematic collection and analysis of information related to the design, implementation, and outcomes of a graduate medical education program for monitoring and improving the quality and effectiveness of the program'58. What distinguishes evaluation from research is that evaluation involves making a value judgement59, and program evaluation use information to decide an educational program's value or worth55.
From the curriculum development perspective, Thomas et al describe the six-step approach in curriculum development in medical education with several underlying assumptions60. The first step starts with problem identification and general needs assessment, and the sixth step is evaluation and feedback, which is particularly relevant for our discussion. Our systematic review revealed that evaluation (step 6) is frequently lacking in curriculum development and program implementation. Thomas et al argue that, 'For a successful curriculum, curriculum development never really ends. Rather, the curriculum evolved, based on evaluation results, changes in resources, changes in targeted learners, and changes in the material requiring mastery.' (pp 8–9)60.
Study limitations
Publication bias exists in this study. There is a significant limitation in omitting grey literature and unindexed research in the inclusion strategy. This resulted in the included studies mainly being based on self-reported surveys and interview results, which can be highly subjective. The identified body of evidence from this synthesis of peer-reviewed studies reporting the program and training outcomes may not be proportional to the actual activities occurring worldwide. The authors are aware that a substantial number of evaluation reports, case studies, and related grey literature exist, some of which have been conducted by external evaluators, including auditing and consulting companies. An example is the Evaluation and Investigative Study of the Queensland Rural Generalist Program conducted by Ernst & Young for Queensland Health, Office of Rural and Remote Health61. In 2012, Ernst and Young reported an external evaluation of the rural generalist pathway. The Evaluation and Investigative Study of the Queensland Rural Generalist Program explored the pathway’s conception, implementation and progress. In addition, the Australian College of Rural and Remote Medicine has recently published an evaluation report62. It includes evaluation questions that focus on clarification – what is working and how it works – by asking in-depth, exploratory questions.
Conclusion
This systematic review of postgraduate medical training programs explored three questions related to program evaluation and training outcomes. The findings suggest the need for rigorous evaluation of postgraduate medical programs that demonstrate effectiveness of educational interventions. The evidence gap between program content/delivery and effectiveness needs critical attention. WHO encourages more case studies in monitoring and evaluation practices that include contextual issues, intervention details with evaluation, confidence levels readily identified and the opportunity to investigate the effectiveness in sensitive information63.
Evaluation plays an essential role in analyzing the program's effectiveness and producing a sufficient number of influential rural generalist physicians to provide comprehensive and high-quality medical care specific to the needs of rural communities. Challenges exist, especially in developing mechanisms to track health workforce data, securing funding for evaluation to understand the life cycle of rural and remote health workers. Lack of funding is also a challenge. Longitudinal studies on the impact of rural and remote training programs can build evidence and increase research and evaluation transparency. Furthermore, information sharing and identifying specific predictors for future programs would benefit both education and policy arenas in rural and remote medicine.
Acknowledgements
We thank Ronald McCoy at the Royal Australian College of General Practitioners and Jasmine Millman at the University of the Ryukyus, Graduate School of Medicine in Okinawa, Japan, for reviewing the manuscript. We also thank Chris L. Coryn and Michael A. Harnar at Interdisciplinary Ph.D. in Evaluation at Western Michigan University, USA for providing valuable comments in the earlier version presented in a poster format.
references:
You might also be interested in:
2008 - Forty years of allocated seats for Sami medical students - has preferential admission worked?
2007 - 'Perimeteritis' and rural health in Manitoba, Canada: perspectives from rural healthcare managers