download PDF
full article:
The Rural Clinical Schools (RCS) program was implemented by the Australian Government in the year 2000 as a workforce strategy to address the shortage of medical practitioners within rural and remote Australia1. The RCS program was merged with the Rural Undergraduate Support & Coordination Program (RUSC) in 2011 and is currently known as the Rural Clinical Training & Support (RCTS) program1. The RCTS program is implemented across Australia in 18 RCS and regional medical schools (RMS).
In the early years of the RCS, the primary focus was recruiting medical students and addressing health workforce shortages, as well as encouraging health professionals to become academics2. In 2004, there were only 10 RCSs in operation, with a further seven being established throughout the decade. Research, although clearly important, was not a part of their operational parameters in this era. However, RCSs and RMSs are well placed to perform targeted rural health research to serve the unique needs of rural residents and better inform rural health policy3,4. In recent years, with the primary workforce and educational objectives having been satisfactorily addressed, rural health research has been included as one of the nine funding parameters of the RCTS program5.
Currently, there is a large number of publications about the RCTS program and medical student cohorts that have undertaken short- and long-term rotations6-8. However, very little is known about the academic and professional staff involved in the program, and their contribution to rural health research. The first 2014 RCTS Snapshot survey of professional and academic staff employed by the Australian RCTS program was recently completed with the support of the Federation of Rural Australian Medical Educators (FRAME)9. This article reports the results of a PubMed review of RCTS research for the decade 2004-2013.
A bibliometric method using PubMed was carried out to retrieve RCTS publications. When searching PubMed, advanced queries were used that included keywords (Medical Subject Headings - MeSH) and 'Tags'10. Tags are used to identify specific parts of a PubMed article. The following tags were used to retrieve relevant RCTS publications: Affiliation (AD) - identifies an organisation or postal/email address of the first author, and occasionally co-authors; Medical Subject Heading (MeSH) - papers linked to a specific topic/subject; and Title/Abstract (TIAB) - papers with a word or phrase in the title or abstract. The LIMIT option is used to define specific time periods.
A condensed version of a validated PubMed query to retrieve Australian publications3 was combined with RCTS-related search terms targeting the affiliation and title/abstract tags (Table 1, query 1) to accurately retrieve MEDLINE-indexed RCTS publications from 2004 to 2013. Two authors (JB & KM) independently checked all retrieved RCTS publications for validity.
Table 1: PubMed search queries (original queries first published in Mendis et al3)
Australian rural health (ARH) publications were defined as Australian publications that explore issues relevant to the health of the regional, rural or remote Australian population3. An ARH query (Table 1, query 2) was used to determine total ARH publications in the decade 2004-2013 so that the proportion of RCTS publications in this total could be determined.
To selectively enumerate RCTS publications related to rural health (Table 1, query 4), a rural health query (Table 1, query 3) was combined with the refined RCTS query (Table 1, query 1). Other publications from RCTS researchers were calculated by subtracting the rural health publication numbers from total RCTS publication numbers for each year examined. Confirmation of correct assignment into both categories (rural health/other) was determined during the validation process described above.
Medical education publications in the rural health category and other category were enumerated by checking all retrieved RCTS publications from 2004 to 2013.
Medical subject heading (MeSH) terms 'rural health services' and 'rural population' were selectively enumerated by searching within the RCTS publication results to tabulate the respective subject areas (Table 1, queries 5 and 6). RCTS publications specific to Indigenous health (Table 1, query 7), National Health Priority Areas (NHPA; Table 1, query 8), and National Rural Health Alliance Priority Areas (NRHAPA; Table 1, query 9) were also enumerated (queries first published in Mendis et al3).
The journals most frequently publishing RCTS research articles in the decade examined were determined by inspection of search results.
A total of 280 RCTS publications were retrieved for the decade analysed, increasing from 10 in 2004 to 49 in 2013 (Fig1). ARH publications from RCTS researchers increased from 8 (80% of total) in 2004 to 30 articles (61% of total) in 2013 (Fig1). RCTS rural health publications increased as a proportion of all ARH publications from 3.4% in 2004 to 7.7% in 2013.
Figure 1: Total PubMed publications by Rural Clinical Training and Support researchers from 2004 to 2013.
Other RCTS publications increased from 2 (20% of total) in 2004 to 19 (39% of total) in 2013 (Fig1). Examination of the abstracts of all other RCTS publications showed that 23 of the 104 articles (22%) related to medical education. Other topics covered in a general, or non-rural, context included mental health, cancer, diabetes, obesity and bariatric surgery, health informatics, complementary and alternative medicines, point of care testing, and asthma.
RCTS medical education publications increased from 3 in 2004 to 14 in 2013 (Fig2), giving a total of 81 articles (28.9% of all RCTS publications). In 2004, 30.0% of all RCTS publications related to medical education in either a rural or a general context. Although there were some fluctuations in the intervening years, in 2013 medical education publications still comprised 28.6% of all RCTS publications. Rural health-related medical education publications outnumbered general medical education publications by at least 2 to 1 in all years, with the exception of 2011 and 2012 where their numbers were relatively equal.
Figure 2: Medical education publications in PubMed by Rural Clinical Training and Support researchers from 2004 to 2013 (RCTS, Rural Clinical Training and Support).
By using MeSH and relevant keywords, other important categories relevant to the RCTS funding parameters were analysed, namely Indigenous health, rural health services and rural population (Fig3). Indigenous health publications increased from 0 in 2004 to 4 in 2014. Four 'rural health services' articles were published in 2004, increasing to 11 in 2014, while 'rural populations'-related articles increased from 0 to 8 over the decade.
Figure 3: Rural Clinical Training and Support publications from 2004 to 2013 in rural health services, rural population and Indigenous health.
Figure 4: Rural Clinical Training and Support publications addressing National Health Priority Areas.
RCTS publications addressing NHPA (Fig4) show that mental health (n=15), diabetes (n=11) and cardiovascular disease (n=11) research articles were the top three areas covered. In contrast, no RCTS research publications related to dementia were found in the decade examined.
RCTS publications addressing the NRHAPA (Fig5) show that aged care (n=38) and mental health (n=15) articles were the most frequently published areas. In contrast, only one oral health research article was found and this was published in 2010.
Figure 5: Rural Clinical Training and Support publications addressing National Rural Health Alliance Priority Areas.
The top five journals publishing RCTS research articles in the decade 2004-2013 were Rural and Remote Health (16%; n=45), Australian Journal of Rural Health (13%; n=35), Australian Family Physician (9%; n = 25), the Medical Journal of Australia (7%; n=19), and Australian Health Review (6%; n=16) (Table 2). Only two of these have a current journal impact factor greater than 3. Of the 103 journals in total that have published RCTS research, only 19 have a current journal impact factor of 3 or more.
Table 2: Frequency of publication of Rural Clinical Training and Support research by journal in the decade 2004-2013
Discussion
To the best of the authors' knowledge, this is the first detailed analysis of a decade of publications emanating from the RCTS program, and highlights that RCTS researchers are fulfilling their current operational parameter of progressing the rural health agenda through targeted research. This was seen by an overall increase in RCTS rural health publications plus an increase in the proportion of RCTS rural health publications making up all ARH publications in the decade 2004 to 2013. It is not surprising that total publication rates were low during the first few years from 2004 to 2006, as the focus of RCSs would have been primarily on establishing a quality medical curriculum for students2. It is encouraging, however, to see the upward trend in publications that has occurred throughout the decade, as the schools and academics became established and their focus was directed towards research as the funding parameter of rural research came into place.
ARH topics were the major focus of RCTS publications throughout the decade of our analysis. Around one-third of all RCTS publications related to medical education, and those with a rural context were more frequent than general medical education publications. The research focus on medical education aligns well with the initial objectives of the rural clinical schools of delivering one clinical year of their medical curriculum in rural/regional settings. A large proportion of RCTS publications (37%) are aligning with the current NHPA11 and NRHAPA12. In particular, RCTS research is addressing areas such as mental health, diabetes, cardiovascular disease and aged care. However, some other rural health priority areas such as obesity and oral health might need a greater focus13,14.
RCTS publications were also detected that did not fit this review's criteria for rural health research3 and were categorised as 'other' research. In spite of this, many of these publications also addressed NHPA and NRHAPA. Although not in a strict rural context, these research publications are of relevance to the rural health agenda, since this general research will most likely be used to inform more specific rural research of this nature. Further reasons for publication of 'other' research from RCTS may be: (a) collaborations with metro centres15; (b) researchers with urban backgrounds who continue working on their research interest that started while in urban settings; and (c) RCTS staff continuing to work in their primary disciplines such as public health, general practice and informatics.
The authors note that among the top 10 journals that published RCTS papers, only two have a current impact factor of more than 3. The impact factor provides one means of assessing the quality of a journal and is sometimes used to infer the quality of the articles within. As such, it could be seen as one way of assessing the research coming out of the RCTS program. However, the journal impact factor alone is not the best indicator of the quality a journal article16,17 although it is one of the indicators used to inform the Excellence in Research for Australia (ERA)18. A more extensive examination of citation rates per article, study design types and altmetrics19, such as number of downloads and sharing through social media, would provide a more holistic measure of research quality16.
The data shown here for the previous decade of RCTS research provide a useful benchmark to which future data can be compared. Continued assessment of RCTS research outputs against these parameters using the reproducible/transparent methods employed here would provide a useful point of reference for monitoring the performance of incumbent rural research academics, as well as providing a guide for new academics in targeting their research to the rural health agenda. The advantage of this approach is that it is focused on rural health research priorities, as opposed to research outputs in a more general context, and does not attempt to compare rural academics with those academics in metropolitan areas. In a recent study of the current RCTS workforce9, only 26% of participating academics indicated that they worked full-time, as compared to the 71% reported by the Federal Government for academics Australia-wide20. Furthermore, just over 58% of RCTS academics reported working at 0.5 full-time equivalent or less9. Given this disparity in working hours, it would be unhelpful to assess rural researchers against their urban counterparts. An approach that assesses their research against health priority areas seems more appropriate for this context.
The current analysis was limited to MEDLINE-indexed journals in PubMed. This approach could have missed some RCTS publications not indexed in MEDLINE. However, Australia's health and medical research has been established to have high international visibility21, and so the numbers missed should be relatively small. Sensitivity testing was carried out using publication listings for several RCSs and RMSs that are reported on the FRAME website22. All listed publications emanating from these institutions were retrieved using the authors' queries. However, it is uncertain how accurate this listing is because all RCS/RMS have not listed their publications on this website and therefore it cannot be ruled out that some RCTS publications may have been missed by these methods.
The classification of rural health and related publications was limited by the MEDLINE MeSH tree classification and indexing by librarians. In addition, publications are tagged by multiple MeSH terms, which can be multidisciplinary and not include 'rural health'. In these instances, recognised primary disciplines such as public health will be used and the context of rural or regional health may not be in its MeSH term. Visibility of RCTS research publications is critical to achieving their research objective of addressing the distinctive needs of rural populations to better inform rural health policy3,4. Ensuring author affiliations are accurate and include the institution title, along with the secondary department or unit, such as rural clinical school or regional medical school, will maximise visibility, particularly when evaluating RCTS outcomes using bibliometric methods such as those used here. Visibility of medical education publications could be further enhanced by including this term in the keywords and title/abstract and not relying on a MeSH keyword assignment. For example, it was found that searching for 'medical education'[MeSH] only detected around 30% of RCTS medical education publications. Therefore the method of detecting these articles had to be altered by manually searching all retrieved RCTS publications.
This first study on the research outputs of RCTS researchers has shown that RCTS researchers are making a valuable contribution not only to medical education research but to rural health research more broadly. These data represent a benchmark of current research strengths, and highlights research areas that need greater focus in a rural context. The research parameters analysed here also provide a means of monitoring and assessing the outputs of future research academics that take up positions at RCSs. As many newer RCTS continue to build their research capacity in the coming years, tracking the contribution of RCTS research to rural health research in this manner can provide a useful outcome measure.
Acknowledgements
The authors especially thank Professor Jonathan Newbury for providing valuable feedback on the manuscript.
References
1. Australian Government Department of Health. Rural Clinical Training and Support (RCTS) 2011-2014 - Operational Framework: Introduction (Online) 2013. Available: http://www.health.gov.au/internet/publications/publishing.nsf/Content/work-st-rcts-2011-14-toc~work-st-rcts-2011-14-int (Accessed 23 September 2014).
2. Australian Government Department of Health. Rural Clinical Schools Program (Online) 2008. Available: http://www.health.gov.au/internet/main/publishing.nsf/Content/A3760E61F341B7F5CA257BF0001D73AB/$File/udrh5.pdf (Accessed 3 June 2015).
3. Mendis K, Edwards T, Stevens W, McCrossin T. Bibliometric analysis on Australian rural health publications from 2006 to 2012. Australian Journal of Rural Health 2014; 22(4): 189-196.
4. Perkins D. Do we need rural health researchers living in rural Australia? Australian Journal of Rural Health 2013; 21(1): 1-2.
5. Australian Government Department of Health. Rural Clinical Training and Support (RCTS) program parameters. (Online) 2013. Available: http://www.health.gov.au/internet/publications/publishing.nsf/Content/work-st-rcts-2011-14-toc~work-st-rcts-2011-14-atta (Accessed 23 September 2014).
6. Clark TR, Freedman SB, Croft AJ, Dalton HE, Luscombe GM, Brown AM, et al. Medical graduates becoming rural doctors: rural background versus extended rural placement. Medical Journal of Australia 2013; 199(11): 779-782.
7. Walker JH, Dewitt DE, Pallant JF, Cunningham CE. Rural origin plus a rural clinical school placement is a significant predictor of medical students' intentions to practice rurally: a multi-university study. Rural and Remote Health 12: 1908. (Online) 2012. Available: http://www.rrh.org.au (Accessed 15 June 2015).
8. Sen Gupta T, Woolley T, Murray R, Hays R, McCloskey T. Positive impacts on rural and regional workforce from the first seven cohorts of James Cook University medical graduates. Rural and Remote Health 14: 2657 (Online) 2014. Available: http://www.rrh.org.au (Accessed 15 June 2015).
9. Mendis K, Greenhill J, Walker J, Bailey J, Croft A, Doyle Z, et al. 2014 Rural Clinical School Training & Support Program SNAPSHOT Survey. Rural and Remote Health (Online) 2015. Available: http://www.rrh.org.au (Accessed 15 June 2015).
10. US National Library of Medicine. MEDLINE®/PubMed® Data Element (Field) Descriptions (Online) 2014. Available: http://www.nlm.nih.gov/bsd/mms/medlineelements.html (Accessed 16 October 2014).
11. Australian Government. Australian Institute of Health and Welfare. National Health Priority Areas (Online) 2013. Available: http://www.aihw.gov.au/national-health-priority-areas (Accessed 16 October 2014).
12. National Rural Health Alliance. NRHA Priority Recommendations from the 12th National Rural Health Conference, 2013 (Online) 2013. Available: http://ruralhealth.org.au/sites/default/files/documents/nrha-policy-document/positions/nrha-priority-recommendations-17-june-2013.pdf (Accessed 16 October 2014).
13. National Rural Health Alliance. Obesity in rural Australia (Online) 2013. Available: http://ruralhealth.org.au/sites/default/files/publications/nrha-obesity-fact-sheet.pdf (Accessed 16 October 2014).
14. National Rural Health Alliance. Oral and dental health (Online) 2013. Available: http://ruralhealth.org.au/sites/default/files/publications/nrha-oral-health-fact-sheet.pdf (Accessed 16 October 2014).
15. Steinbeck K, Hazell P, Cumming RG, Skinner SR, Ivers R, Booy R, et al. The study design and methodology for the ARCHER study-adolescent rural cohort study of hormones, health, education, environments and relationships. BMC Pediatrics 2012; 12(1): 143.
16. Mendis K, Bailey J, McLean R. Tracking Australian health and medical research expenditure with a PubMed bibliometric method. Australian and New Zealand Journal of Public Health 2015; 39(3): 227-231.
17. Lozano GA, Larivière V, Gingras Y. The weakening relationship between the impact factor and papers' citations in the digital age. Journal of the American Society for Information Science and Technology 2012; 63: 2140-2145.
18. Australian Research Council. Excellence in research for Australia (Online) 2015. Available: http://www.arc.gov.au/excellence-research-australia (accessed 9 June 2015).
19. Priem J, Taraborelli D, Groth P, Neylon C. Altmetrics: a manifesto. (online) 2011. Available: http://altmetrics.org/manifesto/ (accessed 15 June 2015)
20. Australian Government Department of Education and Training. Selected higher education statistics - 2014 staff data: staff full time equivalent (Online) 2015. Available: http://www.education.gov.au/selected-higher-education-statistics-2014-staff-data (Accessed 3 June 2015).
21. Butler L. Explaining Australia's increased share of ISI publications - the effects of a funding formula based on publication counts. Research Policy 2003; 32(1): 143-155.
22. Federation of Rural Australian Medical Educators. FRAME Home (Online) 2014. Available: http://www.ausframe.org (Accessed 23 September 2014).