A return-on-investment analysis of impacts on James Cook University medical students and rural workforce resulting from participation in extended rural placements

Introduction

Positive rural and remote community placements have been shown to have educational benefits for medical students that are at least equal to if not better than traditional urban placements^1-5. Immediate benefits include improving clinical competencies, particularly those specific to the priority health issues of these rural and remote areas⁶, while longer-term benefits include increased intentions for future rural practice and for working with underserved populations^7-10. There is also good evidence that if medical students are not provided with rural training opportunities, then the positive effect of having a rural background on subsequent rural practice diminishes over time^6,11.

In addition, there are further benefits of having extended rural placements^12-14. Longer rural placement duration has been found to be associated with later rural practice^15-17 as well as with a later career in primary care¹⁸. Research has also shown extended rural or urban clinical placements have advantages over much shorter hospital-based clinical blocks with respect to improved academic results^19-21, enhanced patient-centredness^19,20, greater exposure to common conditions²¹, more meaningful learning relationships with patients and academic mentors²², and quality of student feedback²³.

In 2012, the James Cook University (JCU) medical school implemented two types of extended rural placement programs for final-year students – the Integrated Rural Placement (IRP) and Longitudinal Integrated Clerkship (LIC) programs. These extended placements base students in small rural and remote towns across northern Australia for periods of 20 and 35 weeks, respectively. However, community-based training programs have extra expenses around student living and travel for both the students and the medical school, and local preceptors and administrators may need to be paid to coordinate and administer the rural programs. Quantifying the return on investment (ROI) from extended community-based medical student training would be important evidence to shift the conversation from one of cost to one of value. If the approximate overall value of extended placements to participating students and later rural medical workforce can be calculated, it may be easier to obtain community and government support for extended rural placements into the future. This study aims to quantify the financial ROI from key participant and rural workforce benefits arising from year 6 JCU medical students participating in extended rural placements from 2012 to 2018.

Methods

Study design

This study was based on accepted social ROI (SROI) methodology that was simplified to involve just a single survey of 2012–2018 JCU MBBS graduates who had experienced extended placements. The survey involved a question to quantify the costs incurred by respondents while undertaking the program as year 6 students, and a combination of closed and open-ended questions to approximate the ‘standard’ quantitative and qualitative components of an ROI evaluation for quantifying the key benefits to stakeholders. The costs incurred by the JCU medical school in delivering the programs from 2012 to 2018 were obtained from school records.

Description of the medical school’s extended placements 

The JCU medical school rural placement program underpins the JCU medical curriculum. All students undertake a total of 20 weeks of mandatory placements in rural and remote communities during the second (4 weeks), fourth (6 weeks) and sixth years (10-week ‘rural internship’) of study. Rural placements aim to expose students to rural and remote health and medical care, as well as provide experiences of living in these communities. In addition, year 6 students are able to choose a 20-week or 35-week extended placement, which includes the 10-week rural internship.

What is social return on investment methodology?

Social return on investment (SROI) methodology is distinguishable from other forms of economic valuation by the engagement of stakeholders in identifying and then valuing personal, social and community outcomes in addition to economic outcomes, the creation of a ‘theory of change’ for the program to capture the linear association between inputs (costs) and program activities leading to stakeholder outcomes/benefits (short-, intermediate- and longer-term).

A unique aspect of ROI evaluations is the assigning of a financial proxy value to each key stakeholder benefit to provide an estimation of the social and economic value generated by the program²⁴. Financial proxies allow the ROI to be expressed as a ratio of the total monetised value of benefits (social value) divided by the total program investment. To make the estimated ratio more accurate, the total calculated social value is then ‘discounted’; that is, reduced, to account for other contributing factors. In this instance, the total value of extended placements will be reduced by taking into account: (1) how much of the key benefits would have occurred anyway if the respondents had not undertaken an extended placement (deadweight), (2) given respondents experienced an extended placement, how much of the key benefits was due to other programs or experiences (attribution), and (3) how long the key benefits last after the placement is completed (duration).

ROI evaluation of the extended rural programs

The ROI evaluation of the JCU medical school’s extended placements follows the six stages of an SROI described by Laing and Moules²⁵, which in turn are based on the stages established in the SROI Network’s guide to SROI²⁶. (Note: This article does not follow the traditional approach of data collection, data analysis and description of results; these are instead described under each of the six SROI stages.)

Stage 1: Establishing scope and identifying key stakeholders:  The authors, all of whom are rural workforce academics and/or rural practitioners, set the scope for the ROI analysis, and identified the stakeholder groups benefiting from the extended rural placement program. Stakeholder beneficiaries are defined as people, organisations or specific populations that experience change (positive or negative) resulting from program activities²⁶. The scope was set at two key stakeholder groups potentially experiencing benefits from extended placements: JCU MBBS graduates who had undertaken an extended placement in their final year of medical school, and rural Australian medical workforce. In SROI evaluations, ‘true beneficiaries’ are suggested to be better placed to determine the outcomes accrued as a result of an intervention compared with ‘funder’ stakeholders²⁷. Therefore, the stakeholder group ‘rural Australian medical workforce’ was chosen for this evaluation rather than the JCU medical school, which funded and implemented extended placements, as ‘rural Australian medical workforce’ is the intended key long-term beneficiary.

The author group also considered all potential benefits and costs to these two key stakeholder groups and summarised these into a draft survey. This draft survey was piloted with two JCU year 6 MBBS students who were currently undertaking an IRP to ensure that the full range of potential benefits for JCU graduates and rural workforce were included in the survey, and that survey questions were likely to be well understood by respondents. After validity testing, an electronic SurveyMonkey survey was sent to the 46 of the total 72 JCU MBBS graduates who had undertaken an extended placement between 2012 and 2018 (as identified through school records) and who had consented to be contacted for further studies in the final-year exit survey.

Stage 2: Mapping costs and benefits:  Stakeholder engagement via the graduate survey was used to develop a theory of change, as well as identifying and quantifying the most important outcome (key benefit) for both the graduate and rural Australian medical workforce stakeholder groups in this impact map. Mapping a theory of change is a key component of a ROI analysis and describes the relationship between inputs (money and other resources that go into running a program), activities of the program, and outcomes/benefits (short-, intermediate- and longer-term) for the stakeholders as a result of the program²⁶. Thus, the theory of change is the ‘story’ of how extended placements make a change or difference in the participants and other stakeholders.

To map the costs to students, the survey asked JCU graduates to self-report the costs accrued to them on their extended placement. To understand the key placement activities and outcomes for students, the survey explored their intentions for a rural career and vocational specialty training pathway, benefits to their clinical competencies and confidences from their placement, how much the extended placements contributed to their clinical skills and career intentions, and any potential negative outcomes from undertaking placements. All questions are described in Appendix I, while the reported costs, activities and resulting benefits are summarised in Figure 1.

The costs to the medical school for the extended rural placements were based on the amount of money paid to students as travel subsidies to move from their home clinical school to the placement site, and for subsidies to connect to the internet via a domestic arrangement if this technology was not available at the placement accommodation. School records identified each of the 72 students’ home clinical school site in year 6 and their rural placement site, allowing calculation of the associated travel subsidy and internet subsidy, if appropriate.

The most common benefit from extended placements reported by 96% of the JCU graduates (and identified as the ‘most important’ by 56% of respondents) was ‘greater depth and breadth of clinical skills’ from the extended placements compared with the mandatory 10-week rural term and other non-rural placement experiences. An increase in clinical confidence was the second most significant outcome for graduates (reported as ‘most important’ by 38% of respondents), with 88% of respondents reporting increased clinical confidence from their placement. Most students also reported an increase in communication skills (94%) and cross-cultural communication skills with Aboriginal and Torres Strait Islander patients (78%).

From the outcome map, the most common benefit for the rural Australian workforce beneficiary group was chosen by the research team as ‘more JCU graduates choosing a full-time rural career’. Overall, 75% (18/24; one missing response for this question) of JCU medical graduate respondents reported intending to have a career in rural and remote practice (defined as 50 or above on a 0–100 scale, with 100 being ‘definitely will be a rural doctor’), with 92% (23/25) of responders also reporting they were currently pursuing a career as a generalist (rural generalist 68%, general practitioner (GP) 16%) or general specialist (8%), appropriate for rural practice. While most of the respondents were still in their early career (postgraduate years 1–3), 10 of the 25 (40%) were practising in a rural or remote town (Modified Monash Model (MMM) 4+) in 2019. The MMM is a widely accepted system in Australia for classifying metropolitan, regional, rural and remote areas into seven rurality categories according to geographical remoteness and town size.

Stage 3: Evidencing key outcomes and giving them a value:  This stage involves uncovering collaborating objective quantitative data to confirm whether the key benefits have occurred, and then to value these benefits through a ‘monetisation’ process involving an appropriate financial proxy. In SROI methodology, financial proxies are used to estimate the net social value, in present-day dollar terms, of key benefits that do not have a market price²⁶.

For the medical student beneficiary group, analysis of the medical school’s final-year exit surveys from 2012 to 2018 (n=738, including 46 who had undertaken an extended placement) was used to confirm that ‘greater depth and breadth of clinical skills’ had occurred in those undertaking an extended placement (Table 1). The exit surveys ask year 6 students how well the MBBS course prepared them to be a doctor with regard to ‘your clinical skills’, ‘ability to apply your knowledge’ and ‘your readiness to work in the first week of internship’. Between 2012 and 2018, year 6 students who undertook an extended placement had significantly higher mean scores than the other students in self-reported ‘clinical skills’ (mean of 4.49 vs 4.29 out of 5; p=0.042), ‘readiness to work in first week of internship’ (mean of 4.47 vs 4.09 out of 5; p=0.001) and ‘ability to apply knowledge’ (mean of 4.33 vs 4.07 out of 5; p=0.006). Statistical comparisons were undertaken using SPSS v24 for Windows (IBM; http://www.spss.com) via two-sided Student’s t-tests, with statistical tests considered significant with p<0.05.

As JCU graduate responders reported increases in both depth and breadth of clinical skills as well as clinical confidence from undertaking extended placements, the financial proxy assigned to value this outcome was based on the personal cost to students to be trained as doctors. Through the Higher Education Contribution Scheme, Australian medical students are required to pay training costs of $9792 per year at 2014 levels. Thus, JCU medical students are required to pay 6 × $9792 = $58,752 for their 6 years of training; that is, the value to JCU students of the standard 6 years training to be ‘work ready’ for internship is $58,752. However, using a ‘revealed preference’ approach²⁷, JCU students who undertake an IRP/LIC experience report (via the exit surveys), on average, an additional 9.3% of ‘work readiness for internship’ (mean of 4.47 vs 4.09). Note that, for this key benefit, the 9.3% is also used as the percentage of deadweight in the later discounting calculation. Therefore, if students have to pay $58,752 for 6 years of training to be ‘work ready’ interns, then, using standard ROI methodology, of the 72 JCU graduates who undertook an extended placement from 2012 to 2018, 88% (based on 22 of the 25 survey responders who reported increased clinical competencies and confidence) would have each received an additional $5464 of value from their extended placement.

For the beneficiary group ‘rural medical workforce’, analysis of the JCU medical school’s graduate tracking database demonstrates that 16 of the total 72 (22.2%) JCU graduates who undertook an extended placement were working in rural or remote towns (MMM 4+) in 2019, compared with 131/1524 (8.6%) of JCU graduates who did not do an extended placement (Table 1). Thus, the extended year 6 rural placement programs are associated with an extra 10 JCU graduates ((22.2% – 8.5%) × 72 = 9.8) practising in a rural or remote town during 2019 than would have been expected without the extended rural programs.

Further evidence of the impact of extended rural placements on promoting willingness towards rural practice was also observed in the study findings. The current study found that the 25 respondents reported their extended placement contributed an average of 67% towards ‘their desire to be a rural or remote doctor’ (attribution); with seven of the 25 respondents also stating they were currently working more rurally in 2019 than they would have had they not undertaken an extended rural placement. These seven included four respondents who reported currently working (2019) in a regional city when they otherwise (if not for their extended rural placement) would have been working in a major city, plus a further one respondent now working in a rural town rather than in a regional city, and two respondents now working in a remote rather than a rural town.

These seven respondents who self-reported that they are practising more rurally than they would have without their extended placement experience were also pursuing rural generalist (5) or generalist specialist (one emergency medicine and one anaesthesia) careers – appropriate choices for a longer-term rural career. Overall, 21/25 (84%) of survey respondents reported currently pursuing a career as a rural generalist or GP, while the school’s graduate tracking database shows 57% of graduates to be currently training or have achieved fellowship as rural generalists or GPs. This increase in vocational training choice for generalist medicine after undertaking an extended placement is supported by the finding that 16/25 (64%) of respondents reported that their placement had ‘influenced the type of doctor they want to be’, with all 16 describing a change towards rural practice and/or rural generalism.

For the stakeholder group ‘rural medical workforce’, the financial proxy assigned for the key benefit ‘JCU graduates willing to work full-time in a rural or remote town’ was based on the willingness of many respondents in the current study to fill a vacant position in a rural or remote town for less than their expected salary. Overall, 50% (12/24; one missing response for this question) of respondents reported they would be willing to accept lower remuneration to secure work in a rural or remote town – a ‘stated preference’ approach²⁷. Of the 12 reporting a willingness to accept a rural doctor position, two reported being willing to accept 0–10% less, five would accept 10–20% less, two would accept 20–30% less, two would accept 30–40% less, and one would accept 40–50% less. Combining this question with another query asking what specific salary they would expect for working in a rural town, the 24 respondents, on average, were willing to take $23,479 less remuneration to work in a rural town (using the midpoint for each percentage category; for example, for the 10–20% less category, a midpoint of 15% was used). Therefore, using standard ROI methodology, of the 72 JCU graduates who undertook an extended placement from 2012 to 2018, 75% (based on 18 of the 24 survey responders who reported a desire to work as a doctor full-time in a rural or remote town) would create $23,479 of savings to rural Australian medical workforce per year.

This financial proxy approximates the potential savings to rural workforce from more quickly filling positions in rural and remote towns because of more doctors willing to accept an available position in these areas as a result of undertaking an extended rural placement. This approximation does not necessarily mean that these graduates would accept less money; it may also cover situations where graduates take a position in a rural or remote town that historically has found it difficult to recruit permanent doctor positions even at full pay. This is a reasonable approximation given current difficulties associated with filling Australian rural doctor vacancies with graduates having appropriate procedural skills²⁸.

Stage 4: Establishing impact:  Calculating a more accurate monetary value to the key benefits must entail ‘discounting’ of the net social value of the key outcomes. Discounting relates to the proportion of the total population who experienced the outcome/benefit, how much of the key outcomes would have happened without the extended placement experience (deadweight), whether any other organisations or programs contribute to key outcomes in addition to the extended placements (attribution), and how long outcomes will last after the program has been completed (duration). In this ROI evaluation, the level of discounting for ‘proportion’, ‘deadweight’, ‘attribution’ and ‘duration’ was established through respondent self-report in the survey and external databases using the formula:

total population ×  proportion ×  $ value of financial proxy ×  % deadweight ×  attribution × duration

The financial value to the 72 students (total population) who undertook an extended placement from 2012 to 2018 is based on the self-reported key benefit ‘work ready for internship’. The survey found 88% of the 25 respondents reported both increased clinical skills and clinical confidence that improved their work readiness for up to 12 months (ie their internship year), based on survey responses where 100% of respondents gave a 6–12-month period of this benefit (duration), and, as previously stated, the deadweight is 9.3%. As there are no other programs that are involved in teaching students clinical skills and confidences in rural towns during extended placements, and as the deadweight also took into account attribution towards being work ready for internship outside of the extended placement experience, the attribution for this benefit is 100%. With an approximate value of $5464 worth of extra clinical skills and confidences to each student who reported experiencing this benefit, the value calculation to the overall population of students undertaking an extended placement is:

72 (total population) ×  88% (proportion) ×  $5464 ×  9.3% (deadweight) ×  100% (attribution) × 1 (duration in years)

= $32,197 total value for all JCU MBBS students undertaking an extended placement between 2012 and 2018.

The total financial value to ‘rural medical workforce’ between 2013 and 2019 from the 72 students (total population) who undertook an extended placement from 2012 to 2018 is based on the self-reported key benefit ‘willingness to work in a rural or remote town’. The survey found 75% (18 of 24; one missing data) of respondents reported a desire to work full-time as a doctor in a rural or remote town (proportion), and that respondents reported their extended placement contributed an average of 67% towards ‘their desire to be a rural or remote doctor’ (attribution). In addition, the school’s tracking database showed JCU graduates who experienced an extended rural placement were more likely to be working rurally in 2019 (22.2%) than those who did not (8.6%). Deadweight is calculated by removing the proportion of key outcomes expected to occur if participants had not undertaken the program; therefore, the calculation of deadweight for this benefit is: 100% – (8.6 ÷ 22.2)% = 61%. Further, the 18 graduates who reported a desire to work full-time in a rural or remote town had worked an average of 1.3 years (duration) in a rural or remote town (MMM 4–7) between 2013 and 2019. Therefore, with an approximate value of $23,479 worth of ‘willingness to work in a rural or remote community’, the value calculation from the overall population of students undertaking an extended placement from 2012 to 2018 is:

72 (total population) ×  75% (proportion) ×  $23,479 ×  61% (deadweight) ×  67% (attribution) × 1.3 (duration in years)

= $673,630 total value for ‘rural medical workforce’ between 2013 and 2019.

Stage 5: Calculating the SROI and undertaking a sensitivity analysis:  Once the total net value had been calculated for 2013 to 2019 (allowing time for students to graduate and enter the workforce) and the various discounts applied, this value was compared with the total investment from 2012 to 2018 (ie the monetised costs of running the IRP/LIC programs) to determine the ROI ratio²⁶. In this SROI study, the net financial value of the IRP and LIC programs from 2013 to 2019 from the key benefits was $705,827 – calculated from the total value to students ($32,197) and the total value to rural medical workforce ($673,630). The total costs of the IRP/LIC program from 2012 to 2018 were calculated at $92,824, including costs to the students of $60,264, and costs to the JCU medical school for sending 72 students out on extended rural placement of $32,560. The SROI ratio, therefore, is $705,827 ÷ $92,824, equating to a ratio of 7.6:1. In other words, for every dollar invested in extended placements from 2012 to 2018, $7.60 of approximate financial value was created.

In calculating the total ROI, it was necessary to make an assumption about the reduction in salary 12 respondents reported being willing to accept for a rural doctor position. When calculating the average reduction of $23,479, only the midpoint was used for each percentage category. For example, for the 10–20% less category, a midpoint of 15% was used, although the true reduction could be anywhere between 10% and 20%. To assess how much influence this assumption may have had on the final ROI ratio of 7.6:1, a sensitivity analysis was carried out to allow the overall value to be expressed between lowest case scenario and highest case scenario for each of the 24 examples. Thus, the sensitivity analysis on the lowest percentage range for each of the 24 respondents’ reported reduction in salary gave a minimum average value of $16,667 and a maximum average value of $29,458. Thus, with an SROI ratio of 7.6:1, there is a degree of confidence that extended rural programs create between $5.50 and $9.50 of social and economic benefits for the key stakeholders for every dollar invested.

Ethics approval

Ethics approval for the study was obtained from the JCU Human Ethics Committee (#H6921).

Discussion

Traditional approaches to which decisions in health professional education are made and analysed often take only financial measures into account, or use simplistic statistical associations with a single outcome (eg future practice location). This ROI study of the value of extended rural placements, the first in health professional education, shows significant benefits to medical students in terms of clinical skills and confidence, and to rural Australian workforce in terms of increased numbers of medical graduates willing to work full-time in rural and remote areas as rural generalists, GPs or generalist specialists. These benefits have far greater value than the costs associated with running the program, and will continue to accrue benefits to rural Australian medical workforce past the 2019 evaluation period, assuming at least some of these graduates continue to work in underserved rural and remote towns – a likely outcome with 40% of the respondents in this study practising in a rural or remote town (MMM 4+) in 2019.

Furthermore, while this study shows JCU’s extended rural placements create considerable benefits to rural Australian medical workforce by producing more full-time regional, rural and remote rural generalists/GPs, other financial benefits would also accrue to rural workforce and rural community members. These other benefits would probably include a reduction in the costs of bringing in locums or international medical graduates to cover vacant positions, or the cost to rural practices and community members if a GP locum or an international medical graduate could not be employed to fill that position. Recruiting difficulties can lead to rural health services needing to employ temporary locum GPs on thousands of dollars a day – endangering the financial sustainability of practices already under fiscal strain²⁹. A study in rural USA found each full-time equivalent rural physician vacancy carried an estimated US$990,000 annualised loss to the medical practice due to costs of employing short-term locums³⁰.

However, much of the value in health professional education is psycho-social in nature, rather than just financial. For example, studies strongly suggest that the social value to rural Australian communities from having a permanently based, longer-term rural doctor would probably be worth considerably more than just financially based outcomes^31,32. Similarly, previous research demonstrates that a strong level of clinical skills and clinical confidence has considerable positive psychological effects for intern-year doctors. The transition from medical student to intern is very stressful, with anxiety and burnout common^33,34, and any negative personal experiences during this period may significantly impact on future confidence levels and professional identity³⁵. It is likely that the extra skills and confidence that most JCU graduates received from their extended rural placement experiences might somewhat alleviate their anxiety, stress and chances of negative experiences during their intern year, which would be very much valued during internship.

In summary, this ROI study shows extended undergraduate rural placement programs result in ‘real’ value for both student participants and rural health workforce, including strong evidence of increased rural medical practice in the short and longer term. Thus, findings strongly suggest rural health workforce policy and strategy should focus on encouraging longer-term undergraduate (and probably postgraduate) rural training posts.

Limitations

There is inherent subjectivity involved with using financial proxies to represent social value, and a different research team may produce different results. ROI methodology is an approximation, as any time value is in question, it is inherently subjective. However, ROI will always tend to underestimate the total impact of intervention programs²⁶. In particular, the current study has not factored in benefits to the community. Thus, while this evaluation counts the expenses students had while living in the community as a program cost, this is actually a financial benefit to the community. In addition, some students would also positively impact on the social fabric of the community, and, as final-year students, assist with clinical activities in hospitals and medical centres, which can also be a significant benefit to the community’s health services.

The main advantage of ROI studies is that they measure ‘true’ program impact by providing a more accurate approximation of causality through the discounting process. Discounting takes into account deadweight (how likely key outcomes are to still occur had participants not undertaken the program), attribution (other influences on participants in addition to the program’s influences) and duration of benefits. Without discounting, it would not be possible to tease out the ‘true’ influence of extended rural placements on participants’ motivations for a rural career and clinical competencies from other ‘non-programmatic’ influences that may be contributing to these outcomes. For example, the opt-in nature of the extended placement programs at JCU means students who are already motivated towards a rural career and/or are confident in their clinical skills will probably be more likely to choose an extended rural placement; this influence can be removed from the final calculation of program impact by adjusting for deadweight. The other advantages of ROI studies are that the full range of benefits arising from the program are usually uncovered, and that the financial proxy for the key benefit to each stakeholder group can approximate the overall value of the program in relation to costs – critical factors in determining whether programs should be further funded or potentially expanded.

However, it must be stated that the ROI methodology relies on approximations and assumptions from self-report to calculate the financial proxies for the key benefits. For example, the calculation for financial proxy for the student stakeholder group – $5464 each for those reporting an increase in clinical skills and confidences – is based both on a 9% increase in self-reported ‘work readiness’ in the exit survey (an approximation of increased benefit derived from a Likert scale), and the assumption that this equates to an extra $5464 of value based on students’ overall HECS fees for undergraduate medical training. Similarly, the calculation of the financial proxy for benefits to ‘rural medical workforce’ is based on graduates’ willingness to accept a lower remuneration as a direct result of their extended placement experiences, leading them to quickly fill an available position in a less desirable rural town or take lower remuneration than might be expected for such positions.

Overall, however, the key to an effective ROI is quality data. While the sample size for the survey was small at 25 respondents, the response rate was reasonable at 54%. In addition, this ROI used several approaches to improve the trustworthiness of the findings: involving multiple sources of objective data from other sources to confirm survey findings, and open-ended questions to allow mapping of a simple theory of change showing the association between key placements activities and key outcomes. However, due to budgetary constraints, this ROI evaluation, while adapted from accepted SROI methodology, only collected quantitative data from a survey without any qualitative investigations, nor were interpretations of this ROI evaluation verified with survey participants – both strongly recommended by the SROI Network²⁶. This survey-only approach was developed as a quick and cost-efficient way of undertaking an ROI evaluation in response to a standard SROI evaluation not being feasible. Indeed, reviews of ROI in health and education^24,36 have identified a significant lack of standard ROI program evaluations around health workforces due to the financial and time constraints involved. Thus, this cost-efficient, survey-based ROI approach has the advantage of usability; however, some richness and diversity of information will have been reduced by using content analysis of open-ended survey questions to approximate qualitative investigations.

In addition, the possibility exists that selection bias may be influencing the study inferences and findings, as only 25 of the total 72 graduates who had undertaken an extended rural placement in their undergraduate year 6 responded to the graduate survey, from which all inferences and findings arose. However, the authors accessed JCU medical school’s graduate tracking database to statistically compare the 25 respondents with the remaining 47 non-respondents across every variable collated in the tracking database. The analysis found the two groups were statistically similar (p>0.100) across all demographic variables (age, gender, rurality of hometown at application to medical school, ethnicity, international origin, lateral entry, accepted under a bonded medical place), undergraduate variables (scholarships awarded, Honours undertaken) and postgraduate training variables (rurality of practice location from postgraduate years 1–5, choice of specialty training college). Therefore, potential for selection bias in this study appears to be minimal.

While the findings of this survey-based ROI are encouraging, ‘traditional’ SROI evaluations involving associated qualitative investigations are further required to confirm the financial ROI of extended rural placement programs identified in this pilot study, as well as to estimate the true social value of program benefits to all student, community and health workforce stakeholders.

It should also be noted, however, that at present it is unlikely that many health professional education schools would easily be able to reproduce studies of this nature. Few researchers are experienced in the relatively new SROI methodology, and not many schools maintain a graduate tracking database containing the objective data required for SROI analyses. Alternatively, this argues for health professional education schools with government-funded rural or other special programs to develop appropriate graduate tracking databases that collate key outcome data, and to collaborate with impact evaluators experienced in SROI analyses.

Conclusion

This ROI study confirms that undertaking an extended rural placement has significant impacts on medical students’ clinical skills, clinical confidence and communication skills for their internship year, as well as having eventual longer-term impacts on rural Australian medical workforce by motivating more graduate doctors to choose a rural career.

The ROI analysis further confirms the financial value to medical students and the rural Australian health system as a direct result of an extended rural placement program – $7.60 for every dollar spent – far outweighing the costs of running the program. However, the overall value of extended placements is likely to be far greater; for example, the potential social and financial value to rural and remote community members from having more locally based doctors skilled in rural medicine. This positive ROI from extended rural placements is important evidence for shifting the conversation around supporting or expanding these programs from one of cost to one of value.

Acknowledgements

The authors acknowledge the JCU medical graduates who participated in this study.