Medical students' assessments of skill development in rural primary care clinics

full article:

Introduction

Background and objectives

Although over 50 million people - or approximately 20% of the US population - live in rural areas, only 9% of physicians in the US practice in a rural setting¹. Targeted recruitment of students from rural areas who express a desire to practice in rural communities is one strategy that medical schools have used to graduate physicians committed to careers as rural primary care physicians^2,3. Another approach to address the need to graduate more generalist physicians for rural settings has been to provide rural training experiences for medical students and family practice residents^4,5. Targeted recruitments can be combined with rural training experiences to maintain and even boost interest for a career in a rural setting^6,7.

The Rural Medical Education (RMED) Program at the University of Illinois, College of Medicine, Rockford, is an example of a combined approach. After initial review of an application by RMED staff to ensure that an applicant meets the college's admission criteria, a special screening committee applies a second level of criteria to applicants for admission, such as examining an individual's rural roots, commitment to primary care, and community involvement⁷. Students meeting criteria are offered an interview and if accepted to the RMED Program, agree to participate in the RMED curriculum. This includes a 16 week fourth-year clerkship experience that moves beyond graduation requirements to reinforce the students' commitment to a career in rural medicine⁸. The RMED Program has been very successful in returning primary care physicians to practice in rural settings. There have been 187 RMED matriculants (with 105 either still in medical school or in residency). Of those in medical practice (n = 82), 82% are in primary care and over 90% are practicing in rural communities.

Since rural practice differs from urban settings, it seems likely that a rural clerkship might vary from students' experiences in a more traditional setting⁹. A rural patient population tends to be less diverse in terms of racial and ethnic background and, on average, older with more chronic health conditions and with less education¹⁰. The scope of rural practice also differs because rural family physicians are more likely to provide obstetric care and to perform more procedures than their urban counterparts¹¹.

Despite the probability that the student experience differs in the rural setting, little is known about students' perceptions of rural training. For example, although rural training is associated with increased clinical skills¹¹, it remains uncertain what specific skills are enhanced by this training. Research, demonstrating that residency trainees who spend time in a rural setting are more likely to see a wider range of patients and experience more continuity of care, also suggests that students might experience similar benefits¹². Rural medicine training also appears to be more multidisciplinary, community-based, and more likely to involve participants in a wide range of community organizations¹³. However, a less academic setting might compromise a student's clinical maturation and have a negative impact on a student, for such reasons as less structure, faculty not having as much formal teaching training or experience, a less 'rich' case mix, and potentially fewer specialists available for teaching^14,15.

In order to examine the impact of a rural training experience on medical students, this study compared the pre- and post-clerkship self-assessments of RMED students participating in a 16 week rural training experience. Specifically, the present study assesses students' self-perceptions of their ability to provide acute and chronic care, preventive care, to do procedures, to communicate with patients, and their understanding of the community and the healthcare system. The study was approved by the university Institutional Review Board.

Methods

Rural Medical Education Program background

The RMED Program students are part of the medical school class at the University of Illinois College of Medicine at Rockford. Class size at the Rockford campus generally varies from 45 to 55, of which approximately one-third are RMED students. The RMED applicants apply to both the College of Medicine and the RMED Program. Applicants are interviewed by an RMED Recruitment and Retention Committee, with recommended students going before the college's Admissions Committee for final approval. Once accepted, students take the regular University of Illinois College of Medicine curriculum, and in addition participate in a specially designed RMED course of study. In the first 3 years of medical school, this curriculum consists of required, monthly evening sessions (3 hours in length) addressing issues in community and population health, medical practice in rural settings, and development of a community-oriented primary care (COPC) project. These sessions are coordinated by RMED faculty. Faculty and outside speakers (eg, rural physicians, health educators, social service and hospital administrators, and other health professionals) facilitate the sessions.

During the major participatory experience, all RMED students spend 4 months living and working in a rural Illinois community in which the students learn clinical, office, and hospital-based skills working with a primary care physician - most often a family physician⁸. During the preceptorship, students spend approximately 70% of their time in clinical activity, sharing training with their assigned physician preceptor as well as with other health professionals in the community. The remaining 30% of the time is devoted to implementing a COPC project that was designed previously by the student. The projects generally fall into four categories: health education/health promotion; access to health care; disease prevalence in the rural sector; and the organization of health care¹⁶. Projects are developed in a service-learning mode and represent opportunities for the student to give to and receive from the rural community in which they are training.

This RMED immersion experience can take place in one of 23 rural Illinois communities. Site locations are presented (Fig 1). It is important to realize that Illinois is a complex state with the fifth largest population of 12 763 371 spread out over 55 584 square miles. While Illinois is often associated with the city of Chicago, of Illinois' 102 counties, 84 (82%) are designated as rural by the Illinois Department of Public Health's Center for Rural Health. The preceptorship sites are located in communities ranging in size from 1100 to 35 000. Just over 50% of the preceptorship communities are less than 10 000 in population size. The communities are deliberately located across the state, allowing students a range in selection of sites. In each community, the local hospital has signed a letter of agreement with the university to allow students privileges with the hospital and agreeing to provide room and board in the community for the medical students. Physicians, who have unpaid faculty appointments, agree to precept the students, serving as their teachers and mentors for clinical activities.

Sample

Since the beginning of the RMED Program in 1993, 131 students have participated in the fourth year, 16 week rural preceptorship. At the beginning of each clerkship, students are asked to complete a pre-clerkship self-assessment of their confidence in their knowledge and ability followed by a post-clerkship evaluation of the same items related to patient and population-based healthcare delivery. The students' assessments are strongly encouraged (but not required), confidential, and do not affect course grades. These pre- and post-surveys comprise the dataset for the present project.



Figure 1: Site locations for rural medical education immersion experience.

Pre- and post-preceptorship surveys

The instrument students complete asks them to assess their level of confidence in their ability and knowledge in 11 dimensions (Fig 2, Table 1).



Figure 2: The 11 dimensions of RMED students' self-assessments.


Table 1: Dimensions and items of RMED students' self-assessments pre and post the 16-week preceptorship



The 11 dimensions cover 97 items. Specifically, the students are instructed: 'For each item, choose the one response that most accurately describes your level of confidence in your present knowledge/ability'. The same list of items is presented to the students pre- and post-preceptorship. The 11 dimensions and component items were developed through review of RMED goals and objectives, and through discussion within the faculty and with other rural programs regarding skill sets important for a rural practitioner. Students assess their level of confidence in their knowledge/ability for each item on a 6-point Likert scale, with values ranging from 0 = no confidence to 5 = high confidence.

For the most part, the individual items within the 11 dimensions are self-explanatory. However, some items are more general and subject to the interpretation of the medical students. For instance, 'social service agencies' (in Dimension 10) may be interpreted by some students as meaning Public Aid or a church-run shelter or a teen pregnancy program. This assessment can vary by community and according to the types of contacts the student has had in the community. The purpose is to emphasize social service agencies in general. Similarly, 'community-based physician practice pattern' and 'routine of physician's role in the community' (in Dimension 10) are general concepts to get the student to think of linkages between the medical care system and the community at large.

Analysis

Analysis was conducted in three stages. First, students' mean scores (pre and post) were compared on each survey item, using a paired samples t-test. Second, items were collapsed into the summary dimensions representative of the clinical and community skills. In both stages, student self-assessments were compared prior to and on completion of the preceptorship. Finally, bivariate analysis was used to look for possible patterns in student responses to each dimension and overall dimension in relation to gender and age of medical student and state designated primary medical care status of the local county (ie, not underserved, partially underserved, whole county underserved).

Results

To date, 96 (73%) of the 131 RMED students completed both pre- and post-clerkship surveys. Each year 12 to 18 students completed the clerkship. The mean age of the 96 students was 27.8 years (sd = 2.7), with 58.6% male. The average size of the community in which the students completed their preceptorship was 11 688 (sd = 8749), with a range in population size from 1124 to 35 076. Two-thirds of the preceptorship sites were located in Illinois counties with partial or whole county primary care shortage designations.

For all 97 items, student mean assessments post-preceptorship were higher than the pre-preceptorship rating 4 months earlier (Table 1), with all differences statistically significant. On average, the most highly assessed skills at the end of the 16 week preceptorship were: urinary tract infection (UTI - 4.69); otitis media (4.62); gastroesophageal reflux disease or reflux (4.57); health maintenance - adult 50 (4.53); health maintenance - adult 25 (4.52); strep screen/culture (4.52); hyperlipidemia (4.50); health maintenance - 75 (4.47); depression (4.45); bronchitis (4.43); cough (4.43); hypertension (4.41); diabetes mellitus (4.39); osteoarthritis (4.38); provider to patient education (4.34); and chest pain (4.33). At the other end of the spectrum, after the 16 week preceptorship were: central line replacement (1.32); paracentesis (1.40); thoracentesis (1.49); episiotomy repair (2.06); arterial blood draw (2.09); and hospital insurance/billing (2.45). At the start of the preceptorship, only one item was self-assessed at a level above 4: UTI (4.33). After the preceptorship, 35 of the 97 items (36%) were assessed above the level of 4.

Table 2 presents the average ratings for each of the dimensions at pre- and post- preceptorship, and the difference between the two ratings. For the 11 summary dimensions, none was self-assessed prior to the preceptorship at an average level above 4, although the average of two of the dimension assessments were below 2: primary care system (1.92) and hospital care system (1.97). At the end of the preceptorship, no average rating by dimensions fell below 2, while 5 of the 11 dimensions (45%) were assessed above the level of 4: common undifferentiated problems (4.05); common acute problems (4.05); delivery of health maintenance (4.13); provision of patient education (4.20); and common chronic problems (4.29). The correlation between pre- and post-preceptorship assessment rankings was high - Spearmans rho = .873.

The largest differences between pre- and post-assessments were found in primary care system (1.30), other healthcare system (1.26), hospital care system (1.15), and community (1.03). The lowest difference in self-assessed skills was in the dimension of procedures (0.55).

Table 2: Comparison of students' skill assessment pre- and post-preceptorship



There was no association between student self-assessments of confidence in abilities post-preceptorship and the variables of age and state designated medical care status of the county. In comparison of self-assessments and gender, only one component emerged as statistically significant: males self-assessed their procedural skills at the end of the 16 week preceptorship more highly than female students (mean = 2.99 compared with mean = 2.73; p = 0.050).

Discussion

This study demonstrated that a 16 week rural clerkship results in a broad-based increase in students' self-assessed knowledge and skills. Generally, the students' level of confidence improved from moderately high to high confidence in their abilities and knowledge. The results highlight the potential value of a rural training experience on knowledge and skill development of medical students in multiple dimensions of training. As Worley et al. have pointed out, there is very little published research that explores the actual learning activities undertaken by students in different environments¹⁷. The present work provides some details on a community-based program providing evidence in support of such programs as credible educational experiences to traditional teaching models, and adds to the body of literature about primary care clerkship experiences^18,19.

Within this domain, the lowest mean difference between pre- and post-assessment was in the dimension of procedures, which was also the lowest rated dimension at the conclusion of the preceptorship (mean = 2.86). Because rural family physicians tend to perform more procedures compared with urban-based physicians²⁰, it is somewhat surprising that this dimension demonstrated the smallest amount of change.

There are several explanations for this finding. One might be the nature of the 16 week RMED rotation. The clerkship emphasizes primary care rather than procedures, an emphasis which might differ from rural training at other institutions. It is the RMED Program philosophy that the students will be able to acquire the necessary procedural skills in other rotations or later during residency training, but that the opportunity to teach core primary care values to students may not be available in other rotations. The RMED preceptors are aware of this emphasis and it is possible that this might lessen their commitment to teaching procedures. In addition, a significant portion of time is devoted to completing the community-oriented primary care project, which might also reduce the time available for procedures. Finally, RMED is one of the shortest of the US longitudinal preceptorship experiences, compared with those of 6 or 9 months where preceptors and students may have more time to develop a relationship and share expertise involving procedures.

Even though the RMED Program places less emphasis on procedures, the relatively low rating and limited improvement warrants concern. In the future, the program plans to work with preceptors and students to develop a core set of procedures that students could reasonably expect to be exposed to during their 16 week rotation. Setting clear expectations for students to seek opportunities for these procedures, and faculty development for preceptors to provide training and sign off on a required student procedure log should bolster this dimension.

However, it is encouraging that given the emphasis on primary care that the knowledge and skills central to the development of primary care and family medicine are rated highly at the finish of the preceptorship. These include confidence in understanding and treating chronic, acute, and undifferentiated medical problems. Since rural physicians devote less time to health maintenance than their urban counterparts²¹, one potential concern is that the training in a rural setting might be deficient in this area. However, student confidence increased both in the delivery of health maintenance and patient education, suggesting that training in these dimensions is adequate.

Another important finding is that the students' levels of confidence increased most in the dimensions of the primary care system, ancillary health care systems, hospital care system, and community - areas not usually well-addressed in the medical school curriculum. However, none of the post-assessment scores in these dimensions were evaluated at level 4 or above. Although this suggests the opportunity for greater improvement, it is still reassuring that the self-assessed confidence in understanding and addressing these types of issues had increased at least one point. Most significantly, the students expressed a major change in understanding the physician's role in the community (from 2.70 to 4.26). In the future, data will be collected regarding students' specific comments on how their understanding of the physician's role changed.

Additionally, further study comparing the RMED Program outcomes to non-rural rotations, as well as to other rural medical education clerkships, will be useful in confirming the equivalency of training and for exploring whether rural settings might even be superior for some dimensions of training. This should aid in planning rural experiences that complement traditional student training experiences

A limitation of this study is that pre- and post-assessments of medical and healthcare skills are self-reports by the medical students. The students' subjective assessments may or may not be accurate in relation to other evaluations of their performance. However, the pre- and post-assessments represent a means of obtaining feedback on the value of the 16 week rural preceptorship. The assessments are similar to the use of self-report surveys designed to provide some information on the impact of educational interventions on knowledge, attitude, and skill development. Also, studies indicate that trainees often rate their competencies more harshly than others would rate them, making the students' consistent perception of improvement and high level of self-perceived post-clerkship skills even more remarkable^22,23.

It is also possible that, as part of the maturation and training during the fourth year, students could exhibit similar or even greater improvement in non-rural rotations in the 11 assessed dimensions. However, the breadth and extent of improvement evidenced in the present analysis would suggest that the rural preceptorship provides a valuable experience in the improvement and development of clinical skills related to primary care. Overwhelmingly, self-assessed skill development occurred regardless of demographic and community characteristics. This is consistent with past research demonstrating that training in community settings matches training in more traditional settings¹⁴.

Conclusion

In conclusion, this study examined the skill development before and after a rural clerkship. From the student's perspective, the 16 week preceptorship appears to be of major educational benefit, with students reporting significant improvement over a broad range of content areas and clinical skills. Future studies using objective measures of performance, such as knowledge exams and objective structured clinical examinations, are needed to examine outcomes of training in rural primary care settings and to document the students' self-perceived improvements.

Acknowledgements

This project was supported, in part, by Grant Number P20 MD000524 from the National Center on Minority Health and Health Disparities, National Institutes of Health. Contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health.

References

1. Norris TE, House P, Schaad D, Mas J, Kelday JM. Student providers aspiring to rural and underserved experiences at the University of Washington: promoting team practice among the health care professions. Academic Medicine 2003; 78: 1211-1216.

2. Kassebaum DG, Szenas PL. Rural sources of medical students' and graduates' choice of rural practice. Academic Medicine 1993; 68: 232-236.

3. Brooks RG, Mardon R, Clawson A. The rural physician workforce in Florida: a survey of U.S. and foreign-born primary care physicians. Journal of Rural Health 2003; 19: 484-491.

4. Brandau D, Huen L. Effect of clerkship training at rural sites on students' clinical skills. Academic Medicine 1994; 69: 691.

5. Bowman RC, Penrod JD. Family practice residency programs and the graduation of rural family physicians. Family Medicine 1998; 30: 288-292.

6. Rabinowitz HK, Diamond JJ, Markham FW, Hazelwood CE. A program to increase the number of family physicians in rural and underserved areas. Journal of the American Medical Association 1999; 281; 255-260.

7. Glasser M, Stearns JA, Stearns MA, Londo R. Screening applicants for a rural medical education program. Academic Medicine 2000; 75: 773.

8. Stearns JA, Stearns MA, Glasser M, Londo R. Illinois RMED. A comprehensive program to improve the supply of rural family physicians. Family Medicine 2000; 32: 17-21.

9. Probst JC, Moore CG, Baxley EG, Lammie JL. Rural-urban differences in visits to primary care physicians. Family Medicine 2002; 34: 609-615.

10. Wackerbarth SB, Johnson MM, Markesbery WR, Smith CD. Urban-rural differences in a memory disorders clinical population. Journal of the American Geriatrics Society 2001; 49: 647-650.

11. Levy BT, Merchant ML. Factors associated with higher clinical skills experience of medical students on a family medicine preceptorship. Family Medicine 2005; 37: 332-340.

12. Jensen CC, DeWitt DE. The reported value of rural internal medicine residency electives and factors that influence rural career choice. Journal of Rural Health 2002; 18: 25-30.

13. Bourke L, Sheridan C, Russell U, Jones G, DeWitt D, Liaw ST. Developing a conceptual understanding of rural health practice. Australian Journal of Rural Health 2004; 12: 181-186.

14. Worley P, Esterman A, Prideaux D. Cohort study of examination performance of undergraduate medical students learning in community settings. BMJ 2004; 328: 207-209.

15. Delaney G, Lim SE, Sar L, Yang SC, Sturmberg JP, Khadra MH. Challenges to rural medical education: a student perspective. Australian Journal of Rural Health 2002; 10: 168-172.

16. Londo R, Glasser M. Community-oriented primary care: an early report on a promising innovation. Education for Health 1999; 12: 299-307.

17. Worley P, Prideaux D, Strasser R, March R, Worley E. What do medical students actually do on clinical rotations? Medical Teacher 2004; 26: 594-598.

18. Maple SA, Jone TA, Bahn TJ, Kiovsky RD, O'Hara BS, Bogdewic SP. Tracking the contribution of a family medicine clerkship to the clinical curriculum. Family Medicine 1998; 30: 332-337.

19. O'Hara BS, Maple SA, Bogdewic SP, Saywell RM Jr, Zollinger TW, Smith CP. Gender and preceptors' feedback to students. Academic Medicine 2000; 75: 1030.

20. Chaytors RG, Szafron O, Crutcher RA. Rural-urban differences in procedures performed by family practice residency graduates. Family Medicine 2001; 33: 766-771.

21. Probst JC, Moore CG, Baxley EG, Lammie JJ. Rural-urban differences in visits to primary care physicians. Family Medicine 2002; 34; 609-615.

22. Zonia SC, Stommel M. Interns' self evaluations compared with faculty's evaluations. Academic Medicine 2000; 75: 742.

23. Gordon MJ. A review of the validity and accuracy of self-assessments in health training. Academic Medicine 1991; 66: 762-769.