download PDF
full article:
Introduction
Publishing research findings in peer-reviewed journals is an important aspect of research knowledge exchange1. However, the motivators to publish vary between individuals and settings. In academic settings, the drive to publish health research findings can potentially be motivated by a need to maximise publication rates for academic advancement or promotion as much as the desire to share findings for the good of health consumers and providers2,3. Within health services, research and the publication of findings is increasingly being linked to translation of research into policy or practice change processes4 aimed at improving health service delivery or health outcomes for consumers.
While clinician-led research has the distinct benefit of being close to practice and therefore of high relevance to translating health improvement, the researchers are not highly trained in writing for publication (WFP), and do not usually have quarantined time to write5. This means that much of the important evidence-based research conducted in health settings is not being shared formally with peer health professionals or between health services5,6.
The publication process can be stressful and challenging, even for experienced academics and researchers1. Publication rates continue to be used as a measure of individual and institutional performance3,6, yet average publication rates per institution remain low and vary dramatically between academics. Two large Australian studies7,8 both showed that a high proportion of publications was contributed by a small number of staff, supporting the ‘many published by the few’ phenomena first reported by Lotka9.
Despite awareness of the need to share research findings, awareness in academic circles of the need to increase academic publication rates, and awareness within health services that publication of clinician-led research findings could be considerably enhanced, published research about the effectiveness of practical strategies to support academic writing is minimal6. A comprehensive review showed that publication rates increase following participation in a writing support group or writing course, or having a writing coach, with no intervention type being substantially more effective than the others2. Where pre- and post-data were available, publication rates improved at least twofold2. While it is not known which aspects of these programs are most facilitative, a multifaceted rather than single-intervention approach was recommended for future efforts at increasing publication rates.
Evidence that a multifaceted approach could be effective was also demonstrated by Kamler6. In this small cohort study, the effectiveness of a one-week WFP course combined with a monthly writers support group to increase publication rates was assessed. Submissions increased from 9 to 33 articles in peer-reviewed journals. Publications (in print) per person increased from a baseline of 0.5 to 1.2 per year. Participants reported increased writing confidence and greater satisfaction with the publishing process. Peer support and receiving recognition and encouragement from line managers were also cited as incentives to publish.
A series of mentored WFP retreats was extremely effective in increasing publication rates of Australian nurses, with 37 out of 39 participants from three retreats submitting for publication, resulting in 17 publications and two book chapters5. A WFP program established in the UK for 10 university academics was much less intensive, involving four face-to-face meetings over 6 months, but regular meetings with mentors and peer program writing buddies, and a feedback mechanism within the program10. The program resulted in four manuscripts being accepted for publication, a further six being submitted, and three conference papers and three book chapters being completed.
In addition to support groups, courses and coaching, co-authorship with supervisors has been reported as vital in getting a profile for student writing in both education and science. Co-authorship has produced international refereed publications, helped students move through the struggles and anxieties of publishing and taught them how to be robust in the face of rejection and ongoing revision6.
Some argue that while publications, conference presentations and book chapters are the objective of WFP, the ‘softer’ skills of writing are equally important and include developing skills in reflexivity, teamwork, collegiality, peer feedback, talking about research and developing critical thinking about research5,10. Kamler proposes that productive writing is both a philosophy and a commitment that is undertaken at an organisational level6,11.
WFP requires a specific style and standard and the use of technical skills that may seem unattainable to novice researchers who describe emotions such as ‘agitation, resentment, despair and fear’ in relation to writing tasks5,11. Novice researchers may not see themselves as published authors, particularly if they are not supported by an experienced research team or are limited by a lack of resources and opportunities to develop adequate writing skills.5.
The demands associated with clinical work and teaching are often prioritised over activities such as WFP by clinician researchers who work within primary health or educational organisations5. Novice clinician researchers working in rural areas also experience geographic isolation from research peers. While the expansion of web-based publications and health-service-based library services has increased access to resources and scientific literature for rural clinicians, the barriers to publication remain substantial.
Between the commencement of the New South Wales (NSW) Health Education and Training Institute Rural Research Capacity Building Program (RRCBP) in 2006 and the end of 2011, three of the 56 (5%) then RRCBP graduates published a total of three papers in peer-reviewed journals, a publication rate of 0.05 per graduate. These publication outcomes were despite participation in a 2-year researcher development program, and submission of a 25-page research outcomes report, which was often well suited to be adapted into a manuscript for publication.
In 2012, RRCBP initiated a WFP ‘bootcamp’. This was a novel opportunity for novice clinician researchers to write for publication, which transcends distance and isolation. The WFP bootcamp was developed with the aim of transitioning novice rural researchers to completion of a draft manuscript for their selected journal by the end of the bootcamp, and being submission-ready 6 weeks after completion of bootcamp. On the basis of existing literature2, the goal of a 50% submission rate was set. The teleconference format of bootcamp was decided upon to facilitate participation by rural clinicians.
The aim of this study was to evaluate the impact and outcomes of a teleconference-delivered, WFP bootcamp. Specifically, the objectives were to determine (i) whether participation in bootcamp increased publication rates for novice researchers, (ii) whether bootcamp provides novice researchers with the knowledge, experience and confidence to submit a manuscript for publication, (iii) whether bootcamp structure and delivery was suitable for novice rural researcher health clinicians, and (iv) the cost of bootcamp per submitted and published paper.
Methods
Participants
The participants in this study are employees in the public hospital system in rural NSW (Australia) who had graduated from the RRCBP (n=112) prior to 30 June 2016. The background, features and outcomes of the RRCBP have been described and evaluated previously12; however, an important feature is that to be eligible for the RRCBP, candidates must be bone fide first-time researchers. WFP bootcamp was a voluntary, structured extension of the RRCBP, which involved both teaching and practical experience in each stage of the WFP process. Theses stages were undertaken sequentially during six 1-hour group teleconferences held on a weekly basis on a working day, but at 8 am to ensure clinical commitments were not impeded. Bootcamp teleconferences were facilitated by RRCBP program managers who had training and experience in the academic publication process. Individual homework tasks that involved incremental writing, sharing work with writing buddies and providing written feedback to other participants occurred in the intervening time between teleconferences. The structure and content of the bootcamp program is shown in Appendix I. Fifty RRCBP graduates (bootcamp group) completed at least one WFP bootcamp, and two commenced but did not complete bootcamp. The 62 RRCBP graduates who did not complete a bootcamp therefore constituted the control group. For those participants who participated in more than one WFP bootcamp (n=9), only evaluation data from their first bootcamp enrolment was included in analysis (Fig1).
Data collection
Data for this study were from four different sources: existing evaluation data from each bootcamp held by program coordinators, existing data on reported RRCBP publications, a WFP questionnaire, and RRCBP budget and program information.
Bootcamp evaluation data: The eight WFP bootcamp programs and end-of-program evaluations were conducted between 2012 and 2015. To evaluate the structure and content of bootcamp, evaluations were completed at the conclusion of each program, using an identical 28-item questionnaire consisting of a combination of multiple-choice, Likert scale and open-ended questions. All participants who commenced each WFP bootcamp were provided with evaluation surveys within a week of the 6-week teleconference (Fig1). Qualitative analysis of open-ended questions was conducted by the cut-and-paste method, to represent common topics and the range of responses13.
Publication outcomes data: In July 2016, all 112 RRCBP graduates were emailed an invitation to participate in the study; the invitation included a participant information sheet (PIS) and a Word document questionnaire about publication outcomes (here on referred to as WFP questionnaire). Questionnaires were sent by and returned to a member of the research team who was not directly involved in bootcamp program delivery. Participants were advised in the PIS that completion and return of the questionnaire was regarded as consent to participate.
To measure and compare publication outcomes, publication-specific information about completion of manuscript, submission of manuscript and publication in a peer-reviewed journal was collected from all RRCBP graduates. As shown in Figure 1, the bootcamp group completed WFP questionnaire version 1 and the control group completed WFP questionnaire version 2. Publication outcomes were checked by two researchers independently by conducting an author search in Google Scholar and in one other database specific to the study topic. Any publication in a peer-reviewed journal (or peer-reviewed conference proceedings) published after the candidate had completed the RRCBP and by July 2016 was added to the database and cross-checked against publications documented by participants. Publication rates for bootcamp participants (n=50) were compared using the same criteria against publication rates of RRCBP graduates who chose not to participate or did not complete bootcamp (n=62).
Publication efficacy data: For bootcamp participants, additional information was collected on ‘Knowledge’, ‘Experience’ and ‘Confidence’ in ‘Writing a research report’ and ‘Publishing research’ from the WFP questionnaire version 1. These two dimensions were selected from the 10 research skill dimensions in the Research Spider, a tool that was developed in consultation with practice-based researchers and academics to measure research skills14. Participants retrospectively self-reported these efficacy measures prior to participation in their first bootcamp and after the most recently completed bootcamp (Fig1). A 5-point Likert scale, ranging from 1 (no confidence/no experience) to 5 (very confident/very experienced) was used to measure these dimensions.
Emails were sent to participants on a 3-monthly basis to update the publication status of manuscripts. Evaluation data were collated on an Excel spreadsheet for analysis for the purposes of evaluating and improving the program.
The cost of conducting a WFP bootcamp was calculated by summations of wages for program development and revision, recording costs, evaluation costs and incidentals. This was divided by total number of bootcamp participant episodes (n=59) and by publications to determine cost per participant, cost per submission and cost per publication.
Figure 1: Flowchart demonstrating allocation of Rural Research Capacity Building Program (RRCBP) graduates to bootcamp or control groups, completion of post-bootcamp evaluation and completion of writing for publication (WFP) questionnaires.
Statistical analysis: Descriptive statistics were used for analysis of participant information and program component completion. Data from the existing evaluation and additional data collected for this study were collated in Excel and then transferred into STATA statistical software v10 (College Station, Texas USA) for statistical analysis by an RRCBP program manager who had not been involved in the delivery of bootcamp.
Publication outcomes relative to bootcamp participation were calculated by assigning one manuscript to each bootcamp episode. Overall publication rates were calculated by dividing the number of total publications (including peer-reviewed journal and published conference proceedings) by number of candidates in the group (bootcamp or control) to determine the mean publications per candidate. A two-sample χ2 test was used to compare publication rates between groups.
The Wilcoxon rank sum test was used to compare participants’ levels of involvement in the peer feedback component of the program and their submission status. Bootcamp participants reported whether they gave feedback and received feedback at six points in the program (maximum of 12 feedback opportunities, one point each). The total score out of 12 for giving and receiving feedback, and out of six for receiving feedback was measured against the submission outcome (yes/no).
Pre- and post-program confidence, experience and interest in ‘writing a research report’ and ‘publishing research’ of bootcamp participants were compared individually by Wilcoxon signed rank tests for non-parametric data. The mean pre-bootcamp scores for ‘writing a research report’ and ‘publishing research’ were calculated by averaging scores for confidence, experience and interest and compared with post-bootcamp scores by Wilcoxon signed rank test to determine changes in overall writing and publishing qualities15,16.
The cost of delivering bootcamp was assessed using a combination of actual and estimated costs to the program coordinators. These were then divided by the number of publications, submitted manuscripts and participants. Bootcamp was offered free to participants. Other costs to participants such as time or phone costs were not considered in the calculations.
Ethics approval
Ethics approval for this study was granted by the Greater Western Human Research Ethics committee in June 2016, approval number LNR/16/GWAHS/62.
Results
Fifty-two out of 112 RRCBP graduates commenced bootcamp, with two who did not complete bootcamp excluded from the study. Of the 50 graduates in the bootcamp group, 88% completed and returned program evaluation questionnaires. Each bootcamp group was multidisciplinary in nature, consisting of nursing (n=20), physiotherapy (n=10), management (n=7) and other allied health professions (n=13).
Submission and publication outcomes
The response rate for the WFP questionnaire was 26% (29/112), 17 from the bootcamp group and 12 from the control group. Self-reported publication and program records were combined and independently verified to determine that 26 out of 50 (52%) bootcamp participants had submitted manuscripts to peer-reviewed journals, of which 21 (42%) had been published. Four of the 21 published papers were in international and 17 were in Australian peer-reviewed scientific journals. Bootcamp publication outcomes compared favourably with the 15% of non-participants who achieved publication (χ2(2, N=112)=10.02, p=0.0015).
The overall publication rate of bootcamp participants (total publications since bootcamp/participants) was 0.80 (40/50), including 37 peer-reviewed publications and three published conference papers. The overall publication rate of 60 RRCBP graduates who did not participate in bootcamp was 0.23.
Knowledge, experience and confidence in writing for publication
Bootcamp participants rated their pre-program research-related knowledge, experience and writing higher than their knowledge, experience and confidence in the publication process. Significant increases in all measures were achieved post-program (p<0.01), with publishing experience increasing from 0.8/5 to 3.6/6 (+2.8/5.0) (Table 1). The mean increase in scores were 1.5 for writing (p<0.01) and 2.3 for publishing (p<0.01). The level of engagement in receiving feedback (p=0.66) or providing and receiving feedback (p=0.96) between peers within the program did not impact on submission rate (Table 1).
The responses to open-ended questions were consistent and detailed. Participants valued the collegiate atmosphere created by sharing their bootcamp experience, particularly giving and receiving feedback on their writing (Appendix II). A particular aspect of bootcamp that was valued by the novice researchers was feeling comfortable in their relative inexperience in academic writing.
Bootcamp participants reported increased confidence in approaching the challenge of WFP following bootcamp, and this was enhanced if they did so collaboratively. The increased confidence reported by bootcamp participants has been applied more widely than to their own publication efforts, to assisting work colleagues in the publication process and for other scientific writing tasks.
Some suggestions to further enhance the bootcamp program related to participants’ needs for the ongoing or subsequent involvement of an experienced writer. The need for time to complete the WFP task was underestimated by some participants, and considered necessary to complete the submission process.
Some interesting insights were offered by the respondents who did not participate in WFP bootcamp. A graduate from the first RRCBP cohort in 2006, and who has subsequently published seven-peer reviewed journal papers, reported that all of their publications have their origin in early skill development and encouragement that was derived from the RRCBP.
Table 1: Self-rated retrospective changes on writing and publishing knowledge, experience and confidence assessed on a 5-point scale pre- and post-program (from writing for publication questionnaire n=17)
Structure and content of bootcamp
All of the 44 questionnaire respondents would recommend WFP bootcamp to others and 43 (98%) rated it as a positive experience, giving it a ‘thumbs-up’ (Table 2). Other features of the program that rated highly were rotation of writing ‘partners’ (n=42, 96%), receiving feedback on writing (n=41, 93%), weekly workload (n=41, 93%) and reviewing the work of others (n=40, 90%). One quarter of participants (n=11) indicated the estimated time commitment of 40 hours over the 6-week program was not sufficient or that they did not have time to commit to the estimated 40 hours (Table 2). Comments from participants on the time commitment revealed that they either completed their manuscript in their own time, or had management support to utilise research or study leave to fulfil bootcamp commitments.
Table 2: Writing for publication bootcamp post-program evaluation from 2012 to 2015
Cost of bootcamp per submission and publication
The cost of conducting a bootcamp was determined from the perspective of the delivery organisation and included program manager wages for program development and revision, teleconference recording expenses, evaluation costs and incidentals. The cost per publication (n=21) was $230.40, per submission (n=31) was $156.08 and per bootcamp participant (n=59) was $82.01.
Discussion
The WFP bootcamp outcomes indicate that conducting a short course over 6 weeks by teleconference is effective in increasing the publication rates of novice clinician researchers. The submission rate (52%) exceeded the 50% submission goal set for bootcamp and compares well with publication rates for programs that involve experienced researchers and academics2, particularly considering the relatively low intensity (weekly teleconferences) and low cost per publication of the bootcamp intervention.
Having completed a 2-year researcher development program, undertaken a research project and submitted a 25-page dissertation, RRCBP graduates were well positioned to publish. However, prior to the inception of the WFP bootcamp program, 5% of RRCBP graduates had published, with a publication rate of 0.05. Bootcamp was introduced as a strategy to increase publication rates. The 42% result for papers published and 0.80 publication rate per participant compare favourably with publication statistics for non-bootcamp RRCBP graduates (15% published, 0.23 publication rate) and with the publications per year reported by McGrail of 0.25-4.4 for support groups, 0.02-1.1 for writing courses and 0.4-0.9 for writing coaches2. Considering the predominantly clinical roles and lack of allocated research and writing time of the rural health clinicians, the bootcamp publication outcomes are encouraging. An important predictor of future publishing success has been noted to be publishing from a dissertation6. This positions RRCBP graduates to contribute to the literature in the future.
The WFP bootcamp helped participants develop knowledge, experience and confidence related to publishing, and experiences as a writing partner learning to give and receive critical review were generally well received. The teleconference format did not seem to limit the collegial approach. Despite differing strategies to increase publication rates, the findings that related to open-ended questions were consistent with those from qualitative studies of WFP interventions2,10,12. The support participants reported experiencing from their peers and the facilitators resulted in increased confidence and motivation in writing. Similarly, Morss and Murray reported that:
the phenomenon that ‘others are in the same boat’ was reassuring for writers, and that increased motivation to write was linked to an increase in confidence to engage with editors, to ask for advice and editorial comments from peers and to submit papers to conferences and journals10.
Responses to open-ended questions indicated that collaboration with experienced peers, the facilitated nature of the program and perseverance were key factors that determined submission of manuscripts to peer-reviewed journals. These results are highly consistent with the findings of McGrail2, Kamler6 and Rickard et al17, suggesting that skilled support from knowledgeable supervisors is critical to publication outcomes.
Although the time to complete the WFP task was underestimated by some participants, the high submission rate indicates that participants found ways to make time to write. This is consistent with previously reported willingness and ability of WFP program participants to use brief periods of time for productive writing10. The varying levels of organisational support for allocation of writing time to complete manuscripts aligns with Kamler’s proposition that WFP needs to be a philosophy supported by the organisation, in order to further improve publication outcomes6.
The retention rate in the program was high, despite the high commitment and expectation that program requirements would be completed predominantly in the participants’ own time. The program evaluation indicates that the workload was high but as expected, and that the program was both challenging and rewarding to participants. The format of six teleconferences (weekly at 8 am on a workday) with ‘homework’ between was selected to minimise impact on the working day of busy clinicians.
With the majority of previous interventions in the field being conducted for academics rather than novice researchers who work predominantly as clinicians, comparison of outcomes is challenging. Previously published programs aimed at increasing publication rates have used a range of approaches to provide participants with the time, support and space they need to write quality manuscripts. Those that would be appropriate for health services include use of study leave or allocation of time for writing in roles in which there is a research component to the position description.
The bootcamp delivery costs were estimated at $230 per publication, and considered to be reasonable to publish unique health-service-level clinician-led research in the scientific literature. Comparison between the WFP bootcamp and similar WFP programs was not possible, as costs of other interventions have not been made available in the literature. As bootcamp has been demonstrated to be as effective as other more intensive interventions, there is potential to promote and deliver the program more widely and to apply the teleconference format to other forms of publication support, such as systematic reviews. Furthermore, the teleconference format could complement other WFP formats such as face-to-face courses, writing retreats, coaching and co-authorship, adding to the multi-faceted programming suggested by others2.
Strengths and limitations
To the authors’ knowledge, this is the first study to report the outcomes of a remotely conducted WFP program in increasing publication rates. The high retention within the program and high program evaluation rates have allowed for comprehensive program and outcome measurement.
The voluntary nature of enrolment in WFP bootcamp may mean groups were not equivalent on relative interest in publishing or perhaps research findings worthy of publication. Our inability to capture differences between the bootcamp group and the control group on these variables is a limitation of this article.
It was not possible to estimate publication rates for the 2011, 2012 to 2013 RRCBP cohorts if bootcamp had not proceeded. However, it is reasonable to estimate that they would have been similar to pre-bootcamp publication rates and therefore the publication rate increased from 5% to 42% following introduction of bootcamp.
Publication rates for bootcamp participant group and the control group were both affected by the ‘many publications by few’ bias but this was consistent with the RRCBP publication outcomes prior to bootcamp and with previously cited literature2.
Perceptions of knowledge, confidence and experience were measured retrospectively, with participants being asked to remember how they would have rated their writing and publishing skills prior to commencing bootcamp. While recall bias could be a factor here, it is also possible that participants were better equipped to assess writing and publishing knowledge, confidence and experience retrospectively.
Conclusion
The completion and publication of health service level research is becoming increasingly prioritised as a means of improving patient outcomes and experiences18. The translation of research into practice is much improved if the research is of importance and relevance to patient needs and if findings about practice improvements are accessible to other clinicians and researchers. The WFP bootcamp was effective in increasing the knowledge, experience and confidence in the WFP process for novice researchers. The flexible delivery approach across a diverse geographic range makes this program format generalisable to a range of settings, disciplines and researchers and is particularly suited to the rural and remote setting.
This research shows that novice researchers respond to similar intervention features as experienced researchers when engaging in WFP, and that WFP outcomes can be increased substantially with modest investment of funding and resources by the host organisation. Remote program delivery by teleconference increased the access of rural clinicians to a WFP program, and was not reported to impede the development of writing skills or program support.
Health professionals have variable writing skills for many reasons, one of which is the lack of emphasis on academic writing development while at university. Realising that variability in writing skills will remain a continuing issue, the outcomes of bootcamp have the potential to be shared within health service organisations to show that publication outcomes can be improved with organisational commitment and structure that facilitate academic writing skill development.
