download PDF
full article:
Introduction
Conducting research in rural and remote areas is compounded by challenges associated with accessing relatively small populations spread over large geographical areas. Individuals residing in rural and remote areas are often underrepresented in research, despite them constituting a vital part of the population, and offering unique experiences and perspectives. This is particularly worrisome at a time when user-driven research has been identified as vital in ensuring the ongoing utility of health research1.
Postal surveys have multiple advantages as a research method, including being the most cost-effective method for population-based research, and obtaining a considerably higher response rate than internet surveys2. The option for open-ended questions means that it is possible to ask for in-depth perspectives, from a large sample, in a relatively resource-efficient way, which is particularly beneficial when targeting rural and remote populations because the geographical isolation of these areas means that research is typically resource-intensive. Such questions are frequently included in population-based surveys; however, they are rarely analysed3. In part this is due to the resources required to manually evaluate free-text responses. Another deterrent is that researchers are not able to prompt respondents, hence responses are often briefer than those obtained by more resource-intensive methods such as telephone interviews4. Options for analysing short-text responses are limited because these responses are not appropriate for detailed qualitative analysis, nor can quantitative methods be applied. Yet, recent research has begun to validate the use of these comments as research studies are beginning to draw on the value of open-text responses5,6.
The aims of this article are to (1) explore word cloud analysis and evaluate the utility of automated programs to supplement the analysis of open-ended survey responses, (2) determine whether meaningful information can be extracted from short-text responses from rural residents who were asked to report their priorities for local health services and (3) identify the characteristics of survey respondents who answer open-ended survey questions. If successful, this method could be usefully applied to countless population-based studies that include open-ended questions, by providing a method to utilise short-text responses while reducing the time required to manually evaluate open-text responses.
Methods
Sample and data collection
This analysis uses data from the Australian Rural Mental Health Survey (ARMHS), a longitudinal population-based study exploring the determinants of mental health in rural and remote New South Wales7. Participants completed self-report postal surveys related to mental health and wellbeing (which we acknowledge may have introduced bias to the present analysis; this is further discussed in the conclusion). The present study uses data collected in Wave 4 (2012) of ARMHS, which included the open-ended question ‘What health services would you like to see the local health district providing that are currently not available in your area?’ ARMHS received ethics approval from the Human Research Ethics Committees of the Universities of Newcastle and Sydney, and the relevant area health services.
Data analysis
Initial familiarisation of the data took place using NVivo software v10 (QSR International; http://www.qsr.com). When the researchers became aware of the brevity of responses (often just a few words), a summative approach to content analysis was applied to the data; that is, the frequency of words used was examined to assess meaning8. The analytical software Wordle (http://www.wordle.net.au) was employed to complete the word-cloud analysis. Wordle is a relatively new text frequency program that presents data in a word cloud. The size of the word in the cloud illustrates how many times, in proportion to other words, a word has appeared in responses9. While this approach is similar to a word frequency table, it has an additional pictorial aspect that makes results more attractive and easy to assess visually, and may be a more appealing and acceptable way to present research results to non-academic audiences. This research tool has increasingly been used in educational, qualitative research and in the media10.
The Statistical Package for the Social Sciences v20 (SPSS; http://www.spss.com) was used to explore sample characteristics. Univariate analyses were conducted using χ² tests.
Results
In total, 388 of the 1165 Wave 4 ARMHS participants (33.3%) provided a response to the free-text question. A description of the sample is provided in Table 1, with responses to the research question depicted in Figure 1. From the overall ARMHS sample, those who did not write a response to the open-ended question were more likely to be males (n=777 (40.7%) vs n=127 (32.7%); χ²(1)=6.9, p=0.009), and less likely to report a mental health problem (n=762 (21.7%) vs n=105 (27.3%); χ²(1)=4.6, p=0.032).
Using the word cloud as a visual guide, key words were identified and used to locate relevant quotes from the full open-text responses. ‘Mental health’ was the most frequent request, cited by 81 people (20.8%). Many participants differentiated between youth mental health services (eg ‘Mental health services for young adults with Aspergers’ and ‘Child mental health team’) and adult mental health (eg ‘Mental health for middle aged, not necessarily medication’ and ‘More community mental health services for the aged’), with some responses also calling for specific multicultural mental health support in their communities. Participants noted the need for increasing accessibility to mental health services by ensuring that ‘bulk billing’ (where there is no out-of-pocket expense for the patient, and costs are paid to the service provider by the federal government) (n=8) was available. When calling for mental health services, some participants commented on long waiting times for appointments (eg ‘Trained psychologists that you can access immediately when needed, not wait for weeks’ and ‘Getting a mental health plan through to GP in less than 3–4 weeks’).
Following ‘mental health’, requests for more ‘specialists’ (n=59) and ‘services’ (n=53) were the second and third most frequent responses, respectively. Visiting specialists were requested by multiple respondents (n=14). Participants described their need for specialists in a variety of ways; some responses were specific to a medical condition (eg heart or urology), while others were more general. Participants described the transport burden associated with service access in rural and remote areas, writing that they would like ‘any service that doesn’t require a 240km round trip!’
As can be seen in Figure 1, ‘GP’ (general practitioner; n=41) and ‘dental’ (n=31) were frequently prioritised by this rural sample. Specifically, many called for a 24-hour medical practice or increased opening times, as well as bulk billing for GPs and dental services. Requests included ‘low cost dental services’ and ‘public dental clinic’.
Less frequent requests illustrated in the word cloud are important when considering representatives from smaller population groups – such as those with specific health needs or conditions including ‘maternity’ services (n=13), ‘cancer’ (n=10), ‘drug and alcohol’ services (n=8), and ‘aged care’ (n=7) services – are all core services, even though they were being called for by fewer people. This lesser frequency may suggest that these services are already considered as available in some rural and remote communities.
Table 1: Demographic characteristics of the ARMHS sample participants responding to the open-ended question ‘What health services would you like to see the local health district providing that are currently not available in your area?’ (n=388)
Figure 1: Word cloud depicting requests for additional health services by rural and remote community residents.
Conclusions
This research aimed to determine whether meaningful and informative data could be obtained from short responses from open-ended survey questions using an automated data analysis technique to supplement a more in-depth analysis. Our findings showed that, while not as detailed as interview responses, the open-ended survey questions provided sufficient information to develop a broad overview of the health service priorities identified by this large rural sample. After identifying the key themes, these responses were easily contextualised and expanded upon through the full free-text data. Such automated data analysis techniques are rarely employed3; however, the current research provides valuable support for their utility in rural and remote health research.
We also aimed to determine the characteristics of survey participants who respond to open-ended questions, with our findings suggesting that these respondents largely reflect the overall sample, with a few exceptions. First, males were less likely to respond than females, which is a common occurrence in research in general. Second, those with a mental health problem were more likely to respond, which may reflect the fact that this was a mental health survey. Because ARMHS is a study on rural mental health, responses to our question may have been biased by the broader questionnaire, potentially accounting for the high demands for mental health services; however, other research concurs that this finding is relevant and called for by rural residents11.
This study demonstrates that it is feasible and resource-efficient to obtain open response data from rural and remote residents, and that such data are a useful and valid source for qualitative analysis. The ARMHS data is unique because it was obtained from a large random sample of rural and remote residents, which is often expensive and difficult12. This research has implications for researchers interested in engaging rural and remote residents. Although short-text responses are not suitable for all qualitative or quantitative data analysis methods, the word cloud methodology employed has demonstrated that meaningful information can be extracted from this data, contributing a resource-efficient supplement to a more detailed analysis. Open-ended questions are often asked in population-based studies yet they are rarely analysed, posing an opportunity yet concurrent challenge for researchers in utilising such participant-driven responses. The lessons learned from the methodology applied can be transferred to other population-based survey studies more widely, not just in rural and remote research. In particular, findings from the present study may be used to determine health service priorities for rural and remote areas, to ensure that services are meeting the needs of the target population.
Acknowledgements
We wish to recognise the contribution of the ARMHS chief investigators: Professor Brian Kelly, Professor David Lyle, Associate Professor Lyn Fragar, Professor John Beard, Professor Vaughan Carr, Professor Jeffrey Fuller, Associate Professor Helen Stain, Professor Prasuna Reddy and Senior Project Co-ordinator Dr Clare Coleman. We wish to acknowledge the support of directors of mental health services in the relevant local health districts during the course of this phase of the study: Drs Russell Roberts, Richard Buss and Dinesh Arya, and particularly acknowledge the research site coordinators in each site: Jan Sidford, John Ogle (Broken Hill), Trim Munro, Amy Strachan (Moree), Louise Holdsworth, Kath O’Driscoll (Lismore), Cheryl Bennett, Jannelle Bowler (Orange), along with Fleur Hourihan, Dr Gina Sartore and Denika Novello. Tonelle Handley is supported by a postdoctoral fellowship from Australian Rotary Health, which is acknowledged with gratitude.