Evaluation of a rural community-based disability service in Queensland, Australia

full article:

Utilising a largely unexploited implementation strategy and service framework, a semi-formal, rural disability service delivery project was established in central Queensland, Australia, with considerable support from and participation of the community¹. This initiative, undertaken by two of the authors (PK, EK), sought to implement a community-based model for responding to disability concerns in a rural setting and to explore outcomes according to a number of parameters. The present article describes the framework developed to evaluate this service and documents a number of findings of interest. Given a recent call for improved evaluation techniques in participatory rural research projects², the suggestions proposed in the innovative evaluation framework developed for this study are timely.

Rural community-based disability service

Characteristics of the service

Through an eight-phase implementation strategy adapted from Participatory Rural Appraisal (PRA)³, an informal community disability service was developed which operated with research funding in the town and surrounding district of Taroom, central Queensland for 2 years until late 2001¹. At the time of publication, the service continues in a voluntary format, in much the same as it was implemented, although in a slightly reduced capacity.

Individuals originally involved in the service included:

• A worker (coordinator) identified through the eight-phase PRA approach who, until completion of the project, was paid a nominal weekly amount to assist with expenses. This person responded to identified needs where appropriate but mostly linked people with skills and time to others with expressed need.
  
• A core group of community members who were closely involved were called on if meetings were required or decisions had to be made.
  
• A larger group of registered volunteers who were called upon from time to time to provide assistance to particular people in specific areas they nominated.
  
• A number of individuals (people with disabilities or some other health-related need) or families who requested some form of support or assistance.

In response to the wishes of community members (expressed in two community meetings) and the skills of the people involved, the assistance provided by the service was not disability-specific and was largely non-professional. The semi-formal service sought to provide assistance with day-to-day practical issues and to maintain a broad focus. Volunteers provided assistance to people with disabilities and others who may have had short- or long-term need in the following areas:

• transport
  
• meals
  
• respite
  
• information dissemination
  
• help in negotiating the 'service maze'
  
• raising community awareness of concerns and needs of people with disabilities
  
• home help
  
• assistance with child care
  
• basic independence training
  
• basic advocacy

The service has established links with people who may require assistance through one of four channels: (i) an individual who contacts the service directly to make an enquiry or request; (ii) a family member or friend who makes an enquiry or request; (iii) a formal referral from the health or other service system; or (iv) an offer made by the service. In the latter instance, the coordinator of the service may be alerted to a potential need through informal (or formal) channels, and an offer of assistance is made to an individual or family. As will be discussed, these channels are dependent on a high level of community awareness, maintained through comprehensive local distribution of materials, informal networks and displays at public events.

The evaluation

Characteristics of the evaluation

From its inception, the evaluation of this community-based service development project was recognised as challenging. Some of the self-evident complexities of evaluating initiatives that are characterised by their community orientation, participatory approach and informal methods have been noted^4,5. Such services are often multi-dimensional in approach⁶, and their focus may extend beyond health or disability per se, to broader social concerns⁷. The nature of the interventions is such that outcomes may only be evident years after an intervention⁸. Most importantly, the success of this type of service depends not only on the actual assistance provided, but also on the nature of the relationship and the level of trust established among people with disabilities, their families and their communities (BG Hope, Oral Presentation, Adelaide, Australia, 1997). Further, because community health and disability programs often seek to be responsive to the local community, their focus and activities may change over time, as may the stakeholders involved.

In recognition of these and other aspects of the project, it was evident from the outset that traditional project evaluation formats would have limited application. As a result, a unique evaluation framework, derived from a number of sources, provided an analytical and descriptive overview, reflecting strengths and weaknesses of the service. All authors played a role in the evaluation; the first author (PK) proposed the initial structure and format for the evaluation.

Process of the evaluation

Some traditional evaluation concerns formed the basis of the evaluation (eg the number of people assisted, the activities of the project and the nature of the assistance provided). However, of greater interest to the researchers were some of the less tangible - and less measurable - evaluation questions regarding the impact of the project and its place within the community. Information about concerns such as the quality of linking, community ownership, sustainability and the capacity of the service to be responsive to community needs, were of particular interest to the researchers and others involved in the project. Consequently, a considerable body of qualitative data was gathered throughout the project. Each of the data sources (A-F) was intended to present part of an overview of the context and activities of the project and the perspectives of those directly and indirectly affected by it. The qualitative data sources were:

1. Responses from written surveys/questionnaires conducted with 75 local people who expressed some interest in the project following a public meeting. Respondents included people with disabilities, family members and interested community members.
   
2. Transcripts of interviews and meetings with key community members, service providers in health, disability and community sectors. Conducted in Taroom and surrounding districts, these interviews explored current service realities and future service plans.
   
3. Notes from personal interviews with people with disabilities and family members who expressed an interest in the project.
   
4. Minutes of community and committee meetings that documented the discussions that guided the project.
   
5. Records of activities of the program, including direct contact with individuals and family members, liaison with various organisations and advocacy with other organisations.
   
6. Transcripts of interviews conducted with workers through the course of the project.

At the conclusion of the project, it became evident that a comprehensive evaluation framework would be required to make sense of the voluminous dataset. Thus, a literature review was conducted to identify appropriate evaluation dimensions and to document characteristics of optimal community rehabilitation principles or best practice in community building initiatives. Based on relevance of content area and context, the review identified five major publications that proposed a number of principles or concepts considered pertinent to the evaluation.

The first of the five studies, an extensive local evaluation study of community capacity and health⁹, identified 13 key concepts that comprised an index of the quality of community capacity building initiatives in rural areas. The second study drew from a large-scale review of studies of community participation¹⁰ to indicate 15 relevant concepts. Five hallmark principles of community-based rehabilitation were drawn from a seminal work in this area¹¹. A total of 18 concepts were drawn from a review of factors relevant to community capacity and service delivery in a local social planning study¹². Finally, a further 28 concepts were derived from an analysis of scores of successful community building projects around the world¹³. The five publications cumulatively documented a total of 79 concepts which were taken to be key characteristics of effective community-based practice, and so relevant to the current project. A substantial overlap was noted among the 79 concepts (some of which were themselves drawn from reviews of concepts). Consequently, the researchers categorised and synthesised the concepts in to 15 fundamental characteristics (Table 1).

Table 1: Fundamental characteristics and descriptors of effective community-based practice identified for the current evaluation with references



Having identified key characteristics of effective community-based practice relevant to the current project, the primary function of the evaluation was to extract available evidence to indicate the performance of the project against these 15 characteristics. The researchers scrutinised each data source (A-F above) for themes relevant to any aspect of the 15 concepts and entered these in a spreadsheet. A priority for this activity was to record as much information as possible, regardless of whether the information showed evidence for or against the existence of that characteristic in the project.

Using the data in the completed spreadsheet table, a SWOT (Strengths, Weaknesses, Opportunities and Threats) analysis was performed for each of the 15 fundamental characteristics¹⁴. After reading the statements derived from all the data sources for each characteristic, the researchers independently determined whether the data indicated that each fundamental characteristic was a strength, weakness, opportunity or threat for the project.

Strengths: were defined as positive characteristics that were internal to the project and could support the future delivery of the service, an advantage that the service had or something the service did well.

Weaknesses: were defined as negative internal characteristics, disadvantages of the service or activities performed poorly by the service.

Opportunities: were defined as positive characteristics or favourable trends that were external to the project and could be harnessed.

Threats: were considered to be negative characteristics that were external to the service and acted as obstacles, competing demands or forces that could damage the service in future.

In classifying each of the characteristics, consensus was reached among the researchers by discussion. An important aspect of this evaluation was the objectivity gained by determining the 15 characteristics after the data collection was complete. This reduced the potential bias of seeking-out certain types of data. A summary of the conclusions in SWOT format is provided (Table 2), as an example of the analysis (Appendix I).

Evaluation findings

An overview of the key quantitative aspects of the evaluation revealed that in the first 2 years of the project, 39 people and their families with various needs were assisted, six of whom had urgent needs. Through advocacy and negotiation with potential funding bodies, the service was instrumental in assisting 11 families to obtain funding to meet their disability-related needs (computers, appliances, modifications etc) to the order of a total of AU$18 000. The 39 recipients of the service included families of children with congenital disabilities, people with arthritis, chronic conditions, degenerative conditions and cancer, people who were temporarily hospitalised, adults with injuries and aged people with ageing issues.

The project was evaluated as being particularly cost effective, having been established and run for 2 years on a budget of approximately AU$28 000 (75% of a $37 000 research budget which also covered travel expenses and the incidentals of two metropolitan based researchers). Apart from the coordinator, the service was entirely voluntary.

Since the evaluation was completed, 31 volunteers remain registered with the service to provide some form of assistance to people with disabilities in this community. The service has developed a resource booklet and conducted a comprehensive local information and resource campaign. It continues to be well linked to local providers and stakeholders. It would appear that the service is becoming a reference point for formal service providers and plays a key role in local negotiations over future health-related services in the area.

SWOT analysis
Results overview of the evaluation according to the SWOT analysis results is summarised (Table 2).

Table 2: Categorisation of the 15 fundamental characteristics according to SWOT analysis



Strengths: Multiple sources, particularly interviews with community members and service providers, indicated that the project had a strong community focus. Feedback from volunteers and people assisted indicated that it both relied on and built social cohesion. The service was proactive in problem solving and responsive to changes in local needs as they evolved. For example, it was intended that the service would be a disability project but it expanded to include a variety of needs, such as assisting with transport for people undergoing medical treatment, assisting families with a child in hospital and assisting people with age-related and chronic health conditions. A significant strength of the service model as identified in interviews and feedback was its ability to be flexible and to provide a good balance of outcomes and process.

The project used a partnership model and a relationship-focussed approach to meeting local need that depended heavily on the 'connectedness' or degree of community involvement of the local coordinator. Given this local focus, the responsiveness of the service and the attention to relationships, it is not surprising that the evaluation revealed a strong level of trust in the service.

Weaknesses: Notes from meetings with service providers indicated that although important partnerships had been established with government departments, these appeared one-way, somewhat tenuous relationships that did not address the financial requirements of the service. Although the service operated according to a strong client-focused and community-based philosophy, minutes of meetings reflected that it had no policy of its own and did not appear to have influence beyond the immediate district. This situation was both a weakness inherent in the structure of the service and a threat to its longevity.

Opportunities: The degree of community knowledge and acceptance of the service indicated that the community building skills of the people involved were considerable, despite their lack of formal training. Interviews revealed that for some, this strong community connection was seen as an ideal foundation upon which to base formal training to strengthen the level of skills and the quality of the service. To date, however, knowledge transfer has been informal and support-oriented.

While the project was collaborative by nature and responsibility was shared, the project was clearly led by one individual. There appeared to be little community control over the daily decision-making, although a core steering group was responsible for its development. Despite the high level of local community awareness the project enjoyed, records showed that there were only a few people involved at any point in time. However, given the existing levels of trust and partnership, there appeared to be substantial opportunities and capacity for developing wider community participation and control over decision-making.

Threats: The limited size of the project meant no need was identified for formal communication systems or procedures, which may also have been a potential threat to sustainability in the long term. Despite the fact that the project was extremely cost effective (and is now entirely voluntary), funding to ensure sustainability of the service has been elusive. The finance and resource infrastructure of the service was seen as both a current weakness and future threat. Paradoxically, the informal nature of the intervention was simultaneously an obstacle and a strength in relation to the establishment of links with potential funders.

Other issues for consideration

The breadth of scope of the evaluation also led to the identification of some further general issues of relevance to the potential extension of this or similar projects.

Local community 'ally'

As described earlier, the researchers in this study relied heavily on a local community 'ally' in the form of the community coordinator/researcher. Anecdotal evaluation information from local sources indicated that the success of the project may largely be attributable to that person in their role. While the PRA process involved metropolitan-based researchers establishing a connection with rural community members, the magnitude of community outcomes achieved may not necessarily be attributable to this method. In addition to assisting researchers with 'entry' into the community, the community coordinator/researcher provided crucial information about the community, limited the potential bias of external researchers and facilitated rapport, thereby enhancing the reliability of information and interventions. While serendipity rather than technique played a role in the current study, the issue of the selection of such allies is crucial for future research and projects, given the importance of this person as gatekeeper to the community and its information.

Volunteer burden

While the project was highly successful and some community members were enthused and involved, many were not. As a result, a core group of three or four guided the project and a network of volunteers provided direct assistance. The risk of overburdening a small number of people was present in this initiative, however the small scale of the project and the community in which it was conducted, appeared to mitigate this risk. Clearly, a challenge for participatory projects such as this is to identify ways of minimising volunteer burnout. One such possibility is to identify strategies to foster and maintain the interest and involvement of a cross-section of people over time.

Ethical issues in rural community research

Ethical issues presented some challenges in this participatory research project. For example, data collection and interventions in this study involved the entire community, so traditional notions of consent lacked relevance. Ensuring confidentiality in a small town was also problematic, given that the intervention involved sharing of personal information, skills and resources. In this instance, the researchers sought to meet ethical obligations by ensuring adherence and commitment to ethical principles, rather than being able to guarantee to the ethics committee that certain actions or particular criteria would be fulfilled. Since inception of the project it is noted that the Australian National Health and Medical Research Council¹⁵ has released guidelines regarding research involving collectivities, which pertains to this form of research. These guidelines suggest that researchers should provide evidence of negotiation with representatives on issues associated with consent, privacy, confidentiality, harm, data ownership and dissemination. This process enables researchers to begin such projects with a range of community consultations and negotiations prior to obtaining ethical consent. Thus, by the time ethical consent is sought, clarity and agreement has been obtained. While this process is not flawless, it substantially facilitates the process by which ethical approval can be achieved and progress can be advanced in a project such as this.

Conclusion

The project described in the current article and elsewhere¹, reflects a trend in rural health services towards more flexible service provision with greater community orientation and community partnership¹⁶. The decision to use an evaluation framework based on key principles and a SWOT analysis has led to the identification of key positive and negative attributes, and has been informative for the service and other stakeholders. Although somewhat subjective and not without limitations, this approach to evaluation has ensured a direct link between the perspectives of key stakeholders and the conclusions drawn.

The role of the authors as instigators of the project (PK, EK) and local coordinator (TC) as well as evaluators may have increased the subjectivity of: (i) the evaluation process; and (ii) the interpretation of findings. This was mitigated to some extent by the use of colleague checking, multiple data sources, triangulation between data sources and a variety of methods of obtaining data (interviews, minutes of meetings, questionnaires etc). The authors chose not to introduce external evaluators or impose artificial constructs on the participants in the evaluation in order to remain consistent with the community philosophy of the service. This community-driven focus has been maintained throughout the project, from identification and implementation to evaluation.

Acknowledging recent suggestions that rural health research should be grounded in the rural and remote community perspective¹⁷ the current project has made some contribution by fostering and equipping a local ally to work with urban researchers by way of a community-driven action research methodology. The promotion of community participation and the development of community leadership in health and disability projects would appear both worthwhile and powerful. While the establishment of a community approach is a lengthy process, it also appears to be highly resource-effective and relevant in the long-term. Nevertheless, the current article has articulated several issues that require further attention if the sustainability of such projects is to be maximised.

Acknowledgements

The authors gratefully acknowledge the support of the Australian Centre of National Research on Disability and Rehabilitation Medicine (CONROD) and the Motor Accident Insurance Commission, Queensland, Australia, for partial funding of this project. The contribution and support of the Acquired Brain Injury Outreach Service and community members in the Taroom Shire is also gratefully acknowledged.

References

1. Kuipers P, Kendall E, Hancock T. Developing a rural community-based disability service: (I) Service framework and implementation strategy. Australian Journal of Rural Health 2001; 9: 22-28.

2. Campbell J. Participatory rural appraisal as qualitative research. Human Organisation 2001; 60: 380-389.

3. Chalmers R. The origins and practice of participatory rrural appraisal. World Development 1994; 22: 953-969.

4. Jaffer R. Monitoring and evaluation of community based rehabilitation - Need for a participatory approach. In: H Finkenflügel (Ed.) The handicapped community: the relation between primary health care and community based rehabilitation. Amsterdam: VU Press, 1993; 51-64.

5. Stilwell P, Stilwell J, Hawley C, Davies C. Measuring outcome in community-based rehabilitation services for people who have suffered traumatic brain injury: the Community Outcome Scale. Clinical Rehabilitation 1998; 12: 521-531.

6. Peat M. Community based rehabilitation. London: Saunders, 1997.

7. Miles S. Engaging with the disability rights movement: the experience of community based rehabilitation in Southern Africa. Disability and Society 1996; 11: 501-517.

8. Lipsey MW, Cordray DS. Evaluation methods for social intervention. Annual Review of Psychology 2000; 51: 345-375.

9. Bush R, Mutch A. The community capacity health development index - manual. Brisbane: Centre for Primary Health Care, University of Queensland, 1999.

10. Plummer J. Municipalities and community participation: a sourcebook for capacity building. London: Earthscan, 2000.

11. Helander E, Mendis P, Nelson G, Goerdt A. Training in the Community for People with Disabilities. (1989). In: M Sharma, S Deepak. A participatory evaluation of community based rehabilitation programme in North Central Vietnam. Disability and Rehabilitation 2001; 23: 352-358.

12. Soule D. Western Gateway Area Strategy. Brisbane: Department of Families, Youth and Community Care, 2001.

13. Mattessich P, Monsey B. Community building: what makes it work? A review of factors influencing successful community building. St Paul, Mn: Wilder, 1997.

14. Sharma M, Deepak S. A participatory evaluation of community based rehabilitation programme in North Central Vietnam. Disability and Rehabilitation 2001; 23: 352-358.

15. National Health and Medical Research Council. Human research ethics handbook. Canberra, Australia: NHMRC, 2001; Chapter 8. 2001. NHMRC Research Ethics Handbook.

16. Humphreys J, Hegney D, Lipscombe J, Gregory G, Chater B. Whither Rural Health? Reviewing a Decade of Progress in Rural Health. Australian Journal of Rural Health 2002; 10: 2-14.

17. Strasser R. Rural health research: Have we turned the corner? Australian Journal of Rural Health 2000; 8: 249-253.

_______________________________________

Appendix I: SWOT analysis table




First published 3 June 2003; Table 1 references modified 8 October 2004.