Problems measuring community health status at a local level: Papua New Guinea's health information system

full article:

Introduction

Health care in Papua New Guinea (PNG) is provided through a unified system of community aid posts, rural health centres and provincial hospitals¹. A network of over 2400 aid posts, 500 health centres and 45 urban clinics are supported by 18 provincial hospitals and one national hospital². Those who provide health care include community health workers, nursing officers, health extension officers and doctors³.

At the village level, the aid post serves a population of between 500 and 3000^4,5. As a traditional village community has less than 500 people, each aid post serves between one and 6 villages⁵. Aid-post health workers provide basic primary health care and provide information on healthy lifestyle practices to prevent illness⁶.

A health centre and its staff serve a population of between 2000 and 20 000⁶. Each health centre provides curative and preventive health services and acts as a referral centre for between 3 and 5 aid posts (Fig1)⁶. District health centres and hospitals coordinate health services within the district and act as a referral point for acute conditions for their sub-district level health centres⁶.

Figure 1: Health system in Papua New Guinea.

The provincial health office coordinates health services within the province, while the National Department of Health provides an advisory role to the provinces⁶. Staff within the National Department of Health have overall responsibility for maintenance of policy and standards, providing technical advice, coordination of the health information system, health planning and data systems⁶.

The PNG Department of Health monitors the performance of the health system using a computerised national health information system. Health information systems are considered vital for assessing population health needs, managing health service delivery and evaluating program coverage and effectiveness^7,8. The computerised health information system was established in PNG in 1989⁹. By 2002 the system was centrally managed and regarded as providing quality data for monitoring and planning, with links across all health system levels⁹. The system uses a well-defined set of performance indicators for monitoring performance of health facility, district, province and nationally⁶.

Each aid post forwards their monthly statistics to the supervisory health centre¹⁰. Information gathered monthly from each health centre is sent via the district health office to the provincial health office where it is entered into a computerised database¹¹. The system calculates percentages against census population data for immediate analysis¹¹. The data is then sent to national level where it is transferred into the national computerised health information system¹¹. Research in 2006 confirmed this is happening routinely¹⁰.

This article draws on the recent evaluation of a nation-wide donor project community development initiative implemented over 7 years (1998-2004). The initiative sought to increase community support for improved health of women and children through attitude and behavioural changes. Activities focused on strengthening and expanding existing village health volunteer programs and engaging community leaders in health development. Details of the quality and extent of the data collected and findings can be found in recently published articles¹². The study found there was an absence of village-level health data. The health information system does not allow reporting of villages separately or the tracking of changes in health status over time according to identifiable villages¹². Because policy dictates exclusion of village data from the national health information system, qualitative data could not confirm village claims of improved health¹².

National policy dictates that aid-post data is not included in health centre statistics⁶. This means aid-post data is not forwarded on from health centre to national level, thus there is a failure to show true and accurate health status. This policy is not always adhered to because some health centres, within the same province or district, include aid-post data in their monthly statistics. Findings from the evaluation indicated this was so in numerous settings¹⁰. The reason given is that aid-post data is considered inaccurate and inappropriate for inclusion in the national system. Aid-post health workers are considered not to have the skills required to competently diagnose illness. However, the ageing population of aid-post orderlies (those trained prior to 1987), is gradually being replaced with a new cadre of better educated community health workers. Community health workers, who graduated after 1988, are better qualified and routinely diagnose, treat and report on life-threatening illness.

The purpose of this article was to highlight problems in evaluating community interventions or local service performance if reliable village-level data is absent. This data could otherwise be used to monitor health status, performance of health workers or the impact of interventions designed to improve health.

Methods

The 2006 outcome evaluation of a Donor Aid Project community health intervention throughout PNG used both qualitative and quantitative methods. The collection of qualitative data, its synthesis and analysis, used the rapid assessment, response and evaluation (RARE) process¹³. Qualitative data were obtained through 175 informal interviews with health personnel at national (n = 18), provincial (n = 34) and district health facilities (n = 58), as well as from 77 community focus group discussions. Data were collected from 10 provinces (50%) and 19 districts (21%) in PNG in communities that were recipients of the donor project input¹⁰. Communities and health facilities were selected based on the likely variability of uptake of the donor project community intervention. Quantitative data were obtained from a review of national census population figures for communities and selected health-centre statistics over a 7 year period (1999 to 2005, inclusive). Analysis of community use of health services included antenatal, childbirth and immunization indicators, as well as the environmentally induced illnesses of malaria, pneumonia and diarrhoea.

Results

The qualitative data showed improved physical health had enhanced social and economic wellbeing as a result of the donor-aid project¹⁰. The changes initiated led to a healthier lifestyle and an improved hygienic living environment with improved behaviour in maternal and child health. Communities claimed to be positively influenced by these changes and there were obvious physical differences between villages that received successful intervention and those that did not. Local ownership of changes and expressed pride in community achievements helped sustain changes over time. Communities that partnered with the local health system increased their use of health services and were helped to reduce maternal and child morbidity and mortality.

In areas with no successful project outcome or interventions, it was observed by the researcher that the workload of the aid-post worker was excessive. These health workers spent as much as 10 hours per day providing curative treatments¹⁰. In areas where behavioural and environmental health changes had been made following the project, decreased illness meant health workers spent only one or 2 hours per day providing curative treatments. This freed the health worker's time for health education and to support community health initiatives, which were observed in action. Numerical data on key maternal health, child health and illness trend indicators over 7 years (1999-2005), however, provided little evidence of improvement in the health of women and children nationally. The relatively small increases in aggregated selected maternal and child health indicators were insufficient to claim a statistically significant impact on improved maternal and child health. Where aggregated health-centre data showed significant increase in maternal or child health indicators or decrease in illness trend in one year, this did not appear to have been maintained in succeeding years. Some communities reported having active village-level health volunteers educating pregnant women on safe childbirth practices, and these were apparently making a difference in qualitative measures or observation. However, the statistical impact was diminished by the large proportion of communities without health volunteers when these village statistics were aggregated at the health centre.

Anomalies in Papua New Guinea health information system

Qualitative data indicated information supplied by the national health information system had poor interpretive value in terms of monitoring key maternal and child health indicators at village level. Field work suggested problems with the quality of data overall. Direct observations suggested under-reporting, over-reporting, and errors with calculation and transcription. Frequent discrepancies were observed when provincial and national level records were compared with health-centre records¹⁰.

Calculations frequently produced percentages over 100% and sometimes as high as 200% or 300%, especially for antenatal visits and supervised births. Data validity and reliability is questionable when three National Department of Health documents reporting five-year performances did not match for any year¹⁰. The authors were told the variability in these documents was due to the practice of correcting data errors when identified, even if this was from years already reported.

Human error in transcribing data from daily tally sheets to monthly health information forms occurred frequently at health centre level, with calculation errors also being common. At provincial level there was risk of human error occurring during computerised data entry. This risk was reduced considerably because the computerised form mimics the hard copy of the health information system forms. Errors at national level were less likely to occur because data is electronically transferred from the province.

The health information system records the total number of births; for example all known births occurring within the health centre catchment area². Confusion in interpretation arises over Department of Health definition of 'supervised birth'. A supervised birth is classified as a birth taking place under the supervision of a qualified health worker at a health centre or hospital². This excludes village births assisted by a qualified health worker or village-level health volunteer. Observation and interviews from this study indicated health workers interpret this policy differently. Therefore, supervised birth data may or may not include village-level births supervised by health volunteers.

Village-level deaths are not required to be included in health-centre statistics. However, there may be over-reporting when a woman dies en-route from an aid post or health centre to hospital because both the health centre and hospital may report the death. This appeared to have occurred in at least one province visited during this study. Confusion can easily occur when filling in the monthly health information reports. The national health information system form is supplied to health facilities by the PNG Department of Health. The form clearly instructs to 'include ... transferred patients in total' (p1). Further confusion occurs when completing the section on page 3 that instructs: '...report all maternal deaths here, even if they occur in the village...'. The inclusion of provincial hospital data with rural health-centre data impacts significantly on maternal mortality rates and reflects poorly, and possibly inaccurately, on district performance¹⁰.

Absence of village-level data

The reliability of data collected at health centre level was questionable. The almost random inclusion or exclusion of aid-post statistics with health centre figures made it impossible to compare health centres over time, let alone villages. This research showed the situation varied between health centres, districts and provinces. In one district, government health-centre statistics included aid-post data while church health centres did not, despite being the supervisory health centre for those aid posts¹⁰. In another province, health-centre data may or may not include services provided by village-level health volunteers. In yet another province, data concerning curative, maternal and child health services provided by village-level health volunteers as part of a large non-government program is regularly included with individual district totals at provincial level.

Health centre staff may or may not utilise aid-post data to monitor health services and illness trends locally and to assess community level health workers' performance. If not used in this way, aid-post data is superfluous because it is not included in the national health information system.

Aid-post and health-centre data does not differentiate by village. Therefore, there is no direct or precise way to measure the impact on physical health in communities that have taken steps to prevent illness occurring. Nor can communities needing assistance because of a disease outbreak be identified. Aid-post data from identifiable villages is not tracked over time or reported separately. Therefore, without village-level baseline data the system cannot monitor change among villages or aid posts, evaluate community health activities or review performance. Illness trends cannot be monitored among villages nor can improvements in village health be determined¹⁰. Similarly, data collected at health centre level cannot be disaggregated by aid posts, so it is not possible to compare performance over time. Any evaluation data from community interventions or trend data from villages is diminished in value if aggregated totals include the large proportion of communities within the health centre's catchment area, which could mean combining 15 to 30 or more villages.

Discussion

This study suggests that national statistical data in PNG has little interpretive value in terms of effectively monitoring health status indicators at village level and may be unreliable. There is a lack of consistency across all levels of the health system. Inaccurate population denominators were being used to calculate health statistics, and this showed discrepancies across all levels. Problems of under- and over-reporting, inaccurate calculations, transcription errors and discrepancies in recorded data occur regularly. The random inclusion of aid-post data in health centre reports renders health-centre statistics unreliable. Valuable aid-post data is lost to the system although routinely collected. Although the national data is crude, it does provide a view of the health of the country as an aggregated whole. However, this cannot be taken to local level, guide action or evaluate interventions.

It is paradoxical that community health workers are considered competent to diagnose and treat when stationed at a health centre, but incompetent to reliably record data when working at aid-post level. Similarly, community health workers are considered competent when reporting MCH service delivery at static health centre clinics, yet are deemed incapable of reporting these same services at aid-post level.

The absence of data reporting the village-level use of health services means there is no baseline for assessing or monitoring changes in health status. Nor, indeed, for measuring disease trends or performance or workload of staff at this level despite clinical data being available. Aid-post and health-centre data needs to be disaggregated by village to enable comparison.

If aid-post data over the past decade had been separately identified in the statistical collection at health centre level, it appears likely quantitative data in this research would have shown changes in the health status of communities that received the intervention. In areas where one or two villages have taken positive steps to prevent illness, there is no numerical way to monitor the impact on this community's health. Similarly, if vector borne disease becomes evident in a district, its precise location or individuals at risk cannot be targeted for immediate assistance. Similarly, health centre staff need to be able to disaggregate data by villages if they are to identify which villages are improving and/or needing assistance where problems have arisen. One health centre in 2005 applied a system of rewarding and acknowledging communities that took the initiative to improve the health of their population. The health centre kept village-level data and, at year-end, health workers announced to the 'healthiest' community to all communities¹⁰. Fawkes and Lin emphasise the importance of observing changes in the community over time¹⁴. Local collections of health information already exist in clinical log books, both in community aid posts and in peripheral health centres but identifying markers are removed when this is entered into the computerised database. This has resulted in a serious information shortfall that can result in inaccurate and unreliable data when monitoring health status and planning at a local level.

Recommendations

The following changes are recommended:

• Change national policy to include aid-post data in health-centre statistics.
  
• Differentiate data by village at aid-post and health centre level. Aid-post and health-centre data already collected could record village identity next to client diagnosis/ service provided. This would allow data to be disaggregated by village and for communities to be compared within the aid-post and/or health centre catchment area.
  
• Collect selected data relating to village-level health volunteer activities at aid-post and health centre level. Compare these across villages over time to monitor health volunteer performance and community response to their input.
  
• Train health centre staff to scrutinise aid-post data and monitor health services and illness trends and assess community level health worker performance.
  
• Baseline data should be collected prior to and after interventions to measure their effect.

Problems evaluating community interventions or local service performance in PNG occur because there is an absence of reliable village-level data. The data collected through the health information system is crude and needs improvement for accuracy as well as monitoring and planning at local level. While aid-post data is available it is not always included in the health information system nor is it disaggregated according to village to show changes in health status. This has resulted in a serious information shortfall that can result in difficulties assessing changes in physical health status without village-level baseline data to measure illness trends and improvements in health in identifiable villages.

Qualitative data from a recent extensive evaluation showed community health interventions were effective in improving health behaviour and the health of women and children. However, statistical verification of improvement in health was not possible using the health information system and consistency in data collection. Health-centre data failed to provide a complete and accurate assessment of community health status because it does not disaggregate by village.

Policy changes must occur centrally to prevent the loss of aid-post data to the system. Aid-post and health-centre data needs to be disaggregated according to village to allow for changes in community health status to be monitored over time. Future planning for community health intervention strategies should take account of the need for baseline data against which to measure change over time.

Acknowledgements

The Australian Agency for International Development (AusAID) is acknowledged for funding the Women and Children's Health Project and field work during 2006 in Papua New Guinea. Also acknowledged is the Charles Darwin University for providing a scholarship for this study.

References

1. PNG Ministry of Health. Program policies and strategies: Volume II. National health plan: 2001-2010. Port Moresby: Papua New Guinea Department of Health, 2000.

2. PNG Department of Health. Annual health sector review preliminary national report: 2005 performance review. Port Moresby: Papua New Guinea Department of Health, 2006.

3. PNG Ministry of Health. Minimum standards for rural health services in Papua New Guinea. Port Moresby: Papua New Guinea Department of Health, 2000.

4. PNG Department of Health. A guide to the health sector improvement program. Port Moresby: Papua New Guinea Department of Health, 2004.

5. Campos-Outcalt D. Health services in Papua New Guinea. Public Health 1989; 103: 161-169.

6. PNG Ministry of Health. A guide to implementing the plan (Volume IV). National health plan 2001-2010. Port Moresby: Papua New Guinea Department of Health, 2000.

7. Finau SA. National health information systems in the Pacific Islands: in search of a future. Health Policy and Planning 1994; 9(2): 161-170.

8. Azubuike MC, Ehiri JE. Health information systems in developing countries: Benefits, problems, and prospects. Journal of the Royal Society for the Promotion of Health 1999; 119(3): 180-184.

9. Cibulskis R, Hiawalyer G. Information systems for health sector monitoring in Papua New Guinea. Bulletin of the World Health Organization 2002; 80(9): 752-758.

10. Ashwell HES. Evaluation of a program to improve interaction between community and health systems in Papua New Guinea: Its impact on maternal and child health (PhD thesis). Charles Darwin University, 2008.

11. Newbrander WC, Thomason JA. Computerizing a national health system in Papua New Guinea. Health Policy and Planning 1988; 3(3): 255-259.

12. Ashwell HE, Barclay L, Outcome evaluation of community health promotion intervention within a donor funded project climate in Papua New Guinea. Rural and Remote Health 9: 1219. (Online) 2009. Available: http://rrh.org.au (Accessed 29 November 2010).

13. Trotter II RT, Needle RH, Goosby E, Bates C, Singer M. A methodological model for rapid assessment, response, and evaluation: The RARE program in public health. Field methods 2001; 13(2): 137-159.

14. Fawkes S, Lin V (Eds). Effectiveness of health promotion in changing environment and lifestyles in developing countries of the Western Pacific Region: a review and a proposed framework. In: WHO Western Pacific Regional Office Workshop on Health Promotion Capacity Building; November 2003; Manilla. World Health Organisation Western Pacific Regional Office, 2003.