An evaluation study on the relevance and effectiveness of training activities in Northern Uganda

full article:

Introduction

Ignorance of all things is an evil neither terrible nor excessive, nor yet the greatest of all; but great cleverness and much learning, if they be accompanied by bad training, are a much greater misfortune. (Plato)

Adequate in-service training is necessary for quality health service delivery in a primary health care setting¹. Policy changes, new clinical guidelines and general management tasks all require effective training of the highest number of health workers possible in order to reach program objectives. Professional development can also influence health workers' motivation and practices². However, training is generally expensive, time-consuming and energy-absorbing and can divert staff from their routine duties, especially when training is repeated. In addition, training may not produce an expected 'cascade effect' of benefits to other health workers, with skills and capacity developed in one individual not transferring to others.

A detailed evaluation of training effectiveness is seldom performed. Some trainers simply rely on the first two elements of Kirkpatrick's evaluation framework³: assessing the improved theoretical knowledge of trainees during and immediately after training. A systematic follow up of trainees after training is rare; however, this could assist in modifying the curriculum, reviewing outcomes and guiding future training development⁴. It is often assumed that the performance of those trained will improve after the activity, although the improvement may not last for long⁵.

Human resource constraints are particularly acute in post-conflict Northern Uganda where insecurity, difficult working and living conditions and limited incentives all contribute to poor retention of health workers and staffing shortages⁶. In a context of scarcity of manpower and a recommended 'task-shifting' approach (the delegation of healthcare tasks to less specialized health workers)⁷, one of the most serious challenges to training effectiveness is that these 'multi-purpose' health workers are usually assigned several competing duties. This may limit both the newly acquired skills and the benefits of the training activity.

This study looked at the results of training activities, with a particular focus on the retention of specifically-trained health workers in their original facility 12-15 months after the training event; trainees' use of acquired skills; and the training's relevance to and usefulness in trainees' routine duties and overall performance, from the perspective of the health workers. The assessment was conducted on training activities organized by the Northern Uganda Malaria, AIDS & HIV and TB Program (NUMAT), a five-year USAID-funded project, the goal of which is to increase access to and utilization of quality HIV, TB and malaria services in Northern Uganda.

Methods

The attendance lists of training events organized and funded by NUMAT in the period July 2007 to February 2008 were retrieved and a master list compiled, indicating the date of training, the name, cadre, designation and workplace of all participants, as well as their telephone contact if available. In the case of missing or contradictory records, additional information was sought from fellow health workers or district health offices. No program training had taken place prior to July 2007.

Face-to-face interviews in English were conducted by trained data collectors with trained health workers, mainly paramedics, using a pre-tested questionnaire. The interviewees were selected after a purposive sampling procedure that fulfilled one or more of the following criteria:

• the time elapsed since attending the training course was less than 12 months
  
• the aim of the training was to provided specific skills for service delivery, in particular to strengthening or increasing related health services (including the creation of district trainers);
  
• the duration of the training (at least one week).

Five training activities were chosen: three related to clinical management of HIV-related conditions; one covered the logistic management of antiretroviral treatment supplies; and one aimed at creating district trainers for home-based management of malaria.

Every effort was made to inform health workers of interview times and locations, which were preferably at the health facility where they currently worked. A telephone interview was conducted with respondents who had been transferred to locations that were difficult to reach.

The questionnaire investigated the following areas: basic demographic characteristics; duration of service in the district health system; other training events attended during the period of study; workplace at the time of training; current workplace; main duties in the current work place; subjective perception of the usefulness of the training for the tasks currently performed; availability of training materials; and exposure to any coaching or mentoring after training.

Two separate questionnaires were completed for the few health workers who participated in two training events, and these were treated as two separate training participants in the analysis, rather than a single individual. Quantitative analysis was performed to measure the rate of participants' retention at their original workplaces; the proportion of health workers currently using the skills acquired; the degree of perceived training relevance; the proportion of those who received any follow up after the training, among other variables. Data were entered in EpiData (The EpiData Association; Odense, Denmark) and analyzed using MS Excel software.

Participation was voluntary. District health officers of districts where the study was conducted were informed about the activity; however, identifiers were removed from the data to protect participants' anonymity.

Results

In total, 115 health workers from 9 districts who participated in 5 training events were identified. It was possible to establish current work place and measure retention rate for 112. However, only 104 participants (92%) were interviewed (Fig1).



Figure 1: The recruitment of participants.

Face-to-face interviews were possible in 63 cases (61%); the remaining 41 were interviewed by telephone.

The respondents' characteristics are summarized (Table 1). Of the 112 participants whose whereabouts could be established, 79 (71%) were still deployed at the original work site and 8 (7%) of these were currently undergoing long-term training (Table 2). Thirty-three participants (29%) had left the facility: 25 had moved to another unit in the same district and 8 had left the district.

When asked about the training they had attended (Table 2), 78 of 104 respondents reported they were using skills acquired during the training, although in some cases this was only partial. The question did not apply for those undergoing further training and one participant did not provide an answer.

Regarding the perceived usefulness of the training content in relation to the health workers' main tasks, 91 (87%) of the 104 respondents reported they found the event useful to some extent (Fig2); only 6 (6%) reported that the training had not been useful at all.

Table 1: Respondents' characteristics




Table 2: Respondents' reported training characteristics






Figure 2: Usefulness of the training attended, as perceived by respondents.


Few of the participants (11%) had received a detailed program prior to the training attended; however, 100 (97%) were provided hand-outs in the form of training notes, participants' manual or implementation guidelines. It is notable that 15 of the 100 did not share the hand-outs with colleagues, mainly due to busy working schedules and a lack of time.

Follow up in the form of support supervision, coaching or mentoring was provided for only 40% of respondents, mostly by Ministry of Health Officials (MOH) or NUMAT staff. This occurred erratically and infrequently, and for over half this was quarterly or less often.



Figure 3: Suggestions to improve training organization.

When asked for suggestions to improve the organization of courses of similar content (Fig3), 25% of the 99 respondents who provided an answer said that follow-up of trainees in their respective work places would be very useful; 22% mentioned conducting longer courses; 21% wanted more training and more people trained; 15% suggested improvement in the participants' 'welfare', suggesting that out-of-pocket allowances were unfairly paid to trainees at different rates.

A quarter of those interviewed (26 of 104) reported having attended another training event on the same topic at some time, and in most cases (19 of 26) this was organized by the MOH or district.

Discussion

Retention

In measuring the retention rate of trainees, it was discovered that after a period of between 12 and 16 months, a substantial number of health workers were still deployed at their original work site. However, those who left to pursue further courses should not be ignored. Although training participants remained attached to their original health units, being away on training is intended to improve their skills. Therefore, after training, newly acquired knowledge may increase participants' eligibility for higher positions. Those who leave a facility shortly after training create an unplanned demand for further training as they are replaced. For practical purposes the number of staff in training cannot exceed a certain number. Scaling up training for specific skills to all facilities would require more time than was initially expected, quickly consuming the scarce resources available.

However, district transfer policies should accommodate the particular skills required according to health worker duties. In this study, the proportion of transferred trainees who remained within the same district was notable; however, in some cases the health worker was redeployed to a facility not equipped to provide the specialized service for which he or she was trained. This was found mainly among clinical personnel promoted to an administrative role in other units. A simple database at district level that specifies staff training would assist in the logical deployment of staff among programs. While this information sometimes exists it is often fragmented among district officials. A more centralized, regularly updated record system under the supervision of the district health officer would be an improvement.

Skills utilization

The use of skills gained during training was compared with reported the tasks and assignments of the health workers. After excluding those undergoing further study, 18% reported not using their newly acquired skills after 12-16 months. This was commonly associated with transfer to another department, ward or facility and was common to all types of training. This situation could be reversed with the consistent use of a transfer scheme that takes into account health workers' ability to replace each other, according to their training and specialized task ability.

It is remarkable that the majority of nursing assistants reported using skills to perform clinical duties that should be outside their role after receiving highly technical training. This is a result of the current practice of gradually shifting tasks to lower cadres in order to address shortages in qualified staff, especially in peripheral health units.

With few exceptions, respondents generally felt their training was useful, irrespective of whether they were using the skills acquired. Exposure to in-service training was perceived as beneficial in terms of broadening knowledge and learning new techniques. Those who rated their training 'not useful' reported that a lack of practice opportunities had rendered the training futile, and assumed that they would have to be retrained in those skills at some future time.

The majority of health workers who did not consider the training attended to be useful had been transferred to a lower level unit that provided a more limited range of services. Others were unable to commence practising new skills due to system issues, particularly disruption to the supply of consumables necessary for the implementation of a new policy, such as the introduction of anti-malarial medicines for home based management of fever (HBMF). The sequential timing of the stages of a new intervention (eg a new clinical approach) can be difficult as it may include policy dissemination, extensive staff training and orientation, public awareness raising, and the provision of all items necessary for the intervention's execution. Training commonly raises expectations among trainees. A delay in implementing the subject of the training is likely to diminish trainees' interest and frustrate their efforts.

Training usefulness and effectiveness

Although the majority of interviewed health workers considered their training useful, some also suggested more training should be offered, and for longer periods. The latter suggestions may have been motivated by the financial advantage (in terms of allowances) associated with training participation.

While the objective of this study was not to conduct an economic analysis, when assessing the usefulness of training, the ultimate question is whether it provides value for the money. Affordable training can be widely offered in order to reach as many health workers as possible. Even if training is organized externally (eg by the MOH or a specific project), the issue of cost remains valid in terms of what else could have been achieve with the same money, time and other resources.

The definition of 'effective training' is debatable. However, this study's findings question the 'effectiveness' of the current training's medium and long term outcomes. Is training effective when it leads to trainees moving to another workplace, even if they maintain and use the acquired skills? The same question applies if trainees are promoted to higher responsibility in the same workplace, and potentially removed from the tasks for which they have been trained. Both situations occurred among participants and would be common experience in the health services. This suggests that defining an acceptable level of retention among trainees should be included in the objectives and expectations of training.

Training organization and follow up

Most participants suggested that receiving complete and prior information about the training event would be an improvement. Information about the training program was rarely communicated to participants prior to the event, with most attending for training with little advance knowledge of the content. This lead to some participants attending training for skills already developed.

The provision of training hand-outs was more consistent. However, sharing the recently acquired knowledge with fellow health workers is not a yet practiced widely. In some cases, briefing other staff is institutionalized in continuing medical education sessions where training materials are made available for all. More often, health workers retain training material for personal use, even taking it with them when transferred to another workplance. However, this study was not able to assess the physical presence of training materials within a health facility, nor was it possible to measure the adequacy and practice usefulness of hand-outs.

Assessing that training is being correctly applied in the workplace once trainees resume duty requires a worksite visit. Regular follow up is instrumental in both identifying gaps in the training and offering technical guidance to trainees as acquired skills are consolidated. This factor was identified as lacking by a number of respondents. Only 40% reported receiving follow up, with half of these reporting it as a rare occurrence. Instituting a steady follow-up process from all training events is essential to increasing staff confidence and morale, and informing trainers of potential weaknesses in the curriculum.

In order to sustain close and regular follow-up long-term, technical supervision and coaching could be provided by district-based supervisors, supported by central facilitators from MOH. This would require district- or regional-level capacity building so that regional trainers and district staff could always be involved in training programs, even when conducted by MOH officials.

Coordination among training organizers must be improved to maximize the financial and human resource investment in training. For example, inviting health workers who have already participated is a missed opportunity to expand knowledge and skills to others. A number of factors contribute to this issue. Donor programs often operate autonomously; MOH departments organize centralized training that is not tailored to district requirements. A solution to the duplication issue may be to involve workplace supervisors in the selection of training participants. District health officers should have an overview of all planned training in their district. The inventory or database of staff training already suggested would serve as a starting point to assess individual and workplace training needs. This could be shared among agencies, including non-government organizations training providers.

Limitations of the study

The limitations of this study include the methodology. Because the sampling was purposive the findings may not generalise to the whole population of health workers in training. A substantial proportion of the respondents were interviewed by telephone, rather than face-to-face. While no sensitive questions were asked, the differences in interview settings may have affected some responses. Additionally, some health workers could not be found or contacted. The time elapsed since training could have introduced recall bias to questionnaire responses; some participants said they could not remember very well or declined to answer to some questions. Finally, although this study attempted to assess whether skills were being used by trainees, it did not examine how they were being utilized which would have required direct observation of health workers' practices. Without the critical component of direct observation, it is difficult to evaluate long-term training effectiveness and usefulness.

Recommendations

Regular in-service health worker training that aims to improve the quality of services must reach as many providers as possible. Evaluating the immediate output of training activities is not a reliable indicator of training value. This study, although limited to a few training courses, has highlighted some issues that need to be addressed.

Retention and follow-up of trained staff: To maintain a relatively high retention rate of health workers at their original worksites, all those providing training must collaborate closely with district authorities. By providing information centrally they can assist in establishing a district inventory or database of staff trained and training events. However, this does pose the problem of collecting a large amount of data retrospectively. An embedded, functional system of trainee follow up is essential. Staff training must include addressing difficulties encountered when implementation occurs at the workplace; and the usefulness of the training program can be greatly enhanced by acting on the constructive feedback of past trainees.

Coordination of training: District health system coordination of all training providers is the only solution to duplicated activities and will maximise the effective use of available resources. Before any training is undertaken - especially when it is extensive - all agencies should be required to share the training plan and objectives. In this way training organizations will have adequate information about similar efforts being undertaken by others, and can assess the actual local training needs realistically.

Advocating for the real principles of training: There is need for a campaign aimed at restoring the real principles behind training. Trainees' subjective impressions of training are almost always positive, advocating for more training and more participants. This may be regarded as a 'bias' towards training due to the financial benefits from associated allowances. The value of acquired knowledge is thus diminished. When focused on personal benefits from training activities, staff underestimate the impact of prolonged absence from workplace, where the delivery of health services is the ultimate purpose of training. Consideration should always be given to the number of trainees absent from a health facility for any one program. This could include the establishment of a systematic workplace method of sharing knowledge and skills among fellow health workers to limit the disrupting effects of staff absence for training.

Expanding evaluation to other trainings: This study was limited to a review of training events for health workers. However, training in the HIV-AIDS field often targets non-medical personnel who are trained and recruited for community-based activities. Assessing the retention and functionality (and other relevant factors) of lay workers is crucial. They often fill a gap in service delivery left by established health staff, and their retention impacts on sustainability when a program concludes.

Conclusions

Successful in-service training broadens the skills and competencies of health workers, ultimately improving the quality of health services. Adequate evaluation of a training event provides invaluable information to the organizers and offers accountability to those who funded it. Customary appraisal of trainees (eg the pre-test/post-test approach or an evaluation form on completion) are insufficient. The value of training is best assessed at some time after the event. Among useful and easily assessed evaluation end-points are staff retention at their original worksites and their current use of acquired skills.

All programs with a substantial training component should commit to periodic, methodical evaluation. Failure to do so may undermine the value of training by perpetuating common training issues that require correction. Simple and effective training evaluation can only enhance the work of the program or organization.

References

1. Guiscafré H, Gutiérrez G, Verver H, Palafox M, Lopez A, Martinez H. Quality improvement of integrated child health care management after in-service training for physicians. Family Practice 2003; 20: 74-76.

2. Rowe AK, Savigny D, Lanata CF, Victora CG. How can we achieve and maintain high-quality performance of health workers in low-resource settings? Lancet2005; 366: 1026-35

3. Kirkpatrick DL. Evaluating training programs: the four levels, 2nd edition. San Francisco, CA: Berrett-Koehler, 1998.

4. Maye O, Gerein N, Tarin E, Butcher C, Pearson S, Heidari G. Training evaluation: a case study of training Iranian health managers. Human Resources for Health 2009: 7: 20.

5. Abbatt FR, Mejia A. Continuing the education of health workers. A workshop manual. Geneva: World Health Orgnization, 1988.

6. Youngs D. Human resources for health programs for countries in conflict and post-conflict situations. Chapel Hill: IntraHealth International, 2006. Available: http://www.capacityproject.org/images/stories/files/post_conflict_resource_paper.pdf (Accessed 16 November 2009).

7. Anon. Task shifting may prove key to tackling infectious diseases. Lancet 2008; 8: 81. (Editorial)