Evaluation of a training program for device operators in the Australian Government's Point of Care Testing in General Practice Trial: issues and implications for rural and remote practices

full article:

Introduction

Point-of-care testing (PoCT) is defined as a pathology test performed on-site by or on behalf the treating doctor at the time of patient consultation, allowing the test result to be used to make an immediate decision about patient treatment¹. Technological advances in the manufacture and design of PoCT devices has resulted in the global uptake of PoCT, currently increasing at a rate of 12% per annum². The scope and application of PoCT is now expanding rapidly from the traditional hospital base to community-based primary care settings where the care of patients, particularly with chronic diseases, is now often focussed¹.

In Australia, PoCT for diabetes management is now firmly established in the Indigenous health sector through the national Quality Assurance for Aboriginal and Torres Strait Islander Medical Services (QAAMS) Program and it has proven robust, safe and clinically and culturally effective^3-6. There is growing interest in PoCT in the Australian general practice setting and a review of the role and value of PoCT in this setting highlighted that rural and remote practices could be the main beneficiaries of PoCT⁷. However the review concluded that further work was needed to determine the clinical and economic benefits of PoCT in general practice. As a result, the Australian Government recently funded the PoCT in General Practice Trial, one of the largest and most comprehensive studies of PoCT ever conducted in this primary care setting⁸.

The PoCT in General Practice Trial was a multi-centre, cluster randomised controlled trial to determine the safety, clinical effectiveness, cost-effectiveness and satisfaction of PoCT in general practice. Three lead organisations were contracted by the Australian Government to deliver the Trial, working collaboratively from an Adelaide base. They were the Discipline of General Practice from the University of Adelaide (who were responsible for overall Trial management and evaluation), the Community Point-of-Care Services (CPS) unit at Flinders University and the RCPA Quality Assurance Programs (QAP) Pty Ltd.

The Trial ran for 18 months in the period 2005-2007. In total, 53 practices based in urban, rural or remote locations across three states (South Australia, New South Wales and Victoria) participated in Trial. In all, 23 practices were randomised to the control group, which had pathology testing performed by their local laboratory; and 30 practices were randomised to the intervention group, which conducted pathology testing by PoCT. A total of 4968 patients (1958 control and 3010 intervention) participated in the Trial, all of whom had either diabetes, hyperlipidaemia and/or were taking anticoagulation therapy (warfarin). Full details on the Trial methodology, rationale, recruitment and baseline patient characteristics have been reported elsewhere⁸.

The following point-of-care (PoC) tests were performed by intervention practices: haemoglobin A1c (HbA1c) and urine albumin:creatinine ratio (ACR) on patients with diabetes; total cholesterol, triglyceride and high density lipoprotein (HDL) cholesterol on patients with hyperlipidaemia; and international normalised ratio (INR) on patients on warfarin therapy. Three PoCT devices were used to measure these tests; the DCA 2000 (Siemens HealthCare Diagnostics; Melbourne, VIC, Australia [formerly Bayer Australia]) for HbA1c and urine ACR; the Cholestech LDX analyser (Point of Care Diagnostics; Sydney, NSW, Australia) for total and HDL cholesterol and triglyceride; and the CoaguChek S (Roche Diagnostics; Sydney, NSW, Australia) for INR.

Point-of-care testing at the intervention practices was underpinned by a quality management framework consisting of a training and competency program for device operators and the routine conduct of internal quality control and external quality assurance testing procedures (standard laboratory practices for monitoring analytical quality that were adapted for use in a general practice setting). Training for device operators is a crucial component of PoCT because device operators should have a sound knowledge and understanding of basic analytical concepts and the technical skill set required to perform PoCT to an analytical standard that is equivalent to a laboratory and safe for patient care. Point-of-care testing training and competency and the quality control program for the Trial were delivered by the PoCT Device Group, a consortium comprising scientists from the Flinders CPS unit working with industry partners from Bayer Australia, Point of Care Diagnostics and Roche Diagnostics. The external quality assurance program was delivered by the RCPA QAP group.

This article describes the design and implementation of the training and competency program for device operators from intervention practices and assesses the effectiveness of the training program as judged by these operators.

Methods

Ethics approval

The PoCT in General Practice Trial was approved by five relevant independent Australian Human Research Ethics Committees. The Trial is registered with the Australian Clinical Trial Registry, Number 12612605000272695.

Design of point-of-care testing training resource package

An education and training resource package was developed for the Trial consisting of a training manual, a set of A3 laminated posters and a CD ROM. This resource package was written by the PoCT Device Manager (MS), with input from scientists from the PoCT Device Group and the RCPA QAP.

The 162 page colour training manual contained both an introduction section and a test-specific section. The introduction covered the theory of PoCT and discussed the importance of quality management of PoCT devices, in particular the principles behind internal quality control and external quality assurance testing.

The test-specific section covered the practical side of performing each of the PoC tests measured in the trial (HbA1c, urine ACR, total cholesterol, triglyceride, HDL cholesterol and INR), and described each test systematically under the following common headings:

• the clinical use of the PoC test
  
• a description of the PoCT device used to measure the test
  
• a description of the PoCT method
  
• how to perform the PoC test on a patient sample
  
• how to test a quality control sample
  
• how to test a quality assurance sample
  
• how to perform basic maintenance procedures on the PoCT device.

The manual also contained an appendix which provided information on topics including:

• selected papers on the clinical use of each of the PoC tests
  
• selected papers on how the PoCT method compared with the laboratory method for each test
  
• specifications of the PoCT devices used in the Trial
  
• common error messages for each PoCT device.

A spiral bound set of 12 laminated A3 posters was also prepared for the Trial. The posters were designed to provide device operators with a user-friendly, step-by-step guide on how to perform a patient test, a quality control test and a quality assurance test for each PoC test. An example of the posters is shown (Fig1). The poster concept had previously proven very popular among Aboriginal health professionals trained as device operators for the QAAMS PoCT program for diabetes management in Aboriginal medical services³.

The CD ROM contained an electronic copy of the training manual and poster set and associated spreadsheets.



Figure 1: Example of a poster showing how to conduct point-of-care testing for INR on a patient sample (with permission of the Australian Government Department of Health and Ageing).


Point-of-care testing training workshops

Five Initial Training Workshops were held for intervention practices from each geographic region between August and October 2005 at three centres - Adelaide, South Australia (to cater mainly for urban practices), Bendigo, Victoria (for rural practices) and Dubbo, New South Wales (for remote practices).

These workshops were each of two-day's duration. They combined theoretical training in the principles and practice of PoCT, the tests and devices used and quality management procedures with 'hands on' practical training delivered in interactive small group sessions, with one scientist from the PoCT Device and/or RCPA QAP groups working with a maximum of three general practice staff. The training workshops also included sessions on PoCT accreditation and Trial protocol implementation provided by the Trial management and evaluation team.

At the completion of training, practice staff were required to correctly answer a set of written competency questions related to each test and to perform a PoC test in the presence of a scientist. The written questions related to the clinical utility of tests and issues of technical or procedural nature covered in the theory section of training. The PoC test performed by the operator as part of initial competency assessment was either a quality control or quality assurance sample both of which had a set target value and an acceptable limit of performance around that target. Device operators were required to obtain a value for the sample tested that was within the acceptable limits to pass their practical competency. On successful completion of this written and practical assessment a competency certificate was presented to each practice staff member.

From January to August 2006, further training sessions were held for intervention practices that required new practice staff to be trained. These were generally conducted as 'one-on-one' sessions between the new operator and a CPS scientist and were often held on-site at the practice concerned.

Five refresher training workshops covering all geographic regions were delivered during late August 2006, coinciding with the 12 month point of the live phase of the Trial, as a commitment to continuing education and training for device operators. Refresher training workshops included an overview of the Trial thus far from the PoCT device, trial management and RCPA QAP groups and a feedback session for device operators to discuss issues related to the use of the PoCT devices, quality management and trial management procedures.

Only device operators who had undergone full training and received a competency certificate were able to conduct routine PoCT during the Trial. A competency register was maintained by the PoCT device group throughout the Trial.

Assessment of point-of-care testing training methods

At the completion of each initial training workshop, practice staff completed a satisfaction survey from the PoCT device group designed to assess the general appropriateness of the methods employed and the overall effectiveness of PoCT training provided by the device group. The survey contained six questions, with respondents rating their level of satisfaction with each question according to a Likert scale⁹.

At the completion of the Trial in February 2007, device operators completed a further questionnaire devised by the Trial evaluation team (from the Trial management group) in which they were asked to indicate how strongly they agreed/disagreed with eight statements concerning their satisfaction with aspects of PoCT using a visual analogue scale (VAS). A VAS is a horizontal line, 10 cm in length, with the left end labelled as 'strongly disagree' and the right end labelled as 'strongly agree'. Participants mark on the line the point they feel represents their level of satisfaction and the distance from the left end of the line to the mark is measured. A larger value indicates a higher level of agreement with the statement. The median score (and inter-quartile range) was calculated for each statement using Statistical Program for Social Sciences v 13.0 (SPSS Inc; Chicago, IL, USA). Scores were split by geographic region to ascertain whether there was any difference in satisfaction levels with PoCT between urban, rural and remote practices.

Results

Practice staff trained as point-of-care testing device operators

At the conclusion of the initial training workshop series 60 device operators from 31 practices completed training and received competency certificates, having passed both written and practical assessments.

From January to August 2006, a number of further training requests from practices were received due to either: (i) new nursing staff needing to be trained because their previous device operator had left the practice; or (ii) the practice requested training for further nursing staff in addition to their existing operators. In total 20 staff (1 urban, 6 rural and 13 remote) from 12 practices (1 urban, 4 rural and 7 remote) underwent training and competency certification during this period.

Thus, in total, 80 device operators comprising 74 practice staff and six GPs from 31 practices were trained and received competency certificates as part of Trial. All 6 GPs who completed training as device operators were from rural (n = 2) and remote (4) locations. A summary of these practices and operators, split by geographic region, is provided (Table 1).

Table 1: Practices and device operator training during the Trial by geographic region



In total, 19 device operators (3 urban, 6 rural and 10 remote) from 11 practices (3 urban, 2 rural and 6 remote) left during the life of the Trial. Of these, 8 device operators (3 urban, 2 rural and 3 remote) from 6 practices (3 urban, 1 rural and 2 remote) resigned for personal reasons. The remaining 11 device operators (4 rural and 7 remote, including one GP) left the Trial because their practice withdrew from the Trial.

At the completion of the Trial, 61 device operators were still actively conducting PoCT at their practice. Of these active operators, 42 (69%) attended refresher training workshops held during the second half of 2006 (Table 2).

Table 2: Practices and device operators undertaking refresher training by geographic region



Satisfaction questionnaires and general feedback from point-of-care testing device operators

Initial training: In response to the question 'How would you describe the level of instruction on the theory of PoCT?' 56/57 respondents (95%) stated the level of instruction was appropriate. For the questions 'Do you feel the posters will be useful for you on a day to day basis?' and 'Was working in small groups with a supporting scientist for practical instructions satisfactory as a training method?', there was unanimous agreement among the 60 respondents that the posters were useful and the mode of practical training was satisfactory. The quality and appropriateness of the PoCT training resources was rated as either good or excellent by all 60 respondents (8 and 52, respectively). Figure 2 shows the responses of device operators to the questions 'How would you rate the support given to you during training by the scientific team?' and 'How would you rate the PoCT device training workshop overall?' The response rate to these questions was 85% and 78%, respectively.



Figure 2: Point-of-care testing device operator response to selected questions at the completion of initial training workshop series.


Refresher training: From the feedback sessions conducted as part of refresher training, PoCT operators were in general agreement that:

• the poster set provided for practices remained useful, its size and clarity was good
  
• PoCT devices had generally proven robust during the Trial
  
• patients were generally satisfied with the PoC testing process and they felt a greater sense of ownership of their pathology results.

End-of-trial satisfaction survey: The responses by device operators to the Trial evaluation team's post-Trial satisfaction questionnaire are displayed (Table 3). In total, 90% (55/61) of the active operators at the end of the Trial completed the questionnaire. Overall, a high level of satisfaction with PoCT was reported by operators across all geographic regions. Device operators from remote practices had the highest satisfaction levels for quality of training, usefulness of the training manual, ease of use of devices, confidence in the accuracy of PoCT results and preference for PoCT over laboratory testing. Rural device operators showed lowest levels of satisfaction for quality of training, competency in using the devices and confidence in the accuracy of PoCT results. Urban device operators recorded the highest levels of satisfaction for usefulness of posters and competency in using the devices, but a lower degree of satisfaction with ease of device use and preference for PoCT over laboratory testing. The usefulness of the posters for conducting PoCT achieved the highest satisfaction rating among operators from all three geographic regions.

Table 3: Point-of-care testing device operator responses to end-of-trial satisfaction questionnaire by geographic location



Discussion

A quality framework that ensures continuous training and on-going surveillance of analytical quality is a critical and fundamental element for underpinning a successful and sustainable PoCT model, particularly in a community setting^10-16. The methods for education, training and competency assessment developed by the PoCT Device Group for use in this Trial were consistent with those: (i) used successfully in PoCT programs for chronic disease management in the Aboriginal community setting in Australia, notably the QAAMS and Point-of-Care Testing in Aboriginal Hands Programs^3-6,17; and (ii) recommended for PoCT by the International Organisation for Standardization (ISO) and the Clinical and Laboratory Standards Institute (CLSI; formerly the NCCLS)^18-19.

The Initial Training Workshop series for the PoCT in General Practice Trial featured the novel use of laminated poster series for day-to-day conduct of PoCT and interactive small group sessions for practical training managed by medical scientists and supported by industry. The use of primary trainers with strong medical science backgrounds, experience in delivering PoCT training programs and having expertise in tailoring training methods to targeted audiences of different health professional groups contributed significantly to the success of the training program. Of particular importance in this regard was the ability to translate complex laboratory terms such as accuracy, precision, quality control and quality assurance into readily understandable concepts for non-laboratory trained device operators, while at the same time adhering to the requirements of international (NCCLS/ISO) guidelines for the conduct of PoCT.

Responses from device operators from all geographic regions to satisfaction surveys conducted by the PoCT Device Group (after the initial training workshop series) and by the Trial evaluation team (at the conclusion of the Trial) indicated that there was widespread acceptance of training methods and their effectiveness. The acceptance of the poster sets as a user-friendly and practical training resource was confirmed by the very high satisfaction levels reported among device operators.

Point-of-care testing has particular application for rural and especially remote health practices where access to laboratory services may be limited, turnaround time for receipt of results may be delayed and rate of patient return for follow up of laboratory results may be low²⁰. The positive responses from remote operators relating to ease of use of PoCT devices, confidence in the accuracy of results and preference for PoCT over laboratory testing augurs well for the acceptance of PoCT in this geographic sector. However the challenges faced by the rural and remote health services, particularly in relation to sustaining both workforce capacity and the ability to conduct PoCT are considerable. High rates of staff turnover are a constant problem for rural and remote services. In this study, 16 (84%) of the 19 device operators who left the study were from rural (6) or remote (10) practices; 5 (63%) of the 8 device operators who resigned were from these geographic regions. Four (80%) of the 5 practice withdrawals were from rural and remote areas. Maintaining and delivering staff training and the capacity to conduct quality-assessed PoCT can also be strained in the face of high staff turnover. It is noteworthy that 11 (92%) of the 12 practices that requested additional training in the 9-12 month period post the initial training workshop series were from rural (4) or remote (7) practices. Further 19 (95%) of the 20 new staff trained were from rural (6) or remote (13) practices. These observations highlight that not only do rural and remote practices have a greater need for training and support compared to their urban counterparts but also require more flexible training options to cater for much higher rates of staff turnover. Regular face-to-face training by a primary training team may be impractical and too expensive for a remote general practice experiencing high staff turnover. Access to electronic training resources such as DVD or web-based training through web-streamed video presentations (such as those being currently used in the QAAMS Program^5-6) may provide alternative training options for this geographic niche.

During the live phase of the Trial, on-going device operator competency was assessed by the routine conduct of quality testing. The results of this testing have been reported elsewhere and indicate that device operators conducted PoCT to a generally acceptable standard²¹.

In conclusion, the methods established for the implementation and delivery of training and competency assessment were appropriate for the PoCT in General Practice Trial. However, findings from this study have emphasised the greater need for training and support for PoCT services in rural and remote practices and the necessity for more flexible training options to address much higher rates of staff turnover.

Acknowledgements

The authors acknowledge the significant support received throughout the life of the Trial from members of the Pathology Section, Department of Health and Ageing, in particular Fifine Cahill, Debbie Stanford, Pamela McKittrick, Helen Blackall, Rob Lillie and Robert Walsh. The authors acknowledge the members of the other two lead organisations responsible for delivery of the PoCT in General Practice Trial, namely the University of Adelaide (Trial Management Group) led by Professor Justin Beilby, Briony Glastonbury and Caroline Laurence, and the RCPA Quality Assurance Programs Pty Ltd (Quality Assurance Program) led by Janice Gill and Meredith Liddy. Dr Andrew St John is thanked for his work and incisive input as Consultant to the PoCT Device Group. Also thanked and acknowledged is the outstanding efforts of industry partners (Dean Whiting, Alison Halfnights, Jan Gilbert and Jonathan Trengove from Siemens HealthCare Diagnostics; Rupert Haines and Peter Merilees from Point of Care Diagnostics Australia; and Dr George Koumantakis, Adrienne Ripley, Kevin Falzon and Agnes Godfrey-Roberts from Roche Diagnostics). Julie Gardner, Ross Forbes, Andy Dallison, Ben Flink and Deb Kelly from Flinders Consulting organised all administrative matters for the PoCT Device Group and managed finances for the Trial with great efficiency.

References

1. Price CP, St John A, Hicks JM (Eds). Point-of-care testing, 2nd edn. Washington DC: AACC Press, 2004.

2. Huckle D. Point-of-care diagnostics: will the hurdles be overcome this time? Expert Reviews of Medical Devices 2006; 3: 421-426.

3. Shephard M, Gill J. Results of an innovative education, training and quality assurance program for point-of-care HbA1c testing using the Bayer DCA 2000 in Australian Aboriginal Community Controlled Health Services. Clinical Biochemist Reviews 2003; 24: 123-130.

4. Shephard M, Gill J. An innovative Australian point-of-care model for urine albumin:creatinine ratio testing that supports diabetes management in indigenous medical services and has international application. Annals of Clinical Biochemistry 2005; 42: 208-215.

5. Shephard M. Cultural and clinical effectiveness of the 'QAAMS' point-of-care testing model for diabetes management in Australian Aboriginal Medical Services. Clinical Biochemist Reviews 2006; 27: 161-170.

6. Shephard M, Gill J. The analytical quality of point-of-care testing in the 'QAAMS' model for diabetes management in Australian Aboriginal Medical Services. Clinical Biochemist Reviews 2006; 27: 185-190.

7. Guibert R, Schattner P, Sikaris K, Churilov L, Leibowitz R, Matthews M. Review of the role and value of near patient testing in general practice. Report to the Pathology Section, Diagnostics and Technology Branch, Commonwealth Department of Health and Aged Care. Melbourne, VIC: Department of Health and Aged Care, 2001.

8. Laurence C, Gialamas A, Yelland L, Bubner T, Ryan P, Willson K et al. A pragmatic cluster randomised controlled trial to evaluate the safety, clinical effectiveness, cost effectiveness and satisfaction with point of care testing in a general practice setting - rationale, design and baseline characteristics. Trials 2008; 9: 50.

9. Esterman A. The Likert scale. Australian Epidemiologist 2003; 10: 46-48.

10. Australasian Association of Clinical Biochemists (AACB). Point of care testing implementation guide. Perth, WA: AACB, 2008. Available: http://www.aacb.asn.au/admin/?getfile=1442 (Accessed 19 January 2008).

11. Wood JF, Burnett D. Training and certification for point-of-care testing. In: CP Price, A St John, JM Hicks (Eds). Point-of-care testing, 2nd edn. Washington DC: AACC Press, 2004; 117-125.

12. Kost GJ. Goals, guidelines, and principles for point-of-care testing. In: G Kost (Ed.). Principles and practice of point-of-care testing. Philadelphia, PA: Lippincott, Williams and Wilkins, 2002.

13. Price CP, St John A. Point-of-care testing. In: C Burtis, E Ashwood, D Bruns (Eds). Tietz textbook of clinical chemistry and molecular diagnostics. Washington, DC: AACC Press, 2005; 299-320.

14. Sawchuk ME. Ensure staff competency with point-of-care testing. Nurse Management 2004; 35: 24.

15. Kost GJ. Preventing problems, medical errors, and biohazards in point-of-care testing. Point of Care 2003; 2: 78-88.

16. Shephard M. How to set up and manage a point-of-care testing service. Clinical Biochemist Newsletter 2004; 153: 36-41.

17. Shephard M, Mazzachi B, Shephard A, Burgoyne T, Dufek A, Ahkit J et al. Point-of-care testing in Aboriginal hands - a model for chronic disease prevention and management in Indigenous Australia. Point of Care 2006; 5: 168-176.

18. International Organisation for Standardization ISO/FDIS 22870. Point-of-care testing (POCT) - Requirements for quality and competence. Geneva: International Organisation for Standardization, 2005.

19. National Committee for Clinical Laboratory Standards. Quality management for unit-use testing; approved guideline. NCCLS document EP18-A. Wayne, PA; NCCLS, 2002.

20. Shephard MDS. Point-of-care testing in the Indigenous rural environment - the Australasian experience. In: Point-of-care testing. CP Price, A St John, JM Hicks (Eds). Washington DC: AACC Press, 2004; 293-301.

21. Shephard M. Final report by the PoCT Device Group on the Point of Care Testing in General Practice Trial. Canberra, ACT: Australian Government Department of Health and Ageing, 2007.