Download - RCPAQAP Training and Competency Assessment Program · February 2014. Communication with Cornerstone for Salesforce (LMS product, US) was difficult and was exacerbated by the time

RCPAQAP Training and Competency Assessment Program

FINAL REPORT | JANUARY 2015

A project funded under the Australian Government’s Quality Use of Pathology Program

January 2015 RCPAQAP Training and Competency Assessment Program

1

Contents Contents 1

Executive Summary 2

Background 3

Learning Management System 5

Development of Modules 8

Evaluation of the online modules 13

Summary 19

Appendix A 21


2

Executive Summary Training and competency assessment of scientific staff within pathology laboratories is critical to providing a quality service and is an accreditation requirement for pathology services in Australia. Currently each pathology service prepares training and competency assessment for all key tasks within the laboratory; however, this is not currently performed in a structured or systematic manner.

Providing training and competency assessment on a web-based platform and in structured modules will deliver a standardised approach to training and competency assessment for scientists and meets both accreditation requirements and national competency based standards, as developed by the Australian Government (2009). It will also significantly reduce resources needed to provide the training and reduce duplication of effort in each pathology service.

The modules are designed to support training conducted in the workplace and act as a resource for laboratories in continuing education. These modules are intended to provide ongoing training and competency assessment for laboratory personnel, at the basic and senior levels. Certificates of completion provide evidence of training and competency for employers and regulators.

A new Learning Management System was developed, and included the capability for managers responsible for workplace training, to monitor staff progress and identify training needs for their pathology service. This has all been embedded within current RCPAQAP systems to ensure a streamlined experience for all customers.

The project was launched on 1 September 2014 with five (5) online modules;

• Quality Assurance and Quality Control – suitable for multi-test analysers in Chemical Pathology/Haematology/Immunology – (Basic scientists level) • Snake and snake bite: the role of the pathology laboratories • Pre-Analytical Module - Select safe sites for venepuncture • Cytopathology Module 1 - Anatomy of the Female Genital Tract • Cytopathology Module 2 - Cytological Identification of normal epithelial cells and other cellular components of cervical and vaginal smears

RCPAQAP has processed 65 enrolments in the first 11 weeks since launching the modules. Early feedback is very positive with scores of 8-9/9.


3

Background Pathology services within Australia are required to prepare and deliver training and competency assessment, of all scientific staff, within their organisation in order to meet accreditation requirements and provide a quality service. Pathology services usually consist of several pathology sites, both within metropolitan and rural areas. Senior scientific staff may have responsibility for as many as 30 different sites/pathology laboratories.

Senior scientific staff (also sometimes referred to as Training Managers) within a pathology service, are responsible for ensuring scientific and technical staff receive appropriate training and are assessed for competency. Each pathology service develops training activities/material and ensures that competency is assessed once the training is completed. This represents a significant amount of duplication across pathology services in Australia and is a very inefficient use of limited resources for delivering training and competency assessment.

Training is delivered face-to-face thus requiring staff to regularly travel to each pathology laboratory to deliver this service. Alternatively, staff may be required to travel to the main laboratory of the pathology group to attend training for several days or weeks. This method of delivering training has a significant impact on both staff and budget resources for the pathology service.

The online training and competency assessment modules are a cost-effective and efficient method for delivering this service to all staff in all laboratories, regardless of geographical location.

There is a need to provide training in an efficient and easy to access platform, especially for those scientists in rural areas, where training may be difficult to achieve. Online modules can be accessed easily and quickly, at any time and best of all, the modules can be undertaken at the individuals own pace.

Delivery of training and competency assessment via a web-based platform will assist in reducing the burden of increased workloads on senior scientific staff within pathology services, and also decrease administrative loads required to meet accreditation requirements. A recent survey of the pathology workforce (by the Department of Health and Ageing, May 2011) showed that more than a third of employed scientists were 50 years of age and older, indicating that a large number of qualified staff would be leaving the profession in the foreseeable future. In addition, over 80% of senior scientists were based in metropolitan areas. These are the staff who usually deliver the training and competency assessment to junior staff. This survey also found that training of staff was one of the major secondary issues of concern, after workforce shortages.


4

The target audience of the program is scientific staff, although modules for Point-of-Care (POC) and collection of pathology specimens will be aimed at other sections of medicine such as nurses, doctors and allied health staff that collect pathology specimens or use POC devices. The pilot module on ‘correctly identifying the patient’ when collecting a pathology specimen was very well received and requests for this module included allied health staff and collection staff. Similarly the POC module was popular and interest exists for more modules in this area.

While there are some organisations that currently deliver some training online they do not address the broad needs of the Australian pathology industry and competency assessment is limited.

The RCPAQAP online modules are being developed to meet the specific requirements of Australian pathology services across all disciplines of pathology. The modules are specifically designed to meet accreditation requirements, APACE and professional development requirements and the national competency based standards for Australian scientists.

The modules should enhance the training and competency assessment protocols of laboratories and compliment training that is provided for technical activities. The modules will act as an adjunct to existing training and will assist in increasing the resources available for the training and competency of scientists.

Ensuring pathology services are provided by a well trained and competent scientific workforce will provide both consumers (the patient) and referrers (the doctors) with the confidence that pathology laboratories will meet their needs and provide a quality service.

Government, as major funders of pathology in Australia, will also benefit from this initiative, as they will be assured that quality pathology services are being provided by a highly skilled scientific workforce. This will be particularly relevant in the next 5-10 years when it is expected that a significant number of highly trained senior scientists plan to retire and pathology services will experience workforce shortages. It is expected that remaining scientific staff will be under significant workload pressures as the increase in demand for pathology will come from an ageing population.


5

Project Officer Recruitment of an Education Project Manager began in July 2013; a Position Description was prepared and the position was advertised on Seek.com. Unfortunately, no suitable candidates were identified that met the selection criteria. The position advertised was a one year contract and this may have deterred suitable candidates from applying.

RCPAQAP then conducted a search for an internal staff member that may be suitable for this position. A suitable internal candidate was identified but she had not applied because the position was a short term contract. This staff member was seconded into the role of Project Officer for one year with the option of moving back into her previous position after the end of the contract.

Fifin Intan commenced in the role of Education Project Manager on 26 August 2013. Fifin commenced training to gain proficiency in Articulate, the authoring tool software. Fifin comes with experience she gained as part of the pilot team that developed the original LMS. She produced the modules, instructional documentation and Frequently Asked Questions. Fifin is a great addition to the project and has helped drive the project forward.


6

Learning Management System (LMS) An LMS Provider was identified and engaged to develop a general LMS for RCPAQAP; RCPAQAP had well documented requirements for the development of the LMS. In planning the development of the RCPAQAP LMS, consideration was given to implementing this system within the current customer portal, where participants/customers currently conduct all activities including processing sales (the enrolment process), accessing information and documents, customer relationship management (CRM) system as well as reports. The sale of the online modules had to be implemented within existing systems, as the target audience is predominantly, existing customers. Integration with the current system (customer portal) was essential.

The first stage of preparing online modules is the development of a learning management system (LMS), a system that delivers the online modules and their content. The development of the LMS was a two-fold process, firstly developing the LMS itself then development for Training Manager functionality, within the LMS.

Some technical issues were identified during the early development of the LMS, and these were related to the customer portal. This resulted in delays in the delivery of the LMS. ProQuest, the LMS development team, continued the development, and testing was conducted January-February 2014. Communication with Cornerstone for Salesforce (LMS product, US) was difficult and was exacerbated by the time differences between the US and Australia and this caused further delays. Cornerstone for Salesforce was engaged to provide the training manager or administrator functionality required in the LMS. Requirements for the Administration (Training Manager) functionality were correctly documented.

As a result of the poor communication with the US based team, the development phase had fallen behind. Some technical issues were causing delays and proved difficult to resolve. The main difficulty was that the functionality would not work within the RCPAQAP environment. The best way to move forward with the project was to engage Proquest to develop the Training Manager functionality. Despite some staffing and further technical issues (at Proquest), the delivery of the LMS with Training Manager functionally was an excellent outcome.

This delay impacted on the delivery dates of the modules and although the impact was minimal it meant RCPAQAP was not able to meet the deadlines set in the Work Plan submitted to the Department and the project was extended by six months.


7

The administration section of the LMS provides a powerful monitoring tool for Training Managers. It allows them to allocate modules and to monitor the progress, completion and pass rates of all staff under their management. This functionality will assist managers to determine training needs within their organisation.

The LMS was thus developed within the RCPAQAP customer portal (myQAP) and is displayed as the ‘Training’ tab.

The LMS and Training Manager functionality were completed late July 2014 and User Acceptance Testing (UAT) was conducted August 2014. Although some difficulties were encountered causing delay in deployment, the end product has met all expectations.


8

Development of Modules Training and competency assessment delivered in an efficient and standardised manner supports laboratory requirements, and ensures staff are trained and competent to perform key tasks. The modules are being developed to meet the requirements of the National Competency Based Standards for Scientists in Australia (2009), which have been developed by government in consultation with the pathology industry.

The 2012 pilot results were vital in determining what modules were needed and these provided a good starting point and direction for RCPAQAP to plan and develop the first modules to launch as part of this project. Once the modules were decided, subject matter experts/authors were identified for each module and negotiations ensued.

The content provided by each author was in PowerPoint format. The content provided the basis for the online module but required significant time and effort to transition from PowerPoint slides to an interactive module with audio, and finally to the test environment. Continuous communication between authors and the Project Manager ensured the modules had the look and feel that each author was aiming for and that the final product met both the RCPAQAP and author’s expectations.

Identifying and engaging subject matter experts (SME)

RCPAQAP has been in negotiations with several subject matter experts to development content for the online modules. Feedback from the pilot phase formed the basis of the planning phase and as well as decisions relating to which modules would meet the need of scientists and would be received well. In addition, some experts contacted RCPAQAP expressing an interest in working on this project.

Several SMEs were contacted to develop specialised modules in some areas. Prof Julian White from the University of Adelaide developed the Snake Venom Detection module; Associate Prof Tony Badrick from Bond University developed the QC/QA module; Rosemary Cooper from the Australian Pre-Analytical Network (APaN) developed the pre-analytical module and Ms Jennifer Ross developed the two Cytopathology modules.

Each author was provided with guidelines (prepared by RCPAQAP) to ensure they had a clear understanding of the expectations of RCPAQAP, and that modules were consistent in terms of presentation, content elements, assessment element presentation and length of the module. The guidelines also ensured the learning modules were delivered in a structured and consistent manner and met the requirements of the Competency Based standards.


9

RCPAQAP experienced delays in receiving the content material on some of the modules. Predominantly the delay was a result of heavy workloads of the subject matter expert. To ensure better management of the subject matter experts and the expectations of the RCPAQAP, the guidelines will be strengthened to include deadlines for each stage of the process, development of the module, beta testing, and adjustments to various elements including the assessment element of the module. This will provide a more realistic guide for the timelines involved in developing and producing an online module from start to finish. Consideration is also being given to implementing service level agreements with each new SME.

Engaging an external provider to develop modules

Engagement of an external provider to produce two online modules was conducted in late 2013. The use of an external provider to develop modules enabled RCPAQAP to utilise experts using high quality graphics and animations to enhance the quality of the module and produce a highly professional product. The external supplier has unlimited resources both in staff expertise and software products to produce high resolution graphics and audio content.

The Snake Venom and Pre-analytical modules (safe site for venepuncture) were most suited to the external provider as there were significant graphics utilised in these modules. The Snake Venom Detection module was ideal for this development as high resolution images or videos of the various snakes were added, as well as high quality animations showing effects on the body and explanations on how test results are impacted.

Animated Biomedical Productions (ABP) was selected as the external provider and worked with the Project Officer to ensure optimum results. The Project Officer monitored progress of the module development work and testing. The modules were completed on time and to a high standard.

Two modules were chosen to be developed by an external provider for two reasons: firstly to facilitate a faster production time of the modules and secondly to test whether a module developed externally by experts was significantly superior to those produced in house. While the two modules were of a high quality, the cost was significantly higher than those produced internally. Utilisation of an external provider will be considered in future, when increased funds are available.


10

Allocation of professional development points and recognition of modules

During the pilot phase of the training and competency assessment project in 2012, feedback was gathered which provided significant input into the development of the current modules. One major factor that was important to scientists was that they were awarded professional development points for successful completion of the modules.

The Australasian Professional Acknowledgement of Continuing Education (APACE) scheme recognises continuing education and training that contributes to professional development of scientists (http://www.aims.org.au/apace/overview). APACE is administered by the Australian Institute of Medical Scientists (AIMS), the professional association representing all disciplines of pathology.

Organisations which produce training material/activity are required to submit the material to the APACE committee, a panel of experts that allocate the relevant professional development points to the material/training activity. As a guide, APACE allocates 1 APACE point to a one hour online learning activity. Where that activity also has a competency/assessment element a further 5 APACE points may be allocated.

RCPAQAP submitted the following three modules to the APACE committee for review and allocation of professional development points in August 2014;

• Quality Assurance and Quality Control (Basic scientists level) – suitable for Chemical Pathology, Haematology and Immunology scientists • Snake and snake bite: the role of the pathology laboratories • Pre-Analytical Module - Select safe sites for venepuncture

Each module was awarded 6 APACE points upon successful completion of the module. Each module has been assigned a pass mark by the subject matter expert.

The two Cytopathology modules were reviewed by the Australian Society of Cytology as follows:

• Cytopathology Module 1 - Anatomy of the Female Genital Tract • Cytopathology Module 2 - Cytological Identification of normal epithelial cells and other cellular components of cervical and vaginal smears

These modules are eligible for professional development points according to the Australian Society of Cytology (ASC) Continuing Education for Cytologists (CEC) Program for non-medical members. Refer to Category 2 “Computer based learning”.

http://www.aims.org.au/apace/overview�


11

All five modules were successfully awarded the highest possible professional development points by the relevant association/society.

NATA

National Association of Testing Authorities (NATA) recognition of the online modules was sought, as this was considered an important issue by laboratory scientists as indicated by the feedback during the pilot phase. NATA, as Australia’s accreditation body, cannot endorse any particular product and as such cannot endorse the modules prepared by RCPAQAP.

While each pathology service must decide what type of competency assessments are appropriate, NATA does acknowledge that online training programs and competency assessments may have a role in the overall training & competency procedures that can be utilised to demonstrate technical competence. Such activities may also allow pathology services to demonstrate the ability to comply with ISO and NPAAC requirements.

Many pathology services utilise online products in the assessment of theoretical and knowledge based competency assessments and the use of these are assessed as part of the accreditation process.

It is noted that the online modules produced by RCPAQAP have been prepared by highly regarded experts within Australian pathology.

Any On-line competency product used by pathology services needs to be assessed in the context of suitability in meeting that facility’s training and competency needs. Further online competency assessment is usually only part of the process and other technical demonstrations are generally also required.

Other work activities

While work was in progress on the LMS and associated functionality, a significant body of work was completed on;

• developing the Training and Competency website

• establishing an enrolment process for this new customer group

• developing a process to assign the modules to the relevant trainee

• preparing instructions and other relevant documentation e.g. enrolment forms

• setting prices and fees


12

Launching the modules

Preparations for the launch of the modules included a significant amount of work and included developing the website, establishing an enrolment process for this new customer group, developing a process to assign the modules to the relevant trainee and preparing instructions and other relevant documentation e.g. enrolment forms and setting prices.

The five modules developed for this project were launched on 1 September 2014 at the opening of the 2015 RCPAQAP enrolments;

• Quality Assurance and Quality Control – suitable for multi-test analysers in Chemical Pathology/Haematology/Immunology – (Basic scientist level)

• Snake and snake bite: the role of the pathology laboratories

• Pre-Analytical Module - Select safe sites for venepuncture

• Cytology module 1 - Anatomy of the Female Genital Tract

• Cytology module 2 - Cytological Identification of normal epithelial cells and other cellular components of cervical and vaginal smears


13

Evaluation of the online modules Evaluation of the online training and competency assessment modules consists of several activities including monitoring the sales of each module, seeking feedback from customers who complete the modules and assessment by professional bodies allocating professional development points.

All of the above evaluation techniques are currently employed by RCPAQAP to evaluate and determine the usefulness and appropriateness of each module when compared to expected outcomes.

Customer feedback is vital in determining what modules are useful and relevant for scientists. This feedback will form the basis upon which decisions are made in the development of future modules to ensure customer’s needs are being met. The feedback will also serve to identify any amendments or updates that are required to ensure the content remains consistent with any changes in standards and industry. The modules will be reviewed annually to ensure any required changes are implemented.

Marketing and advertising the modules

A concerted and planned marketing strategy to launch the modules commenced in the first week of September 2014. It included bulk email notification to all customers in the RCPAQAP customer portal using a Flyer to announce the launch of the modules. Refer Appendix A.

The customer portal was targeted, as this is the most visited section of the RCPAQAP website. Information on the modules was also available on the public RCPAQAP website (http://www.rcpaqap.com.au/training-competency-assessment/) to capture visitors who are not currently enrolled in RCPAQAP programs and may be seeking these services.

The Flyer was also distributed to international agents who manage various regions overseas and include large numbers of RCPAQAP enrolled participants. In addition, advertising of the modules with major scientific bodies is in progress; Australian Association of Clinical Biochemists (AACB), Australian Institute of Medical Scientists (AIMS) and Australian Society of Cytology (ASC).

Future marketing plans include advertising the online modules on professional bodies’ websites or newsletters. The modules will also be promoted at major pathology/scientific conferences such as Pathology Update.

http://www.rcpaqap.com.au/training-competency-assessment/�


14

RCPAQAP advertises the modules in the company Product Catalogue which is available in the customer portal as well as the company website. The modules are also actively promoted to international markets through agents collaborating with RCPAQAP.

Feedback from customers using the modules

A feedback survey has been prepared seeking feedback on various aspects of the modules, including relevance of material, navigation of the module, appropriateness of the content and improvement suggestions. Customers who complete the modules are invited to participate in the feedback survey.

Feedback will be most useful not only to evaluate the modules but to assist with planning future modules. Some of the feedback from the pilot, such as including assessment elements throughout the module rather than a single quiz at the end of the module, was a significant improvement and ensured the learner was engaged for the duration of the online module. This format also served to reinforce the learning material.

In the short time since the modules were launched, the feedback has been very positive and the modules have been highly rated. Despite a good number of sales, only a small number of customers have completed their training and only a small percentage of those completed the feedback survey (n=8), which is disappointing. Below is an example of the some of the survey feedback;

Navigation & Instructions


15

Learning Objectives


16

Content & Presentation


17

Future Planning

Comments

‘Easy to follow’

‘These online training and assessments were the most helpful I have had to do, I wish there were more of this standard’

‘Easy to use and informative’

‘User friendly’

Although the feedback was from a small number of customers, it is worthwhile to gauge customer satisfaction.


18

Monitoring sales of each module

Monitoring the sales of the modules is critical in determining not only the usefulness and appropriateness of the modules but also will assist RCPAQAP in determining if the modules have been priced at a level the market deems appropriate.

Sales of training modules will be measured and monitored on a regular basis to evaluate the usefulness and appropriateness of each module when compared to expected outcomes. Modules are being developed to meet participant needs, so it is imperative to monitor sales to determine if those needs are being met.

Sales will be monitored throughout the year, and where expected sales figures are not met, follow-up will include communication with participants to determine the main cause of poor sales, i.e. inappropriate or perceived lack of need for a particular module.

Since the launch of the modules (in the first three months) sales are as follows;

• Quality Assurance and Quality Control – suitable for multi-test analysers in Chemical Pathology/Haematology/Immunology – (Basic scientist level), 32 modules sold

• Snake and snake bite: the role of the pathology laboratories, 9 modules sold

• Pre-Analytical Module - Select safe sites for venepuncture, 15 modules sold

• Cytology module 1, 2 modules sold

• Cytology module 2, 2 module sold


19

Summary The support from the Quality Use of Pathology Program has enabled RCPAQAP to develop a Learning Management System which delivers online Training and Competency Assessment modules for scientists, with Training Manager capability included. This facilitates monitoring of training and competency assessment of staff and identifies training needs within a pathology service.

The five modules launched 1 September 2014 have been well received with good initial sales to date and positive feedback. RCPAQAP continues to receive a significant number of inquiries about the online modules and with further advertising and marketing of these products, sales are expected to increase.

In addition to the five modules initially launched, other modules are currently being developed for QC/QA and Pre-analytical modules for senior scientists and should be available in the first quarter 2015. Negotiations are also underway to develop modules for Transfusion and Microbiology. Negotiations with an external organisation are almost complete. This is to develop a Haematology Morphology course (Red cells, White Cells and Platelets) suitable for both junior and senior scientists, as well as modules on Paediatric morphology for senior scientists. This course will be supplemented with Case Studies to further assess competence on an ongoing basis.

Problems Encountered

During the development of the LMS within the RCPAQAP customer portal, delays were experienced with the software product selected to establish the LMS functionality. This set back resulted in a 6 month delay in the delivery of the project. The main issue encountered was the lack of expected, off-the-shelf capability of the software with the licenses RCPAQAP held (despite the functionality being available normally). The solution offered by the software team was a total re-development of the LMS and the required Training Manager functionality. This solution would have been costly and time consuming. RCPAQAP was able to identify an alternate solution and developed the functionality with an alternate software development team.

Some delays were also experienced with subject matter experts unable to deliver the content as per agreed deadlines. It is acknowledged that many of the experts with whom RCPAQAP collaborates with are very busy people and the work they undertake for RCPAQAP is usually completed in their own time. This will require better management in future, possibly with the use of service level agreements which include requirements to meet deadlines for the provision of the content and for beta testing of the online modules prior to release. An alternative may be to request documented work plans and even consider an alternate subject matter expert in some cases, where a particular expert’s schedule is too restrictive.


20

Lessons Learnt

In relation to the delays and problems encountered with software development, in future RCPAQAP will undertake a more rigorous and thorough assessment process of new software, prior to committing to any work activities.

Subject matter experts present a particularly unusual and delicate management issue, as these individuals have busy full time jobs, leaving them little time to dedicate to extracurricular activities such as collaboration with RCPAQAP to develop online training and competency assessment module or modules. In future the RCPAQAP may consider engaging SMEs under service level agreements to endure the timely delivery of completed modules.

Final Comments

This is an exciting venture for RCPAQAP that will help to ensure laboratory training managers provide quality pathology services delivered by a trained scientific workforce, which will ensure confidence to patients, requesters of pathology and government. The online training and competency assessment modules also address the current workforce shortages by providing, efficient and easy to access training material that compliments on the job technical training. It focuses on meeting the Competency Based standards for scientists in structured and standardised manner and offers continuing professional development credits.

Easy access to the online modules regardless of geographical location is especially important for rural areas where training can be difficult to access. The online solution is particularly ideal in these cases.


21

Appendix A Training and Competency Assessment –2014 The Training and Competency Assessment Project was successfully piloted in 2012 and further modules have been developed since then.

The online Training & Competency Assessment modules are intended to support laboratory technical training and provide a resource for continuing education. RCPAQAP is working with experts to ensure the modules meet participants learning needs. Web-based delivery offers flexible learning where learners can access the modules anywhere and at anytime.

Modules available for 2015 are;

• Chemical Pathology – Quality Assurance and Quality Control (Basic scientists level) * • Snake and snake bite: the role of the pathology laboratories * • Pre-Analytical Module - Select safe sites for venepuncture * • Cytopathology Module 1 - Anatomy of the Female Genital Tract ** • Cytopathology Module 2 - Cytological Identification of normal epithelial cells and other cellular components of cervical and vaginal smears **

* These modules are eligible for Australian Institute of Medical Scientists (AIMS) APACE accreditation - 6 APACE points upon successful completion of each module.

**These modules are eligible for the Australian Society of Cytology (ASC) Continuing Education for Cytologists (CEC) Program for non-medical members. Refer to Category 2 “Computer based learning”.

RCPAQAP acknowledges the Department of Health and Ageing for its support of this project via funding under the Australian Government’s Quality Use of Pathology Program.

Participants will be able to enrol in these modules via the Enrolments tab in the myQAP (customer portal) and then view the modules in the Training tab once payment is received.

For further details please Contact RCPAQAP:

Phone (+ 612) 9045 6000

Email: [email protected]

mailto:[email protected]�