Enhanced Challenge Pathway Project Completion Report ECP Project... · Enhanced Challenge Pathway...

Enhanced Challenge Pathway Project

Completion Report

Final Report – March 2015

Enhanced Challenge Pathway Project Final Report

March 2015

2

Table of Contents` 1 Introduction .......................................................................................................................................................... 4 2 Project History ...................................................................................................................................................... 6

2.1 Phase 1 ............................................................................................................................................................ 6 2.2 Phase 2 ............................................................................................................................................................ 7

3 Project Methodology ............................................................................................................................................ 8 4 Industry Consultation Update ............................................................................................................................... 9

4.1 Background – Initial Industry Consultation (2011) ............................................................................................ 9 4.2 Enhanced Challenge Pathway Industry Consultation (2013) ............................................................................ 9 4.3 Conclusion ...................................................................................................................................................... 10 4.4 Purpose of 2014 Industry Consultation ........................................................................................................... 11 4.5 Interview Process ........................................................................................................................................... 11 4.6 Interview Topics and Questions ..................................................................................................................... 11 4.7 Interview Participants ..................................................................................................................................... 12 4.8 Carpenter Group Interview ............................................................................................................................. 12 4.9 Construction Electrician Group Interview ........................................................................................................ 13 4.10 Metal Fabricator Group Interview ................................................................................................................... 14 4.11 Millwright Group Interview .............................................................................................................................. 14 4.12 Heavy Duty Equipment Technician Group Interview ...................................................................................... 15 4.13 Summary – Industry Interviews ...................................................................................................................... 15

5 Summary of Registration Practices of Select Non-trades Occupations ............................................................. 16 5.1 Comparison .................................................................................................................................................... 16 5.2 Support Practices for Internationally Trained Professionals (ITPs)................................................................. 17

6 Summary of Practical Assessment Practices for Canadian Apprenticeship Authorities ..................................... 20 7 Practical Assessment Pilots ............................................................................................................................... 22

7.1 ITA Registered Assessor and Assessor Training ........................................................................................... 24 7.2 Enhanced Challenge Operating Models ......................................................................................................... 25

7.2.1 Centralized Model (ITA as Assessment Agency) ....................................................................................... 25 7.2.2 Distributed Model (ITA as Assessment Agency) ........................................................................................ 26 7.2.3 Third Party Assessment Agency Model ...................................................................................................... 26

7.3 Heavy Duty Equipment Technician Practical Assessment Pilot ..................................................................... 27 7.3.1 Purpose ...................................................................................................................................................... 27 7.3.2 Assessment Agency Model ........................................................................................................................ 27 7.3.3 Practical Assessment Tools ....................................................................................................................... 27 7.3.4 Venues ....................................................................................................................................................... 27 7.3.5 Assessor Orientation .................................................................................................................................. 28 7.3.6 Pilot Results ............................................................................................................................................... 28 7.3.7 HDET Moderation Session ......................................................................................................................... 28 7.3.8 Findings ...................................................................................................................................................... 29 7.3.9 Conclusions: ............................................................................................................................................... 29

7.4 Metal Fabricator Practical Assessment Pilot .................................................................................................. 30 7.4.1 Purpose ...................................................................................................................................................... 30 7.4.2 Assessment Agency Model ........................................................................................................................ 30 7.4.3 Assessment Lab, Tools and Equipment, Consumables ............................................................................. 30 7.4.4 Assessment Tools ...................................................................................................................................... 31 7.4.5 Assessor Orientation Session .................................................................................................................... 31 7.4.6 Candidate Assessment Session ................................................................................................................. 32 7.4.7 Moderation Session .................................................................................................................................... 33 7.4.8 Assessment Venue Debrief ........................................................................................................................ 34 7.4.9 Venue ......................................................................................................................................................... 34 7.4.10 Costs .......................................................................................................................................................... 35


March 2015

3

7.4.11 Findings ...................................................................................................................................................... 35 7.4.12 Conclusions: ............................................................................................................................................... 36

7.5 Millwright Practical Assessment Pilot ............................................................................................................. 37 7.5.1 Purpose ...................................................................................................................................................... 37 7.5.2 Assessment Agency Model ........................................................................................................................ 37 7.5.3 Venue ......................................................................................................................................................... 37 7.5.4 Practical Assessment Tools ....................................................................................................................... 37 7.5.5 Candidate Assessment Session ................................................................................................................. 38 7.5.6 Findings ...................................................................................................................................................... 38 7.5.7 Conclusions: ............................................................................................................................................... 40

8 Summary of Findings – Enhanced Challenge Pathway Completion Project ...................................................... 41 8.1 Practical Assessment Pilots – Candidate Results .......................................................................................... 42

9 What Worked? What Did Not Work? .................................................................................................................. 44 9.1 What Worked? ................................................................................................................................................ 44 9.2 What Did Not Work? ....................................................................................................................................... 45

10 Conclusion ......................................................................................................................................................... 46 10.1 National FQR Initiative .................................................................................................................................... 46 10.2 National Harmonization Initiative .................................................................................................................... 48 10.3 Findings for National Initiatives ....................................................................................................................... 49 10.3.1 Challenge requirements ............................................................................................................................. 49 10.3.2 Challenge processes, supporting tools and forms ..................................................................................... 49 10.3.3 Costs ......................................................................................................................................................... 50 10.3.4 Implementation .......................................................................................................................................... 50 10.3.5 Candidates................................................................................................................................................. 50 10.3.6 Assessors .................................................................................................................................................. 50 10.3.7 Enhanced Challenge Pathway System ...................................................................................................... 51

Appendix A. Project Methodology ...................................................................................................................................... 54 Appendix B. Industry Interview Questionnaire ................................................................................................................... 55 Appendix C. List of Interviewees ....................................................................................................................................... 57 Appendix D. Carpenter Group Interview ............................................................................................................................ 58 Appendix E. Construction Electrician Group Interview ...................................................................................................... 60 Appendix F. Metal Fabricator Group Interview .................................................................................................................. 62 Appendix G. Millwright Group Interview ............................................................................................................................. 64 Appendix H. Heavy Duty Equipment Technician Group Interview ..................................................................................... 66 Appendix I. Industry Interview Summary ........................................................................................................................... 68 Appendix J. Summary of Registration Practices of Select Non-trades Occupations ......................................................... 70 Appendix K. ITA Registered Assessor Training ................................................................................................................. 73


March 2015

4

1 Introduction

British Columbia is on the cusp of a development boom, driven by growth in liquefied natural gas and other key sectors.

This has the potential to create generations of employment for British Columbians, especially in the skilled trades sector.

However, a shortage of skilled tradespeople threatens to undermine the province’s ability to capitalize on these

opportunities.

As the provincial body with authority to issue trade credentials, the Industry Training Authority (ITA) is in a unique position to

influence the supply of skilled workers that British Columbia needs. The ITA's mandate is to both encourage participation in

formal apprenticeship training and to recognize those individuals with existing skills and knowledge that were acquired

outside of a formal Canadian apprenticeship.

ITA oversees two pathways to certification: training (apprenticeship) for new workers and assessment for experienced

workers (challengers). ITA is looking at ways to ensure that experienced workers – especially those who have the skills for

the identified growth sectors – are able to achieve credit for knowledge and skills they already possess; and where required

are appropriately trained and credentialed such that they can enter the workforce as quickly as possible at the appropriate

skill level. ITA is directly positioned to have the potential for significant impact in the area of Foreign Qualification

Recognition (FQR) and assessment in the trades.

In 2008 ITA embarked on a process to establish the best ways to achieve these ends. An in-depth Foreign Qualifications

Recognition (FQR) Review of the pros and cons of various systems and methodologies from around the world was

undertaken including desk research and face to face discussions exploring best practices in the study of New Zealand,

Australia and South Africa practices. Through this process, ITA identified successful practices other jurisdictions had used to

address the issues raised. These were learned from the implementation of systems and methodologies that provided a

variety of ways to assess the competencies of challengers. This enabled ITA to formulate the guiding principles for model

development and testing in BC.

ITA decided on an enhanced assessment approach (The Enhanced Challenge Pathway) for challengers. This approach

uses a broader range of assessment tools to judge the competency of challengers. Challengers are assessed on skills

previously obtained through formal and informal learning and work experience against a set of Occupational Performance

Standards.

The goals of the enhanced challenge pathway are:

• To strengthen assessment and recognition of individuals’ current skills and experience to enable successful continuation in the certification process.

• To address barriers to certification experienced by British Columbians trained within or outside of the Canadian apprenticeship system (with work experience from within or outside Canada)

• To ensure that the challenge pathway is robust, credible, and responsive to the needs of workers and industry to increase employer confidence in those who earn trade certification through the challenge pathway.


March 2015

5

The FQR Review completed in August 2012 found that moving away from credential and academic assessment towards

recognition of competencies is an emerging best practice within Foreign Qualifications Recognition. Beginning in 2008, the

ITA researched and piloted new forms of enhanced assessment through its "Multiple Assessment Pathways" (MAP)

initiative.

This project has built on the expertise and lessons learned from the MAP project to develop five occupation specific

competency assessments. The five trades were selected by the Ministry of Jobs, Tourism and Skills Training because they

were high volume, high-demand, and integrated into the Red Seal interprovincial standard across Canada.

The purpose of this project was for the ITA to develop and pilot competency-based assessments for internationally trained

tradespeople in five trades. The trades selected were identified as high priority occupations through employer consultation

done by the Ministry of Jobs, Tourism and Skills Training (JTST), and the FQR Review. The project was also intended for

forming a foundation from which pre-arrival assessments of skills and competencies would be built.

This report summaries the work undertaken on the project and outlines findings that can be used to inform a certification

challenge process to increase employer confidence in workers certified through challenge and apprenticeship.

The report starts with a brief history of the project and summary of work completed. It continues with a summary of

outcomes from three research activities: follow-up, more granular interviews with industry, an environmental scan of the

challenge process for selected non-trades occupations, and a survey conducted by HRSDC of practical assessment

practices in other Canadian jurisdictions. From there it provides an overview of the practical assessments piloted and

findings from each, and concludes with some next steps.

This report represents fully the challenges and successes of this project and provides evidence to direct further investment

and decisions.


March 2015

6

2 Project History

In January 2013, ITA undertook a contract with JTST to develop and pilot five competency based assessments for

internationally trained trades’ people, for the purpose of providing a more accurate assessment of skill levels. This was

intended to provide:

• A more granular understanding of skills for preparation for the Red Seal exam • The development of specific modular gap training to address skills shortages • A process that informs employers about the skill levels and employment potential of internationally trained

trades’ people prior to Red Seal certification • Better preparation for the Red Seal exam with greater potential for success

Contract deliverables were:

1. Five competency assessments for the following trades: Carpenter, Construction Electrician, Heavy Duty Equipment Technician, Millwright and Metal Fabricator. Deliverable to Ministry was a detailed framework, operational procedures/policies and methodology for each competency assessment.

2. Advisory Committees for each trade that included stakeholders, subject matter experts and representatives from the appropriate Industry Training Organizations (ITOs).

3. Assessor training program and materials to deliver the competency assessments and sustainability plan for training assessors on an on-going basis.

4. Training materials for assessors to ensure consistent and equitable delivery of the competency assessments. 5. Competency assessment process that described the steps involved in the assessments and what the applicants

could expect to gain from completing an assessment. 6. Competency assessment piloted for each trade at least once by the end of the contract term.

The project underwent several disruptions that affected the project schedule and deliverables and ultimately led the project

to be defined in terms of two phases.

2.1 Phase 1

Phase 1 of this project began in January 2013. In this phase ITA contracted the services of three ITOs1 to develop the

practical assessment tools for the 5 trades. Each ITO received $100,000 per trade to develop these tools over a period of

several months. All deliverables were due by March 31, 2014; however, not all ITOs met this deadline. The last set of

partial deliverables was received on June 11, 2014 from CITO for Construction Electrician.

To ensure that industry had opportunity to provide input throughout the project, Advisory Groups comprised of industry,

training providers and subject matter experts were formed for each trade. These Advisory Groups were scheduled to meet

4 times throughout the life of the project. The objective of those meetings was to:

• Learn about the background of Enhanced Challenge Pathway options for those not trained in the Canadian apprenticeship system

• Learn about the successful Challenge Pathway practices nationally and internationally in like jurisdictions

1 ITOs include: Transportation Career Development Association of BC (TransCDA): Heavy Duty Equipment Technician; BC Construction Industry Training Organization (CITO): Construction Electrician, Carpenter; Resource Training Organization (RTO): Millwright, Metal Fabricator.


March 2015

7

• Provide feedback and guidance on the consultation process to ensure a broad spectrum of the industry training system are engaged in the process

• Involve constituent stakeholder group in the consultation process • Provide leadership and strategic guidance to ITA • Build relationships and liaise with community stakeholders as required • Recognize the interests and needs of those seeking entry into industry via the Challenge Pathway process • Review and comment on the summary of the consultation to ensure the input and feedback heard was captured

appropriately • Provide input into the development of scenarios for future Challenge Pathway options and strategy development

as requested.

Due to staff changes at ITA in November 2013, it became clear the original deadline would not be met and an extension

was granted till December 2014.

The ITOs developed the practical assessment tools with the assistance of Subject Matter Experts (SMEs) from each of the

trades. These SMEs were instructors and industry resources with considerable knowledge and experience in the trade.

In April 2014 the BC government made public the ITA mandate review report as part of a larger initiative--B.C.’s Skills for

Jobs Blueprint. Among other recommendations, the Government recommended ITA no longer contract program

development work to ITOs. This precipitated a pause to the project while ITA developed a transition plan to implement the

changes. ITOs discontinued work on the project.

The assessment tools for one trade—Heavy Duty Equipment Technician—were ready for pilot testing and ITA worked with

the ITO to conduct this from May-July 2014. Given partially completed deliverables and discontinued ITO involvement, ITA

requested the project be put on hold while a new completion strategy was developed. This was the end of phase 1.

2.2 Phase 2

In September 2014, ITA and JTST jointly decided to adjust the remaining project deliverables and timeline. The remaining

deliverables would be completed by March 2015:

• Pilot test practical assessments for two more trades—Millwright and Metal Fabricator • Pilot test two different delivery models for practical assessments • Conduct a follow-up survey with industry to determine continued support of the model, now that more details

were understand • Submit a final report with findings on the use of practical assessments as a mandatory component of the

challenge application process

ITA’s new Manager of Assessment became the Project Manager and a consultant was hired to support the project. As part

of Phase 2, ITA also decided to include:

• An environmental scan of the challenge application, verification and certification process for three non-trades-related occupations

• An overview of practical assessments conducted by Apprenticeship Authorities in other Canadian jurisdictions


March 2015

8

3 Project Methodology

The project methodology was built on four lines of inquiry:

1. Comparative analysis – Review of Phase I ECP activities, processes and outcomes. 2. Environmental scan – Review of application registration practices. 3. Qualitative research – Industry consultation and key informant interviews. 4. Pilot studies – Practical assessment pilots.

The project benefited from a full review and comparative analysis of Phase I ECP results (1 above). The review provided a

comprehensive analysis of the project results by trade. It was an important step in developing a good understanding of

trade-specific challenge approaches, noting lessons learned and identifying necessary process improvements.

The scan (2 above) provided detailed information on registration/licensure requirements, including specific processes for

internationally trained applicants, from general supports and information provided to applicants to particular policies and

practices. Analyzing the scan brought up several concepts and practices for engaging and supporting internationally trained

professionals that could be adapted and used for the ITA’s trades challenge processes.

The project benefited from rich information collected though industry interviews (3 above). Interviewees provided

suggestions on potential improvements in the following areas: experience validation, practical assessment delivery options,

employer and trades training provider roles in practical assessment.

Two pilot projects – Industrial Mechanic and Metal Fabricator practical assessments – advanced the study further by testing

tools, protocols and other enhanced challenge components in real life situations.

The combination of desktop and primary research provided well-rounded information to inform the Project.

The Table 3.1 ECP Project Methodology (Appendix A) describes project approaches in greater details.


March 2015

9

4 Industry Consultation Update

4.1 Background – Initial Industry Consultation (2011)

In 2011, ITA conducted an extensive industry consultation on Enhanced Assessment methods comprising of surveys,

webinars, and face-to-face meetings in Kamloops, Vancouver and Victoria locations.

85 participants attended the face-to-face meetings from across the stakeholder communities, with more than a dozen trades

represented, and excellent representation from unions, educators, associations, industry training organizations and some

employers.

Table 4.1 – Feedback on the Current Challenge Process (2011)

Positive Issues

� Perspective of quick, inexpensive, simple to use, understood by employers and potential challengers

� Lack of testing of the full scope of the trade � Lack of verification of practical experience � Exam questions can be compromised � No approval or sponsor required � Written exam poses barriers to some � Difficult to verify quality of time in trade.

Table 4.2 – Feedback on the Enhanced Assessment Approach (2011)

Positive Issues

� Four assessment tools represent a significant departure from the multiple choice test in the current challenger process – and people are keen to learn more about how it might work.

� Nearly unanimous agreement that practical assessments have great potential as a measure of a candidate’s competency and verification of submitted work experience

� Enhanced assessment options were widely perceived to be a significant improvement over the current challenger process – much more likely to ensure that successful candidates have earned their certification;

� Tools such as the portfolio, competency interview and practical assessments introduce a measure of subjectivity that is not present in the multiple choice exam and a clear and transparent process of assessor accountability and management needs to be in place

� Challengers will need guidance in the preparation of portfolios

� There is going to be an on-going educational challenge, and engagement challenge to insure a new approach is well designed and understood

� Introduction of a new enhanced assessment approach for challengers will require adjustments; this is not going to be a one-size-fits-all proposition

4.2 Enhanced Challenge Pathway Industry Consultation (2013)

The Enhanced Challenge Pathway project team created Industry Advisory Groups for each of the five trades to provide the

industry with an opportunity to share their experiences, challenges and opportunities in the trades. The Industry Advisory

Groups engaged the services of Subject Matter Experts from Trade Schools and the industry to develop the practical

assessment tools.

These Advisory Groups were scheduled to meet 4 times throughout the life of the project. The objectives of the Advisory


March 2015

10

Group meetings were to:

� Learn about the background of Enhanced Challenge Pathway options for those not trained in the Canadian apprenticeship system.

� Learn about the successful Challenge Pathway practices nationally and internationally in like jurisdictions. � Provide feedback and guidance on the consultation process to ensure a broad spectrum of the industry training

system is engaged in the process. � Involve constituent stakeholder group in the consultation process. � Provide leadership and strategic guidance to ITA. � Build relationships and liaise with community stakeholders as required. � Recognize the interests and needs of those seeking entry into industry via the Challenge Pathway process. � Review and comment on the summary of the consultation to ensure the input and feedback heard was captured

appropriately. � Provide input into the development of scenarios for future Challenge Pathway options and strategy development

as requested.

The Advisory Group members were engaged in the consultation process and demonstrated high interest in developing and

implementing enhanced challenge pathway methods. In spite of this, the consultation was only partially completed (one or

two meetings per Advisory Group –TBC). The main reasons for the incomplete consultation include ambiguous project

directions, project interruptions, staff and organizational changes.

4.3 Conclusion

The 2011 consultation examined the views of the consultation participants on the Enhanced Challenge approach used in

the Multiple Assessment Pathways (MAP) projects involving Heavy Duty Technicians and Cooks. A MAP trade challenge

process used a variety of methods (portfolio of evidence, interview, written assessment, practical assessment) to determine

a candidate’s competency against industry performance standards. The Candidate is deemed competent if he/she can

demonstrate evidence of competency through the combination of the assessment methods (blended mark).

The 2013-14 Enhanced Challenge Pathway (ECP) project is built on the notion of introducing additional assessment

methods to the existing interprovincial multiple-choice exam. Achieving a 70% mark on the IP exam is a mandatory

requirement. Additional evaluation methods such as practical assessment would be used to provide supplementary

evidence of the Candidate’s practical skills and overall competency in a trade. The Candidate is evaluated on his or her

ability to perform practical assessment tasks against established performance standards. It is a mandatory requirement to

meet these standards during the assessment.

The major difference between the MAP and ECP pilots is the certification achievement criteria. For the MAP project, it is a

cumulative mark from multiple assessment methods. For the ECP project, the criteria require two independent, not

combined, results from the written IP exam and practical assessment against established standards.

It is important to consider these differences while conducting the analysis of the results of the initial consultation and current

industry interview.


March 2015

11

4.4 Purpose of 2014 Industry Consultation

The purpose of the Industry Consultation Update (2014) was to consult with industry on the importance of the practical

assessment component in the certification challenge process and obtain recommendations on making the challenge

pathway robust and credible, and responsive to the needs of workers and industry.

The objectives of the consultation were to:

1. Collect industry opinions on current challenge requirements and challenge process; 2. Identify ways to strengthen current challenge process; 3. Identify ways to increase employers’ confidence in the skills of workers who achieved trade certification through

challenge rather than formal apprenticeship; 4. Determine the value of trade certification in selecting and hiring skilled trades workers, including internationally

trained workers.

The consultation was focused on five trades: Carpenter, Construction Electrician, Heavy Duty Equipment Technician,

Industrial Mechanic, and Metal Fabricator.

4.5 Interview Process

A three-step process was used in the organization and completion of the industry interview.

1. Interview preparation, including: • Identifying and approving potential industry contacts for each trade with the Industry Relations Manager and

Project Authority • Determining and approving interview questions

2. Conduct interviews:

• Minimum five interviews per trade

• Collect feedback from a range of employers including large companies and small- and medium-sized enterprises, subject matter experts working in the industry, union representatives and members of industry associations.

3. Summarize and analyze findings

4.6 Interview Topics and Questions

The interview focused on determining whether the current challenge process reflects industry needs and on strengthening

the challenge process to increase employer confidence in Candidates who acquired certification by challenge. The

interview was also a way to discuss the perceived overall value of trade certification and the employers’ confidence in the

skills of certified employees who went through the challenge process compared to those who completed a Canadian

apprenticeship to become certified.

The following four topics were selected for examination:

1. Challenge requirements.


March 2015

12

2. Enhanced challenge process – practical assessment, interview, others – and its impact on employers’ confidence in the skills of challenge Candidates;

3. Enhanced challenge costs; 4. Importance of certification for workers and for making hiring decisions.

A total of 12 questions were developed for the interview. Most of the questions were open-ended or combination questions.

These question types allowed for an open discussion and exploration of solutions. Other than the first question that

addressed trade-specific challenge requirements, the questions were identical. A copy of the questionnaire is enclosed in

Appendix B.

4.7 Interview Participants

The interviewees included trade-certified subject matter experts, HR managers, operations managers, supervisors, and

union and industry group representatives. The list of interviewees by trade is available in the Appendix C.

The interviewees were selected from the current ITA sponsors, industry representatives from Industry Task Forces and

workforce committees, unions and employer associations. A consultant created an initial list that was later updated by the

ITA Industry Relations Managers. The final list of potential participants was reviewed and confirmed by the Project Authority.

An e-mail invitation to participate in the interview was sent out to potential participants who had been determined and

confirmed earlier. The e-mail response was low; direct follow up phone calls helped achieve the desired number of interview

participants for each trade group.

Although the timelines for organizing and conducting interviews were constricted, the project exceeded the minimum

number of 25 interviews, achieving 33. The interviews took place over the period of November 14 and December 5, 2014.

4.8 Carpenter Group Interview

The group consisted of five respondents.

The interview revealed the following:

1. There is low confidence in the effectiveness of the current Employer/Experience Declaration Form. Specifically employers are not confident in whether the declared hours cover the full scope of the trade.

2. There is an agreement that experience declaration and verification processes need to be strengthened. 3. Suggestions on how to strengthen these processes include:

a. Improved employer declaration form – include more details on skills, duties, within the scope the trade b. Practical assessment – will allow demonstration and observation of skills c. Verification interview conducted by a subject matter expert.

4. Respondents support having a practical assessment for challengers. However, there are concerns related to assessment design and implementation, specifically:

a. Carpentry trade is not mandatory b. Scope of practical assessment in relation to core competencies c. Assessment administration d. Industry support and employer buy-ins

5. There is strong support for on-site practical assessments conducted by the employer – Employer Sign-Off.


March 2015

13

6. If the employer does not conduct the practical assessment, the majority of the respondents agree on an 8-hour maximum assessment. This would cover core competencies and be sufficient for getting evidence of the candidate’s core practical skills.

7. Proposed practical assessment fees are in the range of $300-$500 excluding challenge application fees (currently $120).

8. Although the respondents strongly support having a technical interview by a subject matter expert, they do not think it should substitute the practical assessment

A detailed interview summary is available in Appendix D.

4.9 Construction Electrician Group Interview

Eight industry representatives participated in the interview (two companies provided two representatives each).

The key findings for the Construction Electrician group include:

1. Four out of seven respondents think that the current challenge process should be strengthened. 2. Other three respondents are satisfied with the current challenge requirements; they think that additional

requirements may create barriers to certification and hiring. 3. All agree that there is a need to strengthen candidate screening and experience verification processes. 4. Suggestions for strengthening the overall challenge process include:

a. Improved validation of claimed experience (detailed employer/experience declaration and verification interview conducted by subject matter expert)

b. Mandatory Canadian Electrical Code Course c. Completion of a college-based practicum (4-8 weeks).

5. Two respondents suggested that the challenge requirements for internationally trained professionals should be different. Suggestions include:

a. “Preferred countries” list; b. Provisional certification.

6. Practical assessment as a mandatory part of the challenge process is viewed positively but there are concerns related to assessment design and delivery format.

7. Those who support the practical assessment have more confidence in a college-based practicum. 8. If the practical is not college-based, then the suggested duration is 6-8 hours. 9. The range of the assessment fees is broad – from $300 to $1000 – depending on the assessment design and

delivery format. 10. There is agreement that a technical interview is valuable, but that it should not be a substitute for a practical

assessment. 11. There is an observation that those who deal with selection and hiring are more likely to see practical assessment

as a barrier; the majority of operational representatives do not see it as an obstacle.

Additional comments and suggestions include:

1. Develop strategies for attracting Electricians from the U.S. to meet growing demand for trained and certified electrical workers.

2. Consider a provisional certification for electrical workers. 3. Add college practicum to the certification challenge process as a substitute for the practical assessment. This

would better align apprenticeship and challenge processes than a practical test for challengers. It would also increase employer confidence in Challengers’ skills and knowledge. A practical assessment delivered through a college would be more standardized and may have more value.

A detailed interview summary is available in Appendix E.


March 2015

14

4.10 Metal Fabricator Group Interview

Seven industry representatives participated in the interview.

The key findings for the Metal Fabricator group include the following:

1. The majority of respondents (5 out of 7) have low confidence in the results of the current experience declaration and verification processes.

2. There is an agreement that experience verification interviews (with both employer and challenger) and practical assessment should strengthen the current challenge processes.

3. Practical assessment is viewed as a stronger assessment method than an interview. 4. It was mentioned that blueprint reading and interpretation is one of the core skills and it must be incorporated in the

challenge process. 5. Proposed practical assessment duration varies from a 4-8 hour assessment to a 4-6 week college-based

practicum. 6. Proposed fees vary from $300 to $2000; the latter fee is related to a college-based practicum. 7. Respondents believe that the practical assessment business model needs to be built on an effective cost recovery

rather than profit generation basis. 8. Respondents agree that challenge requirements need to be the same for different candidates including those who

gained experience in, across and outside Canada. 9. Employers hire workers with and without certification. For unionized companies, a trade certificate is a requirement

for employment as a journeyperson.

A detailed interview summary is available in Appendix F.

4.11 Millwright Group Interview

Seven industry representatives participated in the Millwright group interview.

The key findings for this group include:

1. All respondents want to see a stronger challenge application process that will include an enhanced Employer and Experience declaration form.

2. Respondents want to ensure that the experience candidates submit is within the scope of the trade. 3. There is an agreement that employers should be accountable for reporting on the working hours of their workers. 4. There is support for making practical assessment a part of the challenge process; it is believed that the practical

assessment will strengthen not only the challenge process and its outcomes but also the initial candidate screening process by attracting the “right” candidate.

5. When asked to define the “right” candidate, respondents said – “someone who has direct industry experience within the scope of the trade, understands the trade, and is able to demonstrate practical skills”.

6. The suggested duration for a practical varies from 4 - 16 hours to a 40-hour college-based practicum. 7. Millwright trade is highly diversified so practical assessment has to cover core competency areas. 8. Assessment fees vary from $500 to $3000 depending on the type of practical assessment. The majority of

responses about assessment fees fall in the $500-$1000 range. 9. Most (6 out of 7) respondents do not hire workers without trades certification for Millwright positions. 10. Employers want to see up to 95-100% of their Millwright workers certified. They also value direct industry

experience; experience has high importance in employee selection and hiring process.

A detailed interview summary is available in Appendix G.


March 2015

15

4.12 Heavy Duty Equipment Technician Group Interview

Six industry representatives participated in this interview.

The key findings for the HDET group include:

1. The majority (5 out of 6) of respondents think that the current challenge process needs improvement. Specifically, they want to see improved experience declaration and verification process and a possible addition of practical assessment.

2. Challenge requirements need to be the same for all challenge candidates no matter where they acquired their work experience – in or outside Canada.

3. Most respondents (5 out of 6) support adding a practical assessment as a mandatory component of the challenge process.

4. The majority (4) of respondents suggest a 4-8 hour assessment that covers core skills. 5. Fees vary from $250 to $1000. Assessment fee is viewed as a barrier for some candidates but not a concern for

the industry; industry wants to attract and support the “right” candidate. 6. There is support for having an interview conducted by a subject mater expert as an additional assessment method

but not as a substitute for the practical assessment. 7. The majority of the interviewees (4 out of 6) think that employers have more confidence in the skills of candidates

who achieved certification through formal apprenticeship rather than through the challenge process.

A detailed interview summary is available in Appendix H.

4.13 Summary – Industry Interviews

In summary, seven major themes emerged in the industry interview:

1. Lack of satisfaction with the current challenge process. 2. Lack of satisfaction with experience declaration and experience verification processes. 3. Practical assessment is a preferred method for measuring Candidates practical skills. 4. Different level of support for the practical assessment as a mandatory component of the challenge process across

five trades (non-regulated trades – lower support). 5. Concerns about practical assessment design, development, delivery, costs, long-term maintenance. 6. Interest in a mandatory college-based practicum for challenge candidates. 7. Concerns from some industry representatives in Human Resources roles who see a mandatory practical

assessment as a barrier to certification and, consequently, hiring.

The findings from industry interviews with 5 trades groups (Carpenter, Construction Electrician, HDET, Metal Fabricator,

and Millwright) are provided in the Table 4.3 (Appendix I).


March 2015

16

5 Summary of Registration Practices of Select Non-trades Occupations

The review of practices for registration of Engineers, Technologists and Registered Nurses included an examination of

current certification and licensing processes used by regulatory bodies to assess applicants. The review focused on

identifying current processes and practices for internationally trained candidates and evaluating whether these practices can

be adapted for assessing applications and trades equivalency of internationally trained tradespersons.

In addition, it was expected that the review would help develop a model that could streamline the application, verification

and assessment processes for internationally trained tradespersons including those who reside in and outside of Canada.

Further examination of the resources and their application is recommended to evaluate how this approach can be

implemented for internationally trained tradespersons.

A detailed summary of registration practices for three non-trades occupations (Engineers, Technologists and Nurses) is

provided in the Appendix J.

5.1 Comparison

The table below provides a summary of current registration/licensure requirements, categories and processes for three

organizations representing Engineers, Technologists and Registered Nurses.

Table 5.1 – Registration/Licensure Requirements (APEG BC, ASTT BC, CRN BC)

Organization

Candidate Support

APEG BC ASTT BC CRN BC

Pre-registration support • Yes/APEGBC Online Self Assessment

• Yes/ Technology registration Canada (TRC) Online Self Assessment;

• ASTTBC Online Self Assessment;

• ITTP web portal

• Yes/ CRN BC website; • National Nursing

Assessment Service (NNAS) – FQR support across Canada

Application

requirements

• Engineering Degree from recognized university or college;

• 4 years of engineering experience with references;

• Canadian environment experience;

• English language competency

• Diploma of Technology (2-3 years);

• Min 2 years of experience;

• Main requirements: Graduation from a Nursing program;

• Different requirements for 1

st time applicants and

renewal applicants


March 2015

17

Organization

Candidate Support

APEG BC ASTT BC CRN BC

Registration

categories

• 14 different categories, e.g.

• Professional Engineer, Engineer-in-Training, Limited Licensee (Eng.L.) Professional Geoscientist, Geoscientist-in-Training, Non-Resident Licensee (NRL) P.Eng., International Mobility P.Eng., other.

• Applied Science Technologist (AScT); Certified Technician;

• Provisional certification for internationally trained (have to complete 1-year work experience requirement within 5-year period after registration)

• Associate member; Other.

• Registered Nurse (RN); • Nurse Practitioner (NP); • Nurse Practitioner

Grandparented (GNP); • Licensed Graduate Nurse

(LGN); • Other

Assessment and

registration

process/requirements

• Application approval; • Completion of the

Law & Ethics Seminar;

• Completion of the Professional Practice Examination (PPE);

• Application approval; • Completion of Professional

Practice and Ethics Examination;

• 1 year of experience in Canada for internationally trained professionals with Provisional Certification

• Application approval; • Written examination; • Objective structured

clinical examination (OSCE);

• Experience/special conditions for members with Provisional certificate/license

Costs • Application fee - $472;

• Academic Examination(s) - $338 per examination;

• Professional Practice - $240 (in Canada)-$340 (outside of Canada);

• Law& Ethics seminar - $400.

• Application fee - $250 • Annual dues – Various

depending on type of certification

• ($365- AScT, $260 – Graduate Technologist)

• Registration fees – Registered Nurse -$450;

• Nurse practitioner - $650; • RN exam-$615;

NP written exam - $700; • NP clinical exam - $2,000 • Other fees – renewal,

reinstatement

5.2 Support Practices for Internationally Trained Professionals (ITPs)

The scan has identified current support practices that help internationally trained professionals (ITPs) prepare for

registration/licensure process in B.C. It is clear that both the industry and the government make significant investments into

the attraction of skilled workers. Both direct their efforts to reducing barriers to achieving professional certification, and to

facilitating successful skilled immigrant labour market integration.


March 2015

18

A brief description of these support practices is provided in the table below.

Table 5.2 – Support practices for Internationally Trained Professionals (APEGBC, ASTTBC, CRNBC)

APEG BC

ASTT BC CRN BC

Support Process • APEG BC web info and links to ITPs

• Online self assessment tool

• ASTTBC web info and links to ITPs

• Online Self assessment tools (ASTTBC and Technology Registration Canada- TRC)

• Internationally Trained Technology Professionals web portal (www.itppbc.com)

• CRN BC info and links to ITPs

• National Nursing Assessment Service

• Provisional Certification

Purpose • Information for applicants; • Help Candidates determine

their skill level and decide on certification type;

• Targets internationally trained Candidates; Canadian Candidates can use the tool as well.

• Information for applicants;

• Help Candidates determine their skill level and decide on certification type;

• Targets internationally trained Candidates; Canadian Candidates can use it too.

• Information for l applicants;

• Provide harmonized approach towards assessment of internationally trained nurses across Canada.

• Reduce or eliminate employment barriers associated with a lack of Canadian work experience.

Outcome • Starting point • Self-assessment results

(printout form) as a recommendation to apply to APEG BC or ASTT BC;

• The self-assessment is not an official assessment of qualification for registration or license in B.C

• Starting point • Self-assessment

results (printout form) as a recommendation to apply to APEG BC or ASTT BC.

• Self-assessment is not an official assessment of qualification for registration or license in B.C.

• Starting point • Information and support

systems (bilingual) • Certification as

employment requirement;

• Faster employment.

In summary, these are four models that the regulatory bodies use to guide internationally trained applicants through the

registration and licensure processes:

1. Online self-assessment tools (Technology Registration Canada (TRC), APEGBC, ASTT BC). 2. Internationally Trained Technology Professionals web portal (www.itppbc.com). 3. Provisional certification (technologists and nurses). 4. National Nursing Assessment Service.


March 2015

19

The first two models could be adapted to develop online self-assessment tools for foreign trained tradespersons. Such tools

would help foreign trained workers determine their skill set and decide on challenge level. The third model – provisional

certification – could be adapted to help reduce barriers to employment for internationally trained trades persons. The forth

model, NNAS, focuses on aligning national requirements.

Further review is required to determine how these practices fit current ITA priorities.


March 2015

20

6 Summary of Practical Assessment Practices for Canadian Apprenticeship Authorities

In October and November 2014, a jurisdictional survey was conducted on the use and structure of practical tests in

Canadian provinces and territories. 10 provinces and 3 territories submitted responses.

Key findings for this survey include:

1. There are no specific or consistent criteria to determine which trade has a practical assessment. a. In Ontario, practical exams were developed as an alternative to the written exam to provide an avenue for

individuals who have been unsuccessful in passing the written exam; b. In Alberta, the provincial or local apprenticeship committee makes a recommendation for a practical and

the Program Development Officer (PDO) and the Provincial Apprenticeship Committee (PAC) develop the assessment together;

c. In Saskatchewan, the criteria is typically established when a trade is being designated; d. In Manitoba, there are no criteria.

2. Three (BC, ON, NL) apprenticeship authorities have been approached to develop a practical assessment for the following trades: Hairstylist, Construction Electrician, Plumber, Automotive Technician, and Heavy Duty Equipment Technician, Machinist (for apprentices in NL).

3. In BC, apprenticeship authority receives regular requests for trade challengers, but employers are discouraged due to cost and complexity to manage. In NB newly designated trades have the option of requiring a practical assessment for a period of time to “grandfather” experienced candidates

4. In most cases, practical assessments are delivered by training providers, apprenticeship authorities, and the department of labour. In BC, practical assessments are also delivered by Third Party Assessment Agencies under contract with ITA.

5. The level of support for challenge Candidates or apprentices to prepare for a practical assessment varies across the provinces; some offer basic online information and notification letters, others offer no support.

6. The duration of a practical assessment across different trades is 1.5-8 hours. 7. All jurisdictions require assessors to be certified in their trade with a minimum amount of experience (varies by

trade and jurisdiction). Ratio of assessors to candidates varies by trade and jurisdiction and range from 1:1 (ON) to 1:6 (BC).

8. There is no consistency in Assessor training. a. In British Columbia, there is an assessor training program comprised of three (3) days of generic training

and one (1) day of trade-specific training; b. In Saskatchewan, markers may also have a “trial marking session” and in some cases, large groups of

markers will have an annual marking seminar; c. In Manitoba, most assessors are college instructors and do regular apprenticeship training. Other

examiners may train examiners, plus they have the occasional professional development session; d. In New Brunswick, new assessors work with a seasoned assessor; e. CWB and Department of Labour train their own assessors.

9. Practical assessment fees vary and are trade specific. At the moment, the lowest fee is $200 (Carpenter/NT), the highest is $750 (Cranes/BC).

10. Practical assessment cost is one of the biggest concerns for all apprenticeship authorities. 11. The biggest challenges apprenticeship authorities face are:

a. Costs associated with the development of practical assessment tools b. Costs associated with assessment administration and management; c. Low number of challenge candidates for some trades; d. No standard practical assessment practices; e. Lack of suitable testing locations; f. Lack of qualified assessors; g. Colleges’ availability to hold practical assessments;


March 2015

21

h. Candidates’ readiness to take practical assessment; i. Internal resources and staff to develop and coordinate practical assessment; j. Maintenance of assessment tools.

12. Practical assessment benefits: a. Increased employer confidence; b. Opportunity for Candidates to demonstrate competency in the trade versus ability to write an exam; c. Results help to confirm the written assessment results; d. Industry engages with and supports the process; e. Verification of practical skills – encourages buy-in from industry


March 2015

22

7 Practical Assessment Pilots

Three practical assessment pilots were conducted over the period April 2014 to February 2015. Heavy Duty Equipment

Technician pilot was completed over the period April 2014 to June 2014. Practical assessment pilots for the Metal

Fabricator and Industrial Mechanic trades were part of the ECP Phase Two and were completed over the period from April

2014 to January 2015.

The purpose of the pilots was twofold – to test the validity and usability of tools and to determine the effectiveness of the

enhanced challenge models and their compatibility with existing organizational functions.

In order to recruit Candidates to participate, ITA offered a financial incentive: $200 Canadian Tire gift certificate. The profile

of the Candidates varied across the pilots and trades. Because these were only pilots, the Candidates were informed that

their results would not count toward their certification.

The table below provides a brief summary of the pilots’ characteristics, tools under revision and validation, and operational

models under review.

Table 7.1 – Summary: Enhanced Challenge Pathway Practical Assessment Pilots Trade

Criteria

HDET Metal Fabricator Millwright

No. of Candidates

Assessed/Passed

12/10 2/2 2/1

No. of Assessor

Trainees

12 2 2

Assessor to Candidate

Ratio

1:1 1:1 1:1

Assessor Rate

$55/hour $75/hour $500/day

Estimated Cost

(practical only)

$1,500 $1,450 $900

Operational Model

(pilot only)

Decentralized/College Centralized/College Third Party


March 2015

23

Trade

Criteria

HDET Metal Fabricator Millwright

Tools Tested • Self-assessment; Checklist tested – found not useful;

• Interview tested – found subjective, requires changes;

• Practical Assessment Tool tested;

• Assessment Venue criteria confirmed;

• Assessor Qualifications confirmed.

• Practical Assessment Tool reviewed and undergone significant changes; Revised version tested;

• Validation with broader industry is required; Assessment Venue criteria confirmed;

• Assessor Qualifications confirmed.

• Practical Assessment Tool reviewed and undergone significant changes.

• Revised version tested; • Candidate Handout

developed and tested; • Validation with broader

industry is required; • Assessment Venue

criteria revised; • Assessor Qualifications

confirmed. • Assessor Toolkit

developed NEW* Level of Efforts Significant Significant Moderate

Successes • Approved practical assessment tool;

• Assessor contracts; • Decentralized model

tested; • Suggestions on

improving tools and processes;

• Candidates were engaged.

• Improved and simplified practical assessment tool;

• Centralized model tested;

• Assessor contracts and scope of work;

• Candidates were engaged.

• Finalized Candidate handout, including work order and drawing;

• Improved and simplified practical assessment tool;

• Developed Assessor Toolkit - flexible delivery;

• Third party model tested; • Assessment time

reduced; • Candidates were

engaged and felt comfortable in worksite setting;

• Employer partner provided support and endorsement;

• Venue costs; • Per diem Assessor rate.

Issues • Additional costs due to travel (location of venues and potential assessors);

• Pilot documentation not completed or completed poorly;

• Standard of performance was missing.

• Small sample size (2 Candidates);

• Lack of time to test other assessment tools;

• Assessor rate ($75/hr) is high; Total cost of $1450 might be a barrier;

• Pilot documentation not completed due to ongoing revisions of the tool.

• Small sample size (2 Candidates);

• Lack of time to test other assessment tools;

• Pilot delay due to unforeseen events;

• Pilot documentation not completed due to ongoing revisions of the tool


March 2015

24

In summary, the pilots revealed the following:

1. Conducting small-scale pilots was a cost effective way to test the tools, various operating models, make necessary revisions and adjustments.

2. Practical assessment tools were designed to test both assessment process and outcomes. This format is more suitable for practical assessments in a college lab environment.

3. The pilots determined that the assessment needed to focus more on the performance outcomes. Focusing on performance outcomes allowed for tools to be simplified and be more applicable across various assessment settings (trade training lab, worksite).

4. Practical assessment tools (for three trades) have undergone reviews and revisions to streamline the assessment process.

5. Candidates from all three trades confirmed that practical tasks have a sufficient level of complexity and reflect basic core skills that a qualified tradesperson must possess.

6. Assessors demonstrated relevant expertise for conducting practical assessments and managing required protocols.

7. ITA Registered Assessor training is required to ensure that Assessors have a strong understanding of assessment rules and protocols (including safety, privacy, duty to accommodate).

8. The Third Party model proved to be the most flexible and cost effective. 9. Candidates requested detailed information on the practical assessment (online practical assessment guide,

instructions or checklist) that would help them prepare accordingly. 10. The tools that have undergone changes as result of the pilots require review and validation by Advisory Group

members and /or broader industry. 11. Practical assessment tools (for all three trades) require final amendments prior to implementation. 12. Enhanced challenge forms (e.g. self-assessment checklist, experience declaration form) require final amendments

prior to implementation.

The detailed descriptions of the ITA Assessor training, operating models and three pilots are provided in the following

sections.

7.1 ITA Registered Assessor and Assessor Training

Assessor Training

ITA’s Registered Assessor Course is designed around 5 areas of Assessor performance, specifically:

1. Prepare for Assessment. 2. Demonstrate Effective Communication and Interpersonal Skills. 3. Conduct Assessments and Make Assessment Decisions. 4. Record and Report Assessment Decisions and Recommendations. 5. Apply Continuous Improvement to the Assessment Process.

The course consists of a pre-course study, 4 days of classroom and shop instruction and activities, and an assessment

practicum (2 supervised assessments).

By the end of the course participants will be able to:

1. Explain the certification system and roles of ITA, Assessors, Assessment Agencies, and certification candidates 2. Interpret the competency standard for a specific trade or occupational qualification. 3. Define the qualities and characteristics of a competent assessor. 4. Explain the code of conduct, privacy, and conflict of interest requirements. 5. Identify the parts and steps of the assessment process. 6. Prepare candidates for assessment.


March 2015

25

7. Conduct assessments and make decisions in accordance with the principles of assessment and rules of evidence. 8. Complete necessary paperwork. 9. Provide candidates with helpful and objective feedback. 10. Describe service standards and ITA-approved operational procedures.

Upon successful completion of required assessor training, ITA will issue an Individual Identification Number to an Assessor.

ITA will maintain a registry of Assessors in ITA Direct Access.

The detailed information of the ITA Registered Assessor training is available in the Appendix K.

7.2 Enhanced Challenge Operating Models

Enhanced Challenge Operating models determine how the new assessment functions will operate within the established

ITA structure and processes.

It was important to test the models and identify the best match with existing organizational processes and functions. It was

anticipated that the successful model would support, strengthen and accelerate enhanced challenge and be flexible as

economic events and industry demands change over time.

The project included the testing and analysis of three Enhanced Challenge operating models:

1. Distributed Model a. ITA as assessment agency b. Different training institutions as venue

2. Centralized Model a. ITA as assessment agency b. One training institution as venue

3. Third Party Model a. Third party assessors approved by ITA b. Work site as venue

7.2.1 Centralized Model (ITA as Assessment Agency)

In the Centralized Model the ITA serves as the assessment agency. The ITA identifies an independent assessment venue

(e.g. public or private school with trades training lab) that has demonstrated the willingness and capacity to deliver

specialized practical assessments for a trade in a safe manner.

The ITA oversees the following activities:

� Identify and confirm appropriate assessment venue for a trade; � Develop and sign assessment venue agreement (includes scope of activities and expected outcomes); � Identify, train and register Assessors as per ITA policy framework; � Organize and conduct Assessor training; � Develop and manage Assessor contracts; � Manage Candidate application and other required documentation; � Develop and coordinate assessment schedule with the venue. � Manage Assessor performance


March 2015

26

� Maintain an Assessor registry � Manage Assessor Moderation Session and Professional Development Activities

The Assessment Venue is responsible for the following:

� Assessment lab; � Assessment materials, tools and equipment; � Access to the facility; � Site security; � First Aid and emergency contacts; � Scheduling required shop, classroom or other space.

* Note: other activities may be included as per specific trades’ needs

7.2.2 Distributed Model (ITA as Assessment Agency)

The Distributed Model has the same characteristics as the Centralized Model except that different training institutions

assume the role of the assessment venue for a given trade. All training institutions/venues involved in the Enhanced

Challenge activities must meet established Assessment Venue Criteria.

7.2.3 Third Party Assessment Agency Model

In the Third Party Assessment Agency Model the ITA outsources the enhanced challenge operation through a bidding

process.

The ITA oversees the following activities:

� Identify successful bidder though an RFP process; � Develop and sign Assessment Agency Agreement; � Assign a staff person to oversee the Assessment Agency Agreement and reporting.

The Assessment Agency provides the following services:

� Work in collaboration with ITA staff responsible for the Assessment Agency agreement; � Identify and confirm assessment venue; � Manage and coordinate ITA-registered Assessors; � Coordinate assessment applications; � Develop and coordinate assessment schedules; � Monitor and record assessments as per established ITA standards; � Submit assessment records and recommendations to the ITA; � Establish safety, First Aid, emergency response procedures in collaboration with assessment site(s); � Confirm all applicable insurances (for Candidates, Assessors and other participating parties).

* Note: other activities may be included as per specific trades’ needs


March 2015

27

7.3 Heavy Duty Equipment Technician Practical Assessment Pilot

7.3.1 Purpose

The purpose of the HDET Practical Assessment pilots was to:

� Test the tools specific to the practical assessment � Identify opportunities to improve the practical assessment process � Confirm that the Assessors know and understand their roles and responsibilities � Identify which Venue(s) would be most appropriate to launch the practical assessments � Identify real costs associated with planning, organizing and implementing the practical assessments for the trade � Provide a learning experience for ITA participants who had never been involved in practical assessments.

It should be noted that only practical assessments, and no other sub-processes or tools related to the overall practical

assessment process (such as candidate scope of work verification interviews) were piloted. The project team was directed

to “test” them during the launch.

7.3.2 Assessment Agency Model

This pilot used the Distributed Model.

7.3.3 Practical Assessment Tools

The practical assessment pilot for Heavy Duty Equipment Technicians was conducted during Phase 1 of the project.

The following practical assessment tools were used during the HDET practical assessment pilots:

� Assessor’s Practical Skills Assessment Task Summary � Assessor’s Information Sheet � Candidate’s Work Order � Marking Sheet � Confidentiality Agreement

7.3.4 Venues

Lessons learned regarding venue requirements and issues is summarized in the list below:

� Not all Venues provide First Aid Contacts. Must ensure that 1 person participating during the practical assessment has First Aid Level One training.

� Need to ensure that Assessors have a designated area to conduct initial instructions for the practical assessment participants.

� All participants, not just the Candidates, must participate in Safety Orientation. � Despite the fact that discussions took place with Venues and agreements were put in place for the practical

assessments, it became evident that ITA must be very specific in terms of its expectations as to the services provided by the Venues.

� Because the Venues did not provide the Assessors to conduct the pilots and the Candidates did not attend the trade schools, ITA had to purchase liability and errors and omissions insurance at a cost of $11,000 for all the pilots for all 5 trades. Should the ECP project go live, ITA could negotiate with the trade schools, a fee that would cover the insurance for both the Assessors and the Candidates.


March 2015

28

7.3.5 Assessor Orientation

Although all the HDET Assessors participated in the Assessor training/tool development workshop, not all Assessors

demonstrated the skills required to become a Registered Assessor and will require additional training and mentored practice

to administer practical assessments. All will require training and practice using any other assessment tools such as work

verification interviews if ITA adopts these for use in the future. Interview training and successful completion of a mentored

practicum is standard procedure for new assessors.

The cost of HDET Assessor “Training”, or rather Assessor tool development & trial, was approximately $17,600 for 11

Assessors or $1,600 per person. This amount includes a daily rate of $175 and their travel and accommodation expenses.

This training did not include training on conducting or evaluating technical interviews or completing assessment

documentation.

7.3.6 Pilot Results

The BCIT pilot was the first pilot conducted and was the most valuable pilot in terms of learning experience. Six ITA staff

participated in that pilot throughout the weekend; some participated over the 2 day period while other attended only 1 day.

Because these were only pilots, the Candidates were informed that their results would not count toward their certification.

The profile of the Candidates varied across the HDET pilots.

The table below provides more information on the HDET pilots.

Table 7.1 – HDET Pilots: dates, locations, and participants

Location BC Institute of

Technology

Vancouver

Thompson

Rivers

University

Kamloops

Okanagan

College

Kelowna

Northern Lights

College

Fort St. John

Vancouver

Island

University

Nanaimo

Dates May 3-4, 2014 May 31, 2014 June 1, 2014 June 7-8, 2014 June 14, 2014

# of Assessors 5 2 2 3 Pilot Cancelled –

No Candidates # of Candidates 4 2 2 4

Two of the 12 HDET Candidates did not pass the practical assessments. The first Candidate who did not pass was a Truck

and Transport Mechanic and was not familiar with hydraulics. The second Candidate who did not pass had language

challenges as English was not his first language. Both of these Candidates were approved to write the HDET IP Exam but

have not yet written it.

7.3.7 HDET Moderation Session

The HDET Moderation was conducted over two days on July 5 and 6, 2014 at the ITA Richmond Office. The objectives

were to:

� Review assessments performed and discuss each as a case study to evaluate for validity, consistency and fairness


March 2015

29

� Discuss challenger performance and assessor marking to reach consensus on consistency in marking to ensure inter-rater reliability of judgments

� Incorporate any required changes into tasks, procedures, and supporting documents.

Eight assessors participated: 4 from the Lower Mainland (2 Instructors and 2 from Industry), 1 from Nanaimo (Associate

Dean), 1 from Kamloops (Industry), 1 from Kelowna (Instructor – now retired), and 1 from Dawson Creek (Instructor).

Seven ITA employees attended: 2 Program Development Officers (PDO) (1 facilitating), Manager of Program Standards,

Manager of Assessment, Manager of Policy & Research, ECP Project Lead, Independent Consultant.

The planned interview role-play on the agenda was scrapped when it was determined that the interview script was not

designed to adequately evaluate scope of work.

7.3.8 Findings

The following are key findings during pilot activities:

Assessors

� The importance of having the Assessors review their pilot documentation and training material prior to the practical assessments was verified.

� The Assessors would like a more robust marking guide with the measures of success for performance of practical tasks. Further investigation is required to determine what this requirement might be.

� There is a need for Assessors to confirm that the Candidate has heard and understood the instructions correctly. � The Assessors would like practical assessment guidelines including:

- Briefing the Candidates on practical assessment tasks and expected performance; - Candidates’ questions and requests for assistance;

Assessment Tools:

� Required simplification/clarification/addition of wording

� Added boxes, columns, headers, partial reformatting

� Confidentiality agreement could be incorporated into the Assessor’s Practical Assessment Task Summary.

Shop Tools and Equipment:

� Tool list for Candidates required improvements. � The need was identified to have the Venue prepare a toolbox containing all the tools for the practical assessment. � Include additional tools required for the practical assessment and tools that are not required for the practical

assessment and incorporate tool selection into the marking guide. � Have the Venue provide the manuals for the practical assessments in both hard copy and electronically and have

them readily available for the practical assessment.

7.3.9 Conclusions:

� A decentralized practical assessment model with multiple colleges across the province requires significant time and resources to coordinate and manage.

� An increase in coordination time increases the total assessment costs, and, therefore a Candidate’s practical assessment fees.

� It is recommended to validate tested tools, forms and operational model with broader industry representation prior to launch.

� The model demonstrates low to moderate level of readiness for full implementation.


March 2015

30

7.4 Metal Fabricator Practical Assessment Pilot

7.4.1 Purpose

The goal of the Metal Fabricator Practical Assessment pilot was to:

• Test and analyze the assessment venue model; • Test practical assessment tools; • Conduct a two-day Assessor orientation session including two assessment simulations where trainees acted as

candidates and evaluated one another’s performance; • Conduct two practical assessments on Candidates; • Confirm Assessors’ readiness to conduct assessments to ITA standards; • Record all findings and suggest changes to prepare the tools for full implementation; • Gather feedback from the Assessors and the Candidates.


The Metal Fabricator practical assessment pilot was built on the Centralized Assessment Venue model.

Below is the description of how the model worked and the roles and responsibilities of participating parties.

ITA:

1. Selected the venue, developed and signed the assessment venue agreement. The agreement outlined all required conditions, including insurance, and deliverables.

2. Identified and selected Metal Fabricator Assessor trainees. Assessor trainee selection was based on the Assessor qualifications established by the industry and approved by the ITA.

3. Determined the Assessors’ pay rate based on industry consultation. A $75/hour rate was established for the Metal Fabricator Assessors.

4. Hired Assessors on contractual terms to conduct assessment and complete all relevant activities. 5. Identified and confirmed pilot candidates and coordinated all assessment preparation and logistics. 6. Prepared and conducted the Assessor, Candidate, Moderation and Venue Debrief sessions.

This model requires the direct involvement of ITA staff and/or consultants in the assessment preparation and coordination.

Piping Industry College of BC (PIC BC):

1. Prepared assessment lab and classroom, including required tools and equipment set up. 2. Ordered and delivered consumables according to the assessment requirements. 3. Arranged on-site First Aid. 4. Assigned First Aid and emergency contact for the time of the assessment. 5. Provided access to the facilities during the assessment. 6. Attended debrief meeting and provided feedback and findings.

7.4.3 Assessment Lab, Tools and Equipment, Consumables

The Assessment lab was set-up as per practical task requirements. Lab set-up and teardown requires between four and six


March 2015

31

hours to set up for two candidates. Having a dedicated shop would be more effective for future operations. However, if the

number of the Challengers is low, the shop cannot be dedicated to challenge assessments due to insufficient use of

valuable space and equipment.

The tools and equipment for mandatory and elective tasks were provided by the venue and met assessment requirements.

Consumables included steel beam, plates, and cardboard/paper, cost approximately $175 per Challenger including delivery

cost. Consumables need to be ordered and delivered to the venue one or two days prior to the assessment.

The venue used during the pilot project has a strong internal security system with restricted access to labs and classrooms.

Should the same venue be used in future assessments, UAPIC BC will need to determine a more efficient way to provide

access to Assessors, Candidates and ITA support staff to the operative classroom and labs.

7.4.4 Assessment Tools

The following practical assessment forms were used during the Metal Fabricator practical assessment pilot:

1. Assessor’s Practical Assessment Task Summary: 2. Confidentiality Agreement and Code of Conduct 3. Assessor Information Sheet 4. Marking Guide 5. Candidates’ Work Order. 6. Technical Drawings.

7.4.5 Assessor Orientation Session

The assessor orientation session was conducted on November 17 - 18, 2014 and included an overview of project

background, discussion of project goals, introduction to assessment rules, preliminary review of tasks and forms, shop tour

and safety talk, and practical assessment role-play. The ITA staff observed the practical assessment role-play and an

independent consultant and the findings on process improvements were recorded.

Session attendees included two Assessor trainees, ITA Manager of Assessment, ITA Manager of Policy and Research, ITA

Assessment Coordinator, and an independent consultant.

The two Assessor trainees who participated in the orientation session trialed all assessment tasks – one mandatory in the

shop and three electives in the classroom – and demonstrated strong knowledge of the tools and good understanding of the

protocols. Based on the trials they confirmed that there should be sufficient time for challenge candidates to complete the

tasks, and made the following suggestions for revisions to the tools to clarify and streamline the assessment/evaluation

process:

1. Include the following sections in the Practical Assessment Tool: a. Confidentiality agreement b. Candidate orientation checklist c. Assessment task description d. Mandatory and elective task combination options e. Performance criteria for each task f. Assessment summary sheet


March 2015

32

g. Decision whether or not candidate performance met the approved standard h. Suggestions to ITA based on outcomes

2. Revise the mandatory task performance criteria

a. Remove most of the operational process criteria as per findings b. Incorporate safety criteria c. Establish ‘Set up an oxy-acetylene unit’ as a ‘pass/fail’ criterion d. Establish ‘Beam measurements’ as a pass/fail criterion

3. Revise the elective tasks performance criteria

a. Remove the operational process criteria b. Include measurements established during the pilot c. Include photographs of process and product at specified junctures

4. Redesign the Practical Assessment Tool:

a. Use A4 format b. Consider the development of an electronic PDF version for record keeping and reporting efficiency c. Include ITA assessment staff contact information

The Assessor orientation session, as mentioned earlier, included a brief overview of the assessment rules and

requirements. It is recommended to conduct additional Assessor training on the ITA assessment administration and

reporting procedures to ensure that assessment activities are aligned with current ITA policies. If interviews are incorporated

into enhanced challenge protocols in future, training will be provided for conducting interviews and scoring responses.

7.4.6 Candidate Assessment Session

The goals of this session were to:

1. Assess two Candidates using the revised tools. 2. Confirm appropriate time allowed to complete mandatory and elective tasks. 3. Confirm appropriate time allowed for full assessment, including lunch and breaks. 4. Verify if tasks’ complexity reflects requirements (i.e. includes key competency areas) for evaluating candidate

performance against approved performance standards. 5. Monitor assessment processes and record recommended changes. 6. Gather Candidates’ feedback and suggestions.

The Candidate assessment session took place on November 20, 2014. The attendees included two Candidates, two

Assessor trainees, ITA Manager of Assessment, ITA Manager of Policy and Research, ITA Assessment Coordinator, and

an independent consultant.

Of the two Candidates: one holds a Metal Fabricator Certificate of Qualification and is interested in writing the Red Seal

examination, the other wrote but did not pass the IP/Red Seal exam.

Both Candidates completed mandatory and elective tasks within the allowed time. One Candidate completed the mandatory

task in two hours, the other in two and a half hours.

Candidates worked on different elective tasks. One Candidate completed the task within 1 hour and 30 minutes, the other

completed the within 1 hour and 45 minutes. Time allowed was two hours.


March 2015

33

The Assessor trainees assessed Candidates’ performance using the revised assessment criteria developed during the

Assessor preparation/training session. The Assessor trainees recorded assessment results in the Assessor’s Practical Task

Summary form; they also took photos of the Candidates’ work to support assessment records. Both Candidates met the

criteria for passing the practical assessment.

The session was completed on time; the estimated time for the practical assessment – seven hours – was accurate.

Candidates participated in the assessment follow-up discussion and provided their feedback and comments on the

assessment process and tools. The conversation included the following questions:

1. How do you feel about the practical assessment component in the challenge process? Both candidates were positive about the practical assessment as part of the challenge process

• “I feel like a real metal fabricator” • “This is what we are asked to do in the workplace”

2. What do you think about the assessment venue (location, lab set up, classroom set up)? • No problem finding the venue; • No problem with parking; • Lab and classroom were good and served the purpose of the assessment.

3. Did you find the tasks too hard/ or too easy?

• The tasks were just right; • Mandatory task is designed to use core fabricating skills; • Elective tasks require refreshment and that is good.

4. What is your overall impression about the assessment? • It was interesting to participate in the process.

5. Other suggestions included: • It would be helpful to have assessment information such as task description, time and fees online; • Including FAQ section would be helpful; • Reference to learning materials will help prepare for the assessment.

7.4.7 Moderation Session

The moderation session took place on November 27, 2014 from 9 a.m. until 1 p.m. at UAPIC BC in Delta, B.C. The

attendees included two Assessors, the ITA Manager of Assessment, ITA Manager of Policy and Research, ITA Assessment

Coordinator and an independent consultant.

The purpose of the moderation session was to:

1. Briefly describe the moderation and its role in the enhanced challenge process. 2. Review and discuss pilot results. 3. Review and validate recommended changes to the tools and forms. 4. Gather feedback and findings on assessment implementation.

One of the Assessors and the consultant met prior to the session to review changes to the practical tasks and revise the

assessment tool accordingly. These changes were incorporated into the Assessor Practical Task Summary form and

presented to all session attendees for review and further comments.


March 2015

34

It is important to remember that the moderation session focused on the practical assessment only and did not include a

review of other enhanced challenge tools and forms (e.g. self-assessment, employer declaration).

The moderation session resulted in the following:

1. Confirmed standard of performance for mandatory and elective tasks. 2. Confirmed evaluation criteria for mandatory and elective tasks. 3. Noted suggestions to make changes to the Assessor Practical Task Summary form as per 1 and 2 (detailed

described in the Assessor session section). 4. Established the time to complete full assessment as seven (7) hours:

- Four (4) hours for mandatory task - Two (2) hours for each elective task - 30 minutes for lunch - 30 minutes for breaks - Total time: seven (7) hours;

5. Suggestion to increase the Assessor to Candidate ratio to 1:2 if possible. Further evaluation of this proposition is required;

6. Assessor trainees completed the Assessor Practical Task Summary form but did not complete the tasks evaluation summary and recommendation to certification sections. These sections require further changes. Note, if the IP is still a certification requirement, the Assessors will not be performing a recommendation for certification based on practical assessment results unless the IP has already been written and passed by the candidate.

7. Both Candidates passed the practical assessment as per new criteria. 8. The final version of the tool is to be developed and circulated to the Assessors and other SMEs who participated in

the development process for review and validation.

7.4.8 Assessment Venue Debrief

The venue debrief session took place on December 4, 2014 from 10 a.m. until 12 p.m.

The purpose of the debrief was to:

• Review and evaluate the pilot under the Assessment Agency model; • Gather venue’s (PIC BC) feedback on assessment set up and operations; • Discuss and evaluate cost efficiency; • Discuss venue’s interest in conducting future practical assessments; • Discuss the approximate number of assessments per year and confirm venue’s capacity; • Determine a necessary minimum number of assessments per year; • Discuss the Assessor and the Assessment Venue roles and responsibilities.

7.4.9 Venue

1. The venue met the criteria for conducting the practical assessment. 2. The venue prepared the assessment lab, classroom, tools, equipment and materials as per assessment agency

agreement. There is no dedicated shop for conducting Metal Fabricator practical assessments at this time. Establishing a permanent or dedicated lab could be considered if assessment demand/volume warrants.

3. One of the Assessors was an UAPIC BC instructor. In the future, school instructors might not be involved in the assessment process since under the Assessment Venue model the ITA selects and hires Assessors. There are some conflict of interest restrictions on which candidates instructors can assess. A number of instructors are ITA-registered assessors for other trades programs.


March 2015

35

4. Consumables (steel beam, set of steel plates, card board) were ordered and delivered in advance. Consumables are not reusable and need to be recycled after the assessment. Recycling will require additional time.

5. Shop rental rate includes the costs for required tools and equipment. 6. Should the model be used in the future, further discussion on lab set up, roles, responsibilities and costs will be

required.

7.4.10 Costs

1. Assessor rate: a. $75/hour for ten (10) hours = $750 (based on seven (7) hours for the assessment time, and three (3)

hours for preparations and marking). b. The assessor wage rate for the pilot was $75/hour. It became evident that the $75/hour rate and 1:1

Assessor to Candidate ratio results in high assessment costs and associated fees. To address high assessment costs it was recommended to change the Assessor to Candidate ratio from 1:1 to 1:2.

2. Shop rent - $400/day. 3. Classroom rent - $200/day. 4. Consumables - $150/person. 5. Delivery of materials (five (5) sets) - $125. 6. Shop set-up - $250. 7. Shop teardown - $150.

a. As mentioned earlier, a shop could be dedicated if a sufficient number of assessments is scheduled on a monthly basis. If the assessment numbers are low, the shop set up and tear down time will be included in the assessment costs. This will increase assessment fees.

b. Currently, approximately 20 – 30 Candidates apply to challenge the Metal Fabricator certification per year (TBC with ITA). This would not justify a dedicated space.

8. Total estimated cost per Candidate is $1,450 based on 1:1 Assessor to Candidate ratio used during the pilot. 9. The number of challenge Candidates per month is important; it will impact the development of the ITA enhanced

challenge operational and business models.

7.4.11 Findings

The table below summarizes key variables tested in the Metal Fabricator pilot, as well as findings and suggestions. The

suggestions are based on pilot findings and feedback from Candidates, Assessors, and the Venue, collected during the

candidate follow up, moderation and assessment venue debrief sessions respectively.

Table 7.2 – Metal Fabricator Pilot Summary

Criteria Findings Suggestions

Number of

Assessors

2 N/A

Assessor to

Candidate Ratio

• 1:1 ratio did not prove cost effective; The ratio can be reduced to improve Assessor time.

• Change from 1:1 to 1:2; This change will help use Assessor time more effectively and reduce assessment costs and Candidate fees.

Assessor Rate • $75/hour used in the pilot • Review Assessor rate and discuss possible hourly rate reduction or establish per diem rate in the range of $450-$500/day


March 2015

36


Estimated Cost • Cost ($1,450 per Candidate) • is high and may create barriers

to certification.

• Change Assessor to Candidate ratio from 1:1 to 1:2; Determine if Assessor rate can be reduced;

• Determine if group assessment is possible for elective tasks and how it can help reduce Assessor time.

Operational Model • Centralized. • Moderate effort.

• This model is difficult to implement if the number of challenge Candidates is small;

• Look into other models (Third Party, College Practicum) and evaluate cost effectiveness;

Tools and Forms • Practical assessment tool required significant changes and undergone revisions;

• Other tools and forms were not tested.

• Complete final revisions to the practical assessment tool as per pilot findings and validate with a boarder SMEs/industry group;

• Review and validate other assessment tools and forms as per decision to implement Metal Fabricator challenge pathway.

Level of Effort • Substantial - Required significant coordinating time.

• Review other operating models for cost effectiveness.

Assessors • Demonstrated skills to become ITA Registered Assessors;

• Required more in depth knowledge of ITA protocols.

• Assessors to attend ITA Assessor training to learn assessment recording and reporting protocols.

• Develop Assessor guide that outlines assessment process, Assessor roles, and responsibilities and provides samples of assessment forms to be completed.

Candidates • Were engaged in the process; • Supported the enhanced

challenge process; • Confirmed that practical tasks

cover core skills and competencies;

• Required more information on the practical assessment process.

• Develop online challenge guide, including specific information on practical assessment;

• Develop FAQ, resources that help candidates prepare for the challenge process.

7.4.12 Conclusions:

� A centralized practical assessment model (one college) with requires less time and resources to coordinate and manage than decentralized.

� A centralized model requires high number of challengers to be feasible. � Assessor rate ($75/hour) is high; it increases the total assessment costs, and, therefore a Candidate’s practical

assessment fees. Review and adjustment is required to ensure that fees do not create barrier to certification. � It is recommended to validate tested tools, forms and operational model with broader industry representation prior

to launch. � The model demonstrates moderate level of readiness for full implementation.


March 2015

37

7.5 Millwright Practical Assessment Pilot

7.5.1 Purpose

The purpose of the Millwright Practical assessment pilot was to:

1. Complete the development of the Candidate Handout (work order and technical drawing) based on the practical assessment tools developed by the industry.

2. Conduct Assessor orientation session including review and revision of the practical assessment tool. 3. Conduct practical assessment for two Candidates, test and revise practical assessment tools and work order as

per pilot findings. 4. Provide suggestions on effective business model, estimate assessment fees, suggest improvements.


The Industrial Mechanic practical assessment pilot was built on the “Assessment agency” or Third Party model. This model

is not new to the ITA; it has been effectively used for conducting practical assessments for a number of trades and

occupations including Cooks, Shipyard Laborers, Climbing Arborists, Crane Operators and Parks and Public Works

Operations Workers. Although the model was proven successful for some of these trades and occupations, it needed

further testing to ensure its suitability for conducting practical assessments for the five chosen trades.

Five potential assessment agencies were identified and invited to respond to the request for proposal (RFP) to conduct a

practical assessment pilot for the Industrial Mechanic trade. These agencies were:

1. Kwantlen Polytechnic University. 2. Industrial Marine Training and Applied Research Centre (IMTARC). 3. Christian Labour Association of Canada (CLAC). 4. Fulford Certification. 5. Millwrights Union, Local 2736.

One of the agencies, Fulford Certification (who act as an Assessment Agency for ITA Crane and Funeral Director trades),

provided their response at a later date and was awarded the contract.

7.5.3 Venue

Vancouver Pile Driving Ltd’s welding and fabrication shop in North Vancouver was selected as an assessment venue.

The shop provided flexible working space and access, as well as the required materials and tools.

The venue tour and safety orientation were provided to the assessment team and ITA Assessment staff observers prior to

assessment day.

7.5.4 Practical Assessment Tools

The pilot identified that the original assessment tool was more suited for conducting assessments in a standard Millwright

college lab and required significant resources (tools, equipment, materials, lab space).

The suggested assessment tool did not meet the requirement that all practical assessments need to be universally

applicable. This requirement ensures that assessments can be conducted with the same criteria across various work place


March 2015

38

settings and are fair for all candidates regardless of their work location or residence in the province.

As the result of the pilot, the Millwright practical assessment tool has undergone significant revisions and resulted in the

following:

1. Modified practical assessment tool that can be used in an industrial setting across the spectrum of users of the trade.

2. Completed Candidate Handout including work order, drawing and task instructions. 3. Portable assessment kit that allows conducting assessments in workplace settings across the province.

The following modification were completed and tested during the pilot:

1. Use Precision Measuring Tools (Mandatory) – 45 minutes were deemed the maximum allowable time for this task.

2. Mount and Remove Bearings (Mandatory) – 90 minutes were deemed the maximum allowable time for this task.

3. Perform Shaft-to-Shaft Rough Alignment (Mandatory) – 45 minutes were deemed the maximum allowable time for this task.

4. Drill and Tap Holes in Steel Plates (Elective) – 45 minutes were deemed the maximum allowable time for this task.

5. Assemble a Hydraulic Circuit (Elective) – 60 minutes were deemed the maximum allowable time for this task while using computer based hydraulic simulators.

6. Total assessment time for three mandatory and one elective task – 3.5 hours. This is far below the estimated 8 hours.

7.5.5 Candidate Assessment Session

The assessment took place in the Welding Shop of Vancouver Pile Driving on January 24, 2015. The services of two Red

Seal Millwright assessors had been secured – one from the original development team (Al Story) and the other, his

colleague (Chris Mellors).

Two Challengers from the provided list volunteered their time to help pilot the practical assessment. One challenger was a

senior maintenance supervisor at a waste processing industry in Vancouver, while the other was in charge of maintenance

for a Lower Mainland municipality aggregate plant.

The assessment proved to be a viable and effective way of determining workplace performance to a standard.

One candidate met standards as per the pilot’s requirements, while the other candidate did not meet the required standards.

7.5.6 Findings

The pilot resulted in an effective and valid assessment process supported by a tested assessment tool and Candidate

handout. In addition, the pilot demonstrated that the assessment could be conducted anywhere in the province by a skilled

Assessor using a portable assessment kit.

The Third party model proved to be cost effective. The model offers flexible assessment delivery and stronger employer

engagement in the assessment process.

The assessment tasks chosen by the development team and modified by the practical assessment pilot team cover core

areas of the Trade and are simple to coordinate.


March 2015

39

Having the practical component in the challenge would ensure that a higher level of industry-defined skill is required before

Certification is offered. Finding a cost effective, simple way to ensure a higher skill level for challengers is a strong outcome

of this pilot.

Table 7.3 – Millwright Pilot Summary


Number of

Assessors

2 N/A

Assessment time 3.5 hours (below est. 8 hours)

Assessor to

Candidate Ratio

1:1 proved to be effective 2 assessments per day

Assessor Rate $500/day -

Estimated Cost $900 (below est. $2,000) Further discussion on pricing model with ITA.

Operational Model • Third Party • This model is flexible. Determine if assessments can be delivered across the province.

• Identify potential designated employers – “ITA Millwright Assessment Employer”

Tools and Forms • Practical assessment tool was modified for use in an industrial setting.

• Portable Assessor toolkit was assembled for the pilot and tested.

• Other tools and forms were not tested.

• Complete final revisions to the practical assessment tool as per pilot findings and validate with a broader industry group;

• Review and validate other assessment forms as per decision to implement Millwright challenge pathway.

• Confirm portable Assessor toolkit efficiency for the challenge process.

Level of Effort • Moderate

Assessors • Demonstrated skills to become ITA Registered Assessors;

• Required more in depth knowledge of ITA protocols.

• Assessors to attend ITA Assessor training to learn assessment recording and reporting protocols.

• Develop Assessor guide that outlines assessment process, Assessor roles, and responsibilities and provides samples of assessment forms to be completed.

Candidates • Were engaged in the process; • Supported the enhanced

challenge process; • Confirmed that practical tasks

cover core skills and competencies;

• Required more information and support on the challenge process.

• Expressed lack of satisfaction with the current challenge process.

• Develop online challenge guide, including specific information on practical assessment;

• Develop resources that help candidates prepare for the challenge process.

• Review gap analysis feedback for the written and practical components.


March 2015

40

7.5.7 Conclusions:

� A Third Party practical assessment model required the lowest time and resources to coordinate and manage than college based models (Decentralized and Centralized).

� Venue rate and Assessor rate per diem are more manageable and cost effective. � It is recommended to validate tested tools, Assessor toolkit and operational model with broader industry

representation prior to launch. � The model demonstrates high level of readiness for full implementation.


March 2015

41

8 Summary of Findings – Enhanced Challenge Pathway Completion Project

The findings from the enhanced challenge pathway projects are organized into tables (Table 8.1; Table 8.2; Table 8.3).

Each table provides description of critical findings for each stage of the project.

Table 8.1 - Feedback on the Current Challenge Process

Positive Issues

• Sufficient amount of working hours • Written tests required knowledge

• Lack of testing of the full scope of the trade • Lack of verification of practical experience within the scope of the

trade • Written exam poses barriers to some • Written is not sufficient to evaluate practical skills • Lack of feedback on knowledge/experience gaps

Table 8.2 - Feedback on the Practical Assessment Component in the Enhanced Challenge Process

Positive Issues

• Practical is viewed as strong mechanism to evaluate a candidate’s competency and verify submitted work experience

• Practical increases employers’ confidence in the skills of candidates who achieved certification through challenge rather than apprenticeship

• Practical attracts experienced workers, strengthens the challenge process and improves challenge outcomes

• As an enhanced assessment method, practical is considered stronger than technical interview

• Non-regulated trades and lack of support from employers • All five trades require highly diversified skill set that is difficult to

measure in a 1-day assessment • Practical assessment tasks need to include core trade skills • Practical assessment fees and affordability for unemployed

workers • Education and engagement of apprenticeship stakeholders to

ensure practical assessment is well designed and understood • Challenger education and preparation will require additional

resources.

Table 8.3 – Enhanced Challenge Pathway Operational Models

Criteria Decentralized (HDET) Centralized (MFAB) Third Party (IM)

Time to Coordinate

Candidates

High High Moderate

Time to Coordinate

Assessors

Moderate/

Contract with ITA

Moderate/

Contract with ITA

Low/Contract with Third

Party

Venue Cost Lab rate Lab rate Negotiable/Possible

industry in kind contribution

Assessor Rate Hourly - $55/hr Hourly- $75/hr $500 per diem

Assessment Time 8 7 3.5

Venue College Lab College Lab Worksite(s)/ across B.C.

Assessor/Candidate Ratio 1:1 1:2 recommended 1:1/2 candidates per day

Flexible Delivery No No Yes


March 2015

42

Readiness for

Implementation

60%

TBC w/ITA

75%

TBC w/ITA

85%

TBC w/ITA

Based on the above-presented analysis the Millwright pilot provides a basis for launching practical assessment for the

Millwright Trade.

Other two trades tested in the pilots require more time to finalize operating model, costing, number of challenge Candidates

per year, and Assessor readiness and availability.

8.1 Practical Assessment Pilots – Candidate Results

The table below describes the qualifications of the pilot Candidates and summarizes the results they achieved during

practical assessments for their trade.

Table 8.4 – Practical Assessment Pilots: Candidate Results

Trade

School

Location Candidate

Name

Employment

Status

At time of HDET

Pilot

Pilot

Results

Comments

BCIT Vancouver LC Truck & Transport

Mechanic

Approved to write

HDET IP Exam; wrote

2 times and failed.

Did Not

Pass

MS Automotive

Service

Technician at

Airport

Approved to write

HDET IP Exam

Pass

RB Unknown Approved to write

HDET IP Exam; wrote

2 times and failed.

Did Not

Pass

English not

first language

SD Automotive

Service

Technician

Approved to write

HDET IP Exam

Pass

NLC Fort St.

John

MH Industrial

Mechanic

(Millwright)-

deregistered

apprentice;

HDET apprentice

Pass Physical

Disability

MS 4th year HDET

apprentice;

certified at the

time of the Pilot

Pass

KH Automotive

Service

Applied to challenge

in May 2014

Pass


March 2015

43

Technician

DL HDET apprentice Pass

Okanagan

College

Kelowna AB HDET apprentice

- (deregistered as

an HDET

apprentice)

Approved to write

HDET IP Exam

Pass

KM HDET apprentice Pass

TRU Kamloops CR Truck & Transport

Mechanic -

Certified

Pass

RC Outdoor Power

Equipment

Technician

Pass

PIC BC Vancouver EQ Unknown Approved to write

Metal Fabricator IP

Exam; written and

failed the exam with

58%

Pass English not

first language

PD UAPICBC/

Shop staff

Completed

apprenticeship in

19??(TBC)

Not a challenger

Pass

Van Pile North

Vancouver

EK Journeyman

Machinist. Senior

maintenance role.

Certified Machinist.

Approved to write

Millwright IP Exam;

wrote on July 10,

2014 (53%)

Did not

pass

KW Senior

maintenance

supervisor.

Approved to write

Millwright IP Exam;

wrote and failed on

July 23, 2014 (64%)

Pass

16

Candidates

13 pass/

3 didn’t

pass


March 2015

44

9 What Worked? What Did Not Work?

9.1 What Worked?

� Phase II Project Team, comprised of ITA staff and an independent consultant, had good team dynamics, energy, commitment, experience and knowledge.

� Engaging an independent consultant to assist with project management and delivery allowed for more flexibility and efficient use of internal and external resources.

� Small-scale pilots were effective in testing assessment tools and making necessary adjustments. The revised tools will help streamline the assessment process, decrease assessment time and costs.

� The pilots were effective in testing three operational models. � Piloting two trades, not five, at the same time worked well and provided more time to complete required changes

and adjustments to the tools. � Call for Proposals process was useful in identifying organizations that currently have the capacity to deliver

practical assessments as per ITA established standards. � Assessor contracts (used for Decentralized and Centralized models) were useful and helped specify the scope of

work for each pilot. � Focusing Assessor recruitment for the Millwright and Metal Fabricator pilots on both industry and post-secondary

representatives worked well. Millwright Assessors from the industry contributed to the development of the portable Assessor Toolkit.

� Industry interviews, conducted over the phone, had a positive impact on industry engagement and participation. � Industry interviews provided valuable information and feedback on the current challenge process and ways to

improve it. � Industry interviews helped determine specific demands and preferences for practical assessments. � Industry interviews helped prove the Enhanced Challenge Pathway concept. � The Centralized model proved practical and efficient; it needs costs adjustment to make it more cost effective. � The Third Party model proved to be practical and the most cost effective. � Phase II project plan was clearly defined and all project goals and objectives were achieved as planned. � Assessor qualifications that were developed and approved by the industry worked well and helped select qualified

Assessor Trainees who were instrumental during the pilot trial and validation. � The Millwright team felt that having Assessor Candidates from the industry was particularly useful for adapting

practical assessment tools to a shop environment. � Assessor peer assessments were useful in making continuous revisions and modifications to the tools during the

trial process. � A follow up review of the modified tools with Assessors and project consultants who are familiar with the

assessment tools development process helped make final changes to the tools (Metal Fabricator and Millwright pilots).

� The Assessor Marking sheet worked well. However, the format (size, boxes) of the document was not easy to use during practical assessment; marking sheet requires redesign and improvement to reflect pilot modifications to the tools.

� Industry questionnaire covered key areas of the pilot study. The questionnaire could be used for a broader industry survey if further data validation is required.

� One-day Assessor orientation session including a brief overview of practical tools worked well for the pilot. There is no need for detailed training on the tools prior to the pilot since tools are being revised during the pilot. Once assessment tools are tested and confirmed, a more detailed Assessor training session is required.

� Venue for the millwright pilot – medium size industrial shop – worked well. The pilot showed that practical assessment could be conducted in smaller millwright shops as well. Smaller shops will need to meet Venue criteria (e.g. tools, equipment, ventilation, safety).

� Venue for the metal fabricator pilot – training lab at PIC BC – worked well. Standard lab setting, low safety risks, availability of food, washrooms, study space, and staff’s familiarity with practical assessment procedures


March 2015

45

contributed to the pilot quality. Approved Venue criteria worked well. It is important to specify venue requirements prior to the pilot and ensure safety and insurance needs. Venue criteria should be reviewed during the pilot and revised if needed.

9.2 What Did Not Work?

� Lack of clarity provided to the ITOs on the enhanced challenge pathway framework, newly introduced assessment methods and their purpose for the challenge process. Lack of clarity provided to the ITOs on requirements for tool development resulted in tools that are not standard and difficult to manage within standard protocol.

� Tools developed by the ITOs and the SMEs are too complex and require significant time and resources to implement.

� Tools’ complexities significantly increase the cost of practical assessments and, therefore, challenge fees. Candidates may simply go to another jurisdiction to become certified.

� Piloting only the practical assessment is not sufficient for the full implementation of enhanced challenge pathway. There is a need to pilot other assessment tools and forms to ensure the quality and efficiency of the enhanced challenge process.

� Making practical assessment part of the challenge process for non-mandatory trades, such as Carpenter, may result in Candidates not seeking certification or getting Certification in other jurisdictions.

� Some tools, such as HDET scope verification interview, require substantial revisions to be used according to their purpose (e.g. scope verification rather than competency validation).

� Assessor trainees who participated in the pilots would need to complete full ITA Registered Assessor Training Course to ensure they have a strong understating of assessment tools and forms and can effectively manage all required protocols.

� Assessor recruitment for the HDET that focused on college instructors did not generate full support for the project, as the objective of the project is to certify Candidates outside the trade school/apprenticeship programs.

� Assessor recruitment for the HDET pilot did not consider Assessors’ and Venues’ location. Travel and accommodation costs increase the costs to the Candidate.

� Unforeseen circumstances (business and personal emergencies) delayed the implementation of the Millwright project. Consequently, the project deadline required extension.

� No guidelines on enhanced challenge pathway development and implementation processes resulted in lack of understanding about project goals and expected outcomes among ITOs. The guidelines would help align and project manage several projects with multiple stakeholders simultaneously.

� Practical assessment tools were designed to evaluate standard work procedures (process) and performance outcomes (product). This made the tools too complex. The need for these types of practical assessment tools was not clearly defined. Therefore, tools needed to be revised and simplified.

� Millwright practical assessment tool did not have a completed Candidate Handout. A Candidate Handout, including clear instructions on practical tasks and achievement criteria, was completed during the pilot.

� Small-scale pilot did not provide sufficient time for Assessors to properly work with revised practical assessment forms (e.g. marking guide, assessment record, assessment report) and learn standard ITA assessment protocols. Additional training is required.

� One elective task for the Millwright pilot – assembling hydraulic circuit – could only be realistically performed on a hydraulic training board in a College Lab. The pilot team suggested using computer based hydraulic simulators. Further examination of this option is required.

� Review and analysis of the enhanced challenge assessment documentation (tools and supporting forms) showed that the documentation’s format varies. That may create administrative inefficiencies in the future. Standard templates (need to be developed) would help with building standard enhanced challenge pathway administrative system and process.


March 2015

46

10 Conclusion

Since the inception of this project, a series of initiatives with overlapping scope have commenced. ITA is involved and fully

committed to the success of these initiatives and will ensure they benefit from any lessons learned from this project. As a

consequence, ITA will not be moving forward at this time with any of the recommendations flowing directly out of this

project.

Below is a description of two key national initiatives whose scope overlaps that of this current project:

10.1 National FQR Initiative

In September 2013, CCDA launched the FQR Initiative with the intent of creating an optimal process and standards

framework that is timely, fair, transparent and consistent across all jurisdictions for assessing the eligibility of internationally

trained workers to become certified in any one of the Red Seal trades.

A Project Task Team of representatives from six Canadian jurisdictions has been formed for this initiative. Representatives

are from the provinces of Alberta, British Columbia, Manitoba, New Brunswick, Ontario and Saskatchewan along with an

ESDC staff member. The Government of Alberta has received funding from ESDC to manage the CCDA FQR Initiative.

Ideal Process Model

As one of the first steps in the CCDA FQR Initiative, an Ideal Process Model was developed that assists in determining

eligibility decisions for the Red Seal Exam. The model is broken into five distinct phases:

1. Pre-application 2. Application 3. Application assessment 4. Competency assessment 5. Decision and communication points

Below is a graphical representation of the Ideal Assessment Process


March 2015

47

Projects Summaries:

Project #1: Application and Assessment Process project

Application and application assessment were identified as the best place to start strengthening the Ideal Process Model. A

contractor was hired in March 2014 and launched the Application and Assessment Process project by reviewing the

application documents and assessment processes used in each participating jurisdiction. The Application and Assessment

Process project produced a set of documents and processes that will increase consistency, transparency and fairness

across jurisdictions while respecting differences in standards and legislation. The work on the Application and Assessment

Process project wrapped up in December 2014.

Project #2: Application and Assessment Pilot and Implementation Planning Project

The Application and Assessment Process work continued in January 2015. In this project there will be pilot testing of the

revised application package in six jurisdictions across Canada. Once the outcomes of the pilots are evaluated, preparation

will be made for full implementation.

Project #3: Content for Web Based Resources Project

In January 2015 work also began to support the pre-application phase of the Ideal Process Model by identifying, gathering

and analyzing information and materials intended to populate several web based tools that are under development. These

tools include a website, an online self-assessment tool and an inventory of out-of-country credentials and training systems.


March 2015

48

The website will help internationally trained workers and employers navigate Canada’s foreign qualification recognition

systems and become better informed about working in the trades before submitting an application. The website will house

information, links and tools including self-assessment tools designed to compare an applicant’s experience against a trade-

specific Red Seal National Occupational Analysis. Content for the website will come from a combination of pre-existing and

new information and resources.

The inventory of out-of-country credentials and training systems will help jurisdictions review applications from

internationally trained workers in a more efficient manner. Details gathered could include descriptions of trades-related

training programs, regulatory bodies, credentials issued, sample documents as well as useful websites and contact

numbers. Content for the inventory will come from a combination of pre-existing and new information and resources.

10.2 National Harmonization Initiative In July 2014, the Forum of Labour Market Ministers (FLMM) recognized the need to enhance and improve apprenticeship in

Canada. Ministers agreed to a pan-Canadian approach to advance apprenticeship initiatives for implementation of the

harmonization of 10 Red Seal Trades in 18 months by Fall 2015.

The Canadian Council of Directors of Apprenticeship (CCDA) identified Harmonization as a strategic priority for 2013

through 2016 and has done extensive research and consultation with Industry and training providers nationally. The FLMM

has charged the CCDA with moving this project forward.

On February 27, 2014, the CCDA approved the following recommendations for the Harmonization project on 10 targeted

trades:

• Sequencing of technical training curriculum content and total training hours (in-school and on-the-job combined)

• Jurisdictional trade names and definitions;

• Use of the most recent national occupational analysis for P/T technical training curriculum development; and

• Specific to the Mobile Crane Operator trades, more consistent weight restrictions and equipment classifications.

These recommendations were subsequently confirmed with national industry and training stakeholders at consultation

meetings help in February and April 2014 as well as the CCDA National Stakeholder meeting in March 2014. Stakeholders

were supportive of work on harmonization and indicated their interest in continued engagement on the Initiative. Industry

also confirmed that sequencing of technical training was the key priority to enhance apprentice mobility.

In conjunction with the Forum of Labour Market Ministers (FLMM) and the Canadian Council for Directors of Apprenticeship

(CCDA), the ITA was tasked with providing a report that represented the consultations with BC’s Industry and Training

Providers. As shown in the National Ideal Assessment Process Model above, the work on competency assessments is

scheduled for after steps 1-3 are completed.


March 2015

49

10.3 Findings for National Initiatives

ITA will pass on the following 26 findings to the national initiatives. These are based on the outcomes from the four research

streams of this study: ECP project review, three practical assessment pilots, industry consultation update and environmental

scan.

The findings are organized by topics used in the industry interviews and are not listed in any priority order. Before pursuing

enhanced challenge for the five trades in this study, or other additional trades in the future, prioritization of these findings

would help determine short, medium and long-term priorities in order to build an effective enhanced challenge system.

The 2013 industry interview sample size for each trade group was small (5-8 interviewees). The reader should keep this in

mind when interpreting results and findings. Also, it is recommended that project findings be validated with Advisory Groups

and broader industry representation.

10.3.1 Challenge requirements

According to the industry interviews, the majority of respondents view the current challenge process as lacking rigor.

Findings for strengthening the challenge process include requiring additional proof of experience within the scope of the

trade, increased employer sign-off responsibility, assessment of core practical skills, completion of a college-based

practicum, and subject matter interviews. Additional findings include:

1. Developing an enhanced challenge application package that includes specified employer and statutory declarations. 2. Developing online self-assessment forms to familiarize candidates with challenge requirements and help determine

their skill set and appropriate challenge level. 3. Developing certification challenge guide for employers that outlines employer responsibilities in declaring and

verifying candidates’ working and on-the-job training experiences. 4. Determining trade(s) for practical assessment implementation (consider regulated and non-regulated trades,

number of challenge candidates per year, trade harmonization priorities).

10.3.2 Challenge processes, supporting tools and forms

Per the consultation findings, an interview with a subject matter expert and a practical assessment are viewed as two

additional methods to evaluate experienced trades workers’ competency. The practical assessment, as data shows, has

more value and weight for employers than the interview. The consultation confirmed that adding the practical assessment

component to the challenge process would help increase employers’ confidence in the skills of workers who achieved

trades certification through challenge. As such, findings include:

5. Including practical assessment in the challenge process and develop supporting forms. 6. Reviewing and validate revised practical assessment tools and protocols with Trade Advisory Groups. Finalize

accordingly. 7. Developing standard templates that can be populated with available trades-specific information. Support with

samples that contain instructions. 8. Reviewing and validate operating models with Trade Advisory Groups. Finalize accordingly.


March 2015

50

9. Conducting trade specific surveys on different practical assessment formats (college-based practicum, employer sign-off, worksite standardized assessment) to prove the concept with broader industry.

10. Developing new employer declaration forms by including more detailed information on the scope of the trade. 11. Increasing employer responsibility in signing off candidates’ industry experience by introducing “Sign-off Liability

Statement” or other relevant measures. 12. Developing challenge pathway process maps for candidates and employers to help prepare and manage enhanced

challenge processes. 13. Developing enhanced challenge guide for candidates and employers that outlines challenge completion

requirements and processes. 14. Developing online challenge preparation tools and resources (e.g. self-assessment checklist, competency profiles,

challenge guide for employers)

10.3.3 Costs

Findings regarding costing include the need to:

15. Discuss assessment costs determined through the pilots with potential assessment agency partners. 16. Re-assess assessor pay rates used during the pilots and determine a benchmark rate. Test to see if assessors will

agree to perform services at that rate. 17. Confirm assessor competency, training and currency requirements meet any policy guidelines and estimate costs of

providing the training. 18. Determine processes, estimate required internal and external resources for launching assessment for one trade.

10.3.4 Implementation

If implementing practical assessments, findings include:

19. Setting up practical assessment launch criteria and choosing one out of three piloted trades for implementation. 20. Analyzing effectiveness of operational models for each of the three piloted trades and developing ECP operating

manual(s). 21. Exploring industry proposed college-based practicum model and its potential for enhanced challenge pathway.

10.3.5 Candidates

Developing effective mechanisms for attracting challenge candidates is crucial for building skilled workforces in BC. To

ensure that the candidates from different groups of the province’s labour pool can apply, prepare for and complete the

challenge process, findings included:

22. Developing an online self-assessment tool that would help the candidates assess their skills and determine their readiness for certification challenge versus entering the apprenticeship training system or doing gap training.

23. Developing online challenge resources (e.g. process map, FAQ, training programs, courses and learning materials).

10.3.6 Assessors

Registered Assessors are an integral part of a practical assessment process. Assessors would need to complete a robust


March 2015

51

assessor training program, maintain currency in their trade and well as assessment methodology. As such, findings include

the need to:

24. Conduct ongoing assessor training, moderation, and professional development to ensure that the Assessors stay current on industry performance standards.

25. Build an assessor candidate pool.

10.3.7 Enhanced Challenge Pathway System

Enhanced challenge is intended to strengthen both the challenge process and the validity of its outcomes. The system

needs to provide an effective framework for evaluating trades equivalency of a broad pool of experienced trades workers

including those who gained their experience and education in and outside of Canada. The system needs to be efficient with

its resources, budget and time requirements to deliver clear benefits which outweigh costs and risks to all parties involved.

As such, findings include:

26. Determining core operational processes (e.g. assessment, training, evaluation, insurance, audit, appeal) and required resources.

ENHANCED CHALLENGE PATHWAY COMPLETION PROJECT

APPENDICES


March 2015

53

1. Appendix A. Project Methodology 54 2. Appendix B. Industry Interview Questionnaire 55 3. Appendix G. List of Interviewees 57 4. Appendix D. Carpenter Group Interview 58 5. Appendix E. Construction Electrician Group Interview 60 6. Appendix F. Metal Fabricator Group Interview 62 7. Appendix G. Millwright Group Interview 64 8. Appendix H. Heavy Duty Equipment Technician Group Interview 66 9. Appendix I. Industry Interview Summary 68 10. Appendix J. Summary of Registration Practices of Select Non-trades Occupations 70 11. Appendix K. ITA Registered Assessor Training 73


March 2015

54

Appendix A. Project Methodology

Table 3.1 ECP Project Methodology

Source Information collected

1. Comparative analysis – Review of Phase I ECP activities, processes and outcomes

Assessment tools and forms for five trades, their purpose and practical use, content, alignment with standards, benchmark for developing practical tasks, challenge process, new roles and responsibilities for conducting practical assessment.

2. Environmental scan – Review of application registration practices

Requirements for registration/licensure as engineers (APEGBC), technicians, technologists, (ASTTBC), and registered nurses CRNBC. Provisional certification practices. Registration requirements, registration/licensure processes for internationally trained professionals. Information and resources for internationally trained professionals.

3. Qualitative research - Industry consultation and key informant interviews

Perspectives of employer, labour and industry association representatives, human resources and apprenticeship program representatives on current challenge requirements and challenge process, practical assessment for challengers (Total number of interviewees=33).

4. Pilot study - Practical assessment pilots

Tools and forms design and content. Assessor readiness and training. Operational models, cost effectiveness, usability of assessment tools, relevance of practical tasks in evaluating candidate performance against approved industry standards, estimated assessment fees, continued process improvements.


March 2015

55

Appendix B. Industry Interview Questionnaire

Interview: Practical Assessment Component in the Certification Challenge Process

The purpose of this interview is to consult with industry on the importance/value of the practical assessment component in

the certification challenge process. In addition, this interview is intended to clarify how trade certification helps employers to

source, select and hire experienced tradespersons from a diverse pool of job candidates, including those who obtained their

trades work experience in, across and outside Canada.

Currently, in order to achieve trade certification through exam challenge, experienced workers must prove required working

hours within the scope of the trade and pass a multiple-choice exam (passing mark is 70% or higher).

I. Challenge Requirements

1. Challenge candidates for your trade must have completed a certain number of work based hours. In addition, they must have a past (or current) employer declare witnessing the candidate having performed a specific range of tasks (unique for each trade). Do you think this is enough or does it need to be strengthened?

2. Challenge candidates include representatives from the following groups: a. Experienced trades people who gained their work experience in B.C.; b. Experienced trades people who gained their experience in other Canadian provinces; a. Experienced trades people who gained their trades experience and education outside of Canada.

Should the certification challenge requirements for different groups be the same or different?

II. Enhanced Challenge Process and Components

3. Would adding a practical assessment as a mandatory component of the certification challenge process for your trade strengthen the process and/or increase your confidence in the abilities of workers who achieve certification through the challenge process?

4. How long do you think a practical assessment needs to be to determine a worker’s level of competence in your trade (e.g. to give you sufficient confidence in the challenger’s abilities to hire them?)

5. Do you think that requiring a practical assessment for certification challengers would deter appropriate candidates from challenging for certification in your trade?

6. Other than conducting a practical assessment, there may be ways that the challenge process for your trade could be strengthened. One way is by requiring candidates to be interviewed by a subject matter expert about their knowledge and experience. Do you think this could replace a practical assessment for your trade?

III. Challenge costs and value of certification

7. Enhanced assessments cost a great deal more to administer than machine-scored multiple-choice examinations. How much do you think a certification challenge candidate would be willing to pay for assessment (e.g. At what price do you think some or most candidates would decline assessment and not apply for challenge?) Price I think candidates would be willing to pay for enhanced challenge in my trade: _$____________

8. If the fee for assessment was higher than the cost you estimated above, what do you think the impact would be in your industry?

IV. Importance of Certification for your workers and in making hiring decisions

9. Do you hire experienced trades workers without formal Canadian credentials? (C of Q/ Red Seal)?


March 2015

56

10. At your firm, what is the optimal percentage of certified tradespersons in Carpentry? (e.g. is it a requirement for all the workers to be certified or are there lower-level jobs, which don’t require certification that make up a significant portion of your skilled trades workforce?)

11. It has been stated that employer confidence in skills and abilities of workers certified by exam challenge is lower than for those of workers who completed apprenticeships. What is your level of agreement with that statement? If you agree, how would the current challenge process have to change to increase your confidence in the knowledge

and abilities of candidates who achieved certification through challenge?

12. Can you please describe the process your organization uses for screening, selecting and hiring trades workers and what, if any, consideration is given to whether or not the applicant is certified in their trade?


March 2015

57

Appendix C. List of Interviewees

33 industry representatives total

The Carpenter Group (5)

1. Abigail Fulton, Vice-President BCCA, Victoria, BC 2. Eric Bell, Northern Legendary Construction, Owner, Charlie Lake, BC 3. Larry Richardson, Director of Training CLAC, Langley, BC 4. Randy Callaghan, Personnel Manager, PCL Construction, Vancouver 5. Steve Haab, Haab Home Construction, Owner, Fort St. John, BC

The Construction Electrician Group (8)

6. Andy Cleven, Electrical Training Joint Committee 7. Amy Davidson, HR Manager, CCT Controls 8. Carol Rycrofft/HR Manager, Superpoint Technologies Group 9. Kevin Shillito/Manager of BC Operations Superpoint Technologies Group 10. Abigail Fulton, Vice President, BC Construction Association 11. Cathy Thomson, Payroll Administrator, Status Electrical, Abbotsford, BC 12. Foreman (name TBC) Status Electrical, Abbotsford, BC 13. Ted Pimms, General Manager, Pimm’s Production Equipment, Fort St. John, BC

The Metal Fabricator Group (7)

14. Allen White, Fabrication Manager, Teck, BC 15. Dave Marcinew, Plant Manager, CANRON, Delta, BC 16. John Folkers, Fabrication Foreman, Weldco Beals, Langley, BC 17. Ian Baxter, Apprenticeship Manager, National Defense, Victoria, BC 18. Kevin Leyenhorst, Manager, Intercontinental Truck Body, TB 19. Mike Kuzinuk, President, Fabrite Services Ltd., Cranbrook, BC 20. Trevor Hill, Metal Fabricator Lead/Manager National Defense, Victoria, BC

The Millwright Group (7)

21. Al Storey, Millwright, Harmac Pacific, Nanaimo, BC 22. Chris Babcock, Manager Field Services, Service on Site, Fort St. John, BC 23. David Hiltz, Millwright, Catalyst Paper, Port Alberni, BC 24. Gordon Balfour, Millwright, Training Coordinator, Millwrights Union, Langley, BC 25. Mike Cass, Vice-President HR, Western Forest Products, Vancouver/Nanaimo, BC 26. Vern Phillips, General Manage HR, Catalyst Paper, 27. Richard Turnbull, Millwright and Machinist, FMF, National Defense, Victoria, BC

The Heavy Duty Equipment Technician Group (6)

28. Devon Lock, Mechanic Nels Ostero Sand and Gravel, Taylor, BC 29. Doug Richardson, General Manager, Inland Kenworth, Langley Branch, BC 30. Gordon Armour, BC Mining Task Force. 31. Ian Simpson HSE Manager/Edmonton Parts, Compression Technology Inc., Fort St. John B.C. 32. Richard Demchuk, Manager Human Resources Howe Sound Pulp and Paper Corporation 33. Alejandra Posadas, Apprenticeship Coordinator, Finning, Edmonton, AB


March 2015

58

Appendix D. Carpenter Group Interview

The group consisted of five respondents:

1. Abigail Fulton, Vice-President BCCA, Victoria, BC 2. Eric Bell, Northern Legendary Construction, Owner, Charlie Lake, BC 3. Larry Richardson, Director of Training CLAC, Langley, BC 4. Randy Callaghan, Personnel Manger, PCL Construction, Vancouver 5. Steve Haab, Haab Home Construction, Owner, Fort St. John, BC

1. Current challenge requirements. 4 out of 6 respondents think that the current requirements for the challenge are satisfactory.

4 out of 6 said that the challenge process needs to be strengthened by enhancing the experience

verification procedure and possibly including employer based practical assessments.

2. Challenge requirements for different groups. All six (6) participants agree that the challenge requirements for all individuals (with domestic or foreign

experience) should be the same.

3. Practical assessment as a mandatory challenge component. Respondents agree that the practical assessment will strengthen the challenge process and increase

employers’ confidence in Candidates.

There is some support for introducing the practical assessment as a mandatory component of the

challenge process; however, there are concerns about practical assessment scope, delivery format, and

industry and Candidate buy-ins.

A practical assessment on the employer’s site – Employer Sign Off – is preferable.

4. Practical assessment duration. The suggested duration for the practical assessment is from three (3) to eight (8) hours.

5. Practical assessment as a barrier. Some respondents think that the mandatory practical assessment will deter some Candidates. Others think

that the appropriate Candidate will not be discouraged from taking the practical assessment.

6. Interview as a substitute for practical. All view the Candidate Interview useful but not as a substitute for the practical assessment.

7. Practical assessment fee. Respondents think that the fee for the practical assessment should be in the range of $300-$500

8. Fee as a barrier. The fee should be reasonable.

If it is too high it will create barriers to certification.

*Note that current estimates of cost-recovery fee for practical assessment are significantly higher than the

range suggested by interviewees.

9. Trades workforce supply pool. Most (4 out of 5) employers hire Candidates without certification.

These Candidates come from the two groups identified in the interview: 1) with experience across Canada

and 2) with foreign experience and certification;


March 2015

59

10. Optimal percentage of trades certified workers. The optimal percentage of certified Carpenters in the companies represented by the interviewees varies

from 50% to 90% and depends on projects and operations.

One respondent was not sure about these numbers and said that the number of certified Carpenters in the

company is market driven – when the demand is high, employers are not looking for a certified Carpenter,

but a Carpenter who can complete a project on time.

11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates. Three (3) out of five (5) disagree that employers have lower confidence in the competencies of those who

challenged the certificate than those who completed formal apprenticeship. One (1) support that statement

and one (1) is not sure.

12. Certification and hiring process. The hiring process for both big and small employers is focused on finding the most experienced Candidate

for the role.

Although trade certification is viewed as valuable, years, breadth and depth of experience outweigh the

value of the certification.

Most of the employers provided various examples of their “star” performers who are not certified and

compared them with a Red Seal certified journeyman who completed a 4-year apprenticeship but knows

only the basics of the trades and the industry.

All employers agreed that the depth and the breadth of work experience is what makes a real journeyman

Carpenter.

A combination of certification and experience is what employers are looking for in the right candidate.

The hiring process for many employers is focused on determining practical skills and the scope of trade

experience.

Certification is valued but preference is often given to strong industry experience.

Canadian experience is generally valued higher.


March 2015

60

Appendix E. Construction Electrician Group Interview

Eight industry representatives participated in the interview (two companies provided two representatives each).

1. Andy Cleven, Electrical Training Joint Committee 2. Amy Davidson, HR Manager, CCT Controls 3. Carol Rycrofft/HR Manager, Superpoint Technologies Group 4. Kevin Shillito/Manager of BC Operations Superpoint Technologies Group 5. Abigail Fulton, Vice President, BC Construction Association 6. Cathy Thomson, Payroll Administrator, Status Electrical, Abbotsford, BC 7. Foreman, Status Electrical 8. Ted Pimms, General Manager, Pimm’s Production Equipment, Fort St. John, BC.

Interview summary is provided below:

1. Current challenge requirements. 3 out of 7 interviewees think that the current challenge requirement is enough.

4 think that it should be strengthened.

One suggests that the amount of required working hours could be increased to 10,000 hours.

One suggested five years of industry experience in Canada.

Other suggestions include Grade 12 diploma, completion of the Canadian Electrical Code course, and

completion of college training (e.g. practicum, level 4) similar to apprenticeship.

Great importance was given to the validation of experience within the scope of the trades.

2. Challenge requirements for different groups of candidates. Most (5) employers think that the challenge requirements for different groups need to be the same (hours,

education requirements)

Some (2) suggest that internationally trained professionals may need to go through a different process of

evaluation to ensure that their experience matches Canadian requirements.

A comment about trades training systems in immigrants’ home countries pointed out that while some

systems are similar to Canadian systems, others are significantly different.

One representative suggested developing a “preferred countries” list and streamlining the process for

Candidates from those countries by recognizing education and issuing “provisional certification.” In this

case, the next step could be job placement and employer sign-off. After that the candidate would receive

full certification.

3. Practical assessment as a mandatory challenge component. The idea of practical assessment as a mandatory component of the challenge process is perceived

positively.

Employers agree that this would help determine the Candidate’s level of expertise. Many want to see a

practical assessment that would help identify gaps, and help suggest further training and job placement.

There are a number of concerns about mandatory practical assessments, including the scope of trade,

assessment efficiency, format and costs.

4. Practical assessment duration. The suggested duration of practical assessment is broad – from six (6) to eight (8) hours for an

assessment that targets core competencies, to a two-month continuous assessment on the employer’s site

or at the college.


March 2015

61

5. Practical assessment as a barrier. Employers’ opinions on whether the practical assessment is going to deter appropriate Candidates vary.

Two (2) think that this may create unnecessary barriers.

Others think that it will not be a problem for the ‘appropriate’ Candidates.

Generally, those who deal with selection and hiring are more likely to see the practical assessment as a

barrier; the majority of operational representative do not see it as an obstacle.

The respondents prefer a school-based practical assessment such as the completion of a practical course,

practical modules, or projects.

6. Interview. All respondents agree that the interview is a valuable tool but it cannot replace the practical assessment for

this trade.

7. Practical assessment fee. The suggested fee for practical assessment is in the range of $350-$1000.

8. Fee as a barrier. In general, a fee higher than $1000 should not have an impact in the industry. However the fee structure

needs to be reasonable and comparable with trades training tuition fees.

9. Job candidates. As for the job Candidate pool, most respondents hire workers without Canadian credentials; most workers

have Canadian experience, followed by workers with foreign experience and credentials. Two respondents

said that they also hire workers with foreign experience and without credentials.

10. Optimal percentage of trades certified workers in a company. The optimal percentage of certified workers working for firms represented by the interviewees varies from

25% to 100%.

11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates The respondents’ support for the statement regarding a lack of confidence in the skills of Candidates who

challenged their certification differs.

Two out of seven disagree, one is not sure and the rest agree. The latter group suggests that adding a

practical test, interview, or completing training can strengthen the current challenge process.

12. Certification and hiring process. In recruitment, all of the employers focus on previous employment, direct industry experience, certification

and safety training.

Some employers include tests (e.g. math, codes, safety) in their hiring process.

All employers look for the right attitude and a good fit with company culture.

All employers put job candidates on probation (two weeks – three months) prior to offering full-time

employment.

Additional comments and suggestions include:

4. Develop strategies for attracting Electricians from the U.S. to meet growing demand for trained and certified electrical workers.

5. Consider a provisional certification for electrical workers. 6. Add college practicum to the certification challenge process as a substitute to practical assessment. This would

better align apprenticeship and challenge processes than a practical test for challengers. It would also increase employer confidence in Challengers’ skills and knowledge. A practical assessment delivered through a college would be more standardized and may have more value.


March 2015

62

Appendix F. Metal Fabricator Group Interview

Seven industry representatives participated in the interview:

1. Allen White, Fabrication Manager, Teck, BC 2. Dave Marcinew, Plant Manager, CANRON, Delta, BC 3. John Folkers, Fabrication Foreman, Weldco Beals, Langley, BC 4. Ian Baxter, Apprenticeship Manager, National Defense, Victoria, BC 5. Kevin Leyenhorst, Manager, Intercontinental Truck Body, TB 6. Mike Kuzinuk, President, Fabrite Services Ltd., Cranbrook, BC 7. Trevor Hill, Metal Fabricator Lead/Manager National Defense, Victoria, BC

The summary below outlines the key findings of the interview:

1. Current challenge requirements. Two (2) out of seven (7) respondents think that the current challenge requirements are enough. Five (5) out of seven (7) think that although the requirements are reasonable, there are many concerns about

experience validation and competency evaluation processes.

There is low trust in the current employer declaration – respondents think that it is too general and requires

more in depth description of the Candidate’s skills and experience.

Many believe that writing an exam is good, but not enough for evaluating or predicting candidate performance

to approved industry standards.

They suggest including a practical assessment, an interview with Challengers, and an interview with

employers to determine Challengers’ competencies and practical skills.

2. Challenge requirements for different Candidate groups. Participants think that the challenge criteria should be the same for all Challenger groups.

3. Practical assessment as a mandatory challenge component. Five (5) out of seven (7) interviewees said that a mandatory practical assessment would increase their

confidence in the skills of workers who challenged their Metal Fabricator trade certification.

4. Practical assessment length. The suggested length of the practical assessment component varies from four (4) hours to a 4-6 weeks

college-based practicum.

The college-based practicum is viewed as an opportunity to bridge the gap between the Challengers and

Apprentices. This type of practicum is viewed as more in-depth standardized industry training.

5. Practical assessment as a barrier. Those who supported the practical assessment as part of the challenge process do not think that it will deter

‘appropriate’ Candidates.

An appropriate Candidate, according to an employer is someone who is serious about his/her career and

supports his/her own professional development.

6. Interview as a substitute for practical assessment. Interview with the Challengers can be helpful but cannot substitute practical assessment.

7. Practical assessment fee. Assessment fee varies from $300 to $2000 where the former is for a 4– 8 hour assessment and the latter is

for a 4-6 weeks college-based practicum.


March 2015

63

8. Fee as a barrier. Participants think that the higher fee (more than $300-$500) will not have a significant impact on the industry.

Assessment fees need to be reasonable and built on a specific formula.

There is an understanding that the assessment business model needs to be built on an effective cost recovery

model.

9. Job candidates. Four (4) out of seven (7) interviewees hire workers without certification, mostly with Canadian experience.

One hires all three groups of workers.

Some employers (2) (mostly those who operate under union agreement) require C of Q or Red Seal for

employment.

Others hire non-certified workers in trades and trades related positions.

10. Optimal percentage of trades certified workers in a company. All employers would like to see most of their trades workers certified – they report that the optimal percentage

of trades workforces in their companies is 100% but current percentage varies from 10% to 75% depending

on operational needs.

11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates. Five (5) out of seven (7) interviewees agree that employers have lower confidence in the competencies of

those who challenged the certificate than those who completed formal apprenticeship.

12. Certification and hiring process. Hiring practices have a high degree of similarities and are based on a screening process that includes

verification of qualifications, training, and industry experience. A small number of respondents hire entry-level

workers and provide in-house training and/or apprenticeship.


March 2015

64

Appendix G. Millwright Group Interview

Seven industry representatives participated in the interview:

1. Al Storey, Millwright, Harmac Pacific, Nanaimo, BC 2. Chris Babcock, Manager Field Services, Service on Site, Fort St. John, BC 3. David Hiltz, Millwright, Catalyst Paper, Port Alberni, BC 4. Gordon Balfour, Millwright, Training Coordinator, Millwrights Union, Langley, BC 5. Mike Cass, Vice-President HR, Western Forest Products, Vancouver/Nanaimo, BC 6. Vern Phillips, General Manage HR, Catalyst Paper, 7. Richard Turnbull, Millwright and Machinist, FMF, National Defence, Victoria, BC

1. Current challenge requirements The current hours should be the same. Most of the respondents would like the application and verification

processes to be strengthened; the suggestions on how to strengthen the processes are different and

include stronger more detailed employer declaration, increased employer responsibility for signing working

hours, interview and practical assessment.

2. Challenge requirements for different Candidate groups. The majority agrees that the challenge requirements should stay the same for different groups of

challengers.

3. Practical assessment as a mandatory challenge component. The practical assessment is viewed as a strong component that would strengthen the challenge process

and increase employers’ confidence in Challengers’ competencies.

4. Practical assessment length. The suggested duration for a practical assessment is from 4 to 16 hours. There was one suggestion to

have a 40-hour practicum at the college that would cover all major skills of a fully qualified Millwright.

5. Practical assessment as a barrier. Respondents do not think that a mandatory practical assessment would deter appropriate candidates from

challenging certifications.

6. Interview as a substitute for practical assessment. Interview with the Challengers can be helpful but cannot substitute practical assessment.

7. Practical assessment fee Assessment fees vary from $500 to $3000.

8. Fee as a barrier. If the fee were higher than suggested, impact on industry would not be significant. The fee needs to be

reasonable and need not create a barrier for those who are willing to challenge.

9. Job candidates. Most (6 out of 7) respondents do not hire workers without formal credentials.

One employer hires Canadian workers without certification and workers with international experience and

certification; they are hired on a condition to challenge trade certification.

10. Optimal percentage of trades certified workers. The optimal percentage of certified Millwrights for the majority of respondents is 95% -100%, excluding

apprentices.

11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates. Five (5) out of seven (7) interviewees agree that their confidence in the Challengers’ skills and

competencies is lower than in skills of those who completed formal apprenticeship.


March 2015

65

12. Certification and hiring process. Hiring practices include screening for Red Seal, relevant industry experience, interview process, some

employment testing, probationary period and onboarding.


March 2015

66

Appendix H. Heavy Duty Equipment Technician Group Interview

This group included six interviewees:

1. Devon Lock, Mechanic Nels Ostero Sand and Gravel, Taylor, BC 2. Doug Richardson, General Manager, Inland Kenworth, Langley Branch, BC 3. Gordon Armour, BC Mining Task Force. 4. Ian Simpson HSE Manager/Edmonton Parts, Compression Technology Inc., Fort St. John B.C. 5. Richard Demchuk, Manager Human Resources Howe Sound Pulp and Paper Corporation 6. Alejandra Posadas, Apprenticeship Coordinator, Finning, Edmonton, AB

1. Current challenge requirements One (1) respondent said the challenge requirements are fully sufficient.

Other five (5) respondents agree that the current hours requirements are enough, but they would like to

see some improvements within the challenge process, specifically:

Improved verification process (e.g. ask employers to provide more details on the candidate’s experience

within the scope of the trade, add practical assessment);

Possible practical assessment.

2. Challenge requirements for different Candidate groups The majority (5 out of 6) of respondents think that the challenge requirements should be the same for

different groups of Challengers.

One respondent said that the challenge requirements should be different for the Candidates with non-

Canadian experience and education to ensure that foreign trained candidates meet Canadian standards.

To complete college training would help align the skills of foreign trained mechanics with Canadian

workplace requirements and standards.

3. Practical assessment as a mandatory challenge component Five out of six (5 out of 6) respondents think that adding a practical assessment as a mandatory

component would strengthen the challenge process and increase employers’ confidence in the abilities of

workers who achieved certification through the challenge process. All respondents agree that a practical

assessment would help demonstrate critical skills required for the workplace. In addition, most of the

respondents think that a practical assessment would help showcase their trade related skills.

4. Practical assessment length The suggested assessment duration varies: two hours, one full day, two full days, one month on the

employer’s site.

Those who suggested that the practical assessment be longer than one day think that eight (8) hours is not

enough to conduct an effective and meaningful assessment for this trade.

5. Practical assessment as a barrier Three out of six (3 out of 6) respondents do not think that requiring a practical assessment for certification

challengers would deter appropriate Candidates.

Two (2) respondents said that it might deter some Candidates.

Possible deterrents include: assessment fees, assessment delivery (lab vs. workplace) with the preference

for a workplace based assessment, or a Candidate’s readiness to challenge.

Overall, this should not be an issue as industry is looking for the appropriate candidates that meet industry

standards.


March 2015

67

6. Interview as a substitute for practical assessment The interview is seen as a positive component of the challenge process but not as a substitute for the

practical assessment.

One respondent suggested that challengers could go through a two to three hour interview with subject

matter experts to help determine the Candidates’ knowledge of processes and identify any possible gaps.

7. Practical assessment fee Assessment fee varies from $250 to $1000.

8. Fee as a barrier Increased assessment fee should not create any impact on industry. Assessment should be cost effective

and reasonable.

9. Job candidates – three groups One (1) respondent hires all three groups of workers;

One (1) respondent hires workers with Canadian experience and workers with international experience and

certification;

One (1) hires only workers with Canadian experience;

Two (2) do not hire uncertified workers;

One (1) skipped this question.

10. Optimal percentage of trades certified workers in company Three out of six (3 out of 6) respondents were not sure.

Other three responses varied from 65% to 100%.

11. Employer confidence in skills of Challenge vs. Apprenticeship Candidates One (1) disagrees, four (4) agree, one (1) is unsure about the statement.

Suggestions to strengthen the challenge process include incorporating practical assessment, interview and

verification process.

12. Certification and hiring process Respondents’ hiring practices are similar and include the following:

Job advertising;

Internal hiring for unionized companies and mandatory certification;

Screening for certification, industry experience, relevant training;

Interview and tests (math, aptitude, personality);

Probationary period


March 2015

68

Appendix I. Industry Interview Summary

The information included in this table is based on the majority of responses for each question.

Table 4.3 – Industry Interview Summary

Trade

Question

Carpenter Construction

Electrician

HDET Metal Fab Millwright

1. Current challenge

requirements

Need to be

strengthened

Need to be

strengthened

Need to be

strengthened

Need to be

strengthened

Need to be

strengthened

2. Challenge requirements

for different groups

Same Same Same Same Same

3. Practical assessment

as a mandatory

challenge component

Support

Support

Support Support Support


duration

4-8 hours or

Employer

Sign Off

8 hours or

College based

practicum

8 hours or

College based

practicum

8 hours

8 hours

College based

practicum


as a barrier

No

No No No

No

6. Interview

Only to support,

not substitute

practical

Only to support,

not substitute

practical

Only to support,

not substitute

practical

Only to support,

not substitute

practical

Only to support,

not substitute

practical


fee

$350-$600 range $750-$1000

range

$500-$1000

range

$500-$750 range

$500-$1000

range

8. Fee as a barrier

No/ only

for some

candidates

No/ only

for some

candidates

No/ only

for some

candidates

No/ only

for some

candidates

No/ only

for some

candidates

9. Job candidates

Hire all 3 groups

plus apprentices

Mostly with

Certification,

experience

across Canada

plus apprentices

Mostly with

experience

across Canada

plus apprentices

Mostly with

Certification,

Experience

across Canada

plus apprentices

Mostly with

experience

across Canada

plus apprentices

10. Optimal % of trades

certified workers in a

journeyperson position

Union – 100%

other depending

on market

demand

Up to 100%

Up to 100%

80%-100% Up to %100

11. 50:50 Lower in Lower in Lower in Lower in


March 2015

69

Trade

Question

Carpenter Construction

Electrician

HDET Metal Fab Millwright

Employer Confidence challenge

candidate

challenge

candidate

challenge

candidate

challenge

candidate

12.

Value of Certification in

hiring process

Certification

preferable

Experience

mandatory

Certification

mandatory

Experience

mandatory

Certification

mandatory

Experience

mandatory

Certification

preferable

Experience

mandatory

Certification

mandatory

Experience

mandatory


March 2015

70

Appendix J. Summary of Registration Practices of Select Non-trades Occupations

Association of Professional Engineers and Geoscientists of B.C. (APEGBC)

APEGBC is the licensing and regulatory body for B.C.’s professional Engineers and Geoscientists. Only engineers licensed

with APEGBC, or those who work under a direct supervision of P.Eng. licensed with APEGBC, are allowed to practice

engineering in B.C.

Registration Requirements

To register as a Professional Engineer (P.Eng.) with APEGBC, the Candidate needs the following:

� Acceptable academic qualifications; � A minimum of four years of satisfactory engineering experience with references; � Completion of the Law & Ethics Seminar requirement; � Successful completion of the Professional Practice Examination (PPE); � English language competency.

All applicants must demonstrate that they have Canadian Environment experience. The term Canadian Environment is

defined as:

� Work experience obtained in Canada, supervised by a Professional Engineer, registered or licensed in the applicable Canadian jurisdiction; or

� Work experience acquired outside Canada, where applicants demonstrate good knowledge of local Canadian engineering laws, practices, standards, customs, codes, conditions and climates

Membership and licensure include 14 categories, such as Professional Engineer, Engineer-in-Training, Limited Licensee

(Eng.L.) Professional Geoscientist, Geoscientist-in-Training, Non-Resident Licensee (NRL) P.Eng., International Mobility

P.Eng.

Self-Assessment Tool for Internationally Trained Engineers

APEGBC, in collaboration with the Applied Science Technologists and Technicians of BC (ASTT BC) and the Ministry of

Jobs, Tourism and Skills Training, has developed an online self-assessment tool. The tool was designed to help

internationally trained Candidates determine whether they should pursue a career as a Professional Engineer or as an

applied science Technologist or Technician.

The self-assessment tool includes a series of questions on education and work experience and provides information on how

declared professional qualifications are likely to be assessed by APEGBC or ASTTBC. The applicants can print the results

of the self-assessment. The applicants are informed that the self-assessment is not an official assessment of qualification

for registration or license in B.C.

The applicants must submit a formal application for registration or license to the appropriate regulatory body supplemented

by the required documentation and fees.

Prospective applicants that completed training in Canada can also use the self-assessment tool; they may find it helpful in

evaluating their qualifications and learning more about the application process.


March 2015

71

Association of Technicians and Technologists of B.C. (ASTTBC)

ASTTBC is responsible for the assessment and certification of Technologists and Technicians in B.C. ASTTBC Certification is not required to practice. Professional designations include Applied Science Technologist (AScT) and Certified Technician (CTech). Other classifications include: Graduate Technologist/Technician (GradTech), Associate and Student.

Registration and Certification Requirements

The minimum education required for AScT certification is a Diploma of Technology (two or three years) from a nationally

accredited program or equivalent. AScT and CTech certifications require a minimum of two years of related work

experience, with at least one year at the level of certification requested.

Provisional Certification for Internationally Trained Professionals

Internationally trained professionals may qualify for provisional membership if their education qualification and work

experience are acceptable. Reclassification from provisional status is possible after one year of work experience that

demonstrates an understanding of Canadian standards.

The provisional members have to complete the one-year work experience requirement within a five-year period after

registration. After the five years, their membership status would automatically be required to be reassessed.

Internationally Trained Technology Professional Portal

ASTTBC, with the help from the BC Government and industry support, has created a web portal for internationally trained

professionals – www.itppbc.com - to guide the internationally trained applicants through the certification process and provide

information on working conditions, language requirements, and employer expectations.

Other available resources include a self-study guide, workshops and webinars, information on codes and standards,

professional practice and ethics; they are designed to help internationally trained professionals navigate the certification

process.

Further examination of these practices and resources is required to determine how the ITTP concept can be used and

applied for engaging and guiding internationally trained tradespersons through the trade certification process in B.C.

College of Registered Nurses of B.C.

In British Columbia, only individuals registered with CRNBC can legally call themselves a Registered Nurse or provide

duties legally restricted to the profession of registered nursing.

Registration requirements

First time applicants include:

• B.C. educated Nurses. Graduates from a nursing program in B.C. must apply for CRNBC registration and apply to write the national nursing exam;

• Canadian Nurses. Registered Nurses with current or previous registration in another Canadian jurisdiction, or recent graduates from a Canadian RN nursing program, are eligible to apply for registration in British Columbia;

• Internationally Trained Nurses. Internationally trained Nurses who have never been registered in Canada may be eligible for registration to


March 2015

72

practice in B.C. First, they must complete the National Nursing Assessment Service (NNAS) process. After completing the NNAS process, CRNBC may require internationally trained nurses to undergo a competency assessment.

Provisional registration

Provisional registration is granted to qualified applicants who have to meet outstanding registration requirements. The

provisional registration is assigned specific conditions that must be met to be eligible for the practicing registration. When

these conditions are met, the provisional registration is automatically converted to the practicing registration.

Examples of provisional registration conditions include:

1. For Canadian RN: a. A condition that the applicant writes and passes the national nursing exam b. A condition that the applicant completes supervised practice experience (SPE)

2. For International RN: a. A condition that the applicant writes and passes the national nursing exam b. A condition that the applicant completes coursework in a specific area

(Note – these courses will normally be determined after reviewing the results of a Substantially Equivalent Competency (SEC) Assessment)

c. A condition that the applicant’s employment is limited to specified facilities

Internationally Educated Nurses

National Nursing Assessment Service (NNAS) is a national body of nurse regulatory bodies funded by the Government of

Canada's Foreign Credential Recognition Program (FCRP).

It developed a harmonized approach to the assessment of internationally educated nurses (IENs). NNAS coordinates a

nation-wide approach for IENs seeking registration and licensure to practice in Canadian jurisdictions.

Significant features of the NNAS include:

• A single national web portal;

• A common approach to initial document collections, authentication, validation and fraud screening;

• A consistent competency based assessment of the IEN application file;

• A consistent format for advisory reports to regulatory bodies to inform eligibility assessments;

• A bilingual customer care centre for IEN applicants;

• A national IEN database providing enhanced reporting and analysis capabilities.

The self-selection guide helps applicants decide which nursing profession is best suited to their experience, education and

credentials obtained outside of Canada. The guide describes three regulated nursing groups – Registered Nurse (RN),

Licensed Practical Nurse (LPN), and Registered Psychiatric Nurse (RPN) – and requirements for registration.

The website contains numerous resources to assist applicants with certification and licensure process.


March 2015

73

Appendix K. ITA Registered Assessor Training

Pre-course Participant Knowledge Check

For each item below, please rate your current level of knowledge and understanding on a scale of 1 to 5

(1= no Knowledge; 5= very high)

Do not sign the form; your response is anonymous. At the end of the classroom session and again at the end of your

practicum mentored assessments you will be asked to update the rating and indicate where you feel you need additional

information or training.

Assessor Course Content Area My rating of my

knowledge

Knowledge of BC’s Trades and Occupational Certification Framework 1 2 3 4 5

Roles and Responsibilities of ITA Approved Assessment Agencies, Registered Assessors, and

Candidates

1 2 3 4 5

Qualities and Competencies of an Effective Assessor 1 2 3 4 5

Principles of Assessment and Rules of Evidence 1 2 3 4 5

Occupational Performance Standard Against Which Candidates are evaluated in my Trade or

Occupation

1 2 3 4 5

ITA Enhanced Assessment Operational Procedures 1 2 3 4 5

Methods of Assessment for my Trade or Occupation 1 2 3 4 5

How to Use Approved Trade or Occupation-specific Assessment Tools 1 2 3 4 5

Use of Effective Questioning Techniques 1 2 3 4 5

How to Complete ITA Forms, Keep Records and Report Results 1 2 3 4 5

Providing Constructive Written Feedback 1 2 3 4 5

Access to Information, Protection of Privacy, Conflict of Interest, and Assessor Code of Conduct 1 2 3 4 5

Procedures around Requests for Reconsideration & Appeals of Decisions 1 2 3 4 5

TOTAL

Date:________________________________

ITA ASSESSOR

TRAINING

Pre-Course Study


March 2015

74

Assessor Definition

An ITA-registered Assessor is authorized to conduct competency-based assessments for specified trades or

occupations and to make professional judgments on whether evidence provided by candidates is sufficient to

meet approved certification standards.

Assessor eligibility and criteria are recommended by industry and approved by ITA. Assessors must be

selected based on the assessor eligibility criteria and, in order to conduct enhanced assessments, must be

registered with ITA and meet currency requirements. Hard copies of assessor files (including registration forms

and performance management documentation) must be kept with the Assessment Agency and must be

available for audit by ITA at any time.

Assessor currency requirements are defined in the relevant ITA Program Profile. The Assessment Agency is

responsible for annual verification of assessor currency. Records of assessor currency must be available for

audit by ITA at any time.


March 2015

75

ITA OPERATIONS POLICY PA3008

V.15.01.2013

POLICY: ITA-REGISTERED ASSESSORS

1.0 Scope This policy governs the registration and activities of individuals contracted or employed by ITA or by an ITA-approved

Assessment Agency. These candidates may be either completing apprentices or individuals challenging for certification.

2.0 Enhanced Assessment Enhanced Assessment describes the evaluation of competencies against occupational standards for the trade or

occupation through means other than writing a standardized examination.

3.0 Authorized Administration of Enhanced Assessment

Where completion of an ITA apprenticeship training program or certification challenge requires demonstration of

competency through a formal evaluation of underpinning knowledge and applied skills as well as, or in place of, a written

Certification examination, such evaluation must be conducted by an ITA-registered Assessor who meets requirements

prescribed in the relevant ITA program standard.

4.0 Eligibility for Assessor Registration

4.1 Registered Assessors must:

• have successfully completed ITA-approved Assessor training;

• have submitted, by way of criminal record check, evidence of no record of criminal offences related to the duties which the applicant would be required to perform as an assessor;

• have signed and agreed to follow ITA’s conflict of interest, confidentiality, and code of conduct rules; • have provided contact information for a minimum of 2 industry colleagues who support their application.

4.2 In addition, the relevant program standard defines Assessor requirements for each of the following:

� minimum amount, scope, and currency of experience in the trade or occupation; � whether or not supervisory experience is required, and if so, how much; � prerequisite certifications and endorsements; � number and type of industry references/recommendations; � minimum number of mentored assessments which must be successfully completed; � currency requirements for maintaining Active Assessor status, including:

o minimum annual number of assessments performed (may vary by program) o minimum annual participation in moderations, professional development or upgrading (may vary by

program) o satisfactory annual performance evaluation by Assessment Agency or ITA (mandatory for all

programs) o contract or employment with an Assessment Agency or ITA. (mandatory for all programs)

5.0 Maintenance of Registry

ITA will maintain an up-to-date registry of all Assessors. ITA-approved Assessment Agencies are responsible for

immediately notifying ITA of changes such as termination of employment or cancellation of a contract for an Assessor.


March 2015

76

6.0 Cancellation of Registration or Re-assignment from Active to Inactive Status

ITA reserves the right at its sole discretion to cancel an Assessor registration, or to re-assign an active registration to

inactive status with or without conditions.

Approved: 24 July.12

Updated 15.Jan.13


March 2015

77

Course Description

ITA’s Registered Assessor Course is built around 5 areas of assessor performance detailed in the Assessor Occupational

Analysis Chart. The course comprises pre-course study materials; 2 days of classroom instruction and activities including a

homework assignment after day 1; and a practicum consisting of the conduct of 2 mentored assessments. Practicum

assessments will be conducted under the supervision of an experienced ITA Registered Assessor and a representative of

ITA.

Course Content

Course content covers principles and rules of assessment and application of the principles and rules using occupation-

specific assessment tools and methods.

In-Class Training Outcomes By the end of the in-class training session you will be able to:

• Explain the certification system and roles of ITA, Assessors, Assessment Agencies, and certification candidates • Interpret the competency standard for a specific trade or occupational qualification • Define the qualities and characteristics of a competent assessor • Explain code of conduct, privacy, and conflict of interest requirements • Identify the parts and steps of the assessment process • Prepare candidates for assessment

Pre-Study Package

• Principles of assessment and rules of evidence

• Roles and attributes of an assessor

• Assessment methods and tools

• Operational Procedures

Day 1 In-class Training

• Demonstrations of Practical Assessment & Classroom Debrief & Moderation

Day 1 - 4 Shop Training & Debrief

• Conduct 2 mentored assessments under the supervision of an ITA experience assesor

Practice Assessment

Registration


March 2015

78

• Conduct assessments and make assessment decisions in accordance with the principles of assessment and rules of evidence

• Complete necessary paperwork • Provide candidates with helpful and objective feedback • Describe service standards and ITA-approved operational procedures

Practice Assessment Outcomes At the end of the Assessor Training course you will be able to demonstrate competence in the following assessor activities

required to become an ITA-registered assessor:

• Prepare for assessments

• Demonstrate effective communication and interpersonal skills

• Conduct assessments and making assessment decisions in accordance with ITA Enhanced Assessment

Operational Procedures

• Record and report assessment decisions and recommendations

• Apply continuous improvement to the assessment process.

Assessor Occupational Analysis Chart

The occupational analysis chart (OAC) is a document that identifies the distinct competencies required province-wide to achieve competency in an occupation. An OAC is not a training program or instructor guide; it informs the development of appropriate training and assessment methods for the occupation training program.

Occupation Description: An “Assessor” is authorized to conduct competency-based assessments for specified trades or occupations and to make professional judgments on whether evidence provided by a certification candidate is sufficient to meet approved certification standards. The Assessor documents assessment outcomes, provides feedback on assessment results, and manages assessment records in accordance with approved policies and procedures.

Line A Prepare for Assessment

A1 Prepare assessment resources, logistics, documentation and venue.

A2 Make professional judgments on the validity, authenticity, currency and sufficiency of evidence provided in

portfolio of evidence

A3 Inform and clarify assessment details with candidate

A4 Identify required accommodations for candidates with special needs for assessment activity

A5 Identify real and perceived conflict of interest scenarios

Line B

Demonstrate Effective Communication and Interpersonal Skills

B1 Use effective communication and interpersonal skills to develop and maintain a professional relationship

B2 Demonstrate use of plain language and terminology that is clear and concise

B3 Provide clear and constructive feedback

B4 Demonstrate active participation in moderation sessions using effective communication skills


March 2015

79

Line C

Conduct Assessments and Make Assessment Decisions

C1 Make professional judgments on the validity, authenticity, currency and sufficiency of evidence.

C2 Identify requirements for further evidence.

C3 Make judgments that are consistent with judgments made on similar evidence.

C4 Conduct assessment activities in accordance with principles of assessment and Privacy and Confidentiality

legislation.

Line D

Record and Report Assessment Decisions and Recommendations

D1 Prepare and submit records of assessment outcomes and supporting evidence in accordance with Operational

Procedures and Retention Policy

Line E

Apply continuous Improvement to the Assessment Process

E1 Identify strengths and weaknesses of the process and tools by applying the principles of assessment and rules

of evidence.

E2 Identify opportunities and recommendations to improve the assessment process and their own practice

Legislative and Policy Framework for Enhanced Assessment Legislative Level Section 8 of British Columbia’s Industry Training Authority Act [SBC 2003] Chapter 34 grants ITA the legislative authority and commensurate responsibility to develop criteria to award industry training credentials, and to establish standards and procedures for assessments. The foundation for issuance of credentials, therefore, is based on industry subject matter experts performing a thorough job analysis and recommending measurable indicators of what a competent practitioner should know and be able to do in the workplace to meet current occupational standards. These indicators are used to construct assessment tools, including written exams, and, where enhanced assessment forms part of the evaluation, portfolio evaluations, technical interviews, practical demonstrations of skill, and so forth. The summary description of trade or occupational certification requirements is found in the relevant ITA Program Profile. Policy Level ITA Board policy governs administration of certification exams, other types of evaluations of candidate knowledge and skill which fall under the classification of “Enhanced Assessment”. Administration of Enhanced Assessment is governed by ITA Operations Policy PA3008, ITA-Registered Assessors, which restricts delivery of enhanced assessments to ITA-Registered Assessors working under ITA supervision or the supervision of an ITA-Approved Assessment Agency. Operational Level By way of an Approved Assessment Agency Agreement between the Agency and ITA, roles and responsibilities Assessment Agencies are assigned to the parties. These administrative responsibilities must be carried out in accordance with ITA’s Enhanced Assessment Operational Procedures.


March 2015

80

ASSESSOR POLICY FRAMEWORK

ITA Operations Policy PA3008

Registered Assessor Policy

Industry Specific Assessor Eligibility Criteria

ITA Assessor Training Course

ITA Registered Assessor Assessor


March 2015

81

Declaration of Conflict of Interest and Agreement to Adhere to the Code of Conduct for ITA Registered Assessors

A: CONFLICT OF INTEREST

A conflict of interest exists in any situation in which you have a private or personal interest sufficient to influence or appear to

influence the objective exercise of your official duties as an assessor.

If you identify a real or perceived conflict of interest, ITA will re-assign the file. The sooner this issue is identified, the less

disruptive it is for the candidate. If you or the candidate discover or declare a conflict of interest part way through the

process (e.g. you didn’t recognize the name or information in a portfolio, but upon meeting the candidate recognize the

person and identify a potential conflict of interest), you must terminate the assessment at the time the conflict of interest is

identified and have the file re-assigned without prejudice.

Typical situations which create real or perceived conflicts of interest and are not allowed include assessing:

• Family members or close friends

• Current co-workers, supervisors, subordinates, business associates and contractors you work with

• Your current students or apprentices

• Persons to whom you owe an obligation

• Competitors

When in doubt, err on the side of caution. If you have a question about whether or not something constitutes a conflict of

interest, contact ITA for advice.

B: ASSESSOR’S CODE OF CONDUCT

The assessor code of conduct detailed below supports professionally responsible assessment practice and is a guide to all

assessors on the responsibilities of their work in line with best practice.

• You are expected to treat all challengers with courtesy and respect.

• It is important to maintain a calm demeanour during all communication with the challenger, whether this is in

person, in writing or over the phone. You should not become involved in any altercation or disagreement with

a challenger. Any issues that arise in relation to the challenger should be referred to the credentialing

authority.

• You must comply with any specific instructions from the credentialing authority in relation to the assessment

process.

• You must maintain strict confidentiality at all times relating to information that is disclosed to you or that you


March 2015

82

discover in the performance of your work. Definitive information about an assessment outcome should not be

given to applicants. This will be provided in writing to the applicant after the assessment event. Under no

circumstances should you talk about the assessment content to anyone other than those responsible for

administering the assessment process.

• Consumption of alcohol or improper use of drugs or other substances must not occur while undertaking your

assessment duties or in any activity that relates to your work as an assessor and must not affect your work

performance.

• You must not accept gifts from applicants, agents or other representatives. If you are personally offered any

inducements, you should refuse the offer promptly and report the matter to the credentialing authority.

• You should not seek or accept favours or gifts for services performed in connection with your work. Included in

this category are gifts in kind, such as free accommodation, travel or entertainment vouchers, whether for you

or members of your family. The general principle to be followed is that you should not seek or accept favours

or gifts from anyone who could benefit by influencing you.

• You must not unlawfully or unjustifiably discriminate against any individual in your practice as an assessor.

• You must not discriminate against any individual on the basis of gender, race or disability or allow their views

of the lifestyle, culture, belief, colour, gender, sexuality or age of others to prejudice your decisions as an

assessor.

• You should maintain your professional knowledge and skill and observe and keep up to date with laws and

professional codes of practice that affect your work.

INDUSTRY TRAINING AUTHORITY (ITA)

___________________________________

Signature

___________________________

Print Title and Name

______________________________

Date

)

)

)

)

)

)

)

)

)

)

__________________________

Signature

__________________________

Print Title and Name

___________________________

Date


March 2015

83

Enhanced Assessment Operational Procedures

March 2014


March 2015

84

Table of Contents

Overview ..................................................................................................................................................................... 84 1. Introduction ..................................................................................................................................................... 84 2. Process Overview ........................................................................................................................................... 85 3. Systems Roles and Responsibilities ............................................................................................................... 87

Administrative Process ................................................................................................................................................ 89 1. Information on the Enhanced Assessment Challenge Process ....................................................................... 89 2. Candidate Application, Registration and Approval to Challenge ..................................................................... 90 3. Assessment Activity ........................................................................................................................................ 91 4. Resource Requirements ................................................................................................................................. 95

Quality Assurance Process.......................................................................................................................................... 97 1. Administration ................................................................................................................................................. 97 2. Occupational Performance Standards and Assessment Tools ....................................................................... 97 3. Candidate Documentation and Record Keeping ............................................................................................. 98 4. Assessor Eligibility and Currency .................................................................................................................... 98 6. System Compliance ........................................................................................................................................ 99

Assessor Management Responsibilities ...................................................................................................................... 99 1. Eligibility Criteria .............................................................................................................................................. 99 2. Training ......................................................................................................................................................... 100 3. Registration ................................................................................................................................................... 101 4. Moderation .................................................................................................................................................... 101 5. Performance Review ..................................................................................................................................... 103

Appendix 1 – Glossary of Terms ..................................................................................................................................... Appendix 2 – Audit Checklist ........................................................................................................................................... Appendix 3 – Complaints and Appeals Procedure …………………………………………………………………………………

Overview

1. Introduction

Enhanced assessment refers to a structured evaluation process of certification candidates against program

standards that involves different assessment methodologies beyond the written multiple-choice examination.

This could include review of a portfolio of evidence, written knowledge assessment, technical interview and

practical assessment. It is recognized that each trade/occupation encompasses a specific set of skills and

competencies and therefore the assessment model will be tailored to meet the specific standard requirements

of each trade.

An enhanced assessment system involves performance criteria development, operational processes and quality

assurance. This document outlines the roles, responsibilities and processes associated with the management

of enhanced assessment of candidates and it provides the framework to ensure an efficient, high quality and

client-focused service irrespective of the methodology adopted.

These operational procedures are for the use of the ITA Operations department and the Assessment Agency

involved. It details activities that are related to the administration of applications, the management of

assessment activities and the business requirements of both organizations. It also outlines the audit criteria for

each stage of the administrative process. These criteria are detailed in an audit checklist at the end of each

section; a summary can be found in Appendix 2 for use throughout the quality assurance process.


March 2015

85

2. Process Overview

The following is a process map that details the operational flow for processing a candidate for certification in a

trade or occupation through means different than the standardized multiple choice written examination. The

reassessment process flow follows the same process map as the initial assessment.


March 2015

86


March 2015

87

Enhanced Assessment Roles and Responsibilities

Below are the roles and responsibilities of the ITA and the Assessment Agency to operate the enhanced

assessment system. Note that in some cases ITA may also be responsible for the activities of the assessment

agency.

ITA

Roles and Responsibilities

Assessment Agency


Application

• Provide general information to the candidate

• Receive and process applications and application fees

• Register candidate into ITA Direct Access (ITADA)

• Forward to Assessment Agency

• Receive and process reassessment applications and fees

• Schedule written exam

Results

• Process results submitted by Assessment Agency

• Issue results notification letter to candidates

• Issue Certificates to candidates who meet the certification standard

Records Management

• Store and Maintain electronic candidate files and assessment records.

• Process and respond to complaints and requests for review of assessor decisions

Assessor Training and Registration

• Train assessors

• Register assessors

• Facilitate moderation sessions

• Maintain updated assessor registry

Application

• Provide specific information to the candidate

• Provide advice to candidates on how to collect evidence to complete application forms and build portfolios, where applicable

• Perform initial review of eligibility, where applicable

• Inform ITA of eligibility to challenge, where applicable

• Set and collect fees for assessment and reassessment activities

• Support ITA in managing complaints in accordance with ITA’s complaint management procedure

• Coordinate reassessment

Assessment Activity

• Assign candidate files to assessors

• Schedule assessment activities and notify relevant parties

• Advise ITA of assessment results including feedback and gap training strategies, where applicable

• Manage required venues, including facilities, equipment and supplies as required

Records Management

• Securely store files and assessment results

• Input, catalogue and archive records

Assessor Management


March 2015

88

ITA


Assessment Agency


Quality Assurance

• Quality assure and audit administrative processes and assessor documentation

• Store, update and publish assessment tools with version control

• Maintain audit criteria for assessors

• Maintain audit standards for venues

• Approve Assessment Agencies who sign Assessment Agency Agreements

Strategy and Policy

• Manage disputes, complaints, requests for reconsideration and appeals

• Recruit and recommend assessors for assessor training and registration

• Assist with design and delivery of moderation sessions

• Manage ongoing performance of assessors

• Submit annual performance evaluations to ITA to confirm currency and verify active assessor status

Quality Assurance

• Recommend to ITA industry-specific criteria for assessment venues

• Recommend to ITA industry-specific eligibility criteria for assessors

• Manage audit requirements

Finance/Contracting

• Manage own financing and contractual arrangements


March 2015

89

Administrative Process

Candidate administration consists of a number of activities that are detailed in this section of the procedures. Highlighted

within the text of each section are the roles and responsibilities of each entity involved in the process. The following

stakeholders are directly involved in the administration process:

• ITA

• Assessment Agency

• Registered assessors

• Candidates

1. Information on the Enhanced Assessment Process

Detailed information about the purpose of the assessment, the assessment standards and the assessment

process are provided by the ITA for the candidates to review. General information is provided and kept up-to-

date by both ITA and the Assessment Agency via the website, including:

• The candidate information kit

• Candidate self-assessment tool

• ITA challenge application form

• Other promotional materials

Audit Checklist

Assessment Agency

� Materials in use are current and approved versions

a. Published candidate information kit

b. Published assessor guide

c. Assessment tools (version matched with ITA)

� Advise candidates how to collect evidence for applications and building portfolios, where applicable

a. Provide staff (including backup) and resources to aid potential candidates throughout the

application and portfolio-building process, where applicable

ITA

� Version control protocols are in place to ensure currency of:

a. Published Approved Performance Standards, where applicable

b. Program Outline/NOA

c. Assessment tools

d. Assessment Agency Agreements

e. Assessor registry


March 2015

90

2. Application, Registration and Eligibility approval

Application

An application package, which includes the application form, and if applicable, a self-assessment tool, and a

guideline for preparing the portfolio of evidence, is provided for the candidate. The candidate completes the

package and submits it, along with the challenger application fee, to the ITA. Fees are payable by cheque

(made payable to “Industry Training Authority”) or through the ITA PayPal site.

Registration

When an application is received, the ITA Customer Service creates an individual file in ITADA, following current

registration procedures. Once the candidate has been registered and the Individual Identification Number has

been issued, ITA will forward the application package to the Assessment Agency with any relevant supporting

documentation.

Candidate Approval

The Assessment Agency will evaluate the application and, when applicable, portfolio of evidence for eligibility

against the program standards. If this initial review finds that the individual is not eligible for assessment the

Assessment Agency will notify ITA. If the individual is eligible for assessment the Assessment Agency will

inform ITA and begin the assessment process, which includes assigning an assessor to the file and, scheduling

assessment activities and venues as required.

Audit Checklist

ITA

� Process for receiving and registering application (date stamp) and opening candidate file

a. Candidate information management system to track intake of applications (i.e., Excel spreadsheet,

ITADA notes)

b. Process for registering candidate in ITADA

c. Service standard of 5 business days from receipt of application to registration in ITADA

� Process for collecting application fess

� Communication of eligibility to candidate

a. Letters communicating acceptance and notification of assessment process next steps

Assessment Agency

� Process for receiving application (date stamp) and opening candidate file


March 2015

91

a. Candidate information management system to track intake of applications (i.e., Excel spreadsheet,

ITADA notes)

� Process followed for reviewing application for completeness and relevancy to the eligibility criteria for the

specific trade or occupation.

a. Identify the individual who is conducting the review and methodology

� Communication of eligibility to ITA, and notification to candidate of venue and potential assessment dates

a. Notifications of eligibility and scheduling to all parties

3. Assessment Activity

Approved assessment activities vary by trade/occupation and may involve one or more assessment

methods.

a. Assessment venues2

Assessment venues are identified by the Assessment Agency in accordance with industry-specific criteria.

Venues should be geographically distributed where possible across the province to meet the needs of the

industry and candidates.

Each assessment venue must:

• Meet industry requirements as defined in the program outline

• Facilitate the efficient use of assessment resources and tools

• Meet Occupational Health and Safety (OHS) requirements

b. Scheduling assessment activities

The assessment process involves one or more of the following assessment activities:

• Desktop review of challenge application

• Technical Interview

• Written Knowledge Assessment

• Practical Assessment

The scheduling of assessment activities is conducted by the Assessment Agency. For trades or

occupations in which the assessment process involves all of the above-mentioned activities; written

assessments and Technical Interviews should be scheduled a minimum of two weeks prior to practical

assessments to allow for the planning of the practical assessment.

The Assessment Agency will send a scheduling notification to the candidate a minimum of 10 business

days prior to the assessment activity.

c. Assessor conducts assessment activity

2 For program-specific venue requirements, refer to Program Outline Section 3. Facility Requirements


March 2015

92

The first evaluation is a desktop review of the candidate application and portfolio if one is submitted.

• The flow of the assessment activity is specific to each trade or occupation and detailed in the Assessor

Guide or approved assessment tools.

• The screening interview will be the first face-to-face contact with the candidate and must be conducted

in accordance with approved assessment tools. Audio recordings of technical interviews must be retained

on each candidate’s file.

Based on responses to screening interview questions the assessor may determine that the candidate is

attempting a certification for which he or she does not have the required experience and scope of trade or

occupation. The assessor may recommend challenge of a level or levels of the trade where the applicant is

ineligible to challenge the full certification. Alternatively, further assessment activity may be postponed until

the identified gaps are addressed. In this case, the assessor should advise the Assessment Agency of the

results and recommend an assessment plan. Assessment Agency will notify ITA and the candidate of

assessment options.

Practical assessments will comprise a pre-approved set of tasks determined by the standards set out in the

relevant Program Outline or OPS. Published program standards and any materials necessary for planning

or preparing for the assessment tasks are to be made available to all candidates.

Assessors must use tools provided by the Assessment Agency to conduct the assessment and document

results to ensure that the required performance standard has been demonstrated. Photographs and/or

video of the assessment may be taken to provide a clear record of the practical tasks and finished products

presented.

d. Results of assessment activity

Following the completion of the assessment activity, the assessor has 10 business days to complete and

electronically submit the assessment results package to the Assessment Agency, followed by submission

of hard copies of the documents and, as applicable, audio/video recordings.

The results package will vary depending on the methods of assessment applied, as appropriate may

include:

• Written Assessment Result

• Technical Interview Responses documented with notes by the assessor

• Practical Assessment results forms

• Assessment Summary

• Complete portfolio of evidence, including the self-assessment form and any additional evidence

gathered during the assessment activity, such as pictures, audio/video recordings, etc. as applicable

The information collected will be used to provide the notification of assessment outcomes.

Following receipt of the results package, the Assessment Agency will review the forms to ensure that they

accurately articulate the outcome of the assessment in sufficient detail to support the assessment decision

and to provide clarity to the candidate regarding the outcome.


March 2015

93

If during the assessment process the candidate provided insufficient evidence to meet the standards, the

Assessment Summary must include detailed information on the areas in which the candidate must provide

further evidence or gain further skills and/or knowledge to meet the certification standard. Assessors are

required to provide constructive feedback to the candidate on their performance and information on further

training, work experience or other ways the candidate can address his or her gaps in skills and knowledge.

4. i. Reporting and cataloguing results

Once the Assessment Agency confirms that the assessment activity reporting tool has been

completed accurately and the assessment results package is received from the assessors, they will

send a reply confirming receipt and the assessor should destroy any files on their personal

computer or in their possession that are related to the candidate.

All documentation submitted in the results package from the assessor must be catalogued and

submitted to ITA to be stored in ITA DirectAccess.

These documents provide details of the assessment and why a decision was reached The

Assessment Summary has the same evidentiary value as an examination answer sheet and must

be signed by the assessor, dated and stored according to ITA’s record retention policies and

procedures, along with any original evidence collected during the assessment process.

5. ii. Communication to candidate of outcomes

The Assessment Agency will submit the results to ITA to be stored in ITA DirectAccess.

Notification of Credential Not Achieved

Candidates who are not successful in meeting all of the standards requirements will not receive a

credential. ITA will issue a notification letter detailing the assessment outcome. This notification

will identify the gaps in evidence to meet the standards to achieve certification and will outline the

reassessment process.

The notification letter should include:

• Assessment summary

• Assessor comments and recommendations

Candidates will be advised that they may contact the Assessment Agency for advice on how to

address the gaps identified during the assessment process, including potential gap training options.

It is the candidate’s responsibility to address the identified gaps and to provide new evidence prior

to reassessment.

Notification of Credential Achieved

Candidates who met the certification standard will be issued the relevant credential in their

notification of results.

e. Complaints, requests for reconsideration, and appeals

Any complaint received from a candidate must be submitted in writing and all parties must be notified in

accordance with ITA’s Complaints and Appeals Management Procedure (See attached appendix 3.).


March 2015

94

Complaints should be acknowledge promptly by telephone, within 2 business days of the date of receipt

and followed up by email to advise the candidate how the complaint will be dealt with.

A candidate who believes that the assessment process was not followed correctly or that they were unfairly

treated may request reconsideration by the ITA CEO or appeal the assessment decision to the Industry

Training Authority Appeal Board. A request must be made in writing to ITA in accordance to ITA policy

PA3005 – Appeals/Dispute Resolution.

If a candidate requests reconsideration by the ITA CEO, a formal review of the assessment result, which

will be managed by the ITA Manager of Program Standards, will be performed only for competencies not

achieved. In no case will a unit of competency previously granted be revoked as a result of the review. The

review will be conducted by a panel of a minimum of three ITA registered assessors (reviewers), one of

whom will be nominated as chair by the review panel. If an assessor has been a part of the assessment

under review, he or she cannot be a member of the review panel. Each of the three reviewers will

independently review the documentation. The assessors will then jointly determine the outcome of the

review, either upholding or amending the results. Each of the three reviewers must submit an independent

review document. The chair will prepare and submit the final joint panel decision document

f. Reassessments

A reassessment may take place when a candidate is unsuccessful in providing sufficient evidence to meet the

program certification standards. Fees for Reassessment activities are set and collected by the Assessment

Agency.

Reassessment will follow the same administrative processes described in Sections 1, 2 and 3.

6. ii. Procedures for reassessment

All applicants for reassessment must follow the procedures listed below:

• There is a 30-day waiting period from the date of the results notification letter before applying

for reassessment

• Candidates applying for reassessment must submit new evidence. New evidence is defined as

evidence collected and dated after the previous assessment.

• The evidence provided must apply to gaps identified in the previous assessment result.

• An assessor will evaluate the new evidence to determine candidate readiness/eligibility for

reassessment and determine the appropriate assessment method (e.g. technical interview,

practical assessment, etc.)

• Reassessment must be conducted in accordance with the plan developed by the assessor. Audit Checklist

Assessment Agency

� Venue meets the industry standard for assessment activity

a. Venue meets specified industry criteria and OHS requirements


March 2015

95

b. Completed checklist for each venue to ensure they meet the requirements (with dates and venue

evaluation process identified)

c. Any formal venue complaints investigated in accordance with ITA Complaints Management

Procedures and documented with proposed improvement and evaluation plan where required

� Assessment agency makes any required contractual arrangements with venues which meet program-

specific venue criteria

a. Staff (including backup) and resources are in place to successfully execute the activity

b. Ensure all required contracts are in place and enforced

� Assessment Agency assigns candidate file to an assessor, schedules assessment activity and , secures

venue

a. Member of staff/backup and resources are in place to successfully execute the activity. NOTE:

Assessment date is based on venue schedules, candidate preferences and assessor availability.

b. Service standard of 10 business days from ITADA registration to scheduling of assessment

� Receive and review documentation of assessment results, upload results to ITADA and report results to

ITA

a. Member of staff /backup and other resources are in place to accept and review assessment results

b. Information management system to track receipt of completed assessment tools and any further

evidence gathered throughout the assessment activity

c. Service standard of 15 business days from assessment completion to reporting results to ITA

� Respond to Complaints in Accordance with ITA Procedures

a. Service standard of 2 business days for responding to complaints by phone and following up via

email

ITA

� ITA process assessment results and issue notifications; and where the certification standard is met, issue

the credential

a. Information management system to track results (i.e., Excel spreadsheet, ITADA notes)

b. Service standard of 5 business days from receipt of results to entering the competencies in ITADA

7. Resource Requirements

Assessment Agencies set and collect fees for Assessment and Reassessment activities; ITA does not set or

collect these fees. The following are costs and resources the Assessment Agency should consider when setting

assessment activity fees:


March 2015

96

Candidate Support, Application and Eligibility Reviews

• Assessment Agency Administrative staff time to aid in completion of application and portfolio

development where applicable

• Assessment Agency Administrative staff time to review application and process registration

• Assessor time to review candidate eligibility for assessment.

Arranging Assessments

• Assessment Agency administrative time and overhead to liaise and coordinate assessment, schedule

assessors, candidates and assessment venues.

• Assessment Agency administrative staff time and overhead to implement appropriate contractual

arrangements with assessors and assessment venues (including insurance issues)

• Assessment Agency administrative staff time and overhead to answer candidate queries on the

process

Conducting Assessments and Processing Results

• Assessor remuneration to perform all assessment activities (portfolio review, assessment plan

development, technical interview, written knowledge assessment, and practical assessment as

applicable) for both first-time candidates and reassessments

• Reimbursement of approved Assessor expenses (e.g. travel, accommodation, etc.)

• Applicable assessment venue costs to conduct technical interview, written assessment and practical

assessment (including facility rental, supplies, consumables, tools and equipment)

• Assessment Agency Administrative staff time and overhead to process results

Annual Compliance Audit

• Administrative staff time and overhead to plan and prepare for audit

• Administrative staff time and overhead to support audit while in progress

• Management staff time to review audit findings and agree on a plan of action for improvement Assessor Management

• Administrative staff time and overhead to complete and submit to ITA annual Assessor Performance

Evaluations and attestation of currency

• Assessor recruitment activities

• Participation in assessor moderation and professional development activities


March 2015

97

Quality Assurance Process

To ensure that enhanced assessment operates effectively and that all applicants are treated in a fair, valid and

consistent manner throughout the assessment process, a number of quality assurance requirements have been

established. The ITA Director of Assessment will manage the audit of the Assessment Agency to ensure compliance at

all stages of the assessment process. Note that the audit process could involve an external auditor.

Quality assurance arrangements cover:

• Systems administration for a quality assessment process

• Compliance with jurisdictional legislation and regulatory requirements

• Administrative and records management procedures

• Assessment judgments/decisions

• Access and equity

• Complaint procedures

The following stakeholders are directly involved in the quality assurance process:

• ITA Assessment and Program Standards departments

• Assessment Agency

• Compliance Auditor

1. Administration

The Assessment Agency must operate the enhanced assessment system in compliance with all ITA program

administration policies (PA3001 – 3006) and program policies (PP1000 – 1016). These can be found on ITA’s

website via the following link: http://www.itabc.ca/about-ita/policies .

The Assessment Agency is required to administer the enhanced assessment process in compliance with these

procedures, and in compliance with the ITA Approved Assessment Agency Agreement to which they are a

party. An ITA compliance audit will be conducted a minimum of once per fiscal year.

Should the audit show non-compliance with any ITA policy, non-compliance with these procedures or a

substantial breach of the Approved Assessment Agency Agreement the ITA will develop, in collaboration with

Assessment Agency, an action plan for improvement or, in extreme cases, may recommend suspension or

termination of the Agreement by ITA’s Director of Assessment. Action plans will be based on the

recommendations detailed in the audit report. If the recommendations have not been implemented within the

agreed timeline, ITA may, at its sole discretion, suspend or terminate the Approved Assessment Agreement and

cease accepting assessment results from the Agency.

2. Program Outlines, Occupational Performance Standards and Assessment Tools

Program Outlines, Approved Performance Standards and assessment tools for provincially recognized trades or

occupations must be developed according to the specifications outlined in the ITA Program Development

Process (PDP). New Performance Standards will be approved by the ITA Standards Review Committee (SRC).


March 2015

98

Approval criteria are detailed in the ITA PDP handbook. Revisions to these will be recommended to ITA SRC in

accordance with the agreed program standards review process.

Moderation sessions and assessor meetings should allow for continuous improvement the natural evolution of

the assessment tools. Assessment tools must be developed to Program Development Process specifications

and are auditable by ITA at any time to ensure compliance with development guidelines and appropriate use of

assessment tools.

3. Candidate Documentation and Record Keeping

Record-keeping systems are critical in the assessment process and for the efficient operation of the enhanced

assessment system.

Efficient record-keeping arrangements will be established to cover:

• Processing and administering candidate applications

• Managing the assessment sessions, e.g., scheduling assessors, candidates, venues etc.

• Recording and managing assessment outcomes and issuing results

• Recording and managing reassessment decisions and outcomes

To ensure that the enhanced assessment system operates efficiently, provincial standards in records retention

apply to any information gathered throughout the assessment process. Records will be maintained and securely

stored in ITA Direct Access.

Assessment records include the assessment tools, assessment outcomes and any communications with

candidates. Records include: notes regarding phone or in-person conversations; copies of electronic exchanges

by fax, e-mail, or text; written communications sent by mail, courier, or hand-delivered.

Records of the following will be kept for quality assurance purposes:

• Information provided to candidates

• Versions of assessment tools used

• Records of assessment times, dates, locations, and names of assessor(s) and candidate

• Documentation relating to the assessment decision/judgment

• Documents relating to the assessment process (e.g., comments, complaints, etc.)

• Assessment outcomes

• Records of feedback to candidate

• Credentials issued

Individual assessors will complete all assessment documentation for each assessment to assist with the

provision of feedback to candidates and to clarify issues that may arise in any appeal or reassessment process.

4. Assessor Eligibility and Currency

Assessor eligibility and criteria are recommended by industry and approved by ITA. Assessors must be selected

based on the assessor eligibility criteria and, in order to conduct enhanced assessments, must be registered

with ITA and meet currency requirements. Hard copies of assessor files (including registration forms and


March 2015

99

performance management documentation) must be kept with the Assessment Agency and must be available for

audit by ITA at any time.

Assessor currency requirements are program-specific, and defined in the assessor training application form for

each program. The Assessment Agency is responsible for annual verification of assessor currency. Records of

assessor currency must be available for audit by ITA at any time.

5. System Compliance

The quality assurance strategies of internal and external audit will be used to evaluate compliance with

enhanced assessment system requirements and to promote continuous improvement. Compliance and

continuous improvement will focus on:

• Adherence to policies and procedures

• Record-keeping requirements

• Analysis of candidate feedback and complaints

• Requirements of the agreements established with the assessment venues, the Assessment Agency

and the assessors

Internal Audit

Administration staff involved in the enhanced assessment system will be provided with checklists and guidelines

that enable them to undertake internal audits of relevant processes to ensure compliance against ITA’s

requirements and to identify areas to target improvement. Results of the audits are to be submitted to ITA

annually.

External Audit

To validate the outcomes of internal audits by the Assessment Agency, ITA will undertake its own regularly

scheduled annual audits and spot checks for compliance against enhanced assessment quality assurance

system requirements detailed in Appendix 2.

Assessor Management Responsibilities

Registered assessors who are trained in enhanced assessment methodology play a critical role in competency-based

assessment. After evaluating a range of evidence, they make the professional judgment as to whether the evidence

provided by the candidate meets the standards for certification.

To ensure the consistency and quality of the enhanced assessment process, the Assessment Agency, will recruit

assessors for ITA-approved training and facilitate the ongoing professional development of assessors through

participation in development and delivery of annual moderation activities.

1. Eligibility Criteria3

3 Program-specific eligibility criteria – see Program Outline


March 2015

100

Assessors will be eligible for registration once they have been evaluated on their ability to conduct

assessments, and the specific requirements of each of the following areas have been demonstrated:

• Prescribed years of work experience in the occupation or trade

• Prescribed trade or occupational certification(s)

• Prescribed endorsements (where applicable)

• Prescribed licenses (where applicable)

• Other Industry requirements such as supervisory experience

• Satisfactory criminal record check

• Signed code of conduct, conflict of interest and confidentiality agreements

Note: Criteria are program-specific, recommended by industry and approved by ITA through the Program

Development Process.

An eligible assessor must demonstrate competence in the following areas of the Occupational Analysis Chart:

• Communicates effectively

• Works effectively with others

• Makes professional judgments related to a candidate’s competence

• Follows code of conduct and maintains required confidentiality at all times

2. Training

Assessor training comprises ITA-approved in-class instruction and mentored practice assessments. Assessors

within the enhanced assessment system are required to have successfully completed an approved ITA

assessor training program covering the following key areas:

• Planning assessments

• Conducting assessments

• Making and properly documenting assessment decisions

• Providing support and feedback for assessment decisions

• Contributing to assessment quality assurance processes

• Communicating effectively

In addition, assessors must have current knowledge of industry practice, exhibit strong interpersonal and

communication skills, and exhibit knowledge of language, literacy and numeracy issues in the context of

workplace assessment. Assessors’ readiness to perform solo enhanced assessments will be determined based

on evidence of competence in performing tasks in the Occupational Analysis Chart demonstrated during

mentored assessments.

An Assessor Candidate is eligible for Registration as an ITA-registered Assessor when the candidate

demonstrates to the satisfaction of an experienced assessor supervising mentored assessments the ability to:

• Interpret units of competency

• Select tools to confirm evidence requirements where assessor discretion on tool selection is permitted


March 2015

101

• Use approved assessment methods, tools and procedures to gather and organize evidence

• Apply rules of evidence and principles of assessment to gather evidence

• Evaluate evidence against program performance standards to make a defensible professional

judgment on whether or not the standard is met

• Consult with peers on assessment outcomes and make professional judgment decisions

• Administer assessments according to approved procedures

• Organize and deliver reassessments (if required)

• Document rationale for standard met or not met, and return documentation in accordance within

approved service standards

• Discuss and implement reasonable adjustment strategies for candidates with special needs as

necessary

• Address OHS risks according to workplace and jurisdictional procedures

3. Registration

Upon successful completion of required assessor training, ITA will issue an Individual Identification Number to

an assessor. ITA will maintain a registry of assessors in ITA Direct Access that will include the following

information:

• Individual personal information

• Evidence of completion of the ITA assessor training program

• Records of conducted assessments

• Annual performance evaluation and verification of currency

4. Moderation

Moderation sessions are a venue for quality assurance of the standards, assessment process, assessment

tools, related documentation, assessment methodology and professional judgments, with the intent of

continuous improvement.

The purpose of a moderation session is to provide a consultative forum to:

• effectively compare and evaluate the assessment process, tools and evidence contributing to the

professional judgments made by a range of assessors against the same approved standards, and to

ensure reliable and consistent assessment outcomes

• effectively provide advice on continuous improvement of assessment tools, documentation, industry

standards, and assessment methodology

• provide advice, support, and quality assurance on the planning, implementation and reporting of

assessment activities, including issues with venues, schedules, tools, etc.

• consider issues arising from the assessment activities that may be relevant to group members and

their constituents. This may include discussion of borderline cases that may have challenged

assessors in their decision-making process.


March 2015

102

• maximize opportunities for information sharing about the assessors’ experience of competency-based

assessment

• contribute to the ongoing professional development of assessors and administrators involved in

competency-based assessments. This may include the creation of case studies to facilitate discussion

and assist in reaching consistency in the assessors’ professional judgments.

• develop action plans to improve the quality of assessments and tools

The Assessment Agency will participate in scheduling moderation meeting dates, organizing venues and

related items as required, coordinating with assessors, and helping to plan moderation session agendas and

activities with industry input. Moderation sessions must be conducted at least once per year.

The following roles should be fulfilled at a moderation session:

• Facilitator – ITA Program Standards Manager responsible for assessment standards

• Co-facilitator – Assessment Agency representative responsible for assessment standards

• To be considered an active and current assessor, all registered assessors and mentored assessors (in

training) are required to attend moderation sessions as required by their contractual obligations with

the Assessment Agency. ITA will only accept assessment results where assessments were conducted

by active assessors.

• Other parties may be invited to attend moderation sessionss at the request of the facilitator or the co-

facilitator to provide advice and assistance considered necessary. One of the functions of moderation

is to provide assessors the opportunity to make recommendations to ITA as needed for updates or

changes to assessment processes or standards such as tools. Other than assessors, those attending

meetings are considered to be ex officio and have no voting rights on recommendations for change to

the assessment standards.

ITA is responsible for updating session participants on any proposed or approved changes that relate to

standards, procedures, or documentation, and for updating session participants on the status of previously

submitted recommendations.

The outcomes of a validation process generally result in recommendations for improvements to assessment

tools, processes and outcomes. It is the responsibility of ITA to produce moderation reports documenting

outcomes. This may include recommendations for changes to the:

• Context and condition of the assessment

• Assessment tasks

• Administration processes

• Assessment criteria used for judging the quality of performance (e.g., the decision-making rules,

evidence requirements, etc.)

• Guidelines for making reasonable adjustments to the way that evidence of performance was gathered,

to ensure that the expected standard of performance specified with the unit of competency has not

been altered

• Recording and reporting results


March 2015

103

In addition to making recommendations for improvement to the assessment tools (as outlined above),

moderation may also include giving assessor feedback on past judgments to adjust future assessments.

5. Performance Review

One of the critical cornerstones of any competency-based assessment process is the quality and consistency of

assessors in making judgments, providing feedback and completing administrative duties.

The performance and professional judgment of all assessors needs to be monitored closely by the Assessment

Agency to ensure that standards are being met relating to adhering to roles and responsibilities, preparing

documentation and assessment records, and adhering to the code of conduct at all times. As assessors gain

experience and competency, this review is likely to become less frequent and fall into regular quality assurance

cycles and processes. There are many opportunities for the Assessment Agency to provide feedback to

assessors on their performance, both during initial training and while conducting assessment activities. The

most effective assessment of an assessor’s skills and competency is observation. The Assessment Agency is

responsible for recording any observations of assessor performance that are conducted, and is responsible for

sharing this feedback with the assessor and for providing any relevant information to ITA.

Assessment Agencies must submit to ITA an annual performance evaluation and attestation of currency for

every assessor.

Ongoing Quality Assurance and Currency

When assessors have been registered and are conducting assessments unsupervised by another assessor, the

role of the Assessment Agency becomes one of quality assurance. It is critical to ensure that feedback is given

and corrective action taken, should the following apply:

• Approved tools are not used or are used incorrectly

• Documents not completed correctly, such as providing too little or too much detail in the summary

• Use of unclear language, inappropriate language or comments

• Assessor does not complete assessments within the required time frames

• The assessment summary notes and the assessment decision are at odds or would not be

supportable upon request for reconsideration or appeal of a decision

• Complaints are filed about process or results

• Inconsistent quality of decision making among assessors

Assessors need to maintain currency of their industry knowledge as well as their assessment competence.

Currency means awareness of the up-to-date industry practice, the workplace and the job/role of the persons

being assessed against the Approved Performance Standards.

Assessors also need to ensure that their assessment competence is current. Self-assessment is an integral

strategy for an individual assessor’s professional development. It involves assessors reflecting on their own

skills, knowledge and attributes and determining where they are in relation to where they need to be and

participating in assessor moderation activities.


March 2015

104

In cases of conflict or complaints about the assessment process, the Assessment Agency and ITA will work

collaboratively to ensure that all concerns are addressed in a timely fashion (see Complaints and Appeals

section).

Audit Checklist

Assessment agency

� Assessors administering enhanced assessments meet program-specific eligibility criteria, as prescribed in

the assessor guide, and have completed the assessor training program

a. Assessor’s file contains registration form demonstrating eligibility requirements

b. Records demonstrating successful completion of the assessor training course and mentored

practicum assessments

� Assessor attendance at a minimum of one assessor moderation session within a fiscal year and record of

conducting a required number of assessments (varies by program)

a. Record of assessments and attendance at moderation sessions

� Feedback on assessors performance delivered to assessors on a regular basis

a. Record of assessor reviews and feedback – particularly following initial assessments (new

assessors)

� Annual performance review and attestation of currency completed and submitted to ITA


March 2015

105

Appendix 1 – Glossary of Terms

TERM DESCRIPTION

Assessment The process of reviewing and evaluating the skills, knowledge and qualifications of

a candidate against predetermined workplace performance standards. This process

could include: practical assessments, written assessments, technical interviews and

evaluation of prior work experience, learning or credentials.

Assessment Agency An organization which has entered into an Approved Assessment Agency

Agreement with ITA to manage enhanced assessment activity for specified trades

or occupations in accordance with ITA’s Enhanced Assessment Operational

Procedures.

Assessment method Technique used to gather different types of evidence, which may include

questioning, observation, third-party reports, interviews, simulations and portfolios.

Assessment resources Any materials that assist in any part of the assessment process. These may include

information for the candidate or assessor, assessment tools, or any other resources

for the quality assurance arrangements of the assessment system.

Assessment plan The approach to assessment and evidence gathering used by the assessor,

including the assessment process, methods and assessment tools.

Assessment process The series of steps that the candidate undertakes within the application,

assessment, recording and reporting cycle.

Assessment system A controlled and ordered process designed to ensure that assessment decisions

made in relation to many candidates, by many assessors, in many situations, are

consistent, fair, valid and reliable.

Assessment tool Contains both the instrument and the procedures for gathering and interpreting

evidence:

• Instrument – specific questions or activity developed from the selected

assessment methods

• Procedures – the information given to the candidate and/or assessor on

conditions under which the assessment should be conducted and recorded

Assessor An individual who has successfully completed an ITA Assessor Training program

and is registered with ITA to evaluate competencies of individuals seeking

certification in specified trades. ITA Authorization to perform assessments is

restricted to assessors who meet program-specific currency requirements.

Certificate of Qualification The credential awarded by ITA to individuals who have successfully completed the

requirements of an Accredited Industry Training Program or a Recognized Industry

Training Program.


March 2015

106

TERM DESCRIPTION

Candidate Any individual who has amassed sufficient practical work experience to meet the

established ITA trade-specific criteria to attempt the certification in the trade and

who has obtained their skills and knowledge outside the B.C. apprenticeship

system.

Competency A competency is the smallest unit of job activity that results in a meaningful

outcome (product, service or decision). Competencies describe what the individual

is able to do. They must be observable and measurable. Competencies are

standards requiring the approval of ITA’s Standards Review Committee.

Credential Issued by ITA, indicating that the recipient has met completion requirements for the

program and therefore has demonstrated the competencies, skills and knowledge

associated with the occupation. A credential can be an awarded certification or an

advanced credential. Awarded certifications and advanced credentials are


Dimensions of

competence

The concept of competence includes all aspects of work performance, not just

narrow task skills. The four dimensions of competence are task skills, task

management skills, contingency management skills and job/role environment skills.

Element The basic building blocks of the unit of competency that describe the tasks that

make up the broader function or job, described by the unit.

Enhanced Assessment A structured evaluation process of candidates’ knowledge, skills and attributes

against program standards that involves different assessment methodologies

beyond the written multiple choice exam. e.g., practical assessment, technical

interview, or portfolio review.

Evidence Information that is used to demonstrate competence against agreed standards.

Quality evidence is valid, authentic, sufficient and current.

Gap assessment Assessment of areas of competence in which the candidate was unable to provide

sufficient evidence to meet certification standards in a previous assessment.

Holistic assessment An approach to assessment that covers the clustering of multiple units/elements

from relevant Occupational Performance Standards and focuses on the assessment

of a ‘whole of job’ role or function that draws on a number of units of competency.

Individual Identification

Number

The unique 6-digit identifier for an individual registered in ITADA. All apprenticeship

and certification-related records for the individual bear this identifier. Formerly called

Trade Worker Identification Number (TWID) or Registration Identification Number.

Industry Training

Authority (ITA)

The B.C. Crown corporation that manages B.C.’s Industry Training, Apprenticeship

and Trades Credentialing system. ITA is governed by the Industry Training

Authority Act [SBC 2003] Chapter 34.


March 2015

107

TERM DESCRIPTION

Industry Training

Organization (ITO)

A not-for-profit legal entity incorporated under the provincial Societies Act that

contracts with ITA under an Enterprise Partnership Agreement to take lead

responsibility for industry training within a particular sector. Established and

operated by industry and with a province-wide mandate, ITOs are financially

supported by industry and through Enterprise Partnership Agreements with ITA.

ITA Direct Access (ITADA) The electronic management information system used to house all of the training

records for individuals completing industry training programs and for individuals who

receive certification.

Moderation A process that involves assessors in discussing and reaching agreement about

assessment processes and outcomes in a particular industry.

National Occupational

Analysis (NOA)

A detailed listing of the competencies required for an occupation for an Accredited

Industry Training Program that is nationally developed under the Interprovincial Red

Seal Program. The NOA organizes competencies in blocks, tasks and sub-tasks.

B.C. Industry Training Programs are designed around the Occupational Analysis

Chart (OAC), often derived from the NOA. The OAC organizes competencies as

general areas of competency (GAC) and competencies. Competencies are


Occupational Performance

Standards (OPS)

Statements developed by Industry and approved by ITA that describe effective

performance in the workplace. These are sometimes expressed as units of

competency that describe an aspect of work in a particular trade, what is to be

assessed and the required level of performance.

Performance criteria Evaluative statements that specify what is to be assessed and the required level of

performance.

Portfolio of evidence A package consisting of primary and third-party evidence from prior experience and

training submitted to demonstrate competence.

Practical Assessment An observation of a candidate’s performance, skills or knowledge to the standard

expected in the workplace, as expressed in the relevant industry program

standards.

Program Outline A communications document that represents the full scope of knowledge and skills

required to competently perform an occupation anywhere in B.C. It defines the

knowledge and skills to be gained through training, but does not contain

instructional procedures.

Program Profile A communications document that describes all of the credentialing pathways,

credentialing model, completion requirements and credentials associated with a

program.


March 2015

108

TERM DESCRIPTION

Quality Assurance Quality assurance involves a planned and systematic auditing process of ensuring

that the requirements of the assessment system, Occupational Performance

Standards and any other criteria are applied in a consistent manner. Quality

assurance mechanisms or procedures are an integral part of the assessment

system.

Reconsideration Reconsideration by the CEO of the results of an assessment review is available

upon request by a candidate.

Records of assessment The information of assessment outcomes that is retained by the organization

responsible for issuing the transcript of results or credentials.

Red Seal Program A standardized national endorsement for a specific trade that enables

interprovincial and inter-territorial mobility. On successful assessment, a nationally

recognized Red Seal endorsement is affixed to the provincial Certificate of

Qualification.

Reporting assessment

outcomes

The different ways in which the outcomes of assessment processes are reported to

the person being assessed, employers and other appropriate personnel or

stakeholders.

Review Review of an assessment decision by a different Assessor is available upon request

by a candidate.

Risk management The systematic application of management policies, procedures and practices to

the tasks of identifying, analyzing, evaluating, treating and monitoring risk.

Self-assessment A process that allows candidates to collect and provide evidence on their own

performances against the Occupational Performance Standards.

Simulation A form of evidence gathering that involves the candidate completing or dealing with

a task, activity or problem in an off-the-job situation that replicates the workplace

context.

Stakeholders Assessment Agencies, Industry training organizations, industry employers,

sponsors of apprentices, training institutions, assessors and the Canadian Council

of Directors of Apprenticeship.

Standards ITA is responsible for establishing the standards to which apprentices and

tradespeople must perform in order to receive ITA certification in an occupation or

trade. These standards, often referred to as program standards, include

credentialing, assessment and achievement criteria designed to ensure that B.C.’s

candidates for assessment (whether apprentices or challengers) can demonstrate

their performance at levels established by national and provincial industry groups.

All program standards require the approval of ITA’s Standards Review Committee.


March 2015

109

TERM DESCRIPTION

Standards Review

Committee (SRC)

ITA Committee responsible for the oversight of standards, as part of the ITA’s

regulatory responsibility. The Standards Review Committee is also responsible for

ensuring that the ITA processes for establishing, developing and managing new

and existing programs and assessment methods are followed by industry partners.

Unit of competency Describes an aspect of work in a particular occupation or industry and used as a

benchmark for assessment of competence.

Validation Involves reviewing, comparing and evaluating assessment processes, tools and

evidence contributing to judgments made by a range of assessors against the same

standards. Validation strategies may be internal processes with stakeholder

involvement or external validations with other providers and/or stakeholders.


March 2015

110

Quality Assurance of Enhanced Assessment for Candidates

Audit Checklist

ASSESSMENT AGENCY

� Materials are kept up to date

a. Published challenger information kit

b. Published assessor guide

c. Assessment tools currently in use (current version matched with ITA)

d. Occupational Performance Standards currently in use (current version matched with ITA)

Administrative Process in place4

� Provide advice to candidates on how to collect evidence and build portfolio

a. Process identifying a staff person/back-up and other resources to aid candidates throughout the application and portfolio building process

� Process for receiving application (date stamp) and candidate file opened

� Candidate information management system to track intake of applications (i.e. excel spreadsheet, ITADA

notes)

� Inform ITA to register candidate in ITADA [ITA receiving initial application and fee, then forwarding to

Assessment Agency]

� Service standard of 5 business days from receipt of application to submission of information to ITA for

registration

� Process for reviewing application for completeness and relevancy to the applied program and assignment to

assessor for portfolio review

o Identify staff person who is conducting the review and methodology

� Communication of eligibility for assessment or request for further action from candidate, and where eligible,

scheduling of assessment dates and venues.

o Letters or assessment voucher communicating acceptance and notification of assessment

Scheduling

� Venue meets the industry standard for assessment activity

o Up to date venue industry requirements and OHS

o Completed checklist for each venue to ensure they meet the requirements (with dates and venue

evaluation process identified)

4 Appropriate notes must be captured in ITADA documenting every step of the process


March 2015

111

o Formal venue complaints investigated and documented in accordance with ITA Operational

Procedures

� Assessment Agency schedules assessment activity and secures venue etc. and assigns candidate to an assessor

o staff person/back-up and other resources are in place to successfully execute the activity o NOTE Assessment date is based on venue schedules and candidate availability and where

possible, candidate preferences

o Service standard of 5 business days from ITADA registration to assessment scheduled

� Assessment Agency makes contractual arrangement with venue o staff person/back-up and other resources are in place to successfully execute the activity o Ensure contracts are in place

� Receive and review documentation of assessment results, upload results to ITADA, and report results to ITA

o staff person/back-up and other resources are in place to accept and review assessment results

o Information management system to track receipt of completed assessment tools and any further

evidence gathered throughout the assessment activity

o Service standard of 15 business days from assessment completion to reporting results to ITA

� Document complaints o Service Standard in accordance with ITA Enhanced Assessment Operational Procedures

INDUSTRY TRAINING AUTHORITY

� Materials are kept up to date

b. Published Occupational Performance Standards

c. Assessment tools currently in use

d. Version control protocols are in place

� Process for registering application (date stamp) and candidate file opened

a. Candidate information management system to track intake of applications (i.e. excel spreadsheet,

ITADA notes)

b. Process for registering candidate in ITADA

c. Service standard of 5 business days from notification of eligibility to challenge to registration

� ITA process assessment results and issue notification and/or credential a. Information management system to track results (i.e. excel spreadsheet, ITADA)

b. Service standard of 5 business days from receipt of results to entering the competencies in ITADA

� Document complaints and follow approved ITA procedures a. Service Standard for responding to complaints in accordance with Enhanced Assessment

Operational Procedures


March 2015

112

Appendix 3 – ITA Enhanced Assessment Complaints and Appeals Procedure

ITA Enhanced Assessment Complaints and Appeals Procedure

BACKGROUND

Part 4, Section 11 of the Industry Training Authority Act, and ITA Operations Policy PA3005 – Appeals / Dispute Resolution

govern Reconsiderations and Appeals of ITA decisions. Special administrative procedures apply to complaints related to

enhanced assessment and to requests for CEO reconsideration of decisions made by ITA-registered assessors. These

administrative procedures are documented in ITA’s Operational Procedures for the Enhanced Assessment of Challengers.

This Enhanced Assessment Complaints and Appeals Procedure defines the roles and interaction between ITA, the

Approved Assessment Agency or Industry Training Organization (AAA / ITO).

PRINCIPLE

ITA and the AAA / ITO have a mutual commitment to provide professional and timely service to any individual who raises a

concern about service, assessment, or procedures. The goal is to verify and address all issues brought forward by a

complainant in a timely and customer-centric manner, with escalation to senior management as appropriate.

RATIONALE

Information sharing and timely communication amongst all parties to the complaint is critical for fair and timely resolution of

complaints, so it is important that there is an agreed-to protocol and service standard for information sharing,

documentation, response, and resolution of complaints.

SCOPE

This procedure applies to two types of complaints. One relates to assessment processes and fairness issues; and the other

relates to competencies not granted. It is possible for either one or both types of complaint to arise from a single

assessment.

Under these Procedures, if a complaint cannot be resolved, and a challenger requests CEO reconsideration of an assessor

decision not to grant one or more competencies, a formal review of the assessment result is performed only for

competencies not achieved. In no case will a competency previously granted be revoked as a result of the review. The

review is conducted by a panel of a minimum of three ITA-registered or HRSDC-registered assessors (reviewers), one of

whom is nominated as chair by the review panel. No assessor involved in the assessment under review may serve as a

member of the review panel. Each of the three reviewers independently reviews documentation from the contested

assessment. The assessors then jointly determine the outcome of the review, making a recommendation to the CEO to

either uphold or amend the results. Each reviewer submits an independent review document, and the chair prepares and

submits the final joint panel recommendation to the CEO.

Complaints Related to Assessment Process and Fairness Complaints Related to Competencies Not Granted

Process and Fairness complaints include, but are not

limited to issues concerning:

� Venues � Equipment and supplies � Assessor professionalism

Complaints regarding competencies not granted are

subject to formal appeal to the ITA Appeal Board in

accordance with Part 4, Section 11 of the Industry

Training Authority Act, and ITA Operations Policy


March 2015

113

� Confidentiality � Discrimination � Safety � Time and scheduling

PA3005, Appeals / Dispute Resolution.

COMPLAINT REVIEW TEAM

The following individuals serve as the Enhanced Assessment Complaint Review Team (CRT):

� ITA Manager of Policy and Research � ITA Director of Assessment � ITA Program Standards Manager responsibility for oversight of the program � Program or Customer Service Manager at the relevant AAA / ITO

All members of the CRT need to be aware of the complaint in order to work together to respond and resolve the complaint in

an appropriate and timely manner. Therefore all complaints or issues raised by a challenger will be logged by the receiving

party and communicated by e-mail or telephone to immediately inform the other parties who make up the CRT) of the

receipt and nature of the complaint. A detailed record of the complaint will be recorded via “Notes” on ITA DirectAccess

within 2 business days of receipt.

Complaints may be able to be resolved relatively quickly by agreement of the parties; but where this is not the case, a

formal procedure for managing escalation is required. From the time a complaint is received until it is resolved, there are

three management levels for resolution. These are:

1) resolution by agreement of the CRT and complainant; 2) review and decision by the COO; and 3) final determination by the CEO.

1


March 2015

114

ESCALATION MATRIX

Complaint Received

Level 1 - Initial Complaint

Level 2 - COO Review

Level 3 - CEO Review

Complainant contacts

ITA or AAA / ITO

Complaint immediately

referred to CRT made up of

ITA Manager of Policy and

Research, ITA Program

Standards Manager, ITA

Director of Assessment, and

Program Mgr at relevant AAA

/ ITO for review and decision.

If not resolved, move to level

2.

Complaint referred to ITA COO

for review and decision.

If not resolved, move to level 3.

ITA CEO for final

determination.2

1 NOTE: If complainant is appealing a decision regarding units of competency not granted then the complainant will

be provided with the procedure as to how to file an appeal with the ITA Appeal Board. 2As Above: competency not granted is subject to formal appeal to the ITA Appeal Board.

PROCEDURE

Level 1 – Initial Complaint Management By CRT

1. Complaints received by ITA or the AAA /ITO must be logged in ITA DirectAccess, and must be immediately referred to the CRT. (Documentation received by ITA will be put into ITA Service Requests (aka.Biz) which will automatically populate ITA DirectAccess.)

a. ITA Supervisor, Quality Service to assign a staff member to scan and save all documents into ITA DirectAccess and distribute via email to Complaint Review Team.

b. PSM organize a conference call for joint review and resolution of the complaint. Review includes, but is not limited to:

• Agreeing on method of verifying complaint • Review of all documentation/evidence • Development of a written action plan

c. PSM will contact the complainant within 2 business days, by phone or e-mail to: • acknowledge the complaint has been received and will be actively reviewed by the CRT • gain additional information regarding events leading up to the complaint • confirm what resolution is being sought • provide a single point of contact for any further communication on this issue • describe next steps and approximate timelines for the complaint review process • In accordance with the Operational Procedures for the Enhanced Assessment of

Challengers, If the resolution being sought by the complainant is regarding competency not granted, then an assessment review panel of 3 independent assessors who were not involved with the initial assessment will be engaged to review the file and make a judgment regarding competency for each contested block or unit of competency.

i. All assessment documentation, including the report from the assessment review panel, to be filed and forwarded to the CEO to assist in the CEO’s decision regarding a request for reconsideration.


March 2015

115

ii. inform the complainant about the review process and that an initial response will be sent within approximately 30 business days

d. If the contact referenced in section c. above was by telephone, PSM to send follow-up e-mail, courier, or registered letter summarizing the conversation and next steps. This will be copied to the CRTand saved in ITA DirectAccess.

e. CRT to meet by phone or face-to-face within 5 business days of being notified of the complaint to discuss the complaint to determine a plan to communicate back to the complainant and to resolve the issue. The agreed-upon plan must be logged in ITA DirectAccess.

2. The PSM will coordinate all further activity in consultation with the Manager, Policy and Research, to ensure: a. Follow-up actions are completed b. A joint resolution of the CRT is agreed-upon c. A formal response letter is approved by the CRT

3. The PSM will communicate the decision to the complainant verbally, and send the formal response letter by e-mail,

courier, or registered letter.

4. All communications and documentation of any kind must be retained for evidence (including informal e-mails, notes regarding telephone conversations, letters, and records of any kind, including audio tapes and photographs of assessment activities or outcomes.

5. The owner of the Complaint procedure will be the Manager, Policy Research who will be responsible for ensuring the process is adhered to, leading the CRT meetings and coordinating records of complaints received and actions taken.

Level 2 – COO Review

1. If the complainant is not satisfied with the decision of the CRT the complaint will be referred to the ITA COO for review and decision.

2. All documentation will be provided to the COO for review.

Level 3 – CEO Review

1. If the complainant is not satisfied with the decision of the COO, the complaint will be referred to the ITA CEO for reconsideration and decision.

a. All documentation will be provided to the CEO for review The CEO decision is final except as noted above where competencies not granted are subject to formal appeal to

the ITA Appeal Board.


March 2015

116


March 2015

117


March 2015

118


March 2015

119


March 2015

120


March 2015

121


March 2015

122


March 2015

123


March 2015

124


March 2015

125


March 2015

126


March 2015

127


March 2015

128


March 2015

129


March 2015

130


March 2015

131


March 2015

132


March 2015

133


March 2015

134


March 2015

135


March 2015

136


March 2015

137


March 2015

138


March 2015

139


March 2015

140


March 2015

141


March 2015

142


March 2015

143


March 2015

144


March 2015

145


March 2015

146


March 2015

147


March 2015

148


March 2015

149


March 2015

150


March 2015

151


March 2015

152


March 2015

153


March 2015

154


March 2015

155

Enhanced Challenge Pathway Project Completion Report ECP Project... · Enhanced Challenge Pathway...

Documents

Transcript of Enhanced Challenge Pathway Project Completion Report ECP Project... · Enhanced Challenge Pathway...