CESA 2017 Survey Results APA’s Office of Continuing Education Sponsor Approval

Purpose

• To evaluate needs of Continuing Education (CE) sponsors

• To improve effectiveness and performance of Office of CE Sponsor Approval

• To measure level of satisfaction with new online application system (CESA OAS)

• To enhance available resources for CE Sponsors

Overview

I. Sponsor Demographics

II. Program Offerings

III. Application Process

IV. Annual Reporting Process

V. CESA OAS

A. Application

B. Annual Report

C. General Features

VI. Resources: Marketing & Communication

VII. Reflections: Strengths & Anticipations

A. Strengths

B. Anticipations and Next Steps

Sponsor Demographics

Response Rate

• 166 Total Respondents1

– 126 complete

– 40 partial2

– 0 disqualified3

Complete

76%

Partial

24%

1Total response rate was 20.47% and represents 166 out of 811 who received the survey for completion.2Partial=submission page not reached3Disqualified=response eliminated by survey logic or quota

Approval Status of Current Sponsors

Newly Approved

15%

Renewing Approval

85%

Approval Status

23%

18%

29%

10%

20%

Duration of Approval Status

0 to 3 years 4 to 8 years 9 to 14 years

15 to 19 years 20+ years

n=157 n=154

Lapses in Sponsorship

33%

22%

11%

6%

28%

Unsuccessful

application

Missed

application

deadline

Insufficient

resources

Did not want

to re-apply

Other

Reasons for Lapse in Sponsorship

Yes

13%

No

82%

Unsure

5%

n=134 n=18

CE Program Offerings

Current CE Program Offerings of Sponsors

Currently offer CE

95%

Do not currently

offer CE

5%

63%

25%

13%

Newly approved

Lack of

interest/audience

Insufficient resources

Why Not?

n=151 n=8

Types of Sponsor Applications

64%6%

24%

7%

CESA only (live CE offerings)

Homestudy only (any CE program where there is no live

interaction with presenter)

CESA and Homestudy Supplemental (simultaneous

applications)

CESA and Homestudy Supplemental (Homestudy

Supplemental added after CESA approval)

96%

11% 7% 6% 9%

Main Reasons for Type(s) of Application(s)

n=151 n=71

Homestudy Offerings of CE Sponsors

57% 57%

39%

32%

11%7%

Online course Pre-recorded

event (e.g.,

workshop,

webinar)

Video on-

demand (i.e.,

digital video

accompanied by

PowerPoint)

Publication (e.g.,

book, magazine,

journal, article)

Audio-only

recording (e.g.,

CD, podcast)

Other

n=110

Target Audiences of CE Programs

99%

83%

74%

60%

48%

46%

30%

21%

18%

15%

Psychologists

Social workers and/or human services workers

Mental health counselors

Marriage and family therapists

Graduate or doctoral students

Psychiatrists

Physicians, nurses, and other medical personnel

Educators

Undergraduate students

Othern=143

1Examples of “other” include: occupational therapists, speech pathologists, attorneys, first responders, court reporters, and juvenile justice workers

1

Application Process

Types of Applications Previously Completed by Sponsors

54%

14%

13%

11%

5% 4%

1%

CESA OAS

(2016 online

system)

Paper Paper + CESAS

(2011 system) +

CESA OAS

(2016 system)

Paper + CESA

OAS (2016

system)

Paper + CESAS

(2011 system)

CESAS (2011

online system)

CESAS (2011

system) + CESA

OAS (2016

system) n=141

Time Spent on OAS Applications

65%13%

23%

Time Spent on OAS Application vs Other

Formats

Less time

More time

The same amount

of time

n=31

15%

25%

14%

8%

38%

Time Spent on Most Recent (OAS)

Application

0-5 hours

6-10 hours

11-15 hours

16-19 hours

20+ hours

n=138

Satisfaction with Application Process via OAS

2%

n=109

9%

18%

53%

17%

Very Dissatisfied

Dissatisfied

Neutral

Satisfied

Very Satisfied

Desired Additional Resources in Application Process

n=138

48%

41%38%

34%31%

10%

20%

Sample

promotional

materials

Flowchart or

timeline of

approval

process

Online

tutorials

Educational

and

Technical

Assistance

Glossary of

CE terms

None Other

Annual Reporting Process

Annual Report (AR) Experience

Yes

76%

No

21%

Don't know 4%

Sponsors Who Completed ARs

Less time

51%

More

time 12%

The same

amount

of time

19%

N/A 18%

Time Spent on AR Submission via

OAS vs Previous Submissions

n=139n=104

Satisfaction with Annual Report Process via OAS

2%

n=104

Very Dissatisfied

Dissatisfied

Neutral

Satisfied

Very Satisfied

10%

23%

48%

17%

Efficiency of Annual Reporting (AR) Process

2%

7%

20%

36%

19%

16%

76%

4%

1%

19%

Is the current AR format effective?

Keep as is

Include

portfolio/other

materialsInclude site

visits

Other

n=129

n=97

1Examples of “other” include: requiring fewer submissions, collecting samples of promotional materials, requiring submission of new activities only, and improving uploading process

1

Is the AR an effective quality assessment?

CESA OAS

Introduction to CESA OAS

This section reflects CE sponsors’ experiences using the online application system (OAS). The chart below represents the processes that sponsors have conducted via the OAS.

Completed an

application for

sponsorship

approval 27%

Completed

an annual

report 28%

Both

(application

and annual

report) 46%

n=131

CESA OAS: Application

This section highlights features of the OAS specifically related to the application for sponsor approval.

Ease of Application Submission

n=93

3%

15%

25%

40%

17%

Very difficult Difficult Neutral Easy Very easy

Helpfulness of OAS Features Related to Application

3% 4%7% 6%

13% 14%

26%

21%

42%39%

24%

39%

25%

21%

10%

19%

2% 2%4% 3%

14%

19%

29%

12%

Hyperlinked

appendices for

relevant standards

(located in top left-

hand corner under

standard)

Hyperlinked

Educational and

Technical Assistance

for each standard

(located in top left-

hand corner under

standard)

"Saving vs Validating"

video

"CESA OAS

Application Tips and

Information" section

(located in "Info" tab)

1 (Not at all helpful)

2 (Not helpful)

3 (Neutral)

4 (Helpful)

5 (Very helpful)

N/A

Did not know about

n=91

n=90

n=90

n=90

CESA OAS Application Pre-Checklist

Yes

49.60%

No

50.40%

Use of App Pre-Checklist

Neutral

13%

Helpful

63%

Very

helpful

24%

Helpfulness of App Pre-Checklist

n=127 n=63

1Of those who used the checklist, none ranked it below neutral (on 5-point scale from “not at all helpful” to “very helpful”)

CESA OAS: Annual Report

This section highlights features of the OAS specifically related to the annual reporting process.

Ease of Annual Report Submission

n=95

1%

6%

23%

51%

19%

Very difficult Difficult Neutral Easy Very easy

Helpfulness of OAS Features Related to Annual Reports

n=91

n=92 n=90n=91

1% 1% 2% 3%3% 2%4%

7%

25%

21%23%

18%

43%

47%

34%

44%

20%

15%18%

23%

2% 2% 3%1%

5%

11%

15%

3%

Hyperlinked "CESA

Instructions for Annual

Reporting" document

(located in top left-hand

corner)

"Report Quick Info"

section

Option to view legacy

attachments

Option to manage

attachments

1 (Not at all helfpul)

2 (Not helfpul)

3 (Neutral)

4 (Helpful)

5 (Very helpful)

N/A

Did not know about

CESA OAS: General Features

This section highlights general features of the OAS.

Ease of Logging Into OAS

0%

8% 7%

46%

40%

Very difficult Difficult Neutral Easy Very Easy

n=131

Ease of Navigation through OAS Tabs

1% 1% 1%2%

7%5%

3%5% 4%

12%

17% 16%

19%17% 16%

42%39%

34%32%

33%

42%43%

33%31%

39%

43%

36%

1%

4%

13%

7%

2%1%

Profile Applications Annual reports Certificates Payments Overall

navigability

1 (Very difficult) 2 (Difficult) 3 (Neutral) 4 (Easy) 5 (Very Easy) N/A

n=129 n=128 n=126 n=127

n=128

n=128

Helpfulness of General Features

1% 1% 1% 1% 1% 1%

6%2%

2%3% 2% 2%

25%21%

18%

27%

19%

12%

43% 42%

37% 37%

47%

40%

12%

19%

26%

15%

25%

39%

1% 1% 2% 2% 0% 1%

13% 14% 14% 15%

6% 6%

CESA OAS

updates

"Quick Tips"

sections

"Notice" sections Action Center "General

Information and

Guidelines" page

Ability to review

applications and

reports as read-

only documents

after submission

1 (Not at all helpful) 2 (Not helpful) 3 (Neutral) 4 (Helpful) 5 (Very Helpful) N/A Did not know about

n=126n=126

n=127 n=126

n=125

n=125

Submission of Payments

1%

2%

25%

73%

Wire Transfer

Electronic funds

transfer (EFT)

Check

Credit card

Preferred Method of Submission

36%39%

19%

4%2%

Very easy Easy Neutral Difficult Very

difficult

Ease of Submission

n=130 n=131

Resources: Marketing & Communication

Resources: Marketing

Yes

49%

No

51%

Use of APA Approval Icon

17%

35%

48%

Website Promotional

materials

Both (website and

promotional

materials)

Placement of Icon

n=123n=60

Resources: Communication

48%

26%

23%

3%

Go to the website

Call

Email

Other

Sponsors’ Preferred Methods

n=122

5%

41%

9%

39%

6%

Weekly Monthly Several

times a

month

Annually Less than

annually

Frequency of Website Visits

n=122

Mechanisms to Ensure Adherence to Deadlines

76%

43%

29%

15%

12%

0.8%

2%

CESA OAS email reminder

Personal calendar reminder

CESA website

Contact Office of CESA

CESA OAS Action Center

None

Other

n=123 (respondents chose all that applied)

Topics that Bring Sponsors to CESA Website

71%

59%56%

44%41%

35%

3%

Application

deadline

dates

FAQ sections Resources for

CE sponsor

approval

Updates CESA contact

information

CESA OAS

information

Other

n=117 (respondents chose all that applied)

Use of CESA Resources

88%

77%

75%

66%

48%

37%

31%

Policies and Procedures or Standards and Criteria

Annual report instructions

Sample documents

OAS login link

Estimated timeline for application cycles

Reference guides

List of current APA-approved Sponsors

n=65 (respondents chose all that applied)

Most Frequently Asked Questions by Topic Area

62% 59% 56% 53%43%

29%

18%12%

8%3%

n=120 (respondents chose all that applied)

Reflections: Strengths & Anticipations

StrengthsThis section discusses several strengths of the CESA Office and OAS that were highlighted by the survey findings.

Strengths

• 70.6% of respondents were satisfied or very satisfied with the application process via OAS (n=109)1, compared with 16% (n=56) and 25% (n=28) of respondents who were equally as satisfied with the previous paper and CESAS processes, respectively.

• 65.4% of respondents were satisfied or very satisfied with the AR process via OAS (n=104)2

Sponsor levels of satisfaction with

application and AR formats

• 64.5% of respondents spent less time on the application via OAS compared to other methods (n=31)3

• 51% of respondents spent less time on the annual report via OAS compared to other methods (n=104)4

Decreased time spent on applications and annual

reports via OAS

• 57% of respondents rated application submission as easy or very easy (n=93)5

• 69.5% of respondents rated AR submission as easy or very easy (n=95)6

Ease of application and annual report submission

via OAS

1See Slide 16 2See Slide 21 3See Slide 15 4See Slide 20 5See Slide 26 6See Slide 30

Helpfulness of OAS Features

• The OAS offers a collection of general features to assist sponsors in the application and AR processes.

• These features include: OAS updates, “Quick Tips” sections, a “Notice” section, the Action Center, the “General Information and Guidelines” page, and the ability to review applications and reports as read-only document.

• All features were rated as helpful or very helpful by the majority of respondents1.

General Features of OAS

• The OAS offers a collection of features specifically related to the application process designed to aid sponsors as they complete their applications.

• These features include: hyperlinked appendices for relevant standards, hyperlinked Educational and Technical Assistance for each standard, the “Saving vs Validating” video, and the “CESA OAS Application Tips and Information” section.

• Most features were rated as helpful or very helpful by the majority of respondents2.

OAS Features Related to

Application

• The OAS offers a collection of features specifically related to the AR process designed to aid sponsors as they complete their ARs.

• These features include: hyperlinked “CESA Instructions for Annual Reporting,” the “Report Quick Info” section, the option to view legacy attachments, and the option to manage attachments.

• All features were rated as helpful or very helpful by the majority of respondents3.

OAS Features Related to Annual Reports

1See Slide 35, responses of 4 or 5 on 5-pt. scale of helpfulness counted2See Slide 27, responses of 4 or 5 on 5-pt. scale of helpfulness counted

3See Slide 31, responses of 4 or 5 on 5-pt. scale of helpfulness counted

Resources for CE SponsorsSeveral resources preferred

or requested by sponsors are currently available on the

CESA website

These resources include:

Flowchart/timeline of approval process

Requested by 41.3% of respondents1

Educational and Technical Assistance

Requested by 34.1% of respondents1

Glossary of CE terms

Requested by 31.2% of respondents1

Other requested resources are currently in development (See Slides

49-52).

1See Slide 18

Anticipations and Next Steps

This section discusses anticipated future actions of the CESA Office to better support and provide guidance to CE Sponsors.

Key Areas for Development: Awareness of OAS Features and CESA Resources Application Pre-Checklist1

• 50.4% of respondents have not used the application pre-checklist.

• Out of 49.6% of respondents who have used the checklist, 87.3% rated it as helpful or very helpful

• The remaining 12.7% rated it as neutral; no respondents rated it as unhelpful or very unhelpful.

Educational and Technical Assistance (ETA)2

• 18.89% of respondents were unaware of the ETA offered with each standard (hyperlinked under standard title and included in Standards and Criteria for approval of Sponsors of Continuing Education for Psychologists).

• Out of 78.89% of respondents who rated the helpfulness of ETA, 60% rated it as helpful or very helpful.

Option to view legacy attachments on AR3

• 15.22% of respondents were unaware of the ability to view legacy attachments on ARs.

• Out of 81.52% of respondents who rated the helpfulness of this option, 52.17% rated it as helpful or very helpful.

OAS Action Center4

• 15.1% of respondents were unaware of OAS Action Center (found on profile page of OAS).

• Out of 83.3% of respondents who rated the helpfulness of the Action Center, 52.4% rated it as helpful or very helpful.

1See Slide 28 2See Slide 27 3See Slide 31 4See Slide 35

Key Areas for Development: Visibility, Accessibility, and Redundancy

Increase visibility and accessibility of CESA resources

• Sign-posting sponsors to improve visibility and accessibility of all existing resources

Build in redundancies for adhering to deadlines

• Sponsors cite OAS email reminder as the most-utilized mechanism for adhering to CESA deadlines (used by 76% of respondents)1.

• CESA Office strongly recommends that all sponsors create additional reminders independent of CESA Office and OAS, such as personal calendar reminders (currently used by 43.1% of respondents)1.

1See Slide 40

Next StepsOngoing

review of and updates to

online application

process based on sponsor feedback:

First application updates will go live on November 1st

Ongoing review of

resources for CE sponsors:

Inventory and development of general resources as well as

those focusing on the application and annual

reporting processes

Development of online tutorial video series is underway

Thank you!