SAA Peer Review of State Assessment Systems PPT 10-8 ... - ed · PDF fileED does not review or...

PEER REVIEW OF STATE

ASSESSMENT SYSTEMS

Overview of Peer Review

of State Assessment

Systems October 8, 2015

AGENDA

1. Purpose and role of peer review

2. Understanding the process

3. Timeline for peer review

4. Understanding the critical elements

a. What’s new?

b. Map of critical elements

c. How to read the critical elements

d. Introduction to the sections

5. Future supports for States

6. Questions

3

PURPOSE AND ROLE OF PEER REVIEW

States have been required to have statewide assessment

systems since the 1994 reauthorization of the ESEA

Section 1111(b)(3) of the ESEA lays out the requirements of

state assessments:

• The same annual assessments used to measure the

achievement of all students in reading/language arts and

mathematics in grades 3-8 and once in high school and in

science at least once in grades 3-5, 6-9, and 10-12

• Aligned with the full range of the State’s academic content

standards

• Valid, reliable, and consistent with relevant, nationally

recognized professional and technical standards for the

purposes for which they are used

4


Section 1111(e) of the ESEA requires that ED conduct a peer

review of State plans, including State assessment systems

ED does not review or approve a State’s academic content

standards

ED started conducting peer review of assessment systems in

2000

ED released revised non-regulatory guidance for the peer

review of State assessment systems in 2004 to reflect

changes in the ESEA in 2002

Between 2005 and 2012, all States went through peer

review for their assessment systems

5


In 2012, in light of significant changes in the field, ED paused

peer review

The AERA/APA/NCME Standards for Educational and

Psychological Testing were being udpated (the revised version

was released in 2014)

Most States were in the midst of implementing newly adopted

content standards and developing new assessment systems

ED wanted to revise our guidance to reflect lessons learned

over the past dozen years, changes in the field and current

best practices, revised professional standards, and feedback

received from States, experts, and other stakeholders

6


The purpose of ED’s assessment peer review guidance is

three-fold:

To support States by identifying expectations that they

can use as they develop, administer, and improve their

assessment systems in order that they provide valid and

reliable information on how well students are achieving

a State’s challenging academic standards

To help States prepare for the peer review of their

assessment systems

To guide the peer reviewers in the review of the State

assessment systems

7

UNDERSTANDING THE PROCESS

The assessment peer review process is:

Evidence-based – the peer review is, by nature,

backward-looking in order to confirm the technical

quality of the assessments based on full administration of

the assessments

Focused on two primary aspects:

Documentation of the process used to develop and

administer the assessments

Data to confirm the quality of the system (i.e., did the

system operate as intended?)

8

WHAT IS REVIEWED?


Assessment peer review is conducted by external assessment

experts, including nationally recognized assessment experts,

State and local assessment directors, and educators

ED will soon put out a call for individuals to serve as peer

reviewers

From the full list of peers, ED will develop a small panel of 3-

5 peers for each State to review that State’s evidence

ED will conduct an introductory training on the assessment

peer review process and criteria prior to beginning the peer

review and again before each subsequent review

9

WHO ARE THE PEER REVIEWERS?


Each peer reviewer uses professional judgement to develop

individual recommendations to ED regarding whether the

documents are sufficient to address the critical element

ED facilitates a meeting of the peer reviewers to review and

discuss the State’s documents

Shortly following the review, ED will provide the State with

the notes from the peer reviewers

Information and technical assistance to the State

Suggest best practices for the State to consider

Following ED’s review and decision, ED will provide formal

feedback to the State

10

HOW DOES THE REVIEW OPERATE?


Almost every State has developed new academic content

standards and assessments since ED paused peer review in

2012

As a result, we believe all States will need to submit

documents for peer review for reading/language arts,

mathematics, and science

Includes all tests needed to meet section 1111(b)(3) of the

ESEA

General assessments

Alternate assessments based on alternate academic

achievement standards (for students with the most significant

cognitive disabilities), and

Other assessments (e.g., native language translation, etc.)

11

WHO SHOULD SUBMIT FOR REVIEW?


Some States will be administering new assessments in

reading/language arts and/or mathematics in 2015-2016

It does not make sense for the State to submit documents about

the assessments it administered in 2014-2015

We believe all States have revised their science assessments

since the last time they were peer reviewed

We know that many States have recently adopted new science

content standards and are developing new science assessments

If that is the case, let your program officer know and indicate

the timeline for the new science assessments

12

WHO SHOULD SUBMIT FOR REVIEW?

TIMELINE FOR REVIEW

November 18, 2015 Deadline for States to indicate to their

program officer when they will submit documents for peer

review

Window 1: Tentatively scheduled for Jan. 25-29, 2016

State will need to submit documents by January 11, 2016

Window 2: Tentatively scheduled for Mar. 28 – Apr. 1, 2016

State will need to submit documents by March 14, 2016

Window 3: Tentatively scheduled for May 16-20, 2016

State will need to submit documents by May 2, 2016

13

Understanding the Critical

Elements

WHAT’S NEW?

General updates – The guidance is revised to follow the new

AERA/APA/NCME standards and to reflect changes in

assessments over the past 10 years, such as the increased

prevalence of technology

Test security – The guidance includes a larger section focusing

on test security before, during, and after the assessments are

administered and the State’s process to protect the integrity

of assessment-related data

Alternate assessments based on alternate academic

achievement standards – The guidance includes much greater

detail on evidence related to alternate assessments

15

WHAT’S NEW?

Coordinated work across States – In cases where States are

administering the same assessments, the revised peer review

process reduces burden for individual States and ensures

consistency by reviewing those submissions together

Alignment – This is not a new aspect of the review but worth

emphasizing—the assessment system is required to cover the

full range of the State’s academic content standards

For example, if a State has writing as part of its reading/language

arts standards, the assessments should include writing

Speaking and listening – ED recognizes that large-scale, Statewide

assessments may not be ready to include speaking and listening at

this time and invites States that have speaking and listening as part of

their standards to submit a waiver for these standards, provided the

State is working to include them in the future. More details on this will

be forthcoming

16

MAP OF CRITICAL ELEMENTS

[Insert map]

17

1. Statewide system of

standards & assessments

1.1 State adoption of academic content

standards for all students

1.2 Coherent & rigorous academic content

standards

1.3 Required assessments

1.4 Including all students

1.5 Participation

data

2. Assessment system operations

2.1 Test design & development

2.2 Item development

2.3 Test administration

2.4 Monitoring test admin.

2.5 Test

security

2.6 Systems for protecting data

integrity & privacy

3. Technical quality—validity

3.1 Overall Validity,

including validity based on test

content

3.2 Validity based on cognitive processes

3.3 Validity based on

internal structure


relation to other variables

4. Technical quality—other

4.1 Reliability

4.2 Fairness & accessibility

4.3 Full performance

continuum

4.4 Scoring

4.5 Multiple assessment forms

4.6 Multiple versions of an

assessment

4.7 Technical analyses &

ongoing maintenance

5. Inclusion of all students

5.1 Procedures for including

SWDs

5.2 Procedures for including ELs

5.3 Accommoda-

tions

5.4 Monitoring test admin. for special populations

6. Academic achievement standards &

reporting

6.1 State adoption of academic

achievement standards for all students

6.2 Achievement standards

setting

6.3 Challenging & aligned academic

achievement standards

6.4 Reporting


[Insert map]

18






standards



1.5 Participation

data






2.5 Test

security


integrity & privacy




content



internal structure




4.1 Reliability



continuum

4.4 Scoring



assessment


ongoing maintenance



SWDs


5.3 Accommoda-

tions



reporting




setting



6.4 Reporting

Sections

Critical

elements


[Insert map]

19






standards



1.5 Participation

data






2.5 Test

security


integrity & privacy




content



internal structure




4.1 Reliability



continuum

4.4 Scoring



assessment


ongoing maintenance



SWDs


5.3 Accommoda-

tions



reporting




setting



6.4 Reporting

Shaded boxes

– likely

addressed by

coordinated

evidence


[Insert map]

20






standards



1.5 Participation

data






2.5 Test

security


integrity & privacy




content



internal structure




4.1 Reliability



continuum

4.4 Scoring



assessment


ongoing maintenance



SWDs


5.3 Accommoda-

tions



reporting




setting



6.4 Reporting

Clear boxes –

likely

addressed by

State-specific

evidence for all

States


[Insert map]

21






standards



1.5 Participation

data






2.5 Test

security


integrity & privacy




content



internal structure




4.1 Reliability



continuum

4.4 Scoring



assessment


ongoing maintenance



SWDs


5.3 Accommoda-

tions



reporting




setting



6.4 Reporting

Dashed lines –

likely

addressed by a

mix of State &

coordinated

evidence


[Insert map]

22






standards



1.5 Participation

data






2.5 Test

security


integrity & privacy




content



internal structure




4.1 Reliability



continuum

4.4 Scoring



assessment


ongoing maintenance



SWDs


5.3 Accommoda-

tions



reporting




setting



6.4 Reporting

Ovals – will be

checked for

completeness by ED

HOW TO READ THE CRITICAL ELEMENTS

23

Element: State

must meet this

level of quality


24

Examples of

evidence –

not

exhaustive


25

Examples of

evidence

specific to AA-

AAAS


26

Examples of

evidence

specific to AA-

AAAS


27

Examples of

evidence

specific to

technology-

based

assessments

FUTURE SUPPORT FOR STATES

Call for peer reviewers

State submission cover sheet and index template

Additional ED webinars

Understanding the elements

Multiple States submitting documents for a common assessment

Lessons learned from window 1

Resources: www.ed.gov/admins/lead/account/saa.html

Contact: Patrick Rooney, [email protected]

28

http://www2.ed.gov/admins/lead/account/saa.html

mailto:[email protected]


CCSSO technical assistance

SCASS meetings, October 27-28 in Atlanta

One-day meeting, November 10 in DC

One person from each state

Supported by The Center for Assessment

For questions, contact: Scott Norton,

[email protected]

29

mailto:[email protected]


• What questions do you have?

• What support do you need to understand the peer review

process?

• What support do you need to understand the critical elements

and examples of evidence?

• What support do you need preparing the submission?

Please type into the chatbox and send a private message to the host.

HELP US HELP YOU -- IDENTIFY THE SUPPORTS THAT ARE NEEDED

30

QUESTIONS?

31

SAA Peer Review of State Assessment Systems PPT 10-8 ... - ed · PDF fileED does not review or...

Documents

Transcript of SAA Peer Review of State Assessment Systems PPT 10-8 ... - ed · PDF fileED does not review or...