Role of the Research Partner in the EVEA Project: The State of Washington Alison L. Bailey, UCLA...
-
Upload
molly-shaw -
Category
Documents
-
view
217 -
download
0
Transcript of Role of the Research Partner in the EVEA Project: The State of Washington Alison L. Bailey, UCLA...
Role of the Research Partner in the EVEA Project:
The State of Washington
Alison L. Bailey, UCLA
CCSSO, NCSA,Detroit, June 22, 2010
Outline
• Highlight the role of the research partnerships
• Chronology of activities as a Research Partner (RP) with the State of Washington
• Example deliverables and products
• Reflections/lessons learned
EVEA Project Goals
Addressing the validity of English LanguageProficiency Assessments (ELPA), researchpartners and states work together to:1. Build:
- individual State Interpretive Arguments (SIA), - a Common Interpretive Argument (CIA),
2. Design a set of studies and instruments to support and test these arguments (pilot level)
3. Make instruments publicly available
Role of Research Partnerships
• Collate information about the partner State’s ELPA system
• Identify and prioritize validity issues; input from Expert Panel (language and measurement experts, on-going contact)
• Create and pilot validation plans
• Foster collaboration between all RPs on protocol development, common issues, etc.
WA State Project Goals
• Create State Interpretative Argument (SIA) for the validity of the ELPA
• Identify:- Claims/assumptions in the ELPA- Dimension of validity evidence- Source of evidence
• Prioritize, design and pilot validation plans
1. Collating Information on WA State ELD Tests and Standards (Assessment
& Student Information Division; Migrant & Bilingual Education Program)
Description of the WA State ELPA System
System Key Components
1. Identifying population of potential ELL students with the Home Language Survey: “Triggered” by affirmative response to
Question#2: Is your child’s first language a language other than English?
2. Screening students with WLPT-II Placement Test
3. Requiring WLPT-II Annual Assessment of all Transitional Bilingual Instructional Program (TBIP)-eligible students (EL services)
The Washington Language Proficiency Test-II (WLPT-II)
• Augmented Stanford English Language Proficiency Test (Pearson/Harcourt)– additional items aligned with WA ELD Standards– 4 levels of proficiency: Beginning, Intermediate, Advanced
& Transitional
• First administered 2005-2006 school year• WLPT–II Placement Test
– Transitional (Level 4) not eligible for TBIP services
• WLPT–II Annual Assessment – Level 1 indicates minimal or no English language proficiency– Level 4 indicates a level of English language proficiency
sufficient to be instructed through an English-only instructional program
Intended Purposes and Uses of The WLPT-II Annual Assessment
Federal Accountability:• Annual Measurable Achievement Objective 1
(progress)
• Annual Measurable Achievement Objective 2 (proficiency)– Criterion for AMAO 2 (achieving proficiency)
» Transitional (Level 4) on WLPT-II Annual Assessment
Additional Uses
• Reclassification to Fluent English Proficient (R-FEP):– Transitional (Level 4) to exit TBIP services
• Program evaluation instrument
2. Identify and Prioritize ELPA Validity Issues
(OPSI Staff, Expert Panel, RP & EVEA team)
WLPT-II Validity Plans and Creation of the WA SIA
P rograms successfully
moving more E L L s tudents into the
P roficient category and exit them out
of the E L P programs
T he E L D s tandards have been developed
to support the acquis ition of E nglis h language proficiency necessary to achieve academic content and
performance expectations .
T he E L P A has been des igned to yield s cores that reflect
s tudents ’ knowledge and skills in relation to
academic E nglish language expectations
defined in the E L D s tandards .
AMAO 1:E L P A scores /
performance levels are used
appropriately to inform decis ions
about progress in attaining E nglish
language proficiency
AMAO 2:E L P A scores /
performance levels accurately reflect s tudents ’ E nglish
language proficiency
E L L s tudents becoming
proficient in E nglish, acquiring
the academic language skills necessary to
participate fully in ins tructional
discourse conducted in
E nglish.
Student Intake: Identifying the ELL population: HLS > Placement Test
(School/teacher role in this process)
External documentation/judgment: For adequacy of SLA theory, articulation of progressions of ELD/P and ELD/P construct definition adopted by ELD standards and assessments
ELPA (annual) administered as intended to yield scores for…
(School staff, teacher, TA roles)
Use and interpretation:Consequences for…
(Teacher role)
ELPA WA State Interpretive Argument
Foundations Document (Generic)• Identifies the external linguistic and
developmental factors by which to judge the legitimacy of the assumptions expressed in the existing ELPA system, namely:– second language acquisition theories – articulation of learning progressions for English language
proficiency (ELP), and– the ELP construct adopted (or implied) by the ELD
standards and assessments
• Raises issues that need to be considered in light of these assumptions, and
• Makes suggestions for moving forward.
Foundations Document (WA Specific)
• SIA
• Description of the WLPT-II
• Description of the WA State ELD Standards
• Catalogue of technical reports, existing validity studies:– Example: Evaluations of the Content of WLPT-II:
» Studies of alignment with WA ELD Standards (2005-6) » Item writers trained to write augmented items aligned
with the test blueprint» Existing SELP items modified if necessary» Items were sampled in ELL classrooms (directions clear/items “reliable” indictors of students achievement)» Results from IRT and DIF analyses for different groupsof test-takers (limited to potential gender biases)
P rograms successfully
moving more E L L s tudents into the
P roficient category and exit them out
of the E L P programs
T he E L D s tandards have been developed
to support the acquis ition of E nglis h language proficiency necessary to achieve academic content and
performance expectations .
T he E L P A has been des igned to yield s cores that reflect
s tudents ’ knowledge and skills in relation to
academic E nglish language expectations
defined in the E L D s tandards .
AMAO 1:E L P A scores /
performance levels are used
appropriately to inform decis ions
about progress in attaining E nglish
language proficiency
AMAO 2:E L P A scores /
performance levels accurately reflect s tudents ’ E nglish
language proficiency
E L L s tudents becoming
proficient in E nglish, acquiring
the academic language skills necessary to
participate fully in ins tructional
discourse conducted in
E nglish.
Student Intake: Identifying the ELL population: HLS > Placement Test
(School/teacher role in this process)
External documentation/judgment: For adequacy of SLA theory, articulation of progressions of ELD/P and ELD/P construct definition adopted by ELD standards and assessments
ELPA (annual) administered as intended to yield scores for…
(School staff, teacher, TA roles)
Use and interpretation:Consequences for…
(Teacher role)
ELPA WA State Interpretive Argument
Claim/AssumptionDimension of Validity
EvidenceSources of Evidence
The ELPA is administered as intended
Evidence based on internal structure
Study monitoring administration (e.g., two teachers, same student)Cluster analysis by administrator
ELPA scores accurately reflect student’s ELP
Evidence based on response processes
Triangulate with other measures of student ELP
Scales have been created to accurately reflect growth
Evidence based on internal structure
Statistical criteria in technical manual
Growth scores are used appropriately to make judgments about program effectiveness
Evidence based on test consequences
Focus group on interpreting dataTriangulate with other measures of school effectiveness
The ELPA is used to calculate growth in English Language acquisition over the year to determine school/program effectiveness
EXAMPLE Organization Chart (EVEA Project)Purpose 1:
Identifying & Prioritizing Issues
• Vertical scaling/year-to-year fluctuations: – Variation in exit (Transitional Level 4)
percentages year-to-year by grade level and test form
– Concerns: used as the TBIP services exit criterion (R-FEP)
Identifying & Prioritizing Issues
• Different ELL program eligibility and exit criteria:– Home Language Survey (HLS) and WLPT-II
Placement Test used to qualify new students for TBIP services
– WLPT-II Annual Assessment used as program exit criterion (and as program evaluation instrument)
– Concerns: HLS used as the initial identifying instrument (false positives/negatives)
Ideas for Investigating Fluctuations• Study of Test-level factors:
– Comparable forms: Issues with equating forms? – Document and evaluate item selection process
(Content analysis?)
• Study of External factors: – ELL program changes – Reduction in funding (less admin. training)– Changes in enrollment demographics
» Language background/new immigrant groups – direct and mediated by HLS less “accurate” for some groups (over/ under-identified)
Ideas for Improving Initial Identification of ELL Population
• Creation of an “Enhanced” HLS:– Pilot additional questions based on language
use in specific activity settings (e.g., research-base showing predictive validity of parent questionnaires about oral and print home practices and later oral language and reading outcomes in English; Reese & Goldenberg, 2008).
• Possible oral interview protocol for non-literate parents– Responsive to recent Somali refugee
population
Additional Areas of Study
• The Role of the WLPT-II in Language and Content-Area Instruction (surfaced by EP): - Survey teachers to determine:
- Whether teachers use WLPT-II results to plan content-area instruction for students (adaptation of existing CRESST teacher survey of science OTL and academic language exposure), and
- What level of instruction supports the English language needed for success on the WLPT-II
Reflections/Lessons Learned1. Liaising across OSPI Divisions and Programs:
– Dialogue across “silos”, often for first time – Easier for an outsider?
2. Meeting with Expert Panel & 5 EVEA States (tailoring SIA surfaced additional issues):– What are the boundaries of the Assessment Division’s
responsibility and sphere of influence?– Do teachers use ELPA scores? If so, how? – Need for the Foundations Document & more time!
Reflections/Lessons Learned3. RP Monthly Discussions:
– Address individual state and larger, across-state issues/RP interests (e.g., wider adoption of Foundation Document; White paper on HLS practices: differences and similarities)
– Accountability (impetus for steady progress)
4. Time(ing) & Attention: – Variation in size/capacity of state Assessment
Divisions (recommend staff designated to work with RP)
– EVEA versus… RTTT.
Thank You!
Contact Information:Alison Bailey: [email protected] Website: eveaproject.com
Additional Background Slides
Programs successfully move more ELL students into the Proficient category and exit
them out of the ELP programs
Programs successfully move more ELL students into the Proficient category and exit
them out of the ELP programs
The ELD standards have been developed to
support the acquisition of English language
proficiency necessary to achieve academic content
and performance expectations.
The ELD standards have been developed to
support the acquisition of English language
proficiency necessary to achieve academic content
and performance expectations.
The ELP assessment has been designed to yield scores that
reflect students’ knowledge and skills in relation to academic
English language expectations defined in the ELD standards.
The ELP assessment has been designed to yield scores that
reflect students’ knowledge and skills in relation to academic
English language expectations defined in the ELD standards.
ELP assessment scores/ performance
levels are used appropriately to inform decisions about progress in attaining English
language proficiency
ELP assessment scores/ performance
levels are used appropriately to inform decisions about progress in attaining English
language proficiency
ELP assessment scores/ performance
levels accurately reflect students’ English language
proficiency
ELP assessment scores/ performance
levels accurately reflect students’ English language
proficiency
The ELP assessment is administered as intended
The ELP assessment is administered as intended
ELL students become proficient in
English, acquiring the academic language skills necessary to
participate fully in instructional
discourse conducted in English.
ELL students become proficient in
English, acquiring the academic language skills necessary to
participate fully in instructional
discourse conducted in English.
EVEA Project: ELPA Common Interpretive Argument (2nd Draft)