STEM Assessment: Bridging Professional Practices and ... · tests are scored accurately and...

STEM Assessment:

Bridging Professional Practices and University Education

1 | Copyright © 2015 by Educational Testing Service. All rights reserved. | November 9, 2015

André A. Rupp, Research DirectorIDEAS and STANDARDS Groups, Capability Division

McBride E-290 (Office) & E-18 (Mail)Educational Testing Service (ETS)

660 Rosedale RoadPrinceton, NJ 08541

Phone: (609) 252-8545Fax: (609) 734-1309

E-mail: [email protected]


The Story Today

• Personal Background• STEM Assessment I: Professional Perspective• STEM Assessment II: University Perspective • Pathways for Integrative Experiences


What we can Learn from Broadway


Case Study: Aladdin (Fabulous!)

Personal Background


Cross-cultural Experiences


7

There are a total of 1246 staff in R&D in specialized areas of expertise:

• Assessment developers (571) Responsible for designing and building tests and producing test questions

• Psychometricians (86) – Provide statistical expertise to ensure that tests are scored accurately and reported on time

• Research scientists (125) – Conduct original research advancing the fields of educational measurement and public policy and folding these innovations into ETS assessments to assure quality and innovation

• Test creation, data, research, and knowledge management support staff (423) – Provide technical, data, editorial, and communications support

• Project managers (41) – Create and manage project plans to ensure work is executed on schedule and on budget

R&D Expertise at ETS


Noteworthy Identities

• Research director at ETS for measurement and validation work around automated systems for scoring and feedback

• Involved in innovative assessment work at ETS and other institutions through project work and advisory functions

• Known for connecting statistical theory and practical uses of diagnostic measurement techniques

• Interested in connections between broader frameworks, ideas, and research findings and best practices

• Passionate about development and mentoring of high-performing professionals


Handbook of Cognition and Assessment:

Frameworks, Methods, and Applications

André A. Rupp Jacqueline Leighton

Wiley-Blackwell

2012 2016

STEM Assessment:Professional Perspective


Core Focus: Powerful Innovations


Assessment = Product + Process


Selected R&D Goals at ETS

−Designing, deploying, and reporting results for large-scale educational surveys and summative assessments

−Developing, testing, and refining digitally-delivered learning and assessment tools for predominantly formative low-stakes needs, in particular innovative stealth assessments (e.g., simulations, games-based assessments, assessments with multimodal outputs)

−Developing automated scoring capabilities for responses to such performance tasks that involve text, speech, mathematics, and multimodal outputs

−Refining models of cognition, meta-cognition, socio-cognition for innovative assessments, including assessments of professional skill sets for the global work force


Project-based Interdisciplinarity

The successful design, implementation, analysis, and use of innovative assessments requires the constant iterative collaboration between – at least – the following teams of experts within any given project:

- Teachers / instructors - Learners / Test-takers

- Learning scientists - Design-based researchers

- Curriculum developers - Program evaluators

- Measurement specialists - Policy-makers

- Applied cognitive psychologists - Computer scientists


Methodological Interdisciplinarity

This work requires the alignment and integration of various methodological disciplines and approaches!


21st Century SkillsCognitive skills:

problem solving, critical thinking, systems thinking, design thinking, reasoning (inductive and deductive), analysis-synthesis-argumentation skills, innovation

Intrapersonal skills:

adaptability/flexibility, self-regulation, metacognition, self-reflection, self-motivation, persistence

Interpersonal skills:

tolerance, empathy, communication, collaboration/teamwork, negotiation skills, conflict resolution, leadership.


Lots of Collaboration with Interesting Characters!

STEM Assessment:University Training

Perspective


Core Focus: Technical Skills?


Methodological Training

- Latent-variable methods / modern psychometrics- Statistical methods for the social sciences- Research design structures for applied studies- Research design structures for simulation work- Statistical properties of assessment scores- Types of assessment formats and uses- Assessment design and delivery- Validation frameworks and approaches- Quality-control and fairness standards


Core Activities

- Course and project work• time management• organization and planning

- Written dissemination• journal articles• technical reports• book chapters• encyclopedia entries• policy documents

- Presentations, discussions, negotiations


21st Century SkillsCognitive skills:

problem solving, critical thinking, systems thinking, design thinking, reasoning (inductive and deductive), analysis-synthesis-argumentation skills, innovation

Intrapersonal skills:

adaptability/flexibility, self-regulation, metacognition, self-reflection, self-motivation, persistence

Interpersonal skills:

tolerance, empathy, communication, collaboration/teamwork, negotiation skills, conflict resolution, leadership.


Does it Matter?

In what combination and to what degree these skills are required to be successful in an assessment-oriented career

depends on the career track and career trajectory!

Academic career in similar degree programWork at large-scale testing company (“core business”)

Work at large-scale testing company (“research / innovation”) Work at small data-analytic start-up

Work at a non-assessment company in other domainsWork for district or state divisions or national consortia

Pathways for Integrative Experiences


Develop all Three Areas!


Project Experiences for Students

- exposure through courses, small projects, and independent self-study

- hands-on experience through research projects funded by NSF, IES, MacArthur Foundation, Bill & Melinda Gates Foundation, and so on

- professional experience through internships / practicums and fellowships at companies

This is partly under control of learners but largely dependent upon direct and indirect opportunity structures at university!


Possible Support Structures (I)

- support effective cross-team collaboration within campus schools /colleges and with regional partners

- make grant writing / funding acquisition as low-burden onprofessors as possible

- encourage and support faculty to revamp curricula and besttraining practices to allow for modularization and rich profession-like experiences

- minimize professional development distractions from non-criticalcommittees, administrative work, and the like for faculty


Possible Support Structures (II)

- develop more flexible intellectual property rules and guidelines that facilitate partnerships

- take some financial risks to develop partnerships with for-profit and not-for-profit companies

- create value systems for academic advancements that support contributions to team projects, including publication pipelines

- create cultures of engagements where 21st century skills are modeled and developed among faculty, students, business divisions, and administrators


Lots of Collaboration with Interesting Characters!

STEM Assessment:

Bridging Professional Practices and University Education

. 30 | Copyright © 2015 by Educational Testing Service. All rights reserved. | November 9, 2015

André A. Rupp, Research DirectorIDEAS and STANDARDS Groups, Capability Division

McBride E-290 (Office) & E-18 (Mail)Educational Testing Service (ETS)

660 Rosedale RoadPrinceton, NJ 08541

Phone: (609) 252-8545Fax: (609) 734-1309

E-mail: [email protected]

STEM Assessment: Bridging Professional Practices and ... · tests are scored accurately and...

Documents

Transcript of STEM Assessment: Bridging Professional Practices and ... · tests are scored accurately and...