4. SA3 Module 2 Designing an assessment study 2015 · into a program or who gets a scholarship, ......

2/19/2015

1

DIVISION OF STUDENT AFFAIRS

Module 2: Designing an Assessment Study

What Will We Do?February, 2015

Module 2: Designing an Assessment Study

What Will We Do?February, 2015

Jeremy Penn, Ph.D.Director

Jeremy Penn, Ph.D.Director

DIVISION OF STUDENT AFFAIRSASSESSMENT

Plan

Implement

Analyze and use

Follow-through

SA3 Modules

1. Asking a Question (Purpose)

2. Designing an Assessment

Study

3. Assessment Instruments (validity and reliability)

4. Analyzing Results

5. Using and Communicating

Results

6. Celebrating results (Follow-

through and reflection)


Resources

• Let’s spend the next 45 minutes quietly reading Schuh, chapters 3 and 4

• Just kidding.

• But I do recommend reading those chapters after we conclude today

• Chapter 2 would be useful if you are planning on using existing institutional data

2/19/2015

2


What will we do?

• Time to start designing your study!• Each group will need to address:

– Assessment method and instrument(s) you will need

• We will work on locating / creating / modifying instruments in later module and we will focus on method in this session

– Practical considerations (who will participate? How will they be encouraged to participate?)

– Timeline for the study– Ethical and IRB considerations


Study Type

• Gets at the reason / purpose for the project

• E.g., Needs Assessment

Method• Strategy or way

assessment will be conducted

• E.g., Focus group

Instrument

• Tool used in the method

• E.g., Focus group prompts


General Principles of Assessment Methods

• In order to design a study, it helps to understand some key principles of assessment methods

• We will look at some commonly-held beliefs about assessment and see if that can help us understand some of these principles

2/19/2015

3


Inaccurate Belief #1

There is “An Assessment Method” that is to be used in all situations and circumstances.

The job of the assessment team is to apply the Assessment Method to all programs / services etc. regardless of context.


Principle #1• No “gold standard”

assessment method• What works in one

place, for one group of students, may not work well somewhere else

• What is best for one assessment project may not work at all in another

Auric Goldfinger



Assessment projects are massive in scale and scope and should attempt to cover everything we do all the time.

2/19/2015

4


Principle #2

Assessment should be reasonable, manageable, and meaningful

• Is this something that could realistically be implemented?

• Do we have the expertise, time, and resources to implement this?

• Will the results be meaningful to the staff and to external constituents?



Assessment requires a measurement approach, including a belief about the nature of knowledge, insists on the use of numbers, and can only be understood through statistics.


Principle #3

• Assessment is about answering a question: what would it take to convince you that a question has been answered?

• Can use quantitative, qualitative, or mixed methods

• Whatever the choice, the methods should be sufficiently rigorous

2/19/2015

5


Qualitative or Quantitative Study?

Quantitative:

• More closed-ended

• Cause / effect (why did it happen?)

• Use of theory

• Assessing differences and magnitude

Qualitative:

• More open-ended

• Descriptive (what happened?)

• Interpretive (what was the meaning to people?)

• Process-orientedSource: Creswell, 2002


Qualitative or Quantitative Study?

• As a pragmatist, I suggest using whatever “works”

• Some questions are more qualitative and others are more quantitative (and some are mixed methods)



A single assessment method is sufficient for answering our question.

2/19/2015

6


Principle #4

• Use multiple assessment methods, try not to over-emphasize indirect methods

• Unlikely any single method will give a complete picture or allow you to completely answer your question(s)– Focus group with follow-up survey– Satisfaction survey and rubric evaluation of minute

papers

• This is particularly important if you are making important decisions about students (e.g. who gets into a program or who gets a scholarship, etc.)


Direct methods• Standardized or locally

developed test or instrument (e.g. depression screening)

• Existing institutional data• Survey• Papers, projects, written

assignments (evaluated with rubrics)

• Presentations• Portfolios• Observation /

performance• Evaluation by supervisor• External reviewers

Indirect methods (perceptions)

• Satisfaction survey

• Participation counts

• Focus groups

• Interviews

• Self-assessment

• Case studies

• Narratives


Pros and Cons of Assessment Methods

(see handout)

2/19/2015

7


Practical Considerations for Your Assessment Method

• Who will participate in your assessment project? • How will participants be encouraged to

participate? • What’s in it for them? Will participants receive

feedback or information on their level of performance?

• Who will be responsible for encouraging participation?

• Who will be responsible for gathering the data? For writing the report and sharing results?


Who should participate?

• Which participants are best able to answer your question? What is your target group (sampling frame)?

• An appropriate sampling strategy should be considered– Sampling allows you to use more resources to

obtain a higher response rate from a smaller number of students


How will participants be encouraged to participate?

• Follow Division Guidelines (http://www.ndsu.edu/fileadmin/vpsa.ndsu.edu/Document_Forms/Best_Practices_and_Guidelines_for_Promotions_Incentives_Gifts.pdf)

• Relevance of study for the individual works better than almost any prize or food

2/19/2015

8


How will participants be encouraged to participate?

• Use existing group of students (e.g., advisory board for a focus group)

• “Captive audience” – get them when you can!• A small up-front token works better than a

few large prizes in a drawing• “Large” may not have to be as “large” as you

think• Phone calls work• Make assessment a routine part of the

program or service – require it!


Benefits of the study to participants

• Participants are giving their time to you – can you give them something in return (beyond the incentive to participate)?

• If you are gathering information on student learning / development, can you share that information with the participants?

• Can you share information about how you will (or have) used the results?

• Could you share examples from how you’ve used results in previous years?


Study Timeline

• Work backwards, starting with when you want to have results analyzed and shared

• General elements:– Identify purpose of study and study questions– Design study– Create or locate assessment tools– Administer assessment tools according to study

design– Analyze results– Write report– Share results and take action

2/19/2015

9


SA3 Timeline

Spring and early summer 2015: Modules 1-3 (planning)

Summer / Fall, 2015: Implement and analyze results

Spring, 2016: Use results

April, 2016: Celebration event (2015 cohort ends)


Ethical and IRB considerations


Penn, Campbell, Valente, Wehlburg, 2012

Making the news for the wrong reason

2/19/2015

10


Potential Assessment Ethical Issues (examples)

• Collecting samples of student work for assessment without consent from students, staff members, or faculty members

• Breach of confidentiality of sensitive information (e.g., sexually transmitted disease status)

• Releasing only those assessment results that make the program / department / institution look good– Intentionally misreporting results to make the

institution look good• Frugging: a survey under the guise of assessment

but with an actual agenda of soliciting donations


Share at your table

Have you ever been a participant in a study where you felt like your rights were violated (that you know of)?

What ethical issues concern you most regarding your assessment work?


Many Professional Standards Available to Guide Practice

• Follow your professional standards

• SEPT includes guidance on:– Reliability– Validity– Test development– Scales, scores,

comparability– Fairness in testing and

test useStandards for Educational and Psychological Testing, APA, AERA, NCME (1999)

2/19/2015

11


Division Expectations

1. Right vs. Wrong: (“Is it the right thing to do?”)

2. Legal vs. Illegal (and follow NDSU and NDUS policy)

3. Do the right thing for the right reason (must hold up to public scrutiny – would you want your assessment project to be on the front page of the Forum?)


General Ethical Principles

• Highest standards of honesty and integrity• Respect the rights of students, faculty and

staff members, and community members• Avoid conflicts of interest• Follow the principles of the local IRB (even

if you do not submit your project for IRB approval)– Strongly recommend IRB training for

everyone involved in your assessment project


General Ethical Principles

• Recognize strengths and limitations of assessment instruments and methods

• Assessment as a responsibility to students, the Division, and the public

• Score use:– Fairness (for both students and those to whom

inferences of scores are applied)• (e.g., evaluating staff performance only on a

satisfaction survey filled out after an event)– Evidence of validity needed when the use has big

consequences (will discuss more in future module)

2/19/2015

12


IRB• IRB approval not required for assessment projects• IRB approval should be pursued when:

– You want to publish or present results at national conference (many publishers and conferences require IRB approval for projects) or if it is a research project

– Topic of assessment is of sensitive nature (e.g., illegal activity, sexual behaviors, drug or alcohol use, etc.)

– Involves protected populations (e.g., children, prisoners, pregnant women) or non-minimal risk

– Need access to sensitive data gathered elsewhere• I can help you with this! • ndsu.edu/irb


Worksheets

Module 2 worksheet will get your team started on this portion of the task


Presentations

• Each team will prepare a short presentation for the group

• Please use PowerPoint or some visual display

• Will have time for a few questions from the other groups

2/19/2015

13


Viewers of the Presentation

• Use the rubrics and provide feedback

• Will collect rubrics and give to the team

• Be prepared to ask questions or provide feedback

• Be constructive – coaching model

4. SA3 Module 2 Designing an assessment study 2015 · into a program or who gets a scholarship, ......

Documents

Transcript of 4. SA3 Module 2 Designing an assessment study 2015 · into a program or who gets a scholarship, ......