Helping Program Assessment Using Automated Text Processing

Anne Gilman

SoTL Brown Bag

25 Sept 2013

9/25/13

Finding questions we hadn't thought to ask...

..in what students say.

Assesment Tools

The Law of the Instrument:

“Give a small boy a hammer, and he will find that everything he encounters needs pounding.”

-Abraham Kaplan

Assesment Tools

Are there tools that can help us ..to articulate goals

we take for granted? ..to point in new

directions?

Cool Hand LIWC

Data

LiM participant essays 57 from 2007-2012

Holistic ratings 23 Director’s Cut Totals:

9 Excellent 11 Good 3 Just barely makin’ it

9/25/13

LiM AssignmentIn a paper of at least two pages, discuss what the Language in

Motion experience has meant for YOU. Be as specific as possible and provide examples where relevant.

 Consider the following:A. Why did you choose to participate in Language in Motion? What

personal goals, both short and long term, did you have for this experience? Were they met? If so, how well? If not, why not?

 B. In what areas have you improved your skills and/or knowledge?

Consider what you learned about the following:

Your second language, if any, The topic(s) of your presentations, Your home culture, Your second culture, American education, Presentation skills: Organization, ability to give clear directions,

adjustment of content to audience (age and knowledge level), etc. Yourself: Communication or cross-cultural communication abilities, time

management, self-confidence, flexibility, leadership, etc.

C. Was there something you wanted to learn through this

experience that you didn’t? If so, what was it and why did you not learn it? Is there something we should do differently?

9/25/13

What did LIWC say?

What did liwc say?

Next Steps:

Analyze by paragraph Consider value of enjoyment Look under the hood

9/2

5/1

3

Can we look under the hood?

Topic sorting c. 1974

Self-organizing maps c. [guess!]

Are these more familiar?

Hoping the XPS viewer works in our room!

9/2

5/1

3

9/25/13

Sample LiM Wordles

9/25/13

What's a trigram?

Demo

Results: Cluster-to-Category Match9

/25

/13

Rating Cl-1 Cl-2 Cl-3

Adequate 0 2 1

Good 6 2 3

Excellent 6 1 1

Better: Talking less about

your own country zzt, zto, top

TBD: Less: tha More: ent, eth

9/25/13

Next Steps

Get more ratings Add more trigrams Compare to LiM, ILAC goals Consider POEs

9/25/13

Many Many Thanks

LiM Director: Dr. Deb Roney Gilman Lab Assistants

LIWC project: Helen Hu, Aric Koestler, Olivia Moody, Sungouk Park, Seth Weil

Trigram project: Tori Rehr, Tori Buser Dr. Kim Roth The Juniata Department of Psychology The SoTL Leadership & the Lakso Fund

Gory Details > aggregate(limtrunc[-1], by=list(cluster=fittrunc.km$cluster), mean) cluster age all and ati den 1 1 0.002255296 0.002815929 0.007260874 0.006236264 0.002169403 2 2 0.004112582 0.002038445 0.006445021 0.007287239 0.003207497 3 3 0.001498536 0.002247922 0.007557460 0.006659985 0.004155337 ent ere ers ese est 1 0.007423606 0.003938757 0.002885631 0.003315418 0.003273650 2 0.010172528 0.003758447 0.001438849 0.005238279 0.003749762 3 0.010626493 0.004926549 0.002321635 0.004856809 0.003323920 eth eve for. hat her 1 0.003958753 0.002694240 0.003256768 0.005716957 0.003714643 2 0.002908776 0.002441420 0.002277172 0.004412824 0.002856234 3 0.003351811 0.002440801 0.003763736 0.002332821 0.003069785 ing ion zto zzt ver 1 0.008014828 0.005659334 0.001344568 0.001344568 0.003019872 2 0.008223715 0.009158820 0.001807482 0.001807482 0.001872784 3 0.005631146 0.006514531 0.004178863 0.004178863 0.003330204 ter tha the thi tio 1 0.002892176 0.005949131 0.01666473 0.003739603 0.005432872 2 0.002505902 0.004485792 0.01535570 0.003083640 0.008802554 3 0.002784424 0.003212007 0.01513814 0.004036677 0.006142843 top tth nts ons opi 1 0.002193457 0.003513021 0.002240106 0.002399789 0.001670029 2 0.002822616 0.002456230 0.002990221 0.002544836 0.002204109 3 0.005067310 0.002195498 0.003912161 0.002463628 0.004443421 ore pia pre rea res 1 0.002220215 0.001284905 0.003154103 0.003108067 0.004400721 2 0.001488311 0.001586731 0.005186367 0.002198381 0.005775334 3 0.002816267 0.004117230 0.004174962 0.003289315 0.005404926 sth 1 0.002374196 2 0.002569780 3 0.002009747