cic.uts.edu.au

Writing Analytics workshop

Simon Buckingham Shum, Simon Knight, Andrew Gibson, Philippa Ryan

Australian Learning Analytics Summer Institute 2015

You  have  1000  students  who  would  like  detailed  feedback  on  their  reports.  

2

Now.  Please.  

(PS:  It’s  1.30am,  deadline  9am)  

DraFs,  not  submiGed  assignments  

How  could  this  be  possible?  

How  do  we  design  and  validate    wriKng  analyKcs?  

What  issues  arise?  3

TEXT  ANALYTICS  IS  WELL  OUT  OF  THE  LAB.  THE  WRITING  ANALYTICS  PRODUCT  SPACE  IS  GROWING…  

4

http://turnitin.com/en_us/features/turnitin-scoring-engine

http://www.pearsonassessments.com/products/100000681/writing-coach.html

Session plan

5

1.30  Welcome  and  overview  of  session  (SBS)    1.35  What  are  the  biggest  challenges  you  face  in  your  students’  wriKng?  (talk  in  pairs)  1.40  Quick  feedback  1.45  What’s  out  there?  Some  products  and  free  tools,  and  issues  they  raise  (SK)  2.00  Introducing  reflecKve  wriKng  +  GoingOK  demo  (AG)  

2.15  How  AWA  handles  reflecKve  wriKng  +  demo  (SBS)  2.30  How  AWA  handles  analyKcal  wriKng  +  demo  (SBS)  2.45  An  academic’s  experience  of  co-­‐designing  AWA  (PR)  2.55  Comments?  3.00  Hands-­‐on  with  GoingOK  or  AWA  or  other  free  web  tools  

3.25  Final  comments  3.30  End  

WHAT  ARE  THE  BIGGEST  CHALLENGES  YOU  FACE  IN  STUDENT    

ACADEMIC  WRITING?  

6

WHAT’S  OUT  THERE  ALREADY  PROVIDING  AUTOMATED  

ASSESSMENT  OF  ACADEMIC  WRITING?  

7

REFLECTIVE    ACADEMIC  WRITING  

ANALYTICS  

8

DO  STUDENTS  WRITE  REFLECTIVELY  IN  YOUR  SUBJECT/DISCIPLINE?  

uts.edu.au

 “Wr i&ng  br ings   together  past ,    present  &   future .  When  we  wr i te  we    ca l l  on   the  past  and  an&c ipate   the     future  even  as  we   inhabi t   the    present ;  at   the  same  &me,  we    engage   in  an  act   that   i s  both  deeply     inte l lectua l  and  overt ly  phys ica l”    Yagelski  p.192  2012.  

 

 

REFLECTIVE  WRITING:  

uts.edu.au

Provides  opportuniKes  for  a  deeper  approach  to  learning  Looks  back  on  the  moment  and  looks  forward  Encourages  shiFs  in  perspecKve  

Is  personal    SubjecKve    An  individual  interpretaKon  of  experiences      

Asks  students  to:  Comment  on  what  they  DON’T  know  Express  uncertainty  Convey  changes  in  understanding  

A  VERY  DIFFERENT  GENRE  

uts.edu.au

REFLECTIVE  WRITING  POSES  SOME  PROBLEMS:  

Challenging  to  teach:  

Different  interpretaKons  of  

what  reflecKve  wriKng  is/

what  it  looks  like    

uts.edu.au

Challenging  to  assess:  

How  do  you  idenKfy  a  deep  or  a  

superficial  reflecKon?  What  are  you  

marking  the  students  on?  Can  some  

students  be  advantaged/disadvantaged?  

A  RUBRIC:  LANGUAGE  &  DISCOURSE  FEATURES  &  TYPE  OF  INFORMATION     Context  of  the  event  that  triggers  the  

reflecKon  (why,  when,  where,  who,  how  much,  what):  the  more  detail  the  beGer,  as  long  as  the  event  is  non-­‐trivial  Specific  informaKon  about  the  event;  specific  examples  of  acKons  ,  ideas,  learning  experiences  Change  between  assumpKons  of  learning  and/or  prior  knowledge  and  actual  performance    

uts.edu.au

Reference  to  the  past:  Kme  markers  and  use  of  past  tense  (when  I  started;  before  my  internship);  shiF  between  habitual  past  tense  (I  used  to)  and  the  present  or  the  recent  past  (since  then  I  have  …)  

FROM  INFORMAL  RUBRIC  TO  FORMAL  RHETORICAL  PATTERN  

14

15

FROM  INFORMAL  RUBRIC  TO  FORMAL  RHETORICAL  PATTERN  

AWA:  REFLECTIVE  ACADEMIC  WRITING  

16

COMPARISON  OF  HUMAN  AND  MACHINE  ANNOTATION  

17

human machine

This counts as a “True Positive”

COMPARISON  OF  HUMAN  AND  MACHINE  

18

human highlighting automated highlighting

19

AWA:  REFLECTIVE  ACADEMIC  WRITING  

ANALYTICAL  ACADEMIC  WRITING  

ANALYTICS  

20

TYPICAL  ANALYTICAL/CRITICAL  RHETORICAL  MOVES  BACKGROUND  KNOWLEDGE  

Recent  studies  indicate  …  

…  the  previously  proposed  …  

…  is  universally  accepted  ...    

 

 

 

 

NOVELTY  

...  new  insights  provide  direct  evidence  ...  

...  we  suggest  a  new  ...  approach  ...  

...  results  define  a  novel  role  ...  

 

 

 

 

 

 

OPEN  QUESTION/MISSING  INFORMATION  

…  liGle  is  known  …  

…  role  …  has  been  elusive  

Current  data  is  insufficient  …  

TREND  

...  emerging  as  a  promising  approach    

Our  understanding  ...  has  grown  exponenKally  ...  

...  growing  recogniKon  of  the  importance  ...  

CONTRASTING  IDEAS  

…  unorthodox  view  resolves  …  paradoxes  …  

In  contrast  with  previous  hypotheses  ...  

...  inconsistent  with  past  findings  ...  

SIGNIFICANCE  

studies  ...  have  provided  important  advances  

Knowledge  ...  is  crucial  for  ...  understanding  

valuable  informaKon  ...  from  studies  

SURPRISE  

We  have  recently  observed  ...  surprisingly  

We  have  idenKfied  ...  unusual  

The  recent  discovery  ...  suggests  intriguing  roles  

SIGNALLING  AUTHOR  INTENT  

The  goal  of  this  study  ...  

Here,  we  show  ...  

Altogether,  our  results  ...  indicate  

21  

AWA: ACADEMIC WRITING ANALYTICS TOOL

22

23

AWA: ACADEMIC WRITING ANALYTICS TOOL

AWA: ACADEMIC WRITING ANALYTICS TOOL

24

Highlighted sentences are colour-coded according to their broad type

Sentences with Function Keys have more precise functions (e.g. Novelty)

AWA:  ANALYTICAL  ACADEMIC  WRITING  

25

Roll over sentences with Fkeys for a popup reminding you of their meaning

AWA:  ANALYTICAL  ACADEMIC  WRITING  (UTS  CIVIL  LAW)  

26

ENGAGING  THE  ACADEMIC  IN  CO-­‐DESIGNING  AWA  

27

1.  Trigger  curiosity  through  CIC  briefings  

2.  Submit  collaboraKon  proposal  to  CIC  

3.  Agree  Kmelines  

4.  Think  aloud  user  interface  walkthrough  

5.  Preliminary  analyses  

6.  SystemaKc  analyses  

7.  Refine  parser  

8.  Design  student  pilot  

9.  Gather  student  feedback  

28

STRUCTURED  ANNOTATION  FROM  THE  CIVIL  LAW  ACADEMIC  TO  TUNE  THE  PARSER  FOR  HER  DISCIPLINE    (classified  comments  in  word)  

29

STARTING  FROM  AWA  OUTPUT.  ..  

30

STRUCTURED  ANNOTATION  FROM  THE  CIVIL  LAW  ACADEMIC  TO  TUNE  THE  PARSER  FOR  HER  DISCIPLINE  

ENGAGING  THE  ACADEMIC  IN  CO-­‐DESIGNING  AWA  

31

1.  Trigger  curiosity  through  CIC  briefings  

2.  Submit  collaboraKon  proposal  to  CIC  

3.  Agree  Kmelines  

4.  Think  aloud  user  interface  walkthrough  

5.  Preliminary  analyses  

6.  SystemaKc  analyses  

7.  Refine  parser  

8.  Design  student  pilot  

9.  Gather  student  feedback  

32

UTS  CIVIL  LAW  STUDENT  VIEWS  

“takes  the  emo&on  out  of  having  your  work  scru&nized”  respondent  12;  “it  was  not  embarrassing  in  the  way  that  it  can  be  when  a  tutor  or  marker  gives  feedback”  student  7  reflec1on  notes    

“I  definitely  found  it  useful.  It  also  made  me  realise  that  I  tend  to  use  bold,  certain  language  in  making  my  point  towards  the  end  of  each  paragraph  rather  than  up  front  at  the  beginning  (when  introducing  my  point).”  Respondent  5  

“I  realise  now  what  descrip&ve  wri&ng  is  -­‐  the  soKware  had  quite  a  bit  to  say  about  my  lack  of  jus&fica&on  -­‐  also  true  -­‐  pressed  for  &me  and  difficult  circumstances  have  caused  this  for  me  in  this  instance  -­‐  good  to  see  it  sampled.”  Respondent  9  

33

UTS  CIVIL  LAW  STUDENT  VIEWS  

it  is  possible  to  make  a  clearly  stated  point  in  an  academic  way  without  using  one  of  the  markers  …saying  that  if  a  paper  does  not  use  specified  'signposts'  suggests  that  the  wri&ng  is  not  clear  and  ‘academic’  (see  ’&ps’  on  the  results  page),  constricts  wri&ng  style.  I  think  it  is  possible  to  be  clear  about  your  posi&on  without  explicitly  saying  'my  posi&on  is…’.  respondent  11  

“found  that  the  tool  was  limited  in  its  ability  to  pick  up  on  summary  sentences.  It  was  able  to  detect  phrases  such  as  ‘ul&mately,  this  essay  will  conclude,’  or  ‘in  conclusion,’  but  the  use  of  adverbs  such  as  ‘thus,’  and  ‘evidently,’  in  conclusive  statements  failed  to  be  recognized.”…  

“Another  limita&on  is  that  certain  sentences,  which  were  recounts  or  mere  descrip&ons  were  deemed  important,  whilst  more  substan&ve  parts  of  the  essay  containing  arguments  and  original  voice  failed  to  be  detected.”    

ONGOING  WORK  

From  highligh&ng  to  ac&onable  reports  •  How  to  bridge  the  gap  between  the  current  ability  to  highlight  sentences,  and  

capability  to  generate  a  meaningful,  ac&onable  report    “Does  this  highligh&ng  mean  it’s  good?”  •  Without  geYng  into  automated  grading,  students  (and  educators)  are  s&ll  keen  

to  know  if  there  are  signature  paZerns  that  are  proxies  for  quality  –  a  research  ques&on  we  are  now  inves&ga&ng  

 “Algorithmic  Accountability”  •  Cri&cal  to  forge  trust  between  the  stakeholders:  Educators,  Analysts,  Students  

 34

WRITING  ANALYTICS  HANDS-­‐ON  

Wri&ng  Tools  tour:  hGp://utscic.edu.au/events  UTS  —  AWA:  hGp://bit.ly/utsawa    

password:  alasi15  QUT  —  GoingOK:  hGp://alasi.nlytx.io