GEDU 6170 Reviewing

Saad Chahine, PhD May 15, 2014

Initial Questions

• Why should I be interested in this article?– Relationship to my current work/research

• Does this article state its problems clearly and succinctly?– Difference between difficult (high level of ability) and unclear (poorly

written)

• Is the literature that is cited relevant, important, and current?– Seminal articles & recent work

• Do the findings in this article address the issues that it raises?– Find the links to the RQs/Hypotheses

(Shank & Brown, 2007)

Initial Questions • Are the measurements and findings reliable?

– Checks of consistency

• Are the measurements and findings valid?– Checks for conceptual meanings and interpretations

• Are the statistical tests overly sensitive, so that they might be “Finding” results that are not really there?– E.g., Correlations are very sensitive to sample size…they will often be

significant. You need to check the actual r value!

• Are the statistical tests overly conservative, so that they might be missing results that are really there?– E.g., Aggregated data to the school level. The grouping averages

individual student’s variation. (Shank & Brown, 2007)

Detailed Questions

• How generalizable are the results and findings?– Be carful of absolute statements with small sample sizes…

• Generalizability • Representativeness • Imbalance• Qualitative?

• How realistic are the results and findings?– Do you think this could really happen in real life?

• Ecological validity • Independent variables • Dependent variables • Interferences & collinearity• Hawthorn Effect (Qq) • Novelty • Experimenter effect • Issues with pre/post designs (Shank & Brown, 2007)

Detailed Questions

• Does the article fulfill its potential? (More focused on Qual) – Competiveness: Proving a point– Appropriation: Denying personal beliefs – Rigidity: Method applied needed to be modified – Superficiality: Do you think you would get the same results in different

settings? – Sentimentality: Empathy or Manipulation – Narcissism: Sometimes its too much about he researcher – Timidity: What did they leave out and why?

(Shank & Brown 2007)

Can you answer these questions?

• How can an understanding of the literature help me improve my skills as a personal reviewer?

• Do I have enough content knowledge to understand this article? Do I need more content background? Where and how can I get it?

• Do I have the technical skills I need in order to understand this article?

(Shank & Brown, 2007)

Activity

• In groups examine the reviews and address the following:

– Elements of the review that were really done well– Elements that are missing from the review – Does the review help you decide whether to read

the book?– What else would you have liked to have seen in

the review?