The ISTE National Educational Technology Standards (NETS S) and Performance Indicators for Students.
ISTE Workshop Research Methods in Educational Technology
description
Transcript of ISTE Workshop Research Methods in Educational Technology
ISTE Workshop Research Methods in Educational
Technology
IIT BombayFebruary 2-9, 2013
Status: Have you done your Homework?
Pre-workshopAssignmentSubmission
Feb.2 Feb.3-8‘Idea proposal’ submission
Feb.9In-workshop
Feb.10-20‘Study-planning
submission
Feb.20 ~ T4E 2013 paper draft
Registered participants
6357 4934 2054+ ? ?4934 2054*
Not doing the assignments and directly submitting something on Feb 20 will not earn a certificate
*Submissions as on Feb. 9, 6 am
Path to course certificate & T4E paper
Recap (Revisiting what we have done so far)
What is this workshop about?
• This workshop is about ET research.
• This workshop is about how you solved the teaching-learning "problem" in your class.
What is Educational Technology?
Tech For Education• Creation and use of
technologies for teaching-learning.
• Creation and use of technology tools to facilitate teaching-learning.
Tech Of Education• Creation and use of
strategies for teaching-learning.
• Focus on what to do with the technology, rather than the technology itself.
This Workshop is a journey
ToET Researcher
• conduct systematic studies to get data about whether our ideas are working.
• provide evidence to support our conclusions.
From ET Practitioner
• teach students;facilitate theirlearning.
• come up with ideas for doing the above.
What route are we taking
Recall: There are no fast-lanes; no short-cutso We cannot simply *listen* to 'lectures' and
expect to absorb the required knowledge.
o We have to *DO* the activities sincerely.o We have to challenge ourselves to go
beyond the obvious ideas.
Recall: The theories underlying this workshopo Spiral Curriculumo Active Learning
Our Journey at a glance
Your idea of:- a teaching-
learning problem
- what teacher (you) & the students will do
- How to
prove your idea works?
Evaluate pre-w/s assignment for strong paper features
In-workshop activities
Guidelines
Moodle resources/ Self-study
Moodle queries
Mentoring by TAs
Fill up & submit ‘Idea proposal’ form
Fill up ‘Study planning template’ for your idea
In-workshop activities
Guidelines
Moodle resources/ Self-study
Moodle queries
Mentoring by TAs
Submit your ‘Study planning’proposal
Moodle queries
TA mentor
Submit paper draft
Pre-workshop Feb. 2 Feb. 3-8 Feb. 9
Feb. 10-20 Feb. 20 ~
In-workshop activities
Guidelines
Research Methodology
Moodle resources for Self-study
i. What qualifies as a research paper ii. What points to address in abstractiii. How to evaluate a research paperiv. Features of a strong research paperv. What referees look for in a paper
i. Peer discussion on pre-workshop assignmentii. Clicker questions on acceptable research paper iii. TPS on : given scenario to research study iv. TPS on : your idea to ET research study v. Evaluation of research papers
i. Workshop slidesii. Videos: reading research papers + doing literature survey
iii. Mid-Workshop assignment
i. Overview of different research paper sections ii. Overview on how to plan a research study
(Novelty, positioning, Procedure soundness, Evidence to collect)
Feb. 2
In-workshop activities
Guidelines
Research Methodology
Moodle resources for Self-study
i. T4E checklistii. Citing research papersiii. Paraphrase to avoid plagiarism
i. What are types of ET research studiesii. What is soundness of procedureiii. Clicker question on what evidence to collectiv. Clicker question on research ethicsv. How to paraphrase other research paper findings
vi. Addressing queries and clarifications
i. Workshop slides +
i. What are types of ET research problemsii. How to plan and execute ET research studiesiii. What are research ethics
Feb. 9
What are referees looking forReferees look
forYour paper must have
Novelty Analysis of prior work to show that your idea is unique
Positioning Analysis to show that your work is required, how your work advances the state of the art
Soundness of procedure
Steps to show that you have implemented solution carefully
Evidence to support claim
Data to show that your solution works as claimed
Overall coherence Consistency between parts of your paper – treatment should address problem, results should give answer to problem
2nd Feb 2013
What exactly is meant by ‘Novelty’?Dictionary: “The quality of being new, unique, original,
innovative, or unusual”.
What has to be novel? At least one of the below:• Your Problem – Research Question(s).• Your Solution – Strategy to solve a known problem.• Your Domain – Adapt a known solution to your context
One of your Moodle queries:• Can a non-innovative strategy be developed into a
strong research paper?• Yes, provided it is positioned well (See next slide).
Strong to Weak
What exactly is meant by ‘Positioning’?Dictionary: “situation/relation with respect to others”.
How to do positioning? Do both of the below:1) Have you shown analysis of related prior work to
bring out the gaps?• papers that have addressed a problem similar to yours• papers that have a solution approach similar to yours
2) Does your solution address any of the gaps above?
As the novelty of your problem or solution decreases, the accuracy of your positioning must increase!
Explain the relation to other work clearlyAwful The galumphing problem has attracted much
attention [3,8,10,18,26,32,37]Bad Smith [36] and Jones [27] worked on galumphing.
Poor Smith [36] addressed galumphing by blitzing, whereas Jones [27] took a flitzing approach
Good Smith's blitzing approach to galumphing[36] achieved 60% coverage [39]. Jones [27] achieved 80% by flitzing, but only for pointer-free cases [16].
Better (Good Above) + We modified the blitzing approach to use the kernel representation of flitzing and achieved 90% coverage while relaxing the restriction so that only cyclic data structures are prohibited.
Source: Mary Shaw, Writing good Software Engineering Research Papers, ICSE 2003
One-line summary – What is the referee’s job?
Referee's job is NOT to find reasons to accept your paper, but to find reasons to reject your paper!
So, not doing all the parts required for a paper – Novelty, Positioning, Soundness, Evidence => Wasted time and effort for you and for referee
The order is important. Even if your work is sound, it can get rejected if it is not positioned properly.
Submitted paper
Positioning wrt related work
Soundness of procedure
Paper recommended for Acceptance
Soundness of evaluation
Novelty of problem/solution If it does not pass the filter
Again: What is the referee’s job?
Typical Acceptance Ratio: 15-20%
80%
Moving Ahead(Today and Beyond):
Towards paper acceptance
First Activity for today!Pair Activity: Peer-review of Idea Proposal assignment.
10 Minutes1. Form pairs and exchange your idea proposal
assignment submission with your partner
2. Read your partner’s answer to Q3.
Are you able to understand the idea? Yes/ No
Does the idea sound exciting to you?Yes/No
3. Read your partner’s answer to Q5.
Is the gap in prior work evident to you?Yes/No
4. Explain to your partner what aspects of his/her answers are not coming out clearly.
PollHow many of you felt that:
You have explained well but your partner still did not understand either your idea or your gap analysis?
Co-ordinators quickly poll and send response by chat.
Value of peer-reviewNever Forget:It is YOUR responsibility to write your paper in such a way
that the reader can understand and follow it easily.
If your friend/colleague has difficulty following your paper, what do you think the referee is going to do?
Value to writers: Every peer-review comment tells you what part of your paper you need to improve upon.
Value to readers: Every paper you peer-review improves your analytical skills which will improve your writing!
Continue with the activityPair Activity: Peer-review of Idea Proposal assignment.
20 Minutes5. Read your partner’s answer to Q6. Suppose you have to
replicate the study,
Do you think that the procedure is described in sufficient detail? Yes/ NoDo you think that the data being collected can give evidence that the idea works? Yes/No
6. Explain to your partner what additional information you require to replicate the procedure of his/her study.
7. Explain to your partner what additional data is required.
PollHow many of you observed that:
Your partner pointed out something that you had overlooked? Something that will improve your study.
Co-ordinators do local poll. We don’t need this answer.
Check if our prediction is true: At least 80% of the participants would have found that
their partner’s comments help to improve their study.
Moral of the story (activity)Don’t wait to complete your study and write your paper
before you seek feedback from peers/colleagues.
Don’t wait to send your work to an ‘expert’ for getting feedback. First get it from your immediate peers.
The more we seek feedback, the more our work will improve, provided we act on the feedback.
Peer-review ethicsAs reviewers, we have to follow three main norms:Non-disclosure: Since the work that we are reviewing
is unpublished, we should maintain its confidentiality. We should not disclose its details to anyone without the author’s consent.
Non-plagiarism: We should not ‘steal’ the author’s ideas and directly use it in our study. More on this later.
Conflict-of-Interest: If we are already working on the same topic as the author, then we should not be the reviewer, to avoid any conflict-of-interest.
Your queriesWe have got your queries in three ways:
Log of chat session from Feb 2nd workshop.
Previous video Q&A session.
Queries posted on Moodle
The core queries that are relevant to all participants will be addressed in today’s sessions; Some will be answered through Moodle.
You can also send queries as chat messages, or Moodle posts, during the tea and lunch breaks.
Some queries to be taken up todayWhat to measure for learning, other than marks?How to setup study and measure systematically? Next session
How to find prior work in a systematic manner? If there is no related work yet, of the present work, what is
to write down in the research paper? Journal or conference, which is good for research
purposes? Last session
Next session: Delving into RMETReferees look
forYour paper must have
Novelty Analysis of prior work to show that your idea is unique
Positioning Analysis to show that your work is required, how your work advances the state of the art
Soundness of procedure
Steps to show that you have implemented solution carefully
Evidence to support claim
Data to show that your solution works as claimed
Overall coherence Consistency between parts of your paper – treatment should address problem, results should give answer to problem
9th Feb 2013