Welcome to the 2014 ITEST Panelist Orientation

First, Thanks!

• Your contribution of time and your insights are immensely important to us.

• You have each been recruited for specific reasons; express yourself.

• Our aim is (mostly) not consensus building; voice the range of perspectives.

• We do hope you’ll find the panel experience interesting and rewarding.

Overview of Presentation

• Quick overview of the ITEST program

• The panel process and preparing reviews

• Conflicts of Interests and Confidentiality

• Avoiding implicit bias

• Some procedural details

First Things First

If you haven’t yet done so –

Please read the Solicitation!

Program Solicitation

NSF 14-512

Aim of the ITEST Program

Ensure a high-quality STEM workforce by supporting projects that:

- Increase student awareness of career opportunities in STEM and cognate fields.

- Motivate students to pursue appropriate educational pathways to STEM-related careers.

- Provide technology-rich experiences that develop disciplinary knowledge, practices, and non-cognitive skills needed in STEM fields.

STEM Workforce & Cognate Fields

• Traditional STEM Disciplines

• Information and Communications Technology (ICT)

• Computing, Computer Sciences, Data Analytics, and related fields.

• Professionals at all levels, including technicians, technologists, scientists, engineers, computer scientists, and mathematicians.

ITEST Projects…

• Must involve students.

• Are informed by relevant research.

• Will focus on workforce development for youth or school-to-work transitions.

• Will conduct foundational or design-based research of strategies, conditions and contexts that improve student STEM learning pathways and STEM-focused career preparations and mentorships.

Two Types of Projects Supported

Strategies projects that address the creation and implementation of innovative workforce-related activities or programs.

– Awards for up to $1.2M for projects lasting up to 3 years.

– Approximately 15-20 projects to be supported.

SPrEaD (Successful Project Expansion and Dissemination) projects that support the wider and broader dissemination and examination of innovative strategies.

– Awards for up to $2M for projects lasting 3-5 years.

– Approximately 5-10 projects to be supported.

Strategies Projects must address one or more of the questions listed in the solicitation

• Experiences that foster student competency

• Instructional and Curricular Models

• Roles of business and workforce members

• Strategies for parents, mentors, & caregivers

• Strategies for principals, counselors, & other school administrators

• Engaging diverse underrepresented populations

See page 6 of the solicitation for details.

SPrEaD proposals must…

• Describe the innovation and the contexts and conditions for broadening and scaling.

• Present evidence for the feasibility of impacts.

• Explain how the proposed project builds on previous implementations.

• Identify anticipated contributions to knowledge.

• Present a study design capable of generating robust evidence of the strategy’s potential.

• Include plans to document the implementation.

• Involve a partnership with another type of institution.

Organizatio

n

submits

via

FastLane

NSF

Program

Program

Officers

Division

Director

Concur

DGA

Organization

Ad hoc

Panel

Award

Proposal Review Process and Timeline

Decline

6 Months 30 Days

Proposal Receipt

at NSF DD Concur DGA

Award

Recommend Advise

Panel Organization

• Panels were organized primarily according to the disciplinary content area

• Most panels include either Strategies proposals or SPrEaD proposals, though a few panels are a blend.

Function of Peer Review at NSF

• You are experts, providing advice to the ITEST program at NSF, and to PIs.

• Your verbatim comments and ratings are sent to PIs.

• Your individual reviews combined with the panel summary are critical in funding decisions

Reviewer Responsibilities • Primary panelists read and write reviews on

assigned proposals. They introduce the proposal at the panel meeting and write the panel summary.

• Secondary panelists read and write reviews on assigned proposals and contribute to the panel discussion.

• Panelists are encouraged to read all proposals so long as there are no Conflicts of Interests.

Parts of an Individual Review

• Rating

• Intellectual Merit

• Broader Impacts

• Summary Statement

Ratings – select only one Excellent, Very Good, Good, Fair, Poor

• Excellent - Outstanding proposal in all respects; deserves highest priority for support.

• Very Good - High quality proposal in nearly all respects; should be supported if at all possible.

• Good - A quality proposal, worthy of support.

• Fair - Proposal lacking in one or more critical aspects; key issues need to be addressed.

• Poor - Proposal has serious deficiencies.

Rating & Review • Make sure your rating is consistent with your

written text.

• You may change your rating and edit the written review anytime prior to the conclusion of the panel.

• It is imperative that your ratings and reviews are entered into Fastlane prior to the panel meeting. We strongly encourage you to upload your reviews at least two days prior to the panel meeting.

Intellectual Merit and Broader Impacts

• Intellectual Merit – the potential to advance knowledge

• Broader Impacts – the potential to benefit society and contribute to the achievement of specific, desired societal outcomes

“Consider what the proposers want to do, why they want to do it, how they plan to do it, how they will know if they succeed and what benefits would accrue if they are successful.”

NSF Review Criteria All proposals are reviewed under two criteria: Intellectual Merit and Broader Impact 1. What is the potential for the proposed activity to:

a. advance knowledge and understanding within its own field or across different fields (Intellectual Merit); and

b. benefit society or advance desired societal outcomes (Broader Impacts)?

2. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?

3. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?

4. How well qualified is the individual, team, or institution to conduct the proposed activities?

5. Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities?

Results of Prior Research

• Does this project build on the results of related prior projects by the PI’s?

• If yes, is there evidence provided about the intellectual merit and broader impacts of the prior project(s)?

• How has the prior project influenced this project?

External Evaluation A proposal must describe appropriate project-specific external review and feedback processes.

• The review might include an external review panel or advisory board or a third-party evaluator.

• The review must independent and rigorous

• The proposal must – Describe the expertise of the external reviewer(s);

– Explain how that expertise relates to the goals and objectives of the proposal;

– specify how the PI will report and use results of the project's external, critical review process.

• There can be different groups providing formative and summative evaluation

Dissemination Plan • Proposals should include plans for creative

and effective dissemination of findings to researchers, policymakers, practitioners, or other relevant constituency groups.

• Applicants are encouraged to bring the same levels of insight and creativity to the dissemination aspect of their proposal as other activities.

Summary Statement of the Review

• The summary should be your overall analysis of the strengths and weaknesses of the intellectual merit and broader impacts of the proposal.

• Do not omit this section.

• Do not summarize the proposal itself in this section.

What makes a Good Review?

• Analytic and not just descriptive.

• Uses criteria consistent with the solicitation.

• Adheres to the Merit Review criteria.

• Uses grammatically correct form whether done in paragraphs or bullet form.

• Usually one to two pages; succinct but substantive.

Supplementary Documents • Because they are not required, you cannot base

your rating on the presence or absence of supplementary material, in the case of ITEST - letters of support.

• You may indicate that additional letters of commitment should be obtained if negotiations go forward.

Common Guidelines for Education Research and Development

• This solicitation has been revised to incorporate the publication jointly developed and issued by the National Science Foundation and the Institute of Education Sciences in the U.S. Department of Education.

• The Guidelines are just that, “guidelines.” They are intended to be helpful for PIs, but they are not to be used as review criteria. For purposes of the review process, awareness of the guidelines may help you judge the appropriateness of methodologies being proposed if the language of the guidelines is being used.

Guidelines

• The Guidelines describe six types of research studies that can generate evidence about how to increase student learning or achieve other outcomes.

• For each research type, there is a description of the purpose and the expected empirical and/or theoretical justifications, types of project outcomes, and quality of evidence.

Minimizing Bias in Reviews Implicit bias toward a group:

• Non-conscious hypotheses/stereotypes – often a small bias, when in same direction, has a large effect over

time

– often about competence

• Widely culturally shared – all people, even members of under-represented groups, hold

implicit biases about these groups

– people are often not aware of them

• Effects greater under circumstances of: – lack of information

– stress from competing tasks

– time pressure

Minimizing Bias in Evaluation

• In your consideration of ITEST Proposals, try to:

– Remain aware how implicit biases might affect your review.

– Decrease time pressure and distractions during the review process.

– Rate on explicit criteria rather than global judgments.

– Point to specific evidence supporting judgments.

Questions?

Conflicts of Interest and Confidentiality

There is nothing wrong with having conflicts of interests.

Conflicts of Interests (COI)

• The importance of being impartial when reviewing for NSF

• How to identify a COI

• Examples of COI

• How to handle a COI

COI Examples

You have a Conflict of Interest if you or your family:

• Are at the institution of the PI or Co-PI.

• Have a financial interest in the proposal.

• Received income from the proposing institution in the last 12 months.

• Your spouse or child has a relationship with a proposal

• Are involved in the proposal.

COI Examples

You have a Conflict of Interest with

• Your thesis advisor: lifetime COI

• Your thesis student: lifetime COI

• A co-author: within the last 48 months

• A co-editor: within the last 24 months

• Close personal friend

• If you feel you cannot be impartial

Not sure? Check with your Program Director.

Not a COI

“Professional friends”

You may know many individuals in the proposals. In general, that is fine, and usually is not a COI. We need your expertise and knowledge of your field to review the proposals.

If you have a COI

• Contact the Program Director in charge of your panel right away.

• Do not read or review the proposal.

• In FastLane check ―”I am declining to review this proposal,” and

• Enter a statement that you have a COI in the box at the bottom of the review form.

Confidentiality • All discussions are confidential.

• Other panels will be meeting. Keep discussions confidential within the building…. And Please…No Photos, Tweeting, Slideshare, Flickr or Facebook posts about the panel or proposals.

• Take care when discussing with fellow panelists in case someone has a COI.

• Leave all documents at NSF and destroy any you left at home.

• Remember: Panels make recommendations. The final outcome may be different than anticipated.

• Identities of panelists and reviewers are confidential.

Questions?

FastLane

• Logging in and Changing Password

• Accessing your proposals

• Print-On-Demand Option

• Updating Personal Information

• Banking information

Reviews in FastLane

• All NSF grant proposals are submitted and reviewed using the FastLane system.

• You will have received an email with your access and log-in information.

• A demonstration version of FastLane is available online.

– Go to: www.FastLane.nsf.gov

4

0

Getting Started in FastLane

• Logging in • Changing Your Password • Entering/Updating Your Personal

Information • Reimbursement/Banking information • Accessing Proposals & Requesting Copies

*Be sure to refer to your Panel Confirmation

Letter for more details.

4

1

Nati

on

al

Scie

nce F

ou

nd

ati

on

https://www.fastlane.nsf.gov/NSFHelp/flashhelp/fastlane/FastLane_Help/fastlane_help.htm#introduction_to_fastlane.htm

4

2

4

3

Nati

on

al S

cie

nce F

ou

nd

ati

on

Panel Review System

Nati

on

al

Scie

nce F

ou

nd

ati

on

It helps us if you list strengths

and weaknesses of the

proposal in the Summary

Statement

4

4

During the Panel • Program Officers may chair the panel or may have

assigned a chair.

• The Panel Chair will facilitate the panel meetings to assure that all proposals are discussed in a timely manner.

• For each proposal, the Primary Reviewer will start the discussion of the proposal by providing a brief overview and their own review. The Primary Reviewer will take notes of the discussion and write the Panel Summary.

• Each proposal receives a panel rating: highly competitive, competitive, or not competitive

• Panel Summary should encapsulate major points of the discussion. All reviewers must approve the Panel Summary.

4

5

FYI: Panel Summary Parts

• Intellectual Merit (Strengths & Weaknesses)

• Broader Impacts (Strengths & Weaknesses)

• Statement about the Data Management Plan

• Statement about the Post-Doc Mentoring Plan if applicable

• Statement about prior NSF Support

• Summary Statement

Travel and Hotel

Travel arrangements can be made only using SATO Travel at 1-800-817-5257

Hotel (if needed): Accommodations Unlimited at (703) 385-5680 or toll free (800) 201-4005.

Flat Rate Fee • Out of area: A flat rate of $480 for 2 meeting days

attended and 2 travel days at $280 per day.

• Local area: $280 for each full meeting day attended.

• This fee is meant to cover all expenses, and an IRS 1099 will be generated at the end of the year.

• All payments to panelists are made by direct deposit.

Thank You!

If you have more questions…….

Contact: Your Panel Program Director