ProgramEvaluationPlan ’ for’ HarrisburgParks’and ... · The program that this evaluation plan...
Transcript of ProgramEvaluationPlan ’ for’ HarrisburgParks’and ... · The program that this evaluation plan...
1
Program Evaluation Plan
for
Harrisburg Parks and Recreation
Youth Soccer Program
By: Shawn Marble
February 17, 2012
2
Table of Contents
History
3-‐4
Programs to be Evaluated
4-‐5
Stakeholders
5-‐6
Purpose of the Evaluation
7-‐8
Structure and Context of the Program
8
Evaluation Questions
9-‐11
Population Description
12-‐14
Body of Knowledge
14-‐16
Evidence Reliability and Validity
16
Resources Needed
17
Appendix A
18
3
History:
The Harrisburg Parks and Recreation Department was formed in January 2007 with one full time
staff member. In January 2009, the Town of Harrisburg hired their first Parks and Recreation
Director to bring the department up to two full time people. This was done because the Town
was planning on undertaking the operations of the Harrisburg Youth Association (HYA). HYA
had existed in the Town of Harrisburg for sixteen years. It was a youth association run entirely
by volunteers and parents. The youth association had provided youth athletics in the spring, fall
and winter seasons including baseball, softball, soccer, basketball, cheerleading and football. The
participant numbers had grown so much that it was no longer manageable by volunteers. The
HYA board approached the Town and after months of preparation and a vote, the Town of
Harrisburg took over the operations of the youth athletic program in January 2009.
The Harrisburg Parks and Recreation Department offers a variety of programs to include special
events such as the Harrisburg Halloween Festival and the Harrisburg July 4th Celebration. It
offers classes such as art, yoga and bridge. By far, the biggest program the Parks and Recreation
Department runs is the youth athletic program. In the spring of each year, the department offers
baseball for age 3-15, softball for ages 7-15 and soccer for ages 3-15. In the fall the department
offers those three sports along with Pop Warner Cheerleading and Pop Warner Football for
children age 7-15. In the winter the department offers basketball for ages 5-15. In 2011, the
department registered 3,281 for youth athletics. Soccer was the largest activity with 1,400
participants.
volunteer coaches and several volunteer coordinators. There are sport specific coordinators that
4
volunteer to help direct each league. For soccer there is a ¾ and 5/6 coordinator, a 7/8 and 9/10
coordinator and an 11/12 and 13/15 coordinator. These coordinators work very closely with the
parents, coaches, participants and staff and help with the draft process, forming practice and
game schedules and overseeing the league in play.
Program to be Evaluated:
The program that this evaluation plan will focus on is the Harrisburg Parks and Recreation youth
soccer program for ages 11, 12 and 13 (11/13) male and female division and the 14, 15 and 16
(14/16) coed division. Up until spring 2011, the Harrisburg Parks and Recreation Department
offered soccer for the following age groups: ¾ coed, 5/6 coed, 7/8 coed, 9/10 coed, 11/12 coed
and 13/15 coed. In spring of 2011, the 9/10 age group was split into boys and girls divisions.
There was an overwhelming support of this and there were more than enough participants
registered to make this happen. This continues today. The reason this split occurred is because
the 9/10 soccer coordinator had heard from enough parents that they felt this age group should be
split up by gender. He also had observed that the registration numbers had dropped off for
females when they went from 7/8 to 9/10. It was decided that by splitting this age group, a better
opportunity could be provided to the males and females in this league.
In January 2012, before registration for the spring season began, the department decided to make
some additional changes. The 11/12 age division was changed to 11/13. It was also split up by
gender. The oldest age group was changed from 13/15 to 14/16. These changes were made for
several reasons. The 11/12 age group was split up to continue what had been done in 9/10. The
coordinators and staff felt that since the males and females were playing separately in 9/10, they
should continue that in 11/12.
5
We also added 13 year olds to that division. That was so that we could add 16 year olds to the
older league by forming a 14-16 division. It was thought that many sixteen year olds may still
want to play recreational soccer, but that once they turned sixteen they could no longer play in
Harrisburg. By changing the age breakdowns a little, we could include these sixteen year olds.
The only reason this age division was not broken down into male/female is because it has been
difficult in the past to get enough participants registered in this age group to have enough teams.
It was feared that if it was split, there would not be enough females or males alone to have a
league. So this league was left coed.
The ultimate goal of these changes it to provide a better program for the youth involved. These
decisions were made based on parent and participant requests and staff and coordinator
observations. Since spring 2012 will be the first season these changes will be in effect, the
Harrisburg Parks and Recreation Department feels it would be highly beneficial to evaluate the
program at the end of the season to make sure that participants and parents are satisfied and that
the participants are benefitting from the changes that have been made. This will also tell the
Parks and Recreation Department whether to continue with these changes for the fall 2012
season, or whether to go back to the original age groups.
Stakeholders:
A successful program evaluation is developed using questions posed by the key stakeholders,
individuals, groups, or organizations that have a significant interest in how well a program
Freeman, 2004, p. 18).
The major stakeholders in this case are the parents, soccer participants, coaches, coordinators,
parks and recreation staff, the Town Administrator and Town Council members.
6
The parent, participants, coaches and coordinators all have a stake in this project because
changes to the program directly affect them. They will be the ones providing the majority of the
feedback regarding the changes to the program. The staff have a stake because it is their
responsibility to run a youth soccer program that benefits its participants in the best way
possible. The Town Administrator and Town Council have a stake because they are managing
the tax dollars that fund this project and are held accountable by citizens of the Town of
Harrisburg.
As the recreation coordinator for the Parks and Recreation Department I have a strong
relationship to most of the stakeholders involved and the program. As a staff member I work
under the Town Council and Town Administrator. It is my responsibility to make sure I am using
our budget wisely and providing the best customer service I can. I work closely with parents and
participants during registration and the season to ensure that each participant is having a good
experience. I address complaints and issues with any parents, players or coaches. I work hand in
hand on a daily basis with the coordinators and coaches to help with the draft process and
forming practice and game schedules. We all support each other to make sure these tasks are
website. I work daily with the other staff in the department to make sure each detail of the
program from registration to ordering uniforms and equipment to scheduling picture day is done
proficiently. The organization and management of this program is largely a group effort and
every piece of the puzzle must be in place for the program to run smoothly.
7
Purpose of the Evaluation:
The type of evaluation that is being done is a summative evaluation. This main purpose of this
evaluation process is to determine whether the programs performance meets the criteria that are
expected. This evaluation will help the department and upper management make a decision about
whether to continue the changes made to the youth soccer program or whether or not it needs to
be re-structured. It could also affect the funding and other resources, such as staff, that is
provided (Rossi, Lipsey and Freeman, 2004).
In this case, the staff would be primarily responsible for conducting the evaluation. Certain
stakeholders such as program participants, parents and volunteers may provide input but will not
provide direct input in the decision making process. Therefore it is especially important that the
information that is found be scientifically credible. It could potentially be used by the Town
Council or Town Manager to make important decisions about the continuation of the program. If
the program evaluation results are not what the Council wants to see, staff positions and other
important resources such as money and equipment, could be eliminated. However, positive
results from the evaluation process could help convince decision makers that additional resources
are needed. Even if the program changes are not successful, staff will be needed to re-structure
the program and reallocate resources. In either situation, this evaluation can help provide
important information needed to make educated decisions by upper management and elected
officials (Rossi, Lipsey and Freeman, 2004).
8
Structure and Context of the Program:
The purpose of the evaluation is to see if the outcomes projected by the Parks and Recreation
staff occur from the changes that are made in the 11/13 and 14/16 age groups for soccer. The
short term goals of the youth soccer program are the same for all participants: to learn the basic
fundamentals of soccer; to get weekly physical activity and interaction with other soccer players;
to get weekly exposure to the outdoors and Harrisburg parks; and to learn to follow directions
and play as a member of a team. It will be challenging to measure the number of participants
who learn the basic fundamentals of soccer and if they learn to play as a member of a team, but
this may be the individual opinion of each participant. The department may be able to determine
this by the number of participants who continue in the program season after season. Additional
intermediate and long term outcomes can be seen in the program logic model in Appendix A.
By making the said changes to the soccer program, it is not expected that the outcomes will
necessarily change, but that they will be met. The overall goal of the department would be for
each child to meet the short term, intermediate and long term outcomes of the program. These
changes to the program will hopefully allow more participants to succeed in having these
outcomes than if the changes had not been made.
9
Evaluation Questions:
Below is a list of sample questions that could be used to evaluate the changes to the youth soccer
program. These questionnaires would only be given to participants (or the parents) of the 11/13
male and female division and the 14/16 coed division.
1. Have you participated in the Harrisburg Youth Soccer program prior to this season?
A. Yes B. No
If you answered yes, how many seasons have you played?
Answer: _______________________________________
2. What age group did you participate in?
Answer: _______________________________________
3. How would you rate your experience this season?
A. Great B. Good C. Neutral C. Ok D. Poor
4. (11/13 Participants Only) Did you agree or disagree with the change made to the division
of the 11/13 coed league into 2 divisions: 11/13 male and 11/13 female?
A. Strongly Disagree
B. Disagree
C. Neutral
D. Agree
10
E. Strongly Agree
5. (11/13 Participants Only) Would you like to see the 11/13 age group continue to be split
into male and female?
A. Yes B. No
6. (11/13 Participants Only) Would you or your child continue to play in the Harrisburg
Parks and Recreation Youth Soccer Program if the 11/13 age division stayed divided by
gender?
A. Yes B. No
Why or Why Not?
Answer: _______________________________________
7. (14/16 Participants Only) This age division was previously 13/15. This season it changed
to 14/16. Did you agree with the inclusion of 16 year olds in this age division?
A. Yes B. No
8. (14/16 Participants Only) Would you like to see this age group continued to be for 14, 15
and 16 year olds next season?
A. Yes B. No
11
9. (14/16 Participants Only) Would you or your child continue to play in the Harrisburg
Parks and Recreation Youth Soccer Program if the 14/16 age division stayed the same
next season?
A. Yes B. No
Why or Why Not?
Answer: _______________________________________
10. Approximately how many days did you or your child meet to have soccer practice?
A. 0-5 B. 6-10 C. 11-15 D. 16-20
11. Do you or your child feel they have learned the fundamentals of soccer this season?
A. Yes B. No
12. Did you or your child feel they were part of the team?
A. Yes B. No
13. Please feel free to provide any additional comments or feedback regarding the changes
made to the 11/13 male/female or 14/16 coed soccer divisions.
________________________________________________________________________
________________________________________________________________________
12
The questions that are listed here will help determine how many participants felt satisfied with
the age group they or their child was in. It also provides them a chance to let us know how many
days there child was at soccer practice which also tells us how many times during the soccer
season they were outside at Harrisburg Parks. In addition, by providing a blank question at the
end for feedback, parents or participants can write anything additional they wish to include about
the program.
t we could set a definite percentage that would determine if the program would
continue or not. It would depend on the number of respondents we received and the use of other
evaluation methods such as registration numbers, observations made by staff and coordinators
and the feedback received from participants and their parents. However, I would feel confident
that if over 60% of respondents were satisfied or strongly satisfied with the program that we
should continue it. I would also conclude that if 60-70% said they plan on registering for the
program next season that we should continue with the current changes. If the response rate is
lower than that, I would be hesitant to continue without some hard thought and discussion with
other staff members, participants, upper management, parents and volunteers.
Population Description:
The population that I will be evaluating will be that of the participants and parents of the 11/13
male and female soccer division as well as those participating in the 14/16 coed soccer division
for the spring 2012 season. This population consists of 11-16 year olds and their parents who live
in Cabarrus and Mecklenburg Counties. They are predominately Caucasian and African
American, middle class citizens.
13
It will be easy to identify this population because they will all be registered with the Harrisburg
Parks and Recreation Department and are in a computer database system which has their address,
email and phone number. For the evaluation questions, it will be easiest to reach them by the use
of email and the internet. Questionnaires will be emailed out to each participant in the 11/13 and
well as paper copies could be kept at the Parks and Recreation office. Coaches could assist by
handing out questionnaires at their practices and games. We will not use a sample, but will
collect date from the entire population.
Data Collection Techniques and M easurement Tools:
The main forms of evaluation instruments that I plan on using to evaluate the Harrisburg Parks
and Recreation youth soccer program would include questionnaires, observations and the use
of registration numbers. From the evaluation questions listed on page 9-‐11, both quantitative
and qualitative date will be collected. The quantitative date will include the numbers of
participants who have participated before, those that plan on participating again and the
number of participants who answered yes or no to each of the yes/no questions. The
qualitative data will be the information gathered that describes why or why not the participant
does or does not plan to participate again and the open ended question at the end of the
survey which allows the participant to provide feedback.
We will use quantitative data also when looking at the registration numbers for each age group
when comparing them from the fall of 2011 to the spring of 2012. We will gather additional
14
qualitative date when observing the 11/13 and 14/16 age divisions during practice and games.
We will also gain qualitative date from feedback during these observations.
We plan on collecting the majority of our data by the use of questionnaires that will be
distributed at the end of the spring soccer season. This would occur in late May or early June
2012. In order to ensure a strong response rate, an incentive may be offered, such as a small
discount on a future registration fee or a parks and recreation item such as a t-‐shirt. If an
incentive cannot be offered, parents would be emailed and/or called to reiterate the
importance of providing feedback on the questionnaires. If coaches distribute and collect these
questionnaires at the last few practices, they can help the department ensure they are
collecting the responses from the majority of participants.
Body of K nowledge:
Title IX played a huge role in bringing sports equality to both males and females. In an excerpt
taken from Title IX gender-
permitted prior to puberty and in coed sports where there are equal numbers of males and
females on both teams
par. 6).
It has been found that prior to puberty, there is no physiological reason to separate boys and girls
in sports leagues. However, sometimes it is important to split these leagues based on differences
in experience or skill level. Furthermore, when soccer first became popular in the United States
in the 1970s, it was rare that there were enough girls to form female only divisions. However,
15
with the continued growth of soccer, especially since the 1990s, it is feasible in some leagues to
have enough males and females to split divisions by gender (Snow, 2007).
According to Snow, males and females should play each other prior to puberty if the competition
many cases, girls have not had the same experience or skill instruction
interest of the participants to split the league by gender. This is not to say that boys and girls
cannot play together in a coed league, however, the main goal would be that each group is of the
performance and skill level and of the relatively same size (Snow, 2007).
From this research, a case can be formed as to why the 11/13 age group should be divided by
gender. Through observations and parental and participant feedback, it was determined that the
size of the participants and the skill level was not the same when children reached this age
division. Therefore, it was more beneficial to both genders to divide this division by gender.
Again, this was also done because there were enough females to form their own division. For the
spring 2012 season so far, there are 55 girls registered. This is enough for five teams of eleven
which is more than enough for a league. The male division has 103 11/13 year olds registered.
Together, this totals 158 participants. In the fall 2011 season, when this age group was coed,
there were only 111 participants.
The important of youth athletics for teenagers seems like an obvious thing. Team sports help
teenagers grow physically, emotionally and socially. Team sports teach important skills such as
competition. Although the main goal of youth athletics should be for participants to learn to work
as a team, it is important to learn competition a skill most will face in life in academics, sports
children an opportunity to understand the healthy aspects of competition in a friendly
-2).
16
Physically, youth who participate in athletics are more likely to describe themselves as being in
good shape and are more likely to make wise healthy food choices. Teenagers also benefit
including those in high school, are likely to receive an end of season trophy for participating in
the team sport. Receiving a trophy with their teammates is intended to help youth feel a sense of
This is another case for why youth athletics should be offered for teens of all ages. Previously
our leagues stopped at age fifteen. By extending that age group to sixteen, we are allowing youth
to get the benefits of youth athletics for one more year.
Evidence Reliability and Validity:
In order for data collection procedures to be considered accurate, they must be valid and
reliable. Some participants may be hesitant to answer the questionnaires honestly and may
want to answer the questions the way they think the evaluator wants or expects (Evaluation
Primer, 1998, par, 1). One way to ensure this will not occur is to make the questionnaires
anonymous.
We could also try the questionnaire out on a small group of participants and give them the
same questionnaire twice at different times. If they respond with the same answers, this will
help ensure that the questionnaire is reliable and valid. We would also want to ensure that the
questions we are asking are providing the information that is needed (Evaluation Primer, 1998).
This can be accomplished by allowing input from the stakeholders and making sure the
17
questionnaire is tested by staff and coordinators to make sure the questions are easily
understandable.
Resources Needed:
The biggest resource that will be needed will be staff and time. We will need to have at least
one staff person in place who can take the time needed to develop a good questionnaire and to
analyze the results. This person will also need to be available to observe practices and games
and to spend time talking with parents and coaches. This would preferably be someone who
currently works in the parks and recreation department who has thorough knowledge of the
youth soccer program and the changes that will be made. A small budget will be needed to
print questionnaires.
In addition, support from the Town Administrator and Town Council will be needed. It would
not be worth the time and effort to collect this data if upper management and the elected
officials did not support staff in doing so. However, it is to their benefit to evaluate their
programs. If staff does not have the support of its stakeholders or the necessary resources to
complete the evaluation, the evaluation will be difficult to conduct. In turn, there will be no way
for the staff to know whether to continue with program changes. If they do continue and
registration numbers drop, the Parks and Recreation Department could lose funding and
support for the Harrisburg Youth Soccer program. Elected officials may need some convincing in
order to allocate staff time and money to performing a large evaluation such as this, but
ultimately it could save the Town money and any long term negative repercussions of not
evaluating the program.
18
APPENDIX A
19
References
Evaluation primer - ensuring evaluations yield valid and reliable f indings. (1998). Retrieved
2/18, 2012, from http://www2.ed.gov.prox.lib.ncsu.edu/offices/OUS/PES/primer6.html
Payne, C. & Fogarty, K. Importance of youth involvement in sports. Retrieved 2/17, 2012, from
http://edis.ifas.ufl.edu/fy1001
Rossi, P., Lipsey, M., & Freeman, H. (2004). In Shawn L. C. (Ed.), Evaluation A systematic
approach (7th ed.). Thousand Oaks, CA: Sage Publications, Inc.
Snow, S. (2007). The coed question. Retrieved 2/17/12, 2012, from
www.socceramerica.com/article/22428/the-coed-question.html