EVALUATION OF THE LANGUAGE CAMP 2012: TOWARDS...

Jurnal Penyelidikan Kent Bil. 12/2014

43

EVALUATION OF THE LANGUAGE CAMP 2012: TOWARDS USEFUL EVALUATION

Shubashini Suppiah

ABSTRACT This article reports on a formative and summative evaluation of the English Language Camp which is a compulsory course in B.ED TESL foundation programme. A need for an evaluation of the English Language Camp programme was initiated in the hope that the effectiveness of the programme can be articulated. In doing so, a utilization-focused evaluation approach was adopted to focus on two key areas namely to see if the activities planned had fulfilled the objectives stated in the course syllabus and to identify areas of strengths and shortcomings that could further be improved on. In envisioning the centrality of evaluation use and responding to issues of under-use, non-use or even misuse of evaluation findings, Michael Quinn Patton (1997) developed the utilization focused evaluation (UFE). The main crux behind the Utilization Focused Evaluation (UFE) is that it is „done for and with specific, intended primary users for specific intended uses” (Patton, 1997:23). In line with this particular purpose, data was gathered through the use of a perception questionnaire and reflective logs on a purposive sample of 24 participants. While most of the responses from the perception questionnaire yielded positive feedback in that they fulfilled the stated outcomes in the course objectives, the reflective logs on the other hand provided some valuable feedback into addressing the strengths and areas that could be improved. Articulating the findings of the evaluation would be essential as it would provide the necessary feedback for the next English Language Camp which would once again be organised at the institute level.

INTRODUCTION

The English Language Camp has been an annual event for all 27 teacher training institutions across Malaysia which offer the B.ED (TESL) foundation course since 2003. The English Language Camp may be carried out at national, zonal or institutional level. Prior to 2012, the English Language Camp was always conducted at national or zonal level. A selected teacher training institute would be selected to organize and run the English Language Camp. In 2010 however, a mutual decision was reached in which it was unanimously decided that the subsequent language camps would be organized and run at the individual institutional levels.

Kent Teacher Training Institute organized and ran its very first English Language Camp in March 2012. As a teacher trainer who was involved in the planning and organisation of the English Language Camp, I felt it was imperative to evaluate the language camp programme for three main purposes which will be further explained in the following sections of this paper.


44

BACKGROUND OF THE PROGRAMME The English Language Camp is a 2 credit component in the B.ED TESL foundation course. It is carried out in the second semester of the B.ED TESL foundation syllabus. It is a mandatory 60 hour or 3 day programme in which all B.ED TESL pre service teacher trainees‟ are required to participate. This camp would introduce students to new and interesting ways of learning and improving their command of the English Language. The pre service TESL teacher trainees‟ would also be provided with the opportunities to develop their planning and decision making skills and leadership skills.

In the three day English Language Camp that was held in Kent Teacher Training Institution, various activities which included an Institute Treasure Hunt, Field Trips, Drama and Skits Workshops and Mini Language Games were carried out. These activities were designed to afford the pre service teacher trainees‟ the opportunity to utilise the English Language in a fun, meaningful, creative and interactive manner. The pre service teacher trainees‟ were also given the opportunity for managing certain activities under the different committees. This is in line with leadership qualities and team building which is also emphasised in the course syllabus.

A need for an evaluation of the English Language Camp programme was initiated by the B.ED TESL foundation course coordinator and the head of the language department. The proposal for an evaluation that would focus on two key areas that we intended to evaluate emerged from the need to see if the activities we had planned had fulfilled the requirements stated in the foundation course syllabus as well to identify areas of strengths and shortcomings that could be improved upon and to specify the intended outcomes for the next English Language Camp in 2013. This evaluation would be essential as it would provide the necessary feedback for the next English Language Camp which would once again be organised by us at the institute level.

OBJECTIVES OF EVALUATION The two main objectives of this evaluation are : 1. To identify the perception of the pre service teacher trainees‟ involved

as to whether they felt that the Language Camp was beneficial to them in what ways.

2. To identify areas of strengths and specific areas that could be improved on for the upcoming English Language Camp which will be held in 2013.

EVALUATION QUESTIONS

The two evaluation questions that were derived from the evaluation objectives and would frame the basis of this evaluation are : 1. What are the perception of the pre service B.ED TESL teacher

trainees‟ of the English Language Camp? Were the outcomes achieved?

2. What are the areas of strengths/specific areas that could be improved on?


45

DEFINITION OF EVALUATION According to Lynch (1996), evaluation is a systematic attempt to gather information in order to make judgements or decisions. Patton (1982), describes the process of evaluation as (1) systematic collection of information (2) for use by specific identifiable people or groups for the purpose of (3) making decisions about and/or improving program effectiveness. In the same vein, Kemmis (1983), asserts that evaluation will assist in clarifying program concerns and provide fair, relevant and accurate information to help those in and around the program to understand it better. In view of this, language programme evaluation have been receiving renewed recognition as of late. Brown (1995), metaphorically described evaluation as being the heart that connects and gives blood to all other program elements. Despite realising its importance, the primary focus of language programme evaluation in its earlier days were concerned with making judgements based on experimental designs and limited quantitative analysis. (Beretta &Davies, 1995; Smith, 1970). As of late however,education practitioners have begun to recognize the benefits of the broader concept of evaluation as a means of informing programme development . In this regard, language programme evaluation have begun looking at the process of language programmes as a means of gaining important information concerning the programme. (Beretta, 1992; Kiely & Rea-Dickins, 2005; Lynch, 1996).

While there appears to be rather postive developments in the field of language programme education, there has been an urgent call for the maximization uses and usefulness of an evaluation particularly in language education programs. In view of this Norris (2008), has called for evaluators to engage proactively in the design and implementation that could maximally contribute to the understanding and improvement specifically for language programmes.

MODELS OF EVALUATION

Prior to the 1970‟s the main focus of educational programme evaluators were to determine as to whether a programme has met its stated objectives. One of the earliest advocates of this objective model is Tyler (1942). In the 1970‟s the need to look beyond the stated goals of a programme emerged with Scriven (1972) emphasising on other outcomes that could potentially be equally as important as the stated goals of the programme. This gave rise to the term formative and summative evaluations in which a programme can be assessed while it is still under its developmental stage or the outcomes of a completed programme.

One of the the major revolution in evaluation emerged from Stufflebeam (1973) who viewed evaluation as a process of providing meaningful and useful information for decision making. Stufflebeam proposed his 4 dimension context, input, process and product (CIPP) model which describes four kinds of evaluative methods. Context evaluation assesses the problems, needs and opportunities present in the educational programme setting. Input evaluation assesses complete strategies and the work plans and budget. Process evaluation monitors documents and assesses program activities while product evaluation examines the impact of the program, the quality and significance of


46

the outcomes and the extent to which the programme is sustainable and transferable. (Owston, 2000 ).

The two fundamental requirements that gave rise to Patton (1978)‟s Utilization Focused Evaluation were the need for decision makers and audiences to be clearly identified and evaluators must be actively involved in all aspects of the evaluation. The elements of the evaluation here includes the formation of the evaluation questions, the research design, the data analysis, the interpretation and the dissemination of the findings. Patton‟s approach towards making evaluation useful is echoed by Cronbach (1980), who stressed on the political context in decision making. Cronbach (1980) posits that decisions are likely to be made in a political setting by a political shaping community. Thus, in this educative process the evaluator is constantly giving feedback to the clients and the final evaluation reports is a menas of communication.

The naturalistic evaluation approach is yet another appraoch to evaluation that emerged in the midst of the growing body of evaluation models. One such model is the Stake (1975)‟s responsive model. Stake‟s models stemmed from the concern that conventional approaches were not sufficiently receptive towards the needs of the clients for information and present different value perspectives when reporting on the success and failure of a programme.

UTILIZATION FOCUSED EVALUATION

Utilization focused programme evaluation is evaluation done for and with specific intended primary users for specific intended uses (Patton, 1997). It is a comprehensive framework within which to develop and implement an evaluation with attention to use built in‟ (Patton, 1997, p: 20). Utilization focused evaluation starts with identifying primary intended users (PIU) of an evaluation, negotiating with them the purposes and intended uses of the evaluation and generating among the PIUs a commitment to these uses. In this manner, uses of the evaluation are clarified and built in from the very beginning, with evaluation users committed to participation throughout the process. Subsequent evaluation activities, including the design, data collection, analysis, and presentation of evaluation findings, all find their guidance from and strong association with the pre-specified intended uses.

Different from other evaluation approaches, the evaluators in UFE work collaboratively with the primary intended users and facilitate their decisions about the purposes, uses, focus, design, and implementation of the evaluation.The evaluators‟ role is more of a facilitator, negotiator, collaborator, coordinator,and consultant, as contrasted with received views of evaluators as decision-maker or judge. The primary intended users(PUIs) are the people who make major decisions like what the evaluation is about, how it shall proceed, and how it shall be used. In this vein, evaluators need to be „situationally responsive‟ (p: 134), to be able to recognize and respond to distinct program situations, and to conscientiously „act, react, and adapt‟ (p: 136) while working with PUI‟s throughout the evaluation process. The goals are to meet the information needs of a distinct program and its people, and to respond to changing, evolving, and dynamic program situations.


47

THE PRINCIPLES OF UTILIZATION FOCUSED EVALUATION There are fourteen general principles that underpin the utilization focused evaluation framework. In the context of this evaluation only ten of the fourteen will be highlighted and described. Principle 1 Commitment to intended use by intended users shpuld be the driving force in an evaluation. According to Patton (1997), at every decision point whether the decision concerns purpose, focus, design, methods, measurement, analysis or reporting, the evaluator must include the intended users. Principle 2 Strategizing about use is ongoing and continuous from the very beginning of the evaluation. Use is not something one becomes interested in at the end of the evaluation. Principle 3 The personal factor contributes significantly to use. The personal factor according to Patton (1997), refers to the research finding that the personal interests and commitments of those involved in an evaluation use. In line with this, Patton contends that evaluations should be specifically user-oriented aimed at the interests and information needs of specific and identifiable people not vague and passive audiences. Principle 5 In outlining the fifth principle in his utilization focused evaluation framework, Patton emphasizes on the need for evaluations to be focused. This is due to the fact that no evaluation can serve all potential stakeholders‟ interests equally, therefore it is of paramount importance that stakeholders representing various constituencies should come together to negotiate issues and questions that are of priority focus within the evaluation. Principle 6 Focusing on intended use requires making deliberate and thoughful choices is the sixth principle in Patton‟s utilization focused evaluation framework. There are three primary uses of evaluation findings which are judging merit or worth, improving programmes and generating knowledge. Similarly, there are four primary uses of evaluation proccesses which are enhancing shared understandings, reinforcing interventions, supporting participant engagement and developing programmes and organizations. Principle 8 Intended users‟ commitment to use can be nurtured and enhanced by actively involving them in a making significant decisions about the evaluation. Patton, (1987) postulated that involvement increases relevance, understanding and ownership of the evaluation all of which in turn facilitates informed and approriate use.


48

Principle 9 and 10 Utilization focused evaluation advocates high quality participation and not high quantity participation. High quality involvement of intended users will result in high quality and useful evaluations. With such high quality placed on the ue of the evaluation, validity and utility are interdependent. Patton, (1997), claim that threats to utility are just as important to counter as threats to validity. Principle 11 The eleventh principle outlined within the Utilization focused evaluation framework stipulates the need for evaluators to be active-reative –adaptive. Evaluators play an active and important role in presenting to intended users their own best judgements about appropriate evaluation focus and methods,they are reactive in listening attentively and respectful to others concerns and lastly they are adaptive in finding ways to design evaluations that incorporate diverse interests including their own all the while maintaining a high standards of professional practice. Principle 13 Use within the utilization focused evaluation framework is different from reporting and dissemination. Patton stresses that reporting and dissemination may be means to facilitate use but they should not be confused with such intended uses such as making decisions, improving programmes, changing thinking, empowering participants and generating knowledge. KEY STAGES IN UTILIZATION FOCUSED EVALUATION

UTILIZATION FOCUSED EVALUATION STUDIES The application of utilization focused evaluation in language education programmes has been scarce to date. Currently there appears to be two studies

Commitment to evaluation progress.

Identify intended users of evaluation

Engage primary users in decisions related to evaluation

(i) Evaluation Focus/Evaluation Questions

Identify methods and formulate evaluation plan

Involve uses in interpreting findings, generating recomendations

and disseminating results


49

that have adopted this evaluation framework. One such study was conducted by Norris (2004), who proposed a reconceptualization of assessment validation. Norris provided a framework for adopting a utilization focused evaluation approach to the validation process. The three year evaluation of the placement assessment in a college foreign language programme depicted the potentials, procedures and benefits of a utilization focused validity evaluation. According to Norris (2004), the most significant contribution of the utilization focused evaluation framework in his study is the ensurity that the evaluation process and findings were put to use to develop and revise the programme and more importantly the placement assessment was serving its intended users.

Yang Wei Wei‟s (2009), evaluation of teacher induction practices in a US university academic English language programme was another study that applied a utilization focused evaluation framework as a basis of her evaluation. Wei Wei (2009)‟s study not only produced useful findings but the findings were actually put to use in developing and improving the teacher induction programme as intended thereby promoting positive programme change.

EVALUATION DESIGN

A mixed method evaluation design incorporating both quantitative and qualitative methods were utilised in this evaluation to gather the pertinent data. In this evaluation the data was gathered using a perception questionnaire and the analysis of reflective journals.

PARTICIPANTS

The participants of this study were identified as the Primary Intended Users (PIUs) as stipulated in Patton‟s Utilization Focused Evaluation framework.The three main PIUs in the current evaluation were the Head of the Language Department, the TESL coordinator and the 24 pre service B.ED TESL teacher trainees‟. The sampling of this evaluation is based on purposive sampling. According to Frankel, Wallen and Hyun (2011) purposive sampling is selected „based on the prior information that will provide the data required‟. (Frankel, et al : 100).

Figure 2. The Primary Intended Users (PUIs)

HEAD OF DEPARTMENT

TESL COORDINATOR

B.ED TESL TEACHER TRAINEES


50

INSTRUMENTS

In this evaluation two instruments were utilised to gather the data namely a perception questionnaire and the pre service teacher trainees‟ reflective logs.

PERCEPTION QUESTIONNAIRE

Evaluation Question 1 : What are the perception of the pre service B.ED TESL teacher trainees‟ of the English Language Camp? Were the outcomes achieved? The perception questionnaire administered to the 24 pre service B.ED TESL pre service teacher trainees consists of 10 items aimed at examining their perception on the outcomes of the English Language Camp. The teacher trainees are required to respond to the items on a 4-point rating scale ranging from (1) strongly disagree to (4) strongly agree.

REFLECTIVE JOURNALS Evaluation Question 2 : What are the areas of strengths/specific areas that could be improved on? In order to identify the strengths and specific areas for improvement the pre service teacher trainees‟ refletive logs were collected and analysed. The reflective log is divided into 5 sections which include; Section 1 : Me (My experiences Prior to the Community Outreach Project experience), Section 2 : Nature/Account/Significant Points of the Community Outreach Project, Section 3 : Feelings (Positive and Negative), Section 4 : Learning (What have I learnt from this experience and where does it link with my current existing knowledge) and Section 5 : Conclusion (Areas that could be improved on/discontinued).The different sections in the reflective log seeks to delve into the very process of English Language camp very much based on what the trainees themselves think. The journals were then collected and analysed.

DATA COLLECTION PROCEDURES AND ANALYSIS The data collection procedures and analysis for the evaluation are as follows :

DATA PROCEDURES ANALYSIS

Perception Questionnaire

Administered to the 24 B.ED TESL pre service teacher trainees‟ after the English Language Camp

Quantitative analysis/ Responses will be reported in terms of percentages/frequencies

Reflective Journals B.ED TESL pre service teacher trainees‟ after the English Language Camp

Qualitative Content Analysis Coded/Themed for salient patterns


51

EVALUATION FINDINGS

The Perception Questionnaire The perception questionnaire consisted of ten items which intended to investigate the pre service teacher trainees‟ perception of the language camp. The items in the perception questionnaire were formulated after discussing with the primary intended users as to the priority purposes and intended uses of the evaluation. In what follows I will report on the frequency of responses in terms of percentages of each item. A total of twenty three pre service teacher trainees‟ completed the perception questionnaire. (n=23)

Item Statement SD D A SA

1 The Language Camp has given me the opportunity to enhance my creative skills in various language based activities

- 4.3% 56.5% 39.1%

2 The Language Camp has allowed me to use the English language in a friendly and non-threatening environment

- - 52.2% 47.8%

3 The Language Camp has helped me to develop my ability to work as a team member.

4.3% - 47.8% 47.8%

4 The Language Camp has helped me develop leadership qualities.

4.3% 4.3% 56.5% 34.8%

5 The Language Camp has helped me improve my communication skills in organising and carrying out activities

4.3% - 52.2% 43.5%

6 The 3 day duration for the language camp is suitable

4.3% 13.0% 47.8% 34.8%

7 The activities that were carried out during the language camp are fun, interesting and motivating.

4.3% - 39.1% 56.5%

8 The Language camp has increased my self confidence in planning and organising events in English.

4.3% - 56.5% 39.1%

9 The Language Camp is an important event.

4.3% - 47.8% 47.8%

10 Overall I am satisfied with the outcome of the Language camp.

4.3% - 65.2% 30.4%

Note : SD= strongly disagree, D= disagree, A= agree and SA= strongly agree. The perception questionnaire in general illustrated rather positive comments on almost all the 10 statements.Item 1 which states if the pre service teacher


52

trainees‟ were of the perception that the language camp has given them the opportunity to enhance creativity through the various activities, saw a total of 56.5% and 39.1% agreeing and strongly agreeing respectively. Only a small number, 4.3% disagreed with the statement. As for item 2, stating the view if the language camp has allowed the use of the language in a non threatening environment had a majority of the pre service teacher trainees‟ agreeing and strongly agreeing to it.

The two following items in the perception questionnaire intended to investigate if the language camp has helped developed team skills and leadership skills displayed almost similar results with 47.8 % agreeing and strongly agreeing that it does help them work as a team and 56.5% and 34.4% who agreed that it did develop leadership skills. 12.9 % of the teacher trainees‟ on the other hand were of the opinion that the language camp did not really develop team and leadership skills. In view of item 5 which states the perception that the language camp had improved communication skills had in sum a majority of the teacher trainees‟ agreeing and strongly agreeing to it. As for item 6 in the perception questionnaire however which concerns the three day duration of the language camp had 17.3% of the pre service teacher trainees‟ strongly disagreeing and disagreeing with the statement. Nevertheless a majority of them still agreed that a three day duration was suitable for a language camp.

Item 7 which was a rather important item in the perception questionnaire, stating if the activities that were planned for the language camp were fun, interesting and motivating had a huge majority agreeing (56.5%) and strongly agreeing (39.1%), while 4.3% strongly disagreed with the statement. The statement in item 8 of the perception questionnaire concerns the issue of confidence in organising and planning events in English once again depicted a majority agreeing and strongly agreeing to it. The final two items in the perception questionnaire which states if the pre service teacher trainees‟ were of the opinion that the language camp is an important event and if they were satisfied with the overall outcome of the language camp once again illustrated 47.8% agreeing and strongly agreeing that the language camp is an important event . As for the satisfaction with the overall outcome of the language camp had 95.6% agreeing and stronly agreeing that they were on the whole satisfied with it.

In general, it was rather heartening to note that the planned activities for the first organised language camp in IPG Kent was found to be satisfactory to the B.ED TESL teacher trainees‟ . The outcome of the perception questionnaire would be presented to the primary intended users of this evaluation which are the head of the language department and the TESL coordinator.

THE REFLECTIVE LOGS

At the end of the three day language camp, the pre service teacher trainees‟ were given a reflective log in which they were required to reflect on the experience as well as to provide some useful suggestions on improving specific areas of the programme. In line with that, my third source of data comprised of 23 reflective logs written by the 23 teacher trainees‟. This data would provide me with the developmental elements of the Language Camp as experienced by


53

the teacher trainees‟ pointing out the strengths and weaknesses of the project as well as highlighting specific areas for improvement. The findings will be based on the 5 sections of the reflective logs in which the most salient responses will be highlighted and discussed.

Section 1 :Me (My experience Prior to the Language Camp experience) Some responses : “I was hoping the language camp would be something that was organised outside the campus” (RJ 1) “A camp where a group of students gather together to learn a language in a fun way” “A camp that will help me with my English” (RJ 16) “I want to learn how to communicate better in English” (RJ 7) “I thought the language camp is all about the English Language, no games, no physical activities” (RJ14) “The camp should realised us that speaking English all the time is really needed and important” (RJ 18) “We will do a lot of activities in English, activities like treasure hunt...more English activities” (RJ21)

Note : RJ = Reflective Journal As illustrated in some of the reflective journal in regards to their expectations of what a language camp should incorporate, most of them had the view that language camp should use the English language through various activities and that communication is of utmost importance.

Section 2 : Facts (Nature, Account, Significant Points) Some responses : “Lots of activities which help improve our English Language...improve our collaboration skills among our fellow classmates..” (RJ 2) “Many activities like the treasure hunt...to answer questions related to grammar and English literature..” (RJ 15) “Full of activities/Fun!...enhances discussions, decisions and teamwork” (RJ 17) “Various interesting activities...Choral speaking covered all the aims like enhance our confidence level, sharpen our communication skills and leadership skills..” (RJ 5) “build up our confidence level. cooperation among the members..communication skills” “exposed to English and how to use it in daily life”.. (RJ 18) „improving our skills in the language...improving communication” (RJ 22) “meetings and discussions among members are very important... (RJ 10) “fun while learning outside the classroom...especially the treasure hunt..” (RJ 7)

Note : RJ = Reflective Journal Most of the responses in regards to the second section of the reflective log in which the teacher trainees‟ were asked to write about their thought and feelings


54

about the way the language camp was conducted as well as significant points of the camp were very positive. Many felt that the camp had achieved its purpose in enhancing their confidence in using the English language and communicating in the language in fun ways. Some were also of the opinion that the language camp allowed them to work as a team and build up leadership skills.

Section 3 :Feeling (Positive and Negative) Some responses : Positive “most of the activities...like the treasure hunt and language games” (RJ 3) “the treasure hunt activity as it really helped the participants to enhance and develop the communication skills, critical thinking skills, leadership skills among team members” “the activities like the treasure hunt, choral speaking, skits competition...fun” (RJ 12) the activities planned went very well...” (RJ 11) “the organisation and planning of most of the activities went well” (RJ 23) “the cooperation of the team members was a positive aspect” (RJ 9) “the team work and everyone participated and gave their full support” (RJ 5) Some responses : Negative “the organisation of some of the events were not well planned like the opening and closing ceremony” (RJ 16) “ team work in certain events were lacking...some were not doing their designated jobs” “time management and lack of responsibility of certain teams”.. (RJ 4) “the organisation of certain activities..but we are still learning” (RJ 18) “time management and cooperation was still not so well” (RJ 20) “self discipline, time management and cooperation among team members still not strong” “Time management, planning” (RJ 8)

Note : RJ = Reflective Journal The third section of the reflective journals probed into the thoughts and feelings of the pre service This is perhaps the most valuable section of the reflective log as data retrieved from here would shed some light into what the teacher trainees‟ had thought about the positive and negative elements of the Language camp as a whole This section was divided into positive and negative feelings. Perhaps the most prevalent theme that recurred in most of the journals in regards to the positive experiences were the activities that were carried out. In most of their journals, the teacher trainees‟ felt that all the activities carried out for the three days were appropriate and of benefit to them in terms of using the English language especially to enhance the communication skills and confidence level. The activity that most of them felt were both enjoyable and beneficial was the treasure hunt. On a more negative note, the phrases lack of cooperation, time management and organisation were the three phrases that kept recurring in most of the teacher trainees‟ reflective journals.


55

Section 5 :Conclusion (Areas that could be improved on/Strategies that should be discontinued) Some Responses : “Participation and cooperation of the class as a whole can be further improved” “involve our seniors in activities” (RJ 11) “all the activities were done well...more language games to enhance critical thinking should be added (RJ 7) “the schedule and planning could be improved..work with our seniors and juniors” (RJ 5) “the management of certain events could be improved...we should work with our TESL seniors” (RJ 17) “time management could be improved..but we are still learning...more language based activities” (RJ 23) “team work between the lecturers and teachers could be improved” (RJ 5) “time management of the way the activities were run could be improved..we should collaborate with the senior TESL for some language activities”. (RJ 19) “time management, planning..we should incorporate talent shows as part of the activities and perhaps a TESL night with all other TESL classes” (RJ 3) “inviting speakers of motivation from outside...it would be great to have a motivational slot” (RJ 16) “more trips and outings” (RJ 11) “involve our seniors in the language camp...have more performances..” (RJ 22) “organization and more language activities could be improved...we should involve our seniors in the language camp” (RJ 10)

Note : RJ = Reflective Journal Finally, Section 5 of the reflective logs expected the trainee‟s to reflect on areas of the Language camp that could be improved on and strategies which they felt should be discontinued. Responses from this section of the reflective log would be extremely helpful in identifying specific areas for improvement. Perhaps the one suggestion that was prevalent in most of the reflective logs was the idea of a collaboration of sort with the TESL seniors.In the foundation syllabus it is clearly stated that the Language Camp be held in the second semester of the three semester foundation programme specifically for the pre degree TESL teacher trainees‟. Neverthless the notion of working together with the seniors in conducting the language camp is indeed an interesting avenue to be explored. This suggestion was reported to the primary intended users of this evaluation and the feedback from them was encouraging. The other aspects that the pre service teacher trainees‟ felt that could be improved on include better time management, better cooperation among team members, and better management of certain events.

The findings from perception questionnaire and the reflective logs revealed some interesting insights into the teacher trainees‟ perception and experiences respectively during the three day language camp. The finding were then reported to the two primary intended users of the evaluation and the interview questions were based on the findings. The interview with the two


56

primary intended users of the evaluation was fruitful in that seven objectives for the next Language camp was outlined to be further discussed. The next chapter of the evaluation report would discuss the findings of the evaluation in view of the three evaluation questions as well as a brief discussion of the uses of the present evaluation.

DISCUSSION, IMPLICATION AND CONCLUSION

This section provides a discussion of the reported findings of the evaluation in regards to the three evaluation questions that had framed the evaluation. A brief discussion of the uses of the evaluation will also be included. The chapter would end with the evaluation implications and conclusion. Evaluation Question 1: What are the perception of the pre service B.ED TESL teacher trainees of the English Language Camp? Were the outcomes achieved? It is rather evident based on the perception questionnaire that the majority of the pre service B.ED TESL teacher trainees‟ were of the opinion that the language camp was of a great benefit to them in terms of improving their communication skills, their ability to work as a team and increasing their confidence level. A majority of them also agreed that the activities that were carried out during the three day language camp were fun, interesting and motivating. A majority of the teacher trainees‟ on the whole appeared to be satisfied with the overall outcome of the language camp that was organised by the language department . Therefore it can be safely concluded that the outcomes of the 2012 English language camp were achieved based on the perception questionnaire that was administered to the twenty three pre service teacher trainees‟ at the end of the three day language camp. Evaluation Question 2 : What are the areas of strengths/specific areas that could be improved on? The reflective logs which were also administered to the twenty three teacher trainees‟ at the end of the language camp shed some valuable insights into understanding some of the strengths and areas that could be further improved for the next language camps. Some of the suggestions like having more language based games, getting the seniors more involved in language camps as well as improving more on time management, and planning and organisation of events were certainly taken into due consideration when the outcomes was presented to the two primary intended users of this evaluation. This is the very first language camp organised at the institution level and all the suggestions on improvement were indeed extremely useful in the planning and organising of the next language camps.

THE USES OF THE EVALUATION FINDINGS AND PROCESS

The present evaluation applied certain principles as outlined in Patton‟s utilization focused evaluation framework as a basis of evaluating this


57

programme. Utilization focused evaluation stresses on the need for the evaluation be of use to its primary intended users. The two main uses of this evaluation were (1) were the outcomes of the 2012 language camp achieved? (2) what are the specific areas of strength and specific areas to improve on. In being able to present the findings from the perception questionnaire and the reflective logs of the teacher trainees‟ to the two primary intended users of this evaluation, an outline of the specific outcomes for the next language camp could be listed down.

In addition to the anticipated uses of the evaluation, the process of the evaluation in which the evaluator and the primary intended users were constantly working together in determining the uses of the evaluation. Through this evaluation, the head of the language department and the TESL coordinator were able to develop a better understanding of the evaluation process and the benefits of it in improving and developing the programme.

By using the findings from the teacher trainees‟ through the perception questionnaire and the reflective logs, the main decision makers of the language camp programme were able to specify an outline for the intended outcomes in the forthcoming language camp. Thus by specifying the outline the primary intended users were able to consider seriously the main objectives and goals that they wish the programme would achieve in its next phase. Moreover this evaluation also empowered the pre service teacher trainees‟ in that their voices and thoughts were actually heard, valued and taken into consideration in specifying the intended outcomes for the next language camp.

IMPLICATIONS AND CONCLUSION

The report of this evaluation study illustrated the uses of evaluation through the adoption of a utilization focused approach (Patton, 1997) in a language programme setting. It is evident in the present evaluation study that in utilising this approach, not only did the evaluation produce useful findings, but the findings were actually put into use in determining the outcomes as well as developing and improving the programme. The uses were made possible by working closely with the primary intended users throughout the evaluation process.

The main contention in using the utilization focused approach in evaluating this particular language programme was based on the priority of use and utility as well as a working together of all the stakeholders towards the betterment of the programme. Although this evaluation was done on a much smaller scale in using the utilization focused approach nevertheless it has certainly reflected the use and users‟ decisions in the evaluation effort.

BIBLIOGRAPHY

Beretta, A & Davies, A (1985). Evaluation of the Banglore Project. ELT Journal

39(2), 121-127. Beretta, A (1992). Evaluation of language education. An overview. In J.C.

Alderson & A. Beretta (Eds). Evaluating second language education (pp: 15-24). Cambridge: Cambridge University Press.


58

Carr,W & Kemmis, S (1983). Becoming Critical : Knowing through action research. Deakin University.

Cronbach, L.J (1980). Towards reform of programme evaluation. San Francisco

CA: Jossey Bass Frankel,J .R, Wallen, N.E and Hyun, H.H (2012). How to Design and Evaluate

Research in Education. McGraw-Hill International Edition. Kiely, R & Rea-Dickins, P (2005). Program evaluation in language education.

New York : Palgrave Macmillan Lynch, B (1996). Language Programme Evaluation. Cambridge. Cambridge

Univeristy Press Norris, J.M (2008).Validity evaluation in language assessment. New York: Peter

Lang. Ownston, R.D (2000). Evaluating web-based learning environments: strategies

and insights. Cyber Psychol Behaviour., 3(1) (pp: 79-87) Patton, M.Q (1982). Practical Evaluation. Thousand Oaks. Sage Patton, M.Q (1997). Utilization Focused Evaluation: The New century Text (3

rd

Ed). London :Sage Publications Patton, M.Q (1978). Utilization Focused Evaluation. Beverly Hills, CA : Sage

Publications Tyler, R.W (1942). General statements on evaluation. J. Education. Res., 35

(pp: 492-501) Scriven, M (1972). Pros and cons about goal free evaluation. Evaluation

Comm., 3(4), (pp : 1-7). Stufflebeam, D.C (1973). An Introduction to the PDK books: Educational

evaluation and decision making. In Educational Evaluation: Theory and Practice, edited by B.L Worthen and J.R. Sanders (pp: 128-142), Belmont, CA: Wadsworth

Weir,C. & J. Robert. (1994). Program Evaluation in ELT. Oxford: Blackwell

Publishers.

EVALUATION OF THE LANGUAGE CAMP 2012: TOWARDS...

Documents

Transcript of EVALUATION OF THE LANGUAGE CAMP 2012: TOWARDS...