Www.educationscotland.gov.uk Scottish Survey of Literacy and Numeracy Improving Numeracy.
Report of the Program Evaluation of QuickSmart Numeracy · 2019. 10. 23. · March 2012. QuickSmart...
Transcript of Report of the Program Evaluation of QuickSmart Numeracy · 2019. 10. 23. · March 2012. QuickSmart...
-
QuickSmart Numeracy
Report of the Program Evaluation of
March 2012
Student Engagement and Program Evaluation Bureau
-
AcknowledgementsThe Student Engagement and Program Evaluation Bureau would like to thank the following people for their time and expertise in assisting with the evaluation of QuickSmart Numeracy:• The Program Evaluation Reference Group (PERG) members:
- Dr Paul Chesterton, Board member, Australasian Evaluation Society- Bronwyn Hession, Senior Education Officer, Catholic Schools Office, Diocese of Broken Bay- Dr Kate O’Brien, Assistant Director, Teaching and Learning, Catholic Education Commission, NSW- Dr Max Smith, Senior Manager, Student Engagement and Program Evaluation (SEPEB) - Chair- Mikael Smith, NSW Aboriginal Education Consultative Group Inc. (AECG)
• Frank Pearce, State Coordinator Aboriginal Education, Catholic Education Commission NSW• Gloria Provest, Leader, School Programs, Aboriginal Education and Training Directorate, DEC• Paul Rodney, Assistant Director, Catholic Education Commission, NSW • Raylene Saunders, R/PEO, Schools, Aboriginal Education and Training Directorate, DEC• Kate O’Donnell, R/Director, Educational Measurement and School Accountability Directorate (EMSAD), DEC• John Pegg, Director, National Centre of Science, ICT, and Mathematics Education for Rural and Regional Australia
(SiMERR), University of New England• Lorraine Graham, University of New England
The evaluation team wishes to thank the principals and staff of the following schools for sharing their experiences of implementing QuickSmart Numeracy:
• Telarah Public School• Barraba Central School• Glen Innes Public School• Oxley Vale Public School• Walcha Central School • Aldaville Public School• Lismore South Public School
• Port Macquarie Public School• Tweed Heads Public School• Wauchope Public School• Wingham Public School• Oxley Park Public School• St Francis Xavier’s Primary School, Narrabri
Additional thanks to the students at Rosemeadow Public School and Mayfield East Public School for their original graphics contributed to the evaluation report.
Evaluation teamMeg Dione-Rodgers Senior Evaluation Officer Dr Susan Harriman Manager, Evaluation PracticeBarry Laing Evaluation Consultant
Program Evaluation UnitStudent Engagement and Program Evaluation Bureau (SEPEB)
© State of New South WalesMarch 2012
-
IIIQuickSmart Numeracy - Evaluation final report
Contents
I
ContentsExecutive summary ..............................................................................................................................................1
Part 1 Introduction ...............................................................................................................................................5
1 Evaluating QuickSmart Numeracy ...................................................................................................................5
1.1 Policy context .........................................................................................................................................5
1.2 QuickSmart ............................................................................................................................................6
1.3 Evaluating the literacy and numeracy programs ........................................................................................7
1.4 Terms of Reference for the evaluation ......................................................................................................7
1.5 Methodology ..........................................................................................................................................8
1.6 Presentation of the evaluation findings ...................................................................................................13
2 QuickSmart program details .........................................................................................................................14
2.1 Aim .....................................................................................................................................................14
2.2 Background .........................................................................................................................................14
2.3 Features of QuickSmart ........................................................................................................................15
2.4 Training and support .............................................................................................................................15
2.5 Resources ...........................................................................................................................................16
2.6 Program support...................................................................................................................................18
2.7 Previous evaluations .............................................................................................................................18
Part 2 Evaluation findings ...................................................................................................................................19
3 Implementation of QuickSmart ......................................................................................................................19
3.1 Program choice ....................................................................................................................................19
3.2 Selection of students ............................................................................................................................20
3.3 Staffing .................................................................................................................................................22
3.4 Professional learning .............................................................................................................................22
3.5 Space and timetabling ..........................................................................................................................24
3.6 Costs of implementing QuickSmart .......................................................................................................25
4 Effects for students ......................................................................................................................................27
4.1 External performance measures ............................................................................................................27
4.2 Local assessments by staff, students and parent community ..................................................................31
4.3 Effects of withdrawing students from class .............................................................................................39
5 Teacher and tutor perspectives ....................................................................................................................41
5.1 Teachers’ and tutors’ overall views of QuickSmart...................................................................................41
5.2 Changed practices ...............................................................................................................................42
5.3 Support for implementation ...................................................................................................................43
5.4 Assessment .........................................................................................................................................45
-
II
6 Impacts in schools .......................................................................................................................................47
6.1 Limited impact on the whole school .......................................................................................................47
6.2 Continuing with QuickSmart practices ...................................................................................................47
6.3 Parent and community involvement .......................................................................................................49
6.4 Sustainability ........................................................................................................................................50
7 Summary of findings and conclusions...........................................................................................................53
7.1 Assessment of the effectiveness of QuickSmart ....................................................................................53
7.2 Assessment of efficient achievement of goals .......................................................................................55
7.3 Addressing the mandatory reform elements of the NPLN ........................................................................56
7.4 Improving educational outcomes of Aboriginal students ..........................................................................57
7.5 Supporting schools to participate in the evaluation ................................................................................57
7.6 Support for reforms to be incorporated into school practice ....................................................................57
7.7 Sustainability ........................................................................................................................................58
Glossary .............................................................................................................................................................59
References ....................................................................................................................................................................................................................................................60
Appendix 1 - Evaluation instruments ...................................................................................................................61
Appendix 2 - Online teacher and tutor survey responses ....................................................................................67
-
VQuickSmart Numeracy - Evaluation final report III
List of tables and figuresTables
Table 1.1: Summary of participation in school interviews ......................................................................................... 9
Table 1.2: Student performance assessment data sets ....................................................................................... 10
Table 1.3: Aggregate data sets ........................................................................................................................... 11
Table 4.1: NAPLAN Numeracy, 2008 Year 3 QuickSmart cohort ........................................................................... 28
Table 4.2: NAPLAN Numeracy, 2009 Year 3 QuickSmart cohort ........................................................................... 28
Table 4.3: NAPLAN Numeracy, 2010 Year 5 QuickSmart cohort ........................................................................... 29
Table 4.4: NAPLAN Numeracy, 2011 Year 5 QuickSmart cohort ........................................................................... 29
Table 4.5: NAPLAN numeracy for QuickSmart comparison to National Minimum Standards ................................... 30
Table 4.6: NAPLAN gain scores and differences in effect size from Year 3 to Year 5 .............................................. 30
Figures
Figure 1.1: National Partnership on Literacy and Numeracy implementation model ................................................... 6
Figure 3.1: Implementation in Year Levels 2009-2011 .......................................................................................... 20
Figure 3.2: Usefulness of professional learning .................................................................................................... 23
Figure 4.1: Improved numeracy outcomes ........................................................................................................... 31
Figure 4.2: Improved numeracy outcomes for Aboriginal students ......................................................................... 35
Figure 4.3: Increased engagement with numeracy ............................................................................................... 37
Figure 5.1: Teacher and tutor understanding of the purpose of QuickSmart ........................................................... 41
Figure 5.2: Teacher and tutor understanding of the principles of QuickSmart ......................................................... 41
Figure 5.3: Improved teaching practice ................................................................................................................ 42
Figure 5.4: Role of QuickSmart coordinator .......................................................................................................... 43
Figure 5.5: Aboriginal education officer participation ............................................................................................. 44
Figure 5.6: Use of the QuickSmart resources ....................................................................................................... 45
Figure 5.7: QuickSmart assessments identify students’ numeracy needs .............................................................. 45
Figure 6.1: Increased parental involvement .......................................................................................................... 49
Contents
-
VI
-
1QuickSmart Numeracy - Evaluation final report 1
Executive Summ
ary
Executive summaryProgram evaluation: QuickSmart Numeracy NSW Department of Education and Communities1 commissioned the Student Engagement and Program Evaluation
Bureau to conduct an evaluation of QuickSmart Numeracy, one of several initiatives being funded under the National
Partnership on Literacy and Numeracy (NPLN).
The Terms of Reference for the program evaluation of QuickSmart Numeracy required that the evaluation:
• assess the effectiveness of the program
• assess the extent to which the program achieves its goals in an efficient manner and where applicable, addresses
the mandatory reform elements of the National Partnership on Literacy and Numeracy, which are:
– effective and evidence-based teaching of literacy and numeracy
– strong school leadership and whole-school engagement with literacy and numeracy
– monitoring student and school literacy and numeracy performance to identify where support is needed
• assess the extent to which the program has improved the educational outcomes of Aboriginal students
• investigate the most effective ways for schools to be supported to participate in the evaluation and for the reforms
to be incorporated into school practice.
Fourteen NPLN schools have chosen to implement QuickSmart Numeracy as their intervention targeting individual
students who are experiencing difficulty with mathematics. Many of these schools have a high proportion of Aboriginal
students.
QuickSmart NumeracyThe QuickSmart suite of programs has been developed by the University of New England to address both literacy
and numeracy needs of middle years students (Years 5 to 8). Only the numeracy component was offered as part of
the NPLN.
For the purposes of this report, QuickSmart will be understood to refer to the numeracy program only.
QuickSmart is a basic skills intervention designed for students who demonstrate persistently low achievement. The
program aims to improve students’ ability to easily and quickly recall number facts and perform basic computation
skills, referred to as ‘automaticity’.
QuickSmart is a research-based program that is supported, resourced and built around a professional learning
program for principals, supervising teachers, teachers and teacher aides. The program emphasises both deliberate
practice and strategy instruction in basic mathematics.
In the QuickSmart program, pairs of students participate in 30-minute lessons, three times a week for 30 weeks.
National test data provide a compelling case for the need to implement research-based programs that improve the
numeracy outcomes for students who are performing at or below the National Benchmarks.
1 On 3 April 2011, the NSW Government changed the name of the Department of Education and Training to the Department of Education and Communities. In this document, a reference to the Department of Education and Training is to be construed as a reference to the Department of Education and Communities.
-
2
The QuickSmart program fills some of the identified gaps in research and practice. The QuickSmart program was
designed as a long-term program for middle school students performing in the lowest 30% of the achievement range,
who need to improve their basic mathematics skills. The program targets those students who have been unable to
draw benefits from other in-class and withdrawal instructional activities (NSW, 2009a).
Evaluation methodologyThe evaluation methodology comprised the following components:
• knowledge review: a review of program documentation relating to QuickSmart
• stakeholder interviews with program developers
• online survey of teachers in all 14 schools that implemented QuickSmart as their individual-student intervention
• visits to four of the 14 schools, to conduct interviews and focus groups with school principals, leaders, teachers,
students, parents and community members
• analysis of school documentation including school Numeracy Plans and Annual School Reports
• analysis of National Assessment Program - Literacy and Numeracy (NAPLAN) and National Partnership on
Literacy and Numeracy (NPLN) assessment data.
The methodology was developed in consultation with, and approved, by the NPLN Program Evaluation Reference
Group (PERG).
Key findingsEvidence from all data sources indicates that student learning outcomes in numeracy have improved for students
engaged in the QuickSmart intervention. This effect is more strongly supported by teachers’, parents’ and students’
impressions of effective learning, than by broad-scale testing measures.
Other major findings relate to the implementation of the program, impacts on teacher work practices and on the
school as a whole, and issues of sustainability. These are detailed in the following sections.
Implementation of QuickSmart Numeracy
Ease of implementation, resources and assessment software were instrumental in schools’ decisions to implement
QuickSmart as their individual-student intervention.
Program choice and selection of students
Schools targeting numeracy as their NPLN programs had only two individual-student interventions to choose
from: Taking Off With Numeracy (TOWN) and QuickSmart Numeracy. Several schools were already implementing
QuickSmart Numeracy at the commencement of NPLN, so chose to continue and expand the program.
In each school, places in the QuickSmart program were limited. Selection of participants was usually based on
perceived need, determined by NAPLAN or other assessment data. Concerns were raised in several schools about
the appropriateness of this method, suggesting that discussions with teachers as to who might benefit most, should
also be used to inform the decisions.
Staff and training
Finding suitable tutors was regarded as the critical influence on the success of the program. Some schools chose to
re-direct existing school support staff to act as QuickSmart tutors; others employed tutors, often parents from within
the school community.
-
3QuickSmart Numeracy - Evaluation final report 3
Teachers and tutors participated in training provided by the QuickSmart SiMERR group, which included three, two-
day sessions for teachers and support staff. A one-day workshop was also provided for principals. This training was
seen to be effective, although expensive.
Some support was provided to schools by the National Centre of Science, Information and Communication
Technology and Mathematics Education for Rural and Regional Australia (SiMERR) at the University of New England,
as well as by regional and diocesan consultants. The level of support available was determined by each sector on a
regional or diocesan basis.
The QuickSmart Cognitive Aptitude Assessment System (CAAS) was seen by students as engaging, as well as
providing useful feedback on student progress to both staff and students.
Managing QuickSmart: learning spaces, computers and timetables
The role of the QuickSmart coordinator was critical in solving challenges around access to suitable rooms and
computers, and in creating timetables that provided continuity for students, and balanced students’ time away from
classes with the number of sessions each tutor could do in a day or week.
Effects for students
Both data sets (NAPLAN and NPLN assessments) demonstrated improvement in numeracy performance for students
in QuickSmart schools. A range of limitations on the reliability and validity of results observed in these data sets have
been outlined in this report; these should be considered when drawing conclusions from the results discussed.
As the majority of students participating in QuickSmart, are in Years 5 or 6, the influence of QuickSmart on student
performance is highly unlikely to be evident in these results.
Principals, school leaders, teachers, students and parents all identified positive impacts of QuickSmart on student
learning outcomes in numeracy. Improvements were also reported in other aspects of learning, such as: enjoyment of
and engagement in numeracy activities; greater willingness to participate in class activities; improved social skills and
behaviour; and better attitudes to homework.
Outcomes for Aboriginal students
Both data sets (NAPLAN and NPLN assessments) also demonstrated improvement for Aboriginal students in
QuickSmart schools.
Most teachers and tutors reported improved numeracy outcomes for all Aboriginal students, as a result of their
participation in QuickSmart. Other academic and social outcomes were also observed, including improved
attendance, increased participation in class activities, as well as increased self-esteem, confidence and improved
behaviour.
Teacher and tutor perspectives
Most staff agreed that the program was effective in meeting the needs of most or all of the targeted students.
Teachers reported some difficulties with the withdrawal program; mostly in terms of the disruption to lessons and the
difficulty some students face in rejoining class lessons.
The QuickSmart professional learning program was judged to be effective. However, only one or two schools trained
all their teachers. Others schools gave priority to program coordinators and tutors or support staff who would work
directly with students. Class teachers in these schools reported that they had, at least, become familiar with the
QuickSmart strategies, and saw value in their use with all students.
Executive Summ
ary
-
4
Impacts on whole-school practices
The introduction of QuickSmart appears to have had minimal impact on overall school culture. Nevertheless, some
key impacts identified by principals included: strengthening of relationships between the school and community;
improved attitudes to mathematics among students; and teachers reflecting on the potential benefit of using the
QuickSmart strategies with students in class numeracy lessons.
Factors leading to success
Several factors were identified as contributing to the program’s overall success. These included:
• selecting and retaining the right tutors
• building strong relationships: between the school and the tutors; between teachers and tutors; and between
tutors and their students
• regular monitoring of student progress using school-based assessments, QuickSmart CAAS assessments and,
to a lesser extent SMART data
• quality training, promoting teacher and tutor confidence in implementing QuickSmart
• establishing a QuickSmart coordinator to provide dedicated program support to tutors, students, teachers, and
parents.
Sustainability
While QuickSmart is labour-ntensive and complex to organise, its results are impressive. This, and the potential to
expand practices beyond the student-withdrawal program, made it an attractive proposition for the NPLN schools to
maintain beyond the period of funding.
Four factors are identified as being critical to the sustainability of QuickSmart in schools:
• securing a funding source for ongoing employment of tutors
• supporting the QuickSmart coordinator with relief time, to complete the complex management tasks
• providing additional resource kits
• ensuring a strategy for ongoing training for new teachers or tutors.
The main obstacle to continued implementation of QuickSmart was the substantial funding required to employ tutors.
Three distinct possibilities emerged:
• obtaining additional funding from a new external source, or cancelling the program
• identifying a funding source within the existing school budget
• incorporating the tutoring role within the workload of support teachers, or relying on volunteers.
Each of the principals in the schools visited, confirmed that it’s likely that they would ‘find’ alternative funds to maintain
the individual-student intervention.
-
5QuickSmart Numeracy - Evaluation final report
Evaluating QuickSm
art Numeracy
5
Part 1 Introduction This evaluation report is presented in two parts:
Part 1 provides the background to the QuickSmart Numeracy program evaluation and to the QuickSmart
program itself.
Part 2 details the findings and conclusions of the evaluation.
1 Evaluating QuickSmart NumeracyQuickSmart Numeracy is one of four literacy and numeracy programs evaluated by the Department’s Student
Engagement and Program Evaluation Bureau, under the National Partnership on Literacy and Numeracy (NPLN).
1.1 Policy context
The NPLN aims to bring about sustainable improvements in the reading and numeracy achievement of students in
NSW schools. ‘Closing the Gap’ for Aboriginal students and improving outcomes for individual students experiencing
difficulty in numeracy, are specific program targets.
The partnership is operating over the four-year period from 2009 to 2012, to facilitate and reward literacy and
numeracy models or approaches that support teachers and demonstrate evidence of accelerating improvement in
student learning achievement.
The focus is on development of:
• effective, evidence-based teaching
• strong leadership and whole-school engagement in literacy and numeracy
• effective use of student performance information to identify where support is needed.
Schools in all three education sectors in NSW (government, independent and Catholic) were identified for
participation, using a range of criteria including:
• the 2008 National Assessment Program - Literacy and Numeracy (NAPLAN) data: schools where the percentage
of all students in Years 3 and 5 at or below minimum standard is above the state percentage in reading and
numeracy
• the school’s suitability and readiness to participate in the NPLN, as advised by regional and diocesan offices
• each school’s student background characteristics, including: enrolment; student language background;
student enrolment data, for instance, proportion of refugee students or Aboriginal students; and the degree of
disadvantage of the school or group of schools.
Participating schools were allocated funding, tied to the reform priorities, to review and refine their school
improvement plan. Schools were expected to use their funding to:
• undertake a self-evaluation on reading or numeracy
• involve the school leadership team in a leadership capacity-building program
• implement an intervention program targeting individual students in Years 3 to 6 who are experiencing difficulty in
reading or numeracy
• implement an intervention program at a whole-class level in Years 3 to 6
• focus on improving teaching and learning in Years 3 to 6, by having teachers participate in a sustained
professional learning program on reading or numeracy.
-
6
Each school was required to complete professional learning in the use of SMART data and school leadership
capacity building as part of the NPLN. Teachers were expected to complete the online Data Analysis Skills
Assessment (DASA) to gauge their own development of skills in analysing data and using results to inform practice.
Figure 1.1: National Partnership on Literacy and Numeracy implementation model
(NSW Government, 2009a)
Within the context of the school improvement plan the funding could also be used to:
• support staff to participate in a local Aboriginal cultural awareness program
• provide release for collaborative programming, resource development, shared reflection and team teaching
• work with school QuickSmart tutors or teaching and learning leaders
• purchase equipment or resources essential to the implementation of the professional learning program, or
• employ teacher aides where they are essential to the implementation of an intervention.
A total of 147 schools in NSW participated in a range of programs offered under the NPLN. Some of these programs
have been developed internally by NSW Department of Education and Communities (DEC), others by the Catholic
education sector, and others by external developers. Each program targets literacy or numeracy, taking either an
individual-student or a whole-class approach, and has been included on the basis of a sound evidence base that
supports its potential to achieve the intended improvements.
Numeracy programs that take a balanced approach to teaching numeracy were selected. Programs were chosen
that are designed to accelerate student outcomes in number and multi-unit place value (NSW Government, 2009c).
1.2 QuickSmart
The QuickSmart suite of programs has been developed by the University of New England to address both literacy and
numeracy needs of middle years (Years 5-8) students. Only the numeracy component was offered as part of the NPLN.
-
7QuickSmart Numeracy - Evaluation final report
Evaluating QuickSm
art Numeracy
7
For the purposes of this report, QuickSmart will be understood to refer to the numeracy program only. If there is a
need to distinguish between the two programs, both QuickSmart Literacy and QuickSmart Numeracy will be referred
to in their full forms.
QuickSmart was nominated within the NPLN Agreement as one of the programs to be evaluated by the Program
Evaluation Unit of the Department’s Student Engagement and Program Evaluation Bureau (SEPEB).
Thirteen public schools, across four regions, implemented the QuickSmart program through participation in the NPLN.
Of these, three schools participated in the qualitative field visit component of the evaluation.
One Catholic Education Commission school chose QuickSmart as their individual-student intervention program for
numeracy, and was included in qualitative field work.
A description of the QuickSmart program is provided in Section 2.
1.3 Evaluating the literacy and numeracy programs
The NPLN Cross-sectoral Working Group determined that eight of the available programs would be formally evaluated. It
was agreed that only programs with seven or more participating schools would be involved in the evaluation.
The following criteria were used to determine whether each program evaluation would be conducted within the DEC
or by an external evaluator:
• The four literacy and numeracy programs developed by organisations outside education systems were evaluated
by SEPEB.
• Literacy and numeracy programs developed within education systems (three programs) were evaluated by an
external evaluator, Urbis Pty Ltd.
• MultiLit was evaluated by Urbis Pty Ltd.
Independent sector schools were not involved in the NPLN program evaluations.
The purpose of the program evaluations is to assess the effectiveness of the selected literacy and numeracy
programs, as defined in the Terms of Reference of the evaluation, below.
1.4 Terms of Reference for the evaluation
The Terms of Reference for the program evaluation of QuickSmart required that the evaluation:
• assess the effectiveness of the program
– assess the extent to which the program achieves its goals in an efficient manner and where applicable,
addresses the mandatory reform elements of the NPLN, which are:
– effective and evidence-based teaching of literacy and numeracy
– strong school leadership and whole-school engagement with literacy and numeracy
– monitoring student and school literacy and numeracy performance to identify where support is needed
• assess the extent to which the program has improved the educational outcomes of Aboriginal students
• investigate the most effective ways for schools to be supported to participate in the evaluation and for the reforms
to be incorporated into school practice.
-
8
1.5 Methodology
The QuickSmart program evaluation employed a mixed-method design, drawing on both quantitative and qualitative
components, as described in the following sections.
The evaluation was undertaken during the second full year of implementation of the QuickSmart program. At this early
stage in the program, the most credible data comes from the experiences of participants and results of in-school
assessment of student achievement. In addition to a survey of teachers, qualitative methods have been employed as
they allow greater depth of enquiry, placing value on the accounts of teachers, school executive members, students
and parent representatives.
The NPLN Program Evaluation Reference Group (PERG) was established to provide advice and guidance to the
evaluation team. The evaluation methodology was approved by the PERG and each of the interview and observation
schedules was developed in consultation with the expert members of the PERG.
1.5.1 Qualitative component
The qualitative component of the evaluation comprised:
• a review of program documentation for QuickSmart
• an interview with Professor John Pegg and Dr Lorraine Graham, program developers, from the National Centre
of Science, Information and Communication Technology and Mathematics Education for Rural and Regional
Australia (SiMERR), University of New England
• visits to four of the 14 schools implementing the program.
Background document review
Prior to the commencement of the evaluation, Program Evaluation team members undertook a thorough review
of program materials and associated literature, to better inform their understanding of the program itself, and their
interpretation of the evaluation data, especially that drawn from conversations with school staff and students.
Interview with program developers
Evaluation team members met with Professor John Pegg and Dr Lorraine Graham (SiMERR) at the commencement
of the evaluation. They provided insights into the philosophy underpinning the development of both the QuickSmart
program and the professional learning model. They also provided recommendations on the types of questions to
direct to teachers, tutors and students.
School visits
The evaluation team engaged in a purposive selection of schools for fieldwork. A random selection of schools was
not possible given the multiple literacy and numeracy programs in use across NSW, the small numbers of schools in
some program cohorts, and the other commitments of some participating schools.
Schools to be visited were selected to provide a range of implementation contexts, in terms of:
• the size and location of schools, ensuring demographic diversity within the sample
• characteristics of student population, including the proportions of Aboriginal students, students from culturally
and linguistically diverse backgrounds
• the mix of NPLN programs undertaken in each school
• schools’ engagement with other initiatives under the Smarter Schools National Partnerships, including the
National Partnership for Low SES Schools, to avoid logistical difficulties for the school.
-
9QuickSmart Numeracy - Evaluation final report
Evaluating QuickSm
art Numeracy
9
Every attempt was made to ensure that selected schools were not involved in multiple other learning initiatives or
funded programs.
In most cases, two members of the evaluation team visited each school for one day. Each visit included:
• an interview with the principal and relevant school leaders
• classroom observation of local program implementation, to allow students to meet evaluators prior to participation
in the focus group
• focus groups with students
• group interviews with teaching staff
• group interviews with the local Aboriginal Education Consultative Group (AECG) and parent group representatives
(where available)
• document review including the school plan and the school numeracy plan.
There are 13 public schools and one Catholic Primary School implementing the QuickSmart program through NPLN
funding. Three public schools and one Catholic school were chosen for qualitative analysis.
One public school was unable to organise an evaluation visit due to school commitments and other demands on
teachers and students. As an alternative, interviews were conducted by video conference with students, teachers
and the principal. There was no classroom observation.
Table 1.1 provides a summary of sources of data collected across the seven schools.
Table 1.1: Summary of participation in school interviews
Methods No of interviews No of participants
Principal and school leaders 4 4
Classroom observations 3 n/a
Student focus groups 4 22
Teacher interviews 3 5
Tutor interviews 3 5
Parent and community interviews 3 6
The inclusion of Aboriginal parents and local community members was seen as an important part of the qualitative
data gathering. Schools and local AECGs were contacted well ahead of time, and provided with information flyers
regarding the evaluation for distribution to members of their community.
Data recording and analysis
All interviews and focus group sessions were recorded using both audio recording and note taking. Field notes
and recordings were consolidated into a single record of interview, which was quality reviewed before analysis
commenced.
Systematic coding commenced with an initial set of categories generated from the Terms of Reference and the
background document review. Original categories were expanded and refined based on common issues and themes
emerging from participants’ responses. Counter-examples were sought, with recognition given to dissenting opinions,
particularly from within teacher group interviews.
-
10
The narrative responses to the teacher survey were included in the consolidated data set for each program. In the
case of QuickSmart, this coded data amounted to approximately 1050 individual thematic records. It is this material
that has been used to provide the majority of the qualitative evidence of the evaluation.
1.5.2 Quantitative component
The quantitative component of the evaluation included an online survey for teachers, and analysis of the available
student assessment data provided by the Department’s Educational Measurement and School Accountability
Directorate (EMSAD).
Teacher online survey
An online survey was made available to teaching staff in all 14 schools, to capture teachers’ experiences and views
on the effectiveness of QuickSmart.
The online survey was completed by a total of 17 school staff including some principals and school executive, from
nine of the 14 schools implementing QuickSmart. Full results of the online survey are attached at Appendix 2.
Note that the sample size of 17 is relatively small and this limits the strength of the generalisations which can be made
from the survey data. However, the feedback and opinions of those who did respond provide a valuable insight into
their experiences of the program.
The charts and graphs used throughout this report present responses to the teacher survey.
NAPLAN and NPLN assessment data
In schools implementing QuickSmart as their individual-student intervention, the majority of students taking part in the
program were in Year 5 or Year 6 (as demonstrated in Figure 3.1, p. 20). Accordingly, the influence of I on student
performance is highly unlikely to be evident in these results. Analysis of the NAPLAN and NPLN assessments is
provided in the interests of completeness of the evaluation.
A data model developed by EMSAD provided the quantitative analysis for the program evaluations. The Department
holds data for both government and Catholic schools. The NSW Catholic Education Commission (CEC) authorised
the preparation of this analysis by the Department for the purpose of this evaluation. The data model included
NAPLAN and NPLN assessment data.
For NAPLAN assessments which occur only for Years 3 and 5 in primary school, a pre-NPLN benchmark was
established using results for students in Year 3 in 2008 and 2009, for comparison with results of the same classes in
Year 5 in 2010 and 2011, as shown in Table 1.2. Individual students cannot be identified and a significant turnover of
students in some schools makes comparisons between students uncertain.
Table 1.2: Student performance assessment data sets
NAPLAN results NPLN assessment results
Cohort 1 Cohort 2
2008 Year 3
2009 Year 3 April, 2009 Year 2 Year 3 Year 4
2010 Year 5 August, 2010 Year 3 Year 4 Year 5
2011 Year 5 August, 2011 Year 4 Year 5 Year 6
-
11QuickSmart Numeracy - Evaluation final report
Evaluating QuickSm
art Numeracy
11
The preliminary NAPLAN data sets provided by EMSAD are provided as:
• test means and standard deviations
• gains in mean scores from Year 3 to Year 5
• percentages below, at and above National Minimum Standards.
The results are provided as aggregated for all NPLN program schools and also separated by program and by various
cohort groups of students, as shown in Table 1.3.
Data for students for whom English is a second language (ESL) are not provided, due to a change in the use of that
cohort group in the NAPLAN test data in recent years.
The key analysis in the use of the NAPLAN data is the effect size measure. Effect size is an indication of the meaning
attached to a difference between the mean of a sample and the mean of the population from which the sample
is drawn. Numerically it is the difference between the sample mean and the population mean, expressed as a
proportion of the standard deviation for the population. It describes the effect of being in a program school on student
performance.
Table 1.3: Aggregate data sets
Data set Program comparison groups Student comparison
NAPLAN• Program schools (aggregated data for
all NPLN schools implementing a given literacy or numeracy program)
• All NPLN schools (literacy/numeracy)• All NSW
• All students• Boys• Girls• Aboriginal• Non-Aboriginal• LBOTE• Non-LBOTE
NPLN assessments• Program schools (aggregated data for
all NPLN schools implementing a given literacy or numeracy program)
• All NPLN schools (literacy/numeracy)
• All students• Boys• Girls• Aboriginal• Non-Aboriginal• LBOTE • Non-LBOTE
In this analysis the population is ‘all the students in the state’ in most cases, but in other cases it is the students in
schools in a NPLN program group, as will be evident in the tables and text throughout this report. The sample is, in
most cases, ‘all NPLN program schools’ but it may be a cohort only, such as the boys or the Aboriginal students only.
An effect size of zero to 0.2 or -0.2 indicates that the means are virtually the same, and as such, the measure of
performance is the same for sample and population. An effect size greater than 0.2 or less than -0.2 indicates that
the mean of the sample is different from the population mean and the performance is different.
Positive effect sizes indicate a sample performance above the population performance, while negative effect sizes
indicate that the performance of the sample is below the population performance. An effect size greater than 0.5 or
less than -0.5, in other words a difference of half a standard deviation or more, indicates that the mean of the sample
is very different from the population mean and the sample performance is well above or well below the population.
-
12
1.5.3 Limitations of the methodology
The major limitation in the design of this program evaluation comes from the short period of time between the
commencement of the interventions in late 2009, and the points of data collection, both quantitative and qualitative,
to mid-2011.
The NAPLAN data can only be viewed as a preliminary and incomplete set because of the program timeframe. For
Cohort 1 (Table 1.2) the period of program implementation prior to the May 2010 NAPLAN testing in Year 5, allows an
effect time of, at most, one full term of the program intervention.
EMSAD has advised that further caution should be exercised in interpreting the results of NAPLAN and NPLN
assessments. A number of limitations impact on the validity of findings from the analysis, including:
• the variation in the focus of tests each year
• considerable student mobility
• small sample sizes for some comparison groups
• the impact of other literacy and numeracy initiatives operating in NPLN schools
• the use of these same programs in other NSW schools (not funded under the NPLN), and
• the lack of a comparable control group against which to benchmark results for NPLN schools.
The NPLN tests are adequate for whole-cohort assessment but are too brief to use for diagnostic assessment of
individual students. As the NPLN tests are half the length of the Basic Skills Tests (BST) on which they are based,
they cannot be used to compare with statewide performance on the former BSTs.
As NAPLAN and the NPLN tests are presented on different scales the results of these two assessments cannot be
compared.
The use of qualitative methods, as well as performance measures or surveys, provides a balancing effect, allowing
the experiences of teachers, students and parents to be presented for interpretation by others. Qualitative methods
do not seek to identify a simple consensus or give extra weight to frequent comments or repeated evidence of
similar experiences. It is the ‘atypical’ that also provides insight into the educational situation, especially if events are
experienced differently in different contexts, or by a variety of participants.
While this may suggest a limitation in the ability to provide general conclusions, what it does offer is recognition of the
diversity of experiences within and between school situations.
The small sample of schools involved in this evaluation, and the relatively low number of responses to the survey,
further limit any generalisations that may be inferred from the evaluation. However, as stated previously, the opinions
of those who did respond provide an insight into their experiences of the QuickSmart program.
1.5.4 Attribution
Finally, concurrent with the introduction of QuickSmart as the individualised program targeting those students in
greatest need of support, each school was required to implement a whole-class numeracy program. In all cases
where QuickSmart Numeracy was introduced as the individual-student intervention, Taking Off With Numeracy
(TOWN) was implemented as the whole-class intervention.
A challenge exists in attributing effectiveness to any one program or intervention, and in isolating that effect from other
influences in classrooms and schools.
-
13QuickSmart Numeracy - Evaluation final report
Evaluating QuickSm
art Numeracy
13
1.6 Presentation of the evaluation findings
Section 2 provides an overview of the QuickSmart program, including its goals, major features and proposed model
for teacher professional learning.
The evaluation findings are presented in Part 2 of this report, in the following sections:
Section 3. Implementation of QuickSmart
Section 4. Effects for students
Section 5. Teacher and tutor perspectives
Section 6. Impact in schools
Section 7. Summary of findings and conclusions.
The interview and observation schedules and the teacher and tutor survey results are included in two appendices.
-
14
2 QuickSmart program detailsThis section provides an overview of the QuickSmart approach and has been drawn from research literature (Bellert
& Graham, 2006; Graham, Bellert, & Pegg, 2007), the QuickSmart Program website (SiMERR, nd), the National
Partnership on Literacy and Numeracy information package for schools (NSW Government, 2009a) and the interview
with Professor John Pegg and Dr Lorraine Graham conducted by the program evaluators (Dione-Rodgers, 2011).
2.1 Aim
QuickSmart is a basic skills intervention designed to assist students who demonstrate persistently low achievement.
The program aims to improve students’ ability to easily and quickly recall number facts and perform basic
computation skills, referred to as ‘automaticity’.
“Automaticity is inferred when such lower-order processes become fast, routine, and independent, and require only small amounts of cognitive resources.” (SiMERR National Centre, n.d.a)
The aim of the QuickSmart numeracy program is to improve students’ information retrieval times, reducing the focus
on mundane or routine tasks. Students become better able to undertake higher-order mental processing and to
develop age-appropriate mathematics skills.
2.2 Background
“The main aim of the QuickSmart research program is to investigate the effect of improved ‘automaticity’ of basic skills on higher-order processes such as problem solving” (Graham, Pegg, Bellert, & Thomas, 2004, p. i)
The development of QuickSmart interventions in literacy and numeracy began in 2001 by Professor John Pegg
and Dr Lorraine Graham, from the University of New England’s National Centre of Science, Information and
Communication Technology and Mathematics Education for Rural and Regional Australia (SiMERR). It is a research-
based program, currently being used in 450 schools nationally, and built around a professional learning program for
principals, school leaders, teachers, teacher aides and school learning support officers.
The QuickSmart program was designed as a long-term, cost-effective program for middle school students performing
in the lowest 30% of the achievement range, who need to improve their basic mathematics skills. The program
targets those students who have been unable to draw benefit from other in-class and withdrawal instructional
activities.
Working intensively with individual students, QuickSmart emphasises both strategy instruction and deliberate practice,
to enhance students’ automaticity. Professor Pegg claims that while intervention programs have been used for many
years, none have achieved long-term impact.
QuickSmart focuses only on the number strand of the NSW mathematics syllabus. The development of automaticity
makes skills transferable to other areas such as space and measurement.
According to the program developers, it is expected that students will attain an effective level of automaticity, on
average, after a period of 30 weeks consistent involvement in the program. Variation in the length of engagement
required, either shorter or longer, may be influenced by the existing knowledge of number facts and the ability of
students. Professor Pegg noted that Aboriginal students may require longer engagement with the program, as
automaticity “is not the norm for them” (Dione-Rodgers, 2011).
-
15QuickSmart Numeracy - Evaluation final report
Program details
15
2.3 Features of QuickSmart
The QuickSmart program focuses on improving students’ understanding and recall of basic number
facts, performance of elementary calculations, and problem solving skills.
Both structured and incidental strategy instruction are important features of numeracy lessons, with the aim of moving
students on from relying on slow and error-prone strategies (especially count-by-one strategies) to the use of more
sophisticated and efficient strategies, including automatic recall.
Key features of the program include:
• pairs of students participate in 30-minute lessons, three times a week, for 30 weeks
• tutors use explicit strategy instruction, including modelling, focussed discussion, specific questioning sequences
and guided and independent practice opportunities
• instruction focuses on the relationship between number facts and ways to recall them
• ‘fact practice’ games and activities that are integral to every lesson
• independent practice
• regular computer-based assessment tasks, used to provide immediate feedback to students and to
• track performance
• close collaboration with parents, teachers and principals.
Students learn “it is ok to be wrong”. Errors are used for correction and improvement. QuickSmart is
about “learning to trust my head” (Dione-Rodgers, 2011).
The intention is to address individual student needs over an extended period, fostering learning in a
non-competitive environment that focuses on individual improvement.
Assessment provides ongoing formative information to guide the design of each individual student’s
program. Most lessons conclude with the program’s Cognitive Aptitude Assessment System (CAAS)
assessment. The resultant graphs of performance are engaging and easy enough for the students to interpret.
There is an important relationship between tutor and students. Originally developed for one-to-one
implementation, developers discovered that when students work in pairs there are additional benefits, including peer
learning and sharing of experiences, as well as increased numbers of students able to take part, and reduced costs
to schools.
2.4 Training and support
QuickSmart requires that teachers and tutors delivering the program participate in intensive training
through the professional learning program. It is recommended that a school based team be established to manage
implementation and that networks be formed between schools to provide further support for teachers.
2.4.1 Professional learning program
The SiMERR group provides all training, which occurs over three, two-day sessions for teachers and
support staff. There is also a one-day workshop for principals.
-
16
All potential instructors, including tutors, class teachers, Student Learning Support Officers and other support
personnel, may participate in the professional learning program. Training is focused on supporting QuickSmart
instructors to understand and provide:
• learning scaffolds and motivational activities
• strategy instruction and concept development
• focused, fun, and successful practice that is integral to every lesson and involves guided and independent timed
practice activities;
• appropriate teacher and peer modelling
• integrated assessment tasks in each lesson.
Teachers are encouraged to incorporate concepts of automaticity (Quick) and accuracy (Smart) regularly in their
teaching.
2.4.2 Costs of training and materials
The costs of the professional learning program were presented as follows:
• between $6000 and $12,000 per school for the full professional learning program, depending on number of
participants
• $3000 per kit of materials (flashcards, timer, workbook, etc). A minimum of one kit is required per school.
Costs are dependent on factors such as the number of schools in a cluster, the implementation activities at each
level of the model, and the type of commitment that different tiers of education are prepared to make to support the
program (NSW Government, 2009a, p. 26).
The program providers suggest that training for non-teacher participants, such as School Learning Support Officers
(SLSO) and Aboriginal Education Officers (AEO), is more cost effective as they are less expensive to replace for each
training day.
SiMERR is a ‘not for profit’ organisation. All profits made are returned to the program to develop resources for
schools.
2.4.3 School-based support
It is expected that each school will establish a QuickSmart team consisting of a teacher coordinator, QuickSmart
instructors and a member of the school executive. The role of the team is to manage the organisation of the program
and provide collegial support to both teachers and tutors.
School network
QuickSmart uses a model of implementation that sets up:
• groups within a school working at the student level, supported by the coordinator and team
• groups of schools within a cluster working to support teacher learning through a ‘school network’
• clusters of schools within a region working at the policy level, while also ensuring the fidelity of implementation
• regions of schools within a state working to support and evaluate the program.
2.5 Resources
Many of the materials required to implement the program are provided in the QuickSmart Numeracy Resource Folder
and the QuickSmart Numeracy Program User Guide, organisation folders and a range of teaching and learning
resources.
-
17QuickSmart Numeracy - Evaluation final report
Program details
17
2.5.1 QuickSmart Numeracy Resource Folder
The folder includes:
• focus fact sheets
• proformas for flashcard sets
• speed sheets
• strategy guidelines
• worksheets
• games
• graph sheets.
The folder also includes administrative and organisational information, teaching and learning resources, and a QuickSmart
DVD. Master copies of all the forms, permission notes, planning notes, evaluation, assessment and graphing sheets
required to implement the program are included. Electronic copies of all the documents are also included.
Teaching and learning resources have been developed in response to the individual learning needs of QuickSmart
students since its inception in 2001. It is noted that these may need to be supplemented to meet the specific learning
needs of particular students.
Electronic copies of the resources may be used to modify existing QuickSmart material or as templates for developing
new resources, as required.
2.5.2 QuickSmart Numeracy User Guide
The QuickSmart Numeracy User Guide provides detailed guidelines for the planning and implementation of the
program. These guidelines include:
• criteria for the selection of targeted and comparison students
• an overview of the QuickSmart instructional cycle
• suggested QuickSmart lesson formats and content
• explanations of assessment procedures used within the QuickSmart program
• a description of the instruments used to record students’ progress within the QuickSmart program
• an outline of how the QuickSmart program itself is evaluated.
2.5.3 Monitoring student progress
The CAAS software is provided on a CD and is used for ongoing assessment throughout the duration of the
QuickSmart program to monitor students’ response times and accuracy.
The CAAS system measures how rapidly students complete tasks. The tutor scores each response for accuracy. Students’
assessment results are automatically averaged per item and made available as either a graph or report form that is easily
interpretable by both students and teachers. Most QuickSmart lessons conclude with an assessment on the CAAS system.
In order to gain a clearer indication of the effectiveness of the QuickSmart intervention for improving accuracy and
automaticity of basic academic skills, pre- and post-test for students are provided. Results are submitted for each
student. SiMERR collects CAAS and standardised test data from other students in the same grade, to be used as a
comparison group to assess program effectiveness.
In general, the group of comparison students included in the assessments consists of average-achieving students as
nominated by their teachers. These comparison students complete the selected CAAS sub-tests in numeracy at the
beginning and the end of the intervention, and also participate in the standardised testing sessions (SiMERR National
Centre, n.d.a).
-
18
2.6 Program support
For Departmental schools QuickSmart support was provided by regional consultants or QuickSmart trainers. The level
and availability of support is determined by each region.
2.7 Previous evaluations
Based on program reviews by the developers, Professor John Pegg and Dr Lorraine Graham, it is claimed that after
three years of using QuickSmart, schools report whole-school growth regardless of which students have been on the
program.
The developers purport to have evidence demonstrating that students, both Aboriginal and non-Aboriginal, have
made academic improvement over the course of involvement in the QuickSmart Program.
Examples of research evidence (SiMERR National Centre, n.d.b) indicate that:
• In the Northern Territory during 2006, 2007 and 2008 the effect size growth of many hundreds of QuickSmart
students based on statewide tests was 0.68, 0.60, 0.78, respectively, and compared to a considerably lower
effect size of approximately 0.3 or less calculated for the average-performing comparison cohort. Data collected
over three years indicates that schools can expect on average a 10% improvement on standardised test results
of QuickSmart students in the first year of implementation and that jumps to approximately 20% improvement in
the second and subsequent years of implementation.
• At Orara High School, 44 Year 7 students undertook the QuickSmart program in 2006 and presented for the
2008 National NAPLAN test in Year 9. The results show that 42 students were above benchmark. The two
students below benchmark were identified as IM2 students in Year 7; however each student experienced above
average growth for the period.
• Results from eight schools who participated in QuickSmart in the North Coast Region of NSW in 2007 had an
effect size 0.75 on the Progressive Achievement Tests in Mathematics (PATMaths) Third Edition (ACER, 2005).
The comparison cohort’s effect size was 0.19. The test cohort’s results represented in excess of an threefold
growth over the year. This result improved in 2008 with an effect size of 0.801 for the new QuickSmart cohort of
238 low-achieving students, of whom 52 were Aboriginal.
• An analysis by independent statistician, using the large data sets of several hundred students in NSW over
several years whose performance on QuickSmart has been evaluated using the Progressive Achievement
Tests in Mathematics (PATMaths) Third Edition (ACER, 2005) found that the effect size for QuickSmart students
ranged from 0.59 to 0.69, with the latter figure representing those students who completed the full 30 weeks of
instruction.
2 IM classes are provided for students with a mild intellectual disability.
-
19QuickSmart Numeracy - Evaluation final report
Implem
entation
19
Part 2 Evaluation findings
Part 2 presents the findings of the program evaluation, organised in terms of priority areas identified in the Terms of
Reference for the evaluation, and in response to issues that emerged through the data collection.
3 Implementation of QuickSmartSection 2 provided an overview of the QuickSmart program as it is intended to be implemented.
This section sets out the major implementation influences that made a difference to schools’ experiences of
QuickSmart in practice.
Key findings:• Ease of implementation, resources and assessment software were instrumental in schools’
decisions to implement QuickSmart as their individual-student intervention.
• Decisions about which children may take part in QuickSmart need to consider a range of criteria in addition to NAPLAN or other assessment results.
• The costs of training placed limits on the number of teachers and other staff who could take part in the QuickSmart professional learning program.
• The role of the coordinator was essential in establishing and managing the withdrawal program.
• The professional learning model was effective in increasing teacher and school capacity to implement QuickSmart strategies beyond the individual-student intervention.
3.1 Program choice
For schools choosing numeracy as their target area for improvement, the choice of programs was limited. Schools
had only one choice of whole-class numeracy program: Taking Off With Numeracy (TOWN). TOWN could also be
used as the targeted intervention for individual students. The only other program on offer was QuickSmart. One
principal expressed his displeasure at the lack of choice:
“[That] there weren’t many intervention options available to us to choose from in the first place was one part of it. So that’s like a gun to the head. We just had to choose what was there.”
The additional frustration expressed by principals was the short time frame in which they had to make a professional
decision regarding program choice. One expressed it bluntly:
“The biggest difficulty in choosing any of them was the ridiculously short timeframe to make judgements of expenditure of hundreds of thousands of dollars without access to asking people questions about how things actually work.”
Two of the visited schools had been using QuickSmart for a period prior to the commencement of the National
Partnership on Literacy and Numeracy (NPLN). The additional funding allowed them to expand the program, beyond
the bare minimum. One principal explained:
“We’d already been involved in QuickSmart for two years. We were a pilot school from 2009 and in that time we had seen... the confidence of the kids was building. So we thought it was something to have a go at and sort of explore a bit more.”
-
20
The other two principals indicated that they were more impressed with QuickSmart than TOWN, so made the
decision to go with the alternative program for their individual numeracy intervention. At one school they used TOWN
and QuickSmart “as a package… that creates an impact.” A number of teachers indicated that the ease of program
implementation, the provision of the resource kit and online assessments, also contributed to the decision to
implement QuickSmart within their schools.
For several principals, knowledge of the program had come through existing association with the University of New
England (UNE). Proximity to UNE and SiMERR was a strong motivator in program selection for schools in the New
England area. One principal explained:
“QuickSmart originates from UNE. We knew about [QuickSmart] through our prac students. We have a bit of a relationship with UNE anyway, so we thought that might be helpful seeing as the originating organisation isn’t too far away.”
3.2 Selection of students
QuickSmart is program designed primarily for middle years’ students. In order to meet the criteria for participation,
primary school students must:
• demonstrate persistent difficulties in numeracy
• display a good attitude to working in small groups
• not have intellectual disabilities (NSW Government, 2009a).
In line with QuickSmart program recommendations, students were mainly selected from the Years 4 to 6 bracket. Survey
results indicate that nearly all students targeted were in Stage 3 during the NPLN period, as shown in Figure 3.1.
Figure 3.1: Implementation in Year Levels 2009-2011
One school identified that they had an informal ‘waiting list’ while most of the others identified the need to carefully
select those in greatest need of assistance, as expressed by one principal:
“… were only able, as a result of the funding, to facilitate [participation of] the most needy students - not all who needed it. So we had to prioritise.”
Most schools reported selecting students based on their NAPLAN performance; those with results in Bands 1 or 2.
This was sometimes tempered by discussions with teachers, as to who might benefit most from participating.
-
21QuickSmart Numeracy - Evaluation final report
Implem
entation
21
Teachers had varied views of who such children might be. Some agreed that the lowest-performing students were
most entitled to be involved. The following issues were raised in teacher and principal interviews:
• Some students were identified by teachers to be ‘not ready’ for the intensity of the program, especially those in
Year 4. Other teachers regretted the NPLN criteria of ‘persistent difficulties’, as expressed here:
“… the rigid structure of targeting kids when they had already failed was an initial problem - we wanted to try and catch them earlier.”
• Year 6 students were sometimes seen as most needy, particularly to better equip them for high school.
• Some schools saw value in involving Year 4 so that benefits would be apparent in future NAPLAN results,
although a few teachers expressed reservations about this rationale.
• Others suggested that the nature of individual students should also be taken into consideration.
The most common area of debate related to whether students “other than the lowest [performing]” would benefit
more from participation. As QuickSmart sessions were often reported to be conducted during maths or numeracy
time, teachers debated whether the time spent with the QuickSmart tutor was sufficiently beneficial to offset the
disadvantage of missing further maths learning opportunities occurring in class, particularly for those students
performing least well. As one teacher suggested:
“… and as I said, the kids if they’re low and they’re getting one-on-one and that’s super but if they’re missing that half hour [of class work] and then they miss the next half hour because they’re late for it, that’s a whole sort of hour of class time missed so… Q. So the benefit isn’t great enough? It depends on the kids. I’d say the kids I have this year - no. The kids I had last year, most of them - yes.”
The need to consider students’ ability to rejoin class activities was raised at each of the schools visited. As suggested
in one teacher discussion:
“… sort of kid that needs motivation to work, the last thing they need is to come halfway through a lesson … So you probably need to look at how the kids can work independently; how good they are at coming in and getting into their work; as much as their needs.”
Teachers also suggested that the “middle, just below where they should be, kids” may make greater gains from the
program, as well as being more able to “walk [back] in, look at what everyone else is doing and pick it up.” Some
teachers expressed strong views, such as:
“I believe the QuickSmart program to be of the greatest benefit to students who fall in the middle band for numeracy. I feel they have the ability to transfer knowledge from their time with the QuickSmart tutor into the classroom.”
Other comments from teachers suggested that the program, while quite structured in how it operates for students,
was flexible enough to allow it to fit in with how the school decides to proceed.
Interesting comments came from a number of parents, all of whom suggested that their children would benefit from
QuickSmart being introduced earlier than Year 5:
“… and getting them in earlier grades, not just the 4/5 mark. I know my child would have benefited had he been doing it for all of Year 4 rather than most of Year 5. I know numbers are an issue and tutors and money but… it’s a brilliant program.”
“Only - keep it going and implement it sooner. It works!”
-
22
3.3 Staffing
Finding suitable tutors was reported as being the critical influence on the success of the program.
Some schools chose to re-direct existing school support staff to act as QuickSmart tutors, including library
assistants, Support Teacher Learning Assistance (STLA), SLSOs, and AEOs, and the school community was eager
to participate. As required, extra tutors were hired to cater for the numbers of students involved in the program. All
principals emphasised the importance of getting the right people:
“I think one of the things we concentrated on was finding the right people [tutors] to deliver the program - and keeping them.
“…they all are parents at the school, and have been here for a long time, in volunteer work as well as other aiding work, Special Ed as well as other things… We didn’t advertise the positions, we hand-picked who we thought could do the job well.”
Principals indicated that staff choice was relatively easy as there was support from existing staff. One principal noted
that the willingness of staff made implementing the program less complicated:
“The fact that I’ve had staff willing and keen to do it. We all hear stories where staff say I’m not going to do it and you can’t make me almost. I’ve been lucky that the four ladies, the two initially and now another two that we’ve trained up are very keen to run it. They find it reasonably easy.”
Most schools had a small team of tutors, often three or four to meet the demand. The stability of the tutoring team
was highlighted as a significant benefit in most of the schools involved, attributed to the relationship developed
between the tutor and the school, especially the QuickSmart coordinator. Support for the tutors was important: from
the coordinator, teachers and within the group of tutors themselves. Tutors working collaboratively was seen to benefit
the school as well as the tutors:
“… to maintain that team of tutors - they bounce ideas off each other - they’re supporting each other so there’s a close knit little team there as well.”
3.4 Professional learning
As mentioned in Section 2.4.2 the QuickSmart program provides professional learning opportunities to teachers and
support staff, numeracy coordinators and educational leaders.
The program providers expected the establishment of a school-based support team, and also recommended the
establishment of networks at the local cluster, regional and statewide levels.
Within the NPLN implementation, only the school level and, in some cases, the school network level operated
effectively.
3.4.1 QuickSmart professional learning program
The first professional learning opportunity provided as part of the QuickSmart program implementation is a day for
principals and other administrators to engage with the details of the program, examine the results of the research that
establishes the intervention’s effectiveness, and, where possible, visit a QuickSmart site.
Principals indicated that this was valuable in determining the choice of program for their school, as well as giving them
the necessary professional learning to effectively support their staff throughout the program implementation.
The QuickSmart coordinator and instructors then participated in the series of two-day professional learning workshops.
Nearly all staff considered the professional learning associated with QuickSmart to be useful, as indicated in Figure 3.2.
-
23QuickSmart Numeracy - Evaluation final report
Implem
entation
23
Figure 3.2: Usefulness of professional learning
Organising attendance at the training days presented logistical as well as funding issues. Most teachers had to travel
significant distances for each two-day workshop, either to Sydney or Armidale. In addition to the cost of the training,
schools had to fund travel and accommodation expenses, and relief for teachers or others attending. Additional days
of relief to cover travel time meant that each two-day workshop cost the school up to four days’ relief, as explained by
a teacher:
“The location of training has been a problem for us. We had to go to Sydney which is 4 days really – a day of travel, two days’ of being trained and a day of travel back. That was certainly a hindrance.”
Even schools located in the New England region, closer to Armidale, described attending training as onerous. For
one school where costs for travelling were prohibitive, the UNE team provided one of the three QuickSmart training
sessions at the school.
While one survey respondent reported that all staff in their school had taken part in training, for many others the costs
put severe limits on the number of people who could be involved. School-based training was seen as a lesser but
necessary option. Training of tutors was provided in-house to save money, as reported by one coordinator:
“… we ran it with a couple of aides, so we had a couple of aides trained and when I say trained, I did the training - they didn’t go off to the training… because we’re right at the southern end [of the region].”
The decision to train teachers and tutors together was seen as beneficial by a few schools. The opportunity to
clarify processes and plan for implementation in the school while away together at the training was appreciated, as
one tutor noted:
“…it’s partly what the training days did, we were all together and [teachers] were at those training days… gave opportunities to talk about things; very structured but it gave us some opportunity to [plan together]”
Principals noted that they would like to have all their teachers trained in QuickSmart so that effective strategies can be
included within class practices for all students.
3.4.2 Ongoing support
Each school established the QuickSmart team as required. The QuickSmart (teacher) coordinator was expected to
lead the implementation of the program, and with support from the school executive, organise the timetables, training
and liaison with parents and community, as required. For some Departmental schools, support was available from a
regional QuickSmart or NPLN facilitator.
The school network
Only a few schools reported being part of one of the QuickSmart school networks. In one school where QuickSmart
was in operation prior to the NPLN, they maintained their links with the existing network, meeting two or three times a
year. Sharing ideas, particularly around common problems was seen to be valuable in also keeping support costs down.
-
24
The geographic location of the school seemed to be the other factor influencing the support provided for
implementation of the program. Schools located close to the University of New England (UNE) reported receiving
effective support. Schools that were more geographically isolated found the ‘school network’ support feature a
challenge. As one coordinator explained:
“I also got an email from [coordinator] at Crawford. He asked if we wanted to be part of a network and I said ‘Yes’, but I’ve never heard anything back from that… He initially said they would catch up, have meetings and see how things were going. That was in Term 1 but I’ve never heard anything since so…”
Overall, school leaders indicated that sufficient support was available from the initial professional learning activities,
from school-based QuickSmart coordinators and through the regional program facilitators.
3.5 Space and timetabling
A few principals commented on the need for extensive planning in order to locate appropriate spaces for QuickSmart
to function effectively, and to enable scheduling of students to attend three half-hour sessions per week. As noted by
one assistant principal:
“Withdrawal programs are always a challenge when it comes to timetabling, coordination of staff and combining appropriate pairs of students. It is always a concern when time is short.”
3.5.1 Locating suitable work spaces
The work space required for QuickSmart had to accommodate a tutor and two students working intensively for half
an hour without interruption, with access to networked computer software and online assessment activities. In some
cases, up to three tutors may be working with students at any given time. A few school leaders indicated that it was
vital to have a designated space so that the resource kits could be easily accessible.
In some schools NPLN funds were used to redesign rooms to ensure an appropriate environment was available for
learning. One principal advised:
“We didn’t have enough [space] because we had two tutors operating - that was the decision we made - so we needed to redesign a room and have some building works done over the Christmas holidays, in readiness.”
For others the computing infrastructure was the issue:
“…it comes down to the availability of the resource (computers) on a network that is reliable.”
“…the QS program needs PC computers, and we’re a Mac school. …so it has probably cost us a bit more than other schools because we have had to purchase computers that we wouldn’t normally have purchased…”
3.5.2 Timetabling
Once spaces had been established, school leaders highlighted the challenges in timetabling all students to fit in with
class demands and tutor availability. It was complicated to balance individual student’s time away from classes with
the number of sessions each tutor could do in a day or week.
Coordinators spent considerable time trying to avoid situations where students consistently missed the same
lessons, especially “literacy time”. Several coordinators expressed frustrations such as:
“We’ve tried to consider things like reading groups, and not taking kids out at that time and adjusting different times throughout the week.”
-
25QuickSmart Numeracy - Evaluation final report
Implem
entation
25
They reported creating timetables where:
“… we do a different time for each day of the week that they have QuickSmart, so that they don’t miss the same thing every day. So they might have a 9:00 to 9:20 on one day and 11:20 to 11:50 on another day, so they don’t miss maths every day, or they don’t miss reading groups every day.”
The down-side of this system was that students and teachers struggled to keep track of where they should be, as
one teacher admitted:
“I’m constantly forgetting who’s going when, which is a real pain for the tutors. They’re chasing me up. So where’s W? I forgot it was 10.00 o’clock.”
Other schools tried systems that allocated sessions to coincide with sport or assembly times, or where students from
one class would all go to tutoring at the same time. One coordinator explained their solution:
“From that experimentation we found that we needed to at least each term change the timetable so that the kids were coming out of classrooms at different times.”
Being flexible was important for all participants: teachers, tutors and students. As described by teachers:
“The tutors are pretty good at readjusting the timetable if there is a particular assessment going on in the classroom and the teachers say, do you mind? They’ll probably reorganise things.”
“Some students who know they’ve got something on and will talk to the QuickSmart tutors in terms of reorganising their schedule.”
3.6 Costs of implementing QuickSmart
In addition to the cost of employing tutors, the two major expenses encountered by all schools were the cost of the
professional learning program and the mandatory kit of resources.
The estimated cost of initial professional learning for each school was between $6,000 and $12,000 depending on
the number of teachers or other staff participating, and the input provided at the cluster and region levels. Added to
this was the cost of teacher relief, travel and accommodation for all those attending the training.
Each school needed to purchase the kit of materials valued at $3,000. Some schools chose to purchase more than
one kit, especially where QuickSmart strategies were implemented across classes, as well as in the withdrawal mode.
As mentioned in Section 3.5.1, some schools