Data-Driven Decision-Making for Quality Control: The Power of a Relational Database Vicki L. Cohen,...
-
Upload
gavin-nicholson -
Category
Documents
-
view
219 -
download
0
Transcript of Data-Driven Decision-Making for Quality Control: The Power of a Relational Database Vicki L. Cohen,...
Data-Driven Decision-Making for Quality Control:
The Power of a Relational Database
Vicki L. Cohen, Ed.D.Marlene Rosenbaum, Ed.D.Joshua Cohen
Fairleigh Dickinson UniversityTeaneck, NJ 07666
Presented at the Annual AACTE ConferenceNew Orleans, February 2008
This session will: Describe how using a relational database
becomes the driver of a quality control system;
Describe the development and utilization of a relational database;
Show how data is leveraged to support student learning, program revision, and outcomes based assessment.
The School of Education (SOE) at Fairleigh Dickinson University Comprised of aprox. 1,000 students Multiple programs that apply for state
certification: 5-Year Accelerated QUEST program MAT LD Educational Leadership Reading Specialist
The SOE at a Glance On two campuses (Teaneck and Madison) Located at 3 Community Colleges throughout
the state of New Jersey 15 Full-time faculty members Aprox 35 part-time faculty Place approximately 120 candidates into
student teaching/year Place a total of approximately 700 candidates
into clinical field experiences/year.
FDU at a Glance
FDU has aprox 12,000 students Largest private university in State of NJ SOE is part of University College on Teaneck
campus QUEST program:
45 candidates at Teaneck 200 Madison 75 CC
SOE Is a Complex Program!
Went for TEAC accreditation
Preparation for TEAC School of Education (SOE) needed to collect
accurate information on its claims Started gathering data on programs and student
performance Recognized need to access, organize and
analyze data in meaningful ways Developed a relational database This would become the “driver” of our Quality
Control System
The Quality Control System (QCS)
Every institution and program has a set of procedures and policies to ensure quality in hiring, admissions, curriculum, program design, and student learning.
Together, these procedures and structures function as a Quality Control System (QCS). (TEAC)
Need for Valid and Reliable Data
QCS must yield valid and reliable evidence about the program’s practices and results, which influences its policies and decision making.
What is Valid Evidence? Are we measuring what we intended to
measure? Are we sure that our evidence is pointing
us in the right direction? How confident do we feel about the data
we collected?
“Am I measuring what I think I am measuring?”
What is Reliable Evidence?
Yields results that are accurate and stable
Collected in a consistent way Confident we we are making the right
decision.
SOE TEAC Process
1. Developed a QCS that ensured we are collecting valid and reliable data on our claims and cross-cutting themes
Instruments (rubrics, observation forms, surveys) were validated through external panel of “experts”
Inter-rater reliability is being established
2. Developed infrastructure and system to collect the data
SOE TEAC Process
3. Analyzed data: aggregated and disaggregated
4. Determined strengths and weaknesses with total faculty involvement
5. Analyzed what revisions needed • Programs• Curriculum• Processes and policies
6. Currently making revisions based upon evidence.
SOE Assessment Philosophy
We use multiple sources of data that are designed to assess the performance of teaching candidates as they progress through our program.
Collect data in three areas.
1) Throughout the Program
Continuous assessment of teaching candidates throughout the program from entrance, midpoint, and exit Grades GPA Praxis scores Rubrics Reflections
2) In Clinical Field Experiences
Assessment of pedagogical knowledge and skills that occurs during clinical field experiences; Placement Observation forms
3) Completion of Program
Perceptions of teaching candidates and alumni after they have completed their program, which is used for program improvement Alumni Surveys Exit Surveys Focus Groups
Traditionally….Data Deficient
Schools of Education have not been collecting data systematically
Infrastructure not set up Not able to access multiple
sources of information
Traditionally…..Data Dummies
What data do we want to collect? How can we manage it? What do we do with data? How do we organize it? Access it? Make sense?
Currently….Data-Driven
Systematically collecting data Infusing into our faculty culture Meeting regularly to assess evidence Making decisions
based upon evidence Beneficial process
due to TEAC
Data-Driven Decision-Making
Plan
CollectData
Impl
emen
tA
nalyze
Data
Reflect
& Revise
Example of Data Collection:Praxis Results 2005/2006
Program FDU SOE Pass Rate
NJ Pass Rate
Elementary Ed Content Knowledge 99% 84%
English Lang Lit Content Knowledge 100% 71%
Social Studies Content Knowledge 100% 64%
Secondary: Math Content 100% 92%
Middle School Math 100% 65%
General Science Content Knowledge 100% 78%
Spanish Content Knowledge 100% 87%
Mean Scores of Teaching Candidates on Selected CCI Indicators Related to Pedagogy as Rated by Supervisors (Fall 2005, Spring 2006)
Midpoint Final
Indicator N Mean Standard Deviation
N Mean Standard Deviation
2.2 Creates and implements lessons that are developmentally appropriate.
57 4.47 .630 51 4.75 .483
3.2 Effectively incorporates multicultural information and strategies when appropriate into the lesson; presents issues from a multicultural perspective.
57 2.82 2.010 51 3.59 1.846
4.1 All essential components of a well designed plan for coherent instruction.
57 4.58 .533 51 4.71 .672
4.2 Appropriate instructional objectives that are clearly stated, measurable and aligned with the NJCCCS.
57 4.42 .706 51 4.67 .476
4.3 Effective use of a wide range of resources including technology to enhance instruction and student learning.
56 3.86 1.420 51 4.39 .918
4.4 Demonstrates understanding of curriculum design and implementation, including various curricular approaches (i.e. interdisciplinary, integrated, thematic).
57 3.79 1.278 51 4.12 1.336
Wanted: Database Administrator
New Job Description: full-time professional staff
Resources Support from administrators Ensure candidate had appropriate
knowledge and skills to design database
What Is a Database System? A collection of data organized in tables,
which can be accessed and manipulated, without having to restructure the tables
Elements of a Database System• A storage system • Data structures• Manipulation tools
Database Advantages Analyze sophisticated correlations easier
because relationships are established between data sets
Make decisions based on information derived from data
Streamline business operations Organize data and eliminates:
• Inconsistent data• Missing data• Redundant data
Problem #1: Field Placement Office
Staff was overwhelmed with managing Clinical Placements
In 2007 800+ letters were mailed to 424 schools asking for placements
Previously, these letters were individually prepared in MS Word documents
Problem #1: Field Placement Office (cont’d)
Clinical Placements must be coordinated with• School• Student• Supervisor
Difficult to aggregate data• What districts have most confirmed/declined rates?• What trends are we seeing?• What kind of schools are we sending our candidates to?
Solution #1: Streamlined Field Placement Office Developed a data collection system for
Clinical Placements Data is:
• Entered on 2 campuses• Used to create personalized communications to
schools, students, and supervisors• Used to manage Clinical Placements
• Confirmed/Pending/Declined• Supervisor Assignments
Solution #1: Streamlined Field Placement Office (cont’d)
Gives us the ability to aggregate data, look at trends, and make decisions Confirmed / Declined distribution by district Analyze demographics of cooperating school
districts
Solution #1: Infrastructure
We start with the person record from the university system
Data on the Clinical Placement is entered
Solution #1: Generating letters
Generate standardized reports for “master lists”
Solution #1: Payroll
Problem #2: What Type of Districts Do
We Place Our Candidates Into?
The District Factor Group (DFG) is a socioeconomic indictor used for comparative test reporting of school districts for New Jersey’s statewide programs.
Problem #2: What Type of Districts Do We Place Our Candidates Into? (cont’d)
DFG Factors: % of adult residents failed to complete high school % of adult residents who attended college Occupational status (laborers, service workers, farm
workers, professionals, etc.) Population density Income Unemployment rate Poverty rate
Problem #2: What Type of Districts Do We Place Our Candidates Into? (cont’d)
Eight DFGs have been created based on the 1990 United States Census data
Range from A (lowest socioeconomic district) to J (highest)
A, B, CD, DE, FG, GH, I, J
DFG: State DistributionDistrict Factor Group Distribution
For All Districts In NJ
0
20
40
60
80
100
120
A B CD DE FG GH I J
DFG: Apprenticeship Teaching Distribution
Distribution For Apprenticeship Teaching In Spring 2007
0
5
10
15
20
25
30
35
District Factor Groups
Fie
ld P
lac
emen
ts
A B CD DE FG GH I J
DFG: Alumni DistributionReported Distribution of Working Alumni
0
2
4
6
8
10
12
14
16
18
District Factor Groups
Alu
mn
i
A B CD DE FG GH I J
Solution #2: Share Data, Discuss, Revise Evident discrepancy between where alumni get
jobs and where candidates are placed We share evidence with faculty and key
stakeholders They discuss and make appropriate decisions
DFG: In the future
Will have full record of where candidates performed clinical experience
Will have record of where they are working
Can correlate accordingly
Data Needed
A state database of teacher employment Difficult to track alumni as they move from
school to school Unique teacher & school identifier
State database needs to integrate with University and commercial marketing data systems
Problem #3: Data Systems Not Integrated
SOE’s recordkeeping is not integrated with the University system Student information is entered into SOE
system manually Limits power of reporting Duplicate person records may exist if Student
ID is not entered correctly
Problem #3: Data Systems Not Integrated (cont’d)
A “live” data connection to University system is not possible Technology is not in place Support for integration is needed
More Efficient and Effective Use of Resources
Relational database assists with: Streamlined SOE business operations
Generate mail merge letters Provide reports Automate payroll
Leverages existing data to create information for program improvement
Started Slowly
Started with trying to code and track our students properly
Administrative assistant created rudimentary Access Database
When she left, we hired a consultant to manage database
He totally redesigned and reorganized it
Skill Set for Database Administrator Problem-solving Relational database design skills SQL proficiency Knowledge of structured programming
language Excellent communication skills Work with faculty and technical staff “People-skills”
Conducted Extensive Search Advertised Set up search committee Interviewed many different applicants Required each applicant to take a test Presented problems to candidates to
assess problem-solving abilities Found many could enter data, but not
design or problem solve
Working with the University
Collaborating with the Arts and Sciences Establishing knowledge-base in content
areas and general education Addressing NJCCCS and Professional
Standards in discipline Aligning content courses and standards in
matrices
Working with the University(cont’d)
Meeting with individual departments Establishing long-term relationships Shifting to new paradigm--Learning Outcomes
Assessment Working with A&S to collect data Database Administrator playing key role in
collection of data across college
Leveraging TEAC Across the University Establishing the need for LOA: Middle States Educating the A&S faculty: LOA process Addressing resistance of A&S faculty Establishing a relational database system for
university: program, college Creating the infrastructure to collect data Collecting multiple sources of data for A&S Getting various groups to communicate and
plan.
Conclusion: The Power of a Relational Database
Data-driven Decision-making requires
an integrated system of collecting data from many different sources.
Systemic The whole institution needs to be
vested in the collection of data Data needs to be collected on faculty,
students, courses, grades, scores, rubrics, observations
University and Colleges need to ensure data collection systems are in place early.
Integrated
All data sets need to be connected so relationships can be established
Queries made Reports generated Correlations and relationships analyzed
Benefits for Total University Middle States Accreditation Nursing Engineering College of Business Program improvement Student learning
School of Education Leads the Way
Ultimate goal is to improve teacher quality and impact achievement for all students.
Data provides the means to do this. Relational database is the engine
that makes this possible.
For more information please contact:
Vicki L. Cohen [email protected], School of Education
Marlene Rosenbaum [email protected] Dean, University College
Joshua Cohen [email protected] Administrator