Pokhara University Examination System:

Effectiveness, Challenges and Solutions

Prof. Dr. Hari Krishna ShresthaPrincipal

Nepal Engineering CollegeChangunarayan VDC-9, Bhaktapur

July 26, 2012email: hari@nec.edu.np

Mobile: 9851006010

• Really interested and committed to find and contribute in the solution?

• Or, just a कर्म�काण्ड?

Fundamental (Macro) FactorsPolitical Factors: Government of Nepal’s unclear and

unreliable policy on involvement of private parties in education, including Higher Education

• Political interference: Symbiotic relationship between national level political leaders and student/ teacher/ staff leaders. Unless this relationship is broken all attempts to improve the situation will fail.

Social Factors: Attending class for parents• Equity in access to higher education, not quotaEducational Factors: Accreditation, not affiliation• Monitoring of Quality Assurance, including curriculum

relevance, mismatch between curriculum and final exam question, and skill level certification

The Blame Game Factors1. Class disruption• Students (strikes, pad-locking, class bunk, non submission of assignments,

sports and similar other activities) • Teachers/Staff (pen-down, not showing up in time, coming to class not

fully prepared, leaving college in mid-semester)• Regional/National events (strikes, chakka-jam)2. Internal Evaluation disruption Teachers cannot submit internal marks

in time Colleges cannot submit internal marks to Pokhara University in time

3. Final Exam disruption (schedule not published in time, unacceptable schedule for the students, memory testing type of questions, course not completed, internal evaluation not completed)

4. Result publication disruption: university not distributing answer sheets in time, teachers not returning answer sheets in time, staff not entering and checking data in time, exam committee not convening in time.

Two Stages of Student Evaluation

• Internal Evaluation (50% in undergraduate, 60% in graduate)

• External (Final, Board, Semester End) Evaluation

• I- F result, everyone seems confused. The I result becomes F only if students cannot complete the requirement by the following semester.

Steps in Conduction of Semester End Examination (at each college)

• Distribute, collect and check (as per course structure) Course Registration Form at the beginning of semester. But this step is becoming redundant due to university’s requirement to fill all regular courses every semester.

• Conduct Internal Evaluation (theory and practical)• Distribute, collect and check (as per course structure and

exam rules) Examination Form at the end of semester• Prepare triplicate and submit the triplicates, Course

Registration forms and Exam Forms to the University

Steps in Conduction of Semester End Examination• Preparation and distribution of Exam Form• Exam Routine Preparation• Question Setting (Multiple sets per subject)• Question Moderation (single or multiple sets)• Question Typing, Printing, checking, • Question counting & packaging for each exam center as per Exam form• Question package sealing• Question packages, seal breaking and sealing form, delivery to exam centers• Conduct Exam• Collect Answer Sheet Package• Distribute Answer Sheet to Checkers and Collect checked answer sheets• Scrutiny of checked answer sheets• Data entry, checking, converting to letter grade (relative/absolute)• Exam Committee Approval• Result Publication• Grade sheet preparation, checking and distribution to colleges

(at University Office)

Steps in Conduction of Semester End Examination (at Exam Center)

Pre-exam preparations: (details in University’s Directions to the Superintendent)

• Logistical preparation (exam hall, furniture, communication, human resources, water, glass, jugs, toilet, emergency medicine, food, fan, wax seal, blank papers, paper envelope, cloth sack, polythene bag, glue, stapler, scissor, needle/thread, graph papers, log-graph, bode graph, many types of charts, tables and codes, photocopy, fax, email, candle, lighter, etc.)

• Security (internal and external)• Answer sheets (first copy and additional copy), Seal breaking form, Answer copy seal

form, examinee attendance form, staff attendance form, etc. collection• Collect exam routine, number of examinees in each subject, ERN of examinees,

colleges associated with subject and examinees, contact person and contact number of each college

• Contact person and contact number of university’s exam officials• Seat Plan preparation for each day• Invigilator plan and support staff plan for each day• Inform and get consent from assigned invigilators, support staff

Steps in Conduction of Semester End Examination (at Exam Center-Exam Conduction)

• Check number of seats in each exam hall/room• Seat planning as per triplicate/”received Exam Forms”/ and marking

seat number in each desk• Question package opening in front of University Observers• Question paper counting and separating as per seat plan and room plan• Orientation and Question paper distribution to invigilators• Letting examinees enter exam hall a few minutes before

commencement of exam• Exam form checking, verifying examinee, verifying examinees seated as

per seat plan• Answer sheet (first copy) distribution to examinees• Announcements and question corrections, if any (as per University

instructions)

Steps in Conduction of Semester End Examination (at Exam Center-Exam Conduction)

• Question paper distribution to examinees and mark starting time of exam and let examinees know starting time.

• Get attendance of examinees, check exam form, ERN, first copy number, signature

• Invigilate exam, ensuring proper exam environment• Keep record of temporary leave from exam hall• Periodically check environment in & around exam

center, including boundary & toilets.• Collect question paper if any examinee leave exam hall

before one hour from the start of exam

Steps in Conduction of Semester End Examination (at Exam Center-Exam Conduction)

• Stitch additional answer sheets a few minutes before end of exam time. Check number of additional answer sheets with record of additional answer sheet number on the first answer sheet.

• Collect answer sheets and arrange numerically as per ERN, and match with attendance record, and sign on each answer sheet

• Collect attendance of invigilators and support staff• Package the answer sheets and seal in presence of University

Observers, along with attendance sheets. Record exam center, subject, date, number of answer sheets, number of seals on answer sheet packages

• Safely deliver the sealed answer sheet packages to University, along with Attendance records of examinees, staff/invigilators

Steps in Conduction of Semester End Examination (at University)

• Collect sealed answer sheet packages and examinee and staff attendance sheets from exam centers, keep record of receipt.

• Collect “Expelled” examinee’s copies and keep record.• From the roster of copy checkers, for each subject, decide on

answer sheet checker (checkers, if number of answer sheets is very large), prepare appointment letter, contact answer sheet checker, deliver or provide answer sheet packages to the checker, keep record of the delivery of answer sheet packages

• Follow up on checking progress to ensure timely completion of answer sheet checking

• Collect checked answer sheets along with filled Mark Foils. Check number of answer sheets match with record in examinee attendance sheets.

Steps in Conduction of Semester End Examination (at University)

• Verify numerical and alphabetic record in Mark Foils• For each subject, decide on Scrutiny Board Members (SBM),

prepare appointment letter, contact SBM, call SBM at University, provide answer sheets, record recommendations of the SBM.

• Enter Marks Secured in computer, check data entry manually, based on Marks Foil and SBM recommendations, and in some specific cases Subject Committee recommendations/PU Examination Committee decisions.

• Using statistical distribution of the marks, convert the marks into grades, adopting relative and/or absolute letter grading system (less than 30 examinees – absolute, generally, not necessarily)

• If difference between internal and external marks consistently differ by more than 25%, apply “down scaling” and re-grade.

Practical Problems in Conduction of Semester End Examination(Question content)

• Questions testing “memory” rather than problem solving skill• Questions not balanced as per syllabus• Questions out of syllabus or from old syllabus or “new” syllabus• Question depth normally unexpected• Questions too long to solve in time• Questions repeated in same paper• Questions matching recent internal evaluation exam question of a

particular college• Question needing specific atypical assumptions, specific atypical steps to

solve in time• Unnecessarily tricky questions• Long question but low mark (for a particular question)• Required data, data table, equations, not provided or unclear, needing

“appropriate assumptions” by the examinees

Practical Problems in Conduction of Semester End Examination(Question typing/printing)

• Too many typos in questions• Figures missing• Wrong figures• Arrows and symbols incorrect, mis-oriented, missing• Dimensions unclear and/or missing• Equations unclear• Data missing in figures• Data in text and figures mismatch• Total mark more or less than full mark• Same question printed on both side, hence number of question papers

inadequate • Inadequate number of questions• Graph paper, charts, code, missing

Practical Problems in Conduction of Semester End Examination(At Exam Center)

• Question transportation from University to Exam Center without protection/security

• Monetary incentive to invigilators and support staff too low• Hot-line not working• No facilities (photocopy, graph papers, codes) etc. unavailable at

Exam Center, especially on public holidays• Security situations at Exam Center• Examinees challenging decisions of Authorized persons inside Exam

Hall• Question corrections almost till the end of exam duration• Chocking and/or breaking of toilet fittings at Exam Center• Answer sheet missing at the end. Everyone submits answer sheets

in the last few minutes.

Practical Problems in Conduction of Semester End Examination(Result Publication)

• Delay in answer sheet distribution, distribute in Pokhara or in Kathmandu?

• concerned subject teachers not available in time, low monetary incentive

• Low HR at university to follow up with answer sheet checkers• Delay in answer sheet returning by specific checker (cannot convert into

relative grading)• Checkers not oriented properly, some apparently considering 60 as “first

division”, no score for process, too little time given for checking• One person checking too many copies• Lack of generally agreed evaluation method• Scrutiny Board Members not available in time• Delay in data entry• Error in data entry

Practical Problems in Conduction of Semester End Examination(Result Publication)

• Internal Marks from colleges not sent in time or sent in piece-meal• Repeated requests from colleges to correct submitted internal

marks• Delay in decisions/recommendations by Subject Committee in

disputed subjects• Complexity added due to mistakes in questions• Exam Committee Meeting infrequent• Result publication takes 60 days, and contains mistakes.• 10 days to apply for rechecking/retotaling (effectively 9 days)• Answer sheets in Pokhara and Kathmandu so long time spent on

finding the requested copies for rechecking/retotaling• Decision on HR for rechecking, person actually coming to university

to re-check takes 15 days. Result becomes final only after 95 days (if everything goes smoothly)

Students’ Complaints• Curriculum out of date or not related to their specialization area. The design of

curriculum are decades old. Delivery of curriculum focused on knowledge transfer only, not useful to get job

• Teachers not trained in teaching methods or qualified to teach• Teachers biased against particular student, particular organization, particular

college• No uniformity across different colleges in depth and width of coverage, teaching

methodology, teaching style, internal evaluation method. “Our college is too strict, while other colleges give marks very liberally.”

• Relative grading system not clear. • One particular college has dominance• Teachers’ close to university officials has more say• Teachers or University officials leak questions• Even the question setter cannot solve all the questions in time. Questions expect

students to memorize long formulas, data and equations. • Answer sheet checking not done properly, no evaluation of solution process.• Sudden change in question format, without any pre-warning.

Students’ Complaints

• Exam Officer’ carelessness, absent students “pass”, exam attendees “absent”, “correctly solved all questions” but “F” or “D” in result

• Retotaling / Rechecking not really done• Exam Routine not properly prepared; “No gap” for difficult subjects • PU Registration Number not provided in time• Difference between result in web and grade sheet• Course registration not possible without result (including result of

retotaling/rechecking) publication before beginning of semester• Exam registration not possible without result publication in time• Assessment or “unit test”/”term test” question of a particular

college in final question.

Colleges’ Complaints• Result not published in time so cannot schedule classes of retake

subjects, course registration not possible• Rules change frequently, often at the pressure of student organizations• No coordination with colleges, when changing rules, and does not

inform colleges after changing rules• University money minded, reduced examinee: invigilator ratio from

10:1 to 16:1, increases risk of answer sheet disappearance, reduces control of exam hall discipline, invigilator cannot go out even for toilet break, or additional answer sheet collection (class room designed for 8 x 3 = 24 students can be used only for 8 x 2 = 16 examinees when conducting exam).

• Letters/submitted internal evaluation data disappear, University asks for same data many times

• University asks for immediate submission of data through email, but later responds that no data sent through regular mail in hard copy

Exam Controller Office’s Complaints• Internal Marks submitted very late, and/or submitted with

mistakes, and repeated requests for corrections• Colleges do not check forms properly before submitting to

the University• Exam Center careless in packaging answer sheet, copy of

more than one subject in same package, mismatch in package content with declaration

• Question paper not well prepared, not balanced as per syllabus, college officials do not orient concerned teachers

• Question paper moderator not serious• Answer sheet checkers do not return answer sheets in time• Biased or not-well-oriented answer sheet checkers, “miser”

checkers

Main Problems

• Internal Mark collection, checking and submission in time

• Question Problem• Result Publication (including re-totaling and

rechecking) in time

Root causes of the problemsCentralized Student Evaluation – Decentralized Education Delivery

• Curriculum outdated and unavailable• Some parts of the curriculum updated but not made available, not circulated in

time to all concerned colleges• No detailed (Micro) syllabus, “education should be broad, but testing

undergraduate students’ performance should not be broad”• Lack of inter-college and college-university coordination• Teachers changing very frequently• Different level teachers in different colleges (from recent graduates to Doctorate

degree holders with many years to experience)• Full time versus part time• Teachers’ ego; no requirement of pedagogic training• Evaluators not properly oriented (Mentality of 32 as pass mark, 60 as first

division)• Confusion on “relative” and “absolute” letter grading systems

Potential Solutions (Schedule)• The Semester End Examination (sometimes also referred as Final

Examination) should start and end as per the Academic Schedule of the Pokhara University.

• The Examination Routine should be published before the beginning of semester classes so that the students know which courses/subjects to register. Alternatively, a fixed examination routine can be published. However, the fixed examination routine may need to be changed periodically as the subjects offered may change with time, or when new programs are offered.

• To prepare the examination routine before the beginning of the semester classes, each college under the University has to provide the list of subjects the colleges are offering in that particular semester as regular subjects well before the commencement of the regular classes. Unless this step is implemented, the existing redundancy of filling Course Registration Form and Examination Form simultaneously will remain.

Potential Solutions (Question)The steps of question preparation are:• preparation of the Terms of Reference (TOR) of question setter and question paper

moderator (one time only)• determination of the subjects in which the University has to prepare question: this step

requires coordination with all the affiliated and constituent colleges• identification of potential question setters and/or question moderator for each subject;

including background check of potential question setters• clear practical guidelines for setting question papers, including format and depth coverage• latest updated detailed curriculum, in coordination with the decisions of the concerned

Subject Committee• contact identified question setters, with the Terms of Reference, requesting them to prepare

multiple sets of question papers, with fixed deadline, following the format and depth coverage, as per latest approved curriculum detail

• collection of the prepared question• initial checking of the prepared question: format and depth coverage balance (based on

weightage (proportion) of question from each chapter as per weightage of chapter in latest approved course detail)

• Contact the question setter if any problem found in the format and/or weightage balance of the question, and resolve any problem found.

Potential Solutions (Question)• Contact the identified question paper moderator, with the TOR, requesting to prepare at

least two sets of questions based on the submitted sets of questions, including expected time required by the examinees to solve/answer the question.

• Type and print all sets of questions prepared by the moderator and let the moderator check the printed version of the questions and sign on the printed versions of the questions. Extra care should be taken in questions involving figures, tables, graphs, symbols, and equations. Sometimes the font becomes too small and the numbers 3 and 8 can look alike. Due to mismatch between the computer and printer, the lines, arrows, text boxes etc. in the figure can move from one location to a different location when printed. When the soft copy of the question includes lines or fonts in different colors, some of the colors may not be printed or printed very lightly in black and white printing.

• Examination Committee decide on the specific question set to be used in each subject.• Print the selected question set for each subject and verify with the signed (by the

moderator) printed question set• Seal the printed questions for later use. Optionally, include one copy of question with the

moderator's signature distorted/over written, in each pack of the question. There should be at least one such question for each subject at each examination center.

• Most of the steps mentioned above are already in practice at the Pokhara University. Some of the steps mentioned above but are yet to be put in practice needs to be discussed and implemented if deemed appropriate.

Potential Solutions (Exam Conduction)• ERN to be changed every semester, ERN not to be related to particular

college• University to deliver sealed question packages to the Exam Centers an hour

before the exam begins. Current practice of college principals carrying questions from university to exam centers is “very risky”.

• Increase security of exam centers, using police force, if necessary. Security personnel to check “Admit Card” before examinees enter Exam center. Police force to be used if examinees start disrupting exam conduction.

• University to collect sealed answer sheets from Exam Center to University.• University to deliver answer sheets to answer sheet checkers, and collect

from them after completion of checking. Current practice of concerned answer sheet checkers collecting answer sheets from university and returning them is “very risky”.

• Not more than 200 answer sheets to any answer sheet checkers; Checked answer sheets to be collected within 10 days maximum. 2 days to enter data, results to be out within 15 days, before beginning of next semester classes.

Potential Solutions (Copy Checking)The copy checking includes the following steps:

• preparation of the guidelines and ToR for the copy checking• identify the potential copy checkers, including necessary back ground check. Make decision

on the copy checkers for each subject. This step should be completed well before the beginning of the examination so that the copies can be dispatched on the same day or next day of the examination.

• contact the identified copy checker and make arrangement to deliver the copy (answer sheets). There is a danger of losing the answer sheets on the road when individual copy checker has to take the answer sheets to his/her home on their own, especially if they are using public transportation.

• Ask the copy checker to provide a break-down of the guideline as to the basis of assigning marks in each question.

• Contact the copy checker on timely basis to complete the copy checking in time.• Collect the copy from the copy checker. • Set up a scrutiny board of experts in the subjects and request them to scrutinize the checked

answer sheets for (a) consistency in checking, (b) bias for or against a particular type of answer, (c) bias for or against answer sheets from a particular examination center, and (d) excessively loose or excessively strict copy checking.

• Make appropriate corrections in the copy checking by the Scrutiny Board of Pokhara University

Potential Solutions (Result Publication)The following is the steps in result publication.• The university system compiles the marks of each subject and enters and

checks the score in the computer system.• Tentative letter grades are assigned as per the university system. As per the

current system, the result of the examinees scoring less than 45% (for graduate programs it is 60%) is separated, and relative grading system is applied on those securing 45% (for graduate programs it is 60%) or more, if the number of examinees in a particular subject is more than 29. Else, an absolute grading system is applied (normally).

• The Examination Board convenes and makes appropriate decisions on the results and uploads the results on the internet, and prints and sends the printed result to concerned colleges.

• Before publishing the results, a checking mechanism should be established to make sure that there are no inadvertent mistakes in the results.

• The result of the semester end examination should be published before the commencement of the next semester regular classes if the spirit of the semester system is to be honored.

Potential Solutions (Maintenance of Secrecy)• The university should main the secrecy of the name of the people (or institution)

involved in different stages of conduction of the examination. If this secrecy is breached it will be difficult to attract qualified persons in the centralized examination system. To maintain secrecy, persons not concerned with a particular phase of the examination should have no access to that phase of examination. If possible, code system needs to be developed, rather than the name of the persons involved.

• In the recent changed socio-political context of Nepal and the penetration of national politics in the grass-root level of education system in Nepal, the conduction of centralized examination has become highly sensitive. Any inadvertent human mistake quickly becomes a disproportionately big issue. For example, mistakes in questions, if not corrected immediately (within 5 to 10 minutes) after the commencement of an examination, can lead to disorderly act by the examinees. Question papers that are considered to be too long or too difficult or out of syllabus can also lead to disorderly acts by the examinees. The University authorities and the persons involved in examination system needs to make sure that the evaluation of the examinees is fair. Everyone concerned needs to be extra careful in discharge of their duties for smooth and reliable centralized examination system.