EVIDENCE COLLECTION AND ANALYSIS IN PERFORMANCE …iced.cag.gov.in/wp-content/uploads/BT-03/kanawat...

EVIDENCE COLLECTION AND ANALYSIS

IN PERFORMANCE AUDITING

BY

PRITHAVIPAL SINGH KANAWAT

PA_Guidelines2014.pdf

Evidence gathering techniques

EVIDENCE GATHERING

TECHNIQUES

The module contains a broad framework of evidence gathering techniques:

1. Survey

2. Statistical sampling

3. Benchmarking

4. Focus groups

5. Interviews

6. Case studies

Appropriate evidence

gathering techniques

enhances the authenticity

of audit findings

This is not an exhaustive

list

Appropriate other

techniques can also be

used

1.SURVEY

Gathering specific information from a group of

people or an organization by questionnaire

Used in planning and execution phases

Planning – identifying issues or key concerns in

an issue

Execution phase – necessary audit evidence

Used to collect quantitative information to

estimate output or evaluate process of a project

STEPS TO USING SURVEY Decide the objective and

target population

Decide the sample size and method of drawing sample

Decide the nature of survey-direct interview, through post, phone, email

Frame the questionnaire

Test the validity of the questionnaire-use of counter questions, language, words etc.

Administer the survey and collect information

Checking the completed questionnaire

Analyze the results of the survey-qualitative and quantitative techniques, avoid bias, errors

Need for expert in large surveys

Strengths and concerns

EVIDENCE GATHERING

TECHNIQUES


1. Survey


3. Benchmarking

4. Focus groups

5. Interviews

6. Case studies




of audit findings


list

Appropriate other


used

Need for Sampling

It saves time

it involves less cost

In some cases it may not be possible to

check 100%

It may yield better results than 100%

checking in some cases

It means----

Audit sampling is

the technique of evaluating the population

by testing a part of the population

Two types of sampling

1. Statistical

2. Judgmental

Statistical sampling is important as it

gives almost accurate results.

Audit sampling Types

Statistical sampling can be defined as any

sampling procedure that uses the Laws of

probability for selecting and evaluating a

sample from a population for the purpose

of reaching a conclusion of population.

Implication of sampling

If the auditor states that he has 95% audit

assurance it means he is confident of a

statistical sampling technique for the audit

assurance.

In non- statistical sampling he can not

state the accuracy in terms of certain

amount.

Statistical Sampling

Sampling means testing less than 100% of the cases in the population for some characteristic and then drawing a conclusion about that characteristic for the entire population.

Traditionally, auditors use ‘test check’ (or judgmental sampling, non-statistical sampling) approach. This means checking a pre-determined proportion of the cases on the basis of the auditor’s judgment. This sampling technique can be effective if properly designed.

Test check does not have the ability to measure sampling risk and thus audit conclusions reached becomes rather difficult to defend.

For statistical sampling techniques, there is a measurable relationship between the size of the sample and the degree of risk.

Statistical sampling procedure uses the laws of probability and provides a measurable degree of sampling risk. Accepting this level of risk, (or conversely at a definite assurance level) the auditor can state his conclusions for the entire population.

In sum, statistical sampling provides greater objectivity in the sample selection and in the audit conclusion.

Attributes and Variable sampling Statistical sampling may be used in different auditing situations.

The auditor may wish to estimate how many departures haveoccurred from the prescribed procedures; or estimate a parameterin the population. Based on whether the audit objective is todetermine a qualitative characteristic or a quantitative estimate ofthe population, the sampling is called an attribute or variablesampling.

Attributes sampling estimates the proportion of items in apopulation having a certain attribute or characteristic. In an auditsituation, attribute sampling could estimate the existence orotherwise of a error. Attribute sampling could be used whendrawing assurance that prescribed procedures are being followedproperly. For example, attribute sampling may be used to deriveassurance that procedures for classification of vouchers have beenfollowed properly. Here, the auditor estimates through attributesampling the percentage of error (vouchers that have been mis-classified) and sets an upper limit of error that he is willing toaccept and still be assured that the systems are in place. Variablessampling would estimate a quantity, e.g., the underassessment in atax circle.

Attribute sampling for test of

controls.

Attribute sampling is defined as a

sampling plan in which the sampling unit

is defined as an account balance, a

purchase invoice or any other constituent

of the accounting population.

Attribute sampling is required to test

whether the internal control is working or

not.

Stages involved in Attribute

sampling

Determining the sample size

Selecting the sample

Performing test of control procedures

Evaluating the test results

How to calculate the sample size?

Define population

Determine the controls

Define exception/error

Stages Attribute Sampling

Contd...

Determine the tolerable error

– This is the maximum deviation the auditor is willing to tolerate and still conclude that the audit assurance is there

Determine the expected occurrence rate

Select the statistical table

Locate the TE

Read the table in the that column to the line that contains error to obtain the sample size

Sampling Methods

Random Selection - each item in the population has a equal chance of selection.

Random selection assumes that the population is homogeneous.

Stratified selection - Population is non-homogeneous. Population is sub-divided into homogeneous groups and then a random sampling is done on the groups, ensuring a better representative sample.

Auditor to use his judgment in determining which kind of sampling is best suited to his audit job

Random selection

In this each and every item has an equal

chance of selection

For.Eg.

If 5 items out of 1000 are to be selected, it

can be 1,4,6,8,9 or 14,25,36,998.999.

Random number tables can be used.

Random Sampling Methods

Simple random sampling :- each member of the

population has an equal chance of

selection.Useful when the population is uniform.

Stratified random sampling :-Population is

divided into strata and random sample is drawn

from each strata. This is useful when there exists

stratification in the data and the method will

ensure that members from each strata are

represented

Systematic sampling :-Population members at equalintervals get selected. Often it might be easier todraw systematic sample than random sample. Thiswould be particularly useful when cases are orderedby size, type or region. Then by selectingsystematically one can ensure that cases havingdifferent attributes have been adequately represented.

This method should be used for samples to evenlycover a population range.

Multistage sampling :- In this sampling one canextract random sample of a sample. Ex. – Samplestates, Sample districts within select states, sampleblocks within selected district, sample villages withinselected blocks, sample beneficiaries within selectedvillages. At each stage a suitable method of samplecould be used

Random Sampling Methods

Methodology in selecting sample

Once the method is finalized design the

actual sample.

For simple random sampling the following

method could be used. For other types of

sampling it is advisable to consult expert.

Simple Random Sampling

(Attribute Sampling)

Used when audit desires to estimate an attribute in a population.

Useful for testing internal control

If errors above certain level- auditor may conclude – Internal Controls are weak

The attribute, which the auditor is interested here are errors/ abbreviations from process

Basic stages involved are :-

a) Determining the sample size,

b) Selecting the sample and perform

substantive audit tests on the sample,

c) project the results



(a) Determining sample size

After defining the target population and the

attribute that audit wishes to test, the size of the

sample required to be tested need to be

determined. This can be done with through

understanding of the following parameters :-• Precision

• Confidence level

• Occurrence rate (p) or population proportion



Determining sample size

Parameters

Precision (E)

Audit test on the sample will throw up an

estimate of the attribute for the population. The

true population value of the attribute could be

more/less than this estimate. The gap between the

sample estimate and the actual population is the

precision. The auditor has to decide the precision

he desires to provide in his estimates.


Parameters

Confidence Level

The confidence level or the level of assurance

that audit needs to provide is to be defined.

Confidence level states how certain the auditor is,

that the actual population measure is within the

sample estimate and its associated precision

level. In case of performance audit, this level can

be taken at 95 percent.


Parameters

Occurrence Rate

The rate (p) or population proportion which

is the proportion of items in the population

having the attribute that audit wishes to test.

This is based on the judgment of the auditor.

Basic stages involved are :-

a) Determining the sample size,

b) Selecting the sample and perform

substantive audit tests on the sample,

c) project the results



(b) Simple Random Sampling

( Attribute sampling)

The sample could be selected using random number

tables or through computers. Auditing software, e.g.

IDEA is an efficient tool for sample selection. Once the

sample selected, identified audit tests are to be applied on

the sample. The proportion of the sample having the

attribute that is under test is determined through audit.

( c ) Projecting the results

The test results are to be projected to the population. Precision can be calculated at the desired confidence level and sample size. Loading the precision on the sample value the upper estimate for the population can be made.

In example of testing internal controls, this estimate is the maximum error/aberration that is expected the the given confidence level. In case this estimate is less than the threshold of error/ aberration that the auditor can tolerate, the auditor can place assurance on the controls. When estimate is higher than the tolerable error/ aberration the auditor can not derive assurance from the controls. The auditor may, in such situations reduce the assurance he derives from the controls and increase the assurance required from substantive tests.

EVIDENCE GATHERING

TECHNIQUES


1. Survey


3. Benchmarking

4. Focus groups

5. Interviews

6. Case studies




of audit findings


list

Appropriate other


used

3. BENCHMARKING

Benchmarking is a process for measuring an organization’s performance or process against such organizations that consistently distinguish themselves in the same categories of performance

External or internal benchmarking can be done to identify opportunities of achieving better economy, efficiency and effectiveness

Can be used in planning and execution phases

Planning-setting the audit criteria

Execution-cause effect analysis by comparison

Steps for using benchmarking Deciding the aspects of

performance or process that will be benchmarked

Deciding the type of comparison and benchmarking partners

Collect data

Determine the performance gap

Framing conclusions and recommendations for betterment

Strengths:

Objective review of processes, policies and systems

Objective data on methods of operation

Better ways of operating

Supports recommendations for change

Target for improvement

Concerns:

High degree of skill

Acceptability of the findings

EVIDENCE GATHERING

TECHNIQUES


1. Survey


3. Benchmarking

4. Focus groups

5. Interviews

6. Case studies




of audit findings


list

Appropriate other


used

4. Focus groups

A qualitative research technique

Selection of some individuals to discuss specific issues in an informal setting

Reactions of the group are used to explore attitudes, beliefs, perceptions, and problems

Causes and effects of problems to achievement of economy, efficiency and effectiveness

6 to 12 participants, auditee staff or beneficiaries of the programme assisted by facilitator, 90 to 180 minutes

Used both during the planning and execution phases

Execution phase for validation of findings

Steps for using this technique

Selecting a facilitator

Determining the number of focus groups

Deciding the participants of the focus groups

A topic guide

Conducting the focus group

Recording the results of a focus group

Analyzing the results of a focus group

Strengths:

Different perspectives, opinions and ideas

Express opinions freely in a group

Less expensive

Concerns:

Not selected statistically

Cannot be projected to the population at large

Needs to be backed up

EVIDENCE GATHERING

TECHNIQUES


1. Survey


3. Benchmarking

4. Focus groups

5. Interviews

6. Case studies




of audit findings


list

Appropriate other


used

5. Interviews

An interview is a question-answer session to elicit specific information

Structured or individual (unstructured)

Structured-specific wording and are asked in set order

Open ended or closed ended questions

Elicit explanations, impressions and opinions

Used both in the planning and execution phases

Planning stage-identifying potential key issues

Execution phase-to corroborate evidence

Steps for using interviews

Preparing the questions-determine the minimum data to be obtained

Determining the interviewees-structured interview selection by statistical sampling

Conducting and recording the results-video audio recording, notes of interview; expert interview and auditor interview in complex/controversial issues

Analyzing the results

Individual interview-strength:

Flexible

Used for probing

New areas broadens the perspective

Concerns

Weak evidence needs collaboration

Problem in keeping focus

Can guess answers, not well formulated

EVIDENCE GATHERING

TECHNIQUES


1. Survey


3. Benchmarking

4. Focus groups

5. Interviews

6. Case studies




of audit findings


list

Appropriate other


used

6. Case studies

Case study is the examination of a selection of incidents, events, transactions or item in order to understand or examine a programme or activity. It is an in depth study of individual cases to explore the audit issues

Used in the planning and execution stages

Case study can help to develop key questions to be focused on later in the main study

Through examination of specific cases and can identify reasons for bad performance

Causes of bad performance and impact of a specific act

Steps for using case studies

Deciding the issue to be audited: specific question to be studied using the criteria need to be identified

Selecting the areas to be studied, judgmental selection, rationale clear and defensible

Conducting the case study: PA techniques used, document review, interviews, focus groups

Analyzing the results: qualitative and

Strengths:

Cheaper than studying a larger sample

More accurate as greater in depth investigation is possible

Problems, cause and effect analysis and pragmatic recommendations possible

Concerns:

Judgment in selection

Open to bias

Difficult to determine generic or an aberration

1

Objective of this session is to test the

understanding of the participants with

reference to the previous Theory Session

which covered the following:

Competence, relevance and sufficiency of

evidence

Factors affecting the evidence

Types of evidences

Audit criteria & evidence

Sources of evidence

Sampling techniques

iCED Jaipur

2

Cont…

Evidence analysis

Evidential process

Documentation

Characteristics of good quality working

papers

Quality assurance in evidence gathering,

analysis and working papers

iCED Jaipur

3

Evidence and Documentation

Q.1. What is meant by Audit Evidence?

iCED Jaipur

4


Solution:1

Audit evidence is the information to be

collected to support audit findings. The

recommendation and conclusions in the p.a.

report are framed based on the audit

evidence.

iCED Jaipur

5


Q.2. What are the techniques followed to

collect Audit Evidence and Characteristic of

Good Evidence?

iCED Jaipur

iCED Jaipur 6


Solution:2

The evidence that are gathered should becompetent i.e. valid and reliable and actuallyrepresent the observations of audit

The evidence should be relevant and have a clearbearing and logical relationships to auditobjectives.

Sampling techniques should be carefully chosen.

Inspection,observation,enquiry and conformationare good procedure of techniques followed

iCED Jaipur 7


– Sufficiency and reasonableness is the other

requirement of valid evidence to strengthen the

performance audits

– The following documents were relied upon as

substantive evidence in the MAT review

iCED Jaipur 8


Q.3 Explain competence relevance and

reasonableness of Audit Evidence?

iCED Jaipur 9


Solution: 3

A valid and reliable evidence is called competent evidence.Documentary evidence are move valid than oral evidencesimilarly evidence obtained by direct observation are morereliable than indirectly obtained. Oral evidence should becorroborated in writing and attestations. Entity generatedinformation is a reflection of reliability of internal controlsystem. Photo copies are to be certified with reference to.their source and identity.

iCED Jaipur 10


Relevance of the evidence is determined the clear andlogical relationship it bears to the audit objective andcriteria. The objectives and criteria are known when thesubject is selected for p.a. Relevant evidence, its natureand location are indicated in advance in accordance withthe audit objectives and criteria to support the views in p.a.

While sufficiency of evidence is quantitative, recommendations is qualitative in p.a. Evidence is sufficient if the fact is carried out on a reasonable population selected on representative basis. It should be convincing when the observation is sensitive and controversial.

iCED Jaipur 11


Q.4. Classify the following evidences into

competence, relevance and reasonableness?

iCED Jaipur 12


(i) sample selected was not representative

(ii) evidence collected relate to insolated

occurrence

(iii) does not establish cause and effective

relationship

(iv) conflicting

(v) biased

(vi) obtained orally

(vii) unattested photo copy

iCED Jaipur 13


Solution: 4

(i) sufficiency

(ii) sufficiency, validity

(iii) sufficiency, reliability

(iv) reliability

(v) reliability

(vi) importance

(vii) competence

iCED Jaipur 14


Q.6 A well prepared documentation of

evidence is a measure of quality assurance

of P.A. Do you agree?

iCED Jaipur 15


Solution 6

Meticulous documentation of evidence in

working paper include:

Planning

Work performed

Findings of Audit

It should be sufficient enough to enable

EVIDENCE COLLECTION AND ANALYSIS IN PERFORMANCE …iced.cag.gov.in/wp-content/uploads/BT-03/kanawat...

Documents

Transcript of EVIDENCE COLLECTION AND ANALYSIS IN PERFORMANCE …iced.cag.gov.in/wp-content/uploads/BT-03/kanawat...