QUALITY OF EVIDENCE

FRCC Compliance WorkshopSeptember/October 2008

2

Evidence is essential

The Registered Entity has the burden to prove compliance through presentation of quality evidence to a standard or requirement.

“Provide Quality Evidence ”

3

Objective

Objective – What is quality evidence? – Define evidence and the types of evidence.– Describe what constitutes quality evidence.– Define appropriate and sufficient evidence.– Provide a better understanding of evidence and its

role in compliance monitoring activities.

4

Definition of Evidence

Evidence is:– Data or information on which factual

statements can be based.

– A collection of relevant and sufficient information offered to verify a presented

fact.

– Obtained through examining and analyzing records, observing events, or

conducting interviews.

5

Presenting Your Evidence • Auditors gather evidence to draw conclusions on findings in a

logical method based upon factual evidence.

• Facts are to be valid and relevant for the point needed to be proven.

– A fact is something that can be objectively verified, is real and does exists.

• Facts must be verifiable.

– A verifiable fact is one whose data or information can be substantiated by comparison, investigation or confirmation.

• Entities should consider presenting and submitting their evidence in the same manner.

6

Types of Evidence

Evidence can be provided as:

– Physical evidence

– Direct observations

– Corroborating evidence• Corroborating evidence is evidence that tends to support a

proposition that is already supported by some evidence. (Wikipedia 9/11/2008)

• To strengthen or support with other evidence; make more certain (the Free dictionary 9/11/2008)

7

Types of Evidence

Physical Evidence is direct inspection of activities of people, property, or events: – Some examples include:

• Schedules• Charts• Maps• Graphs• One-lines• Manuals• Procedures and Policies• Plans

8

Types of Evidence

Documentation: – Physical forms of evidence which already exists– Can be both external or internal– Some examples include:

• Letters• Contracts • Assessments• Agreements• Spreadsheets• Database Extracts• E-mails • Studies

9

Types of Evidence

Direct Observations:– What an auditor sees during your review.

– Strengthened through corroboration with other evidence.

– By itself is almost always unreliable.

– Is achieved by monitoring of personnel, facilities, equipment, and tools.

10

Types of Evidence

Corroborating evidence is:

– Additional information from interviews, logs, etc provides clarity to make the appropriate compliance decision.

– Used to establish the appropriateness of evidence.

– Tests the reliability of evidence and provides confirmation.

– Some examples include:• Reliability Coordinator or Neighboring questionnaires• Voice recordings• Logs • Interviews

11

An Example of Corroborating Evidence

The auditor requests evidence that a Registered Entity has followed all directives from the Reliability Coordinator

– The Registered Entity states it has not received any directives:• Response is neither sufficient, nor adequate

– Testimonial – Low quality– May not withstand scrutiny

Corroborating evidence could be an e-mail from the Reliability Coordinator stating the Reliability Coordinator did:– Not send any directives to the Registered Entity during the

specified time frame, or– Did send a directive and it was followed

12

Quality Evidence

Quality evidence is evaluated by its:

– Appropriateness• relevant, valid and reliable to support findings.

– Sufficiency• enough information to lead another person to the same

conclusion.

– Quality• high quality evidence to be used as proof of findings.

13

Appropriateness of Evidence

Quality evidence must be appropriate:

– Relevant to the reliability standard requirements:

• Is there enough to persuade the auditor his findings are reasonable?

• Is it logically related to the requirement?

– Valid as a document:

• Is the evidence based upon sound reasoning and accurate information?

– Reliable information: • Can the evidence be substantiated when tested?

14

Example of Appropriate Evidence

An agreement is offered as evidence.

The auditor will seek to determine:– Is this the original document?– Does the agreement contain the Registered Entity

letterhead, approval signature and date, revision date, etc?

– Does the agreement address the subject of the requirement?

– Is this version the current version?

15

Sufficient Evidence

Quality evidence must be sufficient:

– Is there enough evidence to lead an auditor to the same findings that you have reached.

– Strong evidence may be better than a large volume of weak evidence.

16

Example of Sufficient Evidence

• The auditor is provided three documents to determine if a procedure exists and is coordinated with neighboring entities:– A procedure– E-mails – Previous version of procedure

• The above evidence is sufficient as:– A valid procedure is provided which addresses all sub-

requirements even if they do not apply to the audited entity.– E-mails provide evidence the procedure was sent, received and

confirmed as coordinated.– Previous version of procedure shows the procedure existed for

the audit period.

17

Quality Evidence

• The evidence must be of the highest quality to be used as proof of findings.

• Quality evidence should contain: – Documents title– Purpose– Date approved– Revision level– Effective date– Authorizing signatures

18

Testimonial Evidence

Written or oral statements in response to inquiries or interviews.

– Examples are:

• Reliability Coordinator or Neighboring questionnaires• Interviews• Letters sent for confirmation

19

Computer Processed Evidence

Computer evidence can be provided if: – The evidence can be determined to be sufficient and

appropriate.

• Is the information considered quality evidence?• Can the evidence be validated from electronic logs to confirm

creation?• Some examples include:

– Printouts (titled, dated)– Screen shots– Logs (titled, dated)– Study results (titled, dated)

20

Professional Judgment

All evidence reviewed requires some level of professional judgment.

“Professional judgment of the auditors is based on their: – Knowledge– Skills– Experiences– Good Faith Application – Integrity

21

Final Thoughts

Some things to remember:

– Who you are trying to convince, the auditors.

– Compliance is based upon the evidence provided.

– Would you objectively make a determination of compliance or possible violation? 

22

Conclusion

At the end of the review, enough quality evidence must have been provided to:

Soundly substantiate the teams findings

and

Support your expected findings

23

“Provide Quality Evidence ”

24

Questions ?