Conformance 101 - The Open Group• UoC Variants are versions of a UoC generated to a specific set...

FACE™ is a Trademark of The Open Group

Conformance 101

February 2016

2

Purpose of this Presentation

•  Overview of the FACE Conformance Program and Related Policies and Procedures

•  Audience: – New members of the FACE Consortium

– New participants in the Conformance Subcommittee

– Other interested parties

3

What is FACE Conformance?

•  FACE Conformance is the assessment of a Software Item, known as a Unit of Conformance (UoC), to the applicable Conformance Requirements contained in the FACE Technical Standard

•  Those Applicable Requirements are determined based on the segment and profile selected in the design of the particular UoC

•  Verification of Conformance is conducted utilizing automated test tools and inspection of design and test documents

•  The specific requirements, method of verification, and associated verification evidence is detailed in the Conformance Verification Matrix (CVM)

4

What can be Certified as FACE Conformant? •  Certification is for Units of Conformance (UoC) or

UoC Packages – Defined on next slide

•  There is – No “compliance”; things are either “certified

conformant” or not. – No FACE certification for entire systems

• Systems can be comprised completely of Certified UoCs or a mix of Certified UoCs and other software

– No FACE certification for independent libraries, runtimes, frameworks

•  These can be included in a certification of a larger set

5

What is a Unit of Conformance (UoC)? •  A Unit of Portability (UoP) is …

–  Items that fit completely within one of the FACE Architecture portable segments

•  Platform Specific Segment

•  Transport Services Segment

•  Portable Component Segment

•  A Unit of Conformance (UoC) is … –  Items that fit completely within one of the five FACE Architecture

segments •  Platform Specific Segment (also a UoP)

•  Transport Services Segment (also a UoP)

•  Portable Component Segment (also a UoP)

•  Operating System Segment (is not a UoP)

•  I/O Services (is not a UoP)

6

What is a Unit of Conformance Package?

•  A Unit of Conformance (UoC) Package is … –  A Collection of UoCs meeting the FACE Standard definition of

packages •  Same Partition

•  Allowable Segments

7

What are UoC variants?

•  UoC Variants are versions of a UoC generated to a specific set of functionality and target environment from the same UoC source code. UoC Variants which conform to the Technical Standard may be certified and listed together on a single Conformance Certificate. If the Software Supplier is submitting UoC variants, conformance evidence must detail the conformant variants and their differences.

8

Conformance Processes

•  FACE Verification is the act of determining the conformance of an implementation to specification requirements. Verification is handled through an entity known as a Verification Authority (VA), a technical expert on the FACE Technical Standard and Verification process and approved by the FACE Consortium Steering Committee

•  FACE Certification is the process of applying for a FACE Conformance Certificate once verification has successfully been completed. Certification is processed through the FACE Certification Authority (CA)

•  FACE Registration is the process of listing FACE Certified UoCs in a public listing of FACE Certified UoCs known as the FACE Registry. The FACE Registry is accessed from the FACE Landing Page

9

Conformance Program and Processes

Software Supplier

FACE Verification

Authority (VA)

FACE Certification Authority

(CA)

FACE Library Administrator

(LA)

Initiate Verification Initiate

Certification Initiate

Registration

FACE Verification FACE Certification FACE Registration

10

Driving Factors

•  Software Supplier is in control – Software Supplier initiates the steps

– No listing of UoCs “in process”

– Software Supplier provides information for the Registry (the public listing)

– FACE Registry does not contain a way to download; Software Supplier must be contacted

– Software Supplier IP is protected

11

Driving Factors

•  No Functional or Performance Testing – Interfaces are tested

– Other Verification Evidence is inspected (evaluated)

– Functional Testing is assumed as part of other development processes and is not required for FACE Conformance

12

Driving Factors

•  Certification is for Portable Software structure – The source code and design are certified conformant

to the FACE Technical Standard – One instance of the object code, compiled for the test

environment, is tested for conformance to the FACE APIs and data model

– Recompiling to a different target does not cause a loss of FACE Certification

13

Roles

•  Software Supplier: The Software Supplier is anyone providing software (UoC) to be certified. This may include the original software developer, an integrator, or another entity wishing to certify software developed from another party.

•  FACE Verification Authority: A FACE Verification Authority (VA) is one of several organizations approved by the FACE Consortium to evaluate software against the FACE Technical Standard. The VA is an expert on the FACE technical standard and verification process. The VA conducts or witnesses conduct of the For-the-Record Test, utilizing an approved Conformance Test Tool, and inspects the Verification Evidence.

14

Roles

•  FACE Certification Authority: The FACE Certification Authority (CA) is the singular organization approved by the FACE Consortium that can provide a FACE Conformance Certificate

•  FACE Library Administrator: The FACE Library Administrator manages a listing of FACE Certified UoCs known as the FACE Registry

•  FACE Trademark Licensor: The FACE Trademark Licensor issues the FACE Conformance Certification Trademark for Certified Units of Conformance and Certified Unit of Conformance Packages

15

Conformance OV-2

SWVerifica+onPackage

Verifica+onResultsPackage

ConformanceCer+ficate

Cer+fica+onPackage

So9wareSupplierCer+ficateID

Verifica+onAuthority(VA)

Verifica+onResultsPackage

Verifica+onReten+onRepository

ConformanceAuthori+es

ReferenceRepository

Cer+fica+onAuthority(CA)

LibraryAdministrator

Consor+umDocumentsToolsPublicSDMSnapshot

Consor+umDocum

ents

Registra+on

Registry

LibraryRepositoryRequirementsandProceduresDocuments

Cer+fica+onReten+onRepository

LibraryRepositoryRequirements

LibraryRepositoryRequirements

SharedData

Model

ITARSDMSnapshot

16

Key FACE References

•  Technical Standard – Requirements for the FACE Architecture

•  Conformance Verification Matrix – Guidance and Verification Methods

•  Conformance Policy – Policy for certification of UoCs

•  Conformance Certification Guide – Guidance on the Policy and Program

17

Software Supplier Provides

•  The Unit of Conformance (UoC) for Verification – Object code, supporting files, and verification evidence

–  Supplied to the Verification Authority (VA) only

–  All UoCs included in a UoC Package

•  Conformance Statement –  Identification of the Unit of Conformance (UoC) – How the Unit of Conformance (UoC) meets FACE Technical

Standard requirements

•  Registry Entry – Description of the Unit of Conformance (UoC), associated

conformant UoC variants, and UoC Package as applicable

– Contact Information for Obtaining the Certifed Unit of Conformance (UoC) or Certified UoC Package

18

VA Provides

•  Verification Statement – Indication that the UoC Passed Verification

– For each UoC comprising a UoC Package, as well for the UoC Package itself

•  Failed Verification is communicated to the Software Supplier specifying what and why failure occurred

19

CA Provides

•  Conformance Certificate – Indication that the Unit of Conformance (UoC) Passed

Certification – For each UoC comprising a UoC Package, as well for

the UoC Package itself

•  Failed Certification is communicated to the Software Supplier with sufficient detail for the problems to be resolved

20

FACE Conformance Program Steps and Processes

Software Supplier

FACE Verification

Authority (VA)


(CA)


(LA)



Registration


Conduct Preparation

21

FACE Landing Page

Conformance Preparation

Software Supplier

FACE Reference Repository

Supplier Obtains References and Tools •  FACE Technical Standard •  Reference Implementation Guide (RIG) •  Automated Tools, SDK, ITK •  Conformance Certification Users Guide •  Conformance Policy •  Verification Matrix •  Matrix Users Guide (MUG) •  Conformance Test Suite

FACE Conformance Tab

22

FACE Library Overview

LibraryAdministration MainLandingPage

Registry

ReferenceRepository

ProductRepositories

· Certified FACE Conformant Software and associated artifacts

· Dev/Test Tools· Business Guide· Technical Standard· Contract Guide

VerificationRetention

Repositories

· Artifacts submitted for FACE conformance verification / certification

CertificationRetentionRepository

Library Portal

FACE Library Infrastructure

CA SiteVA Site Vendors’ Site

PR/CRSystem

· Problem reports and change requests for Consortium artifacts and tools

· Manages conformance and registration workflow

· VA CoP

· FACE Consoritum information

· FACE Events· Procurements

· Listing of FACE Certified UoCs

· Searchable Metadata

23


•  Identify FACE Criteria •  Number of UoCs for each:

o  FACE Architecture Segment o  Profile (General, Safety,

Safety Extended or Security) o  Supporting Libraries for UoC o  Test Environment

24


•  Supplier provides Verification Evidence o  A trace of the FACE requirements to specific

documents supporting the requirements o  Required for all items in the Tech Standard

identified as needing inspection in the Conformance Verification Matrix (CVM) including applicable conditional requirements

Verification Needed (Y or N)

FACE Segment

Technical Standard for the FACE Reference Architecture Edition 1.0 Verification Method

Conformance Artifacts (DID or

equivalent)

SW Supplier Artifact Cross-

Reference

Verification Notes Conditional Reqs

N 3.5.6 PSS Segment Requirements

Y

PSSS 9. All communication with the IOSS shall go through the I/O Services Interface. Test Test Suite

Y

PSSS 10. Messages communicated through the I/O Services Interface shall be in the format defined in Section D.11.

Inspection SDD

Y PSSS 11. All components of the PSSS shall use the interface defined in Section 3.11, Section 3.12, or

Section 3.13 to access the functions provided by the OSS. Test Inspection

Test Suite SAD SDD

Inspection is only of Java frameworks or Ada run-times.

25


•  Supplier Selects a Verification Authority o  List of Approved VAs from the Landing Page o  Meets supplier needs

o  Not limited to Internal Verification o  Willing to perform verification for the UoC’s

applicable FACE Architecture segment, e.g., Operating System

26

FACE Verification Authority (VA)


Software Supplier

FACE Certification Authority (CA)

Supplier Establishes Legal Agreements

•  Conformance Certification Trademark License Agreement with TM Licensor

•  Verification Agreement with Selected VA •  Certification Agreement with CA

FACE Trademark Licensor

27

Conformance Verification Process

Software Supplier

FACE Verification

Authority (VA)

Initiate Verification

•  Select & Establish Contractual Relationship with VA

•  Develop SW Verification Package o  Verification Agreement o  Verification Evidence o  Conformance Statement o  SW Product Set

28


•  SW Verification Package

o  Verification Agreement - defines the conformance verification services to be provided by the VA and acceptance by the SW Supplier to provide the required verification evidence and SW Product Set

o  Verification Evidence - supporting verification documentation submitted by the SW Supplier to provide evidence of FACE Conformance to the applicable conformance requirements of the Technical Standard that are not directly tested by the Test Suite. The verification evidence is organized to correlate with the specific conformance requirements and verification approach contained in the applicable segment of the Conformance Verification Matrix

29


•  SW Verification Package (cont)

o  Conformance Statement – SW Supplier’s response to a standard questionnaire, tailored to the appropriate Segment of the Technical Standard, structured to obtain precise identification of the software product and conformance evidence. It includes SW product description documentation to uniquely identify and configuration manage the SW product through the conformance process. The Conformance Statement identifies the specific edition of the Technical Standard against which the SW product is being certified, the applicable set of conformance requirements, the corresponding Conformance Verification Matrix version, and the version of Conformance Test Suite used for verification. If the Software Supplier is submitting UoC variants, a Conformance Statement must detail the conformant variants and their differences.

30


•  SW Verification Package (cont)

o  SW Product Set – Includes object code representing all source code compiled for the test environment, appropriate data model files, associated information for set-up with interfacing segments, and minimum computer operating environment requirements

If verification will include UoC Variants, the Software Product Set must include multiple versions of the object code. The versions of object code files must be representative of all UoC Variants for which verification is sought. The Software Product Set must include an analysis of this coverage.

31


Software Supplier

FACE Verification

Authority (VA)


FACE Verification

•  Inspect SW Verification Package

•  Evaluate the Verification Evidence

•  Conduct/Witness FTR test using Approved Conformance Test Suite

•  Issue Verification Statement

•  Archive Data

32

Conduct/Witness the FTR Test

•  Uses the Applicable Approved FACE Conformance Test Suite

•  “Witness” means watching execution of the FTR test – not inspection of test results or test report

•  Evaluates the Object Code through Linking (not executing) the product code

•  Evaluates the Data Model for Conformance (IDL in 1.0)

•  Does no functional testing; only ensures the appropriate FACE APIs are present and correct

33

Evaluate the Verification Evidence

•  Inspect the trace from Technical Standard to specific locations in Designs, Requirements, Test Procedures, Test Reports, etc.

•  Ensures the product meets requirements not covered by the Conformance Test Suite (FTR Test)

Verification Needed (Y or N)

FACE Segment

Technical Standard for the FACE Reference Architecture Edition 1.0 Verification Method

Conformance Artifacts (DID or

equivalent)

SW Supplier Artifact Cross-

Reference

Verification Notes Conditional Reqs

N 3.5.6 PSS Segment Requirements

Y

PSSS 9. All communication with the IOSS shall go through the I/O Services Interface. Test Test Suite

Y

PSSS 10. Messages communicated through the I/O Services Interface shall be in the format defined in Section D.11.

Inspection SDD

Y PSSS 11. All components of the PSSS shall use the interface defined in Section 3.11, Section 3.12, or

Section 3.13 to access the functions provided by the OSS. Test Inspection

Test Suite SAD SDD

Inspection is only of Java frameworks or Ada run-times.

34


Software Supplier

FACE Verification Authority (VA)


FACE Verification Verification Retention Repository

Verification Results Pkg & SW Verification Pkg

35

Conformance Certification Process

Software Supplier


(CA)

Initiate Certification

•  Establish Contractual Relationship with CA

•  Submit Legal Agreements o  Certification Agreement o  Trademark License

Agreement

•  CA Requests Verification Result Package from VA

FACE Verification

Authority (VA)

Verification Results Package

36


Software Supplier


(CA)


•  Ensure legal agreements are in place (Certification Agreement and TMLA)

•  Review Conformance and Verification Statements for completeness and correctness

FACE Certification

37


Software Supplier

FACE Certification Authority (CA)


Certification Retention Repository

FACE Certification Conformance Certificate, Conformance Statement, Verification Statement, TMLA

38

FACE Registration Process

Software Supplier


(LA)

Initiate Registration

•  Submit Product Description & Conformance Certificate ID to Library Administrator

39

FACE Registration Process

Software Supplier


(CA)


(LA)

Initiate Registration

FACE Registration Conformance Certificate ID

•  Request & Receive conformation of Conformance Certificate from CA

•  Populates FACE Registry with Product Description & Conformance Certificate

40

Conformance Program and Processes

Software Supplier

FACE Verification

Authority (VA)


(CA)


(LA)



Registration


41

Conformance Packages SW

Verification Package

Verification Results Package

Verification Retention Repository

Certification Package

Certification Retention Repository

Verification Agreement X X

Verification Evidence X X

SW Product Set X X

Conformance Statement X X X X X

Verification Statement X X X X

Certification Agreement X X

TMLA X X

Conformance Certificate X

42

Other Conformance Topics

•  Recertification/Modification of a Certified UoC

•  PR Process

•  Appeals

•  Auditing

43

Recertification/Modification of a Certified UoC

Section Type of Modification

Verification Requirement

Certification Requirement

5.1.1 Conformance maintenance release

Delta-verification Certification information update

5.1.2 Software modification release

Delta-verification New certification

5.1.3 Renamed UoC None Certification information update

5.1.4 Modification of FACE Registry information

None Certification information update

5.1.5 Other modifications

As determined by the VA

As determined by the CA

44

Recertification/Modification of a Certified UoC •  Conformance Maintenance Releases are the result of

modifications made to the UoC because of conformance-related issues that arise following FACE Conformance Certification; for example, conformance-related problems that arise during system-level test and evaluation or later during the porting of the UoC. Regardless of whether the problem was due to an improper implementation of a Conformance Requirement or a problem with the Technical Standard, the problem solution required a modification to the UoC. The Software Supplier will provide an impact assessment report resulting from the modification to the Software Verification Package contents and resubmit the Software Verification Package with the changes incorporated. The VA will conduct an assessment on the report and updated submittal, determine whether further information or testing is required, evaluate any additional data or test results, and upon satisfactory results issue a delta-verification assessment report to the CA. The CA will re-issue the FACE Conformance Certificate.

45

Recertification/Modification of a Certified UoC •  Software Modification Releases are the result of any

software changes made to the UoC not associated with a conformance maintenance release. The requirement for change could be due to Software Supplier market assessment or customer demand.

•  For these changes, the Software Supplier must submit an impact report on the change(s) and submit a new Software Verification Package. The VA will determine the level of verification required.

•  Upon successful conformance verification, the process will follow the certification process for new UoCs.

46

Recertification/Modification of a Certified UoC •  Renamed UoC If a FACE Certified UoC is to be renamed, with no change to the UoC, the Software Supplier may request such changes at any time by contacting the CA. The Software Supplier will be required to affirm to the CA that there has been no change other than name. Renaming a UoC will additionally require an update of FACE Registry information. •  Modification of FACE Registry Information If the Software Supplier wishes to modify information in the FACE Registry that has no impact on the conformance of the FACE Certified UoC – such as contacts, UoC descriptive text, or other metadata – the Software Supplier may request such changes at any time by contacting the LA.

47

Recertification/Modification of a Certified UoC •  Other Modifications Except where specifically stated in this document, any other variant of a FACE Certified UoC which is deemed by the CA to have a material effect on the conformance of the UoC to the Technical Standard constitutes a new UoC, which will be subject to the full conformance verification and certification processes.

48

Problem Reporting (PR) Process

A Problem Report (PR) may be submitted against any problem encountered with the Technical Standard, FACE Conformance Test Suite(s), or conformance verification and certification processes that inhibit or will inhibit the conformance effort. The PR submitter will utilize the Consortium-wide Problem Report/Change Request (PR/CR) process.

49


The types of problems that may be reported include:

•  Errors, conflicts, or ambiguities in the Technical Standard. •  Errors in the test suite(s) used to assess conformance with the

specifications; specifically, in the FACE Conformance Test Suite(s), or other test suites referenced by the FACE Conformance Program (if any). Errors, conflicts, or ambiguities in the set-up requirements of the FACE Conformance Test Suite(s) may also be addressed through PRs.

•  Errors, conflicts, or ambiguities in the conformance verification and certification processes; specifically, those related to the Registration process, legal agreements, and completion of the Software Verification Package (set of artifacts to support conformance verification).

50


A submitter may file PRs to address issues that arise for any of these items. The identity of the submitter will be protected throughout this process. The PR is used specifically for the types of errors listed above which are inhibiting the conformance effort. For general questions on the conformance verification and certification processes, running the test suites, or other problems not covered above, the CA can provide information on obtaining further assistance. Issues not resolved through this assistance may be followed-up with a PR.

51

Problem Reporting and Approved Corrections Flow

Product CCBFACE CCBFTGSteering CommitteePR/CR AdministratorSubmitter

Entry

Proposal

More Information

ConfirmRejection

INITIATED

INITIATED

RETURNED

RECOMMEND REJECTREJECTED

PROPOSED

REJECTED

Rejected

Proceed With

Correction

Provide Feedback

Publish Correction

Satisfied

File Appeal

Proceed with New

Information

Triage

Yes No

Schedule

SatisfiedYesNo

Approve Correction

52

Appeals

Occasions that may give rise to an appeal include, but are not limited to, the following:

•  The Software Supplier disagrees with the resolution of a Problem Report (PR).

•  The Software Supplier disagrees with the CA’s grounds for denying the award of certification or the removal of certification.

•  The Software Supplier of a FACE Certified UoC disagrees with a formal notification from the CA of the need to rectify a non-conformance. The incidence of a UoC having been found to be non-conformant will be handled on a case-by-case basis under Conformance Maintenance Release policies.

•  Appeal requests will be made to the Steering Committee in writing within 90 calendar days of issue occasion.

53

Appeals

Request Appeal

Yes

No

Appeal Granted

Submitter Steering Committee

Appeal DeniedNo

Changes Made

Appeal AcceptedChanges Made

54

Audits

•  The primary purpose of the audit is to ensure the overall integrity of the FACE Conformance Program and the software contained in FACE Product Repositories. An audit may be conducted on a Software Supplier, a VA, or the CA.

•  Audits will be conducted or directed by the Steering Committee (SC) when deemed necessary. The SC designates an auditor to perform an audit. The auditor may request the applicable information from any of the following entities in order to conduct the audit:

•  Software Supplier – Verification Package and configuration-

managed records •  VA – verification records •  CA – certification records

55

Audits

Conformance 101 - The Open Group• UoC Variants are versions of a UoC generated to a specific set...

Documents

Transcript of Conformance 101 - The Open Group• UoC Variants are versions of a UoC generated to a specific set...