Auto-Certification Platform (TP202)

Auto-Certification Platform User Guide 2019

1 Official Use Only

VZW Proprietary & Confidential

Auto-Certification Platform (TP202)

User Guide Auto-Certification Platform

Copyrights and Proprietary Notices

The information in this document is subject to change without notice.

This document contains material that is proprietary of and confidential to Verizon Wireless. Disclosure outside VERIZON WIRELESS is prohibited except by written permission, license agreement, or other confidential agreement. This document may not be reproduced in any form or by an electronic or mechanical means.

The information contained herein is prepared by VERIZON WIRELESS solely for use by VERIZON

WIRELESS employees, agents, and customers. Dissemination of the information and/or concepts

contained herein to other parties is prohibited without VERIZON WIRELESS prior written permission.

Disclosure outside VERIZON WIRELESS is prohibited except by license agreement or other

confidentiality agreements


2 Official Use Only


Change History

Version Description of changes Date

Version 1 Initial draft 10/11/2016

Version 2 Added memory requirements 10/25/2016

Version 3 Updated for new UIs and Functionalities Included Installation details Included Account Registration and Licensing details

03/07/2017

Version 4 Added comments with suggested changes/modifications 05/30/2017

Version 5 Updated the document as per the comments / suggestions provided by Steven Mancuso.

07/10/2017


07/17/2017

Version 7 Added the details regarding software and hardware prerequisites.

07/25/2017

Version 8 Added the Cygwin and RTPG installation steps, added appendix.

08/14/2017


08/15/2017

Version 10 Updated the document per the feedback received from the team.

08/28/2017

Version 11 Updated the document with new features which were added to the application.

08/30/2017

Version 12 Updated the document per the feedback received from the team.

09/05/2017

Version 13 Updated the document with new functionality which was added to the application.

09/28/2017

Version 14 Updated the document per the feedback received. 10/04/2017

Version 15 Updated the reference document link. 10/13/2017

Version 16 Added the COMPORT related information, band config details and made few changes regarding reference document link.

10/31/2017

Version 17 Updated the reference document link. 11/08/2017



Version 20 Updated the document for the newer AUTO-CERTIFICATION PLATFORM release

3/6/2018

Version 21 Updated the document for v3.1AUTO-CERTIFICATION

PLATFORM release 5/21/2018

Version 22 Updated the document for v3.4 release 12/10/2018

Version 23 Updated the document for v3.6 release 8/5/2019


3 Official Use Only


Table of Contents

USER MANUAL 4

1 OVERVIEW 5

1.1 About Auto-Certification Platform (ACP) 5

2 GLOSSARY AND DEFINITION OF TERMS 6

3 AUTO-CERTIFICATION PLATFORM SETUP 7

3.1 Software System Configuration 7

3.2 Before You Start 7

4 GETTING STARTED WITH AUTO-CERTIFICATION PLATFORM 7

4.1 Starting Auto-Certification Platform 7

4.2 AUTO-CERTIFICATION PLATFORM Login 9

5 USER INTERFACE 11

5.1 Select a Device Category 11

5.2 Add Device 12

5.2.1 Select Device 15

5.3 Select Test Plans 18

5.4 Select Test Cases 20

5.5 Test Campaign Execution 23

5.5.1 Re-Run the Campaign 26

5.6 Multiple Campaigns 28

5.7 Reporting 29

5.8 Menu Bar 32

6 RETURN INSTRUCTIONS 37

7 PRODUCT SUPPORT 39

8 ADDITIONAL INFORMATION 39


4 Official Use Only


USER MANUAL

This User Manual contains following sections:

Overview describes Verizon's Auto-Certification Platform application and its purpose.

Glossary and Definition of terms provides the terms used in the document.

Getting Started with Auto-Certification Platform section explains how to access Auto-Certification

Platform application.

User Interface section provides a detailed description of the user interfaces and system functions.

Reports section describes how the collected information is presented by the application. Also, how to

access the information and submit device test reports to Verizon.

Product Support section provides the details to contact support.


5 Official Use Only


1 OVERVIEW

1.1 About Auto-Certification Platform (ACP)

Auto-Certification Platform provides test automation via a web based interface. It covers Verizon test

plans and GCF test plans for LTE only devices which are designed for Verizon Network.

The Auto-Certification Platform system elements include:

● Auto-Certification Platform- Self-contained test platform.

● Auto-Certification Platform companion application - To be installed on customer provided

Windows computer

Note: Reference documents for setup diagrams and various testing steps are provided in below link.

https://vzwdt.com/dtportal/sites/default/files/ACP_Files/Reference-Documents-For-Various-Test-Plans-

and-Testing-Steps.docx

https://vzwdt.com/dtportal/sites/default/files/ACP_Files/Reference-Documents-For-Various-Test-Plans-and-Testing-Steps.docx

https://vzwdt.com/dtportal/sites/default/files/ACP_Files/Reference-Documents-For-Various-Test-Plans-and-Testing-Steps.docx


6 Official Use Only


2 GLOSSARY AND DEFINITION OF TERMS

ACP Auto-Certification Platform

TECC Test Entrance Criteria Checklist

VZW Verizon Wireless

PTC Product Test Campaign

OEM Original Equipment Manufacturer

IOT Interoperability Testing

OD Open Development

DT Device Technology

GCF Global Certification Forum

BYOD Bring your own device

UE User Equipment

DUT Device under Test


7 Official Use Only


3 AUTO-CERTIFICATION PLATFORM SETUP

3.1 Software System Configuration

The Auto-Certification Platform application needs to be setup on laptop/PC which has internet

connection. It operates as a service on windows platform.

3.2 Before You Start

Set of prerequisites are required to run the Auto-Certification Platform application.

You may download Installation and Configuration Guide from link below for instructions: https://vzwdt.com/dtportal/sites/default/files/ACP_Files/Auto-Certification Platform Installation and Configuration Guide.pdf

4 GETTING STARTED WITH AUTO-CERTIFICATION PLATFORM

4.1 Starting Auto-Certification Platform

Please follow below steps to launch Auto-Certification Platform application and start the testing.

1. Double click on Start Auto-Certification Platform shortcut from your desktop as shown below.

The window shown below will appear and start loading the application.

https://vzwdt.com/dtportal/sites/default/files/ACP_Files/Auto-Certification%20Platform%20Installation%20and%20Configuration%20Guide.pdf


8 Official Use Only


2. The application will launch with the default port 8080, which cannot be changed. The server

will be started and will create the required files in your “C:/ACP” directory to run the

application.

3. Auto-Certification Platform application will automatically launch using your default web

browser.

Note: If Auto-Certification Platform server restart is required, please double click on Stop Auto-

Certification Platform server first and then start the server again.


9 Official Use Only


4.2 Auto-Certification Platform Login

The Auto-certification Platform application is launched in the device's native web browser.

Enter the user credentials (user credentials that were used during registration) in login screen. First time

users will be required to agree to terms and conditions immediately after login.

Once Terms & conditions are accepted, there will be a series of screens that will show the flow of the

application.


10 Official Use Only


The Software and Hardware check will ensure that user’s laptop/PC has all required software installed

for execution and confirm the Hardware (ACP) box’s connectivity. First time users are required to submit

values for “System Information” to proceed further.

After saving the system information, the application will display the Auto-certification Platform Device

Testing Home Page shown below.




5 USER INTERFACE

The Auto-certification Platform device certification testing is a stepped process as outlined below.

AUTO-CERTIFICATION PLATFORM Device Testing Process

Please refer below screens for step by step process.

5.1 Select a Device Category

This is the first step within the device testing process.

The device categories such as Retail Device, IoT/Open Development and Chipset are listed as below.

Select Device Category

SELECT DEVICE

CATEGORY SELECT A DEVICE

SELECT

TEST PLANS

SELECT

TEST CASES

EXECUTE DEVICE

TEST

TEST RESULTS

1

2




1. Please select the device category for the device that needs to be tested.

A device category can be selected / unselected by clicking on the category tile. Only one device category

can be selected in this step. The selection is required to proceed to the next step.

2. Proceed to the next step for adding a device.

5.2 Add Device

Click on “Add Device” icon as shown in the below screen:

Please enter all the required details and click “Save” to add a device. Please note, you can also add your

device in “Make/Model” field if you don’t have the device you need in that list.




Add Device

1. Select connected device from drop down

2. Click on Refresh icon to update the connected device

3. Select device COM port if available from drop down. If COM port is not available for selection,

then add it as part of PIXIT & PICS parameters. Connect the DUT (Device under Test) with the

laptop/PC and follow below steps for updating DUT COM Port.

1.) Open Windows Device Manager.




2.) Under Modems, select the device under test.

NOTE: The device will be shown under modems when respective drivers are installed.

3.) Note the “COM Port” under the ‘Modem’ tab [Let us call it <UE_COM_PORT>]




Sync TECC Test Plans – If the devices are present in ORBIT and if you have corresponding license, you

will get “Sync TECC Test Plans” checkbox as shown below. Once you select this checkbox, the test cases

will be synced with ORBIT application. These test cases will be part of TECC.

Note: TECC test plans are device specific and a subset of test cases from the default test plans. The

default test plans are not product specific and contains all test cases, which may or may not be

applicable for the device that is being tested.

5.2.1 Select Device

This is the next step within the device testing process.

Device selection is required to proceed to the next step.

Before executing the test, the actual device that needs to be tested such as a Smartphone, Router (etc)

should be connected to the laptop/PC and Auto-certification Platform.

Note - Appropriate OEM device specific drivers need to be installed on the laptop/PC.

1. A device can be selected / unselected by clicking on the device tile. Only one device can be selected in

this step.

2

3 4

6

5 1 8

7




2. If no devices are available for selected device category or specific device that needs to be tested is not

available, you can add a new device using Add Device icon. Please see below section 5.2 for details.

3. “Edit Device” icon allows editing the device information.

4. “Delete Device” icon allows deleting the device. Please see below section for details

5. Return to the previous step to select a different product category.

6. “PIXIT & PICS Editor” allows editing PIXIT & PICS parameters. Below window will open, you can make

the changes and click “SAVE”.

Note - Please make sure to complete the configuration for primary and secondary bands shown below

before executing the test. Provide the configuration based on the need for your test.




If there is a list of same settings which you want to use for no. of devices, you can make the required

changes and click on “Export as excel” button as shown above. You can save that spreadsheet for later

use.

You can click on “Import settings” and import the saved spreadsheet, and use those settings for the new

device that you add.

7. “AT & MMI command”




If there is a list of same commands which you want to use for no. of devices, you can make the required

changes and click on “Export as excel” button as shown above. You can save that spreadsheet for later

use.

You can click on “Import commands” and import the saved spreadsheet, and use those commands for

the new device that you add.

9. Proceed to the next step to select the test plans for testing.

5.3 Select Test Plans

This step allows selecting one or more test plans for device testing. Test plan visibility is based on the

Hardware (ACP box) type and connectivity.

Test Plan selection is required to proceed to the next step in the process.

1

5

2

4

3

6




1. Device test plans (if available from associated TECC) can be selected / unselected by clicking on the

test plan tiles listed under PTC section. Any number of test plans can be selected for testing a device.

2. Default test plans (non PTC) can be selected / unselected by clicking on the test plan tiles listed under

Non-PTC section. Any number of test plans can be selected for testing a device.

3. The number of iterations to execute a test plan can be entered on each test plan tile. The default is ‘1’

iteration for each test case within the test plan.

4. You can return to the previous step to select a different device if needed.

5. Proceed to the next step to select the test cases for execution.

6. You can select test plans by using the SIM Profile based test case selection. The selected SIM Profile

will filter the test plans to display only the test plans related to that SIM filter. The SIM Profile can be

found on your SIM card as displayed below.




5.4 Select Test Cases

This step allows selecting the test cases to be executed for device testing.

Test case selection is required to proceed to the next step in the process. Under each test plan, one or

more test cases can be selected/deselected for execution.

Select Test Cases

1. All test plans selected in earlier step are displayed as a separate tab with associated test cases

displayed under each tab. The test plans (tabs) are displayed in the order in which the test plans will be

executed. The order for test plan execution can be defined by dragging-and-dropping the tabs to

rearrange the execution order. A test plan can also be removed from execution by deleting the

associated test plan tab. The test plan which is shown here is GCF test plan.

2. You can use the “Filter” criteria to filter the test cases by “# of Cells”. And then select the required

test cases for testing. Similar way, you can use the filter for other fields such as “Test Case ID”,

“Description” and “Iteration”.

3. The test cases can be selected / unselected for execution by selecting the corresponding checkbox for

the test cases. Under each test plan, multiple test cases can be selected for execution.

4

5

1

3

9

6

2

7

8




4. The number of iterations to execute the test case can be entered for each test case. The default

selection is ‘1’ iteration, but that can be modified if more iterations are needed.

5. Return to the previous step to select test plans.

6. You can change the settings for each test case by clicking on the settings icon.

Make the required changes in below screen, and click “Save” to save the settings.

7. You can “Filter” the test cases by selecting “Automated” or “Manual”.

For “Manual” test cases, please refer the reference document to follow the required manual steps as

mentioned in section 1.1.

https://vzwdt.com/dtportal/sites/default/files/ACP_Files/Reference-Documents-for-Various-Test-Plans-

and-Testing-Steps.docx

8. You can provide the “Name” to the test campaign. By default, the campaign name will be “OEM Name

Device Model”.

9. Click Play icon to run the test campaign that will start the execution of all test cases selected as part

of the process. Below message box is displayed to confirm if you want automatic re-run for failed test

cases.

https://vzwdt.com/dtportal/sites/default/files/ACP_Files/Reference-Documents-for-Various-Test-Plans-and-Testing-Steps.docx

https://vzwdt.com/dtportal/sites/default/files/ACP_Files/Reference-Documents-for-Various-Test-Plans-and-Testing-Steps.docx




Auto re-run is useful when you want to re-run the failed test cases multiple times to verify. Please select

“Execute” to run the campaign after providing the number for auto re-run.

Some test cases require RTPG/NAT servers to properly execute the test cases. If you are running a

campaign and the RTPG/NAT confirmation message is displayed in the Execute Campaign panel, please

confirm whether or not RTPG/NAT is properly installed on your test bed.

Please refer to the reference document to set up RTPG/NAT servers on your machine.

https://vzwdt.com/dtportal/sites/default/files/ACP_Files/NAT_and_RTPG_Installation_Guide.pdf

https://vzwdt.com/dtportal/sites/default/files/ACP_Files/NAT_and_RTPG_Installation_Guide.pdf




5.5 Test Campaign Execution

This step displays the overall progress of test execution along with the progress of each test plan and

test cases along with the test results.

Test Campaign Execution

1. Displays the overall progress (% complete) of the test campaign execution.

2. Displays the test plan which is being executed.

3. Displays the execution status of test cases - pending, in progress or completed.

4. Displays the live log that shows the progress of the test case.

5. Displays the individual test result for executed test cases - passed, failed or in conclusive.

1

2 3

4

5 6

10

7

8

11 9 12

13




6. Displays the campaign test results summary as shown below.

7. You can click on Auto Re-run status icon to view iteration specific results.

8. You can click on the Stop icon to stop the campaign.

9. “Download” link is enabled to download the “Campaign Results” once the test cases execution is

completed.




10. When you click on the log icon, below window will open to show the completed log for an executed

test case.

- You can click on “DOWLOAD AS HTML” button to download the logs. That would download the HTML

logs. You will have to provide your username and password to view the logs.

- You can also click on “DOWNLOAD WIRESHARK LOG” to download the logs. Please follow below

instructions regarding Wireshark Plug-in to decode RRC and NAS messages.

- The test case log directory is displayed to show where all logs are stored for each test case on your test

bed.

Instructions:

1.) Download lterrc.lua file from this URL: https://vzwdt.com/dtportal/sites/default/files/AUTO-

CERTIFICATION PLATFORM-downloads/2.0/lterrc.lua

2.) Copy lterrc.lua file under Wireshark install directory. Default Wireshark install folder would be ‘C:\Program Files\Wireshark’.

3.) Wireshark install folder will include init.lua file. Edit init.lua file to include following line at the end of the file. Edit to init.lua would need ‘Administrator’ rights. Add the following line as shown in below figure.

dofile (DATA_DIR.."lterrc.lua")

https://vzwdt.com/dtportal/sites/default/files/InstaCert-downloads/2.0/lterrc.lua

https://vzwdt.com/dtportal/sites/default/files/InstaCert-downloads/2.0/lterrc.lua




4.) Step 2 must be performed each time user accepts any updates from Wireshark for installation.

11. Allows to Re-run the campaign.

12. If you want to change the device settings for re-run, please click on settings icon. It will display the

popup window for settings; make the required changes and click “Save”.

Then, click on Re-run, please see below section 5.5.1 for more details.

13. Allows creating a new campaign.

5.5.1 Re-Run the Campaign

When you click on “Re-run” the campaign as part of above step #11, popup window will appear as

below.

You can select one of the options as shown below:




- If you select “Automatic re-run of failed test cases – Number of Cycles” option, you can provide the

number for auto re-run. And click either “Re-run failed” or “Re-run all”. Re-run failed will run only failed

test cases from the campaign while Re-run all will run all test cases from the campaign.

- If you select “Add/Remove Test cases” checkbox and click on either of Re-run option below screen will

appear.

1. You can select additional test cases from the same test plans.

2. You can go to test plans tab to select test cases from different test plans.

3. Once you have selected all required test cases, click Play icon to “Run” the campaign. Popup will

appear to confirm if you want automatic re-run for failed test cases. Please select “Execute” to

run the campaign after providing the number for auto re-run.

2

1

3




5.6 Multiple Campaigns

After starting a first campaign, you can set up a multiple campaigns for the same device or a different

device. These campaigns will run sequentially in the order they are set up.

1. You can create a new camping after first campaign is in progress by following the steps mentioned in

sections 5.1 to 5.5.

1




1. Shows in progress campaign.

2. Show in queue campaign.

3. You can see completed campaign results.

5.7 Reporting

The product's final test results can be submitted to Verizon from the campaign summary report. The

final report could also be submitted later, if more tests are required for the product.

1

2

3




1. After a test campaign execution is complete, the campaign results display a detailed test report. This

includes the test results for each test cases executed from each test plan. If you have selected multiple

“Run#” for the campaign, you will get the details for each Run#.

2. The campaign log directory is displayed to show where the campaign logs can be found.

3. You can click on the “Submit results” button to submit the campaign report to Verizon. A sample of a

campaign report can be seen below:

1

2

3




5.8 Menu Bar

1. The menu bar provides access to the following information: New campaign, Prerequisite check,

Software and box information, All Results, Compare results, Software updates, Help, Email notifications,

Reset password, and Logout.

2. New Campaign: You can create new campaign and perform the testing as shown in above sections.

3. All Results: Below screen will be displayed when “All Results” submenu is selected. It displays all the

campaigns which are created by the user.

You can select multiple campaigns for selected device as shown below and click “Submit” to submit

the results to Verizon.

2

3

1

6

5

7

4




You can preview the report which you are submitting to Verizon by clicking on “Download Report”

(sample report can be found on page 31).

You can click on “Download Report” to download the report for the selected campaigns.

4.

Compare Results: You can select up to three devices to compare their test results against each other.

After selecting a maximum of three devices, you can press the “View results” button to begin comparing

the selected devices’ campaign results as shown below.




5. By clicking on Email notifications submenu, you can setup the email notifications for various actions.

1

3

2




1. Select email action for which you want to send out the email notification.

For example, you can select “Campaign Completed”. Once the campaign is completed, email

notifications will be sent out with the campaign summary report.

2. After entering all the details, click “Save preferences” to save the changes.

3. Close the window.

6. Software Updates: The screen below will be displayed when “Software updates” submenu is selected.

1. Please select the newer version (if available).

2. Click “Upgrade [Selected Version #]”.

You will see below screen with progress indicator while upgrade is in process.

1

2




Once the upgrade is completed, below message box is displayed. Please click “Ok”, and close the current

browser.




Start the application from the shortcut menu; it will open in your native browser. And login to the

application with your login credentials.

7. Logout - You can log out from the application.

6 RETURN INSTRUCTIONS

The following items are included in the package:

Item No. Name Quantity Label

1 Auto-Certification Platform Hardware 1 A

2 BYOD Test SIM 1 B

3 GCF Test SIM 1 C

4 Antenna 8 D & G

5 Power Adapter 1 E

6 Power Cable 1 F




Auto-Certification Platform Hardware and SIMs:

Accessories (Power Adapter, Power Cable, Antennas):




After your 60 days demo period is over, please return Auto-Certification Platform hardware

including all of the accessories listed in above tables (items 1- 6) back to Vienna Channel at below

address.

Attn: Lauren Castillo

1905 Industrial Blvd.

Colleyville, TX 76034

7 PRODUCT SUPPORT

Please reach out to [email protected] for any queries/support.

8 ADDITIONAL INFORMATION

The Global Certification Forum (GCF) is the organizational backbone to GCF Certification. Founded in

1999, this membership organization brings together technical experts from the world's leading

Manufacturers, Operators and the Test Industry. Together members define the certification

requirements that really work for the industry - because they have been developed by the industry. GCF

Certification follows the principles of a supplier's declaration of conformity (SDoC) as defined in ISO/IEC

17050.Please follow this link for more information:

https://www.globalcertificationforum.org/about.html

mailto:[email protected]

https://www.globalcertificationforum.org/about.html

Auto-Certification Platform (TP202)

Documents

Transcript of Auto-Certification Platform (TP202)