Running in the Pipeline and How to Read the Results

Michelle L. Crane and Frédéric FusierDevOps Transformation and Enablement18 Feb 15

CSPF Scenarios

Contents

CSPF Scenarios and UCD D1 Environment

Alm.60.all Build–Moving from Alm.60.all to individual scenario–Scenario build information (tabs)

Problems–How to investigate–How to handle

Miscellaneous–How to access servers

2

Scenarios and UCD D1 Env

CSPF Scenarios in the Pipeline

Currently running scenarios– CLM BVT (also launched by RelEng)– CLM BVT NOPLE– CLM Licensing– JTS BVT– QM BVT– RM BVT– SVT GOV– SVT MTM – SVT MTM NOPLE

New– DCC BVT– SVT NONPLE RDNG

4

UCD D1 Environment

• Four machines1. CLM all-in-one2. LQE for reporting3. DB24. IHS

https://w3-connections.ibm.com/wikis/home?lang=en#!/wiki/Rational%20SVT%20Automation/page/UCD%20-%20CLM%20Departmental%20Linux%20DB2%20-%20Stream%20600

1

2

3

4

6

Scenario Build Activities

• These are the steps used to build/run the scenarios in the pipeline

• When deploying a manual UCD D1 environment, it is a two step process:

1. Create a new UCD environment…wait…

2. Deploy D1 to that environment

• Same idea for pipeline, …wait… between Create and Deploy

• That wait is waiting for IWD to provision the environments

• Timeouts• 150 minutes for D1• 60 minutes for DataGen

• Personal builds do not time out

Create D1 Env

Create DataGen

Env

Deploy to D1 Env

Deploy to DataGen

Env

Publish Properties

Run Tests

ALM.60.all Build

ALM.60.all Build

• The main build watched is the ALM.60.all build

• Current status of the build is noted in the Tracking Build Item, e.g., Track CLM 6.0 Sprint 9 (January 19 , 2015 - February 13, 2015) (343344)

• The ALM.60.all builds are usually run twice a day, 1130 EST and 1930 EST.

• The CSPF scenarios are run as children of this build

• The scenarios depend on SSE (which depends on CLM), CDSETUP and CSPF builds. If one of these fails, the scenarios will not run

8

9

Sample ALM.60.all build

• Let’s look at sample ALM.60.all build ALM.60.all ALM60-20150210-2056

• Look at the build-log and you can see the status of all of the child builds, including the scenarios

• Easiest way to get more detail is to use External Links tab and follow to the Build Tracking Item

10

Tracking Build

• These tracking items cover all of the scenarios being run, including the CSPF scenarios

• Approved = PassRejected = Fail

• See links to the specific scenario child builds in the comments

• Follow a link to see information about a specific child

11

Overview Tab

• Overview tab1. Pass/fail overall2. List of work items already

filed against all scenarios3. Link to create a new

‘problem with build’ work item against scenario

12

3

12

Activities Tab

• Activities Tab1. Shows activities that make

up this build2. And how long they took

1

2

13

Tests Tab

• Tests Tab1. Shows exactly what step

and test have failed2. Shows how long tests take

as well3. Stack trace details; linked to

code in your workspace

1

2

3

14

Logs Tab

• Logs Tab1. Log file for every activity,

plus master build log

1

15

External Links Tab

• External Links Tab1. Back to build tracking item2. RQM result, where more

details, including screenshots, are found

3. UCD processes4. UCD environments

1

23

4

16

Pass Along Properties Tab

• Pass Along Properties Tab1. URL of deployed server

• Actually, IHS server2. URL of DataGen server

• DataGen = rqmAdapterFQDN = clmSeleniumAdapter

1

2

Problems with Scenarios

Tags

Four tags– Implies four types of problem

Issue Tag

Test defect (test/framework has a defect)

cdp_test_issue

Product defect cdp_product_issue

Test adoption defect (test needs to adopt product UI changes)

cdp_adopt_test

Infrastructure defect cdp_infra_issue

18

Product Defect

1. Overview: see failure

2. Tests: see exact test and stack trace

3. External Links: follow link to RQM result

4. See screenshot

run.cspf.tests CPSF600-CLM_BVT_NO_PLE-20150211-0345

12

3

4

20

How to Handle Product Defects

Find/file defect against the product

Find/file ‘problem with build’ work item against build– Link the two work items– @ffusier, @craneml– Tag with “cdp_product_issue”– Tag with “clm_blocker”

If it’s a quick fix for the product, okay to let scenario fail

But if fix is slow, best to workaround the failure in the scenario, e.g., not run that test or automate some manual workaround to get around the issue

– Slow > 1 week

Web UI Changeor Adoption

1. Overview: see failure

2. Tests: see exact test and stack trace

3. External Links: follow link to RQM Result

4. See screenshot

run.cspf.tests CPSF600-CLM_BVT-20150114-0227

1

2

3

4

22

How to Handle UI or Adoption Defects

Find/file “problem with build” work item against build– @ffusier, @craneml– Tag with “cdp_adopt_test”– Make sure link to RQM result is included (and/or attach screenshots and logs)

Scenario owners: fix Some fixes can be quick (e.g., name of GC product); some are slow (e.g., introduction of GC in the first place). Do as much as you can and call of @ffusier and @craneml if required

UCDInfrastructure Problem

1. Overview: see failure

2. Logs: see error in d1Deploy.log

3. In log, see that ‘application process failed’, i.e., the UCD process failed to deploy D1

4. External Links: follow the link to the UCD process

5. Drill down to find the step that failed

6. Check the output log for clues

run.cspf.tests CPSF600-JTS_BVT-20150126-0545

1 2

3

4

5

6

24

How to Handle UCD Infrastructure Problems

Two kinds of problems with UCD1. Something wrong with application or script in UCD itself, e.g.,

Change in product affecting how product is installed/configured in WAS and UCD process has not caught up yet

2. Something wrong with something UCD depends on, e.g., Site it down (broken or maintenance) e.g., jazz.net, java, other install sites IWD (separate topic)

Solutions1. Find/file “problem with build” work item against build

@ffusier, @craneml Tag with “cdp_infra_issue” Main IWD application contact is Ye (Jennifer) Liu

2. Site down, maintenance window: nothing Jennifer can do to fix Wait and try again later

* Note: Michelle or Frederic are monitoring the daily CT pipeline for infrastructure issues; contact them first before filing a new defect.

IWDInfrastructure Problem

(Special case)

1. Overview: see failure

2. Logs: see failures in create*Env logs

3. In log, see “Timed out waiting” error

1. D1: 9000 sec = 150 min2. DataGen: 3600 sec = 60 min

4. clmdops@us.ibm.com can log into IWD gss5 to see what’s wrong with the environment

run.cspf.tests CPSF600-SVT_GOV-20150210-0528

1 2

3

*Timeouts only affect pipeline builds, not personal builds.

26

How to Handle IWD Infrastructure Problem

Not much to do about IWD issues except– Comment in TBI about infrastructure issue – Wait and try again

Let @ffusier and @craneml know of issues– FYI, we do:

IWD defects Filed against ITE at https://csnext.ibm.com:8001/jazz/web/projects/ITE Current problem:

https://csnext.ibm.com:8001/jazz/web/projects/ITE#action=com.ibm.team.workitem.viewWorkItem&id=46047

- GSS5 requires to be rebooted every day to maintain the pipeline runs

* Note: Michelle or Frederic are monitoring the daily CT pipeline for infrastructure issues; contact them first before filing a new defect.

Test or Framework Problem

Similar to product defect or adoption issue – failure shows up as a test failure

But not a product defect or change– Instead, something’s wrong with the scenario or framework– e.g., NPE or something unexpected

Could be triggered by product change, but not the product’s fault– Instead, scenario/framework didn’t handle every possibility

27

28

How to Handle Test or Framework Problem

These are our most important problems – we cannot allow these!

Find/file “problem with build” work item against build– @ffusier, @craneml– Tag with “cdp_test_issue”– Make sure link to RQM result is included (and/or attach screenshots and logs)

These problems must be fixed right away. Scenario owners should fix any scenario problem, but call on @ffusier or @craneml if required. Feel free to fix framework issue if you can find the problem.

Should be fixed the same day.

Miscellaneous

Miscellaneous

D1 and DataGen environments do not last very long. – DataGen environments can be deleted almost as soon as they finish– D1 environments for a successful run can be deleted quickly– D1 environments for a failed run can last a day or so

Much information is preserved in the build logs and the UCD application process itself

But if environment is needed (e.g., IWD information, or contents of DataGen machine), need to work fast

30

31

Accessing Servers

Possible to use VNC and ssh to connect to CLM, IHS and DataGen servers– Only do this when you are sure the environments still exist!

All properties show one URL for the deployed server: the IHS URL

VNC Ports, Usernames and Passwords

Server Protocol VNC Port Username Password

CLM/IHS vnc/ssh 1 clmadmin ec11ipse

DataGen vnc/ssh 3 root aut0mat10n

Getting CLM URL from IHS

Sometimes you need the CLM URL, e.g., to get logs

Easiest way is to ssh into the IHS machine and read the URL from its configuration

ssh -l clma fit-vmXX-YYY.rtp.raleigh.ibm.comAnswer yes to prompt and enter clmadmin’s password; can also use root to log in

cd /opt/IBM/Plugins/config/webservermergegrep rtp.raleigh plugin-cfg.xml

This searches the config for what other URLs are being used. As of 6.0, there will be two pairs of URLs returned. One URL will be for the CLM server, the other for the LQE machine

Log into <url>:9443/jts/admin to double-check that you have the CLM URL

32

33

Orchestrator Properties File& Build Definitions

Properties file Build definition

34

Build definitions to test changes

Test major or risky changes before requesting review– Analyze changes impacts

35

Build definitions to test changes

– Impacts can be serious

References

References

UCD D1 Environment Instructions: UCD - CLM Departmental Linux DB2 - Stream 600 - Rational SVT Automation Wiki

Working With CSPF Page: https://jazz.net/wiki/bin/view/Main/CspfWorkingWith

Working With the UCD Pipeline: https://jazz.net/wiki/bin/view/Main/CspfWorkingwithUcd

37

Questions???