Are you covered? New tooling for Quality Assurance...© 2018 IBM Corporation Command Line Code...
Transcript of Are you covered? New tooling for Quality Assurance...© 2018 IBM Corporation Command Line Code...
© 2018 IBM Corporation© 2018 IBM Corporation
Are you covered?
New tooling for Quality Assurance
Edmund Reinhardt – IBM i
Application Development Tooling
© 2018 IBM Corporation
2
Agenda
• What is code coverage
• Code Coverage while developing
• Demo with RPGUNIT and Code Coverage and Debug
• Code Coverage in DevOps pipeline (integrated with build)
• Integration with Test Automation products
© 2018 IBM Corporation
The Vision
• IBM i developers are able to confidently change their code
– Because their code is tested 100%
– They can achieve this through Test Driven Development
– They can prove this coverage level (this session)
• Detect untested code
• Detect unreachable code
– They can refactor the code confidently to make it readable, well-
structured and taking advantage of the latest technology.
– As a result the platform is not known for old, hard to modify code,
• Rather as the most reliable, cost-effective platform for business
© 2018 IBM Corporation
Code Coverage while developing
• Expected to do more with less
• Cannot afford long cycle times to test
• Cannot afford to let a bug go through because of a missed path
• Need to work smarter
• Solution
– Use technology to help you
• Develop productively using an IDE with rich language understanding
• Test thoroughly – automatically detect untested branches in code
© 2018 IBM Corporation
5
RDi expands into new territory – Quality Assurance
• RDi Line level Code Coverage Analysis Capability
• You can see exactly which lines were covered and not.
• These results can be accumulated see the coverage of the total suite of tests as well as speeding up the analysis of
subsequent test runs.
• Can determine effectiveness of tests, and guide improvements
• Can help locate dead code
© 2018 IBM Corporation
6
Launching Code Coverage
• Code coverage can be launched on any program or service program that can be debugged – independent of language
• Batch program analysis supported as of RDi 9.1
• Interactive and SEP supported added in RDi 9.1.1 (December 2014)
© 2018 IBM Corporation
7
Code Coverage Report
• After running code coverage, a report is shown as an editor.
• You can drill down through programs, modules and
procedures and see the coverage statistics for each
© 2018 IBM Corporation
8
Coverage annotated in the editor
• Drilling down from the report,
• editor will be opened on the related member
• green and red annotations showing which lines covered.
© 2018 IBM Corporation
9
PDF Report
• The results can shown via PDF reports so they are
available for those who are not in the Eclipse client, i.e.
quality assurance, management etc.
• Note: this functionality was missing in 9.6 until 9.6.0.5
© 2018 IBM Corporation
How to generate PDF reports
© 2018 IBM Corporation
11
Launch Options
Allow code coverage analysis of programs that touch data in
production libraries
Use *LIST if executable code in /COPY, availability depends on
DBGVIEW compile option.
© 2018 IBM Corporation
12
Customize how the program is invoked
© 2018 IBM Corporation
13
Customize the Code Coverage Analysis
Can track the coverage at the higher level of
procedures or programs
Option to generate PDF reports from the initial launch.
Can always export it later.
Associated metadataCould be used track which
testcase covered which lines.
© 2018 IBM Corporation
14
Code Coverage History view operations on reports
Coverage History can be exported
and imported into other
workspaces
Can generate workspace or
comparison reports and
export them in HTML or PDF
14
© 2018 IBM Corporation
15
Compare 2 runs to see improvement in coverage
• Select 2 runs in the Code Coverage Results view and Compare –
see % improvement for different procedures
© 2018 IBM Corporation
16
Service Entry Points
• A special kind of entry breakpoint that can be set directly from Remote
System Explorer
• Debugger is automatically started when first line of program is executed
• Supports batch, interactive, and Web service programs
© 2018 IBM Corporation
17
Use Service Entry Breakpoints with Code Coverage
• In SEP view, toggle to Code Coverage mode from Debug mode
• Set the SEP breakpoint just as you would for debugging
• Invoke the application in whatever manner you normally would
Switch to Code
Coverage mode
Optionally, customize the
launch configuration
© 2018 IBM Corporation
18
RDi 9.1.1 COBOL Code Coverage Enhancement
• The coverage analysis is now reported by paragraph as
well as by line
© 2018 IBM Corporation
New Code Coverage Features in RDi 9.5
• Much, much, much faster! Typically 20x or better improvement
– The architecture was completely redone to streamline code coverage to a
minimum of interaction between the RDi client and the IBM i server
– Solution is ready for real world scale of programs and tests
• Ability to use shared results, so coverage results of test cases can be computed
once and reused for the team
• Ability to merge code coverage results from multiple runs
– Compute the combined coverage and show exactly which lines are missing
– Exactly which modules/files/procedures do not meet specified thresholds
19
© 2018 IBM Corporation
20
The results of multiple test runs can be merged
• The bucket of manual or automatic tests are all run
• The coverage results are all analyzed and merged
into a single baseline
© 2018 IBM Corporation
21
Holes in the Testbucket can now be analyzed
• Procedures and files below a set threshold
are highlighted in red and need more testingClick on Column Headings To Sort
© 2018 IBM Corporation
22
Results can be given meaningful names
© 2018 IBM Corporation
One advantage of the new (9.6.0.5) Web View of coverage
© 2018 IBM Corporation
Code Coverage now supports SonarQube integration!
© 2018 IBM Corporation
25
Results can be exported to a shared location
Export to:
• Your PC
• IFS
• Shared drive
© 2018 IBM Corporation
26
Shared result can be viewed & analyzed from any RDi client
© 2018 IBM Corporation
Some may be color-blind and so the annotations can be customized
to show different symbols as well!
27
Customize coverage annotations
© 2018 IBM Corporation
28
Code Coverage – Filtered File Report
• On any Code Coverage Result, the Open Filtered… action is ava
• Specify a filter string and the report will show code coverage results for lines that match
the filter string
© 2018 IBM Corporation
Integration with RPGUNIT• Follow https://irpgunit.sourceforge.io/help/
© 2018 IBM Corporation
Check Code Coverage of Automated Test
Set a service breakpoint on our service program modules
Set "Code
Coverage"
mode
© 2018 IBM Corporation
Oh oh … our tests don't test all our code
Only 86% of our code is tested!
Drill down into the code. We haven't tested the "found = *off" line
© 2018 IBM Corporation
Introduction to IBM i Code Coverage Command
© 2018 IBM Corporation
33
Imagine…
• Having code coverage on an individual invocation
is interesting
• But for transforming your quality, things have to
be automated
• What if you could check the coverage of your
regular test runs?
• And this all ran on the host with no dependency
on any client software
• And it was FREE!
© 2018 IBM Corporation
Continuous Integration / Continuous Delivery
© 2018 IBM Corporation
Command Line Code Coverage
• Freely available with Rational Development Studio (5770-WDS)
option 60 on the IBM i
IBM i MRM MRI
V7R3 PTF 5770WDS SI65229 2924 PTF 5770WDS SI64655
V7R2 PTF 5770WDS SI65228 2924 PTF 5770WDS SI64544
Option Charge
Base No
Option 60 Workstation Tools No
ILE Compilers Yes
Heritage Compilers Yes
Application Development Toolset Yes
© 2018 IBM Corporation
36
How command line code coverage is structured
IBM i
QSYS
Application PGM/SRVPGM
Under analysis
CODECOV CMD
IFS
Analysis files (CCZIP)
In directory
PC
RDi
Result Directory
Views
Net share or copy
CcApiExtAPI
Your Tools!
© 2018 IBM Corporation
37
Requirements on job environment
• ILE Programs and service programs for instrumentation must be compiled with one of the
following debug options:
• DBGVIEW(*SOURCE)
• DBGVIEW(*LIST)
• DBGVIEW(*ALL)
• Main command: CODECOV
© 2018 IBM Corporation
Features of Headless Code Coverage Tools
• Start a code coverage session against a run command• Add programs and service programs into the instrumentation list• Module filter support
• A list of modules • (M1 M2 M3)
• All modules • (*ALL)
• EXCEPT keyword: • (*EXCEPT M1 M2) - All modules except M1 and M2
• Support both *SOURCE and *LIST views• Support LINE and PROCEDURE level coverage• Save code coverage result in designated location
• When output directory is specified, save result as a generated file under the target directory• When output stream file is specified, save result in the designated file
© 2018 IBM Corporation
Prompt screens for CODECOV command
© 2018 IBM Corporation
Prompt screens for CODECOV command (cont.)
© 2018 IBM Corporation
Include one program into the MODULES list
• Enter one program into the Modules list• Enter “+” into the second “+ for more values” field
© 2018 IBM Corporation
Include another service program into the MODULES list
© 2018 IBM Corporation
Specify a list of modules
• Enter the first module into the Module field and then enter “+” into the first “+ for more values”
© 2018 IBM Corporation
Specify a list of modules (cont.)
Enter other modules
© 2018 IBM Corporation
Code Coverage Mandatory Parameters
• Two parameters are mandatory:• Command to run: The full command line that will run in code coverage instrumentation mode
• Command length restriction: 6000• At least one entry in the MODULES list
• Include at least one program or service program for code coverage instrumentation
• Does not have to be the main program
© 2018 IBM Corporation
Code Coverage View
• Three code coverage view configurations are supported:• *SOURCE• *LIST• *DFT
• Programs and service programs included in instrumentation must be compiled with one of the following debug options:
• DBGVIEW(*SOURCE)• DBGVIEW(*LIST)• DBGVIEW(*ALL)
• *SOURCE:• Use the *SOURCE view for source file display. The programs must be compiled with
DBGVIEW(*SOURCE) or DBGVIEW(*ALL)• *LIST:
• Use the *LIST view for source file display. The programs must be compiled with DBGVIEW(*LIST) or DBGVIEW(*ALL)
• *DFT: The default behavior• Use the *SOURCE view when the program is compiled with DBGVIEW(*SOURCE) or DBGVIEW(*ALL)• Use the *LIST view when the program is compiled with DBGVIEW(*LIST)
© 2018 IBM Corporation
Code Coverage Level
• *LINE: Line level
• Include line-by-line statistics
• The default level
• *PROC: Procedure level
• Only capture the information whether a procedure entry point is hit or not
• No line-by-line statistics
• Run faster, but capture less information
© 2018 IBM Corporation
Output options
• Two parameters to control output options:• Output directory and Output stream file • They cannot be specified at the same time• Error message is issued if they are specified at the same time:
• Parameters OUTDIR and OUTSTMF cannot be used at the same time• Output directory:
• Code coverage result file is saved to this directory. • Use generated name for code coverage result file
• Generated result name format:• ProgramName_TimeStamp.cczip
• The default output directory is *CURRENT.• Output stream file:
• Fully qualified IFS file name for code coverage result file• If the base name does not have extension
• Extension .cczip is appended to the base name• If the base name has an extension, it must be .cczip, otherwise an error is issued:
• The extension for the result name … is not .cczip.
© 2018 IBM Corporation
Code Coverage Result File
• Code coverage result file has .cczip extension• Internally it is a ZIP file• An XML entry (ccdata) contains code coverage statistics• Source files are stored under the “src” folder inside the .cczip file
© 2018 IBM Corporation
Code Coverage Test Identifier
• An identifier to set on the current code coverage session• Not used for any feature right now• Business Partners can use it• IBM ADDI uses it to identify tests and optimizes the minimal set of tests
to get the desired shelving
© 2018 IBM Corporation
Requirements: Viewing Code Coverage Data
• CCZIP results generated by CODECOV can be viewed on RDi v9.5.1.2 or later
• Install RDi 9.5.1.2 or laterNOTE: If you don’t have access to the above location, install as follows:
• Go to: http://ibm.biz/rdi_download• Click on:
• Select Window (Automated web install using IBM Installation Manager) • Follow the steps in Automated procedure for full instructions.
© 2018 IBM Corporation
Viewing Coverage Results in RDi
• Assume CODECOV generated into /home/REINHARD/CodeCoverage
© 2018 IBM Corporation
RDi –- Code Coverage report and Code Coverage Results view
© 2018 IBM Corporation
RDi –- Code Coverage Annotation in source editor
© 2018 IBM Corporation
Limitations
• It is currently not possible to generate HTML/PDF reports from
the .cczip results from CODECOV
– PDF report support delivered with 9.6.0.5
• CODECOV only works with ILE languages
© 2018 IBM Corporation
57
Vision
• Can raise the maturity of your software development process
– Get the code coverage on your regular build/test on the host
– Detect drops in coverage on each build
– Mandate a minimum threshold for your tests
– Integrate with automated test solution
– Capture analysis with only dependence on a command shipped with
IBM i
– Analyze the results with the full power of RDi
© 2018 IBM Corporation
Part 3 – Vendors lining up to support this
• Freshe Legacy
• XaTester from Xact
• Original Software
• ILEditor by Liam
• Integrate Code Coverage with their automated testing
• Include Code Coverage statistics with their reporting
© 2018 IBM Corporation
ILEditor
• Save command,
modules as a
named test
• Invoke test
• See Coverage
result display in
the editor
© 2018 IBM Corporation
Example TestBench by Original
• Existing tools can be used by
wrapping calls with the CODECOV
command.
• In this case, the CMD would simply
be
CALL PGM(TB_OEFIL/OEORDRR
PARM(‘DEMO’).
• TestBench would then ‘wrap’ the
program call inside CODECOV and
handle specifying all the necessary
MODULE parameters for ‘child’
programs etc.
© 2018 IBM Corporation
TestBench Results – Test Run Details
© 2018 IBM Corporation
TestBench Results – Program Calls
© 2018 IBM Corporation
TestBench Results – Database Effects Summary
© 2018 IBM Corporation
TestBench Results – Database Effects Detail
© 2018 IBM Corporation
TestBench Results – Database Effects Update Details
© 2018 IBM Corporation
TestBench Results – Data Queue Activity
© 2018 IBM Corporation
TestDrive Results – Showing Capture and Inclusion of Code
Coverage Results – Future support (not available today)
Selecting the htmlfolder opens the folder in File Explorer so the HTML file can be launched.
© 2018 IBM Corporation
Fresche - X-Analysis and Code Coverage
• The regression testing capabilities in X-Analysis enables the generation of Code Coverage reports
through the creation and configuration of Test Processes.
– Test Process defines the setup, the program to be executed, the generation of a Test Result, Test
Result report and teardown after the test has completed. The Test Process can be run on
demand or submitted to batch.
– Test Result report provides a detailed report of any data area or table that differs from the original
Test Result.
– X-Analysis will also detect problems associated with the object that can be used to alert the
developer of the problem. The results of Problem Analysis are also available via an API to
condition the promotion of the object into production.
• Some of the Code Coverage integrations that are planned for the next release are:
– Setup & Configuration – Enhanced control of which Programs/Modules are included in the Code
Coverage report.
– Measurement - Code Coverage percentages will be tracked over time in order to track trends
– Application Management - Problem Analysis will provide the ability to identify code coverage
conditions that require attention before an object should be moved into production.
© 2018 IBM Corporation
ARCAD Verifier Integration
• ARCAD Verifier is a very capable tool that not only automates batch and interactive tests, but also
detects and validates changes in the user interface, database and spools
• By the end of the year they are planning to release integration with IBM’s CODECOV
© 2018 IBM Corporation
Conclusion
• Code coverage gives you concrete measurement of your test
effectiveness
• At development time, use RDi to launch and view code coverage
• At build time, use the command line CODECOV to test at scale
and the use RDi to view results
• As a command line it is easy to integrate CODECOV in your own
scripts or in automated test tooling
© 2018 IBM Corporation
Where to learn about Code Coverage
• Learn More about Code Coverage
– Charlie Guarino Debugging and Code Coverage 1/3
– Charlie Guarino Debugging and Code Coverage 2/3
– Charlie Guarino Debugging and Code Coverage 3/3
– Code Coverage Product Documentation
– Code Coverage article on DeveloperWorks
– IBM Application Delivery Intelligence based on Code Coverage available
for z
• Test Automation
– RPGUnit integration for RDi – free, applicable for batch programs only
– ARCAD Verifier product description
– Original Software –TestDrive
– Xact XaTester
71
© 2018 IBM Corporation
For More Information:
Some Links You Need Twitter #Hashtags
IBM i Home Page:
www.ibm.com/systems/i
IBM 30th Anniversary
http://ibmi30.mybluemix.net/
IBM Systems Magazine IBM i Edition:
http://ibmsystemsmag.com/ibmi/
Support Life Cycle:
https://www-01.ibm.com/software/support/ibmi/lifecycle/
License Topics:
https://www-01.ibm.com/support/docview.wss?uid=nas8N1022087
@IBMSystems
@COMMONug
@IBMChampions
@IBMSystemsISVs
@IBMiMag
@ITJungleNews
@SAPonIBMi
@SiDforIBMi
#IBMi30
#PowerSystems
#IBMi
#IBMAIX
#POWER8
#LinuxonPower
#OpenPOWER
#HANAonPower
#ITinfrastructure
#OpenSource
#HybridCloud
#BigData
© 2018 IBM Corporation
For More Information:
Blogs
IBM Blogs:
• IBM Systems Magazine You and i (Steve Will)
• IBM Systems Magazine i-Can (Dawn May)
• IBM Systems Magazine: Open your i (Jesse Gorzinski)
• IBM DB2 for i (Mike Cain)
• IBM DB2 Web Query for i (Doug Mack)
IBM Champion’s Blogs:
• IBM Systems Magazine: iDevelop (Jon Paris and Susan Gantner)
• IBM Systems Magazine: iTalk with Tuoy
http://ibmsystemsmag.com/blogs/you-and-i/
http://ibmsystemsmag.com/blogs/i-can/
http://ibmsystemsmag.com/blogs/open-your-i/
http://db2fori.blogspot.co.uk/
http://db2webqueryi.blogspot.co.uk/
http://ibmsystemsmag.com/blogs/idevelop/
http://ibmsystesmag.com/ibmi/trends/italk-with-tuohy/