10 September 2018

Geoff Meyer, Test Architect

geoff_meyer@dell.com

A testing journey in the age of smart assistants

TestExpo2018

Agenda

•Context of DellEMC Server

•AI/Machine Learning 101

•Pinpoint your Testing Painpoints

•Our Superpowers in action

Context at DellEMC Servers

Server Configuration ElementsChassisProcessorMemory DIMMMemory ConfigurationHard Disk Drive (HDD)Non-Volatile Memory (NVM)Embedded Systems ManagementPower Management BIOSPower SupplyBezelNetwork Daughter CardRAID ControllerNetwork Interface Card (NIC)Host Bus Adapter (HBA)Additional PCIe CardsCooling

465 Trillion Test Configurations!!

Analytics Engine

Domain Knowledge

(i.e. Rules)

Data Sources

Data Cleanse

Insights,

Predictions,

Recommendations

Feedback

Data Analytics Modeling

Getting Computers to learn without being explicitly programmed

• Supervised Learning

• Unsupervised Learning

• Reinforcement Learning

Machine Learning

Getting Computers to learn without being explicitly programmed

• Supervised Learning

• Unsupervised Learning

• Reinforcement Learning

Machine Learning

Getting Computers to learn without being explicitly programmed

• Supervised Learning

• Unsupervised Learning

• Reinforcement Learning

Machine Learning

MarI/O - https://www.youtube.com/watch?v=qv6UVOQ0F44

It’s all about the data

“Nobody really goes out of their way to point out the importance of data…”

~ Brian Sletten, Bosatsu Consulting

Data Sources - Product Engineering

What if we had a Smart Assistant?

What are the high-value SUT configurations? What test scripts

should be retired rather than be re-factored?

What tests can detect the maximum number of defects

given the changes in the current build

What is the release risk given the testing that’s been completed?

What’s the optimal coverage for this build/test cycle? What automated test

failures appear to be duplicates?

The Smart Assistant

Selecting our Technology Partners

Q

Challenges

➢ 465 Trillion possible server configs!!!➢ Which are the High Value Configs?➢ How to ensure Optimal Configs

Coverage?

Current Testing Scenario

➢ Quickly predict “best-available” SUT configurations during planning and test execution

➢ Ensure Optimal Configs Coverage➢ Prioritizes High-Value Configs

Objectives

SUT Configuration Model“Q” - System Under Test

Data Sources

Team

Technology Partner: Dell Performance Analytics Group

Q Process FlowDATA SETS

FUNCTIONS

TOOLS & TECH

HIGH VALUE

SUT CONFIGS

FEEDBACK LOOP MACHINE LEARNING

DEFECTS

FEEDBACK LOOP MACHINE LEARNING

TEST HISTORY

100 Configs/RISER+

RISER SLOTS

STORAGE MEMORY THERMAL

BUILD VALID CONFIGS

CONFIGURATION - RESTRICTIONS

PROBABILISTIC RANDOM SAMPLING

100K Configs/RISER

500K Configs/RISER+

CLUSTERING

AS SOLD CONFIGS

NUDDs

SCORING MODEL

GEN OVER GEN

MAPPING

How Does Q Help?

• Increase precision of predictions by extending data sets:

• Test Execution & Defect History (JARVIS integration)

• As-Deployed customer configurations

• As-Tested configurations

• Cost Optimization of prototypes/parts for new platforms

1 Improved Test Coverage - Maximizes Test Coverage given limited availability

Cost Savings - Optimizes number of prototype configurations needed for test

Data-driven - Institutionalizes the Test Configuration Planning process2

3

Time Savings - Reduces Test Configuration Planning from weeks to hours4

Maintains Optimized Coverage – When components/features are delayed1

Automation candidates

De-prioritization candidates

Test Planning Model“JARVIS”

Technology Partner:

Objectives• Use historical test data and defects

as predictors and to expose patterns• Automate deep-think testing tasks• Codify Subject-Matter Expertise• Real-time access to active

repositories

How can I accelerate discovery of break/fix?

Which manual tests are most effective, and should be automated?

Am I over-testing or under-testing?

Which of my test cases appear to be obsolete?

Fast Find of Break/Fix

Reduced test cycle time

Increased Test Capacity

Re-allocate toExploratory Testing

Testbots are hereAI-assisted UI Automation

• Increases UI test coverage

• At substantially less cost of creation and maintenance

Duplicate Defect PredictionDellEMC XtremIO

2018 2019

SUT Config

Develop Deploy Extend

Integrate

JARVIS

Pilot

Deploy (EST)

Test Suite

Workload

AI POC’s Change-Based Regression

AI-Assisted Testbots

Data Curation

Visualization

Visualization

Deploy +

Deploy+

Dell Performance Analytics Group

Cognitive Automation Roadmap

2017

Develop Deploy

Proof-of-Concept

Proof-of-Concept

Staffing

Today

Deploying Smart Assistants

Partner with the Machine

Pinpoint your pain points

Capture your data

Re-imagine Testing

Thank you

ResourcesBooks• Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die: https://www.amazon.com/dp/B019HR9X4U/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1• What To Do When Machines Do Everything: http://www.whenmachinesdoeverything.com/• Weapons of Math Destruction: https://weaponsofmathdestructionbook.com/• Race against the Machine: https://books.google.com/books/about/Race_Against_the_Machine.html?id=IhArMwEACAAJ• Super Freakonomics: http://freakonomics.com/books/• Humans are underrated: http://geoffcolvin.com/books/humans-are-underrated/• Life 3.0: Being Human in the Age of Artificial Intelligence: https://www.amazon.com/Life-3-0-Being-Artificial-Intelligence/dp/1101946598• The Four: http://www.thefourbook.com/

Research• When will AI Exceed Human Performance: https://arxiv.org/pdf/1705.08807.pdf• World Quality Report 2016-17 (Capgemini): https://www.capgemini.com/thought-leadership/world-quality-report-2016-17• World Quality Report 2017-18 (Capgemini): https://www.capgemini.com/thought-leadership/world-quality-report-2017-18• The next era of Human|Machine Partnerships: https://www.delltechnologies.com/en-us/perspectives/realizing-2030.htm• Towards a Reskilling Revolution: A Future of Jobs for All: http://www3.weforum.org/docs/WEF_FOW_Reskilling_Revolution.pdf• Special report: Tech and the future of transportation: http://b2b.cbsimg.net/downloads/Gilbert/SF_feb2018_transport.pdf• How AL will Change Software Development: https://www.slideshare.net/WillyDevNET/how-ai-will-change-software-development-and-applications• 21 Jobs of the future: https://www.cognizant.com/whitepapers/21-jobs-of-the-future-a-guide-to-getting-and-staying-employed-over-the-next-10-years-codex3049.pdf• Wait but why: Artificial Intelligence Revolution Part 1: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html• Wait but why: Artificial Intelligence Revolution Part 2: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html• What’s Next | Artificial Intelligence Part 1: https://www.youtube.com/watch?v=2br8yji-rcM• What’s Next | Artificial Intelligence Part 2: https://www.youtube.com/watch?v=_WKyiGBYFrU• TensorFlow by Brian Sletten: https://www.youtube.com/watch?v=RlrBKYehcNg• Wolfram Alpha: www.wolframalpha.com

ResourcesArticles• The Future of Jobs: http://www3.weforum.org/docs/WEF_Future_of_Jobs.pdf• This Technology Will Upend the Entire Automotive Industry: https://moneywise411.com/new-automotive-technology/?ppc=743242• 5 ways AI will change software testing - https://techbeacon.com/5-ways-ai-will-change-software-testing• What’s Everybody So Afraid of: http://www.popularmechanics.com/technology/robots/news/a28645/googles-alphabet-astro-teller-ai/• Robots Are Coming for Jobs of as Many as 800 Million Worldwide: https://www.bloomberg.com/news/articles/2017-11-29/robots-are-coming-for-jobs-of-as-many-as-800-million-

worldwide• Self-Driving Cars Could Save 300,000 Lives: https://www.theatlantic.com/technology/archive/2015/09/self-driving-cars-could-save-300000-lives-per-decade-in-america/407956/• The 10 Biggest AI Failures of 2017: https://www.techrepublic.com/article/the-10-biggest-ai-failures-of-2017/?ftag=TRE684d531&bhid=24345184115902224026945549370599• Technology has created more jobs than it has destroyed: https://www.theguardian.com/business/2015/aug/17/technology-created-more-jobs-than-destroyed-140-years-data-

census• AlphaZero Is the New Chess Champion: https://www.extremetech.com/extreme/260215-alphazero-new-chess-champion-harbinger-brave-new-world-ai• The tech industry needs one million workers now: https://www.yahoo.com/finance/news/tech-industry-needs-one-million-workers-now-130452775.html• Only 4% of CIOs have deployed AI:https://cio.economictimes.indiatimes.com/news/business-analytics/only-4-pc-of-cios-have-deployed-ai-despite-huge-interest-levels-in-ai-

technologies/62900459• Towers Watson & Oxford Economics: Global Talent 2021: https://abhishekmittal.com/2012/08/04/towers-watson-oxford-economics-global-talent-2021/• Meet the 13-year-old prodigy taking IBM and AI by storm: http://www.abc.net.au/news/2017-07-26/meet-the-teen-taking-ibm-and-artificial-intelligence-by-

storm/8743880?pfmredir=sm• Testing and Management Efficiency: http://www.developsense.com/blog/2018/02/Efficiency/• How reinventing software testing can transform your business: https://techcrunch.com/2018/03/13/how-reinventing-software-testing-can-transform-your-business-and-change-

the-world/?utm_content=68756890&utm_medium=social&utm_source=linkedin• Top 5: Things to know about AI: https://www.techrepublic.com/article/top-5-things-to-know-about-ai• Test.AI nabs $11M Series A funding: https://techcrunch.com/2018/07/31/test-ai-nabs-11m-series-a-led-by-google-to-put-bots-to-work-testing-apps/• Understanding the differences between AI, machine learning, and deep learning: https://www.techrepublic.com/article/understanding-the-differences-between-ai-machine-

learning-and-deep-learning• AI in software testing has arrived. Here's why robots rule: https://searchsoftwarequality.techtarget.com/feature/AI-in-software-testing-has-arrived-Heres-why-robots-rule• A 5-second test for AI fever: https://www.linkedin.com/pulse/5-second-test-ai-fever-g-%C3%B8stby-sol%C3%A5s/

Title: A testing journey in the age of Smart Assistants

Description: The hype surrounding Artificial Intelligence (AI) has rocketed beyond its second peak. AI has undoubtedly gained traction and delivered

value to traditional businesses by automating away manual processes, tasks and jobs in industry after industry. On top of that, entire industries have been disrupted by those that that have used this opportunity to re-invent processes and business models to create new markets. As the general purpose technology of the 4th Industrial Revolution, the emerging chapters of these fascinating Machines demands our attention as AI continues to be applied in ways that directly affect the workplace, one in which the Test community won’t be immune.

In this latest hype cycle, new products and consultants are everywhere and inundating us with solutions that may or may not be applicable to our organizational context. We find ourselves having to sort out fact from fiction and often underestimate the effort in assessing the viability of these new practices. At times like this, shared insight from a fellow practitioner who’s been down this road could be a most welcome assist.

Geoff shares the "in-progress" journey at DellEMC as they drive to optimize and re-invent their testing practices with the application of data-driven smart assistants, powered by Analytics and Machine Learning. At a macro level, Geoff identifies opportunities across the Product Engineering and Test landscape for the application of Analytics and AI. Key ingredients in moving toward solutions that matter is the identification of organization-specific pain points, their prioritization, and the availability and cleanliness of essential data. Geoff shares the process of experimentation, staffing and implementation that his team approached these new opportunities with and then delves into the Smart Assistants that they’ve created to automate deep-think, cognitive-based testing tasks. “'Q", "JARVIS", and “Goose” automate many of the time-consuming and deeply analytical tasks such as determining high-value test configurations, defining high-value/maximum coverage regression test suites, and identifying market-demanding solution workloads when time is not an ally. Most importantly, Geoff shares insights on the activities that should get the highest levels of attention and those that you might want to de-prioritize to later phases of your own Analytics and AI journey.

Abstract

geoff_meyer@dell.com

https://www.linkedin.com/in/geoff-meyer-02b1aa3/

https://twitter.com/geoffrey_meyer

A Test Architect in the Dell EMC Infrastructure Solutions Group, Geoff has 30+ years of industry experience as a software developer, manager, program manager, and director.

He drives the Test Strategy and Architecture for 400+ SW and HW Testers across India, Taiwan, and the United States. His initiatives include Agile Testing, Continuous Testing, Infrastructure as a Service(IaaS), Predictive Analytics and Test.AI

Geoff is a member of the Agile Austin community and is a speaker at Agile, STAR, and international testing conferences. He is an active mentor to Veterans participating in the Vets4Quality.Org program, which provides them an on-ramp to a career in software quality assurance.

Geoff Meyer