Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis
-
Upload
rik-marselis -
Category
Technology
-
view
85 -
download
6
Transcript of Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis
![Page 2: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/2.jpg)
1980
2007 2012 20122008
Workgroup-member
20142009
Rik Marselis@rikmarselis
![Page 3: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/3.jpg)
Who uses a robot today?
![Page 4: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/4.jpg)
You will use multiple within just a few years!!
![Page 5: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/5.jpg)
And please realise: these are robots too!
Chatbot Self-driving mini-bus
![Page 6: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/6.jpg)
WePod: self-driving bus
This minibus is allowed on the road now in Wageningen (the Netherlands) andwas tested by mySogeti-colleaguesDragan, Hugo andRemco.
![Page 7: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/7.jpg)
Robots, ok, but in less than 10 years????
Well;
Did you understand this
trafficsign 10 years ago?
That’s how fast technology
progresses!!
![Page 8: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/8.jpg)
What is Robotics?
• Internet of Things (IoT)
• Big data analytics
• Cloud technology
• Machine learning
• Artificial Intelligence
Robotics
Definition: A robot is a machine that gathers information about its environment by input of sensors and based on this input changes its behavior. Combined with machine learning and artificial intelligence the robot’s reactions over time get more and more adequate. The use of Internet of Things, Bid Data Analytics and Cloud technology make a robot versatile. Robots may appear in a physical shape (e.g. a tin man) or non-physical (e.g. chatbot).
![Page 9: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/9.jpg)
Traditional: Robots for dull & dangerous work
![Page 10: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/10.jpg)
New possibilities, just 2 examples
Exo-skeleton
Healthcare disabled peopleTransport drone delivery
![Page 11: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/11.jpg)
New risks
The bull doesn’t likethe manure robot andconstantly pushes it off its track. (to much frustration of the farmer who needs to cometo rescue the robot ;-)
![Page 12: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/12.jpg)
New angles of quality, aimed at roboticsMechanical
Electrical
Information Processing
Machine Intelligence
Business Impact
Social Impact
e.g. Ethics
e.g. Embodiment
e.g. Empathy
![Page 13: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/13.jpg)
Part 1:
Testing machine learning
![Page 14: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/14.jpg)
Machine Learning in the 80’s
![Page 15: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/15.jpg)
Machine learning today: demonstrating
![Page 16: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/16.jpg)
Machine learning today: demonstrating
![Page 17: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/17.jpg)
Learning by creating rules
How can we find the rules that self-drivingcars must use?
Ask you!
moralmachine.mit.edu
![Page 18: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/18.jpg)
Some
results
![Page 19: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/19.jpg)
Machine learning or machine training?
• A learning machine consists of a complex network
of algorithms
• The learning machine has very many “dials and
switches”
• Making sure the machine learns the right things is
just like ordinary training (well, in a different way…)
![Page 20: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/20.jpg)
Testing Machine Intelligence
• Human: “Dear robot please get me a softdrink”
• Robot: “Dear human here is your cola”
• Is this the correct output?
• No!
In Sweden I want:
![Page 21: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/21.jpg)
Testing Machine Intelligence
• Human: “Dear robot please get me a softdrink”
• Robot: “Dear human it’s Saturday
evening now, so here is your beer”
• Is this the correct output?
• After studying your behavior,
the robot knows your patterns
![Page 22: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/22.jpg)
Testing traditional IT vs. machine intelligence
Traditional (example):
• 1 + 1 = ?
• There’s only one correct output
With Machine Intelligence (example):
• Prepare a meal
• When is the meal “good enough”?
• Is it always the same meal?
2
![Page 23: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/23.jpg)
Two sides of testing machine intelligence
• Testing the output
This we know from traditional testing,
but it’s hard to predict outputs, the
result may be different every time.
• Testing the input
If the machine gets the right input
it will learn and do the right things.
![Page 24: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/24.jpg)
To control the results of AI:
control the inputs…
![Page 25: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/25.jpg)
Helpful chatbots?
• There’s a very thin line between a robot that is not helpful,
a robot that is helpful and a robot that’s too helpful
![Page 26: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/26.jpg)
Helpful chatbots?
The uncanny valley:
![Page 27: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/27.jpg)
What is the role of the tester?
• The tester must see whether the intelligent
machine will maintain the intended behavior in live
operation
• In a way the tester is the psychologist of the
intelligent machine
• A tester must ask even more questions than
before !!
Note: I picked up the idea of the tester being the psychologist of robots from James Bach.
![Page 28: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/28.jpg)
It’s no longer about right or wrong
• Testing intelligent machines is not an easy check
whether the result is right or wrong
• It becomes an appraisal of a specific response in a
certain situation with a certain amount of
knowledge for a certain audience
• If these parameters change the right behavior will
also change
![Page 29: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/29.jpg)
After finishing the test…
• The machine continues to learn and adapt…
• What about confidence?
• The test must result in the confidence that the
machine will learn and adapt in the right way…
![Page 30: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/30.jpg)
Part 2:
Learning machine testing
![Page 31: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/31.jpg)
Using a robot for dull or dangerous testing
• Mechanical tests use robot-arm
![Page 32: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/32.jpg)
Using a robot for intelligent testing
• Functional tests use machine intelligence to
– create test cases and
– execute test cases
– and (important!!) to judge the results
![Page 33: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/33.jpg)
Using a robot for testing physical objects
![Page 34: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/34.jpg)
Using a robot for intelligent testing
• Non-functional tests use machine intelligence
For example intelligent performance testing
(not just simple transactions but a normal variety)
![Page 35: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/35.jpg)
People and Robots testing together
• Teamwork of person & robot
• Each does what it is best at
– Robot repetitive work
– Human creative work
![Page 36: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/36.jpg)
Machine learning: unsupervised & supervised
Deep learning:
Robot learns at various
layers. Lower layers
are unsupervised.
Highest layer(s) is (are)
supervised.
![Page 37: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/37.jpg)
Machine Intelligence generates test data
• Based on Big Data Analytics of live data
• This is so-called “weak intelligence”
The machine is only intelligent in a limited field
![Page 38: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/38.jpg)
Machine Intelligence generates test cases
• Evolutionary algorithms
• Brute force testing
The machine is able to generate and execute
many many more test cases than a human tester
![Page 39: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/39.jpg)
Conclusion
![Page 40: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/40.jpg)
Testing machine learning
• Testing inputs is very important because the output
is not predictable
• The objective of testing is to gain confidence that
the machine will learn and develop itself in a right
way
![Page 41: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/41.jpg)
Learning machine testing
• Test automation will go to the next level
• Use machine intelligence (e.g. big data analytics)
to create your optimal test data
• Use machine intelligence to create many more test
cases, execute them and analyze results
![Page 42: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/42.jpg)
Conclusion…
As a tester better prepare yourself for:
• Testing of Robots
and
• Testing with Robots
![Page 43: Testing machine learning, learning machine testing. EuroSTAR 2016 Rik Marselis](https://reader034.fdocuments.net/reader034/viewer/2022042619/58e4a40f1a28aba3458b6791/html5/thumbnails/43.jpg)
Thanks for attending my talk.
Any questions ???????????
Rik Marselis, Sogeti Nederland