Fall 2015CISC/CMPE320 - Prof. McLeod1 CISC/CMPE320 Assignment 4 is due Nov. 22 (this Sunday). Presentation stuff. Today, More Software Engineering: System Failure and A Movie! Testing, Cont. Week 12 Team Presentations Demonstrate of the fruit of your efforts! And, summarize (in any order you wish): How work ended up being distributed and when it was completed. Difficulties faced and overcome. Difficulties not overcome. Good/bad library use. Work left to do, if any. Team techniques that worked and those that did not. What you would do differently if you had to do this again Fall 2015CISC/CMPE320 - Prof. McLeod2 Presentation Schedule Week 12 Tuesday, Dec. 1 Basswood, Beech, Cherry, Tamarack. Thursday, Dec. 3, Lecture Time Hickory, Maple, Oak, Birch. Thursday, Dec. 3, Tutorial Time (JEF 128) Poplar, BalsamFir, Spruce, Cedar. Friday, Dec. 4 Hemlock, JackPine, Walnut, WhitePine. Fall 2015CISC/CMPE320 - Prof. McLeod3 Peer Grading, Again. Sign-up sheets will be circulated again during presentations. Project Wikis will be make public on Nov. 25. As for last time two Moodle surveys: The Wiki: Is the SDD up-to-date and does it accurately describe the architecture? Is the Wiki looking better than the last time you saw it? The Presentation: Organized, effective, met initial goals, good discussion of problems overcome? Fall 2015CISC/CMPE320 - Prof. McLeod4 Fall 2015CISC/CMPE320 - Prof. McLeod5 System Failure Often the result of a combination of events that is nearly impossible to predict or test for. See presentation from Dr. Terry Shepard linked to the lecture notes page in particular the slides on the FBI VCF (Virtual Case Files) failure. Possibly the most highly publicized software failure. A few more examples ranging from just annoying to deadly: Fall 2015CISC/CMPE320 - Prof. McLeod6 Microsoft Zune 30 Meltdown Apparently 2006 models of this MP3 player just froze up at midnight, Dec. 31, Eventually Microsoft acknowledged the problem. Here is the offending C code: Can you spot the problem? Fall 2015CISC/CMPE320 - Prof. McLeod7 #define ORIGINYEAR 1980 BOOL ConvertDays(UINT32 days, SYSTEMTIME* lpTime) { int dayofweek, month, year; UINT8 *month_tab; //Calculate current day of the week dayofweek = GetDayOfWeek(days); year = ORIGINYEAR; while (days > 365) { if (IsLeapYear(year)) { if (days > 366) { days -= 366; year += 1; } else { days -= 365; year += 1; } Fall 2015CISC/CMPE320 - Prof. McLeod8 Communications Problems The designers of the New England branch of the ARPAnet decided on seven-fold redundancy. Unfortunately all seven lines were sent through the same fibre optic trunk line, which was severed on Dec. 12, (Not all systems problems are a result of software failure!!) Fall 2015CISC/CMPE320 - Prof. McLeod9 More Communications Problems In late 1989, AT&T installed faulty software in 114 electronic switching systems. The problem was traced to a break statement inside an if statement inside a switch statement in a C program. A cascading problem caused all 114 switches to break down. AT&T was shut down nationwide for 9 hours, with an estimated 5 million calls blocked as a result. Fall 2015CISC/CMPE320 - Prof. McLeod10 Nuclear Power Bad software contributed to the accident at Three Mile Island 2 in 1979 Core temperatures were being measured with thermocouples, but the software would not report temperatures over 700 degrees ?????? was reported instead, even though the thermocouples were capable of measuring higher temps. Also intended valve settings were displayed by the control software rather than actual valve settings Fall 2015CISC/CMPE320 - Prof. McLeod11 Defense and Warfare Problems Patriot Missiles during the Gulf War Reported to the media that they were 95% effective. Turns out that they were really only about 13% effective due to a clock drift problem. This caused a Patriot missile to miss the Scud missile that hit an American Army barrack in Dhahran, killing 29 and injuring 97. This is a famous and classic example of a failure of software engineering. Fall 2015CISC/CMPE320 - Prof. McLeod12 Defense and Warfare Problems, Cont. Iran Air Flight 655 was shot down by USS Vincennes missile in 1988 290 People were killed. The error was blamed mostly on the poor interface used by the Aegis radar tracking system used aboard the Vincennes. Fall 2015CISC/CMPE320 - Prof. McLeod13 The F.I.R.S.T. Rules Fast tests are more likely to be run as often as they should be. Independent tests will not depend on each other. You should be able to run tests in any order. Repeatable tests can be run in any hardware/OS environment. Self-validating tests either pass or fail. Timely tests follow the three laws of TDD. Fall 2015CISC/CMPE320 - Prof. McLeod14 Overview of the Testing Process ProcessTesting CodingUnit Testing Design Integration Testing Requirements Analysis Validation Testing System Engineering System Testing 15 Overview, Cont. Test Planning Allocate and schedule resources. Do this early on as soon as models stabilize. Usability Testing Test user interface. (See Paper Prototyping, for example.) Unit Testing Test individual methods and classes. Integration Testing First separate classes by threads. Start by identifying independent classes and test them first, then test dependent classes, layer by layer. Cluster testing test a bunch of collaborating classes together. Use regression testing, stub and driver code as needed (more about these later!). Fall 2015CISC/CMPE320 - Prof. McLeod16 Overview, Cont. System Testing Test all the classes together. Functional Testing tests the requirements listed in the RAD and user manual. Performance Testing checks non-functional requirements and additional design goals from the SDD. (Done by developers). Acceptance and Installation Testing Check the system against any agreement made. (Done by clients with help from developers, if needed). Fall 2015CISC/CMPE320 - Prof. McLeod17 More Definitions A Component is the class, the cluster of classes, the package or one or more subsystems that is being tested. A Test Case is a set of inputs and expected results that exercises a component with the purpose of causing failures and detecting faults. A Test Stub is partial implementation of a component upon which the component being tested depends. A Test Driver is a partial component that depends on the component being tested. Stubs and Drivers allow the component to be harnessed isolated from the rest of the system. Fall 2015CISC/CMPE320 - Prof. McLeod18 More Definitions, Cont. A Correction is a change to a component made to repair a fault. A correction can introduce new faults Round and round we go where well stop nobody knows! Fall 2015CISC/CMPE320 - Prof. McLeod19 More Definitions, Cont. A system that fails after a correction has regressed. Carrying out the previous set of integration tests again is called Regression Testing. Fall 2015CISC/CMPE320 - Prof. McLeod20 Unit Testing Common types of unit testing: Equivalence testing Boundary testing Path testing State-based testing Fall 2015CISC/CMPE320 - Prof. McLeod21 Equivalence Testing A blackbox technique. (Deal with the input/output of the component, leave the internal behaviour alone.) Minimal number of test cases. First, determine a set of equivalence classes. You assume that the component behaves the same for all members of the equivalence class. Fall 2015CISC/CMPE320 - Prof. McLeod22 Equivalence Testing, Cont. Criteria for equivalence classes: Coverage every possible input belongs to one of the equivalence classes. Disjointedness no input belongs to more than one equivalence class. Representation if one member of an equivalence class causes an erroneous state then the same state can be caused by any other member of that class. For each class, select at least two test inputs a valid input and an input that should exercise your exception handlers. Fall 2015CISC/CMPE320 - Prof. McLeod23 Equivalence Testing, Cont. Example suppose you are testing a class that stores dates, as given by a year and month. You would need six equivalence classes: 31 day months, non-leap years 31 day months, leap years 30 day months, non-leap years 30 day months, leap years 28 day month, non-leap years 29 day month, leap year Minimum of 12 tests! Fall 2015CISC/CMPE320 - Prof. McLeod24 Boundary Testing Look at the edges of the equivalence classes. For example: Leap year divisible by 400 (year 2000, for example) Non-leap year divisible by 100 (year 1900, for example) Invalid month (at negative edge of legal months): 0 Invalid month (at positive edge of legality): 13 Fall 2015CISC/CMPE320 - Prof. McLeod25 Equivalence and Boundary Testing The problem with these techniques is that they do not explore many combinations of input data. Often a strange combination will cause an error, and this will take many more tests to locate. (My instincts say that there are just not enough tests!) Fall 2015CISC/CMPE320 - Prof. McLeod26 Path Testing A whitebox technique. (Focus on the internal structure of the component. Test every state of the component and every possible interaction of objects.) Exercise every possible path through the code at least once. Exercise every branch of every if statement. Run each loop can you cause a loop not to run? Throw each exception. Fall 2015CISC/CMPE320 - Prof. McLeod27 Path Testing, Cont. The problem with this technique is that it cannot show where code is missing it only tests existing code. Fall 2015CISC/CMPE320 - Prof. McLeod28 State Based Testing Developed for GUI systems. A new technique that is not fully fleshed out. Based on the idea that objects (like a button) can have different states under different conditions (like active, inactive, pressed, mouseover, etc.) See: WindowTester from Google, for example: https://developers.google.com/java-dev- tools/wintester/html/index Fall 2015CISC/CMPE320 - Prof. McLeod29 Polymorphism Testing OOP provides additional challenges to testing when a program takes advantage of polymorphism. You must make sure that every state of an object is tested. This can lead to many more tests especially when polymorphism is used by several objects in a component.