The case of the Killer Robot by Richard Epstein Killer Robot Richard Epstein Killer Robot Richard...

22
The case of the The case of the Killer Robot Killer Robot by by Richard Epstein Richard Epstein As retold by As retold by Netiva Caftori Netiva Caftori NEIU NEIU Oct. 2004 Oct. 2004

Transcript of The case of the Killer Robot by Richard Epstein Killer Robot Richard Epstein Killer Robot Richard...

The case of theThe case of theKiller RobotKiller Robot

byby

Richard EpsteinRichard Epstein

As retold byAs retold by

Netiva CaftoriNetiva CaftoriNEIUNEIU

Oct. 2004Oct. 2004

When software engineering and When software engineering and computer ethics collidecomputer ethics collide

The newsThe news

A robot operator, Bart A robot operator, Bart Matthews, was killed Matthews, was killed by his robot, Robbie by his robot, Robbie CX30, while at work.CX30, while at work.

A programmer, Randy A programmer, Randy Samuels, who wrote Samuels, who wrote the faulty code was the faulty code was indicted for indicted for manslaughter.manslaughter.

Silicon Techchronics is Silicon Techchronics is Randy’s employer.Randy’s employer.

What really happened?What really happened?

The robot The robot malfunctioned & malfunctioned & crushed its crushed its operator to deathoperator to death

Blood all overBlood all over Decapitated headDecapitated head Num-lock key light Num-lock key light

not onnot on Numeric key pad Numeric key pad

bloodybloody

Who is at fault?Who is at fault?

The programmer The programmer who made a who made a careless mistake?careless mistake?

The designer?The designer? The robotics The robotics

division?division? The company?The company? The operator of the The operator of the

robot?robot? The robot?The robot?

Robbie CX 10 & 20 were Robbie CX 10 & 20 were experimental.experimental.

The robotics division chief, Johnson, put The robotics division chief, Johnson, put pressure on the project manager Reynolds pressure on the project manager Reynolds to finish the project by January.to finish the project by January.

Johnson hired 20 new programmers in June Johnson hired 20 new programmers in June against Reynolds’ will, by shifting against Reynolds’ will, by shifting resources. Robbie CX 30 had to succeed.resources. Robbie CX 30 had to succeed.

Johnson only knew manufacturing Johnson only knew manufacturing hardware. One cannot speed software hardware. One cannot speed software coding by adding more programmers.coding by adding more programmers.

Robbie CX 30 was a step ahead in Robbie CX 30 was a step ahead in sophistication.sophistication.

Johnson: Perfect software is an oxymoron.Johnson: Perfect software is an oxymoron.

Programmer personalityProgrammer personality Randy’s home page: Randy’s home page:

freedom for freedom for programmersprogrammers

A hacker typeA hacker type Enormous stressEnormous stress Programmer was a Programmer was a

prima donna: could prima donna: could not accept criticism or not accept criticism or his own fallibilityhis own fallibility

Helpful but arrogantHelpful but arrogant Under pressure to Under pressure to

finish project on timefinish project on time

A good programming teamA good programming team

Needs to have a whole array of Needs to have a whole array of personalities types including:personalities types including:

An interaction-oriented personAn interaction-oriented person Someone who keeps the peace and Someone who keeps the peace and

helps move things in a positive helps move things in a positive directiondirection

Task-oriented people, but not allTask-oriented people, but not all A democratic teamA democratic team Egoless programmingEgoless programming

Team dynamicsTeam dynamics

The project was controversial from the The project was controversial from the beginningbeginning

Jan Anderson, a programmer, was fired Jan Anderson, a programmer, was fired after she attacked project manager after she attacked project manager Reynolds on his decision to use the Reynolds on his decision to use the waterfall methodology versus the waterfall methodology versus the prototype methodology.prototype methodology.

Reynolds was replacing a dead project Reynolds was replacing a dead project manager as a cost saving measure instead manager as a cost saving measure instead of hiring someone from outside the of hiring someone from outside the company. He never worked with robotics company. He never worked with robotics before.before.

More explanationsMore explanations

The earlier Robbie models processed The earlier Robbie models processed transactions. Robbie CX30 was to transactions. Robbie CX30 was to interact with its operator, therefore interact with its operator, therefore the interface cannot be designed as the interface cannot be designed as an afterthought. Thus, a prototyping an afterthought. Thus, a prototyping model is preferable where the users model is preferable where the users can use a prototype robot while in can use a prototype robot while in the design stage.the design stage.

Software development life-cycleSoftware development life-cycleWater fall methodologyWater fall methodology

RequirementsRequirements AnalysisAnalysis DesignDesign CodingCoding TestingTesting ImplementationImplementation MaintenanceMaintenance

No standard No standard practices practices

Quality controlQuality controlor quality assuranceor quality assurance

Dynamic testingDynamic testing Static testingStatic testing Unit testing: Black and white box Unit testing: Black and white box

testingtesting Integration testingIntegration testing

Prof Silber attested that test results Prof Silber attested that test results were inconsistent with actual killer were inconsistent with actual killer code test resultscode test results

When is the software good When is the software good enough?enough?

Errors can happen in any stageErrors can happen in any stage Design stage: 2 alternatives:Design stage: 2 alternatives:Waterfall & prototypingWaterfall & prototyping TestingTesting User interfaceUser interface The complexity of the task of The complexity of the task of

building real-world softwarebuilding real-world software Enormous stressEnormous stress

The company-Sili-TechThe company-Sili-Tech Two environments: Two environments:

Worker-friendly: Chip Worker-friendly: Chip Creek facility. Creek facility. Prevent repetitive Prevent repetitive strain injuries.strain injuries.

Special training for Special training for employees.employees.

Well designed Well designed workstations.workstations.

Frequent breaks.Frequent breaks.

Worker-unfriendly: Worker-unfriendly: Silicon Valley plant.Silicon Valley plant.

Workers criticized Workers criticized for novelties.for novelties.

No exercises or No exercises or training training encouraged.encouraged.

RSI frequentRSI frequent More compensation More compensation

claimsclaims

Law suitsLaw suits

The wife of the The wife of the murdered operator.murdered operator.

The indicted The indicted programmer programmer himself.himself.

More possible More possible indictments.indictments.

Why was Randy Samuels indicted?Why was Randy Samuels indicted?

Sili-Tech promised to deliver robots that would Sili-Tech promised to deliver robots that would cause no bodily harm.cause no bodily harm.

So Randy was not legally responsible for the So Randy was not legally responsible for the death of the robot operator.death of the robot operator.

Waterson, the president of Sili-Tech has Waterson, the president of Sili-Tech has contributed large sums to the re-election of judge contributed large sums to the re-election of judge McMurdock, who indicted Randy.McMurdock, who indicted Randy.

Functional requirements specify the behavior of Functional requirements specify the behavior of the robot under exceptional conditions. Operator the robot under exceptional conditions. Operator intervention may be needed.intervention may be needed.

Exceptional conditions were not mentioned in Exceptional conditions were not mentioned in training of operators.training of operators.

The charactersThe characters

Jane

Cindy

Prof SilberProf Gritty

Bart

E-mail was not secure at Sili-TechE-mail was not secure at Sili-TechIt was found that:It was found that:

Cindy Yardley faked the test results Cindy Yardley faked the test results of the Killer codeof the Killer code

Johnson told Cindy that the robot was Johnson told Cindy that the robot was safe and all will lose their jobs if the safe and all will lose their jobs if the robot were not shipped on timerobot were not shipped on time

Randy stole some of his software, but Randy stole some of his software, but admitted he had bugs in his Killer admitted he had bugs in his Killer codecode

Ethical issuesEthical issues

Email privacyEmail privacy Professional ethics: Professional ethics:

delivering bug-free delivering bug-free softwaresoftware

Plagiarism-Plagiarism-intellectual intellectual propertyproperty

Honor Honor confidentialityconfidentiality

ACM code of ethicsACM code of ethics

Five ethical tests:Five ethical tests:by Kallman & Grilloby Kallman & Grillo

The mom test: would you tell your The mom test: would you tell your mother?mother?

The TV test: would you tell your story The TV test: would you tell your story on national TV?on national TV?

The smell test: does it smell bad?The smell test: does it smell bad? The other person’s shoes test: would The other person’s shoes test: would

you like it if done to you?you like it if done to you? The market test: would your action The market test: would your action

make a good sale pitch?make a good sale pitch?