©Brian Whitworth 2005, [email protected] THE WEB OF SYSTEM PERFORMANCE (WOSP) Excellence requires...

©Brian Whitworth 2005, [email protected]

1

THE WEB OF SYSTEM PERFORMANCE (WOSP)

Excellence requires Balance


2

Overview

I. System performance issues & problems

II. The Web of System Performance (WOSP) model

III. Implications for system design

IV. Validation experiments on how users evaluate information systems

V. Summary tips


3

Past Predictions– Leisure society (4 day week)- cf a 6 day week!– Paperless office (Toffler) – cf more paper than ever before!– No more programmers (Martin)- programmers still in demand– AI will replace people – we still prefer human telephone help – Star-Trek’s video-phone – we have the technology, but it didn’t happen

– Video-conferencing – millions $ invested, but didn’t happen – Virtual-reality (headset) games – cf MMORGs & WADs – A mainly multi-media Internet - the Internet is still mainly text.

Major advances like blogs and wikis are just text! – E-commerce will dominate trade – dot-coms note, it is < 10%

People who invested in predictions, like Internet bandwidth and video-conferencing, lost money


4

Bleeding Edge Theory• Media Richness Theory

– Rich medium rich message?

– A “richness dimension?

– “Lean” e-mail, chat & text-messaging are the success stories!

• Technology Acceptance Model (TAM)– Usefulness+Usability define end-user acceptance

– Yet Security, Privacy and Reliability are key issues!• Microsoft spent millions on NT to make windows more reliable

– “Mr Clippy” (Word’s Assistant) was a Bayesian smart friendly animation, yet was such a failure, its exclusion was an XP selling point!


5

Leading Edge Practice• Practitioners develop “unpredicted” products like:

– E-mail: Considered a too simple text messaging system

– Chat rooms: Not multi-media, so not predicted

– WWW: Pundits argued it needed centralized control

– HTML: Experts felt it was a too-simple tag language

– Cell phones: Originally thought to be a yuppy toy

– Google: Was a simple search engine (unlike multi-mediaYahoo)

– Text messaging: Why text with a multi-media phone? Because!

Is IS theory out of touch with IS Practice?


6

Yet we need theory• Imagine if we ran an atomic or nuclear program

by trial and error. • Imagine if we ran the space shuttle program by

trial and error • Are we developing the technical infrastructure

of the future World Society by trial and error? – Theory distills experience, and lets us avoid costly

mistakes. It prevents predictable errors.

We Need Better IS Theories!


7

Question

• Propose: The problem is a limited view of system performance.

• Question: What is system performance?– Is it one, two or multi-dimensional?– Is it absolute or relative to the system environment?– What happens when performance increases?– How does one produce high performance systems?


8

Performance Tensions• Alexander’s “Notes on the Synthesis of Form” raised

the idea of “design tension” over 40 years ago. IS developers now use it as “Pattern Theory”. It argues that system performance is always a balancing act.

• A vacuum cleaner is a design “form” in the following problem design space:

Reliability

Usability

Functionality

Cost


9

Evolution Resolves Performance Tensions

• Successful biological systems range from simple viruses to powerful predators– They balance performance design tensions differently– Not just the strong are “fit”– One sided “excellence” can cause extinction

• Likewise, information systems may have a virtual “evolution”, where success has many forms, e.g.– Mobile laptops vs more powerful desktops– “Light” background utilities vs mainline applications– One purpose add-ins vs all purpose suites– Open source public vs closed private systems


10

System Levels

1. Social: norms, culture, laws, zeitgeist, sanctions, roles (Sociology)

2. Personal: semantics, attitudes, beliefs, opinions, ideas (Psychology)

3. Software: programs, data, bandwidth, memory, processing (Computing)

4. Hardware: computers, wires, printer, keyboard, mouse (Engineering)

The model proposes four levels of “system” :


11

Overlapping levels• Each level is a different “view”, not a different

system! (See Alter, Grudin, Kuutti, 1996)

– Meaning --> Culture: Personal meanings common across a group that continue over time create a culture Data --> Meaning: Data/Information creates personal meaning (based on human psychological processes)

– Physics --> Data: Physical hardware actions create data or information (as Shannon & Weaver define it)

• Each level depends on and “emerges” from the previous

• Higher levels offer higher performance benefits e.g. the benefits of cooperative society


12

People vs Technology?• A plane is a technical system, but plane plus the

pilot is also a system. How should people relate to technology? – Human like technology? - machines that replace people,

e.g. the Terminator, IBM's Deep Blue, AI the movie.– Technologized people? - machines that mechanize people,

e.g. from the industrial dark ages, to the Matrix. – People dominated by technology add-ons? - e.g. Star

Trek's Borgs, Star War’s Darth Vader.

• Human and machine can only work together if technology supports human processes


13

• Define the human process, then design the technology– Human Process Technology Support– 1:1 conversation Email – Many sense processing Multi-media system– Trading Ebay type systems– Human foraging Browser design– Group conversations Chat– Associative memory Hypertext– People learn by feedback The wonderful back button– Normative behavior Reputation systems

• NOTE: THERE IS NOT JUST ONE SOLUTION!

Technology support for human processes


14

Levels combine

Social

Personal

Informational

Mechanical Hardware system

Software

system

Human-computer interaction (HCI) system

Socio-technical system (STS)


15

Social-Technical Systems• Hardware Systems involve physical exchanges• Technology Systems are hardware + data flows• HCI Systems are technology plus people • Social-technical Systems (STS) are social systems,

built on HCI systems, built on software systems, built on hardware systems

• A social-technical systems involves all four system levels

• The Web of System Performance (WOSP) is a multi-goal model of the IS design space for any level of a social-technical system


16

System performance involves?

• Four common system elements:– Boundary: monitors system entry/exit– Internal structure: supports and controls – Effectors: generate output effects– Receptors: process input signals

• Examples:– People: skin, brain and organs, muscles and

senses– Computers: case, mother-board architecture,

peripheral output and input


17

System performance is?

How well a system interacts with its environment

• An environment can offer any system: – Opportunity – environment gives benefits– And/Or– Risk – environment gives damage or loss


18

WOSP Goals• Each system element can be designed to gain value

(opportunity) and/or avoid loss (risk):– Boundary

• To enable useful entry (extendibility).• To deny harmful entry (security).

– Internal structure• To accommodate external change (flexibility).• To accommodate internal change (reliability).

– Effector• To maximize external effects (functionality).• To minimize internal effort (usability).

– Receptor• To enable meaning exchange (connectivity).• To limit meaning exchange (privacy)


19

A System is not “high performance” if:

1. Ineffectual – it cannot do the job.2. Unusable – you cannot make it work.3. Unreliable – it breaks down often.4. Insecure – it succumbs to viruses.5. Inflexible – it fails when things change.6. Incompatible – it cannot import standard

plug-ins or data. 7. Disconnected – it cannot communicate.8. Indiscreet – it reveals private information.


20

Effectors: change external environment

Functionality(what it can do)

Usability(reduced effort to to)


21

Structure: controls and maintains

Flexibility(performs differently)

Reliability(performs the same)


22

Boundary: controls entry/exit

Security(can reject)

Extendibility(can accept)


23

Receptors: control communication

Connectivity

(can communicate)

Privacy

(can hide)


24

Functionality

Usability

Connectivity

Security

Privacy

Reliability

Flexibility

Extendibility


25

• Goal: To act on the environment• What can the system do?• Also called: functionality,

capability, power, usefulness, effectiveness

• Examples:– A car that goes fast, stops well,

turns well etc

– Software that does what the user wants

Functionality


26

• Multi-functional• All purpose (e.g. Office Suites)• Feature driven (“bloatware”)• Can always “get the job done”• Complicated - large menus• Most people only use 50% or less

of their potential • Hard to use

Functional systemsTend to be:


27

• Goal: To operate efficiently or easily, to minimize the resource costs of action

• How easy is it to operate/run?• Also called: easy to use, simple,

user friendly, efficient• Examples:

– A car that is easy to drive, with simple controls

– People can use usable software without a lot of training or help

– HTML is usable, email is usable

Usability


28

• Easy to use, learn and operate (intuitive)

• Require little training, help or documentation

• Provide only what is necessary now (contextual screen information)

• Simple, focus on essentials, remove unnecessary “bells or whistles”

• Less powerful

Usable systemsTend to be:


29

• Goal: To continue operating despite internal data or component failure (errors)

• How often can it perform?

• Also called: Dependability, stability, trustworthy, ruggedized, MTBF, robustness, durability, maintainability, recoverability.

• Examples:– A reliable car starts every day

– Reliable software doesn’t “hang” with internal “bugs”, or data errors

– Windows reliability has increased over the years - XP was sold on this

Reliability


30

• Be long lasting (so lifetime warrantees are economic)

• Be predictable (so users can plan)

• Have Redundant parts - if one part fails another takes over

• Be minimally coupled - one error does not cause another• Recover quickly from breakdown (module or data recovery)

• Have Undo/Back operations to reverse errors

• Be hard to change

Reliable systemsTend to:


31

• Goal: To still work in new environments, e.g. by changing its operation

• Where can it perform?• Also called: Adaptability, tailorability,

customizability, portability, platform independence, plasticity, agility, modifiability, pervasive

• Examples:– An all-terrain vehicle works anywhere– Mobile/pervasive computing lets people use

software in any environment

Flexibility


32

• Easily customized to individuals (e.g. control panel)

• Easily “fitted” to different situations (e.g. hardware independent software, O/S independent programming languages)

• Able to learn usage trends (e.g. cache prediction algorithms)

• Demand driven (e.g. CSMA/CD Ethernet vs polling networks)

• Interconnected, holistic - one change can affect all • Error prone (e.g. changing Windows 95 screen display could be

catastrophic! - until an Undo was provided)

Flexible systemsTend to be:


33

• Goal: To use outside components/data• With what can the system perform?• Also called: Tool use, openness, scalability,

interoperability, standards, permeability, compatibility

• Examples:– Cars can fit add-ons (e.g. towbar, spoiler, rack ..)

(if towbar ball is standard size)– Open software, like Netscape, accepts “plug-ins”– Clip and paste between applications allows one

system to use another’s data easily

Extendibility


34

• Be able to accept third-party “add-ons” (Open IBM PC vs McIntosh sealed system)

• Be open source (system code openly displayed)• Support general standards • Be easily added to - Open Systems Architecture

“Plug & Play”• Be easily scalable (like the World Wide Web)• Have decentralized control – object orientated • Not be as secure

Extendible systemsTend to:


35

• Goal: To resist or repel unauthorized entry, misuse or take over

• Who controls the system?• Also called: Anti-virus, firewall,

protectiveness, defense, integrity, safety, integrity

• Examples:– A secure car’s lock cannot be picked– Secure software (and data) cannot be

used or modified by hackers or viruses

Security


36

• Centrally controlled, bureaucratic• Strong on boundary checks (logon codes,

passwords)• Resistant to external attack or entry• Internally idiosyncratic• Risk avoiding rather than opportunity

enhancing • Hard to add on to, not opportunistic

Secure systemsTend to be:


37

• Goal: To communicate with other similar systems

• Who can we talk to?• Also called: Communication,

information exchange, networking, sociabilty, interactivity

• Examples:– A connected car could sense other cars

and avoid a crash– Connected software uses the Internet:

email, chat rooms and groupware

Connectivity


38

• Be informed by other systems, and “group aware”

• Be up to date by download/upload • Have many channeled (multi-media), two-

way duplex (reciprocal), high linkage (many-to-many) communication

• Have a low “cycle time” (user complaints, software patches)

• Experience information overload/distraction

Connected systemsTend to:


39

• Goal: To control internal information release and disclosure

• Who sees us?

• Also called: Tempest proof, opaqueness, privacy, autonomy, camouflage, stealth confidentiality, secrecy

• Examples:– Car with tinted windows, a radar scrambler

– Privacy software controls interactions with others (e.g. Black Ice, Zone Alarm)

Privacy


40

• Be regenerative (“packing” a file needs privacy)

• Restrict information use (e.g. digital signatures copy protection)

• Be information encrypted • Respect rights to own information, where one

needs permission before data access • Protect personal information (documents, credit cards) • Hard to connect to

Private systemsTend to:


41

Performance is a Balance


42

Derived vs Observed

• WOSP goals were derived from the general nature of systems

• Yet match observed system requirements quite well

• Current system requirements are very confused and confounded

• E.g. are privacy and reliability part of security?


43

WOSP Area = Performance• The WOSP area best estimates “performance”• Performance (the area) needs all the WOSP dimensions• ‘Experts” predict based based on recent advance(s)• If the most recent advance is the strongest dimension• It also has the greatest “tension” with other factors• So it soon gives diminishing returns• Then, IS progress moves “unexpectedly” to a dimension where the tension is less• The gaming development of the last decade was not graphics but

social gaming (MMORPGs massively multiplayer online role-playing games)


44

WOSP Lines = Tensions • WOSP lines are cross-cutting tensions• Increasing any aspect of performance means any other

can “bite back” (Edward Tenner)

– “Open” systems can increase security risks, and more security makes systems less open eg US.

– More functions can mean more for users to learn– Flexible control panel can reduce reliability, when users

change the wrong things

• Yet one can have security and openness, ease of use and functionality, connectivity and privacy, etc

• As systems advance, progress needs tension resolution


45

Innovation can resolve tensions

• Find innovative ways to reconcile“opposites”– Functionality + Usability = elegance – Extendibility + Security = discrimination– Reliability + Flexibility = autonomy – Connectivity + Privacy = legitimacy

• Avoid one-dimensional progress (it bites back)

• Find progress at the points of least tension

System design is an art as well as a science


46

Expanding the Web• In 1992, Apple CEO John Sculley introduced the hand held Newton, saying portability (flexibility) was the wave of the future – he was right

• But the Newton’s portability reduced its data input usability, and in 1998 Apple dropped the line.

• When Palm’s Graffiti language improved handwriting recognition, the PDA market took off.

• Innovations may need combination advances


47

Why are “killer apps” functionally simple?

• A basic email system can be written in a weekend, yet email is a killer app

• When systems begin, the web is “slack”. As they evolve, tensions arise

• A simple new application allows WOSP performance expansion

• A powerful new function, like video e-mail, is much more difficult to support all-roundMinimize functionality to increase your system

performance!


48

WOSP shape/profile and Environment “Fit”

• Performance has no “perfect” form• In opportunity environments right action gives benefit,

favoring the four success creating goals:– Functionality, Reliability, Extendibility and Connectivity

• In risk environments, wrong action gives great loss, favoring the four failure avoiding goals:– Security, Privacy, Usability, Flexibility

• Different environments, organizations and applications favor different performance “shapes”


49

Analyze the performance profile

• What % of your project should be spent on: – Security Resist outside attack/take-over?– Extendibility Use outside components/data?– Functionality Required functionality?– Usability Conserve user/system effort?– Reliability Avoid/recover from internal failure?– Flexibility Predict/adapt to external changes?– Connectivity Communicate with other systems?– Privacy Control self-disclosure?

• Don’t spend 90% of project effort on what will be only 50% of system performance


50

Convert WOSP goals to performance requirements

• For a browser– Privacy

• Any sensitive information I give the browser, like logon passwords, is encrypted, so others can’t see it.

• Password information always shows as asterisks, so others cannot look over my shoulder to see them

• It stops web sites from getting my name or email from my computer’s data.

– Security• It can detect and prevent spyware from installing• It can detect and prevent popup ads


51

System design “layers”• Specialist specifications/teams for:

– Functionality: Traditional specs (main module)

– Usability: User costs (HCI) (interface, help, wizards)

– Reliability: Error analysis (error & recovery code)

– Flexibility: Portability analysis (control panel, preferences)

– Extendibility: Compatibility analysis (plug-in manager, import/export data, clip and paste)

– Security: Threat analysis (logon/registration module)

– Connectivity: Internet/network analysis (comms module)

– Privacy: Legitimacy analysis (rights control module)


52

Design Integration

• For projects to produce known and established systems, whose goals are well known and well defined, specialization and specialists maximize results (they give more local output per local criteria)

• For projects to produce new systems, whose goals are not well known or defined, requires integration as well as specialization, to increase synergy and decrease cross-cutting conflicts


53

Increasing Integration

• Cross-disciplinary leader or project “guru”– Hard to find

• Increase cross-discipline communication: Regular cross-specialist meetings under multi-discipline chairs – hard to run

• Combine specialty teams. e. g. for: actions (functionality + usability), interactions (security + extendibility), contingencies (reliability + flexibility) and sociability (connectivity + privacy).


54

Extreme Programming

• For small teams producing innovative/new projects– Everyone contributes to every system part– All team members involved in every aspect of

system design– Common entry point for all system changes

• Maximizes system integration (goal synergies)

• Minimizes design conflicts


55

Validation – Conjoint Analysis

• Subjects were told they were company managers who had to choose a new common browser for their company

• They were given 30 browsers to choose from, each with different ratings on the eight WOSP factors

• Then asked to rate and rank the browsers


56

Results: Criteria Weights

Performance Factor

Avg. Importance

Std Dev. 99% Conf % > 12.5

Security 22.78 12.78 16.07-29.50 70.83 Privacy 15.47 9.19 20.30-10.64 58.33

Usability 14.16 9.88 19.36-8.97 50.00 Functionality 12.02 8.21 16.33-7.70 29.17

Reliability 11.64 8.15 15.93-7.36 33.33 Connectivity 9.24 6.54 12.68-5.80 33.33 Extendibility 7.69 4.56 10.09-5.30 16.67

Flexibility 6.99 6.46 10.39-3.59 16.67

Security, Privacy & Usability came before Functionality!


57

Analytical Hierarchical Processing Study

• TAM proposes perceived usefulness and ease of use affect technology acceptance (2 factors)

• Subjects used both TAM and WOSP criteria

• Subjects rated 3 browsers using AHP pair-wise comparisons

• They then compared the two methods


58

Results: TAM vs WOSP

Effect N TAM WOSP df MS F p

Confidence 20 3.7 2.4 1 16.9 12.8 .002*

Accuracy 20 3.6 2.45 1 13.2 12.4 .002*

Completeness 19 4.21 2.42 1 30.4 16.8 .001*

Outcome Satisfaction 18 3.14 2.35 1 5.64 9.63 .006*


59

CA vs AHP WOSP Ranks

CRITERIA AHP WEIGHTS RANK

LOCAL GLOBAL AHP CA

Extendibility 0.21 0.04 8 7

Security 0.79 0.17 2 1

Flexibility 0.43 0.09 7 8

Reliability 0.57 0.12 4 5

Usability 0.58 0.09 3 3

Functionality 0.42 0.12 6 4

Privacy 0.64 0.20 1 2

Connectivity 0.36 0.11 5 6

Has the Web become a threat environment?


60

What Users Want• Functionality and Usability

• Reliability and Flexibility

• Security and Extendibility

• Connectivity and Privacy

• They want it all!

• Why?

• We ourselves are such a balanced system, so we expect no less of the systems we work with


61

Design Tips Summary• Define the human process, then design the technology• Less (functionality) may mean more performance (avoid

the Version 2 flop problem)

• Match project effort to your app’s performance profile• For new applications, increase design integration by

cross-disciplinary staff, or more/better communication

• Expect cross-cutting conflicts• Convert WOSP goals to technology requirements• Check your web of system performance - success

needs many causes, but system failure needs only one


62

Design Questions

• What is the human process/task your technology will support?

• What is your system level? (S/W,HCI, STS)• What is your application’s performance profile?

(e.g. your top 3-4 criterion goals are?)• What are specific requirements of each goal?• Do you have cross-cutting design issues? • What designs satisfy this “performance space”?


63

ConclusionExcellence requires Balance

See brianwhitworth.com for example papers: 1. The Web of System Performance2. Polite Computing3. Spam and the Social-Technical Gap4. Legitimate by design5. Voting before discussing


64

TAM• TAM proposes perceived usefulness and perceived ease

of use affect technology acceptance

• Is a system characteristics tension model

Perceived Usefulness

Perceived Ease of Use

Behavioural Intention

Use Behaviour


65

Unified Theory of Acceptance and

Use of Technology (UTAUT)Performance Expectancy

Effort Expectancy

Behavioral Intention

Use Behavior

Social Influence

Facilitating Conditions

Voluntariness of use

ExperienceAgeGender


66

WOSP & UTAUT

WOSP Criteria

Behavioral Intention

Use Behavior

Social Influence

Facilitating Conditions

Voluntariness of use

ExperienceAgeGender


67

AHP Design

Select for system performance

WOSP dimensions

TAM dimensions

Software

WOSP rankings TAM rankings

Main Goal

Criterions (Cx,,cxi)

Alternatives (ay)

Output (wxy,wxiy,wy).


68

Technology acceptance

• Three factor types affect technology acceptance:1. System Characteristics: Is it useful, easy to use, secure,

etc? e.g. TAM (original)

2. Individual Characteristics: Age, gender, experience, attitude to computers. e.g. social cognitive theory

3. Organizational Characteristics: Corporate values, technology infrastructure, social structure, normative effects. e.g. innovation diffusion theory

WOSP is a system characteristics tension model, but still compatible with theories 2. and 3.


69

Summary

• One-dimensional progress bites back• Progress is a train on many tracks at once• Advanced system performance has many goals • Reconciling these goals requires innovation• Innovation finds the points of least tension• System design is an art as well as a science,

needing integration as well as specialization• The “killer” advances of the last decade (email,

browsers, chat, blogs) succeeded by balance as well as excellence

©Brian Whitworth 2005, [email protected] THE WEB OF SYSTEM PERFORMANCE (WOSP) Excellence requires...

Documents

Transcript of ©Brian Whitworth 2005, [email protected] THE WEB OF SYSTEM PERFORMANCE (WOSP) Excellence requires...