Dependable Systems Case Studies - uni-potsdam.de · •Three primary flight control computers...
Transcript of Dependable Systems Case Studies - uni-potsdam.de · •Three primary flight control computers...
Dependable Systems Course PT 2012
Boeing 777
• $4 billion-plus development, expected half-century of service, 1994 first lift-off
• Fresh start for Boeing in plane design and building
• 100% ,paper-less‘ design with 3D modeling on computer
• 2200 designer work stations, hooked to 8-node cluster of IBM3090-600 mainframe, 3 TB data
• Saving in engineering mock-ups due to simulation (cable and wire runs)
• First time complete involvement of the airlines, shifted from streamlined development model to parallel design teams
• Concepts adopted in re-newed 737, F-22 fighter and ISS
• Constant development of new versions (6 models so far)
2
Dependable Systems Course PT 2012
Boeing 777 - Reliability
• Mechanical maintainability - line-replacable unit
• Sealed modular component of an airplane, ship, or spacecraft
• Designed to be replaced quickly at an operation location for cost reduction
• Designed to a common specification: plugs, installation tools, bulk, weight, flammability, resistance to radio emissions and damage from environment
• Hardware faults vs. software errors
• Increasing problem in LRU‘s
• Hardware MTBF steadily got better, but MTBUR (Mean Time Between Unscheduled Removals) has not kept pace
• Replacing ,good‘ with ,good‘ in the hope to solve a problem
• Reasoned by software or design errors that did not anticipate flight conditions
3
Dependable Systems Course PT 2012
Boeing 777 - Design Diversity
• Based on Boeing experience, the most likely design errors are
• Requirement errors and implementation misunderstanding
• Software design or coding error
• Future process errors in previously qualified procedures
• Semiconductor parts
• Non-deterministic modern circuit design
• Dissimilarity design through
• Dissimilar software / hardware architectures, based on different designs
• Ada remains accepted standard for embedded programming
4
Dependable Systems Course PT 2012
Boeing 777 - Common Mode Fault Model
• Electrical faults or electrical power failure, hydraulic failure, structural damage
• Impact of objects, electromagnetic environment, lightning strike
• Radiation or ash cloud environment in the atmosphere
• Rough or unsafe installation and maintenance
• Basic counter-measures
• Triple redundancy for all hardware resources: computing system, airplane electrical power, hydraulic power, communication paths
• Fail-passive electronics
• Computer architecture must consider common mode faults and dissimilarities
5
Dependable Systems Course PT 2012
Boeing 777 - Electromagnetic Threats
• Increasing threat from electromagnetic energy
• Increased reliance on electronic systems for safe flight and landing
• Reduction of operating power of electronic devices
• Increased percentage of composite materials with less inherent shielding
• Lightning produces the most intense EME - large inducted voltages
• Increased probability of digital circuit upset resp. ,soft fault‘ occurrence
• Options:
• Distributed bus architecture, error detection and corrective schemes, fiber optic data transfer, computation recovery
6
Dependable Systems Course PT 2012
Boeing 777 - Software
• First digital airliner - only partial use of digital avionics until then
• More than 2.5 million lines of code (400.000 in the 747-400), including avionics and cabin-entertainment
• 600.000 lines for Airplane Information Management System (AIMS) by Honeywell
• Dual cabinets, each with 4 core processors and 4 I/O modules; can operate with 3
• Handles flight management, cockpit displays, central maintenance, condition monitoring, communication management, engine data interface
• Functions share processors, memory system, operating system, utility software, hardware test-equipment, and I/O ports
• Reliability based on software partitioning and standardized hardware modules
• Interfaces with airplane through standardized busses - ARINC 629 and ARINC 429
7
Dependable Systems Course PT 2012
Boeing 777 - AIMS Approach
• Hardware monitoring on every CPU clock cycle
• All computing and I/O resources are self-checking based on lock-step
• Immediate trap on error detection to avoid further data exchange
• Critical functions have shadowing standby resource
• Master self-checking pair is decoupled by SafeBUS on detected error ->shadow output is shown instead
• Duplicated state data allows automated recovery on soft error
8
Dependable Systems Course PT 2012
Boeing 777 - Fly-By-Wire
• First with Airbus 320 in 1988
• Fly-by-wire is now standard in all airplanes
• Lighter wing and tail structures, since no more need for complex and heavy mechanical cables, pulleys, and brackets
• Three primary flight control computers (PFCs)
• Each PFC with 132.000 lines of Ada code
• Calculates control commands for actuators, trim system, and control column feel system (haptical feedback as in mechanical steering)
• Input from control yoke (manual mode) or triplex autopilot
• Airbus A320 fly-by-wire additionally performs flight envelope protection
• Electrical command transmission demands heavy shielding, fly-by-light in future
9
Dependable Systems Course PT 2012
Boeing 777 - PFC
• Flight computer controls electric and hydraulic actuators by sending commands
• Three data buses, physically and electrically isolated, not synchronized
• Three internal computational lanes per PFC, compiled with different Ada compilers
• Each lane receives from all busses, but a PFC sends only to one of them
• Processors: Intel 80486, Motorola 68040, AMD 29050
• Wiring separation and protection from foreign object collision
• Left - Center - Right distributionfor critical components - power, busses, flight controller, hydraulics
10
Dependable Systems Course PT 2012
Boeing 777 - Median Value Select
• Three lanes per PFC, one in command mode, the others in monitoring mode
• Command lane sends proposed command output to PFC-assigned output bus
• Performs median select between two other PFC results and own result
• Send chosen result on the bus
• Ensures fault blocking until PFC reconfiguration
• Monitoring lanes monitor their own command lane
• Cross-lane inhibit hardware logic for automated fault treatment
• Channel outages are detected by cross-channel comparison
11
Dependable Systems Course PT 2012
Boeing 777 Fly-By-Wire
• On total PFC failure, backup analog path is available through actuator electronics
• A/D converts transfer stick input into a ,mechanical link‘-alike direct command
• Multi-redundant power system needed for PFC operation
• Three constant-speed drive generators at engines, based on hydraulics
• Backup power shaft per engine, also based on hydraulics availability
• Backup ram-air turbine in the wing root, drops into airstream on demand
• 12 data bus networks (ARINC real-time bus system, 2Mbps, TDMA)
• Data conversion units for analog bus connectivity to low volume devices
13
Dependable Systems Course PT 2012
Boeing 777 - Fault Diagnosis
• Fault diagnosis and isolation capabilities with On-Board Diagnostic & Maintenance System (OMS) - optimized to deal with system complexity
• Major problem: Visual inspections is not enough to perform root cause analysis
• Maintenance crew deals with a set of black boxes full of microelectronics
• Heavy need for Built-In Test Equipment (BITE) to find faulty module reliably
• Central Maintenance Function (CMF)
• Assist in the analysis of flight deck effects and crew complaints
• Diagnose reasons behind symptoms, isolate to the Line Replacable Module (LRM), bring airplane back to full operation within allocated time
• Airplane Condition Monitoring Function (ACMF)
• Captures parameters based on trigger conditions for long term analysis
15
Dependable Systems Course PT 2012
Boeing 777 - OMS Engineering Approach
• OMS implementation based on model of the airplane, instead of many logical equations
• Individual models per sub-system for fault-response behavior
• Alignment with BITE coverage specifications
• Results in diagnostic model
• Model maps symptoms to faults at runtime
• Maintenance personnel has to trust CMF, rather than replacing boxes under suspicion
• Reduction of nuisance messages extremely important for credibility of solution
17
Dependable Systems Course PT 2012
Boeing 777 - Central Maintenance Function
• Central reasoning system consolidates symptoms from multiple LRUs
• Suppresses secondary symptoms from downstream LRUs, which are not aware of the global airplane state that might render their information invalid
• Correlation of flight deck effects to fault reports possible
• Modules store fault information in non-volatile memory, analyzed by Honeywell
• Flight number, date, airplane ID, LRU position (center, left, right), software part number, selected options, fault code, temperature, number of times occurred, flight phase and UTC when the first fault occurred
• Sending in a black box, instead of traditional custom repair
18
Dependable Systems Course PT 2012
Telephone Switching Systems
• Classical field for high-availability systems since the 60‘s
• 24/7 availability expectation from customer, designed for long-range operation
• Expected downtime less than 1 hour for 20 years (USA)
• Allowed to interrupt 1 out of 10.000 calls, dial tone within 1 sec
• Database, software and configuration changeable without affecting call processing
• Initiation of new calls might be delayed, calls in progress should never be interrupted
• Popular examples and publications for AT&T 5ESS system
• Statistics: 30%-60% hardware failures, 20%-25% software, 30% operational
• Traditional strategies: High reliance on defect elimination and fault recovery
19
Dependable Systems Course PT 2012
Outcomes after 10 years of operation
• Software defects tend to concentrate in specific parts of the system
• Eliminating clusters of defects is more economical than eliminating isolated defects
• Knowledge of clustering can help in placing test code
• Large class of software bugs is caused by human inability to catch complexity
• Many software bugs arise from design incompleteness
• Design holes are reduced with increasing software maturity
• 38% of software deals with recovery
• Increased utilization of reusable robust software blocks
• Distribute when appropriate, move from hardware to software (cost effectiveness)
20
Dependable Systems Course PT 2012
Commercial Fault-Tolerant Systems
• Few major players since the 60‘s
• IBM: From S/360 to zSeries, Tandem / HP NonStop, Stratus, Marathon
• Application examples: automatic teller systems, credit card authorization, retail point-of-sale, stock trading, funds transfer, cellular phone tracking and billing, 911 emergency calls, electronic medical records, travel and hotel reservations
• Different availability demands
• No downtime at all (ATM, three-shift manufacturing)
• 100% availability within given time frame (stock market)
• Infrequent short interruptions are allowed (medical records)
21
[Bar
tlett
& S
pain
how
er]
Dependable Systems Course PT 2012
Stratus Technologies
• Founded in 1980, competitor to Tandem
• Product line originally based on Motorola MC68000, then HP PA-RISC, now Xeon
• Multics-inspired operating system VOS for fault-tolerant OLTP
• Entry-level fault tolerance solution for Windows / Linux with ftServer product line
22
Dependable Systems Course PT 2012
Stratus ftServer
• Logic separation between CPU / memory subsystem and I/O subsystem
• Custom chipset as PCI bridge between CPU and I/O
• Rely on custom backplane for message exchange
• Provides error detection, fault isolation, and lockstep synchronization
• Custom logic within the CPU / memory subsystem
• Primary PCI interfaces, interrupt management, transaction ordering logic
• Standard DMR mode
• Custom logic within I/O subsystem
• Voting logic compares I/O output from all motherboards
• Fault-tolerant I/O through replicated busses, I/O adapters, and devices
23
Dependable Systems Course PT 2012
Stratus ftServer Software
• Unmodified standard Windows Server / Linux
• Only hardened drivers for PCI devices in use, supporting hot-pluggability
• On problem, PCI I/O adapter is isolated from the rest of the system
• Software is shielded from transient hardware errors
• Automatic Reboot
• On reboot, OS crash information is kept in replicated CPU memory
• Failsafe System Software
• System software tracks configuration and revision levels
• Active Upgrade - split redundant system online, apply patches to one side, re-sync physical servers to act as one logical server again
24
Dependable Systems Course PT 2012
Stratus ftServer Software
• Protection of in-memory data against hardware failures by lockstep approach
• Quick dump
• On OS failure, only one half is restarted, the other is used for crash dump analysis
25
Dependable Systems Course PT 2012
NonStop
• 1974: James Treybig + some HP people founded Tandem Computers
• Business goal to build fault-tolerant computer systems for transaction processing (banks, stock market)
• NonStop architecture - Independent processors and storage devices, hardware-based fast failover
• 1997 taken over by Compaq (Himalaya servers), which was bought by HP in 2002
• Well-maintained system manage 5 nines
• Linear scalability and fault intolerance resp. fail fast behavior as design goals
26
Dependable Systems Course PT 2012
NonStop
• NonStop architecture
• Independent nodes communicate with each other and with shared I/O modules
• Self-checked processor modules - either provide correct result, or stop silently
• Shift from custom processors (Cyclone, ‘91) to commodity (MIPS R3000,’91)
• Custom solutions had dual path redundancy for duplicated CPU parts and according redundancy codes
• With commodity processors, lock-step operation was introduced to form a logical processor from one clock
• Redundancy codes in memory and caches to stop on memory errors
• Supported by operating system error detection and task migration capabilities
27
Dependable Systems Course PT 2012
NonStop
• Each hardware component is self-checking and has ,fail-fast‘ behavior
• Improves fault containment and makes isolation of fault easier
• Custom operating system Guardian / NonStop Kernel (NSK)
• Special programming paradigm for ,always‘ fail-safe transaction processing
• Standard OS services, optimized messaging system, task migration support
• Applications run in primary / backup copy mode
• State changes are communicated to backup instance with system mechanisms
• Automated routing of messages to backup system
• Today application implementation typically on-top-of Tuxedo transaction monitor
• Example: NonStop SQL database with linear scaling
29
Dependable Systems Course PT 2012
NonStop
• ServerNet
• High-speed, low latency, packet switching network for IPC and I/O
• DMA transfers
• Two independent fabrics used at the same time
• CRC recalculation at every router, to isolate link failures
• Fault tolerance for storage data by end-to-end checksums and mirroring
• Support for long distance
30
Dependable Systems Course PT 2012
NonStop Advanced Architecture
• Updated approach: Bernick, D. et al., 2005. NonStop Advanced Architecture. In: International Conference on Dependable Systems and Networks.
• Duplication and tight lock-stepping no longer works
• Power management with variable frequencies do not work with lockstepping
• Multiple clocks and asynchronous interfaces on one die
• Higher soft error rate through density is fixed with low-level fix-up routines
• Modern processors with CMP design - disruption of multiple processor TMR
• Market price pressure
• New approach retains the logical structure, but introduces new error detection
• Compare IPC and I/O output from multiple servers
31
Dependable Systems Course PT 2012
NonStop Advanced Architecture
• Modular redundant servers, built from standard 4-way Itanium systems
• Unique synchronization mechanism for fully compared operations -loose lockstep
• Different clock rates, independent error retries and fixup routines, independent cache hit / miss, independent instruction streams
• All two / three server outputs are compared
• Partitioned memory, basedon Itanium protection key mechanism
• Voting units compare output fromdifferent slices for one logicalprocessor
32
Dependable Systems Course PT 2012
NonStop Advanced Architecture
33
Logical Synchronization Unit (LSU)
Independent 4-way Itanium node (Slice)
Dependable Systems Course PT 2012
NonStop Advanced Architecture
• Each logical processor has one or two LSU‘s
• LSU failure does not affect the slices
• Two LSUs in TMR protects against any two hardware faults
• All PEs of a logical processor must take page faults at the same point
• Different cache states and TLB entries do not influence voting
• Data and control speculation in Itanium disabled for symmetric behavior
• Input data from SAN is written into slice memory
• I/O complection notification to all PEs to make data readable (ensured by operating system through page fault mechanisms)
• Same for active outbound I/O buffer
34
Dependable Systems Course PT 2012
NonStop Advanced Architecture
• LSU is designed with complete self-checking design
• Can be replaced online without affecting any logical processor
• First implementation with FGPA voter and ASIC ServerNet interface
• With DMR, error might be self-identifying (e.g. bus), otherwise logical processor shutdown, which is accepted by the NonStop architecture
• Probation bit
• Simple heuristic to disambiguate a DMR voter miscompare
• Probation vector with one bit per slice, voter against slice with bit set
• Set for short time after slice restart (on error) or exceeding correctable error rate
35
Dependable Systems Course PT 2012
NonStop Advanced Architecture
• Reintegration of processing elements
• Copies state of running PE into memory
• Happens while logical processors are online
• <10 min for 32 GB logical processor
• Based on reintegration link hardware betweenprocessor chip set and main memory
• Slice in normal operation ignores memorywrites announced on the link
• CRC-protected, checked always
• Copying by background process
• Source PE flushes caches as part of the process
36
Dependable Systems Course PT 2012
IBM zEnterprise
• The flag ship in Mainframe technology, dedicated ,world‘ of terminology
• Initial concepts of original S/360 (1964)
• Ease assembling of redundant I/O, storage, and CPUs
• Built-in checking against hardware faults, regardless of application
• Built-in hardware fault-locating aids are essential
• How to react on faults
• Ensure system integrity
• Provide uninterrupted applications
• Repair without disruption
• Fail fast philosophy, transparent retry and recover
37
12
IBM System/360 – 19643rd generation
• Customers were frustrated with the migration costs that came with each processor upgrade
• IBM developed a family of processors using the same durable architecture– published in the S/360 Principles of Operations– 24-bit addressing
• Solid logic circuit cards• Common peripheral devices• One operating system - initially
– Operating System/360 (OS/360)
36
2011• IBM System z10 Business Class
– based on innovative System z10 technology
– 130 capacity settings• 1 to 5 Processing Units (PUs)• 26 – 3000+ MIPS
– 4 to 248 GB Processor Storage– HiperSockets– ESCON, FICON Express, FCP Channels– PCICA encryption assist – OSA Express & OSA-ICC– SAP, IFL, ICF, zIIP, zAAP– IBM Enterprise Storage Server (Shark)
• high availability, high performance• Flashcopy and PPRC• 1 to 55.9 TB capacity
– IBM 3592 TotalStorage Enterprise Tape • 900 GB/cartridge at 3:1 compression• up to 40 MB transfer rates
• z/VSE V4.3– 64-bit real addressing– Turbo dispatcher
• support for n-way processors– CICS Transaction Server
• availability and z/OS affinity– TCP/IP
• offered under agreement with CSI– z/VSE Navigator– z/VSE e-business connectors– z/VSE Web services SOAP/XML– HiperSockets– PCICA encryption assist– TS1120 tape encryption support– Shark Flashcopy and PPRC– FCP-attached SCSI disks– Sub-capacity pricing
Dependable Systems Course PT 2012
zEnterprise: RAS Strategy
38
!"
# !$%%&'()&*+,-+,./0+122
'()&341/5,-,065
3%78&(++9&:.;+</
!"!#$#%&''()*+,-.*,/0-12#"113*4#1,
(/0*,#"113*45/6789#:-,#;3*28<
&#=>?#@:A#?BC#D#@E;
FG H": %IJ#H=!!K%''<<#L-.M
%NJ#H=!!K%''<<#L-.M
%%#OC!#"/,4#:KK*<P3-*K&#O*,0-6/3F#L1,-Q120/3
)*/,@,120
H":#;1R*,#E8993-*K
@/2180
"/,4K
"113-2.+,1<>01#!)B
!"!!*<1,S
!*<1,S
# !$%%&'()&*+,-+,./0+128
'()&341/5,-,065
=>?5@<A5@&B*'4CD06,<-/ 0E5&F./ >?56&C&4*6G
FA.115@&H &B)4=&C&D,0E5, &I-J,.@56G&
I16>?5@<A5@&BI'KLG
E18,6*K#1+#?80/.*K##;,*#QT#
UL,K>V*/,>ESK0U;,-1,#E*,W*,K QT#X" Y%'#X" Q%TN
B2K6M*483*4#?80/.*K
E6M*483*4#?80/.*K
;3/22*4#?80/.*K
;,*93/22-2.#,*Z8-,*<*20K
;1R*,#[#CM*,</3#
!/2/.*<*20
!"#$%&'%()*+#,')+-%$)./0%
! ! !! !!
! !!
!!!!
M5N-5,./<,5&O&=0A0>+1&K5A0.P0A0/;&Q+,6/&415N;
Q5.,+</ O&)5>?.10>.A&*+N-+151/6&K5A0.P0A0/;&Q+,6/&415N;R
=;6/5N&3&+E5,.AA&KL=&=/,./5J;
SRR*+1/01<01J&+<,&KL=&T+><6&?5A-6&.E+0@&+</.J56
=<9/60#1+#?
80/.*
Dependable Systems Course PT 2012
zEnterprise: RAS Strategy
• A recovery routine is invoked by the system when the mainline code of the function that it controls fails unexpectedly.
• System provides a system diagnostic work area (SDWA) to the recovery routine.
• Mainline code can set up parameter list to be passed to the recovery routine
• The recovery routine may
• ... attempt to retry the function by invoking the defined retry point.
• Mainline routine can provide footprints indicating recovery/retry actions
• Clean up resources (memory, locks,...)
• ... terminate and escalate to a higher level recovery routine (percolate)
• Multiple recovery mechanisms exist as assembler macros
39
Dependable Systems Course PT 2012
zEnterprise: Outage Types
• Unscheduled Outages
• Advanced Memory RAIM (Redundant Array of Independent Memory) design
• Enhanced Reed-Solomon code (ECC) – 90B/64B
• Protection against Channel/DIMM failures
• Chip marking for fast DRAM replacements
• Mirrored (Storage) Key cache
• Improved chip packaging and condensation management
• Integrated TCP/IP checksum generation/checking
• Integrated EPO switch cover (protecting the switch during repair actions)
• Main focus on implementation in Firmware
40
Dependable Systems Course PT 2012
zEnterprise: Outage Types
• Scheduled Outages
• Double memory data bus lane sparing (reducing repair actions)
• Single memory clock bus sparing
• Support of field repair of interface between processor chip and cache chip and between cache chips (fabric bus)
• Fast bitline delete on L3/L4 cache (largest caches)
• Power distribution using N+2 Voltage Transformation Modules (VTM)
• Redundant (N+2) Humidity Sensors
• Redundant (N+2) Altitude Sensors
• Unified Resource Manager for zBX
41
Dependable Systems Course PT 2012
zEnterprise: Facts
• Memory hierarchy uses write-through cache design and ECC on all levels
• Transient L1 cache error is recovered by CPU instruction retry
• Shared L2 cache is ECC-protected
• Cache delete capability for treating permanent faults
• Redundant paths to I/O modules, all used at normal operation
• Complex inline error checking circuitry
• Support for different operating systems - z/OS (MVS successor), Linux
• Half of the z/OS code is devoted to error handling
42
Dependable Systems Course PT 2012
zEnterprise: Processor
• Instruction fetch and execution units are replicated
• Error check at the end of the pipeline
• R-unit keeps CPU registers and processor checkpoint
• E-units have shadow copy of registers for speed improvement
• All register / cache writes are compared, instruction retry in case
• On fault, overwrite with R unit state
• Since z6, reverted to non-lockstepping and many fault sensors
43
Dependable Systems Course PT 2012
zEnterprise: z196 RAIM
44
!"
# $%""&'()&*+,-+,./0+12"
'()&341/5,-,065
3"72&859:19.1/&;,,.<&+=&'195-51951/&)5>+,<&?8;')@
! !"#$%&'()*'+,'&%&-."'/%#0123A B+:,&)5>+,<&*+1/,+CC5,6&?)*D6@&+,E.10359&01&/F+&-.0,6G&5.HI&)*D&F0/I&=+:, HI.115C6A J'))&/5HI1+C+E<&06&K+L.&MNG&"2&/+&NO&J'))6 -5,&P++QG&-C:EE59&01&E,+:-6&+=&O&A O&J'))6 ?N&+,&O&R(@&-5,&=5./:,5&A !$&+,&2N&R(&-I<60H.C&>5>+,<&-5,&=5./:,54S:.C6&!$&+,&2N&R(&=+,&TU;&.19&H:6/+>5,&-:,HI.65&-5,&=5./:,5&
A 2N&/+&!ON&R(&-I<60H.C&>5>+,<&-5,&P++Q&V&2N&/+&!ON&R(&=+,&:65&?TU;&.19&H:6/+>5,@
! ()45'&%&-."'/%#0123A WI,55&)*D6G&5.HI&F0/I&=0L5 HI.115C6X&WI5&=0=/I&HI.115C&01&5.HI&3"72&)*D&06&,5S:0,59&/+&0>-C5>51/&&>5>+,<&.6&.&859:19.1/&;,,.<&+=&'195-51951/&)5>+,<&?8;')@X WI06&/5HI1+C+E<&.996&60E10=0H.1/&5,,+,&95/5H/0+1&.19&H+,,5H/0+1&H.-.P0C0/056X&(0/G&C.15G&J8;)G&J'))G&6+HQ5/G&.19&H+>-C5/5&>5>+,< HI.115C&=.0C:,56&H.1&P5&95/5H/59&.19&H+,,5H/59G&01HC:901E&>.1<&/<-56&+=&>:C/0-C5&=.0C:,56X
A J'))&/5HI1+C+E<&06&U:-5,K+L. MO"G&"%&/+&!%&J'))6 -5,&P++QG&-C:EE59&01&E,+:-6&+=&Y&Y&J'))6 ?NG&"2&+,&!$&R(@&-5,&=5./:,5&A $%G&O%&+,&"2%&R(&-I<60H.C&8;')&-5,&=5./:,54S:.C6&"2G&2N&+,&"$O&R(&=+,&:65&-5,&=5./:,5X&8;')&/.Q56&$%ZX&?WI5,5&06&1+&1+1[8;')&+-/0+1X@
A N%&/+&72%&R(&8;')&>5>+,<&-5,&P++Q&V&!$&/+&\2O&R(&+=&>5>+,<&=+,&:65&?)010>:>&8;')&=+,&/I5&)"Y&06&2%&R(&V&NO&R(&V&"2&R(&TU;&-C:6&!$&R(&H:6/+>5,&>5>+,<@
! 6-.'7-$8'()45'92/'()*A WI5&T.,9F.,5&U<6/5>&;,5.&?TU;@&06&"2&R(&=0M59G&+:/6095&H:6/+>5,&>5>+,<A '1&6+>5&H.656G&+==5,01E&E,.1:C.,0/<&H.1&-,5L51/&-:,HI.65&+=&.CC&.L.0C.PC5&>5>+,<&01&.&P++Q
# $%""&'()&*+,-+,./0+12$
'()&341/5,-,065
*IN*I!*I$*I"
;U'*
;U'*
*I%
J'))
,:;
<0==
,:;
<0==
,
>
,
,
>
,
J8;)
]
]
]
]
:9"%.#'-='?%&-."'>%@-A%."
+,,! ^+F5,=:C&7%(_2N(&8559&U+C+>+1&H+95
<>B?'690CD.%
! ).,Q01E&/5HI1+C+E<`&1+&I.C=&6-.,01E&155959
! $&J8;)&H.1&P5&>.,Q59! *.CC&=+,&,5-C.H5>51/&+1&/I0,9&J8;)
:92%'690CD.%! *8*&F0/I&85/,<! J./.&A C.15&6-.,01E! *ab&A 8;')&F0/I&C.15&6-.,01E
<E??'690CD.%'F/0#@.%$%'@-&G-2%2$#H'
IJJ'>%1KL! *8*&F0/I&85/,<! J./.&A C.15&6-.,01E! *ab&A 8;')&F0/I&C.15&6-.,01E
<E??',-2$.-CC%.'B!E,'690CD.%! 8;')&85H+L5,<
,8922%C'690CD.%! 8;')&85H+L5,<
)*D%
4**
8;')
]]
]
$[ J55-&*.6H.95&D601E&c:.9&T0EI&J'))6
3"72&8;')&)5>+,<&*+1/,+CC5,&dL5,L05F
Dependable Systems Course PT 2012
zEnterprise: z10 EC Memory Structure
45
!"
# "$%%&'()&*+,-+,./0+12!
'()&341/5,-,065
3%$&4* )57+,8&9/,:;/:,5
<=>=
*?4*@
<=>=
*?4*@4**
A5B5C&"&*.;D5
@58&*.;D5
!"#$%
%2( %2(
"(
@58&*.;D5
!"#$&
%2( %2(
"(
@58&*.;D5
!"#$'
%2( %2(
"(
@58&*.;D5
!"#$(
%2( %2(
"(
# "$%%&'()&*+,-+,./0+12E
'()&341/5,-,065
3%F2&G5H:1H.1/&=,,.8&+I&'1H5-51H51/&)57+,8&JG=')K&9/,:;/:,5
<=>=
*?4*@
<=>=
*?4*@4**G=')&L.,0/8
A5B5C&!&*.;D5
@58&*.;D5
!"#$%
%2( %2(
"(
@58&*.;D5
!"#$&
%2( %2(
"(
@58&*.;D5
!"#$'
%2( %2(
"(
)*+,-$./0123$4,/56789$:;<!$
=13.+6/3
Dependable Systems Course PT 2012
zEnterprise: RAIM Structure
46
!"
# "$%%&'()&*+,-+,./0+12!
'()&341/5,-,065
3%$&4* )57+,8&9/,:;/:,5
<=>=
*?4*@
<=>=
*?4*@4**
A5B5C&"&*.;D5
@58&*.;D5
!"#$%
%2( %2(
"(
@58&*.;D5
!"#$&
%2( %2(
"(
@58&*.;D5
!"#$'
%2( %2(
"(
@58&*.;D5
!"#$(
%2( %2(
"(
# "$%%&'()&*+,-+,./0+12E
'()&341/5,-,065
3%F2&G5H:1H.1/&=,,.8&+I&'1H5-51H51/&)57+,8&JG=')K&9/,:;/:,5
<=>=
*?4*@
<=>=
*?4*@4**G=')&L.,0/8
A5B5C&!&*.;D5
@58&*.;D5
!"#$%
%2( %2(
"(
@58&*.;D5
!"#$&
%2( %2(
"(
@58&*.;D5
!"#$'
%2( %2(
"(
)*+,-$./0123$4,/56789$:;<!$
=13.+6/3
Dependable Systems Course PT 2012
zEnterprise: Virtualization
• Logical partitioning (LPAR)
• Multiple operating system instances on one mainframe
• Splitting through Processor Resource / system manager (PR/SM) in firmware
• Depending on operating system, repartitioning support at runtime
• Support for micro-partitions on one CPU, capped vs. uncapped mode
47
Dependable Systems Course PT 2012
zEnterprise: Redundant I/O
48
!"
# !$%%&'()&*+,-+,./0+123
'()&451/6,-,076
4%"8&9:;&<670=1&+>&??@@ABCCD&96EB1E.1/&'FG&;BH7D7/6I&J +>&6K07/01=&'G&L.=6&.1E&E,.M6,7
!"##$%&'(")(*)+%,-.%/'012)! ;:N&F&*N&7-.,01=! ;:N&96.770=1I61/! 'FG&9676/&O&A.0C+P6,! 'FG&)BK 9676/&F&A.0C+P6,! 96EB1E.1/&'FG&:E.-/6,! 96EB1E.1/&'FG&01/6,L+116L/! 96EB1E.1/&Q6/M+,R&:E.-/6,7! 96EB1E.1/&';*&C01R7! 96EB1E.1/&*,D-/+&-,+L677+,7! 'FG&;M0/LS6E&A.H,0L! Q6/M+,R&;M0/LS6EF9+B/6,&A.H,0L! T0=S&:P.0C.H0C0/D&NCB==01=&9BC67! 'FG&.1E&L+B-C01=&>.1+B/
,6H.C.1L01=&+1&*(:! *S.116C&'10/0./6E&96/,D! T0=S&<./.&'1/6=,0/D&'1>,.7/,BL/B,6! 'FG&:C/6,1./6&N./S! Q6/M+,R&:C/6,1./6&N./S! U0,/B.C04./0+1&V6LS1+C+=D
-,+L677+,
A.0C+P6,
'FG&)BK
'FG&TBH 'FG&TBH
'FG&)BK 'FG&)BK 'FG&)BK
A.0C+P6,
'FG&&.E.-/6,@
'FG&&.E.-/6,@
Q6/M+,R&&.E.-/6,@
Q6/M+,R&.E.-/6,@
*,D-,+:LL6C6,./+,
*,D-/+&:LL6C6,./+,
';*&&W01R7
';*&W01R7
:C/6,1./6&N./S
'FG&;M0/LS
*+1/,+CX10/
'FG&;M0/LS
*+1/,+CX10/
Q6/M+,R;M0/LS
51E&X76,
Q6/M+,R;M0/LS
51EX76,
# !$%%&'()&*+,-+,./0+12Y
'()&451/6,-,076
N,6P61/01=&:CC&GB/.=67
! 3)045'("#'(%."+*2'0" :EP.1L6E&)6I+,D&9:')&Z96EB1E.1/&:,,.D&+>&'1E6-61E61/&)6I+,D[&E670=1"51S.1L6E&966E\;+C+I+1&L+E6&Z5**[&J "$(F8]("N,+/6L/0+1&.=.017/&*S.116CF<'))&>.0CB,67"*S0-&I.,R01=&>+,&>.7/&<9:)&,6-C.L6I61/7
" )0,,+,6E&^6D&L.LS6" 'I-,+P6E&LS0-&-.LR.=01=" 'I-,+P6E&L+1E617./0+1&I.1.=6I61/" '1/6=,./6E&V*NF'N&LS6LR7BI&=616,./0+1FLS6LR01=" '1/6=,./6E&5NG&7M0/LS&L+P6,&Z-,+/6L/01=&/S6&7M0/LS&EB,01=&,6-.0, .L/0+17[" *+1/01B6E&>+LB7&+1&A0,IM.,6
Dependable Systems Course PT 2012
zEnterprise: Parallel Sysplex
• Multi-system data sharing facility
• No single point of failure, cluster-wide coherency and locking protocols
• Coupling facility (CF) works as shared memory, running on separate machine
• Custom CPU commands to talk to CF
• Node clocks are synchronized
• Single operator interface
• Software / hardware updates supported at runtime
• LPARs can act as participants
49
Dependable Systems Course PT 2012
Example: IBM CICS
• “Although most people are blissfully unaware of CICS, they probably make use of it several times a week, for almost every commercial electronic transaction they make. In the whole scheme of things, CICS is much more important than Microsoft Windows." Martin Campbell-Kelly, From Airline Reservations to Sonic the Hedgehog (A History of the Software Industry, MIT Press 2003
• Intended for business transactions
• Many users with repetitive activitites
• Short interactions on shared data
• Data integrity is crucial, low cost per transaction neccessary
• Examples: Banking, insurance, airlines, stock trading, order processing, payroll, ...
52
Dependable Systems Course PT 2012
CICS Management Services
53
18
© 2011 IBM Corporation
CICS Management Services
Web Service
3270
SystemServices
Recovery
Security
Dispatcher
File
Program
CICSCICS
Storage
CommunicationServices
ExternalResourceManagers
Enterprise server
Server
!"#"$%
%&'"())))))))*+,-./012- 3
45"6& 5'78%"
Workstation
Browser
?!àéµ$Web Application Server
SNA
TCP/IP
HTTP
CORBA
SOAP
3270
Queue
Communication Services handles communications with the outside world. The figure shows CICS evolution over time, starting from the legacy 3270 terminal, right up to themost recent Web Service implementation. In case you are wondering about the “?!ae!$” access type, this signifies that CICS will adapt to any future strategic IT communication protocol.From top to bottom, you can see that CICS supports:• 3270 data stream• SNA, including Advanced Program-to-Program Communication (APPC)• TCP/IP, including socket applications with the help of Communication Server for z/OS• HTTP Internet protocol, meaning that CICS supports native browser access and HTTP client or server applications• CORBA, meaning that CICS supports the Java 2 Enterprise Edition (J2EE) Enterprise Java Bean (EJB) specification, which defines the contract between an EJB and an execution container such as CICS.• SOAP, meaning that CICS supports language-independent Web service requester or provider applications.
Dependable Systems Course PT 2012
CICS Recovery
54
10
© 2011 IBM Corporation
Data Recovery Overview
Backout to rollback in-flight Units of Work
after failure
Restore updates made to a file after file has
been lost
The large number of users and complex interactions involved in e-business make data integrity and recovery an extremely important requirement. CICS provides facilities to protect the integrity of user data, rather than placing this burden on the application programmer.CICS provides for recovery and restart from system and transaction failures. Recovery is the process of restoring resources to their status prior to a failure, while restart is concerned with fast restoration of CICS services following a recovery action. The benefits of data integrity, recovery, and restart are increased system availability and reliability, thus enabling a better service to be provided to customers and users.
Dependable Systems Course PT 2012
CICS Recovery
55
11
© 2011 IBM Corporation
Backout: Logging
! CICS automatically captures a copy of the resource that is about to be updated (e.g. CICS writes record to system log f you read with intent to update it)
! Under z/OS, CICS Transaction Server uses the z/OS system logger service to direct these records to a log stream called DFHLOG
In order to perform backout, CICS automatically captures a copy of the resource that is about to be updated. For example, if you read a record with the intent to update it, CICS writes the record to the system log.• In the event of a task or system abnormal termination, the before image, a copy of the record as it appeared before update, enables CICS to perform backout.• Under z/OS, CICS Transaction Server uses the z/OS system logger service to direct these records to a log stream called DFHLOG. In all other CICS environments, the log is a sequential file.
Dependable Systems Course PT 2012
CICS Backout
56
12
© 2011 IBM Corporation
Transaction Backout/ Rollback
! If abnormal termination " use before image to perform backout! Types of backout:
– Dynamic transaction backout """" one task fails, but CICS continues to run– Emergency restart backout " during emergency restart after CICS system failure
This process of removing the effects of changes made by a failing task is called backout, or rollback. If a transaction fails for any reason, CICS automatically reverses its updates.• If a task has made multiple updates, all are removed, so that the data appears as if the transaction had never run.• In the event of a failure for an in-flight UOW, CICS simply uses the before image from the system log to replace the updated resource, thus restoring it to its prior state.• The process is basically the same, whether the failure is confined to one task or the entire system goes down.CICS supports two types of backout:• Dynamic transaction backout occurs when one task fails, but CICS continues to run.• Emergency restart backout is performed during an emergency restart following a CICS system failure.
Dependable Systems Course PT 2012
CICS Forward Recovery
57
13
© 2011 IBM Corporation
Forward Recovery: Recovering from Disk Failure – The problem
What happens if an updated file is destroyed?
! Users make 1000 changes to a file successfully.! Then the file is destroyed and all changes are lost.
" Even if a backup copy of the file is available, then either the users must reenter their changes, or we must provide some automated form of reapplying their updates.
So far we have considered the process of backing out uncommitted updates. What happens if an updated file is destroyed?• Users make 1000 changes to a file successfully.• Then the file is destroyed and all changes are lost.• Even if a backup copy of the file is available, then either the users must reentertheir changes, or we must provide some automated form of reapplying their updates.
Dependable Systems Course PT 2012
CICS Forward Recovery
58
14
© 2011 IBM Corporation
Forward Recovery: Journaling
! To recover committed updates: write after images to journal file/ log stream" forward recovery
To prepare for recovery of committed updates, which is sometimes called forward recovery, CICS will automatically journal all updates if you request this service in your file definition.This means that an after image — the way a record looks after it is updated, deleted, or added — is recorded to a journal file or log stream.
Dependable Systems Course PT 2012
CICS Forward Recovery
59
15
© 2011 IBM Corporation
Forward Recovery
To bring the file up to date, you need three things:1. A backup copy of the file.2. All changes to the file since the backup was taken.3. A program to perform forward recovery.CICS provides the first two items; you must provide the forward recovery
program.
To bring the file up to date, you need three things:1. A backup copy of the file.2. All changes to the file since the backup was taken.3. A program to perform forward recovery.CICS provides the first two items; you must provide the forward recovery program.One such program, CICS VSAM Recovery (CICS VR), is available from IBM as a licensed program.
Dependable Systems Course PT 2012
HP Superdome
• Up to 128 cores (64 x Itanium 2 Dual Core),up to 2 TB of memory, 192 PCI slots
• Up to 16 hardware partitions
• Hot-swapping, redundant fans / power supply
• HP-UX operating system, support for other operating Systems in partitions (Windows Server, Linux, OpenVMS)
• 1200 kg
• > 99.99% system availability reported - finance, airlines, ...
• Maximizing transaction throughput in SAP and database systems
60
(C) H
P