TMDE AND FLEET SYSTEM RISK
description
Transcript of TMDE AND FLEET SYSTEM RISK
![Page 1: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/1.jpg)
1MSDJ
TMDE ANDFLEET SYSTEM RISK
TMDE ANDFLEET SYSTEM RISK
Dennis JacksonNSWC Corona Division (MS-20)
7 May 2009
![Page 2: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/2.jpg)
2MSDJ
Overview
• System Specifications
• System Testing and Risk
• System Adjustment and Specifications
• System Specifications
• System Testing and Risk
• System Adjustment and Specifications
![Page 3: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/3.jpg)
3MSDJ
Prime System Traceability
• Program / acquisition managers will provide measurement requirements that are in support of prime systems to the TMDE EA via MIL-STD-1839C CMRS data submitted. -- NAVSEAINST 4734.1B
• Recommended TMDE shall be capable of measuring or generating to a higher accuracy than the measurement parameters being supported. Unless otherwise specified, a minimum Test Uncertainty Ratio (TUR) of 4 to 1 is desired. The actual TUR shall be documented. -- MIL-STD-1839C
• Development procedures shall result in MRCs that ensure system or equipment operation is within performance standards and established readiness criteria. -- MIL-P-24534A
• Program / acquisition managers will provide measurement requirements that are in support of prime systems to the TMDE EA via MIL-STD-1839C CMRS data submitted. -- NAVSEAINST 4734.1B
• Recommended TMDE shall be capable of measuring or generating to a higher accuracy than the measurement parameters being supported. Unless otherwise specified, a minimum Test Uncertainty Ratio (TUR) of 4 to 1 is desired. The actual TUR shall be documented. -- MIL-STD-1839C
• Development procedures shall result in MRCs that ensure system or equipment operation is within performance standards and established readiness criteria. -- MIL-P-24534A
![Page 4: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/4.jpg)
4MSDJ
Test Accuracy Ratio (TAR)
• TAR = The ratio of the system specification to the TMDE tolerance
• TAR is a key driver for test decision risk and cost
• TAR = The ratio of the system specification to the TMDE tolerance
• TAR is a key driver for test decision risk and cost
1001.0
1.0TAR (Expressed as 10:1)
Example:
![Page 5: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/5.jpg)
5MSDJ
Test Uncertainty Ratio (TUR)
• TUR = The ratio of the system specification to the TMDE uncertainty
• Since the TMDE tolerance is often assumed to be the uncertainty, TAR and TUR are often used as equivalent.
• The new ANSI/NCSLI Z540.3 standard defines TUR as the ratio of the TMDE tolerance to the 95% Calibration Process uncertainty.
• The Z540.3 provides a less ambiguous definition.
• The Z540.3 definition applies to calibration rather than system testing.
• TUR = The ratio of the system specification to the TMDE uncertainty
• Since the TMDE tolerance is often assumed to be the uncertainty, TAR and TUR are often used as equivalent.
• The new ANSI/NCSLI Z540.3 standard defines TUR as the ratio of the TMDE tolerance to the 95% Calibration Process uncertainty.
• The Z540.3 provides a less ambiguous definition.
• The Z540.3 definition applies to calibration rather than system testing.
![Page 6: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/6.jpg)
6MSDJ
Example System
• Controls the loading and arming of the selected weapon
• Launches the weapon
• Provides terminal guidance for AAW (Anti-Air Warfare) missiles
• Controls the target illumination for the terminal guidance of SM-2
Mk 99 Missile Fire Control System
![Page 7: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/7.jpg)
7MSDJ
![Page 8: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/8.jpg)
8MSDJ
![Page 9: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/9.jpg)
9MSDJ
Specification Closeup
![Page 10: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/10.jpg)
10
MSDJ
Specifications
• An MRC directs testing with a TMDE to ensure a system parameter is within a specification.
• For example:
– Ensure voltage measured is 27.0 V (± 2 V)
– Ensure voltage measured is 28.0 (26.6 to 29.4) VDC
• An MRC directs testing with a TMDE to ensure a system parameter is within a specification.
• For example:
– Ensure voltage measured is 27.0 V (± 2 V)
– Ensure voltage measured is 28.0 (26.6 to 29.4) VDC
![Page 11: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/11.jpg)
11
MSDJ
Specification Choice
• What does ± 2.0 V mean?
• At 29.0 V, is performance degraded?
• At 25.0 V, is performance degraded?
• Specifications should not be determined solely on TMDE accuracy
• What does ± 2.0 V mean?
• At 29.0 V, is performance degraded?
• At 25.0 V, is performance degraded?
• Specifications should not be determined solely on TMDE accuracy
![Page 12: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/12.jpg)
12
MSDJ
Utility Curve
Degraded performance
Loss of Utility
Acceptable performance
![Page 13: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/13.jpg)
13
MSDJ
Testing
• Navy systems are tested using test equipment (TMDE)• Navy systems are tested using test equipment (TMDE)
Test Equipment
Navy SystemMeasures
![Page 14: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/14.jpg)
14
MSDJ
SCAT 4245
![Page 15: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/15.jpg)
15
MSDJ
TMDE Index
How do TMDE tolerances relate to testing systems?
![Page 16: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/16.jpg)
16
MSDJ
TMDE Measurement Error
• All Measurements have some error
• TMDE errors need to be small
• Large TMDE errors cause bad test decisions
• All Measurements have some error
• TMDE errors need to be small
• Large TMDE errors cause bad test decisions
Test Equipment
Navy SystemMeasures
• If, for example the test equipment measurement above was off by 1 volt, the system would fail incorrectly
• This would cause unnecessary maintenance
• If, for example the test equipment measurement above was off by 1 volt, the system would fail incorrectly
• This would cause unnecessary maintenance
![Page 17: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/17.jpg)
17
MSDJ
Utility Curve
Uncertainty for TMDE
Measurements tested inside the specification could actually be in the degraded performance region
Degraded performance
Loss of Utility
Acceptable performance
![Page 18: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/18.jpg)
18
MSDJ
MeasurementRisk Events
• False Accept occurs when a parameter is observed through testing by a TMDE to be acceptable, but is actually outside specifications
• False Reject occurs when a parameter is rejected through testing by a TMDE , but is actually inside specifications
• False Accept occurs when a parameter is observed through testing by a TMDE to be acceptable, but is actually outside specifications
• False Reject occurs when a parameter is rejected through testing by a TMDE , but is actually inside specifications
![Page 19: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/19.jpg)
19
MSDJ
Good Test Decisions
![Page 20: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/20.jpg)
20
MSDJ
Risk Consequences
• False Accepts directly harm the Fleet
– Degradation of mission capability
– Failure of mission
– Injury
– Loss of life
• False Rejects indirectly harm the Fleet
– Increased maintenance cost
– Decreased availability
• False Accepts directly harm the Fleet
– Degradation of mission capability
– Failure of mission
– Injury
– Loss of life
• False Rejects indirectly harm the Fleet
– Increased maintenance cost
– Decreased availability
![Page 21: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/21.jpg)
21
MSDJ
TMDE Measurement Model
TMDESYSTEM
PARAMETER
Measures
TMDE Measurement = Parameter Value + Test Error
• The parameter value is the true output from the system
• The TMDE measurement estimates the parameter value
• The test error is due to inaccuracy in the TMDE as well as in the test setup
![Page 22: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/22.jpg)
22
MSDJ
True In Tolerance
A parameter is truly in tolerance if:
Lower Spec < Parameter Value < Upper Spec
NominalLower Spec (-L) Upper Spec (L)
ParameterValue
![Page 23: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/23.jpg)
23
MSDJ
Observed In Tolerance(Acceptance)
A parameter is observed in tolerance if:
Lower Spec < TMDE Measurement < Upper Spec
0Lower Spec (-L) Upper Spec (L)
TMDEMeasurement
![Page 24: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/24.jpg)
24
MSDJ
False Accepts
TMDE Measurement
Nominal- L L
Parameter Value
TestError
False Accept (FA):
• The TMDE Meas is observed in tolerance [ -L < Meas < L ]
• The Par Value is out of tolerance [ Value > L or Value < -L ]
• The decision to accept the Parameter is incorrect
![Page 25: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/25.jpg)
25
MSDJ
Probability of False Accept
Probability of False Accept (PFA):
PFA = Pr( [Observed In Tolerance] and [True Out Of Tolerance] )
= Pr( [-L < Meas < L] and [Value > L or Value < -L] )PFA is the probability of making an incorrect acceptance decision
TMDE Measurement
Nominal- L L
Parameter Value
TestError
![Page 26: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/26.jpg)
26
MSDJ
True Out Of Tolerance
A parameter is truly out of tolerance if:
Parameter Value < Lower Spec or Parameter Value > Upper Spec
NominalLower Spec (-L) Upper Spec (L)
Parameter Value
![Page 27: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/27.jpg)
27
MSDJ
Observed Out Of Tolerance(Rejection)
A parameter is observed out of tolerance if:
TMDE Measurement < Lower Spec or TMDE Measurement > Upper Spec
NominalLower Spec (-L) Upper Spec (L)
TMDE Measurement
![Page 28: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/28.jpg)
28
MSDJ
False Rejects
TMDE Measurement
Nominal- L L
Parameter Value Test
Error
False Reject (FR):
• The Par Value is in tolerance [ -L < Value < L ]• The TMDE Meas is observed out of tolerance
[ Meas > L or Meas < -L ]• The decision to reject the Parameter is incorrect
![Page 29: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/29.jpg)
29
MSDJ
Probability of False Reject
Probability of False Reject (PFR):
PFR = Pr( [Observed Out Of Tolerance] and [True In Tolerance] )
= Pr( [Meas > L or Meas < -L] and [-L < Value < L] )PFR is the probability of making an incorrect reject decision
TMDE Measurement
Nominal- L L
Parameter Value Test
Error
![Page 30: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/30.jpg)
30
MSDJ
Inputs Needed to Calculate Risk Probability
• The Specification Limits (-L, L)• The MRC Card
• The Measurement Uncertainty for the Test Process• Generally, this can be considered the TMDE
uncertainty• Should also include test set-up uncertainty
(cables, etc.)
• The Observed Test Parameter Reliability• 3M Data
![Page 31: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/31.jpg)
31
MSDJ
Risk Tool
![Page 32: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/32.jpg)
32
MSDJ
Risk Examples
![Page 33: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/33.jpg)
33
MSDJ
Not Calibrating Costs Money
• Inaccurate test equipment cause bad test decisions
• Wrong test decisions mean unnecessary maintenance
• Maintenance cost is driven by TMDE uncertainty, system spec’s, cal periodicity, and maintenance periodicity
• Inaccurate test equipment cause bad test decisions
• Wrong test decisions mean unnecessary maintenance
• Maintenance cost is driven by TMDE uncertainty, system spec’s, cal periodicity, and maintenance periodicity
![Page 34: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/34.jpg)
34
MSDJ
The Key Is to Minimize Cost
• More calibration increases calibration budget
• Less calibration increases maintenance budget, but allows extended deployment and fewer personnel
• Minimize the total budget (calibration + maintenance)
• More calibration increases calibration budget
• Less calibration increases maintenance budget, but allows extended deployment and fewer personnel
• Minimize the total budget (calibration + maintenance)
![Page 35: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/35.jpg)
35
MSDJ
Testing vs Adjusting
• For some parameters, the TMDE is used to adjust the parameter value rather than testing it
• Often, no specification is given for this situation
• With no specification, there is no basis for choosing the TMDE since TAR is unknown
• The following shows an example.
• For some parameters, the TMDE is used to adjust the parameter value rather than testing it
• Often, no specification is given for this situation
• With no specification, there is no basis for choosing the TMDE since TAR is unknown
• The following shows an example.
![Page 36: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/36.jpg)
36
MSDJ
![Page 37: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/37.jpg)
37
MSDJ
![Page 38: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/38.jpg)
38
MSDJ
Utility Curve
Uncertainty for TMDE
TMDE adjustment accuracy should be 4 times better than the system requirement
Degraded performance
Loss of Utility
Acceptable performance
![Page 39: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/39.jpg)
39
MSDJ
Adjustment and TUR
• Assume TUR is the ratio of the 95% TMDE uncertainty to the system specification
• With a 4:1 TUR, an adjusted parameter has very close to a 0% chance of being out of tolerance after adjustment.
– Assuming a stable system parameter (good repeatability)
• With a 1:1 TUR, an adjusted parameter has about a 5% chance of being out of tolerance after adjustment.
– A 1:1 TUR occurs when the system specification is set to the TMDE tolerance
• Assume TUR is the ratio of the 95% TMDE uncertainty to the system specification
• With a 4:1 TUR, an adjusted parameter has very close to a 0% chance of being out of tolerance after adjustment.
– Assuming a stable system parameter (good repeatability)
• With a 1:1 TUR, an adjusted parameter has about a 5% chance of being out of tolerance after adjustment.
– A 1:1 TUR occurs when the system specification is set to the TMDE tolerance
![Page 40: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/40.jpg)
40
MSDJ
Conclusions
• Specifications are needed for every measurement event requiring a TMDE
• Specifications should be directly related to system performance
• Specifications should answer the question: “Does system performance degrade when the parameter is outside the specification?”
• Specifications are needed for every measurement event requiring a TMDE
• Specifications should be directly related to system performance
• Specifications should answer the question: “Does system performance degrade when the parameter is outside the specification?”
![Page 41: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/41.jpg)
41
MSDJ
Conclusions
• TMDE tests are used to make decisions about Fleet systems.
• Wrong decisions cause consequences to the Fleet:
– Degradation of mission capability
– Loss of mission
– Injury
– Loss of life
– Unnecessary maintenance cost
– System unavailability
• The probability of wrong decisions can be calculated using risk methods
• TMDE tests are used to make decisions about Fleet systems.
• Wrong decisions cause consequences to the Fleet:
– Degradation of mission capability
– Loss of mission
– Injury
– Loss of life
– Unnecessary maintenance cost
– System unavailability
• The probability of wrong decisions can be calculated using risk methods
![Page 42: TMDE AND FLEET SYSTEM RISK](https://reader035.fdocuments.net/reader035/viewer/2022062309/568158ce550346895dc61a04/html5/thumbnails/42.jpg)
42
MSDJ
Conclusions
• To assess the probability of wrong decisions (PFA and PFR), you need:
– System specifications
– TMDE tolerances/uncertainty
– System reliabilities
• If the system specifications are related to system performance, the risk measures directly relate to system reliability
– Allows assessment of impact of TMDE testing on system performance
– Allows assessment of impact of calibration of TMDE on system performance
• To assess the probability of wrong decisions (PFA and PFR), you need:
– System specifications
– TMDE tolerances/uncertainty
– System reliabilities
• If the system specifications are related to system performance, the risk measures directly relate to system reliability
– Allows assessment of impact of TMDE testing on system performance
– Allows assessment of impact of calibration of TMDE on system performance