FT4 Data Analysis Summary (SSI-ARC) · FT4 Data Analysis Summary (SSI-ARC) Doug Isaacson, Chester...
Transcript of FT4 Data Analysis Summary (SSI-ARC) · FT4 Data Analysis Summary (SSI-ARC) Doug Isaacson, Chester...
1
FT4DataAnalysisSummary(SSI-ARC)DougIsaacson,ChesterGong,ScottReardon,ConfesorSantiago12DEC2016
AbstractStandardsforUnmannedAircraftSystem(UAS)Detect-and-Avoid(DAA)systemsarecurrentlybeingdevelopedundertheauspicesoftheRTCASpecialCommittee228(SC-228).Tosupportthedevelopmentofthesestandards,aseriesofflighttestshasbeenconductedatNASA’sArmstrongFlightResearchCenter(NASA-AFRC).Thefourthinthisseriesofflighttestactivities(FlightTest4,orsimplyFT4)wasconductedduringtheSpringandSummerof2016.FT4supportedtheobjectivesofnumerousorganizationsworkingtowardUASDAAMinimumOperationalPerformanceStandards(MOPS)andUASDAARadarMOPS.Thesummaryprovidedhereinislimitedtotheobjectives,analysisandconclusionsoftheNASAAmesResearchCenter(NASA-ARC)SSIteamtowardtherefinementofUASDAAMOPS.Thisdocumentprovidesahigh-leveloverviewofFT4andtheSSI-ARCobjectives,asummaryofthedataanalysismethodologyandrecommendationsforUASDAAMOPSrefinementsbasedonthedataanalysisresults.Atotalof72encounterswereflowntosupportSSI-ARCobjectives.TestresultsweregenerallyconsistentwithacceptableUASDAAsystemperformanceandwillbeconsideredinbroaderSC-228requirementsvalidationefforts.ObservedalertleadtimesindicatedacceptableUASDAAalertingperformance.EffectiveinteroperabilitybetweentheUASDAAsystemandtheTrafficAlertandCollisionAvoidanceSystem(TCAS)wasobservedwithonenotableexception:TCASResolutionsAdvisories(RA)wereobservedintheabsenceofanyDAAalertontwooccasions,indicatingtheneedforalertparameterrefinement.FindingsfurtherindicatedtheneedforcontinuedworkintheareasofDAAWellClearRecoverylogicandalertstabilityforMode-C-onlyintruders.Finally,resultsdemonstratedahighlevelofcompliancewithasetofevaluationcriteriadesignedtoprovideanecdotalevidenceofacceptableUASDAAsystemperformance.
SystemUnderTest:JADEMFT4wasconductedatNASA-AFRCbetweenAprilandJuneof2016.NASA’sIkhanaresearchaircraftandGroundControlStation(GCS)wereequippedwiththenecessaryhardware,displaysandsoftwaretoevaluatethreeprototypeUASDAAsystems:NASADaedalus,NASAJavaArchitectureforDetectandAvoidExtensibilityandModeling(JADEM)andGeneralAtomicsAeronauticalSystems,Inc.ConflictPredictionandDisplaySystem(CPDS).WhileeachsystemwasdevelopedtobeasconsistentwiththedevelopingSC-228MOPSaspractical,therewerekeydifferencesintheirimplementationthatprecludeaggregatingFT4resultsacrossthesystemsundertest(SUTs).ThisdocumentprovidestheanalysismethodologyandresultsforflighttestingoftheNASASSI-ARC-developedJADEMsystem.
AlertingCriteriaTheFT4JADEMimplementationemployedthesamealertingcriteriaanddisplaysymbologyutilizedintheNASAPartTask6(PT6)study,asshowninTable1.Alertinghysteresis,asdefinedinthedraftDAAMOPS,wasimplementedforFT4(minimum4-secondalertpersistenceunlesssupersededbyhigherpriorityalert).
2
OmnibandsGuidance
TheJADEMsystemprovidesguidancetotheUASpilotintheformofheading/altitudebands,thecolorofwhichindicatesthelevelofalertthatwouldresultiftheownshipweretoinitiateaturn/altitude-changetotheheading/altitudedepicted.Greenbandingindicatesaturn/altitudechangetothedepictedheading/altitudeisnotpredictedtoresultinaLossofWellClear(LoWC).Yellowbandingindicatesaturn/altitude-changeispredictedtocauseaDAACorrectiveAlert(i.e.potentialLoWCinthenext25-55s).Lastly,redbandingindicatesaturn/altitude-changeispredictedtocauseaDAAWarningAlert(i.e.potentialLoWCinthenext25sorless).Figure1depictssampleOmnibandsguidanceaswouldbedisplayedintheUASGCS;a“Corrective”DAAAlertisdisplayedfortheonlydisplayedintruder.TheguidanceillustratedinFigure1indicatesviablemaneuverstoremainwellclearareavailableinboththeheading(~15degreesleftorright)andaltitude(+600ftor-400ft)bands.
Table1:JADEMFT4AlertingCriteria
3
Figure1:SampleJADEMOmnibandsGuidance
Omnibandsguidanceformultipleintrudersisproducedbyformingtheunionofindividualomnibandsguidanceforeachintruderwithinthefieldofregard,withhigherpriorityalertsoverridinglowerprioritieswhenalertsdifferbetweenintrudersforagivenheading/altitudelevel.Additionally,thefollowingassumptionsandparameterswereusedintheJADEMguidanceandalertinglogicforFT4:
1. Ownshipassumedtoexecuteturnsataconstantrateof3deg/s2. Ownshipassumedtoclimbordescendatconstantrateof1000ft/min3. Omnibandsheadingguidancelimitedwithin+/-100degreesofcurrentownshipheading4. Omnibandsheadingguidanceprovidedwith1degreeresolution/discretization5. Omnibandsaltitudeguidancelimitedwithin+/-3000feetofcurrentownshipaltitude6. Omnibandsaltitudeguidanceprovidedwith100ftaltitudediscretization
WellClearRecoveryGuidance
WellClearRecovery(WCR)guidancereferstoguidancethatisprovidedtotheUASpilottoregainWellClearfromallintruders.JADEMwasconfiguredforFT4toprovideWCRguidancewhennoallowableownshipmaneuverispredictedtoremainwellclearofallintruders.ItshouldbenotedthatthisimplementationexceedstheminimumMOPSrequirements(whichonlyrequiresWCRguidancewhenaLoWChasalreadyoccurred).Figure2illustratesWCRguidanceasimplementedinJADEMforFT4;theredhorizontalbandingindicatesnoviablemaneuversarepredictedtoremainWellClear,andthegreenWCR“wedge”indicatestheadviseddirectionandroughmagnitudeofhorizontalmaneuverthatwillresultinregainingWellClearinasafeandtimelymanner.Determinationofa“safeandtimely”maneuverisdictatedbyevaluationofaWCRcostfunction.
HeadingBands
AltitudeBands
4
TheJADEMWCRcostfunctionforFT4considersanumberoffactors,butismostinfluencedbytheproximityofthepredictedClosestPointofApproach(CPA)totheacceptedNMACvolumeandsecondarilybythedesiretomaintainconsistentWCRguidance(changesinturndirectionincuracostpenalty).Finally,verticalJADEMWCRguidancewasnotprovidedforFT4topreventtheUASpilotfrommaneuveringverticallynearthecollisionavoidanceboundaryagainstcooperativeintruders(whichmaydegradeTrafficAlertandCollisionAvoidanceSystem(TCAS)IIperformance)andduetoverticalstateestimationuncertaintiesforintruderstrackedsolelybytheradar(i.e.non-cooperativeintruders).
Figure2:SampleWCRGuidance
TrafficAlertandCollisionAvoidanceSystem(TCAS)InteroperabilityImplementation
OneoftheprimaryobjectivesofFT4wastoevaluatetheUASDAA-TCASInteroperabilityconceptdevelopedsinceFT3.ThefollowingfourheuristicssummarizetheDAATCASInteroperabilityConceptasimplementedinJADEMforFT4:
1. AnyintruderwithanactivecorrectiveResolutionAdvisory(RA)isremovedfromallDAAguidancecalculations
a. HorizontalDAAguidancewillbeshownfornon-RAaircraftb. AllDAAverticalguidanceissuppressedduringacorrectiveRA
2. DuringapreventiveRA,TCASguidanceisaninputtotheDAAverticalguidancetoensureanyverticalDAAguidanceisconsistent(e.g.DONOTCLIMB)withtheRAguidance
3. Anytimeownship’scompliancewithacorrectiveRAleadstoasecondaryDAAWarningalert(maneuvernow),DAAguidanceshallreverttowellclearrecoveryinordertobemoredirectwithguidance,e.g.:
a. CompliancewithTCAS‘DESCEND’RAleadstoasecondaryDAAWarningb. RatherthanshowthepilotfullOmnibandssuggestiveguidance,limitedsuggestive
guidanceisdisplayed(e.g.maneuverleft)
HighBound
LowBound
5
FT4ObjectivesandMethodologyforJADEMAnalysisEachoftheFT4systemsundertest(SUTs)wasevaluatedindependently,andeachaccordingtoobjectivesandanalysismethodologiestailoredtotheirspecificimplementationofthedraftSC-228UASDAAMOPSrequirements.ThissectiondescribestheFT4objectives,scenariodevelopmentandanalysismethodologydevelopedandemployedfortheJADEMsystemundertest.
SSI-ARCFT4ObjectivesandScenarioGeneration
Asetofhigh-leveltestobjectivesweredevelopedtosupportthedevelopmentoftheSC-228UASDAAMOPS.Thefollowingfourhigh-leveltestobjectiveswereusedtoguidetheFT4planning,conductandanalysisofJADEMalertingandguidancelogic:
1. ValidateDAArequirementsinstressingcasesthatdriveMOPSrequirements,including:High-speedcooperativeintruder,Low-speednon-cooperativeintruder,highverticalclosurerateencounter,andModeC/S-onlyintruder(i.e.withoutADS-B).
2. ValidateTCAS/DAAalertingandguidanceinteroperabilityconceptinthepresenceofrealisticsensor,trackingandnavigationalerrorsandinmultiple-intruderencountersagainstbothcooperativeandnon-cooperativeintruders.
3. Validate“WellClearRecovery”guidancerequirementsinthepresenceofrealisticsensor,trackingandnavigationalerrors.
4. ValidateDAAalertingandguidancerequirementsinthepresenceofrealisticsensor,trackingandnavigationalerrors.
Insupportoftheseobjectives,aseriesofencounterswerescriptedtoaddresseachobjective.Thedevelopmentofeachscenariorequiredidentificationof:theprimaryhigh-levelobjectivebeingaddressed,theencountergeometry,intruderequipage,datacollectionrequirements,encountermethodology(includingpilotinstructions),checkstoensuredataquality,draftMOPSreference,andevaluationcriteriatodeterminewhetherornotthetestobjectivesweremet.AsamplescenariotemplateresultingfromthisprocessisincludedasFigure3.TheSSI-ARCteamsubsequentlycoordinatedwithAFRCflighttestpersonneltorefinethescenariosoutlinedinthetemplatesforproductionofflight“testcards”providingthelevelofdetailnecessarytotrainownshipandintruderpilotsandexecutethescriptedscenarioonthedayoftheflight.Intotal,72testcardsweredevelopedandflownforFT4JADEManalysis;asampletestcardisincludedasFigure4.
6
Figure3:Sam
pleFT4ScenarioTemplate
7
Figure4:Sam
pleFT4TestCard
8
AnalysisMethodology
Theanalysismethodologywasdesignedtoaddressthe“evaluationcriteria”identifiedforeachscenariotemplate.Assuch,thefollowingkeyeventdatawereextractedfromtherecordedflighttestdataforeachtestcard:
• FirstTrack:Firstrecordedtrackforeachintruderintheencounter• DAAPreventiveAlert:Onset(s)ofDAAPreventivealert(s)(foreachnewoccurrence)• DAACorrectiveAlert:Onset(s)ofDAACorrectivealerts(s)(foreachnewoccurrence)• DAAWarningAlert:Onset(s)ofDAAWarningalerts(s)(foreachnewoccurrence)• LoWC:OnsetofeachnewLossofWellClear(maybemultipleLoWCsand/orLoWCfor
multipleintruders)• RegainWellClear:firsttimeatwhichWellClearisregainedafteraLoWCevent.• ClosestPointofApproach:time(s)atwhichminimumslantrangeisachievedduringthe
encounterforeachintruder• TCASEvent:TimeofinitialTCASPreventiveorCorrectiveAlertandanyassociated
guidance(orTCASallclearmessage)• WellClearRecovery:OnsetofWCRguidanceandassociateddirection(e.g.turnright)as
wellasanychangestodirectionofguidance• FinalAlert:Thelastrecordedalertforeachintruderintheencounter• LastTrack:Lastrecordedtrackforeachintruderintheencounter
Theseeventswereloggedtoadatabasethatwasusedintheanalysistodetermineifeventsoccurredinthepropersequence(e.g.TCASCorrectiveRAswereprecededbyDAAcorrectiveorwarningalerts)andwithappropriatetiming,asdictatedbyeachencounter’sevaluationcriteria.Table2includestheextractedeventsfromasingleFT4JADEMencounter.
Table2:SampleEncounterEventTableCycleCount Intruder Time ClockTime Event
191 N3GC 1461942437.39 15:07:17 FIRST_TRACK
198 N3GC 1461942445.37 15:07:25 DAA_PREVENTATIVE
203 N3GC 1461942450.38 15:07:30 DAA_CORRECTIVE
208 N3GC 1461942455.41 15:07:35 DAA_PREVENTATIVE
215 N3GC 1461942462.35 15:07:42 DAA_CORRECTIVE
233 N3GC 1461942480.40 15:08:00 DAA_WARNING
257 n/a 1461942504.39 15:08:24 WCR_TURN_RIGHT
258 N3GC 1461942505.38 15:08:25 LOWC
270 N3GC 1461942518.39 15:08:38 TCAS_CLIMB
291 N3GC 1461942539.39 15:08:59 CPA
291 N3GC 1461942539.39 15:08:59 TCAS_LEVEL_OFF
296 N3GC 1461942544.36 15:09:04 TCAS_AA_CLEAR
297 N3GC 1461942545.39 15:09:05 DAA_PREVENTATIVE
297 N3GC 1461942545.39 15:09:05 REGAIN_WC
300 N3GC 1461942548.40 15:09:08 FINAL_ALERT
308 N3GC 1461942556.36 15:09:16
LAST_TRACK
9
Alsosupportingtheassessmentofevaluationcriteriawereaircraftstatedata,alertingthreatparameterdata,intruderrelativestatedata,andomnibandsguidancedata.ExamplesofdataplotsforeachoftheseareshowninFigures5-8.
Figure5:SampleAircraftStateDataPlots
Figure6:SampleAlertingThreatParameterDataPlots
10
Figure7:SampleIntruderRelativeStateDataPlots
Figure8:SampleOmnibandsDataPlots
11
Ineachoftheaircraftstateplots(Figure5),relativestateplots(Figure6)andalertingthreatdataplots(Figure7),thecolorofthefilled-circlemarkerstransposedonthedataplottraceindicatesthetypeofDAAalert:DAApreventive(black),DAAcorrective(orange),andDAAwarning(red)alerts.Additionally,thebackgroundcolorcodingontheOmnibandsdataplots(refertoFigure8)isconsistentwiththeOmnibandscoloringpredictingnoLoWC(green),LoWCwithin25-55s(yellow)andLoWCwithinthenext25s(red)attheindicatedheading/altitude.Lastly,theOmnibandsdataplotsalsoindicatewhenWCRguidance(yellowtrianglemarkers)and/orTCASRAs(magentatrianglemarkers)arepresent.OmnibandsdataplotswereusedtoassessDAA-TCASinteroperability.
Finally,videorecordingsoftheUASDAAJADEMdisplaywerereviewedtoprovideanyadditionaldetailsnecessarytoassesstheperformanceoftheDAAsystemagainstthescenarioevaluationcriteria.
ResultsScenarioEvaluationCriteriaCompliance
ThefirstelementoftheanalysisfocusedonevaluationoftheBooleanevaluationcriteriaforeachscenario.Eightevaluationcriteriawereassessedaspartofthisanalysis(FT4JADEMcompliancewitheachcriterionisprovidedinparenthesis)
1. ADAACorrectiveAlertisissuedtotheUASPilot(94%)2. OwnshipremainsWellClearofintruders(90%)3. DAAAlertsareremovedonceclearofthreat(100%)4. DAAAlertsandGuidanceareprovidedtotheUASPilotfollowingtheexpirationofaTCAS
RA,ifappropriateaccordingtotheDAAalertingthresholds(100%)5. DAAAlertsandGuidanceareremovedforintruderswithCorrectiveTCASRAGuidance
(90%)6. WellClearRecoveryGuidanceisprovidedwhennoviablemaneuversarepredictedto
remainwellclearofallintruders(100%)7. UASPilotmaneuversinresponsetoWCRguidanceandregainsWellClear(100%)8. UASPilotmaneuversinresponsetoandconsistentwithcorrectiveTCASRAguidance
(100%)
Dataanalysisdemonstratedagenerallyhighlevelofcompliancewiththeevaluationtestcriteria,withafewnotableinstancesofnon-compliance.Thesenon-complianceswereevaluatedonacase-by-casebasisasfollows:
FailuretoProvideCorrectiveAlert:AllcasesinwhichaDAACorrectivealertwasnotissuedpriortohigherpriorityalertsoccurredduetotheDAACorrectiveandWarningAlertshavingthesameHMDthreshold.ScenariosdevelopedtoevaluateTCASinteroperabilitywerescriptedwithintrudersmaneuveringtointerceptthecourseoftheUASownshipwithinclosetemporalproximitytothepredictedCPA.TheresultofthisisthatthepredictedmodifiedtauoftheseencounterswasgenerallybelowtheDAAWarningAlertthresholdpriortotheinterceptmaneuver,whilethepredictedHMDdidnotmeettheHMDcriterionforDAACorrective.AstheintruderturnedtointercepttheUASownship,thepredictedHMDrapidlydecreasedinvalueuntilitmettheDAAWarningalertHMDthresholdandaDAAWarningAlertwasissuedwithoutapriorcorrectivealertduetothe
12
lowmodifiedtauvalueatthetimeofthemaneuver.WhileincreasingtheDAACorrectiveAlertHMDvalueinthesecaseswouldleadtothepreferredalertsequencing(DAACorrectiveAlertprecedingDAAWarningAlert),itisunclearifsuchachangeisnecessary.IncreasingtheDAACorrectiveAlertHMDvalueshouldbeconsideredinthecontextofthetradeoffbetweenincreasedDAACorrectiveAlertrateandtherateofoccurrenceofencountersinactualoperationsthatwouldotherwiseleadtotheundesirablealertsequence.Thesenon-compliancesarebelievedtobeanartifactofacarefullyplannedtestmaneuverandareexpectedtobeuncommoninactualoperations;ananalysisofencountersanticipatedinactualoperationsisnecessarytocorroboratethisexpectation.
FailuretoRemainWellClear:TheevaluationcriteriatoremainwellclearonlyappliedtomitigatedencountersinwhichpilotswereinstructedtofollowDAAguidance.Forsuchencounters,UASpilotswereinstructedto:1)postponemaneuversuntilaDAAWarningAlerthadbeenissued,2)maneuvertotheedgeofthegreenguidancebandinthedirectionoftheirchoice(ifmorethanoneoptionwasprovided),and3)toonlymaneuveronce(nocorrectionstoinitialmaneuverallowed).TheseinstructionswereprovidedwiththeintentionofisolatingJADEMperformancefromthepilot’sabilitytoaccommodateshortcomingsinDAAsystemperformanceinremainingWellClear(e.g.byaddinghis/herownexcessmaneuverbuffersormakingadjustmentstotheinitialmaneuvertocounterpoorJADEMtrajectorypredictionsorguidanceinstabilities).SuchpilotresponsetoaDAACorrectiveAlertinoperationaluseofJADEMwouldbeconsistentwithapilotassessingthepredictedLoWC,anddelayingmaneuveringpendingfurtherdevelopmentoftheencounteroruntilahigherprioritytaskiscompleted.WhiledelayingUASmaneuveringtoavoidLoWCuntilaDAAWarningAlertisreceivedshouldgenerallynotleadtoLoWC,suchdelayswouldresultinhigherrateofLoWC(acceptableratesofLoWCinthiscasearebeyondthescopeofFT4).Thepilotinstructiontoonlymaneuveronceforagivenencounterremovesthepilot’sabilitytoreacttotheuncertaintiesofdevelopingencountersandartificiallyincreasestherateofLoWC;refinementstoUASmaneuverstoremainWellCleararetobeexpectedinactualuseofaDAAsystem.GiventhepilotinstructionsintheuseofJADEMinFT4,itissomewhatsurprisingthatLoWCsonlyoccurredin2ofthe20mitigatedencounters.Bothinstancesweretheresultofmodestintruderaccelerations:increasedgroundspeedinonecaseandaslowlyarcingturntowardtheUASinthesecond.ItislikelythatbothLoWCswouldhavebeenavoidedwithearlierownshipmaneuvering(inresponsetotheinitialDAACorrectivealerts)andtheallowanceofmaneuveradjustmentstoaccountformaneuveringintruders.
FailuretoRemoveDAAGuidanceforIntruderswithCorrectiveRAGuidance:TheTCASinteroperabilityconceptimplementedforFT4requiredthatintrudersinvolvedinaTCAScorrectiveRAberemovedfromconsiderationforDAAAlertingandGuidanceconsideration.WhilethiswasimplementedintheJADEMsoftware,3instancesofconcurrentJADEMDAAandTCASRAguidancewereobservedduringFT4.Eachoftheseinstancesweredeterminedtobetheresultofasoftwareissueunrelatedtotheinteroperabilityconceptandisconsideredatestartifactnotindicativeofsystemperformance.
DAAAlertTiming
DAAalerttimingwasanalyzedtoassesstheefficacyofJADEMDAAalertsinarealisticenvironment,includingstressingencountergeometriesandsurveillancesensorerror.TheDAAalerttiminganalysiswaslimitedtounmitigatedencounterstoremovetheinfluenceofpilotactionsontheresultsandbecausesomeofthemetricsarereferencedtoinitialLossofWellClear.
13
DAAAlertLeadTime:Figures9and10providehistogramsofDAACorrectiveAlertleadtimeandDAAWarningAlertleadtime,respectively.Leadtimeisdefinedastheelapsedtimebetweentheinitialalertofagivenpriority(e.g.firstDAAWarningalertissuedtotheUASpilotduringanencounter)andtheinitialLoWC.TheobservedmeanDAACorrectivealertleadtimewas46sandthemeanDAAWarningalertleadtimewas23s.TheleadtimeforDAACorrectivealertsintheabsenceofaDAACorrectivealertwasassignedthesameleadtimeastheDAAWarningalert;thatis,thefirsttimeatwhichallthealertingthresholdsfortheDAACorrectivealertaremet.Whentheseencountersareexcludedfromtheanalysis,themeanDAACorrectivealertleadtimeishigher:49s.
LeadTime,tLEAD(s)
Figure9:DAACorrectiveAlertLeadTimeHistogram
LeadTime,tLEAD(s)
Figure10:DAAWarningAlertLeadTimeHistogram
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80
Frequency
Frequency
14
DAAAlertTransitionTime:Alerttransitiontimeisdefinedastheelapsedtimebetweenagivenalertprioritylevelandthenexthighestalertprioritylevel(e.g.transitiontimebetweeninitialDAACorrectivealertandinitialDAAWarningalert).Table3providesthemeantransitiontimesbetweenalertprioritylevels;theresultsareconsistentwithexpectationsgiventhemodifiedtauvaluesforeachalertlevel,asspecifiedinTable1.NofirmconclusionscanbedrawnfromthemeantransitiontimesinTable3duetothelimitedsamplesizeandoperationalcoverageoftheFT4encounterset.
Table3:DAAAlertTransitionTime
AlertTransition MeanTransitionTime(s)
DAAPreventivetoDAACorrective 3.2s
DAACorrectivetoDAAWarning 24.9s
DAAWarningtoWellClearRecovery 19.3s
WellClearRecoverytoTCAS 9.7s
DAATCASInteroperability
Asdiscussedintheprecedingevaluationcriteriacompliancesection,compliancewiththerequiredelementsoftheDAA-TCASinteroperabilityconceptswaslargelymet,withfewexceptionsbeingtheresultofJADEMsoftwareimplementationissues.Ofnote,twoscenariosresultedincorrectiveTCASRAswhileJADEMdeemedtheownshipWellClearoftheintruderwithgreenbandingguidanceatthetimeofthecorrectiveTCASRA.FurtheranalysisbySC-228personnelandTCAS-IIexpertsconcludedthatsuch“WellClearRAs”observedinFT4aretheresultofincompatibilitybetweenDAACorrectiveAlertHMDthreshold(0.75nm)andhowHMDisused(orinthiscase,NOTused)tofilterTCASRAs.Inthesetwocases,itwasconcludedthatTCASdidnotconsiderHMDinissuingacorrectiveRAduetothecloseproximityoftheintruderandtheexpectederrorintheHMDprediction.BasedonthisfindingandconsistentobservationswithallsystemsundertestinFT4,itisrecommendedthattheMOPSAlertingrequirementsberefinedtoconsiderTCASemploymentoftheHMDfilterandtomitigateitsimpactonalertingperformance.
ModeCIntruderGuidanceStability
AnalysisofJADEMguidancestabilityforintruderslackingADS-Bequipmentwasconsideredasecondarypriorityofthedataanalysis.WhenModeCsurveillanceistheonlyavailablebearingsourcefortheDAAsystem,ahighdegreeofbearinguncertaintyistobeexpected.Unfortunately,onlythreeFT4JADEMencountersincludedintruderswithtranspondersbutwithoutADS-Bequipage.Further,theJADEMguidancefortheseencounterswasbasedonafusiontrackthatincludedradarsurveillance(i.e.,haslowbearingerrorwithintheradardetectionrange).Thus,whiletheobservedguidancedemonstratedexcellentstabilityforthesethreeencounters,moreanalysisisneededtodrawanyconclusionsaboutthestabilityofJADEMDAAguidanceforModeCIntruders.However,itisunclearifthisshouldbeconsideredapriorityforfurtherinvestigationgiventheexpectationthatintruderswillgenerallybewithinradarrangepriortotheprescribedalertingthresholds(i.e.bearingerrorisnotexpectedtoinfluenceDAAalerting).
15
WCRPerformance
WhileWCRperformancemettheobjectivecriteriaprescribedinthescenarioevaluationcriteria,FT4UASpilotparticipantssubjectivelyassessedWCRperformanceaslackinginanumberofareas.First,WCRguidancestabilitywasdeemedpoorforintruderslackingADS-Bequipage;suchencountersdemonstratedmultiplechangesintheWCRguidancedirectionthatpilotsfounddistractingorlackinginformativevalue.Second,UASpilotsobjectedtoWCRguidancethatprescribedturnstoheadingsaftofthecurrentUAScourse;suchturnswouldrequiresignificantdeviationfromplannedrouteofflight,wouldresultinsignificanttimeincloseproximityforhead-onencountergeometries,andoftenplacenon-cooperativeintrudersoutsidethesurveillancevolume.
ConcludingRemarksandRecommendationsFT4providedauniqueopportunitytoevaluatetheperformanceofaprototypeUASDAAsystem(NASAJADEM)acrossarangeofencounterconditionsandinarealisticflightenvironmentincludingsurveillancesystemerrors.However,aswithallflighttests,FT4wasatightlycontrolledexperimentwithpriorpilotknowledgeofencountergeometryandintruder“escape”procedures.Assuch,whiletheperformanceoftheUASDAAsystememployedinFT4isevaluatedforthepurposeoffurtheringMOPSdevelopment,theresultsmustbeconsideredinthecontextofFT4andincombinationwithresultsfromotherUASDAAexperiments(e.g.NASAPT5andNASAPT6,wheretheUASpilotshadnoadvanceknowledgeofencountersandintruderactions).However,whereashuman-in-the-loopsimulationsusemodelsofaircraftdynamics,surveillancesensorsandatmosphericphenomena,flighttestoffersaglimpseattheperformanceoftheprototypeUASDAAsystemunderreal-worldconditions;itisthroughthislensthattheperformanceoftheprototypeUASDAAsystemisassessed.DAAsystemperformancewasassessedacrossfivecategoriesofmetrics:AlertTiming,UASDAA-TCASInteroperability,ModeCIntruderAlertStability,WellClearRecoverPerformanceandabroadsetofScenarioEvaluationCriteria.
ObservedalertleadtimesindicatedacceptableUASDAAalertingperformance;meanDAACorrectiveandDAAWarningalertleadtimeswere46and23secondsrespectively.Excludingencountersthatincludedturnstowardtheintruder(withtheexplicitpurposeofcreatingimmediateTCASRAs),theDAACorrectiveAlertleadtimeincreasedto49s.Thesemeanleadtimesareconsistentwith(albeitslightlybelow)theMOPSaveragealertleadtimes(Section2.2.4.3.4.),andareindicativeofacceptablealertingthresholdsimplementedwithinJADEMforFT4.
FT4included29JADEMscenariostoinvestigatetheUASDAA-TCASinteroperabilityconcept.Resultsindicateeffectiveinteroperabilityacrosstherangeoftestconditionswithonenotableexceptioncase:the“WellClearRA”.Observedforasmallnumberofscenarios(andacrossallsystemsundertest),UASpilotswerepresentedwithaTCASRAwhiletheUASDAAsystemhaddeterminedtheintrudertobeWellClear.ItwasdeterminedthatsuchRAswereduetosuppressionoftheHorizontalMissDistance(HMD)criteriontestwhenTCASdeemedtheintruderbearinginformationtobeofinsufficientquality.BecauseTCASdoesnothaveaqualitybearingsurveillancesource(e.g.ADS-Borairborneradar),suchcasesaretobeexpected,anditisrecommendedfurtheranalysisbeconductedtodeterminetherateofoccurrenceofsuchRAsintheNASandpotentialmitigations,ifneeded.
Observationsonthestabilityofalertsfor“ModeC-onlyIntruders”werelimitedtoasinglescenarioinFT4.Assuch,moredataisnecessarytodrawfirmconclusionsontheacceptabilityofUASDAA
16
systemperformanceforsuchintruders.TheconcernregardingalertinstabilityforMode-C-onlyintrudersstemsfromthepoorbearinginformationderivedfromtransponderrepliesalone.However,giventheairborneradarequipagerequirementandtypicalradardetectionrange,itisunclearifsuchintrudersrepresentacredibleconcernregardingalertingstability;bearingerrorshouldbeanon-issueoncetheintruderiswithinradardetectionrange.FurtheranalysisofModeCintruderalertstabilityshouldthusconsiderradardetectionrangeandthebearingperformanceofanintegratedintrudertrack.
NinescenarioswereconductedtoassesstheperformanceofUASDAAWellClearRecovery(WCR)guidancetotheUASpilot.WhileobservationsindicatedeffectiveWCRguidanceforADS-Bequippedintruders,guidancewasdeemedlargelyineffectiveforotherintrudersforanumberofreasons.First,UASpilotsindicated(anddataconfirmed)somedirectionalinstabilityinWCRguidance;itisrecommendedthatadditionalheuristicsand/orhysteresisbeincludedinWCRlogictopreventfrequentchangesinthedirectionalWCRguidance.Second,WCRguidanceoccasionallyincludedturnswellbeyond90degreesfromthecurrentcourse;UASpilotsfoundthisobjectionableandineffective.ItisrecommendedthatWCRdirectionalguidancebelimitedtoturnsoflessthansomereasonablebound(e.g.90degreesfromcurrentcourse).
Finally,abroadsetofscenarioevaluationcriteriawerecollectedtoassesshigh-levelUASDAAsystemperformance.CompliancewithscenarioevaluationcriteriaisconsideredindicativeofacceptableUASDAAperformance.FT4JADEMscenarioevaluationcriteriacomplianceexceeded90%andnon-complianceswereassessedtobeeithertestartifactsortheresultsofa(sincecorrected)softwarecodingerror.Whilenoacceptancethresholdforevaluationcriteriacompliancewasestablished,theobservedhighlevelofcomplianceprovidesanecdotalevidenceforrequirementsvalidationofthealertingparameters,WCRguidancelogicandTCASinteroperabilityconceptimplementedwithinJADEMforFT4.Finally,itisimportanttoreiteratethattheresultspresentedhereinarebasedonalimitedsetofscriptedscenariosexecutedinatightlycontrolledflighttestenvironmentwithwell-rehearsedandlimitingpilotproceduresforuseofthesystemundertest;theresults,conclusionsandrecommendationsincludedinthisdocumentprovidekeyinsightsandanecdotalevidence,butrepresentasmallfractionoftheanalysesnecessarytofullyvalidatetheSC-228UASDAAMOPSrequirements.