April 18, 2018 Sam Siewert, ICARUS Group

CU Denver, 2018

Drone Net Architecture

UAS Traffic Management Multi-modal Sensor Networking Experiments

Drone NetVideo overview of project -Watch video

Motivation – Growth of sUAS– Droneii, FAA, Sandia, ASSURE– Counter UAS, NASA UTM (UAS

Traffic Mgt.)

Problem – Default solution – Part 107, Restrictions, ADS-B Rx– Ground RADAR, LIDAR high cost– sUAS ADS-B Tx/Rx alone

insufficient– Flight LIDAR & Vision (last 50 feet)

Sam Siewert, ICARUS Group CU Denver 2018 - Drone Net 2

Method – Information FusionEO/IR Multi-spectral imaging

– Visible, NIR, LWIR pixel-level fusion– Lower-cost than common bore-sight

Acoustic cameras & Microphone arrays– Beam-forming microphone array (Azimuth & Elevation)

All-sky camera - hemispherical, high resolution– Azimuth and Elevation

Link Drone Net Nodes– Detect and Track Compliant and Non-compliant sUAS– Wireless to Acoustic arrays, wired All-sky and EO/IR

ADS-B aggregation and improvement– Comparing ADS-B to OEM and Custom IMU+GPS (e.g.

ping2020)

Flight LIDAR and EO/IR - last 50 foot navigation

© Sam Siewert, ICARUS Group CU Denver 2018 - Drone Net 3

Drone Net - UAS Traffic Management Test System

Air-column Test Range[Drone Net NodeSensor Network]

1 Km

1 Km

ADS-B[Ping 2020i]

ADS-B truth

Machine Vision & Learning[Real-Time and Simulation]

sUAS

Compliant Flight Configuration

Passive Sensing Net

AcousticArray

EO/IR Narrow Field

Performance

ROC, PR, F-measure,new metrics

S or X-band RADAR

Active Sensing

Ground LIDARLimited Range

Detect, Track, Classify, Identify, Localize

Cluster &GP-GPU

GA Traffic

MAVlink

UAS LIDAR For Proximity

Operations

NASDatabase

All-skyHemispherical

Navigation Logtruth

© Sam Siewert, ICARUS Group CU Denver 2018 - Drone Net 4

Method – Machine Vision & LearningMachine Vision using SoC Linux– Salient Object Detection

Shape, Behavior and Contrast/Color/Texture in Multiple BandsPerformance [ROC, PR, F-measure, confusion matrices]

– Real-Time Detection, Segmentation, Tracking, Classification, Identification

Machine Learning (Traditional, ANN)– Expert systems– Bayesian inference, Deep Belief Net– PCA [Principal Component Analysis]– SVM [Support Vector Machines]– Clustering [e.g. K-means]– GPU Accelerated DNN (cuDNN)– Supervised, Unsupervised learning

Leverage Open Source: ROS, OpenCV, PyBrain, PyML, MLpack, cuDNN, Caffe, Tensorflow

© Sam Siewert CU Denver 2018 - Drone Net 5

617.5 meters [diagonal measure of cell that is 437m2]

Mavic Pro [335mm diagonal size]

FreeFly ALTA6 [1.126 m diameter]

Drone Net - Feasibility Test Range

© Sam Siewert, ICARUS Group CU Denver 2018 - Drone Net 6

874 m

874 m

EO/IR Cost estimate - $33K1. Qty 5, $5K EO/IR = $25K2. Qty 4, $2K All-sky = $8K

Aviation Weather RADAR1. Garmin X-band GWX-70 - $33K+ (Weather only)

Ground RADAR comparison - $120K to $610K1. EEC Ranger X5 dual-polarization X-band - est. $610K2. Echodyne - MESA-DAA - est. $120K for 4 panels, (750m)3. SRC Gryphon R1400 - unknown cost, (8.5Km)

Target Feature

EO / IR

Acoustic SpectrumAnalyzer

RADAR

Shape X X(dual-polar)

Track X X X

Texture,color

X

Thermal X

Acoustic Spectrogram

X

E-magsignature

X

Range <1Km <100m <5Km <10Km

Drone Net - RADAR Comparison

© Sam Siewert, ICARUS Group CU Denver 2018 - Drone Net 7

617.5 m

437 m

© Sam Siewert, ICARUS Group CU Denver 2018 - Drone Net 8

Cover ERAU Campus

5 EO/IR Nodes4 All-sky CamerasAcoustic (TBD)

874m x 874m

Approximately1Km GridImagined for ATC

VLOS from Physicsto DLC Parking lot

VLOS from AXFAB toDLC Parking lot

300m

300m

MV/ML Flight EO/IR Frames[OEM Snapshot for prototype,MV/ML future enhancement]

MV/ML Ground EO/IR Frames[Detection, Classification and Identification

Subset of frames fromContinuous 10Hz baseline]

MATLABGeometric Analysis

& Re-Simulation

OEM NavigationLog Data

HF NavigationLog Data

[future enhancement]

ADS-B Log Data[sUAS, GA compliant

identification]

MV/MLDetection Performance

HRV ROC, PR, F-measure

Human ReviewDetection, Classification, and Identification

{TP, FP, TN, FN}

Localization Error &ADS-B Identification, Detection

{TP, FP, TN, FN}

Simulated HFOV, VFOVAnd Cross Section of Tracked sUAS

Synthetic Frame Generation

Time Correlated Frame Retrieval

HF truthOEM truth

Optical Navigation truth

Frame Compare

ADS-B truth

Actual

sUAS Not sUAS

PredictedsUAS 250 TP 43 FP

Not sUAS 0 FN 3 TN

© Sam Siewert, ICARUS Group CU Denver 2018 - Drone Net 9

Acoustic Detection/LocalizationResearch Topic #1

Dr. Mehran Andalibi, Omkar Prabhu (CU MS ESE), Dr. Siewert– Goal - Perimeter Detection (all

conditions)– Elevation/Azimuth Estimate for EO/IR– General Localization to narrow EO/IR

tilt/pan search for re-detection– Characterization of Drone acoustic

signatures in progress with ANSYS

Sam Siewert CU Denver 2018 - Drone Net 10

All-sky acoustic evaluation kit for beam-forming DOA

5 meters from sourceFrequency [KHz] 1.5 2.5 3.5 Actual AngleDetected angle 22.5 22.5 22.5 35

67.5 67.5 67.5 70202.5 202.5 202.5 230

10 meters from sourceFrequency [KHz] 1.5 2.5 3.5 35Detected angle No detection 22.5 22.5 35

No detection 67.5 67.5 70No detection 202.5 202.5 230

Angle Estimation Compared to Actual in Feasibility Testing

MV/ML Classification/IDResearch Topic #2

Dr. Siewert, Soumyatha Gavvala (CU MS ESE)– Tensorflow R-CNN and Single Shot Detector– ROC comparison to Motion-Detect Baseline

Goal - Outperform Motion-Detect with filters, train on workstation, deploy to Tegra TX2 for real-time operations

Sam Siewert CU Denver 2018 - Drone Net 11

Results from R-CNN MV/ML Classification with Drone Net Detection Images

Large Aerial ObjectCatalog collected atEach Drone Net site

All-Sky Detection/LocalizationResearch Topic #3

Dr. Siewert, Aasheesh Dandupally (CU MS ESE)– Simple Salient Object / Motion Detect in Hemisphere (all-sky)– 6 Cameras with overlapping FoV

Goal - detect and estimate aerial object elevation and azimuth to notify EO/IR for re-detection and tilt/pan trackingEliminate simple false alarms (birds, bugs, clouds)

Sam Siewert CU Denver 2018 - Drone Net 12

6 x 2 Mpixel MPTSScale up to

6 x 20 Mpixel

EO/IR RT Detect, Classify, Track and LocalizeResearch Topic #4

Dr. Siewert, Garrison Bybee (ERAU SE)– Notification from All-sky or Acoustic provides

coarse Elevation, Azimuth– Re-detect and classify as sUAS (CNN, DBN,

SVM, …)– Tilt/pan tracking of sUAS targets of interest

and UTM disposition (on-flight plan, off-flight-plan, no-flight-plan)

Goal - Track, shape, texture, color (multi-spectral) for all sUAS of interest, logged to Aerial Catalog

DSRC protocol between EO/IR nodes and Overly-Compliant EO/IR/LIDAR nodes

Support UTM 2020+ Urban ConOps

Sam Siewert CU Denver 2018 - Drone Net 13

Dynamixel MX series servo3D printed gimbalNEMA enclosure2 OTS RGB/Panchromatic1 OTS LWIRJetson TX2RT MV/ML for SOD

Over-Compliant Aerial NodeResearch Topic #5

Dr. Bruder, Gentilini, Siewert, Jonathan Buchholz (ERAU MS UASE)– HF Navigation compared to

ADS-B (Bruder)– LIDAR + LWIR Fusion

(Gentilini, Siewert, Buchholz)– Last 50 foot Urban Navigation

(Buchholz)Goal - Determine safe urban operation, GPS-denied, for parcel delivery scenarios with Sense-and-Avoid

NASA UTM Challenge (2020)

Sam Siewert CU Denver 2018 - Drone Net 14

LIDAR Point Cloud from Lab Bench Test with LWIR image fusion

Data Management and AnalyticsResearch Topic #6

Dr. Siewert, Prof. Perry, Prof. Young

MV/ML on Workstation ground nodes (Lambda DevBox, 20TB RAID)– R-DBMS for Aerial Catalog– Blockchain research for Drone

catalog and tracking events (shared in Drone Net network)

– File management of raw images

– Automated human review (Auto-it)

– Real-Time ATC NOTAMs– Forensic browsing

Goal - Human Review Truth model and Secure sharing of aerial catalog for registered and non-compliant sUAS

Sam Siewert CU Denver 2018 - Drone Net 15

ERAU ICARUS STEM 125 LabCompliance Description Flight

PlanADS-B Complia

nceATC

notification

Registered sUAS, flight plan filed, following flight plan, ADS-B, safe navigation.

X X Full None

No ADS-B, unknown navigation equipment, standing waiver with Part 101 registered drone (e.g. hobby)

X Full None

Registered sUAS, ADS-B, but not on filed flight plan.

X Partial Warning

No ADS-B, unknown navigation equipment, no standing waiver or filed flight plan

Partial Warning

No ADS-B, large visual size, no standing waiver or filed flight plan, not classified or identified as hobby drone, unexpected track, shape, texture, and color in visible and LWIR.

None Safety Alert

Verification with Re-simulationResearch Topic #7

Dr. Andalibi, Bruder– MATLAB simulation to verify

detection in HFOV, VFOV– Track history and geometric

observability– Add virtual cameras to explore

potential improvementsGoal - Geometric Truth model for compliant HF Nav, OEM Nav, ADS-B tracked drone

Sam Siewert, ICARUS Group CU Denver 2018 - Drone Net 16

Track segmentshown in “a”

SummaryDrone Net Architecture Defined and Shown Feasible– 2 Rural Tests Completed [IEEE Aerospace data]– 2 Urban Tests Completed [2/25/18, 3/25/18]

Promise to match RADAR at lower cost

Forms Reference Design for UTM collaboration research

Next Steps …– Further develop research topics, analyze, replicate and scale– Enable simulation of larger systems [MATLAB]– RADAR simulation compare– Work with partner with ground RADAR– Test UTM Last 50 Foot Scenario (Parcel Delivery)

Questions?

Sam Siewert, ICARUS Group CU Denver 2018 - Drone Net 17

2017/18 Team – ERAU SponsoredERAU – Drone Net

– Dr. Sam Siewert, PI, Assistant Prof.– Dr. Iacopo Gentilini, Co-I– Dr. Stephen Bruder, Co-I– Dr. Mehran Andalibi, Co-I

– Jonathan Buchholz - ME Robotics– Dakota Burklund - AE Student– Garrison Bybee - SE Student– Jacqueline Abendroth-Smith - ME Robotics– Evan Atlas - AE Student– Demi Matthew Vis – AE/SE Student (graduated)– Ryan Claus – SE Student (graduated)

CU Boulder – Embedded Systems Graduate– Aasheesh Dandupally – MS, ESE– Omkar Prabhu – MS, ESE– Soumyatha Gavvala – MS, ESE– Surjith Singh – MS, ESE (graduated)– Akshay Singh – ME, ESE (graduated)– Shivasankar Gunasekaran – ME,ESE (graduated)– Omkar Seelam – ME, ESE (graduated)– Kalyan Pingali – ME, ESE (graduated)

Industry Advising/Collaboration Participants– Randall Myers, Mentor Graphics (PCB, CAD, Systems)

Sam Siewert CU Denver 2018 - Drone Net 18

Research References

Sam Siewert 19

References

Sam Siewert 20

S. Siewert, M. Andalibi, S. Bruder, I. Gentilini, A. Dandupally, S. Gavvala, O. Prabhu, J. Buchholz, D. Burklund, “Drone Net, a passive instrument network driven by machine vision and machine learning to automate UAS traffic management”, AUVSI Xponential poster, Denver, Colorado, May 2018.

S. Siewert, M. Andalibi, S. Bruder, I. Gentilini, J. Buchholz, “Drone Net Architecture for UAS Traffic Management Multi-modal Sensor Networking Experiments”, IEEE Aerospace Conference [presentation], Big Sky, Montana, March 2018.

S. Siewert, M. Vis, R. Claus, R. Krishnamurthy, S. B. Singh, A. K. Singh, S. Gunasekaran, “Image and Information Fusion Experiments with a Software-Defined Multi-Spectral Imaging System for Aviation and Marine Sensor Networks”, AIAA SciTech 2017, Grapevine, Texas, January 2017.

S. Siewert, V. Angoth, R. Krishnamurthy, K. Mani, K. Mock, S. B. Singh, S. Srivistava, C. Wagner, R. Claus, M. Demi Vis, “Software Defined Multi-Spectral Imaging for Arctic Sensor Networks”, SPIE Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXII, Baltimore, Maryland, April 2016.

S. Siewert, J. Shihadeh, Randall Myers, Jay Khandhar, Vitaly Ivanov, “Low Cost, High Performance and Efficiency Computational Photometer Design”, SPIE Sensing Technology and Applications, SPIE Proceedings, Volume 9121, Baltimore, Maryland, May 2014.

Piella, G. (2003). A general framework for multiresolution image fusion: from pixels to regions. Information fusion, 4(4), 259-280.

Blum, R. S., & Liu, Z. (Eds.). (2005). Multi-sensor image fusion and its applications. CRC press.

Liu, Z., Blasch, E., Xue, Z., Zhao, J., Laganiere, R., & Wu, W. (2012). Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(1), 94-109.

References

Sam Siewert 21

Simone, G., Farina, A., Morabito, F. C., Serpico, S. B., & Bruzzone, L. (2002). Image fusion techniques for remote sensing applications. Information fusion, 3(1), 3-15.

Mitchell, H. B. (2010). Image fusion: theories, techniques and applications. Springer Science & Business Media.

Szeliski, R. (2010). Computer vision: algorithms and applications. Springer Science & Business Media.

Sharma, G., Jurie, F., & Schmid, C. (2012, June). Discriminative spatial saliency for image classification. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (pp. 3506-3513). IEEE.

Toet, A. (2011). Computational versus psychophysical bottom-up image saliency: A comparative evaluation study. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(11), 2131-2146.

Valenti, R., Sebe, N., & Gevers, T. (2009, September). Image saliency by isocentric curvedness and color. In Computer Vision, 2009 IEEE 12th International Conference on (pp. 2185-2192). IEEE.

Wang, M., Konrad, J., Ishwar, P., Jing, K., & Rowley, H. (2011, June). Image saliency: From intrinsic to extrinsic context. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on (pp. 417-424). IEEE.

Liu, F., & Gleicher, M. (2006, July). Region enhanced scale-invariant saliency detection. In Multimedia and Expo, 2006 IEEE International Conference on (pp. 1477-1480). IEEE.

Cheng, M. M., Mitra, N. J., Huang, X., & Hu, S. M. (2014). Salientshape: Group saliency in image collections. The Visual Computer, 30(4), 443-453.

http://global.digitalglobe.com/sites/default/files/DG_WorldView2_DS_PROD.pdf

http://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf

Richards, Mark A., James A. Scheer, and William A. Holm. Principles of modern radar. SciTech Pub., 2010.

References

Sam Siewert 22

Brown, Christopher D., and Herbert T. Davis. "Receiver operating characteristics curves and related decision measures: A tutorial." Chemometrics and Intelligent Laboratory Systems 80.1 (2006): 24-38.

Wang, Bin, and Piotr Dudek. "A fast self-tuning background subtraction algorithm." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2014.

Panagiotakis, Costas, et al. "Segmentation and sampling of moving object trajectories based on representativeness." IEEE Transactions on Knowledge and Data Engineering 24.7 (2012): 1328-1343.

Public SDMSI shared data web site for video sequences captured and used in two experiments presented in this paper - http://mercury.pr.erau.edu/~siewerts/extra/papers/AIAA-SDMSI-data-2017/

Perazzi, Federico, et al. "Saliency filters: Contrast based filtering for salient region detection." Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012.

Achanta, Radhakrishna, et al. "SLIC superpixels compared to state-of-the-art superpixel methods." IEEE transactions on pattern analysis and machine intelligence 34.11 (2012): 2274-2282.24Hou, Xiaodi, and Liqing Zhang. "Saliency detection: A spectral residual approach." 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2007.

Global Contrast based Salient Region Detection. Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, Shi-Min Hu. IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE TPAMI), 37(3), 569-582, 2015.

flightradar24.com, ADS-B, primary/secondary RADAR flight localization and aggregation services.

Birch, Gabriel Carisle, John Clark Griffin, and Matthew Kelly Erdman. UAS Detection Classification and Neutralization: Market Survey 2015. No. SAND2015-6365. Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2015.

Drone Detection and Neutralization Companies

Leading Drone Detection Companies

• https://www.blacksagetech.com/• https://www.droneshield.com/• http://www.dedrone.com/en/• https://www.kongsberggeospatial.com/applications/argus-cuas• https://fortemtech.com/

List of Drone Detection Companies and Experiments

• DJI Aeroscope• Test at JFK by FBI• SRC Gryphon Sensors Counter UAS• SPI Infrared Drone Detection• Industrial Camera Drone Detection• HGH Infrared Drone Detection• http://www.cerbair.com• Israel Defense Counter UAS• AARONIA Drone Detection

© Sam Siewert 23