ENSIGN on HPE Superdome Flex ENSIGN on HPE ......ENSIGN® on HPE Superdome Flex How In-Memory...

7
ENSIGN ® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic ENSIGN ENSIGN ® is a scalable high-performance data analytics software package developed by Reservoir Labs that decomposes large datasets (for example, network logs) into recognizable patterns of behavior and provides actionable insights into the data. The core of ENSIGN is a suite of tensor decomposition routines that are optimized for both shared memory and distributed memory systems. ENSIGN has successfully extracted normal and off-normal patterns, leading to the discovery of anomalous and alarming behaviors, from logs with billions of entries. URL: www.reservoir.com/ensign-cyber/ ENSIGN Features and Benefits Feature Benefit No up-front specification of what to look for Enables the user to detect unknown unknowns Easy-to-use command-line tools to process, ingest, and export network data Simplifies data ingest/export and avoids complex feature engineering on input data Scriptable and modular tools for running the workflow Enables automated workflow and easier integration into existing workflow/platform Integration with modern ML packages and workflows Integrates with Python tools and environments such as Anaconda and Jupyter notebook – makes ENSIGN usable by a data analyst along with other tools Joint analysis of multiple data logs Enables comprehensive and deeper analysis and insights

Transcript of ENSIGN on HPE Superdome Flex ENSIGN on HPE ......ENSIGN® on HPE Superdome Flex How In-Memory...

Page 1: ENSIGN on HPE Superdome Flex ENSIGN on HPE ......ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic ENSIGN® on HPE Superdome Flex How In-Memory Computing

ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic

ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic

ENSIGN ENSIGN® is a scalable high-performance data analytics software package developed by Reservoir Labs that decomposes large datasets (for example, network logs) into recognizable patterns of behavior and provides actionable insights into the data. The core of ENSIGN is a suite of tensor decomposition routines that are optimized for both shared memory and distributed memory systems. ENSIGN has successfully extracted normal and off-normal patterns, leading to the discovery of anomalous and alarming behaviors, from logs with billions of entries.

URL: www.reservoir.com/ensign-cyber/

ENSIGN Features and Benefits

Feature Benefit

No up-front specification of what to look for

Enables the user to detect unknown unknowns

Easy-to-use command-line tools to process, ingest, and export network data

Simplifies data ingest/export and avoids complex feature engineering on input data

Scriptable and modular tools for running the workflow

Enables automated workflow and easier integration into existing workflow/platform

Integration with modern ML packages and workflows

Integrates with Python tools and environments such as Anaconda and Jupyter notebook – makes ENSIGN usable by a data analyst along with other tools

Joint analysis of multiple data logs

Enables comprehensive and deeper analysis and insights

Page 2: ENSIGN on HPE Superdome Flex ENSIGN on HPE ......ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic ENSIGN® on HPE Superdome Flex How In-Memory Computing

Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic

ENSIGN Features and Benefits (continued)

Feature Benefit

Patented and patent-pending high-performance optimizations for core analysis methods

Makes ENSIGN tensor decompositions applicable for large-scale cyber data analysis (level of capability and scalability is well beyond what open source tensor tools could offer)

Advanced patent-pending data structures (“mode specific sparse tensor” and “mode generic sparse tensor”) for tensor methods

Enables effective utilization of compute and memory resources in target compute systems and thereby enables the application of advanced tensor methods for extracting complex patterns

HPC deployments [Large shared or distributed memory systems. Local or in the cloud.]

Leverages 100s or 1000s of cores in the system with maximum utilization and minimum data footprint – enables scalable large-scale deployment

Streaming (“low-rank”) updates Helps the user understand, in near-real-time, how the network behavior is evolving over time

Richer set of advanced analysis methods

Enables identification and interpretation of sophisticated network behaviors

Page 3: ENSIGN on HPE Superdome Flex ENSIGN on HPE ......ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic ENSIGN® on HPE Superdome Flex How In-Memory Computing

Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic

ENSIGN Requirements

Operating Environment

Linux

Hardware Platform

Desktop PC 100,000 scale*

Multi-core Server (20 cores, 500 GB) 10-100 million scale

High-end In-memory Server (896 cores, 48 TB) 100 million - 100 billion scale

Cluster of Multi-core Nodes 100 million - 10 billion scale

Human Resources

No tensor expertise, familiarity with data

Software Components

Primary software: ENSIGN v4.2

Dependent software packages: a. Python (v3.7.0) b. Dask (v1.0.0) c. numpy (v1.15) d. matplotlib (v3.0.3)

*scale: indicates the number of log entries in the data file

Page 4: ENSIGN on HPE Superdome Flex ENSIGN on HPE ......ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic ENSIGN® on HPE Superdome Flex How In-Memory Computing

Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic

Superdome Flex Superdome Flex is a uniquely modular, highly flexible and reliable platform that scales seamlessly to help businesses of any size turn critical data into real-time business insights.

Superdome Flex Features and Benefits Turns data into actionable insights at unparalleled scale and in real-time

• HPE Superdome Flex Server offers unparalleled scalability of up to 32 sockets • In-memory design and unmatched memory capacity of up to 48 TB in a single

platform

Superdome Flex Full rack

(Hero shot)

Superdome Flex (Rear 16-slot)

SuperdomeFlex(Topinternal)

Page 5: ENSIGN on HPE Superdome Flex ENSIGN on HPE ......ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic ENSIGN® on HPE Superdome Flex How In-Memory Computing

Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic

• Gives the compute power needed for most demanding in-memory workloads with groundbreaking performance at scale, ultra-low latency, and high bandwidth.

• Designed for the future based on Memory-Driven Computing design principles to boost analytics performance

Enables to keep pace with evolving in-memory computing demands

• HPE Superdome Flex Server has a unique modular design that scales flexibly and seamlessly from 4- to 32-sockets in 4-socket increments; utilizes Intel Xeon Scalable processors both Gold and Platinum.

• Offers cost efficient entry point for mission-critical workloads at 4 sockets and the ability to scale beyond 8 sockets with both cost-efficient Gold processors and premium Platinum processor

• Allows to scale up or scale out the environment with 4-socket modular building blocks and customize each building block to match workload needs with choice of memory size and capacity, processor and core count, and amount and type of I/O

Safeguards mission-critical workloads

• The HPE Superdome Flex Server delivers the highest service levels on industry standards with extreme and proven RAS capabilities not available on other x86 platforms

• Reduces human error with best-in-class predictive fault handling Error Analysis Engine, which predicts hardware faults and initiates self-repair without operator assistance

• Contains errors at the firmware level, including memory errors, before any interruption can occur at the Operating System layer with HPE's “Firmware First” approach

• Allows to isolate workloads and/or consolidate multiple workloads onto a single managed complex with Hewlett Packard Enterprise unique x86 hard partitioning (HPE nPars) and service individual partitions and/or reconfigure while other partitions continue to run undisturbed

• Delivers continuity for Linux® workloads with HPE Serviceguard for Linux (SGLX) high availability and disaster recovery clustering solution, protecting from a multitude of infrastructure and application faults across physical or virtual environments over any distance

Page 6: ENSIGN on HPE Superdome Flex ENSIGN on HPE ......ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic ENSIGN® on HPE Superdome Flex How In-Memory Computing

Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic

ENSIGN on Superdome Flex ENSIGN and Superdome Flex offer a unique software-hardware solution to turn massive amounts of data into critical actionable insights. In particular, “ENSIGN on Superdome Flex” has been demonstrated in the cybersecurity space to accelerate data analytics, tackle large-scale workloads, and enable critical applications. ENSIGN fully and seamlessly utilizes the compute power offered by Superdome Flex for demanding in-memory workloads and enables quick turnaround time for cyber analytics on massive rapidly growing data. ENSIGN enables high performance at scale effectively exploiting the ultra-low latency and high bandwidth benefits offered by Superdome Flex for high-end in-memory computing.

ENSIGN-Superdome Flex performance on analyzing a large cyber log from a real operational network – showing near-ideal scaling of computation time, good scaling of communication time, and good scaling of

overall execution time of analysis

Page 7: ENSIGN on HPE Superdome Flex ENSIGN on HPE ......ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic ENSIGN® on HPE Superdome Flex How In-Memory Computing

Reservoir Labs ENSIGN® on HPE Superdome Flex How In-Memory Computing Accelerates a Modern Analytic

The highlights from the experiments of running ENSIGN on a “16 socket 288 core 12 TB” Superdome Flex box (as illustrated in the figure above) for analyzing a large cyber dataset collected from a real operational network are as follows:

• Able to run large workloads that was not previously possible (for example, workloads not possible with running ENSIGN on a commodity scale-out cluster of Intel Xeon nodes)

• Reduced the end-to-end cyber analysis workflow time from 5 hours down to 5 minutes for a ~10 GB cyber log collected over one day

The accelerated ENSIGN analytic solution is made possible by key optimizations that exploit the distinctive features of Superdome Flex.

Superdome Flex Feature ENSIGN Optimization

Low latency & high bandwidth memory performance

Reduced communication volume and synchronization in ENSIGN methods

Large scaled-up in-memory capacity Memory-efficient parallel computations in ENSIGN methods

Avoiding writing data to disk and keeping data in memory through the ENSIGN workflow

Scalable processing power Operation-minimal parallel computations in ENSIGN methods enabling near-ideal scaling of computations

(212) 780-0527 www.reservoir.com/ensign-cyber/ [email protected]