© 2014 VMware Inc. All rights reserved.

Log Insight Technical OverviewLog Insight 3.3

David PasekVMware TAM

vRealize Log Insight 3.0 - Agenda

• Overview

• Log Insight for vCenter

• Usability & Visualization Options

• Administration & Scale Details

• DEMO

CONFIDENTIAL 2

Log Insight Overview

Hybrid Cloud(Private / Public)Physical Infrastructure

SOFTWARE-DEFINED DATA CENTER

Compute Network Storage

End-User Computing

Extensibility

Applications

Cloud Management Platform (CMP)

Virtualized Infrastructure

Introduction: Environment Landscape

VMware Logs

OS andApp Logs

200 ESXi Host + VMs = 200GB or 2B log events per day

Physical Infrastructure Logs

5

Primary Use Cases

Troubleshooting and Root Cause Analysis

• Follow the trail from vRealize Operations Manager to logs to get to root cause to an observed problem

• Identify the needle in the haystack in real time when troubleshooting a problem

Monitoring

•Monitor metrics and events (performance & change) that are visible only in logs •Identify problems proactively, ensure SLAs and comply to IT policies

Unstructured Data Warehouse

• Collect all the data in one place without the need for custom parsing, transformation of data

• Get full visibility across all your IT environment from a single place

vRealize Log Insight Overview

Intelligent Operations•Enterprise Scale •Predictive Analytics/Machine Learning for faster problem resolution

Built for the Software Defined Data Center•Base version now included with vCenter• Insight into VMware products incl. NSX, vRealize Automation, Horizon View

•Attractive pricing model for customers of all sizes – not based on log volume

Unified Management• Integration with vRealize Operations Management

Suite Inventory integration, alert notifications

Extensible• Over 40 Content Packs Available

Operating system

vSphere

System statistics

Applications

Security

Infrastructure

Logs

Log Insight 2.0

AnalyzeDiscover

SearchVisualize

IT Operations

Security

Compliance

40B events 10 event types…by machine learning

OverviewAppApp

CONFIDENTIAL

vRealize Log Insight for vCenter

vRealize Log InsightAvailable with vCenter Server Standard

Intelligent Log Analytics for vCenter• Free 25-OSI Log Insight pack per vCenter Benefits of vRealize Log Insight for vCenter• Powerful big data log management built for vSphere• Includes all VMware Content Packs • Extensive Log Management – Captures log data from

physical servers, network and storage devices, OSs, applications, and more

• Intuitive on-the-fly keyword filtering and custom dashboards

• Integration with vRealize Operations – Inventory integration, 2-way Alert Visualization

NEW

The best real-time management for SDDC

Security

App

Operating System

vSphere

System Statistics

Applications Other IT

App

Logs

Upgrade to Full vRealize Log Insight for:

• Extensibility – 3rd Party and Custom Content Packs such as Microsoft, Cisco, EMC, NetApp (29+ available)

• Scalability – Cluster Support and Event Forwarding• High availability, Archiving and SSL

Technical Overview

Log Insight Technical Overview

Cloud / Data Center

Log Management

OSLogs

VCLogs

AppLogs

SystemStats

SecurityLogs

API Syslog

Analyze• Can analyze any unstructured time-series data,

configuration etc.• Automatically identifies structures in the data

Scale• Central, scale-out store (no-SQL) for all collected logs• Configurable retention and archiving• Maintenance free

Best for SDDC• Queries, alerts, fields, charts in the vSphere Content Pack

Intelligent Operations

Predictive Analytics

• Machine Learning based Automatic Data Consolidation

• Intelligent data summarization• Cluster similar messages together

• Automatic Schema extraction• Automatically understand message

structure• Intelligent automatic field extraction

Technical Overview

It’s like ‘Rosetta Stone’ for logs

Log Insight proactively learns:

from:

Then you can query it like a database!

Machine Learning• Automatic event clustering

– Cluster similar events: reverse-engineer line of code that generated events– Happens at ingestion time so zero impact on interactive analytics– Example: Search retrieves 10,000, but summarized as just 10 event types

• Schema discovery– Automatically understand the structure each event– Automatic field extraction: “smart fields” defined for each event, including their data

types

vSphere Content Pack• Ships out of the box

• Knowledge about ESXi and vSphere logs as well as vCenter Alarms, Events & Tasks

• It consists of: Queries, alerts, dashboards, group templates, and field extractions

• Divided into functional categories– Including ESXi, Storage, and

vCenter Alarms

• vSphere and Content Pack dashboards cannot be modified – users can clone them into their workspace

Visualize Log Data Using Dashboards

Run all queries in the list

Dashboard Filters

Launch into Interactive Analytics

Types of Dashboards:PersonalShared by AdminsContent Pack

Choose Dashboards

Interactive Analytics

Fields with breakdown charts

Query time range

Overview Chart:By default: count of events over time

Time bar length

Multiple Aggregation functions/analytics

Search Box and Query Builder

Events List

Interactive Analytics – Events List

• Content pack fields belong to a namespace (e.g. vSphere) so that they don’t collide

• The system timestamp on the left is the arrival timestamp that Log Insight uses

• Clicking a field in the Events list or a bar in the overview chart list creates a constraint

• The constraints can form a logical AND (match all) or logical OR (match any)

Message arrival timestamp, can differ from the one embedded in the message

Total matches

Standard syslog fields (defined in syslog RFC)

CP and Integration fields

vRealize Log Insight – Usability & Visualization

Intelligent Visualizations & Extensibility

• Multiple-Function charts• Chart Options• Snapshots Visualization• Share a Query• Event Type Colorizing, Highlighting

Technical Overview

Smarter Visualizations

Snapshots Visualization

Multi-Function Charts

vRealize Log Insight – Multiple Function Charts

CONFIDENTIAL 20

Add additional function to

chart

Overview Chart Visualizations• Table View and Additional Visualizations

– Field Table• Show/hide columns• Add to Dashboard

– Additional chart types• Column Chart• Line Chart• Area Chart• Bar Chart• Pie Chart• Bubble Chart• Data Table

vRealize Log Insight – Snapshots Visualization

CONFIDENTIAL 22

Create Snapshot

vRealize Log Insight – Share a Query!

Log Insight 3.x Auto-Shortened, friendly URLs for

sharing

CONFIDENTIAL 25

Event Types - Highlight and Colorize• Easier visual analysis of events

• Purpose: Troubleshooting and RCA

• How it works: Select the gear icon to the left of any event and select Highlight/Colorize option

Quickly Identify Distinct Event Types, Within Events Tab

26

View Chart Data as a Table

Build data visualizations using a table

Choose Table to change the visualization

•  

vRealize Log Insight – Administration & Scale

Highly Available, Simplified, and Scalable• Ingestion and Query HA

• 12 Cluster Nodes, 48TB of live log data, 2.7 TB per day

• Integrated Load Balancer with Multiple VIPs

• Enhanced vSphere Integration

• APIs

• Authentication and Simple Query

• Agent & Agent Management

• Client-side event parsing

• Agent Groups for Centralized configuration

• Agent SSL Support

• Webhooks

• Rolling Upgrades & Rollback for Cluster Nodes

Technical Overview

Simplified and Powerful Administration

Agent Configuration

Groups

vRealize Log Insight – Query HA

Log Insight 2.5 Log Insight 3.x

Integrated Load Balancer – Multiple VIP

30

Define multiple VIPs with ILB Associate tags with

each VIP. Useful for RBAC and Content

Packs

ILB Election process is unchanged

LI 3.x - Query HA

Log Search Worker

Log Search Worker

Log Search Worker

Log Search Worker

Log Search Master

UI

Load Balancer

Client

VIP

UI Query Request

Log Search Master

Log Search Master

Log Search Master

UI UI UI

Master

Log Search Master

UI

LI 3.0 Query HA – Master Dies

Log Search Worker

Log Search Worker

Log Search Worker

Log Search Worker

Log Search Master

UI

Load Balancer

Client

VIP

UI Query Request

Log Search Master

Log Search Master

Log Search Master

UI UI UI

Master

35

Authentication API• Use-case: Required to leverage the query API (discussed next)

Request Response

GET /api/v1/sessions HTTP/1.1Host: localhost:9543Accept: application/json

HTTP/1.1 200 OKContent-Type: application/json; charset=utf-8 {  "userId": ”<user_guid>",  "ttl": <int>}

POST /api/v1/sessions HTTP/1.1Host: localhost:9543Accept: application/jsonContent-Type: application/json { ”provider”: “<Local|ActiveDirectory>”,  "username": ”<username>",  "password": ”<password>"}

HTTP/1.1 200 OKContent-Type: application/json; charset=utf-8 {  "userId": ”<user_guid>",  "sessionId": ”<session_id>",  "ttl": <int>}

vRealize Log Insight – Agent Groups

Predefined “Agent Groups” configuration can now be included in

content packs

vRealize Log Insight – Agent Groups (con’t)

CONFIDENTIAL 38

Define Agent Configuration for

Group

Create New or select existing

Agent Group

Define Filter(s) to apply configuration to specific

systems

Results Based on Defined Filter(s)

vRealize Log Insight – Client-Side Event Parsing

CONFIDENTIAL 39

Example: CSV Log Parser

liagent.ini

Example: Configuration

[parser|myparser] base_parser = csv fields = field_name1, field_name2, field_name3 delimiter = ";"

[filelog|some_csv_logs] directory=D:\Logs include=*.txt;*.txt.* parser=myparser

; define a parser section with an arbitrary name ; which can be referred from log sources[parser|myparser]

; parser section refers to any defined parser such as csv, kvp, etcbase_parser = csv

; define extracted fields to include separated by commasfields = field_name1, field_name2, , field_name4

; define delimiter to be used by the parser enclosed in quotesDelimiter = “;”

; after defining parser, refer to it from winlog or filelog sources[filelog|some_csv_logs]

; define source directoryDirectory=csvsource1

; define log files to includeInclude=*.txt

; refer to the parser name you defined aboveparser=myparser

Integrations and Extensibility

vSphere Integration

42

Add tags for vCenter and ESXi events. RBAC use case

Automatically add unconfigured

Hosts

Choose ILB VIP

Better Together: vRealize Operations and vRealize Log Insight

Leverage all your IT data for comprehensive visibility in one place

Structured Data

Metrics Alerts Events

VMware vRealize Operations

Capacity, Performance and Configuration Management Events

Launch in Context

Unstructured Data

Logs Messages

VMware vRealizeLog Insight

Log analytics, aggregation, and search

App App

App

App

App

App

App

App

App

App

App

Public Cloud

VMware vRealize Log Insight Extensibility

Highly Extensible• Captures log data from physical servers, network and storage devices, OSs, applications, VMs, and hosts, and more

Log Insight Content Packs• Encapsulate, pre-built dashboards and product-specific alerts from vRealize Log Insight• Provide vendor specific guidance and insight into which logs really matter

Log Insight Marketplace• Built into the UI or available at www.solutionexchange.vmware.com

Content Packs

Overview

• Operating System• Application• Network• Storage• SDDC• Security

CONFIDENTIAL

Demonstration

Learn More

Try the Hands-on Lab. Nothing to download!

47

Visit the website for resources, 60-day free trial,

evaluation guide, and purchasing information.

@VMLogInsight

www.vmware.com/products/vrealize-log-insight

vmware.com/go/vRealize-Ops-Insight-HOL

loginsight.vmware.com/

Website:

Hands-on Lab:

Log Insight Community:

Thank You