Big Data Impacts on Hybrid Infrastructure and Management
-
Upload
enterprise-management-associates -
Category
Data & Analytics
-
view
56 -
download
2
Transcript of Big Data Impacts on Hybrid Infrastructure and Management
Shamus McGillicuddy
Senior Analyst
@shamusEMA
Big Data Impacts on Hybrid Infrastructure and Management
April 18, 2023
Today’s Presenters
Slide 2
Shamus McGillicuddy, Senior Analyst, EMA Shamus has more than nine years of experience in the IT industry, primarily as a journalist covering the network infrastructure market. At Enterprise Management Associates (EMA), he is the senior analyst for the network management practice.
Prior to joining EMA, Shamus was the news director for TechTarget's networking publications. He led the news team's coverage of all networking topics, from the infrastructure layer to the management layer. He has published hundreds of articles about the technology and competitive positioning of networking products and vendors. He was a founding editor of TechTarget's website SearchSDN.com, a leading resource for technical information and news on the software-defined networking industry.
Slide 3
Logistics for Today’s Webinar
An archived version of the event recording will be available at www.enterprisemanagement.com
• Log questions in the Q&A panel located on the lower right corner of your screen
• Questions will be addressed during the Q&A session of the event
Questions
Event recording
Shamus McGillicuddy
Senior Analyst
@shamusEMA
Big Data Impacts on Hybrid Infrastructure and Management
April 18, 2023
Research Goals
• Investigate the impacts that big data collection and analysis have on infrastructure performance and behavior.• Determine whether these infrastructure
impacts affect engineering and operational practices
• Examine how IT organizations make big data work for them• Identify the infrastructure monitoring data IT
organizations export to big data environments• Determine how analysis of that data supports
IT planning, monitoring and troubleshooting • Identify the key organizational benefits
experienced by successful users of these techniques
Slide 6 © 2015 Enterprise Management Associates, Inc.
Research Demographics
• 156 participants• Mix of IT professionals and IT directors/executives
• 43% executives and 57% technical staff
• Technology savvy professionals• Strong IT/IS/Network focus: 69%• The rest are telecommunications, engineering/R&D, technical customer
support, quality assurance and general management
• North American focus (97%)• Broad distribution of company size
27% small enterprise (1-999 employees) 35% medium enterprise (1,000-4,999) 38% large enterprise (5,000 and up)
• Two statically significant vertical industries 16% Finance/Banking/Insurance 15% Manufacturing (excluding computer hardware)
Slide 7 © 2015 Enterprise Management Associates, Inc.
Big Data Maturity Perspective
• Research participants identified the number of big data projects they have in production• Reveals levels of experience and maturity organizations have
with big data• Does an organization's approach to big data analytics for IT
change as they gain experience?
• Classifying organizational experience by number of big data projects• 29% Beginners: 1 or 2 big data projects in production• 38% Intermediates: 3 to 5 projects• 33% Advanced: 6 or more projects
Slide 8 © 2015 Enterprise Management Associates, Inc.
Big Data Impacts on Infrastructure
Slide 9 © 2015 Enterprise Management Associates, Inc.
Increased network traffic load due to big data collection
Increased network traffic load for backup of big data repositories
Increased compute demands for big data processing/analytics
Increased storage needs for big data stores
Increased use of cloud services for big data storage
Increased use of cloud services for big data analytics
No changes observed
45%
46%
42%
58%
42%
30%
13%
Have you witnessed any changes in the behavior or performance of your orgs IT infrastructure due to any types of big data projects?
Sample Size = 156, Valid Cases = 156
Big Data Impacts on Infrastructure
• Beginner big data users feel fewer infrastructure impacts• 35% of beginners report no infrastructure impacts• But that won’t last
94% of advanced firms observe infrastructure impacts
• Beginners typically only experience increased storage demands (46%)• Other tech domains impacted only 20% to 26% of the time
• Advanced firms feel infrastructure impacts everywhere• 63% increased storage demands• 61% increased network traffic from big data backups• 55% increased network traffic from big data collection• 55% increased use of cloud services for big data storage• 53% increased compute demands for big data analytics
Slide 10 © 2015 Enterprise Management Associates, Inc.
Big Data Impacts on Infrastructure: Vertical insights
• Manufacturers are leaning on the cloud for big data storage and analytics• 54% see increased use of cloud-based big data storage (vs. 42% of
general sample)• 46% see increased use of cloud-based big data analytics (vs. 30% of
general sample)• Manufacturers demonstrate an affinity for the cloud throughout this
research
• Finance uses cloud for big data storage (44%) but not for analytics (8%)
Slide 11 © 2015 Enterprise Management Associates, Inc.
Big Data Impacts on Infrastructure Planning and Engineering
Slide 12 © 2015 Enterprise Management Associates, Inc.
Datacenter systems managers
Database managers
Cloud managers
Security managers
Application developers
Storage managers
Network Managers
Other
None of the above
46%
52%
47%
47%
41%
45%
53%
1%
6%
Which teams have seen impact to infrastructure planning and design practices due to the presence of big data projects?
Sample Size = 156, Valid Cases = 156
Big Data Impacts on Infrastructure Planning and Engineering
Slide 13 © 2015 Enterprise Management Associates, Inc.
Datacenter systems managers
Database managers
Cloud managers
Security managers
Application developers
Storage managers
Network Managers
Other
None of the above
35%
39%
24%
39%
26%
43%
46%
0%
13%
46%
54%
47%
39%
44%
37%
47%
2%
3%
55%
61%
67%
65%
51%
55%
67%
0%
4%
Which teams have seen impact to infrastructure planning and design practices due to the presence of big data projects?
Advanced
Intermediate
Beginner
Sample Size = 156, Valid Cases = 156
Big Data Impacts on Infrastructure Operations
Slide 14 © 2015 Enterprise Management Associates, Inc.
Network ops
Security ops
Datacenter systems ops
Application support
Cloud ops
IT (cross-domain) ops
Other
None of the above
49%
49%
52%
37%
43%
49%
0%
4%
Which teams have seen significant impact to their daily ops practices due to big data projects?
Sample Size = 156, Valid Cases = 156
All aspects of operations equally affected, except app support and cloud ops
Big Data Impacts on Infrastructure Operations
Slide 15 © 2015 Enterprise Management Associates, Inc.
Network ops
Security ops
Datacenter systems ops
Application support
Cloud ops
IT (cross-domain) ops
Other
None of the above
46%
35%
37%
22%
35%
48%
0%
9%
42%
51%
49%
32%
36%
46%
0%
2%
61%
61%
69%
55%
59%
55%
0%
4%
Which teams have seen significant impact to their daily ops practices due to big data projects?
Advanced
Intermediate
Beginner
Sample Size = 156, Valid Cases = 156
Advanced big data firms have ops issues across the board.Beginners concentrate on network and cross-domain operations.
Big Data Analytics for IT
Slide 16 © 2015 Enterprise Management Associates, Inc.
Yes, currently using
No, but planning to use in next 12 months
No, but considering at some point
No use or plans
71%
18%
11%
0%
Are you currently using or planning to use big data systems and technologies to collect and/or analyze
IT infrastructure monitoring data?
Research participants: EMA sought enterprises that were considering, planning or currently applying big data analytics to IT infrastructure monitoring data.
Organizations with no plans to do so were excluded.
IT Monitoring Data Exported to Big Data Repositories
Slide 17 © 2015 Enterprise Management Associates, Inc.
Time series performance data
Log entries
Flow records (NetFlow, sFlow, IPFIX)
Raw network packets
Interpreted packet flow/wire data
Application performance data
Transaction records
Cloud provider API metrics
Other
46%
41%
43%
30%
39%
59%
52%
50%
1%
Which types of IT data are you collecting and adding to a big data repository?
Sample Size = 122, Valid Cases = 122
Exporting IT Monitoring Data to Big Data Repositories
Slide 18 © 2015 Enterprise Management Associates, Inc.
Time series performance data
Log entries
Flow records (NetFlow, sFlow, IPFIX)
Raw network packets
Interpreted packet flow/wire data
Application performance data
Transaction records
Cloud provider API metrics
Other
28%
28%
28%
10%
28%
69%
55%
34%
3%
47%
36%
45%
30%
32%
53%
47%
49%
0%
57%
54%
50%
41%
54%
59%
57%
61%
0%
Advanced
Intermediate
Beginner
Sample Size = 122, Valid Cases = 122
Other IT monitoring data types becomes more popular as organizations gain big data experience.
Advanced users are exporting everything.
Advanced users find other data types more valuable; APM and transaction records lose relative importance
Slide 19 © 2015 Enterprise Management Associates, Inc.
Time series performance data
Log entries
Flow records (NetFlow, sFlow, IPFIX)
Raw network packets
Interpreted packet flow/wire data
Application performance data
Transaction records
Cloud provider API metrics
Other
24%
14%
21%
3%
10%
55%
45%
31%
3%
23%
13%
32%
15%
23%
43%
32%
32%
0%
37%
22%
22%
17%
22%
39%
35%
39%
0%
Which types of IT data are the three most important for big data analytics for IT?
Advanced
Intermediate
Beginner
Sample Size = 122, Valid Cases = 122
What IT Management Practices are Supported by Big Data Analytics?
Slide 20 © 2015 Enterprise Management Associates, Inc.
Internal infrastructure planning
Hybrid/Cloud infrastructure planning
Ops monitoring
Infrastructure security
Troubleshooting/diagnostics
Service quality analysis/improvement
Business activity insights
None of the above
50%
44%
56%
53%
37%
45%
35%
4%
Sample Size = 156, Valid Cases = 156
The top 3:• Operations monitoring• Infrastructure security• Internal Infrastructure planning
What IT Management Practices are Supported by Big Data Analytics?
• A deeper look into these supported management practices• Troubleshooting becomes more important as
users gain experience with big data Nearly half of advanced users apply big data
analytics to troubleshooting and diagnostics
• Service quality analysis/improvement becomes a majority use case for advanced users (55%)
• The discovery of business activity insights (33%) is the the least supported management practice
Key enabler of IT-business alignment No variation with big data experience
Slide 21 © 2015 Enterprise Management Associates, Inc.
Three Constituencies Benefit from Big Data Analytics for IT
Slide 22 © 2015 Enterprise Management Associates, Inc.
IT Infrastructure Engineering
IT Infrastructure Ops
Architecture
DevOps
Service portfolio planning
Service Delivery
Other cross-domain service mgmt
Configuration mgmt
Change mgmt
Asset mgmt /Financial planning
Capacity planning
Vendor mgmt
User Experience Mgmt
On-line Ops
Executive IT
Other
37%
49%
10%
9%
9%
9%
6%
14%
9%
17%
19%
8%
15%
11%
39%
1%
Which functional groups/practices will most benefit from leveraging the results of big data analytics for IT?
Sample Size = 150, Valid Cases = 150
How Do IT organizations Use Big Data Analytics
• The research explores use cases across three key practice areas• Planning and Engineering• Technical performance monitoring• Trouble triage and diagnostics
• Which types of IT monitoring data are valuable to each of these analytical tasks?
Slide 23 © 2015 Enterprise Management Associates, Inc.
Planning & Engineering Tasks: Capacity Planning is THE Use Case for Big Data Analytics for IT
Slide 24 © 2015 Enterprise Management Associates, Inc.
Network capacity planning
Server capacity planning
Storage capacity planning
DevOps
External Cloud capacity planning
SLA/SLO planning
Other
None of the above
57%
66%
70%
29%
41%
23%
0%
6%
For which types of planning & engineering tasks does your org use or plan to use big data analytics for IT?
Sample Size = 150, Valid Cases = 150
Planning & Engineering Tasks Supported by Big Data
• Advanced users are more innovative with:• External cloud capacity planning (54% of advanced users versus 41%
of general research pool)• SLA/SLO planning (38% of advanced firms versus 23% of general pool)
• Manufacturers also more active around external cloud capacity planning (54%)
• Finance lags in network capacity planning (36%), external cloud (20%) and SLA/SLO planning (4%)
Slide 25 © 2015 Enterprise Management Associates, Inc.
Most Important Data Types for Planning & Engineering Tasks
Slide 26 © 2015 Enterprise Management Associates, Inc.
Events
Time series performance data
Log entries
Flow records (NetFlow, sFlow, IPFIX)
Raw network packets
Interpreted packet flow/wire data
Application performance data
Transaction records
Cloud provider API metrics
Other
26%
44%
35%
34%
32%
36%
51%
51%
48%
0%
Which types of data are most important for planning & engineering via big data analytics for IT?
Sample Size = 116, Valid Cases = 116
Application performance data, transaction records, & cloud provider API metrics are top three.
Most Important Data Types for Planning & Engineering Tasks
Other data types gain importance for advanced big data users
Slide 27 © 2015 Enterprise Management Associates, Inc.
Flow records (39% of advanced users
versus 34% of general pool)
Interpreted packet flow (43% of advanced versus
36% of general)
Raw network packets (41% of advanced versus
32% of general)
Technical Performance Monitoring Tasks Supported by Big Data
Slide 28 © 2015 Enterprise Management Associates, Inc.
Cross-domain service performance and availability
Network availability and performance
Systems availability and performance
Storage availability and performance
Service transaction response time
Configuration/change mgmt effectiveness
Infrastructure optimization
Application performance optimization
Security-related issues
Other
None of the above
25%
53%
52%
49%
36%
32%
45%
36%
27%
0%
3%
For which technical performance issues does your org use or plan to use big data analytics for IT?
Sample Size = 150, Valid Cases = 150
Technical Performance Monitoring Tasks Supported by Big Data
• Advanced big data users• Especially focused on systems availability and performance monitoring
(68%)• More focused (44%) on application performance optimization• Other advanced use case
Service transaction response time monitoring: 42% Configuration/change management effectiveness: 44%
• Executive IT• Bigger focus on infrastructure optimization (54% versus 39% of IT staff)• Like advanced users, exploring configuration/change management
effectiveness (40% versus 26% of staff)
Slide 29 © 2015 Enterprise Management Associates, Inc.
Most Important Data Types for Technical Performance Monitoring
Slide 30 © 2015 Enterprise Management Associates, Inc.
Events
Time series performance data
Log entries
Flow records (NetFlow, sFlow, IPFIX)
Raw network packets
Interpreted packet flow/wire data
Application performance data
Transaction records
Cloud provider API metrics
Other
32%
47%
48%
48%
33%
37%
63%
36%
25%
0%
Which types of data are most important for technical performance monitoring via big data analytics for IT?
Sample Size = 117, Valid Cases = 117
Triage & Diagnostics Tasks Supported by Big Data: No Dominant Use Case
Slide 31 © 2015 Enterprise Management Associates, Inc.
Isolate problem to application or infrastructure
Triage across application tiers and/or middleware
Isolate the relevant service provider
Triage across virtualized systems
Isolate infrastructure issues in the network
Isolate infrastructure issues internal to systems
Isolate infrastructure issues within the DB tier
Isolate infrastructure issues in storage
Isolate non-mobile end device issues
Isolate mobile-specific end-device issues
Isolate security-related issues
Other
None of the above
30%
37%
37%
41%
35%
37%
29%
43%
25%
21%
27%
0%
7%
For which types of technical triage and diagnostics tasks does your org use or plan to use big data analytics for IT?
Sample Size = 150, Valid Cases = 150
Triage & Diagnostics Tasks Supported by Big Data: Advanced Users Identify Key Use Cases
Slide 32 © 2015 Enterprise Management Associates, Inc.
Isolate problem to application or infrastruc-ture
Triage across application tiers and/or middleware
Isolate the relevant service provider
Triage across virtualized systems
Isolate infrastructure issues in the network
Isolate infrastructure issues internal to sys-tems
Isolate infrastructure issues within the DB tier
Isolate infrastructure issues in storage
Isolate non-mobile end device issues
Isolate mobile-specific end-device issues
Isolate security-related issues
Other
None of the above
34%
27%
22%
32%
24%
37%
20%
44%
22%
22%
44%
0%
12%
25%
41%
36%
46%
32%
31%
32%
42%
20%
24%
20%
0%
5%
32%
42%
52%
44%
46%
44%
34%
44%
34%
18%
22%
0%
4%
Advanced
Intermediate
Beginner
Sample Size = 150, Valid Cases = 150
Most Important Data Types for Triage and Diagnostics
Slide 33 © 2015 Enterprise Management Associates, Inc.
Events
Time series performance data
Log entries
Flow records (NetFlow, sFlow, IPFIX)
Raw network packets
Interpreted packet flow/wire data
Application performance data
Transaction records
Cloud provider API metrics
Other
28%
42%
38%
35%
28%
35%
56%
41%
47%
0%
Which types of data are most important for triage and diagnostics via big data analytics for IT?
Sample Size = 116, Valid Cases = 116
What Benefits Do Enterprises Experience From Big Data Analytics for IT
Slide 34 © 2015 Enterprise Management Associates, Inc.
Faster time to repair problems
Proactive ability to prevent problems
More efficient use of infrastructure capacity
Better correlation between change and performance
Better alignment with the business
Improved OpEx efficiencies within IT
Improved compliance with industry requirements
Fast identification of security threats
Faster time to deliver new IT services
Less overhead writing/maintaining rules and thresholds
Surprises / unexpected insights
Other
None so far, it's still too soon to tell
31%
45%
46%
36%
41%
45%
29%
36%
35%
23%
9%
0%
4%
Sample Size = 124, Valid Cases = 124
Summary
• Big data most often impacts storage capacity and network performance• All infrastructure planning & operations practices are affected, especially as
firms gain more big data experience
• IT monitoring data imported into big data environments• Application performance data, transaction records, cloud provider API
metrics are most popular
• Use cases for big data analytics for IT• Storage, systems and network capacity planning are tops• Availability and performance monitoring of these 3 technologies also
important• No dominant troubleshooting use case, but storage problem isolation is
prominent
• Key benefits:• Efficient use of infrastructure and OpEx, proactive problem prevention
Slide 35 © 2015 Enterprise Management Associates, Inc.
Download the report for more insights
• The report identifies:• The backend databases these
enterprises use• The analytical tools they apply to these
big data environments
• More insights by vertical industries, size of company, and role
• More Analysis of how big data impacts and practices vary by big data maturity levels
• The technical and cultural issues that impact big data success
Slide 36 © 2015 Enterprise Management Associates, Inc.