Open source: Top issues in the top enterprise packages
-
Upload
rogue-wave-software -
Category
Technology
-
view
283 -
download
0
Transcript of Open source: Top issues in the top enterprise packages
![Page 1: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/1.jpg)
1© 2016 Rogue Wave Software, Inc. All Rights Reserved.
1
Top open source lessonsfor every enterpriseEpisode 4:
Top issues in the top enterprise packages
![Page 2: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/2.jpg)
2© 2016 Rogue Wave Software, Inc. All Rights Reserved.
2
Bill CrowellEnterprise architect, OpenLogic supportRogue Wave Software
Vince CoxEnterprise architect, OpenLogic supportRogue Wave Software
Presenters
![Page 3: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/3.jpg)
3© 2016 Rogue Wave Software, Inc. All Rights Reserved.
3
Poll #1What percentage of your mission critical software is open source?
A: 0 to 25%B: 26 to 50%C: 51 to 75%
D: 75%
![Page 4: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/4.jpg)
4© 2016 Rogue Wave Software, Inc. All Rights Reserved.
4
1. Introduction2. Setting the context3. Top issues: middleware + runtimes4. Top issues: databases5. Top issues: security6. Conclusion7. Q&A
Agenda
![Page 5: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/5.jpg)
5© 2016 Rogue Wave Software, Inc. All Rights Reserved.
5
Who wrote LevelDB and what version of ActiveMQ did it debut in?
What percentage of web server market share does Apache HTTP Server hold?
Pop quiz
![Page 6: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/6.jpg)
6© 2016 Rogue Wave Software, Inc. All Rights Reserved.
6
Introduction
What problems do our clients commonly run into?
From our experience…• ActiveMQ High Availability Shared Databases Using NFS• PostgreSQL Database Performance• Request Header Vulnerabilities
![Page 7: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/7.jpg)
7© 2016 Rogue Wave Software, Inc. All Rights Reserved.
7
Poll #2What type of ActiveMQ persistence store do you use?
A: KahaDBB: LevelDB
C: Replicated LevelDBD: We don’t use message persistence with ActiveMQ
E: We don’t use ActiveMQ
![Page 8: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/8.jpg)
8© 2016 Rogue Wave Software, Inc. All Rights Reserved.
8
Top issues:middleware +
runtimes
![Page 9: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/9.jpg)
9© 2016 Rogue Wave Software, Inc. All Rights Reserved.
9
Scenario #1
Observing crashing or unresponsive broker
Typical configuration/requirements• Master/slave network broker setup with NFS mount• NFSv4 (SoftNAS) cloud-based shared file system available
with Amazon Web Services EC2 or GlusterFS• Often a high-throughput requirement (10k messages/second
of 1-3kb message size)
One of the following problems…• Master dies resulting in “no master”
scenario• Slave prematurely claims lock resulting in 2
master brokers leading to message loss• Continuous master/slave re-election• Increasing CPUs/memory doesn’t help• Non-existent prior to production
![Page 10: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/10.jpg)
10© 2016 Rogue Wave Software, Inc. All Rights Reserved.
10
Scenario #1 solution
Use SAN, Replicated LevelDB, or Pluggable Storage Lockers
Three solutions• Use block-level iSCSI driver with Storage Area Network (SAN)• Master-slave for HA and Replicated LevelDB managed by
Zookeeper• Pluggable Storage Lockers
More points• Inherent flaws in OS-level filesystem locking mechanism• Exclusive file locks work great with a SAN but is most
expensive• Replicated LevelDB requires a more configuration and a
quorum of nodes (replicas / 2 + 1)• Pluggable Storage Locker/Lease Database Locker
![Page 11: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/11.jpg)
11© 2016 Rogue Wave Software, Inc. All Rights Reserved.
11
Scenario #1 solution
Lease Database Locker
Points• Master must renew lease before lease expires• The lease period can be configured• If not renewed, then the slave takes ownership of the
lease becoming the new master
More points• Leased locks can survive database replica failovers• Can be used with any JDBC-compliant database• Make sure to uniquely name your brokers• Keep master/slave clocks synchronized with NTP
service• Uniquely name your brokers. Use connection pooling
![Page 12: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/12.jpg)
12© 2016 Rogue Wave Software, Inc. All Rights Reserved.
12
Poll #3Do you use PostgreSQL?
A: We don’t use PostgreSQLB: We don’t use PostgreSQL but plan on using it in the future
C: We use PostgreSQL as a mission-critical application databaseD: We use PostgreSQL as part of another open source project
E: We are migrating away from PostgreSQL
![Page 13: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/13.jpg)
13© 2016 Rogue Wave Software, Inc. All Rights Reserved.
13
Top issues:databases
![Page 14: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/14.jpg)
14© 2016 Rogue Wave Software, Inc. All Rights Reserved.
14
Scenario #2
Database performance is poor
Symptoms• Sorting and querying take a long time• One particular query or web page is hanging the
database• “Sorry, too many clients already” or connection pool
is full
Where do I start?• Localize if possible: Data center, network,
database or application server?• When did it start• What changes took place?
![Page 15: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/15.jpg)
15© 2016 Rogue Wave Software, Inc. All Rights Reserved.
15
Scenario #2 solution
Identify and analyzeQuick checks• top with ‘c’ command shows process ID, CPU, and
memory utilization• ”iostat –x –m 5” reveals disk IO wait times• ELK = Elasticsearch + Logstash + Kibana• “EXPLAIN ANALYZE <SQL>” shows execution time and
table scans
Tools• psql: SELECT pid, datname, usename, query FROM pg_stat_activity;
pid datname usename query42102 jboss jboss SELECT pid, datname, usename,
query FROM pg_stat_activity;
42103 jboss jboss SELECT video FROM news where...;
![Page 16: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/16.jpg)
16© 2016 Rogue Wave Software, Inc. All Rights Reserved.
16
Scenario #2 solution
Identify and analyze
Tools• pg_stat_statmentswww.postgresql.org/docs/current/static/pgstatstatements.html
Note: pg_stat_statements requires more shared memory
postgresql.conf:shared_preload_libraries = ‘pg_stat_statements’
Server restart is required after enabling the shared library
Reset statistics: select pg_stat_reset();
![Page 17: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/17.jpg)
17© 2016 Rogue Wave Software, Inc. All Rights Reserved.
17
Scenario #2 solution
Identify and analyze
ToolsMore on PostgreSQL performance: www.craigkerstiens.com/2013/01/10/more-on-postgres-performance/
SELECT (total_time / 1000 / 60) as total_minutes, (total_time/calls) as average_time, query FROM pg_stat_statements ORDER BY 1 DESC LIMIT 100;
The QueryTotal Query Time (in minutes)Average Time (in milliseconds)
![Page 18: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/18.jpg)
18© 2016 Rogue Wave Software, Inc. All Rights Reserved.
18
Scenario #2 solution
Identify and analyze
Tools• pbBadger: dalibo.github.io/pgbadger/
• Requires Perl
pgBadger: dalibo.github.io/pgbadger/postgresql.conf• log_min_duration_statement = 0• log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d ’• log_checkpoints = on• log_connections = on• log_disconnections = on• log_lock_waits = on• log_temp_files = 0After making changes run: SELECT pg_reload_conf();
![Page 19: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/19.jpg)
19© 2016 Rogue Wave Software, Inc. All Rights Reserved.
19
Scenario #2 solution
Implement and test (…and repeat...)
Supporting points• What if I don’t find the offending SQL or table? • Start looking at the database configuration.• Did you benchmark and tune?
More points• Most parameters are automatically adjusted.• We find that many people make minimal changes to the default
configuration. • As always, remember to test any configuration changes in a non-
prod environment first, and implement changes incrementally. • Do not make numerous configuration changes all at once. • Systematically measure your performance tests. Use a tool like
pgbench.
![Page 20: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/20.jpg)
20© 2016 Rogue Wave Software, Inc. All Rights Reserved.
20
Scenario #2 solution
Implement and test (…and repeat...)
Configuration Settings in postgresql.conf• max_connections: Maximum simultaneous connections to
the database (default is 100).• shared_buffers: database cache size (default is 128MB).
25% of total RAM. Windows-based should be 64-512MB.• effective_cache_size: Tells the query planner how much
RAM there is to execute. 50-75% of total RAM.• work_mem: Used for complex sorts (default is 4MB). Check if
it is uncommented and what the value is.• maintenance_work_mem: Amount of memory for background
processes for pgdump, pgrestore, vacuum, indexing, and bulk loads (default is 64MB). 256MB-1GB for large databases.
• checkpoint_segments: Maxiumum # of log file segments between WAL checkpoints (default is 3).
![Page 21: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/21.jpg)
21© 2016 Rogue Wave Software, Inc. All Rights Reserved.
21
Scenario #2 solution
Implement and test (…and repeat...)
Configuration Settings in postgresql.conf• wal_buffers: Write ahead log buffer used for writing a
transaction to disk.
What should I really set these values to?• pgTune: pgtune.leopard.in.ua/
SELECT name, current_setting(name), SOURCE FROM pg_settings WHERE SOURCE NOT IN ('default', 'override'); PostgreSQL 9.0 High Performancewww.amazon.com/PostgreSQL-High-Performance-Gregory-Smith/dp/184951030X/163-3733534-8577963
![Page 22: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/22.jpg)
22© 2016 Rogue Wave Software, Inc. All Rights Reserved.
22
Poll #4Are you using SSLv3?
A: YesB: No
![Page 23: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/23.jpg)
23© 2016 Rogue Wave Software, Inc. All Rights Reserved.
23
Top issues:security
![Page 24: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/24.jpg)
24© 2016 Rogue Wave Software, Inc. All Rights Reserved.
24
Scenario #3
Pen testing reveals the application server is vulnerable
Supporting points• Cache-control, Pragma, Expires headers are not
set• X-Powered-By reveals the application server
typeMore points• A proxy can inject an intermediate
page and compromise the site• Certain versions of JBoss allow
little to no control of request header manipulation
![Page 25: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/25.jpg)
25© 2016 Rogue Wave Software, Inc. All Rights Reserved.
25
Scenario #3 solution
Request header manipulation strategiesSupporting points• Proxy requests through Apache HTTP Server before
sending them to the application server• Mod_expire could be used to explicitly set them. Does
not guarantee header will be set• Set org.apache.catalina.connector.X_POWERED_BY to
falseMore points• Using Apache HTTP Server’s mod_header module is easiest and
preferred option• mod_headers module allows manipulation before and after the
request• If this is not an option, then a filter can do the same function• There should never be a situation where an application server is
internet facing without the protection of a web server in the DMZ
![Page 26: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/26.jpg)
26© 2016 Rogue Wave Software, Inc. All Rights Reserved.
26
Scenario #3 solution
Request header manipulation strategiesDMZ• The front end should always be in the DMZ• This should never be an application server • Web servers are “far more” capable than
application servers in this role
Best practice• Traditional 3-headed monster, Web/App/DB should always
reside at unique layers in the infrastructure• There should be horizontal and/or vertical separation between
them at all times• This provides greater opportunities for other security gear to
potentially sniff and sort out other possible vulnerabilities. Like HIDS/NIDS, etc.
![Page 27: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/27.jpg)
27© 2016 Rogue Wave Software, Inc. All Rights Reserved.
27
Scenario #3 solution
Request header manipulation strategiesHTTPD Mod_expires• Mod_expires can set the max_age and
expirations• ExpiresByType text/html M604800 (Expires a
week after modification)HTTPD Mod_Header• Much more flexible than mod_expires• Header echo ^KC (copies all request headers starting with KS
to response headers• Header set TESTHEADER “Hi, The request ran in %D uSEC’s”• Env variables can be set as well. Much more capable than
mod_expires
![Page 28: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/28.jpg)
28© 2016 Rogue Wave Software, Inc. All Rights Reserved.
28
Scenario #3 solution
Header manipulation examplesHTTPD Mod_header Example• <ifModule mod)headers.c>• Header set DateTimeHEader “%D %t”• </ifModule>
Header merge Cache-Control no-cacheHeader merge Cache-Control no-store
YieldsCache-Control: no-cache, no-store
![Page 29: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/29.jpg)
29© 2016 Rogue Wave Software, Inc. All Rights Reserved.
29
Scenario #3 solution
Header manipulation examplesHTTPD Mod_header Example• Header set Set-Cookie testcookie “expr=-z %
{req:Cookie}”• Header merge Cache-Control no-cache env=CGI• Header merge Cache-Control no cache
end=NO_CACHE• Head merge Cache-Control not-store
env=NO_STORE
Non-exclusive conditions. If all are set: Cache-Control: no-cache, no-store
Using append instead of merge would cause a duplicate no-cache message
![Page 30: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/30.jpg)
30© 2016 Rogue Wave Software, Inc. All Rights Reserved.
30
Scenario #3 solution
Header manipulation examplesHTTPD Mod_header (Powerful module)• Add, Append, echo, edit, merge, set setifempty,
unset• Allows setting of environment variables to use
are triggers• Expressions can be set as well for more
additional control
Examples prove that web servers are well suited for this type of work.
![Page 31: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/31.jpg)
31© 2016 Rogue Wave Software, Inc. All Rights Reserved.
31
Scenario #3 solution
Request header manipulation strategiesJBoss Wildfly : Header Manipulation• You shouldn’t, but if you absolutely must• You could modify system properties to override
certain values. Or set a filter.• Newer versions of JBoss provide header
modificationExample• <system properties>• <property
name=“org.apache.coyote.http11.Http11Protocol.SERVER” value=“someserver”/>
• <system-properties>• For each version of Jboss, you may need to use the CLI to
determine if the necessary keys are available in that version
![Page 32: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/32.jpg)
32© 2016 Rogue Wave Software, Inc. All Rights Reserved.
32
Scenario #3 solution
Request header manipulation strategies
JBoss Wildfly : Header Manipulation• You can, but apache httpd is still a good accomplice• Remove the following the alter the headers• <filter-ref name=“x-powered-by-header”/>• Performed in the undertow segment of
standalone.xml
Example• As newer versions of Wildfly emerge there is a bit more control
of headers• This does not remove the need to have a web server in front• A reverse proxy is is vital piece of application server security
![Page 33: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/33.jpg)
33© 2016 Rogue Wave Software, Inc. All Rights Reserved.
33
Scenario #3 solution
Request header manipulation strategies
JBoss Wildfly : Header Manipulation• CLI command structure will provide a view of current
header manipulation capabilities Wildfly has• More flexibility in each new release• Wildfly’s capabilities are far behind that of apache’s• Having this level of control at your reverse proxy just
makes the most sense• These features have been requested for quite some
time from the user base.• Not a replacement for a frontend DMZ based web
server
![Page 34: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/34.jpg)
34© 2016 Rogue Wave Software, Inc. All Rights Reserved.
34
Who wrote LevelDB and what version of ActiveMQ did it debut in?Written by Google: Jeff Dean who also has contributed to MapReduce and Google TranslateSanjay Ghemawat who also contributed to MapReduce and iCal which is a popular calendar application in Unix/Linux.
LevelDB debuted in ActiveMQ 5.10.0 but was not production ready.What percentage of web server market share does Apache HTTP Server hold?July 2016Apache HTTP Server: 52%nginx: 30.5%Microsoft: 12%
Pop quiz answers
![Page 35: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/35.jpg)
35© 2016 Rogue Wave Software, Inc. All Rights Reserved.
35
Conclusion
![Page 36: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/36.jpg)
36© 2016 Rogue Wave Software, Inc. All Rights Reserved.
36
JBoss/Wildfly: Newer versions of Wildfly allow administrators to control HTTP headers and older versions of JBoss can be augmented with a filter or Apache HTTP Server's mod_header for equivalent functionality.
So much open source!
PostgreSQL: Use tools like pgBench and pgTune.
PostgreSQL: Troubleshooting tools like pg_stat_statements and pgBadger can help narrow down database performance issues.
ActiveMQ: A Lease Database Locker is a viable alternative to NFS where SAN is cost prohibitive. Consider Replicated LevelDB with Apache Zookeeper for performance.
![Page 37: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/37.jpg)
37© 2016 Rogue Wave Software, Inc. All Rights Reserved.
37
Our support
![Page 38: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/38.jpg)
38© 2016 Rogue Wave Software, Inc. All Rights Reserved.
38
Q & A
![Page 39: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/39.jpg)
39© 2016 Rogue Wave Software, Inc. All Rights Reserved.
39
Watch on demand
• Watch this webinar on demand
• Read the recap blog to see the results of the polls and Q&A session
![Page 40: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/40.jpg)
40© 2016 Rogue Wave Software, Inc. All Rights Reserved.
40
Follow up
For OpenLogic support customers:
OSS Radio
Get a free OSS support ticket to experience our expertise
roguewave.com/freeticket
Free open source newsletter:
roguewave.com/products/open-source-support/openupdate
![Page 41: Open source: Top issues in the top enterprise packages](https://reader036.fdocuments.net/reader036/viewer/2022062522/5882b6661a28abd75a8b7431/html5/thumbnails/41.jpg)
41© 2016 Rogue Wave Software, Inc. All Rights Reserved.
41