Linux container performance on HPE ProLiant servers
Transcript of Linux container performance on HPE ProLiant servers
Linux container performance on HPE ProLiant servers Understanding performance differences between containers and virtual machines
Technical white paper
Technical white paper
Introduction ....................................................................................................................................................................................................................................................................................................................................................3 Overview of containers and virtual machines .................................................................................................................................................................................................................................................................3 Overview of testing objectives ..................................................................................................................................................................................................................................................................................................... 4 Complex business applications ................................................................................................................................................................................................................................................................................................... 4
Java-based containers and VMs.......................................................................................................................................................................................................................................................................................... 4 Performance ........................................................................................................................................................................................................................................................................................................................................................ 6 Processor usage ............................................................................................................................................................................................................................................................................................................................................... 7 Memory usage ................................................................................................................................................................................................................................................................................................................................................... 8 Summary ................................................................................................................................................................................................................................................................................................................................................................. 8
Distributed environment ............................................................................................................................................................................................................................................................................................................. 9 Summary .............................................................................................................................................................................................................................................................................................................................................................. 10
Web-based applications ............................................................................................................................................................................................................................................................................................................10 Hard reboot ....................................................................................................................................................................................................................................................................................................................................................... 10 Soft Reboot ......................................................................................................................................................................................................................................................................................................................................................... 13 Summary ............................................................................................................................................................................................................................................................................................................................................................... 15
Pure performance .................................................................................................................................................................................................................................................................................................................................. 16 Storage I/O performance (Sysbench mysql) .......................................................................................................................................................................................................................................................... 16 Floating point computing performance (LINPACK) ........................................................................................................................................................................................................................................ 18 Memory performance (Stream) .......................................................................................................................................................................................................................................................................................... 19 Storage I/O performance (FIO tester) .......................................................................................................................................................................................................................................................................... 19 Network performance ................................................................................................................................................................................................................................................................................................................. 20 Multiuser application environment ................................................................................................................................................................................................................................................................................ 23
Conclusion ................................................................................................................................................................................................................................................................................................................................................... 24 Key takeaways ................................................................................................................................................................................................................................................................................................................................... 25 Resources, contacts, or additional links ...................................................................................................................................................................................................................................................................... 26
Technical white paper Page 3
Introduction When planning your strategy for implementing a solution stack, isolating applications and processes can be a way to create an environment that supports your applications with the best performance.
In a Linux environment, applications and processes can be isolated using either containers or virtual machines (VMs). Containers can yield the best performance; however, they might not be ideal for every solution stack. There are solution stacks where VMs provide advantages and are a better fit than containers. For example, some solution stacks include multi-tenant applications and do not need isolation of any kind—either container or VM; or, a solution stack may have applications that are operating system (OS) dependent and require VMs instead of containers.
It is not the intention of this paper to endorse either containers or VMs. Your solution stack requirements determine the need to separate and isolate an application natively, in a container, or in a VM. This paper provides general information about container technologies from a performance perspective and concludes with some general advice for implementing containers.
For a detailed overview of container technology, its history, how it works, and its benefits and shortcomings, see the Linux container technology technical white paper.
Overview of containers and virtual machines Figure 1 illustrates the basic structure of VMs and containers.
As illustrated in the figure, VMs create a virtualization environment that uses the Linux kernel as a hypervisor. The Linux kernel manages the processor and memory resources by emulating a machine interface for each VM guest OS kernel, initialization system, library, and application(s).
Instead of using a hypervisor to manage VMs, the figure shows how containers isolate applications into separate environments (containers) that include processor, memory, and networking resources as part of the container itself. This environment provides OS-level virtualization. Containers have their own root; and, users and processes do not perform operations outside of the container environment. The host OS kernel manages container workloads directly, which reduces the overhead involved with managing system resources. This improves efficiency and therefore, improves performance.
For more details about container technology, its history, how it works, and its benefits and shortcomings, see the Linux container technology technical white paper.
Figure 1. VM and container comparison
Technical white paper Page 4
Overview of testing objectives Our testing objectives were to understand the scalability of complex business applications and to understand the pure subsystem performance for containers and VMs. We performed benchmark testing that encompassed the following configurations. For each configuration, we ran different workloads to gather the comparative information on resource usage for processor, memory, networking, and storage.
• A bare-metal HPE ProLiant server configured with Red Hat Enterprise Linux (RHEL)
• An HPE ProLiant server configured with RHEL, RHEL Atomic Host, or Red Hat Enterprise Virtualization (RHEV) and applications isolated in Docker containers. Docker is the platform that we used for creating and managing containers.
• An HPE ProLiant server configured with RHEL, RHEL Atomic Host, or RHEV and applications isolated in kernel-based virtual machines (KVMs). We used KVMs for creating the virtualization infrastructure.
Complex business applications To gather information about the relative scalability and performance for complex business applications, we focused on the following environments:
• Java-based containers and VMs
• Distributed environments
• Web-based applications
To understand processor usage and memory usage within these environments, we simulated Java-based applications running in a database online transaction processing (OLTP) workload. We focused on Java-based applications because they are prevalent in these environments.
To understand capacity and performance within these environments, we looked at the following capacity measurements in transitional workloads:
• Performance and deployment speed in a virtualization workload test
• Cloud deployment and performance using an OpenStack-based workload
Java-based containers and VMs For our benchmarks that simulated a scalable, Java-based environment, we captured the following performance values. The number of operations were the same for all workloads.
• Pure throughput: The number of transactions completed during the test.
• Time-critical throughput: The number of required transactions that completed during the test.
Technical white paper Page 5
The results (illustrated in Figure 2 and Figure 3) show that the workloads running in containers run 3% to 17% more Java operations per second (JOPS) than the workloads running in a comparable VM configuration, with the four-container configuration showing a noticeable advantage.
Figure 2. Comparison of two Docker containers and two KVMs (ProLiant DL380p Gen8 with RHEL 7.1)
Figure 3. Comparison of four Docker containers ad four KVMs (ProLiant DL380p Gen8 with RHEL 7.1)
3769
14489
2794
14167
3378
13684
3865
14328
3973
14473
0
2000
4000
6000
8000
10000
12000
14000
16000
Time-critical throughtput Pure throughput
Java
ope
ratio
ns p
er s
econ
d (J
OP
S)
RHEL 7.1 on a bare-metal server KVM on RHEL 7.1 KVM on RHEV 6.6 Docker container on RHEL 7.1 Docker container on RHEL Atomic Host 7.1
28469
75956
18438
61031
18932
65011
27725
75139
28213
75425
0
10000
20000
30000
40000
50000
60000
70000
80000
Time-critical throughtput Pure throughput
Java
ope
ratio
ns p
er s
econ
d (J
OP
S)
RHEL 7.1 on a bare-metal server KVM on RHEL 7.1 KVM on RHEV 6.6 Docker container on RHEL 7.1 Docker container on RHEL Atomic Host 7.1
Technical white paper Page 6
Performance Our benchmark data for pure throughput and time-critical throughput showed that containers performed better than VMs. Figure 4 through Figure 9 illustrate the following comparison summary:
• Comparing one container to one VM:
– There was a 9% performance improvement with the container.
– There was a 6% reduction in processor use with the container.
– There was a 3% increase in memory use with the container.
• Comparing four containers to four VMs:
– There was a 12% performance improvement (throughput) with the containers (see Figure 4).
– There was a 12% reduction in processor use with the containers (see Figure 6).
– There was a 4% increase in memory use with the containers (see Figure 8).
• Comparing ten containers to ten VMs:
– There was a 17% performance improvement (throughput) with the containers (see Figure 5).
– There was a 17% reduction in processor use with the containers (see Figure 7).
– There was a 6% increase in memory use with the containers (see Figure 9).
Figure 4. Comparison of four Docker containers and four KVMs (ProLiant DL380p Gen8 with RHEL 7.1)
Figure 5. Comparison of ten Docker containers and ten KVMs (ProLiant DL380p Gen8 with RHEL 7.1)
2709.5
7512.5
3033.75
8255.5
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
Time-critical throughput Pure throughput
Java
ope
ratio
ns p
er s
econ
d (J
OP
S)
KVM average Docker container average
528.80
2517.70
579.98
3359.85
0
500
1000
1500
2000
2500
3000
3500
4000
Time-critical throughput Pure throughput
Java
ope
ratio
ns p
er s
econ
d (J
OP
S)
KVM average Docker container average
Technical white paper Page 7
Processor usage
Figure 6. Comparison of four Docker containers and four KVMs (ProLiant DL380p Gen8 with RHEL 7.1)
Figure 7. Comparison of ten Docker containers and ten KVMs (ProLiant DL380p Gen8 with RHEL 7.1)
0
20
40
60
80
100
120
1 11 21 31 41 51 61 71 81 91 101
111
121
131
141
151
161
171
181
191
201
211
221
231
241
251
261
271
281
291
301
311
321
331
341
351
361
371
381
391
401
411
421
431
441
451
461
471
481
491
Pro
cess
or u
se (p
erce
nt)
Monitoring interval
KVM Docker container
0102030405060708090
1 13 25 37 49 61 73 85 97 109
121
133
145
157
169
181
193
205
217
229
241
253
265
277
289
301
313
325
337
349
361
373
385
397
409
421
433
445
457
469
481
493
505
517
529
541
553
565
577
589
Pro
cess
or u
se (p
erce
nt)
Monitoring interval
KVM Docker container
Technical white paper Page 8
Memory usage
Figure 8. Comparison of four Docker containers and four KVMs (ProLiant DL380p Gen8 with RHEL 7.1)
Figure 9. Comparison of ten Docker containers and ten KVMs (ProLiant DL380p Gen8 with RHEL 7.1)
Summary Our testing shows that a Java-based business workload performs better on containers than on VMs.
VMs generally use more processor and memory because each VM has its own kernel. Each VM is overhead for the host because every kernel requires memory and processor resources.
Containers use control groups (cgroups) to allocate resources, including processor and memory, which allows a container to occupy more memory temporarily as needed. To get higher performance, a container might occupy around 3% more memory during a heavy workload; however, it releases that extra memory after the workload finishes. A VM uses the /dev/kvm module to isolate memory; therefore, a VM does not require more memory to handle different workloads but does require more memory overall because of the kernel.
0
20
40
60
80
100
120
140
1 11 21 31 41 51 61 71 81 91 101
111
121
131
141
151
161
171
181
191
201
211
221
231
241
251
261
271
281
291
301
311
321
331
341
351
361
371
381
391
401
411
421
431
441
451
461
471
481
491
Mem
ory
use
(GB
)
Monitoring interval
KVM Container
020406080
100120
1 13 25 37 49 61 73 85 97 109
121
133
145
157
169
181
193
205
217
229
241
253
265
277
289
301
313
325
337
349
361
373
385
397
409
421
433
445
457
469
481
493
505
517
529
541
553
565
577
589Mem
ory
use
(GB
)
Monitoring interval
KVM Container
Technical white paper Page 9
Distributed environment Our distributed environment evaluation simulates a scalable, distributed environment that includes extensive communication among containers, which is a common characteristic of microservices-based and business-to-business applications. For the test platform, we used a ProLiant DL580 Gen8 server running RHEL 7.1.
To show scaling, we consolidated multiple servers running different applications onto a single system with multiple VMs or containers until the overall throughput peaked. This scenario models the multi-tiered applications and database environments that require enterprise-class VMs based on real-world traffic demanded from a web server, including the following:
• Java application server
• Internet Message Access Protocol (IMAP) server
• Batch server
As the number of servers increases from one to three, performance (requests per second) is comparable between containers and VMs (see Figure 10). Although performance is similar, resource consumption declines with containers compared to VMs (see Figure 11). Resource consumption increases with VMs because each VM has a kernel, which requires its own processor and memory resources. Because the host kernel manages container resources directly, the container resource consumption declines as the number of servers increases.
Figure 10. Comparison of Docker containers and KVMs (ProLiant DL580 Gen8 with RHEL 7.1)
Figure 11. Comparison of Docker containers to KVMs (ProLiant DL580 Gen8 with RHEL 7.1)
94.93
188.3
269.2
94.68
188.7
270.3
0
50
100
150
200
250
300
1 Server 2 Servers 3 Servers
Req
uest
s pe
r sec
ond
KVM Docker container
-43% -46%-40%
-34%
-61%
-39%
-80%
-60%
-40%
-20%
0%
Processor use Memory use
Per
cent
1 Tile 2 Tiles 3 Tiles
Technical white paper Page 10
Summary Regardless of the workload, the resource consumption of containers is markedly less than VMs. Each VM has overhead because each VM’s kernel requires processing resources and up to 1 GB of memory. The resources that the VM kernel requires is most apparent when a system is fully loaded or when there are many VMs. The OS kernel manages containers directly, so containers do not have the extra overhead.
Web-based applications To evaluate how OpenStack works at scale, we simulated scalable, web-based applications running in an Openstack framework. We achieved this by automating and unifying multi-node OpenStack deployments, cloud verification, and profiling. We tested two scenarios that reflect impact of application service start up:
• Hard reboot, which power cycles the instance.
• Soft reboot, which performs a graceful shut down and restart of the instance.
For the test platform, we used a ProLiant DL580 Gen8 server running Ubuntu Server 14.04.
Hard reboot This test showed that containers provide better performance to boot the server or to perform delete functions. When compared to a VM, the container performed as follows (see Figure 12):
• The average boot time for a container is 28.21% faster.
• The average delete time for a container is 34.35% faster.
• The average total time for a container is 30.57% faster.
4.338
2.473
6.8126.043
3.767
9.811
0
2
4
6
8
10
12
Server boot time (average) Server delete time (average) Total time (average)
Tim
e (s
econ
ds)
Docker container KVM
Technical white paper Page 11
Figure 12. Comparison of Docker container and KVM (ProLiant DL580 Gen8 with Ubuntu Server 14.04)
As illustrated in Figure 13 and Figure 14, containers run at a more stable, steady state, utilizing less peak processor bandwidth than VMs. When rebooting a container, the processor resource consumption is more stable because the container only needs to stop and restart a process. By comparison, a VM must go through the entire reboot process.
Figure 13. Docker container processor use (ProLiant DL580 Gen8 with Ubuntu Server 14.04)
Figure 14. KVM processor use (ProLiant DL580 Gen8 with Ubuntu Server 14.04)
0123456789
10
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75
Pro
cess
or u
se (p
erce
nt)
Time (seconds)
User System
0
2
4
6
8
10
1 15 29 43 57 71 85 99 113
127
141
155
169
183
197
211
225
239
253
267
281
295
309
323
337
351
365
379
393
407
421
435
449
463
477
491
505
519
533
547
561
575
589
603
617
631
645P
roce
ssor
use
(per
cent
)
Time (seconds)usr sys
Technical white paper Page 12
As illustrated in Figure 15 and Figure 16, the characteristics of containers and VMs result in better memory use with containers.
Figure 15. Docker container memory use (ProLiant DL580 Gen8 with Ubuntu Server 14.04)
Figure 16. KVM memory use (ProLiant DL580 Gen8 with Ubuntu Server 14.04)
0
1000
2000
3000
4000
5000
6000
7000
8000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37
Mem
ory
use
(MB
)
Time (seconds)
010002000300040005000600070008000
1 15 29 43 57 71 85 99 113
127
141
155
169
183
197
211
225
239
253
267
281
295
309
323
337
351
365
379
393
407
421
435
449
463
477
491
505
519
533
547
561
575
589
603
617
631
645
Mem
ory
use
(MB
)
Time (seconds)
Technical white paper Page 13
Soft Reboot Our testing shows that containers provide better performance during a soft reboot, start, and stop. When compared to a VM, the container performed as follows (see Figure 17):
• The average soft reboot time for a container is 96.65% faster.
• The average start time for a container is 45.99% faster.
• The average stop time for a container is 96.13% faster.
• The average total time for a container is 90.76% faster.
Figure 17. Container and VM soft reboot, start time, and stop time comparison (ProLiant DL580 Gen8 with Ubuntu Server 14.04).
4.209 1.623 2.493
19.008
125.463
3.005
64.422
205.789
0
50
100
150
200
250
Soft reboot time (average) Start time (average) Stop time (average) Total time (average)
Tim
e (s
econ
ds)
Docker container KVM
Technical white paper Page 14
As illustrated in Figure 18 and Figure 19, containers run at a more stable, steady state, utilizing less peak processor bandwidth than VMs. When rebooting a container, the processor resource consumption is more stable because the container only needs to stop and restart a process. By comparison, a VM must go through the entire reboot process.
Figure 18. Docker container Compute Node processor use (ProLiant DL580 Gen8 with Ubuntu Server 14.04)
0
1
2
3
4
5
6
7
8
9
10
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75
Pro
cess
or u
se (p
erce
nt)
Time (seconds)
User System
0123456789
10
1 15 29 43 57 71 85 99 113
127
141
155
169
183
197
211
225
239
253
267
281
295
309
323
337
351
365
379
393
407
421
435
449
463
477
491
505
519
533
547
561
575
589
603
617
631
645P
roce
ssor
use
(per
cent
)
Time (seconds)
User System
Technical white paper Page 15
Figure 19. KVM Compute Node processor use (ProLiant DL580 Gen8 with Ubuntu Server 14.04)
As illustrated in Figure 20 and Figure 21, the characteristics of containers and VMs result in better memory use with containers.
Figure 20. Docker container Compute Node memory use (ProLiant DL580 Gen8 with Ubuntu Server 14.04)
Figure 21. KVM Compute Node memory use (ProLiant DL580 Gen8 with Ubuntu Server 14.04)
Summary Containers perform better and consume fewer resources when booting. The container boot process involves starting a process in a cgroup. Without initializing emulated hardware, as a VM must do, the container allocates resources by cgroup and namespace and then mounts the filesystem.
Booting a VM simulates a separate system, which requires more time and processor consumption. Additionally, because each VM has a kernel, more memory resources are required.
0
1000
2000
3000
4000
5000
6000
7000
8000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73
Mem
ory
use
(MB
)
Time (seconds)
010002000300040005000600070008000
1 15 29 43 57 71 85 99 113
127
141
155
169
183
197
211
225
239
253
267
281
295
309
323
337
351
365
379
393
407
421
435
449
463
477
491
505
519
533
547
561
575
589
603
617
631
645
Mem
ory
use
(MB
)
Time (seconds)
Technical white paper Page 16
Pure performance To measure performance in specific intensive situations, we tested intensive processor, memory, file system, and networking workloads comparing containers and VMs. To understand the pure performance characteristics of containers and VMs, we tested utilization of the following subsystems:
• Processor and memory performance (using the Sysbench CPU/memory benchmark)
• Floating point computing performance (using the Linear Algebra PACKage (LINPACK) benchmark)
• Memory performance (using the Stream (Sustainable Memory Bandwidth in High Performance Computers) benchmark)
• Storage I/O performance (using the Flexible I/O (FIO) tester and Sysbench mysql benchmarks)
• Network performance (using the Netperf benchmark)
• Multiuser application environment (using the AIM7 benchmark)
Storage I/O performance (Sysbench mysql) Figure 22 shows results for storage I/O comparisons in an OLTP application environment running on MySQL, as follows:
• For read and write transactions per second for a container volume, which is a partition that is formatted into a file system is 37% to 86% higher than VM virtual I/O (VirtIO), which is a partition space of a disk.
• The maximum gap is 86% for 25 threads.
Figure 22. Results for OLTP application environment running on top of MySQL Server (ProLiant DL380p Gen8 Server with RHEL 7.1)
5 10 25 32 50 100 150KVM (virtual I/O) 1460.66 2210.16 2422.76 2596.08 2950.34 3162 3264.1KVM (virtual I/O and dedicate disk) 1484.8 2188.58 2334.27 2520.12 2842.62 3146.7 3050.6Docker container (device mapper) 2155.75 3686.52 4517.3 4686.49 4727.64 4642.68 4464.37Docker container (volume) 2451.42 4244.8 5177.24 5179 5043.17 5028.2 4116.72
0
1000
2000
3000
4000
5000
6000
Tran
sact
ions
opr
sec
ond
Thread
Technical white paper Page 17
Processor and memory performance (Sysbench CPU/memory)
Figure 23 and Figure 24 show results for processor and memory performance comparisons in VMs/containers, which are as follows:
• For one container, the processor testing (total time) is 904.7625.
• For 30 containers the average time (total time) is 1089.704.
Figure 23. Processor test results (ProLiant BL460c Gen8 Server with RHEL 7.1
Figure 24. Memory test results (ProLiant BL460c Gen8 Server with RHEL 7.1)
3800
4000
4200
4400
4600
4800
5000
5200
5400
5600
5800
0 3 6 9 12 15 18 21 24 27 30
Tim
e (s
econ
ds)
Number of Docker containers and KVMs
VM
Container
200
400
600
800
1000
1200
1400
1600
1800
2000
0 3 6 9 12 15 18 21 24 27 30
Tim
e (s
econ
ds)
Number of Docker containers and KVMs
VM
Container
Technical white paper Page 18
Floating point computing performance (LINPACK) To measure floating-point computing power, we evaluated a ProLiant DL580 Gen8 server running RHEL 7.1 and RHEL Atomic Host. Containers showed better performance (see Figure 25):
• With a floating point problem size of 79k, containers performed as follows:
– Containers have the same performance with VMs on RHEL 7.1
– Containers performance on RHEL Atomic Host improves by 7.8% compare to containers on RHEL7.1
• With a floating point problem size of 39k, containers performed as follows:
– Containers perform 1.6% higher than VMs on RHEL 7.1
– Containers performance on RHEL Atomic Host improves by 7% compare VMs on RHEV
Figure 25. Results (ProLiant DL580 Gen8 with RHEL 7.1)
1.88 1.871.87 1.841.87 1.862.03 2.00
1.89 1.86
0.00
0.50
1.00
1.50
2.00
2.50
79k 39k
Gig
a Fl
oatin
g P
oint
Ope
ratio
ns p
er
Sec
ond
(GFL
OP
S)
Native KVM Docker container on Native Docker container on RHEL Atomic Host KVM on RHEV
Technical white paper Page 19
Memory performance (Stream) The platform used for our memory evaluation was a ProLiant DL580 Gen8 server running RHEL 7.1. As illustrated in Figure 26, we found comparable results for memory use between containers and VMs.
Figure 26. Memory performance results (ProLiant DL580 Gen8 with RHEL 7.1)
Storage I/O performance (FIO tester) The platform that we used for our storage I/O evaluation was a ProLiant DL580 Gen8 server running RHEL 7.1. As illustrated in Figure 27, we found comparable storage use between containers and VMs.
Figure 27. Storage I/O Reads (ProLiant DL580 Gen8 with RHEL 7.1)
4972
7
4860
5
5462
0
5430
7
4911
1
4926
1
5489
8
5543
3
4666
0
4668
1 5215
3
5234
0
4916
8
4871
1
5404
4
5410
6
4936
3
4835
2
5415
8
5427
6
0
10000
20000
30000
40000
50000
60000
Copy Scale Add Triad
Rat
e (M
B/s
)
RHEL 7.1 bare metal VM on RHEL 7.1 VM on RHEV 6.6 Container on RHEL 7.1 Container on RHEL Atomic Host 7.1
3910
14
2443
8
3845
10
2403
2
4025
94
2516
2
4132
05
2582
5
4082
16
2551
3
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
Bandwidth (KB/s) Input/output per second (IOPS)
RHEL 7.1 KVM Docker container on RHEL 7.1 Docker container on RHEL Atomic Host 7.1 KVM on RHEV 6.6
Technical white paper Page 20
As illustrated in Figure 28, the results show that the bandwidth (KB/s) of containers on RHEL 7.1 is 4.5% higher than VMs on RHEL 7.1. Regarding the IOPS, we found container performance on RHEL Atomic Host comparable to RHEL 7.1.
Figure 28. FIO tester write results (ProLiant DL580 Gen8 with RHEL 7.1)
Network performance As illustrated in Figure 29 and Figure 30, container performance is similar to VM performance in the TCP Stream test, but in the TCP RR test, containers perform much better:
• Transmission Control Protocol (TCP) Stream (10^6 bits/sec):
– Container performance on RHEL 7.1 is 6% lower than VM performance on RHEL 7.1.
– Container performance on RHEL Atomic Host is 1.7% higher than VM performance on RHEL 7.1.
• TCP Request/Response (TCP RR):
– Container performance on RHEL 7.1 is 27.1% better than VM performance on RHEL 7.1.
– Container on RHEL Atomic Host performance is 23.4% better than VM performance on RHEL 7.1.
3910
14
2443
8
3845
10
2403
2
4025
94
2516
2
4132
05
2582
5
4082
16
2551
3
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
Bandwidth (KB/s) Input/output per second (IOPS)
RHEL 7.1 KVM Docker container on RHEL 7.1 Docker container on RHEL Atomic Host 7.1 KVM on RHEV 6.6
Technical white paper Page 21
Figure 29. TCP Stream network performance (ProLiant DL580 Gen8 with RHEL 7.1)
Figure 30. TCP RR network performance (ProLiant DL580 Gen8 with RHEL 7.1)
9347 91918639
9346 9105
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
Thro
ughp
ut (1
0^6b
its/s
ec)
RHEL 7.1 bare metal KVM on RHEL 7.1 Docker container on RHEL7.1 Docker container on RHEL Atomic host 7.1 KVM on RHEV 6.6
18876
11611
1593015158
12340
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
Tran
sact
ion
rate
(per
sec
ond)
RHEL 7.1 bare metal KVM on RHEL 7.1 Docker container on RHEL 7.1 Docker container on RHEL Atomic Host 7.1 KVM on RHEV 6.6
Technical white paper Page 22
Figure 31 and Figure 32 illustrate the performance of containers in bridge network mode running on RHEL Atomic Host. Our results show that the bridge network mode of containers on RHEL Atomic host is not very stable.
• TCP Stream (10^6 bits/sec):
– There is 141% variance between the MAX and the MIN transaction rate among the 10 tests.
• TCP RR:
– There is 39.03% variance between the MAX and the MIN transaction rate among the 10 tests.
Figure 31. TCP Stream throughput (ProLiant DL380 Gen8 with RHEL 7.1)
Figure 32. TCP RR transaction rates per second (ProLiant DL580 Gen8 with RHEL 7.1)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
1 2 3 4 5 6 7 8 9 10
Thro
ughp
ut (1
0^6
bits
per
sec
ond)
Transaction rate per second
0
5000
10000
15000
20000
25000
1 2 3 4 5 6 7 8 9 10
Thro
ughp
ut (p
acka
ges
per s
econ
d)
Transaction rate per second
Technical white paper Page 23
Multiuser application environment AIM7 is a throughput benchmark that exercises many aspects of the OS by using different workloads such as, disk-file operations, process creation, user virtual memory operations, pipe I/O, and compute-bound operations. The configuration of Docker containers and KVMs for this environment are as follows:
• KVM configuration:
– virtual CPU (vCPU) cores: 48 cores (vCPU=0–23, 48–71)
– Memory: 128 GB (m=128 g)
– Disk: VirtIO
• Docker container configuration
– Cpuset: 48 cores (cpuset=0–23, 48–71)
– Memory: 128 GB (m=128 g)
– Disk: Use default Docker storage driver: Device Mapper
– Data loop file: /var/lib/docker/devicemapper/devicemapper/data
– Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
As illustrated in Figure 33 and Figure 34, Sometimes container provide better performance and sometimes VM provide better performance:
Figure 33. AIM7 throughput (ProLiant DL580 Gen8 with RHEL 7.1)
0
100000
200000
300000
400000
500000
600000
new_dbase dbase custom shared alltests compute long all_utime high_systime
Jobs
per
min
ute
(JP
M)
RHEL 7.1 bare metal KVM on RHEL 7.1 Docker container on RHEL 7.1 Docker container on RHEL Atomic Host 7.1 KVM on RHEV 6.6
Technical white paper Page 24
Figure 34. Additional AIM7 workload results (ProLiant DL580 Gen8 with RHEL 7.1)
Conclusion Containers and VMs are both good technologies that provide an environment for hosting multiple applications on a given server. Containers can be a good choice for environments where there is homogeneity in the expectations among the hosted applications. VMs can be a better choice when stacking applications that require a mixed environment, including things like completely different OSes or significantly different OS versions.
Our testing shows that Containers usually have a performance edge, as long as the mix of apps is suitable to containerization. This performance edge comes usually from the lack of hypervisor overhead, elimination of duplicated OS services inside each VM and less waste of available memory. Our testing also shows that containers become more efficient as the number of containers increases. Figure 35 summarizes our pure performance test results. The figure illustrates that containers perform better than VMs in all of the performance testing. In most of the benchmark testing, we found that containers perform better with RHEL Atomic Host than with RHEL 7.1. The Netperf and Sysbench (mysql) results show that containers perform better than VMs because VMs use VirtIO to manage and transfer data from the VM to hardware, and then from hardware back to the VM.
From a resource perspective, VMs require more memory and processors than containers because each VM has its own kernel. Containers use OS-level virtualization and perform well with fewer processors and less memory. If the memory requirements exceed what is available in the server, both containers and VMs will continue to work; however, container performance remains better. If resource needs increase too much beyond the available system resources, some VMs will hang; containers will continue to work, but could become unstable.
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
4000000
new_fserver fserver five_sec disk
RHEL 7.1 bare metal KVM on RHEL 7.1 Docker container on RHEL 7.1 Docker container on RHEL Atomic Host 7.1 KVM on RHEV 6.6
Jobb
s p
er m
inut
e (J
PM
)
Technical white paper Page 25
Figure 35. Comparison of workload testing
Key takeaways As we were conducting our testing, we learned a few things that are useful to consider:
• When using Docker, do not use the container file system (loopback devices), which is the default file system. With this file system, the process for creating containers becomes progressively slower. Instead, use another storage option, such as `--storage-opt dm.thinpooldev`.
• If you have many Docker containers that do not require network access, use the container option “—net=none;” Unless you set this option, the network resources are assigned to Docker0, which decreases performance.
• Keep important or rapid read/write data outside of a container. A container’s file system has layers and the layers can decrease performance. For containers in a micro-service system, this means that the same image could replace the data and run on a new file system or on another machine. To avoid this issue, save the data to the host or to storage.
• Consider how you assign processor and memory resources for each container. Resource limits for each container may be required to make sure that containers do not interfere with each other’s resources.
• Allocate a bit more resources to the container than the solution stack requires. If you do not assign adequate resources, a Linux out-of-memory (OOM) killer could occur. OOM killer is a process that the Linux kernel uses when the system is critically low on memory.
• Granting privileges to a container could provide access to non-uniform memory access (NUMA) and result in the potential for improper configuration.
12%
3%
10% 8%
2% 2%5% 3%
37%
-5%
62%
2%
-10%
0%
10%
20%
30%
40%
50%
60%
70%
Docker container on RHEL 7.1 versus KVM on RHEL 7.1 Docker container on RHEL Atomic Host versus Docker container on RHEL 7.1Business Linpack Stream FIO Netperf Sysbench
Technical white paper Page 26
Sign up for updates
Rate this document
© Copyright 2016 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for HPE products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HPE shall not be liable for technical or editorial errors or omissions contained herein.
Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. Red Hat is a trademark of Red Hat, Inc. Linux is a registered trademark of Linus Torvalds.
4AA6-2761ENW, May, 2016
Resources, contacts, or additional links AIM benchmark sourceforge.net/projects/aimbench
Flexible I/O (FIO) tester linux.die.net/man/1/fio
Netlib Repository (includes Linear Algebra PACKage (LINPACK) benchmark information) netlib.org
Netperf Homepage netperf.org/netperf/NetperfPage.html
Rally benchmark wiki.openstack.org/wiki/Rally
STREAM: Sustainable Memory Bandwidth in High Performance Computers cs.virginia.edu/stream/
Sysbench manual imysql.com/wp-content/uploads/2014/10/Sysbench-manual.pdf
HPE Servers technical white papers hpe.com/docs/servertechnology