Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies...
Transcript of Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies...
![Page 1: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/1.jpg)
x
Exploring the New DPDK
Memory SubsystemANATOLY BURAKOV
BRUCE RICHARDSON
![Page 2: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/2.jpg)
2
DPDK Is No Longer As Greedy!
• DPDK can now allocate hugepage memory as needed
• DPDK can also release memory that is unused
• DPDK can put pages into fewer files
• Small page sizes and virtio are not enemies anymore!
• (18.08+) DPDK no longer requires a hugetlbfs mountpoint
![Page 3: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/3.jpg)
Looking Inside ANATOLY BURAKOV
![Page 4: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/4.jpg)
4
What Changed in 18.05?
Main design goal:
Ability to map/unmap hugepages at runtime, not just startup
Everything else is side effect and/or practical necessity!
![Page 5: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/5.jpg)
5
Memory Rework Design Principles
Question:
• How do you keep IOVA-contiguous memory without pre-sorting
pages?
Answer:
• You don’t!
• In 18.05, we deal with pages, not segments
• Memory is no longer guaranteed to be IOVA-contiguous
![Page 6: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/6.jpg)
6
Memory Rework Design Principles
Question:
• What if you need IOVA-contiguous memory?
Answer:
• Chances are, you actually don’t…
• Ask for it!
• Normal malloc API’s will not allocate IOVA-contiguous memory
• Memzone allocator has a flag to request IOVA-contiguous memory
• Use VFIO for everything
• Use legacy mode
![Page 7: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/7.jpg)
7
Memory Rework Design Principles
Question:
• How do we guarantee secondary process has the same view of
memory?
Answer:
• Preallocate all VA space at startup!
• Page table are synchronized over DPDK IPC
• Primary has authority over what pages get used
![Page 8: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/8.jpg)
8
Legacy DPDK Memory Architecture
• VA layout follows PA layout
• VA and PA layout is fixed
Page Page Page Page Page Page Page Page Page
rte_memseg rte_memseg
Page
malloc_elem malloc_elemmalloc_elem
rte_memzone rte_memzone
malloc_elem
rte_memzone
Contiguous VA area Contiguous VA area
Page Page Page Page Page Page Page Page Page Page
![Page 9: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/9.jpg)
9
18.05+ DPDK Memory Architecture
• VA layout is independent from PA layout
• VA layout is fixed, PA layout is not
Page Page Page Page Page
rte_memseg
Contiguous VA area
Page Page Page Page Page
rte_memseg rte_memseg rte_memseg rte_memseg
Page Page Page Page Page
malloc_elem malloc_elem
rte_memzone
![Page 10: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/10.jpg)
Shiny New Stuff BRUCE RICHARDSON
![Page 11: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/11.jpg)
11
Changes & New Features in 18.05+
New API’s:
• New memzone flag:
• RTE_MEMZONE_IOVA_CONTIG
• Memory event and validation callbacks
• Page map/unmap events
• Allow/deny new page mappings over specified limit
• Page walk and lookup API’s
• rte_memseg_walk et al.
![Page 12: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/12.jpg)
12
Changes & New Features in 18.05+
EAL parameters:
• -m/--socket-mem is now a minimum, not a limit• Think guaranteed memory availability
• --single-file-segments• Creates fewer hugepage files
• --legacy-mem• Mimics old DPDK
• --limit-mem (18.08+)• Place upper limit on memory usage, per socket
• --in-memory (18.08+)• Run without hugetlbfs mountpoint
![Page 13: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/13.jpg)
13
Future Changes (18.11+)
External memory support
• Currently RFC, V1 will be submitted for 18.11
• Using normal DPDK allocators with non-DPDK memory!
Memfd hugepages support for --in-memory mode
• Allows running without hugetlbfs and use virtio/vhost
• Patches currently at V1
• Virtio patches currently RFC
• Makes DPDK easier to set up in Cloud Native environments
![Page 14: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/14.jpg)
Case Studies BRUCE RICHARDSON
![Page 15: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/15.jpg)
15
Why You Should Care
Generally, memory in DPDK is designed to be invisible, so why should anyone care?
• Because we can accidentally break stuff!
When changes happen, certain things may break because:
• Code makes assumptions about memory layout
• Code makes assumptions about internals of DPDK
Memory management is fundamental to DPDK, so changes in memory subsystem can potentially affect everyone!
• Call for more reviews of memory-related patches
![Page 16: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/16.jpg)
16
Memory Layout Dependency
Problem:
• Certain drivers in DPDK relied on PA layout for lookups
• Few memsegs to look through => little impact on performance
• After applying 18.05 memory hotplug changes, there was a noticeable performance drop
Solution:
• For affected drivers, stopgap solution was implemented for 18.05
• Performance still impacted for small page sizes
• Proper solution expected for 18.11
![Page 17: Exploring the New DPDK Memory Subsystem · Memory Layout Dependency Problem: • net/virtio relies on valid memory starting from offset 0 into page table • A patch to 18.08 made](https://reader035.fdocuments.net/reader035/viewer/2022071008/5fc650034187eb34041cba4a/html5/thumbnails/17.jpg)
17
Memory Layout Dependency
Problem:
• net/virtio relies on valid memory starting from offset 0 into page table
• A patch to 18.08 made it so that segments are allocated from the top of
VA space
• As a result, net/virtio had issues trying to share more memory than was
needed
Solution:
• Reverted the patch for 18.08
• Investigation still ongoing