Memory Barriers in the Linux Kernel
-
Upload
davidlohr-bueso -
Category
Software
-
view
147 -
download
2
Transcript of Memory Barriers in the Linux Kernel
![Page 1: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/1.jpg)
Memory Barriers in the Linux KernelSemantics and Practices
Embedded Linux Conference – April 2016. San Diego, CA.
Davidlohr Bueso <[email protected]>SUSE Labs.
![Page 2: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/2.jpg)
2
Agenda
1. Introduction
• Reordering Examples
• Underlying need for memory barriers
2. Barriers in the kernel● Building blocks
● Implicit barriers
● Atomic operations
● Acquire/release semantics.
![Page 3: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/3.jpg)
3
References
i. David Howells, Paul E. McKenney. Linux Kernel source: Documentation/memory-barriers.txt
ii. Paul E. McKenney. Is Parallel Programming Hard, And, If So, What Can You Do About It?
iii. Paul E. McKenney. Memory Barriers: a Hardware View for Software Hackers. June 2010.
iv. Sorin, Hill, Wood. A Primer on Memory Consistency and Cache Coherence. Synthesis Lectures on Computer Architecture. 2011.
![Page 4: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/4.jpg)
4
![Page 5: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/5.jpg)
5
Flagship Example
A = 0, B = 0 (shared variables)
CPU0
A = 1x = B
CPU1
B = 1y = A
![Page 6: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/6.jpg)
6
Flagship Example
A = 0, B = 0 (shared variables)
CPU0
A = 1x = B
CPU1
B = 1y = A
(x, y) =
![Page 7: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/7.jpg)
7
Flagship Example
A = 0, B = 0 (shared variables)
CPU0
A = 1x = B
CPU1
B = 1y = A
(x, y) =
(0, 1)
A = 1x = B B = 1 y = A
![Page 8: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/8.jpg)
8
Flagship Example
A = 0, B = 0 (shared variables)
CPU0
A = 1x = B
CPU1
B = 1y = A
(x, y) =
(0, 1)
B = 1 y = AA = 1x = B
(1, 0)
![Page 9: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/9.jpg)
9
Flagship Example
A = 0, B = 0 (shared variables)
CPU0
A = 1x = B
CPU1
B = 1y = A
(x, y) =
(0, 1)
A = 1 B = 1 y = Ax = B
(1, 0)
(1, 1)
![Page 10: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/10.jpg)
10
Flagship Example
A = 0, B = 0 (shared variables)
CPU0
A = 1x = B
CPU1
B = 1y = A
(x, y) =
(0, 1)
x = B y = A A = 1 B = 1
(1, 0)
(1, 1)
(0, 0)
![Page 11: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/11.jpg)
11
Memory Consistency Models
• Most modern multicore systems are coherent but not consistent.‒ Same address is subject to the cache coherency protocol.
• Describes what the CPU can do regarding instruction ordering across addresses.‒ Helps programmers make sense of the world.
‒ CPU is not aware if application is single or multi-threaded. When optimizing, it only ensures single threaded correctness.
![Page 12: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/12.jpg)
12
Sequential Consistency (SC)
“A multiprocessor is sequentially consistent if the result of any execution is the same as some sequential order, and within any
processor, the operations are executed in program order”
– Lamport, 1979.
• Intuitively a programmer's ideal scenario.‒ The instructions are executed by the same CPU in the order in
which it was written.
‒ All processes see the same interleaving of operations.
![Page 13: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/13.jpg)
13
Total Store Order (TSO)
• SPARC, x86 (Intel, AMD)
• Similar to SC, but:‒ Loads may be reordered with writes.
[l] B
[s] B
[l] B
[s] C
[l] B
[s] A
[l] A
[s] B
![Page 14: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/14.jpg)
14
Total Store Order (TSO)
• SPARC, x86 (Intel, AMD)
• Similar to SC, but:‒ Loads may be reordered with writes.
[l] B
[s] B
[l] B
[s] C
[l] B
[s] A
[l] A
[s] B
L→L
![Page 15: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/15.jpg)
15
Total Store Order (TSO)
• SPARC, x86 (Intel, AMD)
• Similar to SC, but:‒ Loads may be reordered with writes.
[l] B
[s] B
[l] B
[s] C
[l] B
[s] A
[l] A
[s] B
L→L
S→S
![Page 16: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/16.jpg)
16
Total Store Order (TSO)
• SPARC, x86 (Intel, AMD)
• Similar to SC, but:‒ Loads may be reordered with writes.
[l] B
[s] B
[l] B
[s] C
[l] B
[s] A
[l] A
[s] B
L→L
S→S
L→S
![Page 17: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/17.jpg)
17
Total Store Order (TSO)
• SPARC, x86 (Intel, AMD)
• Similar to SC, but:‒ Loads may be reordered with writes.
[l] B
[s] B
[l] B
[s] C
[l] B
[s] A
[l] A
[s] B
L→L
S→S
L→S
S→L
![Page 18: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/18.jpg)
18
Relaxed Models
• Arbitrary reorder limited only by explicit memory-barrier instructions.
• ARM, Power, tilera, Alpha.
![Page 19: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/19.jpg)
19
Fixing the Example
A = 0, B = 0 (shared variables)
CPU0
A = 1
x = B
CPU1
B = 1
y = A
CPU1
B = 1
y = A
![Page 20: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/20.jpg)
20
Fixing the Example
A = 0, B = 0 (shared variables)
CPU0
A = 1<MB>x = B
CPU1
B = 1
y = A
CPU1
B = 1<MB>y = A
![Page 21: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/21.jpg)
21
Fixing the Example
A = 0, B = 0 (shared variables)
CPU0
A = 1<MB>x = B
CPU1
B = 1
y = A
CPU1
B = 1<MB>y = A
● Compiler barrier
![Page 22: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/22.jpg)
22
Fixing the Example
A = 0, B = 0 (shared variables)
CPU0
A = 1<MB>x = B
CPU1
B = 1
y = A
CPU1
B = 1<MB>y = A
● Compiler barrier
● Mandatory barriers (general+rw)
![Page 23: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/23.jpg)
23
Fixing the Example
A = 0, B = 0 (shared variables)
CPU0
A = 1<MB>x = B
CPU1
B = 1
y = A
CPU1
B = 1<MB>y = A
● Compiler barrier
● Mandatory barriers (general+rw)
● SMP-conditional barriers
![Page 24: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/24.jpg)
24
Fixing the Example
A = 0, B = 0 (shared variables)
CPU0
A = 1<MB>x = B
CPU1
B = 1
y = A
CPU1
B = 1<MB>y = A
● Compiler barrier
● Mandatory barriers (general+rw)
● SMP-conditional barriers
● acquire/release
![Page 25: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/25.jpg)
25
Fixing the Example
A = 0, B = 0 (shared variables)
CPU0
A = 1<MB>x = B
CPU1
B = 1
y = A
CPU1
B = 1<MB>y = A
● Compiler barrier
● Mandatory barriers (general+rw)
● SMP-conditional barriers
● acquire/release
● Data dependency barriers
● Device barriers
![Page 26: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/26.jpg)
Barriers in the Linux Kernel
![Page 27: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/27.jpg)
27
Abstracting Architectures
• Most kernel programmers need not worry about ordering specifics of every architecture.‒ Some notion of barrier usage is handy nonetheless – implicit
vs explicit, semantics, etc.
• Linux must handle the CPU's memory ordering specifics in a portable way with LCD semantics of memory barriers.‒ CPU appears to execute in program order.
‒ Single variable consistency.
‒ Barriers operate in pairs.
‒ Sufficient to implement synchronization primitives.
![Page 28: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/28.jpg)
28
Abstracting Architectures
mb()mfence
dsb
sync...
● Each architecture must implement its own calls or otherwise default to the generic and highly unoptimized behavior.
● <arch/xxx/include/asm/barriers.h> will always define the low-level CPU specifics, then rely on <include/asmgeneric/barriers.h>
![Page 29: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/29.jpg)
29
A Note on barrier()
• Prevents the compiler from getting smart, acting as a general barrier.
• Within a loop forces the compiler to reload conditional variables – READ/WRITE_ONCE.
![Page 30: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/30.jpg)
30
Implicit Barriers
• Calls that have implied barriers, the caller can safely rely on:‒ Locking functions
‒ Scheduler functions
‒ Interrupt disabling functions
‒ Others.
![Page 31: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/31.jpg)
31
Sleeping/Waking
• Extremely common task in the kernel and flagship example of flag-based CPU-CPU interaction.
CPU0 CPU1
while (!done) { done = true;schedule(); wake_up_process(t);current→state = …;
}
![Page 32: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/32.jpg)
32
Sleeping/Waking
• Extremely common task in the kernel and flagship example of flag-based CPU-CPU interaction.
CPU0 CPU1
while (!done) { done = true;schedule(); wake_up_process(t);current→state = …;set_current_state(…);
}
![Page 33: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/33.jpg)
33
Sleeping/Waking
• Extremely common task in the kernel and flagship example of flag-based CPU-CPU interaction.
CPU0 CPU1
while (!done) { done = true;schedule(); wake_up_process(t);current→state = …;set_current_state(…);
}
smp_store_mb(): [s] →state = … smp_mb()
![Page 34: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/34.jpg)
34
Atomic Operations
• Any atomic operation that modifies some state in memory and returns information about the state can potentially imply a SMP barrier:‒ smp_mb() on each side of the actual operation
[atomic_*_]xchg()atomic_*_return()atomic_*_and_test()atomic_*_add_negative()
![Page 35: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/35.jpg)
35
Atomic Operations
• Any atomic operation that modifies some state in memory and returns information about the state can potentially imply a SMP barrier:‒ smp_mb() on each side of the actual operation
‒ Conditional calls imply barriers only when successful.
[atomic_*_]xchg()atomic_*_return()atomic_*_and_test()atomic_*_add_negative()
[atomic_*_]cmpxchg()atomic_*_add_unless()
![Page 36: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/36.jpg)
36
Atomic Operations
• Most basic of operations therefore do not imply barriers.
• Many contexts can require barriers:cpumask_set_cpu(cpu, vec>mask);/* * When adding a new vector, we update the mask first, * do a write memory barrier, and then update the count, to * make sure the vector is visible when count is set. */smp_mb__before_atomic();atomic_inc(&(vec)>count);
![Page 37: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/37.jpg)
37
Atomic Operations
• Most basic of operations therefore do not imply barriers.
• Many contexts can require barriers:/* * When removing from the vector, we decrement the counter first * do a memory barrier and then clear the mask. */atomic_dec(&(vec)>count);smp_mb__after_atomic();cpumask_clear_cpu(cpu, vec>mask);
![Page 38: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/38.jpg)
38
Acquire/Release Semantics
• One way barriers.
• Passing information reliably between threads about a variable.‒ Ideal in producer/consumer type situations (pairing!!).
‒ After an ACQUIRE on a given variable, all memory accesses preceding any prior RELEASE on that same variable are guaranteed to be visible.
‒ All accesses of all previous critical sections for that variable are guaranteed to have completed.
‒ C++11's memory_order_acquire, memory_order_release and memory_order_relaxed.
![Page 39: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/39.jpg)
39
Acquire/Release Semantics
CPU0
spin_lock … …
CPU0
spin_lock(&l)
spin_unlock(&l)
CPU1
spin_lock(&l)
spin_unlock(&l)
CR
CR
![Page 40: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/40.jpg)
40
Acquire/Release Semantics
CPU0
spin_lock … …
CPU0
spin_lock(&l)
spin_unlock(&l)
CPU1
spin_lock(&l)
spin_unlock(&l)
CR
CR
smp_store_release(lock→val, 0) <> cmpxchg_acquire(lock→val, 0, LOCKED)
![Page 41: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/41.jpg)
41
Acquire/Release Semantics
CPU0
spin_lock … …
CPU0
spin_lock(&l)
spin_unlock(&l)
CPU1
spin_lock(&l)
spin_unlock(&l)
CR
CRRELEASE
ACQUIRE
smp_store_release(lock→val, 0) <> cmpxchg_acquire(lock→val, 0, LOCKED)
![Page 42: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/42.jpg)
42
Acquire/Release Semantics
CPU0
spin_lock … …
CPU0
spin_lock(&l)
spin_unlock(&l)
CPU1
spin_lock(&l)
spin_unlock(&l)
CR
CRRELEASE(LS, SS)
ACQUIRE(LL, LS)
smp_store_release(lock→val, 0) <> cmpxchg_acquire(lock→val, 0, LOCKED)
![Page 43: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/43.jpg)
43
Acquire/Release Semantics
CPU0
spin_lock … …
CPU0
spin_lock(&l)
spin_unlock(&l)
CPU1
spin_lock(&l)
spin_unlock(&l)
CR
CRRELEASE
ACQUIRE
smp_store_release(lock→val, 0) <> cmpxchg_acquire(lock→val, 0, LOCKED)
![Page 44: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/44.jpg)
44
Acquire/Release Semantics
CPU0
spin_lock … …
CPU0
spin_lock(&l)
spin_unlock(&l)
CPU1
spin_lock(&l)
spin_unlock(&l)
CR
CRRELEASE
ACQUIRE
smp_store_release(lock→val, 0) <> cmpxchg_acquire(lock→val, 0, LOCKED)
![Page 45: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/45.jpg)
45
Acquire/Release Semantics
CPU0
spin_lock … …
CPU0
spin_lock(&l)
spin_unlock(&l)
CPU1
spin_lock(&l)
spin_unlock(&l)
CR
CRRELEASE
ACQUIRE
smp_store_release(lock→val, 0) <> cmpxchg_acquire(lock→val, 0, LOCKED)
![Page 46: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/46.jpg)
46
Acquire/Release Semantics
• Regular atomic/RMW calls have been fine grained for archs that support strict acquire/release semantics.
• Currently only used by arm64 and PPC.‒ LDAR/STLR
cmpxchg()cmpxchg_acquire()cmpxchg_release()cmpxchg_relaxed()
smp_load_acquire()smp_cond_acquire()smp_store_release()
![Page 47: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/47.jpg)
47
Acquire/Release Semantics
• These are minimal guarantees.‒ Ensuring barriers on both sides of a lock operation will require
therefore, full barrier semantics:
smp_mb__before_spinlock()
smp_mb__after_spinlock()
• Certainly not limited to locking.‒ perf, IPI paths, scheduler, tty, etc.
![Page 48: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/48.jpg)
48
Acquire/Release Semantics
• Busy-waiting on a variable that requires ACQUIRE semantics:CPU0
while (!done)cpu_relax();
smp_rmb();
CPU1
smp_store_release(done, 1);
![Page 49: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/49.jpg)
49
Acquire/Release Semantics
• Busy-waiting on a variable that requires ACQUIRE semantics:CPU0
while (!done)cpu_relax();
[LS] smp_rmb(); [LL]
CPU1
smp_store_release(done, 1);
![Page 50: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/50.jpg)
50
Acquire/Release Semantics
• Busy-waiting on a variable that requires ACQUIRE semantics:CPU0
while (!done)cpu_relax();
[LS] smp_rmb(); [LL]
CPU1
smp_store_release(done, 1);
smp_load_acquire(!done);
![Page 51: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/51.jpg)
51
Acquire/Release Semantics
• Busy-waiting on a variable that requires ACQUIRE semantics:CPU0
while (!done)cpu_relax();
[LS] smp_rmb(); [LL]
CPU1
smp_store_release(done, 1);
• Fine-graining SMP barriers while a performance optimization, makes it harder for kernel programmers.
![Page 52: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/52.jpg)
52
Concluding Remarks
• Assume nothing.
• Read memory-barriers.txt
• Use barrier pairings.
• Comment barriers.
![Page 53: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/53.jpg)
Thank you.
![Page 54: Memory Barriers in the Linux Kernel](https://reader035.fdocuments.net/reader035/viewer/2022062316/5882954a1a28abca6d8b7357/html5/thumbnails/54.jpg)
54