Improving IPC by Kernel DesignJochen Liedtke

Shane MatthewsPortland State University

3/12/2004 Portland State University

Summary

• Review• Performance improved

– Architecture Level– Algorithmic Level– Interface Level– Coding Level

3

Micro-kernels• Minimal OS, providing

a set of primitives used to implement thread/address space management and IPC [1]

• Everything else is moved to user-space (servers)

4

Terminology (L3)• Dataspace

– Memory object, mapped into address space• Task

– Composed of threads, dataspaces, and an address space• Message

– String/memory object

5

L3 Architecture & IPC• Active components communicate via

messages• Applies to:

– Device drivers• Implemented as user level tasks

– Hardware Interrupts• Interrupt message from micro-kernel to thread

6

L3 Redesign Principles• IPC performance is the master

– Security and performance must not be affected• Synergetic effects taken into consideration

– (Think combined effects)– May lead to reinforcement or diminution

• Design must aim at performance goal– Per short message transfer– 350 cycles (7 micro-seconds)

3/12/2004 Portland State University

Architectural Level

• Messages• Process Structure• Control Blocks

3/12/2004 Portland State University

Compound Messages• Multiple

send/receive -> 1 send/receive

• Messages consists of direct/indirect strings, and memory objects

9

Twofold message copy

• [A space] -> [kernel] -> [B space]

• O(20 + .75n) cycles, n:= bytes

• Good for small messages

• Need something better as n grows

10

LRPC and SRC RPC• Client/server share user level memory

– sender -> shared buffer• Problems

– When server to client is 1 to many, shared regions of address space become critical resources

– Shared regions require explicit opens (unlike L3)– Message change during/after checking

11

Direct Message Copy Via Windows

• L3's method– Destination mapped

into window– Message copied to

window

• Window– per address space– Accessed exclusivly

by kernel

12

Communication Windows• Problems

– Must be fast– Different threads

coxisting within address space

• L3 Implementation– One word page

directory B to A.

13

Process Structure• Threads running kernel mode have 1 kernel

stack per thread– Efficient since interupts, page faults, IPC,

already save state on kernel stack• Continuations

– Pro: • Reduce kernel stack

– Cons: • Require additional copies between kernel and

continutation• Interfere with other optimizations

14

Tread Control Blocks• Implemented as large array in kernel

– fast tcb access• Array base + tcb # + tcb size

– Saves TLB misses (IPC)• kernel stacks of sender and reciever located in TCB

page

– Locking done via unmapping on TCB

3/12/2004 Portland State University

Algorithmic Level• Thread Identifier

• Lazy Scheduling

• Short Messages Via Registers

3/12/2004 Portland State University

Thread Identifier

• Thread addressed by 64-bit UID in user-mode

• Thread number in lower 32-bits of UID– AND with bit mask, add to TCB’s array base

3/12/2004 Portland State University

Lazy Scheduling

• IPC operation call or reply & receive next– Delete sending thread from ready queue– Insert into waiting queue– Delete receiving thread from waiting queue– Insert into ready queue

• Too many queue operations!

3/12/2004 Portland State University

Lazy Scheduling cont.

• L3 queue invariants– Ready queue contains all ready threads– Waiting queue contains at least all threads

waiting• TCB contains threads state (ready/waiting)• Scheduler removes all threads not

belonging to queue during queue parsing

3/12/2004 Portland State University

Short Messages Via Registers

• High proportion of messages are short– Ex. Driver ack/error, hardware interrupts

• 486– 7 general registers– 3 needed: sender ID, result code– 4 available

• 8-byte messages using coding scheme

3/12/2004 Portland State University

Interface Level

• Simple RPC stubs– Load registers, system call, check success– Compiler generates stubs inline

• Parameter Passing– Use registers when possible

3/12/2004 Portland State University

Coding Level

• Reduce cache and TLB misses– Short kernel code

• Short jumps, use registers, short address displacements

– IPC kernel code in one page– Handle save/restore of coprocessor lazily

• Delayed until different thread needs to use it

3/12/2004 Portland State University

Results• 100% would indicate

double the time increase

• Removal of all increase IPC time by 134% for 8 byte message

3/12/2004 Portland State University

Results• L3 VS Mach• System

– Intel 486 DX-50– 256 KB external

cache– 16 MB memory

3/12/2004 Portland State University

Results cont.

3/12/2004 Portland State University

Conclusions

• IPC improved by applying– Performance based reasoning– Synergetic effects– Architecture -> coding

26

References• [1]

http://en.wikipedia.org/wiki/Micro_kernel• [2] Improving IPC by Kernel Design -

Jochen Liedtke