xCalls: Safe I/O in Memory Transactions

Haris Volos,Andres Jaan Tack, Neelam Goyal+,

Michael Swift, Adam Welc§

University of Wisconsin - MadisonUniversity of Wisconsin - Madison+ §

• CMP leads to more concurrency within programs• Synchronizing access to shared data via locks is hard

• atomic construct simplifies synchronizing access to shared data

atomic { x = x - c; y = y + c;}

LOCK(L) x = x - c; y = y + c;UNLOCK(L)

store(A)store(B)store(A)

Transactional Memory (TM)

Thread 1

atomic { A = A – 20; B = B + 20;}

atomic { B = B – 10; A = A + 10;}

Thread 2

Atomicity:PERFORM both updates to A and B or none

Isolation:OBSERVE both red transaction’s updates to A and B or none

ABORT conflicting transactions

Abort blue

Conflict

2

A challenging world…• Real world programs frequently take actions

outside of their own memory– Firefox: ~1% critical sections do system call [Baugh TRANSACT ’07]

• Most TM systems apply only to user-level memory

Thread 1

atomic { item = procItem(queue); write (file, principal); write (file, item->header);}

atomic { item = procItem(queue); write (file, principal); write (file, item->header);}

memory file Thread 21a2b2b

1a

Abort

Interleaved writes

File writes not dropped

Memory updates dropped

3

1a

1b

2a

2b

State of the art

• Defer• Undo• Global Lock

Ignore failures

Stop the world

4

atomic { item = procItem(queue); write (file, principal); write (file, item->header); send (socket, item->body);}

LAN

Internet

NAS

COMMITPerform send

Defer

Contribution

5

• xCall programming interface– Exposes transactional semantics to programmer– Enables I/O within transactions w/o stopping the world– Exposes all failures to the program

Transactional Program

Transaction-unaware kernel

System call interface

xCall Library

Legacy calls

xCalls

Runs in user mode

Outline

• Motivation• xCall Design & Implementation• Evaluation• Conclusion

6

Design overview

• Principles1. As early as possible but not earlier2. Expose all failures

• Components– Atomic execution– Isolation– Error handling

7

Atomic execution

• Provide abort semantics for kernel data and I/O

Expose to programmer when action is performed

8

atomic { item = procItem(queue); x_write (file, principal); x_write (file, item->header);}

file

Abort

buffers

Can reverse action? Need result? Execution Example

No Yes Global lock ioctl()

No No Defer x_write_pipe()

Yes -- In-place x_write()

Isolation

• Prevent conflicting changes to kernel data made within a transaction

Sentinels– Revocable user-level locks– Lock logical kernel state visible through system calls

Thread 1

atomic { item = procItem(queue); x_write (file, principal); x_write (file, item->header);}

atomic { item = procItem(queue); x_write (file, principal); x_write (file, item->header);}

memory file Thread 2

Conflict

9

atomic { item = procItem(queue); x_write (file, principal, &err1); x_write (file, item->header, &err2); x_send (socket, item->body, &err3);}if (err1 || err2 || err3) { /* CLEANUP */}

Error handling

• Some errors might not happen until transaction commits or aborts

Inform programmer when failures happen– Handle errors after transaction completes

LAN

InternetDefer

Deferred send: FAILEDerr3 = error

COMMITPerform send

Handle errorhere

10

ssize_t x_write (int fd, void *buf, ssize_t nbytes ) { void *localbuf; ssize_t bytes; get_sentinel(fd); localbuf = alloc_local_buf(nbytes); read(fd, localbuf, nbytes); bytes = write(fd, buf, nbytes); if (bytes != -1) compensate(x_undo_write, fd, localbuf, bytes, result);}int x_undo_write (int fd, void *buf, ssize_t nbytes, int *result){ off_t ret1, ret2; lseek(fd, -nbytes, SEEK_CUR); if (ret1) pwrite(fd, buf, nbytes, ret1); if (ret1 == -1 || ret2 == -1) *result = errno; return (ret1 == -1 || ret2 == -1);

}

Example: file write

, int *result

Atomic execution

Atomic execution

Atomic execution

Isolation

Error handling

Error handling

ret2 =

11

ret1 =

Summary

• xCall API exposes transactional semantics– Atomicity– Isolation– Error handling

• Prototype implementation– Executes as user-mode library– Relies on Intel STM for transactional memory– Provides 27 xCalls including file handling,

communication, threading12

Outline

• Motivation• xCall Design & Implementation• Evaluation– Benefit of xCalls over global lock– Benefit of TM over locks

• Conclusion

13

Evaluation platform

• Transactified three large multithreaded apps– Berkeley DB– BIND– XMMS

• Configurations– Native : locks + system calls – STM : transactions + system calls + global lock– xCalls : transactions + xCalls

• Run on 4 quad-core 2 GHz AMD Barcelona14

Performance: Berkeley DB (1/2)

15

• xCalls scales better than STM with global lock• TM worse than locks due to STM overhead

• Workload: TPC-C

Performance: Berkeley DB (2/2)

16

• xCalls improve concurrency over coarse grain lock• Global lock kills optimistic concurrency

• Workload: Lockscale

Performance: BIND

17

• Transactions scale better than coarse grain locks• xCalls enable additional concurrency

• Workload: QueryPerf

Performance summary

18

• xCalls benefit programs with I/O concurrency• TM benefits programs with contended but not

conflicting critical sections

Conclusion

• xCall programming interface– Brings common OS services to TM programs– Requires no kernel modifications– Improves scalability over state of the art

19

Questions?