Inter-domain Socket Communications Supporting High Performance and Full Binary Compatibility on Xen March 5, 2008 Kangho Kim System Software Team Internet Server Group VEE 08 Outline Motivation Design & Implementation Socket creation Connect Accept Send/recv Close Live migration Evaluation results Conclusion Motivation Low bandwidth (Xen 3.0.3) We need XWAY!! Motivation(contd) Low latency (Xen 3.0.3) Existing socket applications Design & Implementation Shared memory Direct communication TCP/IP bypass Direct data copy Shared memory Hardware VM Kernel User Domain 0Domain U XWAY Shared memory TCP XWAY channel HeadTail SQ HeadTail RQ HeadTail RQ Event channel HeadTail SQ Shared memory for one XWAY channel Domain ADomain B XWAY architecture socket application library VFS INET UNIXX25 XWAY switch TCP IP XWAY protocol Network device driverXWAY driver BSD socket user kernel UDP Dual channel XWAY channel for data TCP/IP channel for socket options and socket state socket application XWAY socket socket application XWAY socket TCP/IP channel XWAY channel xway_sock Socket data structure for XWAY llseek: no_llseek poll: sock_poll open: sock_no_open release: sock_close readv: sock_readv writev: sock_writev sendpage: sock_sendpage release: inet_release bind: inet_bind connect: xsw_stream_connect accept: xsw_accept poll: xsw_poll listen: inet_listen shutdown: inet_shutdown sendmsg: inet_sendmsg recvmsg: sock_common_recvmsg sendpage: inet_sendpage close: xsw_close connect: xsw_v4_connect disconnect: xsw_disconnect accept: xsw_csk_accept ioctl: tcp_ioctl init: xsw_v4_init_sock shutdown: xsw_shutdown setsockopt: tcp_setsockopt getsockopt: tcp_getsockopt remote_dom_id eventchn_port eventchn_irq xring_send xring_recv xring_send_h xring_send_t xring_recv_h xring_recv_t inode socket sock Create Sock Connect TCP channel Sock Connection helper XWAY channel SQ RQ SQ RQ Accept TCP channel XWAY channel Sock XWAY channel Send/Recv TCP channel Sock XWAY channel SQ RQ SQ RQ Close Sock TCP channel XWAY channel SQ RQ SQ RQ XWAY channel Poll TCP session XWAY session Sock Live migration Cut XWAY channels underneath XWAY socket Redirect all data send/receive calls to TCP channels Ready to live migration Hardware VM Kernel User Domain 0Domain U TCP/IP Hardware VM Domain 0 Live migration (contd) TCP channel Sock XWAY channel SQ RQ SQ RQ Evaluation Bandwidth Netperf TCP_STREAM Latency Netperf (TCP_RR) Application performance DBT-1 performance Execution time of API Binary compatibility Scp Ssh Apache TCP_DEFER_ACCEPT Vsftpd connect, accept, MSG_PEEK Proftpd shutdown, connect, SO_LINGER Telnet MySQL Netperf Conclusion XWAY design and implementation Full binary compatibility for socket interface High performance Live migration support Remaining work UDP, Domain crash resiliency Reducing CPU usage Code release April 20, 2007: xway ver 0.6 June 4, 2007: xway ver By the end of March, 2008: xway ver 0.7 (live migration included) Download http://sourceforge.net/projects/xway Questions or Comments Kangho Kim Cheiyol Kim Hyun-Sup Shin