Unix Programming with Perl 2

Unix Programming with Perl 2

Unix Programming with Perl 2

DeNA Co., Ltd.

Kazuho Oku

Writing correct codeWriting correct code

tests aren’t enoughtests don’t ensure that the code is correct

writing correct code requires…knowledge of perl and knowledge of the OS

Oct 15 2011 Unix Programming with Perl 2 2

Last Year’s TalkLast Year’s Talk

Covered these aspects of Unix programming using Perl$! and Errno

how to evaluate the errors

file handlestheir internalsinteraction w. fork(2)

Unix signalsvarious signals and how to handle themwriting cancellable code


Last Year’s SlidesLast Year’s Slides

http://www.slideshare.net/kazuho/unix-programming-with-perl


Today’s TalkToday’s Talk

will cover more advanced topicsinter-process communicationUnix signals and race condition


IPC::Open3


IPC::Open3 - Pipe and DeadlockIPC::Open3 - Pipe and Deadlock

# check syntax errors in perl scripts by using perl –cmy @cmd = ( $^X, (map { "-I$_" } grep { !ref $_ } @INC), '-c', $file,);my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (waitpid($pid, 0) != $pid) {}ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file);

http://d.hatena.ne.jp/tokuhirom/20100813/1281666615



Q. generally speaking, this code might block. But when?

my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (waitpid($pid, 0) != $pid) {}





A1. blocks if the child process reads from STDIN



Close STDIN of the child before calling waitpid

my $pid = open3(my $cin, my $cout, my $cerr, @cmd);close $cin;while (waitpid($pid, 0) != $pid) {}





A1. blocks if the child process reads from STDIN

A2. blocks if the child process writes long data to STDOUT or STDERR


Size of the pipe bufferSize of the pipe buffer

Pipe has size limitcannot write infinitelyunless the other peer reads from pipe

But actually, how large is the size limit?


Size of the pipe buffer (2)Size of the pipe buffer (2)

Checking the size of the pipe buffer

for (my $sz = 1; ; $sz++) { my @cmd = ( $^X, '-e', qq(print "1"x$sz), ); my $pid = open3(my $cin, my $cout, 0, @cmd) or die $!; while (waitpid($pid, 0) != $pid) {} print "size: $sz\n";}


Size of the pipe buffer (3)Size of the pipe buffer (3)

Size of the pipe buffer was…Linux 2.6.32 (x86-64): 65,536 bytesMac OS X 10.6: 16,384 bytes

Size may varyold versions of Linux: 4,096 bytes

TCP streams and Unix sockets have configurable buffer size as well



How should we avoid deadlocks?if we do not need the output of the child

process (ex. perl –c), could this be the right answer?

my $pid = open3(my $cin, my $cout, my $cerr, @cmd);close $cout;close $cerr;while (waitpid($pid, 0) != $pid) {}ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file);



No. The child process may get killed while trying to write to the output streams closed by the parent process

my $pid = open3(my $cin, my $cout, my $cerr, @cmd);close $cout;close $cerr;# if child process tries to write at this moment, it# will be killed by SIGPIPEwhile (waitpid($pid, 0) != $pid) {}# and as a result, $? may become differentok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file);



So how about this? we read all data from the child process

my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (<$cout>) {}while (<$cerr>) {}while (waitpid($pid, 0) != $pid) {}



No. Will deadlock if the child process writes more bytes than the pipe buffer size to STDERR

my $pid = open3(my $cin, my $cout, my $cerr, @cmd);while (<$cout>) {} # may enter deadlock at this pointwhile (<$cerr>) {}while (waitpid($pid, 0) != $pid) {}



Tip: passing undef as CHLD_ERR will send all output to CHLD_OUT

my $pid = open3(my $cin, my $cout, undef, @cmd);while (<$cout>) {}while (waitpid($pid, 0) != $pid) {}


IPC::Open3 – Better to use temporary filesIPC::Open3 – Better to use temporary files

Advice: don’t use pipes, use temporary files (unless you need to read the output of the child process while it is running)

my $cout = File::Temp->new();my $pid = do { local *COUT = $cout; open3(my $cin, ’>&COUT’, 0, @cmd);} or die $!;while (waitpid($pid, 0) != $pid) {}seek($cout, 0, SEEK_SET) or die $!; # seek to the start...Oct 15 2011 Unix Programming with Perl 2 20

IPC::Open3 vs. open |-IPC::Open3 vs. open |-

Q. Why use IPC::Open3 instead of open ’-| cmd > tmpfile’?

A. to skip the shell invocationfor speed and security


IPC::Open3 vs. open |- (2)IPC::Open3 vs. open |- (2)

Escaping for open |- is difficult# using IPC::Open3my @cmd = ( $prog, ’-e’, $arg2,);my $pid = open3(my $cin, my $cout, 0, @cmd) or die $!;

# using open |-my $cmd = ”$prog –e $arg”; # need to escape $argopen my $fh, ’|-’,”$cmd > ” . $tempfh->filename or die $!;


Avoid shell invocationAvoid shell invocation

Shell invocation is evilex. iT○ns upgrade accidentally removes

user filesfailed to quote usernames with a whitespace

Direct invocation is safersystem($args) => system(@args)open | => IPC::Open2 or IPC::Open3


Rewriting open |- using IPC::Open3Rewriting open |- using IPC::Open3

# the originalopen(my $fh,’|-’, $cmd) or die $!;print $fh ”hello\n”;close $fh;# TODO: check $?

# using IPC::Open3my $pid = open3(my $fh,’>&STDOUT’, ’>&STDERR’, @cmd) or die $!;print $fh ”hello\n”;close $fh;while (waitpid($pid, 0) != $pid) {}# TODO: check $?


Rewriting open -| using IPC::Open3Rewriting open -| using IPC::Open3

# the originalopen(my $fh,’-|’, $cmd) or die $!;my $line = <$fh>;close $fh;# TODO: check $?

# using IPC::Open3my $pid = open3(my $cin, my $fh, ’>&STDERR’, @cmd) or die $!;close $cin;my $line = <$fh>;close $fh;while (waitpid($pid, 0) != $pid) {}# TODO: check $?


Signal and Race Condition


How to sleep until receiving a signal?How to sleep until receiving a signal?

Is the code correct?

my $gothup = 0;local $SIG{HUP} = sub { $gothup++ };while (! $gothup) { sleep(30 * 60); # sleep for 30 minutes do_cleanup(); # do some periodical tasks}print ”SIGHUP!\n”;


How to sleep until receiving a signal? (2)How to sleep until receiving a signal? (2)

a race condition exists

my $gothup = 0;local $SIG{HUP} = sub { $gothup++ };while (! $gothup) { # What happens if a signal is arrives here? # sleep() will sleep 30 minutes since it never # gets interrupted by the signal sleep(30 * 60); # sleep for 30 minutes do_cleanup(); # do some periodical tasks}print ”SIGHUP!\n”;


Use POSIX::pselectUse POSIX::pselect

SIGHUP is blocked outside of pselect (and thus no race conditions)

my $blocked = POSIX::SigSet->new(SIGHUP);my $unblocked = POSIX::SigSet->new();my $gothup = 0;local $SIG{HUP} = sub { $gothup++ };sigprocmask(SIG_BLOCK, $blocked, $unblocked);while (! $gothup) { pselect(undef, undef, undef, 30 * 60, $unblocked); do_cleanup(); # do some periodical tasks}print ”SIGHUP!\n”;


The problem of pselectThe problem of pselect

Pselect has race condition on many environments, implemented like…

sub select { my ($rset, $wset, $eset, $secs, $mask) = @_; my $oldmask = POSIX::SigSet->new(); sigprocmask(SIG_SETMASK, $mask, $oldmask); $oldmask); my $ret = select($rset, $wset, $eset, $secs); sigprocmask(SIG_SETMASK, $oldmask); $ret;}

osx has the problem, glibc on linux does not have the problem but bionic (android) has the problem, …


Using eval & die does not solve the problemUsing eval & die does not solve the problem

my $blocked = POSIX::SigSet->new(SIGHUP);my $unblocked = POSIX::SigSet->new();sigprocmask(SIG_BLOCK, $blocked, $unblocked);$SIG{HUP} = sub { die ’sighup:’ };while (! $gothup) { { local $@; eval { sigprocmask(SIG_SETMASK, $unblocked); # what if the signal is delivered at the very moment the # perl interpreter calls sleep(3)? sleep(30 * 60); # sleep for 30 minutes }; $gothup = $@ =~ /^sighup:/; sigprocmask(SIG_SETMASK, $blocked); } do_cleanup(); # do some periodical tasks}


The fix – call syswrite on signalThe fix – call syswrite on signal

# set unsafe signal handler (see perldoc perlipc) that converts a# signal to a message (Q. proper error handling as advised in this# year and last year’s slides are missing. Can you name them?)socketpair(my $sig_rdr, my $sig_wtr, AF_UNIX, SOCK_STREAM, PF_UNSPEC) or die $!;POSIX::sigaction(SIGHUP, POSIX::SigAction->new(sub { syswrite($sig_wtr, "1", 1) == 1 or die $!; }}));while (1) { my $rfds = ''; vec($rfds, fileno($sig_rdr), 1) = 1; if (select($rfds, undef, undef, 30 * 60) > 0) { sysread($sig_rdr, my $buf, 1) == 1 or die $!; last; } do_cleanup(); # do some periodical tasks}


Summary


SummarySummary

buffer size is not infinite, be aware of deadlocks on inter-process / network communication

avoid shell invocation, use system(@args) or IPC::Open3

be careful of race conditions when handling Unix signals


Unix Programming with Perl 2

Technology

Transcript of Unix Programming with Perl 2