Bigbadwolf
-
Upload
osfameron -
Category
Technology
-
view
8.323 -
download
0
description
Transcript of Bigbadwolf
30 agosto 20073
Debugging web applications● Debugging is hard● Complex, legacy applications...● Superstition● Panic● A metaphor involving pigs and wolves to the
rescue!
30 agosto 20074
This metaphor is brought to you by
● firenze.pm● perl.it● beer
– (Belhaven Best, James Pub, Firenze)
30 agosto 20075
What happens when a little piggy goes in the forest?
The Forest
30 agosto 20076
What happens when a little piggy goes in the forest?
The ForestCGI
30 agosto 20077
What happens when a little piggy goes in the forest?
The ForestCGI
mod_perl
30 agosto 20078
What happens when a little piggy goes in the forest?
The ForestCGI
mod_perl
Perl
30 agosto 20079
What happens when a little piggy goes in the forest?
The ForestCGI
mod_perlPerl
DBI
30 agosto 200710
What happens when a little piggy goes in the forest?
The ForestCGI
mod_perlPerl
DBI
modules
30 agosto 200711
What happens when a little piggy goes in the forest?
The ForestCGI
mod_perlPerl
DBI
modules
ourcode
30 agosto 200712
The Forest● And more!
– Network– HTML– XML– 3rd party services– AJAX– Configuration– Unix permissions
30 agosto 200713
Werewolves!
30 agosto 200714
Werewolves!
There!Wolves!
30 agosto 200715
What happens when a little piggy goes in the forest?
30 agosto 200716
What happens when a little piggy goes in the forest?
The Forestpig
16
30 agosto 200717
What happens when a little piggy goes in the forest?
The Forest
YARGGH!
30 agosto 200718
What happens when a little piggy goes in the forest?
The Forest
404
30 agosto 200719
What happens when a little piggy goes in the forest?
The Forest another pig
30 agosto 200720
What happens when a little piggy goes in the forest?
The Forest
30 agosto 200721
What happens when a little piggy goes in the forest?
● It never came out!● It came out limping and wounded● Dead! Gored by werewolves!● Takes a long time to come out
30 agosto 200722
What happens when a little piggy goes in the forest?
● It never came out!● It came out limping and wounded
– Part of the functionality is broken– Information wrong
● Dead! Gored by werewolves!– User sees raw error on screen
● Takes a long time to come out– optimization needed
30 agosto 200723
Werewolves!● Round up the villagers for a lynchmob!● (Bring a pitchfork!)● Burn them!
– (maybe burn a few innocent villagers along the way too!)
30 agosto 200724
Engineers● Luckily things are (a little) better if we know
what we're doing
30 agosto 200725
We shouldn't see this
Scary Forest(eeek!)
30 agosto 200726
But this
30 agosto 200727
Logging● Trace the piggy as he goes through the forest
27
30 agosto 200728
Logging
The Forestpig
PIG1: I'm scared
30 agosto 200729
Logging
The Forest
PIG1: I'm scaredPIG1: I'm very scared
30 agosto 200730
Logging
The Forest
PIG1: I'm scaredPIG1: I'm very scaredPIG1: This is Cruelty to Animals!
30 agosto 200731
Logging
The Forest
PIG1: I'm scaredPIG1: I'm very scaredPIG1: This is Cruelty to Animals!PIG1: Speaking to the Database
30 agosto 200732
Logging
The Forest
PIG1: I'm scaredPIG1: I'm very scaredPIG1: This is Cruelty to Animals!PIG1: Speaking to the DatabasePIG1: Look over there, a wol...
30 agosto 200733
Logging (print statements)● Print statements
– print “I AM HERE!”;
30 agosto 200734
Logging (warnings)● warn
– warn “THIS HAPPENED!”;
– (this will end up in the Apache error log● /var/log/apache2/error.log(.yyyy.mm.dd)
● errors also go to Apache error log– always check the error log
30 agosto 200735
Logging (apache error log)● $ tail f error.log.20070411
● | grep “Media” # your module● | grep “HC” # your identifier
● Point to the right machine– edit /etc/hosts ( ...\drivers\etc\hosts on Windows)– send to a server you have an account on
– (restart Firefox, sigh)
30 agosto 200736
Logging (apache error log)● $ tail f error.log.20070411
● | grep v “uninitialized”
● Too many warnings!● Hard to see what are real errors● (Also annoys the Sysadmins)
30 agosto 200737
Logging (print statements)● Print to HTML
– contenttype: text/html (or text/plain)– <pre> $info </pre>
– You can wrap this in a call● $this>printDebug(“pippo”);
● But this messes up headers!● Label your own piggy!
– $this>printDebug( “HC(contacts): pippo”);
30 agosto 200738
Logging (data structures)● Complex data structures
– Data::Dumper
– $VAR1 = { 'y' => 2, 'x' => 1 };
30 agosto 200739
Logging (manual trace)● When did the script crash?
– $this>printDebug(“HC 1 got here”);
– do something– $this>printDebug(“HC 2 got here”);
– do something– $this>printDebug(“HC 3 got here”);
– do something– $this>printDebug(“HC 4 got here”);
30 agosto 200740
Logging (manual trace)● Yes, that's quite stupid
– (but it beats using a pitchfork)
30 agosto 200741
Logging (end of html debug)
● To avoid messing up headers you can get your framework to print a development footer at the bottom of the HTML page instead
30 agosto 200742
Logging (to file)● save to a file
– open file– write– close file
● Lots of people recommend Log::Log4perl
30 agosto 200743
Logging ● use Log::Log4Perl qw(:easy);
● Log::Log4Perl>easy_init($ERROR);
● my $log = Log::Log4perl>get_logger('foo');
● $log>debug(“Only show this to logs”);
● $log>info(“This is more important”);
● $log>warn(“Oh dear”);
● $log>error(“Eek! This error on screen”);
● $log>fatal(“Werewolves!”);
30 agosto 200744
Logging● Log files eventually get really big● Sysadmins will want to rotate them● This may mean appending
– .yyyy.mm.ddto the filenames
30 agosto 200745
Error handling
The Forest
another pig
If a pig screams in the forest and
there's noone there to hear it... did it
really die?
45
30 agosto 200746
Error handling● Sometimes it's OK to die
– use CGI::Carp qw( fatalsToBrowser );
– die “HC: eeeeeek! The piggy is dead”;
30 agosto 200747
Error handling (CSI!)● But how did the piggy die?
– Bad parameter at TestModule.pm line 10.● If only we knew who had called TestModule!
30 agosto 200748
Error handling (Carp)● package TestModule;
● use Carp qw( croak );
● ... croak “Bad parameter”;...
– Bad parameter at test.pl line 7
30 agosto 200749
Error handling (stacktraces)● What if it was another subroutine or module
called?● use Carp qw(confess);
● confess “Ooops!”;
● Ooops! at Pippo.pm line 13 Pippo::three() called at Pluto.pm line 10 Pluto::two() called at Pinco.pm line 11 Pinco::one() called at test.pl line 7
30 agosto 200750
Error handling (framework)● If your framework handles this, you may get
stacktraces by default– They aren't always easy to read, but they contain
lots of useful information!
– Catalyst– In house framwork
30 agosto 200751
Error handling (Database)● We don't want the user seeing DB errors in live!
– BUT– We probably want to see them in staging!– classic idiom
● $sth>execute() or die $sth>errstr;
– You may want to:● have a “fatal” function which either dies informatively (in
staging) or produces an appropriate error document (in live)
30 agosto 200752
Error handling (Database)● Do you really want to fail silently...?
– $sth>execute()or warn $sth>errstr;
– (or $logger>log($sth>errstr));
30 agosto 200753
Testing
The Forest
I hope someone
tested this parachute
54
30 agosto 200754
Burn the Witch!● Sometimes we think we know where the
problem is– Sometimes we're wrong
30 agosto 200755
How do we check● A test!
– does it weigh more or less than a duck?● use Test::More tests => 1;
– is( $possible_witch>weight(), $duck>weight(), “It's the witch!” );
30 agosto 200756
Testing in the forest● Sometimes we need to test:
– user clicks on link in browser, logs in– user puts in information– the script talks to the Database– the script talks to a 3rd party API– the script talks to a module which returns some
information– the HTML is processed
● (e.g. the forest)
30 agosto 200757
Testing in the forest● But...
– what if we suspect that the problem is with formatting of display name?
– MyModule:: createDisplayName();
– Just test that!
30 agosto 200758
See the trees...● Write a simple test
– use Test::More;
– use MyModule;
– is ($m>createDisplayName('robin'), “Robin Hood”, “display name returned correctly” );
● If the test returns correctly, then maybe the problem is before (or after)
● (we could use logging to help find out which!)
30 agosto 200759
See the trees...
● MVC!
30 agosto 200760
Testing● Helps clarify what the problem is● Helps clarify where the problem is
– (or isn't)
● Also– vampires are repelled by tests!
30 agosto 200761
Interactive Testing● Perl has a debugger● Handy for running little snippets of code to
check how it works
30 agosto 200762
Interactive Testing~$ perl debug
Enter h or `h h' for help, or `man perldebug' for more help.
main::(e:1): bug DB<1> @a = (1,2,3,4,5)
DB<2> splice(@a, 2,1) = ("one", "two");Can't modify splice in scalar assignment at (eval 18)[/usr/share/perl/5.8/perl5db.pl:628] line 2, at EOF
DB<3> splice(@a, 2,1, "one", "two");
DB<4> x @a 0 1 1 2 2 'one' 3 'two' 4 4 5 5
30 agosto 200763
Interactive Debugging● Perl's debugger can also do much more
– Load modules– Set breakpoints– Step through file, etc.
● It could also be connected to remotely, for example in Apache sessions
● see also Devel::ebug● ... and Devel::REPL
30 agosto 200764
Mapping the Forest
64
30 agosto 200765
Mapping the Forest● Documentation!
– not just a boring task– it's a way to make sure the piggies don't get eaten
by wolves next time
● POD is good!● but
# Watch out! We're doing this because # otherwise {...} would happen!
is good too.
30 agosto 200766
Timing (bonus)
The Forest
what's the
hurry?
66
30 agosto 200767
Timing the piggies● By hand
– my $timer = time;
– ...do stuff...– my $delta = time $timer;
30 agosto 200768
Timing the piggies (accurately)
● By hand– use Time::HiRes qw(time);
– my $timer = time;
– ...do stuff...– my $delta = time $timer;
30 agosto 200769
What to optimize● Sometimes a script/subroutine may not be the
guilty party● We should measure...
– everything?– by hand?
30 agosto 200770
What to optimize● Devel::DProf
– Show which subroutines took most time to run in your script
30 agosto 200771
test.pl#!/usr/bin/perluse strict; use warnings;
print_hello();do_do_sleep();
sub print_hello { for (1..100_000) { print "HELLO"; }}
sub do_do_sleep { for (1..2) { do_sleep(); }}sub do_sleep { sleep 1;}
30 agosto 200772
running test.pl under DProf● $ perl d:DProf test.pl
● HELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLOHELLO
30 agosto 200773
dprofpp$ dprofpp Total Elapsed Time = 2.379776 Seconds User+System Time = 0.059776 SecondsExclusive Times%Time ExclSec CumulS #Calls sec/call Csec/c Name 66.9 0.040 0.040 1 0.0400 0.0400 main::print_hello 16.7 0.010 0.010 3 0.0033 0.0033 main::BEGIN 0.00 0.000 1 strict::bits 0.00 0.000 1 strict::import 0.00 0.000 1 warnings::import 0.00 0.000 1 bytes::import 0.00 0.000 1 DynaLoader::dl_load_fi 0.00 0.000 1 DynaLoader::dl_undef_s 0.00 0.000 1 DynaLoader::dl_find_sy 0.00 0.000 1 DynaLoader::dl_install 0.00 0.000 1 warnings::BEGIN 0.00 0.000 1 Data::Dumper::bootstra
30 agosto 200774
dprofpp timings● Note that sleep wasn't counted
– real times not counted, just CPU of local machine– sleep doesn't affect local CPU– neither does remote DB call !
30 agosto 200775
dprofpp -r (real)$ dprofpp rTotal Elapsed Time = 2.379776 Seconds Real Time = 2.379776 SecondsExclusive Times%Time ExclSec CumulS #Calls sec/call Csec/c Name 84.4 2.010 2.010 2 1.0050 1.0050 main::do_sleep 14.7 0.350 0.350 1 0.3500 0.3500 main::print_hello 0.42 0.010 0.010 1 0.0100 0.0100 warnings::BEGIN 0.42 0.010 0.010 1 0.0100 0.0100 overload::BEGIN 0.00 0.000 1 strict::bits 0.00 0.000 1 strict::import 0.00 0.000 1 warnings::import 0.00 0.000 1 bytes::import 0.00 0.000 1 DynaLoader::dl_load_fi 0.00 0.000 1 DynaLoader::dl_undef_s 0.00 0.000 1 DynaLoader::dl_find_sy 0.00 0.000 1 DynaLoader::dl_install 0.00 0.000 1 warnings::BEGIN
30 agosto 200776
dprofpp -r timings● Note that do_do_sleep wasn't counted
– inclusive times not counted (child calls). Just the time a single subroutine takes to run.
– Usually, you want to know what all of your subroutine was doing!
30 agosto 200777
dprofpp -rI (real, inclusive)
$ dprofpp rITotal Elapsed Time = 2.379776 Seconds Real Time = 2.379776 SecondsInclusive Times%Time ExclSec CumulS #Calls sec/call Csec/c Name 84.4 2.010 2.010 2 1.0050 1.0050 main::do_sleep 84.4 2.010 1 2.0100 main::do_do_sleep 14.7 0.350 0.350 1 0.3500 0.3500 main::print_hello 0.84 0.020 3 0.0066 main::BEGIN 0.42 0.010 0.010 1 0.0100 0.0100 warnings::BEGIN 0.42 0.010 0.010 1 0.0100 0.0100 overload::BEGIN 0.00 0.000 1 strict::bits 0.00 0.000 1 strict::import 0.00 0.000 1 warnings::import 0.00 0.000 1 bytes::import 0.00 0.000 1 DynaLoader::dl_load_fi 0.00 0.000 1 DynaLoader::dl_undef_sy 0.00 0.000 1 DynaLoader::dl_find_sym
30 agosto 200778
DProf stability● Perl debugging hooks are... a little fragile● dprofpp is particularly so● Depending on the host/perl you may find
perl d:Whatever will crash up to ~20% of time
30 agosto 200779
DProf in web scripts● Not run from command line...● but we can add most command line flags to
shebang!● Change
– #!/usr/bin/perl– #!/usr/bin/perl d:DProf
● This works– (except the 20% of time that script will just crash)
30 agosto 200780
DProf in web scripts● By default, DProf will create a file
– $current_dir/tmon.out
● (This may be an issue if you are developing without individual sandboxes)
30 agosto 200781
DProf in web scripts (tmon.out)
– #!/usr/bin/perl – #!/usr/bin/perl d:DProf
30 agosto 200782
DProf in web scripts● There are other modules!
– Devel::SmallProf
– Devel::FastProf
– Devel::DProfLB (“Less Bad”)
– Apache::DProf (If running on mod_perl)
● Search “prof” on http://search.cpan.org– look at documentation– look at reviews!
30 agosto 200783
Profiling: what next?● You found a hotspot
– takes lots of time...– ... or called lots of times
● What next?– make the hotspot faster– find a way to rewrite it– PROFIT?
30 agosto 200784
Profiling: what next?● You found a hotspot
– takes lots of time...– ... or called lots of times
● What next?– make the hotspot faster– find a way to rewrite it
– make sure everything still works!
30 agosto 200785
testing optimization● things to test
– 1) does it still work– 2) do other things that used the same code still
work– 3) is it faster
● 1. you can do by hand● 2. it's nicer to have automated tests
30 agosto 200786
testing optimization● old version of conv
sub conv { my $id = shift;
my %hash = ( 0 => 'a', 1 => 'b', 2 => 'c', 3 => 'd', 4 => 'e', 5 => 'f', 6 => 'g', 7 => 'h', 8 => 'i', 9 => 'l' ); my @single = split //, $id; my $res; foreach (@single){ $res = $hash{$_}.$res; } return $res;}
30 agosto 200787
testing optimization● new version of conv
sub conv { my $id = shift; $id=~tr/0123456789//cd;
# delete things not in the range $id=~tr/0123456789/abcdefghil/d;
# do the transliteration return reverse $id;}
30 agosto 200788
testing optimization● The testuse Test::More tests=>1;
# test the "conv" function in sn_contacts/main.chm
is (conv( "0123" ), "dcba" );is (conv( "hello 0123 world" ), "dcba" ); # sanity checkis (conv( "890" ), "ali" );is (conv( "0123456789" ), "lihgfedcba" );
is (conv2( "0123" ), "dcba" );is (conv2( "hello 0123 world" ), "dcba" ); # sanity checkis (conv2( "890" ), "ali" );is (conv2( "0123456789" ), "lihgfedcba" );
30 agosto 200789
testing optimization (3. is it faster?)
use Benchmark qw(:all);use Test::More tests => 1;
my @LIST = '0000'..'9999';
is_deeply( [ conv1(@LIST) ], [ conv2(@LIST) ], "structures are identical",);
sub do_conv1 { return map { conv( $_ ); } @_;}sub do_conv2 { return map { conv2( $_ ); } @_;}
30 agosto 200790
testing optimization (3. is it faster?)
my $results = timethese( 1_000_000, { conv1 => \&conv1, conv2 => \&conv2, }, "none");cmpthese( $results );
30 agosto 200791
testing optimization (3. is it faster?)
$ perl bench.pl 1..1ok 1 structures are identical
Benchmark: timing 1000000 iterations of conv1, conv2... conv1: 1 wallclock secs ( 0.16 usr + 0.00 sys = 0.16 CPU) @ 6250000.00/s (n=1000000) (warning: too few iterations for a reliable count) conv2: 0 wallclock secs ( 0.16 usr + 0.00 sys = 0.16 CPU) @ 6250000.00/s (n=1000000) (warning: too few iterations for a reliable count)
Rate conv2 conv1conv2 6250000/s 0%conv1 6250000/s 0%
30 agosto 200792
testing optimization (3. is it faster?)
$ perl bench.pl 1..1ok 1 structures are identical
Benchmark: timing 1000000 iterations of conv1, conv2... conv1: 1 wallclock secs ( 0.16 usr + 0.00 sys = 0.16 CPU) @ 6250000.00/s (n=1000000) (warning: too few iterations for a reliable count) conv2: 0 wallclock secs ( 0.16 usr + 0.00 sys = 0.16 CPU) @ 6250000.00/s (n=1000000) (warning: too few iterations for a reliable count)
Rate conv2 conv1conv2 6250000/s 0%conv1 6250000/s 0%
30 agosto 200793
testing optimization (3. is it faster?)
● Benchmark compares on a micro level, running thousands (or millions) of times
● Now test on macro (single script)– Redo dprofpp, compare results
– (Or even save the old tmon.out)
Grazie
Web Developer @ Dada GroupDada Group is seeking Perl and/or PHP Developers to join our passionate & international Production Team.Multiple locations are available in Italy, Spain, the United States, Brazil & China, for on-site work.
Your profile: you have passion for the Internet, new technologies and systems integration. Moreover you have extensive experience developing applications for the web, especially in the consumer entertainment (web & mobile) field.
If you think you are up to the challenge, please send us your resume !!!
Contact: [email protected]: www.dada.net