Post on 10-May-2015
description
The Perl APIfor the
Mortally Terrified
Mike Friedman
YAPC::NA 2014
Orlando, FL
@friedo
This talk is the product of a lengthy journey.
The Perl APIWhat’s it good for?
The Perl APIWhat’s it good for?
Make things fast:
The Perl APIWhat’s it good for?
Make things fast: - Write your slow Perl code in C
The Perl APIWhat’s it good for?
Make things fast: - Write your slow Perl code in C - Keep your fast Perl code in Perl
The Perl APIWhat’s it good for?
Make Perl more useful:
The Perl APIWhat’s it good for?
Make Perl more useful: - Interface with libraries written in C
The Perl APIWhat’s it good for?
Make Perl more useful: - Interface with libraries written in C - Create modules exposing external APIs to Perl
The Perl APIWhat’s it good for?
Make Perl do...stuff
The Perl APIWhat’s it good for?
Make Perl do...stuff - Change opcodes on the fly?
The Perl APIWhat’s it good for?
Make Perl do...stuff - Change opcodes on the fly? - Extend the parser?
The Perl APIWhat’s it good for?
Make Perl do...stuff - Change opcodes on the fly? - Extend the parser? - Make exceptions work?
The Perl APIWhat’s it good for?
Make Perl do...stuff - Change opcodes on the fly? - Extend the parser? - Make exceptions work? - Why not?
The Perl APIWhat’s it good for?
Make Perl do...stuff - Change opcodes on the fly? - Extend the parser? - Make exceptions work? - Why not?
XS
What is XS?
perldoc perlxs:
XS is an interface description file format used to create an extension interface between Perl and C code (or a C library) which one wishes to use with Perl. The XS interface is combined with the library to create a new library which can then be either dynamically loaded or statically linked into perl. The XS interface description is written in the XS language and is the core component of the Perl extension interface.
XS is a specialized templating language for C.
Some XS Code
Some XS Code
XS Directives
Some C Code made by xsubpp
XS != Perl API
For most things
For most things
You don't actuallyhave to use XS.
(OK. You don’t have to know that you’re using it.)
Inline::C
use Inline C;hello('Mike');hello('Cookie Monster');
__END__
__C__void hello(char* name) { printf("Hello %s!\n", name);}
use Inline C;hello('Mike');hello('Grover');
__END__
__C__void hello(char* name) { printf("Hello %s!\n", name);}
use Inline C;hello('Mike');hello('Grover');
__END__
__C__void hello(char* name) { printf("Hello %s!\n", name);}
use Inline C;hello('Mike');hello('Grover');
__END__
__C__void hello(char* name) { printf("Hello %s!\n", name);}
Inline::C Code in Perl
Inline::C Code in Perl
Digest::MD5
Inline::C Code in Perl
Digest::MD5
C Code Parse::RecDescent
Inline::C Code in Perl
Digest::MD5
Create build in a temp dir
C Code Parse::RecDescent
Inline::C Code in Perl
Digest::MD5
Create build in a temp dir
C Code Parse::RecDescent
Write XS
Inline::C Code in Perl
Digest::MD5
Create build in a temp dir
C Code Parse::RecDescent
Write XS
Compile and install
Inline::C Code in Perl
Digest::MD5
Create build in a temp dir
C Code Parse::RecDescent
Write XS
Compile and install
�
How do we make Perl do stuff in C?
How do we make Perl do stuff in C?
We need to understand Perl’s data structures.
Handwaving Ahead
A $scalar in Perl is represented by a struct called SV.
SvNULL
SvRV SvNVSvPV SvIV
(These are not all of them.)
What does SV look like?
ANY
What does SV look like?
Points to a struct with the scalar's metadata.
ANY
What does SV look like?
What does SV look like?
REFCNT
What does SV look like?
Reference count for garbage collection
REFCNT
What does SV look like?
What does SV look like?
FLAGS TYPE
What does SV look like?
Flags and type information
FLAGS TYPE
What does SV look like?
What does SV look like?
SV_U
What does SV look like?
Points to a union with the actual scalar's data
SV_U
What does SV look like?
What does SV look like?
What does SV look like?
The simplest scalar:undef
What does SV look like?
The simplest scalar:undef
The "ANY" pointer is NULL, and TYPE is zero.
undef is, like, totes boring.
What about strings?
What does SV look like?
What does SV look like?
Strings
What does SV look like?
Strings
"ANY" points to a "xpv", and TYPE is 4. *
* in Perl 5.20.
What does SV look like?
Strings
"ANY" points to a "xpv", and TYPE is 4. *
* in Perl 5.20.
SV becomes SvPV
What does SV look like?
What does SV look like?
Integers
What does SV look like?
Integers
"ANY" points to a "xpviv", and TYPE is 1. *
* in Perl 5.20.
What does SV look like?
Integers
"ANY" points to a "xpviv", and TYPE is 1. *
* in Perl 5.20.
SV becomes SvIV
What does SV look like?
What does SV look like?
Floats
What does SV look like?
Floats
"ANY" points to a "xpvnv", and TYPE is 2. *
* in Perl 5.20.
What does SV look like?
Floats
"ANY" points to a "xpvnv", and TYPE is 2. *
* in Perl 5.20.
SV becomes SvNV
WRITE CODE NOW?!??!!11!?
Let's make something fast.
#!/usr/bin/env perluse strict;use warnings;use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;
print 'Give me another number: ';my $num_b = <STDIN>;
printf "The sum is: %s\n", add( $num_a, $num_b );__END____C__
int add( int a, int b ) { return a + b;}
#!/usr/bin/env perluse strict;use warnings;use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;
print 'Give me another number: ';my $num_b = <STDIN>;
printf "The sum is: %s\n", add( $num_a, $num_b );__END____C__
int add( int a, int b ) { return a + b;}
#!/usr/bin/env perluse strict;use warnings;use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;
print 'Give me another number: ';my $num_b = <STDIN>;
printf "The sum is: %s\n", add( $num_a, $num_b );__END____C__
int add( int a, int b ) { return a + b;}
#!/usr/bin/env perluse strict;use warnings;use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;
print 'Give me another number: ';my $num_b = <STDIN>;
printf "The sum is: %s\n", add( $num_a, $num_b );__END____C__
int add( int a, int b ) { return a + b;}
#!/usr/bin/env perluse strict;use warnings;use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;
print 'Give me another number: ';my $num_b = <STDIN>;
printf "The sum is: %s\n", add( $num_a, $num_b );__END____C__
int add( int a, int b ) { return a + b;}
#!/usr/bin/env perluse strict;use warnings;use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;
print 'Give me another number: ';my $num_b = <STDIN>;
printf "The sum is: %s\n", add( $num_a, $num_b );__END____C__
int add( int a, int b ) { return a + b;}
bash-3.2$ perl cexp1.plGive me a number: 42Give me another number: 11The sum is: 53
Run it!
bash-3.2$ perl cexp1.plGive me a number: 42Give me another number: 31.337The sum is: 73
Run it again!
bash-3.2$ perl cexp1.plGive me a number: 42Give me another number: 31.337The sum is: 73
Run it again!
What happened?
Inline::C is
CRAZY SMART
intintintint
Inline::C is
CRAZY SMART
Adding numbers:What will we need?
•SvIOK Is it a SvIV?
•SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?
•SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?
•SvIV Get the C int value from SV
•SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?
•SvIV Get the C int value from SV
•SvNV Get the C float value from SV
•SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?
•SvIV Get the C int value from SV
•SvNV Get the C float value from SV
•newSViv Make a new integer SV
•SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?
•SvIV Get the C int value from SV
•SvNV Get the C float value from SV
•newSViv Make a new integer SV
•newSVnv Make a new float SV
use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;print 'Give me another number: ';my $num_b = <STDIN>;$num_a += 0; $num_b += 0;printf "The sum is: %s\n", add( $num_a, $num_b );__END____C__SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) { return newSViv( SvIV( a ) + SvIV( b ) ); } else if ( SvNOK( a ) && SvNOK( b ) ) { return newSVnv( SvNV( a ) + SvNV( b ) ); } else { croak( "I don't know what to do!" ); }}
use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;print 'Give me another number: ';my $num_b = <STDIN>;$num_a += 0; $num_b += 0; # coerceprintf "The sum is: %s\n", add( $num_a, $num_b );__END____C__SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) { return newSViv( SvIV( a ) + SvIV( b ) ); } else if ( SvNOK( a ) && SvNOK( b ) ) { return newSVnv( SvNV( a ) + SvNV( b ) ); } else { croak( "I don't know what to do!" ); }}
use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;print 'Give me another number: ';my $num_b = <STDIN>;$num_a += 0; $num_b += 0;printf "The sum is: %s\n", add( $num_a, $num_b );__END____C__SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) { return newSViv( SvIV( a ) + SvIV( b ) ); } else if ( SvNOK( a ) && SvNOK( b ) ) { return newSVnv( SvNV( a ) + SvNV( b ) ); } else { croak( "I don't know what to do!" ); }}
use Inline 'C';
print 'Give me a number: ';my $num_a = <STDIN>;print 'Give me another number: ';my $num_b = <STDIN>;$num_a += 0; $num_b += 0;printf "The sum is: %s\n", add( $num_a, $num_b );__END____C__SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) { return newSViv( SvIV( a ) + SvIV( b ) ); } else if ( SvNOK( a ) && SvNOK( b ) ) { return newSVnv( SvNV( a ) + SvNV( b ) ); } else { croak( "I don't know what to do!" ); }}
bash-3.2$ perl cexp2.pl Give me a number: 42Give me another number: 11The sum is: 53bash-3.2$ perl cexp2.pl
Run it!
bash-3.2$ perl cexp2.pl Give me a number: 42Give me another number: 11The sum is: 53bash-3.2$ perl cexp2.pl
Run it!
bash-3.2$ perl cexp2.pl Give me a number: 53.4Give me another number: 6.54The sum is: 59.94
bash-3.2$ perl cexp2.pl Give me a number: 42Give me another number: 11The sum is: 53bash-3.2$ perl cexp2.pl
Run it!
bash-3.2$ perl cexp2.pl Give me a number: 53.4Give me another number: 6.54The sum is: 59.94
Give me a number: 42Give me another number: 6.54I don't know what to do! at cexp2.pl line 16, <STDIN> line 2.
Once you get this far, everything you need
is in perlapi.
Let's do something with arrays.
What does SV look like?
What does SV look like?
Arrays
What does SV look like?
Arrays
"ANY" points to a "xpvav"
What does SV look like?
Arrays
"ANY" points to a "xpvav"
SV becomes AV
Squaring numbers:What will we need?
•SvRV Get SV from reference
•SvRV Get SV from reference
•newAV Make a new array AV
•SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array
•SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvIOK Is it an int?
•SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvIOK Is it an int?
•SvIV Get C int from SV
•SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvIOK Is it an int?
•SvIV Get C int from SV
•av_push Obvs.
#!/usr/bin/env perluse strict;use warnings;use Inline 'C';
print "Give me some numbers: ";my @numbers = map { $_ += 0 } split /,/, <STDIN>;
my $result = squares( \@numbers );printf "The squares are: %s\n", join ", ", @$result;
__END__
#!/usr/bin/env perluse strict;use warnings;use Inline 'C';
print "Give me some numbers: ";my @numbers = map { $_ += 0 } split /,/, <STDIN>;
my $result = squares( \@numbers );printf "The squares are: %s\n", join ", ", @$result;
__END__
__C__
SV *squares( SV *numbers ) { AV *array = (AV *)SvRV( numbers ); AV *return_array = newAV(); SV **tmp;
int len = AvFILL( array ); int i, val;
for( i = 0; i <= len; i++ ) { tmp = av_fetch( array, i, 1 ); if( !SvIOK( *tmp ) ) { croak( "Can't handle this value!" ); } val = SvIV( *tmp ); val = val * val;
av_push( return_array, newSViv( val ) ); }
return newRV_inc( (SV *)return_array );}
__C__
SV *squares( SV *numbers ) { AV *array = (AV *)SvRV( numbers ); AV *return_array = newAV(); SV **tmp;
int len = AvFILL( array ); int i, val;
for( i = 0; i <= len; i++ ) { tmp = av_fetch( array, i, 1 ); if( !SvIOK( *tmp ) ) { croak( "Can't handle this value!" ); } val = SvIV( *tmp ); val = val * val;
av_push( return_array, newSViv( val ) ); }
return newRV_inc( (SV *)return_array );}
__C__
SV *squares( SV *numbers ) { AV *array = (AV *)SvRV( numbers ); AV *return_array = newAV(); SV **tmp;
int len = AvFILL( array ); int i, val;
for( i = 0; i <= len; i++ ) { tmp = av_fetch( array, i, 1 ); if( !SvIOK( *tmp ) ) { croak( "Can't handle this value!" ); } val = SvIV( *tmp ); val = val * val;
av_push( return_array, newSViv( val ) ); }
return newRV_inc( (SV *)return_array );}
__C__
SV *squares( SV *numbers ) { AV *array = (AV *)SvRV( numbers ); AV *return_array = newAV(); SV **tmp;
int len = AvFILL( array ); int i, val;
for( i = 0; i <= len; i++ ) { tmp = av_fetch( array, i, 1 ); if( !SvIOK( *tmp ) ) { croak( "Can't handle this value!" ); } val = SvIV( *tmp ); val = val * val;
av_push( return_array, newSViv( val ) ); }
return newRV_inc( (SV *)return_array );}
__C__
SV *squares( SV *numbers ) { AV *array = (AV *)SvRV( numbers ); AV *return_array = newAV(); SV **tmp;
int len = AvFILL( array ); int i, val;
for( i = 0; i <= len; i++ ) { tmp = av_fetch( array, i, 1 ); if( !SvIOK( *tmp ) ) { croak( "Can't handle this value!" ); } val = SvIV( *tmp ); val = val * val;
av_push( return_array, newSViv( val ) ); }
return newRV_inc( (SV *)return_array );}
__C__
SV *squares( SV *numbers ) { AV *array = (AV *)SvRV( numbers ); AV *return_array = newAV(); SV **tmp;
int len = AvFILL( array ); int i, val;
for( i = 0; i <= len; i++ ) { tmp = av_fetch( array, i, 1 ); if( !SvIOK( *tmp ) ) { croak( "Can't handle this value!" ); } val = SvIV( *tmp ); val = val * val;
av_push( return_array, newSViv( val ) ); }
return newRV_inc( (SV *)return_array );}
__C__
SV *squares( SV *numbers ) { AV *array = (AV *)SvRV( numbers ); AV *return_array = newAV(); SV **tmp;
int len = AvFILL( array ); int i, val;
for( i = 0; i <= len; i++ ) { tmp = av_fetch( array, i, 1 ); if( !SvIOK( *tmp ) ) { croak( "Can't handle this value!" ); } val = SvIV( *tmp ); val = val * val;
av_push( return_array, newSViv( val ) ); }
return newRV( (SV *)return_array );}
bash-3.2$ perl cexp3.pl Give me some numbers: 4,5,6,23The squares are: 16, 25, 36, 529
Run it!
Let's do something with hashes.
What does SV look like?
What does SV look like?
Hashes
What does SV look like?
Hashes
"ANY" points to a "xpvhv"
What does SV look like?
Hashes
"ANY" points to a "xpvhv"
SV becomes HV
Finding uniques:What will we need?
•SvRV Get SV from reference
•SvRV Get SV from reference
•newHV Make a new hash HV
•SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array
•SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvPOK Is it a string?
•SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvPOK Is it a string?
•SvPV Get C string from SV
•SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvPOK Is it a string?
•SvPV Get C string from SV
•hv_store Obvs.
#!/usr/bin/env perluse strict;use warnings;use Inline 'C';
print "Give me some words: ";my @words = split /,/, <STDIN>;chomp @words;my $result = uniques( \@words );printf "The uniques are: %s\n", join ", ", %$result;
__END__
__C__
SV *uniques( SV *words ) { AV *array = (AV *)SvRV( words ); HV *result = newHV(); SV **tmp;
int len = AvFILL( array ); int i; char *val;
for( i = 0; i <= len; i++ ) { tmp = av_fetch( array, i, 1 ); if( !SvPOK( *tmp ) ) { croak( "Can't handle this value!" ); }
val = SvPV_nolen( *tmp );
hv_store( result, val, strlen( val ), newSV(0), 0 ); }
return newRV( (SV *)result );}
bash-3.2$ perl cexp4.pl Give me some words: foo,bar,baz,baz,foo,quux,narf,poitThe uniques are: bar, narf, baz, poit, quux, foo
Run it!
What isMagic?
Not that terrifying.
Magic is a linked list of function pointers attached to a *SV.
Magic is a linked list of function pointers attached to a *SV.
The functions implement custom read/write behavior.
Magic is a linked list of function pointers attached to a *SV.
The functions implement custom read/write behavior.
Special vars like %ENV are implemented via Magic.
$ENV{foo} = "blah";
$ENV{foo} = "blah";
assignment op
$ENV{foo} = "blah";
Magic on %ENV ?
assignment op
$ENV{foo} = "blah";
Magic on %ENV ?
assignment op
Look up write function
$ENV{foo} = "blah";
Magic on %ENV ?
assignment op
Look up write function
call libc setenv()
tie() is also implemented via Magic.
Terror defeated!
Where do we go from here?
Resources:
perlgutsFor details on how to manipulate Perl's
data structures.
Resources:
perlapiFor a comprehensive reference to (almost)
everything you need to make perl do stuff.
Resources:
Perlguts Illustrated
For pretty diagrams and explication.
Resources:
Inline::C Cookbook
For common solutions to problemsinterfacing between Perl and C
Resources:
Resources:
Code samples
http://github.com/friedo/perl-api-terror
YAPC::NA
Thank you.
Mike Friedmanfriedo@friedo.com
http://friedo.com/http://github.com/friedo/perl-api-terror