【16-B-3】パネルディスカッション「エンジニアが創るプロダクトの未来 ~エンジニアからプロダクトマネージャーへ~」 #devsumi #devsumiB
これからのPerlプロダクトのかたち(YAPC::Asia 2013)
-
Upload
goccy -
Category
Technology
-
view
5.297 -
download
0
Transcript of これからのPerlプロダクトのかたち(YAPC::Asia 2013)
A brand new way of Perl product, and it’s future
Masaaki Goshima (@goccy54)mixi Inc.
Me
• Name : Masaaki Goshima (@goccy54)• Job : mixi (Tanpopo Group)– Development for Developers– Developing a platform for refining legacy
software– Managing Jenkins
• Personally, interested in Perl and developing yet another Perl, gperl– (YAPC::Asia 2012) Perl と出会い Perl を作る
gperl
• Fastest perl-like language– Aiming perl5 compatible syntax–Written with C++
• The last commit for repository was 11 months ago...– “Is this already departed !?”
NO! ITS STILL ALIVE!!!
• However, gperl itself has departed• Modules(lexical analyzer, parser,
code generator, interpreter) are useful– Lexical analyzer: implementing perl5
syntax highlighter– Parser: static analysis tool (used as
refinement of legacy perl5 code)
Input -> Lexer -> Parser -> CodeGenerator -> VM (JIT) -> output
This presentation’s Goal
• Today, we’re going to present “Spinout projects” for each modules
gperl
Compiler::Lexer Compiler::Parser Compiler::CodeGenerator::LLVM
LexerParser CodeGenerator
???::???::??? ?????Next Module
Agenda
• Section 1– Introduction of Compiler::* Modules
• Section 2 – (Application1) Running on multi
platforms
• Section 3 – (Application2) Static analysis tool
Section1
• Introduction of Compiler::* Modules1. Compiler::Lexer – Lexical Analyzer for Perl5
2. Compiler::Parser – Create Abstract Syntax Tree for Perl5
3. Compiler::CodeGenerator::LLVM– Create LLVM IR for Perl5
Compiler::Lexer
• Lexical Analyzer for Perl5• Features
– Simple (return Array Reference of Token Object)– Fast (faster 10 ~ 100 times than PPI::Tokenize)
• Wirtten with C++• Perfect hashing for reserved keywords• memory pool for token objects
– Readable Code• Nothing use parser generator like yacc/bison
Example1: my $v = sprintf(<<'TEMPLATE', 'yapc');2: %s::Asia3: TEMPLATE4: $v =~ s#yapc#YAPC#;
%s::Asia : HereDocumentTEMPLATE : HereDocumentEnd$v : Var=~ : RegOKs : RegReplace# : RegDelimyapc : RegReplaceFrom# : RegMiddleDelimYAPC : RegReplaceTo# : RegDelim; : SemiColon
my : VarDecl$v : LocalVar= : Assignsprintf : BuiltinFunc( : LeftParenthesis<< : LeftShiftTEMPLATE : HereDocumentRawTag, : Commayapc : RawString) : RightParenthesis; : SemiColon
tokenize
Tips
• Cannot tokenize pattern1. func * v • 「 * 」 is Glob or Mul
2. func / ..• 「 / 」 is RegexDelimiter or Div
3. func <<FLAG• 「 FLAG 」 is HereDocumentTag or Constant
It may make a mistake
Future plan
• Supporting recursively tokenizing–More wide range of parsing application– Recursive parsing will take time,
optimizing will be next future work
func * v => func(*v) or func() * vfunc / … => func(/../) or func() / vfunc <<FLAG => func(<<FLAG) or func() << FLAG
sub func($) {} or sub func() {}
Compiler::Parser
• Create Abstract Syntax Tree for Perl5• Features– Fast (faster than PPI)– Readable Code• Simple design• Nothing use generator
Can generate Virtual Machine code as walking tree by post order
->
$a->{b}->[0]->c(@args)
->
->
$a {}
b
[]
c
0
@args
left
left
left
right
argsright
right
data
data
Example
my $v = sin $v + $v * $v / $v - $v && $v
my ($v = (sin((($v + (($v * $v) / $v)) - $v)) && $v))
=
$v &&
left right
rightleft
The most difficult part of Perl5 parser:
hidden(optional) parenthesis
Example2) Can parse BlackPerl
BEFOREHAND: close door, each window & exit;wait until time;open spell book; study;read (spell, $scan, select); tell us;write it, print(the hex) while each watches, reverse length, write again;kill spiders, pop them, chop,split, kill them. unlink arms, shift,wait and listen (listening, wait).sort the flock (then,warn "the goats", kill "the sheep");kill them, dump qualms,shift moralities, values aside, each one;die sheep; die (to, reverse the => systemyou accept (reject, respect));next step, kill next sacrifice,each sacrifice, wait, redo ritualuntil "all the spirits are pleased";do it ("as they say").do it(*everyone***must***participate***in***forbidden**s*e*x*).return last victim; package body;
exit crypt(time, times & “half a time”)& close it.select (quickly) andwarn next victim;AFTERWARDS: tell nobody.wait, wait until time;wait until next year, next decade;sleep, sleep, die yourself,die @last
BlackPerl for Perl5
Future plan
• Replace PPI !!!– Supporting some expressions• ThreeTermOperator, Glob, given/when, goto
etc..
– Supporting compatible methods PPI provides• PPI::Document::find• PPI::Document::prune
Compiler::CodeGenerator::LLVM
• Create LLVM IR for Perl5
->
$a->{b}->[0]->c(@args)
->
->
$a {}
b
[]
c
0
@args
left
left
left
right
argsright
right
data
data1 2
3
4 5
6
7 8
9
10Compiler::Parser Compiler::CodeGenerator::LLVM
LLVM IR
Running with JIT
Native Code
Other Language
LLVM(Low Level Virtual Machine)
• Compiler Infrastructure• Better than GNU GCC
LLVM
X86
ARM
Power
C/C++/Objective-C
JavaScriptpy2llvm
MacRuby
clang
?? ?
How to use
• Dependency: clang/llvm version 3.2 or 3.3
use Compiler::Lexer;use Compiler::Parser;use Compiler::CodeGenerator::LLVM;
my $tokens = Compiler::Lexer->new('')->tokenize($code);my $ast = Compiler::Parser->new->parse($tokens);my $generator = Compiler::CodeGenerator::LLVM->new();my $llvm_ir = $generator->generate($ast); # generate LLVM IR$generator->debug_run($ast); # run with JIT
Benchmark(Fibonacci)
Languagetime(real
)[sec] X
gperl 0.10 X81.2
LuaJIT(2.0.0-beta10) 0.12 X67.6
v8(0.8.0) 0.18 X45.1
Compiler::CodeGenerator::LLVM 0.72 X11.2
Perl(5.16.0) 8.12 X1.0
Environment
MacOSX(10.8.4)2.2GHz Intel Core
i7Memory: 8GB
N=35
Section 2
• Application1)– Running on multi platforms
!?
1. Running on Web Browser
• Recently, way of writing code that runs on web browser is not only JavaScript– e.g.) CoffeScript, JSX, TypeScript, Dart
• I wrote module for Perl5!
SEE ALSO : perl.js (@gfx)
Compiler::Tools::Transpiler
• Translate Perl5 codes to JavaScript codes– (Step1) translate Perl5 codes to LLVM-IR with
Compiler::* modules– (Step2) translate LLVM-IR to JavaScript codes
with emscripten
Com
pile
r::Le
xer
Com
pile
r::Pa
rser
Com
pile
r::Co
deGe
nera
tor::
LLVM
Perl5 LLVM emscripten
How to use
use Compiler::Tools::Transpiler;
my $transpiler = Compiler::Tools::Transpiler->new({ engine => 'JavaScript’, library_path => [‘lib’]});
open my $fh, '<', 'target_application.pl';my $perl5_code = do { local $/; <$fh> };my $javascript_code = $transpiler->transpile($perl5_code);
open $fh, '>', 'target_application.js';print $fh $javascript_code;close $fh;
※ Needs clang-3.2 and LLVM-3.2 (Not ver. 3.3 or later) because emscripten has still not support LLVM 3.3 or later
ROADMAP/Repository
• Fix some emscripten’s bugs• Supporting accessors for HTML objects• Supporting Canvas API• etc..
• Repository– https://github.com/goccy/p5-Compiler-Tools-Transpiler.git
2. Running on iOS and OSX
• Recently, software that can develop iOS or OSX applications using light weight language has being released– RubyMotion– mocl– Titanium
• I wrote module for Perl5!
PerlMotioniOS and OS X development
using the Perl5 programming language
PerlMotion’s Architecture
UIKit
Cocoa Touch FrameworksCore Animation Core Audio Core Data
Perl5 x Cocoa Touch FrameworksBindings Library
LLVM IR(Links Perl5 codes and Cocoa Touch Frameworks)
Perl5 Codes
Compiler::Lexer
Compiler::Parser
Compiler::CodeGenerator::LLVM
iOS Simulator iOS DeviceMacOSX
Generates Native Code for each target Architecture
DEMO
• HelloWorld by PerlMotion
Current Status
Still not support functions
Symbol Management System : GlobGarbage Collection
RegexpThree Term Operator
etc..
Not support functionsDynamic evaluation like eval,
s/../../e, require
Supported functions
Primitive Types : Int, double, String, Array, Hash, ArrayRefence,
HashReference,IOHandler, BlessedObject …
Operators : binary operator, single term operator …
BuiltinFunction : print, say, shift, push, sin, open …
Variable Definition, Function Definition
OOP System : package, bless, @ISA
ROADMAP
• Supporting deployment on device• Preparing debugging environment– (stdout/stderr/gdb..etc)
• Supporting existing framework (e.g. UIKit)• Supporting CPAN modules written by Pure
Perl
Support CPAN modules
Support some Frameworks
Support deployment on device
Support debugging environment
2014/4
Will release at April, 2014!
Welcome your Contribution!
• I want discuss design or objective– Especially, I need advice on “What Perl
community likes?” (Naming rule, etc…)
https://github.com/goccy/perl-motion.git
Conclusion at Section 1 and 2
• Perl5 codes Run Anywhere!!!– iOS,OSX,Web Browser and others
• Welcome your Contribution– Compiler::Lexer
• https://github.com/goccy/p5-Compiler-Lexer.git
– Compiler::Parser• https://github.com/goccy/p5-Compiler-Parser.git
– Compiler::CodeGenerator::LLVM• https://github.com/goccy/p5-Compiler-CodeGenerator.git
– Compiler::Tools::Transpiler• https://github.com/goccy/p5-Compiler-Tools-Transpiler.git
– PerlMotion• https://github.com/goccy/perl-motion.git
• Application2)– Static analysis tool
Section3
Compiler::Tools::CopyPasteDetector
• Detect Copy and Paste for Perl5• Features– Fast (detecting Engine written by C++
with POSIX Thread)– Accuracy (based on Compiler::Lexer
and B::Deparse)–High functionality Visualizer
(some metrics or scattergram etc…)
DEMO
• Detect Copy & Paste of Catalyst and Mojolicious
Another Modules
• Perl::MinimumVersion::Fast(@tokuhirom)– Find a minimum required version of perl for
Perl code
• Test::LocalFunctions::Fast(@papix)– Detect unused local function
Compiler::Lexer or Compiler::Parser provide that you can write faster modules than existing module that uses PPI
Future plan
• Perl::Metrics::Simple::Fast–Will release faster module than
Perl::Metrics::Simple by using Compiler::Lexer and Compiler::Parser
• Next future, I will CPANize of static alnalysis module that has been used at mixi– It includes rich Visualizer
Conclusion at Section 3
• Introduction of copy and paste detecting tool for Perl5– Compiler::Tools::CopyPasteDetector
• Compiler::Lexer or Compiler::Parser provide that you can write faster modules than existing module that uses PPI
Finally
Compiler::Lexer Compiler::Parser Compiler::CodeGenerator::LLVM
gperl will reborn
Type Inference Engine
Static Typing
- Coming soon -gperl will assemble these brand new modules...