1. Open Source Compiler Construction (for the JVM) Tom Lee Shine Technologies OSCON Java 2011

2. Overview

About the Java Virtual Machine 3. About Scala 4. About Apache BCEL 5. A Generalized Compiler Architecture 6. Introducing Awesome 7. Writing a Stub Code Generator for Awesome 8. Abstract Syntax Trees (ASTs) 9. Parsing Awesome with Scala 10. What Next?

11. About the Java Virtual Machine

Stack-based machine.

• Operands pushed onto a stack, operated upon by opcodes.

Heavily influenced by Java The Language. 12. Allows interop between various source languages. 13. JVM bytecode is architecture-independent. 14. Multiple implementations.

• Oracle / HotSpot 15. Apache / Harmony 16. OpenJDK 17. Many more...

18. About Scala

http://www.scala-lang.org/ 19. Functional/OOP hybrid programming language

• First class functions 20. Pattern matching 21. Classes & traits 22. Etc.

Runs on the JVM.

• And .NET.

Own standard library (on top of the Java standard library).

23. About Apache BCEL

http://jakarta.apache.org/bcel/ 24. Bytecode Engineering Library 25. Emit JVM bytecode with a reasonably straightforward API. 26. Also supports reading class files, modifying classes, etc.

• But we won't use that stuff here.

27. A Generalized Compiler Architecture

Scannermatches patterns in source code, outputstokens. 28. Parserorganizes tokens into anAbstract Syntax Tree (AST).

• Or aParse Tree .

Semantic checks or optimizations of the AST may occur here. 29. Code Generatortraverses the AST to produce target code.

30. Introducing Awesome program::= ( expression';')* expression::=sum sum::=product(('+' | '-')product )* product::=number(('*' | '/')number )* number::= /[0-9]+/ The Awesome compiler will generate JVM bytecode to display each parsed expression. (Awesome, eh?) 31. Writing a Stub Code Generator for Awesome

Work backwards andwrite a stub code generator first .

• Immediate feedback! 32. A solid foundation on which to build the rest of the compiler.

Generate a Hello World class file withBCEL . 33. We'll make the code generator do real stuff later.

34. Abstract Syntax Trees (ASTs)

A logical, in-memory representation of the source program. 35. Constructed by the parser. 36. Semantic checks and optimizations possible at the AST level.

• Outside the scope of this presentation sorry!

Let's add the beginnings of an AST to our compiler.

37. Parsing Awesome with Scala

Useparser combinators .

• Combine small parsing functionstogether to describe a language. 38. Scannerandparser wrt the Generalized Compiler Architecture.

Describe languages using something of a pseudo-EBNF. 39. You can write your own parsers too.

• e.g. for string literals.

We'lluse built-in parsersto identify integers & semi-colons.

40. The Awesome Grammar Revisited program::= ( expression';')* expression::=sum sum::=product(('+' | '-')product )* product::=number(('*' | '/')number )* number::= /[0-9]+/ 41. What next?

Variables? 42. Conditional logic? 43. For/while loops? 44. Function calls?

45. Summary

Write the code generator first for instant gratification. 46. Use Scala parser combinators to build an AST from your input. 47. Update the code generator to walk the AST, emitting equivalent bytecode for each node using BCEL. 48. Iterate & add new features!

49. That's All, Folks! Shine Technologies wwwhttp://www.shinetech.com githubshinetech twitter@realshinetech Tom Lee wwwhttp://tomlee.coemail me@tomlee.cogithub thomaslee twitter@tglee All code from this presentation will be available from http://github.com/thomaslee/oscon2011-awesome