Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf ·...
Transcript of Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf ·...
![Page 1: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/1.jpg)
Grammatech, Meet Grammatech, Meet BasilBasil
(or “Geeks, Know Thine Selves”)
Jonathan RiehlGrammatech, Inc.
July 29th, 2005
![Page 2: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/2.jpg)
OutlineOutline
● Why I Deserve to Live(Or: What have I done, and what I want to do.)
● Please Give Me a Reason to Live(Or: Why you should care.)
● Please Give Me a Place to Live(Or: No, really I need a place to crash soon.)
![Page 3: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/3.jpg)
I Want to LiveI Want to Live
● What I have done.– GRAD– Paths– PyFront
● What I want to do.– More of the same, but with a catchier
name. (Or is that a more obscure name?)
![Page 4: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/4.jpg)
What I Have DoneWhat I Have Done
● GRAD - “Grammar Based Rapid Application Development”
● Paths● PyFront● And beyond...
![Page 5: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/5.jpg)
Back in the day...Back in the day...
● Shuttle flight design required a team of software developers.
● Problem: Large C/C++ simulation code base and custom input language.
● Solution: Python
![Page 6: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/6.jpg)
PythonPython
● Provided a simpler language that could be used by aerospace engineers.
● Could ideally leverage the Python interpreter for simulation input language.
● New problem: Legacy investment vs. migration costs.
![Page 7: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/7.jpg)
GRADGRAD
● Grammar Based Rapid Application Development– Given a language's grammar, we
could automatically generate a Python extension module for that language.
● Isn't this SWIG?
![Page 8: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/8.jpg)
GRAD vs. SWIGGRAD vs. SWIG
.hFile
GRAD
SWIG.iFile
WrapperCode
C++Code
C++Compiler
PythonExtensionModule
User modified
![Page 9: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/9.jpg)
Paths (GRAD/Paths)Paths (GRAD/Paths)
● Find out what your boss hates to do and specialize in it...
● Idea: Rapid application development should feature rapid application testing.
● Can leverage GRAD technology to do this.
![Page 10: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/10.jpg)
Rapid Application Rapid Application Testing RoadmapTesting Roadmap
● Step 1: Perform control flow analysis.– Identify structural test cases.– Instrument code for coverage
analysis.● Step 2: Perform data flow analysis.● Step 3: Generate test vectors.
(Ooops. Funding got cut.)
![Page 11: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/11.jpg)
PyFrontPyFront
● Meanwhile, back at comp.lang.python...
● Lot of talk about a Python compiler.● I was building dataflow models of both
Python and C/C++, what if I “reversed the stream” of the C/C++ analyzer?
● Originally called Basil, but was able to keep compiler semantics true to Python.
![Page 12: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/12.jpg)
And beyond...And beyond...
● Everything is starting to look like nails...
● We begin to see patterns which imply opportunities for reuse.
SourceCode
SyntaxTreeParser Back End
ProgramModel
![Page 13: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/13.jpg)
And now a whopping And now a whopping huge intermission.huge intermission.
● A life hangs in the balance, and this guy wants to get philisophical?!
![Page 14: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/14.jpg)
The N Languages The N Languages ProblemProblem
● Originally termed the 500 languages problem.
● But in a world where everyone has their own language, it isn't so outrageous to generalize to an arbitrary number of languages.
![Page 15: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/15.jpg)
In the Beginning...In the Beginning...
This, is GCC.
![Page 16: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/16.jpg)
And Then Came And Then Came Analysis...Analysis...
But, that's like, n*m components!
![Page 17: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/17.jpg)
Hence .NETHence .NET
Wow, we go from n*m to n+m.
![Page 18: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/18.jpg)
But Should That Be All?But Should That Be All?
We go back to n*k + k*m, but gain flexibility.(And avoid lock out...)
![Page 19: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/19.jpg)
What I Want to DoWhat I Want to Do
● Provide a framework for analysis and integration of multiple programming languages.
● Provide the ability to prototype new languages.
● These goals are complimentary.
![Page 20: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/20.jpg)
Language AnalysisLanguage Analysis
● Follow the original GRAD approach: start with parsers.
● Use these to build:– Control Flow Models– Data Flow Models– Object Models– Wacky User Defined Models
![Page 21: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/21.jpg)
Language IntegrationLanguage Integration
● So we have analysis, integration comes with applications of the analysis.
● These applications would support multiple languages right off the bat.
● Example: Provide a code generator for an object model and type map and you have SWIG.
![Page 22: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/22.jpg)
Language PrototypesLanguage Prototypes
● Since we are building language parsers, we already have to play nice with parser generators.
● Idea: Expose this infrastructure to assist with language grammar development.
![Page 23: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/23.jpg)
Project DesignProject Design
● Integrations– Gateways to domains beyond the
wainscoting and/or framework.
● Parser Generator Integrations● Language Integrations● Model Integrations● Applications
– The glue between integrations.
![Page 24: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/24.jpg)
Parser Generator Parser Generator IntegrationsIntegrations
GrammarInputFile
GrammarSyntaxTree
GrammarModel
GrammarInput Parser
GrammarGenerator
GrammarBack End
Applications
ParserGenerators
Partial Integrations for: Bison, pgen
![Page 25: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/25.jpg)
Language IntegrationsLanguage Integrations
ParserGenerators
LanguageParser
SourceCode
SyntaxTree
Applications
Current Parsers: Python, C
![Page 26: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/26.jpg)
Model IntegrationsModel Integrations
SyntaxTree
XMLDocument
OtherModel
ModelInstance
OtherApplications
LanguageBack End
XML to ModelMapping
ModelTranslator
![Page 27: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/27.jpg)
Modeling Madness!Modeling Madness!
MetaModel
Instance
Meta ModelCode Generator
ModelClass
Library
3rd Party ModelApplication
ModelInstance
![Page 28: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/28.jpg)
ApplicationsApplications
● Code generators– Translators, Compilers
● Model viewers and utilities– Paths– Test Generation
● Model editors– Grammarian
![Page 29: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/29.jpg)
For Instance: A M.S. For Instance: A M.S. ThesisThesis
● The Basil Test String Generator● Application that maps from Basil grammar
model to test strings (code generator).
TesterizerGrammarModel
TestStrings
EBNFGrammar
pgenGrammar
BisonGrammar
EBNFIntegration
pgenIntegration
BisonIntegration
![Page 30: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/30.jpg)
Put it all together...Put it all together...
SourceCode
Parser
Back EndsBack Ends
ProductsBack Ends
LanguageGrammar
GrammarWorkshop
SemanticWorkshop Models
GrammarModel
SyntaxTree
![Page 31: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/31.jpg)
The Grammar The Grammar WorkshopWorkshop
● Grammarian– Allows grammar models to be
emitted as meta-models.– Generates some tests.– Working on generating code.
● Take model and generate a Bison input that will output a syntax tree only – allows actions to be stripped.
![Page 32: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/32.jpg)
The Semantic WorkshopThe Semantic Workshop
● Have been doing tree rewriting recently...– Isn't that one way of specifying
parser actions?● Yes, indeed. Reduction semantics by
Felleisen and the PLT, is one example.
– Solution: PyPat, a top down tree matching language.
![Page 33: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/33.jpg)
Related WorkRelated Work
● Inten(s|t)ional Programming– http://intentsoft.com/
– What does Charles Simonyi smoke and where can I get some?
● Stratego/XT (& SDF) (Visser et al.)– http://www.program-transformation.org/
● Ralf Lämmel● SUIF and related plug ins
– Recently became aware of OpenAnalysis, an open IR component.
![Page 34: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/34.jpg)
Can I Get a Witness?Can I Get a Witness?
● Or possibly just “borrow a feeling”?
http://wildideas.org/basil/
![Page 35: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/35.jpg)
No Sudden Moves, or No Sudden Moves, or the Engineer Gets It!the Engineer Gets It!
● Okay, maybe I should calm down now.
● And we can talk business...
![Page 36: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/36.jpg)
Why Grammatech?Why Grammatech?
● Isn't it obvious now?● Okay, how about this:
– I work on Basil for free. All of it is under the BSD license (extend and embed away).
– Grammatech's tool chain could gain functionality by adding very few adapter components.
![Page 37: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/37.jpg)
Getting From Now to Getting From Now to WowWow
● Good engineering, but I'm at a theory school. Here's most of what I see in my bag of tricks:
– Asking software engineering tool makers to eat their own dog food.
– (Sane) decoupling of syntax and actions.
● In many ways, Basil is a solution looking for a problem.
– Problems your customers might just happen to have.
● My intuitions can use a tweak.
– What kind of functionality in this space would wow you?
![Page 38: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/38.jpg)
Work, work, work...Work, work, work...
● PyPat adds a key piece to the puzzle. It is time to demonstrate how it all works.
● What? More Python Compilers?– Refactor standard compiler module.
● Provide support for multiple bytecode outputs (Python, Lua, Parrot).
– PyPy - Python written in Python.
– Son of PyFront – PyFront but with type inference.
![Page 39: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/39.jpg)
Even more work!Even more work!
● More language integrations!– Parsers for C++, Java, SML, Fortran(?!)
● UML/XMI Support– Will allow modeling using UML tools!– Had to use DTD to model translator to
date.
● Models!– Don't even have old school models in there
yet!
![Page 40: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/40.jpg)
AcknowledgementsAcknowledgements
● The GRAD development team– Charlie Fly, Greg Boes, Mark Guerra
● Python community– Guido van Rossum, Jeremy Hylton, Fred Drake Jr.
● PyPy community– Holger Krekel, Armin Rigo, Samuele Pedroni
● University of Chicago– David Beazley, John Reppy
![Page 41: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/41.jpg)
And now my excuse for And now my excuse for a punchline...a punchline...
![Page 42: Grammatech, Meet Basil - University of Chicagopeople.cs.uchicago.edu/~jriehl/Grammatech.pdf · Grammatech, Meet Basil (or “Geeks, Know Thine Selves”) Jonathan Riehl Grammatech,](https://reader034.fdocuments.net/reader034/viewer/2022042917/5f5875090c32b25f4322208d/html5/thumbnails/42.jpg)
No, really, I need a No, really, I need a place to crash soon.place to crash soon.
● School starts here on the 25th?● The students are going to take
back my sublet on the 15th!● I'd at least like to stay to the
19th.● Willing to trade labor, money,
possibly some gratitude...