Download - The Art Of Parsing @ Devoxx France 2014

@dbolkensteyn @_godin_#parsing

The Art of Parsing

Evgeny Mandrikov @_godin_Dinesh Bolkensteyn @dbolkensteynhttp://sonarsource.com

@dbolkensteyn @_godin_#parsing 2/56

The Art of Parsing// TODO: don't forget to add huge disclaimer that all opinions hereinbelow are our own and not our employer (they wish they had them)

Evgeny Mandrikov@_godin_

Dinesh Bolkensteyn@dbolkensteyn


I want to create a parser

«Done»!

Use Yacc, JavaCC, ANTLR, SSLR, …

or hand-written ?


What is the plan?

Why• javac and GCC are hand-written• do we use parser-generators ?

Together we will implement parser for• arithmetic expressions• common constructions from Java• C++ ;)


Java formal grammar

JLS8

JLS7


Answer is

42


Pill of theory

NUM ➙ 42Nonterminal

Productions

Terminals(tokens)


Grammar for numbers

NUM ➙ NUM DIGIT | DIGITDIGIT ➙ 0|1|2|3|4|5|6|7|8|9

4, 8, 15, 16, 23, 42,…

Alternatives


Arithmetic expressions

4 – 3 – 2 = ?


expr ➙ expr – expr | NUM


4 – 3 – 2 = ?



expr

4 3

2

expr


(4 – 3)– 2 =-1



4

3 2

expr

expr


(4 – 3)– 2 =-1 4 –(3 – 2)= 3

expr

4 3

2

expr


Arithmetic expressionsexpr ➙ NUM – expr | NUM


(4 – 3)– 2 =-1 4 –(3 – 2)= 3

expr

4 3

2

expr 4

3 2

expr

expr


Arithmetic expressionsexpr ➙ NUM – expr | NUM


expr ➙ expr – NUM | NUM

(4 – 3)– 2 =-1 4 –(3 – 2)= 3

4

3 2

expr

expr

expr

4 3

2

expr


Show me the code

int expr() { int res = expr(); if (token == '–') return res – num(); return num(); }




Show me the code right code

? ? int expr() { int res = expr(); if (token == '–') return res – num(); return num(); }




Show me the code right code




int expr() { int res = num(); while (token == '–') res = res – num(); return res; }

int expr() { int res = num(); while (token == '–') res = res – num(); return res; }



4 – 3 * 2 = ?



4 – 3 * 2 = -2

expr ➙ expr – NUM | expr * NUM | NUM



4 –(3 * 2)= -2(4 – 3)* 2 = 2 expr ➙ expr – NUM | expr * NUM | NUM



subs ➙ subs – mult | mult mult ➙ mult * NUM | NUM

4 –(3 * 2)= -2


Show me the code

int subs() { res = mult() ; while (token == '–') res = res – mult(); return res; }

int mult() { int res = num(); while (token == '*') res = res * num(); return res; }

int subs() { res = mult() ; while (token == '–') res = res – mult(); return res; }

int mult() { int res = num(); while (token == '*') res = res * num(); return res; }

subs ➙ subs – mult | multmult ➙ mult * NUM | NUM


LL(1)

● back to 1969● one token lookahead● no left-recursion


What is the plan?

✔ arithmetic expressions✔ LL(1)

• a few common constructions from Java• C++ ;)


The real deal

expr-stmt ➙ expr ; obj.method(); a = obj.field; obj.method(); a = obj.field;


The real deal

expr-stmt ➙ expr ;

expr ➙ field-access

| method-call

| assignment

obj.method(); a = obj.field; obj.method(); a = obj.field;


The real deal



| method-call

| assignment

field-access ➙ qualified-id

qualified-id ➙ qualified-id . id

| id



The real deal



| method-call

| assignment



| id

method-call ➙ qualified-id ()



The real deal



| method-call

| assignment



| id


assignment ➙ qualified-id = expr



int qualified_id() { /* easy */ } int field_access() { /* easy */ } int method_call() { /* easy */ } int assignment() { /* easy */ }

int expr() { // ??? }

int qualified_id() { /* easy */ } int field_access() { /* easy */ } int method_call() { /* easy */ } int assignment() { /* easy */ }

int expr() { // ??? }

Show me the code



| method-call

| assignment



| id


assignment ➙ qualified-id = expr



int expr() { String id = qualified_id(); if (token == '(') return method_call(); else if (token == '=') return assignment(); else return field_access(); }

int expr() { String id = qualified_id(); if (token == '(') return method_call(); else if (token == '=') return assignment(); else return field_access(); }

The LL(1) wayexpr ➙ field-access

| method-call

| assignment



Realityhttp://hg.openjdk.java.net/jdk8/jdk8/langtools/.../JavacParser.java

http://hg.openjdk.java.net/jdk8/jdk8/langtools/.../JavacParser.java


The better wayexpr ➙ field-access

| method-call

| assignment

int expr() { try { return field_access(); } catch (RE e1) { try { return method_call(); } catch (RE e2) { return assignment(); } } }

int expr() { try { return field_access(); } catch (RE e1) { try { return method_call(); } catch (RE e2) { return assignment(); } } }



int expr() { try { return method_call() ; } catch (RE e1) { try { return assignment(); } catch (RE e2) { return field_access(); } } }

int expr() { try { return method_call() ; } catch (RE e1) { try { return assignment(); } catch (RE e2) { return field_access(); } } }

Show me the code right codeexpr ➙ method-call

/ assignment

/ field-access



Parsing Expression Grammars

● 2002● ordered choice «/»● backtracking● no left-recursion


enum Nonterminals { EXPR, METHOD_CALL, … }

void grammar() { rule(EXPR).is( firstOf( METHOD_CALL, ASSIGNMENT, FIELD_ACCESS)); }

enum Nonterminals { EXPR, METHOD_CALL, … }

void grammar() { rule(EXPR).is( firstOf( METHOD_CALL, ASSIGNMENT, FIELD_ACCESS)); }

DSL for PEGexpr ➙ method-call

/ assignment

/ field-access



What is the plan?

✔ arithmetic expressions✔ LL(1)

✔ common constructions from Java✔ PEG

• C++ ;)

@YourTwitterHandle#DVXFR14{session hashtag} @dbolkensteyn @_godin_#parsing

Tea

Break


if (false) if (true) System.out.println("foo"); else System.out.println("bar");


Quiz




«Dangling else»

if-stmt ➙ IF (cond) stmt ELSE stmt / IF (cond) stmt


Java is awesome

(A)*B

(A)*B


C++ all the pains of the world

int *B; typedef int A; (A)*B; // cast to type 'A' ('int' alias) // of dereference of expression 'B'

int A, B; (A)*B; // multiplication of 'A' and 'B' // with redundant parenthesis around 'A'

int *B; typedef int A; (A)*B; // cast to type 'A' ('int' alias) // of dereference of expression 'B'

int A, B; (A)*B; // multiplication of 'A' and 'B' // with redundant parenthesis around 'A'

Java is good, because itwas influenced by bad experience of C++ (A)*B (A)*B


rule(MUL_EXPR).is( UNARY_EXPR, zeroOrMore('*', UNARY_EXPR)); rule(UNARY_EXPR).is( firstOf( sequence('(', TYPE_ID, ')', UNARY_EXPR), PRIMARY, sequence('*', UNARY_EXPR))); rule(PRIMARY).is( firstOf( sequence('(', EXPR, ')'), ID));


Hit the wall !

(A)*B (A)*B




Hit the wall !

(A)*B (A)*B


Dreammul-expr ➙ mul-expr * unary-expr | unary-exprunary-expr ➙ ( type-id ) unary-expr | * unary-expr | primaryprimary ➙ ( expr ) | id

(A)*B (A)*B


Generalized parsers

● Earley (1968)● slow

● GLR (1984)● complex


Chicken and egg problem

(A)*B

unary-expr mul-expr

(A) (A)*B

B*...

(A)*B (A)*Bmul-expr ➙ mul-expr * unary-expr

| unary-expr

unary-expr ➙ ( type-id ) unary-expr

| * unary-expr

| primary

primary ➙ ( expr )

| id


Back to the future «dangling else»

if (…) if (…) then-stmt else else-stmt

if (…) if (…) then-stmt else else-stmt

outer-if

inner-if inner-if

then-stmt else-stmt

inner-if · else-stmt


GLL : How does it work ?

mul-expr ➙ mul-expr * unary-expr

| unary-expr


Generalized LL

● 2010● no grammar left behind (left-recursive, ambiguous)

● simpler than GLR● syntactic ambiguities


Sum

mar

y


Summary

LL(1)

• trivial• major grammar changes• only good for arithmetic expressions• on steroids as in JavaCC usable for real languages


Summary

PEG

• trivial• fewer grammar changes• no ambiguities• usable for real languages• nice tools such as SSLR• dead-end for C/C++


Summary

GLL

• any grammar• relatively simple• ambiguities• reasonable performances• the only clean choice for C/C++• only «academic» tools for now... ;)


Summary

Hand-written

● based on LL(1)● precise error-reporting and recovery

● best performances● maintainance hell


Q & A