Regexes vs Regular Expressions; and Recursive Descent Parser
description
Transcript of Regexes vs Regular Expressions; and Recursive Descent Parser
1
Regexes vs Regular Expressions; andRecursive Descent Parser
Ras Bodik, Thibaud Hottelier, James IdeUC Berkeley
CS164: Introduction to Programming Languages and Compilers Fall 2010
Expressiveness of recognizersWhat does it mean to "tell strings apart"?
Or "test a string" or "recognize a language", where language = a (potentially infinite) set of strings
It is to accept only a string with that has some property
such as can be written as ('1'*k)*m, k>1, m>1or contains only balanced parentheses: ((())()(()))
Why can't a reg expression test for ('1'*k)*m, k>1,m>1 ?
Recall reg expression: char . | *We can use sugar to add e+, by rewriting e+ to e.e*We can also add e++, which means 2+ of e: e++ --> e.e.e*
2
… continuedSo it seems we can test for ('1'*k)*m, k>1,m>1, right?
(1++)++ rewrite 1++ using e++ --> e.e+(11+)++ rewrite (11+)++ using e++ --> e.e+(11+)(11+)+
Now why isn't (11+)(11+)+ the same as (11+)\1+ ?
How do we show these test for different property?
3
A refresherRegexes and regular expressions both support operators in this grammar
R ::= char | R R | R* | R ‘|’ R
Regexes suppot more operators, such as backreferences \1, \2,Capturing groups
but let’s ignore this for now.
4
Regexes vs RERegexes implemented with backtracking
This regex requires exponential time to discover that it does not match the input string X==============.
X(.+)+X
REs implemented by translation to NFA, which is then translated to DFA.
Corresponding regular expression requires only linear time, after converted to DFA.
5
MatchAllOn the problem of detecting whether a pattern (regex or RE) matches the entire string, both regex and RE interpretation of a patter agree
– After all, to match the whole string, it is sufficient to find any number of times that a Kleene star matches
6
Let’s now focus on when regex and RE differ
7
Example from Jeff Friedl’s book Imagine you want to parse a config file:
filesToCompile=a.cpp b.cppThe regex for this command line format:
[a-zA-Z]+=.*Now let’s allow an optional \n-separated 2nd
line:filesToCompile=a.cpp b.cpp \<\n> d.cpp e.h
We extend the original regex: [a-zA-Z]+=.*(\\\n.*)?
This regex does not match our two-line input. Why?
What compiler textbooks don’t teach youThe textbook string matching problem is
simple:Does a regex r match the entire string s?– a clean statement and suitable for theoretical
study– here is where regexes and FSMs are equivalent
The matching problem in the Real World:Given a string s and a regex r, find a substring
in s matching r.Do you see the language design issue here?
– There may be many such substrings. – We need to decide which substring to find.
It is easy to agree where the substring should start:– the matched substring should be the leftmost
match
Two schools of regexesThey differ in where it should end:Declarative approach: longest of all matches
– conceptually, enumerate all matches and return longest
Operational approach: define behavior of *, | operatorse* match e as many times as possible while
allowing the remainder of the regex t o match
e|e select leftmost choice while allowing remainder to match
[a-zA-Z]+ = .* ( \\ \n .* )?
filesToCompile=a.cpp b.cpp \<\n> d.cpp e.h
These are important differencesWe saw a non-contrived regex can behave
differently– personal story: I spent 3 hours debugging a
similar regex– despite reading the manual carefully
The (greedy) operational semantics of * – does not guarantee longest match (in case you
need it)– forces the programmer to reason about
backtrackingIt seems that backtracking is nice to reason
about– because it’s local: no need to consider the
entire regex– cognitive load is actually higher, as it breaks
composition
11
Where in history of re did things go wrong?
It’s tempting to blame perl– but the greedy regex semantics seems older– there are other reasons why backtracking is used
Hypothesis 1:creators of re libs knew not that NFA can – can be the target language for compiling regexes– find all matches simultaneously (no backtracking)– be implemented efficiently (convert NFA to DFA)
Hypothesis 2: their hands were tied– Ken Thompson’s algorithm for re-to-NFA was
patentedWith backtracking came the greedy semantics
– longest match would be expensive (must try all matches)
– so semantics was defined greedily, and non-compositionally
Concepts• Syntax tree-directed translation (re to NFA)• recognizers: tell strings apart• NFA, DFA, regular expressions = equally
powerful• but \1 (backreference) makes regexes
more pwrful• Syntax sugar: e+ to e.e*• Compositionality: be weary of greedy
semantics• Metacharacters: characters with special
meaning13
Summary of DFA, NFA, RegexpWhat you need to understand and remember
– what is DFA, NFA, regular expression– the three have equal expressive power– what is the “expressive power”– you can convert
• RE NFA DFA• NFA RE • and hence also DFA RE, because DFA is a special
case of NFA– NFAs are easier to use, more costly to execute
• NFA emulation O(S2)-times slower than DFA• conversion NFADFA incurs exponential cost in space
Some of these concepts will be covered in the section 14
Recursive descent parser
15
16
Recursive Descent ParserPoor man’s backtracking parser
does not do full backtracking you must be a bit carefulbut quite fast, despite backtracking, and simple to implement
many successful languages implemented with r.d. parser
– in many situations, this parser is all you will needwhen could you use an even simpler parser?
– when the grammar is not (heavily) recursive • ex: parse a formatted email message for answers to a
quiz– you could use the “spaghetti code” parser from
last lecture– but this simplification may not be worth it– because r.d. parser makes the grammar clear
maintainable
17
A Recursive Descent Parser (1)
write a function for each terminal, production, non-terminal
– return true iff input matches that terminal, production, n/t
– advance next
Terminals:bool term(TOKEN tok) { return in[next++] == tok; }
nth production of non-terminal S:bool Sn() { … }
non-terminal S: bool S() { … }
18
A Recursive Descent Parser (2)
For production E T + E bool E1() { return T() && term(PLUS) && E(); }For production E T bool E2() { return T(); }For all productions of E (with backtracking)
bool E() { int save = next; return (next = save, E1()) || (next =
save, E2()); }
Ras Bodik, CS 164, Spring 2007
19
A Recursive Descent Parser (4)Functions for non-terminal Tbool T1() { return term(OPEN) && E() &&
term(CLOSE); }bool T2() { return term(INT) && term(TIMES) &&
T(); }bool T3() { return term(INT); }
bool T() { int save = next; return (next = save, T1()) || (next = save, T2()) || (next = save, T3()); }
20
Recursive Descent Parsing. Notes.To start the parser
– Initialize next to point to first token– Invoke E()
Notice how this simulates our backtracking parser
– but r.d. parser does not perform full backtracking
– this is important to remember (see example in a HW)
LL and LR parsing algorithms are more efficient
– see a compiler textbook if interested
Ras Bodik, CS 164, Spring 2007
21
First problem with Recursive-Descent Parsing
Parsing: – given a string of tokens t1 t2 ... tn, find its parse
treeRecursive-descent parsing, backtracking parsing
– Try all the productions (almost) exhaustively– At a given moment the fringe of the parse tree is:
t1 t2 … tk A …– ie, parser will eventually derive a string starting
with terminals– parser compares this prefix with the remainder of
the input– if mismatch, parser backtracks
• but there are grammars such that – parser will NEVER derive a string starting with a
terminal
Eliminating left recursion
23
When Recursive Descent Does Not Always WorkConsider a production S S a:
– In the process of parsing S we try the above rule
– What goes wrong?A left-recursive grammar has a non-terminal S
S + S for some + : derives in one or more steps
Recursive descent may not work in such cases
– It may go into an loopYou say “may”?
– is there a left-recursive. grammar that r.d. parsers can handle?
24
Elimination of Left Recursion• Consider the left-recursive grammar S S |
• S generates all strings starting with a and followed by a number of
• Can rewrite using right-recursion S S’ S’ S’ |
25
Elimination of Left-Recursion. Example• Consider the grammar
S 1 | S 0 ( = 1 and = 0 )
can be rewritten as S 1 S’
S’ 0 S’ |
26
Oops, didn’t we break anything in the process?Consider the grammar for additions:
E E + id idAfter left-recursion elimination:
E id E’ E’ + id E’
Draw the parse tree for id+id+idyour figure comes here
Ras Bodik, CS 164, Spring 2007
27
More Elimination of Left-RecursionIn general S S 1 | … | S n | 1 | … | m
All strings derived from S start with one of 1,…,m and continue with several instances of 1,…,n Rewrite as
S 1 S’ | … | m S’ S’ 1 S’ | … | n S’ |
28
General Left Recursion• The grammar
S A | A S is also left-recursive because
S + S
• This left-recursion can also be eliminated• See [ALSU], Section 4.3 for general
algorithm
29
A comment on removing left recursion• Not a big deal in practice
– ie, you won’t have to convert from left recursion too often
• Just define a right-recursive grammar from the start– works for many cases
• Example: list of arguments – btw, lists are common in programming
language grammars– Left recursive: LIST id LIST , id– Right recursive: LIST id id , LIST – Just opt for the second alternative!
Left Factoring
Ras Bodik, CS 164, Spring 2007
31
Are all grammars equally efficient for r.d.p.?• Consider this grammar:
E T + E T – E TT id * T id / T id
• Parse this stringid * id
• Do you see the inefficiency?– the parser will repeat this derivation three
times (try it)T id * T id * id
32
Left Factoring• reduces backtracking in r.d. parser
• beforeE T + E T – E TT id * T id / T id
• afterE T E’E’ + E – E T id T’T’ * T / T
Limited Backtracking
34
Order of productions may matter in r.d. parser• Consider this grammar
E T + E T – E TT F F * T F / T ---- here we are
trying T F firstF id n ( E )
• Now try to parseid * id
• Why does the r.d. parser return “syntax error”?– it never backtracks and tries T F * T– it only tries T F and succeeds
• Lesson: put longer productions first
35
Summary of Recursive Descent• Simple and general parsing strategy
– Left recursion must be eliminated first– Left factoring not essential but helps reduce
backtracking– Ambiguity must be removed– Order of productions compensates for limited
backtracking
• Do you have to do all these by hand?– first two can be done automatically– third needs intelligence– last could perhaps be automated, too