Implementation differences between parser combinators and packrat algorithm

Question

In order to have a better understanding of packrat I've tried to have a look at the provided implementation coming with the paper (I'm focusing on the bind):

instance Derivs d => Monad (Parser d) where 

  -- Sequencing combinator
  (Parser p1) >>= f = Parser parse

    where parse dvs = first (p1 dvs)

          first (Parsed val rem err) = 
            let Parser p2 = f val
            in second err (p2 rem)
          first (NoParse err) = NoParse err

          second err1 (Parsed val rem err) =
            Parsed val rem (joinErrors err1 err)
          second err1 (NoParse err) =
            NoParse (joinErrors err1 err)

-- Result-producing combinator
return x = Parser (\dvs -> Parsed x dvs (nullError dvs))

-- Failure combinator
fail [] = Parser (\dvs -> NoParse (nullError dvs))
fail msg = Parser (\dvs -> NoParse (msgError (dvPos dvs) msg))

For me it looks like (errors handling aside) to parser combinators (such as this simplified version of Parsec):

bind :: Parser a -> (a -> Parser b) -> Parser b
bind p f = Parser $ \s -> concatMap (\(a, s') -> parse (f a) s') $ parse p s

I'm quite confused because before that I thought that the big difference was that packrat was a parser generator with a memoization part. Sadly it seems that this concept is not used in this implementation.

What is the big difference between parser combinators and packrat at implementation level?

PS: I also have had a look at Papillon but it seems to be very different from the implementation coming with the paper.

The implementation you linked to mixes tabs and spaces, and uses a different tab width than Stack Overflow does, so your copy/paste looks mis-indented. Obviously fixing this will not answer your question, but it will make it easier to read. — amalloy, Nov 21 '18 at 23:09
"Parser generators vs. parser combinators vs. hand-written parsers" and "packrat vs. predictive parsing vs. recursive descent with backtracking vs. bottom-up algorithms" are two completely separate classifications. One's about the tools you use to write the parser, the other's about the algorithm the parser uses. Your linked implementation is a parser combinator implementing packrat parsing. That's not a contradiction. — sepp2k, Nov 22 '18 at 02:06
@sepp2k you're right that his question should have said "what's the difference between the packrat library implementation and recursive descent or predictive parser combinator libraries?". However, the implementation he links to does not actually implement packrat parsing: you can use it, but you have to apply the packrat trick yourself in your own code. See my answer below. — Dominique Devriese, Jan 25 '19 at 13:59

Dominique Devriese · Accepted Answer · 2019-01-25T13:52:44.720

2

The point here is really that this Packrat parser combinator library is not a full implementation of the Packrat algorithm, but more like a set of definitions that can be reused between different packrat parsers.

The real trick of the packrat algorithm (namely the memoization of parse results) happens elsewhere. Look at the following code (taken from Ford's thesis):

data Derivs = Derivs {
                   dvAdditive :: Result Int,
                   dvMultitive :: Result Int,
                   dvPrimary :: Result Int,
                   dvDecimal :: Result Int,
                   dvChar :: Result Char}


pExpression :: Derivs -> Result ArithDerivs Int
Parser pExpression = (do char ’(’
                         l <- Parser dvExpression
                         char ’+’
                         r <- Parser dvExpression
                         char ’)’
                         return (l + r))
                     </> (do Parser dvDecimal)

Here, it's important to notice that the recursive call of the expression parser to itself is broken (in a kind of open-recursion fashion) by simply projecting the appropriate component of the Derivs structure.

This recursive knot is then tied in the "recursive tie-up function" (again taken from Ford's thesis):

parse :: String -> Derivs
parse s = d where
  d = Derivs add mult prim dec chr
  add = pAdditive d
  mult = pMultitive d
  prim = pPrimary d
  dec = pDecimal d
  chr = case s of
          (c:s’) -> Parsed c (parse s’)
          [] -> NoParse

These snippets are really where the packrat trick happens. It's important to understand that this trick cannot be implemented in a standard way in a traditional parser combinator library (at least in a pure programming language like Haskell), because it needs to know the recursive structure of the grammar. There are experimental approaches to parser combinator libraries that use a particular representation of the recursive structure of the grammar, and there it is possible to provide a standard implementation of Packrat. For example, my own grammar-combinators library (not maintained atm, sorry) offers an implementation of Packrat.

edited Jan 25 '19 at 13:52

answered Nov 22 '18 at 07:22

Dominique Devriese

2,998
1
15
21

Thanks for your answer, but I'm not sure to understand how this co-recursion make the memoization work. I have done some tests with `fix` (on a fibonacci sequence) for example and it have to impact on performances. If you can add some link/insights about it, thanks. – GlinesMome Nov 25 '18 at 21:53
Sorry, but it's not clear to me what you're asking. The memoization is the core trick of the packrat algorithm. Essentially, the only values of type Derivs that will ever be generated for a given input, is the initial result of `parse "input"`, as well as the ones generated in the chr case of parse above. For those latter ones, only one value of type Derivs is generated for every character consumed. So there will be as many Derivs values as inputs in the string. – Dominique Devriese Nov 26 '18 at 22:20
Those Derivs values have one field for every non-terminal (plus the chr primitive), so in summary, there is one Result ArithDerivs x for every non-terminal (plus the chr primitive) at every input position. All other non-terminals only ever return one of these pre-existing Derivs values, because they just project a field of one of those pre-generated Derivs values. – Dominique Devriese Nov 26 '18 at 22:24
Any field of those Derivs values will only be evaluated once, by virtue of Haskell's lazyness, so this means you get the desired memoization. – Dominique Devriese Nov 26 '18 at 22:25
In fact I have hard time to figure out how memoization works here. Let's take a parser based on `ArithMonad.hs` `(Parser advDigit >> symbol '+' >> Parser advDigit) > (Parser advDigit >> symbol '-' >> Parser advDigit)`, given that expression, how/why the first `Parser advDigit` is memoized in `58-16` for example? – GlinesMome Nov 28 '18 at 22:08
Well you would invoke it by evaluating `advDigit (parse someInput)`. By definition of `parse`, that evaluates to `advDigit d` for the d in the definition of `parse`. That evaluates to `pAdditive d`. – Dominique Devriese Nov 30 '18 at 08:03
1

Evaluating `pAdditive d` will then first evaluate `advMultitive d`. It's important that it's `advMultitive d` that's being evaluated (a field of the record d), not `pMultitive d` (the actual parse of a multiplicative expression a this position). – Dominique Devriese Nov 30 '18 at 08:12
The reason this is important is that if the field `advMultitive d` has already been evaluated earlier (for example in another attempted but failed branch of the grammar), then it will not be evaluated again, per Haskell's lazy evaluation. – Dominique Devriese Nov 30 '18 at 08:14
In other words, it's important that recursive references in parsers project out the appropriate fields of `d` rather than call the corresponding parsers directly. Concretely, it's important that ArithMonad.hs:75 says `Parser advMultitive` rather than `Parser pMultitive`. The former would be memoized but the latter wouldn't. – Dominique Devriese Nov 30 '18 at 08:16
I've added a note that my point about observing the recursive structure of the grammar only applies to pure languages like Haskell. – Dominique Devriese Jan 25 '19 at 13:53

PaulMcG · Answer 2 · 2018-12-29T17:07:55.373

As stated elsewhere, packrat is not an alternative to combinators, but is an implementation option in those parsers. Pyparsing is a combinator that offers an optional packrat optimization by calling ParserElement.enablePackrat(). Its implementation is almost a drop-in replacement for pyparsing's _parse method (renamed to _parseNoCache), with a _parseCache method.

Pyparsing uses a fixed-length FIFO queue for its cache, since packrat cache entries get stale once the combinator fully matches the cached expression and moves on through the input stream. A custom cache size can be passed as an integer argument to enablePackrat(), or if None is passed, the cache is unbounded. A packrat cache with the default value of 128 was only 2% less efficient than an unbounded cache against the supplied Verilog parser, with significant savings in memory.

Implementation differences between parser combinators and packrat algorithm

2 Answers2