Using FParsec to parse possibly malformed input

Question

I'm writing a parser for a specific file format using FParsec as a firstish foaray into learning fsharp. Part of the file has the following format

{ 123 456 789 333 }

Where the numbers in the brackets are pairs of values and there can be an arbitrary number of spaces to separate them. So these would also be valid things to parse:

{  22 456              7 333     }

And of course the content of the brackets might be empty, i.e. {}

In addition I want the parser to be able to handle the case where the content is a bit malformed, eg. { some descriptive text } or maybe more likely { 12 3 4} (invalid since the 4 wouldn't be paired with anything). In this case I just want the contents saved to be processed separately.

I have this so far:

type DimNummer = int
type ObjektNummer = int
type DimObjektPair = DimNummer * ObjektNummer
type ObjektListResult = Result<DimObjektPair list, string>


let sieObjektLista = 
    let pnum = numberLiteral NumberLiteralOptions.None "dimOrObj"
    let ws = spaces
    let pobj = pnum .>> ws |>> fun x -> 
        let on: ObjektNummer = int x.String
        on
    let pdim = pnum |>> fun x -> 
        let dim: DimNummer = int x.String
        dim

    let pdimObj = (pdim .>> spaces1) .>>. pobj |>> DimObjektPair

    let toObjektLista(objList:list<DimObjektPair>) = 
        let res: ObjektListResult = Result.Ok objList
        res

    let pdimObjs = sepBy pdimObj spaces1
    let validList = pdimObjs |>> toObjektLista

    let toInvalid(str:string) = 
        let res: ObjektListResult = 
            match str.Trim(' ')  with 
            | "" -> Result.Ok []
            | _ -> Result.Error str
        res

    let invalidList = manyChars anyChar |>> toInvalid
    let pres = between (pchar '{') (pchar '}') (ws >>. (validList <|> invalidList) .>> ws)
    pres

let parseSieObjektLista = run sieObjektLista

However running this on a valid sample I get an error:

{ 53735        7785  86231   36732         }
                     ^
Expecting: whitespace or '}'

just a hint: at least for debugging (but for my taste generally) i would take all the nested functions out of sieObjektLista, which allows you easily to test each (sub) parser function in isolation. you debug that in f# interactive. in general, nested functions avoid polluting the namespace but make code maintenance much harder (cannot run them in isolation anymore), so generally its better to avoid them unless good reasons are in their favor. — citykid, May 18 '21 at 07:09
Thanks for the advice. I wanted to keep the module interface fairly clean, but I think you are right. It makes it easier to debug etc if all parser functions can be accessed even if they're just used internally. — Emil L, May 18 '21 at 17:29

Brian Berns · Answer 1 · 2021-05-18T14:41:01.733

1

The main problem here is in pdimObjs. The sepBy parser fails because the separator spaces following each number have already been consumed by pobj, so spaces1 cannot succeed. Instead, I suggest you try this:

let pdimObjs = many pdimObj

Which gives the following result on your test input:

Success: Ok [(53735, 7785); (86231, 36732)]

edited May 18 '21 at 14:41

answered May 18 '21 at 14:25

Brian Berns

15,499
2
30
40

There is no need for `spaces`, they're already consumed in `pobj`. – Fyodor Soikin May 18 '21 at 14:28
Good point. Whitespace management in FParsec can be tricky. – Brian Berns May 18 '21 at 14:29

Fyodor Soikin · Accepted Answer · 2021-05-18T14:37:21.053

You're trying to consume too many spaces.

Look: pdimObj is a pdim, followed by some spaces, followed by pobj, which is itself a pnum followed by some spaces. So if you look at the first part of the input:

{ 53735        7785  86231   36732         }
  \___/\______/\__/\/
    ^      ^    ^   ^
    |      |    |   |
   pnum    |    |   |
    ^   spaces1 |   |
    |           |   ws
   pdim        pnum  ^
     ^          ^    |
     |          \    /
     |           \  /
     |            \/
      \          pobj
       \          /
        \________/
            ^
            |
          pdimObj

One can clearly see from here that pdimObj consumes everything up to 86231, including the space just before it. And therefore, when sepBy inside pdimObjs looks for the next separator (which is spaces1), it can't find any. So it fails.

The smallest way to fix this is to make pdimObjs use many instead of sepBy: since pobj already consumes trailing spaces, there is no need to also consume them in sepBy:

let pdimObjs = many pdimObj

But a cleaner way, in my opinion, would be to remove ws from pobj, because, intuitively, trailing spaces aren't part of the number representing your object (whatever that is), and instead handle possible trailing spaces in pdimObjs via sepEndBy:

let pobj = pnum |>> fun x ->
    let on: ObjektNummer = int x.String
    on
...
let pdimObjs = sepEndBy pdimObj spaces1

That works great for valid input. Any suggestion on how to make the `(validList <|> invalidList)` use the invalidList if the validList fails? Currently it seems to expect a `dimOrObj` or a `}`. But I would like it to use the invalidList parser function basically as a fallback. — Emil L, May 18 '21 at 17:59
Managed to figure it out by using `attempt` and some other minor adjustments. — Emil L, May 19 '21 at 03:44

Using FParsec to parse possibly malformed input

2 Answers2