I'm streaming chess PGNs from the Lichess API. Streaming is recommended, since any given user could have up to ~500,000 games stored. The goal is to parse the PGNs in a parsePGNStream conduit as they're coming in. Unfortunately, the stream chunking is arbitrary and PGN-agnostic -- meaning there has to be some kind of intermediary conduit which concatenates chunks so that each piece of input is a complete PGN ByteString once it reaches parsePGNStream.
reqBr GET (url) NoReqBody opts $ \r -> do
runConduitRes $ (responseBodySource r .| repairChunk .| parsePGNStream .| BS.sinkFile "./test.pgn")
I have a partial PGN parser which recursively finds the first complete PGN-match in a given ByteString
stepParse i
| valid = Just i
| BS.null i = Nothing
| otherwise = stepParse (BS.init i)
where
r = MP.parseMaybe pgn i
valid = isJust r
If I had to guess, the solution probably involves some kind leftover which feeds unconsumed input back to the conduit input -- but everything I do seems to be a dead end. How would one go about implementing this?
