5

I need to convert a few hundred text files using TWiki markup to a more standard markup language (e.g. Markdown or Asciidoc) and, as there doesn't seem to be any usable tool for doing this (pandoc supports TWIki but very poorly, i.e. it irretrievably loses not only much of the markup but also a lot of contents), I'd like to write my own converter in Raku.

At first sight, it looks like it would be appropriate to use Raku grammars for this task, but I'm a bit worried that I could have problems with some broken markup -- which I'd still need to handle in some way. So would it be wiser to stick to manual regexes to have more flexibility?

If anybody has any experience with using Raku grammars for parsing markup, could you please share it? TIA!

VZ.
  • 21,740
  • 3
  • 39
  • 42
  • 2
    I can promise you will not lose any flexibility by using Raku grammars. They are the natural extension of "traditional" regexes but written in a way that's *much* more readable and is actually designed to scale to arbitrary grammars. 100% you have found the right tool for the job. Happy parsing! – Silvio Mayolo Apr 09 '22 at 23:57
  • 1
    What Silvio says. Raku grammars = Raku regexes = arbitrary programs written in a Turing complete suite of sub-languages. You can parse *anything* that a computer program can parse, no matter what weird wrinkles the format has, or broken markup a particular file has. You "just" get to write your parser with convenient mechanisms and notation. Top two tips: 1) Install [CommaIDE](https://commaide.com/) and use its [Grammar Live View](https://commaide.com/docs/grammar-live-view). Really. Don't even think about not doing that. 2) Lean on the community. Keep asking questions here, reddit, IRC, etc. – raiph Apr 10 '22 at 01:05
  • 1
    To get a sense of the way Raku essentially eliminates the distinction between regexes and grammars, consider reading [my answer to SO **Extracting from `.bib` file with Raku**](https://stackoverflow.com/questions/45172113/extracting-from-bib-file-with-raku-previously-aka-perl-6/45181464#45181464). – raiph Apr 10 '22 at 01:16
  • 1
    Thanks for all the replies! I understand that, in theory, grammars can do anything and I've already used them and appreciate them, but this was for much better defined/constrained input, so I've just been wondering if, in _practice_, it's a good idea to use them when you may have to deal with some less well-formed input (e.g. unclosed/mismatched tags etc). For now I'll check the CommaIDE (I'm a Vim guy normally, so it's a big leap for me!) and the `.bib` example, thanks @raiph! – VZ. Apr 10 '22 at 13:42
  • 1
    Thanks, let me experiment with writing a grammar for TWiki and I'll update the answer if I run into any problems -- or if I don't! – VZ. Apr 10 '22 at 17:40
  • 2
    I just wrote my first serious grammar to slice and dice a vimwiki file (mardown with some taskwiki (vim plugin) and other weirdness mixed in). Works very well. I did a little writeup about some of the pitfalls and traps I ran into: https://www.reddit.com/r/rakulang/comments/tsu1sr/q_how_long_does_it_take_to_get_fairly_proficient/ – StevieD Apr 11 '22 at 13:12

0 Answers0