2

What's the best practice for dealing with complex literals in Rascal?

Two examples from JavaScript (my DSL has similar cases):

  • Strings with \ escapes - have to be unescaped into actual value.
  • Regular expression literals - need their own sub-AST.

implode refuses to map lexicals to abstract trees, they are obviously handed differently from syntax productions, despite having complete parse trees available. For example, the following parser fails with IllegalArgument("Missing lexical constructor"):

module lexicals

import Prelude;

lexical Char = "\\" ![] | ![\\]; // potentially escaped character
lexical String = "\"" Char* "\""; // if I make this "syntax", implode works as expected

start syntax Expr = string: String;

data EXPR = string(list[str] chars);

void main(list[str] args) {
    str text = "\"Hello\\nworld\"";
    print(implode(#EXPR, parse(#Expr, text)));
}

The only idea I have so far is to capture all lexicals as raw strings and later re-parse them (implode and all) using separately defined syntaxes without layout whitespace. Hopefully, there's a better way.

Nikita Nemkin
  • 2,780
  • 22
  • 23

1 Answers1

2

The way implode converts a parse tree into an ast is document in the rascal tutor:implode. This contains the following rule:

Unlabeled lexicals are imploded to str, int, real, bool depending on the expected type in the ADT. To implode lexical into types other than str, the PDB parse functions for integers and doubles are used. Boolean lexicals should match "true" or "false". NB: lexicals are imploded this way, even if they are ambiguous.

So, solution 1 is to add a label to your production:

lexical String = string: "\"" Char* "\"";

Also, perhaps you do not need to have an AST next to your parse tree? At least not one that has to closely match your grammar. The two common scenario's are:

  1. You need an AST, since the structure of the grammar is unsuited for your purpose. In that case, you have to manually write your implode function.
  2. The structure of your parse tree is good enough. In that case, checkout the example for Concrete Syntax. It is a very clean way work with the target language nested inside rascal.

We are leaning more and more to deprecating the implode function since our concrete syntax is powerfull enough for most cases.

Davy Landman
  • 15,109
  • 6
  • 49
  • 73
  • Labeling with "string:" conflicts with existing constructor. Labeling with something else ("string1:") results in "Cannot find a constructor for EXPR". (I've got the latter error a lot with my actual definitions, using version 0.7.2.201501130937) My ASTs are very close to the grammar (it's a typical "Expr" grammar), but I need to apply a good deal of term rewriting (canonicalization and optimization), which is why I went with Rascal in the first place: it's a simple but practical language that has both parsing and rewriting built-in. – Nikita Nemkin May 27 '15 at 19:00
  • I think this can be solved by splitting them over 2 modules. And indeed, rascal is a very nice language for parsing and rewriting. And to a certain extend, this is also possible on parse trees. – Davy Landman May 27 '15 at 19:32
  • Side-stepping implode and manually mapping parse trees to "core language" is probably better approach than reparsing lexicals, I'll go with it. Discussion: I understand the ideas behind working within concrete syntax (TXL fan here), but it's REALLY unfortunate that Rascal is moving into this direction. Immutable data and built-in backtracking (`fail`) make it an ideal host for classical "parser with AST building actions" approach. Lightweight tree-building syntax (a la ANTLR 3) + automatic decoration with layout data (whitespace and comments) + automatic unparsing... One can dream. – Nikita Nemkin May 27 '15 at 19:44
  • Intresting discussion, did you know that `fail` is also possible in concrete syntaxes? Or even during parsing, but than it's `filter`. But perhaps your ideas are better suited in some feature requests on [github](https://github.com/cwi-swat/rascal/issues)? We are always interested in users and their usage of Rascal. – Davy Landman May 27 '15 at 19:50
  • The main problem with the automatic `implode` approach is that you now have to maintain the same structure twice. In the grammar and in the ADT. The coupling between those two are very fragile and not something our type-checker can help you with. – Davy Landman May 27 '15 at 19:53