0

For context (skip to question if you like): I am learning ocaml and started reading the Real World Ocaml.

So far, really (really!) liking the language and the book.

One thing I'm starting to not like so much is this mysterious 'ppx' stuff that is starting to show up everywhere I look when I try to 'dig a little deeper'(typically this digging involves opening some source code of a dependency and then and stumbling onto some 'magic' things like [@@deriving sexp]).

From lots of googling / various scattered sources... this seems to be some kind of syntax extension mechanism based on AST transformations. It seems to be heavily used (at least in janestreet libs). Near I can tell this is an 'unofficial' or undocumented feature of Ocaml. For example, looking at the Ocaml manual I couldn't find any trace of it.

Now, the question. How official/supported is the ppx mechanism for extending ocaml with new syntax by means of AST transforms?

EDIT/note: It was pointed out in the answer by @octachron that there is in fact a 'trace' of the feature in the manual that I missed. The syntax for these [@@..] bits at least is documented in the manual / ocaml grammar.

But the answer i.m.o really should address two things: syntax and semantics.

  • syntax is about "how do we write these magic new things".
  • semantics is about how do we attach meaning to them.

A partial answer was already given by @octachron. It is clear now that the 'syntax' is official. But it is still a bit unclear to me how 'official' the support is for attaching semantics to these syntax extensions. So I am holding out for accepting an answer that is a bit more complete/explicit about semantics).

PS: Near as I can tell, there is nothing in the manual about this, and the Ocaml compiler ostensibly does nothing with the annotations at all (except, I suppose, parsing them). So unless I missed something else in the manual, then there isn't an 'official' mechanism in the language that lets you access the data in these annotations and do anything meaningful with them. Is that correct?

Kris
  • 3,898
  • 1
  • 23
  • 32
  • 3
    This doesn't seem like a good fit for StackOverflow, as it's asking for opinion and advice rather than help with a specific problem. However I will say that I never use the syntactic preprocessing tools with my OCaml code unless they're required by some dependency. In those cases they can be very convenient, of course. But I see a problem with fragmenting OCaml into a large number of incompatible syntactic city states. It's possible to build large systems without using ppx at all. – Jeffrey Scofield Apr 23 '22 at 17:02
  • Thanks @JeffreyScofield for sharing your opinion. Personally, I don't get why 'asking for advice or opinions' isn't a good fit. Sure it may be the 'officical position' of SO stated somewhere. But I read many such questions and answers to them with interest and find them often quite useful. So there may not be a single right answer but IMO, those kinds of questions and answers are often the most interesting, and a spread of well stated and motivated opinions can be quite helpful in making up one's own mind, or decide what/where to look for more insight. – Kris Apr 23 '22 at 18:02
  • ^^^ That being said though if indeed it doesn't look like this question is really not welcome here or is not a good fit, or not well formulated enough, or raises other objections... I will delete it in a couple of days. In the mean time I am genuinely curious to see what people have to say about it :-) – Kris Apr 23 '22 at 18:08
  • 3
    Turns out, the is a forum for this kind of question (which is, "why you can't ask for advice or opinions on SO?"), it's meta.stackoverflow.com. However, the general idea is that when there are good answer to questions that ask about an opinion on something, there is also a way to formulate the question that fits better SO policy, while still allowing these interesting answers. See the meta forum for more details. If you'd like to know how to reformulate this specific question in a more SO-ish way, you can also ask on the meta forum. – jthulhu Apr 23 '22 at 20:13
  • Okay thanks, given the answer I got maybe I can rephrase it to be more about the 'official status' of the ppx feature and less about opinions on how to live with or without it. I'll give it a try. – Kris Apr 23 '22 at 21:06
  • @JeffreyScofield and BlackBeans . I tried to make the question more specific and less 'opinion based'. Hopefully enough to consider removing your 'Close' votes :-) – Kris Apr 23 '22 at 21:47

2 Answers2

1

To answer the question about the official status of ppxs, they are officially supported syntax extensions. The syntax for attributes and extension nodes is described in the manual at https://ocaml.org/manual/attributes.html and https://ocaml.org/manual/extensionnodes.html. There is also a basic framework available for writing ppx exported in the compiler library (https://ocaml.org/api/compilerlibref/Ast_mapper.html), even if ppxlib is the currently advised option.

octachron
  • 17,178
  • 2
  • 16
  • 23
  • What do you mean with "they are officially supported"? From your answer it is clear that the official support at least defines the syntax for attributes/nodes. But about what the mechanism for their semantics. (I mean a syntax extension like `[[@@deriving-fields]]` is a bit more than just a piece of metadata you can add to an AST. How 'official' is the support for attaching/implementing semantics? Is there a section in the manual explaining how you can attach meaning to these bits of metadata? – Kris Apr 23 '22 at 21:04
  • I reworde my question to be more explicit about asking about the mechanism for attaching semantics to a syntax extension (without this arguably, a syntax extension is of very limited use. This answer is a bit 'confusing' for not clearly distinsguishing the two (that is on me, for not asking it more explicitly). As it is however, the answer makes it seem like it is 'all official' whereas my impression ATM is that only the 'syntax' part is officially supported, and the mechanism for attaching semantics is not. So if you could clarify that, I can accept the answer. – Kris Apr 23 '22 at 21:42
  • Ppxs are AST transformers. In other words, there are AST-level macros and they don't add any new semantics to the language. In particular, it is (nearly) always possible to expand a ppx-annotated code source into an equivalent code source without ppx annotations or extension nodes. – octachron Apr 23 '22 at 21:47
  • Arguably they do add/change semantics because code that actually relies on what these transformer do will not compile if you just give it to the compiler... does it? (I.e. if the transformer adds some new functions or whatnot, any code that actually calls this functions will not compile if you just give it to the compiler. (Unless the compiler knows how to resolve and execute the AST transforms as part of compilation) – Kris Apr 23 '22 at 21:51
  • If you skip a preprocessor phase, you may end up with different code source indeed that may behave differently indeed. And yes, it best to see ppxs are independent AST-level preprocessor binaries built by using the compiler library. – octachron Apr 23 '22 at 21:56
  • So then the question becomes really 'how official' is this mechanic for finding and running the right preprocessors. Seems to be all very 'ad-hoc'. I.e. there is no official way to give meaning to any of this. It is you, as a user, who is utimately responsible for finding the right preprocessor and calling it using some version of makefile or whatnot. So its not an official part of the language at all? – Kris Apr 23 '22 at 22:04
  • Ppxs are syntax extension that can be de defined using AST transformer built upon the officially released compiler library and using mappers written in the compiler for this exact purpose. So no, there are totally official ways to implement ppx extensions. And finding and running the right ppx preprocessor is not different at all from finding and linking the right library. Nevertheless, ppxs extensions are not part of the language, that's the point: there are (well-delimited) extensions. – octachron Apr 23 '22 at 22:10
  • "And finding and running the right ppx preprocessor is not different at all from finding and linking the right library". Not quite, when a library is missing, I think the compiler understands that you are referring some thing that it is supposed to be able to find, but it can't. Whereas the 'syntax' extensions are essentially no different from structured comments. They have no meaning at all and nobody can tell what, if anything, they are supposed to do by looking at the code. – Kris Apr 23 '22 at 22:24
  • Anyway, thanks for the answer and discussion, I think I understand the 'officiality' of it more or less now. I still think the answer is 'incomplete' without talking more explcitly about the fact that there is no offical way to attach semantics to the annotations in the language. – Kris Apr 23 '22 at 22:28
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/244140/discussion-between-kris-and-octachron). – Kris Apr 23 '22 at 22:33
0

There are two parts to this question:

The first is syntax. That part was answered very well already by @octachron. The syntax for these [@@..] bits are an official part of Ocaml syntax. They are described in the manual at https://ocaml.org/manual/attributes.html and https://ocaml.org/manual/extensionnodes.html.

The second part is semantics. The answer here is more complex.

Note: Octachron already gave some answer here. However, the answer to me was a bit confusing. So I'll try and write an more detailed answer here myself that tries to address that confusion directly.

The tricky part (for me at least) to understand here was that, what you might call 'official support', depends a bit on your expectations, or point of view.

Looking at it from the point of view of the compiler and Ocaml as a programming language... The attributes and annotations have no meaning whatsoever. To the Ocaml compiler they are essentially treated much like comments. They are simply recognised by the parser and completely ignored.

So if the question is... "does the Ocaml language/compiler define an official way to attach semantics to extensionnodes and annotations?"; then the answer is "no". To the compiler... they simply have no semantics.

Any meaning comes from a 'ppx' pre-processor. A ppx pre-processor is a separate Ocaml program that reads in Ocaml source code and parses it into an AST, then transforms the AST. This process is totally separate from the Ocaml compiler and is a true pre-processor.

There is however an official/standard/recommended way that such pre-processors are created and used.

In a nutshell:

  • For creating a ppx you use the offical ppxlib, which is an Ocaml library.
  • To use a ppx extension in your code, you configure a build system like dune to execute it before the ocaml compiler.

These things are explained in great detail in the 'official' ppxlib User manual.

Kris
  • 3,898
  • 1
  • 23
  • 32