Embedding other language in Flex/Bison

Question

The bottom line:

If you would like to add one, very small feature into C++ using Flex/Bison, how would you do that? For example, ability to declare void xxx() functions with syntax: foo%%: xxx?

The whole story:

Once I have coded a custom shader processing program that built ready-to-use Cg/GLSL pixel and vertex shaders from a number of blocks. I've added few features, mostly related to static compilation (something like a "better preprocessor").

For example, something that would look like

#if LIGHT_TYPE == POINT
    float lightStrength = dot(N, normalize(pos - lightPos));
#elif LIGHT_TYPE == DIRECTIONAL
    float lightStrength = dot(N, lightDir);
#endif

with pure macros, looks like

[LightType = Point]
[Require: pos, lightPos]
float LightStrength()
{
    !let %N Normal
    return dot(%N, normalize(pos - lightPos));
}

in my "language". As you can see, "functions" can be provided for variable light/material types. There is also a possibility to "call" other function and to mark what uniform/varying attributes are required for specific shader.

Why all this effort? Because (especially in early cards like SM 2.0) there were numerous limitations of attributes, and my "compiler" produced ready-to-use shader with list of required attributes/variables, managed parameters between pixel and vertex shaders, optimized some static stuff (just for readability, as Cg compiler would optimize it later anyway).

OK, but I am not writing all of this to praise myself or something.

I've written this "compiler" in C# and it initially was a very small program. But as time passed, many features were added and now the program is a complete mess with no options of refactoring. Also, being coded in C# means I cannot embed this compiler directly in C++ game, forcing me to spawn processes to compile shaders (this takes a lot of time).

I would like to rewrite my language/compiler using C++ and Flex/Bison toolkit. I've already coded efficient math parser in Flex/Bison so I am somewhat experienced in this matter. However there is one thing that I cannot resolve myself and this is a very topic of my question.

How can I possibly embed GLSL into my language? In C# compiler, I've just went line-by-line and checked if line starts with special characters (like % or [) and later did many tricks&hacks using string_replace and/or regexes.

Now I would like to write down a clean Bison grammar and do it properly. But including whole syntax of GLSL scares me. It is quite a big and complicated language, constantly evolving. And most I would do would be to pass-through all this GLSL code anyway.

If you are not experienced with GLSL/Cg/graphics programming at all, this is not quite important. The question can be rewritten into the "bottom line".

So, if you would like to add one, very small feature into C++ using Flex/Bison, how would you do that? For example, ability to declare void xxx() functions with syntax: foo%%: xxx?

Adobe hijacking that `flex` flag really sucks. (But using `yacc` and `lex` might be better anyway.) — sbi, Nov 12 '10 at 19:29
Bison is quite easy to hack. I modded it to emit C# instead a few years back. — leppie, Jun 09 '11 at 09:37

score 1 · Answer 1 · edited Jun 20 '20 at 09:12

I added multithreading mechanisms to pascal for a school project once, so I've had to deal with this. What I did was I found a BNF definition of pascal and copied that, mainly making my tokens equal to the text in 99% of the cases, and adding the new intermediate language code in the 1% of new tokens I added. Pascal is simple, so the BNF definition is relatively simple. C++, not so much.

The C++ Programming Language by Stroustroup has the language grammar that is practically parser-ready, but that's a lot of code to copy by hand. Maybe a certain website can help you find a bison/lex set for "standard" C++, and then you can modify that.

EDIT: I found my yacc files, so here's a small example from my pascal code

procheader: PROCEDURE ID paramslist ';'{thread::inLocalDecl=true; $$="procedure "+$2+$3+";\n";}
|            FUNCTION ID paramslist ':' ID ';'{thread::inLocalDecl=true; $$="function "+$2+$3+":"+$5+";\n";}
|  THREAD {thread::inThreadDecl=true;
       thread::inArgDecl=true;}
   ID {thread::curName=$3;} paramslist { thread::inArgDecl=false;} ';'
{ 
   /*a lot of code to transform this custom construct into standard pascal*/
}

the first two elements were just standard pascal: I copied the input tokens into the output strings verbatim (ie I didn't actually modify anything from the input file).

the third element(my THREAD keyword) was, obviously, not standard pascal. So I transformed the output into something that I could actually compile in standard pascal.

Basically to compile my "threaded" pascal, I had to take my source file, pass it through my parser, then compile the output.

Embedding other language in Flex/Bison

1 Answers1