How to handle a newLine properly

Question

I'm trying to make a language that ignores NewLines in my code anywhere

so i declared in my lexer just [\n] without returning anything so it can be ignored this is working well and it is ignoring every newLine in the code but i wanted to implement one rule where it shouldn't ignore it the rule is : if the code is like this :

!Array_Library;

!Loop_Library;

it should be accepted

!Array_Library;!Loop_Library;

shouldn't be accepted because there has to be a newLine between the two libraries

i tried to make my lexer return NewLine when it encounters \n but then i'll lose the "ignore NewLines anywhere" rule

Why? Is it ambiguous without a newline? Or do you just not like the way it looks? — rici, Mar 03 '21 at 20:32
i want to make newLine between each declared library but anywhere else is to be ignored — Tony Tony, Mar 03 '21 at 20:39
Sure, I understand that. But my question is **why** is it important? It makes a difference to the answer. Does putting two of those commands on the same line create an ambiguity? If so, please show enough of the grammar to understand the ambiguity. Or, is it just that you don't like the way it looks without a new line? In that case, there are completely different strategies. — rici, Mar 03 '21 at 21:47
language has some certain rules , one of the rules i want to implement is that when you write the code and you declare multiple libraries without NewLines between them should be a syntax error — Tony Tony, Mar 03 '21 at 22:04
now when i return the newLine in my lex without ignoring it , sure i can make every library in a line , but then in my whole code i have to make sure that the newLines have to be ignored ( between instructions , declarations ) — Tony Tony, Mar 03 '21 at 22:07
i want the newLine between each library declaration and at the same time it should be ignored in my whole code , brother i think i answered your question and made it clear that declaring Libraries in the same line should be a syntaxic error , and making other declarations or instructions with or without newLines shouldn't matter it should still be correct — Tony Tony, Mar 03 '21 at 22:20
But you haven't said *why.* It isn't the job of the compiler to enforce style rules or coding standards. The compiler should be as simple as possible. Just ignore it. — user207421, Mar 03 '21 at 22:42
yes but it's one of the syntaxic constraints of this compiler — Tony Tony, Mar 03 '21 at 22:49

score 0 · Answer 1 · answered Mar 04 '21 at 00:40

The usual way of doing this sort of thing is to use lexer start conditions (sometimes called start states), and have either the lexer or parser set the current start condition at the appropriate time. You can find info on this in the documentation for flex, but basically, your lexer would be something like:

%s SPECIAL

%%

<INITIAL>[\n]     ;   /* normally ignore newlines */
<SPECIAL>[\n]     return *yytext;   /* but not in the SPECIAL state */

/* other flex rules */

%%

void enableNewlines() { BEGIN(SPECIAL); }
void disableNewlines() { BEGIN(INITIAL); }

then in your bison code, you might have actions like

input: bang command '\n'  {
    /* handle the command */
    disableNewlines(); }

bang: '!'  { enableNewlines(); }

the basic idea is that when you recognized enough of something to know that there's a required newline coming up, you go into that special start state, and then once you've parsed the newline, you go back into the 'normal' start state. You need to be careful of the exact sequencing as in some cases, bison will read ahead a token (getting the lookahead) before reducing an action.

An alternative approach that may be simpler or more complex is to have the lexer always return newlines and have your grammar explicitly ignore them everywhere they might occur. This may be verbose (lots of places in the grammar).

How to handle a newLine properly

1 Answers1