Define and in a lexer

Question

I am trying to implement a front end which attempts to conform to a subset of this specification.

It seems that many things are clearly defined in the reference, except <LINE-START> and <LINE-END>, which are nevertheless often used.

Here is a citation: "For ease of specification it is convenient to be able to explicitly refer to the point that immediately precedes the beginning of a logical line and the point immediately preceding the final line-terminator of a logical line. This is accomplished using <LINE-START> and <LINE-END> as terminal symbols of the VBA grammars. A <LINE-START> is defined to immediately precede each logical line and a <LINE-END> is defined as replacing the <line-terminator> at the end of each logical line:"

Here are some examples:

line-terminator = (%x000D %x000A) / %x000D / %x000A / %x2028 / %x2029
line-continuation = *WSC underscore *WSC line-terminator
WS = 1*(WSC / line-continuation)

EOL = [WS] LINE-END
logical-line = LINE-START *extended-line LINE-END

if-statement = LINE-START “If” boolean-expression “Then” EOL
               statement-block
               *[else-if-block]
               [else-block]
               LINE-START ((“End” “If”) / “EndIf”)

else-if-block = LINE-START “ElseIf” boolean-expression “Then” EOL
                LINE-START statement-block

else-block = LINE-START “Else” statement-block

Does anyone know where and how to define <LINE-START> and <LINE-END>?

Welcome to rolling your own lexer and parser, based on real documents. Much of the energy that goes into implementing such devices is caused by the lack of precision of specification, followed by the differences in the real implementation vs. whatever was specified, which you get to discover and fix by trial and error. — Ira Baxter, Aug 08 '12 at 17:28

Chris Walton · Answer 1 · 2012-08-08T16:54:04.847

Given the multiple languages described in the document, I would expect the definition of line start and line end to be part of table 3.2.2 Logical Line Grammar, if these are (as suggested elsewhere in the document) tokens. As you say, they are not defined very clearly.

There is another possibility. The way these terms are used suggests an analogy with BOF/EOF. This would suggest that they are states, independent of the token system used. If this is the case you will need to define these as part of your parser when the appropriate conditions hold. The definition will depend on several factors - the position in the file; the previous, the current, and possibly the next token. (For example, Start Line is the state the document needs to be put into, after advancing from the End Line state, but not if also EOF). These rules will have to be derived from the (fragmentary) definition given, and from appropriate assumptions.

This is not a complete answer, but given the vagueness of the specification on this point, and the multiple usage of these tokens?/states? I find it difficult to see how a more complete answer can be given. HTH.

user1587368 · Answer 2 · 2012-08-11T20:20:56.190

In VBA, you can write that command

Dim from
Dim to
FakeTest from, to

LINE-START refers to the start of a line but before the first word of line and not the very start line.

Also, LINE-END refers to the end of line but not so because you can add whitespace and tabulation before the end of line.

Note, that spaces in VBA are recognized as valid separators to make a difference between two words, especially when you are calling a method :

CallMethod param1, param2, _
           param3

VBA doesn't allow to insert parenthesis expressed if there is a result, as you can see in that example:

result = CallAMethod(param1, param2)

Also, you don't need to add a semi-colon, nor a colon to separate statements.

I say : whitespace and tabulation are part of the language syntax. This is not the case in others programming languages.

Define and in a lexer

2 Answers2