ECMAScript 2017: Parsing from nonterminal StringLiteral to String values

Question

I am trying to understand the translation of a string literal to a final String value (consisting of code unit values), following ECMAScript 2017.

---

Relevant Excerpts

5.1.2 The Lexical and RegExp Grammars

A lexical grammar for ECMAScript is given in clause 11. This grammar has as its terminal symbols Unicode code points that conform to the rules for SourceCharacter defined in 10.1. It defines a set of productions, starting from the goal symbol InputElementDiv, InputElementTemplateTail, or InputElementRegExp, or InputElementRegExpOrTemplateTail, that describe how sequences of such code points are translated into a sequence of input elements.

Input elements other than white space and comments form the terminal symbols for the syntactic grammar for ECMAScript and are called ECMAScript tokens. These tokens are the reserved words, identifiers, literals, and punctuators of the ECMAScript language.

5.1.4 The Syntactic Grammar

When a stream of code points is to be parsed as an ECMAScript Script or Module, it is first converted to a stream of input elements by repeated application of the lexical grammar; this stream of input elements is then parsed by a single application of the syntactic grammar.

and

11 ECMAScript Language: Lexical Grammar

The source text of an ECMAScript Script or Module is first converted into a sequence of input elements, which are tokens, line terminators, comments, or white space. The source text is scanned from left to right, repeatedly taking the longest possible sequence of code points as the next input element.

11.8.4 String Literals

StringLiteral ::
    " DoubleStringCharacters_opt "
    ' SingleStringCharacters_opt '

SingleStringCharacters ::
    SingleStringCharacter SingleStringCharacters_opt

SingleStringCharacter ::
    SourceCharacter but not one of ' or \ or LineTerminator
    \ EscapeSequence
    LineContinuation

EscapeSequence ::
    CharacterEscapeSequence
    0 [lookahead ∉ DecimalDigit]
    HexEscapeSequence
    UnicodeEscapeSequence

CharacterEscapeSequence ::
    SingleEscapeCharacter
    NonEscapeCharacter

NonEscapeCharacter ::
    SourceCharacter but not one of EscapeCharacter or LineTerminator

EscapeCharacter ::
    SingleEscapeCharacter
    DecimalDigit
    x
    u

11.8.4.3 Static Semantics: SV

A string literal stands for a value of the String type. The String value (SV) of the literal is described in terms of code unit values contributed by the various parts of the string literal.

and

The SV of SingleStringCharacter :: SourceCharacter but not one of ' or \ or LineTerminator is the UTF16Encoding of the code point value of SourceCharacter.

The SV of SingleStringCharacter :: \ EscapeSequence is the SV of the EscapeSequence.

---

Question

Assume we have string literal 'b\ar'. I now want to follow the above lexical grammar and semantic grammar, to turn the string literal into a set of code unit values.

1. b\ar is recognized as a CommonToken
2. b\ar is further recognized as a StringLiteral
3. StringLiteral is translated to SingleStringCharacters
4. Each code point in SingleStringCharacters is translated to SingleStringCharacter
5. Each SingleStringCharacter without a \ infront is translated to a SourceCharacter
6. \a is recognized as \ EscapeSequence
7. EscapeSequence (a) is translated to NonEscapeCharacter
8. NonEscapeCharacter is translated to SourceCharacter
9. All SourceCharacter's are translated to any Unicode code point
10. Finally, the SV rules are applied to get string values and thus code unit values

The problem I have is that the StringLiteral input element is now:

SourceCharacter, \ SourceCharacter, SourceCharacter

There is no SV rule for \ SourceCharacter, only for \ EscapeCharacter.

Which makes me wonder if I have the order wrong, or misunderstood how the lexical and syntactic grammar is applied.

I am also confused about how the SV rules are applied altogether. Because they are defined to apply to nonterminal symbols, as opposed to terminal symbols (which should be the result after the lexical grammar has been applied).

Any help is deeply appreciated.

Answer 1

Alright, assuming we're going in with a single token 'b\ar', which is as you've said a StringLiteral token. Applying the algorithm defined in 11.8.4.3 Static Semantics: SV as well as 10.1.1 Static Semantics: UTF16Encoding(cp), we follow the SV rules:

1. The SV of StringLiteral:: ' SingleStringCharacters ' is the SV of SingleStringCharacters.
   • Unwrap the quotes since we're recursively running SV on just the SingleStringCharacters part, e.g. SV(b\ar)

2. The SV of SingleStringCharacters:: SingleStringCharacterSingleStringCharacters is a sequence of one or two code units that is the SV of SingleStringCharacter followed by all the code units in the SV of SingleStringCharacters in order.

   This says "call SV every SingleStringCharacter appending results".

   1. SV(b)
      1. The SV of SingleStringCharacter:: SourceCharacter but not one of ' or \ or LineTerminator is the UTF16Encoding of the code point value of SourceCharacter.
         • The codepoint "b" is codeunit \x0062 so the result here is essentially a code unit sequence of a single 16-bit unit \x0062
   2. SV(\a)
      1. The SV of SingleStringCharacter:: \ EscapeSequence is the SV of the EscapeSequence.
         • Essentially SV(EscapeSequence) this SV(a) (no \ prefix)
      2. The SV of EscapeSequence:: CharacterEscapeSequence is the SV of the CharacterEscapeSequence.
         • Basically just passing through SV(a)
      3. The SV of CharacterEscapeSequence:: NonEscapeCharacter is the SV of the NonEscapeCharacter.
         • More pass-through
      4. The SV of NonEscapeCharacter:: SourceCharacter but not one of EscapeCharacter or LineTerminator is the UTF16Encoding of the code point value of SourceCharacter.
         • The codepoint "a" is code unit \x0061, so this results in a single-unit sequence of just \x0061.
   3. SV(r)
      • Following the same steps as for SV(b) this results a single-unit sequence containing \x0072.
3. Merging the sequence SV(b) + SV(\a) + SV(r) back together, the value of the string is the sequence of UTF16 code units [\x0062, \x0061, \x0072]. That sequence of code units results in bar.

Edit:

I though we should first apply the lexical grammar and end up with tokens, and then subsequently apply the SV rules?

The "token" from a lexer's standpoint, is StringLiteral, everything within that is just information on how to parse. EscapeSequence is not a type of token.

SV defines how to break down the StringLiteral token into a sequence of code units.

As states in 11 ECMAScript Language: Lexical Grammar

The source text of an ECMAScript Script or Module is first converted into a sequence of input elements, which are tokens, line terminators, comments, or white space. The source text is scanned from left to right, repeatedly taking the longest possible sequence of code points as the next input element.

These "input elements" are the tokens used by the parser grammar.

Assuming the order of events is right, my second questions is around SV(\a). The first escape sequence rule is applied and we are left with SV(a), which should follow the same path as SV(b) no?

There's more than just the value, there is also the datatype. Using Flow/Typescript-style annotations, you can kind of think of the steps above for

1. The SV of SingleStringCharacter:: \ EscapeSequence is the SV of the EscapeSequence.
2. The SV of EscapeSequence:: CharacterEscapeSequence is the SV of the CharacterEscapeSequence.
3. The SV of CharacterEscapeSequence:: NonEscapeCharacter is the SV of the NonEscapeCharacter.
4. The SV of NonEscapeCharacter:: SourceCharacter but not one of EscapeCharacter or LineTerminator is the UTF16Encoding of the code point value of SourceCharacter.

as if it were an overloaded function, e.g.

function SV(parts: ["\", EscapeSequence]) {
    return SV(parts[1]);
}
function SV(parts: [CharacterEscapeSequence]) {
    return SV(parts[0]);
}
function SV(parts: [NonEscapeCharacter]) {
    return SV(parts[0]);
}
function SV(parts: [SourceCharacter]) {
    return UTF16Encoding(parts[0]);
}

So SV(a) would be kind of like SV("a": [CharacterEscapeSequence]) whereas SV(b) has a different type.