0

I am using text/scanner package to parse some arbitrary expressions. I am currently trying to implement a not in option, that is, if the current identifier is not, and the next is in, parse it using function notin(left, right), and otherwise we parse it as negate(right).

I've essentially got the code to manage these cases however, I am unable to rewind the scanner in case the next token is not in. I've tried by recording the position and then reassigning it later, but to no avail and haven't been able to find a different solution.

func readToken(stream *scanner.Scanner) {
    switch stream.Scan() {
    case scanner.Ident:
        switch stream.TokenText() {
        case "in":
            in(left, right)
        case "not":
            oldPosition := stream.Position
            nextToken := stream.Scan()
            if nextToken == scanner.Ident {
                switch stream.TokenText() {
                case "in":
                    fmt.Println("notin")
                default:
                    // how do we rewind the scanner?
                    stream.Position = oldPosition 
                    fmt.Println("negate default")
                }
            } else {
                fmt.Println("negate no-ident")
            }
        }
    }
}

How can I rewind the scanner when I don't find a valid identifier?

Edit, I also tried using Peek() as below, but that still changes the state to the point that I'd need to rewind as well.

// other code
case "not":
    nextIdent, err := getNextIdent(stream)
    if err != nil {
        fmt.Println("negate no-ident")
    } else {
        switch nextIdent {
        case "in":
            fmt.Println("notin")
        default:
            fmt.Println("negate default")
        }
    }
// other code


func getNextIdent(s *scanner.Scanner) (string, error) {
    var nextIdent string

    ch := s.Peek()

    // skip white space
    for s.Whitespace&(1<<uint(ch)) != 0 {
        ch = s.Next()
    }

    if isIdentRune(ch, 0) {
        nextIdent = string(ch)
        ch = s.Next()
        nextIdent += string(ch)
        for i := 1; isIdentRune(ch, i); i++ {
            ch = s.Next()
            if s.Whitespace&(1<<uint(ch)) != 0 {
                break
            }
            nextIdent += string(ch)
        }
        return nextIdent, nil
    }

    return "",errors.New("not a ident")
}

Note, the code I've got is a forked from Knetic/govaluate combined with a PR from GH user generikvault and some other forks. The full code can be found on my Github profile

GManz
  • 1,548
  • 2
  • 21
  • 42
  • 1
    Have you tried replacing `nextToken := stream.Scan()` with `nextToken := stream.Peek()`? Of course, you'll have to adapt some of your code. – Federico Oct 13 '19 at 01:03
  • 2
    You can't rewind a scanner. Your only option, if `Peek` is not sufficient, is to buffer the input, and scan it twice. – Jonathan Hall Oct 13 '19 at 09:22
  • You don't mention how you are actually parsing, but if you are using something like recursive descent, there is no need for the scanner to make this distinction because the two uses of `not` are in disjoint contexts: one is a prefix operator; the other is part of an infix operator. You should know before you request the token which one would be valid. After you see a `not` token, a following `in` would either be required or illegal, depending upon which type of `not` it is. – rici Oct 14 '19 at 05:56
  • @Federico `Peek()` gives you a single rune, which isn't enough to figure out what's the next token is. – GManz Oct 14 '19 at 21:21
  • @Flimzy I guess it doesn't necessarily need to be a rewind option as long as I can see if `not` comes after `in`. I've tried to buffer (providing I understood what you mean), but to no avail - see the updated answer. – GManz Oct 14 '19 at 21:39

1 Answers1

2

By looking at the API references of text/scanner, I can't seem to find a way to rewind the scanner the way you want.

However, the Peek() method would get you the next rune without advancing the scanner. Inside the "not" case, you can use it to peek in advance to see if it matches.

masnun
  • 11,635
  • 4
  • 39
  • 50
  • `Peek()` only gives you 1 rune, which doesn't help. I tried using `Next()` too but that just changes the state as well which gets me back to need to rewind. – GManz Oct 14 '19 at 21:20
  • You can keep `Peek()`-ing as far as you wish. Since it doesn't change the state, you may even iterate the entire document. – masnun Oct 15 '19 at 04:03
  • I tried using `Peek()` in a loop but to it only returns the same rune, whereas `Next()` always returns the next run but it does change the state. – GManz Oct 27 '19 at 16:32
  • So `fmt.Printf("%v;%v;%v\n%v;%v;%v\n", s.Peek(), s.Peek(), s.Peek(), s.Next(), s.Next(), s.Next())` produces `32;32;32\n32;105;110` – GManz Oct 27 '19 at 18:17