How can I compare two source code files/ ast trees?

Question

I'm generating some source code using the templates package( is there a better method? )and part of the testing I need to check if the output matches the expected source code.

I tried a string comparison but it fails due the extra spaces / new lines generated by the templates package. I've also tried format.Source with not success. ( FAIL)

I tried to parse the ast of the both sources (see bellow) but the ast doesn't match either even if the code is basically same except the new lines / spaces. (FAIL)

package main

import (
    "fmt"
    "go/parser"
    "go/token"
    "reflect"
)

func main() {
    stub1 := `package main
     func myfunc(s string) error {
        return nil  
    }`
    stub2 := `package main

     func myfunc(s string) error {

        return nil

    }`
    fset := token.NewFileSet()
    r1, err := parser.ParseFile(fset, "", stub1, parser.AllErrors)
    if err != nil {
        panic(err)
    }
    fset = token.NewFileSet()
    r2, err := parser.ParseFile(fset, "", stub2, parser.AllErrors)
    if err != nil {
        panic(err)
    }
    if !reflect.DeepEqual(r1, r2) {
        fmt.Printf("e %v, r %s, ", r1, r2)
    }
}

Playground

Do you want to compare arbitrary trees, or just go trees that you have parsed? — Ira Baxter, May 15 '15 at 04:05

score 8 · Answer 1 · answered May 12 '15 at 17:35

Well, one simple way to achieve this is to use the go/printer library, that gives you better control of output formatting, and is basically like running gofmt on the source, normalizing both trees:

package main
import (
    "fmt"
    "go/parser"
    "go/token"
    "go/printer"
    //"reflect"
    "bytes"
)

func main() {
    stub1 := `package main
     func myfunc(s string) error {
        return nil  
    }`
    stub2 := `package main

     func myfunc(s string) error {

        return nil

    }`

    fset1 := token.NewFileSet()
    r1, err := parser.ParseFile(fset1, "", stub1, parser.AllErrors)
    if err != nil {
        panic(err)
    }
    fset2 := token.NewFileSet()
    r2, err := parser.ParseFile(fset1, "", stub2, parser.AllErrors)
    if err != nil {
        panic(err)
    }

    // we create two output buffers for each source tree
    out1 := bytes.NewBuffer(nil)
    out2 := bytes.NewBuffer(nil)

    // we use the same printer config for both
    conf := &printer.Config{Mode: printer.TabIndent | printer.UseSpaces, Tabwidth: 8}

    // print to both outputs
    if err := conf.Fprint(out1, fset1, r1); err != nil {
        panic(err)
    }
    if err := conf.Fprint(out2, fset2, r2); err != nil {
        panic(err)
    }


    // they should be identical!
    if string(out1.Bytes()) != string(out2.Bytes()) {
        panic(string(out1.Bytes()) +"\n" + string(out2.Bytes()))
    } else {
        fmt.Println("A-OKAY!")
    }
}

Of course this code needs to be refactored to not look as stupid. Another approach is instead of using DeepEqual, create a tree comparison function yourself, that skips irrelevant nodes.

The comparison function may not be trivial so I'm trying to avoid that. printer seems to fail on more "complex" structures http://play.golang.org/p/I9cAVEYLAm — themihai, May 14 '15 at 14:43
@mihai maybe traverse the tree and filter it, then use DeepEqual? — Not_a_Golfer, May 14 '15 at 20:02
Any idea how exactly can I filter the irrelevant nodes? I've tried to remove the nil nodes assuming they represent the new lines / spaces but it doesn't seem to be the case. play.golang.org/p/JVVpKIzela — themihai, May 15 '15 at 01:45
A nit: `bytes.Buffer` has a `String()` method, so you don't need all those conversions. — Ainar-G, May 15 '15 at 06:36

themihai · Accepted Answer · 2015-05-22T10:46:53.337

This was easier than I thought. All I had to do was to remove the empty new lines(after formatting). Below is the code.

    package main

    import (
        "fmt"
        "go/format"
        "strings"
    )

    func main() {
        a, err := fmtSource(stub1)
        if err != nil {
            panic(err)
        }
        b, err := fmtSource(stub2)
        if err != nil {
            panic(err)
        }
        if a != b {
            fmt.Printf("a %v, \n b %v", a, b)
        }
    }

func fmtSource(source string) (string, error) {
    if !strings.Contains(source, "package") {
        source = "package main\n" + source
    }
    b, err := format.Source([]byte(source))
    if err != nil {
        return "", err
    }
    // cleanLine replaces double space with one space
    cleanLine := func(s string)string{
        sa := strings.Fields(s)
        return strings.Join(sa, " ")
    }
    lines := strings.Split(string(b), "\n")
    n := 0
    var startLn *int
    for _, line := range lines {
        if line != "" {
            line = cleanLine(line)
            lines[n] = line
            if startLn == nil {
                x := n
                startLn = &x
            }
            n++
        }
    }
    lines = lines[*startLn:n]
    // Add final "" entry to get trailing newline from Join.
    if n > 0 && lines[n-1] != "" {
        lines = append(lines, "")
    }


    // Make it pretty 
    b, err = format.Source([]byte(strings.Join(lines, "\n")))
    if err != nil {
        return "", err
    }
    return string(b), nil
}

Honestly, i think a better approach would've been to add a `hashCode` method to every AST node, then to match them you just check if both AST have the same hashCode — Lorenzo, Aug 25 '22 at 00:51

How can I compare two source code files/ ast trees?

2 Answers2