gofmt preserving of newlines

Question

When formatting go source code with gofmt, it preserves the newlines so you can group items together. I'm interested on how this is actually implemented. I tried looking at the source code in the github repo golang/go, but couldn't find it immediately. If you look at https://github.com/golang/go/blob/master/src/go/printer/printer.go#L979:

// intersperse extra newlines if present in the source

How does the printer know those extra newlines are present in the source? Can someone point me into the right direction?

Answer 1

Unlike most lexers, the go lexer is including tokens which are often removed or elided by a compiler's lexer. The stream of tokens emitted by the lexer includes, among others, tokens for comments, implied semicolons, newlines, formfeeds (FF), and other whitespace. This allows the same token stream to be used to regenerate the source, and to create structures required by the compiler, such as the AST.

Answer 2

gofmt works on the AST. When you look at https://golang.org/pkg/go/ast you'll see that every node has functions Pos() and End() which return the token.Pos of the beginning and end respectively. These are essentially offsets in the source file and as such know nothing about line numbers/breaks.

But when combined with a token.Fileset such a token.Pos can be converted into a token.Position which includes the line number. gofmt does that in the function printer.go:lineFor().

The actual insertion of linebreaks is done in nodes.go:linebreak(). The first argument to linebreak() is a line number obtained by calling the aforementioned lineFor() on the respective token.Pos. The function computes the difference between this line number and the line number of the last token that was printed (tracked in the pos field of struct printer). This tells it if the token to be printed now is on the same line in the input file as the previous token. If it isn't, that means the programmer included one or more line breaks in the original source and linebreak() will output at most 1 empty line. While it could preserve all input line breaks, gofmt's policy is to compress series of empty lines down to only 1 empty line.

If the reason you're asking this question is that you want to customize gofmt, take a look at https://github.com/mbenkmann/goformat

Answer 3

In the internal.go package `line 40-41 there's this:

// Insert using a ;, not a newline, so that the line numbers

// in psrc match the ones in src.

And then this:

psrc := append([]byte("package p;"), src...)
file, err = parser.ParseFile(fset, filename, psrc, parserMode)

is that what you are looking for? If I got your question right.