6

Most of the output methods in Go's io package return (int, error), for example io.Writer's Write([]byte) method and the io.WriteString(io.Writer, string) function. However, a few of the output methods, such as io.WriterTo's WriteTo method, return (int64, error) instead. This makes it inconvenient to implement WriteTo in terms of Write or WriteString without storing an intermediate value and type converting it from int to int64. What is the reason for this discrepancy?

Matt
  • 21,026
  • 18
  • 63
  • 115

2 Answers2

6

It's possible that WriteTo copies more than int32 bytes of data.

With the io.Readerand io.Writer interfaces, the amount of data is limited by the size of the given slice, which has a length limited by int for the current architecture.

JimB
  • 104,193
  • 13
  • 262
  • 255
  • I might agree outright but it's interesting to note that in the language spec, an int might be either 32 or 64 bits, depending on the implementation/architecture. Without looking at other packages that use the `WriterTo` and `Writer` interfaces my guess is the authors might've intuited that the `WriterTo`/`ReaderFrom` interfaces had more potential for having to deal with larger amounts of data. – Jerry Saravia Apr 15 '15 at 21:06
  • @JerrySaravia: that's the point, WriterTo and ReaderFrom *can* handle larger amounts of data. They're not limited to using a single slice for the operation. There's nothing to guess about. – JimB Apr 15 '15 at 21:11
  • That's a good point. I guess `WriterTo` could be copying a file that is larger than 2GB or something like that. But then why not just use `int64` everywhere for consistency? – Matt Apr 15 '15 at 21:20
  • 2
    @Matt: because it's *more* consistent to return an `int` everywhere where it's the appropriate data type, rather than convert everything to `int64` for the fewer cases where that's possible on some system. The value from Read and Write is in terms of the length of a slice, and is often used to index into slices. There would be a lot more conversion to do the other way. – JimB Apr 15 '15 at 21:47
  • @JimB Are slices limited in size? I thought their only limit was the size of the available memory. – Jerry Saravia Apr 16 '15 at 01:29
  • @JerrySaravia: all that means is the length and capacity of slices are limited by the size of `int`. On 386 or amd64p32 you can allocate the entire 32bit address space, but a slice can only have a capacity of MaxInt32. – JimB Apr 16 '15 at 13:57
  • @JimB Thanks, that makes things clearer to me. Thanks. On 64 bit implementations of, int the two end up being equivalent. But on 32 bit implementations the WriterTo is specific and says, "expect that we might right more than a 32 bit int's worth of data." – Jerry Saravia Apr 16 '15 at 18:13
5

The signature of the Writer.Write() method:

Write(p []byte) (n int, err error)

It writes the contents of a slice. Quoting from the spec: Slice types:

A slice is a descriptor for a contiguous segment of an underlying array...

As we all know, the slice has an underlying array. Quoting again from the Spec: Array types:

The length is part of the array's type; it must evaluate to a non-negative constant representable by a value of type int.

So the maximum length of an array is limited by the maximum value of the int type (which is 2147483647 in case of 32 bit and 9223372036854775807 in case of 64 bit architectures).

So back to the Writer.Write() method: since it writes the content of the passed slice, it is guaranteed that the number of written bytes will not be more that what fits into an int.

Now WriteTo.WriteTo() method:

WriteTo(w Writer) (n int64, err error)

A slice or array is nowhere mentioned. You have no guarantees that the result will fit into an int, so the int64 is more than justified.

Example: BigBuffer

Imagine a BigBuffer implementation which temporarily writes data into an array or slice. The implementation may manage multiple arrays so that if one is full (e.g. reached max int), continues in another one. Now if this BigBuffer implements the WriteTo interface and you call this method to write the content into an os.File, the result will be more than max int.

icza
  • 389,944
  • 63
  • 907
  • 827
  • A more trivial example would be any random type that implemented `WriteTo` for something that could have a >31 bit representation (e.g. a type representing/holding a 2.01 GB video; or a silly `type Zeros int64` type that is defined as writing out that many zeros). – Dave C Apr 16 '15 at 16:15