-1

I'm trying to use a slice, say mySlice, with a very large starting index. Rather than explicitly subtracting the starting index by always using it as mySlice[index - mySliceStartIndex], I am tempted to simply define the slice in such a way that I can use it without such arithmetic as mySlice[index]. Can this be done without allocating memory for all the unused low indices?

The naive way of doing this, allocating a slice and then reslicing it (e.g. mySlice = mySlice[3*1024*1024*1024:4*1024*1024*1024]) is obviously memory inefficient because the underlying array not only needs to be allocated for the entire range, but remains allocated. does not even work, because afterwards the data formerly at index 3*1024*1024*1024 is now at index 0, whilst my goal is to keep it at the original index.

Can I allocate the slice (or its underlying array) in such a way that indices below the slice's start are not allocated, ideally not even initially?

  • What you want is unclear. A slice is allocated over an array. If you don't want the part before the slice to be allocated, simply start the slice at the beginning of the array. – Denys Séguret Mar 10 '14 at 09:24
  • you want indexing to start on index > 0? – Arjan Mar 10 '14 at 09:26
  • @Arjan Yes, exactly. I've edited the question's title to reflect that as early as possible. –  Mar 10 '14 at 09:27
  • I'm still trying to sort of figure out what exactly it is you're concerned about here, so bear with what might sound like an odd question: is it that you're concerned that reslicing allocates a new underlying array for each slice of an original slice? (Bearing in mind that it doesn't.) Or are you worried that you have unused data before/after the ends of a slice that you're no longer using? Or neither? –  Mar 10 '14 at 09:47
  • @nil I'm concerned with the unused data before the beginning of the slice: Assume I have a machine with only 128 GB of memory, and I'd like to only use the last 32 GB of an array that occupies 128 GB in memory. Can I create a slice where only those last elements are actually backed by an array, reducing my memory footprint from 100% of what is available to 25%? And if it is possible at all, would it also work for going from 400% to 25%? –  Mar 10 '14 at 09:50
  • Well, you could allocate a _new_ slice of a smaller size and use `copy` to copy from a portion of the original slice to the new slice. If, then, the original array is collected by the GC, you'll obviously reduce your memory usage. If you're asking if you can resize an array in place, the answer's no. (Alternatively to `copy`, there's also this little trick: http://stackoverflow.com/a/16748424/457812) –  Mar 10 '14 at 09:57
  • @nil Argh. You made me realized I've had a major misconception about slices, thinking I could slice an array in a way such that behind the scenes it subtracts (rather than adds) a positive offset to the index, such that the slice starts not in the underlying array at a non-zero index, but rather in the index to the slice. Which is silly seeing how slices work otherwise---and I have no idea anymore how I got there first. Thanks for putting me on the right track. –  Mar 10 '14 at 10:04
  • I'd just think of slices as windows into an underlying array (i.e., you can see what's visible through the window, but it might not be everything). Keeps it simple. At any rate, glad I could help somehow, have fun coding. –  Mar 10 '14 at 10:05
  • Thanks! Now I wonder if I should delete this question, though, seeing that it most likely won't help anyone else anyways. –  Mar 10 '14 at 10:07

1 Answers1

3

This will not be possible without actually /not/ allocating the unused parts. The way a slice is defined in Go, is through a reflect.SliceHeader

type SliceHeader struct {
    Data uintptr
    Len  int
    Cap  int
}

It contains no starting index field. Merely a reference to an underlying, fixed size array. It is this underlying array which holds your actual data. The slice is simply a 'window' into that array, which always begins at index 0. Wherever 0 may be in the underlying array.

For instance, consider the following code:

a := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
b := a[2:8]
c := a[8:]
d := b[2:4]

This yields a memory layout as follows:

fixed array: [ 0 1 2 3 4 5 6 7 8 9 ]  > [10]int at address 273785072
slice a    :   . . . . . . . . . .    > SliceHeader{Data:273785072 Len:10 Cap:10}
slice b    :       . . . . . .        > SliceHeader{Data:273785080 Len:6 Cap:8}
slice c    :                   . .    > SliceHeader{Data:273785104 Len:2 Cap:2}
slice d    :           . .            > SliceHeader{Data:273785088 Len:2 Cap:6}

The values for Data are simply address offsets into the fixed array and all four slices share the underlying storage.

a =:= $273785072
b =:= $273785080 =:= $a + sizeof(int)*2 =:= $a + 8
c =:= $273785104 =:= $a + sizeof(int)*8 =:= $a + 32
d =:= $273785088 =:= $b + sizeof(int)*2 =:= $a + sizeof(int)*4 =:= $a + 16

At whatever index you re-slice an existing slice, the new slice will always be indexed from 0 to len(s), because the address in the underlying fixed array it points to puts it there.

Memory mapping

If your data is loaded from file on a disk, you can have a different option: use syscall.Mmap to provide access to the data through a slice, starting at the desired index. The returned slice is now index from 0 and it covers only the range you specified.

func mmap(fd *os.File, start, size int) ([]byte, error) {
    _, err := fd.Seek(0, 0)
    if err != nil {
        return nil, err
    }

    return syscall.Mmap(int(fd.Fd()), start, size,
        syscall.PROT_READ, syscall.MAP_SHARED)
}

Do not forget to call syscall.Munmap on the returned slice, when you are done using it.

jimt
  • 25,324
  • 8
  • 70
  • 60
  • +1 Thanks for this great answer! This means my only chance to use a slice would be to modify the Data pointer, which surely isn't possibly since it would wreck havoc with the memory management, presumably causing a runtime error just for pointing at a starting address that is not only not allocated but likely outside the heap as well. The memory mapped IO is an interesting idea, perfect if one actually wants to have the entire array on disk anyways. –  Mar 10 '14 at 13:43
  • Indeed, you would be treading into uncharted territories. There be dragons. – jimt Mar 10 '14 at 14:29