0

So I have this var resolve of type map[string][]byte that is being initialised by calling a method. If I just iterate resolve the value is being printed correctly.

for k, v := range resolve {
        fmt.Printf("%s\t%s\n", k, v)
}

But in the very next line I am trying to iteration over the map to store the values in a db (bolt), in that value for a key (key1) in the map is being changed automatically and I am not able to figure out why. To further simplify that what I did is stored the value for that key in a new var

a:= resolve["key1"]

and then while storing the values in the db I checked if the key is key1 store a. In that case also the value of a is being changed which should not.

This gist has code we would be able to see that resolve that we have in line 30 has been change in line 34.

I have added the code in the go playground here is the link https://play.golang.org/p/2WacK-xxRp_m

viveksinghggits
  • 661
  • 14
  • 35
  • Can you plz post the complete program possibly on go playground? – Bhupinder Singh Narang Jul 15 '20 at 08:28
  • @BhupinderSinghNarang I am not able to replicate that, thats happening with the code that I have where I iterate the tings and insert them in boltstore. – viveksinghggits Jul 15 '20 at 08:29
  • 2
    Please define absolutely precisely what do yo mean by "value for a key is being changed". What I mean is that your values are slices. A slice _value_ is a `struct` with 3 fields: a pointer to the so-called backing array, the length and the capacity. Assigning a slice value to N variables would produce N variables _all sharing the same undelying array._ That is, mutating of any of the slice element via _any_of those variables will be "seen through" all the N-1 remaining variables. So we have do discern changing of map values from changing of element of slices which are map's values. – kostix Jul 15 '20 at 08:33
  • Hey @kostix I understand what you are saying the but lets take an example of this code https://play.golang.org/p/9idNSwfFi2Y, in that case value of `b` is not being changed right, and thats what I was expecting in my code as well. – viveksinghggits Jul 15 '20 at 08:39
  • added code snippet. – viveksinghggits Jul 15 '20 at 09:05
  • @viveksinghggits: so, the code in the playground is behaving like it should? And we still have no idea what happens in your "broken" code? Sounds like succeeding in reproducing this on playground will get you 80% of the solution. – Sergio Tulentsev Jul 15 '20 at 09:14
  • @SergioTulentsev the gist that is added has the broken code if you want to look at that. And yes code in playground has expected behaviour. – viveksinghggits Jul 15 '20 at 09:16
  • In your gist, `someFunction` mutates the input map and returns the same map. Otherwise I see nothing dangrous _unless_ some concurrent code which is not shown is also being run. Can you try building your program with `go build -race` and re-try? If the map is really gets changed "miraculously", the race detector will likely to spot that and crash your process. – kostix Jul 15 '20 at 09:17
  • @viveksinghggits: the output of that gist (from your comment) is what I'd expect to see. You set vivek-11 to key 1 and this value is used after. What is the problem, again? You thought changes to `dbData` from `myFunction` should not have been applied to `resolve`? – Sergio Tulentsev Jul 15 '20 at 09:19
  • Wait, ain't you're trying to explain the _order_ of elements printed is different each time? – kostix Jul 15 '20 at 09:20
  • @SergioTulentsev if we see the output in the gist the problem is why the values of key 2, 3 and 4 are being changed and that is the problem – viveksinghggits Jul 15 '20 at 09:22
  • 1
    @SergioTulentsev here is the code in playground https://play.golang.org/p/2WacK-xxRp_m – viveksinghggits Jul 15 '20 at 09:23
  • 1
    @viveksinghggits: ah, I see. And thanks for the playground. Now let me stare at the code :) – Sergio Tulentsev Jul 15 '20 at 09:26
  • 2
    The problem must be somewhere in BoltDB and how it handles writing bigger values for the same key. If we don't increase the size (`dbData["1"] = []byte("vivek-9")`), then it works (as in, the other values are not affected). Tried a quick glimpse into github.com/boltdb/bolt, but nothing jumps out so far. If I were to guess, the byte slices you get from ReadAll, they're "windows" into the one big backing byte array. When you insert a bigger value, it causes subsequent data to shift. But the windows don't move and so they're now observing different data. Clone your values in ReadAll. – Sergio Tulentsev Jul 15 '20 at 09:46
  • @SergioTulentsev okay so even if we assume the data that is there is in the BoltDB is somehow being corrupted, the in memory var `resolve` that we have, should not be changed right. Because resolve is not being updated from where its works and it doesnt. – viveksinghggits Jul 15 '20 at 09:49
  • 1
    @viveksinghggits it is not changed. This is how slices work in go. They don't hold data. A slice is a tiny data structure that says "there's N bytes of data at address 0x12345". If something were to overwrite memory at that range, the slice would happily read the updated bytes when you iterate it next time. – Sergio Tulentsev Jul 15 '20 at 09:55

2 Answers2

3

On your line in readAll:

lGraceP[string(k)] = v

you're storing the value for later use. The documentation specifies that the value v is not valid after the transaction ends.

From Cursor.First (and there's similar text in Cursor.Next) in the bolt library (see the highlighted text):

First moves the cursor to the first item in the bucket and returns its key and value. If the bucket is empty then a nil key and value are returned. The returned key and value are only valid for the life of the transaction.

The way that the key and value are only valid for the life of the transaction is that the array underlying their slices are re-used. That causes your values to mutate unexpectedly.

Paul Hankin
  • 54,811
  • 11
  • 92
  • 118
  • It seems this limitation is only for the values that are being read using `.Get` method. – viveksinghggits Jul 15 '20 at 10:58
  • Why do you say it seem that the limitation is only for the values being read using `.Get`? The text from the documentation of `Cursor.First` and `Cursor.Next` says that it applies to those methods. – Paul Hankin Jul 15 '20 at 11:30
  • because [this section](https://github.com/boltdb/bolt#iterating-over-keys) doesnt say anything about values not being correct without transaction. – viveksinghggits Jul 15 '20 at 11:34
  • but I agree, I think I will have to copy things to another slice. – viveksinghggits Jul 15 '20 at 11:48
  • this is the solution https://play.golang.org/p/wkRVh_JUiqP – viveksinghggits Jul 15 '20 at 13:38
  • 1
    I think just `lGraceP[string(k)] = append([]byte{}, v...)` is sufficient. No need to copy the key (because you're converting it to a string anyway), and you can duplicate the value inline using `append`. – Paul Hankin Jul 15 '20 at 13:50
2

The problem must be somewhere in BoltDB and how it handles writing bigger values for the same key. If we don't increase the size

dbData["1"] = []byte("vivek-9"))

then it works (as in, the other values are not affected). Tried a quick glimpse into boltdb/bolt, but nothing jumps out so far.

If I were to guess, the byte slices you get from ReadAll are "windows" into the one big backing byte array. When you insert a bigger value, it causes subsequent bytes to shift. But the windows don't move and so they're now observing different data. Clone your values in ReadAll, this will help.

Sergio Tulentsev
  • 226,338
  • 43
  • 373
  • 367