How to expand a Tensorflow Variable

Question

Is there any way to make a Tensorflow Variable larger? Like, let's say I wanted to add a neuron to a layer of a neural network in the middle of training. How would I go about doing that? An answer in This question told me how to change the shape of the variable, to expand it to fit another row of weights, but I don't know how to initialize those new weights.

I figure another way of going about this might involve combining variables, as in initializing the weights first in a second variable and then adding that in as a new row or column of the first variable, but I can't find anything that lets me do that either.

score 12 · Answer 1 · edited May 23 '17 at 12:26

12

There are various ways you could accomplish this.

1) The second answer in that post (https://stackoverflow.com/a/33662680/5548115) explains how you can change the shape of a variable by calling 'assign' with validate_shape=False. For example, you could do something like

# Assume var is [m, n] 
# Add the new 'data' of shape [1, n] with new values
new_neuron = tf.constant(...)  

# If concatenating to add a row, concat on the first dimension.
# If new_neuron was [m, 1], you would concat on the second dimension.
new_variable_data = tf.concat(0, [var, new_neuron])  # [m+1, n]

resize_var = tf.assign(var, new_variable_data, validate_shape=False)

Then when you run resize_var, the data pointed to by 'var' will now have the updated data.

2) You could also create a large initial variable, and call tf.slice on different regions of the variable as training progresses, since you can dynamically change the 'begin' and 'size' attributes of slice.

edited May 23 '17 at 12:26

Community

1
1

answered Jan 21 '16 at 06:02

vrv

411
2
6

If I added a new variable of shape [m, 1], would the final shape be [m, n+1]? – Beez Jan 21 '16 at 18:25
And wouldn't pack just create a variable with a third dimension? That's what the API says, and I'm getting errors when pack is run saying the dimensions are not compatible unless I make the dimensions the same size, in which case it adds a third dimension of size 2. – Beez Jan 21 '16 at 18:59
Ah, you are right about pack. I guess you could do tf.concat(0, [...tensors...]) (or concatenating on whichever dimension you would like to create the shape you desire) I've edited the answer to reflect this. – vrv Jan 21 '16 at 20:35
concat also requires the dimensions to be the same size. I'm currently working on an answer that requires you to unpack and then pack again. I've got it half done, but the other half is giving me errors at the moment. I'll put it up once it's working. – Beez Jan 21 '16 at 21:09

score 4 · Answer 2 · answered Oct 17 '16 at 01:36

4

Simply using tf.concat for expand a Tensorflow Variable,you can see the api_docs for detail.

    v1 = tf.Variable(tf.zeros([5,3]),dtype=tf.float32)
    v2 = tf.Variable(tf.zeros([1,3]),dtype=tf.float32)
    v3 = tf.concat(0,[v1, v2])

answered Oct 17 '16 at 01:36

蔡俊浩

41
2

score 2 · Accepted Answer · edited May 02 '18 at 09:57

Figured it out. It's kind of a roundabout process, but it's the only one I can tell that actually functions. You need to first unpack the variables, then append the new variable to the end, then pack them back together.

If you're expanding along the first dimension, it's rather short: only 7 lines of actual code.

#the first variable is 5x3
v1 = tf.Variable(tf.zeros([5, 3], dtype=tf.float32), "1")

#the second variable is 1x3
v2 = tf.Variable(tf.zeros([1, 3], dtype=tf.float32), "2")

#unpack the first variable into a list of size 3 tensors
#there should be 5 tensors in the list
change_shape = tf.unpack(v1)

#unpack the second variable into a list of size 3 tensors
#there should be 1 tensor in this list
change_shape_2 = tf.unpack(v2)

#for each tensor in the second list, append it to the first list
for i in range(len(change_shape_2)):
  change_shape.append(change_shape_2[i])

#repack the list of tensors into a single tensor
#the shape of this resultant tensor should be [6, 3]
final = tf.pack(change_shape)

If you want to expand along the second dimension, it gets somewhat longer.

#First variable, 5x3
v3 = tf.Variable(tf.zeros([5, 3], dtype=tf.float32))

#second variable, 5x1
v4 = tf.Variable(tf.zeros([5, 1], dtype=tf.float32))

#unpack tensors into lists of size 3 tensors and size 1 tensors, respectively
#both lists will hold 5 tensors
change = tf.unpack(v3)
change2 = tf.unpack(v4)

#for each tensor in the first list, unpack it into its own list
#this should make a 2d array of size 1 tensors, array will be 5x3
changestep2 = []
for i in range(len(change)):
  changestep2.append(tf.unpack(change[i]))

#do the same thing for the second tensor
#2d array of size 1 tensors, array will be 5x1
change2step2 = []
for i in range(len(change2)):
  change2step2.append(tf.unpack(change2[i]))

  #for each tensor in the array, append it onto the corresponding array in the first list
  for j in range(len(change2step2[i])):
    changestep2[i].append(change2step2[i][j])

  #pack the lists in the array back into tensors
  changestep2[i] = tf.pack(changestep2[i])

#pack the list of tensors into a single tensor
#the shape of this resultant tensor should be [5, 4]
final2 = tf.pack(changestep2)

I don't know if there's a more efficient way of doing this, but this works, as far as it goes. Changing further dimensions would require more layers of lists, as necessary.

Note that tf.concat() concatenates tensors. E.g., your example 1 can be: v1 = tf.variable(...[5, 3]...) v2 = tf.variable(...[1, 3]...) final = tf.concat(0, [v1, v2]) You second example can be done: v1 = tf.variable(...[5, 3]...) v2 = tf.variable(...[5, 1]...) final = tf.concat(1, [v1, v2]) I think that's what vrv suggested. — zfc, Jan 23 '16 at 17:11

How to expand a Tensorflow Variable

3 Answers3

Linked