0

I would like to create a function that for every line of a given data X, is applying the softmax function only for some sampled classes, lets say 2, out of K total classes. In simple python the code seems like that:

def softy(X,W, num_samples):
    N = X.shape[0]
    K = W.shape[0]
    S = np.zeros((N,K)) 
    ar_to_sof = np.zeros(num_samples)
    sampled_ind = np.zeros(num_samples, dtype = int)
    for line in range(N):        
        for samp in range(num_samples):
            sampled_ind[samp] = randint(0,K-1)
            ar_to_sof[samp] = np.dot(X[line],np.transpose(W[sampled_ind[samp]])) 
        ar_to_sof = softmax(ar_to_sof)
        S[line][sampled_ind] = ar_to_sof 

    return S

S finally would contain zeros, and non_zero values in the indexes defined for every line by the array "samped_ind". I would like to implement this using Tensorflow. The problem is that it contains "advanced" indexing and i cannot find a way using this library to create that.

I am trying that using this code:

S = tf.Variable(tf.zeros((N,K)))
tfx = tf.placeholder(tf.float32,shape=(None,D))
wsampled = tf.placeholder(tf.float32, shape = (None,D))
ar_to_sof = tf.matmul(tfx,wsampled,transpose_b=True)
softy = tf.nn.softmax(ar_to_sof)
r = tf.random_uniform(shape=(), minval=0,maxval=K, dtype=tf.int32)
...
for line in range(N):
    sampled_ind = tf.constant(value=[sess.run(r),sess.run(r)],dtype= tf.int32)
    Wsampled = sess.run(tf.gather(W,sampled_ind))
    sess.run(softy,feed_dict={tfx:X[line:line+1], wsampled:Wsampled})

Everything works until here, but i cannot find a way to do the update that i want in the matrix S, in python code "S[line][sampled_ind] = ar_to_sof ".

How could i make this work?

Cfis Yoi
  • 241
  • 6
  • 15
  • Maybe [`tf.random_shuffle`](https://www.tensorflow.org/versions/r0.11/api_docs/python/constant_op.html#random_shuffle) would do it? – drpng Nov 12 '16 at 23:05
  • Maybe my question was not clear enough. Sorry for that. I don't want to shuffle my S Variable matrix. I want to create it! S finally would contain zeros, and non_zero values in the indexes defined for every line by the array "samped_ind". – Cfis Yoi Nov 12 '16 at 23:28
  • Oh, maybe something like `tf.one_hot`? – drpng Nov 12 '16 at 23:45
  • Sorry but i can't understand how one_hot could solve my question. Maybe i am thinking of it in a wrong way. More useful seems to me the tf.scatter_update, but the indexing of it, is confusing for me (how to pass the line and the sampled_ind to scatter_update??). However, thanks for trying to answer. – Cfis Yoi Nov 13 '16 at 00:29
  • Would it be accurate to say that you can boil down the problem to having a set of indices, possibly with values attached to them, and you want them in a dense tensor? Tensorflow supports a lot of sparse operations, and sometimes it's not necessary to unpack them. – drpng Nov 13 '16 at 00:43

1 Answers1

0

An answer to my problem was found in the comment of a solution of this problem. Suggests to reshape to 1d vector my matrix S. In that way, the code is working and it looks like:

S = tf.Variable(tf.zeros(shape=(N*K)))
W = tf.Variable(tf.random_uniform((K,D)))
tfx = tf.placeholder(tf.float32,shape=(None,D))
sampled_ind = tf.random_uniform(dtype=tf.int32, minval=0, maxval=K-1, shape=[num_samps])
ar_to_sof = tf.matmul(tfx,tf.gather(W,sampled_ind),transpose_b=True)
updates = tf.reshape(tf.nn.softmax(ar_to_sof),shape=(num_samps,))
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
for line in range(N):
    inds_new = sampled_ind + line*K
    sess.run(tf.scatter_update(S,inds_new,updates), feed_dict={tfx: X[line:line+1]})

S = tf.reshape(S,shape=(N,K))

That returns the result that i was expecting. The problem now is that this implementation is too slow. Much slower than the numpy version. Maybe is the for loop. Any suggestions?

Community
  • 1
  • 1
Cfis Yoi
  • 241
  • 6
  • 15