Load data to placeholder for once and then run multiple computations

Question

Question

How to load data into placeholder for once, and then run multiple computations on data in the placeholder?

Use case

• I have 100 numpy arrays (A₁, ..., A₁₀₀) with the same shape.
• The objective function depends on both the input data and an array of variables B.
• For example, the loss function for A₁ can be `loss_1 = np.sum(A_1) + np.sum(B_1).
• For each A_n, I want to find the array of variables B_n that minimize the corresponding loss function loss_n.
• The result should be 100 arrays of variables, which are: B₁,...,B₁₀₀

I want to load A₁, find B₁, and then repeat for the rest of the A arrays.

Attempt 1

Loading the A arrays with tf.constant would lead to out of memory. After I load A₁ and find B₁. When I load A₂, A₁ will still stay in the memory of the GPU. After a while, the program will use up all memory of the GPU.

Attempt 2

Use placeholder, and load the same data in every step of the minimization. That will be slow because transferring data to the GPU is slow.

import tensorflow as tf

x = tf.placeholder("float", None)
y = x * 2

with tf.Session() as session:
    for j in range(100):  # Iterate over the 100 A arrays
        A_n = [1, 2, 3]  # The 100 different A arrays.
        for i in range(300):  # Steps of one minimization
            # Why do I have to feed the same data for 300 times??
            result = session.run(y, feed_dict={x: A_n})
            print(result)

Answer 1

This can be achieved by converting x to a variable. Variables in TF 1.X are initialized by explicitly running their initializer via Session.run. Therefore, all you need is to initialize the variable x with a placeholder:

x_init = tf.placeholder(tf.float32, shape=(3, ))
x = tf.Variable(x_init)
y = x * 2

with tf.Session() as sess:
    for j in range(100):
        A_n = [j, j + 1, j + 2]
        # Reinitialize `x` with the new A_n.
        sess.run(x.initializer, feed_dict={x_init: A_n})
        # `x` is initialized and therefore there is nothing to feed.
        for i in range(300):
            result = sess.run(y)
            print(result)

Note that this assumes that the shapes of A_n are the same.