tensorflow c++ batch inference

Question

I have a problem with making inference on a batchsize greater than 1 using the c++ tensorflow api. The network input planes are 8x8x13 and the output is a single float. When I try to infer on multiple samples as follows, the result is correct only for the first sample. I used keras2tensorflow tool for converting the graph to .pb format.

node {
  name: "main_input"
  op: "Placeholder"
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "shape"
    value {
      shape {
        dim {
          size: -1
        }
        dim {
          size: 8
        }
        dim {
          size: 8
        }
        dim {
          size: 12
        }
      }
    }
  }
}

Edit: Output node is a scalar. Could the culprit be the keras2tensorflow code that I used to convert keras hdf5 file to pb ? Maybe the output should have been a -1x1 to accept any number of samples just like the input planes). I got the converter code from the following link: keras_to_tensorflow

node {
  name: "value_0"
  op: "Identity"
  input: "strided_slice"
  attr { 
    key: "T"
    value {
      type: DT_FLOAT
    }
  }
}

The input plane dimension is correctly set to -1 x 8 x 8 x 13.

void test() {

    //input planes
    const int nmoves = pstack->count; //This is the number of samples
    TensorShape input_shape({nmoves, 8, 8, CHANNELS});
    Tensor inputs(DT_FLOAT, input_shape);

    //.... Initialize input planes

    //output
    std::vector<Tensor> outputs;

    //run session
    TF_CHECK_OK( session->Run(
        {{input_layer, inputs}}, {output_layer}, {}, &outputs) 
    );

    //get results
    auto outd = outputs[0].flat<float>().data(); //is this correct way to access the data for multiple samples ?
    for(int i = 0;i < nmoves; i++) {
        float p = outd[i];    //The value of p is wrong for all but the first one
        std::cout << "I" << i << " == " << p << std::endl;
    }
}

Example output (p) for each sample where the result is supposed to be between 0 and 1 is shown below. Only I0 is correct while I16 and I18 have very large values. I think the problem is that after running the session the dimension of outputs is still 1, should have been 20. Is it possible at all to do inference on multiple samples using the c++ api ?

I0 == 0.434162
I1 == 0
I2 == 0
I3 == 0.0640963
I4 == 0.0718748
I5 == 0.325485
I6 == 0
I7 == 0
I8 == 0
I9 == 0
I10 == 0.141193
I11 == 0.398055
I12 == 0.237758
I13 == 0.530693
I14 == 2.44527e-42
I15 == 0
I16 == -5.62959e+14
I17 == 4.56697e-41
I18 == -5.62959e+14
I19 == 4.56697e-41

could you give the whole source code, or reference two the code,i suspect that it has to do with data().shape that has extra dimensions, can you paste the pb for output node too — Eliethesaiyan, May 17 '18 at 07:40
I have updated my post with the output node and the code I used to convert keras hd5 file to pb. Thanks. — danny, May 17 '18 at 14:34
since its output is float ,you should be looping through output instead of data then, float p = utputs[i].flat().data(); let me know how it goes — Eliethesaiyan, May 18 '18 at 02:01
depending on yout model definition, output might have some additional values such as indexes which are not related to the output you and i suspect hat the zeros might be the indexes — Eliethesaiyan, May 18 '18 at 02:07
Unfortunately that doesn't work. It seems to recognize only output[0] and fails on output[1] with the message "F tensorflow/core/framework/tensor.cc:617] Check failed: dtype() == expected_dtype (0 vs. 1)" — danny, May 18 '18 at 15:47
Was this resolved? We're seeing something similar, where the C++ API only returns one result for a batch inference request. The values you're seeing might be data outside of the tensor (which is why they're so large). — Dan, Jun 28 '19 at 17:14
My problem was due to a bug in keras_to_tensorflow. Look at the answer I provided for details. — danny, Jun 29 '19 at 15:45

score 0 · Accepted Answer · answered Jun 29 '19 at 15:44

The problem turned out to be due to a bug the keras_to_tensorflow I used for conversion. I reported the issue here. The bug is still there in keras_to_tensorflow

On line 68:

pred[i] = tf.identity(net_model.output[i], name=pred_node_names[i])

The "output" should have been "outputs"

pred[i] = tf.identity(net_model.outputs[i], name=pred_node_names[i])

tensorflow c++ batch inference

1 Answers1