How to decode TFRecord data sample with variable length strings?

Question

Let's say we have a TFRecord file with data samples like this:

def _bytes_feature(value):
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))

def _float32_feature(value):
    return tf.train.Feature(float_list=tf.train.FloatList(value=value))

example = tf.train.Example(features=tf.train.Features(feature={
    'image/encoded': _bytes_feature(encoded_jpg),
    'label': _float_list_feature(label),
}))

Here encoded_jpg is the raw value of encoded 32x32 jpg images, the length of which can be quite different for different images; label is a fixed-length vector.

For fixed-length fields, one can always use something like the following to decode the sample:

features = tf.parse_single_example(
    serialized_example,
    features = {
        'image/encoded': tf.FixedLenFeature([], tf.string)
        'label': tf.FixedLenFeature([], tf.float32)
    }

)

But here the length of image/encoded is not constant, the aforementioned one doesn't work anymore.

If I change the code into this:

features = tf.parse_single_example(
    serialized_example,
    features = {
        'image/encoded': tf.VarLenFeature(tf.string)
        'label': tf.FixedLenFeature([], tf.float32)
    }
)

encoded = features['image/encoded']

image/encoded is something like sparse tensor, I don't know how to decode the image from this stuff.

Does anyone have similar experience before? Any suggestion is appreciated.

Thanks!

You still use the` tf.FixedLenFeature`. This is addressed in the thread :https://stackoverflow.com/questions/41921746/tensorflow-varlenfeature-vs-fixedlenfeature — Vijay Mariappan, Jul 11 '17 at 20:45

score 5 · Accepted Answer · answered Jul 12 '17 at 09:35

Following code may useful:

convert to tfrecord:

ex = tf.train.SequenceExample()
ex.context.feature["length"].int64_list.value.append(label)
ex_tokens = ex.feature_lists.feature_list["image/encoded"]
for value in range(encoded_jpg):
    ex_tokens.feature.add().int64_list.value.append(value)

with tf.python_io.TFRecordWriter(os.path.join(DATA_PATH, filename) + ".tfrecord") as filew:
    filew.write(ex.SerializeToString())

read tfrecord

context_features = {
    "pose": tf.FixedLenFeature([], dtype=tf.float32)
}

sequence_features = {
    "image/encoded": tf.FixedLenSequenceFeature([], dtype=tf.int64),
}

tf_reader = tf.TFRecordReader()
tf_key, tf_serialized = tf_reader.read(tf_file_queue)
tf_context, tf_sequence = tf.parse_single_sequence_example(
    serialized = tf_serialized,
    context_features = context_features,
    sequence_features = sequence_features
)
encoded = tf_sequence['image/encoded']

How do you re-shape the variably-shaped "image/encoded" features using the "length" in the context? — Oriol Nieto, Sep 07 '18 at 19:04
this code example is rather incomplete. I suggest to specify variables like "pose" and "label" — whiletrue, Nov 05 '18 at 06:38

How to decode TFRecord data sample with variable length strings?

1 Answers1