I want to read data from TFRecord

Question

I saved the image date into tfrecord, but I cannot parse it with tensorflow dataset api.

My environment

Ubuntu 18.04
Python 3.6.8
Jupyter Notebook
Tensorflow 1.12.0

I saved the image data by following code,

writer = tf.python_io.TFRecordWriter('training.tfrecord')

# X_train: paths to the image, y_train: labels (0 or 1)
for image_path, label in zip(X_train, y_train):
    image = cv2.imread(image_path)
    image = cv2.resize(image, (150, 150)) / 255.0
    ex = tf.train.Example(
        features = tf.train.Features(
            feature={
                'image' : tf.train.Feature(float_list = tf.train.FloatList(value=image.ravel())),
                'label' : tf.train.Feature(int64_list = tf.train.Int64List(value=[label]))
            }
        )
    )
    writer.write(ex.SerializeToString())
writer.close()

I tried getting a image from tfrecord file like that.

for record in tf.python_io.tf_record_iterator('test.tfrecord'):
    example = tf.train.Example()
    example.ParseFromString(record)

    img = example.features.feature['image'].float_list.value
    label = example.features.feature['label'].int64_list.value[0]

This method works.

But it doesn't when I use Dataset API to get images for my ML model.

def _parse_function(example_proto):
    features = {
        'label' : tf.FixedLenFeature((), tf.int64),
        'image' : tf.FixedLenFeature((), tf.float32)
    }
    parsed_features = tf.parse_single_example(example_proto, features)

    return parsed_features['image'], parsed_features['label']

def read_image(images, labels):
    label = tf.cast(labels, tf.int32)
    images = tf.cast(images, tf.float32)
    image = tf.reshape(images, [150, 150, 3])

# read the data
dataset = tf.data.TFRecordDataset('training.tfrecord')
dataset = dataset.map(_parse_function)
dataset = dataset.map(read_image) # <- ERROR!

The error massage is

ValueError: Cannot reshape a tensor with 1 elements to shape [150,150,3] (67500 elements) for 'Reshape' (op: 'Reshape') with input shapes: [], [3] and with input tensors computed as partial shapes: input[1] = [150,150,3].

I though the cause of this error is the shape of the array is wrong, so I confirmed the element of the "dataset"

<MapDataset shapes: ((), ()), types: (tf.float32, tf.int64)>

"dataset" variable has no data. I don't know why it heppens.

Postscript

I tried the solution from Sharky, as a result,

def parse(example_proto):
    features = {
        'label' : tf.FixedLenFeature((), tf.string, ''),
        'image' : tf.FixedLenFeature((), tf.string, '')
    }
    parsed_features = tf.parse_single_example(example_proto, features)
    img_shape = tf.stack([150, 150, 3])
    image = tf.decode_raw(parsed_features['image'], tf.float32)
    image = tf.reshape(image, img_shape)
    label = tf.decode_raw(parsed_features['label'], tf.int32)
    label = tf.reshape(label, tf.stack([1]))

    return image, label

works, I think. But I cannot get array from this MapDataset type object. How to do that?

Sharky · Accepted Answer · 2019-03-28T13:23:27.590

1

Try using a single parse function

def parse(example_proto):
    features = {
        'label' : tf.FixedLenFeature((), tf.int64),
        'image' : tf.FixedLenFeature((), tf.string)
    }
    parsed_features = tf.parse_single_example(example_proto, features)
    img_shape = tf.stack([height, width, channel])
    image = tf.decode_raw(parsed_features['image'], tf.float32)
    image = tf.reshape(image, img_shape)
    label = tf.cast(parsed['label'], tf.int32)
    return image, label

Ok, it seem's that parse_single_example expects string type instead of float. I'd advise to encode like this

def int64_feature(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))


def bytes_feature(value):
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))

writer = tf.python_io.TFRecordWriter('training.tfrecord')

for image_path, label in zip(X_train, y_train):
    image = cv2.imread(image_path)
    image = cv2.resize(image, (150, 150)) / 255.0
    img_raw = image.tostring()
    ex = tf.train.Example(features=tf.train.Features(feature={                                                                     
                        'image': bytes_feature(img_raw),
                        'label': int64_feature(label)
                         }))
    writer.write(ex.SerializeToString())
writer.close()

edited Mar 28 '19 at 13:23

answered Mar 27 '19 at 15:40

Sharky

4,473
2
19
27

It' strange. One thing I could think of is that you're using `tf.train.FloatList` instead of `BytesList` diring converting to tfrecords. Try substituting float in 'image' : tf.FixedLenFeature((), tf.float32) to `tf.string, ''` – Sharky Mar 27 '19 at 16:20
Thank you for your advice. It appears to work well. I add a code and a comment in the above. – Y. P Mar 27 '19 at 23:23
As I said, try encoding with bytes feature, this should work – Sharky Mar 28 '19 at 09:34
I tried encoding with bytes (img_raw=image.tostring() ?) and saved. It succeeded, but the parse doesn't work (in parse function [height, width, channel]=[150,150,3] ?). The error massage is following, – Y. P Mar 28 '19 at 12:14
ValueError: Tried to convert 'tensor' to a tensor and failed. Error: Argument must be a dense tensor: FixedLenFeature(shape=(), dtype=tf.float32, default_value=None) - got shape [3], but wanted [3, 0]. – Y. P Mar 28 '19 at 12:14
forgot np.tostring. Upadated code. And you can use `cv2.resize(image, (150, 150)) / 255.0` during decoding, i think it may be faster – Sharky Mar 28 '19 at 12:46
In parse function, what is the 'labels' in tf.cast method? When I tried 'labels=features['label]'', I got ValueError as I wrote above. – Y. P Mar 28 '19 at 13:20
Sorry, I forgot. Updated – Sharky Mar 28 '19 at 13:24
Thank you. It worked well. But still I don't know how to convert the MapDataset to the array. I'll make another question. – Y. P Mar 29 '19 at 12:02

I want to read data from TFRecord

My environment

Postscript

1 Answers1

Linked