load tfrecord files into keras model by batch

Question

I want to load tfrecord extracted batch data in keras model

I used the tfrecord to store my data and its label, and I extracted it out using tf.dataset API by batch and created an iterator, but I dont know how to proceed to load data into keras model.

import tensorflow as tf
import keras as k
import numpy as np
num_epochs = 2
data_dim = 75
timesteps = 300
num_classes = 82
batch_size = 128
training_filename = [filepath]
validation_filename = [filepath]


def parse_function(example1):
    features = tf.io.parse_single_example(example1, features={
        'label': tf.io.FixedLenFeature(shape=(), dtype=tf.int64, default_value=None),
        'skeleton': tf.io.FixedLenFeature(shape=(), dtype=tf.string),
        'skeleton_shape': tf.io.FixedLenFeature(shape=(3,), dtype=tf.int64)})
    features['skeleton'] = tf.decode_raw(features['skeleton'], tf.float64)
    skeleton = tf.reshape(features['skeleton'], (300, 75))
    label = tf.one_hot(features['label'], num_classes, dtype=tf.float64)
    return skeleton, label


def load_dataset(filename):
    data_reading = tf.data.TFRecordDataset(filename)
    dataset = data_reading.map(parse_function)
    dataset = dataset.batch(batch_size)
    iterator = dataset.make_one_shot_iterator()
    skeleton, label = iterator.get_next()
    return skeleton, label


x_train, y_train = load_dataset(training_filename)
x_val, y_val = load_dataset(validation_filename)

model = k.Sequential()
model.add(k.layers.LSTM(128, activation='relu', return_sequences=True))
model.add(k.layers.LSTM(128, activation='relu'))

model.add(k.layers.Dense(num_classes, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(x_train, y_train,
          epochs=5, shuffle=True,
          validation_data=(x_val, y_val))

Traceback (most recent call last):
  File "D:/MScProject/extraction/feeder.py", line 60, in <module>
    y=run_model(x_train,y_train,x_val,y_val)
  File "D:\MScProject\extraction\lstm.py", line 22, in run_model
    validation_data=(x_val, y_val))
  File "C:\Users\wille\Anaconda3\lib\site-packages\keras\engine\training.py", line 952, in fit
    batch_size=batch_size)
  File "C:\Users\wille\Anaconda3\lib\site-packages\keras\engine\training.py", line 677, in _standardize_user_data
    self._set_inputs(x)
  File "C:\Users\wille\Anaconda3\lib\site-packages\keras\engine\training.py", line 589, in _set_inputs
    self.build(input_shape=(None,) + inputs.shape[1:])
TypeError: can only concatenate tuple (not "TensorShape") to tuple

score 0 · Answer 1 · answered Aug 05 '19 at 18:38

0

Which version of TF are you using?

If you are with TF 2.0 you can feed directly tf.data.Dataset to model.fit.

Check this entry at TF documentation

answered Aug 05 '19 at 18:38

Guillem

2,376
2
18
35

tf 1.14, is there any way to do it in the current version? – Julian Ji Aug 05 '19 at 18:45
Maybe this [notebook](https://www.kaggle.com/kmader/tf-data-tutorial-with-retina-and-keras) or this [post](https://stackoverflow.com/questions/46135499/how-to-properly-combine-tensorflows-dataset-api-and-keras) can help you – Guillem Aug 05 '19 at 18:48

load tfrecord files into keras model by batch

1 Answers1