4

The following code is a batch data provider for .mat files, but has the following problem when running it:

    TypeError: expected str, bytes or os.PathLike object, not FIFOQueue

The code is:

    import numpy as np
    import tensorflow as tf
    from tensorflow.python.framework import ops
    from tensorflow.python.framework import dtypes
    import h5py

    def Reader(filename):
        with h5py.File(filename, 'r') as f:
            image = np.transpose(np.array(f.get('patch_x'), dtype=np.float32))
            label = np.transpose(np.array(f.get('patch_y'), dtype=np.float32))
        image = ops.convert_to_tensor(image, dtype=dtypes.float32)
        label = ops.convert_to_tensor(label, dtype=dtypes.float32)

        return image, label

    def Inputs(filenames, batch_size, shuffle=True):
        filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
        filename_queue = tf.train.string_input_producer(filenames, shuffle=shuffle)
        image, label = Reader(filename_queue)
        image = tf.cast(image, tf.float32)
        label = tf.cast(label, tf.float32)

        num_preprocess_threads = 4
        if shuffle:
            image_batch, label_batch = tf.train.shuffle_batch([image, label], batch_size=batch_size, num_threads=num_preprocess_threads, capacity=5*batch_size, min_after_dequeue=2*batch_size)
        else:
            image_batch, label_batch = tf.train.batch([image, label], batch_size=batch_size, num_threads=num_preprocess_threads, capacity=5*batch_size, min_after_dequeue=2*batch_size)
        return image_batch, label_batch

Does anyone know how to convert a string tensor to a python string easily? Thanks.

UPDATED 1: when using the filename.dequeue(), the error information is:

    TypeError: expected str, bytes or os.PathLike object, not Tensor
mining
  • 3,557
  • 5
  • 39
  • 66

2 Answers2

3

To convert a string TensorFlow tensor to Python string, run bytes.decode(string_tensor.numpy()).

Filip Bártek
  • 646
  • 7
  • 15
0

tf.train.string_input_producer() return a queue, not a string. You have to use Graph operation, which get string tensor from this queue and read file from disk. For example you can use chain of operation:

image = tf.image.decode_jpeg(tf.read_file(filename_queue.dequeue()))

if you have jpeg files.

In TensorFlow 1.2 there is new structure Dataset for creating input pipeline. I think it is good idea to use Dataset instead queues.

Vladimir Bystricky
  • 1,320
  • 1
  • 11
  • 13
  • Thanks for your answer. I think the data loader in pytorch is more simple. [http://pytorch.org/tutorials/beginner/data_loading_tutorial.html] – mining Jul 21 '17 at 21:43
  • Using Dataset or Queue is not necessary. You can use `tf.placeholder` for graph input data, read data from the files by existing python code, and feed this data to the graph. – Vladimir Bystricky Jul 21 '17 at 21:49
  • Many talent programmers provide a lot of interfaces based on the tensorflow, those are confusing to me. In fact, we just need some simplest interfaces like those classes in `caffe`, give an input, then output a new tensor. I don't know why they developed so many children using many many different `tf` packages. – mining Jul 21 '17 at 21:49
  • There are too many `tfxxxx`. Why don't they use the original classes and functions in official `tensorflow` package? – mining Jul 21 '17 at 21:53
  • I can't ask to this question, I am not developer of TF:) But I think there are not a problem using only "core" part of TF and don't use additional modules. – Vladimir Bystricky Jul 21 '17 at 21:56
  • Yeah, thank you sir! I'm grateful for your comments! I just have a little dislikes on those more and more different but may be redundant (just in my opinion) packages. Maybe I'm out-dated in my mind for the new technology, I still like the simple and light-weight `caffe` way, there is no complex heritages in the implementation and we could easily review the source code and have some modifications. It is more immediate. – mining Jul 21 '17 at 22:04
  • I agree with you in part of your message, because I like light-weight code too. (but may be because i'm out-dated too :) ) But I don't sure, about light-weighting of caffe. I worked with it, and I think TF more simple to use, but it demands to spend time, and understand the general idea of it: you build graph from blocks and then launch it. I found this idea very elegant. – Vladimir Bystricky Jul 21 '17 at 22:34
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/149853/discussion-between-vladimir-bystricky-and-mining). – Vladimir Bystricky Jul 21 '17 at 23:42