2

I have a piece of code to read some binary files which are written in a specific format. I want to write a tensorflow version of my read_file() function. Is there any way to do so? I don't want to wrap my function using tf.py_function().

def read_file(filename):
    f = open(filename, 'rb')
    w = int(np.fromfile(f, np.int32, count=1))
    h = int(np.fromfile(f, np.int32, count=1))
    data = np.fromfile(f, np.float32, count=2 * w * h)
    data = np.resize(data, (h, w, 2))
    f.close()
    return data
srilekha palepu - Intel
  • 2,017
  • 9
  • 22
Ali Salehi
  • 1,003
  • 8
  • 19

2 Answers2

1

You could potentially use tf.data.FixedLengthRecordDataset to read samples encoded with a fixed length of bytes from a binary file.

Scholar
  • 463
  • 5
  • 19
0

Unfortunately, every python code that you want to use beside the tf* functions must be wrapped inside the tf.py_function().

There is no other way to integrate custom Python code with the TensorFlow specialised functions.

You can find even more details about this at this question: Is there an alternative to tf.py_function() for custom Python code?

Timbus Calin
  • 13,809
  • 5
  • 41
  • 59
  • The key function here is `np.fromfile()`, I couldn't find any Tensorflow alternative for that, are you aware of any? With that, it is easy to convert the whole function. – Ali Salehi Jan 08 '20 at 17:42
  • What if use use tf.constant(np.fromfile()) to convert it to Tensor? – Timbus Calin Jan 08 '20 at 17:44
  • Curious about the result, never tried it, tell me :D – Timbus Calin Jan 08 '20 at 17:44
  • Thanks, but that doesn't work here. I'm using a wrapped version of `read_file()` to make my `tf.data.dataset`, and trying to speed up the process wanted to avoid using `tf.py_function()` by converting the entire code to tensorflow functions. I guess I need to write my own custom tf op. – Ali Salehi Jan 08 '20 at 17:55
  • Yes, I know. That's why I put that link to my question, because there was an answer stating there that you cannot do that. – Timbus Calin Jan 08 '20 at 17:56
  • @AliSalehi It's frustrating as it seems like such a simple feature. Did you end up implementing you own ops? Any chance you'd like to share? – Scholar Feb 01 '23 at 22:40