1

I was wondering how is ExampleGen used in production? I understand that their outputs can be feeded into the TFDV components of TFX to validate schema, skews, and others.

But I get lost since ExampleGen generates a train & eval split, and I don’t find why you would split the data in production into train & eval.

As far as I know, TFX is more suitable for deploying models into production, if I'm going to make a non-productive model maybe just using Tensorflow could work.

So ym questions are:

Is TFX are used for the modeling/dev part? i.e. before deploying your model.

Is it suitable to develop a model in Tensorflow and then migrate it to TFX for the production part?

Thanks!

marz
  • 831
  • 1
  • 7
  • 12
  • 1
    I have the same questions as yours. As for the train & eval split, we may just generate the 'eval' split through setting input_config (`proto.Input`) to only have the 'eval' split (e.g., `example_gen_pb2.Input.Split(name='eval', pattern='eval/*')` ) – Maosi Chen Aug 26 '21 at 17:44
  • 1
    @MaosiChen I'm starting to think that it is because TFX also allows you to make the modeling part also. So it is not only for deploying and serving but you can use TFX on your dev/modeling time, and after that continue to use TFX for the production/serving time. But it is just my 2 cents. – marz Aug 27 '21 at 15:01

1 Answers1

0

The ExampleGen TFX Pipeline component ingests data into TFX pipelines.

In simple words, ExampleGen fetches the data from external data sources such as CSV, TFRecord, Avro, Parquet and BigQuery and generates tf.Example and tf.SequenceExample records which can be read by other TFX components. For more info, Please refer The ExampleGen TFX Pipeline Component

Is TFX are used for the modeling/dev part? i.e. before deploying your model.

Yes TFX can be used for modeling, training, serving inference, and managing deployments to online, native mobile, and JavaScript targets. Once you model is trained on TFX, you can deploy your model using TF serving and other deployment targets.

Is it suitable to develop a model in Tensorflow and then migrate it to TFX for the production part?

Once you have developed and trained a model using TFX pipeline, you can deploy it using TF serving system. You can also serve tensorflow models using TF serving. Please refer Serving a TensorFlow Model