1

In the TFX pipelines, how do we make use of BulkInferrer?

It is quite standard to connect the BulkInferrer with a trained model or pushed_model. However, what if I don't want to train the model again, instead I would love to use a previously trained model or pushed_model and use the BulkInferrer for batch inference(kinda like serving with BulkInferrer). Is it possible to do that?

If not, what is the purpose of BulkInferrer component, just to do a one-time prediction or validation after the whole training?

Any suggestions or comments will be appreciated, thanks.

xmindata
  • 97
  • 1
  • 1
  • 10

1 Answers1

1

BulkInferrer is the newest component added in the TFX library to support the batch inference of unlabelled data.

enter image description here Robert Crowe nicely positioned BulkInferrer in the inference pipeline, during the recent BEAM summit

Here is a list of use cases of why would someone use the BulkInferrer, trying to approach it in the case of ml-pipelines rather than data-pipelines:

  1. Periodically in an batch job, such as a nightly run to infer scores for the data received last day. For example when the inference latency is high and needs to run offline to populate a cache with scores, or when large amount of instanced need to be inferred.
  2. When the inference results are not expected to be served directly back to the inference request (ie real-time) but can be executed asynchronously. For example, to shadow test a new model prior to its release.
  3. In an event-driven fashion. For example, trigger a model retraining when inferring batches of data to and model drift is detected.
  4. For cost optimisation. For example on low throughput models/services there might be long idle times of cpu/gpu instances.

To do this in your ML pipeline without retraining your model, you can include BulkInferrer indeed at the end of the pipeline and reuse the results from previous runs if the inputs and configuration has not changed. This is achieved by both Argo and Tekton workflow managers on Kubeflow pipelines, as they implement TFX, see step caching.

Theofilos Papapanagiotou
  • 5,133
  • 1
  • 18
  • 24
  • Thanks a million, this really helps me to understand this component. I think all what I need is to enable the `cache` in the pipeline. Just a follow up question, now that I have the artifact of a `pushed_model` from pipeline `p-1`, is it possible to create a new pipeline `p-2` and load this `pushed_model` from `p-1` to run `BulkInferrer` in `p-2`? – xmindata Nov 02 '21 at 07:16
  • yes of course, you can refer to the artifact (model) produced by p-1 using the `trainer.outputs['model']` channel – Theofilos Papapanagiotou Nov 02 '21 at 13:44
  • Thanks again for the help. I am indeed trying to use the `trainer.outputs['model']` as the model input for `Bulkinferrer`, however, they are from different runs (say `run-1` and `run-2`), so I have to pick up the `trainer.outputs['model']` from `p-1` (of `run-1`) and load as a `Channel` type and feed it to `Bulkinferrer` in `p-2` (of `run-2`). So far I didn't have any luck to load the `trainer.outputs['model']` as a `Channel` type, would you mind give me some elaborate examples how can i achieve that? – xmindata Nov 02 '21 at 13:58
  • how about picking it up from your metadata store? If you're using MLMD you can query the for the model of the previous run, based for example on its name. – Theofilos Papapanagiotou Nov 02 '21 at 14:04
  • The `pushed_model` fetched from mlmd is of type `ml_metadata.proto.metadata_store_pb2.Artifact`, if i feed this into the `Bulkinferrer` i got `Argument model should be a Channel of type ` :( – xmindata Nov 02 '21 at 14:11
  • the `uri` from the `Artifact()` should contain the path to the model file/dir, which you can then add to the tfx channel – Theofilos Papapanagiotou Nov 02 '21 at 14:36
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/238805/discussion-between-theofilos-papapanagiotou-and-xmindata). – Theofilos Papapanagiotou Nov 02 '21 at 14:44
  • 1
    Thank you for the help, it took me a whiel to figure out how the Resolver works. Bedankt! @TheofilosPapapanagiotou – xmindata Nov 13 '21 at 08:07