2

Folks, I am writing an application which will produce recommendations based on ML model call. The application will have different models, some of them should be called in sequence. A data scientist should be able, to upload a model in the system. This means that the application should have logic to store models metadata as well as address of a model server. A model server will be instantiated dynamically on a model upload event. I would like to use a cluster of TensorFlow Serving here, however I am stacked with a question of architecture. Is there a way to have something like service registry for TensorFlow servers? What is the best way to build such a cluster of servers with different models?

Andras Gyacsok
  • 61
  • 7

1 Answers1

1

I need some clarification on what you're trying to do. Is the feature vector for all the models the same? If not then it will be quite a bit harder to do this. Trained models are encapsulated in the SavedModel format. It sounds like you're trying to train an ensemble, but some of the models are frozen? You could certainly write a custom component to make an inference request as part of the input to Trainer, if that's what you need.

UPDATE 1 From your comment below it sounds like what you might be looking for is a service mesh, such as Istio for example. That would help manage the connections between services running inside containers, and the connections between users and services. In this case tf.Serving instances running your models are the services, but the basic request-response pattern is the same. Does that help?

RCrowe
  • 363
  • 2
  • 9
  • Thank you. I would like to concentrate only on model deployment process. Let's assume that output of data-science pipeline is always new model, served by TF server in separate Docker container. The container will be deployed in our cluster aside with other different model containers. Next, this model should be known by the system, which acts as a middleman between user and the model, considering predefined route for the model container does not exists. Is there a sort of good practices to build such applications, e.g service registry, usage of gossip protocol? – Andras Gyacsok Nov 21 '19 at 20:24
  • Thank you, Robert. Yes, I think that's the right way to go. – Andras Gyacsok Nov 26 '19 at 14:47