Stack Overflow
Stack Overflow

Questions tagged [tfx]

TensorFlow Extended (TFX) is an end-to-end platform for deploying production ML pipelines

TFX is a Google-production-scale machine learning platform based on TensorFlow. It provides a configuration framework and shared libraries to integrate common components needed to define, launch, and monitor your machine learning system.

Resources

185 questions
1
vote
0 answers

How can I use BigQuery in a standalone tensorflow transform (TFT) pipeline?

I'm interested in interactive development of a preprocessing_fn for tft.AnalyzeAndTransformDataSet. By interactive development, I mean running a standalone beam pipeline in a Jupyter Notebook and later connecting to the resulting transformed data…
jb_ml_eng
  • 61
  • 4
1
vote
1 answer

How to predict with Bulkinferrer from a previously pushed model in Tensorflow Extended (TFX)

In the TFX pipelines, how do we make use of BulkInferrer? It is quite standard to connect the BulkInferrer with a trained model or pushed_model. However, what if I don't want to train the model again, instead I would love to use a previously trained…
xmindata
  • 97
  • 1
  • 1
  • 10
1
vote
0 answers

How do I find the length of a vocabulary computed during TFX Transform?

I'm currently building a project in TFX and during the Transform step I compute the "vocabulary" for a categorical variable. For later steps (but still during preprocessing), I want to use the length of that vocabulary (i.e. the number of distinct…
Sarah Messer
  • 3,592
  • 1
  • 26
  • 43
1
vote
1 answer

TFX Tensorflow model validator component - You passed a data dictionary with keys ['image_raw_xf']. Expected the following keys: ['input_1']

I'm building a tfx pipeline based on the cifar10 example : [https://github.com/tensorflow/tfx/tree/master/tfx/examples/cifar10] The difference is that I don't want to convert it to tf_lite model and instead use a regular keras based tensorflow…
Juan Acevedo
  • 1,768
  • 2
  • 20
  • 39
1
vote
1 answer

Kubeflow, passing Python dataframe across components?

I am writing a Kubeflow component which reads an input query and creates a dataframe, roughly as: from kfp.v2.dsl import component @component(...) def read_and_write(): # read the input query # transform to dataframe …
FrankNrg92
  • 71
  • 4
1
vote
1 answer

Disabling useless logs/ouputs from TFX/setuptools

I have been working with TensorFlow Extended (TFX) for about a month and a half now and there has always been one thing that really bothers me about it. The logging and mountain of stdout outputs from TFX pipeline (regardless of the…
gcarr7
  • 11
  • 1
1
vote
2 answers

Beam + Flink: No parallelism when using SDFBoundedSourceReader

Background: I am using TFX pipelines with Flink as the runner for Beam (flink session cluster using flink-on-k8s-operator). The Flink cluster has 2 taskmanagers with 16 cores each, and parallelism is set to 32. TFX components call…
rojmor
  • 31
  • 4
1
vote
1 answer

How do you fit a tf.Dataset to a Keras Autoencoder Model when the Dataset has been generated using TFX?

Problem As the title suggests I have been trying to create a pipeline for training an Autoencoder model using TFX. The problem I'm having is fitting the tf.Dataset returned by the DataAccessor.tf_dataset_factory object to the Autoencoder. Below I…
JChandler
  • 51
  • 6
1
vote
1 answer

ExampleGen on production

I was wondering how is ExampleGen used in production? I understand that their outputs can be feeded into the TFDV components of TFX to validate schema, skews, and others. But I get lost since ExampleGen generates a train & eval split, and I don’t…
marz
  • 831
  • 1
  • 7
  • 12
1
vote
1 answer

Tensorflow Extended (TFX): Is there an easy way to debug functions from Transorm component?

I am supposed to modify a function which is a part of Transorm component. It is a long series of tensorflow operations and I am not sure a. how particular steps affect processed variables b. what does my input look like. I can't just run them in…
Brzoskwinia
  • 371
  • 2
  • 11
1
vote
0 answers

tfx.orchestration.experimental has no attribute 'get_default_kubeflow_metadata_config'

I am trying to orchestrate a TFX pipeline using kubeflow and I am getting the following error during the creation of the pipeline. AttributeError: module 'tfx.orchestration.experimental' has no attribute 'get_default_kubeflow_metadata_config' Below…
AshwinSrinivas_1
  • 11
  • 1
1
vote
2 answers

Error output in Google Colab tutorial when installing tfx

I'm running the following Google Colab, but when I run the following command, I get the below error: !pip install -U tfx -- ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is…
Christopher Mills
  • 711
  • 10
  • 28
1
vote
1 answer

Dynamic while loop Tensorflow

I have this numpy function, I simplified it into something like this: def _func(new_inputs, X): """ Basically any operation. Below is just an example """ outputs = new_inputs + X new_inputs.pop(0) return outputs,…
eng2019
  • 953
  • 10
  • 26
1
vote
2 answers

Airflow: How can I create a virtual python environment in distributed airflow in runtime for a tfx-pipeline using the tfx's AirflowDagRunner?

I have DAGS which I want to run in different python environments in the same distributed airflow cluster. Is it possible to create a virtual python environment while executing the DAG? I am looking to add tfx ML pipelines to our distributed airflow…
Surya
  • 11
  • 3
1
vote
0 answers

TFX Pushed model working fine when used in BQML, but error when using with BulkInferrer

Followed https://stackoverflow.com/a/65979056/12559246 to get the o/p of BulkInferrer to a dataframe. Still error persists in BulkInferrer module. Code Snippet:- BQExampleGen BulkInferrer The model created by pusher in same pipeline is able to…
Abhilash
  • 11
  • 1
1 2 3
…
12 13