0

All this while I was using lightweight python function as a component in kubeflow pipeline but recently I switched to use docker container as a component in kubeflow pipeline which goes something like this...

    produce: str = 'hello'
):
    return dsl.ContainerOp(
        name = 'Say hi', 
        image = 'docker.io/playground/comp1:latest', 
        command = ['python', 'msg.py'],
        arguments=[
            '--msg', send_msg
        ],
        file_outputs={
            'output': '/output.txt',
        }
    )

Do i need to to some changes in the above code? do i need to autoscale docker container? if so how will the dsl.ContainerOp know which instance of docker container it should hit?

I did a lot of searching and also explored kfp package to see if that helps but to my luck i could not even find a closer solution to it. First of all is it possible? Am i in the right direction? Can someone please help me in my R&D?

Bernard Vander Beken
  • 4,848
  • 5
  • 54
  • 76
Sagar SN
  • 35
  • 1
  • 8

2 Answers2

3

I was using lightweight python function as a component in kubeflow pipeline but recently I switched to use docker container as a component

  1. Lightweight components are proper container components. You can export them to component.yaml files and share them: create_component_from_func(my_func, output_component_file='component.yaml') or (same) func_to_container_op(my_func, output_component_file='component.yaml')

  2. ContainerOps are not components. They're half-compiled task-like objects. We advice our users not to create them directly, but create real reusable components instead.

component.yaml

name: 'Say hi'
inputs:
- {name: Message}
outputs:
- {name: Output}
implementation:
  container:
    image: 'docker.io/playground/comp1:latest'
    command: ['python', 'msg.py']
    args: [
      '--msg', {inputValue: Message},
      '--output', {outputPath: Output},
    ]

Then load it using load_component_from_file or load_component_from_url or load_component_from_text and use it in multiple pipelines. Reusable components automatically produce ContainerOps when you instantiate them.

  1. Execution-wise there is no difference between lightweight components, components loaded from component.yaml and manually created ContainerOp instances.

Do i need to to some changes in the above code? do i need to autoscale docker container?

Probably not. Kubernetes' autoscaling is an opt-in feature that starts more nodes when they load is high. SO it only kicks in when you run many heavy tasks in parallel. If you just run a single task, there is nothing to scale.

Ark-kun
  • 6,358
  • 2
  • 34
  • 70
0

I think there is no issue with the code and you are on the right track. As per the GCP documentation click here, you can take advantage of GKE autoscaling to scale up your workload horizontally. That being said you can define a pod in a deployment manifest. Please note that, the pod will contain the kubeflow component like a container. For detailed information, please follow the link here here. I hope this helps you to solve the issue.

Mahboob
  • 1,877
  • 5
  • 19