When running a kfp pipeline with custom components (python function wrappers) that use a base image and packages_to_install on top of that, the component may silently fail without any descriptive error.
from kfp.v2.dsl import component
@component(
base_image=f"{MY_BASE_IMAGE}",
packages_to_install=MY_ADDITIONAL_PACKAGES_LIST,
)
def python_function():
.
.
.
The replica workerpool0-0 exited with a non-zero status of 1. Termination reason: Error.
As it fails to produce any logs from the actual function run, my guess is that it fails during packages_to_install phase due to some broken dependencies between the base_image and the packages I try to install on top of it.
To localize the exact problem, I would like to check the logs of this additional package installation (which I imagine is something like a Docker RUN that does pip install), but I haven't found any logs from that step produced in Vertex.
Any ideas on how to get your hands on those logs? Thanks!
Digging through KFP component documentation, haven't found any information about the logs.
Solution to the silent component failure I think would be to limit the usage of this additional packages installation step and pack as much as we can into the base image, but still want to be able to see the logs for additional packages.
