0

I am trying to deploy MLflow on Cloud Run. The Cloud Run Service needs to be connected to CloudSQL. I am trying to establish the connection by running CloudSQL proxy in the same docker container and connecting ML flow with the same. In the logs I am able to see that the CloudSQL proxy is able to connect to the Cloud SQL instance but logs for Mlflow server are not there, nor I am able to access MLflow via UI.

On deploying it on Cloud Run, I am getting this error:

Here is my Dockerfile:

FROM python:3.8-slim-buster

WORKDIR /app

RUN apt-get update && apt-get install -y bash && apt-get install -y wget gnupg2 && \
    wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy && \
    chmod +x cloud_sql_proxy && \
    pip install mlflow pymysql

FROM gcr.io/cloudsql-docker/gce-proxy:1.22.0

# Copy the service account JSON file to the container
COPY ./<path-to-credential-file> .

# Expose the default Cloud SQL port
EXPOSE 5000

ENV INSTANCE_CONNECTION_NAME=<instance-connection-name>
ENV MYSQL_USER=<username>
ENV MYSQL_PASSWORD=<password>
ENV MYSQL_DATABASE=<database_name>

# Start the Cloud SQL proxy
CMD ["/cloud_sql_proxy", "-instances=$INSTANCE_CONNECTION_NAME=tcp:0.0.0.0:3306", "-credential_file=<credential-file>", "-dir=/cloudsql & mlflow server \
    --default-artifact-root gs://<gcs-bucket-uri> \
    --host 0.0.0.0 --port 5000 \
    --backend-store-uri mysql+pymysql://$MYSQL_USER:$MYSQL_PASSWORD@localhost:3306/$MYSQL_DATABASE?unix_socket=/cloudsql/$INSTANCE_CONNECTION_NAME"]

The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information.```
Shobit Jain
  • 308
  • 2
  • 14
  • 1
    Any reason why you're not just using the built-in Cloud SQL integration? Cloud Run will run the Proxy for you in that approach. – enocom Mar 27 '23 at 18:42
  • @enocom, acc. to your approach my steps should be - - Deploy a Cloud Run Service - Then, add CloudSQL Instance in the configuration IMO, these steps work when we write code for the application from scratch. In my case, I am uploading a third party Docker Image (Mlflow) and starting a command to start the server which I think is not able to set up connection. For unix sockets, one has to set up the code which I am not able For public IP address, I am not able to get the URL/IP address of the Cloud Run service as it won't deploy unless it is able to set up connection – Shobit Jain Mar 28 '23 at 07:55
  • 1
    Does mlflow accept a Unix socket? If so, you could configure the Cloud SQL integration and just point mlflow at the Unix socket (/cloudsql/myproj:region:myinst). – enocom Mar 28 '23 at 15:26
  • I am not sure about this and need your suggestion. In the MLflow docs, the format for backend store URI is as follows: +://:@:/ Will it support only IP address and port of the CloudSQL instance? – Shobit Jain Mar 29 '23 at 07:20

2 Answers2

0

This error is usually caused by the container unable to listen to the incoming HTTP requests on the port defined by Cloud Run and provided in the $PORT environment variable. By default, requests are sent to port 8080 but you can configure Cloud Run to send requests to the port of your choice.

You can check this documentation on troubleshooting container failing to start especially number 2:

Check if your container is listening for requests on the expected port as documented in the container runtime contract. Your container must listen for incoming requests on the port that is defined by Cloud Run and provided in the PORT environment variable. See Configuring containers for instructions on how to specify the port.

You may also refer to these community questions as your references:

Robert G
  • 1,583
  • 3
  • 13
0

I am able to deploy via this github repo: https://github.com/dlabsai/mlflow-for-gcp

Shobit Jain
  • 308
  • 2
  • 14