Artifact storage and MLFLow on remote server

Question

I am trying to get MLFlow on another machine in a local network to run and I would like to ask for some help because I don't know what to do now.

I have a mlflow server running on a server. The mlflow server is running under my user on the server and has been started like this:

mlflow server --host 0.0.0.0 --port 9999 --default-artifact-root sftp://<MYUSERNAME>@<SERVER>:<PATH/TO/DIRECTORY/WHICH/EXISTS>

My program which should log all the data to the mlflow server looks like this:

from mlflow import log_metric, log_param, log_artifact, set_tracking_uri

if __name__ == "__main__":
    remote_server_uri = '<SERVER>' # this value has been replaced
    set_tracking_uri(remote_server_uri)
    # Log a parameter (key-value pair)
    log_param("param1", 5)

    # Log a metric; metrics can be updated throughout the run
    log_metric("foo", 1)
    log_metric("foo", 2)
    log_metric("foo", 3)

    # Log an artifact (output file)
    with open("output.txt", "w") as f:
        f.write("Hello world!")
    log_artifact("output.txt")

The parameters get and metrics get transfered to the server but not the artifacts. Why is that so?

Note on the SFTP part: I can log in via SFTP and the pysftp package is installed

2 questions I had: 1 are both servers linux servers? 2 If they are linux have tou tried passwordless sftp server? — pushd93, Oct 08 '20 at 14:58

score 4 · Answer 1 · answered Feb 28 '20 at 15:27

4

I guess your problem is that you need to create also the experiment so using the sftp remote storage

mlflow.create_experiment("my_experiment", artifact_location=sftp_uri)

This fixed it for me.

answered Feb 28 '20 at 15:27

sklingel

169
2
9

score 2 · Accepted Answer · answered Dec 10 '19 at 08:50

2

I don't know if I will get an answer to my problem but I did solved it this way.

On the server I created the directory /var/mlruns. I pass this directory to mlflow via --backend-store-uri file:///var/mlruns

Then I mount this directory via e.g. sshfs on my local machine under the same path.

I don't like this solution but it solved the problem good enough for now.

answered Dec 10 '19 at 08:50

Spark Monkay

422
4
18

Make sure to add the sshfs to your fstab. If you have a heterogeneous environment, you could give nfs (https://linuxconfig.org/how-to-configure-nfs-on-linux) a try as well. – tafaust Oct 28 '21 at 14:46

Artifact storage and MLFLow on remote server

2 Answers2