2

I just started following the dagster tutorial. I managed to get the hello_cereal job running with dagit and the Python API, but for some reason when trying with dagster CLI

dagster job execute -f hello_cereal.py

I am getting a DagsterUnmetExecutorRequirementsError:

2021-10-22 15:33:22 - dagster - ERROR - hello_cereal_job - 272b37fb-9f39-44dc-b63a-dcd3dfbb7956 - 880 - RUN_FAILURE - Execution of run for "hello_cereal_job" failed. Pipeline failure during initialization for pipeline "hello_cereal_job". This may be due to a failure in initializing the executor or one of the loggers.

dagster.core.errors.DagsterUnmetExecutorRequirementsError: You have attempted to use an executor that uses multiple processes with an ephemeral DagsterInstance. A non-ephemeral instance is needed to coordinate execution between multiple processes. You can configure your default instance via $DAGSTER_HOME or ensure a valid one is passed when invoking the python APIs. You can learn more about setting up a persistent DagsterInstance from the DagsterInstance docs here: https://docs.dagster.io/deployment/dagster-instance#default-local-behavior

Indeed, I don't have $DAGSTER_HOME set, but since it is working with the web UI and the Python API versions, I was wondering if I made a mistake elsewhere ?

I am on macOS BigSur (11.6), on a fresh miniconda installation (v4.10.3), Python 3.9.5 and dagster 0.13.0.

alxthm
  • 57
  • 1
  • 6

2 Answers2

1

I had the same issue. I set DAGSTER_HOME and it fixed the problem. From the Dagster docs (https://docs.dagster.io/deployment/dagster-instance)

If DAGSTER_HOME is not set, the Dagster tools will use an ephemeral instance for execution. In this case, the run and event log storages will be in-memory rather than persisted to disk, and filesystem storage will use a temporary directory that is cleaned up when the process exits. This is useful for tests and is the default when invoking Python APIs such as JobDefinition.execute_in_process directly.

davemasino
  • 96
  • 3
1

thanks for the report - this issue was fixed in the 0.13.1 release that went out today on 10/25. That command should work without needing to worry about setting DAGSTER_HOME.

Daniel Gibson
  • 71
  • 1