Stack Overflow
Stack Overflow

Questions tagged [dagster]

Dagster is an open source system for building modern data applications.

Dagster, by Elementl, is a set of abstractions for building self-describing, testable, and reliable data applications. It uses functional data programming, gradual/optional typing, and testability to facilitate composition of data applications from DAGs of solids, its basic computational unit.

142 questions
0
votes
1 answer

Dagster Graphql get all jobs in a Repo

I am looking to use the dagster Graphql as a documented here I want to get all jobs in a repo, I am using the "Get a list of jobs within a repository" query outlined in the above documentation And Get the following error { "error": { "data":…
DavidMul
  • 71
  • 4
0
votes
1 answer

Submit dagster run programmatically (without dagit graphQL)

Is there a way I can submit dagster run programmatically without using dagit graphQL? dagster_instance = DagsterInstance.get() dagster_instance.submit_run( pipeline_run.run_id, workspace=graphene_info.context ) Basically…
Varun Shridhar
  • 113
  • 1
  • 9
0
votes
1 answer

Dagster Daemon sends heartbeats but starts no runs

I created a simplified project with dagster and started the dagster-daemon, however it starts no runs even though it sends heartbeats to the default sqlite daemon_heartbeats table. I put all the files I needed in the same folder (at this point,…
CelioxF
  • 63
  • 5
0
votes
1 answer

(Dagster) Schedule my_hourly_schedule was started from a location that can no longer be found

I'm getting the following Warning message when trying to start the dagster-daemon: Schedule my_hourly_schedule was started from a location Scheduler that can no longer be found in the workspace, or has metadata that has changed since the schedule…
CelioxF
  • 63
  • 5
0
votes
1 answer

Access traceback in dagster's failure hook

I have a failure hook in which I would want to send a mail with the exception that has been raised. Is there any way to access it? # PIPELINE @failure_hook def email_message_on_failure(context: HookContext): logging.exception(e) mail_errors…
Javi Torre
  • 724
  • 8
  • 23
0
votes
1 answer

What is proper Partition configs for Dagster job?

Currently, I am facing with dagster.core.errors.PartitionExecutionError but error logs from Dagster seem not obvious to me. dagster.core.errors.PartitionExecutionError: Error occurred during the evaluation of the `run_config_for_partition` function…
Bryan
  • 1,477
  • 1
  • 21
  • 38
0
votes
1 answer

ChildProcessCrashException in Dagster multiprocess execution in multi-container Docker deployment

I have a Dagster job that is training a CNN (using Keras). The Op that runs fit() is causing the following error: Multiprocess executor: child process for step train unexpectedly exited with code…
Atticus
  • 147
  • 1
  • 9
0
votes
1 answer

How to run tasks in parallel in dagster?

I am using dagster to running into local node.js microservices pipelines, in order to execute test. The ide is execute n docker_files, and n node.js microservices, easily like you can do with dagster. The problem is that when I execute the first…
Tlaloc-ES
  • 4,825
  • 7
  • 38
  • 84
0
votes
1 answer

AWS credentials not found for celery-k8s deployment

I'm trying to run dagster using celery-k8s and using the examples/celery-k8s as a start. upon running the pipeline from playground I get Initialization of resources [s3, io_manager] failed. botocore.exceptions.NoCredentialsError: Unable to locate…
Somasundaram Sekar
  • 5,244
  • 6
  • 43
  • 85
0
votes
1 answer

Dagster pass ordered dict in the config schema

How can I pass an Ordered Dict in Dagster solid's config schema? The simple thing: from dagster import solid, execute_solid, Field @solid(config_schema={'my_dict': Field(dict, is_required=True)}) def test_ordered_dict(context): …
cyau
  • 449
  • 4
  • 14
0
votes
1 answer

Dagster use a solid's AssetMaterialization in another solid (reproducible code included)

So let's say I have two solids. The first does some computations and writes a file to disk. The second solid takes that file and does other things with it, but it needs its filesystem path in order to open it. I can do this with two yields (one for…
cyau
  • 449
  • 4
  • 14
0
votes
2 answers

Dagster Installation on MacOS Big Sur

I am trying to install Dagster and Dagit using:python3 -m pip install dagster dagit --user But, Always when it starts to build wheel for grpcio, it throws an error and stops the installation. ERROR: Command errored out with exit status 1:…
Uttasarga Singh
  • 27
  • 7
0
votes
1 answer

Dagster how to provide presets when creating partitions via PartitionSetDefinition

When creating a PartitionSetDefinition in Dagster you can pass in a 'mode' that will swap the resources used (for testing purposes you may want to use cloud storage in PROD but use local storage for local development A mode requires you to specify a…
j-hulbert
  • 1
  • 1
0
votes
1 answer

Is there a way to get the traceback stack from Dagster failure_hook and success_hook?

from dagster import HookContext, failure_hook, success_hook @success_hook(required_resource_keys={"slack"}) def slack_message_on_success(context: HookContext): message = f"Solid {context.solid.name} finished successfully" …
Learner
  • 1
  • 1
0
votes
2 answers

Daily_schedule triggered runs and backfill runs have different date partition

I have @daily_schedule triggered daily at 3 minutes past 12am When triggered by the scheduled tick at '2021-02-16 00:03:00' The date input shows '2021-02-15 00:00:00', partition tagged as '2021-02-15' While if triggered via backfill for partition…
Isaac
  • 11
  • 1
1 2 3
…
9
10