Stack Overflow
Stack Overflow

Questions tagged [dagster]

Dagster is an open source system for building modern data applications.

Dagster, by Elementl, is a set of abstractions for building self-describing, testable, and reliable data applications. It uses functional data programming, gradual/optional typing, and testability to facilitate composition of data applications from DAGs of solids, its basic computational unit.

142 questions
0
votes
2 answers

stream data between tasks in pipeline orchestration tool Prefect/Dagster/Airflow

How can I stream data between tasks in a workflow with the help of a data pipeline orchestration tool like Prefect, Dagster or Airflow? I am looking for a good data pipeline orchestration tool. I think I have a fairly decent overview now of what…
phobic
  • 914
  • 10
  • 24
0
votes
0 answers

Dagster orchestrator - How can I run commands to process dbt models in another Docker container?

I'm playing a bit with Dagster orchestrator and I have some doubts. My intention is use this tool only as a scheduler and runner for commands since I have the business logic developed in other repositories and containers. So, the architecture would…
RgG
  • 23
  • 2
0
votes
1 answer

How to use dagster with great expectations?

The issue I'm trying out great expectations with dagster, as per this guide My pipeline seems to execute correctly until it reaches this block: expectation = dagster_ge.ge_validation_op_factory( name='ge_validation_op', …
Imad
  • 2,358
  • 5
  • 26
  • 55
0
votes
0 answers

Providing the same resource to every op in a job

So I need myresource in every op. How do I do this without having to specify it in every op, but in the job definition instead? I think this is possible, but I cannot find a specification in the documentation for exactly what the job config looks…
user1943569
  • 21
  • 2
0
votes
1 answer

How do I write a Dagster asset that depends on an earlier partition of itself?

I was using depends_on_past with Airflow. I'm now using Dagster, with software-defined assets, and I was told that the way to get similar functionality is with build_asset_reconciliation_sensor and a daily-partitioned asset where each partition…
Sandy Ryza
  • 265
  • 1
  • 8
0
votes
0 answers

How to allow memoization for jobs defined from Dagster assets

I would like to use Memoization for jobs that I am defining from AssetsDefinitions with define_asset_job. The problem is: define_asset_job does not allow me to set-up a version_strategy. How can I accomplish this, is there a different way to define…
pedrovgp
  • 767
  • 9
  • 23
0
votes
1 answer

How to force some HTML to render inside some other website?

I am working with an open source ETL tool (dagster) that allows me to attach some "metadata" to the results of each operation. Typically the metadata should be text or numbers, but I want to insert an HTML snippet. My issue is that the HTML is not…
MYK
  • 1,988
  • 7
  • 30
0
votes
1 answer

How to run a C# data transformation in dagster

I am currently evaluating dagster to build data engineering pipelines. I have to incorporate a huge body of existing c# code for a key piece of data transformation (from object storage to object storage) that I cannot simply replace in python. Do I…
Panke
  • 156
  • 9
0
votes
1 answer

Is it possible to transform one asset into another asset using ops in dagster?

From what I found here, it is possible to use ops and graphs to generate assets. However, I would like to use an asset as an input for an op. I am exploring it for a following use case: I fetch a list of country metadata from an external API and…
desa
  • 1,240
  • 12
  • 31
0
votes
1 answer

Dagster - running a cross repository job from a sensor

I've learned from the docs that a Dagster sensor could be triggered by a job from a different repo. In a similar manner, is there a way to run a cross repo job using the RunRequest inside the sensor, ie. something like this? @run_status_sensor( …
dreo
  • 910
  • 1
  • 9
  • 20
0
votes
1 answer

How to setup a success_hook to with send messages from telegram bot

I'm new one in Dagster. Could you help me, please? I want to understand how to set up an etl process error notification through a telegram bot My code: import pygsheets import os import telegram from dagster import resource from typing import…
Andrey
  • 75
  • 10
0
votes
0 answers

Docker incorrectly collects the image

After buying a mac on an m1 chip, I was faced with the problem of building images I am working on a test that will check the relevance of the data in the database. After writing this test in Pycharm, I collect the image in docker with docker buildx…
Lia
  • 1
0
votes
2 answers

How to materialize a downstream asset independently in dagster using python API?

I have dagster==1.0.11 and I am trying to materialize a downstream_asset independently from its upstream_asset. I want to do it through the python API. Please consider the code below for reference. # example.py from dagster import asset,…
pedrovgp
  • 767
  • 9
  • 23
0
votes
1 answer

Calling localhost:3000 on the browser after starting dagit returns ERR_CONNECTION_REFUSED

I have just started trying out dagster, I am already stumped at the first steps. In the guide they specify: Then, start the Dagit web server: dagit Open http://localhost:3000 with your browser to see the project. Now, you can start writing assets…
Imad
  • 2,358
  • 5
  • 26
  • 55
0
votes
0 answers

DPI-1047: Cannot locate a 64-bit Oracle Client library: "failed to get message for Windows Error 126"

I am working with dagster and dbt. I have a test repo that I use both on a Debian VM and on my Windows PC. On Debian everything works fine, here on Windows I receive the following Oracle error and can't fix it. ERROR oracle adapter: Got an error…
Lorenzo
  • 180
  • 8
1 2 3
…
9 10