Stack Overflow
Stack Overflow

Questions tagged [dbt-cloud]

16 questions
1
vote
0 answers

Get dbt cloud job id inside a dbt macro

I'm trying to get the DBT_CLOUD_JOB_ID - as dbt_artifact seems to do (here): { set dbt_cloud_job_id = env_var('DBT_CLOUD_JOB_ID', '') %} {{ log("LOG dbt_cloud_job_id: " ~ dbt_cloud_job_id) }} Prints: LOG dbt_cloud_job_id: But I'm not succeeding.…
Lucas Mendes Mota Da Fonseca
  • 2,172
  • 2
  • 18
  • 28
1
vote
1 answer

dbt Cloud airflow conn_id from secret manager breaking parsing

I'm using aws secret manager to store my connections in airflow. So when creating the dbt connection string in the secret , i followed this…
callmedivs
  • 33
  • 5
1
vote
0 answers

How do I start a dbt run from the dbt-cloud cli?

I have a dbt project, and the dbt-cloud cli. I have my account ID and my token. I want to do a dbt run. When I try dbt-cloud job run ... it wants a job-id. But it doesn't seem to like the one I get from dbt-cloud job list Do I need to create a job…
Brian Postow
  • 11,709
  • 17
  • 81
  • 125
1
vote
0 answers

How to trigger a GH Actions workflow once a Slim CI job is completed (and successful)

We are currently running a linter for each of our CI builds within GitHub PRs. This linter, though, requires a dbt manifest.json which we can only build when the full dbt schemas are available in the PR schema. We currently need to manually…
Aleix CC
  • 1,601
  • 1
  • 7
  • 18
0
votes
1 answer

Is it possible to build seed dataset/table over multiple files in DBT?

Is it possible to build seed dataset/table over multiple files in DBT? I have two data files like below in my dbt project Building seed dataset/table on individual file works perfectly fine. However, what I am looking for is to create one seed…
Pravin Singh
  • 79
  • 7
0
votes
0 answers

Weird behaviour of a pre_hook inside a config block in a dbt model

I would like to run a DELETE statement before the execution of a model. The model is called ga_events and the DELETE statement should be based on another model called stg_events_inc. There fore I've created a macro: {%- macro…
MrZH6
  • 227
  • 1
  • 5
  • 16
0
votes
0 answers

DBT jinja code for Generating current date

Before job runs everyday, I trying to create a partition_key variable = currentdate()-1 so that my pipeline would run for previous day but I am getting errors within DBT console No filter named 'format_datetime'. line 2 set partition_key =…
NIRAJ BAIS
  • 41
  • 4
0
votes
0 answers

Run a model in production schema from development environment

We are in dbt Cloud. Many times (especially for incremental models) I would like to run a model only one time with a specific set of parameters, which I don't want to push into the main branch. I would like to test a dbt run in a development…
MrZH6
  • 227
  • 1
  • 5
  • 16
0
votes
1 answer

How trigger dbt cloud job on free version

I wan't to trigger dbt cloud job with API request from a python script, however the API is not accessible for unpaid version as the error suggest : {"status": {"code": 401, "is_success": false, "user_message": "The API is not accessible to unpaid…
user15915737
  • 165
  • 2
  • 15
0
votes
0 answers

How to pass execution_date to DbtCloudRunJobOperator를

I am wondering if there is a way to pass execution_date to DbtCloudRunJobOperator in Airflow. The following is a snippet from its documentation: trigger_job_run1 = DbtCloudRunJobOperator( task_id="trigger_job_run1", job_id=48617, …
kee
  • 10,969
  • 24
  • 107
  • 168
0
votes
0 answers

how to add python library in airflow which is deployed by kubernetes

I have deployed airflow via helmchart in minikube. I have one dag which runs dbtcloud job After deploying airflow sucessfully, when I open my airflow UI, I receive this error on top of window: File "", line 219, in _call_with_frames_removed File…
MariN
  • 11
  • 2
0
votes
0 answers

dbt cloud status bar issue

I have a strange dbt behaviour. When I run models or tests in dbt cloud, some of them passes, some fails. When I click on 'System Logs' dropdown or logs of each models, it shows the result of run and information about the run. The command, which I…
Paul
  • 31
  • 6
0
votes
0 answers

DBT cloud IDE customization possible?

I'm learning DBT, using the cloud IDE. I'm normally a True Believer in the Cult of Emacs. So, I want some control over the IDE, I want to be able to define some editor macros (like "comment out current line") and I want to redefine some keys (like…
Brian Postow
  • 11,709
  • 17
  • 81
  • 125
0
votes
1 answer

Feed Jinja list with results of SQL query

I'm new to DBT and Jinja and wondering if it is possible to dynamically define a list using a SQL query. Instead of manually declaring the items of the list like: {% set myOrders = [123,234, 345, 456, 567] %} Define the list with a SQL query,…
user1288609
  • 17
  • 9
0
votes
0 answers

Is there a way I could call an API which trigger a glue job from dbt?

I have models in dbt but I after dbt test, I want to trigger an API call which could kick-off my AWS glue job. Any suggestions for workaround on this? Approaches tried: Dbt —> fal script (which runs python code) —> py script could call glue…
Oj Sindher
  • 1
1
2