1

Can anyone help me on this? I'm getting error,***Runtime Error: Cannot set database in spark!*** while running dbt model via Spark thrift mode with remote Hive metastore.

I need to transform some models in DBT using Apache Spark as the adapter. Now, I'm running spark locally on my local machine. I started the thrift server as below with remote hive metastore URI.

  1. Started master

./sbin/start-master.sh

  1. Started worker

./sbin/start-worker.sh spark://master_url:7077

  1. Started Thrift Server

./sbin/start-thriftserver.sh --master spark://master_url:7077 --packages org.apache.iceberg:iceberg-spark3-runtime:0.13.1 --hiveconf hive.metastore.uris=thrift://ip:9083

In my DBT project,

project_name:
  outputs:
    dev:
      host: localhost
      method: thrift
      port: 10000
      schema: test_dbt
      threads: 4
      type: spark
      user: admin
  target: dev

While executing dbt run, getting the following error.

dbt run --select test -t dev
Running with dbt=1.1.0
Partial parse save file not found. Starting full parse.
Encountered an error:
Runtime Error 
Cannot set database in spark!

Please note that there is not much info in dbt.log

This error was getting because of the " database" filed in the source yml file.

JensG
  • 13,148
  • 4
  • 45
  • 55
Jithin K J
  • 11
  • 3
  • What do you see when you run `dbt debug`? Your profile has the placeholder name, `project_name` still in it -- that needs to match the name you gave your project in `dbt_project.yml`. When you do `run` you might be grabbing the config from the wrong profile. Additionally, you're specifying a `target` with the `-t` option named `spark`, but the only target you have defined in your profile is named `dev` – tconbeer Jun 13 '22 at 14:59
  • @tconbeer , sorry for that, I forgot to update "dev" in the question. in dbt debug, Connection: host: localhost port: 10000 cluster: None endpoint: None schema: test organization: 0 Connection test: [OK connection ok] All checks passed! – Jithin K J Jun 14 '22 at 11:20
  • This error was getting because of the " database" filed in the source yml file. – Jithin K J Jun 24 '22 at 08:31
  • Yes, I had to remove the "database" setting in source.yml but I kept the "schema", with the database name: `sources: - name: XYZ schema: "{{ var('database') }}"` – Jakub Adamek Mar 05 '23 at 20:57

0 Answers0