Spark temp tables not found

Asked Mar 08 '18 at 21:29

Active Mar 08 '18 at 21:29

Viewed 829 times

I'm trying to run a pySpark job with custom inputs, for testing purposes. The job has three sets of input, each read from a table in a different metastore database.

The data is read in spark with: hiveContext.table('myDb.myTable')

The test inputs are three files. In an attempt to not change any of the original code, I read all three inputs into DataFrames, and attempt to register a temp table with myDF.registerTempTable('myDb.myTable').

The problem is that spark fails with org.apache.spark.sql.catalyst.analysis.NoSuchTableException.

I've also tried:

hiveContext.sql('create database if not exists myDb')
hiveContext.sql('use myDb')
myDF.registerTempTable('myTable')

But that fails as well.

Any idea why the table cannot be found?

Using Spark 1.6

asked Mar 08 '18 at 21:29

summerbulb

5,709
8
37
83

I think the issue is the `.`. I tried something similar on my machine (Spark 2.1) and got `AnalysisException: u'It is not allowed to add database prefix 'myDb' for the TEMPORARY view name.;'` – pault Mar 08 '18 at 21:35
@pault I agree. I've seen similar behavior. How, in that case, do you register a table in a specific database? – summerbulb Mar 08 '18 at 21:53
AFAIK, I don't think you can. I think you're stuck modifying the original code unfortunately :-( – pault Mar 08 '18 at 21:54
did you try with back tick sign ` as myDF.registerTempTable(' ` myDb.myTable ` ') space is not needed though. – Ramesh Maharjan Mar 09 '18 at 03:40
@RameshMaharjan, Same issue. `NoSuchTableException` – summerbulb Mar 09 '18 at 05:47
I haven't worked with hive so I am not of good use on that :( – Ramesh Maharjan Mar 09 '18 at 05:52

Spark temp tables not found

0 Answers0