I created a dataframe of type pyspark.sql.dataframe.DataFrame by executing the following line:
dataframe = sqlContext.sql("select * from my_data_table")
How can I convert this back to a sparksql table that I can run sql queries on?
Asked
Active
Viewed 2.9k times
12
Semihcan Doken
- 776
- 3
- 10
- 23
1 Answers
15
You can create your table by using createReplaceTempView. In your case it would be like:
dataframe.createOrReplaceTempView("mytable")
After this you can query your mytable using SQL.
If your a spark version is ≤ 1.6.2 you can use registerTempTable
Alberto Bonsanto
- 17,556
- 10
- 64
- 93
-
2When I run `dataframe.createOrReplaceTempView("mytable")` I get the following error: 'DataFrame' object has no attribute 'createOrReplaceTempView' – Semihcan Doken Aug 20 '16 at 03:17
-
What version of Apache spark are you using? – Alberto Bonsanto Aug 20 '16 at 10:56
-
1If you want to permanently create a table use this `df.write.format("parquet").saveAsTable("mytable")` – Shrikant Prabhu Jul 21 '21 at 06:05
-
Use following to first drop the table if exists and then create one ` spark.sql("DROP TABLE IF EXISTS " + tableName)` – Shrikant Prabhu Jul 21 '21 at 07:04