I'm using spark.read.format("jdbc").option("query", tmpSql) to load a table from Mysql, and I can see a query select * from (xxx) where 1=0 from database monitor, later I know this query is used for inferring table schema in Spark.
However when I use spark.read.format("jdbc").option("query", tmpSql).schema(xxx), the table schema inferring query is still there.
Why Spark still need to infer table schema when customSchema is already specified?
Asked
Active
Viewed 766 times
0
JianZhang
- 1
- 1
-
Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Dec 15 '21 at 13:33
1 Answers
0
Try to use the customSchema parameter of spark jdbc.
.option("customSchema", schema_str)
过过招
- 3,722
- 2
- 4
- 11