I have a springboot - Spark application that will inject a dataFrame into an SQL Server DB, the app was working fine before we decide to enable Alaway Encrypted for a couple sensible columns. Just after DBA configured the DB with Always Encrypted we get the following error message :
com.microsoft.sqlserver.jdbc.SQLServerException: Source and destination schemas do not match.
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.validateColumnMappings(SQLServerBulkCopy.java:1855)
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(SQLServerBulkCopy.java:1682)
at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(SQLServerBulkCopy.java:669)
at com.microsoft.azure.sqldb.spark.connect.DataFrameFunctions.com$microsoft$azure$sqldb$spark$connect$DataFrameFunctions$$bulkCopy(DataFrameFunctions.scala:127)
at com.microsoft.azure.sqldb.spark.connect.DataFrameFunctions$$anonfun$bulkCopyToSqlDB$1.apply(DataFrameFunctions.scala:72)
at com.microsoft.azure.sqldb.spark.connect.DataFrameFunctions$$anonfun$bulkCopyToSqlDB$1.apply(DataFrameFunctions.scala:72)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
Connexion String is :
this.connection = DriverManager.getConnection("jdbc:sqlserver://" + parameters.server + ";databaseName=XXXX;columnEncryptionSetting=Enabled;keyVaultProviderClientId=xxxxxxxx;keyVaultProviderClientKey=xxxxxxxx",parameters.user, parameters.password);
and the dataFrame BulkCopy config is :
java.util.Map<String, Object> map = new java.util.HashMap<String, Object>();
map.put("url", databaseParameters.server);
map.put("databaseName", databaseParameters.database);
map.put("user", databaseParameters.user);
map.put("password", databaseParameters.password);
map.put("dbTable", databaseTable);
map.put("bulkCopyBatchSize", "200000");
map.put("bulkCopyTableLock", "false");
map.put("bulkCopyTimeout", "600");
map.put("useUnicode",true);
map.put("characterEncoding","UTF-8");
scala.collection.immutable.Map<String, Object> props = JavaConverters.mapAsScalaMapConverter(map).asScala()
.toMap(Predef.$conforms());
config = new com.microsoft.azure.sqldb.spark.config.SqlDBConfigBuilder(props).build();
The injection methode is :
...
dataFrameFunctions = new DataFrameFunctions<Row>(df);
dataFrameFunctions.bulkCopyToSqlDB(config, metadata, false);
I did check the difference between the table after enabling AE and a backup with AE, both tables have the same schema (except the encrypted)
Any idea please !
Thanks in advance