0

I have a springboot - Spark application that will inject a dataFrame into an SQL Server DB, the app was working fine before we decide to enable Alaway Encrypted for a couple sensible columns. Just after DBA configured the DB with Always Encrypted we get the following error message :

com.microsoft.sqlserver.jdbc.SQLServerException: Source and destination schemas do not match.
    at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.validateColumnMappings(SQLServerBulkCopy.java:1855)
    at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(SQLServerBulkCopy.java:1682)
    at com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(SQLServerBulkCopy.java:669)
    at com.microsoft.azure.sqldb.spark.connect.DataFrameFunctions.com$microsoft$azure$sqldb$spark$connect$DataFrameFunctions$$bulkCopy(DataFrameFunctions.scala:127)
    at com.microsoft.azure.sqldb.spark.connect.DataFrameFunctions$$anonfun$bulkCopyToSqlDB$1.apply(DataFrameFunctions.scala:72)
    at com.microsoft.azure.sqldb.spark.connect.DataFrameFunctions$$anonfun$bulkCopyToSqlDB$1.apply(DataFrameFunctions.scala:72)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
    at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)

Connexion String is :

this.connection = DriverManager.getConnection("jdbc:sqlserver://" + parameters.server + ";databaseName=XXXX;columnEncryptionSetting=Enabled;keyVaultProviderClientId=xxxxxxxx;keyVaultProviderClientKey=xxxxxxxx",parameters.user, parameters.password);

and the dataFrame BulkCopy config is :

java.util.Map<String, Object> map = new java.util.HashMap<String, Object>();
        map.put("url", databaseParameters.server);
        map.put("databaseName", databaseParameters.database);
        map.put("user", databaseParameters.user);
        map.put("password", databaseParameters.password);
        map.put("dbTable", databaseTable);
        map.put("bulkCopyBatchSize", "200000");
        map.put("bulkCopyTableLock", "false");
        map.put("bulkCopyTimeout", "600");
        map.put("useUnicode",true);
        map.put("characterEncoding","UTF-8");

        scala.collection.immutable.Map<String, Object> props = JavaConverters.mapAsScalaMapConverter(map).asScala()
                .toMap(Predef.$conforms());
        config = new com.microsoft.azure.sqldb.spark.config.SqlDBConfigBuilder(props).build();

The injection methode is :

...
dataFrameFunctions = new DataFrameFunctions<Row>(df);
dataFrameFunctions.bulkCopyToSqlDB(config, metadata, false);

I did check the difference between the table after enabling AE and a backup with AE, both tables have the same schema (except the encrypted)

Any idea please !

Thanks in advance

M-BNCH
  • 393
  • 1
  • 3
  • 18
  • I believe SQLServerBulkCopy is case sensitive when it comes to the column names - check your maps against the schema very carefully. – Bridge Sep 30 '22 at 14:04

0 Answers0