0

I have an issue in a pyspark dataframe u_final below, when I show the dataframe it looks correct, but when I insert in the table, the insertion shows me a different dataframe with less data, and as ou can see I have nothing between the show command and the insertion command.

u_final.show()
dp.data_jdbc_writer("jdbc:oracle:thin:@//127.0.0.1:1521/JDET","user","pwd","overwrite","IFDWH.IF_DEPARTMENT", u_final)

-- data_jdbc_writer function definition :

def data_jdbc_writer(url:str,truncate:str, user:str,pwd:str,mode:str,table:str, data):
    data.write \
        .format("jdbc") \
        .option("driver", "oracle.jdbc.OracleDriver")\
        .option("url", url) \
        .option("truncate", true) \
        .option("user", user) \
        .option("password", pwd) \
        .mode(mode) \
        .option("CreateTableOptions", '') \
        .option("dbtable", table) \
        .save()

The casting is correct, the same structure and same data types.

sghiar
  • 1
  • 3
  • Please provide some extra information like final dataframe and the data after insertion in Oracle. – Mohammad Sunny Jun 27 '23 at 16:46
  • Thanks Mohammed for your comment. The problème here is weird, because as you can see u_final.show() shows the correct dataframe the I want, but right after I insert it into the table with the overwrite mode I get a different smaller dataframe inserted into the table. – sghiar Jun 28 '23 at 07:24
  • What is the dataframe count?? (U_final.count() ) – Mohammad Sunny Jun 28 '23 at 09:45
  • the count displays 91 but I get only 1 row in the table after insertion – sghiar Jun 28 '23 at 12:36
  • Most probably data column type in table can be the issue. Please try this link https://forums.oracle.com/ords/apexds/post/insert-into-table-with-restrictions-8989 – Mohammad Sunny Jun 28 '23 at 14:09

0 Answers0