NIFI Insert CSV File into Postgres Database with date fields

Question

I would like to insert csv file into my postgres database. I use processors :

Getfiles -> 
Split (cause files are big) ->
UpdateAttribute (to add avro.schema) ->
ConvertCSvToAvro ->
Putdatabaserecord.

If i use only string/text fields (in my avro schema and in column postgres database), the result is ok. But when i tried to format Date fields, i have an error.

My raw data (CSV) is :

date_export|num_etiquette|key
07/11/2019 01:36:00|BAROMETRExxxxx|BAROMETRE-xxxxx

My avro schema is :

{
"type":"record",
"name":"public.data_scope_gp_temp",
"fields":[
{"name":"date_export","type":{ "type": "int", "logicalType": "date"}},
{"name":"num_etiquette","type":"string"},
{"name":"cle_scope","type":"string"}
]}

My postgres schema is:

date_export date,
num_etiquette text COLLATE pg_catalog."default",
key text COLLATE pg_catalog."default"

Any idea ?Regards

score 0 · Answer 1 · answered Nov 12 '19 at 21:33

You don't need UpdateAttribute or ConvertCsvToAvro to use PutDatabaseRecord. You can specify a CSVReader in PutDatabaseRecord, and your CSVReader can supply the Avro schema in the Schema Text property (don't forget to set your Schema Strategy to Use Schema Text).

NIFI Insert CSV File into Postgres Database with date fields

1 Answers1