PDI - PosgreSQL Bulk Load error, Insert/Update and Table Output slow

Question

i wanna ask about this : BTW, I'm new with PostgreSQL and PDI. I have search about this in internet and I don't get it how PostgresSQL Bulk Load work and why this error (image below) show up.

PostgreSQL Bulk Load Error

This error show up when I try to insert CSV into postgres DB.

Why I used PostgreSQL Bulk Load? 'cause in my transformation after doing some ETL to my data and try to insert it directly using Table Output or Insert/Update into postgre DB it take long time (17m rows take 2h55m with speed 1.500-1.750 rows/s). And I will have about 75m+ rows of data that need to process.

That why I use PostgreSQL Bulk Load and change my transformation. I will have 2 tranformations. 1 for data process and another one for insert using bulk load. 1st transformation, Text File Input > do some ETL and process the data > Text Files Output as CSV again.
2st transformation is like the image above.

Or if you guys have any idea/another way to do it please?

which version of PDI? – Zaynul Abadin Tuhin Jun 22 '18 at 06:59 — Zaynul Abadin Tuhin, Jun 22 '18 at 06:59

score 0 · Answer 1 · answered Jun 22 '18 at 06:58

0

you can try by using below fix of PostgreSQL Bulk Loader

https://github.com/dexalex84/PentahoKettlePostgreSQLBulkInsertFix/blob/master/engine/src/org/pentaho/di/trans/steps/pgbulkloader/PGBulkLoader.java

answered Jun 22 '18 at 06:58

Zaynul Abadin Tuhin

31,407
5
33
63

what should I do with that java code? is this a proper way to solve my problem? can u explain it to me? like I said I'm new with these thing – Rio Odestila Jun 22 '18 at 07:33
you need to build that code and make jar file use it inside plugins folder – Zaynul Abadin Tuhin Jun 22 '18 at 07:44
@ZaynulAbadinTuhin can you explain how to use it, and where to put it? i have same issue as the author – Andrey Jul 11 '22 at 00:08

PDI - PosgreSQL Bulk Load error, Insert/Update and Table Output slow

1 Answers1