I'm processing EDI files. I read every line & process it. Every line contains data for multiple insert & from thoses data I'm generating an INSERT ALL
SQL query. I would like to pass it to the writer now so he can send the query to the DB. I have no idea how to do that as this is my 1st time using Spring Batch.
Reader :
@Bean
public FlatFileItemReader<String> reader() {
return new FlatFileItemReaderBuilder<String>()
.name("docItemReader")
.resource(new ClassPathResource("TestEdifact.EDI"))
.delimited()
.delimiter("\\n")
.names("doc")
.fieldSetMapper(new StringFieldMapper())
.build();
}
This is generating a prepared statement, i don't think i should use prepared statement as i'm generating dynamic query based on the data inside the EDI file. I'm not using objects( & ORM ) as there are a LOTS of different objects and that would be painfull to handle(or maybe not ?). Anyway i'm too deep in this to do anyhow else atm. This is how my writer looks like :
@Bean
public JdbcBatchItemWriter<String> writer(DataSource dataSource) {
return new JdbcBatchItemWriterBuilder<String>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.itemPreparedStatementSetter(new StatementSetterTest())
.sql("?")
.dataSource(dataSource)
.build();
}
Can you please help me to pass the generated query to the writer and help me with the writer itself ? Thanks ! By the way if you have explanations on what you will say to me, do not hesitate i will read them !!
EDIT : Example of what my data looks like & what i want at the end :
data - this is one line for me, i splitted for readability:
DTM+137:200105022200:203′
FTX+PUR+1++THIS IS AN EXAMPLE OF EDIFACT FILE.’
NAD+BY+5400110000009::9′
output:
INSERT ALL INTO DTM (column1,column2,column3) values (137, 200105022200, 203)
INTO FTX (column1,column2,column3,column4) values ('PUR',1,'THIS IS AN EXAMPLE OF EDIFACT FILE.')
INTO NAD (column1,column2,column3) values ('BY', 5400110000009, 9) SELECT * FROM DUAL;
This is an example where i insert only, the 1st data will be insert only that's why i'm doing so BUT
i will end up generating 1 query for every line because the line can maybe update existing data instead of insert ( & delete as well ) - BTW maybe i can keep all the queries in a single String (not too good for restartability tho :x). I need the whole line & i cannot read elements 1 by 1 instead of reading the whole line. It can end up being separate line like this :
UPDATE DTM SET (column1,column2,column3) values (137, 200105022200, 203) where somecondition;
UPDATE FTX SET (column1,column2,column3,column4) values ('PUR',1,'THIS IS AN EXAMPLE OF EDIFACT FILE.') where somecondition;
UPDATE NAD SET (column1,column2,column3) values ('BY', 5400110000009, 9) where somecondition;
EDIT 2 :
Here is what my code currently looks like :
@Bean
public FlatFileItemReader<String> reader() {
return new FlatFileItemReaderBuilder<String>()
.name("docItemReader")
.resource(new ClassPathResource("file.edi"))
.delimited()
.delimiter("\\n")
.names("doc")
.fieldSetMapper(new StringFieldMapper())
.build();
}
@Bean
public DocItemProcessor processor() {
return new DocItemProcessor();
}
// is this correct ?
@Bean
public ListUnpackingItemWriter<String> listItemWriter() {
ListUnpackingItemWriter<String> writer = new ListUnpackingItemWriter<>();
writer.setDelegate(writer(dataSource()));
return writer;
}
// how should i build the writer ?
public JdbcBatchItemWriter<String> writer(DataSource dataSource) {
return new JdbcBatchItemWriterBuilder<String>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.itemPreparedStatementSetter(new StatementSetterTest())
.sql("insert into people(col) values (?)")
.dataSource(dataSource)
.build();
}
@Bean
public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
return jobBuilderFactory.get("importUserJob")
.incrementer(new RunIdIncrementer())
.listener(listener)
.flow(step1)
.end()
.build();
}
@Bean
public Step step1(ItemWriter<String> writer) {
return stepBuilderFactory.get("step1")
.<String, List<String>> chunk(10000)
.reader(reader())
.processor(processor())
.writer(listItemWriter())
.build();
}
My processor does now returns List<String>
but i dont get how to write my writer. I added comments to show you where i need help.