0

I'm processing EDI files. I read every line & process it. Every line contains data for multiple insert & from thoses data I'm generating an INSERT ALL SQL query. I would like to pass it to the writer now so he can send the query to the DB. I have no idea how to do that as this is my 1st time using Spring Batch. Reader :

@Bean
    public FlatFileItemReader<String> reader() {
        return new FlatFileItemReaderBuilder<String>()
            .name("docItemReader")
            .resource(new ClassPathResource("TestEdifact.EDI"))
            .delimited()
            .delimiter("\\n")
            .names("doc")
            .fieldSetMapper(new StringFieldMapper())
            .build();
    }

This is generating a prepared statement, i don't think i should use prepared statement as i'm generating dynamic query based on the data inside the EDI file. I'm not using objects( & ORM ) as there are a LOTS of different objects and that would be painfull to handle(or maybe not ?). Anyway i'm too deep in this to do anyhow else atm. This is how my writer looks like :

@Bean
    public JdbcBatchItemWriter<String> writer(DataSource dataSource) {
        return new JdbcBatchItemWriterBuilder<String>()
            .itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
            .itemPreparedStatementSetter(new StatementSetterTest())
            .sql("?")
            .dataSource(dataSource)
            .build();
    }

Can you please help me to pass the generated query to the writer and help me with the writer itself ? Thanks ! By the way if you have explanations on what you will say to me, do not hesitate i will read them !!


EDIT : Example of what my data looks like & what i want at the end :

data - this is one line for me, i splitted for readability:

DTM+137:200105022200:203′
FTX+PUR+1++THIS IS AN EXAMPLE OF EDIFACT FILE.’
NAD+BY+5400110000009::9′

output:

INSERT ALL INTO DTM (column1,column2,column3) values (137, 200105022200, 203)
INTO FTX (column1,column2,column3,column4) values ('PUR',1,'THIS IS AN EXAMPLE OF EDIFACT FILE.')
INTO NAD (column1,column2,column3) values ('BY', 5400110000009, 9) SELECT * FROM DUAL;

This is an example where i insert only, the 1st data will be insert only that's why i'm doing so BUT i will end up generating 1 query for every line because the line can maybe update existing data instead of insert ( & delete as well ) - BTW maybe i can keep all the queries in a single String (not too good for restartability tho :x). I need the whole line & i cannot read elements 1 by 1 instead of reading the whole line. It can end up being separate line like this :

UPDATE DTM SET (column1,column2,column3) values (137, 200105022200, 203) where somecondition;
UPDATE FTX SET (column1,column2,column3,column4) values ('PUR',1,'THIS IS AN EXAMPLE OF EDIFACT FILE.') where somecondition;
UPDATE NAD SET (column1,column2,column3) values ('BY', 5400110000009, 9) where somecondition;

EDIT 2 :

Here is what my code currently looks like :

    @Bean
    public FlatFileItemReader<String> reader() {
        return new FlatFileItemReaderBuilder<String>()
            .name("docItemReader")
            .resource(new ClassPathResource("file.edi"))
            .delimited()
            .delimiter("\\n")
            .names("doc")
            .fieldSetMapper(new StringFieldMapper())
            .build();
    }
        
    @Bean
    public DocItemProcessor processor() {
        return new DocItemProcessor();
    }
    
    // is this correct ?
    @Bean
    public ListUnpackingItemWriter<String> listItemWriter() {
        ListUnpackingItemWriter<String> writer = new ListUnpackingItemWriter<>();
        writer.setDelegate(writer(dataSource()));

        return writer;
    }

    // how should i build the writer ?
    public JdbcBatchItemWriter<String> writer(DataSource dataSource) {
        return new JdbcBatchItemWriterBuilder<String>()
            .itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
            .itemPreparedStatementSetter(new StatementSetterTest())
            .sql("insert into people(col) values (?)")
            .dataSource(dataSource)
            .build();
    }


    @Bean
    public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
        return jobBuilderFactory.get("importUserJob")
            .incrementer(new RunIdIncrementer())
            .listener(listener)
            .flow(step1)
            .end()
            .build();
    }

    @Bean
    public Step step1(ItemWriter<String> writer) {
        return stepBuilderFactory.get("step1")
            .<String, List<String>> chunk(10000)
            .reader(reader())
            .processor(processor())
            .writer(listItemWriter())
            .build();
    }

My processor does now returns List<String> but i dont get how to write my writer. I added comments to show you where i need help.

  • `Every line contains data for multiple insert & from thoses data I'm generating an INSERT ALL SQL query`: this means for every item you generate multiple queries, is that correct? In the chunk-oriented model, the writer receives multiple items to write them in batch mode, so in your case, the writer would receive a list of list of queries and I'm not sure if this model is suitable for your case. Can you give an end to end example with a few records to explain what you are trying to achieve? – Mahmoud Ben Hassine Apr 22 '21 at 10:13
  • Sorry after re reading it's poorly explained. For now i'm generating a single query because i'm only inserting, but later for update purposes i will need to pass multiple queries to the writer for 1 processed line. I will add a (fake) example as i cannot show you the data i'm working with – Paddy Mariage Apr 22 '21 at 10:35
  • Thanks for the example. So if understand correctly, without referring to spring batch, the input is a file and the output is a list of sql queries. Is that correct? Since for each item your processor can generate multiple items, you can check this: https://stackoverflow.com/questions/37866312/spring-batch-using-an-itemwriter-with-list-of-lists – Mahmoud Ben Hassine Apr 22 '21 at 10:56
  • I will have a look at it as soon as i have time, thanks a lot for your help and time ! It does look like this is what i need. – Paddy Mariage Apr 22 '21 at 11:54
  • The answer is good but i'm not keen enough with Spring Batch to come to a solution myself... Can you please help me with the writer ? i dont know how to put the pieces together, i added the code to show you where i'm at – Paddy Mariage Apr 22 '21 at 14:40

0 Answers0