0

I'm trying to read data from table which is having ~13 million records. I used setFetchSize to read the records in batch wise doing some lookups and writing it to csv file. But output csv is generating with more than 50 million records which is not expected. I suspecting the iteration is not working as correctly. Any help will be appreciated.

def processRecords(ParamHelper params){

                 try {
                        CSVWriter writer;
                        writer = new CSVWriter(new FileWriter(params.outputDir.getAbsolutePath()+"/Records_fact.csv"),(Character)'\t',(Character)'\u0000',(Character)'\n');

                String contractRateSql = "select contract_id,season from table";

                ResultSet resRecords = stmt.executeQuery(contractRateSql);
                Map <String,Map<String,String>> masterRecords = new HashMap<String,Map<String,String>>();
                Map<String,String> existingRecords = null;
                int count = 0;
                resRecords.setFetchSize(10000)
                while(resRecords.next()) {
                        try{
                                existingRecords = new HashMap<String,String>();
                                existingRecords.put("cont_id",resRateRecords.getString("contract_id"));
                                existingRecords.put("season",resRateRecords.getString("season"));
                                masterRecords.put(resRecords.getString("contract_id")+"#"+count++,existingRecords);
                        }

                        catch(Exception e){
                                e.printStackTrace();
                        }

                        masterRecords.each{ k, v ->
                                try{
                                    //some process 
                                }catch(Exception e){

                                        e.printStackTrace();

                                }
                        }

                        if(valueList.size()>0)
                                writer.writeAll(valueList);



                        try {
                        if (resRateRecords != null) resRateRecords.close();
                        } catch (Exception e) {

                        };


                }
                writer.close();
                catch(Exception e){
                        e.printStackTrace();
                        println("Occured while fetching the data");
                }

        }
    }
marjun
  • 696
  • 5
  • 17
  • 30
  • Does it also produce this many results when you run it again after you've removed your `Records_fact.csv` file? You're calling `new FileWriter(path, true)` which means it will append the data to your already existing file. – Mark Dec 18 '18 at 15:00
  • Yes, every time its creating this many results. I have even tried with new FileWriter(path) aswell. – marjun Dec 18 '18 at 15:06
  • 2
    What's `valueList`? You're only writing when the size of that `> 0` but I don't see you adding data to it anywhere, also after you've written the data you're never clearing the `valueList`. – Mark Dec 18 '18 at 15:09
  • Simple question: you said your output csv file had 50million records, was it really 50 million records or 50 million lines. Because if an field in your database has a new line character in it then you will get a single record taking up two lines in the output file (i.e. a,b,"this field takes\ntwo lines", c,d) – Scott Conway Jan 14 '19 at 21:41

0 Answers0