0

I have a Flink program which is generating data using a custom generator and at the same time, I want to save that data into SQLite3 database. One way to do this is to parse the data into required format within the map function and then insert data from within map operation. But I am not able to do this. The code that I am using to write data to SQLite3 is as below.I am not mentioning whole code, so database connection is also there in the initial part of code, which is not included below

 DataStream<Event> merged = stream1.union(stream2,stream3);

    merged.print();


    // sending data to Timekeeper via socket


    merged.map(new MapFunction<Event, String>() {


        @Override
        public String map(Event event) throws Exception {


            String tuple = event.toString();
             Integer patient_id = event.getPatinet_id();
             Integer sensor_id = event.getSensor_id();
             Integer uid = event.getUid();
             Long time = event.getTime();
             Integer value = event.getValue();




                 String sql1 =   "INSERT into mobile_events ( patientid , sensorid , uid , eatg ,valuez ) VALUES (" +
                         + patient_id + ","
                         + sensor_id + ","
                         + uid + ","
                         + time + ","
                         + value
                         + ");" ;


                 stmt.executeUpdate(sql1);

            return tuple + "\n";

        }
    });

issue is 

Error:(124, 22) java: local variables referenced from an inner class must be final or effectively final

if I make it final, then I get following error

Caused by: java.io.NotSerializableException: org.sqlite.jdbc4.JDBC4Statement
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:315)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:81)
... 4 more
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
  • 1
    The `stmt` in MapFunction is not serialisable. Try to use `RichMapFunction`, using `transient` keyword before db connection and statement object in the RichMapFunction, and initialise them in `open()` function. – BrightFlow Jan 05 '18 at 06:15
  • making connection object transient gives an error , I tried `transient Connection c ;` I have used `RichMapFunction` but still, I am having the same issue. Another solution I was thinking is to write to database from generator function in a parallel thread. – Amarjit Dhillon Jan 05 '18 at 06:26
  • When using `transient`, `transient Connection c` should be placed in the `RichMapFunction` as a member variable. And also, place `stmt` should be declared `transient` in `RichMapFunction`. Like this: `merged.map(new MapFunction() { private transient Connection c; private transient Statement stmt; @Override open(Configuration conf) throws Exception { // init c and stmt here }` – BrightFlow Jan 05 '18 at 20:11
  • should I make the connection in open() using try catch? – Amarjit Dhillon Jan 05 '18 at 20:12
  • 1
    I think once there is some wrong while setting up the connection(throw exceptions), the map function could not work normally. Just let the task fail without using try/catch. – BrightFlow Jan 05 '18 at 20:15

0 Answers0