0

I am converting some legacy Java code written for Flink version 1.5 to Flink version 1.13.1. Specifically, I'm working with Table API. I have to read data from CSV file, perform some basic SQL and then write results back to a file.

For Flink version 1.5, I used the following code to perform above actions

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
        
TableSource tableSrc = CsvTableSource.builder()
    .path("<CSV_PATH>")
    .fieldDelimiter(",")
    .field("date", Types.STRING)
    .field("month", Types.STRING)
    ...
    .build();
        
    tableEnv.registerTableSource("CatalogTable", tableSrc);

String sql = "...";
Table result = tableEnv.sqlQuery(sql);
DataSet<Row1> resultSet = tableEnv.toDataSet(result, Row1.class);
resultSet.writeAsText("<OUT_PATH>");
env.execute("Flink Table-Sql Example");

In order to convert above code to Flink version 1.13.1, I wrote the following code

import org.apache.flink.table.api.Table;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.table.api.TableEnvironment;
import org.apache.flink.table.api.EnvironmentSettings;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.table.api.bridge.java.BatchTableEnvironment;

EnvironmentSettings settings = EnvironmentSettings
    .newInstance()
    .inBatchMode()
    .build();

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
TableEnvironment tableEnv = TableEnvironment.create(settings);

final String tableDDL = "CREATE TEMPORARY TABLE CatalogTable (" +
    "date STRING, " +
    "month STRING, " +
    "..." +
    ") WITH (" +
    "'connector' = 'filesystem', " +
    "'path' = 'file:///CSV_PATH', " +
    "'format' = 'csv'" +
    ")";

tableEnv.executeSql(tableDDL);

String sql = "...";
Table result = tableEnv.sqlQuery(sql);

// DEPRECATED - BatchTableEnvironment required to convert Table to Dataset
BatchTableEnvironment bTableEnv = BatchTableEnvironment.create(env);
DataSet<Row1> resultSet = bTableEnv.toDataSet(result, Row1.class);

resultSet.writeAsText("<OUT_PATH>");
env.execute("Flink Table-Sql Example");

However, BatchTableEnvironment is marked as "Deprecated" in Flink version 1.13. Is there any alternative to convert Table to Dataset or to directly write a Table to a file?

F Baig
  • 339
  • 1
  • 4
  • 13
  • The docs say to use `TableEnvironment` as you already implemented. Do you have an error? – Felipe Jul 01 '21 at 06:23
  • `TableEnvironment` does not seem to have any method to convert to `Dataset` or to write `Table` to file – F Baig Jul 01 '21 at 14:52
  • what is your `import package....BatchTableEnvironment`? – Felipe Jul 01 '21 at 15:01
  • For version 1.13.1 `import org.apache.flink.table.api.bridge.java.BatchTableEnvironment`. Updated in original as well – F Baig Jul 01 '21 at 21:29
  • Maybe it is the case to use `org.apache.flink.table.api.java.BatchTableEnvironmen` as it is saying here https://stackoverflow.com/a/49098517/2096986 – Felipe Jul 02 '21 at 05:59
  • That is no longer available in recent versions. Interestingly their latest [doc example](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/tableapi/#overview--examples) shows `tableEnv.toDataset()` function being used but it is no longer available in their [Javadoc](https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/table/api/TableEnvironment.html) – F Baig Jul 02 '21 at 22:39
  • Just FYI, the DataSet API is soft-deprecated and there is currently a plan being established on hard-deprecating it. Ideally you'll want to migrate away from the DataSet API. – Ingo Bürk Jul 07 '21 at 06:13

0 Answers0