0

I want to export data from voltdb to kafka , but can't found any updating data for the corresponding Kafka topic.

voltdb deployment.xml:

 <export>
    <configuration target="archive" enabled="true" type="kafka">
        <property name="bootstrap.servers">s7biapp27:9092,s7biapp30:9092,s7biapp31:9092</property>
        <property name="topic.key">sobackup.voltdb_neweggpreso_1_</property>
        <property name="acks">0</property>
    </configuration>
</export>

voltdb stream:

CREATE STREAM SOBACKUP EXPORT TO TARGET archive (
TIMESTAMP varchar(50),
YEAR varchar(4),
QUARTER varchar(10),
MONTH varchar(30),
WEEKGROUP varchar(30),
....
);

voltdb volt.log:

 262 2018-04-23 03:56:18,089   INFO  [SP 0 Site - 0:0] EXPORT: Creating new export generation 2728899562273210367 Rejoin: false
 263 2018-04-23 03:56:18,090   INFO  [SP 0 Site - 0:0] EXPORT: Creating ad for SOBACKUP_4143034886_0
 264 2018-04-23 03:56:18,091   INFO  [SP 0 Site - 0:0] EXPORT: Creating ExportDataSource for table SOBACKUP signature SOBACKUP|vvvvvvvvvvviivvvvvvvvvvvvvvvvvvvvvvivivivvvvvvvvvv    vvvddddid partition id 0
 ...
 277 2018-04-23 03:56:18,099   INFO  [SP 0 Site - 0:0] EXPORT: Creating ad for SOBACKUP_4143034886_7
 278 2018-04-23 03:56:18,100   INFO  [SP 0 Site - 0:0] EXPORT: Creating ExportDataSource for table SOBACKUP signature SOBACKUP|vvvvvvvvvvviivvvvvvvvvvvvvvvvvvvvvvivivivvvvvvvvvv    vvvddddid partition id 7
 279 2018-04-23 03:56:18,188   INFO  [SP 0 Site - 0:0] IMPORT: Final importer count:0
 280 2018-04-23 03:56:18,189   INFO  [SP 0 Site - 0:0] HOST: Elastic duration set to 50 milliseconds
 281 2018-04-23 03:56:18,189   INFO  [SP 0 Site - 0:0] HOST: Elastic throughput set to 2 mb/s
 282 2018-04-23 03:56:18,189   INFO  [SP 0 Site - 0:0] HOST: Max temptable size set to 100 mb
 283 2018-04-23 03:56:18,189   INFO  [SP 0 Site - 0:0] HOST: Snapshot priority set to 6 [0 - 10]
 284 2018-04-23 03:56:18,189   INFO  [SP 0 Site - 0:0] HOST: Query timeout set to 10000 milliseconds
 285 2018-04-23 03:56:18,189   INFO  [SP 0 Site - 0:0] HOST: Resource limit monitoring configured to run every 60 seconds
 286 2018-04-23 03:56:18,189   INFO  [SP 0 Site - 0:0] HOST: RSS limit: 80% (49.22 GB)
 287 2018-04-23 03:56:18,194   INFO  [ExportDataSource gen 2728898351436349439 table SOBACKUP partition 0] EXPORT: Pushed EOS buffer with 0 bytes remaining
 ...
 302 2018-04-23 03:56:18,200   INFO  [ExportDataSource gen 2728898351436349439 table SOBACKUP partition 3] EXPORT: Drained source in generation 2728898351436349439 with 8 of 8 d    rained
 303 2018-04-23 03:56:18,200   INFO  [GuestProcessor gen Export Generation - 2728898351436349439 shutdown task] EXPORT: Finished draining generation 2728898351436349439
 304 2018-04-23 03:56:18,200   INFO  [GuestProcessor gen Export Generation - 2728898351436349439 shutdown task] EXPORT: Creating connector org.voltdb.export.processors.GuestProc    essor
 305 2018-04-23 03:56:18,201   INFO  [GuestProcessor gen Export Generation - 2728898351436349439 shutdown task] EXPORT: Setting master for partition: 0 Table SOBACKUP Replica ru    nning false

So, please help me find out where's wrong. Is my deployment.xml?

Ckll
  • 17
  • 4
  • It's not clear what the problem might be from this. If you call "exec @Statistics TABLE 0;" and filter the output for the SOBACKUP table, the TUPLE_COUNT column will show how many records have so far been inserted into the stream, and the TUPLE_ALLOCATED_MEMORY column will show the total size in bytes that is currently buffered (e.g. has not yet been published to kafka). – BenjaminBallard Apr 23 '18 at 13:45
  • In a word, I can't export data to kafka from voltdb. I tryed "exec @Statistics TABLE 0;" then the values of TUPLE_COUNT column and TUPLE_ALLOCATED_MEMORY column are 0. So, please help me find out what went wrong. – Ckll May 08 '18 at 08:51
  • Since your CREATE STREAM statement includes "EXPORT TO TARGET archive", as long as you ran "voltdb init" using the deployment.xml file including the export configuration you listed above, it should be buffering records in the stream, even if it can't connect to kafka, and tuple_count would not be zero unless you never inserted anything into the stream. – BenjaminBallard May 08 '18 at 15:46

0 Answers0