2

I am using yugabyte db 1.3.0.0 and following https://docs.yugabyte.com/latest/deploy/cdc/use-cdc/ to learn yugabyte db cdc.

Procedure Followed:

a) generate users dynamically and insert into table, yugastore.users continuously through script
b) downloaded yb-cdc-connector.jar using command, wget -O yb-cdc-connector.jar https://github.com/yugabyte/yb-kafka-connector/blob/master/yb-cdc/yb-cdc-connector.jar?raw=true
c)copied yb-cdc-connector.jar to jre/lib/ext using command cp -a /root/yb-cdc-connector.jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/ext/
d) java -jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/ext/yb_cdc_connector.jar --table_name yugastore.users --master_addrs 127.0.0.1 --stream_id 1 --log_only

Error Logs:

[root@srvr0 ~]# java -jar /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/ext/yb_cdc_connector.jar --table_name yugastore.users --master_addrs 127.0.0.1 --stream_id 1 --log_only
Error: Unable to access jarfile /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/ext/yb_cdc_connector.jar

Records Insertion:

yugastore=> select count(*) from yugastore.users;
 count 
-------
 59414
(1 row)

yugastore=> select count(*) from yugastore.users;
 count 
-------
 60066
(1 row)

yugastore=> select count(*) from yugastore.users;
 count 
-------
 79341
(1 row)

yugastore=>

java:

[root@srvr0 ~]# java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

Availability of yb-cdc-connector.jar:

[root@srvr0 ~]# ll /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/ext/
total 54756
-rw-r--r--. 1 root root  4003855 Oct 22 12:18 cldrdata.jar
-rw-r--r--. 1 root root     9445 Oct 22 12:18 dnsns.jar
-rw-r--r--. 1 root root    48733 Oct 22 12:18 jaccess.jar
-rw-r--r--. 1 root root  1204895 Oct 22 12:18 localedata.jar
-rw-r--r--. 1 root root      617 Oct 22 12:18 meta-index
-rw-r--r--. 1 root root  2033680 Oct 22 12:18 nashorn.jar
-rw-r--r--. 1 root root    52079 Oct 22 12:18 sunec.jar
-rw-r--r--. 1 root root   304504 Oct 22 12:18 sunjce_provider.jar
-rw-r--r--. 1 root root   279788 Oct 22 12:18 sunpkcs11.jar
-rw-r--r--. 1 root root 48026360 Jan 16 08:31 yb-cdc-connector.jar
-rw-r--r--. 1 root root    78006 Oct 22 12:18 zipfs.jar

Please help me in printing change logs at stdout!

Update1:

small correction required in the doc: edit yb_cdc_connector.jar to yb-cdc-connector.jar in the document command, as wget downloads as yb_cdc_connector.jar.

Though there are changes happening (inserts and deletes) to the table, it showing polling but no cdc is printed.

Logs:

[root@srvr0 ~]# java -jar ./yb_cdc_connector.jar --table_name yugastore.users --master_addrs 127.0.0.1 --stream_id 1 --log_only
[2020-01-22 01:37:24,221] INFO Starting CDC Kafka Connector... (org.yb.cdc.Main:28)
2020-01-22 01:37:24,393 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Creating new YB client...
[2020-01-22 01:37:28,344] INFO Discovered tablet YB Master for table YB Master with partition ["", "") (org.yb.client.AsyncYBClient:1593)
2020-01-22 01:37:28,839 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Polling for new tablet c6f3d759202341ecad87e2617579371c
2020-01-22 01:37:28,842 [INFO|org.yb.cdc.KafkaConnector|KafkaConnector] Polling for new tablet e94b1748d1a742289500a30a38ff9eda

Insertion:

time: 01:37:49.980 cumulative records: 10
time: 01:37:52.169 cumulative records: 20
time: 01:37:52.410 cumulative records: 30
time: 01:37:52.425 cumulative records: 40
...
time: 01:39:28.139 cumulative records: 970
time: 01:39:28.171 cumulative records: 980
time: 01:39:28.208 cumulative records: 990
time: 01:39:28.246 cumulative records: 1000

Deletion : 

yugastore=# delete from yugastore.users;
DELETE 21314
yugastore=# select count(*) from yugastore.users;
 count 
-------
     0
(1 row)
AVA
  • 2,474
  • 2
  • 26
  • 41

1 Answers1

3

The error shows that the jar file cannot be accessed in the system directory you moved. Most likely a permission issue. Note that you don't even need to move this jar into a system directory. So keeping the jar is some normal directory is the simplest option.

Also, you don't need the stream_id as well since you want the default behavior most times. So try the following command in a user directory where you have the jar file.

java -jar ./yb_cdc_connector.jar \ 
--table_name yugastore.users \
--master_addrs 127.0.0.1:7100 \
--log_only
Sid Choudhury
  • 190
  • 4
  • 1
    now, getting polling but no cdc is printed. help me in resolving the issue – AVA Jan 22 '20 at 06:49
  • you should run the jar only after your `yugastore.users` table is created. then you should start the load generator to populate the table .. these new rows would then be streamed out to the output of the jar. note that CDC is meant for capturing changes (i.e. new inserts) and is not meant for moving existing data. A working example is here for your review: https://docs.yugabyte.com/latest/explore/change-data-capture/ – Sid Choudhury Jan 23 '20 at 20:00
  • whether CDC is supported in version 1.3.0.0. – AVA Jan 25 '20 at 09:26
  • whether any libpq based CDC in c++ is there. – AVA Jan 25 '20 at 09:27
  • CDC was first introduced in the 2.0 release so 1.3 is not a release where CDC is available. Since it has been significantly enhanced after the original 2.0 release, we recommend using the latest release (currently 2.0.10 as per https://docs.yugabyte.com/latest/releases/). – Sid Choudhury Jan 27 '20 at 17:42
  • The sample CDC client implementation today is available only in Java. The plan is certainly to introduce more examples in other languages. If you would like to collaborate on this with a maintainer, recommend filing a github issue at https://github.com/yugabyte/yugabyte-db/issues. – Sid Choudhury Jan 27 '20 at 17:44
  • Submitted request in github for Change Data Capture tool in c++ using libpq #3459(https://github.com/yugabyte/yugabyte-db/issues/3459) – AVA Jan 28 '20 at 13:43