6

I'm building a cdc pipeline to read mysql binlog through maxwell and putting them into kafka my compression type is snappy in maxwell config.But at consumer end in my spring project I'm getting this error.

org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:361) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:195) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:167) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.Snappy.init(Snappy.java:69) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:435) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:466) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at java.base/java.io.DataInputStream.readByte(DataInputStream.java:271) ~[na:na]
at org.apache.kafka.common.utils.ByteUtils.readUnsignedVarint(ByteUtils.java:170) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:205) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:296) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecordBatch$2.doReadRecord(DefaultRecordBatch.java:278) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecordBatch$StreamRecordIterator.readNext(DefaultRecordBatch.java:617) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:582) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:551) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1578) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1613) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1454) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:687) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:638) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1299) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1233) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) ~[kafka-clients-2.7.2.jar:na]

My java version "17.0.2" 2022-01-18 LTS and using Apple M1 chip.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
jss
  • 199
  • 2
  • 13

1 Answers1

6

You'll need to update snappy version to 1.1.8.4 or later:

    <dependency>
      <groupId>org.xerial.snappy</groupId>
      <artifactId>snappy-java</artifactId>
      <version>1.1.8.4</version>
    </dependency>

Source: https://issues.apache.org/jira/browse/BEAM-11863?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17318180#comment-17318180

Son Nguyen
  • 187
  • 1
  • 6