1

while executing below code to connect to apache pulsar using apache beam PulsarIO in java sdk. Getting below error while adding pulsar client in beam pipeline.

Beam version 2.40, 2.41 javaSE 1.8

import java.io.*;
import org.apache.beam.sdk.*;
import org.apache.beam.io.pulsar.*;
import org.apache.beam.sdk.options.*;
import org.apache.beam.sdk.transforms.*;
import org.apache.beam.sdk.values.*;
import org.apache.beam.transforms.SerializableFunction;
import org.apache.pulsar.client.api.*;
import org.apache.pulsar.client.impl.PulsarClientImpl;
import org.apache.pulsar.client.impl.auth.AuthenticationTls;
import java.util.*;

public class BeamPulsar{
 public static void main(String args[]) throws PulsarClientException{
 PipelineOptions options=PipelineOptionsFactory.as(PipelineOptions.class);
 Pipeline p= Pipeline.create(options=options);
 PulsarClient pulsarClient;

 String TLSCERTFILE='filepath'
 String TLSKEYFILE='filepath'
 String CACERTFILE='filepath'
 String BROKER_URL='pulsar+ssl://'
 String TOPIC_NAME='topic'
 Map<String, String> authParams=new HashMap<>();
 authParams.put("tlsCertFile",TLSCERTFILE);
 authParams.put("tlsKeytFile",TLSKEYFILE);
 Authentication tlsAuth =AuthenticationFactory.create(AuthenticationTls.class.getName(),authParams);
 pulsarClient=PulsarClient.builder().serviceUrl(BROKER_URL).tlsTrustCertsFilePath(CACERTFILE).authentication(tlsAuth).build()
PCollection<PulsarMessage> records=p.apply("read from pulsar", PulsarIO.read()
                                    .withTopic(TOPIC_NAME)
                                    .withPulsarClient((SerializableFunction<String, PulsarClient>)pulsarClient)
                                    .withPublishTime()
                                    .withClientUrl(BROKER_URL)
                                    .withAdminUrl(BROKER_URL));
p.run().waitUntilFinish();
}}

Error: Exception in thread "main" java.lang.ClassException: class org.apache.pulsar.client.impl.PulsarClientImpl cannot be cast to class org.apache.beam.sdk.transform.SerializableFunction(org.apache.pulsar.client.impl.PulsarClientImpl and org.apache.beam.sdk.transforms.SerializableFunction are in unnamed module of loader app)

phani geeth
  • 21
  • 2
  • Tried `(s) -> pulsarClient` (instead of `pulsarClient`) ..your code is not compilable ;(;( – xerx593 Oct 03 '22 at 16:37
  • It worked by constructing Pulsar client within an implementations of SerializableFunction. .withPulsarClient(new SerializableFunction(){ @Override public PulsarClient apply(String i){ try{ return PulsarClient.builder().serviceUrl(BROKER_URL).tlsTrustCertsFilePath(CACERTFILE).authentication(tlsAuth).build();} catch (PulsarClientException e){} }}) – phani geeth Oct 07 '22 at 06:32
  • And if you already initialized `pulsarClient` (with `pulsarClient=PulsarClient.builder().serviceUrl(BROKER_URL).tlsTrustCertsFilePath(CACERTFILE).authentication(tlsAuth).build()`), then `(String s) -> pulsarClient` is the "short hand" for that ... A "functional interface"...with String input, and pulsar client output;) – xerx593 Oct 07 '22 at 10:34
  • So just `.withPulsarClient((s) -> pulsarClient)` ..to precise my first comment. – xerx593 Oct 07 '22 at 10:37

1 Answers1

1

You will want to put the construction of the PulsarClient inside of an implementation of SerializableFunction and not in your main. The client needs to be constructed in the remote process actually running the IO code, not the machine that submits the pipeline.

Byron
  • 46
  • 1