How to fix "incompatible types: org.apache.beam.sdk.options.ValueProvider cannot be converted to java.lang.String"

Question

I followed this link to create a template which builds a beam pipeline to read from KafkaIO. But I always encountered " incompatible types: org.apache.beam.sdk.options.ValueProvider cannot be converted to java.lang.String". It is line ".withBootstrapServers(options.getKafkaServer())" which caused the error. Beam version is 2.9.0 and here is part of my code.

public interface Options extends PipelineOptions {
    @Description("Kafka server")
    @Required
    ValueProvider<String> getKafkaServer();

    void setKafkaServer(ValueProvider<String> value);

    @Description("Topic to read from")
    @Required
    ValueProvider<String> getInputTopic();

    void setInputTopic(ValueProvider<String> value);

    @Description("Topic to write to")
    @Required
    ValueProvider<String> getOutputTopic();

    void setOutputTopic(ValueProvider<String> value);

    @Description("File path to write to")
    @Required
    ValueProvider<String> getOutput();

    void setOutput(ValueProvider<String> value);
}

public static void main(String[] args) {
    Options options = PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);
    Pipeline p = Pipeline.create(options);

    PCollection<String> processedData = p.apply(KafkaIO.<Long, String>read()
            .withBootstrapServers(options.getKafkaServer())
            .withTopic(options.getInputTopic())
            .withKeyDeserializer(LongDeserializer.class)
            .withValueDeserializer(StringDeserializer.class)
            .withoutMetadata() 
    )

And following is how I run the code:

mvn compile exec:java \
-Dexec.mainClass=${MyClass} \
-Pdataflow-runner -Dexec.args=" \
--project=${MyClass} \
--stagingLocation=gs://${MyBucket}/staging \
--tempLocation=gs://${MyBucket}/temp \
--templateLocation=gs://${MyBucket}/templates/${MyClass} \
--runner=DataflowRunner"

@IanKemp it is .withBootstrapServers(options.getKafkaServer()) — sheng666, Feb 04 '19 at 04:43

Brachi · Answer 1 · 2019-02-27T11:31:15.573

1

In order to access a value via ValueProvider, you need to use the get method, and then you get the value with its concrete type.

For example: when having option:

ValueProvider<String> getKafkaServer();

you can access it with:

getKafkaServer().get() this will return you String object.

Seems like the KafkaIo Api requires to get string parameter and not ValueProvider, you have to extract the value from the ValueProvider wrapper.

edited Feb 27 '19 at 11:31

answered Feb 19 '19 at 14:39

Brachi

637
9
17

2

Could you give explanation for this answer? Just to make it clear how to use that – Guillaume Racicot Feb 19 '19 at 19:34
Guillaume Racicto, when using `ValueProvider` you can access the value with `get` – Brachi Feb 20 '19 at 07:34
can you edit your answer with that information please? Comments don't really count as answer – Guillaume Racicot Feb 20 '19 at 12:57
2

Edited. is it explained enough? what is missing – Brachi Feb 27 '19 at 11:27

score 0 · Answer 2 · answered Feb 04 '19 at 23:57

I might find the issue, which is that kafkaIO is not supported. Following are from Google create template.

" Some I/O connectors contain methods that accept ValueProvider objects. To determine support for a specific connector and method, see the API reference documentation for the I/O connector. Supported methods have an overload with a ValueProvider. If a method does not have an overload, the method does not support runtime parameters. The following I/O connectors have at least partial ValueProvider support:

File-based IOs: TextIO, AvroIO, FileIO, TFRecordIO, XmlIO BigQueryIO* BigtableIO (requires SDK 2.3.0 or later) PubSubIO SpannerIO "

FYI, by using flex templates, one can us any kind of connector and avoid having to deal with ValueProviders. https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates — robertwb, Oct 01 '21 at 23:57

How to fix "incompatible types: org.apache.beam.sdk.options.ValueProvider cannot be converted to java.lang.String"

2 Answers2

Linked