Apache Pulsar is an open-source distributed pub-sub messaging system
Questions tagged [apache-pulsar]
339 questions
0
votes
1 answer
Apache Pulsar Clickhouse Sink - does it have intervals between inserts?
Clickhouse allows high performance writes but only if they are done in bulk and with intervals (recommended is at least 1 second interval between inserts). In the documentation to JDBC connector for Clickhouse batchSize option exists but there is…
Andrii Rusanov
- 4,405
- 2
- 34
- 54
0
votes
1 answer
How to check for the existence of a Pulsar topic programmatically in Java?
When auto-topic-creation is disabled on Pulsar brokers (eg: using /bin/pulsar-admin namespaces set-auto-topic-creation public/default --disable), it may be useful to check the existence of the topic and eventually create it if missing before…
fhussonnois
- 1,607
- 12
- 23
0
votes
0 answers
Read from Pulsar and load to Postgres with asyncio
I am trying to read data from pulsar and write to postgres table with asyncio and asyncpg in Python.
import asyncio
import asyncpg
import requests
import nest_asyncio
nest_asyncio.apply()
class Connection:
loop =…
jovicbg
- 1,523
- 8
- 34
- 74
0
votes
1 answer
I would like to enable Flink to read from Pulsar
I’m new to Pulsar!
Now, I am trying to implement these codes( https://flink.apache.org/2019/05/03/pulsar-flink.html ) in Scala. However, I can’t find some classes(e.g. PulsarSourceBuild).
How can I do them in Scala? Where should I check?
xksa
- 87
- 1
- 8
0
votes
1 answer
Receiving "No appender found" error for log4j2 when adding Pulsar appender
I'm working on a complex application that uses an Ignite version of log4j2. It works perfectly fine, but when I try to add a Pulsar appender it throws an error:
log4j:WARN No appenders could be found for logger…
doctopus
- 5,349
- 8
- 53
- 105
0
votes
1 answer
What is the best way to parallelize Pulsar consumer workload?
I want to use Pulsar as a message queue using shared consumers and the Java client. For the moment being, there are no strict ordering requirements, and also no partitions. The tasks triggered by the messages usually take up to 2 seconds. Is there…
Seldon
- 388
- 2
- 12
0
votes
1 answer
End to End Encryption Using Apache Pulsar Per Tenant/Topic
I'm trying to incorporate end to end encryption using Apache Pulsar. So far the examples I've seen look similar to http://pulsar.apache.org/docs/en/security-encryption/
I saw that the encryption key (public / private key) is fetched every 4 hours…
Yezen Rashid
- 11
- 1
0
votes
1 answer
Is there a Pub Sub service where publishers only send messages to a given topic if there are subscribers?
I'm looking for a scalable pub sub cluster where it's possible that publishers only send messages for a given topic to socketcluster if there are subscribers?
The reason being is that all of my publishers process millions of messages for many topics…
Fachtna Simi
- 437
- 2
- 13
0
votes
1 answer
Apache pulsar: Akka streams - consumer configuration
I have to write a simple service (Record ingester service) via which I need to consume messages present on apache pulsar and store them to elastic store and for that I am using com.sksamuel.pulsar4s.akka.
Messages on pulsar is produced by another…
user1270392
- 2,981
- 4
- 21
- 25
0
votes
1 answer
"Bystander" Pulsar Consumer for logging
How do I create a Pulsar consumer that listens to topic(s), but does not acknowledge the messages, basically only "eavesdropping" the conversation.
wiradikusuma
- 1,930
- 4
- 28
- 44
0
votes
0 answers
Can't connect to remote dockerized pulsar server
I've set up a dockerized pulsar server on a remote machine, which I'm connected to via VPN on ip address 172.21.11.82. On my local machine, I'm running the pulsar client to try to connect to the remote pulsar server on url…
doctopus
- 5,349
- 8
- 53
- 105
0
votes
1 answer
Using Pulsar as appender to Log4j2: How to stop logging acknowledgements?
I have a Producer that logs messages using Apache Pulsar as an appender to Log4j2.
The Consumer is listening on the same Pulsar topic and writing the received messages to a log file. The Consumer is able to receive the messages properly, but the…
doctopus
- 5,349
- 8
- 53
- 105
0
votes
0 answers
From pulsar function how to send messages to different pulsar topic partition?
may I know from pulsar functions how to publish message(s) to different pulsar topic partition based on hashing? . pulsar producer has various hashing schemes to send messages to different topic, like that i am looking for.
Murugan M
- 13
- 3
0
votes
1 answer
Unable to deploy Apache Pulsar on AKS. 'selectedNode annotation value "" not set'
This is my first attempt to deploy Pulsar on AKS v1.15.11.
I'm getting a not very verbose error messages from 2 pods that are "unscheduled" :
Firtst pod unscheduled "pulsar-zookeeper-0"
[Pod] [pulsar-zookeeper-0] FailedScheduling: selectedNode…
Odubuc
- 656
- 6
- 15
0
votes
0 answers
The python pulsar concumer is block when receiving the message
There two python scripts producer.py and consumer.py in my application.
# producer.py
client = pulsar.Client("pulsar://localhost:6650")
producer = client.create_producer("topic1")
producer.send(pickle.dumps(obj))
producer =…
Huanran Xue
- 1
- 2