0

I've setup a data pipeline using divolte.io to stream click data from website to a server. I'm not sure how can I do this for multiple websites because all the streams can get mixed up. Any ideas on how to do this?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Jaskaran Singh Puri
  • 729
  • 2
  • 11
  • 37

1 Answers1

0

On the same server, you need to bind to different ports

Create more than one config file, setting divolte.global.server.port to different values, then run the application with those configs.

In order to set a new config file, it actually needs to be in it's own directory

Divolte Collector will try to find configuration files at startup in the configuration directory. Typically this is the conf/ directory nested under the Divolte Collector installation. Divolte Collector will try to locate the configuration directory at ../conf relative to the startup script. The configuration directory can be overridden by setting the DIVOLTE_CONF_DIR environment variable. If set, the value will be used as configuration directory

Alternatively, you could run the exact same config within many containers/VMs, then use port mappings around that

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • How many ports will I be able add, say a 1000 users want to use the service. It won't be scalable, is their a different way? Can I allow multiple users to use the same link/port to send data and I do the segmentation at database level, according to domains. Does that make sense? – Jaskaran Singh Puri Jan 22 '19 at 11:17
  • You can use any port from 1024-65535. If you want scalability, I would put it behind a reverse proxy / load balancer, behind a single address & port. I haven't used Divollte, though, but behind the scenes, Kafka scales on its own. You can have one topic for all sites, or likely break it between differerent sites – OneCricketeer Jan 22 '19 at 16:21