Shipping Logs Securely to a Remote Process Group using MiNiFi

Question

I am having a little bit of challenge with NiFi…MiNiFi precisely. We use MiNiFi to ship logs from remote systems to a NiFi Instance, from there to Kafka and into Elasticsearch. We can successfully do this without https, However, recently I was tasked to do same securely using https.

Using certificates, I can connect to the NiFi UI, the challenge is that MiNiFi is unable to connect to the RPG on the remote NiFi with the error "Unable to communicate with Remote NiFi at URI https://xxxx.com:9443/nifi due to: Received fatal alert: handshake_failure" . I suspect this is because of the errors below

2018-07-23 16:27:23,083 INFO [main] o.apache.nifi.controller.FlowController Not enabling RAW Socket Site-to-Site functionality because nifi.remote.input.socket.port is not set

2018-07-23 16:27:23,083 INFO [main] o.apache.nifi.controller.FlowController Not enabling HTTP(S) Site-to-Site functionality because the 'nifi.remote.input.http.enabled' property is not true

I have tried to set these properties in the nifi.properties file of MiNiFi, but the file is always overwritten at each restart with default values loaded. Please, do you have any ideas on how to resolve this?

How can I bootstrap these settings at startup in the config.yml file or any other place?

score 1 · Accepted Answer · answered Jul 23 '18 at 20:57

You'll need to set those in the original flow that you export from NiFi to MiNiFi. The nifi.properties of the MiNiFi instance is automatically generated from the provided config.yml file. That file is generated by using the MiNiFi Converter Toolkit to convert the exported template XML file.

For more, you can watch these videos or read the Getting Started Guide.

You'll want to look for lines like the following in the config.yml:

Security Properties:
    keystore: /tmp/ssl/localhost-ks.jks
    keystore type: JKS
    keystore password: localtest
    key password: localtest
    truststore: /tmp/ssl/localhost-ts.jks
    truststore type: JKS
    truststore password: localtest
    ssl protocol: TLS
    Sensitive Props:
        key:
        algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
        provider: BC

Remote Processing Groups:
- name: http://localhost:8080/nifi
  url: http://localhost:8080/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  Input Ports:
  - id: AUTOGENERATED_NIFI_PORT_ID_HERE
    name: MiNiFi-input
    comment: ''
    max concurrent tasks: 1
    use compression: false
    Properties: # Deviates from spec and will later be removed when this is autonegotiated      
        Port: 1026      
        Host Name: localhost

Thank you @Andy. I gave up on this project, using another workaround and only just returned to it. Amazing your solution worked out for me. — Princewill Ezeidei, Oct 04 '21 at 12:58

Shipping Logs Securely to a Remote Process Group using MiNiFi

1 Answers1