9

We are struggling with trying to figure out the best approach for updating processor configurations as a flow progresses through the dev, test, and prod stages. We would really like to avoid manipulating host, port, etc. references in the processors when the flow is deployed to the specific environment. At least in our case, we will have different hosts for things like ElasticSearch, PostGres, etc. How have others handled this?

Things we have considered:

  • Pull the config from a properties file using expression language. This is great for processors that have EL enabled, but not the case for those where it isn't.
  • Manipulate the flow xml and overwrite the host, port, etc. configurations. A bit concerned about inadvertently corrupting the xml and how portable this will be across NIFI versions.

Any tips or suggestions would be greatly appreciated. There is a good chance that there is an obvious solution we have neglected to consider.

EDIT:

We are going with the templates that Byran suggested. They will definitely meet our needs and appear to be a good way for us to control configurations across numerous environments.

https://github.com/aperepel/nifi-api-deploy

scarpacci
  • 8,957
  • 16
  • 79
  • 144
  • Hi, I'm the creator of this nifi-api-deploy project. I have been making improvements to it based on the feedback and how folks want to use it. Feel free to engage and provide feedback and/or requests. – andrew Apr 21 '16 at 13:25

2 Answers2

9

This discussion comes up frequently, and there is definitely room for improvement here...

You are correct that currently one approach is to extract environment related property values into the bootstrap.conf, and then reference them through expression language so the flow.xml.gz can be moved from one environment to the other. As you mentioned this only works well with properties that support expression language.

In order to make this easier in the future, there is a feature proposal for an idea called a Variable Registry:

https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

An interesting approach you may want to look at is using templates. There is a GitHub project that can be used to help with this:

https://github.com/aperepel/nifi-api-deploy

Bryan Bende
  • 18,320
  • 1
  • 28
  • 39
  • As always, appreciate your input, Bryan. I will take a look at the templates. – scarpacci Apr 14 '16 at 13:16
  • Hi Bryan, I took a look at the nifi-api-deploy utility with @scarpacci and it definitely seems to be on the right track. We have fairly complex flows, and the code as written didn't seem to play nicely with undeploying processors/connections at the root level processor group, or the controllers. Manually deleting and redeploying seemed to work alright. It also seems to auto start processor groups, even if all contained processors are marked as halted. – Josh Harrison Apr 21 '16 at 14:47
  • Hi Josh, glad that that it seems to be on the right track for what you are looking to do. As far as the functionality, the project is maintained by a community member, so if you think something is not working correctly the best option is probably to submit an issue through the GitHub page, I know there are a couple of open issues right now with discussions going. The part about auto starting stopped process groups sounds like something that should be addressed. – Bryan Bende Apr 24 '16 at 13:29
4

You can loook at this post automating NIFI template deployment

For automating NIFI template deployment, there is a tool that works well : https://github.com/hermannpencole/nifi-config

  • Prepare your nifi development

    1. Create a template on nifi
    2. and download it
    3. Extrac a sample configuration with the tools
  • Deploy it on production
    1. undeploy the old version with the tools
    2. deploy the template with the tools
    3. update the production configuration with the tools
Hermann
  • 81
  • 4