How to persist nifi flowfiles by using docker-compose?

Question

right now I'm using nifi and its processors for some streaming stuff (mqtt listener, json evaluating, text replacement, write into db ...). I'm trying to persist the flowfiles and therefore I did some volume mapping (see below). But it doesn't work; after restarting the container it seems the flowfiles arent't saved ...

Could anybody give me a hint how to solve that problem?

nifi:
image: apache/nifi
restart: on-failure
ports:
  - "8000:8000"
networks:
  - traefik
environment:
  - NIFI_WEB_HTTP_PORT=8000
volumes:
  - nifi_conf:/opt/nifi/conf
  - nifi_state:/data/nifi/state
  - nifi_db:/opt/nifi/database_repository
  - nifi_flowfile:/opt/nifi/flowfile_repository
  - nifi_content:/opt/nifi/content_repository
  - nifi_provenance:/opt/nifi/provenance_repository 


volumes:
 nifi_provenance:{}
 nifi_flowfile: {}
 nifi_content: {}
 nifi_db: {}
 nifi_state: {}
 nifi_conf: {}

Thanks.

score 8 · Answer 1 · answered Sep 06 '19 at 06:50

One update from my side. Apache Nifi changed their directories after 1.8.0. So you should use the following:

volumes:
      - ./nifi_state:/opt/nifi/nifi-current/state
      - ./nifi_db:/opt/nifi/nifi-current/database_repository
      - ./nifi_flowfile:/opt/nifi/nifi-current/flowfile_repository
      - ./nifi_content:/opt/nifi/nifi-current/content_repository
      - ./nifi_provenance:/opt/nifi/nifi-current/provenance_repository

score 7 · Accepted Answer · answered Mar 25 '19 at 10:21

7

you could map docker container folders directly to the host machine like this:

services:
  nifi:
    ...
    volumes:
      - ./conf:/opt/conf
      - ./nifi_state:/data/nifi/state
      ...

no additional volume definition required

note that under windows with virtualbox this feature works only in the current user directory.

answered Mar 25 '19 at 10:21

daggett

26,404
3
40
56

For me NiFi can read from such volumes but never write. Do you know how I can give NiFi a write permission to a host folder? – DarkLeafyGreen Jul 05 '19 at 11:03
It writable for me. What OS do you have? If windows, are you using virtual box? – daggett Jul 05 '19 at 13:12
2

I am using docker on Ubuntu-1604. It works for templates: ${PWD}/templates:/opt/nifi/nifi-1.9.2/conf/templates, however it does not work for the following: flowfile_directory, content_directory, provenance_directory. Have you tried it for those folders? – DarkLeafyGreen Jul 08 '19 at 05:54

DiEVeXx · Answer 3 · 2020-04-27T11:28:16.747

FINAL EDIT: After many tests and trials, the only way I have found to persist Nifi with docker swarm has been the following:

Step 1: create nifi_data volume

$ docker volume create nifi_data

Step 2: Start the stack with the following configuration

version: "3.7"
services:
  nifi:
    image: apache/nifi:1.9.2
    ports:
      - target: 8080
        published: 9090
        protocol: tcp
        mode: host
    environment:
      - NIFI_WEB_HTTP_HOST=0.0.0.0
      #- NIFI_HOME=/home/nifi
      #- NIFI_LOG_DIR=/home/nifi/logs

    volumes:
      - nifi_data:/home/nifi

volumes:
  nifi_data:
    external: true

Step 3: Enter the container

$ docker exec -it <container_id> bash

Step 4: Copy the current in the home

$ cd /opt/nifi/nifi-current
$ cp -r ./* /home/nifi

Step 5: Remove Deployed Stack

docker stack rm nifi

Step 6: Deploy stack with the following configuration (just removed the #)

version: "3.7"
services:
  nifi:
    image: apache/nifi:1.9.2
    ports:
      - target: 8080
        published: 9090
        protocol: tcp
        mode: host
    environment:
      - NIFI_WEB_HTTP_HOST=0.0.0.0
      - NIFI_HOME=/home/nifi
      - NIFI_LOG_DIR=/home/nifi/logs

    volumes:
      - nifi_data:/home/nifi

volumes:
  nifi_data:
    external: true

score 0 · Answer 4 · answered Apr 03 '19 at 12:52

Alternatively you could only use docker-compose stop instead of docker-compose down which will not remove your container and thus keeping the volumes mounted.

This means you do not have to do any mapping of volumes and can just use this basic docker-compose file:

version: '2'
services:
  futa-nifi-lsc:
    environment:
      - NIFI_WEB_HTTP_PORT=9000
    image: apache/nifi:1.8.0
    volumes:
      - ./jdbc_driver:/opt/jdbc_driver
      - ./checkin_files:/opt/checkin_files
      - ./truststore:/opt/truststore
    ports:
      - "9000:9000"

For more information read this article here.

timtam · Answer 5 · 2020-04-21T15:35:29.980

version: '3.7'

services:

  zookeeper:  # the configuration manager
    hostname: zookeeper
    container_name: zookeeper
    image: 'bitnami/zookeeper:latest'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  nifi:
    image: apache/nifi:latest
    ports:
      - 8080 # Unsecured HTTP Web Port
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
    volumes:
      - ./nifi/state:/opt/nifi/nifi-current/state
      - ./nifi/db:/opt/nifi/nifi-current/database_repository
      - ./nifi/flowfile:/opt/nifi/nifi-current/flowfile_repository
      - ./nifi/content:/opt/nifi/nifi-current/content_repository
      - ./nifi/provenance:/opt/nifi/nifi-current/provenance_repository
      - ./nifi/logs:/opt/nifi/nifi-current/logs

How to persist nifi flowfiles by using docker-compose?

5 Answers5