Path for Flink state.checkpoints.dir in docker-compose in Windows 10 environment

Question

I have Windows 10 OS, docker-compose and want to work with Apache Flink tutorial Playground, docker-compose starting correctly starting docker-compose but after several minutes of work, Apache Flink has to create checkpoints, but there is some problem with access to the file system.

Exception:

org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize the pending checkpoint 104. Failure reason: Failure to finalize checkpoint.
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1216) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
…..

Caused by: org.apache.flink.util.SerializedThrowable: Mkdirs failed to create file:/tmp/flink-checkpoints-directory/d73c2f87b0d7ea6748a1913ee4b50afe/chk-104
    at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:262) ~[flink-dist_2.11-1.12.1.jar:1.12.1]

Could you please help me with the correct path and docker access?

state.backend: filesystem
state.checkpoints.dir: file:///tmp/flink-checkpoints-directory
state.savepoints.dir: file:///tmp/flink-savepoints-directory

Also I tried use full Windows path but got the same error.

Are you using Windows or Linux Docker containers ? – Guillaume Vauvert Feb 25 '21 at 18:06 — Guillaume Vauvert, Feb 25 '21 at 18:06

score 0 · Accepted Answer · answered Feb 25 '21 at 18:06

Are you using Windows Docker containers or Linux Docker containers ?

Right-click on the Docker Desktop icon to see your current configuration.

Switch to Windows containers

OR

Switch to Linux containers

You have to configure Flink paths according to you target Docker container type.

Note: You cannot use Windows and Linux containers at the same time.

score 0 · Answer 2 · answered May 22 '23 at 16:54

For anyone else struggling with this, I highly recommend using WSL2 instead of WSL1. Additionally, for the volumes I've configured them as so:

diff --git a/operations-playground/docker-compose.yaml b/operations-playground/docker-compose.yaml
index 5478271..85a5fe0 100644
--- a/operations-playground/docker-compose.yaml
+++ b/operations-playground/docker-compose.yaml
@@ -27,7 +27,7 @@ services:
       - kafka
     volumes:
       - ./conf:/opt/flink/conf
-      - flink_data:/tmp/
+      - /tmp/:/tmp/
     environment:
       - JOB_MANAGER_RPC_ADDRESS=jobmanager
   clickevent-generator:
@@ -42,7 +42,7 @@ services:
       - 8081:8081
     volumes:
       - ./conf:/opt/flink/conf
-      - flink_data:/tmp/
+      - /tmp/:/tmp/
     environment:
       - JOB_MANAGER_RPC_ADDRESS=jobmanager
   taskmanager:
@@ -52,7 +52,7 @@ services:
     command: "taskmanager.sh start-foreground"
     volumes:
       - ./conf:/opt/flink/conf
-      - flink_data:/tmp/
+      - /tmp/:/tmp/
     environment:
       - JOB_MANAGER_RPC_ADDRESS=jobmanager
   zookeeper:
@@ -69,5 +69,3 @@ services:
     ports:
       - 9092:9092
       - 9094:9094
-volumes:
-  flink_data:

And I'm running everything from within a WSL2 Ubuntu distro. Follow the guide on https://docs.docker.com/desktop/windows/wsl/ closely to make this work for you.

Path for Flink state.checkpoints.dir in docker-compose in Windows 10 environment

2 Answers2