AccessDeniedException while running Apache Kafka 3 on Windows

Question

Apache Kafka 3.0 was released yesterday and I decided to try it on Windows 10. Although, ZooKeeper starts perfectly, Kafka shuts down after logging few exceptions.

I understand that AccessDeniedException is quite self explanatory but the permission isn't the issue because Kafka is able to create the very files for which AccessDeniedException is logged.

[2021-09-22 23:58:18,295] ERROR Failed to write meta.properties due to (kafka.server.BrokerMetadataCheckpoint)
java.nio.file.AccessDeniedException: D:\tmp\kafka-logs
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941)
        at kafka.server.BrokerMetadataCheckpoint.liftedTree1$1(BrokerMetadataCheckpoint.scala:214)
        at kafka.server.BrokerMetadataCheckpoint.write(BrokerMetadataCheckpoint.scala:204)
        at kafka.server.KafkaServer.$anonfun$checkpointBrokerMetadata$2(KafkaServer.scala:772)
        at kafka.server.KafkaServer.$anonfun$checkpointBrokerMetadata$2$adapted(KafkaServer.scala:770)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889)
        at kafka.server.KafkaServer.checkpointBrokerMetadata(KafkaServer.scala:770)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:322)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2021-09-22 23:58:18,301] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.nio.file.AccessDeniedException: D:\tmp\kafka-logs
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941)
        at kafka.server.BrokerMetadataCheckpoint.liftedTree1$1(BrokerMetadataCheckpoint.scala:214)
        at kafka.server.BrokerMetadataCheckpoint.write(BrokerMetadataCheckpoint.scala:204)
        at kafka.server.KafkaServer.$anonfun$checkpointBrokerMetadata$2(KafkaServer.scala:772)
        at kafka.server.KafkaServer.$anonfun$checkpointBrokerMetadata$2$adapted(KafkaServer.scala:770)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889)
        at kafka.server.KafkaServer.checkpointBrokerMetadata(KafkaServer.scala:770)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:322)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)

and

[2021-09-22 23:58:19,115] INFO Shutting down. (kafka.log.LogManager)
[2021-09-22 23:58:19,200] ERROR Error while writing to checkpoint file D:\tmp\kafka-logs\recovery-point-offset-checkpoint (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: D:\tmp\kafka-logs
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941)
        at kafka.server.checkpoints.CheckpointFile.liftedTree1$1(CheckpointFile.scala:114)
        at kafka.server.checkpoints.CheckpointFile.write(CheckpointFile.scala:92)
        at kafka.server.checkpoints.OffsetCheckpointFile.write(OffsetCheckpointFile.scala:67)
        at kafka.log.LogManager.$anonfun$checkpointRecoveryOffsetsInDir$1(LogManager.scala:675)
        at kafka.log.LogManager.$anonfun$checkpointRecoveryOffsetsInDir$1$adapted(LogManager.scala:671)
        at scala.Option.foreach(Option.scala:437)
        at kafka.log.LogManager.checkpointRecoveryOffsetsInDir(LogManager.scala:671)
        at kafka.log.LogManager.$anonfun$shutdown$9(LogManager.scala:542)
        at kafka.log.LogManager.$anonfun$shutdown$9$adapted(LogManager.scala:535)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.mutable.HashMap$Node.foreachEntry(HashMap.scala:633)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:499)
        at kafka.log.LogManager.shutdown(LogManager.scala:535)
        at kafka.server.KafkaServer.$anonfun$shutdown$18(KafkaServer.scala:701)
        at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:68)
        at kafka.server.KafkaServer.shutdown(KafkaServer.scala:701)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:435)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2021-09-22 23:58:19,202] ERROR Disk error while writing recovery offsets checkpoint in directory D:\tmp\kafka-logs: Error while writing to checkpoint file D:\tmp\kafka-logs\recovery-point-offset-checkpoint (kafka.log.LogManager)
[2021-09-22 23:58:19,233] ERROR Error while writing to checkpoint file D:\tmp\kafka-logs\log-start-offset-checkpoint (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: D:\tmp\kafka-logs
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941)
        at kafka.server.checkpoints.CheckpointFile.liftedTree1$1(CheckpointFile.scala:114)
        at kafka.server.checkpoints.CheckpointFile.write(CheckpointFile.scala:92)
        at kafka.server.checkpoints.OffsetCheckpointFile.write(OffsetCheckpointFile.scala:67)
        at kafka.log.LogManager.$anonfun$checkpointLogStartOffsetsInDir$1(LogManager.scala:698)
        at kafka.log.LogManager.$anonfun$checkpointLogStartOffsetsInDir$1$adapted(LogManager.scala:694)
        at scala.Option.foreach(Option.scala:437)
        at kafka.log.LogManager.checkpointLogStartOffsetsInDir(LogManager.scala:694)
        at kafka.log.LogManager.$anonfun$shutdown$9(LogManager.scala:545)
        at kafka.log.LogManager.$anonfun$shutdown$9$adapted(LogManager.scala:535)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.mutable.HashMap$Node.foreachEntry(HashMap.scala:633)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:499)
        at kafka.log.LogManager.shutdown(LogManager.scala:535)
        at kafka.server.KafkaServer.$anonfun$shutdown$18(KafkaServer.scala:701)
        at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:68)
        at kafka.server.KafkaServer.shutdown(KafkaServer.scala:701)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:435)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2021-09-22 23:58:19,234] ERROR Disk error while writing log start offsets checkpoint in directory D:\tmp\kafka-logs: Error while writing to checkpoint file D:\tmp\kafka-logs\log-start-offset-checkpoint (kafka.log.LogManager)
[2021-09-22 23:58:19,236] INFO Shutdown complete. (kafka.log.LogManager)

ZooKeeper scripts -

bin\windows\kafka-server-start.bat .\config\server.properties

Kafka starting scripts -

bin\windows\kafka-server-start.bat .\config\server.properties

I am using default configuration files for ZooKeeper and Kafka in following environment -

Environment

JDK 11.0.2
Kafka 3.0
Windows 10

There is an [endless list of issues](https://issues.apache.org/jira/browse/KAFKA-1155?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22windows%22%20ORDER%20BY%20affectedVersion%20ASC) related to Kafka running on Windows. The problem is in particular related to the Windows file system and its file lock mechanism. Your issue looks like [KAFKA-8811](https://issues.apache.org/jira/browse/KAFKA-8811) or [KAFKA-7020](https://issues.apache.org/jira/browse/KAFKA-7020). You can try to run Kafka on a Linux file system using WSL 2 (not WSL 1). — G Quintana, Sep 22 '21 at 19:44
@GQuintana Yes. Running on WSL 2 seems to be the only working way. Thanks. — Kedar Joshi, Sep 22 '21 at 19:52
Sounds like you didn't run CMD as Administrator. Also, Windows doesn't have any /tmp folder, and your default user doesn't have access to create folders in C drive, so seems like you didn't really adjust any properties to a file path you do have access to (you'd need to do the same in Unix environment as well) — OneCricketeer, Sep 22 '21 at 21:50
@OneCricketeer I didn't see anything related to running as administrator in the documentation. If you can share the reference, I would appreciate it. Also, like I mentioned in the question, how can it be an `AccessDeniedException` when Kafka is able to create the files in the exact same directory. — Kedar Joshi, Sep 23 '21 at 17:29
The Kafka documentation is Linux focused for the most part. Unclear what other files you're referring to, but the error is explicitly about the actual topic data, which Kafka requires to start — OneCricketeer, Sep 23 '21 at 18:48
Does anybody solved this issues, i tried anything, changed the log path, added user permission, used java8 instead of Java11 , but nothing works. — S.Bozzoni, Sep 25 '21 at 15:26
@S.Bozzoni No. Look at the JIRA link in the first comment. Simply don't use Windows. Use Docker or WSL2 on Windows instead — OneCricketeer, Sep 29 '21 at 14:57
I have the same problem. Previous version of Kafka (2.8.1) works ok. Can't find anything in JIRA for Kafka 3 and Windows BTW - looks like this but isn't yet posted. — mvmn, Sep 30 '21 at 21:10
Ya even I had same problem, downloaded 2.8.1 even with java 14 its working fine. Looks like some issue with 3.0.0 version. — Niru, Oct 03 '21 at 01:37
On several places in the Kafka 3.0.0, `FileChannel.open(path, StandardOpenOption.READ)` is called which will always fail on Windows when path is a directory. See: https://mail.openjdk.java.net/pipermail/nio-dev/2013-February/002123.html So a new release of Kafka is needed to get Kafka 3.0.0 running on Windows (without wsl) — userM1433372, Dec 09 '21 at 07:47
Will be fixed in Kafka 3.0.1. See https://issues.apache.org/jira/browse/KAFKA-13391 — userM1433372, Dec 09 '21 at 09:37

score 0 · Answer 1 · edited Oct 02 '21 at 18:41

Nowdays in Windows 10 it's possible to use wsl2 to run kafka. Just please follow this guide:

https://www.confluent.io/blog/set-up-and-run-kafka-on-windows-linux-wsl-2/#enable-windows-subsystem

then you can write code and launch it in wsl

Sometimes there is an issue with windows and wsl communication so you have to issue in windows command prompt wsl --shutdown and then restart it (issue wsl again), after that you can call your wsl launched application and call its exposed services from windows

AccessDeniedException while running Apache Kafka 3 on Windows

1 Answers1