Run Hadoop with Docker (for both DEV and PROD environments)

Question

What is the easiest way (that works for both dev' and real environments) to run Hadoop with Docker?

That is for local development environment and real environment difference should be only destination machine.

P.S. related to

• Is Hadoop in Docker container faster/worth it?
• Using Hadoop and Spark on Docker containers
• Run a hadoop cluster on docker containers
• How to set up Hadoop in Docker Swarm?
• Is it possible to start multi physical node hadoop clustster using docker?
• Multi-node Hadoop cluster with Docker

and many https://stackoverflow.com/questions/tagged/hadoop+docker

Answer 1

There are a few Hadoop images on Docker Hub, but if you want something suitable for different environments, you'll want to run fully distributed - i.e. with a container for the HDFS and YARN master node, and multiple containers for the worker nodes.

I have an image which works like that which you can use as a starting point: sixeyed/hadoop-dotnet. You can see from the Dockerfile that it starts from the Java base image, installs Hadoop and uses a startup script so containers can be run as a master or a worker.

That means you can run a distributed cluster with Docker, using the latest networking stack:

docker network create hadoop

docker run -d -p 50070:50070 -p 8088:8088 \
    --network hadoop --name hadoop-dotnet-master \
    sixeyed/hadoop-dotnet master

docker run -d -p 50075:50075 -p 8142:8042 -p 19888:19888 \
    --network hadoop 
    sixeyed/hadoop-dotnet worker

Or you can run a fully-distributed cluster with a Docker Compose file.

That image also includes .NET Core, but you can cut out that part if you're not using it.