Using Docker and Testcontainers for database integration testing

Question

After a bit of research I was led to think that Docker containers could be a good fit for database integration testing using test containers, as this would only require a Docker container running a database image reproducing the database schema against which the tests would be executed. This instance can either be run locally on each developer’s machine or, even better, one single instance can be shared by multiple developers: an automated test can be configured to start a database instance from the same image in Docker for every test method if needed using the @Rule annotation.

While trying to install Docker for Windows 7 to play with it, I kept getting the following error which seems to be related to VirtualBox:

docker Error creating machine: Error in driver during machine creation: Unable to start the VM: VBoxManage.exe startvm default --type headless failed:

Result code: E_FAIL (0x80004005)
Component: MachineWrap
Interface: IMachine

I will share how I worked around this problem to help other people who may have run into the same problem.

Taoufik Mohdit · Accepted Answer · 2020-09-28T11:12:55.177

To workaround the issue described in the question, I followed the steps below:

#1. Install VMware workstation player#

#2. Run a Linux Ubuntu virtual machine#

Using iso file from Ubuntu website, run an Ubuntu appliance on VMware player

#3. Install Docker on Ubuntu VM#

$ sudo apt-get install docker.io

#4. Post-install steps#

###4.1. Create the docker group and add your user###

$ sudo groupadd docker

$ sudo usermod -aG docker $USER

###4.2. Log out and log back in## so that your group membership is re-evaluated

###4.3. Verify that you can run docker commands without sudo###

$ docker run hello-world

#5. Download database image#

In this example it's Oracle 11g:

$ docker pull wnameless/oracle-xe-11g-r2

#6. Run Database image#

(if you plan to connect to the Oracle database instance from testcontainers, this step is not necessary)

$ docker run -d -p 49161:1521 wnameless/oracle-xe-11g-r2

For further details/options refer to docker hub:

#7. Connect to the database#

Using with following settings:

hostname: localhost

port: 49161

sid: xe

service name: xe

username: system

password: oracle

To connect from the host machine use the IP address of the VM instead of localhost . Run ifconfig on the guest Ubuntu VM to get the IP address

#8. Configure Docker to be accessed remotely#

Configure where the Docker daemon listens for connections (Required only if Docker needs to be accessed remotely, i.e. not from the guest Ubuntu system), as by default Docker daemon listens on unix sockets (local connections)

###8.1. Create config file###

Create /etc/systemd/system/docker.service.d/override.conf file with content to override default one (ExecStart=/usr/bin/dockerd -H fd://):

# /etc/systemd/system/docker.service.d/override.conf

[Service]

ExecStart=

ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376

To create this file you can run the following unix command:

printf "# /etc/systemd/system/docker.service.d/override.conf\n[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376" > /etc/systemd/system/docker.service.d/override.conf

###8.2. Restart Docker###

After saving this file, reload the configuration by running:

systemctl daemon-reload

Then restart Docker by running:

systemctl restart docker.service

###8.3. Check your Docker daemon###

After restarting docker service, you can see the port number in the output of either:

systemctl status docker.service

(You should see something like: /usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376)

Or

sudo netstat -tunlp | grep dock

(You should see something like: tcp6 0 0 :::2376 :::* LISTEN 121686/dockerd)

###8.4. Useful resources###

How do I enable the remote API for dockerd

How to detect a docker daemon port

Configure where the docker daemon listens for connections

#9. Set Docker Host environment variable#

This step is necessary only if you plan to connect remotely to the database container using testcontainers API (e.g. from a Junit test) from the OS hosting the Ubuntu VM (docker dameon is running on the ubuntu VM)

Define environment variable: DOCKER_HOST = tcp://<Ubuntu machine's IP address>:2376 . Note the hostname is the ubuntu VM's. If this environment variable is not defined, testcontainers API (OracleContainer oracleContainer = new OracleContainer("wnameless/oracle-xe-11g");) will be expecting Docker daemon to be running on localhost (See code snippet further below)

#10. Use the database container from a test class#

Using testcontainer API, a Junit test can start a database instance from a the Docker image on Ubuntu's VM, execute queries against it and, eventually shut it down

###Junit test class###

package com.xxx.yyy.repository;
 
import static org.junit.Assert.assertEquals;
 
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.time.LocalDateTime;
import java.util.concurrent.TimeoutException;
 
import org.junit.ClassRule;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInstance;
import org.junit.jupiter.api.TestInstance.Lifecycle;
import org.testcontainers.containers.OracleContainer;
 
@TestInstance(Lifecycle.PER_CLASS)
public class MyRepositoryIT {
 
    @ClassRule
    public OracleContainer oracleContainer;
 
    @BeforeAll
    public void setup() throws TimeoutException {
        String dockerHost = System.getenv("DOCKER_HOST");
        System.out.println("dockerHost: @" + dockerHost + "@");
 
        System.out.println("Starting Oracle Container... (" + LocalDateTime.now() + ")");
        oracleContainer = new OracleContainer("wnameless/oracle-xe-11g");
        oracleContainer.start();
        System.out.println("Oracle Container started. (" + LocalDateTime.now() + ")");
 
    }
 
    @AfterAll
    public void tearDown() {
        System.out.println("Stopping Oracle Container... (" + LocalDateTime.now() + ")");
        oracleContainer.stop();
        System.out.println("Oracle Container stopped. (" + LocalDateTime.now() + ")");
    }
 
    @Test
    public void whenSelectQueryExecuted_thenResulstsReturned() throws Exception {
 
        String jdbcUrl = oracleContainer.getJdbcUrl();
        String username = oracleContainer.getUsername();
        String password = oracleContainer.getPassword();
        Connection conn = DriverManager.getConnection(jdbcUrl, username, password);
        ResultSet resultSet = conn.createStatement().executeQuery("SELECT 1 FROM DUAL");
        resultSet.next();
        int result = resultSet.getInt(1);
 
        assertEquals(1, result);
 
    }
 
}

###Maven dependencies###

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.xxx.yyy</groupId>
    <artifactId>docker-testcontainers</artifactId>
    <version>0.0.1-SNAPSHOT</version>
 
    <properties>
        <java.version>1.8</java.version>
        <spring.version>5.1.3.RELEASE</spring.version>
        <testcontainers.version>1.10.2</testcontainers.version>
        <junit-engine.version>5.3.2</junit-engine.version>
        <junit-launcher.version>1.3.2</junit-launcher.version>
        <maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
    </properties>
 
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven-compiler-plugin.version}</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
 
    <dependencies>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>testcontainers</artifactId>
            <version>${testcontainers.version}</version>
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>oracle-xe</artifactId>
            <version>${testcontainers.version}</version>
        </dependency>
        <dependency>
            <groupId>com.oracle</groupId>
            <artifactId>ojdbc6</artifactId>
            <version>12.1.0.2</version>
        </dependency>
 
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit-engine.version}</version>
            <scope>test</scope>
        </dependency>
 
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-launcher</artifactId>
            <version>${junit-launcher.version}</version>
            <scope>test</scope>
        </dependency>
         
    </dependencies>

#Miscellaneous notes#

###Useful docker commands###

List images: docker images
List all containers: docker ps -a
Start a container: docker start [container id]
List started containers: docker ps
View logs of started container: docker logs [container id]

###References###

Installing Docker on Ubuntu

Further details about Post-install steps

Using an Oracle image within Docker

Database Testing With TestContainers

###About Oracle 12c image###

I've tried an Oracle 12c image (sath89/oracle-12c from: https://hub.docker.com/r/sath89/oracle-12c)

$ docker run -d -p 8080:8080 -p 1521:1521 --name oracle-db-12c sath89/oracle-12c

but it seems to be so slow starting up from testcontainers that the following exception is eventually (after approximately 4 minutes) thrown:

java.sql.SQLRecoverableException: ORA-01033: ORACLE initialization or shutdown in progress.

If the 12c image is started from docker host itself (i.e. Ubuntu), it does start successfully.

Opening the Docker socket to the network like this _will_ get your system compromised: anyone at all who can reach port 2376 of your system has unrestricted root access over it, with absolutely no authentication or logging. — David Maze, Apr 16 '19 at 10:06
Thanks @DavidMaze for your comment. Is the risk present even if the socket is open to the host OS and not directly to the external network? If so, could you point me to some resources on how to secure access? — Taoufik Mohdit, Apr 16 '19 at 10:13
I used some parts of this answer for installing postgres and redis docker instances in the ubuntu virtual machine for integration testing with testcontainers. I was trying to define DOCKER_HOST in `~\.testcontainers.properties` file and it was not working. As you said DOCKER_HOST needs to be defined in the environment variables. Additionally, @DavidMaze I don't think it will be problem because it's just opening virtual machine port to host OS, not to external network like @Taoufik Mohdit said. — omerhakanbilici, Apr 03 '20 at 12:30
If you'd allow passwordless root logins and unrestricted `sudo` on the VM "for convenience", it's no more or less secure than that. — David Maze, Apr 03 '20 at 12:50
@DavidMaze, this setup was part of an exploratory PoC, so security aspects were not on top of the list. I am still curious to know how exactly this setup makes the system vulnerable, although not exposed to external network directly. — Taoufik Mohdit, May 01 '20 at 13:18
Once I can access the Docker socket, I can `DOCKER_HOST=tcp://10.10.10.10:2376 docker run -v/:/host busybox sh` and edit arbitrary files like `/host/etc/shadow` and `/host/etc/sudoers`. — David Maze, May 01 '20 at 13:27

Using Docker and Testcontainers for database integration testing

1 Answers1