2

I have a mariadb running in a container. On 'docker run', an import script (from a db dump) is run by mariadb, which creates users, builds schema, etc.

As the size of that dump script grows, the time to do the import increases. At this point it's about 8-10 seconds, but i expect amount of data to increase substantially, and the import time will be more difficult to predict.

I'd like to be able to send a signal from the container to the host, to let it know that the data has been loaded, and that db is ready to be used. So far i have found info on how to send signal from one container to another container, but there's no information on how to send signal from container to the host. Also, i need to be able to do this programmatically, as creating container is part part of a larger pipeline.

Ideally, i'd like to be able to do something like this:

client = docker.from_env()
db_c = client.containers.run('my_db_image', ....)

# check for signal from db_c container
# do other things

Thank you!

Shadow
  • 33,525
  • 10
  • 51
  • 64
Alex
  • 111
  • 7
  • The standard approach for this is to poll the database container until it starts accepting inbound TCP connections. This is typically in the container startup sequence, so the containers don't need outside help to wait for the database to be ready. – David Maze Oct 30 '19 at 01:38
  • @DavidMaze Thanks for the reply. The db is IN the container. Container is started by a python script, outside of the container. This script needs to know when the db import (inside the container) is complete, so it can proceed. – Alex Oct 30 '19 at 04:44

2 Answers2

2

AFAIK you cannot send signals from the container to a process running on the host but there are other ways to know when the import has finished. I think the easiest is to start the container in detached mode and wait until a specific line gets logged. The following script for example waits until the line done is logged:

import os
import docker

client = docker.from_env()

container = client.containers.run('ubuntu:latest', 'bash -c "for i in {1..10}; do sleep 1; echo working; done; echo done"', detach=True)
print('container started')

for line in container.logs(stream=True):
    print line.strip()
    if line.strip() == 'done':
        break

print('continue....')

If the output of the import script goes to stdout it could contain a simple print at the end:

select 'The import has finished' AS '';

Wait for this string in the python script.

Another approach is to use some other form of inter-process communication. An example using named pipes:

import os
import docker
import errno

client = docker.from_env()

FIFO = '/tmp/apipe'

# create the pipe
try:
    os.mkfifo(FIFO)
except OSError as oe: 
    if oe.errno != errno.EEXIST:
        raise
# start the container sharing the pipe
container = client.containers.run('ubuntu:latest', 'bash -c "sleep 5; echo done > /tmp/apipe"', volumes={FIFO: {'bind': FIFO, 'mode': 'rw'}}, detach=True)
print("container started")

with open(FIFO) as fifo:
    print("FIFO opened")
    while True:
        data = fifo.read()
        if len(data) == 0:
            print("Writer closed")
            break
        print('Read: "{0}"'.format(data))

print("continue...")

The host shares the named pipe with the container. In the python script the read call to the FIFO is blocked until some data is available in the pipe. In the container the import script writes to the pipe notifying the program that the data has been loaded. The mysql system command, \! command to execute an external command might come in handy in this situation. You could simply add to the end of the script:

\! echo done > /tmp/apipe

In a similar way you could use IPC sockets (aka Unix sockets) or shared memory but things get a bit more complicated.

Yet another solution is to add a health-check to the container. The health status can be polled on the host by inspecting the container. See How to wait until docker start is finished?

Edited: The above approaches assume the container is initialized and accepting connections. If the script is executed as part of the initialization process (Initializing a fresh instance), which seems to be the case here, the database is not ready and accepting connections when the import completes. For the initialization the server is temporarily started with the --skip_networking (allowing only local clients) and only after the initialization completes it is restarted and becomes available remotely.

b0gusb
  • 4,283
  • 2
  • 14
  • 33
  • 1
    I haven't thought of named pipes, that's pretty clever. I ended up just trying to connect and sleeping, in a loop. – Alex Oct 31 '19 at 05:56
  • 1
    I realized that in your case the best way to create the database (you're probably already doing that) is to use the MariaDB initialization mechanism [Initializing a fresh instance](https://hub.docker.com/_/mariadb). During the initialization the server does not accept incoming connections therefore the named pipes approach won't help you much. The server is re-started with TCP connectivity after the initialization, so polling the database as in @LinPy's answer is the right way to go. – b0gusb Nov 01 '19 at 08:35
1

you can add this code to check if the db is ready to accept the connections:

import MySQLdb
import time


db = MySQLdb.connect(host='MYHost', user='MYNAME', passwd='PASS', db='MYDB')

if not db:
    while True:
        db = MySQLdb.connect(host='MYHost', user='MYNAME', passwd='PASS', db='MYDB')
        if db:
            break
        print("Still waiting for the DB")
        time.sleep(10)
LinPy
  • 16,987
  • 4
  • 43
  • 57