Running vespa in separate container for configserver and services

Question

I referenced the link https://docs.vespa.ai/documentation/docker-containers-in-production.html to control the config server and services in two different docker containers.

I have a single machine on which I am trying to run separate containers for configserver(one container) and services(one container)

But, I am unable to understand what is the use of services container, as I am still able to work with only config server container running.

Docker commands I am running:

docker run --hostname vespa-cfg --name vcfg --privileged -e VESPA_CONFIGSERVERS=vespa-cfg -p 19071:19071 -p 7070:8080 -d vespaengine/vespa:6.330.51 configserver

docker run --hostname vespa-ssrv --name vsrv --privileged -e VESPA_CONFIGSERVERS=vespa-cfg -p 7080:8080 -d vespaengine/vespa:6.330.51 services

I have added following in my hosts.xml

<hosts>
  <host name="vespa-cfg">
    <alias>server01</alias>
  </host>
  <host name="vespa-ssrv">
    <alias>service01</alias>
  </host>
</hosts>

And in services.xml

<services version="1.0">
    <admin version="1.0">
    <adminserver hostalias="server01"/>
    <configservers>
        <configserver hostalias="server01"/>
    </configservers>
    </admin>

    <container id="default-container" version="1.0">
        ...
        <nodes>
            <node hostalias="service01" />
        </nodes>
        ...
    </container>

    <content id="default-content" version="1.0">
        ...
        <nodes>
            <node hostalias="service01" distribution-key="0" />
        </nodes>
        ...
    </content>
</services>

I also tried persisting the data on both containers on separate volume points but all the data is still being persisted in the configserver volume point directory.

The query APIs are also working with configserver on port 7070, and not on services' 7080.

Please, help.

score 2 · Answer 1 · answered Jun 27 '19 at 20:05

In short, the configserver(s) configures the services nodes. It is the services nodes that processes and store data for serving. in large systems, you have 1 or 3 configservers, and 10's/100's of services nodes

Ref https://docs.vespa.ai/documentation/overview.html

I have seen elsewhere that some had problems with slobrok (service location broker) on the configserver nodes - that was corrected by running vespa-start-services in the configserver container (just log into the container and run the start command and then check on 7080) - https://docs.vespa.ai/documentation/reference/files-processes-and-ports.html

when the system is working correctly, you should get a response on http://localhost:7080 in your config above

vespa-logfmt should give some ideas of what the problem is

I also recommend you use latest, not 6.330.51 - Vespa 7 is the current major release

I ran vespa-start-services from within my **config server**. But, still wasn't able to get a response on localhost:7080. I checked the logs with vespa-logfmt on config server but there were no warnings or errors. Then when I tried to run vespa-start-services in my **services** container, it got stuck on "Waiting for config proxy to start". When I checked its logs, it had following warnings: **config-sentinel : FRT Connection tcp/localhost:19090 suspended until** ................. **configproxy : configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler Timed out (timeout 25000)** — Vikrant Thakur, Jun 28 '19 at 09:39

score 0 · Answer 2 · answered Aug 05 '19 at 12:17

To be able to use multiple containers like you want to, you need to use the internal Docker DNS so that Vespa can look up the names of all the containers and config servers. This can be done by creating a separate network and following conventions on hostnames and container names. Your example above will work if you start the containers this way:

docker network create --driver bridge vespa-net
docker run --network=vespa-net --hostname vespa-cfg.vespa-net --name vespa-cfg --privileged -e VESPA_CONFIGSERVERS=vespa-cfg.vespa-net -p 19071:19071 -p 7070:8080 -d vespaengine/vespa:6.330.51 configserver
docker run --network=vespa-net --hostname vespa-ssrv.vespa-net --name vespa-ssrv --privileged -e VESPA_CONFIGSERVERS=vespa-cfg.vespa-net -p 7080:8080 -d vespaengine/vespa:6.330.51 services

Remember to update your hosts.xml:

<hosts>
  <host name="vespa-cfg.vespa-net">
    <alias>server01</alias>
  </host>
  <host name="vespa-ssrv.vespa-net">
    <alias>service01</alias>
  </host>
</hosts>

I also recommend to use the most recent version of the Vespa Docker image from here

Running vespa in separate container for configserver and services

2 Answers2