Docker swarm service - running state but no logs

Question

In an existing swarm, I created a service via a docker-compose yaml file using the 'docker stack' command.

When I check the service via 'docker service ls' command, the new service shows up on the list. it shows "0/1" in the REPLICAS column
When I check the service using the command below, it shows 'Running' as the Desired State

docker service ps --no-trunc (service id)
When I check if there is already a corresponding container for the service, I can see none
When I try to access the service via the browser, it seems to be not started.
What is difficult is I cannot see any logs to find the cause of why this is happening

docker service logs (service id)

I figured it may just be slow to start but I waited for about half an hour and it was still in that state. Not sure how can I find out the cause of this without any logs. Can anyone help me on this?

EDIT: Below is the result when I did a docker inspect of the service task

[
    {
        "ID": "wt2tdoz64j5wmci4gr3q3io2e",
        "Version": {
            "Index": 3407514
        },
        "CreatedAt": "2020-08-25T00:58:13.012900717Z",
        "UpdatedAt": "2020-08-25T00:58:13.012900717Z",
        "Labels": {},
        "Spec": {
            "ContainerSpec": {
                "Image": "my-ui-image:1.8.006",
                "Labels": {
                    "com.docker.stack.namespace": "myservice-stack"
                },
                "Env": [
                    "BACKEND_HOSTNAME=somewebsite.com",
                    "BACKEND_PORT=3421"
                ],
                "Privileges": {
                    "CredentialSpec": null,
                    "SELinuxContext": null
                },
                "Hosts": [
                    "10.152.30.18 somewebsite.com"
                ],
                "Isolation": "default"
            },
            "Resources": {},
            "Placement": {},
            "Networks": [
                {
                    "Target": "lt87emwtgbeztof5k2r1z2v27",
                    "Aliases": [
                        "myui_poc2"
                    ]
                }
            ],
            "ForceUpdate": 0
        },
        "ServiceID": "nbskoeofakkgxlgj3utgn45c5",
        "Slot": 1,
        "Status": {
            "Timestamp": "2020-08-25T00:58:13.012883476Z",
            "State": "new",
            "Message": "created",
            "PortStatus": {}
        },
        "DesiredState": "running"
    }
]

@Carlos actually its not composed only of one service but 10 services. its too big. all services in the same docker file encounters the same problem — lecarpetron dookmarion, Aug 24 '20 at 13:17
Do you have other services running without issues in the swarm? Are the services experiencing the issue using custom images or something "standard" from dockerhub? — Carlos, Aug 24 '20 at 13:28
Can you post the yml snippet for just one service? Without more details it will be hard to say if it's an environment issue or a service config issue or something else. — Carlos, Aug 24 '20 at 13:30

score 1 · Answer 1 · answered Aug 24 '20 at 19:28

1

If you store your images in private registry then you must be logged in by command docker login and deploy your services by docker stack deploy -c docker-compose.yml your_service --with-registry-auth.

answered Aug 24 '20 at 19:28

aniucum

151
1
4

the images are also stored in the server, not in a separate artifactory – lecarpetron dookmarion Aug 24 '20 at 23:10
1

It is problem. Master node does not share the local images with other nodes. You can create simple docker registry https://docs.docker.com/registry/ – aniucum Aug 24 '20 at 23:34
ok thanks for the info but this is just one node so far. just a master node – lecarpetron dookmarion Aug 25 '20 at 03:06
you can use option `--resolve-image` in your command `docker stack deploy -c docker-compose.yml myservice-stack --resolve-image never` – aniucum Aug 25 '20 at 03:57
Before deploying try to remove all services `docker service rm $(docker service ls -q) `. If it is does not help we will need to docker-compose.yml – aniucum Aug 25 '20 at 04:46
i think its seems to be a memory problem. when i waited, after 37 mins, I saw the replicas as "1/1" and the service was now usable. – lecarpetron dookmarion Aug 25 '20 at 06:58

score 1 · Answer 2 · answered Aug 24 '20 at 22:43

1

From the docker service ps ... output, you will see a column with the task id. You can get further details of the state of that task by inspecting the task id:

docker inspect $taskid

answered Aug 24 '20 at 22:43

BMitch

231,797
42
475
450

@lecarpetrondookmarion me neither, but then I don't have any way to see any of these commands running in your environment. Without seeing any output from the commands you are running, all we can do is guess. – BMitch Aug 24 '20 at 23:28
I added the result of the docker inspect of the task. See above – lecarpetron dookmarion Aug 25 '20 at 01:02

score 0 · Answer 3 · answered Aug 25 '20 at 03:46

0

My guess is that your app is not redirecting it's output to stdout and that's why you don't get any output when doing "docker service logs...".

I would start by looking at this: https://docs.docker.com/config/containers/logging/

How you redirect the apps output to stdout will depend on what language your app is developed in.

answered Aug 25 '20 at 03:46

Carlos

1,696
1
14
21

When you do "docker service ps (serviceid)", do you only get back one entry with a state of "Running"? or do you get additional entries where the state is "Shutdown"? – Carlos Aug 25 '20 at 11:53

Docker swarm service - running state but no logs

3 Answers3