Exception with connecting to database which runs on separate host in docker stack

Question

I have a trouble with networking in docker compose which runs in docker stack environment. Firt of all I have a project on .NET with dockerized Postgres database which runs in another container. I wrote this two images to docker-compose file (it's content you can see below). If I run this app on the one host everything is fine and app works correctly! But when I try to use docker-compose on multiplie hosts it raises this exception:

2022-12-22 22:44:24 Unhandled exception. System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000001, 11): Resource temporarily unavailable
2022-12-22 22:44:24    at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, ValueStopwatch stopwatch)
2022-12-22 22:44:24    at System.Net.Dns.GetHostAddresses(String hostNameOrAddress, AddressFamily family)
2022-12-22 22:44:24    at Npgsql.Internal.NpgsqlConnector.Connect(NpgsqlTimeout timeout)
2022-12-22 22:44:24    at Npgsql.Internal.NpgsqlConnector.RawOpen(SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken, Boolean isFirstAttempt)
2022-12-22 22:44:24    at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|203_1(NpgsqlConnector conn, SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken, Boolean isFirstAttempt)
2022-12-22 22:44:24    at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.UnpooledDataSource.Get(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|45_0(Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.NpgsqlConnection.Open()
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternal(Boolean errorsExpected)
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
2022-12-22 22:44:24    at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlDatabaseCreator.Exists(Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlDatabaseCreator.Exists(Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlDatabaseCreator.Exists()
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Storage.RelationalDatabaseCreator.EnsureCreated()
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade.EnsureCreated()
2022-12-22 22:44:24    at TimeManager.DAO.UserContext..ctor(IConfiguration configuration) in /src/TimeManager.DAO/UserContext.cs:line 30
2022-12-22 22:44:24    at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
2022-12-22 22:44:24    at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
2022-12-22 22:44:24    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
2022-12-22 22:44:24    at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
2022-12-22 22:44:24    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2022-12-22 22:44:24    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2022-12-22 22:44:24    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
2022-12-22 22:44:24    at Microsoft.AspNetCore.Builder.WebApplication.Run(String url)
2022-12-22 22:44:24    at Program.<Main>$(String[] args) in /src/TimeManager.API/Program.cs:line 51

I guess that this exception somehow based on networking troubles in my docker compose but I don't really understand how to deal with that problem. Swarm only throw container with error from one host to another. The original idea of using stack was to have the database on a separate host but this trouble bothers me. Thank you in advance!

My docker-compose.yml file:

version: '3.4'

services:
    timemanagerapi:
        image: ****/timemanagerapi:latest
        restart: always
        ports:
            - 80:80
        depends_on:
            - postgresserver
    postgresserver:
        image: postgres
        environment:
            - POSTGRES_PASSWORD=******
        volumes:
            - pgdata:/var/lib/postgresql/data     
        ports:
            - 5432:5432


volumes:
  pgdata:

I tried to add parameter deploy to my service in docker-compose.yml but it does not help. In internet I found that I need to add --network host but according to the official docker documentation this feature does not work with Docker Desktop on Windows which I use so I can't use this method

What host name are you using for the DB connection string? What do you mean by "use docker-compose on multiplie hosts"? — gunr2171, Dec 22 '22 at 20:35
I have a docker swarm with two hosts me and remote one from hosting service. And I used docker swarm init on remote one, connected to it using docker swarm join then created stack by command docker stack deploy with my docker compose file as parameter. I guess that it means to run docker-compose on multiple hosts. — Avusar, Dec 22 '22 at 20:40
"depends_on" is not a valid entry for a Swarm., so you'll also need some retry logic on your c# program. You can explicitly make a [network](https://docs.docker.com/compose/compose-file/compose-file-v3/#network-configuration-reference), but a [default](https://docs.docker.com/compose/networking/) is made for you, so your containers can communicate by service name. Is "postgresserver" the host name in your DB connection string? — gunr2171, Dec 22 '22 at 20:47
Sorry but now I can't paste my connection string here because my computer is turned off :) I can say only that instead of using direct IP address as hostname I used name of service with Postgres database in docker-compose.yml ("postgresserver") and everything worked fine on one host — Avusar, Dec 22 '22 at 20:55
According to your message I just need to remove "depends_on" block and swarm will work correct? — Avusar, Dec 22 '22 at 20:57
Swarm deployment will just ignore any invalid keys (depends_on), so removing it won't change anything. If you're using the service name as the host in the connection string (postgresserver) and both containers are in the same network, then it looks like your compose file is correct. Your error does say "Resource temporarily unavailable", so maybe add a ~30 second pause at the start of your program before trying to access the database. Remember that with swarm you can't control the order that containers boot up in. — gunr2171, Dec 22 '22 at 21:01
I noticed that when exception occurs swarm only restart main container with error not database so on second or third restart the database must be already be able to process requests but exceptions still raise. Are you sure that adding pause will solve my problem? — Avusar, Dec 22 '22 at 21:13
You are also going to need to label the node that hosts the database and use that label as a placement constraint, so it doesn't migrate to another node. The default swarm volume driver creates local volumes that are not shared between nodes. — Chris Becke, Dec 23 '22 at 13:36

score 1 · Accepted Answer · answered Dec 22 '22 at 21:25

You Compose file contains the depends_on entry. When you deploy using Docker Compose (docker-compose up -d), this controls the order in which containers are started. In your case, the database container will start first, and then your C# app container.

However, the depends_on entry is ignored when deploying through Docker Swarm (docker stack deploy ...).

When your C# app boots up, you have no way of knowing if the database is fully ready to accept requests or not, so you need to account for it.

Implement a retry mechanism for testing your connection to the database when your app is starting up. As an example, the Polly library can assist with that.
As a quick-and-dirty method, just sleep (or await Task.Delay) for a fixed amount of time to wait for the database to be ready. Try starting at 30 seconds. If this works, you can read the docker logs and adjust the time as needed.

Thank you very much! But can I answer another one question: how can I specify swarm that my c# app can't be deployed on manager and must be deployed only on connected workers — Avusar, Dec 22 '22 at 21:30
https://stackoverflow.com/questions/36609890/can-we-deploy-a-container-into-a-specific-node-in-a-docker-swarm — gunr2171, Dec 22 '22 at 21:31

Exception with connecting to database which runs on separate host in docker stack

1 Answers1