Docker daemon got crashed after short span of time. Lately Docker services of stacks doesn't get properly up and resulted in app crash which only got fixed when i removed all the stacks and redeployed them.
I'm running my whole Android app and Other APIs on docker swarm cluster.
*I have my machine running on Google Cloud platform with around 75 CPUs and 250G memory which is more than enough for all the services I'm running on my machine. I have haproxy in frontend which does reverse proxy, backend as python flask api with 5 replicas, Database connectivity through pgbouncer. Else Logspout, datadog, portainer, redis, etc. *
I couldn't understand that even if i have enough resources, proper setup system with enough max_pids still the daemon crashed.
Generally Necessary: - cgroup hierarchy: properly mounted [/sys/fs/cgroup]
apparmor: enabled and tools installed
CONFIG_NAMESPACES: enabled
CONFIG_NET_NS: enabled
CONFIG_PID_NS: enabled
CONFIG_IPC_NS: enabled
CONFIG_UTS_NS: enabled
CONFIG_CGROUPS: enabled
CONFIG_CGROUP_CPUACCT: enabled
CONFIG_CGROUP_DEVICE: enabled
CONFIG_CGROUP_FREEZER: enabled
CONFIG_CGROUP_SCHED: enabled
CONFIG_CPUSETS: enabled
CONFIG_MEMCG: enabled
CONFIG_KEYS: enabled
CONFIG_VETH: enabled (as module)
CONFIG_BRIDGE: enabled (as module)
CONFIG_BRIDGE_NETFILTER: enabled (as module)
CONFIG_NF_NAT_IPV4: enabled (as module)
CONFIG_IP_NF_FILTER: enabled (as module)
CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
CONFIG_IP_NF_NAT: enabled (as module)
CONFIG_NF_NAT: enabled (as module)
CONFIG_NF_NAT_NEEDED: enabled
CONFIG_POSIX_MQUEUE: enabled
Optional Features:
CONFIG_USER_NS: enabled
CONFIG_SECCOMP: enabled
CONFIG_CGROUP_PIDS: enabled
CONFIG_MEMCG_SWAP: enabled
CONFIG_MEMCG_SWAP_ENABLED: missing
(cgroup swap accounting is currently enabled)
CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
CONFIG_BLK_CGROUP: enabled
CONFIG_BLK_DEV_THROTTLING: enabled
CONFIG_IOSCHED_CFQ: enabled
CONFIG_CFQ_GROUP_IOSCHED: enabled
CONFIG_CGROUP_PERF: enabled
CONFIG_CGROUP_HUGETLB: enabled
CONFIG_NET_CLS_CGROUP: enabled (as module)
CONFIG_CGROUP_NET_PRIO: enabled
CONFIG_CFS_BANDWIDTH: enabled
CONFIG_FAIR_GROUP_SCHED: enabled
CONFIG_RT_GROUP_SCHED: missing
CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
CONFIG_IP_VS: enabled (as module)
CONFIG_IP_VS_NFCT: enabled
CONFIG_IP_VS_PROTO_TCP: enabled
CONFIG_IP_VS_PROTO_UDP: enabled
CONFIG_IP_VS_RR: enabled (as module)
CONFIG_EXT4_FS: enabled
CONFIG_EXT4_FS_POSIX_ACL: enabled
CONFIG_EXT4_FS_SECURITY: enabled
Network Drivers:
"overlay":
CONFIG_VXLAN: enabled (as module)
CONFIG_BRIDGE_VLAN_FILTERING: enabled
Optional (for encrypted networks):
CONFIG_CRYPTO: enabled
CONFIG_CRYPTO_AEAD: enabled
CONFIG_CRYPTO_GCM: enabled
CONFIG_CRYPTO_SEQIV: enabled
CONFIG_CRYPTO_GHASH: enabled
CONFIG_XFRM: enabled
CONFIG_XFRM_USER: enabled (as module)
CONFIG_XFRM_ALGO: enabled (as module)
CONFIG_INET_ESP: enabled (as module)
CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
"ipvlan":
CONFIG_IPVLAN: enabled (as module)
"macvlan":
CONFIG_MACVLAN: enabled (as module)
CONFIG_DUMMY: enabled (as module)
"ftp,tftp client in container":
CONFIG_NF_NAT_FTP: enabled (as module)
CONFIG_NF_CONNTRACK_FTP: enabled (as module)
CONFIG_NF_NAT_TFTP: enabled (as module)
CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
Storage Drivers:
"aufs":
CONFIG_AUFS_FS: enabled (as module)
"btrfs":
CONFIG_BTRFS_FS: enabled (as module)
CONFIG_BTRFS_FS_POSIX_ACL: enabled
"devicemapper":
CONFIG_BLK_DEV_DM: enabled
CONFIG_DM_THIN_PROVISIONING: enabled (as module)
"overlay":
CONFIG_OVERLAY_FS: enabled (as module)
"zfs":
/dev/zfs: missing
zfs command: missing
zpool command: missing
Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000
NEED HELP ON UNDERSTANDING THE ISSUE HERE.
RESULT OF docker info here.
Containers: 170
Running: 167
Paused: 0
Stopped: 3
Images: 144
Server Version: 18.09.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: uqkfe247qfql50b1bft3r205b
Is Manager: true
ClusterID: fe79jmqus0l6zsa7kl41cbqa9
Managers: 1
Nodes: 2
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.160.0.30
Manager Addresses: 10.160.0.30:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version: N/A
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-1040-gcp
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 76
Total Memory: 246GiB
Name: rc-manager-instance
ID: 2PEM:4AF6:47RA:EMDM:CIMD:H4OC:5MNG:SXNI:ERFB:ML5G:O3YI:6VWA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
10.160.0.30:7000
127.0.0.0/8
Live Restore Enabled: false
