google/cadvisor:latest image monitor container run error

Question

I found the bug below on the above "devicemapper" at bugzilla.redhat.com. It let me install the "device-mapper-persistent-data" package. However, when I installed it, I received the following error.

I tried to use this link to solve the error, however, I still haven't managed to solve it.

This is what I tried:

1. update the libcontainer dependency
2. look at all cgroup mounts

Here is the error:

[root@slave712 ~]# docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest
Unable to find image 'google/cadvisor:latest' locally
latest: Pulling from google/cadvisor
e110a4a17941: Pull complete 
e17fa94aae07: Pull complete 
7097ce2382c0: Pull complete 
Digest:     sha256:3b1f3714a27d40bf413d5fde0429ad6410bc14fcdae5a3478daa34cc701901fa
Status: Downloaded newer image for google/cadvisor:latest
a54c09247fd6572f96365dced4178f0cec8696329ed39c0349991f39ab6d660d

[root@slave712 ~]# docker logs cadvisor

I1001 02:41:16.237284 1 storagedriver.go:50] Caching stats in memory for 2m0s
I1001 02:41:16.238324 1 manager.go:140] cAdvisor running in container: "/docker/a54c09247fd6572f96365dced4178f0cec8696329ed39c0349991f39ab6d660d"
W1001 02:41:16.379106 1 manager.go:148] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1001 02:41:16.428790 1 fs.go:116] Filesystem partitions: map[/dev/mapper/centos-root:{mountpoint:/var/lib/docker/devicemapper major:253 minor:0 fsType:xfs blockSize:0} /dev/sda1:{mountpoint:/rootfs/boot major:8 minor:1 fsType:xfs blockSize:0} /dev/mapper/docker-253:0-402876333-34a64b9159b501f40b949fc1ae894ee7fc1985645d65a63f4dfe689c32919e43:{mountpoint:/ major:253 minor:3 fsType:xfs blockSize:0}]
I1001 02:41:16.655301 1 manager.go:195] Machine: {NumCores:4 CpuFrequency:3192628 MemoryCapacity:1025396736 MachineID:f40694ca9e71417fb1af85dcf66dfc39 SystemUUID:564DBE01-1C91-8E79-5C04-6FC1D390B5EB BootID:9008dc53-91fe-4510-b199-8c26241bc0b5 Filesystems:[{Device:/dev/mapper/docker-253:0-402876333-34a64b9159b501f40b949fc1ae894ee7fc1985645d65a63f4dfe689c32919e43 Capacity:10725883904 Type:vfs Inodes:10484736 HasInodes:true} {Device:/dev/mapper/centos-root Capacity:105011822592 Type:vfs Inodes:102600704 HasInodes:true} {Device:/dev/sda1 Capacity:206221312 Type:vfs Inodes:204800 HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size:105063120896 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:2097152000 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:107374182400 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:10737418240 Scheduler:none} 2:0:{Name:fd0 Major:2 Minor:0 Size:4096 Scheduler:deadline} 8:0:{Name:sda Major:8 Minor:0 Size:107374182400 Scheduler:deadline}] NetworkDevices:[{Name:br-92607f6f2739 MacAddress:02:42:9a:50:1f:2b Speed:0 Mtu:1500} {Name:eno16777736 MacAddress:00:0c:29:90:b5:eb Speed:1000 Mtu:1500}] Topology:[{Id:0 Memory:1073205248 Cores:[{Id:0 Threads:[0] Caches:[]} {Id:1 Threads:[1] Caches:[]}] Caches:[{Size:6291456 Type:Unified Level:3}]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[2] Caches:[]} {Id:1 Threads:[3] Caches:[]}] Caches:[{Size:6291456 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I1001 02:41:16.659951 1 manager.go:201] Version: {KernelVersion:3.10.0-327.el7.x86_64 ContainerOsVersion:Alpine Linux v3.4 DockerVersion:1.12.1 CadvisorVersion:v0.24.0 CadvisorRevision:0cdf491}
E1001 02:41:16.698803 1 factory.go:291] devicemapper filesystem stats will not be reported: unable to find thin_ls binary
I1001 02:41:16.698900 1 factory.go:295] Registering Docker factory
W1001 02:41:16.698962 1 manager.go:244] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I1001 02:41:16.698982 1 factory.go:54] Registering systemd factory
I1001 02:41:16.699503 1 factory.go:86] Registering Raw factory
I1001 02:41:16.700006 1 manager.go:1082] Started watching for new ooms in manager
W1001 02:41:16.715521 1 manager.go:272] Could not configure a source for OOM detection, disabling OOM events: unable to find any kernel log file available from our set: [/var/log/kern.log /var/log/messages /var/log/syslog]
I1001 02:41:16.730767 1 manager.go:285] Starting recovery of all containers
I1001 02:41:19.183769 1 manager.go:290] Recovery completed
F1001 02:41:19.254307 1 cadvisor.go:151] Failed to start container manager: inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

[root@slave712 ~]# docker version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:

OS/Arch: linux/amd64

Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built:

OS/Arch: linux/amd64

[root@slave712 ~]# mount -l
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=490460k,nr_inodes=122615,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/centos-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
/dev/sda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=100140k,mode=700)
/dev/mapper/centos-root on /var/lib/docker/devicemapper type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

[root@slave712 ~]# uname -a
Linux slave712 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@slave712 ~]# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core)

Juless-Z · Answer 1 · 2021-04-03T17:28:06.910

for me, it worked with the next docker-compose setup. It's true I have not tested it completely. I will update the post as soon as possible. But if it helps up to this moment to someone, perfect.

  cadvisor:
  image: google/cadvisor:latest
  container_name: monitoring_cadvisor
  restart: unless-stopped
  volumes:
    - /:/rootfs:ro
    - /var/run/docker.sock:/var/run/docker.sock:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
  expose:
    - 8080
  ports:
    - 8084:8080

I updated the docker.sock path. cadvisor wasn't able to find the containers without the direct link.

Cheers

Tim Allclair · Answer 2 · 2016-10-13T22:13:04.957

0

It looks like you're hitting the issue fixed in the link you mentioned. That fix has not yet been incorporated into a cAdvisor release, so it is not included when you run the docker image google/cadvisor:latest (assuming you didn't build your own image and tag it with that).

You can try building and running your own docker image from master, or wait for the next patch release (should be soon), which will include that fix.

EDIT:

The fix has now been incorporated into cAdvisor v0.24.1 (docker image google/cadvisor:v0.24.1)

edited Oct 13 '16 at 22:13

answered Oct 06 '16 at 19:52

Tim Allclair

7,347
2
27
25

Well, it makes people very rogue – Jason Tom Oct 07 '16 at 06:24
I am using Ubuntu OS with docker version "17.05.0-ce", cadvisor image version "google/cadvisor:v0.24.1" and influx docker image "tutum/influxdb :0.8.8". And I am getting error on running cadvisor: "W0913 17:19:43.993296 1 manager.go:148] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused ". I am able to see cadvisor in browser and influxdb is also up. But I am not able to see stats from cadvisor in influxdb. I followed https://www.brianchristner.io/how-to-setup-docker-monitoring/ – Raji Sep 13 '17 at 17:26
@Raji you can ignore that rkt error (unless you're using rkt). Also, I recommend updating to a more recent cAdvisor release. – Tim Allclair Sep 13 '17 at 18:11
Thanks Tim for the response. When I open the influxdb in browser (http://hostmachine:8083), I don't see any data from cadvisor. I checked 'stats' option and 'measurements' option. 'stats' shows me some data about '_internal' database though I selected 'cadvvisor' database. and measurements is empty. Am I looking wrong? – Raji Sep 14 '17 at 08:06

google/cadvisor:latest image monitor container run error

2 Answers2