1

I am currently using graphite to monitor metrics from an API server. I use statsd/carbon to retrieve metrics via graphite (I use the docker from here / https://github.com/hopsoft/docker-graphite-statsd).

The thing is, I cannot keep the carbon data more than one or two days after the carbon data was created. It works perfectly well for statsd - I have monitored the response time and number of requests/errors on my endpoints more than a week ago and can still have access to it. And the very same day that I send this data, carbon data (cpu usage, memory used...) was also available, but now it is not visible any more. Is there any fix I can do?

Thank you very much!

-- smgr

Ram Ghadiyaram
  • 28,239
  • 13
  • 95
  • 121

2 Answers2

0

Did you check your storage-schemas.conf file? That configures the storage retention per metric. You will have to delete or resize existing wsp files after you change it.

Popieluch
  • 180
  • 1
  • 8
  • Ok, the thing is, I cannot find/access any config file whatsoever... When I try to locate them with apt-file, it gives me folder paths that actually do not exist on my computer, I am kinda confused here –  Aug 08 '17 at 09:55
  • according to the link you provided it should be in `/opt/graphite/conf/storage-schemas.conf` within the container. See: [link](https://github.com/hopsoft/docker-graphite-statsd/blob/master/README.md#change-the-configuration) – Popieluch Aug 08 '17 at 13:47
  • I know, but the file does actually not exist... There is no graphite folder in my opt folder. I do not have access to any of the config files, which is quite a bother to be honest –  Aug 08 '17 at 13:54
  • strange, just installed the docker container and found the config files with `docker inspect graphite` as described in the readme. – Popieluch Aug 08 '17 at 15:56
  • I did not read this part of the read me I must confess... Just did the command. It returns what I put in the answer to my owm post, but then, once more, the paths do not seem to actually exist :/ –  Aug 08 '17 at 18:04
  • Do you mean `/opt` as in literally the root on your system? Docker runs applications in containers which cannot access your hosts system. The files reside in separate volumes. Docker mounts the volumes on your filesystem, the destination is shown in the `source` field. Only in your config the source field is empty `state` is created and there is an `error`. It looks like this container never run successfully and did not create the volumes – Popieluch Aug 08 '17 at 19:29
  • Ho my, I was quite lost there, did not know about the source and so on... But then, just like you pointed out, there is no source... Should I try to restart the whole installation process? Seems like the right thing to do, but then, I do not understand what went wrong in the first place... Thanks for the insight anyway ;) –  Aug 08 '17 at 20:24
  • `"Error": "driver failed programming external connectivity on endpoint graphite (c30ef84fc31049bd1f711027996e4619ba3a46e6171901822eb73c951d531803): Error starting userland proxy: listen tcp 0.0.0.0:80: listen: address already in use",` So apparently you have another process listening on port 80, stop that first and restart the container (docker start graphite; docker stop graphite) and rerun the inspect command. I would suggest you read some about [docker](https://docs.docker.com/#introduction) or just not run Graphite in docker. – Popieluch Aug 08 '17 at 22:13
  • I fixed this issue, than you very much! However, the volumes are ow stored in a folder var/lib/docker that I cannot open ('Permission denied...') –  Aug 09 '17 at 18:56
0

This is what I get when I use the command docker inspect graphite... However, none of the paths mentioned actually exist (for instance, I do not have a graphite folder in /opt) ... :/

Did I miss a step in the installation/configuration or something? 

[{"Id": "5864b754f8a9671ad1ad2c48b992e33fea33551f3f7f6b3a9139a2a95dd57c56",

    "Created": "2017-07-26T10:11:29.868707602Z",
    "Path": "/sbin/my_init",
    "Args": [],
    "State": {
        "Status": "created",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 128,
        "Error": "driver failed programming external connectivity on endpoint graphite (c30ef84fc31049bd1f711027996e4619ba3a46e6171901822eb73c951d531803): Error starting userland proxy: listen tcp 0.0.0.0:80: listen: address already in use",
        "StartedAt": "0001-01-01T00:00:00Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "sha256:2cc9a7dfe93b5a3406091869b652df5fab9d0324b2a09d0be8421ea1f75826ca",
    "ResolvConfPath": "/var/lib/docker/containers/5864b754f8a9671ad1ad2c48b992e33fea33551f3f7f6b3a9139a2a95dd57c56/resolv.conf",
    "HostnamePath": "",
    "HostsPath": "/var/lib/docker/containers/5864b754f8a9671ad1ad2c48b992e33fea33551f3f7f6b3a9139a2a95dd57c56/hosts",
    "LogPath": "",
    "Name": "/graphite",
    "RestartCount": 0,
    "Driver": "aufs",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "NetworkMode": "default",
        "PortBindings": {
            "2003/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "2003"
                }
            ],
            "2004/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "2004"
                }
            ],
            "2023/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "2023"
                }
            ],
            "2024/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "2024"
                }
            ],
            "80/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "80"
                }
            ],
            "8125/udp": [
                {
                    "HostIp": "",
                    "HostPort": "8125"
                }
            ],
            "8126/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "8126"
                }
            ]
        },
        "RestartPolicy": {
            "Name": "always",
            "MaximumRetryCount": 0
        },
        "AutoRemove": false,
        "VolumeDriver": "",
        "VolumesFrom": null,
        "CapAdd": null,
        "CapDrop": null,
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": null,
        "GroupAdd": null,
        "IpcMode": "",
        "Cgroup": "",
        "Links": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyRootfs": false,
        "SecurityOpt": null,
        "UTSMode": "",
        "UsernsMode": "",
        "ShmSize": 67108864,
        "Runtime": "runc",
        "ConsoleSize": [
            0,
            0
        ],
        "Isolation": "",
        "CpuShares": 0,
        "Memory": 0,
        "NanoCpus": 0,
        "CgroupParent": "",
        "BlkioWeight": 0,
        "BlkioWeightDevice": null,
        "BlkioDeviceReadBps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteIOps": null,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "Devices": [],
        "DeviceCgroupRules": null,
        "DiskQuota": 0,
        "KernelMemory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": -1,
        "OomKillDisable": false,
        "PidsLimit": 0,
        "Ulimits": null,
        "CpuCount": 0,
        "CpuPercent": 0,
        "IOMaximumIOps": 0,
        "IOMaximumBandwidth": 0
    },
    "GraphDriver": {
        "Data": null,
        "Name": "aufs"
    },
    "Mounts": [
        {
            "Type": "volume",
            "Name": "7c5691b26b893f0e12784bba6ce1616cac9aa942eae0ea0f8de2f9c3c9df0313",
            "Source": "",
            "Destination": "/etc/logrotate.d",
            "Driver": "local",
            "Mode": "",
            "RW": true,
            "Propagation": ""
        },
        {
            "Type": "volume",
            "Name": "2abf2652bb382d78a558f9dbaeb730197312b633276c5b49ea30d9b1c892690b",
            "Source": "",
            "Destination": "/etc/nginx",
            "Driver": "local",
            "Mode": "",
            "RW": true,
            "Propagation": ""
        },
        {
            "Type": "volume",
            "Name": "66373f1180605928d4cb56a9b22fd22a0511f6be3ef1445de7fa7b8f2718794b",
            "Source": "",
            "Destination": "/opt/graphite/conf",
            "Driver": "local",
            "Mode": "",
            "RW": true,
            "Propagation": ""
        },
        {
            "Type": "volume",
            "Name": "30514c45cf7c315010bd2f0d59f8b40c9a8eb2e1d47d0e3dd02cac99cbf40963",
            "Source": "",
            "Destination": "/opt/graphite/storage",
            "Driver": "local",
            "Mode": "",
            "RW": true,
            "Propagation": ""
        },
        {
            "Type": "volume",
            "Name": "0df80d56c1316a79b919b2d170dbee547677f79b7e6de66dc6e7f62e9c423703",
            "Source": "",
            "Destination": "/opt/statsd",
            "Driver": "local",
            "Mode": "",
            "RW": true,
            "Propagation": ""
        },
        {
            "Type": "volume",
            "Name": "ea42f5555bc08b2cd3323a17e7fc957bc82088af504a186f6238a10baef1b118",
            "Source": "",
            "Destination": "/var/log",
            "Driver": "local",
            "Mode": "",
            "RW": true,
            "Propagation": ""
        }
    ],
    "Config": {
        "Hostname": "5864b754f8a9",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "2003/tcp": {},
            "2004/tcp": {},
            "2023/tcp": {},
            "2024/tcp": {},
            "80/tcp": {},
            "8125/udp": {},
            "8126/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "HOME=/root"
        ],
        "Cmd": [
            "/sbin/my_init"
        ],
        "ArgsEscaped": true,
        "Image": "hopsoft/graphite-statsd",
        "Volumes": {
            "/etc/logrotate.d": {},
            "/etc/nginx": {},
            "/opt/graphite/conf": {},
            "/opt/graphite/storage": {},
            "/opt/statsd": {},
            "/var/log": {}
        },
        "WorkingDir": "/",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": {}
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "7a2a711d4a05100058901316718d2165a59c99e19f4df18d3641fa9373a9c911",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {},
        "SandboxKey": "/var/run/docker/netns/7a2a711d4a05",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "bridge": {
                "IPAMConfig": null,
                "Links": null,
                "Aliases": null,
                "NetworkID": "4a497ec3a58479f07a839a23f9611eee37536a519ed3a90f603dc98558ef9f4a",
                "EndpointID": "",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "",
                "DriverOpts": null
            }
        }
    }
}