Flocker data migration

Question

I'm using flocker to persist and migrate docker containers data through ZfS Dataset backend.

docker run -v test:/data --volume-driver flocker busybox 
 sh -c "echo hello world > /data/file.txt"

I had this error for a week:

Unable to find image 'busybox:latest' locally latest: Pulling from library/busybox 583635769552: Pull complete b175bcb79023: Pull complete Digest: sha256:c1bc9b4bffe665bf014a305cc6cf3bca0e6effeb69d681d7a208ce741dad58e0 Status: Downloaded newer image for busybox:latest Error response from daemon: Cannot start container 128ddff1c0e9d6740c23b2f475b14206775a131878b4ed725a3280e22de79666: Timed out waiting for dataset to mount...

Any help would be appreciated.

flocker-docker-plugin.log

{
  "task_uuid": "4001196a-902c-4139-8b4f-e217490242ab",
  "error": true,
  "timestamp": 1459172727.570846,
  "message": "Unhandled Error\nTraceback (most recent call last):\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/dockerplugin/_script.py\", line 93, in docker_plugin_main\n    options=DockerPluginOptions()).main()\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/common/script.py\", line 294, in main\n    self._react(run_and_log, [], _reactor=self._reactor)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/twisted/internet/task.py\", line 882, in react\n    finished = main(_reactor, *argv)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/common/script.py\", line 282, in run_and_log\n    d = maybeDeferred(self.script.main, reactor, options)\n--- <exception caught here> ---\n  File \"/opt/flocker/local/lib/python2.7/site-packages/twisted/internet/defer.py\", line 150, in maybeDeferred\n    result = f(*args, **kw)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/dockerplugin/_script.py\", line 71, in main\n    certificates_path.child(b\"plugin.key\"))\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/apiclient/_client.py\", line 592, in __init__\n    cert_path, key_path)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/ca/_validation.py\", line 137, in treq_with_authentication\n    user_credential = UserCredential.from_files(user_cert_path, user_key_path)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/ca/_ca.py\", line 371, in from_files\n    certificate = load_certificate_file(certificate_path)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/ca/_ca.py\", line 232, in load_certificate_file\n    e.filename, code, failure\nflocker.ca._ca.PathError: Certificate file could not be opened. No such file or directory /etc/flocker/plugin.crt\n",
  "message_type": "twisted:log",
  "task_level": [
    1
  ]
}
{
  "task_uuid": "846b9f01-f618-4723-bf9e-2ce7ac6b79c9",
  "error": true,
  "timestamp": 1459172727.57451,
  "message": "main function encountered error\nTraceback (most recent call last):\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/dockerplugin/_script.py\", line 93, in docker_plugin_main\n    options=DockerPluginOptions()).main()\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/common/script.py\", line 294, in main\n    self._react(run_and_log, [], _reactor=self._reactor)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/twisted/internet/task.py\", line 882, in react\n    finished = main(_reactor, *argv)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/common/script.py\", line 282, in run_and_log\n    d = maybeDeferred(self.script.main, reactor, options)\n--- <exception caught here> ---\n  File \"/opt/flocker/local/lib/python2.7/site-packages/twisted/internet/defer.py\", line 150, in maybeDeferred\n    result = f(*args, **kw)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/dockerplugin/_script.py\", line 71, in main\n    certificates_path.child(b\"plugin.key\"))\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/apiclient/_client.py\", line 592, in __init__\n    cert_path, key_path)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/ca/_validation.py\", line 137, in treq_with_authentication\n    user_credential = UserCredential.from_files(user_cert_path, user_key_path)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/ca/_ca.py\", line 371, in from_files\n    certificate = load_certificate_file(certificate_path)\n  File \"/opt/flocker/local/lib/python2.7/site-packages/flocker/ca/_ca.py\", line 232, in load_certificate_file\n    e.filename, code, e file could nfailure\nflocker.ca._ca.PathError: Certificatot be opened. No such file or directory /etc/flocker/plugin.crt\n",
  "message_type": "twisted:log",
  "task_level": [
    1
  ]
}{
  "task_uuid": "7aab456b-a754-4160-8bcb-0b618a63ecda",
  "error": false,
  "timestamp": 1459172727.575473,
  "message": "Main loop terminated.",
  "message_type": "twisted:log",
  "task_level": [
    1
  ]
}{
  "task_uuid": "5b490165-8eb1-4755-9c87-1b49f00bc700",
  "error": false,
  "timestamp": 1459172728.921761,
  "message": "Log opened.",
  "message_type": "twisted:log",
  "task_level": [
    1
  ]
}

flocker-dataset-agent.log

{
  "task_uuid": "e0a6549f-7515-427f-abb6-b31379895bde",
  "cluster_state": {
    "node_uuid_to_era": {
      "values": [
        [
          {
            "hex": "e4b23086-3d3e-44c4-acc4-8b5d31c8fc9b",
            "$__class__$": "UUID"
          },
          {
            "hex": "c1ccd75f-14c4-4e33-a9a9-b6ea876c5a05",
            "$__class__$": "UUID"
          }
        ],
        [
          {
            "hex": "70f6f5dd-378c-4913-8b2f-0b0f1f55f0a8",
            "$__class__$": "UUID"
          },
          {
            "hex": "f824f593-8723-47f3-8605-a45b2262b268",
            "$__class__$": "UUID"
          }
        ]
      ],
      "$__class__$": "PMap"
    },
    "nodes": [
      {
        "paths": {
          "values": [
            [
              "7645c292-5329-4aa3-b606-981c2f4e4892",
              {
                "path": "/flocker/55ab515f-43eb-488b-b3a3-e65fa5c62249.default.7645c292-5329-4aa3-b606-981c2f4e4892",
                "$__class__$": "FilePath"
              }
            ]
          ],
          "$__class__$": "PMap"
        },
        "uuid": {
          "hex": "e4b23086-3d3e-44c4-acc4-8b5d31c8fc9b",
          "$__class__$": "UUID"
        },
        "$__class__$": "NodeState",
        "hostname": "192.168.224.7",
        "devices": {
          "values": [

          ],
          "$__class__$": "PMap"
        },
        "applications": [

        ],
        "manifestations": {
          "values": [
            [
              "7645c292-5329-4aa3-b606-981c2f4e4892",
              {
                "dataset": {
                  "deleted": false,
                  "dataset_id": "7645c292-5329-4aa3-b606-981c2f4e4892",
                  "metadata": {
                    "values": [

                    ],
                    "$__class__$": "PMap"
                  },
                  "maximum_size": 80530636800,
                  "$__class__$": "Dataset"
                },
                "primary": true,
                "$__class__$": "Manifestation"
              }
            ]
          ],
          "$__class__$": "PMap"
        }
      },
      {
        "paths": {
          "values": [

          ],
          "$__class__$": "PMap"
        },
        "uuid": {
          "hex": "70f6f5dd-378c-4913-8b2f-0b0f1f55f0a8",
          "$__class__$": "UUID"
        },
        "$__class__$": "NodeState",
        "hostname": "192.168.224.89",
        "devices": {
          "values": [

          ],
          "$__class__$": "PMap"
        },
        "applications": [

        ],
        "manifestations": {
          "values": [

          ],
          "$__class__$": "PMap"
        }
      }
    ],
    "nonmanifest_datasets": {
      "values": [

      ],
      "$__class__$": "PMap"
    },
    "$__class__$": "DeploymentState"
  },
  "action_type": "flocker:agent:converge",
  "desired_configuration": {
    "persistent_state": {
      "blockdevice_ownership": {
        "values": [

        ],
        "$__class__$": "PMap"
      },
      "$__class__$": "PersistentState"
    },
    "nodes": [
      {
        "applications": [

        ],
        "manifestations": {
          "values": [
            [
              "7645c292-5329-4aa3-b606-981c2f4e4892",
              {
                "dataset": {
                  "deleted": false,
                  "dataset_id": "7645c292-5329-4aa3-b606-981c2f4e4892",
                  "metadata": {
                    "values": [
                      [
                        "name",
                        "apples"
                      ]
                    ],
                    "$__class__$": "PMap"
                  },
                  "maximum_size": 80530636800,
                  "$__class__$": "Dataset"
                },
                "primary": true,
                "$__class__$": "Manifestation"
              }
            ]
          ],
          "$__class__$": "PMap"
        },
        "uuid": {
          "hex": "70f6f5dd-378c-4913-8b2f-0b0f1f55f0a8",
          "$__class__$": "UUID"
        },
        "$__class__$": "Node"
      },
      {
        "applications": [

        ],
        "manifestations": {
          "values": [

          ],
          "$__class__$": "PMap"
        },
        "uuid": {
          "hex": "e4b23086-3d3e-44c4-acc4-8b5d31c8fc9b",
          "$__class__$": "UUID"
        },
        "$__class__$": "Node"
      }
    ],
    "leases": {
      "values": [

      ],
      "$__class__$": "PMap"
    },
    "$__class__$": "Deployment"
  },
  "timestamp": 1459176487.344602,
  "action_status": "started",
  "task_level": [
    2,
    1
  ]
}{
  "timestamp": 1459176487.345864,
  "task_uuid": "e0a6549f-7515-427f-abb6-b31379895bde",
  "action_type": "flocker:agent:discovery",
  "action_status": "started",
  "task_level": [
    2,
    2,
    1
  ]
}{
  "fsm_next_state": "<ConvergenceLoopStates=CONVERGING>",
  "task_level": [
    3
  ],
  "action_type": "fsm:transition",
  "timestamp": 1459176487.348982,
  "fsm_output": [
    "<ConvergenceLoopOutputs=CLEAR_WAKEUP>",
    "<ConvergenceLoopOutputs=CONVERGE>"
  ],
  "task_uuid": "e0a6549f-7515-427f-abb6-b31379895bde",
  "action_status": "succeeded"
}{
  "task_uuid": "e0a6549f-7515-427f-abb6-b31379895bde",
  "task_level": [
    2,
    2,
    2
  ],
  "action_type": "flocker:agent:discovery",
  "timestamp": 1459176487.354977,
  "state": "NodeLocalState(node_state=NodeState(applications=None, paths=UnicodeFilepathPMap({}), manifestations=UnicodeManifestationPMap({}), hostname=u'192.168.224.89', uuid=UUID('70f6f5dd-378c-4913-8b2f-0b0f1f55f0a8'), devices=UuidFilepathPMap({})))",
  "action_status": "succeeded"
}{
  "timestamp": 1459176487.356563,
  "task_uuid": "e0a6549f-7515-427f-abb6-b31379895bde",
  "message_type": "flocker:agent:converge:actions",
  "task_level": [
    2,
    3
  ],
  "calculated_actions": "NoOp(sleep=datetime.timedelta(0, 1))"
}{
  "timestamp": 1459176487.356925,
  "task_uuid": "e0a6549f-7515-427f-abb6-b31379895bde",
  "action_type": "flocker:change:noop",
  "action_status": "started",
  "task_level": [
    2,
    4,
    1
  ]
}{
  "timestamp": 1459176487.357233,
  "task_uuid": "e0a6549f-7515-427f-abb6-b31379895bde",
  "action_type": "flocker:change:noop",
  "action_status": "succeeded",
  "task_level": [
    2,
    4,
    2
  ]
}

Hey DIStephane, How you got these log files? I am unable to get these logs. — Yogesh Jilhawar, Apr 29 '16 at 09:20
Ok in my case i used ubuntu. So i think this is what you need https://docs.clusterhq.com/en/1.2.0/using/administering/debugging.html — DiStephane, Apr 29 '16 at 13:45

score 2 · Answer 1 · answered Mar 29 '16 at 13:19

2

The logs contain the following error:

_ca.PathError: Certificate file could not be opened. No such file or directory /etc/flocker/plugin.crt

This means you are missing the necessary certificates in order for the docker plugin to communicate correctly.

Please see the below link to create the necessary certificates.

https://docs.clusterhq.com/en/latest/docker-integration/generate-api-plugin.html

answered Mar 29 '16 at 13:19

Ryan Wallner

31
3

Thanks Ryan for your fast and detailed answer. I solved the problem of plugin.crt. But the process of migration still not working. I got this error : – DiStephane Mar 31 '16 at 10:17

score 1 · Answer 2 · answered Mar 31 '16 at 21:17

1

The problem is still there. The full discussion with Ryan can be found in this link: https://groups.google.com/forum/#!topic/flocker-users/l5wCW-U1zKs

answered Mar 31 '16 at 21:17

DiStephane

105
1
7

Flocker data migration

2 Answers2