0

I have been trying to figure this out since 2 days without success. Let me know if you have faced similar issue and how to fix it. Or any pointers will be of help.

Here is what we do.

  1. Create virtual instance.
  2. Create blank volume.
  3. Create volume from snapshot.
  4. Attach blank volume to instance.
  5. Attach volume from snapshot to instance.
  6. Other IP address, security group stuff.

All these are done in separate threads but with dependencies added. Ex. 4 depends on 2 (4->2). 2->1. 5->3->1.

The problem is that sometimes:

  1. One of the devices don’t appear in the device list. Openstack shows both attached.
  2. Both device don’t appear in the device list. Openstack shows both attached.
  3. Both appear but, disks at specified devices get switched! Ex. We attached 1GB disk on /dev/vdb and 10GB disk on /dev/vdc. But when we do fdisk –l /dev/vdc we get 1G and /dev/vdb shows 10G.
  4. Works!!!

I have checked the code and all components are doing what they are supposed to do. That is leading me to suspect openstack itself! (I know… that’s kind of strange.)

I am using allinone with icehouse and RDO packstack setup.

[Update 1]

I am using JClouds APIs to attach the volumes. Also, I am specifying the device names (/dev/xvdb, /dev/xvdc). Yet, I am checking for device names in the response after the call and using them instead.

NovaAPI nova = ContextBuilder
            .newBuilder("openstack-nova")
            .endpoint(endpoint)
            .credentials(getIdentity(), getPassword())
            .modules(modules).buildApi(NovaApi.class);
VolumeAttachmentApi attachment = nova.getVolumeAttachmentExtensionForZone(CONSTANT_REGION).get();
VolumeAttachment attachment.attachVolumeToServerAsDevice(volumeIdExisting, instanceId, deviceDevXVDB);

[Update 2] I created a sample program to isolate the issue. This sample program does the following:

    /-> 2 -> 3 >\
1 -> -> 6 -> -> x
    \-> 4 -> 5 >/

Basically the main thread creates the instance and launches two threads. One thread creates blank volume (10GB) and attaches it. While the other creates a volume from snapshot (1GB) and attaches it. The main thread then creates IP and attaches it and waits for the two threads to complete.

The observation is rather strange. Every single time, the attach volume response says the blank volume got attached to /dev/vdc, and the volume from snapshot attached to /dev/vdb.

But when I checked on the instance

fdisk -l /dev/vdb

returns 10GB and

fdisk -l /dev/vdc

returns 1GB

What could I be missing?? Any pointers/suggestions would be helpful.

jww
  • 97,681
  • 90
  • 411
  • 885
Adam A R
  • 29
  • 5
  • Can you share the commands your using to attach the volumes? Are you letting Openstack assign devices or are you specifying? – Dave Aug 27 '14 at 13:21
  • Hi Dave, updated answers to your questions in the original post – Adam A R Aug 28 '14 at 10:55
  • From what you have said, I can't spot any obvious issue. I'm not overly familiar with jclouds, but my only suggestion would be to try it with the nova client directly to see if the issue is in OS, or the client library your using. I tried the same thing on my deployment (icehouse installed with RDO packstack) and couldn't replicate the issue. – Dave Aug 28 '14 at 20:21
  • Dave, when i try use the same program to do the create+attach in sequence it works without any issue. And yes, if I do it from either Horizon or from command-line, everything works as expected. – Adam A R Aug 29 '14 at 16:21

1 Answers1

1

Just for the record. Attaching volumes simultaneously will not work. The device names assigned to attached volume is dependent on which gets attached first. Even though we may specify the name, it may not be honored by the underlying driver.

Finally, we did away with the parallel attachment of volumes to sequential and now everything works as expected.

Adam A R
  • 29
  • 5