Linux multipath: How to configure a single multibus path group

Question

I have just upgraded a functional Ubuntu 16.04 host to 18.04 and am now having issues with multipath.

Package versions: * multipath-tools 0.7.4-2ubuntu3 * open-iscsi 2.0.874-5ubuntu2.7

I have a Dell PowerVault MD3860i with four paths to the host. Before the upgrade, multipath -ll looked like this:

backupeng (3600a098000b5efae00000e9a5b9b58f5) dm-2 DELL,MD38xxi
size=8.0T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:1 sdb 8:16 active ready running
  |- 4:0:0:1 sdc 8:32 active ready running
  |- 5:0:0:1 sdd 8:48 active ready running
  `- 6:0:0:1 sde 8:64 active ready running

Now it looks like this:

backupeng (3600a098000b5efae00000e9a5b9b58f5) dm-2 DELL,MD38xxi
size=8.0T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=14 status=active
| |- 5:0:0:1 sdd 8:48 active ready running
| `- 6:0:0:1 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=9 status=enabled
  |- 3:0:0:1 sdb 8:16 active ready running
  `- 4:0:0:1 sdc 8:32 active ready running

My /etc/multipath.conf looks like this:

defaults {
    user_friendly_names yes
    path_selector "round-robin 0"
    path_grouping_policy multibus
}

multipaths {
    multipath {
        wwid 3600a098000b5efae00000e9a5b9b58f5
        alias backupeng
    }
}

For performance reasons, I need to have all paths in the same path group, like they were before. My understanding is that path_grouping_policy multibus is supposed to do this. I have tried restarting multipathd, setting up the iscsi and multipath configs on the host from scratch, and so on for the past few hours.

I can paste the full output of multipathd -k -> show config but what I'm seeing in there agrees with my multipath.conf file. Is there any other information I can provide?

Did you try to add `path_grouping_policy multibus` to multipath scope? — sparse, Aug 31 '20 at 05:39

score 5 · Accepted Answer · answered May 17 '19 at 05:46

Have you actually measured the performance and found it degraded? You may actually find it's improved, although the difference may not be very large.

The Dell PowerVault MD3860i seems to be an active/passive (also known as asymmetric, or ALUA) style storage system: it has two storage controllers, and each of the disks are considered "owned by" one of the controllers at any given time. If a disk is being accessed through a controller that doesn't currently "own" the disk, it triggers an ownership switch which adds some extra latency to the disk I/O operation.

As a result, it is preferable to use only the paths that use the controller that currently owns the disk being accessed.

Your new multipath -ll output indicates that dm-multipath is receiving ALUA information from the storage system using the rdac protocol. That is, the storage controller is telling dm-multipath which paths it should currently use. The rdac protocol is an older protocol used by LSI/Engenio/NetApp storage controllers and their OEMs, and it is well understood. There are other similar vendor-specific protocols, and the SCSI-3 ALUA would be the new standard way of conveying this information, although not all storage systems use it yet.

You are setting the default path_grouping_policy to multibus, but it is very likely being overridden by hardware-specific defaults compiled into multipath-tools. (Specific beats general: any settings in applicable device or multipath sections will override the values set in the defaults section.)

These built-in defaults are implemented in cooperation with storage device manufacturers; apparently the multipath-tools version in Ubuntu 16.04 did not yet have specific defaults for Dell PowerVault MD3860i, but in Ubuntu 18.04 it has.

You can view these built-in defaults with sudo multipath -t. For your storage system, the relevant group of settings will probably look like this:

    device {
            vendor "DELL"
            product "(MD34xx|MD38xx)"
            product_blacklist "Universal Xport"
            path_grouping_policy "group_by_prio"
            path_checker "rdac"
            features "2 pg_init_retries 50"
            hardware_handler "1 rdac"
            prio "rdac"
            failback "immediate"
            no_path_retry 30
    }

The multibus path_grouping_policy is intended for true active/active storage systems, which allow you to use any and all paths with no restrictions. These tend to be larger, higher-tier storage products.

You can write your own device { ... } or overrides { ... } block in multipath.conf to override these settings, but you should do so in production only if you have specific information from the storage vendor to do so, or if you have the test results to prove you actually know better than the vendor and the dm-multipath developers.

So `multipath` is doing what it should and I am, in fact, going crazy. Or at least I have some more learning to do where storage is concerned. Everything you said makes sense and I've accepted your answer. Thanks! — Eil, May 17 '19 at 17:21
upvote for one of the rare, rare cases where "are you sure you are having a problem and not a better situation" turned out to be true. — Florian Heigl, Jun 23 '20 at 22:43

Linux multipath: How to configure a single multibus path group

1 Answers1