3

I am writing a block device driver for linux. It is crucial to support unsafe removal (like usb unplug). In other words, I want to be able to shut down the block device without creating memory leaks / crashes even while applications hold open files or performing IO on my device or if it is mounted with file system. Surely unsafe removal would possibly corrupt the data which is stored on the device, but that is something the customers are willing to accept.

Here is the basics steps I have done:

  1. Upon unsafe removal, block device spawns a zombie which will automatically fail all new IO requests, ioctls, etc. The zombie substitutes make_request function and changes other function pointers so kernel would not need the original block device.
  2. Block device waits for all IO which is running now (and use my internal resources) to complete
  3. It does del_gendisk(); however this does not really free's kernel resources because they are still used.
  4. Block device frees itself.
  5. The zombie keeps track of the amount of opens() and close() on the block device and when last close() occurs it automatically free() itself
    1. Result - I am not leaking the blockdevice, request queue, gen disk, etc.

However this is a very difficult mechanism which requires a lot of code and is extremely prone to race conditions. I am still struggling with corner cases, per_cpu counting of io's and occasional crashes

My questions: Is there a mechanism in the kernel which already does that? I searched manuals, literature, and countless source code examples of block device drivers, ram disks and USB drivers but could not find a solution. I am sure, that I am not the first one to encounter this problem.

Edited: I learned from the answer below, by Dave S about the hot-plug mechanism but it does not help me. I need a solution of how to safely shut down the driver and not how to notify the kernel that driver was shut down.

Example of one problem: blk_queue_make_request() registers a function through which my block devices serves IO. In that function I increment per_cpu counters to know how many IO's are in flight by each cpu. However there is a race condition of function being called but counter was not increased yet, so my device thinks there are 0 IO's, releases the resources and then IO comes and crashes the system. Hotplug will not assist me with this problem as far as I understand

DanielHsH
  • 4,287
  • 3
  • 30
  • 36
  • You mean something like hotplugging? – moooeeeep Feb 27 '17 at 08:04
  • Thanks, I am not sure hotplug is relevant, see my comment on the answer below, and also added clarification to my question – DanielHsH Feb 28 '17 at 07:14
  • I think hotplugging or hot removal is what you have in mind. This is probably very specific to the system architecture and the kind of a device you have at hand (usb, pci, something else?). I think you should narrow down your question's scope by adding the relevant details. Can you provide your current state "Hello World" device driver implementation? – moooeeeep Feb 28 '17 at 08:45

2 Answers2

1

About a decade ago I used hotplugging on a software driver project to safely add/remove an external USB disk drive which interfaced to an embedded Linux driven Set-top Box.

For your project you will also need to write a hot plug. A hotplug is a program which is used by the kernel to notify user mode software when some significant (usually hardware-related) events take place. An example is when a USB device has just been plugged in or removed.

From Linux 2.6 kernel onwards, hotplugging has been integrated with the driver model core so that any bus or class can report hotplug events when devices are added or removed.

In the kernel tree, /usr/src/linux/Documentation/usb/hotplug.txt has basic information about USB Device Driver API support for hotplugging. See also this link, and GOOGLE as well for examples and documentation.

http://linux-hotplug.sourceforge.net/

Another very helpful document which discusses hotplugging with block devices can be found here:

https://www.kernel.org/doc/pending/hotplug.txt

This document also gives a good example of illustrating hotplug events handling:

Below is a table of the main variables you should be aware of:

Hotplug event variables:

Every hotplug event should provide at least the following variables:

ACTION
The current hotplug action: "add" to add the device, "remove" to remove it.
The 2.6.22 kernel can also generate "change", "online", "offline", and
"move" actions.

DEVPATH
Path under /sys at which this device's sysfs directory can be found.

SUBSYSTEM
If this is "block", it's a block device.  Anything other subsystem is
either a char device or does not have an associated device node.

The following variables are also provided for some devices:

MAJOR and MINOR
If these are present, a device node can be created in /dev for this device.
Some devices (such as network cards) don't generate a /dev node.

DRIVER
If present, a suggested driver (module) for handling this device.  No
relation to whether or not a driver is currently handling the device.

INTERFACE and IFINDEX
When SUBSYSTEM=net, these variables indicate the name of the interface
and a unique integer for the interface.  (Note that "INTERFACE=eth0" could
be paired with "IFINDEX=2" because eth0 isn't guaranteed to come before lo
and the count doesn't start at 0.)

FIRMWARE
The system is requesting firmware for the device.
Dave S
  • 973
  • 9
  • 17
  • After reading the attached links & with some help from google, i understand that the udev/hotplug mechanism is a framework that standardizes the way the low-level kernel drivers notify higher level kernel layers & user-spade applications about various changes that occur within the kernel/hardware. e.g.: a network drivers notifies that a port is up/down to allow higher level networking code to set it up with resources. however, this doesn't seem to provide any help for the driver owner - in fact, it might add some extra work for integrating with udev, generating the events & testing. – DanielHsH Feb 28 '17 at 07:13
  • the question was specifically aimed at the tasks of the person who writes the block device driver & attempts to remove a block device which is a representation of a networked-attached storage (SAN), while an application has open handles & IO's to that block device. This is the issue were having a hard time with - the procedure to quiescent the IO's in-flight & then dealing with how to free the device resources as handles are closed – DanielHsH Feb 28 '17 at 07:13
  • I also added clarification to my question – DanielHsH Feb 28 '17 at 07:16
  • @DanielHsH - I remember on the project I worked on, we had a problem when the user removed the disk during recording/playback. Our way of knowing of Disc insert/removal was through the hot plug events which triggered our handling code. Recording, being an write operation was very tricky to deal with. I remember there was a problem with a handle not being released due to a semaphore lock. There was a fix made on our module that dealt with the file system. Though, your problem is different, in our case we had to make a custom fix in our middleware code. – Dave S Feb 28 '17 at 07:25
1

If the driver is creating device it could be possible to suddenly delete it:

  1. echo 1 > /sys/block/device-name/device/delete where device-name may be sde, for example,

or

  1. echo 1 > /sys/class/scsi_device/h:c:t:l/device/delete, where h is the HBA number, c is the channel on the HBA, t is the SCSI target ID, and l is the LUN.

In my case, it perfectly simulates scenarios for crushing writes and recovery of data from journaling.

Normally to safely remove device more steps is needed so deleting device is a pretty drastic event for data and could be useful for testing :)

please consider this:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/online_storage_reconfiguration_guide/removing_devices

http://www.sysadminshare.com/2012/09/add-remove-single-disk-device-in-linux.html

aNeutrino
  • 131
  • 3
  • 1
    Worth mentioning: if you later want to get your device back, use `for host in $(ls /sys/class/scsi_host/); do echo "- - -" > /sys/class/scsi_host/${host}/scan; done` – Hi-Angel Nov 06 '20 at 07:21