From Documentation/driver-model/binding.txt:
When a new device is added, the bus's list of drivers is iterated over
to find one that supports it. In order to determine that, the device
ID of the device must match one of the device IDs that the driver
supports. The format and semantics for comparing IDs is bus-specific.
From the same source:
The process is almost identical for when a new driver is added.
The bus's list of devices is iterated over to find a match. Devices
that already have a driver are skipped. All the devices are iterated
over, to bind as many devices as possible to the driver.
So it looks like you have a problem in the ID matching, that is in your case specific to the USB bus, see the usb_device_match function in drivers/usb/core/driver.c.
Or it may be that your device already is bound to a device that is unloaded when the device is unplugged (you can check that by controlling if there is a symlink to a driver in the corresponding /sys/bus/usb/xxxxx directory)