Disclaimer: I work for Kontakt.io, although not as an Android developer
Unfortunately the biggest strength of Android, its openness and variety, is IMHO also a source of the weakness that you see right now. Different manufacturers use different Bluetooth chipsets, antennas, etc. in their devices. All of the possible combinations usually work in typical Bluetooth use cases, like connecting to Bluetooth speakers, keyboards, etc., but some of them fail when something more is required (e.g. scanning beacons).
In my experience latest Samsung flagships are pretty reliable, but e.g. OnePlus phones have a lot of problems with beacons. Similarly, Nexus 5 is one of the best Android phones when it comes to working with beacons, but Nexus 5X really struggles with detecting them. You've mentioned Sony Xperia SP, that it works OK, but we had Xperia Z (don't remember the exact model, sorry) and it was horrible. Apparently your HTC falls into the Bad BLE category :-(