1

I would like to implement the read and write calls of the python hidapi, in pysub.

An example code using the python hidapi, looks like this:

import hid

hdev = hid.device()
h = hdev.open_path( path )

h.write( send_buffer )

res = h.read( 64 )    
receive_buffer = bytearray( res )

The main problem that I have with this is that the python hidapi read() returns a list of ints (one python int for each byte in the buffer received from the hardware), and I need the buffer as bytes and faithful to what was received.(*)

A secondary issue is that open, read and write are the only things I need and I need to keep the system as light as possible. Therefore I want to avoid the extra dependencies.

(*) bytearray() is not a good solution in this case, for reasons beyond the scope of this question.

DrM
  • 2,404
  • 15
  • 29
  • Just to clarify, what's the reason that a list of ints isn't suitable for your use-case? You can iterate over them or do random access into them the same as if it was a byte string. Are you passing it to something else that specifically needs a byte string? – Kemp Jun 23 '21 at 14:24
  • @Kemp, I am passing it to something that needs the data as sent. It comes from data acquisition hardware, and can be two byte ints or floats from imaging sensors and waveform recorders. The problems in having to convert it back to its original format, are performance and throughput related. – DrM Jun 23 '21 at 14:57
  • Reading the documentation for pyUSB it appears to return arrays from the read calls, so you'll have the same problem as you do for hidapi. – Kemp Jun 23 '21 at 22:13
  • @kemp Pyusb, for read, returns an array object of the transfer type. If the transfer type is bytes, it is an array of bytes. The data is faithful to what was sent and we do not have the problem of converting the values as ints back to the actual structure in bytes. Note that the second parameter in the read call is the number of bytes. – DrM Jun 24 '21 at 02:25
  • @kemp see the answer below. It works and the data is indeed an array of bytes. – DrM Jun 24 '21 at 20:52
  • The `read()` return may confusingly appear as a "list of ints" — but it's [actually](https://github.com/pyusb/pyusb/blob/master/docs/tutorial.rst#talk-to-me-honey) a byte [array](https://docs.python.org/3/library/array.html). This type is as efficient, compact and exact as Python gets. Think of it not as `std::list`, but as `uint8_t[]`. – ulidtko Jun 25 '21 at 11:21

2 Answers2

1

I've done some quick benchmarking and it seems that DrM's answer was definitely heading in the right direction with working on arrays, but there is a slightly better option for conversion. Results below for 10 million iterations operating on 64-byte data buffers.

Using

data_list = [30] * 64
data_array = array('B', data_list)

I got the following run times in seconds:

Technique Time (seconds, 10 million iterations)
bytearray(data_list) 12.7
bytearray(data_array) 3.0
data_array.tobytes() 2.0
struct.pack('%uB' % len(data_list), *data_list) 18.6
struct.pack('%uB' % len(data_array), *data_array) 22.5

It appears that using the array.tobytes method is the fastest, followed by calling bytearray with the array as the argument.

Obviously I reused the same buffer on each iteration, probably among other unrealistic factors, so YMMV. These should be indicative results relative to each other though, even if not in absolute terms. Also this obviously doesn't account for the performance of then working on a bytearray versus bytes.

Kemp
  • 3,467
  • 1
  • 18
  • 27
  • I believe the pysub read() returns type array. Would that then be the data_array of your second case above? If so then, i could simple do read().tobytes() ? Interesting, I think I will try it. It would be a 33% saving in time, certainly worthwhile. – DrM Jun 27 '21 at 13:01
0

Here is a minimum code example that seems to work. The HID device is a Teensy board that uses its RAWHID.recv() and RAWHID.send() to exchange text and binary with the host computer.

#!/usr/bin/python

import usb.core
import usb.util

dev = usb.core.find(idVendor=0x16C0, idProduct=0x0486)
    
try:
    dev.reset()
except Exception as e:
    print( 'reset', e)

if dev.is_kernel_driver_active(0):
    print( 'detaching kernel driver')
    dev.detach_kernel_driver(0)

endpoint_in = dev[0][(0,0)][0]
endpoint_out = dev[0][(0,0)][1]

# Send a command to the Teensy
endpoint_out.write( "version".encode() + bytes([0]) )

# Read the response, an array of byte, .tobytes() gives us a bytearray.
buffer = dev.read(endpoint_in.bEndpointAddress, 64, 1000).tobytes()

# Decode and print the zero terminated string response
n = buffer.index(0)
print( buffer[:n].decode() )
DrM
  • 2,404
  • 15
  • 29
  • You may want to check out the `array`'s `tobytes` method, which may be able to work faster given internal knowledge of the object. Worth a benchmark at least. – Kemp Jun 25 '21 at 08:06
  • I should also point out that `bytearray` is a mutable object and will have some overhead compared to `bytes`. Might be worth seeing how simply doing `struct.pack('%uB' % len(buffer), *buffer)` to get `bytes` performs. – Kemp Jun 25 '21 at 08:17
  • My curiosity won out and I've posted my benchmarking results as [an answer](https://stackoverflow.com/a/68128418/3228591) for reference. – Kemp Jun 25 '21 at 09:10
  • I'll note that this solution may work — but is not HID, strictly speaking. Here you do direct barebones USB transfers (bulk or interrupt, depending on what the device declares in its endpoint descriptor). This is 1 level below actual HID. By analogy, this is like doing HTTP requests by opening TCP socket and hand-formatting and handling all the headers, status lines & GET/POST methods, manually. Nothing wrong with the approach per se; just be aware that this is simply PyUSB usage — not HID yet, there remains a layer of detail above. – ulidtko Jun 25 '21 at 11:12
  • @ulidtko Yes this skips the hid completely, That is what I want. For this use case, the HID does not contribute anything and only adds dependencies and a deeper call stack. – DrM Jun 27 '21 at 12:57
  • @kemp Thank you, I confirmed that tobytes() works and edited accordingly. – DrM Jun 28 '21 at 12:07
  • As a small comment, `bytes([0])` ends up being `b'\x00'`. Using the latter directly will avoid you constructing an array and then converting the single element to a byte string. Also, instead of `bytearray(buffer[:n]).decode()` you can just do `buffer[:n].decode()`, avoiding the construction (and immediate disposal) of the `bytearray`. – Kemp Jun 28 '21 at 12:15
  • @kemp, oops you are right. I forgot to change that when i changed the first part to use .tobytes() – DrM Jun 29 '21 at 13:45
  • `usb.core.find` doesn't seem to work for me in Windows 10. – RufusVS Mar 21 '22 at 19:51
  • @RufusVS Hi, do you have libusb installed? There are a few posts on pyusb and windows on stackoverflow. And, I imagine googling the error message will quickly take you to answer. – DrM Mar 22 '22 at 01:26
  • @DrM I'll check on `libusb`, I probably don't have it installed. I actual found another route with `hid` and `hidapi` and was able to build it and change the code. But I'd rather get `pyusb` working, so I imagine if was able to build and install one, I should be able to do the other. – RufusVS Mar 22 '22 at 05:03
  • @DrM To follow up, it was quite simple to install libusb with the instructions at the [Adafruit learning](https://learn.adafruit.com/circuitpython-on-any-computer-with-ft232h/windows) so `pyusb` works fine! That'll be way better than hid! – RufusVS Mar 22 '22 at 05:25
  • @RufusVS very good. If you happen to also be using a Teensy, the serial interface is actually must faster. The trick is to select dual serial ports in the setup and then in the code use one for input and one for output. There is a bug somewhere (host or slave, I am not sure), that makes it slow when one port does both. – DrM Mar 22 '22 at 18:23
  • @DrM Actually, in my case I'm communicating with `Denkovi` 4 port relay board, which uses the `MCP2200` interface chip and doesn't even use its serial port, but another endpoint. But its working great now. I may try to finish the HID version just as an intellectual exercise though. – RufusVS Mar 23 '22 at 04:31
  • With this capricious down vote, I regret posting the question. Moreover, this answer is absolutely correct. I wrote the question, I know what answer I was looking for, this is it and it works fine. The person who downvoted this, posted no explanation and afforded no opportunity to discuss and clarify any misconceptions. This is the behavior, far to common on the site, and usually by uneducated or self educated people, that moved me to stop contributing to stackoverflow. – DrM Jul 30 '23 at 15:37