1

I'm trying to write Android Camera stream frames to the UVC Buffer using FileOutputStream. For context: the UVC Driver is working on the device and it has a custom built kernel.

I get 24 frames per second using imageAnalyzer:

imageAnalyzer = ImageAnalysis.Builder()
.setTargetAspectRatio(screenAspectRatio)
.setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888)
...
imageAnalysis.setAnalyzer(cameraExecutor) { image ->
val buffer = image.planes[0].buffer
val data = buffer.toByteArray()
...
}

Then based on UVC Specifications I build the header of the frame:

val header = ByteBuffer.allocate(26)
val frameSize = image.width * image.height * ImageFormat.getBitsPerPixel(image.format) / 8
val EOH = 0x01
val ERR = 0x00
val STI = 0x01
val REST = 0x00
val SRC = 0x00
val PTS = (System.currentTimeMillis() - referenceTime) * 10000
val endOfFrame = 0x01
val FID = (frameId).toByte()

Add all of the above to the header

header.putInt(frameSize)
header.putShort(image.width.toShort())
header.putShort(image.height.toShort())
header.put(image.format.toByte())
header.put(((EOH shl 7) or (ERR shl 6) or (STI shl 5) or (REST shl 4) or SRC).toByte())
header.putLong(PTS)
header.put(endOfFrame.toByte())
header.put(FID)

Open the FileOutputStream and try to write the header and the image:

val uvcFileOutputStream = FileOutputStream("/dev/video3", true)
uvcFileOutputStream.write(header.toByteArray() + data)
uvcFileOutputStream.close()

Tried to tweak the header/payload but I'm still getting the same error:

java.io.IOException: write failed: EINVAL (Invalid argument)
at libcore.io.IoBridge.write(IoBridge.java:654)
at java.io.FileOutputStream.write(FileOutputStream.java:401)
at java.io.FileOutputStream.write(FileOutputStream.java:379)

What could I be doing wrong? is the header format wrong?

Filip Luchianenco
  • 6,912
  • 9
  • 41
  • 63

1 Answers1

0

I don't know the answer directly, but I was curious to look and have some findings. I focused on the Kotlin part, as I don't know about UVC and because I suspect the problem to be there.

Huge assumption

Since there's no link to the specification I just found this source: https://www.usb.org/document-library/video-class-v15-document-set within the ZIP I looked at USB_Video_Payload_Frame_Based_1.5.pdf Page 9, Section 2.1 Payload Header I'm basing all my findings on this, so if I got this wrong, everything else is. It could still lead to a solution though if you validated the same things.

Finding 1: HLE is wrong

HLE is the length of the header, not the image data. You're putting the whole image size there (all the RGB byte data). Table 2-1 describes PTS and SCR bits control whether PTS and SCR are present. This means that if they're 0 in BFH then the header is shorter. This is why HLE is either 2, 6, 12. Confirmation source + the fact that the field is 1 byte long (each row of Table 2-1 is 1 byte/8 bits) which means the header can be only up to 255 bytes long.

Finding 2: all the header is misaligned

Since your putting HLE with putInt, you're writing 4 bytes, from this point on, everything is messed up in the header, the flags depend on image size, etc.

Finding 3: SCR and PTS flag inconsistencies

Assuming I was wrong about 1 and 2. You're still setting the SRC and PTS bit to 0, but pushing a long (8 bytes).

Finding 4: wrong source

Actually, something is really off at this point, so I looked at your referenced GitHub ticket and found a better example of what your code represents:

Sadly, I was unable to match up what your header structure is, so I'm going to assume that you are implementing something very similar to what I was looking at, because all PDFs had pretty much the same header table.

Finding 5: HLE is wrong or missing

Assuming you need to start with the image size, the HLE is still wrong because it's the image format's type, not in connection with SCR and PTS flags.

Finding 6: BFH is missing fields

If you're following one of these specs, the BFH is always one byte with 8 bits. This is confirmed by how the shls are putting it together in your code and the descriptions of each flag (flag = true|false / 1/0).

Finding 7: PTS is wrong

Multiplying something that is millisecond precise by 10000 looks strange. The doc says "at most 450 microseconds", if you're trying to convert between ms and us, I think the multiplier would be just 1000. Either way it is only an int (4 bytes) large, definitely not a long.

Finding 8: coding assistant?

I have a feeling after reading all this, that Copilot, ChatGPT or other generator wrote your original code. This sound confirmed by you looking for a reputable source.

Finding 9: reputable source example

If I were you I would try to find a working example of this in GitHub, using keyword search like this: https://github.com/search?q=hle+pts+sti+eoh+fid+scr+bfh&type=code the languages don't really matter since these are binary file/stream formats, so regardless of language they should be produced and read the same way.

Finding 10: bit order

Have a look at big endian / little endian byte order. If you look at Table 2-1 in the PDF I linked you can see which bit should map to which byte. You can specify the order you need easily on the buffer BEFORE writing to it, by the looks of the PDF it is header.order(ByteOrder.LITTLE_ENDIAN). I think conventionally 0 is the lowest bit and 31 is the highest. I can't cite a source on this, I seem to remember from uni. Bit 0 should be the 2^0 component (1) and bit 7 is the 2^7 (128). Reversing it would make things much harder to compute and comprehend. So PTS [7:0] means that byte is the lowest 8 bits of the 32 bit PTS number.


If you link to your specification source, I can revise what I wrote, but likely will find very similar guesses.

TWiStErRob
  • 44,762
  • 26
  • 170
  • 254
  • I appreciate you diving deeper into this. The USB_Video_Payload_Frame_Based_1.5.pdf is the source I used as well together with information from Microsoft's UVC driver explanation and with some base explanations from ChatGPT like you mentioned in Finding 8. It obviously can't write the headers for me but it did help explain how the header needs to be formed. I will try fixing the other points you mentioned but upon my latest research it seems like I need to work with IOCTL directly in C++ and write to the /dev/video3 device same way uvc_gadget does(in github ticket above). – Filip Luchianenco Jan 26 '23 at 03:13
  • the main problem with all the sources(github, microsoft, pdf specs you mentioned) is that most of the time they describe how to consume a UVC device source. They all assume we have a producer. In this case we are the producer and need to emulate a UVC device producer. – Filip Luchianenco Jan 26 '23 at 03:19