Speed Up Multiple Model Inference on EDGE TPU

Question

I have retrained a RESNET50 model for reidentification on EDGE TPU. However, it seems to be no way to fetch a batch of image to EDGE_TPU.

I have come up with a solution of running multiple same model for images.

However, is there anyway to speed up the model inference for multiple model? The threading now is even slower than single model inference

score 1 · Answer 1 · answered Sep 14 '20 at 18:38

1

Yeah, the edgetpu's architect won't allow processing in batch size. Have you tried model pipelining? https://coral.ai/docs/edgetpu/pipeline/

Unfortunately only available in C++ right now, but we're looking to extends it to python in mid Q4.

answered Sep 14 '20 at 18:38

Nam Vu

1,727
1
11
24

1

Thank you, Nam. I will give it a try my Vietnamese fellow :D – dtlam26 Sep 15 '20 at 01:25
this is outside of the comment section, but how to post-quantize full integer inference on keras with multiple output. I have try and always fail at edge_tpu compile. Can you check it for me? I attach my model below – dtlam26 Sep 16 '20 at 09:20
https://drive.google.com/drive/folders/1l-8kEOWDWQA7zQD7qU_Vj6Cjvgj8mXf4?usp=sharing – dtlam26 Sep 16 '20 at 09:23
@dtlam26 what does your ptq script looks like? Can you post it on that drive folder also? – Nam Vu Sep 17 '20 at 15:03
I have uploaded it, Nam. Thank you for spending your time for me – dtlam26 Sep 18 '20 at 08:10
I intend to get the quantization with float input and output for regression problem – dtlam26 Sep 18 '20 at 08:11

dtlam26 · Answer 2 · 2020-11-01T18:25:32.380

0

Because batch inference is not available now, so pipelining is another secondary option. However, after experiencing with my model, we can make a psuedo batch by feeding multiple single input for EDGE_TPU as another option

edited Nov 01 '20 at 18:25

answered Oct 08 '20 at 10:48

dtlam26

1,410
11
19

Speed Up Multiple Model Inference on EDGE TPU

2 Answers2