CoreML performance cost of Vision Framework vs. CVPixelBuffer?

Asked Jun 22 '21 at 15:05

Active Apr 02 '22 at 12:30

Viewed 226 times

There are two options for passing images into CoreML models:

Pass in a CGImage to Vision Framework with its niceties
Pass in a CVPixelBuffer directly

Is there any data on the memory and processing overhead associated with using the Vision framework vs. passing a CVPixelBuffer directly to CoreML?

Thoughts based on what I've seen while debugging:

Memory

Assuming we already have the data in a CVPixelBuffer, creating the CGImage to pass to Vision seems to double the memory usage. It looks like Vision is creating a new object in CoreVideo/CoreImage from createPixelBufferFromVNImageBuffer which makes sense as it needs to create a copy of the image to crop/rotate/scale.

Processing

You're going to have to do the rotation and/or scaling either way. I'd assume Vision does those at least as efficiently as you could do by hand with Accelerate. So should not be any overhead here.

edited Apr 02 '22 at 12:30

Andy Jazz

49,178
17
136
220

asked Jun 22 '21 at 15:05

Nate Lowry

2

Use `CVPixelBuffer` if you already have it, preferably in BGRA pixel order (or YUV) as opposed to ARGB. – Matthijs Hollemans Jun 22 '21 at 21:03
The person who wrote the book! I think I might see if I can do the rotation and scaling with Accelerate without too much of a perf or maintainability hit. – Nate Lowry Jun 24 '21 at 01:51
Vision can automatically do cropping and rotation (in steps of 90 degrees). – Matthijs Hollemans Jun 25 '21 at 09:17

CoreML performance cost of Vision Framework vs. CVPixelBuffer?

Memory

Processing

0 Answers0