Getting RGBA values for all pixels of CGImage Swift-CodePudding

I am trying to create a real time video processing app, in which I need to get the RGBA values of all pixels for each frame, and process them using an external library, and show them. I am trying to get the RGBA value for each pixel, but it is too slow the way I am doing it, I was wondering if there is a way to do it faster, using VImage. This is my current code, and the way I get all the pixels, as I get the current frame:

        guard let cgImage = context.makeImage() else {
        return nil
    }
    guard let data = cgImage.dataProvider?.data,
    let bytes = CFDataGetBytePtr(data) else {
    fatalError("Couldn't access image data")
    }
    assert(cgImage.colorSpace?.model == .rgb)
    let bytesPerPixel = cgImage.bitsPerPixel / cgImage.bitsPerComponent
    gp.async {
        for y in 0 ..< cgImage.height {
            for x in 0 ..< cgImage.width {
                let offset = (y * cgImage.bytesPerRow)   (x * bytesPerPixel)
                let components = (r: bytes[offset], g: bytes[offset   1], b: bytes[offset   2])
                print("[x:\(x), y:\(y)] \(components)")
            }
            print("---")
        }

    }

CodePudding user response：

This is the slowest way to do it. A faster way is with a custom CoreImage filter.

Faster than that is to write your own OpenGL Shader (or rather, it's equivalent in Metal for current devices)

I've written OpenGL shaders, but have not worked with Metal yet.

Both allow you to write graphics code that runs directly on the GPU.

CodePudding user response：

You can access the underlying pixels in a vImage buffer to do this.

For example, given an image named cgImage, use the following code to populate a vImage buffer:

guard
    let format = vImage_CGImageFormat(cgImage: cgImage),
    let buffer = try? vImage_Buffer(cgImage: cgImage,
                                    format: format) else {
        exit(-1)
    }

let rowStride = buffer.rowBytes / MemoryLayout<Pixel_8>.stride / format.componentCount

Note that a vImage buffer's data may be wider than the image (see: https://developer.apple.com/documentation/accelerate/finding_the_sharpest_image_in_a_sequence_of_captured_images) which is why I've added rowStride.

To access the pixels as a single buffer of interleaved values, use:

do {
    let n = rowStride * Int(buffer.height) * format.componentCount
    let start = buffer.data.assumingMemoryBound(to: Pixel_8.self)
    let ptr = UnsafeBufferPointer(start: start, count: n)
    
    print(Array(ptr)[ 0 ... 15]) // prints the first 15 interleaved values
}

To access the pixels as a buffer of Pixel_8888 values, use (make sure that format.componentCount is 4:

do {
    let n = rowStride * Int(buffer.height)
    let start = buffer.data.assumingMemoryBound(to: Pixel_8888.self)
    let ptr = UnsafeBufferPointer(start: start, count: n)
    
    print(Array(ptr)[ 0 ... 3]) // prints the first 4 pixels
}