Why does the result of my image filter look too strong, compared to reference?-CodePudding

in my computer vision class, I am asked to write my convolution codes from scatch, and use Pillow to test my results. I verify my convolution results with Scipy, they agree except at the boundary, which I don't care that much at this moment. Then I converted some images into numpy arrays, applied convolutions, and used Image.fromarray to convert it back to image, I found that my result is really different from using Image.filter directly. Here is one of the images I am using:

Image.filter gives me which I think is nice. By when I do my own convolution and use Image.fromarray, it gives me

I am not sure where went wrong. Could anyone help me with it? Any hint is appreciated!

from PIL import Image
from PIL import ImageFilter
import sys
import numpy as np
from scipy.signal import convolve2d
from PIL import ImageCms


def convolve2d_color(A,B):
    # A is the convolution kernel
    # B is matrix representing RGB. B.shape = (n,m,3)
    result = np.zeros(B.shape)
    for i in range(3):
        result[:, :, i] = convolve2d( B[:,:, i], A, 'same', 'fill' )
    return result

im = Image.open(sys.argv[1])


C = np.zeros((3,3))
C[1][0] = -1
C[1][-1] = 1

# doing convolution to images

im_arr = np.asarray(im)/255.0
# print(im_arr[:,:,0])
im_conv = convolve2d_color(C, im_arr)
# print(im_conv[:,:,0])
print(np.uint8(im_conv[1:-1,1:-1,1]*255))
im_output = Image.fromarray(np.uint8(im_conv*255), mode="RGB")
profile = ImageCms.createProfile("sRGB")
im_output.save("save1.png", icc_profile=ImageCms.ImageCmsProfile(profile).tobytes())

profile = ImageCms.createProfile("sRGB")
im_pillow = im.filter(ImageFilter.Kernel((3,3), list(C.reshape(9,)), 1))
print(np.array(im_pillow)[1:-1,1:-1,1])
im_pillow.save("save2.png", icc_profile=ImageCms.ImageCmsProfile(profile).tobytes())

Run python3 pyfile.py image.jpg to see the result. I also print some matrices out:

# My convolution
[[246 246 236 ...   0 254 253]
 [247 247 238 ... 255 254 255]
 [248 249 236 ...   0 254 254]
 ...
 [252 252 252 ...   0  14   4]
 [  2   0 254 ...   8  20   2]
 [  2   1 255 ...  12  14 253]]
# Image.filter
[[10 10 20 ...  1  2  3]
 [ 9  9 19 ...  2  2  1]
 [ 8  7 20 ...  1  2  2]
 ...
 [ 5  5  4 ...  0  0  0]
 [ 0  0  2 ...  0  0  0]
 [ 0  0  1 ...  0  0  3]]

Many entries add up to 256, but some don't.

CodePudding user response：

The main issue involves the conversion of im_conv to uint8.
Without clipping pixel values to range [0, 255], the negative values gets overflowed to high positive values when converting to uint8.

Correct conversion applies scaling by 255, rounding, clipping to [0, 255] and casting to uint8:

im_conv_as_uint8 = np.uint8(np.round(im_conv*255).clip(0, 255))

For getting the same result using Pillow, we have to flip the kernel matrix left/right.
I can't find it in the documentation, but according to the filtered output, it should be flipped left/right before converting it to a list:

C = C[:, ::-1]  # Flip left/right
im_pillow = im.filter(ImageFilter.Kernel((3, 3), list(C.reshape(9,)), 1))

The following code sample shows that the outputs of SciPy and Pillow are equal (except for the margins):

from PIL import Image
from PIL import ImageFilter
import sys
import numpy as np
from scipy.signal import convolve2d
from PIL import ImageCms


def convolve2d_color(A, B):
    # A is the convolution kernel
    # B is matrix representing RGB. B.shape = (n,m,3)
    result = np.zeros(B.shape)
    for i in range(3):
        result[:, :, i] = convolve2d(B[:, :, i], A, 'same', 'fill')
    return result

im = Image.open('parrot.jpg')


C = np.zeros((3,3))
C[1][0] = -1
C[1][-1] = 1

# doing convolution to images

im_arr = np.asarray(im)/255.0
# print(im_arr[:,:,0])
im_conv = convolve2d_color(C, im_arr)
# print(im_conv[:,:,0])
im_conv_as_uint8 = np.uint8(np.round(im_conv*255).clip(0, 255))
print(im_conv_as_uint8[1:-1,1:-1,1])
im_output = Image.fromarray(im_conv_as_uint8, mode="RGB")
profile = ImageCms.createProfile("sRGB")
im_output.save("save1.png", icc_profile=ImageCms.ImageCmsProfile(profile).tobytes())


C = C[:, ::-1]  # Flip left/right (required for getting correct result of im.filter)

profile = ImageCms.createProfile("sRGB")
im_pillow = im.filter(ImageFilter.Kernel((3, 3), list(C.reshape(9,)), 1))
print(np.array(im_pillow)[1:-1,1:-1,1])
im_pillow.save("save2.png", icc_profile=ImageCms.ImageCmsProfile(profile).tobytes())


# Check for equality (ignore the margins)
is_equal = np.array_equal(np.array(im_pillow)[1:-1,1:-1,1], im_conv_as_uint8[1:-1,1:-1,1])
print(is_equal) # True

im_conv_as_uint8:

Note:
Clipping the negative values to zero gives the same output as Pillow, but it may not be the best solution (because we are losing data).

We may apply linear transformation from range [-1, 1] to [0, 255]:

im_conv_as_uint8 = np.uint8(np.round((im_conv 1)/2*255).clip(0, 255))