Reorder numpy array to new bitlength elements without loop-CodePudding

If I have a numpy array with elements each representing e.g. a 9-bit integer, is there an easy way (maybe without a loop) to reorder it in a way that the resulting array elements each represent a 8-bit integer with the "lost bits" at the end of the previous element getting added at the beginning of the next element? for example to get the following

np.array([0b100111000, 0b100101100, 0b110011100, 0b110010100])  # initial array in binarys
# convert to
np.array([0b10011100, 0b01001011, 0b00110011, 0b10011001, 0b01000000])  # resulting array

I hope it is understandable what I want to archive. Additional info, I don't know if this makes any difference: All of my 9-bit numbers start with the msb beeing 1 (they are bigger than 255) and the last two bits are always 0, like in the example above. The arrays I want to process are much bigger with thousands of elements.

Thanks for your help in advance!

edit:

my current (complicated) solution is the following:

import numpy as np
def get_bits(data, offset, leng):
    data = (data % (1 << (offset   leng))) >> offset
    return data

data1 = np.array([0b100111000, 0b100101100, 0b110011100, 0b110010100])
i = 1
part1 = []
part2 = []
for el in data1:
    if i == 1:
        part2.append(0)
    part1.append(get_bits(el, i, 8))
    part2.append(get_bits(el, 0, i)<<(8-i))
    if i == 8:
        i = 1
        part1.append(0)
    else:
        i  = 1
if i != 1:
    part1.append(0)
res = np.array(part1)   np.array(part2)

CodePudding user response：

I think I understood most of what you want, and given that you can do bit operation with numpy arrays in which case you get the desire bit operation element wise if do it with two array (or the same for all if it is an array vs a number), then you need to construct the appropriate arrays to do the thing, so something like this

>>> import numpy as np
>>> x = np.array([0b100111000, 0b100101100, 0b110011100, 0b110010100])
>>> goal=np.array([0b10011100, 0b01001011, 0b00110011, 0b10011001, 0b01000000])
>>> x
array([312, 300, 412, 404])
>>> goal
array([156,  75,  51, 153,  64])
>>> shift1 = np.array(range(1,1 len(x)))
>>> shift1
array([1, 2, 3, 4])
>>> mask1  = np.array([2**n -1 for n in range(1,1 len(x))])
>>> mask1
array([ 1,  3,  7, 15])
>>> res=((x>>shift1)|((x&mask1)<<(9-shift1)))&0b11111111
>>> res
array([156,  75,  51, 153], dtype=int32)
>>> goal
array([156,  75,  51, 153,  64])
>>>

I don't understand why your goal array have one extra element, but the above operation give the others numbers, and adding one extra is not complicated, so adjust as necessary.

Now for explaining the ((x>>shift1)|((x&mask1)<<(9-shift1)))&0b11111111

First I notice you do a bigger shift by element, that is simple

>>> x>>shift1
array([156,  75,  51,  25], dtype=int32)
>>> 
>>> list(map(bin,x>>shift1))
['0b10011100', '0b1001011', '0b110011', '0b11001']
>>>

We also want to catch the bits that would be lose with the shift, with an and with an appropriate mask we get those

>>> x&mask1
array([0, 0, 4, 4], dtype=int32)
>>> list(map(bin,mask1))
['0b1', '0b11', '0b111', '0b1111']
>>> list(map(bin,x&mask1))
['0b0', '0b0', '0b100', '0b100']
>>>

then we right shift that result by the complementary amount

>>> 9-shift1
array([8, 7, 6, 5])
>>> ((x&mask1)<<(9-shift1))
array([  0,   0, 256, 128], dtype=int32)
>>> list(map(bin,_))
['0b0', '0b0', '0b100000000', '0b10000000']
>>>

then we or both together

>>> (x>>shift1) | ((x&mask1)<<(9-shift1))
array([156,  75, 307, 153], dtype=int32)
>>> list(map(bin,_))
['0b10011100', '0b1001011', '0b100110011', '0b10011001']
>>>

and finally we and that with 0b11111111 to keep only the 8 bit we want

Additionally, you mention that the last 2 bit are always zero, then a more simple solution is simple shift it by 2, and to recover the original just shift in back in the other direction

>>> x
array([312, 300, 412, 404])
>>> y = x>>2
>>> y
array([ 78,  75, 103, 101], dtype=int32)
>>> y<<2
array([312, 300, 412, 404], dtype=int32)
>>>

CodePudding user response：

You can do it in two steps with np.unpackbits and np.packbits. First turn your array into a big-endian column vector:

>>> z = np.array([0b100111000, 0b100101100, 0b110011100, 0b110010100], dtype='<u2').reshape(-1, 1)
>>> z.view(np.uint8)
array([[ 56,   1],
       [ 44,   1],
       [156,   1],
       [148,   1]], dtype=uint8)

You can convert this into an array of bits directly by unpacking. In fact, at some point (PR #10855) I added the count parameter to chop of the high zeros for you:

>>> np.unpackbits(z.view(np.uint8), axis=1, bitorder='l', count=9)
array([[0, 0, 0, 1, 1, 1, 0, 0, 1],
       [0, 0, 1, 1, 0, 1, 0, 0, 1],
       [0, 0, 1, 1, 1, 0, 0, 1, 1],
       [0, 0, 1, 0, 1, 0, 0, 1, 1]], dtype=uint8)

Now you can just repack the reversed raveled array:

>>> u = np.unpackbits(z.view(np.uint8), axis=1, bitorder='l', count=9)[:, ::-1].ravel()
>>> result = np.packbits(u)
>>> result.dtype
dtype('uint8')
>>> [bin(x) for x in result]
['0b10011100', '0b1001011', '0b110011', '0b10011001', '0b1000000']

If your machine is native little endian (e.g., most intel architectures), you can do this in a one-liner:

z = np.array([0b100111000, 0b100101100, 0b110011100, 0b110010100])
result = np.packbits(np.unpackbits(z.view(np.uint8), axis=1, bitorder='l', count=9)[:, ::-1].ravel())

Otherwise, you can do z.byteswap().view(np.uint8) to get the right starting order.