I created a memory block with a Byte size of 10 and wanted to create a random number and put it into the Memory block but it always just gives me error messages so I wonder if I am doing it wrong.

from multiprocessing import shared_memory
import random

shared_mem_1 = shared_memory.SharedMemory(create=True, size=10)
num = (random.sample(range(1, 1000), 10))
for i, c in enumerate(num):
    shared_mem_1.buf[i] = c

the error-message:

Traceback (most recent call last):

File "main.py", line 7, in shared_mem_1.buf[i] = c

ValueError: memoryview: invalid value for format 'B'

CodePudding user response：

The problem is that num contains values over 255 and when it's assigned to buf the invalid value for format 'B' error appears. Format B is exactly the format for bytes (Check the table of formats here).

There are 2 options:

1. Change the range of the random numbers to be between 0 and 255; or,
2. Convert to bytes with the int.to_bytes function.

Option 1

from multiprocessing import shared_memory
import random

shared_mem_1 = shared_memory.SharedMemory(create=True, size=10)
num = (random.sample(range(0, 255), 10))
for i, c in enumerate(num):
    shared_mem_1.buf[i] = c
shared_mem_1.unlink()

Option 2

For option 2 you'd need to pay attention to the bytes order (big-endian/little-endian) and how many bytes an integer has in your case (Also, the amount of memory to allocate depends on this length). The assignment to the buffer should calculate the offset it saved already.

from multiprocessing import shared_memory
import random

int_length = 4
shared_mem_1 = shared_memory.SharedMemory(create=True, size=int_length * 10)
num = (random.sample(range(1, 1000), 10))
for i, c in enumerate(num):
    pos = i*int_length
    shared_mem_1.buf[pos:pos int_length] = c.to_bytes(int_length, 'big')
shared_mem_1.unlink()

CodePudding user response：

I find the most useful way to take advantage of multiprocessing.shared_memory is to create a numpy array that uses the shared memory region as it's memory buffer. Numpy handles setting the correct data type (is it an 8 bit integer? a 32 bit float? 64 bit float? etc..) as well as providing a convenient interface (similar, but more extensible than python's built-in array module). That way any modifications to the array are visible across any processes that have that same memory region mapped to an array.

from multiprocessing import Process
from multiprocessing.shared_memory import SharedMemory
import numpy as np

def foo(shm, shape, dtype):
    arr = np.ndarray(shape, dtype, buffer = shm.buf) #remote version of arr
    print(arr)
    arr[0] = 20 #modify some data in arr to show modifications cross to the other process
    shm.close() #SharedMemory is internally a file, which needs to be closed.

if __name__ == "__main__":
    shm = SharedMemory(create=True, size=40) #40 bytes for 10 floats
    arr = np.ndarray([10], 'f4', shm.buf) #local version of arr (10 floats)
    arr[:] = np.random.rand(10) #insert some data to arr
    p = Process(target=foo, args=(shm, arr.shape, arr.dtype)
    p.start()
    p.join() #wait for p to finish
    print(arr) #arr should reflect the changes made in foo which occurred in another process.
    shm.close() #close the file
    shm.unlink() #delete the file (happens automatically on windows but not linux)