I have 4 python files in one project folder. main.py first.py second.py variables.py

I only run main.py. This file sequentially calls first.py, then second.py. Then, main.py, first.py and second.py imports variables.py.

The content of variables.py is simply the declaration of a "shared" variable across the three.

I wanted first.py to modify this shared variable and then I want this change to be carried over when the process goes back to main.py (after returning from first.py) and when the second.py is finally called.

I initially thought I would be able to do this since the variable was declared in a separate py file, but its not working.

My understanding of what's happening is:

1. first.py imports variables.py. This action causes the variable to be declared with it's value set to initial.
2. first.py modifies this shared variable.
3. first.py execution ends and goes back to main.py. At this point, I see that the value of shared variable is back to initial. Why is that? Is it because first.py execution ends? But why did it happen even if the shared variable is declared in another python file?

I appreciate anyone who can enlighten me on what's happening (how the shared variable is stored in memory, what script will determine is lifetime, ending which script will end this variable, etc..). And I appreciate suggestions on how do I go about this. At this point, I am already considering on simply writing the modified shared variable value (on first.py) to an external text file then simply read and re-initialize when second.py is called later.

My codes are below. To run the project, simply run main.py

1. main.py

    import subprocess
    import os
    import variables
    
    programs = ['first.py', 'second.py']
    path=os.getcwd()   '\\running multiple py with shared variables\\'
    for program in programs:
        subprocess.run(['python', path program])
        print('running main.py')
        print(variables.shared_variable)

2. first.py

    import variables
    print('running first.py')
    variables.shared_variable = 'First modification'
    print(variables.shared_variable)

3. second.py

    import variables
    print('running second.py')
    print(variables.shared_variable)

4. variables.py

    shared_variable = "Initial value"

Output of program on terminal:

running first.py
First modification
running main.py
Initial value                   -> I really want this to be "First 
                                   modification"
running second.py
Initial value                   -> I really want this to be "First 
                                   modification"
running main.py
Initial value                   -> I don't really care here but I honestly 
                                   expected this to be "First modification" 
                                   as well` ```

CodePudding user response：

There's no shmem() shared memory going on here.

    for program in programs:
        subprocess.run(['python', path   program])

You spawned a pair of child processes. Which each computed a result. And then called exit(), discarding the result.

If the child doesn't serialize a result which the parent parses, then the result is gone forever. The symptoms you report are exactly what is expected.

The good news is that you have excellent skills for thinking about this problem analytically, and reporting to others what happened. So I am confident you will soon implement a satisfactory solution.

CodePudding user response：

Credits to @J_H for mentioning shared memory concept. I was able to solve my problem using shared_memory class from multiprocessing module.

I am sharing the modified code. With this implementation, the file variables.py is not even needed since you need to track only the decided name of the shared memory block (shared_memory1 here). If you don't specify it, it will be auto-generated which you can check in the object attributes.

1. main.py

import subprocess
import os
import variables
from multiprocessing import shared_memory

shm = shared_memory.SharedMemory(create=True, size=500, name='shared_memory1')
programs = ['first.py', 'second.py']
path=os.getcwd()   '\\running multiple py with shared variables\\'
for program in programs:
    subprocess.run(['python', path program])
    print('running main.py')
    ctr = 0
    retrieved_val = ''
    while (shm.buf[ctr] != 0):
        retrieved_val = retrieved_val   chr(shm.buf[ctr])
        ctr = ctr   1
    variables.shared_variable = retrieved_val
    print(variables.shared_variable)

shm.close()
shm.unlink()

2. first.py

import variables
from multiprocessing import shared_memory

shm = shared_memory.SharedMemory(name='shared_memory1')

print('running first.py')
variables.shared_variable = 'First modification'

ctr = 0
for val in variables.shared_variable:
    shm.buf[ctr] = ord(val)
    ctr = ctr   1
print(variables.shared_variable)

shm.close()

3. second.py

import variables
from multiprocessing import shared_memory

shm = shared_memory.SharedMemory(name='shared_memory1')
ctr = 0
retrieved_val = ''
while (shm.buf[ctr] != 0):
    retrieved_val = retrieved_val   chr(shm.buf[ctr])
    ctr = ctr   1
print('running second.py')
variables.shared_variable = retrieved_val
print(variables.shared_variable)

shm.close()

4. variables.py

shared_variable = "Initial value"

Terminal output after running main.py:

running first.py
First modification
running main.py
First modification
running second.py
First modification
running main.py
First modification