Making all processors in a computing node read the same memory block in OpenMPI-CodePudding

I want to run a numerical simulation in a computing cluster using OpenMPI and C. However, there is a big matrix A that is constant and equal in every proccessor. To avoid alloc'ing too much unnecessary memory, I want to group the processors that have a shared memory (that is, they are in the same computing node) and have only one processor in each node malloc and set that big matrix A. Then all other processors in the node should be able to read that matrix.

This other question is exactly what I need, but it is in Fortran. I took its answer and made a prototype in C, but it is not working as expected. For a reason I can not understand, the memory address returned by MPI_Win_allocate_shared is offset by 1 displacement unit. See code below:

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int
main ()
{
    MPI_Init(NULL, NULL);
    int num_processors;
    MPI_Comm_size(MPI_COMM_WORLD, &num_processors);
    int proc_global_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &proc_global_rank);

    // Split de global communicator, grouping by shared memory
    MPI_Comm compute_comm;
    MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &compute_comm);
    int proc_compute_rank;
    MPI_Comm_rank(compute_comm, &proc_compute_rank);

    // Create the window
    const int ARRAY_SIZE = 1;
    int* shared_data;
    MPI_Aint shared_data_size;
    if (proc_compute_rank == 0) {
        shared_data_size = ARRAY_SIZE * sizeof(int);
    } else {
        shared_data_size = 0;
    }
    int disp_unit = sizeof(int);
    MPI_Win window;
    MPI_Win_allocate_shared(shared_data_size, disp_unit, MPI_INFO_NULL, compute_comm, &shared_data, &window);
    if (proc_compute_rank == 0) {
        shared_data[0] = -10;
    }
    // get the location of the shared memory segment
    if (proc_compute_rank != 0) {
        MPI_Win_shared_query(window, 0, &shared_data_size, &disp_unit, shared_data);
    }
    MPI_Win_fence(0, window);  // wait for shared variables to be ready to be used
    MPI_Barrier(compute_comm);
    printf("rank %d, value[0] = %d\n", proc_compute_rank, shared_data[0]);  // prints a wrong value
    printf("rank %d, value[-1] = %d\n", proc_compute_rank, shared_data[-1]);  // prints correct value
    // do calculations...
    MPI_Win_fence(0, window);  // wait for shared variables to be no longer needed
    MPI_Barrier(compute_comm);
    MPI_Win_free(&window);
    MPI_Finalize();
    return EXIT_SUCCESS;
}

CodePudding user response：

The real problem in your code is that you pass an int* to MPI_Win_query, but that call needs to set the pointer, so you need to pass an int**, meaning in your case &shared_data.

There can potentially be a second problem: by setting the contents of the window buffer explicitly you are assuming the "unified memory model", which is likely the case. If in doubt, test

MPI_Win_get_attr(the_window,MPI_WIN_MODEL,&modelstar,&flag);

and make sure that that parameter is not MPI_WIN_SEPARATE, in which case you'd have to MPI_Put the data into the window.

Btw, you made an observation about the processes getting different addresses. That is going to stay true even after you fix your ampersand error! What a programmer thinks of as an address is only a logical address which is translated by the "page table" into a physical address. But every process has its own page table, so the same physical address will translate to different logical addresses ni the processes, and so, yes, you will get different printed hex addresses, even if you have the right pointer.