MPI Collective communication

Collective communication means all processes within a communicatorcall the same routine. Portable applications should assume thatcollective routines include a global synchronization. The following simple code fragment employs four basic collectiveroutines to manipulate a statically partitioned regular domain(one-dimensional in this case). The full domain length is broadcastfrom a root process to all others. The initial dataset is distributed(scattered) among the processes. After each compute step, a globalmaximum is determined for use by the root. The root then gathersthe final dataset. ~~~ #include <mpi.h> { int i; int myrank; int size; int root; int full_domain_length; int sub_domain_length; double global_max; double local_max; double *full_domain; double *sub_domain; MPI_Comm_rank(MPI_COMM_WORLD, &myrank); MPI_Comm_size(MPI_COMM_WORLD, &size); root = 0; /* * Root obtains full domain and broadcasts its length. */ if (myrank == root) { get_full_domain(&full_domain, &full_domain_length); } MPI_Bcast(&full_domain_length, 1, MPI_INT, root, MPI_COMM_WORLD); /* * Allocate subdomain memory. * Scatter the initial dataset among the processes. */ sub_domain_length = full_domain_length / size; sub_domain = (double *) malloc(sub_domain_length * sizeof(double)); MPI_Scatter(full_domain, sub_domain_length, MPI_DOUBLE, sub_domain, sub_domain_length, MPI_DOUBLE, root, MPI_COMM_WORLD); /* * Loop computing and determining max values. */ for (i = 0; i < NSTEPS; ++i) { compute(sub_domain, sub_domain_length, &local_max); MPI_Reduce(&local_max, &global_max, 1, MPI_DOUBLE, MPI_MAX, root, MPI_COMM_WORLD); } /* * Gather final dataset. */ MPI_Gather(sub_domain, sub_domain_length, MPI_DOUBLE, full_domain, sub_domain_length, MPI_DOUBLE, root, MPI_COMM_WORLD); } ~~~ ### Broadcast ~~~ MPI_Bcast(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm); ~~~ All processes use the same count, datatype, root, and communicator.Before the operation, the root buffer contains a message. After theoperation, all buffers contain the message from the root process. ### Scatter ~~~ MPI_Scatter(void *sndbuf, int sndcnt, MPI_Datatype sndtype, void *rcvbuf, int rcvcnt, MPI_Datatype rcvtype, int root, MPI_Comm comm); ~~~ All processes use the same send and receive counts, datatypes, rootand communicator. Before the operation, the root send buffer containsa message of length `sndcnt * N', where N is the number of processes.After the operation, the message is divided equally and dispersed toall processes (including the root) following rank order. ### Reduce ~~~ MPI_Reduce(void *sndbuf, void *rcvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm); ~~~ All processes use the same count, datatype, reduction operation, rootand communicator. After the operation, the root process has in itsreceive buffer the result of the pair-wise reduction of the sendbuffers of all processes, including its own. MPI predefines reductionoperations, including: MPI_MAX, MPI_MIN, MPI_SUM, MPI_PROD, MPI_LAND,MPI_BAND, MPI_LOR, MPI_BOR, MPI_LXOR, MPI_BXOR. ### Gather ~~~ MPI_Gather(void *sndbuf, int sndcnt, MPI_Datatype sndtype, void *rcvbuf, int rcvcnt, MPI_Datatype rcvtype, int root, MPI_Comm comm); ~~~ All processes use the same send and receive counts, datatypes, root andcommunicator. This routine is the reverse of MPI_Scatter(): after theoperation the root process has in its receive buffer the concatenationof the send buffers of all processes (including its own), with a totalmessage length of `rcvcnt * N', where N is the number of processes.The message is gathered following rank order.