Skip to content

Commit cc4e406

Browse files
rhc54rhc54
committed
Update Group Construct entry
Update the PMIx_Group_construct entry to include a few missing attributes: * PMIX_GROUP_LOCAL_CID - allow a participant to provide a local context ID to be shared with other participants * PMIX_GROUP_BOOTSTRAP - indicate that the "bootstrap" method of group construction is to be used * PMIX_GROUP_ADD_MEMBERS - specify the additional members that are to be included in the final group * PMIX_GROUP_JOB_INFO - used in the data returned at the end of the group construct operation. Includes all job-level info for participants. Extend the PMIx_Group_construct description to explain the new construct method and how it is used. Signed-off-by: Ralph Castain <[email protected]>
1 parent 46c63a5 commit cc4e406

File tree

2 files changed

+252
-2
lines changed

2 files changed

+252
-2
lines changed

‎Chap_API_Sets_Groups.tex

Lines changed: 65 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -279,6 +279,41 @@ \subsection{Process Group Attributes}
279279
\declareAttribute{PMIX_GROUP_LOCAL_ONLY}{"pmix.grp.lcl"}{bool}{
280280
Group operation only involves local processes. \ac{PMIx} implementations are \textit{required} to automatically scan an array of group members for local vs remote processes - if only local processes are detected, the implementation need not execute a global collective for the operation unless a context ID has been requested from the host environment. This can result in significant time savings. This attribute can be used to optimize the operation by indicating whether or not only local processes are represented, thus allowing the implementation to bypass the scan.
281281
}
282+
%
283+
\versionMarkerProvisional{5.1}
284+
\declareAttribute{PMIX_GROUP_LOCAL_CID}{"pmix.grp.lclid"}{size_t}{
285+
Local context ID for the participating process member of a group. Provided
286+
in the \refstruct{pmix_info_t} directives when the process calls construct.
287+
These values are to be returned to all participants with the collected data
288+
at completion of construct.
289+
}
290+
%
291+
\versionMarkerProvisional{5.1}
292+
\declareAttribute{PMIX_GROUP_BOOTSTRAP}{"pmix.grp.btstrp"}{size_t}{
293+
Group construct call will not include all members as individual
294+
participants don't know the entire list but at least
295+
know how may leaders will be involved. Leaders provide only their
296+
own process ID in the \code{procs} parameter to the construct
297+
API, and are \textit{required} to include the \refattr{PMIX_GROUP_BOOTSTRAP}
298+
attribute in their array of \refstruct{pmix_info_t} directives, with
299+
the value in that attribute set to equal the number
300+
of leaders in the group construct operation. They may also provide the
301+
\refattr{PMIX_GROUP_ADD_MEMBERS} attribute with an array of process IDs that are to
302+
belong to the final group - each of those processes will also call the group
303+
construct, but with a \code{NULL} process ID to indicate they are joining
304+
as "add members" and not leaders. Construction will complete once all
305+
leaders and "add members" have participated.
306+
}
307+
%
308+
\versionMarkerProvisional{5.1}
309+
\declareAttribute{PMIX_GROUP_ADD_MEMBERS}{"pmix.grp.add"}{pmix_data_array_t*}{
310+
Array of \refstruct{pmix_proc_t} identifying procs that are not
311+
included in the membership specified in the \code{procs} array passed to
312+
the group construct call, but are to be included in the
313+
final group. See \refattr{PMIX_GROUP_BOOTSTRAP} for a full description
314+
of the associated operation.
315+
}
316+
%
282317

283318
\vspace{\baselineskip}
284319
The following attributes are used to return information at the conclusion of a \ac{PMIx} Group operation and/or in event notifications:
@@ -291,6 +326,11 @@ \subsection{Process Group Attributes}
291326
\declareAttribute{PMIX_GROUP_ENDPT_DATA}{"pmix.grp.endpt"}{pmix_byte_object_t}{
292327
Data collected during group construction to ensure communication between group members is supported upon completion of the operation.
293328
}
329+
%
330+
\versionMarkerProvisional{5.1}
331+
\declareAttribute{PMIX_GROUP_JOB_INFO}{"pmix.grp.jinfo"}{pmix_byte_object_t}{
332+
Job-level info for all group participants, returned at end of group construction
333+
}
294334

295335
\vspace{\baselineskip}
296336
In addition, a process can request (via \refapi{PMIx_Get}) the process groups to which a given process (including itself) belongs:
@@ -353,6 +393,9 @@ \subsection{\code{PMIx_Group_construct}}
353393
The following attributes are optional for host environments that support this operation:
354394

355395
\pasteAttributeItem{PMIX_TIMEOUT}
396+
\pasteAttributeItem{PMIX_GROUP_LOCAL_CID}
397+
\pasteAttributeItem{PMIX_GROUP_BOOTSTRAP}
398+
\pasteAttributeItem{PMIX_GROUP_ADD_MEMBERS}
356399

357400
\optattrend
358401

@@ -363,7 +406,7 @@ \subsection{\code{PMIx_Group_construct}}
363406

364407
Each provided \refstruct{pmix_proc_t} struct can pass \refconst{PMIX_RANK_WILDCARD} to indicate that all processes in the given namespace are participating.
365408
The ordering of the entries in the \refarg{procs} has no significance, neither is the method used for identifying processes.
366-
Some callers may describe the target set of processes using PMIX_RANK_WILDCARD while other
409+
Some callers may describe the target set of processes using PMIX_RANK_WILDCARD while other
367410
callers may list the individual processes of a namespace explicitly.
368411
This is unlike \refapi{PMIx_Connect}, because in this group operation, the group name \refarg{grp} is used to determine the uniqueness of the collective.
369412

@@ -381,8 +424,24 @@ \subsection{\code{PMIx_Group_construct}}
381424

382425
Processes in a group under construction are not allowed to leave the group until group construction is complete. Upon completion of the construct procedure, each group member will have access to the job-level information of all namespaces represented in the group plus any information posted via \refapi{PMIx_Put} (subject to the usual scoping directives) for every group member.
383426

427+
Bootstrap mode is used when the processes leading a group construct operation do
428+
not know the identity of all other processes that will be participating, but at least
429+
know how may leaders will be involved.
430+
Leaders provide only their own process ID in the \code{procs} parameter and are \textit{required} to include the
431+
\refattr{PMIX_GROUP_BOOTSTRAP} attribute in their array of \refstruct{pmix_info_t}
432+
directives, with the value in that attribute set to equal the number
433+
of leaders in the group construct operation. Each leader may optionally provide the
434+
\refattr{PMIX_GROUP_ADD_MEMBERS} attribute with an array of process IDs that are to
435+
belong to the final group - each of those processes must also call the group
436+
construct, but with a \code{NULL} process ID to indicate they are joining
437+
as "add members" and not leaders. While leaders may each uniquely identify "add member" processes, they
438+
are not required to guarantee uniqueness - i.e., two leaders may include the same process in their array
439+
of "add members", though each "add member" must participate only once regardless of the number
440+
of times they might be included in an "add members" array.
441+
Construction will complete once all leaders and "add members" have participated.
442+
384443
\adviceimplstart
385-
At the conclusion of the construct operation, the \ac{PMIx} library is \emph{required} to ensure that job-related information from each participating namespace plus any information posted by group members via \refapi{PMIx_Put} (subject to scoping directives) is available to each member via calls to \refapi{PMIx_Get}.
444+
At the conclusion of the construct operation, the \ac{PMIx} library is \emph{required} to ensure that job-related information from each participating namespace (via the \refattr{PMIX_GROUP_JOB_INFO} attribute) plus any information posted by group members via \refapi{PMIx_Put} (subject to scoping directives) is available to each member via calls to \refapi{PMIx_Get}.
386445
\adviceimplend
387446

388447
\advicermstart
@@ -458,6 +517,10 @@ \subsection{\code{PMIx_Group_construct_nb}}
458517
The following attributes are optional for host environments that support this operation:
459518

460519
\pasteAttributeItem{PMIX_TIMEOUT}
520+
\pasteAttributeItem{PMIX_GROUP_LOCAL_CID}
521+
\pasteAttributeItem{PMIX_GROUP_BOOTSTRAP}
522+
\pasteAttributeItem{PMIX_GROUP_ADD_MEMBERS}
523+
461524

462525
\optattrend
463526

‎sources/group_bootstrap.c

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
/*
2+
* Copyright (c) 2004-2010 The Trustees of Indiana University and Indiana
3+
* University Research and Technology
4+
* Corporation. All rights reserved.
5+
* Copyright (c) 2004-2011 The University of Tennessee and The University
6+
* of Tennessee Research Foundation. All rights
7+
* reserved.
8+
* Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
9+
* University of Stuttgart. All rights reserved.
10+
* Copyright (c) 2004-2005 The Regents of the University of California.
11+
* All rights reserved.
12+
* Copyright (c) 2006-2013 Los Alamos National Security, LLC.
13+
* All rights reserved.
14+
* Copyright (c) 2009-2012 Cisco Systems, Inc. All rights reserved.
15+
* Copyright (c) 2011 Oak Ridge National Labs. All rights reserved.
16+
* Copyright (c) 2013-2020 Intel, Inc. All rights reserved.
17+
* Copyright (c) 2015 Mellanox Technologies, Inc. All rights reserved.
18+
* Copyright (c) 2019 IBM Corporation. All rights reserved.
19+
* Copyright (c) 2021-2025 Nanook Consulting All rights reserved.
20+
* $COPYRIGHT$
21+
*
22+
* Additional copyrights may follow
23+
*
24+
* $HEADER$
25+
*
26+
*/
27+
28+
#include <pthread.h>
29+
#include <stdbool.h>
30+
#include <stdio.h>
31+
#include <stdlib.h>
32+
#include <time.h>
33+
#include <unistd.h>
34+
#include <libgen.h>
35+
36+
#include <pmix.h>
37+
38+
int main(int argc, char **argv)
39+
{
40+
pmix_proc_t myproc;
41+
int rc, ret;
42+
pmix_value_t *val = NULL;
43+
uint32_t nprocs;
44+
pmix_proc_t proc, *parray = NULL;
45+
pmix_info_t *results = NULL, info[3];
46+
size_t nresults, cid, n, m, psize = 0;
47+
pmix_data_array_t dry;
48+
char hostname[1024];
49+
50+
gethostname(hostname, 1024);
51+
52+
/* init us */
53+
if (PMIX_SUCCESS != (rc = PMIx_Init(&myproc, NULL, 0))) {
54+
fprintf(stderr, "Client ns %s rank %d: PMIx_Init failed: %s\n", myproc.nspace, myproc.rank,
55+
PMIx_Error_string(rc));
56+
exit(1);
57+
}
58+
fprintf(stderr, "Client ns %s rank %d: Running on host %s\n", myproc.nspace, myproc.rank, hostname);
59+
60+
PMIX_PROC_CONSTRUCT(&proc);
61+
PMIX_LOAD_PROCID(&proc, myproc.nspace, PMIX_RANK_WILDCARD);
62+
63+
/* get our job size */
64+
if (PMIX_SUCCESS != (rc = PMIx_Get(&proc, PMIX_JOB_SIZE, NULL, 0, &val))) {
65+
fprintf(stderr, "Client ns %s rank %d: PMIx_Get job size failed: %s\n", myproc.nspace,
66+
myproc.rank, PMIx_Error_string(rc));
67+
goto done;
68+
}
69+
nprocs = val->data.uint32;
70+
PMIX_VALUE_RELEASE(val);
71+
if (nprocs < 6) {
72+
if (0 == myproc.rank) {
73+
fprintf(stderr, "This example with add-members requires a minimum of 6 processes\n");
74+
}
75+
exit(1);
76+
}
77+
fprintf(stderr, "Client %s:%d job size %u\n", myproc.nspace, myproc.rank, nprocs);
78+
79+
/* call fence to sync */
80+
PMIX_PROC_CONSTRUCT(&proc);
81+
PMIX_LOAD_PROCID(&proc, myproc.nspace, PMIX_RANK_WILDCARD);
82+
if (PMIX_SUCCESS != (rc = PMIx_Fence(&proc, 1, NULL, 0))) {
83+
fprintf(stderr, "Client ns %s rank %d: PMIx_Fence failed: %d\n", myproc.nspace, myproc.rank,
84+
rc);
85+
goto done;
86+
}
87+
88+
//<EG BEGIN ID="bootstrap">
89+
/* rank=0 and 3 bootstrap a new group */
90+
if (0 == myproc.rank || 3 == myproc.rank) {
91+
fprintf(stderr, "%d executing Group_construct\n", myproc.rank);
92+
PMIX_INFO_LOAD(&info[0], PMIX_GROUP_ASSIGN_CONTEXT_ID, NULL, PMIX_BOOL);
93+
// two procs are performing the bootstrap
94+
n = 2;
95+
PMIX_INFO_LOAD(&info[1], PMIX_GROUP_BOOTSTRAP, &n, PMIX_SIZE);
96+
97+
PMIX_DATA_ARRAY_CONSTRUCT(&dry, 1, PMIX_PROC);
98+
parray = (pmix_proc_t*)dry.array;
99+
if (3 == myproc.rank) {
100+
fprintf(stderr, "[%u]: Adding members\n", myproc.rank);
101+
PMIX_LOAD_PROCID(&parray[0], myproc.nspace, 4);
102+
} else {
103+
fprintf(stderr, "[%u]: Adding members\n", myproc.rank);
104+
PMIX_LOAD_PROCID(&parray[0], myproc.nspace, 5);
105+
}
106+
PMIX_INFO_LOAD(&info[2], PMIX_GROUP_ADD_MEMBERS, &dry, PMIX_DATA_ARRAY);
107+
PMIX_DATA_ARRAY_DESTRUCT(&dry);
108+
rc = PMIx_Group_construct("ourgroup", &myproc, 1, info, 3, &results, &nresults);
109+
if (PMIX_SUCCESS != rc) {
110+
fprintf(stderr, "Client ns %s rank %d: PMIx_Group_construct failed: %s\n",
111+
myproc.nspace, myproc.rank, PMIx_Error_string(rc));
112+
goto done;
113+
}
114+
} else if (4 == myproc.rank || 5 == myproc.rank) {
115+
fprintf(stderr, "%d executing Group_construct\n", myproc.rank);
116+
fflush(stderr);
117+
rc = PMIx_Group_construct("ourgroup", NULL, 0, NULL, 0, &results, &nresults);
118+
if (PMIX_SUCCESS != rc) {
119+
fprintf(stderr, "Client ns %s rank %d: PMIx_Group_construct failed: %s\n",
120+
myproc.nspace, myproc.rank, PMIx_Error_string(rc));
121+
goto done;
122+
}
123+
}
124+
//<EG END ID="bootstrap">
125+
fprintf(stderr, "%d GROUP CONSTRUCT COMPLETE\n", myproc.rank);
126+
fflush(stderr);
127+
128+
if (0 == myproc.rank || 3 == myproc.rank ||
129+
4 == myproc.rank || 5 == myproc.rank) {
130+
/* we should have a single results object */
131+
if (NULL != results) {
132+
cid = 0;
133+
for (n=0; n < nresults; n++) {
134+
if (PMIX_CHECK_KEY(&results[n], PMIX_GROUP_CONTEXT_ID)) {
135+
PMIX_VALUE_GET_NUMBER(rc, &results[n].value, cid, size_t);
136+
fprintf(stderr, "%d Group construct complete with status %s KEY %s CID %lu\n",
137+
myproc.rank, PMIx_Error_string(rc), results[n].key, (unsigned long) cid);
138+
} else if (PMIX_CHECK_KEY(&results[n], PMIX_GROUP_MEMBERSHIP)) {
139+
parray = (pmix_proc_t*)results[n].value.data.darray->array;
140+
psize = results[n].value.data.darray->size;
141+
fprintf(stderr, "[%u] NUM MEMBERS: %u MEMBERSHIP:\n", myproc.rank, (unsigned)psize);
142+
for (m=0; m < psize; m++) {
143+
fprintf(stderr, "\t%s:%u\n", parray[m].nspace, parray[m].rank);
144+
}
145+
}
146+
}
147+
} else {
148+
fprintf(stderr, "%d Group construct complete, but no results returned\n", myproc.rank);
149+
}
150+
if (NULL == parray) {
151+
fprintf(stderr, "%d NULL proc array\n", myproc.rank);
152+
goto done;
153+
}
154+
155+
fprintf(stderr, "%d Executing group fence\n", myproc.rank);
156+
PMIX_PROC_CONSTRUCT(&proc);
157+
PMIX_LOAD_PROCID(&proc, "ourgroup", PMIX_RANK_WILDCARD);
158+
if (PMIX_SUCCESS != (rc = PMIx_Fence(&proc, 1, NULL, 0))) {
159+
fprintf(stderr, "Client ns %s rank %d: PMIx_Fence failed: %d\n", myproc.nspace, myproc.rank,
160+
rc);
161+
goto done;
162+
}
163+
164+
fprintf(stderr, "%d executing Group_destruct\n", myproc.rank);
165+
rc = PMIx_Group_destruct("ourgroup", NULL, 0);
166+
if (PMIX_SUCCESS != rc) {
167+
fprintf(stderr, "Client ns %s rank %d: PMIx_Group_destruct failed: %s\n", myproc.nspace,
168+
myproc.rank, PMIx_Error_string(rc));
169+
goto done;
170+
}
171+
}
172+
173+
done:
174+
/* finalize us */
175+
fprintf(stderr, "Client ns %s rank %d: Finalizing\n", myproc.nspace, myproc.rank);
176+
if (PMIX_SUCCESS != (ret = PMIx_Finalize(NULL, 0))) {
177+
fprintf(stderr, "Client ns %s rank %d:PMIx_Finalize failed: %s\n", myproc.nspace,
178+
myproc.rank, PMIx_Error_string(ret));
179+
rc = ret;
180+
} else {
181+
fprintf(stderr, "Client ns %s rank %d:PMIx_Finalize successfully completed\n",
182+
myproc.nspace, myproc.rank);
183+
}
184+
fprintf(stderr, "%s:%d COMPLETE\n", myproc.nspace, myproc.rank);
185+
fflush(stderr);
186+
return (rc);
187+
}

0 commit comments

Comments
 (0)