25.2.277. PMIx_tool_init

PMIx_tool_init — Initialize the PMIx tool library.

25.2.277.1. SYNOPSIS

#include <pmix_tool.h>

pmix_status_t PMIx_tool_init(pmix_proc_t *proc,
                             pmix_info_t info[], size_t ninfo);

25.2.277.1.1. Python Syntax

from pmix import *

foo = PMIxTool()
# the directives is a list of Python ``pmix_info_t`` dictionaries
pydirs = [{'key': PMIX_TOOL_NSPACE,
           'value': "mytool", 'val_type': PMIX_STRING},
          {'key': PMIX_TOOL_RANK,
           'value': 0, 'val_type': PMIX_PROC_RANK}]
rc, myname = foo.init(pydirs)

25.2.277.2. INPUT PARAMETERS

  • info: Pointer to an array of pmix_info_t(5) structures conveying directives that qualify the initialization and any subsequent server connection (see DIRECTIVES). A NULL value is supported if no directives are desired.

  • ninfo: Number of elements in the info array.

25.2.277.3. OUTPUT PARAMETERS

  • proc: Pointer to a pmix_proc_t(5) structure in which the tool’s namespace and rank are to be returned. Unlike PMIx_Init(3), the initial call to PMIx_tool_init requires a non-NULL value here so that the assigned identifier can be returned; passing NULL on the first call returns PMIX_ERR_BAD_PARAM.

25.2.277.4. DESCRIPTION

Initialize the PMIx tool library, returning the process identifier assigned to this tool in the provided pmix_proc_t struct.

When called, the PMIx tool library will check for the required connection information of a local PMIx server and, unless directed otherwise, will attempt to establish a connection to it. If the information is not found, or the server connection fails, then an appropriate error constant will be returned — unless the caller has requested that the connection be treated as optional (see PMIX_TOOL_CONNECT_OPTIONAL) or has requested that no connection be attempted at all (see PMIX_TOOL_DO_NOT_CONNECT).

If successful, the function will return PMIX_SUCCESS and will fill the provided structure with the server-assigned namespace and rank of the tool. If the tool self-assigns its identity (because it is not connecting to a server, or because connection was optional and failed), the returned namespace is synthesized from the tool’s hostname and process ID and the rank is set to zero.

A process may declare itself to be a launcher (via PMIX_LAUNCHER) or the system scheduler (via PMIX_SERVER_SCHEDULER), in which case the tool library additionally initializes the server-side infrastructure so the process can host its own PMIx server and spawn or accept connections from clients.

25.2.277.4.1. Reference counting

The PMIx tool library is reference counted, and so multiple calls to PMIx_tool_init are allowed. Thus, one way to obtain the namespace and rank of the process is to simply call PMIx_tool_init with a non-NULL proc parameter. Each call must be balanced by a matching call to PMIx_tool_finalize(3); only the last such finalize actually tears the library down.

Warning

Use PMIx with caution in a setuid or setgid program. PMIx takes its configuration from the environment and from locations the environment names: the PMIX_* variables (including those that name the server to connect to and the directories components are loaded from), a per-user parameter file under $HOME, and the session directory under $TMPDIR. In a setuid or setgid program these are set by the user who ran it, not by the program’s owner. PMIx cannot tell the difference, and it cannot ignore them, because the environment is how a process finds its server. A setuid or setgid program may need to set or clear these variables itself before calling this function.

25.2.277.5. DIRECTIVES

The following attributes are relevant to this operation. Support for any given attribute is optional and depends on how the PMIx implementation was built and on the capabilities of the host environment.

25.2.277.5.1. Tool identity attributes

  • PMIX_TOOL_NSPACE (char*) — namespace to use for this tool. Must be accompanied by PMIX_TOOL_RANK; providing one without the other returns an error. If not given, the identity is taken from the PMIX_NAMESPACE and PMIX_RANK environment variables (set when the tool was launched by a PMIx-enabled daemon, and inherited by a tool that a job script starts) or, when those are absent, is assigned by the server’s host. A client takes those variables out of its environment when it initializes, so a tool it starts after that has none of them (see PMIx_Init(3)).

    A server accepts one connection per namespace and rank, and the first process to connect holds it:

    • a tool that takes its namespace and rank from its environment can find them in use - by the script that started it, or by another tool it started - or held for the first client process that connected as that rank, even after that process finalized or exited, when the tool is another process. The server refuses the connection with PMIX_ERR_EXISTS, and the tool connects again without an identity, to be given one by the host: it comes up under a namespace of its own. If the host does not give it one, PMIx_tool_init returns PMIX_ERR_EXISTS. Tools of releases before this one do not connect again, and PMIx_tool_init returns PMIX_ERR_EXISTS;

    • a tool given PMIX_TOOL_NSPACE and PMIX_TOOL_RANK keeps that identity. If it is in use, PMIx_tool_init returns PMIX_ERR_EXISTS;

    • a tool that takes a job script’s rank before the script’s application connects holds the rank, and the application is then refused with PMIX_ERR_EXISTS. The server cannot tell such a tool from a tool that is the script’s only PMIx process. Where a script runs a tool alongside its application, start the tool without PMIX_NAMESPACE and PMIX_RANK in its environment, or give it PMIX_TOOL_NSPACE and PMIX_TOOL_RANK.

    A tool that joins a namespace - a client’s rank, or a namespace another tool of its name holds - must use that namespace’s wire format: every process of a namespace is sent the same. A tool whose library does not have it is refused with PMIX_ERR_NOT_SUPPORTED.

  • PMIX_TOOL_RANK (uint32_t) — rank of this tool within its namespace.

  • PMIX_LAUNCHER (bool) — the tool is a launcher; causes the server-side infrastructure to be initialized, including a listener. Which rendezvous files it writes, and whether it accepts the tools that read them, is decided as for a server, by the attributes in Launcher attributes and those PMIx_server_init(3) takes - among them PMIX_SERVER_TOOL_SUPPORT, without which the launcher refuses every tool that connects to it. See Settings and What They Enable.

  • PMIX_SERVER_SCHEDULER (bool) — the process is hosted by the system scheduler and intends to act as the system-level server. The runtimes that find it connect as tools, so it implies PMIX_SERVER_TOOL_SUPPORT; passing that attribute false as well fails the init.

25.2.277.5.2. Connection attributes

These attributes govern whether and how the tool rendezvous with a PMIx server during initialization.

  • PMIX_TOOL_DO_NOT_CONNECT (bool) — the tool wants to use internal PMIx support but does not want to connect to a server. The tool self-assigns its identity.

  • PMIX_TOOL_CONNECT_OPTIONAL (bool) — the tool shall connect to a server if one is available, but otherwise shall continue (self-assigning its identity) rather than returning an error. This covers a connection that is attempted and fails; it does not cover a malformed connection directive, which is reported as PMIX_ERR_BAD_PARAM whether or not the connection is optional.

  • PMIX_CONNECT_TO_SYSTEM (bool) — connect solely to the system-level PMIx server.

  • PMIX_CONNECT_SYSTEM_FIRST (bool) — preferentially look for a system-level PMIx server first, then fall back to a server identified by another attribute.

  • PMIX_SERVER_URI (char*) — connect to the server at the given URI. The server must be on this host: a TCP address that is not this host’s is not dialled, and the tool fails with PMIX_ERR_UNREACH.

  • PMIX_SERVER_NSPACE (char*) — connect to the server of the given namespace.

  • PMIX_SERVER_PIDINFO (pid_t) — connect to the server embedded in the process with the given PID.

  • PMIX_TCP_URI (char*) — URI of the server to connect to, or a file name containing it in the form file:<name of file>. For this and PMIX_SERVER_URI, a TCP URI is refused with PMIX_ERR_NOT_SUPPORTED and no connection is made: pass the server’s local-socket (usock://) URI, or, for a server of a release before v7.0, its file.

  • PMIX_TOOL_ATTACHMENT_FILE (char*) — file containing the connection information to be used for attaching to a server.

  • PMIX_CONNECT_TO_SCHEDULER (bool) — connect to the system scheduler.

  • PMIX_CONNECT_TO_SYS_CONTROLLER (bool) — connect to the system controller.

  • PMIX_SERVER_HOSTNAME (char*) — the node on which the target server is located.

  • PMIX_MYSERVER_URI (char*) — the URI of this process’s own local socket, used when the process itself accepts connections. It is the URI to hand a process of this release that is to connect back, such as a launcher (PMIX_LAUNCHER_RNDZ_URI). To give this process’s address to a process of a release before v7.0, use PMIX_SERVER_URI instead.

  • PMIX_CONNECTION_ORDER (char*) — comma-delimited list of connection-target attributes (e.g., PMIX_CONNECT_TO_SCHEDULER, PMIX_CONNECT_TO_SYS_CONTROLLER, PMIX_CONNECT_TO_SYSTEM) defining the order, first to last, in which connections are attempted. Unless the final entry is an “only” flag, the tool falls back to the local server if no listed target succeeds. Each entry must be one of PMIX_CONNECT_SYSTEM_FIRST, PMIX_CONNECT_TO_SYSTEM, PMIX_CONNECT_TO_SCHEDULER or PMIX_CONNECT_TO_SYS_CONTROLLER, given by name or by its string value, with no surrounding spaces; any other entry is PMIX_ERR_BAD_PARAM.

  • PMIX_CONNECT_MAX_RETRIES (uint32_t) — maximum number of times to attempt connecting to the server.

  • PMIX_CONNECT_RETRY_DELAY (uint32_t) — time, in seconds, between connection attempts.

  • PMIX_TIMEOUT (int) — time, in seconds, to wait for the connection to be established. 0, or no PMIX_TIMEOUT, waits as long as the ptl_base_handshake_wait_time parameter says (60 seconds by default, and no bound if it is 0).

  • PMIX_PRIMARY_SERVER (bool) — designate the server being connected to as the tool’s primary server once the connection is established.

25.2.277.5.3. Launcher attributes

These attributes apply to a tool that declares itself a launcher via PMIX_LAUNCHER.

  • PMIX_LAUNCHER_RENDEZVOUS_FILE (char*) — pathname of the file in which the launcher’s connection information is to be stored. The launcher writes it whether or not it connects to a server (PMIX_TOOL_DO_NOT_CONNECT). A file already at the name is replaced if the process that wrote it no longer runs; one that a running process holds fails the init, with a message.

  • PMIX_LAUNCHER_DAEMON (char*) — path to an executable to use as the launcher’s backend daemon, replacing the launcher’s own. The user is responsible for ensuring compatibility with the host launcher.

  • PMIX_EXEC_AGENT (char*) — path to an executable that the launcher’s backend daemons fork/exec in place of the actual application processes. The daemon passes the application’s full command line to the exec agent, which is responsible for exec’ing the application in its own place and does not connect back to the daemon.

  • PMIX_LAUNCH_DIRECTIVES (pmix_data_array_t*) — an array of pmix_info_t containing directives for the launcher; a convenience attribute allowing all directives to be retrieved with a single PMIx_Get call.

25.2.277.5.4. Behavioral attributes

  • PMIX_FWD_STDIN (bool) — forward this process’s stdin to the target processes.

  • PMIX_NOHUP (bool) — any processes started on behalf of the calling tool (or the specified namespace) should continue running after the tool disconnects from its server.

  • PMIX_IOF_LOCAL_OUTPUT (bool) — write forwarded output streams to the local stdout/stderr (the default for tools).

  • PMIX_SERVER_TMPDIR (char*) — temporary directory where the PMIx server places its rendezvous connection files. Defaults to the PMIX_SERVER_TMPDIR environment variable, if set.

  • PMIX_SYSTEM_TMPDIR (char*) — temporary directory used for the system-level server rendezvous point. Defaults to the PMIX_SYSTEM_TMPDIR environment variable, if set.

  • PMIX_BIND_PROGRESS_THREAD (char*) — comma-delimited ranges of CPUs to which the internal PMIx progress thread is to be bound.

  • PMIX_BIND_REQUIRED (bool) — return an error if the internal PMIx progress thread cannot be bound as requested.

  • PMIX_EXTERNAL_AUX_EVENT_BASE (void*) — pointer to an event_base the library is to use for auxiliary functions (e.g., capturing signals) that would otherwise interfere with the host.

25.2.277.6. RETURN VALUE

Returns PMIX_SUCCESS on success. On error, a negative value corresponding to a PMIx error constant is returned, including:

  • PMIX_ERR_INIT — the PMIx library could not be initialized (for example, a usock-only transport was requested, or a tight race between concurrent init calls left the library in an unusable state).

  • PMIX_ERR_BAD_PARAM — an invalid combination of directives was supplied (e.g., a namespace without a rank), a connection directive was malformed (a value of the wrong type, a server URI that does not parse, or an unrecognized PMIX_CONNECTION_ORDER entry) — even when the connection was marked optional, since no connection was attempted — or proc was NULL on the initial call.

  • PMIX_ERR_UNREACH — the requested PMIx server could not be reached and the connection was not optional.

  • PMIX_ERR_OUTDATED — the requested PMIx server uses a version of the PMIx library older than this one can connect to (v3.2), and the connection was not optional. A message naming the server’s version is printed.

  • PMIX_ERR_LOCAL_URI_REQUIRED — the tool was given the server’s TCP address - as a URI, or in a file that names no local socket - and the server is of v7.0 or later, which offers TCP only to tools of earlier releases. A message says to pass the usock:// URI the server reports, or its own report file.

  • PMIX_ERR_NOMEM — the library could not allocate required internal storage.

  • PMIX_ERR_EXISTS — the namespace and rank the tool was to connect as already have a connection, and the tool did not come up under a namespace of its own (see PMIX_TOOL_NSPACE).

  • PMIX_ERR_NOT_SUPPORTED — among other causes, the tool would join a namespace whose processes use a wire format its library does not have.

Any other negative value indicates an appropriate error condition. PMIx error constants are defined in pmix_common.h.

A PMIx_tool_init refused for its input before the library is set up — the PMIX_ERR_BAD_PARAM cases above, other than a malformed connection directive — leaves the library as it found it, and the tool may call PMIx_tool_init again. After any other failure the library cannot be initialized again in this process: later calls return PMIX_ERR_INIT.

25.2.277.7. NOTES

PMIx_tool_init is intended for use by tool processes — debuggers, profilers, and workflow managers — that may or may not have been registered with a PMIx server before being started. Processes that were registered as clients before being started should instead call PMIx_Init(3), and processes hosting a PMIx server on behalf of a resource manager should call PMIx_server_init(3).

A tool is restricted to the hash component of the GDS framework for interacting with a server’s data store.

After initialization, a tool may establish additional server connections with PMIx_tool_attach_to_server(3) and select among them with PMIx_tool_set_server(3).