25.2.248. PMIx_server_init
PMIx_server_init — Initialize the PMIx server support library.
25.2.248.1. SYNOPSIS
#include <pmix_server.h>
pmix_status_t PMIx_server_init(pmix_server_module_t *module,
pmix_info_t info[], size_t ninfo);
25.2.248.1.1. Python Syntax
from pmix import *
foo = PMIxServer()
# the directives is a list of Python ``pmix_info_t`` dictionaries
pydirs = [{'key': PMIX_SERVER_NSPACE,
'value': "SERVER", 'val_type': PMIX_STRING}]
# map is a dictionary of server-module-function keys to Python
# callback implementations (e.g. {'clientconnected': myconnfn, ...})
rc = foo.init(pydirs, map)
25.2.248.2. INPUT PARAMETERS
module: Pointer to a pmix_server_module_t(5) structure containing the host environment’s callback functions — the “function-shipped” server module through which client requests are relayed to the resource manager. Any function the host does not support is indicated by leaving its function pointerNULL; client calls to an unsupported operation return a “not supported” error. PassingNULLformodule(or an empty structure) is permitted and indicates that the host will not support multi-node operations such as PMIx_Fence(3), but still intends to give local clients access to job information.info: Pointer to an array of pmix_info_t(5) structures conveying directives that qualify server initialization (see DIRECTIVES). ANULLvalue is supported when no directives are desired.ninfo: Number of elements in theinfoarray.
25.2.248.3. DESCRIPTION
Initialize the PMIx server support library and register the host environment’s
callback module. PMIx_server_init is called by the process that hosts the
PMIx server — typically a resource manager daemon, launcher, or tool
that will accept client and/or tool connections.
The PMIx server provides a function-shipping approach to the server side of
the protocol: each client-visible PMIx operation is mirrored by an entry in the
pmix_server_module_t structure. When a client request arrives, the PMIx
server library invokes the corresponding host function so that resource
managers can implement server behavior without being burdened with PMIx
internal details. For performance and scalability, host module functions are
required to return quickly and to execute their work asynchronously, completing
each operation by invoking the callback function supplied to them.
Important
A host must run against a PMIx library of the same major
version as the PMIx headers it was compiled against. The library takes
the host’s pmix_server_module_t to be the size and layout defined by
its own headers, and expects the host’s functions to behave as that
version specifies. It cannot check either. New members are added to the
structure only in a new major release. A host that runs against a
library of a different major version is not supported, and its behavior
is undefined.
Ownership rules for the module functions are important: all data passed to a host server function is owned by the PMIx server library and must not be freed by the host, while data returned by the host via a module callback is owned by the host and may be released once the callback returns.
The info array supplies additional information the server may need when
initializing — for example, the namespace and rank to assign to the
server itself, the temporary directory in which to place the rendezvous socket,
or the PMIX_SERVER_TOOL_SUPPORT directive announcing that the daemon is
willing to accept connection requests from tools (see DIRECTIVES).
The PMIx server library is reference counted. Repeated calls to
PMIx_server_init are permitted and simply increment the reference count;
if a module is provided on a later call and none had been set previously,
the library adopts it. Each successful call must be balanced by a matching call
to PMIx_server_finalize(3).
PMIx_server_init is a blocking call that completes synchronously; it does
not take a callback.
Warning
Use PMIx with caution in a setuid or setgid program. PMIx
takes its configuration from the environment and from locations the
environment names: the PMIX_* variables (including those that name
the server to connect to and the directories components are loaded
from), a per-user parameter file under $HOME, and the session
directory under $TMPDIR. In a setuid or setgid program these are
set by the user who ran it, not by the program’s owner. PMIx cannot
tell the difference, and it cannot ignore them, because the
environment is how a process finds its server. A setuid or setgid
program may need to set or clear these variables itself before calling
this function.
25.2.248.4. DIRECTIVES
The following attributes are relevant to this operation. Support for any given
attribute is optional and depends on how the PMIx implementation was built and
on the capabilities of the host environment. All are passed in the info
array.
Several behaviors need more than one attribute, or a build option or site setting as well: serving tools, being found as a scheduler, taking clients of earlier releases. Settings and What They Enable sets out each combination and what it does, and Starting the Server what to pass for each goal.
25.2.248.4.1. Server identity and directories
PMIX_SERVER_NSPACE(char*) — namespace to assign to this PMIx server. If not given, thePMIX_SERVER_NSPACEenvironment variable is used - a launcher that starts the server itself sets it - and otherwise the namespace ispmix-server.PMIX_SERVER_RANK(pmix_rank_t) — rank to assign to this PMIx server. If not given, thePMIX_SERVER_RANKenvironment variable is used, and otherwise the server’s pid.PMIX_SERVER_TMPDIR(char*) — temporary directory in which the server is to place its session-level files. The rendezvous files tools use to find it go underPMIX_SYSTEM_TMPDIRinstead (see Rendezvous files). If not given, thePMIX_SERVER_TMPDIRenvironment variable is used, thenTMPDIR,TEMPandTMP, and finally/tmp. The server’s local socket is made here too, and its path must fit a socket address (see The local socket). The directory must already exist. Its path is checked at init, with the system tmpdir’s: see The tmpdirs.PMIX_SYSTEM_TMPDIR(char*) — temporary directory beneath which the server places its rendezvous files: those tools find it by, inpmix-user.<uid>, and its system-level file (system server, scheduler, system controller), which a root server given no system tmpdir puts in/run/pmixinstead. If not given, thePMIX_SYSTEM_TMPDIRenvironment variable is used, then the same fallbacks as forPMIX_SERVER_TMPDIR.
25.2.248.4.2. Role and support attributes
PMIX_SERVER_TOOL_SUPPORT(bool) — the host is willing to accept connection requests from tools, and rendezvous files naming the server by PID and by namespace are written (see Rendezvous files). Without it, every tool is refused withPMIX_ERR_NOT_SUPPORTED- including one a client or job script starts, which has the server’s address in its environment. The files are readable only by the host’s user ID, and tools running under other user IDs are refused, unlessPMIX_SERVER_ALLOW_FOREIGN_TOOLSis also given; such tools also need the server’s local socket widened to them (PMIX_SOCKET_MODE,PMIX_USERID,PMIX_GRPID), which a server does not do by itself. Tools of root, and of the user named byPMIX_USERID, are not refused this way.PMIX_SERVER_SYSTEM_SUPPORT(bool) — the server is the node’s system server: tools asking for the system server find it through a system-level rendezvous file. Takes effect only together withPMIX_SERVER_TOOL_SUPPORT. As for that attribute, the file is readable only by the host’s user ID unlessPMIX_SERVER_ALLOW_FOREIGN_TOOLSis given.PMIX_SERVER_SESSION_SUPPORT(bool) — the server is the session server of its session, found by tools through a rendezvous file named byPMIX_SESSION_RNDZ_ID, else byPMIX_SESSION_ID: without eitherPMIx_server_initfails. Servers of different sessions share a node; a second server of one session on a node failsPMIx_server_init. Takes effect only together withPMIX_SERVER_TOOL_SUPPORT.PMIX_SESSION_RNDZ_ID(uint32_t) — the ID naming the session server’s rendezvous file, unique among the session servers on a node; seePMIX_SERVER_SESSION_SUPPORT. The host chooses it, so that it need not changePMIX_SESSION_ID, the session’s ID as the scheduler assigned it, to name the file - which is named byPMIX_SESSION_IDwhen this is not given.PMIX_SERVER_GATEWAY(bool) — the server is acting as a gateway that can relay operations (e.g., logging) requiring routing to other nodes.PMIX_SERVER_SCHEDULER(bool) — the server is hosting the system scheduler (workload manager) and expects to receive scheduler-related requests. The runtimes that find it connect as tools, so it impliesPMIX_SERVER_TOOL_SUPPORT; passing that attribute false as well fails the init.PMIX_SERVER_SYS_CONTROLLER(bool) — the server is hosting the system controller. As forPMIX_SERVER_SCHEDULER, the processes that find it are tools: it impliesPMIX_SERVER_TOOL_SUPPORT, and passing that attribute false as well fails the init.PMIX_SERVER_LEGACY_PEERS(bool) — take (or refuse) connections from clients and tools of PMIx releases before v7.0, which connect over TCP. Overrides theptl_base_legacy_clientsMCA parameter (andptl_base_legacy_peers, which sets it in a build with legacy clients), which defaults tofalse. With it, the server listens on a loopback interface for them and gives its clients the TCP address in the variables those releases read; without it the server has no TCP listener, and a client or tool of an earlier release cannot connect. Only a PMIx configured--enable-legacy-clientscan take them (the default is not to): in one without it, asking fails the init - by this attribute, or byptl_base_legacy_clients;ptl_base_legacy_peersasks such a build for nothing. A host that does not pass it leaves the decision to the site’s parameter.PMIX_SERVER_REMOTE_CONNECTIONS(bool) — deprecated. A server takes connections from its own host only:truefails the init, andfalsechanges nothing. A client or tool does not dial a server at an address that is not its own host’s, and the server refuses a TCP peer at such an address.PMIX_SERVER_ALLOW_FOREIGN_TOOLS(bool) — mark the rendezvous files as readable by all users and allow tools running under user IDs other than that of the server to connect. Defaults tofalse: without it, a connection request from a tool under another user ID is refused. The host retains ultimate authority over such connections and may restrict what foreign tools are permitted to do (for example, limiting them to queries) as dictated by host policy. The permissions of an existing directory the files are placed in are never changed, so a directory other users cannot search still keeps their tools out (see Rendezvous files).
25.2.248.4.3. Connection attributes
PMIX_SOCKET_MODE(uint32_t) — the mode (nine permission bits) of the server’s local socket. Default 0600: only the server’s user can connect. See The local socket.PMIX_USERID(uint32_t, or a user name as char*) — the owner of the server’s local socket. Default: the server’s user. Its directory stays the server’s. Tools of this user are taken withoutPMIX_SERVER_ALLOW_FOREIGN_TOOLS, as the server’s own user’s are.PMIX_GRPID(uint32_t, or a group name as char*) — the group of the server’s local socket and of its directory. Default: the server’s effective group.PMIX_LAUNCHER_RENDEZVOUS_FILE(char*) — full pathname of an additional rendezvous file in which the server is to write its contact information — for example, so that a debugger can find a launcher. Missing directories on the path are created. The file is written whether or notPMIX_SERVER_TOOL_SUPPORTis given; a tool that reads it is accepted only with that attribute.PMIX_CONNECT_TO_SYSTEM,PMIX_CONNECT_SYSTEM_FIRST,PMIX_CONNECT_TO_SCHEDULER,PMIX_CONNECT_TO_SYS_CONTROLLER(bool),PMIX_CONNECTION_ORDER(char*),PMIX_SERVER_PIDINFO(pid_t),PMIX_TOOL_ATTACHMENT_FILE(char*) — connect the server to the server these name, as for PMIx_tool_init(3), and make it this server’s own server — the one, for example, an allocation request is sent to. A malformed value fails the init. WithPMIX_TOOL_CONNECT_OPTIONALa server that cannot be reached is not an error.PMIX_TCP_REPORT_URI(char*) — report the server’s URI:-writes the local socket’s URI to stdout and+to stderr;tcp-andtcp+write the TCP URI instead, for a reader of a release before v7.0, and fail the init when the server takes no legacy clients and so has no TCP listener. An integer writes to that open file descriptor, which is then closed: the TCP URI’s string and then the local socket’s, each ending in a NUL, when the server takes legacy clients, else the local socket’s alone. Anything else is taken as the name of a file to write it to.PMIX_TCP_IF_INCLUDE(char*) — comma-delimited list of devices and/or CIDR notation to listen on. Cannot be combined withPMIX_TCP_IF_EXCLUDE.PMIX_TCP_IF_EXCLUDE(char*) — comma-delimited list of devices and/or CIDR notation not to listen on.PMIX_TCP_IPV4_PORT(int or char*) — IPv4 port to listen on, or (as a string) a comma-delimited list of ports and ranges to try in order.0or-1lets the kernel choose.PMIX_TCP_IPV6_PORT(int or char*) — asPMIX_TCP_IPV4_PORT, for IPv6.PMIX_TCP_DISABLE_IPV4(bool) — do not listen on IPv4 addresses.PMIX_TCP_DISABLE_IPV6(bool) — do not listen on IPv6 addresses.
25.2.248.4.4. Topology and behavior attributes
PMIX_TOPOLOGY2(pmix_topology_t) — pointer to an implementation-specific description of the local node topology.PMIX_SERVER_SHARE_TOPOLOGY(bool) — the server is to scalably expose the node topology to its local clients (for example, via shared-memory backing stores), cleaning up any artifacts at finalize.PMIX_HOMOGENEOUS_SYSTEM(bool) — the nodes in the system are topologically identical, so the server need not compute or exchange per-node topology descriptions.PMIX_SERVER_ENABLE_MONITORING(bool) — enable the internal monitoring capabilities of the PMIx server.PMIX_EXTERNAL_PROGRESS(bool) — the host will progress the PMIx library as needed via calls toPMIx_Progressrather than PMIx spawning its own internal progress thread.PMIX_SINGLETON(char*) — the server is operating in support of a singleton process; the value is thenspace.rankstring of that singleton.PMIX_IOF_LOCAL_OUTPUT(bool) — write output streams to the server’s own stdout/stderr.PMIX_BIND_PROGRESS_THREAD(char*) — comma-delimited ranges of CPUs to which the internal PMIx progress thread is to be bound.PMIX_BIND_REQUIRED(bool) — return an error if the internal PMIx progress thread cannot be bound as requested.PMIX_EXTERNAL_AUX_EVENT_BASE(void*) — pointer to anevent_basethe library is to use for auxiliary functions (e.g., capturing signals) that would otherwise interfere with the host.
25.2.248.5. RETURN VALUE
Returns PMIX_SUCCESS on success. On error, a negative value corresponding
to a PMIx error constant is returned, including:
PMIX_ERR_INIT— the PMIx server library could not be initialized (for example, a required transport was explicitly disabled, or a tight race between concurrentPMIx_server_initcalls left the library in an unusable state).PMIX_ERR_NOT_SUPPORTED— a provided directive requested behavior the implementation does not support.PMIX_ERR_BAD_PARAM— a directive or an environmental value carried something the implementation cannot use — for example aPMIX_SINGLETONthat is not of the form"nspace.rank", or aPMIX_SERVER_RANKenvironment variable that is not a simple non-negative number.
Any other negative value indicates an appropriate error condition. PMIx error
constants are defined in pmix_common.h.
A failed PMIx_server_init leaves nothing behind. Anything the call had
already brought up is torn down before the error is returned, so the caller may
correct whatever was rejected and call PMIx_server_init again. It must
not call PMIx_server_finalize(3), which
answers PMIX_ERR_INIT because no initialization completed. The one
exception is a failure of the underlying runtime itself, which cannot be
unwound; there the library reports the error and is unusable for the life of
the process.
25.2.248.6. NOTES
PMIx_server_init is intended for use only by processes acting as a PMIx
server — processes that host the server support library on behalf of a
resource manager or launcher. Client processes call
PMIx_Init(3) and tools call
PMIx_tool_init(3) instead.
After a successful PMIx_server_init, the host is expected to register each
participating namespace with
PMIx_server_register_nspace(3) and
each local client with
PMIx_server_register_client(3) before
launching that client, then set up the client’s environment with
PMIx_server_setup_fork.
25.2.248.6.1. The local socket
Every server listens on a local (AF_UNIX) socket, where the clients and
tools of this release on its host connect. It listens on TCP, on a loopback
interface, only for peers of earlier releases (PMIX_SERVER_LEGACY_PEERS).
The kernel identifies the process at
each end of a connection to the local socket: the server knows each peer’s
uid, gid and pid, and each peer can check the server’s uid and gid.
The socket is pmix-<pid>, in a directory of its own the server makes
beneath PMIX_SERVER_TMPDIR: <tmpdir>/pmix-XXXXXX. By default the
directory is mode 0700 and the socket 0600, so only the server’s user can
connect. A host that wants other users’ processes to connect - a client it
starts under another user ID, a tool it allows with
PMIX_SERVER_ALLOW_FOREIGN_TOOLS - must widen it: PMIX_SOCKET_MODE
sets the socket’s mode, and PMIX_USERID and PMIX_GRPID its owner and
group. The directory stays the server’s: it is searchable by whom the mode
lets connect - and by everyone, when the socket is given to another user -
and readable by no one but the server’s user. The server never widens it on
its own.
The socket’s path must fit a socket address - about 100 characters. A
temporary directory too long for it makes PMIx_server_init fail; there
is no fallback. The default $TMPDIR on macOS is too long: pass a shorter
PMIX_SERVER_TMPDIR.
The server removes the socket and its directory at finalize. A server that
is killed leaves them behind, as pmix-XXXXXX directories in its
temporary directory.
PMIx_server_setup_fork gives a child the socket’s address and the
server’s uid and gid in PMIX_SERVER_LOCAL_URI; a child of this release
connects there and nowhere else, and only to a server with that uid and gid.
25.2.248.6.2. Rendezvous files
A tool finds a server through a rendezvous file: a small file holding the server’s URI, version and PID, and the address of its local socket. Which ones are written depends on the directives:
Directive |
File |
Directory |
|---|---|---|
|
the name given |
as given |
|
|
the system directory (below) |
|
|
the system directory |
|
|
the system directory |
|
|
|
|
|
|
A scheduler or system controller writes only its own file; the files under
PMIX_SERVER_TOOL_SUPPORT are written by any other server that gives it.
The tool files go in a directory of the server’s user’s own, whatever the
server’s session tmpdir: pmix-user.<uid> beneath the system tmpdir. The
server makes it mode 0700 - 0711 with PMIX_SERVER_ALLOW_FOREIGN_TOOLS, so
other users’ tools can open the files at their names but not list them. One
already there is used only if it is a directory of the server’s user, not a
link, that no other user can write; anything else is reported, and the server
runs on without writing its tool files. The system directory is
PMIX_SYSTEM_TMPDIR when the host names one; otherwise /run/pmix for a
root server (/var/run/pmix where there is no /run), and the server’s
pmix-user directory for any other.
By default each file is readable only by the server’s user ID. With
PMIX_SERVER_ALLOW_FOREIGN_TOOLS it is also readable by all users.
A file at the same name that no running server holds is removed and replaced.
One a running server holds belongs to that server, and PMIx_server_init
fails rather than take it over: one system server, scheduler or system
controller per node, one session server per session on a node, and one server
per namespace. A server whose host names no namespace takes
pmix-server-<hostname>-<pid>.
A file named by PMIX_TCP_REPORT_URI is not a rendezvous file and none of
the above applies to it. A file already at that name is reused only if the
server’s user owns it; one of any other user’s - root’s too, unless the server
is root - fails PMIx_server_init. The file, reused or new, is given mode
0644 - readable by all, writable only by the server’s user - and is removed at
finalize.
25.2.248.6.3. The tmpdirs
Before it makes anything in them, the server checks the path of its session
and system tmpdirs, links resolved, from /. Every directory above a tmpdir
must be owned by root, the server’s user, or a user from outside the server’s
user namespace, and must not be writable by anyone else unless it has the
sticky bit, or its group’s only member is its owner. The tmpdir itself may be
any user’s - a root server is often given a directory it made for the job’s
user - provided only its owner can write it; or else it must pass the rule
for the directories above, as /tmp does. Otherwise another user could
move the server’s directories aside, and its clients would fail to connect.
A path that fails makes PMIx_server_init fail, with a message naming the
directory. The ptl_base_tmpdir_check MCA parameter changes that:
refuse (the default), warn - report it and continue - or silent,
not to check. The last two are for a directory no other user shares that the
rule misjudges, such as a Kubernetes emptyDir mounted at /tmp.