3.5.4. Host Environments and PMIx v7
For developers of resource managers, launchers, schedulers, and other programs that embed the PMIx server library. PRRTE v5 already does everything listed here, and serves as an example; the contract applies to any host.
This page summarizes. The detail is in Host Environments, which each section links to. A system scheduler should also read Schedulers and PMIx v7.
3.5.4.1. The shape of the change
The division of work is written down. PMIx authenticates each connection and makes the inexpensive first checks. The host is the final arbiter of every connection and every operation passed up to it. See What a Host Must Do.
PMIx tells the host who is asking on every up-call made for a client or tool, and leaves the decision to the host.
A host that has not been updated fails safe. The identity in an up-call is always that of the process connected to the server, as the kernel reported it. A host that authorizes on it judges the process actually there; a relayed claim arrives beside it, not in its place.
3.5.4.2. Building
Build the host against the PMIx major release it runs with. The library is given only a pointer to the host’s
pmix_server_module_t, and cannot tell which release’s definition of it the host was built with.New module entries since v5.0:
tool_connected2,log2,abort2, andresource_block. The oldertool_connected,log, andabortare called only when the new ones areNULL. See pmix_server_module_t(5).Capability flags in
pmix_version.hsay what this library does, for a host’sconfigureto test: for examplePMIX_CAP_REQUESTER_ID,PMIX_CAP_ORIGIN_ID,PMIX_CAP_DMODEX_REQUEST2,PMIX_CAP_ACCESS_CHECK, andPMIX_CAP_DEREG_DEPARTS.A host that uses PMIx’s internal headers, as PRRTE does, has no compatibility guarantee across release series (PMIx and PRRTE Releases).
3.5.4.3. Starting the server
Ask for tool support (
PMIX_SERVER_TOOL_SUPPORT) if tools are to connect. Without it, none can - and that includes the tools that find a system or session server, and the debugger that reads aPMIX_LAUNCHER_RENDEZVOUS_FILE. A scheduler or system controller (PMIX_SERVER_SCHEDULER,PMIX_SERVER_SYS_CONTROLLER) is the exception: it serves only tools, so the role implies tool support, and passingPMIX_SERVER_TOOL_SUPPORTfalse with it fails the init. A launcher passes it toPMIx_tool_init. See Starting the Server, and Settings and What They Enable for every combination.Decide on clients and tools of earlier releases.
PMIX_SERVER_LEGACY_PEERStakes them, if PMIx was built--enable-legacy-clients; the site’sptl_base_legacy_clientsdoes the same when the host does not say. Without either, they are refused, and an older client runs as a singleton. Passing the attribute false overrides the site, and passing it true in a build without the support fails init, so a host that has no reason to decide should not pass it.Remote connections are gone.
PMIX_SERVER_REMOTE_CONNECTIONSset to true fails init.The local socket is owner-only. To take tools of other users, pass
PMIX_SERVER_ALLOW_FOREIGN_TOOLSand widen the socket (PMIX_SOCKET_MODE, withPMIX_USERIDandPMIX_GRPIDnaming its owner and group). A tool of the user named asPMIX_USERIDis taken withoutPMIX_SERVER_ALLOW_FOREIGN_TOOLS, as root’s and the server’s own are.Use a short temporary directory. The server refuses one another user could change, and a socket path is limited to about 100 characters.
A session-level server must name its session (
PMIX_SESSION_RNDZ_IDorPMIX_SESSION_ID).Reporting the address (
PMIX_TCP_REPORT_URI):-and+now give the local socket’s address;tcp-andtcp+the TCP address. A pipe gets both when the server listens on TCP, TCP first.After a failed init, do not call
PMIx_server_finalize. The server library does not change the host’s own environment.
3.5.4.4. Registering jobs and processes
Register each job’s owner (
PMIX_USERID,PMIX_GRPID) withPMIx_server_register_nspace, on every node that hosts part of the job. A host running as a service account must: otherwise the owner falls back to the clients’ user, and on a node with none of the job’s processes to the server’s own user.Register an access list (
PMIX_ACCESS_PERMISSIONS) to widen access beyond the owner, and accept one in spawn requests.Give users’ groups where the kernel cannot. A process’s groups come from the kernel for the local socket. For a request relayed from another node, a TCP peer, or a platform where the kernel gives no list, PMIx uses the groups the host gives with
PMIX_USER_GRPIDS- atregister_nspacefor the job’s owner, orPMIx_server_register_resourcesfor another user. PMIx never looks groups up itself.Prepare every local process before starting it, with
PMIx_server_register_clientandPMIx_server_setup_fork. The environmentsetup_forkproduces carries the local socket’s address and a connection token for that process; the token is bound to the first process that connects with it.Re-registering a client updates it and returns
PMIX_UPDATED. It used to returnPMIX_ERR_DUPLICATE_KEY.Sessions can be registered and deregistered on their own (
PMIx_server_register_session,PMIx_server_deregister_session). Registering a known session again updates it.A rank the host deregisters leaves its job’s collectives, and is counted back in if the host restarts it under the same rank (
PMIX_CAP_DEREG_DEPARTS).
3.5.4.5. Accepting connections
What the host is told.
client_connected2andtool_connected2receive the verified user and group, the authentication method, the kernel’s pid for the process, andPMIX_TIMEOUT.What the host checks. What PMIx cannot: that the host started this client, with this pid; that this tool’s user may do what it asks.
Answer once, within the timeout.
A tool may arrive with no namespace. Assign one, and keep assigned names unique. Do not register a job under the name a tool gave itself.
Handle
PMIX_ERR_LOST_CONNECTION. It is raised for a process that leaves without finalizing, one accepted that never connected, and one that execs.Client clones are gone: one connection per rank.
PMIX_ALLOW_CLIENT_CLONESis deprecated and ignored.
3.5.4.8. Up-call return codes and groups
Returning
PMIX_OPERATION_SUCCEEDEDfromspawn,lookup,direct_modex,get_credential, orvalidate_credentialis treated as an error.A group ID must be unique, non-empty, and may not name a job the server knows. A group controller should refuse a duplicate with
PMIX_ERR_EXISTS.Joining a group construct that is still collecting must be the same operation, and - when the first call listed every participant - by one of them.
Participant checks apply to clients of every release.
A monitor’s alert may raise only its own code, a monitor code, or a user code; it now carries the monitor’s ID and settings and the heartbeats or checks missed.