11.16. Connection Lifecycle: Clients and Tools
This page describes the life of a connection between a PMIx server and one of its clients or tools, from the server opening its listener to the connection ending. It covers:
how a server sets up its listener;
how a server tells the clients it starts where to connect;
how a client or tool opens a connection;
how the server accepts it: the handshake, the security checks, and the call to the host;
how a connection ends: by finalize, by disconnect, by a process dying, and by the host.
Two related subjects are covered elsewhere:
How a tool finds a server - rendezvous files, connection-order directives, the tmpdir search - is in Transport Layer (Client/Server/Tool Connections) for now.
Messages on an open connection - framing, tags, the send and receive handlers - are in Transport Layer (Client/Server/Tool Connections).
The security mechanisms themselves are in Authentication. What a host may do with the identity it is given is in Deciding What a Requester May Do.
11.16.1. The pieces
Piece |
Where |
Role |
|---|---|---|
Listener |
|
Has each transport open its listening socket - the local socket
( |
Connection handler |
|
Reads and checks each new connection’s request, runs the checks, calls the host, replies. |
Connecting side |
|
Builds and sends the connection request, reads the replies. |
Security ( |
|
Creates a credential on the connecting side; checks it, and runs any further exchange, on the server. |
Host up-calls |
|
The host’s decision to accept a connection, and its notice that a process has finished. |
Departure handling |
|
Unwinds a connection and the server state that depends on it. |
There is no listener thread. Accepting, reading a connection request, and every check after it run on the PMIx progress thread.
11.16.2. Rules the connection path follows
Every phase below is written to these rules. They are what makes the path predictable for the peer and for the host.
One identity, settled before anyone acts on it. The peer’s user and group are settled - its credential checked and any further exchange complete - before any policy check is made and before the host is told anything.
A connection changes nothing until it is accepted. Checks run against a scratch copy of the peer’s identity. Nothing the host registered, and nothing another connection relies on, is modified by a connection that may yet be refused.
Every check that can refuse runs before the host is asked. Once the host accepts a connection, only the final replies remain.
The progress thread is never held. No step waits on a peer without a time limit, and a slow peer does not stop the server serving everyone else.
A refused peer is told why. Every refusal sends a status before closing, so the peer can report something more useful than “server unreachable”.
Every failure undoes exactly what was done. Nothing is left behind, and nothing is released twice.
Clients and tools are checked alike. The same checks run in the same order on both paths, except where a check only makes sense for one of them.
The host is told where each item came from: the identity the security component settled, the registration the host made, or what the peer’s PMIx library supplied.
Once the host has accepted a connection, it can learn how it ends: a finalize through
client_finalized, and a lost connection throughPMIX_ERR_LOST_CONNECTIONdelivered to the host’s own event handlers. A client is the host’s child, so the host also sees it end directly.A departure is handled the same way however it is noticed - by the socket closing without a finalize, or by the host deregistering the process. A finalize alone is not a departure: the process may initialize again.
11.16.3. 1. Setting up the listener
A server sets up its listener during PMIx_server_init. A tool sets
one up only if it is a launcher or a scheduler, after it has connected
to its own server.
Setting up happens in two steps:
pmix_ptl_base_create_listeneropens the sockets and publishes how to reach them. Nothing is accepted yet.pmix_ptl_base_start_listening, at the very end of init, starts accepting.
Between the two, the server finishes initializing. A peer that connects in that window waits in the kernel’s backlog until the server starts accepting.
A server listens on its local socket, where every client and tool of
this release on its host connects. It listens on TCP only for those
who use it: clients and tools of earlier releases, when it takes them
(ptl_base_legacy_clients, or the host’s PMIX_SERVER_LEGACY_PEERS;
off by default, and only in a PMIx configured
--enable-legacy-clients). A server takes connections from its own host only: a
host that asks for connections from other hosts
(PMIX_SERVER_REMOTE_CONNECTIONS set to true) fails
PMIx_server_init. A server that cannot open a socket it needs fails
PMIx_server_init too.
Important
A server that does not take legacy clients cannot run clients
of earlier releases. Such a client - an Open MPI 5 process, say, with
its bundled PMIx v5.0 - is given only the local socket’s URI, which it
cannot use. Its connection fails, it logs one PMIX ERROR: BAD-PARAM
line, and it comes up as a singleton: an N-process job runs as N
separate one-process jobs. Nothing the server could set instead makes
every earlier release stop rather than run alone - to such a client, no
server and an unusable one both mean “singleton”. The remedy is the
site’s: build with --enable-legacy-clients and turn legacy clients
on (ptl_base_legacy_clients, or the host’s
PMIX_SERVER_LEGACY_PEERS), or update the clients along with their
servers.
The other direction needs only a build option. In a PMIx configured
--enable-legacy-servers, a client or tool of this release started by
a server of an earlier release (v3.2 on) - a container’s application
under an older resource manager - connects to it over TCP, at the
address and in the wire format the server’s environment or rendezvous
file gives it. Without that option, the default, such a server is
refused: before the process connects, naming the server’s release, and
with PMIX_ERR_NOT_SUPPORTED, which fails a client’s PMIx_Init
rather than making it a singleton. Any server address over TCP is
refused the same way, since only a server of an earlier release is
reached at one. With neither direction built - the default - the library
has no TCP transport at all.
11.16.3.1. The local socket
The local socket is an AF_UNIX socket. The kernel identifies the
process at the far end of each connection to it - its uid, gid and pid -
whatever that process claims.
It is
pmix-<pid>, in a directory of its own that the server makes beneath its temporary directory:<tmpdir>/pmix-XXXXXX.The directory is mode 0700 and the socket 0600: only the server’s own user can connect. A host widens that by passing
PMIX_SOCKET_MODE, andPMIX_USERIDandPMIX_GRPIDfor the socket’s owner and group, toPMIx_server_init. The socket takes its mode and owner while the directory is still closed; the directory, which stays the server’s, is then opened to whom the mode lets in - search permission, never read. The server holds the directory open from the moment it makes it, and makes every change in it through that descriptor - thebindtoo, where/proc/self/fdreaches the directory by it.The path must fit an
AF_UNIXaddress, about 100 characters. A temporary directory too long for it fails init; there is no fallback. The default$TMPDIRon macOS is too long - pass a shorterPMIX_SERVER_TMPDIR.At finalize the server removes the socket and its directory. A server that is killed leaves them behind.
11.16.3.2. Choosing an interface
For TCP, the server considers each network interface that has an IPv4 address
(or IPv6, if enabled), skipping any the if_include / if_exclude
settings rule out. Then:
It listens on loopback only. Only processes on the same node can reach it.
With no loopback interface left, it fails its init. It never takes a public interface instead.
The port is the first free one from ptl_base_ipv4_ports (or the
PMIX_TCP_IPV4_PORT directive), and by default is chosen by the
kernel. Each socket is close-on-exec and non-blocking.
11.16.3.3. Publishing the address
Each address is a URI:
<nspace>.<rank>;usock://<path>
<nspace>.<rank>;tcp4://<address>:<port>
(tcp6:// for IPv6). The server’s URI is its TCP one when it listens
on TCP, and its local socket’s when it does not. The server stores it
under PMIX_SERVER_URI, reports it, and puts it first in its
rendezvous files. A TCP URI reaches older peers, which cannot use the
local socket. PMIX_MYSERVER_URI is always the local socket’s URI:
it is the one to hand a process of this release, such as a launcher
that is to connect back (PMIX_LAUNCHER_RNDZ_URI). To give the
server’s address to a process of a release before v7.0, use
PMIX_SERVER_URI instead. When the
server has one, the local socket’s URI is given beside it where a
process on the host reads it: the environment of the processes the
server starts (see below) and the rendezvous files.
The server can also report the URI on request (ptl_base_report_uri
or the PMIX_TCP_REPORT_URI directive):
Value |
Legacy clients on |
Legacy clients off |
|---|---|---|
|
the local socket’s URI |
the local socket’s URI |
|
the TCP URI, for a reader of an earlier release |
refused at init: there is no TCP listener |
a number (an open pipe, then closed) |
|
|
anything else (a file name) |
line 1 the TCP URI, line 2 the version, then the |
line 1 the local socket’s URI, line 2 the version |
So - and + always print what a client or tool of this release
uses, and tcp-/tcp+ are for a person pointing a reader of an
earlier release at this server. A program reading the pipe and linked to
an earlier release - Open MPI’s, say - takes the first string, as it
always has; one linked to this release takes the second. A file is
left mode 0644: only the server’s user can write it. One already at the
name is reused only if it is that user’s; another user’s stops the
server starting.
11.16.3.4. Rendezvous files
A server that supports tools writes rendezvous files, through which
tools find it, in a directory only its user can write:
<system tmpdir>/pmix-user.<uid> - or, when something other than its
own directory is at that name, $XDG_RUNTIME_DIR/pmix, where that is a
directory of its user’s that no other user can write. A root system
server, scheduler or system controller writes its file in /run/pmix.
A tool looks there,
and in the directories of the other users whose servers it may talk to;
it walks the whole system tmpdir, as an earlier release did, only with
legacy servers on (ptl_base_legacy_servers). The details are in Transport Layer (Client/Server/Tool Connections). Each file holds the URI, the
server’s version, pid, uid and gid, the local socket’s URI, the wire
formats and security mechanisms it accepts, and the lock that shows it
is running (see A server the kernel cannot name). The server removes
the files it wrote when it finalizes.
11.16.3.5. Accepting
Once listening, the progress thread runs the accept handler whenever a connection arrives:
acceptthe connection and make it non-blocking and close-on-exec.Add it to the list of pending connections - accepted, but not yet delivered their connection request.
Start a timer of
ptl_base_connect_ack_timeoutseconds (default 5). A connection that has not delivered its whole request in that time is closed.If
ptl_base_max_pending_connectionsconnections are already pending (default: an eighth of the process’s file-descriptor limit), close the oldest of them. Tools waiting for the host’s decision are not pending connections and are never the ones closed. They have a limit of their own, the same number: once that many tools are waiting, a further tool is refused withPMIX_ERR_OUT_OF_RESOURCE. A user other than the server’s, root and the host’sPMIX_USERIDmay have at most a quarter of that number waiting (at least one); a further tool of that user is refused the same way.Allocate a connection’s request buffer as its bytes arrive: 4 KB at first, doubling up to the length the request names. If the buffers of all pending connections would then pass
ptl_base_max_pending_bytes(default 16 MiB, 0 for no limit), close that connection.
If accept itself fails, the handler retries per-connection errors
at once. When the process is out of descriptors or memory, it pauses
accepting for a second at a time until the condition clears.
At finalize the server closes its listening sockets, closes any pending connections, and removes its rendezvous and URI files.
11.16.4. 2. Telling a client where to connect
A client never searches for its server. The host tells it, through the environment, before the client starts. The host:
registers the job (
PMIx_server_register_nspace);registers the client, with the uid and gid it will run as (
PMIx_server_register_client);calls PMIx_server_setup_fork(3) to add the connection information to the client’s environment;
starts the client with that environment.
A client is local to its server by definition: the host registers it and prepares its environment on the server’s node. It connects to the local socket.
PMIx_server_setup_fork sets:
Variable |
Value |
Used by the client to |
|---|---|---|
|
the client’s identity |
name itself to the server |
|
the local socket’s URI, then |
connect to the local socket, and check that the server there is the one that started it. Clients of earlier releases ignore it |
|
|
present them to the server of that name, however it reaches it, and to no other. Clients of earlier releases ignore it |
|
the server’s TCP URI when it takes clients of earlier releases, else its local socket’s URI |
choose the newest wire format both sides share: the first
variable the client recognizes wins. A client of an earlier
release also connects to this URI. Given the local socket’s,
which it cannot use, it depends on the release: a v4.x or v5.0
client initializes as a singleton and returns
|
|
the server’s version |
know which features the server has |
|
the security mechanisms the server accepts |
choose its own mechanism from that list |
|
the buffer type |
format its messages as the server expects |
|
the datastores the server offers |
choose how to receive job data |
|
the server’s temporary directories |
place its own temporary files |
|
the server’s name for this node |
use the same node name as the server |
A client takes these out of its environment as PMIx_Init begins -
all but PMIX_SERVER_TMPDIR, PMIX_SYSTEM_TMPDIR and
PMIX_HOSTNAME - and keeps them for its own process, where a later
PMIx_Init of the same process reads them. A tool does the same at
PMIx_tool_init. Both also take PMIX_LAUNCHER_RNDZ_URI,
PMIX_SERVER_NSPACE and PMIX_SERVER_RANK, which a launcher sets
for a process it starts itself. The token is overwritten where the
environment held it. A process the client or tool starts once it has
initialized - forked, or forked and exec’d - and a program it execs
inherit none of them, and a process forked from it does not find the
ones it kept. Only the C library’s environment is changed: a language
runtime’s own copy of it, such as Python’s os.environ, still holds
them (the PMIx Python bindings take them out of that copy too).
A server takes none of them out: the host’s environment is the host’s.
The server adds the variables above to each client’s environment one
client at a time, and does not pass its own environment on. The one
place the library does - a process it starts itself, with no host to
start it (PMIx_Spawn from a launcher, or a server whose host has no
spawn up-call) - it passes its environment on less these variables,
and gives the process its own.
A process started with none of these is a singleton. It looks for a
system-level server on its node (the pmix.sys.<host> rendezvous
file) and, if it finds one, connects to it; otherwise it runs without a
server.
11.16.5. 3. Opening a connection
A client connects inside PMIx_Init, and a tool inside
PMIx_tool_init, on the caller’s thread. Each blocks until the
connection is complete or has failed. A tool attaching to another
server later (PMIx_tool_attach_to_server) connects from the
progress thread, without blocking it.
A client takes the local socket’s URI from its environment (or a URI
from a PMIX_SERVER_URI directive to PMIx_Init). A tool finds its
server as described in Transport Layer (Client/Server/Tool Connections); a rendezvous file of this release
gives it the local socket. A tool given a TCP URI directly
(PMIX_SERVER_URI or PMIX_TCP_URI, or the launcher’s
PMIX_LAUNCHER_RNDZ_URI) does not connect, and fails with
PMIX_ERR_NOT_SUPPORTED (help topic tcp-uri-alone): a server of
this release takes it on its local socket, and one of an earlier
release through its rendezvous file (file:<path>), which gives the
server’s version. From there the two do the same:
Open a connection to the URI’s address - only an address of this host. A TCP address that is not this host’s - not a loopback address, and not an address one of its interfaces holds, by the interface list PMIx made at init or by the kernel’s own list - is not dialled, and the process says why. A tool fails with
PMIX_ERR_UNREACH; a client, which was told where its server is, failsPMIx_InitwithPMIX_ERR_COMM_FAILURErather than coming up as a singleton. Each attempt is bounded byptl_base_handshake_wait_time(default 60 seconds). A process given the local socket connects there and nowhere else: if it cannot reach the socket, it fails to initialize, and says why.Check the server (below).
Send the connection request. On the local socket that is an offer, answered by the server’s choice, and then the credential (see The connection request). Each send is bounded by
ptl_base_handshake_wait_timetoo, so a server that stops reading ends the connection withPMIX_ERR_TIMEOUT.Read the replies, each also bounded by
ptl_base_handshake_wait_time.Arm the connection for normal traffic.
11.16.5.1. Checking the server
On the local socket the kernel names the server as well: the uid and gid of the process listening there. A process checks them before it sends its request:
A process the server started was told who the server is, in
PMIX_SERVER_LOCAL_URI. The kernel’s uid and gid must equal those. A process in a user namespace of its own - a rootless container, say - is shown the server’s ids as its namespace maps them; it maps them back to its parent namespace’s, through/proc/self/uid_mapandgid_map, before comparing. A server whose ids its namespace does not map - the kernel shows them as the overflow id, 65534 - cannot be named; see “A server the kernel cannot name” below.Any other process - a tool that found the server through a rendezvous file, say - talks to a server running as its own user or as root, or as a user or group the
ptl_base_server_accessparameter names (users=<list>,groups=<list>). The gid checked is the server’s effective gid.
A server that fails the check is not talked to. The process reports the uid and gid it found and what it expected.
Over TCP, the kernel names the owner of the server’s end of the
connection when the server is on the same host, from its table of TCP
connections - a user, but no group. A process applies the same rule by
user alone: a server running as its own user, as root, or as a user
ptl_base_server_access names. A server the table does not show is
refused (PMIX_ERR_INVALID_CRED), and so is one the process could not
ask about because it is out of descriptors or memory. A table that
cannot be read - /proc mounted with hidepid, or kept from a
container - counts as no table. When the table has no answer:
a connection whose far end is no longer there - the server closed it - fails with
PMIX_ERR_UNREACH;a far end that is not at an address of this host - one reached through another network namespace, say - is refused with
PMIX_ERR_UNREACH, saying the address is not this host’s;a server on this host when the kernel has no table to ask is connected to only when its source vouches for it - see “A server the kernel cannot name” below.
11.16.5.2. A server the kernel cannot name
The kernel’s answer is used whenever it names a real user. It cannot in three cases:
TCP with no table of connections. Linux, macOS and FreeBSD 12 and later have one. Any other platform has none.
TCP with a table that cannot be read, such as
/procmounted withhidepid, or restricted in a container, where neither thesock_diagnetlink interface nor/proc/net/tcpanswers.An id from outside the process’s user namespace. The kernel shows such an id as the overflow id (65534), on the local socket and in the TCP table alike. A process finds it has no entry in
/proc/self/uid_map(orgid_map). This happens in a rootless container whose server runs on the host as root - Slurm’s OCI containers, for one. Linux only: elsewhere every id is named.
Such a server is connected to only when the address came from a source that vouches for it - one the user or the launcher answers for:
A client’s server, always: it is the one named for the client, by the environment it was started with or by its caller.
A server that named itself in the environment of a tool or other process it started.
A URI passed directly (
PMIX_SERVER_URIorPMIX_TCP_URI), including the launcher’sPMIX_LAUNCHER_RNDZ_URI.A file the caller named, as
file:<path>orPMIX_TOOL_ATTACHMENT_FILE, when:it is the file at the name, not a link to one nor one of several names for it, and a regular file only its owner can write;
its owner is the user the process was told to expect, the process’s own user, root, a user
ptl_base_server_accessnames, or an id from outside the process’s user namespace. A process inside a namespace cannot make a file such an id owns, so the file was made outside, and the user named it.
What release the file names, and whether its server is still running, play no part.
A server found by a search, or at a fixed name in a shared directory, never vouches. Neither does anything else.
The user is responsible for passing the right address. A source that vouches is taken at its word: PMIx cannot check, for such a server, who runs it.
A server nothing vouches for is refused with PMIX_ERR_INVALID_CRED,
and the process says to name the server’s file or pass its URI (help
topics server-unnamed and server-unmapped). A server of v7.0 or
later also listens on a local socket, where the kernel names its user
outside a user namespace; a process whose file names that socket
connects there.
A server that runs as nobody, or as a service account, on
the host itself has a mapped id - it is named, not unnamed - so the
normal rule applies: it is refused unless ptl_base_server_access
names that user. That holds too for a client of an earlier-release
server run by a service account.
A process of an earlier release does none of this; the server checks it as any other process on its host - see The native Mechanism for a platform with no table.
11.16.5.3. The connection request
On the local socket, which only peers of this release use, the request is three messages:
The offer, from the peer: a message header, then
the peer’s PMIx version;
the kind of peer (one byte, as in the table below);
its namespace and rank, for the kinds that send them;
the names of its
pseccomponents;the names of its
bfropscomponents, the one it prefers first;the buffer type it prefers (one byte);
the names of its
gdscomponents, the one it prefers first.
Each list is a one-byte count followed by that many names, each ending in a NUL. A list holds at most 16 names (
PMIX_PTL_OFFER_MAX_NAMES) and a name at most 64 bytes (PMIX_PTL_OFFER_MAX_NAMELEN); an offer past either is refused withPMIX_ERR_BAD_PARAM. The offer carries no uid or gid: those are the kernel’s.The choice, from the server: a status, and on success the length of what follows and the chosen names - security mechanism, wire format, buffer type (one byte), datastore. The server chooses:
security: the first of the server’s active mechanisms, in priority order, that the peer offers -
mungewhen both have it, elsenative. If the peer offers none of them it is refused withPMIX_ERR_NOT_SUPPORTED.wire format and buffer type: the namespace’s, if a peer of the namespace has already connected and the kernel’s uid and gid for the peer are those of the rank it names (or, for a rank not yet in the namespace, of the namespace’s owner and every process in it) - a peer that does not offer it is refused with
PMIX_ERR_NOT_SUPPORTED. Otherwise - a new namespace, or a peer of another uid or gid - the first wire format the peer offers that the server has, and the peer’s buffer type.datastore: the first the peer offers that the server has. It is the peer’s own; the first peer’s also becomes its namespace’s default.
The peer refuses a choice it did not offer: its init fails with
PMIX_ERR_NOT_SUPPORTED.The credential, from the peer: a message header, the credential’s length and the credential, made with the chosen security mechanism (see below), and then the information field, packed in the chosen wire format and buffer type.
The server answers the credential with the replies below. The three
messages share one deadline: the connection’s
ptl_base_connect_ack_timeout, from when it was accepted.
Over TCP, which only peers of earlier releases use, the request is one message: a header followed by these fields, in this order. The order is a wire-format contract: a server and its peers can be different PMIx releases, so fields are only ever appended.
# |
Field |
Content |
|---|---|---|
1 |
security mechanism |
name of the |
2 |
credential length |
|
3 |
credential |
created by that component (see below) |
4 |
kind of peer |
one byte: see the table below |
5 |
identity |
depends on the kind: some or all of uid, gid, namespace, rank |
6 |
version |
the peer’s PMIx version |
7 |
wire format |
name of the |
8 |
buffer type |
|
9 |
datastore |
name of the |
10 |
information |
packed |
The kinds of peer:
Value |
Kind |
Sent by |
Identity sent |
|---|---|---|---|
0 |
simple client |
a client started by the server |
namespace, rank |
1, 2 |
legacy tool, legacy launcher |
tools and launchers of old releases only |
uid, gid |
3 |
tool needing an identifier |
a tool that has no namespace or rank of its own |
uid, gid |
4 |
tool given an identifier |
a tool given |
uid, gid, namespace, rank |
5 |
tool started by a server |
a tool whose environment names it ( |
uid, gid, namespace, rank |
6, 7, 8 |
launcher needing / given / started with an identifier |
as 3, 4, 5, for a launcher |
as 3, 4, 5 |
9 |
singleton |
a client with no server of its own, connecting to a system server |
uid, gid, namespace, rank |
10 |
scheduler |
a scheduler |
uid, gid, namespace, rank |
The information field carries, from a client, its pid
(PMIX_PROC_PID, which the client’s PMIx library obtains with
getpid()) and command line (PMIX_CMD_LINE). From a tool it
carries its pid, real and effective uid and gid, its roles
(PMIX_LAUNCHER, PMIX_SERVER_SCHEDULER,
PMIX_SERVER_SYS_CONTROLLER), its command line, and any directives it
was given that are meant for the server.
11.16.5.4. The credential
Each psec component makes its own credential:
native sends the peer’s effective uid and gid. The server takes the uid from its kernel instead where it can, and takes the claim only where its kernel keeps no table of TCP connections and it takes legacy peers: see The native Mechanism.
munge sends a credential encoded by the MUNGE daemon. It carries no payload: the server refuses a
mungecredential that does. On the local socket, where the kernel names the server’s uid, the credential is restricted to that uid (MUNGE_OPT_UID_RESTRICTION), so only a process running as the server’s user can decode it. Over TCP, or where the uid differs between this process’s user namespace and its parent’s, it is unrestricted.
11.16.5.5. The replies
A client reads, in order:
a status: success, or a request to continue with a security exchange;
if requested, the client side of that exchange (no component of this release requests one);
its index in the server’s table of peers.
A tool reads, in order:
if requested, the security exchange;
a status - the host’s decision;
if it asked for one, its assigned namespace and rank;
the server’s namespace and rank;
a final status.
Any status other than success ends the connection, and the peer
reports it. A connection the server closes without sending a status is
reported as PMIX_ERR_UNREACH.
Once the replies are in, the peer marks itself connected, switches the
socket to non-blocking, and arms it for normal traffic. A client then
asks the server for its job information, completing PMIx_Init.
11.16.6. 4. Accepting a connection
Everything in this section runs on the server’s progress thread.
11.16.6.1. Reading the request
The connection handler reads the request as it arrives, without
waiting: each time the socket is readable it takes what is there and
keeps its place. A request larger than PMIX_MAX_CRED_SIZE (128 KB)
is refused. Once the whole request is in, the connection leaves the
pending list and its timer stops.
On the local socket the first message is the offer. The handler takes the peer’s uid and gid from the kernel, reads the offer, checks the version and kind below, and answers with its choice (see The connection request). The connection then goes back to waiting, on the same timer, for the credential, which it reads the same way. Over TCP the one message holds every field.
The handler then reads the fields in order. Two checks apply to every peer:
Kind of peer. An unknown value is refused.
Version. A peer from PMIx v1 or v2.0 is refused with
PMIX_ERR_OUTDATED. See Version Numbers and Binary Compatibility for the supported floors. A peer of any release before v7.0 is refused withPMIX_ERR_NOT_SUPPORTEDunless the server takes legacy clients, and on the local socket whether it does or not: no earlier release has that transport. A peer of v7.0 or later is refused on TCP withPMIX_ERR_LOCAL_URI_REQUIRED: it uses the local socket, where the kernel names it, and the TCP listener exists only to take earlier releases - on their claim, where the kernel cannot name them. The peer says which address it was given and what to use instead (help topicslocal-uri-required-uriandlocal-uri-required-file).
Before the security component is asked, a peer that connected over TCP
from an address that is not this host’s - not a loopback address, and
not the address of one of its interfaces - is refused with
PMIX_ERR_NO_PERMISSIONS. A server takes peers on its own host only.
The rest depends on whether the peer is a client (kinds 0 and 9) or a tool (all the others).
11.16.6.2. Checks for a client
Its components must be available. On the local socket the server chose them from the peer’s offer. Over TCP, the security, wire-format, and datastore components it names must be ones this server has loaded, or it is refused with
PMIX_ERR_NOT_SUPPORTED.The client must be registered. Its namespace, and its rank within it, must have been registered by the host (
PMIx_server_register_client). An unregistered client is refused withPMIX_ERR_INVALID_CRED.Its identity must be the registered one. The security component settles the client’s uid and gid (see Settling the identity), and they must equal the uid and gid the host registered for that rank. On the local socket the uid and gid the kernel reports for the client must equal them too, whichever security component is chosen; a client that differs is refused with
PMIX_ERR_INVALID_CREDbefore its credential is validated.It must present its rank’s token. A rank the host registers with
PMIx_server_register_clientgets a token nobody is handed; preparing a process for it withPMIx_server_setup_forkreplaces it with the one that process is given, and preparing it again replaces that. The server keeps it until the rank or its job is deregistered. A client of this release that names the rank must present it, on whichever socket it comes, or is refused withPMIX_ERR_INVALID_CREDbefore the host is asked - so until the rank is prepared, no client is taken for it. A rank neither registered nor prepared has none, and nothing is asked of it; nor is anything asked of the singleton a server was started for (PMIX_SINGLETON), which no host prepared. A client of an earlier release presents none, and is not asked for one: with legacy clients, a connection over TCP that declares an earlier release is taken without a token.Its wire format must match its job’s. All clients of one job use one wire format, set by the first to connect. This is checked once the client’s identity and token have been, and a client that differs is refused with
PMIX_ERR_NOT_SUPPORTED.
A PMIX_USERID or PMIX_GRPID in a client’s information field is
ignored. The identity the security component settles is the only one
the server uses.
11.16.6.3. Checks for a tool
Its security component must be available - over TCP. On the local socket the server chose it.
Its identity is settled by the security component (see Settling the identity). The settled uid and gid replace whatever the tool claimed.
A tool of another user is refused unless the host started the server with
PMIX_SERVER_ALLOW_FOREIGN_TOOLS. A tool of the server’s own uid, of root’s, or of the uid the host named toPMIx_server_initasPMIX_USERIDis not another user’s.A tool that names an identity must be entitled to it:
a tool started by a server (kind 5 or 8) whose rank is already known must have that rank’s uid and gid;
a tool of this release that names a rank with a token must present it, as a client must. Once a client connection has bound the token to its pid, a tool on the local socket from another pid is refused with
PMIX_ERR_EXISTS, as for a rank that is connected; a tool’s connection binds nothing;a tool joining a namespace the server already knows must have that namespace’s uid and gid, and the namespace must belong to a tool, or to a process this server started.
Its wire format must be one this server accepts.
A tool the host has not given an identity - a self-started tool with no namespace - gets a new namespace, created by the server and named by the host.
11.16.6.4. Settling the identity
The security component - the one the server chose, on the local socket, or the one the peer named, over TCP - settles its uid and gid. How depends on the component:
native, on the local socket, takes the uid and gid the kernel reports for the connection. Over TCP it asks the server’s own kernel who owns the peer’s end of the connection, which gives the uid; a peer the table does not show is refused. A peer at an address that is not this host’s never reaches it (see above). On a platform with no table to ask, a TCP peer is accepted on its claim: the server has a TCP listener only when it takes legacy clients. A table the server could not read - it is out of descriptors or memory, say - is no answer, and the peer is refused. See The native Mechanism.
munge decodes the credential through the MUNGE daemon. The decoded uid and gid must equal the peer’s registered (client) or claimed (tool) identity. On the local socket a tool’s identity is the one the kernel reports, and a client’s kernel uid and gid have already been compared with its registration (see Checks for a client).
The result is one uid and one gid. They are what the host is given, and what every later access decision for this peer uses.
A peer from outside the server’s user namespace. A server in a user
namespace of its own - in a container, say - is shown each peer’s ids as
its namespace numbers them, and an id the namespace does not map as the
overflow id (65534). On Linux, a peer the kernel reports with the
overflow uid or gid, when the server’s /proc/self/uid_map (or
gid_map) has no entry for that id, is refused with
PMIX_ERR_INVALID_CRED - on the local socket at its first message,
and over TCP when the owner the kernel reports is that uid. Where the
map does have an entry for the overflow id - a subordinate-id range
that covers it, as for a container whose processes run as nobody -
that number is the id of a user of the namespace, and a peer reported
with it is taken as that user, as is any other. The kernel shows a peer
from outside the namespace with the same number, and nothing in the
connection tells the two apart.
Settling the identity is a single step: no component runs an exchange with the peer.
It can wait on a service, though, and does so on the progress thread, where the server serves no other connection or event meanwhile. munge’s decode asks the MUNGE daemon, for every connection. And where the kernel does not give the peer’s groups - a TCP peer, or a platform with no such list - its group is checked against the passwd and group databases, which may ask a directory service through NSS; those answers are cached, so a later connection by the same user does not ask again while they last. A host whose MUNGE daemon or name service can be slow to answer should expect each delay to be the server’s too.
11.16.6.5. Calling the host
When every check has passed, the server asks the host. The peer waits.
A client goes to
client_connected2
(or, if that is NULL, the deprecated client_connected). The host
receives:
Item |
Content |
Comes from |
|---|---|---|
|
the client’s namespace and rank |
the host’s registration; the client’s identity was checked against it |
|
the pointer the host gave when it registered the client |
the host |
|
the settled uid and gid |
the security component |
|
the name of the security component that settled them |
the server |
|
the client’s process ID |
the client’s PMIx library, which calls |
|
the client’s real uid and gid, if it sent them |
the client’s PMIx library |
|
the seconds the host is asked to answer within |
the server’s |
A tool goes to
tool_connected2 (or
the deprecated tool_connected). The host receives the tool’s whole
information field, with these set by the server:
Item |
Content |
Comes from |
|---|---|---|
|
the settled uid and gid |
the security component |
|
the name of the security component that settled them |
the server |
|
the identity the tool gave, if any |
the tool; checked as above where the server already knows it |
|
the tool’s PMIx version |
the tool’s PMIx library, in its connection request |
|
the seconds the host is asked to answer within |
the server’s |
The rest is passed on as the tool sent it: PMIX_PROC_PID,
PMIX_REALUID and PMIX_REALGID, which the tool’s PMIx library
obtains from the operating system; PMIX_CMD_LINE; and the roles the
tool asked for.
If the host provides neither connection up-call, a client is accepted, and so is a tool that brought its own identity. A tool that needs an identity assigned is refused.
The host decides on one tool of a name at a time. A tool that brings its own namespace and rank waits in a queue, in the order the tools’ identities were settled, while the host decides on a tool of that name - or while one of that name is not yet done with after a late answer (see When the host’s answer does not stand). So the host never decides on two tools for one name at once:
where a tool belongs - the namespace and rank it joins, and whether that rank already has a connection - is checked when its turn comes, so it sees every tool accepted ahead of it;
when the host accepts a tool, any queued tool that names the same namespace and rank is refused with
PMIX_ERR_EXISTSat once, and any that names the same namespace as another user is refused withPMIX_ERR_NOT_SUPPORTED. The host is not asked about either;a queued tool that has closed its connection by its turn - its own connect timed out, say - is dropped without asking the host;
when the server finalizes, the tools still queued are refused with
PMIX_ERR_UNREACH.
Tools of other names, and tools that need an identity - which have no
name until the host gives them one - are put to the host together. The
host keeps the names it gives unique; it cannot know of a name a tool
took for itself until it is asked. A tool the host names while a tool
that took that name is still being decided on is checked again when it
is accepted: whichever is accepted second is refused with
PMIX_ERR_EXISTS, and a tool given a rank in a namespace that
appeared while the host decided - another tool’s, accepted meanwhile -
joins it only if every process in it is the tool’s user.
A slow answer therefore delays the tools of that name behind it, and their time in the queue counts against their own wait.
11.16.6.6. What the host must do
Decide. Check that the client is a process you started, with the pid given in
PMIX_PROC_PID; check that the tool’s user may use this server, in the roles it asked for. See Accepting Connections.Record the identity. Keep the
PMIX_USERIDandPMIX_GRPIDyou were given. They are the identity for every later request from this peer, and the library passes them again on every up-call it makes for it. See Deciding What a Requester May Do.Answer exactly once. Either return an error, or return
PMIX_SUCCESSand later call the callback, once, from any thread. Never both.client_connected2may instead returnPMIX_OPERATION_SUCCEEDEDto accept at once without calling the callback.tool_connected2may not, because the tool’s assigned identity can only travel through the callback.Answer within
PMIX_TIMEOUT. The peer is waiting, bounded by its ownptl_base_handshake_wait_time, and tools of its name behind it in the queue wait too.PMIX_TIMEOUTis what the peer has left of that wait; a peer with none left is refused withPMIX_ERR_TIMEOUTwithout the host being asked. The up-call itself runs on the progress thread, so it must not block. A host that cannot decide in time should answer no.Handle
PMIX_ERR_LOST_CONNECTIONfor a process it accepted. See When the host’s answer does not stand.
11.16.6.7. When the host’s answer does not stand
The server waits for the host’s answer for PMIX_TIMEOUT and two
seconds more. Then it refuses the peer with PMIX_ERR_TIMEOUT, and
keeps only what it needs for the answer, which the host still owes: the
host holds the callback’s cbdata. Until the answer comes, the peer’s
name stays held - a client’s rank is refused to any other connection,
and a tool of the same name stays in the queue - so that what follows
cannot name a process accepted since.
A yes the server cannot act on is followed by a
PMIX_ERR_LOST_CONNECTION event naming the process, delivered to the
event handlers the host registered in its own process, as for any
process that leaves without finalizing. That happens when:
the yes comes after the server stopped waiting;
the peer has gone by the time the server replies to it;
the host deregistered the client, its job, or the tool’s rank while it decided;
the peer’s wire format no longer fits its job;
the server could not record the connection.
The name stays held until the event has been given to the host’s handlers, as it is while the server waits for the answer, so nothing accepted under the name meanwhile can be the process the event names. The event is not raised when a live connection has the name by then: it would name that connection. A host with no handler for the event is not told, as for any lost connection.
A no, early or late, is followed by nothing.
11.16.6.8. Replying to the peer
When the host accepts, the server:
adds the peer to its table of peers;
sends the remaining replies (see The replies);
makes the socket non-blocking and arms it for normal traffic;
sends the peer any event notifications cached for it.
When the host refuses, the server sends the host’s status to the peer and closes the connection.
11.16.6.9. One connection per rank
A rank has at most one connection at a time. A process a client starts
once it has initialized inherits none of its identity (see 2. Telling
a client where to connect), and calls PMIx_Init as a singleton. One
that does have it - started before the client initialized, or given it
by the host or a job script - and connects as the same rank is a
clone, in the Standard’s terms. The server does not accept clones.
A client connection for a rank that already has a connection - live (not finalized, socket open) or waiting on the host’s answer - is refused with
PMIX_ERR_EXISTS, before the host is asked.A tool joining a rank that already has a connection is refused the same way - checked before its wire format, so a tool is told its rank is taken rather than that its format does not fit the namespace. A tool that took its namespace and rank from its environment - one started from a job script, say, which inherits the script’s - then connects again as a tool that needs an identity, and the host gives it a namespace of its own; if that connection fails too, the tool is told
PMIX_ERR_EXISTS. Tools of earlier releases do not connect again: theirPMIx_tool_initreturnsPMIX_ERR_EXISTS. A tool connects again only whilePMIx_tool_initruns: a tool that attaches to a server later keeps the identity it was initialized with, and so does a tool whose caller gave it one (PMIX_TOOL_NSPACEandPMIX_TOOL_RANK).The first process to connect holds the rank. A tool that takes a job script’s rank before the script’s application connects leaves the application refused: the server cannot tell such a tool from one that is the script’s only PMIx process, so the script must start its tools without the script’s
PMIX_NAMESPACEandPMIX_RANK, or with an identity of their own.A tool with the server’s address in its environment but no namespace and rank asks the host for an identity. A tool of an earlier release - v7.0.0rc1 among them - in that position says instead that it is a tool on a client’s rank, and sends an empty name: the server takes it as a tool that needs an identity, and the host names it, but the name is not sent to the tool, which reads none. Any other connection that names itself with an empty namespace or an invalid rank is refused with
PMIX_ERR_BAD_PARAM: an empty namespace would match every namespace where names are compared.A rank whose earlier connection finalized, or was lost, can connect again.
Every socket the library creates or accepts is close-on-exec, so a process’s connection closes when it execs - on the local socket and over TCP alike, for a client and a tool. The server handles that as it handles any connection that ends without a finalize: the rank is lost at once (see When a connection is lost), whether the process runs on or not. The program the process execs is a new process to the server.
The server never compares a new connection’s pid with the pid of the
rank’s live connection: a second connection for a rank that has one is
refused with PMIX_ERR_EXISTS whatever pid it comes from.
Libraries of earlier releases leave their connection open across an
exec. The program such a process execs inherits the socket, and cannot
use it: the rank keeps its connection until that program exits, and if
the program calls PMIx_Init or PMIx_tool_init for the rank, it
is refused with PMIX_ERR_EXISTS meanwhile. A process of this release
whose server is of an earlier release closes its connection when it
execs, as it does with any server, and that server handles the
connection as lost.
A rank’s process that died can connect again - restarted by the host, say - before the server has read its old connection’s close. Before refusing, the server looks at the old socket: one that has closed in order or been reset is handled as a lost connection there and then, and the new connection goes ahead. One with unread data on it, or one that reports any other error, is live.
PMIX_ALL_CLONES_PARTICIPATE, which the Standard defines for
collectives, has no effect: a rank has one process.
11.16.7. 5. Ending a connection
A connection ends in one of five ways. The server must handle each one completely: release the peer, account for it in every operation it was part of, and let the host know.
11.16.7.1. When a client finalizes
On the client, PMIx_Finalize:
returns
PMIX_ERR_WOULD_BLOCKif called from the progress thread;if the client was started with
PMIX_EMBED_BARRIER, joins a fence of its job;sends
PMIX_FINALIZE_CMDto the server and waits for the reply, at mostpmix_server_client_fintimeseconds (thefinalize_timeoutMCA parameter, default 10);stops its progress thread, closes the connection, and releases its state.
On the server, PMIX_FINALIZE_CMD:
removes the client’s event registrations and output-forwarding requests, fails requests waiting on its data, and drops cached notifications addressed to it;
calls the host’s client_finalized;
once the host calls back, marks the peer finalized and replies.
When the client’s connection then closes, the server releases the connection. It keeps the rank’s entry - a tombstone - so that the same rank can initialize again later and be recognized. See Client Init/Finalize Lifecycle and State Management.
A finalized rank remains a member of its job, because the process may
call PMIx_Init again and rejoin. An MPI Sessions program does this
once per session, and a fast rank can open the next session’s fence
before a slow one has finished the last. So a collective that names a
finalized rank keeps waiting for it to come back and contribute. See
Specification: Tracking Local Collective Participation, “A graceful finalize is not a
departure”.
The server cannot tell a rank that will return from one that will not.
The host can: deregistering the client (see When the host ends it)
tells the server the process is gone for good. Where the host does not
deregister, a collective whose participant finalized and never returns
waits until its PMIX_TIMEOUT, or indefinitely if it has none.
What the host must do:
release what it holds for that process;
call the callback, which releases the client. The client waits only up to its own timeout;
not deregister the client from inside the up-call before calling the callback.
11.16.7.2. When a tool finalizes or disconnects
PMIx_tool_finalize sends PMIX_FINALIZE_CMD to the tool’s
primary server, waits for the reply, and closes all its connections. The
primary server handles a tool’s finalize as it does a client’s, and
calls client_finalized for it too; server_object is then
NULL. (The capability flag PMIX_CAP_TOOL_FINALIZED says the
library does this.) Any other server sees its connection close, and
handles it as a lost connection (see When a connection is lost). That
is deliberate: for a tool, whether it exited or crashed makes no
difference to a server.
PMIx_tool_disconnect ends the connection to one server, without a
message. It is not a finalize, but the server cannot tell it from one
that failed, and handles it as a lost connection. Requests the tool
still had outstanding with that server complete with
PMIX_ERR_UNREACH.
The server treats a disconnected tool as gone, because once a tool has disconnected neither the server nor the host can tell when it later ends: if its state were kept for a return, it would never be released. So a namespace the server created for a self-started tool is removed when the last process using it has gone, whether by finalize, disconnect or failure. A tool that connects again arrives as a new tool.
11.16.7.3. When a connection is lost
A process that exits, or is killed, without finalizing closes its connection without notice; so does a tool that disconnects. The server notices when a read or write on the socket fails - a process that execs closes its connection the same way (see One connection per rank). The server:
closes the connection and stops its events;
accounts for the process in every collective it was part of: a fence, connect, disconnect, or group construct still waiting for it completes with
PMIX_ERR_PARTIAL_SUCCESSif others remain, orPMIX_ERR_LOST_CONNECTIONif not;removes its event registrations and output requests, fails requests waiting on its data with
PMIX_ERR_LOST_CONNECTION, and stops any monitoring of it. A request for its data made afterwards, while the rank is lost, is answeredPMIX_ERR_NOT_FOUNDat once unless the data had already arrived;raises
PMIX_ERR_LOST_CONNECTION, naming the process, with rangePMIX_RANGE_PROC_LOCAL. It goes to the event handlers the host registered in its own process withPMIx_Register_event_handler, and to no client or tool. It is not cached: a handler registered later does not receive it.
A process whose PMIx_Finalize request reached the server has
finalized, even if its connection closes before the host answers that
request - the client gives up waiting after a while. It is not counted
as lost, and no event is raised for it.
If the rank connects again - the host started a new process for it - the server releases what it kept of the lost connection, and the new process starts with no data published: a request for its data waits for the new process’s, not the lost one’s.
For a client this is a second notice: the client is the host’s child, and the host sees it end. For a tool the host did not start, the event is the only notice that it left without finalizing.
A process that hangs without closing its connection is not detected by the server. The host’s own process monitoring covers that case.
What the host must do: register a handler for
PMIX_ERR_LOST_CONNECTION if it accepts tools it did not start, and
release what it holds for such a tool when the event names it (PRRTE
does this); notice a client’s death through its own monitoring;
generate the termination events applications expect
(PMIX_PROC_TERMINATED, and PMIX_ERR_PROC_TERM_WO_SYNC for
processes connected to it), and deregister the process when it no
longer needs it.
11.16.7.4. When the server goes away
When a client’s or tool’s connection to its server fails:
every request waiting on that server completes with
PMIX_ERR_UNREACH;the process raises
PMIX_ERR_LOST_CONNECTION, if the server was its primary one and it has not begun to finalize;from then on, calls that need the server return
PMIX_ERR_UNREACH.
There is no automatic reconnection. A tool can attach to another server.
11.16.7.5. When the host ends it
Deregistering a client (PMIx_server_deregister_client(3)) tells the server that a process has ended and will not connect again. It is the only way the server learns that a finalized rank is gone for good: finalizing alone leaves the rank a member of its job (see When a client finalizes).
A host need not call it. Without it, the rank stays a member until its job is deregistered, and:
a collective that names a finalized rank that never returns waits until its
PMIX_TIMEOUT, or indefinitely;a request for data that rank never posted, made after it finalized, waits instead of being answered.
When to call it: once the process has terminated. Not from
client_finalized - the process may still initialize again. PRRTE
calls it for each local process when the process has terminated: its
exit has been collected and its output drained. Its
client_finalized only records that the process finalized.
What the server does:
if the client is still connected, closes the connection and handles it as a lost connection, without calling the host back or raising an event: its event and output registrations end, and a request waiting on its data is answered
PMIX_ERR_LOST_CONNECTION. So is one waiting on a rank that never connected. The host deregisters a live client only when it is about to kill it or believes it already dead, so the connection may fail on its own while this runs; the departure is still handled once;releases the network and GPU resources assigned to the process, stops monitoring it, and runs any epilogs registered for it;
sends any reply still queued for it - typically the reply to its finalize;
removes the rank’s entry. A later connection by that rank is refused, because it is no longer registered;
records the rank as departed, so a later request for its data is answered
PMIX_ERR_NOT_FOUNDat once;accounts for its departure in every active collective that names it and has not received its contribution, as for a lost connection, and stops counting it among the job’s local processes, so a collective over the whole job started afterwards does not wait for it either. Whichever of this and the lost connection comes first does it, once;
once every local process of the job has finalized or been deregistered, tells the network and GPU support that the job’s local processes are done.
The departure is counted because the rank can never contribute now,
and because a collective’s PMIX_TIMEOUT is optional: without this
step, a job whose rank finalized and exited would hang.
Deregistering a job (PMIx_server_deregister_nspace(3)) deregisters every client in it, removes the job’s data, events, and group memberships, and accounts for the departure of each of its ranks in every active collective that names it and has not received the rank’s contribution, so that collective completes on the participants that remain.
Finalizing the server (PMIx_server_finalize(3)) closes every connection. Clients and tools see their server go away.
11.16.8. Host obligations, in one place
Register a job and each of its clients before starting them, and call
PMIx_server_setup_forkfor each client.Decide every connection in
client_connected2ortool_connected2: check that a client is a process you started, with the pid given inPMIX_PROC_PID, and that a tool’s user may use the server in the roles it asked for.Record the
PMIX_USERIDandPMIX_GRPIDyou are given. They are the peer’s identity for every later authorization.Answer each up-call exactly once, promptly, without blocking - a connection up-call within the
PMIX_TIMEOUTit carries.Be ready to give an identity to a tool you launched: one whose namespace and rank are already in use connects again without one.
In
client_finalized, release your state for the process and call the callback. Do not deregister the process there: it may initialize again.Watch your processes. Generate termination events when they end.
Register a handler for
PMIX_ERR_LOST_CONNECTION: it is how you learn that a tool you did not start has gone without finalizing, and that a process you accepted never became connected.Optionally, deregister each client once its process has terminated. Until you do, a rank that finalized is still expected in its job’s collectives.
A peer that vanishes while the server writes to it costs the server an
error on that connection, never a SIGPIPE: no write PMIx makes to a
connection’s socket can raise it, nor the write of its URI to a pipe the
host names (PMIX_SERVER_REPORT_URI). Nor can a write forwarding output or
input to a pipe or file of the library’s own, such as a child’s stdin. A
host need not change its own handling of that signal for any of these.
Output written to the process’s own stdout and stderr is written as the
process writes it: if their reader goes, that write raises SIGPIPE
as any other write the process makes there would.