24.1. PMIx v7.x series
This file contains all the NEWS updates for the PMIx v7.x series, in reverse chronological order.
24.1.1. 7.0.0 – TBD
Important
This is the first release in the v7 family. As with the v6 series, releases are intended to be infrequent “reference tags” that mark the completion of some significant body of work rather than a regular cadence.
For this release, that work is fivefold: completion of
the Group family of APIs with the invite/join construction
mode (see the
Group Construction
section of the documentation); a substantial reduction in
the amount of data PMIx moves during startup, aided by a
new shared-memory datastore (gds/shmem3); a connection
model in which every client and tool reaches a server on
its own host over a local socket, where the kernel names
both ends of the connection; an access model in which
every job records its owner and the server holding its
data decides, request by request, who may see it; and a
systematic review of every subtree under src that
closed a long list of crashes, leaks, and use-after-free
defects.
Sites, host environments, and tool writers moving from an earlier release should read Moving to PMIx v7, which describes what changed for each role, and the settings matrix, which sets the configure options, MCA parameters, and server attributes that a server’s behavior depends on against one another.
Proper execution of the group construction modes, and of the session and allocation support described below, requires that the host provide the necessary backend support. Please check with your host provider to ensure it is available, or feel free to use the PMIx Reference RunTime Environment (PRRTE).
24.1.1.1. Highlights
Group construction by invitation now works end to end, including endpoint exchange, context-id assignment, and invitations that name a whole namespace.
PMIX_CAP_GROUP_JOIN_COMPLETES.Startup moves far less data.
PMIx_Commitand the collecting fence send only what changed since the last commit rather than the entire store, and a get for a process of another job no longer resends that job’s job-level data when the requester already holds it. The wire format is unchanged.New shared-memory datastore.
gds/shmem3replacesgds/shmem2, with lock-free reads and the ability to update the data of a running job.Access control by user and group. Every job records its owner – named by the host, or the user its clients run as – together with an optional list of others who may read it. The server holding a job’s data decides who may have it, answering
PMIX_ERR_NO_PERMISSIONSotherwise, and a job’s output files and shared-memory segments belong to its owner.PMIX_ACCESS_PERMISSIONS,PMIX_ACCESS_USERIDS, andPMIX_ACCESS_GRPIDS– until now accepted only for published data – name who else may read a job when they are given in the job-level info ofPMIx_SpawnorPMIx_server_register_nspace. A process’s groups are those the kernel reports for its connection, or those the host gives with the newPMIX_USER_GRPIDS.PMIX_CAP_ACCESS_CHECK.Up-calls name the requester. Every call a server makes on a peer’s behalf now says who asked.
PMIX_USERIDandPMIX_GRPIDare always those of the connected peer, as its connection established them, withPMIX_REQUESTORwhere the up-call’s signature cannot carry it andPMIX_REQUESTOR_PIDwhere the kernel reports the peer’s pid. A relaying host names whom it speaks for with the newPMIX_ORIGIN_*attributes, which are passed up for the receiving host to judge, and a requester asks to have work charged to a group withPMIX_REQUESTED_GRPID. A spawn up-call says whether a tool or a client asked.PMIx_server_dmodex_request2takes an info array, and the newabort2up-call carries the requester’s identity, whichabortcannot.PMIX_CAP_REQUESTER_ID,PMIX_CAP_DMODEX_REQUEST2,PMIX_CAP_ORIGIN_ID,PMIX_CAP_ABORT2.Clients and tools connect over a local socket. Every server listens on an owner-only
AF_UNIXsocket, where the kernel reports the pid, uid, gid, and groups of each peer, and takes connections from its own host only; TCP remains solely for clients, tools, and servers of earlier releases (see below). A connecting process offers its security, wire-format, and datastore components and the server chooses among them; the process then presents a credential and, for a client, the connection tokenPMIx_server_setup_forkhands its rank, which is bound to the first process that presents it. A rank has one connection at a time, the host’s decision on a connection has a deadline, and a client or tool checks who runs the server it reached.Servers are found in directories only their user writes. A server’s rendezvous files go in
<system tmpdir>/pmix-user.<uid>, or$XDG_RUNTIME_DIR/pmixwhen that directory cannot be used, and a root system server, scheduler, or system controller writes to/run/pmix. A server holds a lock beside each file while it runs, so a search passes over the file of a server that is gone, and a tool takes a file only of the user who wrote it. A session-level server is named byPMIX_SESSION_RNDZ_IDor itsPMIX_SESSION_ID, so servers of different sessions can share a node. Before it starts, a server checks that no other user can change its tmpdirs (ptl_base_tmpdir_check).Data can be deleted. New
PMIX_DEL_*scopes onPMIx_Putremove a key, including from clients that cached it.PMIX_CAP_DATA_DELETE.Sessions are first class.
PMIx_server_register_sessionandPMIx_server_deregister_sessiondescribe a session independently of its jobs.PMIX_CAP_SESSION_REGISTRATION.Stdin flow control.
PMIx_server_IOF_flow_controllets a host suspend and resume its stdin source, including across chained launchers.PMIX_CAP_IOF_FLOW_CONTROL.Expanded allocation support through new
PMIX_ALLOC_*attributes;palloccan target an existing allocation, and a spawn can carry the allocation request it needs.PMIX_ALLOC_UIDandPMIX_ALLOC_GIDname the user and group an allocation or session belongs to, whichPMIX_USERIDandPMIX_GRPIDcannot, since they name the caller – a scheduler or host acting for that user.PMIX_ALLOC_GRANT_TIMEgives the time the scheduler granted an allocation that the host answered before the grant, so the time a request spent queued can be told from the time the DVM took to grow.New compression components.
zstd(preferred when available, roughly 6x faster than zlib) andlz4. All nodes in a job must use the same component.GPU discovery repaired.
pgpu/amdandpgpu/nvdactivate on the right nodes, including nodes whose AMD GPUs report themselves as processing accelerators, and Intel GPUs get a correctZE_AFFINITY_MASK.ROCR_VISIBLE_DEVICESnames each AMD GPU asGPU-and its 16-digit unique id, the form ROCm reads; it is left unset for a partitioned GPU. A 3D-controller GPU that hwloc reports with no OS device is counted as a GPU and named by its PCI bus id. Where any of a process’s GPUs cannot be named to its runtime, the visible-devices variable is not set, so the process sees every GPU rather than having one it was given hidden.PMIX_CAP_DEVICE_ENUM.NICs are named to the libraries that use them. A process mapped against network devices is told which through the variable its fabric library reads:
NCCL_IB_HCA,UCX_NET_DEVICES, andFI_VERBS_DEVICE_NAMEfor InfiniBand and RoCE adapters from NVIDIA, Broadcom, and Intel (iWARP excluded);FI_EFA_IFACEfor Amazon EFA (newpnet/efa);FI_CXI_DEVICE_NAMEfor HPE Slingshot (newpnet/cxi, which prefers thecxiNdevice and finds it in sysfs under an hwloc older than 2.14); andFI_OPX_HFI_SELECTfor a single Omni-Path adapter, for libfabric’s OPX provider, on OPA100 and on CN5000, whose HFIspnet/opanow recognizes. A value already set is replaced, and a variable that cannot name the assigned devices exactly, as the library will read it, is left unset rather than written to select another device. A device is named as these libraries accept it even when the host mapped the process against another name for it, such as its network interface (ib0,hsn0).OPX gets a job key. A launch that asks for security keys on an Omni-Path node also sets
FI_OPX_UUID, the key OPX uses for its shared-memory segments and hardware contexts. A job and the jobs it spawns share one key, and a key already in the process’s environment is kept.FI_OPX_*variables are now forwarded by default.A loaded topology survives its caller. The structure
PMIx_Load_topologyfills in refers to the library’s own topology. A caller that destructs it – asMPI_Finalizedoes in older MPICH builds – now only clears it, before or afterPMIx_Finalize, where the library’s topology was destroyed a second time. A process that finalizes and initializes again loads its topology afresh rather than being handed none.Distances can be measured from a device.
PMIX_DEVICE_DIST_ORIGINanswers questions such as which NICs are nearest a given GPU, and distances now account for the PCIe tree.PMIX_CAP_DEVICE_DIST_ORIGIN.Connections no longer stall a process. A silent or unresponsive peer can no longer hang a server, client, or tool, and intermittent connection failures under spawn-heavy workloads have been fixed. A connect to a busy server is retried within the connect’s bound, and
PMIX_TIMEOUTbounds a tool’s connect.Collectives account for departed ranks. A rank the host deregisters, or whose connection is lost, leaves every collective of its job exactly once, whichever happens first, and a rank that returns is counted back in. A rank that calls more than once, or is named twice in a participant list, is counted once.
PMIX_CAP_DEREG_DEPARTS.The host learns who took part in a collective. Every fence, connect, disconnect, and group up-call carries
PMIX_LOCAL_PARTICIPANTS, naming the local processes that called, and a host names the members that never took part with the newPMIX_COLLECTIVE_NONPARTICIPANTS, which reaches each caller in the results ofPMIx_Group_construct– now returned on a failure that carries the list as well as on success. A collective whose localPMIX_TIMEOUTfires is handed to the host as it stands, withPMIX_ERR_TIMEOUTas its local status, rather than failed where the host never hears of it, and the newPMIx_server_collective_expirelets a host hand up every server’s collective at its own deadline.A group ID belongs to one group. A group ID is held by one group until it is destructed, and may not name a job the server knows. Every caller of a construct or destruct must be a participant and name the same participants, and a destruct runs over the recorded membership. An invitation is aborted when its leader departs, and a get by group rank is answered only for a member of the group.
Output files are written as the job’s owner. A server that is not the job’s owner creates and opens the job’s output files in a child running as the owner, with the groups the host registered for it, and writes them from a thread of its own. A server that cannot act as the owner leaves the files to a launcher that names itself with
PMIX_IOF_FILE_WRITER, or else tells the job once, when it is registered, that its files are not written. The open is bounded byrunas_timeoutwhether or not the server is itself the owner, so a stalled filesystem no longer holds the server.Log output can be addressed. The new
PMIX_LOG_TARGETdirective names the only processes a server’s log output is for, so a message about one tool’s request – such as a hostfile that would not parse – goes to that tool alone instead of to every tool connected, and is not cached for tools that connect later.Spawned jobs’ output follows the parent’s. A spawned job’s output now arrives formatted as its parent’s (
PMIX_IOF_JOB_DIRECTIVES), and thePMIX_FWD_*channels a spawn names are honored on every server.Self-started tools keep namespaces of their own. The namespace a tool names for itself is never merged into a job the host registers; a host registration of that name takes it back, and the namespace is released when its last tool leaves.
PMIX_CAP_TOOL_NSPACE_RELEASE.Registering a rank again updates it, and returns the new success status
PMIX_UPDATED.New regex interface.
PMIx_generate_regex2andPMIx_parse_regex2work on a structuredpmix_regex2_t; thepreg/nativecomponent has been removed.PMIX_CAP_REGEX2.New wire format.
bfrops/v70carries this release’s data types, andbfrops/v61again carries exactly the set v6.1.0 shipped.No more hangs from event handlers. Blocking APIs called from the progress thread now return
PMIX_ERR_WOULD_BLOCK.Concurrent gets are answered separately. A
PMIx_Getshares an outstanding request to the server only when it asks for the same key of the same process with no directives, so a get for a key a peer never posted no longer fails the other gets for that peer.PMIX_ANL_MAP on request. When the host does not supply it, a server computes
PMIX_ANL_MAPfrom the job’s node list and each node’s local peers, so MPICH need not resolve its peers node by node.Finalize reports its outcome.
PMIx_FinalizeandPMIx_tool_finalizereturn the host’s answer to the finalize request, orPMIX_ERR_TIMEOUT, and a server writes out the replies it has queued before its progress thread stops.Safer file handling. Shared-memory,
hwloc.sm, and rendezvous files are created exclusively and verified before they are reused or re-permissioned.Wire data is bounded before it is trusted. Every count read off the wire is checked against the bytes actually received before it sizes an allocation, and the datatype of each element is verified before it is read – across
bfrops,gds,pstat,ptl, the event system, and the client, server, and tool message handlers.Email logging is configurable.
plog/smtptakes its recipients and sender from MCA parameters, acceptsPMIX_LOG_EMAIL_MSGas the message body, bounds the SMTP dialogue withplog_smtp_timeout, and reports a message as sent only once the server has accepted it.pmix_info fixes.
--paramand--show-versionnow honor the arguments their usage text documents.MCA parameter files are parsed by a new line reader, fixing several silent misreads (CRLF endings, quoted values, a missing final newline).
Command-line matching. The new
pmix_cli_match()resolves an input against all of an option’s choices and reports ambiguous abbreviations.PMIX_CAP_CLI_MATCH.Argv lookup.
PMIx_Argv_idxreturns the position of a string in an argv array.PMIX_CAP_ARGV_IDX.Memory binding.
PMIX_MEMBINDnames the memory attribute (for example,latencyorbandwidth) by whichPMIx_Spawnselects the memory a process is bound to.FreeBSD. A new
pstat/pfreebsdcomponent samples processes, nodes, disks, and interfaces, andpsec/nativefinds the owner of a TCP peer there.Build.
flexis no longer required, the Python bindings now receive the build’s optimization and hardening flags, and the tree builds warning-free with gcc 16. The Cython check, the documentation’s Markdown converter, and the unit tests’ Python scripts all run with the Python interpreter configure found rather than the firstpython3onPATH, and a tarball now finds Cython when it is installed only as a Python package.Documentation. 277 new man pages cover the public API,
docs/how-things-workdescribes the major subsystems, and the security policy now lives inSECURITY.md. New guides cover host environments, system administration, and moving from v5 to v7 for each role, schedulers included. The transition guide lists the NIC variables PMIx sets at launch, and the allocation-request and session-control pages explain how the two calls work together.Testing. New unit tests, multi-node Docker Swarm suites, multi-user output-file tests, AddressSanitizer CI, cross-version CI with clients back to v2.1, a CI build without support for earlier releases, and topologies from Slingshot, CN5000, MI350, and AWS GPU nodes as device-discovery fixtures.
24.1.1.2. Compatibility notes
Out-of-tree MCA components must be rebuilt against the v7 headers; framework versions are now checked at load time, and the
pnetinterface is at 4.0.0.PMIX_CAP_MCA_FW_VERSION.Earlier releases are supported only when built for. A default build speaks only this release’s wire format and has no TCP transport. Configure with
--enable-legacy-clientsfor a server to take clients and tools from v2.1 on, then turn it on at run time withptl_base_legacy_clientsorPMIX_SERVER_LEGACY_PEERS; the server listens for them on loopback TCP only. Configure with--enable-legacy-serversfor a client or tool to connect to servers from v3.2 through v6.x.--enable-legacy-peersandptl_base_legacy_peersask for both. A client of an earlier release under a server that does not take legacy clients cannot connect, and runs as a singleton. Refusals by version reportPMIX_ERR_OUTDATED.Tools on other hosts are not supported. A server takes connections from its own host only, and a process connects only to a server on its own host.
PMIX_SERVER_REMOTE_CONNECTIONSis deprecated: true failsPMIx_server_init, and false changes nothing.A connecting peer must present a credential. The handshake-only model is gone, and with it the
psec/noneandpsec/dummy_handshakecomponents. A host or tool that selected either must choosenativeormunge.One connection per rank. A second connection for a connected rank is refused with
PMIX_ERR_EXISTS, andPMIX_ALLOW_CLIENT_CLONESis deprecated and ignored. Every socket the library makes is close-on-exec, so an exec ends the process’s connection, andPMIx_InitandPMIx_tool_initremove the variables that identify the process and its server from the environment, so a process it starts afterwards does not inherit them.Server URIs.
PMIX_MYSERVER_URIis now always the local socket’s URI;PMIX_SERVER_URIis unchanged, and is what a process of an earlier release should be given. A TCP URI given alone to a tool, an attach, or a launcher is refused withPMIX_ERR_NOT_SUPPORTED: name the server’s local-socket URI, orfile:<path>. A server refuses a peer of this release on TCP with the newPMIX_ERR_LOCAL_URI_REQUIRED. The-and+report-URI outputs print the local socket’s URI, and the newtcp-andtcp+print the TCP URI.No silent singleton fallback.
PMIx_Initfails withPMIX_ERR_COMM_FAILUREwhen it cannot reach a server it was told about, or when it is givenPMIX_NAMESPACEbut no server URI.PMIX_ERR_UNREACHnow means no server was described.Connection waits are bounded by default.
ptl_base_handshake_wait_timenow defaults to 60 seconds and the newptl_base_connect_ack_timeoutto 5 seconds; set either to 0 for an unbounded wait.Registration requirements.
PMIx_server_register_nspacerefuses job data that does not includePMIX_JOB_SIZE, and a client connects only once the host has registered its namespace. A session-level server must be givenPMIX_SESSION_RNDZ_IDorPMIX_SESSION_ID, and a server given no namespace takespmix-server-<host>-<pid>. A host should not register the namespace a self-started tool names.A user’s groups come from the host. The library no longer looks a user’s groups up. A host supplies them with
PMIX_USER_GRPIDS, for a job’s owner inPMIx_server_register_nspaceand for any other user inPMIx_server_register_resources, where a singlePMIX_GRPIDnames one group.A scheduler or system controller serves tools unless the host passes
PMIX_SERVER_TOOL_SUPPORTfalse, which now fails the init. A host is now told throughclient_finalizedwhen a connected scheduler finalizes.A host must not complete a collective whose
PMIX_LOCAL_COLLECTIVE_STATUSisPMIX_ERR_TIMEOUTas a success: it was handed up without the members that never called. On a group construct that asked forPMIX_GROUP_FT_COLLECTIVE,PMIX_ERR_LOST_CONNECTIONis a degraded success, and the group forms on the survivors.Group membership keeps the order the host provided, and group ranks follow it.
PMIX_GROUP_FINAL_MEMBERSHIP_ORDERis deprecated and ignored.A blocking
PMIx_IOF_pullnow returns the registration id (a non-negative value) instead ofPMIX_OPERATION_SUCCEEDED.PMIx_FinalizeandPMIx_tool_finalizecan returnPMIX_ERR_TIMEOUT, andPMIx_tool_finalizereturnsPMIX_ERR_WOULD_BLOCKwhen called on the progress thread.PMIx_Compute_distancesvalues differ from earlier releases because of the new PCIe term; relative order is otherwise unchanged.Directories PMIx creates below a given root are reused only when owned by the effective uid, and existing directories PMIx is given are used with their permissions as found.
PMIX_SERVER_ALLOW_FOREIGN_TOOLSnow defaults tofalse. Hosts that want tools from other users to connect must pass it; PRRTE does. Tools of the user the host names asPMIX_USERIDare taken without it.A command-line option value must now be a leading part of the name it matches, so
packagefoono longer matchespackage.Support for the Solaris, Sun Studio, and KAI toolchains has been removed.
PMIX_COMPRESSED_STRINGis deprecated.A server answers a request for a job’s data only when the requester is permitted it, and returns
PMIX_ERR_NO_PERMISSIONSwhen it is not. A host that registers a namespace for another user should name the owner, since a job with no owner named belongs to the user its clients run as.PMIX_USERIDandPMIX_GRPIDmay now be given as a user or group name as well as a number.pallocdrops--uidand accepts a group name for--gid.A tool looks for a server’s rendezvous file in the
pmix-userdirectories of the users whose servers it may talk to – its own, root’s, and thoseptl_base_server_accessnames – and in/run/pmix. It walks the system tmpdir, at mostptl_base_search_depthdirectories deep (default 8), only for a server of an earlier release whenptl_base_legacy_serversis set, and matches a file it was told to look for by its whole name rather than by prefix.New limits:
psensor_base_max_monitors_per_peerandpstat_base_max_monitors_per_peer(each 16 by default) cap the periodic monitors one process may hold,iof_max_partial_line(64KB) caps how long a partial output line is held before it is written out as it stands, andpmix_hwloc_max_cpu_indexbounds the processor indices accepted in a cpuset list.More limits:
ptl_base_max_pending_connections(a quarter of the descriptor limit) andptl_base_max_pending_bytes(16MB) bound connections still being read;pmix_server_max_event_regs_per_peer,pmix_server_max_iof_reqs_per_peer, andpmix_server_max_invites_per_peer(each 1024) cap what one connection may hold;pmix_max_events_per_source(a quarter ofpmix_max_events) caps one namespace’s share of the event cache;pcompress_base_max_size(512MB) caps a decompressed size;plog_base_max_suppressed(1024) caps the log messages held for duplicate suppression;pmdl_ompi_max_param_file_size(1MB) caps an Open MPI parameter file read for a job;iof_writer_backlog(16384 chunks) caps output queued for a file another user’s job writes;runas_timeout(10 seconds) bounds work done as another user; andpmix_tool_finalize_timeout(5 seconds) bounds a tool’s wait at finalize.PMIx reads a parameter file only if nobody could have written it but root, the user it runs as, the owner of the loaded PMIx library or of a directory above it, and – except when running as root – the members of a group it is in. The same holds for the directory holding the file unless that directory is sticky, and for the directory of a symbolic link’s target. A file that fails is passed over with a message saying why, and a FIFO at the file’s name no longer blocks startup. The MCA home-directory defaults are omitted when the user has no home directory.
pstattakes an empty field list to mean every field, and answers a pid the requester does not own as one with no process.The installed
pmix_getline()utility reads a line of any length, and takes a second argument that reports a read that stopped short.The provisional resource-usage structures of v5.0 (
pmix_proc_stats_tand its relatives), with their data types, functions, and macros, have been removed in favor of the attribute arrays the Standard adopted; usePMIx_Process_monitor. Three attributes that shared another attribute’s string in v5.0 –PMIX_FABRIC_DEVICE_COORDINATES,PMIX_JOB_CTRL_CHECKPOINT_TIMEOUT, andPMIX_QUERY_SUPPORTED_QUALIFIERS– now have strings of their own.PMIX_SESSION_PROVISION, missing since v5.0, is restored, andPMIX_CAP_BASEis again 1, as it was in v5.0.The Autotools versions used to build official release tarballs now live in
VERSIONas*_dist_versionfields rather than insidecontrib/make_dist_tarball, and the required toolchain is m4 1.4.21, autoconf 2.73, automake 1.19, and libtool 2.6.2.
A full list of individual changes will not be provided here, but will commence with the v7.0.1 release.