24.1. PMIx v7.x series

This file contains all the NEWS updates for the PMIx v7.x series, in reverse chronological order.

24.1.1. 7.0.0 – TBD

Important

This is the first release in the v7 family. As with the v6 series, releases are intended to be infrequent “reference tags” that mark the completion of some significant body of work rather than a regular cadence.

For this release, that work is fivefold: completion of the Group family of APIs with the invite/join construction mode (see the Group Construction section of the documentation); a substantial reduction in the amount of data PMIx moves during startup, aided by a new shared-memory datastore (gds/shmem3); a connection model in which every client and tool reaches a server on its own host over a local socket, where the kernel names both ends of the connection; an access model in which every job records its owner and the server holding its data decides, request by request, who may see it; and a systematic review of every subtree under src that closed a long list of crashes, leaks, and use-after-free defects.

Sites, host environments, and tool writers moving from an earlier release should read Moving to PMIx v7, which describes what changed for each role, and the settings matrix, which sets the configure options, MCA parameters, and server attributes that a server’s behavior depends on against one another.

Proper execution of the group construction modes, and of the session and allocation support described below, requires that the host provide the necessary backend support. Please check with your host provider to ensure it is available, or feel free to use the PMIx Reference RunTime Environment (PRRTE).

24.1.1.1. Highlights

  • Group construction by invitation now works end to end, including endpoint exchange, context-id assignment, and invitations that name a whole namespace. PMIX_CAP_GROUP_JOIN_COMPLETES.

  • Startup moves far less data. PMIx_Commit and the collecting fence send only what changed since the last commit rather than the entire store, and a get for a process of another job no longer resends that job’s job-level data when the requester already holds it. The wire format is unchanged.

  • New shared-memory datastore. gds/shmem3 replaces gds/shmem2, with lock-free reads and the ability to update the data of a running job.

  • Access control by user and group. Every job records its owner – named by the host, or the user its clients run as – together with an optional list of others who may read it. The server holding a job’s data decides who may have it, answering PMIX_ERR_NO_PERMISSIONS otherwise, and a job’s output files and shared-memory segments belong to its owner. PMIX_ACCESS_PERMISSIONS, PMIX_ACCESS_USERIDS, and PMIX_ACCESS_GRPIDS – until now accepted only for published data – name who else may read a job when they are given in the job-level info of PMIx_Spawn or PMIx_server_register_nspace. A process’s groups are those the kernel reports for its connection, or those the host gives with the new PMIX_USER_GRPIDS. PMIX_CAP_ACCESS_CHECK.

  • Up-calls name the requester. Every call a server makes on a peer’s behalf now says who asked. PMIX_USERID and PMIX_GRPID are always those of the connected peer, as its connection established them, with PMIX_REQUESTOR where the up-call’s signature cannot carry it and PMIX_REQUESTOR_PID where the kernel reports the peer’s pid. A relaying host names whom it speaks for with the new PMIX_ORIGIN_* attributes, which are passed up for the receiving host to judge, and a requester asks to have work charged to a group with PMIX_REQUESTED_GRPID. A spawn up-call says whether a tool or a client asked. PMIx_server_dmodex_request2 takes an info array, and the new abort2 up-call carries the requester’s identity, which abort cannot. PMIX_CAP_REQUESTER_ID, PMIX_CAP_DMODEX_REQUEST2, PMIX_CAP_ORIGIN_ID, PMIX_CAP_ABORT2.

  • Clients and tools connect over a local socket. Every server listens on an owner-only AF_UNIX socket, where the kernel reports the pid, uid, gid, and groups of each peer, and takes connections from its own host only; TCP remains solely for clients, tools, and servers of earlier releases (see below). A connecting process offers its security, wire-format, and datastore components and the server chooses among them; the process then presents a credential and, for a client, the connection token PMIx_server_setup_fork hands its rank, which is bound to the first process that presents it. A rank has one connection at a time, the host’s decision on a connection has a deadline, and a client or tool checks who runs the server it reached.

  • Servers are found in directories only their user writes. A server’s rendezvous files go in <system tmpdir>/pmix-user.<uid>, or $XDG_RUNTIME_DIR/pmix when that directory cannot be used, and a root system server, scheduler, or system controller writes to /run/pmix. A server holds a lock beside each file while it runs, so a search passes over the file of a server that is gone, and a tool takes a file only of the user who wrote it. A session-level server is named by PMIX_SESSION_RNDZ_ID or its PMIX_SESSION_ID, so servers of different sessions can share a node. Before it starts, a server checks that no other user can change its tmpdirs (ptl_base_tmpdir_check).

  • Data can be deleted. New PMIX_DEL_* scopes on PMIx_Put remove a key, including from clients that cached it. PMIX_CAP_DATA_DELETE.

  • Sessions are first class. PMIx_server_register_session and PMIx_server_deregister_session describe a session independently of its jobs. PMIX_CAP_SESSION_REGISTRATION.

  • Stdin flow control. PMIx_server_IOF_flow_control lets a host suspend and resume its stdin source, including across chained launchers. PMIX_CAP_IOF_FLOW_CONTROL.

  • Expanded allocation support through new PMIX_ALLOC_* attributes; palloc can target an existing allocation, and a spawn can carry the allocation request it needs. PMIX_ALLOC_UID and PMIX_ALLOC_GID name the user and group an allocation or session belongs to, which PMIX_USERID and PMIX_GRPID cannot, since they name the caller – a scheduler or host acting for that user. PMIX_ALLOC_GRANT_TIME gives the time the scheduler granted an allocation that the host answered before the grant, so the time a request spent queued can be told from the time the DVM took to grow.

  • New compression components. zstd (preferred when available, roughly 6x faster than zlib) and lz4. All nodes in a job must use the same component.

  • GPU discovery repaired. pgpu/amd and pgpu/nvd activate on the right nodes, including nodes whose AMD GPUs report themselves as processing accelerators, and Intel GPUs get a correct ZE_AFFINITY_MASK. ROCR_VISIBLE_DEVICES names each AMD GPU as GPU- and its 16-digit unique id, the form ROCm reads; it is left unset for a partitioned GPU. A 3D-controller GPU that hwloc reports with no OS device is counted as a GPU and named by its PCI bus id. Where any of a process’s GPUs cannot be named to its runtime, the visible-devices variable is not set, so the process sees every GPU rather than having one it was given hidden. PMIX_CAP_DEVICE_ENUM.

  • NICs are named to the libraries that use them. A process mapped against network devices is told which through the variable its fabric library reads: NCCL_IB_HCA, UCX_NET_DEVICES, and FI_VERBS_DEVICE_NAME for InfiniBand and RoCE adapters from NVIDIA, Broadcom, and Intel (iWARP excluded); FI_EFA_IFACE for Amazon EFA (new pnet/efa); FI_CXI_DEVICE_NAME for HPE Slingshot (new pnet/cxi, which prefers the cxiN device and finds it in sysfs under an hwloc older than 2.14); and FI_OPX_HFI_SELECT for a single Omni-Path adapter, for libfabric’s OPX provider, on OPA100 and on CN5000, whose HFIs pnet/opa now recognizes. A value already set is replaced, and a variable that cannot name the assigned devices exactly, as the library will read it, is left unset rather than written to select another device. A device is named as these libraries accept it even when the host mapped the process against another name for it, such as its network interface (ib0, hsn0).

  • OPX gets a job key. A launch that asks for security keys on an Omni-Path node also sets FI_OPX_UUID, the key OPX uses for its shared-memory segments and hardware contexts. A job and the jobs it spawns share one key, and a key already in the process’s environment is kept. FI_OPX_* variables are now forwarded by default.

  • A loaded topology survives its caller. The structure PMIx_Load_topology fills in refers to the library’s own topology. A caller that destructs it – as MPI_Finalize does in older MPICH builds – now only clears it, before or after PMIx_Finalize, where the library’s topology was destroyed a second time. A process that finalizes and initializes again loads its topology afresh rather than being handed none.

  • Distances can be measured from a device. PMIX_DEVICE_DIST_ORIGIN answers questions such as which NICs are nearest a given GPU, and distances now account for the PCIe tree. PMIX_CAP_DEVICE_DIST_ORIGIN.

  • Connections no longer stall a process. A silent or unresponsive peer can no longer hang a server, client, or tool, and intermittent connection failures under spawn-heavy workloads have been fixed. A connect to a busy server is retried within the connect’s bound, and PMIX_TIMEOUT bounds a tool’s connect.

  • Collectives account for departed ranks. A rank the host deregisters, or whose connection is lost, leaves every collective of its job exactly once, whichever happens first, and a rank that returns is counted back in. A rank that calls more than once, or is named twice in a participant list, is counted once. PMIX_CAP_DEREG_DEPARTS.

  • The host learns who took part in a collective. Every fence, connect, disconnect, and group up-call carries PMIX_LOCAL_PARTICIPANTS, naming the local processes that called, and a host names the members that never took part with the new PMIX_COLLECTIVE_NONPARTICIPANTS, which reaches each caller in the results of PMIx_Group_construct – now returned on a failure that carries the list as well as on success. A collective whose local PMIX_TIMEOUT fires is handed to the host as it stands, with PMIX_ERR_TIMEOUT as its local status, rather than failed where the host never hears of it, and the new PMIx_server_collective_expire lets a host hand up every server’s collective at its own deadline.

  • A group ID belongs to one group. A group ID is held by one group until it is destructed, and may not name a job the server knows. Every caller of a construct or destruct must be a participant and name the same participants, and a destruct runs over the recorded membership. An invitation is aborted when its leader departs, and a get by group rank is answered only for a member of the group.

  • Output files are written as the job’s owner. A server that is not the job’s owner creates and opens the job’s output files in a child running as the owner, with the groups the host registered for it, and writes them from a thread of its own. A server that cannot act as the owner leaves the files to a launcher that names itself with PMIX_IOF_FILE_WRITER, or else tells the job once, when it is registered, that its files are not written. The open is bounded by runas_timeout whether or not the server is itself the owner, so a stalled filesystem no longer holds the server.

  • Log output can be addressed. The new PMIX_LOG_TARGET directive names the only processes a server’s log output is for, so a message about one tool’s request – such as a hostfile that would not parse – goes to that tool alone instead of to every tool connected, and is not cached for tools that connect later.

  • Spawned jobs’ output follows the parent’s. A spawned job’s output now arrives formatted as its parent’s (PMIX_IOF_JOB_DIRECTIVES), and the PMIX_FWD_* channels a spawn names are honored on every server.

  • Self-started tools keep namespaces of their own. The namespace a tool names for itself is never merged into a job the host registers; a host registration of that name takes it back, and the namespace is released when its last tool leaves. PMIX_CAP_TOOL_NSPACE_RELEASE.

  • Registering a rank again updates it, and returns the new success status PMIX_UPDATED.

  • New regex interface. PMIx_generate_regex2 and PMIx_parse_regex2 work on a structured pmix_regex2_t; the preg/native component has been removed. PMIX_CAP_REGEX2.

  • New wire format. bfrops/v70 carries this release’s data types, and bfrops/v61 again carries exactly the set v6.1.0 shipped.

  • No more hangs from event handlers. Blocking APIs called from the progress thread now return PMIX_ERR_WOULD_BLOCK.

  • Concurrent gets are answered separately. A PMIx_Get shares an outstanding request to the server only when it asks for the same key of the same process with no directives, so a get for a key a peer never posted no longer fails the other gets for that peer.

  • PMIX_ANL_MAP on request. When the host does not supply it, a server computes PMIX_ANL_MAP from the job’s node list and each node’s local peers, so MPICH need not resolve its peers node by node.

  • Finalize reports its outcome. PMIx_Finalize and PMIx_tool_finalize return the host’s answer to the finalize request, or PMIX_ERR_TIMEOUT, and a server writes out the replies it has queued before its progress thread stops.

  • Safer file handling. Shared-memory, hwloc.sm, and rendezvous files are created exclusively and verified before they are reused or re-permissioned.

  • Wire data is bounded before it is trusted. Every count read off the wire is checked against the bytes actually received before it sizes an allocation, and the datatype of each element is verified before it is read – across bfrops, gds, pstat, ptl, the event system, and the client, server, and tool message handlers.

  • Email logging is configurable. plog/smtp takes its recipients and sender from MCA parameters, accepts PMIX_LOG_EMAIL_MSG as the message body, bounds the SMTP dialogue with plog_smtp_timeout, and reports a message as sent only once the server has accepted it.

  • pmix_info fixes. --param and --show-version now honor the arguments their usage text documents.

  • MCA parameter files are parsed by a new line reader, fixing several silent misreads (CRLF endings, quoted values, a missing final newline).

  • Command-line matching. The new pmix_cli_match() resolves an input against all of an option’s choices and reports ambiguous abbreviations. PMIX_CAP_CLI_MATCH.

  • Argv lookup. PMIx_Argv_idx returns the position of a string in an argv array. PMIX_CAP_ARGV_IDX.

  • Memory binding. PMIX_MEMBIND names the memory attribute (for example, latency or bandwidth) by which PMIx_Spawn selects the memory a process is bound to.

  • FreeBSD. A new pstat/pfreebsd component samples processes, nodes, disks, and interfaces, and psec/native finds the owner of a TCP peer there.

  • Build. flex is no longer required, the Python bindings now receive the build’s optimization and hardening flags, and the tree builds warning-free with gcc 16. The Cython check, the documentation’s Markdown converter, and the unit tests’ Python scripts all run with the Python interpreter configure found rather than the first python3 on PATH, and a tarball now finds Cython when it is installed only as a Python package.

  • Documentation. 277 new man pages cover the public API, docs/how-things-work describes the major subsystems, and the security policy now lives in SECURITY.md. New guides cover host environments, system administration, and moving from v5 to v7 for each role, schedulers included. The transition guide lists the NIC variables PMIx sets at launch, and the allocation-request and session-control pages explain how the two calls work together.

  • Testing. New unit tests, multi-node Docker Swarm suites, multi-user output-file tests, AddressSanitizer CI, cross-version CI with clients back to v2.1, a CI build without support for earlier releases, and topologies from Slingshot, CN5000, MI350, and AWS GPU nodes as device-discovery fixtures.

24.1.1.2. Compatibility notes

  • Out-of-tree MCA components must be rebuilt against the v7 headers; framework versions are now checked at load time, and the pnet interface is at 4.0.0. PMIX_CAP_MCA_FW_VERSION.

  • Earlier releases are supported only when built for. A default build speaks only this release’s wire format and has no TCP transport. Configure with --enable-legacy-clients for a server to take clients and tools from v2.1 on, then turn it on at run time with ptl_base_legacy_clients or PMIX_SERVER_LEGACY_PEERS; the server listens for them on loopback TCP only. Configure with --enable-legacy-servers for a client or tool to connect to servers from v3.2 through v6.x. --enable-legacy-peers and ptl_base_legacy_peers ask for both. A client of an earlier release under a server that does not take legacy clients cannot connect, and runs as a singleton. Refusals by version report PMIX_ERR_OUTDATED.

  • Tools on other hosts are not supported. A server takes connections from its own host only, and a process connects only to a server on its own host. PMIX_SERVER_REMOTE_CONNECTIONS is deprecated: true fails PMIx_server_init, and false changes nothing.

  • A connecting peer must present a credential. The handshake-only model is gone, and with it the psec/none and psec/dummy_handshake components. A host or tool that selected either must choose native or munge.

  • One connection per rank. A second connection for a connected rank is refused with PMIX_ERR_EXISTS, and PMIX_ALLOW_CLIENT_CLONES is deprecated and ignored. Every socket the library makes is close-on-exec, so an exec ends the process’s connection, and PMIx_Init and PMIx_tool_init remove the variables that identify the process and its server from the environment, so a process it starts afterwards does not inherit them.

  • Server URIs. PMIX_MYSERVER_URI is now always the local socket’s URI; PMIX_SERVER_URI is unchanged, and is what a process of an earlier release should be given. A TCP URI given alone to a tool, an attach, or a launcher is refused with PMIX_ERR_NOT_SUPPORTED: name the server’s local-socket URI, or file:<path>. A server refuses a peer of this release on TCP with the new PMIX_ERR_LOCAL_URI_REQUIRED. The - and + report-URI outputs print the local socket’s URI, and the new tcp- and tcp+ print the TCP URI.

  • No silent singleton fallback. PMIx_Init fails with PMIX_ERR_COMM_FAILURE when it cannot reach a server it was told about, or when it is given PMIX_NAMESPACE but no server URI. PMIX_ERR_UNREACH now means no server was described.

  • Connection waits are bounded by default. ptl_base_handshake_wait_time now defaults to 60 seconds and the new ptl_base_connect_ack_timeout to 5 seconds; set either to 0 for an unbounded wait.

  • Registration requirements. PMIx_server_register_nspace refuses job data that does not include PMIX_JOB_SIZE, and a client connects only once the host has registered its namespace. A session-level server must be given PMIX_SESSION_RNDZ_ID or PMIX_SESSION_ID, and a server given no namespace takes pmix-server-<host>-<pid>. A host should not register the namespace a self-started tool names.

  • A user’s groups come from the host. The library no longer looks a user’s groups up. A host supplies them with PMIX_USER_GRPIDS, for a job’s owner in PMIx_server_register_nspace and for any other user in PMIx_server_register_resources, where a single PMIX_GRPID names one group.

  • A scheduler or system controller serves tools unless the host passes PMIX_SERVER_TOOL_SUPPORT false, which now fails the init. A host is now told through client_finalized when a connected scheduler finalizes.

  • A host must not complete a collective whose PMIX_LOCAL_COLLECTIVE_STATUS is PMIX_ERR_TIMEOUT as a success: it was handed up without the members that never called. On a group construct that asked for PMIX_GROUP_FT_COLLECTIVE, PMIX_ERR_LOST_CONNECTION is a degraded success, and the group forms on the survivors.

  • Group membership keeps the order the host provided, and group ranks follow it. PMIX_GROUP_FINAL_MEMBERSHIP_ORDER is deprecated and ignored.

  • A blocking PMIx_IOF_pull now returns the registration id (a non-negative value) instead of PMIX_OPERATION_SUCCEEDED.

  • PMIx_Finalize and PMIx_tool_finalize can return PMIX_ERR_TIMEOUT, and PMIx_tool_finalize returns PMIX_ERR_WOULD_BLOCK when called on the progress thread.

  • PMIx_Compute_distances values differ from earlier releases because of the new PCIe term; relative order is otherwise unchanged.

  • Directories PMIx creates below a given root are reused only when owned by the effective uid, and existing directories PMIx is given are used with their permissions as found.

  • PMIX_SERVER_ALLOW_FOREIGN_TOOLS now defaults to false. Hosts that want tools from other users to connect must pass it; PRRTE does. Tools of the user the host names as PMIX_USERID are taken without it.

  • A command-line option value must now be a leading part of the name it matches, so packagefoo no longer matches package.

  • Support for the Solaris, Sun Studio, and KAI toolchains has been removed.

  • PMIX_COMPRESSED_STRING is deprecated.

  • A server answers a request for a job’s data only when the requester is permitted it, and returns PMIX_ERR_NO_PERMISSIONS when it is not. A host that registers a namespace for another user should name the owner, since a job with no owner named belongs to the user its clients run as.

  • PMIX_USERID and PMIX_GRPID may now be given as a user or group name as well as a number. palloc drops --uid and accepts a group name for --gid.

  • A tool looks for a server’s rendezvous file in the pmix-user directories of the users whose servers it may talk to – its own, root’s, and those ptl_base_server_access names – and in /run/pmix. It walks the system tmpdir, at most ptl_base_search_depth directories deep (default 8), only for a server of an earlier release when ptl_base_legacy_servers is set, and matches a file it was told to look for by its whole name rather than by prefix.

  • New limits: psensor_base_max_monitors_per_peer and pstat_base_max_monitors_per_peer (each 16 by default) cap the periodic monitors one process may hold, iof_max_partial_line (64KB) caps how long a partial output line is held before it is written out as it stands, and pmix_hwloc_max_cpu_index bounds the processor indices accepted in a cpuset list.

  • More limits: ptl_base_max_pending_connections (a quarter of the descriptor limit) and ptl_base_max_pending_bytes (16MB) bound connections still being read; pmix_server_max_event_regs_per_peer, pmix_server_max_iof_reqs_per_peer, and pmix_server_max_invites_per_peer (each 1024) cap what one connection may hold; pmix_max_events_per_source (a quarter of pmix_max_events) caps one namespace’s share of the event cache; pcompress_base_max_size (512MB) caps a decompressed size; plog_base_max_suppressed (1024) caps the log messages held for duplicate suppression; pmdl_ompi_max_param_file_size (1MB) caps an Open MPI parameter file read for a job; iof_writer_backlog (16384 chunks) caps output queued for a file another user’s job writes; runas_timeout (10 seconds) bounds work done as another user; and pmix_tool_finalize_timeout (5 seconds) bounds a tool’s wait at finalize.

  • PMIx reads a parameter file only if nobody could have written it but root, the user it runs as, the owner of the loaded PMIx library or of a directory above it, and – except when running as root – the members of a group it is in. The same holds for the directory holding the file unless that directory is sticky, and for the directory of a symbolic link’s target. A file that fails is passed over with a message saying why, and a FIFO at the file’s name no longer blocks startup. The MCA home-directory defaults are omitted when the user has no home directory.

  • pstat takes an empty field list to mean every field, and answers a pid the requester does not own as one with no process.

  • The installed pmix_getline() utility reads a line of any length, and takes a second argument that reports a read that stopped short.

  • The provisional resource-usage structures of v5.0 (pmix_proc_stats_t and its relatives), with their data types, functions, and macros, have been removed in favor of the attribute arrays the Standard adopted; use PMIx_Process_monitor. Three attributes that shared another attribute’s string in v5.0 – PMIX_FABRIC_DEVICE_COORDINATES, PMIX_JOB_CTRL_CHECKPOINT_TIMEOUT, and PMIX_QUERY_SUPPORTED_QUALIFIERS – now have strings of their own.

  • PMIX_SESSION_PROVISION, missing since v5.0, is restored, and PMIX_CAP_BASE is again 1, as it was in v5.0.

  • The Autotools versions used to build official release tarballs now live in VERSION as *_dist_version fields rather than inside contrib/make_dist_tarball, and the required toolchain is m4 1.4.21, autoconf 2.73, automake 1.19, and libtool 2.6.2.

A full list of individual changes will not be provided here, but will commence with the v7.0.1 release.