Access control by user and group
================================

.. note:: This is a **plan**. It describes the access model PMIx is
          moving to, and the order in which the pieces are being
          implemented. The `Status`_ section at the end says which
          parts are in place today. Anything not yet implemented
          behaves as it did before this work began.

This page is for two audiences:

* **Users and administrators**, who need to know who can see a job's
  data and act on a job, and how to widen that.
* **Developers** of PMIx and of host environments, who need to know
  where each check is made and what a host has to do.

Summary
-------

* Access to a job - reading its data, pulling its output, monitoring
  or signalling its processes - is decided by **who is asking** (their
  user ID and groups), not by which job the requester belongs to.
* By default, **only the job's owner** can access it. The owner can
  widen that when the job is started, by naming other users and/or
  groups with ``PMIX_ACCESS_PERMISSIONS``.
* Crossing jobs is normal and supported. A debugger or tool that asks
  about another job succeeds whenever its user is allowed.
* These rules govern requests that **clients and tools** make of a PMIx
  server. They do not restrict the **host environment** itself - see
  below.

Who is governed: the host versus clients and tools
--------------------------------------------------

A PMIx server lives inside a *host environment*: a resource manager,
launcher, or other system daemon that calls ``PMIx_server_init`` and
supplies the ``pmix_server_module_t`` up-calls. Many hosts run as a
privileged service account that is **not** root.

* **The host is not restricted.** The library does not limit what the
  host can do through the server APIs (registering jobs, delivering
  output, answering data requests, and so on), and does not second-guess
  what the host does with a request passed up to it. The host is the
  authority for its own operations.
* **Clients and tools are governed.** A request a client or tool sends
  to a server is checked against the target job's permissions:

  - by the **server library**, for data and operations it handles
    itself;
  - by the **host**, for operations the library passes up to it. For
    these the library tells the host exactly who is asking - the
    requester's process ID, ``PMIX_USERID`` and ``PMIX_GRPID`` (see
    :ref:`pmix_server_module_t(5) <man5-pmix_server_module_t>`) - and
    the host applies the same rule.

The requester's identity is the one established when it connected to
the server (the connection handshake verifies it against the host's
registration). It is not something the request itself can claim.

A request can pass through more than one server: a host relays what one
of its clients or tools asked for to another server - a workflow
manager to the resource manager, a launcher to a scheduler. The next
server is told who is connected to it - the relaying host - and, beside
that, what the relay claims about who first asked
(``PMIX_ORIGIN_USERID`` and the other ``PMIX_ORIGIN_*`` attributes).
Whether to believe that claim is the receiving host's decision, made
from the relay's own verified identity; the library never substitutes
the claim for it. See "Requests that pass through more than one
server" in :doc:`/hosts/authorization`.

Terms
-----

Owner
   The user (and primary group) a job belongs to - normally the user
   who started it.

Requester
   The client or tool making a request, identified by the user ID and
   group ID recorded when it connected and, on the server's local
   socket, by the groups the kernel reported for its process then.

Access list
   Additional users and groups the owner has allowed, given with
   ``PMIX_ACCESS_PERMISSIONS``.

The rule
--------

A requester may access a job if **any** of the following holds:

#. the requester's user ID is 0 (root);
#. the requester's user ID is the server's own effective user ID - the
   account the host runs as, which can already inspect the server
   process itself;
#. the requester's user ID is the job owner's;
#. the requester's user ID is in the job's access list;
#. the requester belongs to a group in the job's access list - one of
   the groups described under **Group membership** below.

Otherwise the request fails with ``PMIX_ERR_NO_PERMISSIONS``.

Establishing a job's permissions
--------------------------------

**The owner.** The host gives the job's owner when it registers the
job, as ``PMIX_USERID`` and ``PMIX_GRPID`` in the info passed to
``PMIx_server_register_nspace``. Either may be a number or a user or
group name; names are resolved when the job is registered. If the host
does not give them:

* the owner is the user and group the job's clients were registered
  with (``PMIx_server_register_client``);
* failing that, the owner is the server's own user.

A tool's owner is the user and group it presented when it connected.

**The access list.** A user widens access when starting a job, by
passing ``PMIX_ACCESS_PERMISSIONS`` in the job-level directives of
``PMIx_Spawn``. Its value is a data array of ``pmix_info_t`` holding
either or both of:

* ``PMIX_ACCESS_USERIDS`` - a data array of user IDs (numbers or names);
* ``PMIX_ACCESS_GRPIDS`` - a data array of group IDs (numbers or names).

The host passes the list on when it registers the job, on every node
that hosts a part of it. The same attributes already govern access to
published data (``PMIx_Publish``), so users meet one vocabulary.

The list widens access and names nobody's ownership. Only the two
attributes above are read inside it: a ``PMIX_USERID`` or ``PMIX_GRPID``
there is not the job's owner, and an access list nested inside another
is not read. ``PMIx_Spawn`` is refused with ``PMIX_ERR_BAD_PARAM`` when
its access list - or a ``PMIX_JOB_INFO_ARRAY`` among its job-level
directives - carries an identity attribute (``PMIX_USERID``,
``PMIX_GRPID``, ``PMIX_REQUESTOR_PID``, ``PMIX_REQUESTED_GRPID`` or one
of the ``PMIX_ORIGIN_*`` attributes), or when one access list is nested
inside another. The owner is the host's to name, at the top level of its
registration.

**Group membership** is the set of groups the requester's process
belongs to - its effective group and its supplementary groups. The set
belongs to the process, not to its user: two processes of one user may
hold different groups. The library never looks a user's groups up: no
directory lookup runs on its behalf, now or later. The set is taken from
the first of these that gives one:

#. **The kernel**, for a client or tool connected to the server's local
   socket: the groups of the connecting process, which the server reads
   from the connection when it is made - on Linux with ``SO_PEERGROUPS``
   (kernel 4.13 or later), and on macOS and FreeBSD with
   ``LOCAL_PEERCRED``, which reports at most 16 groups. On Linux, a group
   not mapped into the server's user namespace - which the kernel reports
   as the overflow gid - is left out. The effective gid the kernel
   reports for the process is always one of them. The set is
   the one the process had when it connected, and is kept for as long as
   the connection lasts.
#. **The host**, for the requester's user: the groups it gives with
   ``PMIX_USER_GRPIDS`` (a data array of group IDs, numbers or names):

   * for a job's owner, in that job's ``PMIx_server_register_nspace``;
   * for any other user - a tool's user, say - in
     ``PMIx_server_register_resources`` with ``PMIX_USERID``. A single
     ``PMIX_GRPID`` there says the user belongs to that group alone.

   These are used for a process on the local socket only where the kernel
   reports no list - on a platform without either option, or when reading
   it fails - and are what a requester connected over TCP, or a request
   the host relays from another node, is judged by.
#. **The one group the request is made under**, where the host gave no
   groups for the user: for a client or tool on the local socket, the
   effective gid the kernel reports for it; for one connected over TCP,
   the gid it connected with; for a request the host relays from another
   node, the ``PMIX_GRPID`` carried with it. A group check then passes
   only if the job's access list names exactly that group.

**Providing the host's groups is the host's responsibility.** The library
uses exactly the groups it was given until the host gives them again; a
host whose users' groups change registers them again (with the user's next
job, for example). A user the host gave no groups for has no record. Root
and the server's own user are never recorded - they need no groups. A
record is dropped when the host deregisters that user
(``PMIx_server_deregister_resources`` with ``PMIX_USERID``), and at
finalize.

A host that keeps its own records and wants the account's groups may call
``pmix_server_user_refresh()`` from its own thread; the library itself
never does.

What is covered
---------------

.. list-table::
   :header-rows: 1
   :widths: 28 42 30

   * - Area
     - Examples
     - Checked by
   * - Job-level data
     - ``PMIx_Get`` of another job's data (rank ``PMIX_RANK_WILDCARD``)
     - server library
   * - Process data on this node
     - ``PMIx_Get`` of a process's committed (modex) data
     - server library
   * - Process data on another node
     - ``PMIx_Get`` answered by direct modex
     - the server that holds the data (see below)
   * - Query and resolve
     - process tables, namespace information, ``PMIx_Resolve_peers``
     - server library for what it answers; host for the rest
   * - Output forwarding
     - ``PMIx_IOF_pull``, and a pull following a job into the jobs it
       spawns
     - host approves each pull; library for inheritance
   * - Output files
     - ``PMIX_IOF_OUTPUT_TO_FILE`` / ``_TO_DIRECTORY``
     - file ownership and mode (see below)
   * - Monitoring
     - ``PMIx_Process_monitor`` targets, process statistics
     - server library for processes on its node; host for others
   * - Job control
     - signal, kill, terminate, abort, stdin
     - host; server library for cleanup directives it handles
   * - Sessions and allocations
     - extending, releasing, spawning into an allocation
     - host (by user, not by namespace)

What the server library checks
------------------------------

Beyond the rule itself, a few things hold for every check the server
library makes:

* **A job's own processes** always access their own job.
* **The server's own namespace** describes the server, not anyone's
  job, and anyone may read it.
* **A job the host has not registered with this server** - one known
  here only because data for it arrived from elsewhere - has no
  permissions here to judge by. Requests for its data are checked by
  the server that holds it (see `Data held on another node`_). Acting
  on such a job - cleanup, monitoring, following its output - is
  allowed only for root and the server's own user.

Where the checks are made:

.. list-table::
   :header-rows: 1
   :widths: 30 70

   * - Request
     - Check
   * - ``PMIx_Get``, and a refresh of the client's cached copy
     - The requester must be allowed the target job. A request waiting
       for a job to be registered is checked again once it is.
   * - ``PMIx_Query_info`` naming a namespace (``PMIX_NSPACE``)
     - The requester must be allowed that job.
   * - ``PMIx_Resolve_peers``, ``PMIx_Resolve_nodes``
     - Naming a job, the requester must be allowed it. Naming none, the
       answer includes only the jobs the requester may access.
   * - ``PMIx_Process_monitor`` of processes on this node
     - Processes named by ``PMIX_MONITOR_TARGET_PROCS`` must all belong
       to jobs the requester may access, or the request is refused; each
       is read at the process ID its host recorded (``PMIX_PROC_PID``).
       Processes named by ``PMIX_MONITOR_TARGET_PIDS`` must all be owned
       by the requester, and -1 means every process it owns. A request
       for every local process leaves out those the requester may not
       access. Requests from the host are not restricted, except one it
       relays from another node (``PMIX_MONITOR_PROXY``), which must name
       the requester's ``PMIX_USERID`` and is checked for that user, and
       is refused without it. Each process must still belong to the
       expected user when it is read.
   * - Cleanup directives (``PMIX_REGISTER_CLEANUP`` and its family)
       on ``PMIx_Job_control``
     - The requester must be allowed every job it names, and the
       cleanup runs as the requester - each entry records who asked -
       never as the job's user or the server. A job the server does not
       know is refused.
   * - Output forwarding inherited by a spawned job
     - A subscription to a job's output follows into the jobs it
       spawns only if its requester may access each of them.

Data held on another node
-------------------------

When a requester asks for data the local server does not have, the
server asks its host, which fetches it from the server that holds it
(*direct modex*). The check is made **by the server that holds the
data**, because that server has the job's permissions:

* The requesting server names the requester in its ``direct_modex``
  up-call - ``PMIX_REQUESTOR``, ``PMIX_USERID`` and ``PMIX_GRPID``, the
  user and group being those established when the requester connected.
  Requesters of different identities are sent up separately; requesters
  of one identity share a fetch.
* The host passes that identity to the holding server through
  ``PMIx_server_dmodex_request2``, a new form of
  ``PMIx_server_dmodex_request`` that takes an info array (every PMIx
  API carries one).
* The holding server applies the rule and answers, or refuses with
  ``PMIX_ERR_NO_PERMISSIONS``.
* The requesting server keeps a copy of what it receives, and remembers
  which identities the holding server approved. From that copy it
  answers only those, the job's own processes, root, and its own user;
  any other requester is asked about as though the copy were not there,
  and gets its own fetch. This applies to ``PMIx_Get``, to a refresh of
  a client's cache, and to query and resolve answered from the copy.

A host that still uses ``PMIx_server_dmodex_request`` passes no
requester identity, and the holding server answers as it does today.
The capability flag ``PMIX_CAP_DMODEX_REQUEST2`` tells a host that the
library provides ``PMIx_server_dmodex_request2``.

Output files and shared memory
------------------------------

**Output files** (``PMIX_IOF_OUTPUT_TO_FILE``,
``PMIX_IOF_OUTPUT_TO_DIRECTORY``). The paths a job names are the job
owner's, and **everything about its output files is done as the owner**:
the directories are created, and the files opened, checked, emptied and
given their mode, by the owner - in place when the server runs as the
owner, and otherwise in a short-lived child of the server that has
dropped to the owner's uid, gid and the groups the host registered for it
(``PMIX_USER_GRPIDS``; with none, the owner's gid alone). A system takes
only so many groups for a process (``sysconf(_SC_NGROUPS_MAX)``: 16 on
macOS); the child has the owner's gid and as many of the others, in the
order the host gave them, as fit. On Linux the child also gives up any
capability it still holds once it is the owner - a server given
``CAP_SETUID`` as a file capability keeps the rest of its capabilities
across the change of user - and does nothing if one remains. So the kernel
applies the owner's own permissions to every path the job named: a
privileged server creates nothing, and writes over nothing, that the
owner could not.

* A file and its directories belong to the owner from creation: ``0600``
  and ``0700``.
* If the access list names groups, the files are made readable by the
  first listed group (``0640``, directories ``0750``), where the owner can
  set that group. File modes can express only an owner and one group, so
  further users and groups in the list are not applied to files. Access
  control lists are not used, as they are not portable.
* An output file that is not a regular file with a single name, **owned by
  the job's owner**, is not written - an existing file of someone else's
  is refused and left as it was, even where the owner could write it.
* **The launcher writes them, when it can.** A tool that spawns a job and
  is sent the job's output - ``prun``, say - names itself the job's file
  writer (``PMIX_IOF_FILE_WRITER``). The server that takes the spawn
  accepts that only from the tool it names, and passes it on to jobs that
  the job's processes spawn with output files of their own. No
  server then writes that job's files: the one sending the job's output
  to the tool sends the file directives with the first of it, and the
  tool writes the files as itself. So a server running as a service
  account other than root can run another user's job with output files.
  The files are written where the tool runs, not where each process
  runs.
* Otherwise a server that can neither run as the owner nor become it -
  one running as a service account other than root - writes no output
  files for that owner's jobs, and says so once per job, when the job is
  registered - before any of its processes run.
* **The owner's child is given a deadline.** It works on paths the owner
  chose, which may be on a filesystem that does not respond. A server
  waits for it no longer than the ``runas_timeout`` MCA parameter (10 seconds by
  default), then kills it and does not write that file. None of the
  job's other output files is tried after that, since each would cost
  the same wait. The files themselves are then written by a thread of
  their own rather than by the one that serves every job on the node,
  so a file that stops answering later holds only that thread. What is
  held for such a file meanwhile is limited by the
  ``iof_writer_backlog`` MCA parameter, and output beyond it is not
  written to the file. The same deadline bounds the cleanup a job registers
  (``PMIX_REGISTER_CLEANUP``, ``PMIX_REGISTER_CLEANUP_DIR``), which is
  done as the requester in the same way. Setting the parameter to 0 waits
  indefinitely.

**Shared-memory job data** (``gds/shmem3``). Only the server writes a
segment: clients open its backing file read-only and map it read-only.
The file belongs to the job's owner, with mode ``0400`` - ``0440`` for
the first group the access list names - and never grants write access.
A client the mode does not admit - of another user's job, or of a job
whose file the server could not give to its owner - receives the job's
data from the server instead, where the rule is applied.

Not covered
-----------

* **Event notification.** The process raising an event chooses who
  receives it through the event's range. The one exception is
  ``PMIX_DEBUGGER_RELEASE``, which lets a process held for a debugger
  continue: a server accepts it from a client or tool only if the rule
  above permits that requester for the job of every process the event can
  reach. A job the server has not had registered is left to the host,
  which relays the event.
* **Published data** (``PMIx_Publish`` / ``PMIx_Lookup``). The host's
  data server already governs it per published item with
  ``PMIX_ACCESS_PERMISSIONS``.

Compatibility
-------------

* **A host that registers no owner** gets the fallback above: the user
  its clients were registered with, then the server's own user. On a
  node that hosts none of a job's processes there are no clients, so
  there only root and the server's own user can access that job. A host
  running as the same user as its jobs sees no change; a host running
  as a service account should register each job's owner.
* **A host that does not pass ``PMIX_ACCESS_PERMISSIONS``** at
  registration gets owner-only access for every job, which is the
  default.
* **Clients and tools of any version** are identified by their
  connection, so nothing changes for them.
* **Hosts** can detect the library's support through capability flags
  in ``pmix_version.h``: ``PMIX_CAP_REQUESTER_ID`` (the requester's
  identity on up-calls), ``PMIX_CAP_ORIGIN_ID`` (an up-call's
  identity is always the connected peer's, with ``PMIX_REQUESTOR_PID``,
  ``PMIX_REQUESTED_GRPID`` and a relay's ``PMIX_ORIGIN_*`` claims
  beside it), ``PMIX_CAP_DMODEX_REQUEST2``
  (``PMIx_server_dmodex_request2``) and ``PMIX_CAP_ACCESS_CHECK``
  (``pmix_server_access_check()``).

What a host needs to do
-----------------------

* Register each job's owner (``PMIX_USERID``, ``PMIX_GRPID``) and access
  list (``PMIX_ACCESS_PERMISSIONS``) with ``PMIx_server_register_nspace``
  on every node that hosts part of the job.
* Accept ``PMIX_ACCESS_PERMISSIONS`` in spawn requests and carry it with
  the job.
* Apply the rule to the operations it performs for clients and tools -
  job control, abort, stdin, IOF pull approval, queries it answers,
  monitoring of other nodes, sessions and allocations - using the
  requester identity the library passes in each up-call. A host linked
  with the library applies exactly the library's rule by keeping its own
  copies and calling ``pmix_server_access_check()`` (declared in the
  installed ``src/server/pmix_server_ops.h``) from its own thread: a
  ``pmix_access_t`` for each job, loaded with
  ``pmix_server_access_load()`` from the info it registers the job with,
  and a ``pmix_user_t`` for each requester. None of these touch the
  library's state. When the host drops a user it deregisters it from the
  library too, so the two stay in step. The ``PMIX_GRPID`` in an up-call
  is the group the requester chose to charge the work to and is not
  verified: it is not an input to the rule.
* Carry the requester's identity when it relays a request to another of
  its daemons.
* Answer direct-modex requests with ``PMIx_server_dmodex_request2``,
  passing the requester's identity.

PRRTE is the reference host, and implements these alongside the library
changes.

Status
------

.. list-table::
   :header-rows: 1
   :widths: 8 62 30

   * - Phase
     - Work
     - State
   * - 0
     - The requester's process ID, ``PMIX_USERID`` and ``PMIX_GRPID`` are
       passed on every up-call made for a client or tool; ``PMIX_REQUESTOR``
       where an up-call has no process argument; user and group names
       accepted; IOF pulls forwarded only after the host approves.
       PRRTE approves an IOF pull only for root, the DVM's user, or the
       job's owner.
     - Done
   * - 1
     - Each job's owner and access list stored by the server library,
       from the host's registration; one check implementing the rule;
       group membership cached per user.
     - Done
   * - 2
     - The server library applies the rule to what it answers itself:
       job and process data, query and resolve, monitoring of local
       processes, cleanup directives, and IOF inheritance.
     - Done
   * - 3
     - ``PMIx_server_dmodex_request2``; the requester's identity on the
       ``direct_modex`` up-call; the check at the holding server; the
       requesting server answers from its copy only the requesters the
       holder approved. PRRTE registers each job's owner
       (``PMIX_USERID``, ``PMIX_GRPID``) with
       ``PMIx_server_register_nspace``, carries the requester's identity
       on its direct-modex relay, and uses
       ``PMIx_server_dmodex_request2``.
     - Done
   * - 4
     - Output file ownership and modes; server-mediated job data for
       requesters the shared-memory file mode does not admit, with
       shared-memory segments read-only to their readers.
     - Done
   * - 5
     - ``pmix_server_access_check()``, the rule as a function a host
       calls with its own copies of a job's access list and the
       requester's groups; users registered and deregistered by the host,
       their groups as the host gives them (``PMIX_USER_GRPIDS``); a monitor
       request relayed from another node checked for the requester it
       names. PRRTE carries each job's access list (``--rtos
       users=,groups=``, and ``PMIX_ACCESS_PERMISSIONS`` at spawn) and
       applies the rule to the operations it performs.
     - Done
