MSC3706: Extensions to /_matrix/federation/v2/send_join/{roomId}/{eventId} for partial state (#3706)

Author: richvdh

proposals/3706-partial-state-in-send-join.md

@@ -0,0 +1,122 @@
+# MSC3706: Extensions to `/_matrix/federation/v2/send_join/{roomId}/{eventId}` for partial state
+
+## Background
+
+It is well known that joining large rooms over federation can be very slow (see,
+for example, [synapse#1211](https://github.com/matrix-org/synapse/issues/1211)).
+
+Much of the reason for this is the large number of events which are returned by
+the [`/send_join`](https://spec.matrix.org/v1.2/server-server-api/#put_matrixfederationv2send_joinroomideventid)
+API, and must be validated and stored.
+
+[MSC3902](https://github.com/matrix-org/matrix-doc/pull/3902) gives an overview
+of changes to the matrix protocol to improve this situation. This MSC focuses
+on a specific aspect of those suggestions by proposing specific changes to the
+`/send_join` API.
+
+## Proposal
+
+[`PUT /_matrix/federation/v2/send_join/{roomId}/{eventId}`](https://spec.matrix.org/v1.2/server-server-api/#put_matrixfederationv2send_joinroomideventid)
+is extended to support "partial state" in its responses. This involves the
+following changes.
+
+### New query parameter
+
+`omit_members` is added as a new query parameter. This can take the values
+`true` or `false`; other values should be rejected with an HTTP 400 error with
+matrix error code `M_INVALID_PARAM`.
+
+Calling servers use this parameter to indicate support for partial state in
+`send_join`. If it is not set to `true`, receiving servers continue to behave
+as they do today.
+
+### Changes to the response
+
+Receiving servers are not obliged to implement partial state: they are also
+free to support it for some rooms and not others.
+
+The following changes are made to the response:
+
+ * `members_omitted`: a new boolean field is added. This should be set to `true`
+   to indicate that `m.room.member` events have been omitted from the response.
+   It must otherwise be set to `false` or omitted.
+
+ * `state`: if partial state is being returned, then state events with event
+   type `m.room.member` are omitted from the response. All other room state is
+   returned as normal. (See 'Alternatives' for discussion on why only
+   `m.room.member` events are omitted.)
+ 
+ * `auth_chain`: The spec currently defines this as "The auth chain for the
+   entire current room state". We instead rephrase this as:
+
+   > All events in the auth chain for the returned join event, as well as
+   > those in the auth chains for any events returned in `state`.
+
+   (Note that in the case that full state is being returned, the two
+   definitions are equivalent.)
+
+ * If the `omit_members` query parameter was set, we make a further
+   optimisation to `auth_chain`:
+
+   > Any events returned within `state` can be omitted from `auth_chain`.
+
+   For example: the `m.room.create` event is part of the room state, so
+   must be included in `state`. However, it also forms part of the auth chain
+   for all of the returned events, so in the current spec, must *also* be
+   included in `auth_chain`. However, this is redundant, so we should omit it
+   for calling servers which opt into that via the `omit_members` query param.
+
+ * `servers_in_room`: A new field of type `[string]`, listing the servers
+   active in the room (ie, those with joined members) before the join.
+
+   This is to be used by the joining server to send outgoing federation
+   transactions while it synchronises the full state, as outlined in [MSC3902](https://github.com/matrix-org/matrix-spec-proposals/pull/3902).
+
+   This field is **required** if the `members_omitted` response field is true; it
+   is otherwise optional.
+
+## Potential issues
+
+None at present.
+
+## Alternatives
+
+ * In future, the list of event types to omit could be expanded. (Some rooms
+   may have large numbers of other state events).
+   
+   Currently, `m.room.member` events are by far the biggest problem. For
+   example, a `/send_join` request for Matrix HQ returns approximately 85000
+   events in `state`, of which all but 44 are `m.room.member`. 
+
+   Additionally: the client-server API already provides a mechanism for
+   omitting `m.room.member` events from the `/sync` response, via
+   [lazy loading](https://spec.matrix.org/v1.4/client-server-api/#lazy-loading-room-members),
+   which means that this change can be implemented without changes to the
+   client-server API. Extending the list of event types to be omitted would
+   require changes to the client-server API and therefore be significantly more
+   involved.
+
+   In order to reduce the scope of the change, we have therefore decided to
+   focus on `m.room.member` events for now. Future MSCs might provide a
+   mechanism for omitting other event types.
+ 
+## Security considerations
+
+No security issues are currently foreseen with this specific MSC, though the
+larger topic of incremental synchronisation of state has several concerns;
+these will be discussed in other MSCs such as MSC3902.
+
+## Unstable prefix
+
+The following mapping will be used for identifiers in this MSC during
+development:
+
+Proposed final identifier | Purpose         | Development identifier
+------------------------- | --------------- | ----
+`omit_members`            | query parameter | `org.matrix.msc3706.partial_state`
+`members_omitted`         | response field  | `org.matrix.msc3706.partial_state`
+`servers_in_room`         | response field  | `org.matrix.msc3706.servers_in_room`
+
+## Dependencies
+
+This MSC does not build on any existing unaccepted MSCs.