MSC3552: Extensible Events - Images and Stickers

proposals/3552-extensible-events-images.md

@@ -0,0 +1,144 @@
+# MSC3552: Extensible Events - Images and Stickers
+
+[MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767) describes Extensible Events in detail,
+though deliberately does not include schemas for some messaging types. This MSC covers only images
+and stickers.
+
+*Rationale*: Splitting the MSCs down into individual parts makes it easier to implement and review in
+stages without blocking other pieces of the overall idea. For example, an issue with the way images
+are represented should not block the overall schema from going through.
+
+This MSC additionally relies upon [MSC3551](https://github.com/matrix-org/matrix-doc/pull/3551).
+
+## Proposal
+
+Using [MSC1767](https://github.com/matrix-org/matrix-doc/pull/1767)'s system, a new event type
+is introduced to describe applicable functionality: `m.image`. This event type is simply an image
+upload, akin to the now-legacy [`m.image` `msgtype` from `m.room.message`](https://spec.matrix.org/v1.1/client-server-api/#mimage).
+
+An example is:
+
+```json5
+{
+  "type": "m.image",
+  "content": {
+    "m.markup": [
+      // Format of the fallback is not defined, but should have enough information for a text-only
+      // client to do something with the image, just like with plain file uploads.
+      {"body": "matrix.png (12 KB) https://example.org/_matrix/media/v3/download/example.org/abcd1234"}
+    ],
+    "m.file": {
+      "url": "mxc://example.org/abcd1234",
+      "name": "matrix.png",
+      "mimetype": "image/png",
+      "size": 12345
+    },
+    "m.image_details": { // optional
+      "width": 640,
+      "height": 480
+    },
+    "m.thumbnail": [ // optional
+      {
+        // A thumbnail is an m.file+m.image, or a small image
+        "m.file": {
+          "url": "mxc://exmaple.org/efgh5678",
+          "mimetype": "image/jpeg",
+          "size": 123
+
+          // "name" is optional in this scenario
+        },
+        "m.image_details": {
+          "width": 160,
+          "height": 120
+        }
+      },
+      // ...
+    ],
+    "m.caption": { // optional - goes above/below image
+      "m.markup": [{"body": "Look at this cool Matrix logo"}]
+    },
+    "m.alt_text": { // optional - accessibility consideration for image
+      "m.markup": [{"body": "matrix logo"}]
+    }
+  }
+}
+```
+
+With consideration for extensible events, the following content blocks are defined:
+
+* `m.image_details` - Currently records width and height (both required, in pixels), but in
+  future could additionally supply other image details such as colour space.
+* `m.thumbnail` - An array of (usually) smaller images the client can use to show in place of
+  the event's image for bandwidth or size considerations. Currently requires two other content
+  blocks nested under it: `m.file` and `m.image_details`.
+  * Clients should find the thumbnail most suitable for them - the array is not ordered, but
+    encouraged to have smaller images (by byte size) first.
+  * Multiple thumbnail formats may be supplied (webp, webm, jpeg, etc) with the same dimensions.
+    Clients should ensure they are capable of rendering the type before picking that thumbnail.
+  * `m.file`'s `mimetype` is a required field in this block.
+  * `m.file`'s `name` is optional in this block.
+* `m.alt_text` - Alternative text for the content, for accessibility considerations. Currently
+  requires an `m.markup` content block to be nested within it, however senders should only
+  specify a plain text body for ease of parsing.
+  * *Note*: We use the full capability of `m.markup` here not for mimetype, but future support
+    for translations and other text-based extensions.
+
+Together with content blocks from other proposals, an `m.image` is described as:
+
+* **Required** - An `m.markup` block to act as a fallback for clients which can't process images.
+* **Required** - An `m.file` block to contain the image itself. Clients use this to show the image.
+* **Optional** - An `m.image_details` block to describe any image-specific metadata, such as dimensions.
+  Like with existing `m.room.message` events today, clients should keep images within a set of
+  reasonable bounds, regardless of sender-supplied values. For example, keeping images at a minimum
+  size and within a maximum size.
+* **Optional** - An `m.thumbnail` block (array) to describe any thumbnails for the image.
+* **Optional** - An `m.caption` block to represent any text that should be shown above or below the
+  image. Currently this MSC does not describe a way to pick whether the text goes above or below,
+  leaving this as an implementation detail. A future MSC may investigate ways of representing this,
+  if needed.
+* **Optional** - An `m.alt_text` block to represent alternative/descriptive text for the image. This
+  is used as an accessibility feature, and per the block's definition above should only contain a plain
+  text representation at the moment. Clients are encouraged to assume there is no alt text if no plain
+  text representations are present. For clarity, this value would be supplied to the `alt` attribute
+  of an `img` node in HTML.
+
+The above describes the minimum requirements for sending an `m.image` event. Senders can add additional
+blocks, however as per the extensible events system, receivers which understand image events should not
+honour them.
+
+To represent stickers, we instead use a mixin on `m.image_details`. A new (optional) boolean field
+called `m.sticker` is added if the client is intended to render the image as a sticker. When rendering
+as a sticker, the `m.caption` can be shown as a tooltip (or similar) rather than inline with the image
+itself. `m.sticker` defaults to `false`.
+
+The [`m.sticker` event type](https://spec.matrix.org/v1.1/client-server-api/#msticker) is deprecated
+and removed, like `m.room.message` in MSC1767.
+
+Note that `m.file` supports encryption and therefore it's possible to encrypt thumbnails and images
+too.
+
+If a client does not support rendering images inline, the client would instead typically represent
+the event as a plain file upload, then fall further back to a plain text message.
+
+## Potential issues
+
+The schema duplicates some of the information into the text fallback, though this is unavoidable
+and intentional for fallback considerations.
+
+## Alternatives
+
+No significant alternatives known.
+
+## Security considerations
+
+The same considerations which currently apply to files, images, stickers, and extensible events also
+apply here. For example, bounds on image size, assuming sender-provided details about the file are
+false, etc.
+
+## Unstable prefix
+
+While this MSC is not considered stable, implementations should use `org.matrix.msc1767.*` as a prefix in
+place of `m.*` throughout this proposal. Note that this uses the namespace of the parent MSC rather than
+the namespace of this MSC - this is deliberate.
+
+Note that extensible events should only be used in an appropriate room version as well.