Events sent immediately after joining can be incorrectly soft-failed

[MadLittleMods]

Description

The Gitter bridge seems to have reproduced a race condition in federation where because the appservice joins and sends a message in quick succession, the message can out run the join which causes the message to soft_fail.

To see an actual reproduction, there is a missing message in the !SrkiFczPSWmYrlSNYF:matrix.org room for matrix.org homeserver users ($mMJdjGPC2GaoWK3pLApa_O2vBoeuQwYtP6iY6XzbrfY). This is tracked on Gitter by https://gitlab.com/gitterHQ/webapp/-/issues/2770#note_583925093

The event exists in the gitter.im homeserver database and is visible to the gitter.im homeserver where the Gitter bridge appservice operates. The message is also visible to any new homeserver that comes in the room and backfills the messages. It's only not visible on the matrix.org homeserver.

The message also exists on the matrix.org homeserver database but is soft_failed (thanks to @richvdh for checking the database)

@Half-Shot has also seen this happen a few times with the IRC bridge.

As @leonerd from #1444 and @Half-Shot mentioned, this is probably not a problem for normal users because the time taken to join and then send a message is some number of seconds. Whereas with appservices, can join and send an event in quick succession (almost instantly).

Related issues:

• events sent immediately after joining can be incorrectly soft-failed #6536: events sent immediately after joining can be incorrectly soft-failed
• events get soft-failed when the federation_inbound worker is busy #7744: events get soft-failed when the federation_inbound worker is busy
• Setting profile + joining a room can race (SYN-787) #1444: Reproduces with a join, set profile, membership event (not over federation though).
• Soft-failures make federated bans racy and frustrate redaction #9329

Steps to reproduce

1. Create a room on HS1, !my-room:hs1
2. Setup HS2 with an appservice
3. Join and send a message through the appservice to !my-room:hs1
   • The appservice is probably not necessary to reproduce. Just send a join and message event from HS2
4. Check whether the message is visible on hs1
5. Since it's a race condition, this probably does not always reproduce

Version information

• Homeserver: gitter.im -> matrix.org

[richvdh]

I think this might be the same as #6536 (though with a much better description)?

[MadLittleMods]

@richvdh #6536 does look like a good culprit! Should we close one in favor of the other?

I've marked it as S-Critical because it causes data loss (message is not visible). Feel free to re-prioritize more appropriately though.

[richvdh]

I closed #6536, since this has a bunch more detail

[MadLittleMods]

Another reproduction at https://gitter.im/matrix-org/gitter?at=60c0ecae84c2f15b796e33a2 which was sent to !OvgDmYaPOFRRWnIdQe:half-shot.uk as $r3aSycTMNPGi6gKK5-N7jw8fAEt3I2Q1lfW0LdwUyns but failed to federate over to matrix.org.

You can only see their join event on matrix.org, https://matrix.to/#/!OvgDmYaPOFRRWnIdQe:half-shot.uk/$wB_7bSk-IGb0Czes_BCV3W3DIRPsxeGw-7ZTSbvkaus?via=half-shot.uk&via=matrix.org&via=gitter.im

-- https://gitlab.com/gitterHQ/webapp/-/issues/2770#note_597176264