WPB-240: Generate and fan out events about stopping to federate (#3397)

lepsa · fisx · battermann · web-flow · commit 6db29bb1b245 · 2023-07-18T09:31:11.000+02:00
* Breadcrumbs. * Better breadcrumbs. We don't know which release this will land in yet, PR is more future-proof. * Polish docs. * afterthought. * Update services/brig/test/integration/API/Federation.hs * More tests. * Remove a lying comment. * Refactor: keep the semantics of `_runSettings` intact. (This is part of the `Opts` data structure read from a file. it's very confusing to start updating part of that on the fly, and ignoring whatever comes from the config file.) * Remove a lying comment. * Fixup / WIP * Fixup * I take it back! * more docs. * rm bogus TODO, add a non-bogus one :) * revert changes of status code in integration tests. (Double-check if we can't keep the prod code behavior intact to avoid upgrade / fed-to-fed api / client api issues.) * Various updates, mainly moving some common code into wire-api and minimising types that are used in several common paths. * FS-1115: More common code for federation domain updates * Use tinylog instead of `print`. * sanitize-pr * Use retry instead of threadDelay-loop. * FS-1115: Updating design docs on how update intervals are supplied * Clarify source comment. * FS-1115: Cleaning up un-needed changes to options * update docs.wire.com * docs. * docs. * Fix: `/i/user/meta-info` (stern) (#3281) * Handle race conditions in /integration (#3278) * Update docs.wire.com (#3284) * Deprecate start-services-only.sh * Fix outdated docs on how to run swagger docs locally. * Restore deleted scripts (#3287) * Refactor federation domain configuration. - move strategy config from federator to brig. - remove domain list from strategy type. - add 'none' strategy (more explicit that 'list' with empty list, but both still work) * Fixup * Complete CRUD api for federator remotes (Update is missing) [WIP]. * Complete CRUD api for federator remotes (Update is missing): tests. * docs. * Cleanup * FS-1179: Setting up a location in galley for processing deleted domains NOTE: Broken! Currently not compiling Adding a new query to cassandra for getting remote members and conversation IDs that they are in, setting up loops for removing these members and potentially conversations using existing APIs. * FS-1179: Fixing the polysemy error. Picking an error type and documenting what might need to be done for better error mapping. * FS-1179: Initial processing for deleting remote domains. Setting up Galley to remove remote users from local conversations, local users from remote conversations, and to delete all user connections for the federation domain that is being dropped. * FS-1179: Fixing an error http reponse code. * wip * Remove dead code. * nit-pick. * Typo. * Fine-tune logging. * Fix haddocks. * Implement put. * Clariy updateFrequency everywhere. * nit-pick. * Fixup * Cleanup * Implement put. (For real this time.) * Move integration tests to /integration. * FS-1179: Work in progress. Tests are broken due to timing issues * FS-1179: Fixing an errant delete * FS-1179: Commiting and switching to FS-1115 * ... * wip * ... * FS-1115: Setting up calls to Brig from integration tests. JSON, calling, etc. * ... * ... * ... * ... * ... * ... * Fix * ... * WIP: Chasing down cache issues between some bug fixes * ... * ... * ... * ... * Tests are passing! * process leif's feedback. * ... * Fixing compile issues after a merge * Updating templates * Removing a JSON roundtrip, better using aeson. * Moving more of the federation domain update code into wire-api. Tests are broken with these changes. * wip * sanitize-pr * FS-1115: Updating brig integration config to help tests * s/AllowList/AllowDynamic/g * rm trailing whitespace. * docs * Updating with PR feedback * Have federation domain tests use MakesValue more * FS-1115: Removing more FedConn type specific code from federation tests * More code leaning on typeclasses * Pre-emptive rework before merging * Mark flaky test case. * docs. * Formatting * FS-1179: Setting up the receiving RabbitMQ loop. Setup a reading loop for RabbitMQ to get the domains that need to be deleted. We're using a queue here so that we have a non-volatile store for these domains that can automatically handle galley instances dying or being scaled down. * wip * FS-1179: Refactoring the RabbitMQ code Refactoring RabbitMQ code so that there is a single channel for both production and consumption of messages. There is also a function for "simple" processing where a message queuing system isn't available, but it still needs to have persistence and task co-ordination defined for it. This may not be a problem if we decide to mandate the use of RabbitMQ, or a similar queue. * Code formatting * FS-1179: Fixing tests using the wrong domain. Using the wrong domain in the galley code, not the test directly, was causing errors that were annoying to debug. * docs. * Changelog. * docs. * sanitize-pr * Fixup * FS-1179: Moving a delete call into Brig, and calling it from Galley. Moving the deletion of one-on-one conversations into Brig, as Galley can't access the tables where that info is stored. This added a new internal API endpoint. Integration tests are currently failing on trying to delete one-on-one connections in Brig. * FS-1179: Fixing tests by telling cassandra to run queries anyway * Update docs/src/understand/configure-federation.md Co-authored-by: Sven Tennie <sven.tennie@gmail.com> * Update docs/src/developer/developer/federation-design-aspects.md Co-authored-by: Sven Tennie <sven.tennie@gmail.com> * FS-1179: Coping code from other tests to setup remote conversations. Setitng up remote conversations, using code from another test. Querying this database status requires messing around with Cql and polysemy rather than being able to look at API requests and reponses. The test is currently broken and I need to dig into why, and it is possible that this is a timing issue between brig and galley, but I'm not certain. * Fix cannon config map * whitespace * FS-1179: Improving tests in galley, and adding a new test to brig * FS-1179: PR formatting * Adding a round-trip test for rabbitmq. Using the pub and sub functions from the main galley code * hi ci * docs. * docs. * FS-1179: Add an internal endpoint for deleting a federation domain * FS-1179: Adding a defederation worker to background worker Adding a new worker on background worker that reads from a queue and calls Galley to delete a federation domain. * FS-1179: Some timeout config for the background worker. * Keeping brig and backend worker in sync * WIP: New tests and various changes to background worker * hi ci * Tweak docs. * nit-pick. * Mark test case as flaky. * Tweak log levels. * Tweak log msg. * wip * Update libs/wire-api/src/Wire/API/Routes/Internal/Brig.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update docs/src/understand/configure-federation.md Co-authored-by: fisx <mf@zerobuzz.net> * Update services/brig/src/Brig/API/Internal.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/brig/src/Brig/API/Internal.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/brig/src/Brig/API/Internal.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/federator/src/Federator/Run.hs Co-authored-by: fisx <mf@zerobuzz.net> * Update services/galley/src/Galley/App.hs Co-authored-by: fisx <mf@zerobuzz.net> * FS-1115: PR notes * try fix helm charts * WIP * hi ci * changed federation strategy in CI from allowDynamic to allowAll * correct error in case of allowDynamic * default case made explicit * FS-1179: Modifications to tests to help stop hangups * Cleaning up comments * Create use domain names in integration tests. (this avoids hypothetical concurrency problems.) * Better errors in /integration. * nit-picks. * Better errors in /integration. * revert commit noise. * Fix `make list-flaky-tests`. * nit-picks, renames, minor refactorings. * s/type/newtype/ * Polish Wire.API.FederationUpdate (names and declaration order). * Fix: Always cancel `syncFedDomainUpdateThread`. * hi ci * Fixing things post-merge * Adding a changelog entry * FS-1179: Brig now deletes the notification queue for deleted domains * FS-1179: Removing and errant import * FS-1179: PR sanitisation. * FS-1179: Updating a TODO * FS-1179: Writing cleanup code so that we don't have dangling threads * FS-1179: PR formatting * PR formatting * More comments and exception handling * Fixing tests now that they have moved to hspec * Formatting * Fix background-worker integration tests. * Updating the internal notes * WPB-240: Initial implementation of notifications sending for defederation * Removing a redundant language extension * Removing a redundant language extension * WPB-240: Matching event fanout count to the limit in options. Adding tests for event fanout. * WPB-240: Comments and code in tests for quickly filling the DB * FS-1179: Reworking after a discussion with Akshay about how to use AMQP. Workers are going to have their own connection management to RabbitMQ which will keep things simpler and help reduce potential blocks in processing. It also helps us not step on each other's toes when writing this service. * PR formatting * Hi CI * FS-1179: Adding galley and brig to background-worker's configmap * FS-1179: Removing the confusing "integration" tests for background-worker They were never actual integration tests, I was just confused about why `make ci` wasn't picking up the existing tests, so I made it run them and that was a mistake and caused confusion. * PR notes * `make sanitize-pr`. * Fixup * Changing the format of federation.delete. * Sanitizing code * FS-1179: Removing dead code and bumping schema migrations * FS-1179: Removing more dead code * PR formatting * Hi CI * WPB-240: Formatting code * WPB-240: Changelog entries * WPB-240: Removing dead code * WPB-240: Moving notification code into an effect, as mentioned in PR. --------- Co-authored-by: Matthias Fischmann <mf@zerobuzz.net> Co-authored-by: Leif Battermann <leif.battermann@wire.com> Co-authored-by: Stefan Matting <smatting@users.noreply.github.com> Co-authored-by: Sven Tennie <sven.tennie@gmail.com>
diff --git a/changelog.d/1-api-changes/WPB-240 b/changelog.d/1-api-changes/WPB-240
@@ -0,0 +1,3 @@
+Added a new notification event type, "federation.delete".
+This event contains a single domain for a remote server that the local server is de-federating from.
+This notification is sent twice during de-federation. Once before and once after cleaning up and removing references to the remote server from the local database.
diff --git a/changelog.d/6-federation/WPB-240 b/changelog.d/6-federation/WPB-240
@@ -0,0 +1 @@
+De-federating from a remote server sends a pair of notifications to clients, announcing which server will no longer be federated with.
diff --git a/libs/wire-api/src/Wire/API/Event/Federation.hs b/libs/wire-api/src/Wire/API/Event/Federation.hs
@@ -0,0 +1,64 @@
+module Wire.API.Event.Federation
+  ( Event (..),
+    EventType (..),
+  )
+where
+
+import Data.Aeson (FromJSON, ToJSON)
+import qualified Data.Aeson as A
+import qualified Data.Aeson.KeyMap as KeyMap
+import Data.Domain
+import Data.Json.Util (ToJSONObject (toJSONObject))
+import Data.Schema
+import qualified Data.Swagger as S
+import Imports
+import Wire.Arbitrary
+
+data Event = Event
+  { _eventType :: EventType,
+    _eventDomain :: Domain
+  }
+  deriving (Eq, Show, Ord, Generic)
+
+instance Arbitrary Event where
+  arbitrary =
+    Event
+      <$> arbitrary
+      <*> arbitrary
+
+data EventType
+  = FederationDelete
+  deriving (Eq, Show, Ord, Generic)
+  deriving (Arbitrary) via (GenericUniform EventType)
+  deriving (A.FromJSON, A.ToJSON, S.ToSchema) via Schema EventType
+
+instance ToSchema EventType where
+  schema =
+    enum @Text "EventType" $
+      mconcat
+        [ element "federation.delete" FederationDelete
+        ]
+
+eventObjectSchema :: ObjectSchema SwaggerDoc Event
+eventObjectSchema =
+  Event
+    <$> _eventType .= field "type" schema
+    <*> _eventDomain .= field "domain" schema
+
+instance ToSchema Event where
+  schema = object "Event" eventObjectSchema
+
+instance ToJSONObject Event where
+  toJSONObject =
+    KeyMap.fromList
+      . fromMaybe []
+      . schemaOut eventObjectSchema
+
+instance S.ToSchema Event where
+  declareNamedSchema = schemaToSwagger
+
+instance FromJSON Event where
+  parseJSON = schemaParseJSON
+
+instance ToJSON Event where
+  toJSON = schemaToJSON
diff --git a/libs/wire-api/wire-api.cabal b/libs/wire-api/wire-api.cabal
@@ -37,6 +37,7 @@ library
     Wire.API.Error.Gundeck
     Wire.API.Event.Conversation
     Wire.API.Event.FeatureConfig
+    Wire.API.Event.Federation
     Wire.API.Event.Team
     Wire.API.FederationStatus
     Wire.API.FederationUpdate
diff --git a/services/galley/galley.cabal b/services/galley/galley.cabal
@@ -99,6 +99,7 @@ library
     Galley.Effects.CodeStore
     Galley.Effects.ConversationStore
     Galley.Effects.CustomBackendStore
+    Galley.Effects.DefederationNotifications
     Galley.Effects.ExternalAccess
     Galley.Effects.FederatorAccess
     Galley.Effects.FireAndForget
diff --git a/services/galley/src/Galley/API/Internal.hs b/services/galley/src/Galley/API/Internal.hs
@@ -65,6 +65,7 @@ import Galley.Effects
 import Galley.Effects.BackendNotificationQueueAccess
 import Galley.Effects.ClientStore
 import Galley.Effects.ConversationStore
+import Galley.Effects.DefederationNotifications (DefederationNotifications, sendDefederationNotifications)
 import Galley.Effects.FederatorAccess
 import Galley.Effects.GundeckAccess
 import Galley.Effects.LegalHoldStore as LegalHoldStore
@@ -85,8 +86,8 @@ import Imports hiding (head)
 import qualified Network.AMQP as Q
 import Network.HTTP.Types
 import Network.Wai
-import Network.Wai.Predicate hiding (Error, err, setStatus)
-import qualified Network.Wai.Predicate as Predicate
+import Network.Wai.Predicate hiding (Error, err, result, setStatus)
+import qualified Network.Wai.Predicate as Predicate hiding (result)
 import Network.Wai.Routing hiding (App, route, toList)
 import Network.Wai.Utilities hiding (Error)
 import Network.Wai.Utilities.ZAuth
@@ -538,12 +539,18 @@ internalDeleteFederationDomainH ::
     Member TeamStore r,
     Member BrigAccess r,
     Member GundeckAccess r,
-    Member ExternalAccess r
+    Member ExternalAccess r,
+    Member DefederationNotifications r
   ) =>
   Domain ::: JSON ->
   Sem r Response
 internalDeleteFederationDomainH (domain ::: _) = do
+  -- We have to send the same event twice.
+  -- Once before and once after defederation work.
+  -- https://wearezeta.atlassian.net/wiki/spaces/ENGINEERIN/pages/809238539/Use+case+Stopping+to+federate+with+a+domain
+  sendDefederationNotifications domain
   deleteFederationDomain domain
+  sendDefederationNotifications domain
   pure (empty & setStatus status200)
 
 -- Remove remote members from local conversations
diff --git a/services/galley/src/Galley/App.hs b/services/galley/src/Galley/App.hs
@@ -272,6 +272,7 @@ evalGalley e =
     . interpretFederatorAccess
     . interpretExternalAccess
     . interpretGundeckAccess
+    . interpretDefederationNotifications
     . interpretSparAccess
     . interpretBrigAccess
   where
diff --git a/services/galley/src/Galley/Cassandra/Conversation/Members.hs b/services/galley/src/Galley/Cassandra/Conversation/Members.hs
@@ -18,6 +18,8 @@
 module Galley.Cassandra.Conversation.Members
   ( addMembers,
     members,
+    allMembers,
+    toMember,
     lookupRemoteMembers,
     removeMembersFromLocalConv,
     toMemberStatus,
@@ -122,6 +124,11 @@ members conv =
   fmap (mapMaybe toMember) . retry x1 $
     query Cql.selectMembers (params LocalQuorum (Identity conv))
 
+allMembers :: Client [LocalMember]
+allMembers =
+  fmap (mapMaybe toMember) . retry x1 $
+    query Cql.selectAllMembers (params LocalQuorum ())
+
 toMemberStatus ::
   ( -- otr muted
     Maybe MutedStatus,
@@ -386,6 +393,7 @@ interpretMemberStoreToCassandra = interpret $ \case
   CreateBotMember sr bid cid -> embedClient $ addBotMember sr bid cid
   GetLocalMember cid uid -> embedClient $ member cid uid
   GetLocalMembers cid -> embedClient $ members cid
+  GetAllLocalMembers -> embedClient allMembers
   GetRemoteMember cid uid -> embedClient $ lookupRemoteMember cid (tDomain uid) (tUnqualified uid)
   GetRemoteMembers rcid -> embedClient $ lookupRemoteMembers rcid
   CheckLocalMemberRemoteConv uid rcnv -> fmap (not . null) $ embedClient $ lookupLocalMemberRemoteConv uid rcnv
diff --git a/services/galley/src/Galley/Cassandra/Queries.hs b/services/galley/src/Galley/Cassandra/Queries.hs
@@ -339,6 +339,9 @@ selectMember = "select user, service, provider, status, otr_muted_status, otr_mu
 selectMembers :: PrepQuery R (Identity ConvId) (UserId, Maybe ServiceId, Maybe ProviderId, Maybe MemberStatus, Maybe MutedStatus, Maybe Text, Maybe Bool, Maybe Text, Maybe Bool, Maybe Text, Maybe RoleName)
 selectMembers = "select user, service, provider, status, otr_muted_status, otr_muted_ref, otr_archived, otr_archived_ref, hidden, hidden_ref, conversation_role from member where conv = ?"
 
+selectAllMembers :: PrepQuery R () (UserId, Maybe ServiceId, Maybe ProviderId, Maybe MemberStatus, Maybe MutedStatus, Maybe Text, Maybe Bool, Maybe Text, Maybe Bool, Maybe Text, Maybe RoleName)
+selectAllMembers = "select user, service, provider, status, otr_muted_status, otr_muted_ref, otr_archived, otr_archived_ref, hidden, hidden_ref, conversation_role from member"
+
 insertMember :: PrepQuery W (ConvId, UserId, Maybe ServiceId, Maybe ProviderId, RoleName) ()
 insertMember = "insert into member (conv, user, service, provider, status, conversation_role) values (?, ?, ?, ?, 0, ?)"
 
diff --git a/services/galley/src/Galley/Effects.hs b/services/galley/src/Galley/Effects.hs
@@ -69,6 +69,7 @@ import Galley.Effects.ClientStore
 import Galley.Effects.CodeStore
 import Galley.Effects.ConversationStore
 import Galley.Effects.CustomBackendStore
+import Galley.Effects.DefederationNotifications
 import Galley.Effects.ExternalAccess
 import Galley.Effects.FederatorAccess
 import Galley.Effects.FireAndForget
@@ -99,6 +100,7 @@ import Wire.Sem.Paging.Cassandra
 type GalleyEffects1 =
   '[ BrigAccess,
      SparAccess,
+     DefederationNotifications,
      GundeckAccess,
      ExternalAccess,
      FederatorAccess,
diff --git a/services/galley/src/Galley/Effects/DefederationNotifications.hs b/services/galley/src/Galley/Effects/DefederationNotifications.hs
@@ -0,0 +1,14 @@
+{-# LANGUAGE TemplateHaskell #-}
+
+module Galley.Effects.DefederationNotifications 
+  ( DefederationNotifications (..),
+    sendDefederationNotifications
+  ) where
+
+import Polysemy
+import Data.Domain (Domain)
+
+data DefederationNotifications m a where
+  SendDefederationNotifications :: Domain -> DefederationNotifications m ()
+
+makeSem ''DefederationNotifications
diff --git a/services/galley/src/Galley/Effects/ExternalAccess.hs b/services/galley/src/Galley/Effects/ExternalAccess.hs
@@ -26,6 +26,7 @@ module Galley.Effects.ExternalAccess
   )
 where
 
+import Data.Aeson
 import Data.Id
 import Galley.Data.Services
 import Imports
@@ -34,7 +35,7 @@ import Wire.API.Event.Conversation
 
 data ExternalAccess m a where
   Deliver :: Foldable f => f (BotMember, Event) -> ExternalAccess m [BotMember]
-  DeliverAsync :: Foldable f => f (BotMember, Event) -> ExternalAccess m ()
+  DeliverAsync :: (ToJSON e, Foldable f) => f (BotMember, e) -> ExternalAccess m ()
   DeliverAndDeleteAsync :: Foldable f => ConvId -> f (BotMember, Event) -> ExternalAccess m ()
 
 makeSem ''ExternalAccess
diff --git a/services/galley/src/Galley/Effects/MemberStore.hs b/services/galley/src/Galley/Effects/MemberStore.hs
@@ -30,6 +30,7 @@ module Galley.Effects.MemberStore
     -- * Read members
     getLocalMember,
     getLocalMembers,
+    getAllLocalMembers,
     getRemoteMember,
     getRemoteMembers,
     checkLocalMemberRemoteConv,
@@ -70,6 +71,7 @@ data MemberStore m a where
   CreateBotMember :: ServiceRef -> BotId -> ConvId -> MemberStore m BotMember
   GetLocalMember :: ConvId -> UserId -> MemberStore m (Maybe LocalMember)
   GetLocalMembers :: ConvId -> MemberStore m [LocalMember]
+  GetAllLocalMembers :: MemberStore m [LocalMember]
   GetRemoteMember :: ConvId -> Remote UserId -> MemberStore m (Maybe RemoteMember)
   GetRemoteMembers :: ConvId -> MemberStore m [RemoteMember]
   CheckLocalMemberRemoteConv :: UserId -> Remote ConvId -> MemberStore m Bool
diff --git a/services/galley/src/Galley/External.hs b/services/galley/src/Galley/External.hs
@@ -21,6 +21,7 @@ import Bilge.Request
 import Bilge.Retry (httpHandlers)
 import Control.Lens
 import Control.Retry
+import Data.Aeson (ToJSON)
 import Data.ByteString.Conversion.To
 import Data.Id
 import Data.Misc
@@ -60,7 +61,7 @@ interpretExternalAccess = interpret $ \case
 -- | Like deliver, but ignore orphaned bots and return immediately.
 --
 -- FUTUREWORK: Check if this can be removed.
-deliverAsync :: [(BotMember, Event)] -> App ()
+deliverAsync :: ToJSON e => [(BotMember, e)] -> App ()
 deliverAsync = void . forkIO . void . deliver
 
 -- | Like deliver, but remove orphaned bots and return immediately.
@@ -69,10 +70,10 @@ deliverAndDeleteAsync cnv pushes = void . forkIO $ do
   gone <- deliver pushes
   mapM_ (deleteBot cnv . botMemId) gone
 
-deliver :: [(BotMember, Event)] -> App [BotMember]
+deliver :: forall e. ToJSON e => [(BotMember, e)] -> App [BotMember]
 deliver pp = mapM (async . exec) pp >>= foldM eval [] . zip (map fst pp)
   where
-    exec :: (BotMember, Event) -> App Bool
+    exec :: (BotMember, e) -> App Bool
     exec (b, e) =
       lookupService (botMemService b) >>= \case
         Nothing -> pure False
@@ -118,7 +119,7 @@ deliver pp = mapM (async . exec) pp >>= foldM eval [] . zip (map fst pp)
 
 -- Internal -------------------------------------------------------------------
 
-deliver1 :: Service -> BotMember -> Event -> App ()
+deliver1 :: ToJSON e => Service -> BotMember -> e -> App ()
 deliver1 s bm e
   | s ^. serviceEnabled = do
       let t = toByteString' (s ^. serviceToken)
diff --git a/services/galley/src/Galley/Intra/Effects.hs b/services/galley/src/Galley/Intra/Effects.hs
@@ -20,16 +20,27 @@ module Galley.Intra.Effects
     interpretSparAccess,
     interpretBotAccess,
     interpretGundeckAccess,
+    interpretDefederationNotifications,
   )
 where
 
+import Cassandra (ClientState, Consistency (LocalQuorum), Page (hasMore, nextPage, result), paginate, paramsP)
+import Control.Lens ((.~))
+import Data.Range (Range (fromRange))
 import Galley.API.Error
+import Galley.API.Util (localBotsAndUsers)
+import Galley.Cassandra.Conversation.Members (toMember)
+import Galley.Cassandra.Queries (selectAllMembers)
+import Galley.Cassandra.Store (embedClient)
 import Galley.Effects.BotAccess (BotAccess (..))
 import Galley.Effects.BrigAccess (BrigAccess (..))
-import Galley.Effects.GundeckAccess (GundeckAccess (..))
+import Galley.Effects.DefederationNotifications (DefederationNotifications (..))
+import Galley.Effects.ExternalAccess (ExternalAccess, deliverAsync)
+import Galley.Effects.GundeckAccess (GundeckAccess (..), push1)
 import Galley.Effects.SparAccess (SparAccess (..))
 import Galley.Env
 import Galley.Intra.Client
+import qualified Galley.Intra.Push as Intra
 import qualified Galley.Intra.Push.Internal as G
 import Galley.Intra.Spar
 import Galley.Intra.Team
@@ -41,6 +52,8 @@ import Polysemy.Error
 import Polysemy.Input
 import qualified Polysemy.TinyLog as P
 import qualified UnliftIO
+import qualified Wire.API.Event.Federation as Federation
+import Wire.API.Team.Member (ListType (ListComplete))
 
 interpretBrigAccess ::
   ( Member (Embed IO) r,
@@ -123,3 +136,36 @@ interpretGundeckAccess ::
 interpretGundeckAccess = interpret $ \case
   Push ps -> embedApp $ G.push ps
   PushSlowly ps -> embedApp $ G.pushSlowly ps
+
+interpretDefederationNotifications ::
+  ( Member (Embed IO) r,
+    Member (Input Env) r,
+    Member (Input ClientState) r,
+    Member GundeckAccess r,
+    Member ExternalAccess r
+  ) =>
+  Sem (DefederationNotifications ': r) a ->
+  Sem r a
+interpretDefederationNotifications = interpret $ \case
+  SendDefederationNotifications domain -> do
+    maxPage <- inputs $ fromRange . currentFanoutLimit . _options -- This is based on the limits in removeIfLargeFanout
+    page <- embedClient $ paginate selectAllMembers (paramsP LocalQuorum () maxPage)
+    void $ sendNotificationPage page
+    where
+      pushEvents results = do
+        let (bots, mems) = localBotsAndUsers results
+            recipients = Intra.recipient <$> mems
+            event = Intra.FederationEvent $ Federation.Event Federation.FederationDelete domain
+        for_ (Intra.newPush ListComplete Nothing event recipients) $ \p -> do
+          -- Futurework: Transient or not?
+          -- RouteAny is used as it will wake up mobile clients
+          -- and notify them of the changes to federation state.
+          push1 $ p & Intra.pushRoute .~ Intra.RouteAny
+        deliverAsync (bots `zip` repeat (G.pushEventJson event))
+      sendNotificationPage page = do
+        let res = result page
+            mems = mapMaybe toMember res
+        pushEvents mems
+        when (hasMore page) $ do
+          page' <- embedClient $ nextPage page
+          sendNotificationPage page'
diff --git a/services/galley/src/Galley/Intra/Push/Internal.hs b/services/galley/src/Galley/Intra/Push/Internal.hs
@@ -40,18 +40,21 @@ import Imports hiding (forkIO)
 import UnliftIO.Async (mapConcurrently_)
 import Wire.API.Event.Conversation (Event (evtFrom))
 import qualified Wire.API.Event.FeatureConfig as FeatureConfig
+import qualified Wire.API.Event.Federation as Federation
 import qualified Wire.API.Event.Team as Teams
 import Wire.API.Team.Member
 
 data PushEvent
   = ConvEvent Event
   | TeamEvent Teams.Event
   | FeatureConfigEvent FeatureConfig.Event
+  | FederationEvent Federation.Event
 
 pushEventJson :: PushEvent -> Object
 pushEventJson (ConvEvent e) = toJSONObject e
 pushEventJson (TeamEvent e) = toJSONObject e
 pushEventJson (FeatureConfigEvent e) = toJSONObject e
+pushEventJson (FederationEvent e) = toJSONObject e
 
 data RecipientBy user = Recipient
   { _recipientUserId :: user,
diff --git a/services/galley/test/integration/API.hs b/services/galley/test/integration/API.hs
diff --git a/services/galley/test/integration/API/Util.hs b/services/galley/test/integration/API/Util.hs

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+Added a new notification event type, "federation.delete".`
	`2`	`+This event contains a single domain for a remote server that the local server is de-federating from.`
	`3`	`+This notification is sent twice during de-federation. Once before and once after cleaning up and removing references to the remote server from the local database.`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+De-federating from a remote server sends a pair of notifications to clients, announcing which server will no longer be federated with.`