/_matrix/federation/v1/state_ids/ is heavy

[richvdh]

in a large room with long auth chains, each request to /_matrix/federation/v1/state_ids/ takes a lot of database time. The requesting servers tend to time out the requests, and then retry, compounding the problem.

[richvdh]

see #6597: it's not obvious that this data is even useful.

[anoadragon453]

It seems now that we do want to keep this behaviour, but optimize the performance of it as per #6597 (comment)

[anoadragon453]

@richvdh Do you think it worth keeping this issue open if we have #6597?

[richvdh]

sorry, yes. This issue is about the database/CPU time taken on the server side; I've actually got a couple of patches which I need to PR to improve it.

#6597 is about handling the response on the client side.

[richvdh]

Hopefully this is improved by #7931 and #7930.

[richvdh]

this is still a problem; matrix.org's federation readers are still pinning the CPU and hammering the database regularly.

[richvdh]

This is exacerbated by rooms which have complex graphs. A single back-pagination request in !YynUnYHpqlHuoTAjsp:matrix.org on matrix.org introduced 51 new backwards-extremities, each of which required a state_ids request to kde.org, each of which takes 3 minutes of database time.