Update to 1.9.0 breaks authentication via LDAP (ma1sd / rest client)

[jacotec]

Description

After updating synapse to 1.9.0 no user can login. Users are stored in a LDAP database, authentication via REST-API / ma1sd.

Steps to reproduce

Open Riot Web
Log in with any existing user
Error message: Fehler: Problem bei der Kommunikation mit dem angegebenen Home-Server. (M_UNKNOWN)

Version information

Synapse 1.9.0 installed via official repo.
Ubuntu 18.04 LTS

Log file of the issue:

2020-01-23 14:09:46,680 - synapse.access.https.8448 - 233 - INFO - POST-44 - 87.79.201.63 - 8448 - Received request: POST /_matrix/client/r0/login
2020-01-23 14:09:46,681 - synapse.rest.client.v1.login - 176 - INFO - POST-44 - Got login request with identifier: {'type': 'm.id.user', 'user': 'admin'}, medium: None, address: None, user: None
2020-01-23 14:09:46,681 - rest_auth_provider - 46 - INFO - POST-44 - Got password check for @admin:jaychat.de
2020-01-23 14:09:46,695 - rest_auth_provider - 62 - INFO - POST-44 - User @admin:jaychat.de authenticated
2020-01-23 14:09:46,699 - rest_auth_provider - 76 - INFO - POST-44 - User @admin:jaychat.de already exists, registration skipped
2020-01-23 14:09:46,699 - rest_auth_provider - 79 - INFO - POST-44 - Handling profile data
2020-01-23 14:09:46,700 - synapse.http.server - 110 - ERROR - POST-44 - Failed handle request via 'LoginRestServlet': <XForwardedForRequest at 0x7f18edeedda0 method='POST' uri='/_matrix/client/r0/login' clientproto='HTTP/1.0' site=8448>
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: {}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: {}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: ('@admin:jaychat.de', '$2b$12$YEY5SVN7Yy43yFq3f6saLuAR/uaNK5m4Res6hgnLLc/70KODPt7ZW')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: @admin:jaychat.de

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/http/server.py", line 78, in wrapped_request_handler
    await h(self, request)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/http/server.py", line 331, in _async_render
    callback_return = await callback_return
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/client/v1/login.py", line 150, in on_POST
    result = await self._do_other_login(login_submission)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/client/v1/login.py", line 281, in _do_other_login
    identifier["user"], login_submission
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/handlers/auth.py", line 622, in validate_login
    is_valid = yield provider.check_password(qualified_user_id, password)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/rest_auth_provider.py", line 82, in check_password
    store = yield self.account_handler.hs.get_profile_handler().store
AttributeError: 'ModuleApi' object has no attribute 'hs'

[MurzNN]

Got the same problem after upgrading to 1.9.0:

2020-01-23 17:31:23,842 - synapse.access.http.8008 - 233 - INFO - POST-222 - 95.79.93.22 - 8008 - Received request: POST /_matrix/client/r0/login
2020-01-23 17:31:23,843 - synapse.rest.client.v1.login - 176 - INFO - POST-222 - Got login request with identifier: {'type': 'm.id.user', 'user': 'korepov'}, medium: None, address: None, user: None
2020-01-23 17:31:23,844 - rest_auth_provider - 46 - INFO - POST-222 - Got password check for @korepov:digiterra.pro
2020-01-23 17:31:23,959 - rest_auth_provider - 62 - INFO - POST-222 - User @korepov:digiterra.pro authenticated
2020-01-23 17:31:23,961 - rest_auth_provider - 76 - INFO - POST-222 - User @korepov:digiterra.pro already exists, registration skipped
2020-01-23 17:31:23,961 - rest_auth_provider - 79 - INFO - POST-222 - Handling profile data
2020-01-23 17:31:23,962 - synapse.http.server - 110 - ERROR - POST-222 - Failed handle request via 'LoginRestServlet': <XForwardedForRequest at 0x7f23a08e2e48 method='POST' uri='/_matrix/client/r0/login' clientproto='HTTP/1.0' site=8008>
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: {}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: {}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: ('@korepov:digiterra.pro', None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: @korepov:digiterra.pro

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/http/server.py", line 78, in wrapped_request_handler
    await h(self, request)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/http/server.py", line 331, in _async_render
    callback_return = await callback_return
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/client/v1/login.py", line 150, in on_POST
    result = await self._do_other_login(login_submission)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/client/v1/login.py", line 281, in _do_other_login
    identifier["user"], login_submission
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/handlers/auth.py", line 622, in validate_login
    is_valid = yield provider.check_password(qualified_user_id, password)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/rest_auth_provider.py", line 82, in check_password
    store = yield self.account_handler.hs.get_profile_handler().store
AttributeError: 'ModuleApi' object has no attribute 'hs'

[jacotec]

Unfortunately downgrade to 1.8.0 does not work, synapse does not start ("ValueError: Cannot use this database as it is too new for the server to understand").

A hotfix is highly appreciated as the only other way is to restore my backup from last night, losing data of today.

[MurzNN]

Seems the source of problem is in https://github.com/ma1uta/matrix-synapse-rest-password-provider/blob/master/rest_auth_provider.py near the:

            store = yield self.account_handler.hs.get_profile_handler().store
            if "display_name" in profile and ((registration and self.config.setNameOnRegister) or (self.config.setNameOnLogin)):
                display_name = profile["display_name"]
                logger.info("Setting display name to '%s' based on profile data", display_name)
                yield store.set_profile_displayname(localpart, display_name)

because self.account_handler have no 'hs' attribute. Is it renamed or moved in Synapse?

[erikjohnston]

Hi all, firstly let me say that this is unfortunate, especially as v1.9.0 doesn't let you downgrade. However, this has been broken because we've changed ModuleApi (the account_handler) to not expose the hs object to make it clear that plugins should not use it to poke arbitrary bits of the code base. This is as we want to try and keep the public interface we expose to plugins stable without having to worry about breaking them if we change the internals of synapse.

As such, ideally this would be fixed in the ldap plugin you used to only use the public API, this may in turn require us to expand the ModuleApi interface, which we're happy to do.

As an immediate and temporary fix you can patch synapse to re-expose the hs object for now, by:

diff --git a/synapse/module_api/__init__.py b/synapse/module_api/__init__.py
index d680ee95e..80298ecee 100644
--- a/synapse/module_api/__init__.py
+++ b/synapse/module_api/__init__.py
@@ -37,6 +37,7 @@ class ModuleApi(object):
 
     def __init__(self, hs, auth_handler):
         self._hs = hs
+        self.hs = hs
 
         self._store = hs.get_datastore()
         self._auth = hs.get_auth()

Or alternatively you could patch the plugin to use account_handler._hs, but again that is a temporary band aid until it can be changed to use public APIs.

If neither of those two are possible we are willing to consider a hot fix this time round, but we would aim to remove support again in the next release (likely in roughly two weeks). We're really not keen on continuing to support this precisely because we want to ensure that these sort of don't breakages happen again in the future.

[MurzNN]

@erikjohnston, great thanks for quick fix - it works well for me!

[MurzNN]

Can you please also provide the right way how to fix this problem in https://github.com/ma1uta/matrix-synapse-rest-password-provider module? If Synapse will not expose hs object, what alternative can be used for get_profile_handler()?

[jacotec]

@erikjohnston Thank you also from my side for the lightning fast reaction and fix! Highly appreciated!

[erikjohnston]

You're welcome 👍

Can you please also provide the right way how to fix this problem in https://github.com/ma1uta/matrix-synapse-rest-password-provider module? If Synapse will not expose hs object, what alternative can be used for get_profile_handler()?

A quick browse of the code suggests its doing three things:

1. Syncing the profiles of existing users
2. Syncing the "threepids" of existing users (e.g. emails, msisdn etc)
3. Getting the current time.

The first two seem like reasonable additions to the ModuleApi, e.g. set_profile_displayname(localpart, display_name) could be dropped straight into the ModuleApi interface. I'm happy to review PRs for doing so.

For the third one either we can add it to the interface as well, or you can simply replace it with the following, as its not doing anything special and just getting the timestamp in milliseconds:

import time;

def time_msec():
    """Get the current timestamp in milliseconds
    """
    int(time.time() * 1000)

[menturion]

@erikjohnston
Please keep Synapse compatible with existing password providers. This is really essential for us.

[michaelkaye]

It looks like a temporary fix is in place in the latest rest_auth_provider: https://github.com/ma1uta/matrix-synapse-rest-password-provider/blob/master/rest_auth_provider.py#L84