Make central IS opt-in (#80)

- Switch to HttpClient for remote fetcher
- Don't fail for remote binding on matrix.org

application.example.yaml

@@ -47,18 +47,33 @@ key.path: ''
 storage.provider.sqlite.database: '/path/to/mxisd.db'
 
 
+####################
+# Fallback servers #
+####################
+#
+# Root/Central servers to be used as final fallback when performing lookups.
+# By default, for privacy reasons, matrix.org servers are not enabled anymore.
+# See the following issue: https://github.com/kamax-io/mxisd/issues/76
+#
+# If you would like to use them and trade away your privacy for convenience, uncomment the following option:
+#
+#forward.servers: ['matrix-org']
+
+
 ################
 # LDAP Backend #
 ################
 # If you would like to integrate with your AD/Samba/LDAP server,
 # see https://github.com/kamax-io/mxisd/blob/master/docs/backends/ldap.md
 
+
 ###############
 # SQL Backend #
 ###############
 # If you would like to integrate with a MySQL/MariaDB/PostgreQL/SQLite DB,
 # see https://github.com/kamax-io/mxisd/blob/master/docs/backends/sql.md
 
+
 ################
 # REST Backend #
 ################

build.gradle

@@ -102,9 +102,6 @@ dependencies {
     compile 'com.sun.mail:javax.mail:1.5.6'
     compile 'javax.mail:javax.mail-api:1.5.6'
 
-    // Google Client APIs
-    compile 'com.google.api-client:google-api-client:1.23.0'
-
     // Google Firebase Authentication backend
     compile 'com.google.firebase:firebase-admin:5.3.0'
 

docs/architecture.md

@@ -18,12 +18,9 @@ TCP 443
           |   +-------------------+
  TCP 8090 +-> | mxisd             |
               |                   |
-              | - Profile's 3PIDs >----+
+              | - Profile's 3PIDs |
-              | - 3PID Invites    |    |             +--------------------------+
+              | - 3PID Invites    |
-              +-|-----------------+    +>----------> | Central Identity service |
+              +-|-----------------+
-                |                      |   TCP 443   | Matrix.org / Vector.im   |
-                |                      |             +--------------------------+
-                +>-------------------->+
                 |
              TCP 443
                 |  +------------------------+

docs/faq.md

@@ -19,8 +19,9 @@ started and answer questions you might have.
 ### Do I need to use mxisd if I run a Homeserver?
 No, but it is strongly recommended, even if you don't use any Identity store or integration.
 
-In its default configuration, mxisd will talk to the central Matrix Identity servers and use other federated public
+In its default configuration, mxisd uses other federated public servers when performing queries.  
-servers when performing queries, giving you access to at least the same information as if you were not running it.
+It can also [be configured](features/identity.md#lookups) to use the central matrix.org servers, giving you access to at
+least the same information as if you were not running it.
 
 It will also give your users a choice to make their 3PIDs available publicly, ensuring they are made aware of the
 privacy consequences, which is not the case with the central Matrix.org servers.
@@ -70,18 +71,15 @@ So really, you should go with mxisd.
 ### Will I loose access to the central Matrix.org/Vector.im Identity data if I use mxisd?
 No.
 
-In its default configuration, mxisd act as a proxy to Matrix.org/Vector.im. You will have access to the same data and
+In its default configuration, mxisd does not talk to the central Identity server matrix.org to avoid leaking your private
-behaviour than if you were using them directly. There is no downside in using mxisd with the default configuration.
+data and those of people you might know.
 
-mxisd can also be configured not to talk to the central Identity servers if you wish.
+mxisd [can be configured](features/identity.md#lookups) to talk to the central Identity servers if you wish.
 
 ### So mxisd is just a big hack! I don't want to use non-official features!
-mxisd primary concern is to always be compatible with the Matrix ecosystem and the Identity service API.  
+mxisd primary concerns are your privacy and to always be compatible with the Matrix ecosystem and the Identity service API.  
 Whenever the API will be updated and/or enhanced, mxisd will follow, remaining 100% compatible with the ecosystem.
 
-Therefore, using mxisd is a safe choice. It will be like using the central Matrix.org Identity servers, yet not closing
-the door to a growing list of enhancements and integrations.
-
 ### Should I use mxisd if I don't host my own Homeserver?
 No.
 

docs/features/federation.md

@@ -5,8 +5,8 @@ Federated Identity server using the DNS domain part of the 3PID.
 Emails are the best candidate for this kind of resolution which are DNS domain based already.  
 On the other hand, Phone numbers cannot be resolved this way.
 
-For 3PIDs which are not compatible with the DNS system, mxisd will talk to the central Identity server of matrix.org by
+For 3PIDs which are not compatible with the DNS system, mxisd can be configured to talk to fallback Identity servers like
-default.
+the central matrix.org one. See the [Identity feature](identity.md#lookups) for instructions on how to enable it.
 
 Outbound federation is enabled by default while inbound federation is opt-in and require a specific DNS record.
 
@@ -17,16 +17,14 @@ Outbound federation is enabled by default while inbound federation is opt-in and
               |                   |   |      +------> +----------+
               |                   |   |      |
               | Invites / Lookups |   |      |
- Federated    | +--------+        |   |      |        +-------------------+
+ Federated    | +--------+        |   |      |
- Identity  ---->| Remote |>-----------+      +------> | Remote Federated  |
+ Identity  ---->| Remote |>-----------+      |
- Server       | +--------+        |          |        | mxisd servers     |
+ Server       | +--------+        |          |
-              |                   |          |        +-------------------+
+              |                   |          |
-              | +--------+        |          |
+              | +--------+        |          |        +-------------------+
- Homeserver --->| Local  |>------------------+
+ Homeserver --->| Local  |>------------------+------> | Remote Federated  |
- and clients  | +--------+        |          |        +--------------------------+ 
+ and clients  | +--------+        |                   | mxisd servers     |
-              +-------------------+          +------> | Central Identity service |
+              +-------------------+                   +-------------------+
-                                                      | Matrix.org / Vector.im   |
-                                                      +--------------------------+
 ```
 
 ## Inbound

docs/features/identity.md

@@ -3,6 +3,16 @@
 
 Implementation of the [Unofficial Matrix Identity Service API](https://kamax.io/matrix/api/identity_service/unstable.html).
 
+## Lookups
+If you would like to use the central matrix.org Identity server to ensure maximum discovery at the cost of potentially
+leaking all your contacts information, add the following to your configuration:
+```yaml
+forward.servers:
+  - 'matrix-org'
+```
+**NOTE:** You should carefully consider enabling this option, which is discouraged.  
+For more info, see the [relevant issue](https://github.com/kamax-io/mxisd/issues/76).
+
 ## Room Invitations
 Resolution can be customized using the following configuration:
 

docs/stores/sql.md

@@ -44,7 +44,7 @@ Example: `/path/to/sqlite/file.db`
 
 #### Others
 ```yaml
-sql.connection: //<HOST[:PORT]/DB?username=USER&password=PASS
+sql.connection: //<HOST[:PORT]/DB?user=USER&password=PASS
 ```
 Set the connection info for the database by replacing the following values:
 - `HOST`: Hostname of the SQL server

docs/stores/synapse.md

@@ -35,7 +35,7 @@ Example: `/path/to/synapse/sqliteFile.db`
 
 ### PostgreSQL
 ```yaml
-synapseSql.connection: //<HOST[:PORT]/DB?username=USER&password=PASS
+synapseSql.connection: //<HOST[:PORT]/DB?user=USER&password=PASS
 ```
 Set the connection info for the database by replacing the following values:
 - `HOST`: Hostname of the SQL server

docs/threepids/session/session.md

@@ -117,6 +117,7 @@ The following example of configuration (incomplete extract) shows which items ar
 **IMPORTANT:** Most configuration items shown have default values and should not be included in your own configuration
 file unless you want to specifically overwrite them.
 ```yaml
+# CONFIGURATION EXAMPLE
 # DO NOT COPY/PASTE THIS IN YOUR CONFIGURATION
 session.policy.validation.enabled: true
 session.policy.validation.forLocal:
@@ -132,6 +133,7 @@ session.policy.validation.forRemote:
     enabled: true
     server: 'configExample'  # Not to be included in config! Already present in default config!
 # DO NOT COPY/PASTE THIS IN YOUR CONFIGURATION
+# CONFIGURATION EXAMPLE
 ```
 
 `session.policy.validation` is the core configuration to control what users configured to use your Identity server
@@ -143,8 +145,8 @@ It is also divided into two sections: `forLocal` and `forRemote` which refers to
 Each scope is divided into three parts:
 - global on/off switch for 3PID sessions using `.enabled`
 - `toLocal` allowing or not local 3PID session validations
-- `toRemote` allowing or not remote 3PID session validations and to which server such sessions should be sent. 
+- `toRemote` allowing or not remote 3PID session validations and to which server such sessions should be sent.   
-`.server` takes a Matrix Identity server list label. Only the first server in the list is currently used.
+  `.server` takes a Matrix Identity server list label. Only the first server in the list is currently used.
 
 If both `toLocal` and `toRemote` are enabled, the user will be offered to initiate a remote session once their 3PID
 locally validated.

src/main/java/io/kamax/mxisd/auth/AuthManager.java

@@ -43,7 +43,8 @@ public class AuthManager {
 
     private Logger log = LoggerFactory.getLogger(AuthManager.class);
 
-    private List<AuthenticatorProvider> providers;
+    @Autowired
+    private List<AuthenticatorProvider> providers = new ArrayList<>();
 
     @Autowired
     private MatrixConfig mxCfg;
@@ -51,10 +52,6 @@ public class AuthManager {
     @Autowired
     private InvitationManager invMgr;
 
-    public AuthManager(List<AuthenticatorProvider> providers) {
-        this.providers = new ArrayList<>(providers);
-    }
-
     public UserAuthResult authenticate(String id, String password) {
         _MatrixID mxid = MatrixID.asAcceptable(id);
         for (AuthenticatorProvider provider : providers) {
@@ -62,7 +59,6 @@ public class AuthManager {
                 continue;
             }
 
-            log.info("Attempting auth with " + provider.getClass().getSimpleName());
             BackendAuthResult result = provider.authenticate(mxid, password);
             if (result.isSuccess()) {
 

src/main/java/io/kamax/mxisd/auth/provider/BackendAuthResult.java

@@ -49,9 +49,8 @@ public class BackendAuthResult {
         return r;
     }
 
-    public BackendAuthResult fail() {
+    public void fail() {
         success = false;
-        return this;
     }
 
     public static BackendAuthResult success(String id, UserIdType type, String displayName) {
@@ -64,11 +63,10 @@ public class BackendAuthResult {
         return r;
     }
 
-    public BackendAuthResult succeed(String id, String type, String displayName) {
+    public void succeed(String id, String type, String displayName) {
         this.success = true;
         this.id = new UserID(type, id);
         this.profile.displayName = displayName;
-        return this;
     }
 
     private Boolean success;

src/main/java/io/kamax/mxisd/backend/google/GoogleProviderBackend.java

@@ -1,134 +0,0 @@
-/*
- * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2018 Kamax Sàrl
- *
- * https://www.kamax.io/
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU Affero General Public License as
- * published by the Free Software Foundation, either version 3 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU Affero General Public License for more details.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- */
-
-package io.kamax.mxisd.backend.google;
-
-import com.google.api.client.googleapis.auth.oauth2.GoogleIdToken;
-import com.google.api.client.googleapis.auth.oauth2.GoogleIdTokenVerifier;
-import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
-import com.google.api.client.http.HttpTransport;
-import com.google.api.client.json.JsonFactory;
-import com.google.api.client.json.jackson2.JacksonFactory;
-import io.kamax.matrix._MatrixID;
-import io.kamax.matrix.ThreePid;
-import io.kamax.mxisd.UserIdType;
-import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
-import io.kamax.mxisd.auth.provider.BackendAuthResult;
-import io.kamax.mxisd.config.GoogleConfig;
-import io.kamax.mxisd.lookup.strategy.LookupStrategy;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Component;
-
-import java.io.IOException;
-import java.security.GeneralSecurityException;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-import java.util.Optional;
-
-@Component
-public class GoogleProviderBackend implements AuthenticatorProvider {
-
-    private final Logger log = LoggerFactory.getLogger(GoogleProviderBackend.class);
-    private final GoogleConfig cfg;
-    private final LookupStrategy lookup;
-
-    private GoogleIdTokenVerifier verifier;
-
-    public Optional<GoogleIdToken> extractToken(String data) throws GeneralSecurityException, IOException {
-        return Optional.ofNullable(verifier.verify(data));
-    }
-
-    public List<ThreePid> extractThreepids(GoogleIdToken token) {
-        List<ThreePid> tpids = new ArrayList<>();
-        tpids.add(new ThreePid("io.kamax.google.id", token.getPayload().getSubject()));
-        if (token.getPayload().getEmailVerified()) {
-            tpids.add(new ThreePid("email", token.getPayload().getEmail()));
-        }
-        return tpids;
-    }
-
-    @Autowired
-    public GoogleProviderBackend(GoogleConfig cfg, LookupStrategy lookup) {
-        this.cfg = cfg;
-        this.lookup = lookup;
-
-        if (isEnabled()) {
-            try {
-                HttpTransport transport = GoogleNetHttpTransport.newTrustedTransport();
-                JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
-
-                verifier = new GoogleIdTokenVerifier.Builder(transport, jsonFactory)
-                        .setAudience(Collections.singletonList(cfg.getClient().getId()))
-                        .build();
-            } catch (IOException | GeneralSecurityException e) {
-                throw new RuntimeException(e);
-            }
-        }
-    }
-
-    @Override
-    public boolean isEnabled() {
-        return cfg.isEnabled();
-    }
-
-    @Override
-    public BackendAuthResult authenticate(_MatrixID mxid, String password) {
-        BackendAuthResult result = new BackendAuthResult();
-
-        try {
-            return extractToken(password).map(idToken -> {
-                GoogleIdToken.Payload payload = idToken.getPayload();
-                if (!payload.getEmailVerified()) { // We only want users who validated their email
-                    return BackendAuthResult.failure();
-                }
-
-                // Get user identifier
-                String userId = payload.getSubject();
-
-                // We validate that the user who authenticated has his Google account associated already
-                return lookup.find("io.kamax.google.id", userId, false).map(r -> {
-
-                    if (!r.getMxid().equals(mxid)) {
-                        return result.fail();
-                    }
-
-                    // Get profile information from payload
-                    extractThreepids(idToken).forEach(result::withThreePid);
-                    String name = (String) payload.get("name");
-
-                    payload.getUnknownKeys().keySet().forEach(key -> {
-                        log.info("Unknown key in Google profile: {} -> ", key, payload.get(key));
-                    });
-
-                    return result.succeed(mxid.getId(), UserIdType.MatrixID.getId(), name);
-                }).orElse(BackendAuthResult.failure());
-            }).orElse(BackendAuthResult.failure());
-        } catch (GeneralSecurityException e) {
-            throw new RuntimeException(e);
-        } catch (IOException e) {
-            log.error("Unable to authenticate via Google due to network error", e);
-            return result.fail();
-        }
-    }
-
-}

src/main/java/io/kamax/mxisd/backend/sql/SqlThreePidProvider.java

@@ -59,10 +59,6 @@ public abstract class SqlThreePidProvider implements IThreePidProvider, ProfileP
         this.mxCfg = mxCfg;
     }
 
-    protected Connection getConnection() throws SQLException {
-        return pool.get();
-    }
-
     @Override
     public boolean isEnabled() {
         return cfg.isEnabled();
@@ -123,7 +119,7 @@ public abstract class SqlThreePidProvider implements IThreePidProvider, ProfileP
         List<_ThreePid> threepids = new ArrayList<>();
 
         String stmtSql = cfg.getProfile().getThreepid().getQuery();
-        try (Connection conn = getConnection()) {
+        try (Connection conn = pool.get()) {
             PreparedStatement stmt = conn.prepareStatement(stmtSql);
             stmt.setString(1, mxid.getId());
 

src/main/java/io/kamax/mxisd/backend/sql/SynapseSqlThreePidProvider.java

@@ -20,50 +20,17 @@
 
 package io.kamax.mxisd.backend.sql;
 
-import io.kamax.matrix.ThreePid;
-import io.kamax.matrix._MatrixID;
 import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.config.sql.synapse.SynapseSqlProviderConfig;
-import io.kamax.mxisd.profile.ProfileWriter;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
-import java.sql.Connection;
-import java.sql.PreparedStatement;
-import java.sql.SQLException;
-import java.time.Instant;
-
 @Component
-public class SynapseSqlThreePidProvider extends SqlThreePidProvider implements ProfileWriter {
+public class SynapseSqlThreePidProvider extends SqlThreePidProvider {
-
-    private final Logger log = LoggerFactory.getLogger(SynapseSqlThreePidProvider.class);
 
     @Autowired
     public SynapseSqlThreePidProvider(SynapseSqlProviderConfig cfg, MatrixConfig mxCfg) {
         super(cfg, mxCfg);
     }
 
-    @Override
-    public boolean addThreepid(_MatrixID mxid, ThreePid tpid) {
-        try (Connection conn = getConnection()) {
-            PreparedStatement stmt = conn.prepareStatement("INSERT INTO user_threepids (user_id, medium, address, validated_at, added_at) values (?,?,?,?,?)");
-            stmt.setString(1, mxid.getId());
-            stmt.setString(2, tpid.getMedium());
-            stmt.setString(3, tpid.getAddress());
-            stmt.setLong(4, Instant.now().toEpochMilli());
-            stmt.setLong(5, Instant.now().toEpochMilli());
-
-            int rows = stmt.executeUpdate();
-            if (rows != 1) {
-                log.error("Unable to update 3PID info. Modified row(s): {}", rows);
-            }
-
-            return true;
-        } catch (SQLException e) {
-            throw new RuntimeException(e);
-        }
-    }
-
 }

src/main/java/io/kamax/mxisd/config/GoogleConfig.java

@@ -1,107 +0,0 @@
-/*
- * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2018 Kamax Sàrl
- *
- * https://www.kamax.io/
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU Affero General Public License as
- * published by the Free Software Foundation, either version 3 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU Affero General Public License for more details.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- */
-
-package io.kamax.mxisd.config;
-
-import org.apache.commons.lang.StringUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.boot.context.properties.ConfigurationProperties;
-import org.springframework.context.annotation.Configuration;
-
-import javax.annotation.PostConstruct;
-
-@Configuration
-@ConfigurationProperties("google")
-public class GoogleConfig {
-
-    public static class Client {
-
-        private String id;
-        private String secret;
-
-        public String getId() {
-            return id;
-        }
-
-        public void setId(String id) {
-            this.id = id;
-        }
-
-        public String getSecret() {
-            return secret;
-        }
-
-        public void setSecret(String secret) {
-            this.secret = secret;
-        }
-
-    }
-
-    private final Logger log = LoggerFactory.getLogger(GoogleConfig.class);
-
-    private boolean enabled;
-    private Client client = new Client();
-    private String medium = "io.kamax.google.id";
-    private String prefix = "google_";
-
-    public boolean isEnabled() {
-        return enabled;
-    }
-
-    public void setEnabled(boolean enabled) {
-        this.enabled = enabled;
-    }
-
-    public Client getClient() {
-        return client;
-    }
-
-    public void setClient(Client client) {
-        this.client = client;
-    }
-
-    public String getMedium() {
-        return medium;
-    }
-
-    public void setMedium(String medium) {
-        this.medium = medium;
-    }
-
-    public String getPrefix() {
-        return prefix;
-    }
-
-    public void setPrefix(String prefix) {
-        this.prefix = prefix;
-    }
-
-    @PostConstruct
-    public void build() {
-        log.info("--- Google config ---");
-        log.info("Enabled: {}", isEnabled());
-        log.info("Client ID: {}", getClient().getId());
-        log.info("Client secret set? {}", StringUtils.isNotBlank(getClient().getSecret()));
-        log.info("3PID medium: {}", getMedium());
-        log.info("MXID prefix: {}", getPrefix());
-    }
-
-}

src/main/java/io/kamax/mxisd/config/ldap/LdapConfig.java

@@ -359,6 +359,7 @@ public abstract class LdapConfig {
 
         log.info("Host: {}", connection.getHost());
         log.info("Port: {}", connection.getPort());
+        log.info("TLS: {}", connection.isTls());
         log.info("Bind DN: {}", connection.getBindDn());
         log.info("Base DN: {}", connection.getBaseDn());
 

src/main/java/io/kamax/mxisd/controller/auth/v1/RegistrationController.java

@@ -1,159 +0,0 @@
-/*
- * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2018 Kamax Sàrl
- *
- * https://www.kamax.io/
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU Affero General Public License as
- * published by the Free Software Foundation, either version 3 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU Affero General Public License for more details.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- */
-
-package io.kamax.mxisd.controller.auth.v1;
-
-import com.google.api.client.googleapis.auth.oauth2.GoogleIdToken;
-import com.google.gson.Gson;
-import com.google.gson.JsonObject;
-import io.kamax.matrix.MatrixID;
-import io.kamax.matrix.ThreePid;
-import io.kamax.matrix._MatrixID;
-import io.kamax.mxisd.backend.google.GoogleProviderBackend;
-import io.kamax.mxisd.dns.ClientDnsOverwrite;
-import io.kamax.mxisd.profile.ProfileManager;
-import io.kamax.mxisd.util.GsonParser;
-import io.kamax.mxisd.util.GsonUtil;
-import io.kamax.mxisd.util.RestClientUtils;
-import org.apache.commons.lang.StringUtils;
-import org.apache.http.client.methods.CloseableHttpResponse;
-import org.apache.http.client.methods.HttpGet;
-import org.apache.http.client.methods.HttpPost;
-import org.apache.http.client.utils.URIBuilder;
-import org.apache.http.impl.client.CloseableHttpClient;
-import org.apache.http.util.EntityUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.http.MediaType;
-import org.springframework.web.bind.annotation.CrossOrigin;
-import org.springframework.web.bind.annotation.RequestMapping;
-import org.springframework.web.bind.annotation.RequestMethod;
-import org.springframework.web.bind.annotation.RestController;
-
-import javax.servlet.http.HttpServletRequest;
-import javax.servlet.http.HttpServletResponse;
-import java.io.IOException;
-import java.net.URI;
-import java.security.GeneralSecurityException;
-import java.util.ArrayList;
-import java.util.List;
-
-@RestController
-@CrossOrigin
-@RequestMapping(produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
-public class RegistrationController {
-
-    private final Logger log = LoggerFactory.getLogger(RegistrationController.class);
-
-    private final String registerV1Url = "/_matrix/client/r0/register";
-
-    private GoogleProviderBackend google;
-    private ProfileManager pMgr;
-    private ClientDnsOverwrite dns;
-    private CloseableHttpClient client;
-    private Gson gson;
-    private GsonParser parser;
-
-    @Autowired
-    public RegistrationController(GoogleProviderBackend google, ProfileManager pMgr, ClientDnsOverwrite dns, CloseableHttpClient client) {
-        this.google = google;
-        this.pMgr = pMgr;
-        this.dns = dns;
-        this.client = client;
-        this.gson = GsonUtil.build();
-        this.parser = new GsonParser(gson);
-    }
-
-    private String resolveProxyUrl(HttpServletRequest req) {
-        URI target = URI.create(req.getRequestURL().toString());
-        URIBuilder builder = dns.transform(target);
-        String urlToLogin = builder.toString();
-        log.info("Proxy resolution: {} to {}", target.toString(), urlToLogin);
-        return urlToLogin;
-    }
-
-    @RequestMapping(path = registerV1Url, method = RequestMethod.GET)
-    public String getLogin(HttpServletRequest req, HttpServletResponse res) {
-        try (CloseableHttpResponse hsResponse = client.execute(new HttpGet(resolveProxyUrl(req)))) {
-            res.setStatus(hsResponse.getStatusLine().getStatusCode());
-            return EntityUtils.toString(hsResponse.getEntity());
-        } catch (IOException e) {
-            throw new RuntimeException(e);
-        }
-    }
-
-    @RequestMapping(path = registerV1Url, method = RequestMethod.POST)
-    public String register(HttpServletRequest req, HttpServletResponse res) {
-        List<ThreePid> ids = new ArrayList<>();
-        try {
-            JsonObject reqJsonObject = parser.parse(req.getInputStream());
-            GsonUtil.findObj(reqJsonObject, "auth").ifPresent(auth -> {
-                GsonUtil.findPrimitive(auth, "type").ifPresent(type -> {
-                    if (StringUtils.equals("io.kamax.google.auth", type.getAsString())) {
-                        log.info("Got registration attempt with Google account");
-                        if (!auth.has("googleId")) {
-                            throw new IllegalArgumentException("Google ID is missing");
-                        }
-
-                        String gId = auth.get("googleId").getAsString();
-                        try {
-                            GoogleIdToken token = google.extractToken(reqJsonObject.get("password").getAsString()).orElseThrow(() -> new IllegalArgumentException("Google ID Token is missing or invalid"));
-                            if (!StringUtils.equals(gId, token.getPayload().getSubject())) {
-                                throw new IllegalArgumentException("Google ID does not match token");
-                            }
-                            log.info("Google ID: {}", gId);
-
-                            ids.addAll(google.extractThreepids(token));
-
-                            auth.addProperty("type", "m.login.dummy");
-                            auth.remove("googleId");
-                            reqJsonObject.addProperty("username", "g-" + gId);
-                            reqJsonObject.addProperty("password", "");
-                        } catch (IOException | GeneralSecurityException e) {
-                            throw new RuntimeException(e);
-                        }
-                    }
-                });
-            });
-
-            log.info("Sending body: {}", gson.toJson(reqJsonObject));
-            HttpPost httpPost = RestClientUtils.post(resolveProxyUrl(req), gson, reqJsonObject);
-            try (CloseableHttpResponse httpResponse = client.execute(httpPost)) {
-                int sc = httpResponse.getStatusLine().getStatusCode();
-                String body = EntityUtils.toString(httpResponse.getEntity());
-                JsonObject json = parser.parse(body);
-                if (sc == 200 && json.has("user_id")) {
-                    // Required here as synapse doesn't call pass provider on register
-                    log.info("User was registered, adding 3PIDs");
-                    _MatrixID mxid = new MatrixID(json.get("user_id").getAsString());
-                    for (ThreePid tpid : ids) {
-                        pMgr.addThreepid(mxid, tpid);
-                    }
-                }
-                res.setStatus(sc);
-                return body;
-            }
-        } catch (IOException e) {
-            throw new RuntimeException(e);
-        }
-    }
-
-}

src/main/java/io/kamax/mxisd/controller/directory/v1/io/UserDirectorySearchResult.java

@@ -20,8 +20,8 @@
 
 package io.kamax.mxisd.controller.directory.v1.io;
 
-import java.util.ArrayList;
+import java.util.HashSet;
-import java.util.List;
+import java.util.Set;
 
 public class UserDirectorySearchResult {
 
@@ -55,10 +55,31 @@ public class UserDirectorySearchResult {
             this.userId = userId;
         }
 
+        @Override
+        public boolean equals(Object o) {
+            if (this == o) return true;
+            if (o == null || getClass() != o.getClass()) return false;
+
+            Result result = (Result) o;
+
+            if (displayName != null ? !displayName.equals(result.displayName) : result.displayName != null)
+                return false;
+            if (avatarUrl != null ? !avatarUrl.equals(result.avatarUrl) : result.avatarUrl != null) return false;
+            return userId.equals(result.userId);
+        }
+
+        @Override
+        public int hashCode() {
+            int result = displayName != null ? displayName.hashCode() : 0;
+            result = 31 * result + (avatarUrl != null ? avatarUrl.hashCode() : 0);
+            result = 31 * result + userId.hashCode();
+            return result;
+        }
+
     }
 
     private boolean limited;
-    private List<Result> results = new ArrayList<>();
+    private Set<Result> results = new HashSet<>();
 
     public boolean isLimited() {
         return limited;
@@ -68,11 +89,11 @@ public class UserDirectorySearchResult {
         this.limited = limited;
     }
 
-    public List<Result> getResults() {
+    public Set<Result> getResults() {
         return results;
     }
 
-    public void setResults(List<Result> results) {
+    public void setResults(Set<Result> results) {
         this.results = results;
     }
 

src/main/java/io/kamax/mxisd/lookup/provider/ForwarderProvider.java

@@ -21,6 +21,7 @@
 package io.kamax.mxisd.lookup.provider;
 
 import io.kamax.mxisd.config.ForwardConfig;
+import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.SingleLookupRequest;
 import io.kamax.mxisd.lookup.ThreePidMapping;
@@ -42,6 +43,9 @@ class ForwarderProvider implements IThreePidProvider {
     @Autowired
     private ForwardConfig cfg;
 
+    @Autowired
+    private MatrixConfig mxCfg;
+
     @Autowired
     private IRemoteIdentityServerFetcher fetcher;
 
@@ -62,10 +66,13 @@ class ForwarderProvider implements IThreePidProvider {
 
     @Override
     public Optional<SingleLookupReply> find(SingleLookupRequest request) {
-        for (String root : cfg.getServers()) {
+        for (String label : cfg.getServers()) {
-            Optional<SingleLookupReply> answer = fetcher.find(root, request);
+            for (String srv : mxCfg.getIdentity().getServers(label)) {
-            if (answer.isPresent()) {
+                log.info("Using forward server {}", srv);
-                return answer;
+                Optional<SingleLookupReply> answer = fetcher.find(srv, request);
+                if (answer.isPresent()) {
+                    return answer;
+                }
             }
         }
 
@@ -77,13 +84,15 @@ class ForwarderProvider implements IThreePidProvider {
         List<ThreePidMapping> mappingsToDo = new ArrayList<>(mappings);
         List<ThreePidMapping> mappingsFoundGlobal = new ArrayList<>();
 
-        for (String root : cfg.getServers()) {
+        for (String label : cfg.getServers()) {
-            log.info("{} mappings remaining: {}", mappingsToDo.size(), mappingsToDo);
+            for (String srv : mxCfg.getIdentity().getServers(label)) {
-            log.info("Querying {}", root);
+                log.info("{} mappings remaining: {}", mappingsToDo.size(), mappingsToDo);
-            List<ThreePidMapping> mappingsFound = fetcher.find(root, mappingsToDo);
+                log.info("Querying {}", srv);
-            log.info("{} returned {} mappings", root, mappingsFound.size());
+                List<ThreePidMapping> mappingsFound = fetcher.find(srv, mappingsToDo);
-            mappingsFoundGlobal.addAll(mappingsFound);
+                log.info("{} returned {} mappings", srv, mappingsFound.size());
-            mappingsToDo.removeAll(mappingsFound);
+                mappingsFoundGlobal.addAll(mappingsFound);
+                mappingsToDo.removeAll(mappingsFound);
+            }
         }
 
         return mappingsFoundGlobal;

src/main/java/io/kamax/mxisd/lookup/provider/RemoteIdentityServerFetcher.java

@@ -23,6 +23,7 @@ package io.kamax.mxisd.lookup.provider;
 import com.google.gson.Gson;
 import com.google.gson.JsonObject;
 import com.google.gson.JsonParseException;
+import io.kamax.matrix.json.GsonUtil;
 import io.kamax.mxisd.controller.identity.v1.ClientBulkLookupRequest;
 import io.kamax.mxisd.exception.InvalidResponseJsonException;
 import io.kamax.mxisd.lookup.SingleLookupReply;
@@ -33,18 +34,20 @@ import io.kamax.mxisd.matrix.IdentityServerUtils;
 import io.kamax.mxisd.util.GsonParser;
 import io.kamax.mxisd.util.RestClientUtils;
 import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
 import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.utils.URIBuilder;
 import org.apache.http.impl.client.CloseableHttpClient;
-import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.context.annotation.Lazy;
 import org.springframework.context.annotation.Scope;
 import org.springframework.stereotype.Component;
 
 import java.io.IOException;
-import java.net.HttpURLConnection;
+import java.net.URISyntaxException;
-import java.net.URL;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Optional;
@@ -59,6 +62,9 @@ public class RemoteIdentityServerFetcher implements IRemoteIdentityServerFetcher
     private Gson gson = new Gson();
     private GsonParser parser = new GsonParser(gson);
 
+    @Autowired
+    private CloseableHttpClient client;
+
     @Override
     public boolean isUsable(String remote) {
         return IdentityServerUtils.isUsable(remote);
@@ -69,24 +75,40 @@ public class RemoteIdentityServerFetcher implements IRemoteIdentityServerFetcher
         log.info("Looking up {} 3PID {} using {}", request.getType(), request.getThreePid(), remote);
 
         try {
-            HttpURLConnection rootSrvConn = (HttpURLConnection) new URL(
+            URIBuilder b = new URIBuilder(remote);
-                    remote + "/_matrix/identity/api/v1/lookup?medium=" + request.getType() + "&address=" + request.getThreePid()
+            b.setPath("/_matrix/identity/api/v1/lookup");
-            ).openConnection();
+            b.addParameter("medium", request.getType());
-            JsonObject obj = parser.parse(rootSrvConn.getInputStream());
+            b.addParameter("address", request.getThreePid());
-            if (obj.has("address")) {
+            HttpGet req = new HttpGet(b.build());
-                log.info("Found 3PID mapping: {}", gson.toJson(obj));
 
-                return Optional.of(SingleLookupReply.fromRecursive(request, gson.toJson(obj)));
+            try (CloseableHttpResponse res = client.execute(req)) {
+                int statusCode = res.getStatusLine().getStatusCode();
+                String body = EntityUtils.toString(res.getEntity());
+
+                if (statusCode != 200) {
+                    log.warn("Remote returned status code {}", statusCode);
+                    log.warn("Body: {}", body);
+                    return Optional.empty();
+                }
+
+                JsonObject obj = GsonUtil.parseObj(body);
+                if (obj.has("address")) {
+                    log.debug("Found 3PID mapping: {}", gson.toJson(obj));
+                    return Optional.of(SingleLookupReply.fromRecursive(request, gson.toJson(obj)));
+                }
+
+                log.info("Empty 3PID mapping from {}", remote);
+                return Optional.empty();
             }
-
-            log.info("Empty 3PID mapping from {}", remote);
-            return Optional.empty();
         } catch (IOException e) {
             log.warn("Error looking up 3PID mapping {}: {}", request.getThreePid(), e.getMessage());
             return Optional.empty();
         } catch (JsonParseException e) {
             log.warn("Invalid JSON answer from {}", remote);
             return Optional.empty();
+        } catch (URISyntaxException e) {
+            log.warn("Invalid remote address: {}", e.getMessage(), e);
+            return Optional.empty();
         }
     }
 
@@ -98,12 +120,15 @@ public class RemoteIdentityServerFetcher implements IRemoteIdentityServerFetcher
         mappingRequest.setMappings(mappings);
 
         String url = remote + "/_matrix/identity/api/v1/bulk_lookup";
-        CloseableHttpClient client = HttpClients.createDefault();
         try {
             HttpPost request = RestClientUtils.post(url, mappingRequest);
             try (CloseableHttpResponse response = client.execute(request)) {
-                if (response.getStatusLine().getStatusCode() != 200) {
+                int statusCode = response.getStatusLine().getStatusCode();
-                    log.info("Could not perform lookup at {} due to HTTP return code: {}", url, response.getStatusLine().getStatusCode());
+                String body = EntityUtils.toString(response.getEntity());
+
+                if (statusCode != 200) {
+                    log.warn("Could not perform lookup at {} due to HTTP return code: {}", url, statusCode);
+                    log.warn("Body: {}", body);
                     return mappingsFound;
                 }
 

src/main/java/io/kamax/mxisd/profile/ProfileManager.java

@@ -20,7 +20,6 @@
 
 package io.kamax.mxisd.profile;
 
-import io.kamax.matrix.ThreePid;
 import io.kamax.matrix._MatrixID;
 import io.kamax.matrix._ThreePid;
 import org.springframework.stereotype.Component;
@@ -33,21 +32,16 @@ import java.util.stream.Collectors;
 @Component
 public class ProfileManager {
 
-    private List<ProfileProvider> readers;
+    private List<ProfileProvider> providers;
-    private List<ProfileWriter> writers;
 
-    public ProfileManager(List<ProfileProvider> providers, List<ProfileWriter> writers) {
+    public ProfileManager(List<ProfileProvider> providers) {
-        this.readers = providers.stream()
+        this.providers = providers.stream()
                 .filter(ProfileProvider::isEnabled)
                 .collect(Collectors.toList());
-
-        this.writers = writers.stream()
-                .filter(ProfileWriter::isEnabled)
-                .collect(Collectors.toList());
     }
 
     public <T> List<T> get(Function<ProfileProvider, List<T>> function) {
-        return readers.stream()
+        return providers.stream()
                 .map(function)
                 .flatMap(Collection::stream)
                 .collect(Collectors.toList());
@@ -61,8 +55,4 @@ public class ProfileManager {
         return get(p -> p.getRoles(mxid));
     }
 
-    public void addThreepid(_MatrixID mxid, ThreePid tpid) {
-        writers.forEach(w -> w.addThreepid(mxid, tpid));
-    }
-
 }

src/main/java/io/kamax/mxisd/profile/ProfileWriter.java

@@ -1,32 +0,0 @@
-/*
- * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2018 Kamax Sàrl
- *
- * https://www.kamax.io/
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU Affero General Public License as
- * published by the Free Software Foundation, either version 3 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU Affero General Public License for more details.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- */
-
-package io.kamax.mxisd.profile;
-
-import io.kamax.matrix.ThreePid;
-import io.kamax.matrix._MatrixID;
-
-public interface ProfileWriter {
-
-    boolean isEnabled();
-
-    boolean addThreepid(_MatrixID mxid, ThreePid tpid);
-
-}

src/main/java/io/kamax/mxisd/session/SessionMananger.java

@@ -277,8 +277,7 @@ public class SessionMananger {
         }
 
         String is = servers.get(0);
-        String url = IdentityServerUtils.findIsUrlForDomain(is)
+        String url = IdentityServerUtils.findIsUrlForDomain(is).orElse(is);
-                .orElseThrow(() -> new InternalServerError(is + " could not be resolved to an Identity server"));
         log.info("Will use IS endpoint {}", url);
 
         String remoteSecret = session.isRemote() ? session.getRemoteSecret() : RandomStringUtils.randomAlphanumeric(16);

src/main/java/io/kamax/mxisd/spring/CloseableHttpClientFactory.java

@@ -30,7 +30,11 @@ public class CloseableHttpClientFactory {
 
     @Bean
     public CloseableHttpClient getClient() {
-        return HttpClients.custom().setUserAgent("mxisd").build();
+        return HttpClients.custom()
+                .setUserAgent("mxisd")
+                .setMaxConnPerRoute(Integer.MAX_VALUE)
+                .setMaxConnTotal(Integer.MAX_VALUE)
+                .build();
     }
 
 }

src/main/java/io/kamax/mxisd/spring/CryptoFactory.java

@@ -24,7 +24,7 @@ import io.kamax.matrix.crypto.KeyFileStore;
 import io.kamax.matrix.crypto.KeyManager;
 import io.kamax.matrix.crypto.SignatureManager;
 import io.kamax.mxisd.config.KeyConfig;
-import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.ServerConfig;
 import org.apache.commons.io.FileUtils;
 import org.springframework.context.annotation.Bean;
 import org.springframework.context.annotation.Configuration;
@@ -50,8 +50,8 @@ public class CryptoFactory {
     }
 
     @Bean
-    public SignatureManager getSignatureManager(KeyManager keyMgr, MatrixConfig mxCfg) {
+    public SignatureManager getSignatureManager(KeyManager keyMgr, ServerConfig cfg) {
-        return new SignatureManager(keyMgr, mxCfg.getDomain());
+        return new SignatureManager(keyMgr, cfg.getName());
     }
 
 }

src/main/java/io/kamax/mxisd/util/GsonParser.java

@@ -45,8 +45,8 @@ public class GsonParser {
         this.gson = gson;
     }
 
-    public JsonObject parse(String raw) {
+    public JsonObject parse(InputStream stream) throws IOException {
-        JsonElement el = parser.parse(raw);
+        JsonElement el = parser.parse(IOUtils.toString(stream, StandardCharsets.UTF_8));
         if (!el.isJsonObject()) {
             throw new InvalidResponseJsonException("Response body is not a JSON object");
         }
@@ -54,10 +54,6 @@ public class GsonParser {
         return el.getAsJsonObject();
     }
 
-    public JsonObject parse(InputStream stream) throws IOException {
-        return parse(IOUtils.toString(stream, StandardCharsets.UTF_8));
-    }
-
     public <T> T parse(HttpServletRequest req, Class<T> type) throws IOException {
         return gson.fromJson(parse(req.getInputStream()), type);
     }

src/main/resources/application.yaml

@@ -24,7 +24,7 @@ matrix:
   domain: ''
   identity:
     servers:
-      root:
+      matrix-org:
         - 'https://matrix.org'
 
 lookup:
@@ -174,9 +174,7 @@ wordpress:
         threepid: 'SELECT DISTINCT user_login, display_name FROM wp_users WHERE user_email LIKE ?'
 
 forward:
-  servers:
+  servers: []
-    - 'https://matrix.org'
-    - 'https://vector.im'
 
 threepid:
   medium:
@@ -226,13 +224,13 @@ session:
         toLocal: true
         toRemote:
           enabled: true
-          server: 'root'
+          server: 'matrix-org'
       forRemote:
         enabled: true
         toLocal: false
         toRemote:
           enabled: true
-          server: 'root'
+          server: 'matrix-org'
 
 notification:
 #  handler:

src/test/java/io/kamax/mxisd/backend/rest/RestDirectoryProviderTest.java

@@ -33,8 +33,7 @@ import org.junit.Test;
 import java.nio.charset.StandardCharsets;
 
 import static com.github.tomakehurst.wiremock.client.WireMock.*;
-import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.*;
-import static org.junit.Assert.assertTrue;
 
 public class RestDirectoryProviderTest {
 
@@ -89,8 +88,8 @@ public class RestDirectoryProviderTest {
 
         UserDirectorySearchResult result = p.searchByDisplayName(byNameSearch);
         assertTrue(!result.isLimited());
-        assertTrue(result.getResults().size() == 1);
+        assertEquals(1, result.getResults().size());
-        UserDirectorySearchResult.Result entry = result.getResults().get(0);
+        UserDirectorySearchResult.Result entry = result.getResults().iterator().next();
         assertNotNull(entry);
         assertTrue(StringUtils.equals(byNameAvatar, entry.getAvatarUrl()));
         assertTrue(StringUtils.equals(byNameDisplay, entry.getDisplayName()));
@@ -132,8 +131,8 @@ public class RestDirectoryProviderTest {
 
         UserDirectorySearchResult result = p.searchBy3pid(byThreepidSearch);
         assertTrue(!result.isLimited());
-        assertTrue(result.getResults().size() == 1);
+        assertEquals(1, result.getResults().size());
-        UserDirectorySearchResult.Result entry = result.getResults().get(0);
+        UserDirectorySearchResult.Result entry = result.getResults().iterator().next();
         assertNotNull(entry);
         assertTrue(StringUtils.equals(byThreepidAvatar, entry.getAvatarUrl()));
         assertTrue(StringUtils.equals(byThreepidDisplay, entry.getDisplayName()));