Properly handle leading @ in search (Fix #79)

Enhance documentation to talk about server.name in DNS override for auth

application.example.yaml

@@ -47,18 +47,33 @@ key.path: ''
 storage.provider.sqlite.database: '/path/to/mxisd.db'
 
 
+####################
+# Fallback servers #
+####################
+#
+# Root/Central servers to be used as final fallback when performing lookups.
+# By default, for privacy reasons, matrix.org servers are not enabled anymore.
+# See the following issue: https://github.com/kamax-io/mxisd/issues/76
+#
+# If you would like to use them and trade away your privacy for convenience, uncomment the following option:
+#
+#forward.servers: ['matrix-org']
+
+
 ################
 # LDAP Backend #
 ################
 # If you would like to integrate with your AD/Samba/LDAP server,
 # see https://github.com/kamax-io/mxisd/blob/master/docs/backends/ldap.md
 
+
 ###############
 # SQL Backend #
 ###############
 # If you would like to integrate with a MySQL/MariaDB/PostgreQL/SQLite DB,
 # see https://github.com/kamax-io/mxisd/blob/master/docs/backends/sql.md
 
+
 ################
 # REST Backend #
 ################

docs/architecture.md

@@ -18,12 +18,9 @@ TCP 443
           |   +-------------------+
  TCP 8090 +-> | mxisd             |
               |                   |
-              | - Profile's 3PIDs >----+
-              | - 3PID Invites    |    |             +--------------------------+
-              +-|-----------------+    +>----------> | Central Identity service |
-                |                      |   TCP 443   | Matrix.org / Vector.im   |
-                |                      |             +--------------------------+
-                +>-------------------->+
+              | - Profile's 3PIDs |
+              | - 3PID Invites    |
+              +-|-----------------+
                 |
              TCP 443
                 |  +------------------------+

docs/faq.md

@@ -19,8 +19,9 @@ started and answer questions you might have.
 ### Do I need to use mxisd if I run a Homeserver?
 No, but it is strongly recommended, even if you don't use any Identity store or integration.
 
-In its default configuration, mxisd will talk to the central Matrix Identity servers and use other federated public
-servers when performing queries, giving you access to at least the same information as if you were not running it.
+In its default configuration, mxisd uses other federated public servers when performing queries.  
+It can also [be configured](features/identity.md#lookups) to use the central matrix.org servers, giving you access to at
+least the same information as if you were not running it.
 
 It will also give your users a choice to make their 3PIDs available publicly, ensuring they are made aware of the
 privacy consequences, which is not the case with the central Matrix.org servers.
@@ -70,18 +71,15 @@ So really, you should go with mxisd.
 ### Will I loose access to the central Matrix.org/Vector.im Identity data if I use mxisd?
 No.
 
-In its default configuration, mxisd act as a proxy to Matrix.org/Vector.im. You will have access to the same data and
-behaviour than if you were using them directly. There is no downside in using mxisd with the default configuration.
+In its default configuration, mxisd does not talk to the central Identity server matrix.org to avoid leaking your private
+data and those of people you might know.
 
-mxisd can also be configured not to talk to the central Identity servers if you wish.
+mxisd [can be configured](features/identity.md#lookups) to talk to the central Identity servers if you wish.
 
 ### So mxisd is just a big hack! I don't want to use non-official features!
-mxisd primary concern is to always be compatible with the Matrix ecosystem and the Identity service API.  
+mxisd primary concerns are your privacy and to always be compatible with the Matrix ecosystem and the Identity service API.  
 Whenever the API will be updated and/or enhanced, mxisd will follow, remaining 100% compatible with the ecosystem.
 
-Therefore, using mxisd is a safe choice. It will be like using the central Matrix.org Identity servers, yet not closing
-the door to a growing list of enhancements and integrations.
-
 ### Should I use mxisd if I don't host my own Homeserver?
 No.
 

docs/features/authentication.md

@@ -148,7 +148,8 @@ dns.overwrite.homeserver.client:
     value: 'http://localhost:8008'
 ```
 `name` must be the hostname of the URL that clients use when connecting to the Homeserver.
-In case the hostname is the same as your Matrix domain, you can use `${matrix.domain}` to auto-populate the `value`
-using the `matrix.domain` configuration option and avoid duplicating it.
+You can use `${server.name}` to auto-populate the `value` using the `server.name` configuration option and avoid duplicating it.
+In case the hostname is the same as your Matrix domain and `server.name` is not explicitely set in the config, `server.name` will default to
+`matrix.domain` and will still probably have the correct value.
 
 `value` is the base internal URL of the Homeserver, without any `/_matrix/..` or trailing `/`.

docs/features/directory.md

@@ -138,5 +138,5 @@ the `matrix.domain` configuration option and avoid duplicating it.
 You can configure if the Homeserver should be queried at all when doing a directory search.  
 To disable Homeserver results, set the following in mxisd configuration file:
 ```yaml
-directory.exclude.homeserever: true
+directory.exclude.homeserver: true
 ```

docs/features/federation.md

@@ -5,8 +5,8 @@ Federated Identity server using the DNS domain part of the 3PID.
 Emails are the best candidate for this kind of resolution which are DNS domain based already.  
 On the other hand, Phone numbers cannot be resolved this way.
 
-For 3PIDs which are not compatible with the DNS system, mxisd will talk to the central Identity server of matrix.org by
-default.
+For 3PIDs which are not compatible with the DNS system, mxisd can be configured to talk to fallback Identity servers like
+the central matrix.org one. See the [Identity feature](identity.md#lookups) for instructions on how to enable it.
 
 Outbound federation is enabled by default while inbound federation is opt-in and require a specific DNS record.
 
@@ -17,16 +17,14 @@ Outbound federation is enabled by default while inbound federation is opt-in and
               |                   |   |      +------> +----------+
               |                   |   |      |
               | Invites / Lookups |   |      |
- Federated    | +--------+        |   |      |        +-------------------+
- Identity  ---->| Remote |>-----------+      +------> | Remote Federated  |
- Server       | +--------+        |          |        | mxisd servers     |
-              |                   |          |        +-------------------+
-              | +--------+        |          |
- Homeserver --->| Local  |>------------------+
- and clients  | +--------+        |          |        +--------------------------+ 
-              +-------------------+          +------> | Central Identity service |
-                                                      | Matrix.org / Vector.im   |
-                                                      +--------------------------+
+ Federated    | +--------+        |   |      |
+ Identity  ---->| Remote |>-----------+      |
+ Server       | +--------+        |          |
+              |                   |          |
+              | +--------+        |          |        +-------------------+
+ Homeserver --->| Local  |>------------------+------> | Remote Federated  |
+ and clients  | +--------+        |                   | mxisd servers     |
+              +-------------------+                   +-------------------+
 ```
 
 ## Inbound

docs/features/identity.md

@@ -3,6 +3,16 @@
 
 Implementation of the [Unofficial Matrix Identity Service API](https://kamax.io/matrix/api/identity_service/unstable.html).
 
+## Lookups
+If you would like to use the central matrix.org Identity server to ensure maximum discovery at the cost of potentially
+leaking all your contacts information, add the following to your configuration:
+```yaml
+forward.servers:
+  - 'matrix-org'
+```
+**NOTE:** You should carefully consider enabling this option, which is discouraged.  
+For more info, see the [relevant issue](https://github.com/kamax-io/mxisd/issues/76).
+
 ## Room Invitations
 Resolution can be customized using the following configuration:
 

docs/threepids/session/session.md

@@ -117,6 +117,7 @@ The following example of configuration (incomplete extract) shows which items ar
 **IMPORTANT:** Most configuration items shown have default values and should not be included in your own configuration
 file unless you want to specifically overwrite them.
 ```yaml
+# CONFIGURATION EXAMPLE
 # DO NOT COPY/PASTE THIS IN YOUR CONFIGURATION
 session.policy.validation.enabled: true
 session.policy.validation.forLocal:
@@ -132,6 +133,7 @@ session.policy.validation.forRemote:
     enabled: true
     server: 'configExample'  # Not to be included in config! Already present in default config!
 # DO NOT COPY/PASTE THIS IN YOUR CONFIGURATION
+# CONFIGURATION EXAMPLE
 ```
 
 `session.policy.validation` is the core configuration to control what users configured to use your Identity server

src/main/java/io/kamax/mxisd/config/ldap/LdapConfig.java

@@ -359,6 +359,7 @@ public abstract class LdapConfig {
 
         log.info("Host: {}", connection.getHost());
         log.info("Port: {}", connection.getPort());
+        log.info("TLS: {}", connection.isTls());
         log.info("Bind DN: {}", connection.getBindDn());
         log.info("Base DN: {}", connection.getBaseDn());
 

src/main/java/io/kamax/mxisd/directory/DirectoryManager.java

@@ -75,6 +75,10 @@ public class DirectoryManager {
     }
 
     public UserDirectorySearchResult search(URI target, String accessToken, String query) {
+        if (StringUtils.startsWith(query, "@")) {
+            query = query.substring(1);
+        }
+
         log.info("Performing search for '{}'", query);
         log.info("Original request URL: {}", target);
         UserDirectorySearchResult result = new UserDirectorySearchResult();

src/main/java/io/kamax/mxisd/lookup/provider/ForwarderProvider.java

@@ -21,6 +21,7 @@
 package io.kamax.mxisd.lookup.provider;
 
 import io.kamax.mxisd.config.ForwardConfig;
+import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.SingleLookupRequest;
 import io.kamax.mxisd.lookup.ThreePidMapping;
@@ -42,6 +43,9 @@ class ForwarderProvider implements IThreePidProvider {
     @Autowired
     private ForwardConfig cfg;
 
+    @Autowired
+    private MatrixConfig mxCfg;
+
     @Autowired
     private IRemoteIdentityServerFetcher fetcher;
 
@@ -62,12 +66,15 @@ class ForwarderProvider implements IThreePidProvider {
 
     @Override
     public Optional<SingleLookupReply> find(SingleLookupRequest request) {
-        for (String root : cfg.getServers()) {
-            Optional<SingleLookupReply> answer = fetcher.find(root, request);
+        for (String label : cfg.getServers()) {
+            for (String srv : mxCfg.getIdentity().getServers(label)) {
+                log.info("Using forward server {}", srv);
+                Optional<SingleLookupReply> answer = fetcher.find(srv, request);
                 if (answer.isPresent()) {
                     return answer;
                 }
             }
+        }
 
         return Optional.empty();
     }
@@ -77,14 +84,16 @@ class ForwarderProvider implements IThreePidProvider {
         List<ThreePidMapping> mappingsToDo = new ArrayList<>(mappings);
         List<ThreePidMapping> mappingsFoundGlobal = new ArrayList<>();
 
-        for (String root : cfg.getServers()) {
+        for (String label : cfg.getServers()) {
+            for (String srv : mxCfg.getIdentity().getServers(label)) {
                 log.info("{} mappings remaining: {}", mappingsToDo.size(), mappingsToDo);
-            log.info("Querying {}", root);
-            List<ThreePidMapping> mappingsFound = fetcher.find(root, mappingsToDo);
-            log.info("{} returned {} mappings", root, mappingsFound.size());
+                log.info("Querying {}", srv);
+                List<ThreePidMapping> mappingsFound = fetcher.find(srv, mappingsToDo);
+                log.info("{} returned {} mappings", srv, mappingsFound.size());
                 mappingsFoundGlobal.addAll(mappingsFound);
                 mappingsToDo.removeAll(mappingsFound);
             }
+        }
 
         return mappingsFoundGlobal;
     }

src/main/resources/application.yaml

@@ -24,7 +24,7 @@ matrix:
   domain: ''
   identity:
     servers:
-      root:
+      matrix-org:
         - 'https://matrix.org'
 
 lookup:
@@ -174,9 +174,7 @@ wordpress:
         threepid: 'SELECT DISTINCT user_login, display_name FROM wp_users WHERE user_email LIKE ?'
 
 forward:
-  servers:
-    - 'https://matrix.org'
-    - 'https://vector.im'
+  servers: []
 
 threepid:
   medium:
@@ -226,13 +224,13 @@ session:
         toLocal: true
         toRemote:
           enabled: true
-          server: 'root'
+          server: 'matrix-org'
       forRemote:
         enabled: true
         toLocal: false
         toRemote:
           enabled: true
-          server: 'root'
+          server: 'matrix-org'
 
 notification:
 #  handler: