Stable implementation of Directory integration

- Documentation
- Allow to specific other attributes in LDAP to include in the search

docs/features/directory-users.md

@@ -1,14 +1,146 @@
-- Only work for LDAP and SQL
+# User Directory
-  - For LDAP: Set LDAP config (new global filter, optional: threepids attributes if the default identity queries were changed)
+This feature allows you to search for existing and/or potential users that are already present in your Identity backend
-  - For SQL: Use `synapseSql` module with `type: {sqlite|postgresql}` and `database` as JDBC url after `jdbc:driver:`
+or that already share a room with you on the Homeserver.
-    - `/path/to/db` for `sqlite`
+
-    - `//host/db?username...` for `postgresql`)
+Without any integration, synapse:
-- Configure DNS overwrite for the Homeserver hostname used when connecting as a client (and mention ${matrix.domain} can be used)
+- Only search within the users **already** known to you
+- Only search on the Display Name and the Matrix ID
+
+With mxisd integration, you can:
+- Search on Matrix ID, Display name and 3PIDs (Email, phone numbers) of any users already in your configured backend
+- Search for users which you are not in contact with yet. Super useful for corporations who want to give Matrix access
+internally, so users can just find themselves **prior** to having any common room(s)
+- Use any attribute of your backend to extend the search!
+
+## Overview
+This is performed by intercepting the Homeserver endpoint `/_matrix/client/r0/user_directory/search` like so:
+```
+           +----------------------------------------------+
+client --> | Reverse proxy                                                                         Step 2
+           |                                                Step 1  +-------------------------+
+           |   /_matrix/client/r0/user_directory/search ----------> |                         |  Search in   +---------+
+           |                        /\                              | mxisd - Identity server | -----------> | Backend |
+           |   /_matrix/*            \----------------------------- |                         |  all users   +---------+
+           |        |            Step 4: Send back merged results   +-------------------------+
+           +--------|-------                                                     |
+                    |                                                          Step 3
+                    |                                                            |
+                    |    +------------+                                Search in known users
+                    \--> | Homeserver | <----------------------------------------/
+                         +------------+   /_matrix/client/r0/user_directory/search
+```
+
+## Requirements
+- Reverse proxy setup, which you should already have in place if you use mxisd
+- Compatible backends:
+  - LDAP
+  - SQL
+  - REST
+  
+## Configuration
+### Reverse proxy
+Apache2 configuration to put under the relevant virtual domain:
+```
+ProxyPreserveHost on
+ProxyPass /_matrix/identity/ http://mxisdInternalIpAddress:8090/_matrix/identity/
+ProxyPass /_matrix/client/r0/user_directory/ http://mxisdInternalIpAddress:8090/_matrix/client/r0/user_directory/
+ProxyPass /_matrix/ http://HomeserverInternalIpAddress:8008/_matrix/
+
+```
+`ProxyPreserveHost` or equivalent must be enabled to detect to which Homeserver mxisd should talk to when building
+results.
+
+### Backend
+#### LDAP
+Configuration structure has been altered so queries are automatically built from a global or specific filter and a list
+of attributes. To ensure Directory feature works, here how the LDAP configuration should look like:
+```
+ldap:
+  enabled: false
+  filter: '(memberOf=CN=Matrix Users,OU=Groups,DC=example,DC=org)'
+  connection:
+    host: 'ldapIpOrDomain'
+    bindDn: 'CN=Matrix Identity Server,OU=Accounts,DC=example,DC=org'
+    bindPassword: 'mxisd'
+    baseDn: 'OU=Accounts,DC=example,DC=org'
+  attribute:
+    uid:
+      type: 'uid'
+      value: 'userPrincipalName'
+    name: 'displayName'
+    threepid:
+      email:
+        - 'mailPrimaryAddress'
+        - 'mail'
+        - 'otherMailbox'
+      msisdn:
+        - 'telephoneNumber'
+        - 'mobile'
+        - 'homePhone'
+        - 'otherTelephone'
+        - 'otherMobile'
+        - 'otherHomePhone'
+  directory:
+    attribute:
+      other:
+        - 'employeeNumber'
+        - 'someOtherAttribute'
+```
+Previous configuration entries that contained queries with the `%3pid` placeholder should not be used anymore, unless
+specifically overwritten. Instead, add all attributes to the relevant sections.
+
+If you would like to include an attribute which is not a display name or a 3PID, you can use the
+`directory.attribute.other` to list any extra attributes you want included in searches.  
+If you do not want to include any extra attribute, that configuration section can be skipped. 
+
+#### SQL
+If you plan to integrate directory search directly with synapse, use the `synapseSql` provider, based on the following
+config:
+```
+synapseSql:
+  enabled: true
+  type: <database ID>
+  connection: ``
+```
+`type` and `connection`, including any other configuration item, follow the same values as the regular `sql` backend.
+
+---
+
+For the regular SQL backend, the following configuration items are available:
+```
+sql:
+  directory:
+    enabled: true
+    query:
+      name:
+        type: 'localpart'
+        value: 'SELECT idColumn, displayNameColumn FROM table WHERE displayNameColumn LIKE ?'
+      threepid:
+        type: 'localpart'
+        value: 'SELECT idColumn, displayNameColumn FROM table WHERE threepidColumn LIKE ?'
+```
+For each query, `type` can be used to tell mxisd how to process the ID column:
+- `localpart` will append the `matrix.domain` to it
+- `mxid` will use the ID as-is. If it is not a valid Matrix ID, the search will fail.
+
+`value` is the SQL query and must return two columns:
+- The first being the User ID
+- The second being its display name
+
+#### REST
+See the [dedicated document](../backends/rest.md)
+### DNS Overwrite
+Just like you need to configure a reverse proxy to send client requests to mxisd, you also need to configure mxisd with
+the internal IP of the Homeserver so it can talk to it directly to integrate its directory search.
+
+To do so, use the following configuration:
 ```
 dns.overwrite.homeserver.client:
   - name: 'example.org'
     value: 'http://localhost:8008'
 ```
-- Configure reverse proxy
+`name` must be the hostname of the URL that clients use when connecting to the Homeserver.  
-  - for `/_matrix/client/r0/user_directory/search` to `http://internalIpOfMxisd:8090/_matrix/client/r0/user_directory/search`
+In case the hostname is the same as your Matrix domain, you can use `${matrix.domain}` to auto-populate the value using
-  - With `ProxyPreserveHost on` on apache
+the `matrix.domain` configuration option and avoid duplicating it.
+
+`value` is the base intenral URL of the Homeserver, without any `/_matrix/..` or trailing `/`.

src/main/java/io/kamax/mxisd/backend/ldap/LdapDirectoryProvider.java

@@ -40,7 +40,6 @@ import org.springframework.stereotype.Component;
 
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Collections;
 import java.util.List;
 
 @Component
@@ -73,6 +72,7 @@ public class LdapDirectoryProvider extends LdapGenericBackend implements IDirect
             attributes.toArray(attArray);
 
             String searchQuery = buildOrQueryWithFilter(getCfg().getDirectory().getFilter(), "*" + query + "*", attArray);
+            log.debug("Query: {}", searchQuery);
             try (EntryCursor cursor = conn.search(getBaseDn(), searchQuery, SearchScope.SUBTREE, attArray)) {
                 while (cursor.next()) {
                     Entry entry = cursor.get();
@@ -102,13 +102,17 @@ public class LdapDirectoryProvider extends LdapGenericBackend implements IDirect
     @Override
     public UserDirectorySearchResult searchByDisplayName(String query) {
         log.info("Performing LDAP directory search on display name using '{}'", query);
-        return search(query, Collections.singletonList(getCfg().getAttribute().getName()));
+        List<String> attributes = new ArrayList<>();
+        attributes.add(getAt().getName());
+        attributes.addAll(getCfg().getDirectory().getAttribute().getOther());
+        return search(query, attributes);
     }
 
     @Override
     public UserDirectorySearchResult searchBy3pid(String query) {
         log.info("Performing LDAP directory search on 3PIDs using '{}'", query);
         List<String> attributes = new ArrayList<>();
+        attributes.add(getAt().getName());
         getCfg().getAttribute().getThreepid().forEach((k, v) -> attributes.addAll(v));
         return search(query, attributes);
     }

src/main/java/io/kamax/mxisd/config/ldap/LdapConfig.java

@@ -32,6 +32,8 @@ import org.springframework.boot.context.properties.ConfigurationProperties;
 import org.springframework.context.annotation.Configuration;
 
 import javax.annotation.PostConstruct;
+import java.util.ArrayList;
+import java.util.List;
 
 @Configuration
 @ConfigurationProperties(prefix = "ldap")
@@ -45,8 +47,31 @@ public class LdapConfig {
 
     public static class Directory {
 
+        public static class Attribute {
+
+            private List<String> other = new ArrayList<>();
+
+            public List<String> getOther() {
+                return other;
+            }
+
+            public void setOther(List<String> other) {
+                this.other = other;
+            }
+
+        }
+
+        private Attribute attribute = new Attribute();
         private String filter;
 
+        public Attribute getAttribute() {
+            return attribute;
+        }
+
+        public void setAttribute(Attribute attribute) {
+            this.attribute = attribute;
+        }
+
         public String getFilter() {
             return filter;
         }

src/main/java/io/kamax/mxisd/config/sql/SqlConfig.java

@@ -1,6 +1,6 @@
 package io.kamax.mxisd.config.sql;
 
-import com.google.gson.Gson;
+import io.kamax.mxisd.util.GsonUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -212,8 +212,9 @@ public abstract class SqlConfig {
             log.info("Type: {}", getType());
             log.info("Connection: {}", getConnection());
             log.info("Auth enabled: {}", getAuth().isEnabled());
+            log.info("Directory queries: {}", GsonUtil.build().toJson(getDirectory().getQuery()));
             log.info("Identity type: {}", getIdentity().getType());
-            log.info("Identity medium queries: {}", new Gson().toJson(getIdentity().getMedium()));
+            log.info("Identity medium queries: {}", GsonUtil.build().toJson(getIdentity().getMedium()));
         }
     }
 

src/main/resources/application.yaml

@@ -69,6 +69,8 @@ ldap:
   auth:
     filter: ''
   directory:
+    attribute:
+      other: []
     filter: ''
   identity:
     filter: ''
@@ -89,8 +91,10 @@ sql:
     enabled: false
     query:
       name:
+        type: 'localpart'
         value: 'SELECT 1'
       threepid:
+        type: 'localpart'
         value: 'SELECT 1'
   identity:
     type: 'mxid'