Improve the docker experience

- Only one env variable to configure on first usage
- Auto-generate a default config
- Improve doc

Dockerfile

@@ -6,6 +6,11 @@ EXPOSE 8090
 
 ADD build/libs/mxisd.jar /mxisd.jar
 ADD src/docker/start.sh /start.sh
+RUN mkdir -p /var/mxisd
 
 ENV JAVA_OPTS=""
+ENV CONF_FILE_PATH="/etc/mxisd/mxisd.yaml"
+ENV SIGN_KEY_PATH="/var/mxisd/sign.key"
+ENV SQLITE_DATABASE_PATH="/var/mxisd/mxisd.db"
+
 CMD [ "/start.sh" ]

README.md

@@ -5,20 +5,20 @@ mxisd - Federated Matrix Identity Server Daemon
 - [Overview](#overview)
 - [Features](#features)
 - [Why use mxisd](#why-use-mxisd)
-- [Quick start](#quick-start)
+- [Getting Started](#getting-started)
 - [Support](#support)
 - [Contribute](#contribute)
 - [FAQ](#faq)
 - [Contact](#contact)
 
 # Overview
-mxisd is a Federated Matrix Identity server for self-hosted Matrix infrastructures with enhanced features.
+mxisd is a Federated Matrix Identity server for self-hosted Matrix infrastructures with [enhanced features](#features).
   
 It is specifically designed to connect to an Identity store (AD/Samba/LDAP, SQL Database, Web services/application, ...)
 and ease the integration of the Matrix ecosystem with an existing infrastructure, or to build a new one using lasting
 tools.
 
-The core principle of mxisd is to map between Matrix IDs and 3PIDs (Thrid-party Identifiers) for the Homeserver and its
+The core principle of mxisd is to map between Matrix IDs and 3PIDs (Third-party Identifiers) for the Homeserver and its
 users. 3PIDs can be anything that identify a user, like:
 - Full name
 - Email address
@@ -29,8 +29,9 @@ users. 3PIDs can be anything that identify a user, like:
 - Facebook ID
 - ...
 
-mxisd is an enhanced Identity service, which implements the [Matrix Identity service API](https://matrix.org/docs/spec/identity_service/unstable.html)
+mxisd is an enhanced Identity service, which implements the
-but also several other features that greatly enhance user experience within Matrix.
+[Matrix Identity service API](https://matrix.org/docs/spec/identity_service/unstable.html) but also several
+[other features](#features) that greatly enhance user experience within Matrix.
 
 mxisd is the one stop shop for anything regarding Authentication, Directory and Identity management in Matrix built as a
 single coherent product.
@@ -64,115 +65,20 @@ currently **cannot be removed**
 - Users can directly find each other using whatever attribute is relevant within your Identity store
 - Federate your Identity lookups so you can discover others and/or others can discover you, all with extensive ACLs
 
-# Quick Start
+# Getting started
-1. [Preparation](#preparation)
+See the [dedicated document](docs/getting-started.md)
-2. [Install](#install)
-3. [Configure](#configure)
-4. [Integrate](#integrate)
-5. [Validate](#validate)
-
-Following these quick start instructions, you will have a basic setup that can perform recursive/federated lookups and
-talk to the central Matrix.org Identity service.  
-This will be a good ground work for further integration with your existing Identity stores.
-
-## Preparation
-You will need:
-- Homeserver
-- Reverse proxy with regular TLS/SSL certificate (Let's encrypt) for your mxisd domain
-
-As synapse requires an HTTPS connection when talking to an Identity service, a reverse proxy is required as mxisd does
-not support HTTPS listener at this time.
-
-For maximum integration, it is best to have your Homeserver and mxisd reachable via the same hostname.  
-You can also use a dedicated domain for mxisd, but will not have access to some features.
-
-Be aware of a [NAT/Reverse proxy gotcha](https://github.com/kamax-io/mxisd/wiki/Gotchas#nating) if you use the same
-hostname.
-
-The following Quick Start guide assumes you will host the Homeserver and mxisd under the same hostname.  
-If you would like a high-level view of the infrastructure and how each feature is integrated, see the
-[dedicated document](docs/architecture.md)
-
-## Install
-Install via:
-- [Debian package](docs/install/debian.md)
-- [Docker image](docs/install/docker.md)
-- [Sources](docs/build.md)
-
-See the [Latest release](https://github.com/kamax-io/mxisd/releases/latest) for links to each.
-
-## Configure
-Create/edit a minimal configuration (see installer doc for the location):
-```
-matrix.domain: 'MyMatrixDomain.org'
-key.path: '/path/to/signing.key.file'
-storage.provider.sqlite.database: '/path/to/mxisd.db'
-```  
-- `matrix.domain` should be set to your Homeserver domain
-- `key.path` will store the signing keys, which must be kept safe!
-- `storage.provider.sqlite.database` is the location of the SQLite Database file which will hold state (invites, etc.)
-
-If your HS/mxisd hostname is not the same as your Matrix domain, configure `server.name`.  
-Complete configuration guide is available [here](docs/configure.md).
-
-## Integrate
-For an overview of a typical mxisd infrastructure, see the [dedicated document](docs/architecture.md)
-### Reverse proxy
-#### Apache2
-In the VirtualHost handling the domain with SSL, add the following line and replace `0.0.0.0` by the right address/host.  
-**This line MUST be present before the one for the homeserver!**
-```
-ProxyPass /_matrix/identity/ http://0.0.0.0:8090/_matrix/identity/
-```
-
-Typical VirtualHost configuration would be:
-```
-<VirtualHost *:443>
-    ServerName example.org
-    
-    ...
-    
-    ProxyPreserveHost on
-    ProxyPass /_matrix/identity/ http://10.1.2.3:8090/_matrix/identity/
-    ProxyPass /_matrix/ http://10.1.2.3:8008/_matrix/
-</VirtualHost>
-```
-
-### Synapse
-Add your mxisd domain into the `homeserver.yaml` at `trusted_third_party_id_servers` and restart synapse.  
-In a typical configuration, you would end up with something similair to:
-```
-trusted_third_party_id_servers:
-    - matrix.org
-    - vector.im
-    - example.org
-```
-It is recommended to remove `matrix.org` and `vector.im` so only your own Identity server is allowed by synapse. 
-
-### Federation and network discovery
-See the [dedicated document](docs/features/federation.md).
-
-## Validate
-Log in using your Matrix client and set `https://example.org` as your Identity server URL, replacing `example.org` by
-the relevant hostname which you configured in your reverse proxy.  
-Invite `mxisd-lookup-test@kamax.io` to a room, which should be turned into a Matrix invite to `@mxisd-lookup-test:kamax.io`.  
-**NOTE:** you might not see a Matrix suggestion for the e-mail address, which is normal. Still proceed with the invite.
-  
-If it worked, it means you are up and running and can enjoy mxisd in its basic mode! Congratulations!  
-If it did not work, [get in touch](#support) and we'll do our best to get you started.
-
-You can now integrate mxisd further with your infrastructure using the various [features](docs/README.md) guides.
 
 # Support
 ## Community
 If you need help, want to report a bug or just say hi, you can reach us on Matrix at 
-[#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io) or [directly peek anonymously](https://view.matrix.org/room/!NPRUEisLjcaMtHIzDr:kamax.io/).
+[#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io) or
+[directly peek anonymously](https://view.matrix.org/room/!NPRUEisLjcaMtHIzDr:kamax.io/).
 For more high-level discussion about the Identity Server architecture/API, go to 
 [#matrix-identity:matrix.org](https://matrix.to/#/#matrix-identity:matrix.org)
 
 ## Professional
-If you would prefer professional support/custom development for mxisd and/or for Matrix in general, including other open source technologies/products, 
+If you would prefer professional support/custom development for mxisd and/or for Matrix in general, including other open
-please visit [our website](https://www.kamax.io/) to get in touch with us and get a quote.
+source technologies/products, please visit [our website](https://www.kamax.io/) to get in touch with us and get a quote.
 
 We offer affordable monthly/yearly support plans for mxisd, synapse or your full Matrix infrastructure.
 
@@ -185,8 +91,8 @@ You can contribute as a community member by:
 - Helping us improve the documentation: tell us what is good or not good (in an issue or in Matrix), or make a PR with
 changes you feel improve the doc.
 - Contribute code directly: we love contributors! All your contributions will be licensed under AGPLv3.
-- Donate! any donation is welcome, regardless how small or big. This will directly be used for the fixed costs and
+- [Donate!](https://liberapay.com/maximusdor/) Any donation is welcome, regardless how small or big, and will directly
-developer time.
+be used for the fixed costs and developer time of mxisd.
 
 You can contribute as an organisation/corporation by:
 - Get a [support contract](#support-professional). This is the best way you can help us as it ensures mxisd is
@@ -194,64 +100,7 @@ maintained regularly and you get direct access to the support team.
 - Sponsoring new features or bug fixes. [Get in touch](#contact) so we can discuss it further.
 
 # FAQ
-### Do I need to use mxisd if I run a Homeserver?
+See the [dedicated document](docs/faq.md)
-No, but it is recommended, even if you don't use any backends or integration.
-
-mxisd in its default configuration will use federation and involve the central Matrix.org Identity servers when
-performing queries, giving you access to at least the same information as if you were not running it.
-
-It will also give your users a choice to make their 3PIDs available publicly, ensuring they are made aware of the
-privacy consequences, which is not the case with the central Matrix.org servers.
-
-So mxisd is like your gatekeeper and guardian angel. It does not change what you already know, just adds some nice
-simple features on top of it.
-
-### I already use the synapse LDAP3 auth provider, why should I care about mxisd?
-The [synapse LDAP3 auth provider](https://github.com/matrix-org/matrix-synapse-ldap3) only handles on specific flow:
-validate credentials at login.
-
-It does not:
-- Auto-provision user profiles
-- Integrate with Identity management
-- Integrate with Directory searches
-- Protect you against the username case sensitivites issues in synapse
-
-mxisd is a replacement and enhancement of it, and also offers coherent results in all areas, which LDAP3 auth provider
-does not.
-
-### I saw that sydent is the official Identity server implemenation of the Matrix team, I should use that!
-You can, but [sydent](https://github.com/matrix-org/sydent):
-- [should not be used and/or self-hosted](https://github.com/matrix-org/sydent/issues/22)
-- is not meant to be linked to a specific Homeserver / domain
-- cannot handle federation or proxy lookups, effectively isolating your users from the rest of the network
-- forces you to duplicate all your identity data, so people can be found by 3PIDs
-- forces users to enter all their emails and phone numbers manually in their profile
-
-So really, you should go with mxisd.
-
-### I'm not sure I understand what an "Identity server" is supposed to be or do
-The current Identity service API is more a placeholder, as the Matrix devs did not have time so far to really work on
-what they want to do with that part of the ecosystem. Therefore, "Identity" is a misleading word currently.  
-Given the scope of the current Identity Service API, it would be best called "Invitation service".
-
-Because the current scope is so limited and no integration is done with the Homeserver, there was a big lack of features
-for groups/corporations/organisation. This is where mxisd comes in.
-
-mxisd implements the Identity Service API and also a set of features which are expected by regular users, truly living
-up to its "Identity server" name.
-
-### So mxisd is just a big hack! I don't want to use non-official features!
-mxisd primary concern is to always be compatible with the Matrix ecosystem and the Identity service API.  
-Whenever the API will be updated and/or enhanced, mxisd will follow, remaining 100% compatible with the ecosystem.
-
-We also directly talk with the Matrix developers to ensure all features we implement have their approval, and that we
-are in line with their vision of Identity management within the Matrix ecosystem.
-
-Therefore, using mxisd is a safe choice. It will be like using the central Matrix.org Identity servers, yet not closing
-the door to very nice enhancements and integrations.
-
-### Should I use mxisd if I don't host my own Homeserver?
-No
 
 # Contact
 Get in touch via:

build.gradle

@@ -47,7 +47,7 @@ String gitVersion() {
     def versionPattern = Pattern.compile("v(\\d+\\.)?(\\d+\\.)?(\\d+)(-.*)?")
     ByteArrayOutputStream out = new ByteArrayOutputStream()
     exec {
-        commandLine = ['git', 'describe', '--always', '--dirty']
+        commandLine = ['git', 'describe', '--tags', '--always', '--dirty']
         standardOutput = out
     }
     def v = out.toString().replace(System.lineSeparator(), '')

docs/_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-cayman

docs/backends/README.md

@@ -0,0 +1,5 @@
+# Identity Stores (Backends)
+- [Samba / Active Directory / LDAP](ldap.md)
+- [SQL Databases](sql.md)
+- [Website / Web service / Web app](rest.md)
+- [Google Firebase](firebase.md)

docs/backends/firebase.md

@@ -1,4 +1,14 @@
 # Google Firebase
+https://firebase.google.com/
+
+## Requirements
+This backend requires a suitable Matrix client capable of performing Firebase authentication and passing the following
+information:
+- Firebase User ID as Matrix username
+- Firebase token as Matrix password
+
+If your client is Riot, you will need a custom version.
+
 ## Configuration
 To be completed. For now, see default structure and values:
 ```

docs/backends/ldap.md

@@ -1,53 +1,99 @@
-# AD/Samba/LDAP backend
+# LDAP (Samba / Active Directory / OpenLDAP)
+## Getting started
+To use your LDAP backend, add the bare minimum configuration in mxisd config file:
+```
+ldap.enabled: true
+ldap.connection.host: 'ldapHostnameOrIp'
+ldap.connection.bindDn: 'CN=My Mxisd User,OU=Users,DC=example,DC=org'
+ldap.connection.bindPassword: 'TheUserPassword'
+ldap.connection.baseDn: 'OU=Users,DC=example,DC=org'
+```
+These are standard LDAP connection configuration. mxisd will try to connect on port default port 389 without encryption.
+
+---
+
+If you would like to use a TLS/SSL connection, use the following configuration options (STARTLS not supported):
+```
+ldap.connection.tls: true
+ldap.connection.port: 12345
+```
+
+---
+
+You can also set a default global filter on any LDAP queries:
+```
+ldap.filter: '(memberOf=CN=My Matrix Users,OU=Groups,DC=example,DC=org)'
+```
+This example would only return users part of the group called `My Matrix Users`.
+This can be overwritten or append in each specific flow describe below.
+
+---
+
+LDAP features are based on mapping LDAP attributes to Matrix concepts, like a Matrix ID, its localpart, the user display
+name, their email(s) and/or phone number(s).
+     
+Default attributes are well suited for Active Directory/Samba. In case you are using a native LDAP backend, you will
+most certainly configure those mappings.
+
+The following example would set the `uid` attribute as localpart and the Matrix display name to `cn`
+```
+ldap.attribute.uid.type: 'uid'
+ldap.attribute.uid.value: 'uid'
+ldap.attribute.name: 'cn'
+```
+
+You can also change the attribute lists for 3PID, like email or phone numbers.  
+The following example would overwrite the [default list of attributes](../../src/main/resources/application.yaml#L67)
+for emails and phone number:
+```
+ldap.attribute.threepid.email:
+  - 'mail'
+  - 'otherMailAttribute'
+
+ldap.attribute.threepid.msisdn:
+  - 'phone'
+  - 'otherPhoneAttribute'
+```
+
+## Identity
+Identity features (related to 3PID invites or searches) are enabled and configured using default values and no specific
+configuration item is needed to get started.
+
+If you would like to overwrite some global configuration relative to filter and/or attributes, see the Identity section
+of the Configuration below.
+
+## Authentication
+No further configuration is needed to enable authentication with LDAP once globally enabled and configured.  
+You have the possiblity to use a different query filter if you wish, see Configuration below.
+
+Profile auto-fill is enabled by default. It will use the `name` and `threepid` configuration options to get a lit of
+attributes to be used to build the user profile to pass on to synapse during authentication.
+
+## Directory
+No further configuration is needed to enable directory with LDAP once globally enabled and configured.
+
+If you would like to use extra attributes in search that are not 3PIDs, like nicknames, group names, employee number:
+```
+ldap.directory.attribute.other:
+  - 'myNicknameAttribute'
+  - 'memberOf'
+  - 'employeeNumberAttribute'
+```
+
 ## Configuration
-### Structure and default values
+Please read the [Configuration](../configure.md) explanatory note if you are not familiar with the terms used below.
-```
+ 
-ldap:
-  enabled: false
-  filter: ''
-  connection:
-    host: ''
-    tls: false
-    port: 389
-    bindDn: ''
-    bindPassword: ''
-    baseDn: ''
-  attribute:
-    uid:
-      type: 'uid'
-      value: 'userPrincipalName'
-    name: 'displayName'
-    threepid:
-      email:
-        - 'mailPrimaryAddress'
-        - 'mail'
-        - 'otherMailbox'
-      msisdn:
-        - 'telephoneNumber'
-        - 'mobile'
-        - 'homePhone'
-        - 'otherTelephone'
-        - 'otherMobile'
-        - 'otherHomePhone'
-  auth:
-    filter: ''
-  directory:
-    attribute:
-      other: []
-    filter: ''
-  identity:
-    filter: ''
-    medium:
-      email: ''
-      msisdn: ''
-```
 ### General
+Base path: `ldap`
+
 | Item      | Description                                                                               |
 |-----------|-------------------------------------------------------------------------------------------|
 | `enabled` | Globaly enable/disable the LDAP backend                                                   |
 | `filter`  | Global filter to apply on all LDAP queries. Can be overwritten in each applicable section |
 
 ### Connection
+Base path: `ldap.connection`
+
 | Item           | Description                                          |
 |----------------|------------------------------------------------------|
 | `host`         | Host to connect to                                   |
@@ -58,6 +104,8 @@ ldap:
 | `baseDn`       | Base DN for queries                                  |
 
 ### Attributes
+Base path: `ldap.attribute`
+
 | Item        | Description                                                                                                            |
 |-------------|------------------------------------------------------------------------------------------------------------------------|
 | `uid.type`  | Indicate how to process the User ID (UID) attribute:                                                                   |
@@ -68,11 +116,15 @@ ldap:
 | `threepid`  | Namespace where each key is a 3PID type and contains a list of attributes                                              |
 
 ### Authentication
+Base path: `ldap.auth`
+
 | Item     | Description                                                                                      |
 |----------|--------------------------------------------------------------------------------------------------|
 | `filter` | Specific user filter applied during authentication. Global filter is used if empty/blank/not set |
 
 ### Directory
+Base path: `ldap.directory`
+
 | Item              | Description                                                         |
 |-------------------|---------------------------------------------------------------------|
 | `attribute.other` | Additional attributes to be used when performing directory searches |
@@ -80,6 +132,8 @@ ldap:
 |                   | Global filter is used if empty/blank/not set                        |
 
 ### Identity
+Base path: `ldap.identity`
+
 | Item     | Description                                                                                       |
 |----------|---------------------------------------------------------------------------------------------------|
 | `filter` | Specific user filter applied during identity search. Global filter is used if empty/blank/not set | 

docs/configure.md

@@ -37,7 +37,7 @@ server:
 **WARNING:** mxisd might overwrite/adapt some values during launch. Those changes will not be reflected into copied keys.
 
 ## Categories
-For each category below, the base configuration path will be given, which needs to be appened to every configuration
+For each category below, the base configuration path will be given, which needs to be appended to every configuration
 item described.
 
 Example: if the base path was `basePath` and the following table was given:

docs/faq.md

@@ -0,0 +1,65 @@
+# FAQ
+### Do I need to use mxisd if I run a Homeserver?
+No, but it is recommended, even if you don't use any backends or integration.
+
+In its default configuration, mxisd will talk to the central Matrix Identity servers and use other federated public
+servers when performing queries, giving you access to at least the same information as if you were not running it.
+
+It will also give your users a choice to make their 3PIDs available publicly, ensuring they are made aware of the
+privacy consequences, which is not the case with the central Matrix.org servers.
+
+So mxisd is like your gatekeeper and guardian angel. It does not change what you already know, just adds some nice
+simple features on top of it.
+
+### I already use the synapse LDAP3 auth provider, why should I care about mxisd?
+The [synapse LDAP3 auth provider](https://github.com/matrix-org/matrix-synapse-ldap3) is not longer maintained and
+only handles on specific flow: validate credentials at login.
+
+It does not:
+- Auto-provision user profiles
+- Integrate with Identity management
+- Integrate with Directory searches
+- Protect you against the username case sensitivites issues in synapse
+
+mxisd is a replacement and enhancement of it, offering coherent results in all areas, which LDAP3 auth provider
+does not.
+
+### Sydent is the official Identity server implementation of the Matrix team, why not use that?
+You can, but [sydent](https://github.com/matrix-org/sydent):
+- [should not be used and/or self-hosted](https://github.com/matrix-org/sydent/issues/22)
+- is not meant to be linked to a specific Homeserver / domain
+- cannot handle federation or proxy lookups, effectively isolating your users from the rest of the network
+- forces you to duplicate all your identity data, so people can be found by 3PIDs
+- forces users to enter all their emails and phone numbers manually in their profile
+
+So really, you should go with mxisd.
+
+### Will I loose access to the central Matrix.org/Vector.im Identity data if I use mxisd?
+In its default configuration, mxisd act as a proxy to Matrix.org/Vector.im. You will have access to the same data and
+behaviour than if you were using them directly. There is no downside in using mxisd with the default configuration.
+
+mxisd can also be configured not to talk to the central Identity servers if you wish.
+
+### I'm not sure I understand what an "Identity server" is supposed to be or do
+The current Identity service API is more a placeholder, as the Matrix devs did not have time so far to really work on
+what they want to do with that part of the ecosystem. Therefore, "Identity" is a misleading word currently.  
+Given the scope of the current Identity Service API, it would be best called "Invitation service".
+
+Because the current scope is so limited and no integration is done with the Homeserver, there was a big lack of features
+for groups/corporations/organisation. This is where mxisd comes in.
+
+mxisd implements the Identity Service API and also a set of features which are expected by regular users, truly living
+up to its "Identity server" name.
+
+### So mxisd is just a big hack! I don't want to use non-official features!
+mxisd primary concern is to always be compatible with the Matrix ecosystem and the Identity service API.  
+Whenever the API will be updated and/or enhanced, mxisd will follow, remaining 100% compatible with the ecosystem.
+
+We also directly talk with the Matrix developers to ensure all features we implement have their approval, and that we
+are in line with their vision of Identity management within the Matrix ecosystem.
+
+Therefore, using mxisd is a safe choice. It will be like using the central Matrix.org Identity servers, yet not closing
+the door to very nice enhancements and integrations.
+
+### Should I use mxisd if I don't host my own Homeserver?
+No

docs/features/authentication.md

@@ -1,6 +1,7 @@
 # Authentication
-Performed via [synapse with REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth/blob/master/README.md)  
+Authentication is an enchanced Identity feature of mxisd to ensure coherent and centralized identity management.
-Point the `endpoint` to mxisd internal IP on port 8090
+
+It allows to use Identity stores configured in mxisd to authenticate users on your Homeserver.
 
 ## Overview
 ```
@@ -23,6 +24,31 @@ Point the `endpoint` to mxisd internal IP on port 8090
               |   user profiles          |    If valid credentials and supported by backend
               +--------------------------+
 ```
+Performed on [synapse with REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth/blob/master/README.md)
 
-## Profile auto-fill
+## Getting started
-To be documented
+Authentication is possible by linking synapse and mxisd together using the REST auth module
+(also known as password provider).
+
+### Synapse
+- Install the [REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth).
+- Edit your synapse configuration:
+  - As described by the auth module documentation
+  - Set `endpoint` to `http://mxisdAddress:8090` - Replace `mxisdAddress` by an IP/host name that provides a direct
+  connection to mxisd.  
+  This **MUST NOT** be a public address, and SHOULD NOT go through a reverse proxy.
+- Restart synapse
+
+### mxisd
+- Configure and enable at least one [Identity store](../backends/)
+- Restart mxisd
+
+### Validate
+Login on the Homeserver using credentials present in your backend.
+
+## Next steps
+### Profile auto-fill
+Auto-filling user profile depends on two conditions:
+- The REST auth module is configured for it, which is the case by default
+- Your Identity store is configured to provide profile data. See your Identity store [documentation](../backends/) on
+how to enable the feature.

docs/features/directory-users.md

@@ -9,6 +9,7 @@
     - [LDAP](#ldap)
     - [SQL](#sql)
     - [REST](#rest)
+- [Next steps](#next-steps)
 
 ## Description
 This feature allows you to search for existing and/or potential users that are already present in your Identity backend
@@ -165,3 +166,11 @@ For each query, `type` can be used to tell mxisd how to process the ID column:
 
 #### REST
 See the [dedicated document](../backends/rest.md)
+
+## Next steps
+### Homeserver results
+You can configure if the Homeserver should be queried at all when doing a directory search.  
+To disable Homeserver results, set the following in mxisd config file:
+```
+directory.exclude.homeserever: true
+```

docs/getting-started.md

@@ -0,0 +1,113 @@
+# Getting started
+1. [Preparation](#preparation)
+2. [Install](#install)
+3. [Configure](#configure)
+4. [Integrate](#integrate)
+5. [Validate](#validate)
+6. [Next steps](#next-steps)
+
+Following these quick start instructions, you will have a basic setup that can perform recursive/federated lookups and
+talk to the central Matrix.org Identity service.  
+This will be a good ground work for further integration with your existing Identity stores.
+
+## Preparation
+You will need:
+- Homeserver
+- Reverse proxy with regular TLS/SSL certificate (Let's encrypt) for your mxisd domain
+
+As synapse requires an HTTPS connection when talking to an Identity service, a reverse proxy is required as mxisd does
+not support HTTPS listener at this time.
+
+For maximum integration, it is best to have your Homeserver and mxisd reachable via the same hostname.  
+You can also use a dedicated domain for mxisd, but will not have access to some features.
+
+Be aware of a [NAT/Reverse proxy gotcha](https://github.com/kamax-io/mxisd/wiki/Gotchas#nating) if you use the same
+hostname.
+
+The following Quick Start guide assumes you will host the Homeserver and mxisd under the same hostname.  
+If you would like a high-level view of the infrastructure and how each feature is integrated, see the
+[dedicated document](architecture.md)
+
+## Install
+Install via:
+- [Debian package](install/debian.md)
+- [Docker image](install/docker.md)
+- [Sources](build.md)
+
+See the [Latest release](https://github.com/kamax-io/mxisd/releases/latest) for links to each.
+
+## Configure
+**NOTE**: please view the install instruction for your platform, as this step might be optional/handled for you.
+
+Create/edit a minimal configuration (see installer doc for the location):
+```
+matrix.domain: 'MyMatrixDomain.org'
+key.path: '/path/to/signing.key.file'
+storage.provider.sqlite.database: '/path/to/mxisd.db'
+```  
+- `matrix.domain` should be set to your Homeserver domain
+- `key.path` will store the signing keys, which must be kept safe!
+- `storage.provider.sqlite.database` is the location of the SQLite Database file which will hold state (invites, etc.)
+
+If your HS/mxisd hostname is not the same as your Matrix domain, configure `server.name`.  
+Complete configuration guide is available [here](configure.md).
+
+## Integrate
+For an overview of a typical mxisd infrastructure, see the [dedicated document](architecture.md)
+### Reverse proxy
+#### Apache2
+In the VirtualHost handling the domain with SSL, add the following line and replace `0.0.0.0` by the internal IP/hostname
+pointing to mxisd.  
+**This line MUST be present before the one for the homeserver!**
+```
+ProxyPass /_matrix/identity/ http://0.0.0.0:8090/_matrix/identity/
+```
+
+Typical VirtualHost configuration would be:
+```
+<VirtualHost *:443>
+    ServerName example.org
+    
+    ...
+    
+    ProxyPreserveHost on
+    ProxyPass /_matrix/identity/ http://10.1.2.3:8090/_matrix/identity/
+    ProxyPass /_matrix/ http://10.1.2.3:8008/_matrix/
+</VirtualHost>
+```
+
+### Synapse
+Add your mxisd domain into the `homeserver.yaml` at `trusted_third_party_id_servers` and restart synapse.  
+In a typical configuration, you would end up with something similair to:
+```
+trusted_third_party_id_servers:
+    - matrix.org
+    - vector.im
+    - example.org
+```
+It is recommended to remove `matrix.org` and `vector.im` so only your own Identity server is authoritative for your HS.
+
+## Validate
+Log in using your Matrix client and set `https://example.org` as your Identity server URL, replacing `example.org` by
+the relevant hostname which you configured in your reverse proxy.  
+Invite `mxisd-lookup-test@kamax.io` to a room, which should be turned into a Matrix invite to `@mxisd-lookup-test:kamax.io`.  
+**NOTE:** you might not see a Matrix suggestion for the e-mail address, which is normal. Still proceed with the invite.
+  
+If it worked, it means you are up and running and can enjoy mxisd in its basic mode! Congratulations!  
+If it did not work, [get in touch](#support) and we'll do our best to get you started.
+
+You can now integrate mxisd further with your infrastructure using the various [features](README.md) guides.
+
+## Next steps
+Once your mxisd server is up and running, here are the next steps to further enhance and integrate your installation:
+
+Enable extra features:
+- [Federation](features/federation.md)
+- [Authenticate with synapse](features/authentication.md), profile auto-provisioning if you wish
+- [Directory search](features/directory-users.md)
+
+Use your Identity stores:
+- [LDAP / Samba / Active directory](backends/ldap.md)
+- [SQL Database](backends/sql.md)
+- [Website / Web service / Web app](backends/rest.md)
+- [Google Firebase](backends/firebase.md)

docs/install/docker.md

@@ -5,10 +5,18 @@ Pull the latest stable image:
 docker pull kamax/mxisd
 ```
 
+## Configure
+On first run, simply using `MATRIX_DOMAIN` as an environment variable will create a default config for you.  
+You can also provide a configuration file named `mxisd.yaml` in the volume mapped to `/etc/mxisd` before starting your
+container.
+
 ## Run
-Run it (adapt volume paths to your host):
+Use the following command after adapting to your needs:
+- The `MATRIX_DOMAIN` environment variable to yours
+- The volumes host paths
+
 ```
-docker run --rm -v /data/mxisd/etc:/etc/mxisd -v /data/mxisd/var:/var/mxisd -p 8090:8090 -t kamax/mxisd
+docker run --rm -e MATRIX_DOMAIN=example.org -v /data/mxisd/etc:/etc/mxisd -v /data/mxisd/var:/var/mxisd -p 8090:8090 -t kamax/mxisd
 ```
 
 For more info, including the list of possible tags, see [the public repository](https://hub.docker.com/r/kamax/mxisd/)

src/docker/start.sh

@@ -1,2 +1,26 @@
 #!/bin/sh
+
+if ! [ -z "$CONF_FILE_PATH" ] && ! [ -f "CONF_FILE_PATH" ]; then
+    echo "Generating config file $CONF_FILE_PATH"
+    touch "CONF_FILE_PATH"
+
+    if ! [ -z "$MATRIX_DOMAIN" ]; then
+        echo "Setting matrix domain to $MATRIX_DOMAIN"
+        echo "matrix.domain: $MATRIX_DOMAIN" >> "$CONF_FILE_PATH"
+    fi
+
+    if ! [ -z "$SIGN_KEY_PATH" ]; then
+        echo "Setting signing key path to $SIGN_KEY_PATH"
+        echo "key.path: $SIGN_KEY_PATH" >> "$CONF_FILE_PATH"
+    fi
+
+    if ! [ -z "$SQLITE_DATABASE_PATH" ]; then
+        echo "Setting SQLite DB path to $SQLITE_DATABASE_PATH"
+        echo "storage.provider.sqlite.database: $SQLITE_DATABASE_PATH" >> "$CONF_FILE_PATH"
+    fi
+
+    echo "Starting mxisd..."
+    echo
+fi
+
 exec java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -Dspring.config.location=/etc/mxisd/ -Dspring.config.name=mxisd -jar /mxisd.jar

src/main/java/io/kamax/mxisd/backend/ldap/LdapAuthProvider.java

@@ -20,7 +20,11 @@
 
 package io.kamax.mxisd.backend.ldap;
 
+import com.google.i18n.phonenumbers.NumberParseException;
+import com.google.i18n.phonenumbers.PhoneNumberUtil;
+import io.kamax.matrix.ThreePidMedium;
 import io.kamax.matrix._MatrixID;
+import io.kamax.mxisd.ThreePid;
 import io.kamax.mxisd.UserIdType;
 import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
 import io.kamax.mxisd.auth.provider.BackendAuthResult;
@@ -41,12 +45,18 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
 import java.io.IOException;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
 
 @Component
 public class LdapAuthProvider extends LdapGenericBackend implements AuthenticatorProvider {
 
     private Logger log = LoggerFactory.getLogger(LdapAuthProvider.class);
 
+    private PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
+
     @Autowired
     public LdapAuthProvider(LdapConfig cfg, MatrixConfig mxCfg) {
         super(cfg, mxCfg);
@@ -57,6 +67,21 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
         return getCfg().isEnabled();
     }
 
+    private Optional<String> getMsisdn(String phoneNumber) {
+        try { // FIXME export into dedicated ThreePid class within SDK (copy from Firebase Auth)
+            return Optional.of(phoneUtil.format(
+                    phoneUtil.parse(
+                            phoneNumber,
+                            null // No default region
+                    ),
+                    PhoneNumberUtil.PhoneNumberFormat.E164
+            ).substring(1)); // We want without the leading +
+        } catch (NumberParseException e) {
+            log.warn("Invalid phone number: {}", phoneNumber);
+            return Optional.empty();
+        }
+    }
+
     @Override
     public BackendAuthResult authenticate(_MatrixID mxid, String password) {
         log.info("Performing auth for {}", mxid);
@@ -74,7 +99,15 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
 
             String userFilter = "(" + getUidAtt() + "=" + userFilterValue + ")";
             userFilter = buildWithFilter(userFilter, getCfg().getAuth().getFilter());
-            try (EntryCursor cursor = conn.search(getBaseDn(), userFilter, SearchScope.SUBTREE, getUidAtt(), getAt().getName())) {
+
+            Set<String> attributes = new HashSet<>();
+            attributes.add(getUidAtt());
+            attributes.add(getAt().getName());
+            getAt().getThreepid().forEach((k, v) -> attributes.addAll(v));
+            String[] attArray = new String[attributes.size()];
+            attributes.toArray(attArray);
+
+            try (EntryCursor cursor = conn.search(getBaseDn(), userFilter, SearchScope.SUBTREE, attArray)) {
                 while (cursor.next()) {
                     Entry entry = cursor.get();
                     String dn = entry.getDn().getName();
@@ -99,7 +132,24 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
                     log.info("DN {} is a valid match", dn);
 
                     // TODO should we canonicalize the MXID?
-                    return BackendAuthResult.success(mxid.getId(), UserIdType.MatrixID, name);
+                    BackendAuthResult result = BackendAuthResult.success(mxid.getId(), UserIdType.MatrixID, name);
+                    log.info("Processing 3PIDs for profile");
+                    getAt().getThreepid().forEach((k, v) -> {
+                        log.info("Processing 3PID type {}", k);
+                        v.forEach(attId -> {
+                            List<String> values = getAttributes(entry, attId);
+                            log.info("\tAttribute {} has {} value(s)", attId, values.size());
+                            getAttributes(entry, attId).forEach(tpidValue -> {
+                                if (ThreePidMedium.PhoneNumber.is(k)) {
+                                    tpidValue = getMsisdn(tpidValue).orElse(tpidValue);
+                                }
+                                result.withThreePid(new ThreePid(k, tpidValue));
+                            });
+                        });
+                    });
+
+                    log.info("Found {} 3PIDs", result.getProfile().getThreePids().size());
+                    return result;
                 }
             } catch (CursorLdapReferralException e) {
                 log.warn("Entity for {} is only available via referral, skipping", mxid);

src/main/java/io/kamax/mxisd/backend/ldap/LdapGenericBackend.java

@@ -25,6 +25,7 @@ import io.kamax.mxisd.config.ldap.LdapAttributeConfig;
 import io.kamax.mxisd.config.ldap.LdapConfig;
 import org.apache.commons.lang.StringUtils;
 import org.apache.directory.api.ldap.model.entry.Attribute;
+import org.apache.directory.api.ldap.model.entry.AttributeUtils;
 import org.apache.directory.api.ldap.model.entry.Entry;
 import org.apache.directory.api.ldap.model.exception.LdapException;
 import org.apache.directory.ldap.client.api.LdapConnection;
@@ -32,6 +33,9 @@ import org.apache.directory.ldap.client.api.LdapNetworkConnection;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import javax.naming.NamingEnumeration;
+import javax.naming.NamingException;
+import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.List;
 import java.util.Optional;
@@ -124,7 +128,6 @@ public abstract class LdapGenericBackend {
     public Optional<String> getAttribute(Entry entry, String attName) {
         Attribute attribute = entry.get(attName);
         if (attribute == null) {
-            log.info("DN {}: no attribute {}, skipping", entry.getDn(), attName);
             return Optional.empty();
         }
 
@@ -137,4 +140,22 @@ public abstract class LdapGenericBackend {
         return Optional.of(value);
     }
 
+    public List<String> getAttributes(Entry entry, String attName) {
+        List<String> values = new ArrayList<>();
+        javax.naming.directory.Attribute att = AttributeUtils.toAttributes(entry).get(attName);
+        if (att == null) {
+            return values;
+        }
+
+        try {
+            NamingEnumeration<?> list = att.getAll();
+            while (list.hasMore()) {
+                values.add(list.next().toString());
+            }
+        } catch (NamingException e) {
+            log.warn("Error while processing LDAP attribute {}, result could be incomplete!", attName, e);
+        }
+        return values;
+    }
+
 }

src/main/java/io/kamax/mxisd/backend/ldap/LdapThreePidProvider.java

@@ -68,13 +68,16 @@ public class LdapThreePidProvider extends LdapGenericBackend implements IThreePi
     }
 
     private Optional<String> lookup(LdapConnection conn, String medium, String value) {
-        Optional<String> queryOpt = getCfg().getIdentity().getQuery(medium);
+        Optional<String> tPidQueryOpt = getCfg().getIdentity().getQuery(medium);
-        if (!queryOpt.isPresent()) {
+        if (!tPidQueryOpt.isPresent()) {
             log.warn("{} is not a configured 3PID type for LDAP lookup", medium);
             return Optional.empty();
         }
 
-        String searchQuery = queryOpt.get().replaceAll(getCfg().getIdentity().getToken(), value);
+        // we merge 3PID specific query with global/specific filter, if one exists.
+        String tPidQuery = tPidQueryOpt.get().replaceAll(getCfg().getIdentity().getToken(), value);
+        String searchQuery = buildWithFilter(tPidQuery, getCfg().getIdentity().getFilter());
+
         try (EntryCursor cursor = conn.search(getBaseDn(), searchQuery, SearchScope.SUBTREE, getUidAtt())) {
             while (cursor.next()) {
                 Entry entry = cursor.get();

src/main/java/io/kamax/mxisd/backend/sql/GenericSqlAuthProvider.java

@@ -24,7 +24,7 @@ import io.kamax.matrix._MatrixID;
 import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
 import io.kamax.mxisd.auth.provider.BackendAuthResult;
 import io.kamax.mxisd.config.ServerConfig;
-import io.kamax.mxisd.config.sql.SqlProviderConfig;
+import io.kamax.mxisd.config.sql.GenericSqlProviderConfig;
 import io.kamax.mxisd.invitation.InvitationManager;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -32,15 +32,15 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
 @Component
-public class SqlAuthProvider implements AuthenticatorProvider {
+public class GenericSqlAuthProvider implements AuthenticatorProvider {
 
-    private Logger log = LoggerFactory.getLogger(SqlAuthProvider.class);
+    private Logger log = LoggerFactory.getLogger(GenericSqlAuthProvider.class);
 
     @Autowired
     private ServerConfig srvCfg;
 
     @Autowired
-    private SqlProviderConfig cfg;
+    private GenericSqlProviderConfig cfg;
 
     @Autowired
     private InvitationManager invMgr;

src/main/java/io/kamax/mxisd/backend/sql/GenericSqlDirectoryProvider.java

@@ -22,8 +22,8 @@ package io.kamax.mxisd.backend.sql;
 
 import io.kamax.matrix.MatrixID;
 import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.sql.GenericSqlProviderConfig;
 import io.kamax.mxisd.config.sql.SqlConfig;
-import io.kamax.mxisd.config.sql.SqlProviderConfig;
 import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult;
 import io.kamax.mxisd.directory.IDirectoryProvider;
 import io.kamax.mxisd.exception.InternalServerError;
@@ -39,16 +39,16 @@ import java.util.Optional;
 
 import static io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult.Result;
 
-public abstract class SqlDirectoryProvider implements IDirectoryProvider {
+public abstract class GenericSqlDirectoryProvider implements IDirectoryProvider {
 
-    private Logger log = LoggerFactory.getLogger(SqlDirectoryProvider.class);
+    private Logger log = LoggerFactory.getLogger(GenericSqlDirectoryProvider.class);
 
     protected SqlConfig cfg;
     private MatrixConfig mxCfg;
 
     private SqlConnectionPool pool;
 
-    public SqlDirectoryProvider(SqlConfig cfg, MatrixConfig mxCfg) {
+    public GenericSqlDirectoryProvider(SqlConfig cfg, MatrixConfig mxCfg) {
         this.cfg = cfg;
         this.pool = new SqlConnectionPool(cfg);
         this.mxCfg = mxCfg;
@@ -72,7 +72,7 @@ public abstract class SqlDirectoryProvider implements IDirectoryProvider {
         return Optional.of(item);
     }
 
-    public UserDirectorySearchResult search(String searchTerm, SqlProviderConfig.Query query) {
+    public UserDirectorySearchResult search(String searchTerm, GenericSqlProviderConfig.Query query) {
         try (Connection conn = pool.get()) {
             log.info("Will execute query: {}", query.getValue());
             try (PreparedStatement stmt = conn.prepareStatement(query.getValue())) {

src/main/java/io/kamax/mxisd/backend/sql/GenericSqlThreePidProvider.java

@@ -0,0 +1,36 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.sql;
+
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.sql.GenericSqlProviderConfig;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+@Component
+public class GenericSqlThreePidProvider extends SqlThreePidProvider {
+
+    @Autowired
+    public GenericSqlThreePidProvider(GenericSqlProviderConfig cfg, MatrixConfig mxCfg) {
+        super(cfg, mxCfg);
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/sql/SqlThreePidProvider.java

@@ -22,7 +22,7 @@ package io.kamax.mxisd.backend.sql;
 
 import io.kamax.matrix.MatrixID;
 import io.kamax.mxisd.config.MatrixConfig;
-import io.kamax.mxisd.config.sql.SqlProviderConfig;
+import io.kamax.mxisd.config.sql.SqlConfig;
 import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.SingleLookupRequest;
 import io.kamax.mxisd.lookup.ThreePidMapping;
@@ -30,8 +30,6 @@ import io.kamax.mxisd.lookup.provider.IThreePidProvider;
 import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Component;
 
 import java.sql.Connection;
 import java.sql.PreparedStatement;
@@ -41,18 +39,16 @@ import java.util.ArrayList;
 import java.util.List;
 import java.util.Optional;
 
-@Component
+public abstract class SqlThreePidProvider implements IThreePidProvider {
-public class SqlThreePidProvider implements IThreePidProvider {
 
     private Logger log = LoggerFactory.getLogger(SqlThreePidProvider.class);
 
-    private SqlProviderConfig cfg;
+    private SqlConfig cfg;
     private MatrixConfig mxCfg;
 
     private SqlConnectionPool pool;
 
-    @Autowired
+    public SqlThreePidProvider(SqlConfig cfg, MatrixConfig mxCfg) {
-    public SqlThreePidProvider(SqlProviderConfig cfg, MatrixConfig mxCfg) {
         this.cfg = cfg;
         this.pool = new SqlConnectionPool(cfg);
         this.mxCfg = mxCfg;

src/main/java/io/kamax/mxisd/backend/sql/SynapseSqlThreePidProvider.java

@@ -0,0 +1,36 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.sql;
+
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.sql.synapse.SynapseSqlProviderConfig;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+@Component
+public class SynapseSqlThreePidProvider extends SqlThreePidProvider {
+
+    @Autowired
+    public SynapseSqlThreePidProvider(SynapseSqlProviderConfig cfg, MatrixConfig mxCfg) {
+        super(cfg, mxCfg);
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/sql/SynapseSqliteDirectoryProvider.java

@@ -21,7 +21,7 @@
 package io.kamax.mxisd.backend.sql;
 
 import io.kamax.mxisd.config.MatrixConfig;
-import io.kamax.mxisd.config.sql.SqlProviderConfig;
+import io.kamax.mxisd.config.sql.GenericSqlProviderConfig;
 import io.kamax.mxisd.config.sql.synapse.SynapseSqlProviderConfig;
 import io.kamax.mxisd.exception.ConfigurationException;
 import org.apache.commons.lang.StringUtils;
@@ -32,9 +32,7 @@ import java.sql.PreparedStatement;
 import java.sql.SQLException;
 
 @Component
-public class SynapseSqliteDirectoryProvider extends SqlDirectoryProvider {
+public class SynapseSqliteDirectoryProvider extends GenericSqlDirectoryProvider {
-
-    private SynapseSqlProviderConfig cfg;
 
     @Autowired
     public SynapseSqliteDirectoryProvider(SynapseSqlProviderConfig cfg, MatrixConfig mxCfg) {
@@ -42,7 +40,7 @@ public class SynapseSqliteDirectoryProvider extends SqlDirectoryProvider {
 
         if (StringUtils.equals("sqlite", cfg.getType())) {
             String userId = "'@' || p.user_id || ':" + mxCfg.getDomain() + "'";
-            SqlProviderConfig.Type queries = cfg.getDirectory().getQuery();
+            GenericSqlProviderConfig.Type queries = cfg.getDirectory().getQuery();
             queries.getName().setValue(
                     "select " + userId + ", displayname from profiles p where displayname like ?");
             queries.getThreepid().setValue(
@@ -51,7 +49,7 @@ public class SynapseSqliteDirectoryProvider extends SqlDirectoryProvider {
                             "where t.address like ?");
         } else if (StringUtils.equals("postgresql", cfg.getType())) {
             String userId = "concat('@',p.user_id,':" + mxCfg.getDomain() + "')";
-            SqlProviderConfig.Type queries = cfg.getDirectory().getQuery();
+            GenericSqlProviderConfig.Type queries = cfg.getDirectory().getQuery();
             queries.getName().setValue(
                     "select " + userId + ", displayname from profiles p where displayname ilike ?");
             queries.getThreepid().setValue(

src/main/java/io/kamax/mxisd/config/DirectoryConfig.java

@@ -0,0 +1,59 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.config;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.context.annotation.Configuration;
+
+@Configuration
+@ConfigurationProperties("directory")
+public class DirectoryConfig {
+
+    private final transient Logger log = LoggerFactory.getLogger(DnsOverwriteConfig.class);
+
+    public static class Exclude {
+
+        private boolean homeserver;
+
+        public boolean getHomeserver() {
+            return homeserver;
+        }
+
+        public Exclude setHomeserver(boolean homeserver) {
+            this.homeserver = homeserver;
+            return this;
+        }
+
+    }
+
+    private Exclude exclude = new Exclude();
+
+    public Exclude getExclude() {
+        return exclude;
+    }
+
+    public void setExclude(Exclude exclude) {
+        this.exclude = exclude;
+    }
+
+}

src/main/java/io/kamax/mxisd/config/ldap/LdapConfig.java

@@ -162,11 +162,14 @@ public class LdapConfig {
             throw new IllegalStateException("LDAP port is not valid");
         }
 
+        if (StringUtils.isBlank(conn.getBaseDn())) {
+            throw new ConfigurationException("ldap.connection.baseDn");
+        }
+
         if (StringUtils.isBlank(attribute.getUid().getType())) {
             throw new IllegalStateException("Attribute UID Type cannot be empty");
         }
 
-
         if (StringUtils.isBlank(attribute.getUid().getValue())) {
             throw new IllegalStateException("Attribute UID value cannot be empty");
         }

src/main/java/io/kamax/mxisd/config/sql/GenericSqlProviderConfig.java

@@ -24,21 +24,14 @@ import org.springframework.boot.context.properties.ConfigurationProperties;
 import org.springframework.context.annotation.Configuration;
 import org.springframework.context.annotation.Primary;
 
-import javax.annotation.PostConstruct;
-
 @Configuration
 @ConfigurationProperties("sql")
 @Primary
-public class SqlProviderConfig extends SqlConfig {
+public class GenericSqlProviderConfig extends SqlConfig {
 
     @Override
     protected String getProviderName() {
         return "Generic SQL";
     }
 
-    @PostConstruct
-    public void build() {
-        super.build();
-    }
-
 }

src/main/java/io/kamax/mxisd/config/sql/SqlConfig.java

@@ -4,6 +4,7 @@ import io.kamax.mxisd.util.GsonUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import javax.annotation.PostConstruct;
 import java.util.HashMap;
 import java.util.Map;
 
@@ -36,22 +37,22 @@ public abstract class SqlConfig {
 
     public static class Type {
 
-        private SqlProviderConfig.Query name = new SqlProviderConfig.Query();
+        private GenericSqlProviderConfig.Query name = new GenericSqlProviderConfig.Query();
-        private SqlProviderConfig.Query threepid = new SqlProviderConfig.Query();
+        private GenericSqlProviderConfig.Query threepid = new GenericSqlProviderConfig.Query();
 
-        public SqlProviderConfig.Query getName() {
+        public GenericSqlProviderConfig.Query getName() {
             return name;
         }
 
-        public void setName(SqlProviderConfig.Query name) {
+        public void setName(GenericSqlProviderConfig.Query name) {
             this.name = name;
         }
 
-        public SqlProviderConfig.Query getThreepid() {
+        public GenericSqlProviderConfig.Query getThreepid() {
             return threepid;
         }
 
-        public void setThreepid(SqlProviderConfig.Query threepid) {
+        public void setThreepid(GenericSqlProviderConfig.Query threepid) {
             this.threepid = threepid;
         }
 
@@ -74,7 +75,7 @@ public abstract class SqlConfig {
     public static class Directory {
 
         private Boolean enabled;
-        private SqlProviderConfig.Type query = new SqlProviderConfig.Type();
+        private GenericSqlProviderConfig.Type query = new GenericSqlProviderConfig.Type();
 
         public Boolean isEnabled() {
             return enabled;
@@ -84,11 +85,11 @@ public abstract class SqlConfig {
             this.enabled = enabled;
         }
 
-        public SqlProviderConfig.Type getQuery() {
+        public GenericSqlProviderConfig.Type getQuery() {
             return query;
         }
 
-        public void setQuery(SqlProviderConfig.Type query) {
+        public void setQuery(GenericSqlProviderConfig.Type query) {
             this.query = query;
         }
 
@@ -138,9 +139,9 @@ public abstract class SqlConfig {
     private boolean enabled;
     private String type;
     private String connection;
-    private SqlProviderConfig.Auth auth = new SqlProviderConfig.Auth();
+    private GenericSqlProviderConfig.Auth auth = new GenericSqlProviderConfig.Auth();
-    private SqlProviderConfig.Directory directory = new SqlProviderConfig.Directory();
+    private GenericSqlProviderConfig.Directory directory = new GenericSqlProviderConfig.Directory();
-    private SqlProviderConfig.Identity identity = new SqlProviderConfig.Identity();
+    private GenericSqlProviderConfig.Identity identity = new GenericSqlProviderConfig.Identity();
 
     public boolean isEnabled() {
         return enabled;
@@ -166,35 +167,33 @@ public abstract class SqlConfig {
         this.connection = connection;
     }
 
-    public SqlProviderConfig.Auth getAuth() {
+    public GenericSqlProviderConfig.Auth getAuth() {
         return auth;
     }
 
-    public void setAuth(SqlProviderConfig.Auth auth) {
+    public void setAuth(GenericSqlProviderConfig.Auth auth) {
         this.auth = auth;
     }
 
-    public SqlProviderConfig.Directory getDirectory() {
+    public GenericSqlProviderConfig.Directory getDirectory() {
         return directory;
     }
 
-    public void setDirectory(SqlProviderConfig.Directory directory) {
+    public void setDirectory(GenericSqlProviderConfig.Directory directory) {
         this.directory = directory;
     }
 
-    public SqlProviderConfig.Identity getIdentity() {
+    public GenericSqlProviderConfig.Identity getIdentity() {
         return identity;
     }
 
-    public void setIdentity(SqlProviderConfig.Identity identity) {
+    public void setIdentity(GenericSqlProviderConfig.Identity identity) {
         this.identity = identity;
     }
 
     protected abstract String getProviderName();
 
-    public void build() {
+    protected void doBuild() {
-        log.info("--- " + getProviderName() + " Provider config ---");
-
         if (getAuth().isEnabled() == null) {
             getAuth().setEnabled(isEnabled());
         }
@@ -206,6 +205,13 @@ public abstract class SqlConfig {
         if (getIdentity().isEnabled() == null) {
             getIdentity().setEnabled(isEnabled());
         }
+    }
+
+    @PostConstruct
+    public void build() {
+        log.info("--- " + getProviderName() + " Provider config ---");
+
+        doBuild();
 
         log.info("Enabled: {}", isEnabled());
         if (isEnabled()) {
@@ -214,6 +220,7 @@ public abstract class SqlConfig {
             log.info("Auth enabled: {}", getAuth().isEnabled());
             log.info("Directory queries: {}", GsonUtil.build().toJson(getDirectory().getQuery()));
             log.info("Identity type: {}", getIdentity().getType());
+            log.info("3PID mapping query: {}", getIdentity().getQuery());
             log.info("Identity medium queries: {}", GsonUtil.build().toJson(getIdentity().getMedium()));
         }
     }

src/main/java/io/kamax/mxisd/config/sql/synapse/SynapseSqlProviderConfig.java

@@ -21,6 +21,7 @@
 package io.kamax.mxisd.config.sql.synapse;
 
 import io.kamax.mxisd.config.sql.SqlConfig;
+import org.apache.commons.lang.StringUtils;
 import org.springframework.boot.context.properties.ConfigurationProperties;
 import org.springframework.context.annotation.Configuration;
 
@@ -36,8 +37,23 @@ public class SynapseSqlProviderConfig extends SqlConfig {
     }
 
     @PostConstruct
-    public void build() {
+    public void doBuild() {
-        super.build();
+        super.doBuild();
+        // FIXME check that the DB is not the mxisd one
+        // See https://matrix.to/#/!NPRUEisLjcaMtHIzDr:kamax.io/$1509377583327omXkC:kamax.io
+
+        getAuth().setEnabled(false); // Synapse does the auth, we only act as a directory/identity service.
+
+        if (getDirectory().isEnabled()) {
+            //FIXME set default queries for name and threepid
+        }
+
+        if (getIdentity().isEnabled()) {
+            if (StringUtils.isBlank(getIdentity().getType())) {
+                getIdentity().setType("mxid");
+                getIdentity().setQuery("SELECT user_id AS uid FROM user_threepids WHERE medium = ? AND address = ?");
+            }
+        }
     }
 
 }

src/main/java/io/kamax/mxisd/directory/DirectoryManager.java

@@ -23,6 +23,7 @@ package io.kamax.mxisd.directory;
 import com.google.gson.Gson;
 import com.google.gson.JsonSyntaxException;
 import io.kamax.matrix.MatrixErrorInfo;
+import io.kamax.mxisd.config.DirectoryConfig;
 import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchRequest;
 import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult;
 import io.kamax.mxisd.dns.ClientDnsOverwrite;
@@ -54,6 +55,7 @@ public class DirectoryManager {
 
     private Logger log = LoggerFactory.getLogger(DirectoryManager.class);
 
+    private DirectoryConfig cfg;
     private List<IDirectoryProvider> providers;
 
     private ClientDnsOverwrite dns;
@@ -61,7 +63,8 @@ public class DirectoryManager {
     private Gson gson;
 
     @Autowired
-    public DirectoryManager(List<IDirectoryProvider> providers, ClientDnsOverwrite dns) {
+    public DirectoryManager(DirectoryConfig cfg, List<IDirectoryProvider> providers, ClientDnsOverwrite dns) {
+        this.cfg = cfg;
         this.dns = dns;
         this.client = HttpClients.custom().setUserAgent("mxisd").build(); //FIXME centralize
         this.gson = GsonUtil.build();
@@ -76,37 +79,41 @@ public class DirectoryManager {
         log.info("Original request URL: {}", target);
         UserDirectorySearchResult result = new UserDirectorySearchResult();
 
-        URIBuilder builder = dns.transform(target);
+        if (cfg.getExclude().getHomeserver()) {
-        log.info("Querying HS at {}", builder);
+            log.info("Skipping HS directory data, disabled in config");
-        builder.setParameter("access_token", accessToken);
+        } else {
-        HttpPost req = RestClientUtils.post(
+            URIBuilder builder = dns.transform(target);
-                builder.toString(),
+            log.info("Querying HS at {}", builder);
-                new UserDirectorySearchRequest(query));
+            builder.setParameter("access_token", accessToken);
-        try (CloseableHttpResponse res = client.execute(req)) {
+            HttpPost req = RestClientUtils.post(
-            int status = res.getStatusLine().getStatusCode();
+                    builder.toString(),
-            Charset charset = ContentType.getOrDefault(res.getEntity()).getCharset();
+                    new UserDirectorySearchRequest(query));
-            String body = IOUtils.toString(res.getEntity().getContent(), charset);
+            try (CloseableHttpResponse res = client.execute(req)) {
+                int status = res.getStatusLine().getStatusCode();
+                Charset charset = ContentType.getOrDefault(res.getEntity()).getCharset();
+                String body = IOUtils.toString(res.getEntity().getContent(), charset);
 
-            if (status != 200) {
+                if (status != 200) {
-                MatrixErrorInfo info = gson.fromJson(body, MatrixErrorInfo.class);
+                    MatrixErrorInfo info = gson.fromJson(body, MatrixErrorInfo.class);
-                if (StringUtils.equals("M_UNRECOGNIZED", info.getErrcode())) { // FIXME no hardcoding, use Enum
+                    if (StringUtils.equals("M_UNRECOGNIZED", info.getErrcode())) { // FIXME no hardcoding, use Enum
-                    log.warn("Homeserver does not support Directory feature, skipping");
+                        log.warn("Homeserver does not support Directory feature, skipping");
-                } else {
+                    } else {
-                    log.error("Homeserver returned an error while performing directory search");
+                        log.error("Homeserver returned an error while performing directory search");
-                    throw new MatrixException(status, info.getErrcode(), info.getError());
+                        throw new MatrixException(status, info.getErrcode(), info.getError());
+                    }
                 }
-            }
 
-            UserDirectorySearchResult resultHs = gson.fromJson(body, UserDirectorySearchResult.class);
+                UserDirectorySearchResult resultHs = gson.fromJson(body, UserDirectorySearchResult.class);
-            log.info("Found {} match(es) in HS for '{}'", resultHs.getResults().size(), query);
+                log.info("Found {} match(es) in HS for '{}'", resultHs.getResults().size(), query);
-            result.getResults().addAll(resultHs.getResults());
+                result.getResults().addAll(resultHs.getResults());
-            if (resultHs.isLimited()) {
+                if (resultHs.isLimited()) {
-                result.setLimited(true);
+                    result.setLimited(true);
+                }
+            } catch (JsonSyntaxException e) {
+                throw new InternalServerError("Invalid JSON reply from the HS: " + e.getMessage());
+            } catch (IOException e) {
+                throw new InternalServerError("Unable to query the HS: I/O error: " + e.getMessage());
             }
-        } catch (JsonSyntaxException e) {
-            throw new InternalServerError("Invalid JSON reply from the HS: " + e.getMessage());
-        } catch (IOException e) {
-            throw new InternalServerError("Unable to query the HS: I/O error: " + e.getMessage());
         }
 
         for (IDirectoryProvider provider : providers) {

src/main/java/io/kamax/mxisd/lookup/strategy/RecursivePriorityLookupStrategy.java

@@ -53,7 +53,10 @@ public class RecursivePriorityLookupStrategy implements LookupStrategy {
     public RecursivePriorityLookupStrategy(RecursiveLookupConfig cfg, List<IThreePidProvider> providers, IBridgeFetcher bridge) {
         this.cfg = cfg;
         this.bridge = bridge;
-        this.providers = providers.stream().filter(IThreePidProvider::isEnabled).collect(Collectors.toList());
+        this.providers = providers.stream().filter(p -> {
+            log.info("3PID Provider {} is enabled: {}", p.getClass().getSimpleName(), p.isEnabled());
+            return p.isEnabled();
+        }).collect(Collectors.toList());
     }
 
     @PostConstruct

src/main/java/io/kamax/mxisd/notification/NotificationManager.java

@@ -46,16 +46,14 @@ public class NotificationManager {
         this.handlers = new HashMap<>();
         handlers.forEach(h -> {
             log.info("Found handler {} for medium {}", h.getId(), h.getMedium());
-            String handlerId = cfg.getHandler().get(h.getMedium());
+            String handlerId = cfg.getHandler().getOrDefault(h.getMedium(), "raw");
-            if (StringUtils.isBlank(handlerId) || StringUtils.equals(handlerId, h.getId())) {
+            if (StringUtils.equals(handlerId, h.getId())) {
                 this.handlers.put(h.getMedium(), h);
             }
         });
 
         log.info("--- Notification handler ---");
-        this.handlers.forEach((k, v) -> {
+        this.handlers.forEach((k, v) -> log.info("\tHandler for {}: {}", k, v.getId()));
-            log.info("\tHandler for {}: {}", k, v.getId());
-        });
     }
 
     private INotificationHandler ensureMedium(String medium) {

src/main/resources/application.yaml

@@ -227,10 +227,14 @@ view:
 storage:
   backend: 'sqlite'
 
+directory:
+  exclude:
+    homeserver: false
+
 ---
 spring:
   profiles: systemd
 
 logging:
   pattern:
-    console: '%d{.SSS}${LOG_LEVEL_PATTERN:%5p} [%15.15t] %35.35logger{34} : %m%n${LOG_EXCEPTION_CONVERSION_WORD:%wEx}'
+    console: '%d{.SSS} ${LOG_LEVEL_PATTERN:%5p} [%15.15t] %35.35logger{34} : %m%n${LOG_EXCEPTION_CONVERSION_WORD:%wEx}'