Support for profile auto-fill with LDAP

README.md

@@ -5,14 +5,14 @@ mxisd - Federated Matrix Identity Server Daemon
 - [Overview](#overview)
 - [Features](#features)
 - [Why use mxisd](#why-use-mxisd)
-- [Quick start](#quick-start)
+- [Getting Started](#getting-started)
 - [Support](#support)
 - [Contribute](#contribute)
 - [FAQ](#faq)
 - [Contact](#contact)
 
 # Overview
-mxisd is a Federated Matrix Identity server for self-hosted Matrix infrastructures with enhanced features.
+mxisd is a Federated Matrix Identity server for self-hosted Matrix infrastructures with [enhanced features](#features).
   
 It is specifically designed to connect to an Identity store (AD/Samba/LDAP, SQL Database, Web services/application, ...)
 and ease the integration of the Matrix ecosystem with an existing infrastructure, or to build a new one using lasting
@@ -29,8 +29,9 @@ users. 3PIDs can be anything that identify a user, like:
 - Facebook ID
 - ...
 
-mxisd is an enhanced Identity service, which implements the [Matrix Identity service API](https://matrix.org/docs/spec/identity_service/unstable.html)
+mxisd is an enhanced Identity service, which implements the
-but also several other features that greatly enhance user experience within Matrix.
+[Matrix Identity service API](https://matrix.org/docs/spec/identity_service/unstable.html) but also several
+[other features](#features) that greatly enhance user experience within Matrix.
 
 mxisd is the one stop shop for anything regarding Authentication, Directory and Identity management in Matrix built as a
 single coherent product.
@@ -64,114 +65,20 @@ currently **cannot be removed**
 - Users can directly find each other using whatever attribute is relevant within your Identity store
 - Federate your Identity lookups so you can discover others and/or others can discover you, all with extensive ACLs
 
-# Quick Start
+# Getting started
-1. [Preparation](#preparation)
+See the [dedicated document](docs/getting-started.md)
-2. [Install](#install)
-3. [Configure](#configure)
-4. [Integrate](#integrate)
-5. [Validate](#validate)
-
-Following these quick start instructions, you will have a basic setup that can perform recursive/federated lookups and
-talk to the central Matrix.org Identity service.  
-This will be a good ground work for further integration with your existing Identity stores.
-
-## Preparation
-You will need:
-- Homeserver
-- Reverse proxy with regular TLS/SSL certificate (Let's encrypt) for your mxisd domain
-
-As synapse requires an HTTPS connection when talking to an Identity service, a reverse proxy is required as mxisd does
-not support HTTPS listener at this time.
-
-For maximum integration, it is best to have your Homeserver and mxisd reachable via the same hostname.  
-You can also use a dedicated domain for mxisd, but will not have access to some features.
-
-Be aware of a [NAT/Reverse proxy gotcha](https://github.com/kamax-io/mxisd/wiki/Gotchas#nating) if you use the same
-hostname.
-
-The following Quick Start guide assumes you will host the Homeserver and mxisd under the same hostname.  
-If you would like a high-level view of the infrastructure and how each feature is integrated, see the
-[dedicated document](docs/architecture.md)
-
-## Install
-Install via:
-- [Debian package](docs/install/debian.md)
-- [Docker image](docs/install/docker.md)
-- [Sources](docs/build.md)
-
-See the [Latest release](https://github.com/kamax-io/mxisd/releases/latest) for links to each.
-
-## Configure
-Create/edit a minimal configuration (see installer doc for the location):
-```
-matrix.domain: 'MyMatrixDomain.org'
-key.path: '/path/to/signing.key.file'
-storage.provider.sqlite.database: '/path/to/mxisd.db'
-```  
-- `matrix.domain` should be set to your Homeserver domain
-- `key.path` will store the signing keys, which must be kept safe!
-- `storage.provider.sqlite.database` is the location of the SQLite Database file which will hold state (invites, etc.)
-
-If your HS/mxisd hostname is not the same as your Matrix domain, configure `server.name`.  
-Complete configuration guide is available [here](docs/configure.md).
-
-## Integrate
-For an overview of a typical mxisd infrastructure, see the [dedicated document](docs/architecture.md)
-### Reverse proxy
-#### Apache2
-In the VirtualHost handling the domain with SSL, add the following line and replace `0.0.0.0` by the right address/host.  
-**This line MUST be present before the one for the homeserver!**
-```
-ProxyPass /_matrix/identity http://0.0.0.0:8090/_matrix/identity
-```
-
-Typical VirtualHost configuration would be:
-```
-<VirtualHost *:443>
-    ServerName example.org
-    
-    ...
-    
-    ProxyPreserveHost on
-    ProxyPass /_matrix/identity/ http://10.1.2.3:8090/_matrix/identity/
-    ProxyPass /_matrix/ http://10.1.2.3:8008/_matrix/
-</VirtualHost>
-```
-
-### Synapse
-Add your mxisd domain into the `homeserver.yaml` at `trusted_third_party_id_servers` and restart synapse.  
-In a typical configuration, you would end up with something similair to:
-```
-trusted_third_party_id_servers:
-    - matrix.org
-    - vector.im
-    - example.org
-```
-It is recommended to remove `matrix.org` and `vector.im` so only your own Identity server is allowed by synapse. 
-
-### Federation and network discovery
-See the [dedicated document](docs/features/federation.md).
-
-## Validate
-Log in using your Matrix client and set `https://example.org` as your Identity server URL, replacing `example.org` by
-the relevant hostname which you configured in your reverse proxy.
-Try to invite `mxisd-lookup-test@kamax.io`, which should be turned into a Matrix invite to `@mxisd-lookup-test:kamax.io`
-  
-If it worked, it means you are up and running and can enjoy mxisd in its basic mode! Congratulations!  
-If it did not work, [get in touch](#support) and we'll do our best to get you started.  
-
-You can now integrate mxisd further with your infrastructure using the various [features](docs/README.md) guides.
 
 # Support
 ## Community
 If you need help, want to report a bug or just say hi, you can reach us on Matrix at 
-[#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io) or [directly peek anonymously](https://view.matrix.org/room/!NPRUEisLjcaMtHIzDr:kamax.io/).
+[#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io) or
+[directly peek anonymously](https://view.matrix.org/room/!NPRUEisLjcaMtHIzDr:kamax.io/).
 For more high-level discussion about the Identity Server architecture/API, go to 
 [#matrix-identity:matrix.org](https://matrix.to/#/#matrix-identity:matrix.org)
 
 ## Professional
-If you would prefer professional support/custom development for mxisd and/or for Matrix in general, including other open source technologies/products, 
+If you would prefer professional support/custom development for mxisd and/or for Matrix in general, including other open
-please visit [our website](https://www.kamax.io/) to get in touch with us and get a quote.
+source technologies/products, please visit [our website](https://www.kamax.io/) to get in touch with us and get a quote.
 
 We offer affordable monthly/yearly support plans for mxisd, synapse or your full Matrix infrastructure.
 
@@ -184,8 +91,8 @@ You can contribute as a community member by:
 - Helping us improve the documentation: tell us what is good or not good (in an issue or in Matrix), or make a PR with
 changes you feel improve the doc.
 - Contribute code directly: we love contributors! All your contributions will be licensed under AGPLv3.
-- Donate! any donation is welcome, regardless how small or big. This will directly be used for the fixed costs and
+- [Donate!](https://liberapay.com/maximusdor/) Any donation is welcome, regardless how small or big, and will directly
-developer time.
+be used for the fixed costs and developer time of mxisd.
 
 You can contribute as an organisation/corporation by:
 - Get a [support contract](#support-professional). This is the best way you can help us as it ensures mxisd is
@@ -193,64 +100,7 @@ maintained regularly and you get direct access to the support team.
 - Sponsoring new features or bug fixes. [Get in touch](#contact) so we can discuss it further.
 
 # FAQ
-### Do I need to use mxisd if I run a Homeserver?
+See the [dedicated document](docs/faq.md)
-No, but it is recommended, even if you don't use any backends or integration.
-
-mxisd in its default configuration will use federation and involve the central Matrix.org Identity servers when
-performing queries, giving you access to at least the same information as if you were not running it.
-
-It will also give your users a choice to make their 3PIDs available publicly, ensuring they are made aware of the
-privacy consequences, which is not the case with the central Matrix.org servers.
-
-So mxisd is like your gatekeeper and guardian angel. It does not change what you already know, just adds some nice
-simple features on top of it.
-
-### I already use the synapse LDAP3 auth provider, why should I care about mxisd?
-The [synapse LDAP3 auth provider](https://github.com/matrix-org/matrix-synapse-ldap3) only handles on specific flow:
-validate credentials at login.
-
-It does not:
-- Auto-provision user profiles
-- Integrate with Identity management
-- Integrate with Directory searches
-- Protect you against the username case sensitivites issues in synapse
-
-mxisd is a replacement and enhancement of it, and also offers coherent results in all areas, which LDAP3 auth provider
-does not.
-
-### I saw that sydent is the official Identity server implemenation of the Matrix team, I should use that!
-You can, but [sydent](https://github.com/matrix-org/sydent):
-- [should not be used and/or self-hosted](https://github.com/matrix-org/sydent/issues/22)
-- is not meant to be linked to a specific Homeserver / domain
-- cannot handle federation or proxy lookups, effectively isolating your users from the rest of the network
-- forces you to duplicate all your identity data, so people can be found by 3PIDs
-- forces users to enter all their emails and phone numbers manually in their profile
-
-So really, you should go with mxisd.
-
-### I'm not sure I understand what an "Identity server" is supposed to be or do
-The current Identity service API is more a placeholder, as the Matrix devs did not have time so far to really work on
-what they want to do with that part of the ecosystem. Therefore, "Identity" is a misleading word currently.  
-Given the scope of the current Identity Service API, it would be best called "Invitation service".
-
-Because the current scope is so limited and no integration is done with the Homeserver, there was a big lack of features
-for groups/corporations/organisation. This is where mxisd comes in.
-
-mxisd implements the Identity Service API and also a set of features which are expected by regular users, truly living
-up to its "Identity server" name.
-
-### So mxisd is just a big hack! I don't want to use non-official features!
-mxisd primary concern is to always be compatible with the Matrix ecosystem and the Identity service API.  
-Whenever the API will be updated and/or enhanced, mxisd will follow, remaining 100% compatible with the ecosystem.
-
-We also directly talk with the Matrix developers to ensure all features we implement have their approval, and that we
-are in line with their vision of Identity management within the Matrix ecosystem.
-
-Therefore, using mxisd is a safe choice. It will be like using the central Matrix.org Identity servers, yet not closing
-the door to very nice enhancements and integrations.
-
-### Should I use mxisd if I don't host my own Homeserver?
-No
 
 # Contact
 Get in touch via:

docs/backends/README.md

@@ -0,0 +1,5 @@
+# Identity Stores (Backends)
+- [Samba / Active Directory / LDAP](ldap.md)
+- [SQL Databases](sql.md)
+- [Website / Web service / Web app](rest.md)
+- [Google Firebase](firebase.md)

docs/backends/firebase.md

@@ -1,4 +1,14 @@
 # Google Firebase
+https://firebase.google.com/
+
+## Requirements
+This backend requires a suitable Matrix client capable of performing Firebase authentication and passing the following
+information:
+- Firebase User ID as Matrix username
+- Firebase token as Matrix password
+
+If your client is Riot, you will need a custom version.
+
 ## Configuration
 To be completed. For now, see default structure and values:
 ```
@@ -6,4 +16,4 @@ firebase:
   enabled: false
   credentials: '/path/to/firebase/credentials.json'
   database: 'https://my-project.firebaseio.com/'
-```
+```

docs/backends/ldap.md

@@ -1,53 +1,97 @@
-# AD/Samba/LDAP backend
+# LDAP (Samba / Active Directory / OpenLDAP)
+## Getting started
+To use your LDAP backend, add the bare minimum configuration in mxisd config file:
+```
+ldap.enabled: true
+ldap.connection.host: 'ldapHostnameOrIp'
+ldap.connection.bindDn: 'CN=My Mxisd User,OU=Users,DC=example,DC=org'
+ldap.connection.bindPassword: 'TheUserPassword'
+ldap.connection.baseDn: 'OU=Users,DC=example,DC=org'
+```
+These are standard LDAP connection configuration. mxisd will try to connect on port default port 389 without encryption.
+
+---
+
+If you would like to use a TLS/SSL connection, use the following configuration options (STARTLS not supported):
+```
+ldap.connection.tls: true
+ldap.connection.port: 12345
+```
+
+---
+
+You can also set a default global filter on any LDAP queries:
+```
+ldap.filter: '(memberOf=CN=My Matrix Users,OU=Groups,DC=example,DC=org)'
+```
+This example would only return users part of the group called `My Matrix Users`.
+This can be overwritten or append in each specific flow describe below.
+
+---
+
+LDAP features are based on mapping LDAP attributes to Matrix concepts, like a Matrix ID, its localpart, the user display
+name, their email(s) and/or phone number(s).
+     
+Default attributes are well suited for Active Directory/Samba. In case you are using a native LDAP backend, you will
+most certainly configure those mappings.
+
+The following example would set the `uid` attribute as localpart and the Matrix display name to `cn`
+```
+ldap.attribute.uid.type: 'uid'
+ldap.attribute.uid.value: 'uid'
+ldap.attribute.name: 'cn'
+```
+
+You can also change the attribute lists for 3PID, like email or phone numbers.  
+The following example would overwrite the [default list of attributes](../../src/main/resources/application.yaml#L67) for emails and phone number:
+```
+ldap.attribute.threepid.email:
+  - 'mail'
+  - 'otherMailAttribute'
+
+ldap.attribute.threepid.msisdn:
+  - 'phone'
+  - 'otherPhoneAttribute'
+```
+
+## Identity
+Identity features (related to 3PID invites or searches) are enabled and configured using default values and no specific
+configuration item is needed to get started.
+
+If you would like to overwrite some global configuration relative to filter and/or attributes, see the Identity section
+of the Configuration below.
+
+## Authentication
+No further configuration is needed to enable authentication with LDAP once globally enabled and configured.  
+You have the possiblity to use a different query filter if you wish, see Configuration below.
+
+Profile auto-fill is not yet supported but is a top priority.
+
+## Directory
+No further configuration is needed to enable directory with LDAP once globally enabled and configured.
+
+If you would like to use extra attributes in search that are not 3PIDs, like nicknames, group names, employee number:
+```
+ldap.directory.attribute.other:
+  - 'myNicknameAttribute'
+  - 'memberOf'
+  - 'employeeNumberAttribute'
+```
+
 ## Configuration
-### Structure and default values
+Please read the [Configuration](../configure.md) explanatory note if you are not familiar with the terms used below.
-```
+ 
-ldap:
-  enabled: false
-  filter: ''
-  connection:
-    host: ''
-    tls: false
-    port: 389
-    bindDn: ''
-    bindPassword: ''
-    baseDn: ''
-  attribute:
-    uid:
-      type: 'uid'
-      value: 'userPrincipalName'
-    name: 'displayName'
-    threepid:
-      email:
-        - 'mailPrimaryAddress'
-        - 'mail'
-        - 'otherMailbox'
-      msisdn:
-        - 'telephoneNumber'
-        - 'mobile'
-        - 'homePhone'
-        - 'otherTelephone'
-        - 'otherMobile'
-        - 'otherHomePhone'
-  auth:
-    filter: ''
-  directory:
-    attribute:
-      other: []
-    filter: ''
-  identity:
-    filter: ''
-    medium:
-      email: ''
-      msisdn: ''
-```
 ### General
+Base path: `ldap`
+
 | Item      | Description                                                                               |
 |-----------|-------------------------------------------------------------------------------------------|
 | `enabled` | Globaly enable/disable the LDAP backend                                                   |
 | `filter`  | Global filter to apply on all LDAP queries. Can be overwritten in each applicable section |
 
 ### Connection
+Base path: `ldap.connection`
+
 | Item           | Description                                          |
 |----------------|------------------------------------------------------|
 | `host`         | Host to connect to                                   |
@@ -58,6 +102,8 @@ ldap:
 | `baseDn`       | Base DN for queries                                  |
 
 ### Attributes
+Base path: `ldap.attribute`
+
 | Item        | Description                                                                                                            |
 |-------------|------------------------------------------------------------------------------------------------------------------------|
 | `uid.type`  | Indicate how to process the User ID (UID) attribute:                                                                   |
@@ -68,11 +114,15 @@ ldap:
 | `threepid`  | Namespace where each key is a 3PID type and contains a list of attributes                                              |
 
 ### Authentication
+Base path: `ldap.auth`
+
 | Item     | Description                                                                                      |
 |----------|--------------------------------------------------------------------------------------------------|
 | `filter` | Specific user filter applied during authentication. Global filter is used if empty/blank/not set |
 
 ### Directory
+Base path: `ldap.directory`
+
 | Item              | Description                                                         |
 |-------------------|---------------------------------------------------------------------|
 | `attribute.other` | Additional attributes to be used when performing directory searches |
@@ -80,6 +130,8 @@ ldap:
 |                   | Global filter is used if empty/blank/not set                        |
 
 ### Identity
+Base path: `ldap.identity`
+
 | Item     | Description                                                                                       |
 |----------|---------------------------------------------------------------------------------------------------|
 | `filter` | Specific user filter applied during identity search. Global filter is used if empty/blank/not set | 

docs/configure.md

@@ -37,7 +37,7 @@ server:
 **WARNING:** mxisd might overwrite/adapt some values during launch. Those changes will not be reflected into copied keys.
 
 ## Categories
-For each category below, the base configuration path will be given, which needs to be appened to every configuration
+For each category below, the base configuration path will be given, which needs to be appended to every configuration
 item described.
 
 Example: if the base path was `basePath` and the following table was given:

docs/faq.md

@@ -0,0 +1,65 @@
+# FAQ
+### Do I need to use mxisd if I run a Homeserver?
+No, but it is recommended, even if you don't use any backends or integration.
+
+In its default configuration, mxisd will talk to the central Matrix Identity servers and use other federated public
+servers when performing queries, giving you access to at least the same information as if you were not running it.
+
+It will also give your users a choice to make their 3PIDs available publicly, ensuring they are made aware of the
+privacy consequences, which is not the case with the central Matrix.org servers.
+
+So mxisd is like your gatekeeper and guardian angel. It does not change what you already know, just adds some nice
+simple features on top of it.
+
+### I already use the synapse LDAP3 auth provider, why should I care about mxisd?
+The [synapse LDAP3 auth provider](https://github.com/matrix-org/matrix-synapse-ldap3) is not longer maintained and
+only handles on specific flow: validate credentials at login.
+
+It does not:
+- Auto-provision user profiles
+- Integrate with Identity management
+- Integrate with Directory searches
+- Protect you against the username case sensitivites issues in synapse
+
+mxisd is a replacement and enhancement of it, offering coherent results in all areas, which LDAP3 auth provider
+does not.
+
+### Sydent is the official Identity server implementation of the Matrix team, why not use that?
+You can, but [sydent](https://github.com/matrix-org/sydent):
+- [should not be used and/or self-hosted](https://github.com/matrix-org/sydent/issues/22)
+- is not meant to be linked to a specific Homeserver / domain
+- cannot handle federation or proxy lookups, effectively isolating your users from the rest of the network
+- forces you to duplicate all your identity data, so people can be found by 3PIDs
+- forces users to enter all their emails and phone numbers manually in their profile
+
+So really, you should go with mxisd.
+
+### Will I loose access to the central Matrix.org/Vector.im Identity data if I use mxisd?
+In its default configuration, mxisd act as a proxy to Matrix.org/Vector.im. You will have access to the same data and
+behaviour than if you were using them directly. There is no downside in using mxisd with the default configuration.
+
+mxisd can also be configured not to talk to the central Identity servers if you wish.
+
+### I'm not sure I understand what an "Identity server" is supposed to be or do
+The current Identity service API is more a placeholder, as the Matrix devs did not have time so far to really work on
+what they want to do with that part of the ecosystem. Therefore, "Identity" is a misleading word currently.  
+Given the scope of the current Identity Service API, it would be best called "Invitation service".
+
+Because the current scope is so limited and no integration is done with the Homeserver, there was a big lack of features
+for groups/corporations/organisation. This is where mxisd comes in.
+
+mxisd implements the Identity Service API and also a set of features which are expected by regular users, truly living
+up to its "Identity server" name.
+
+### So mxisd is just a big hack! I don't want to use non-official features!
+mxisd primary concern is to always be compatible with the Matrix ecosystem and the Identity service API.  
+Whenever the API will be updated and/or enhanced, mxisd will follow, remaining 100% compatible with the ecosystem.
+
+We also directly talk with the Matrix developers to ensure all features we implement have their approval, and that we
+are in line with their vision of Identity management within the Matrix ecosystem.
+
+Therefore, using mxisd is a safe choice. It will be like using the central Matrix.org Identity servers, yet not closing
+the door to very nice enhancements and integrations.
+
+### Should I use mxisd if I don't host my own Homeserver?
+No

docs/features/authentication.md

@@ -1,6 +1,7 @@
 # Authentication
-Performed via [synapse with REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth/blob/master/README.md)  
+Authentication is an enchanced Identity feature of mxisd to ensure coherent and centralized identity management.
-Point the `endpoint` to mxisd internal IP on port 8090
+
+It allows to use Identity stores configured in mxisd to authenticate users on your Homeserver.
 
 ## Overview
 ```
@@ -23,6 +24,26 @@ Point the `endpoint` to mxisd internal IP on port 8090
               |   user profiles          |    If valid credentials and supported by backend
               +--------------------------+
 ```
+Performed on [synapse with REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth/blob/master/README.md)
+
+## Getting started
+### Synapse
+You will need:
+- Configure and enable at least one [Identity store](../backends/)
+- Install the [REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth)
+
+Once installed, edit your synapse configuration as described for the auth module:
+- Set `endpoint` to `http://mxisdAddress:8090` - Replace `mxisdAddress` to an internal IP/Hostname.
+- If you want to avoid [known issues](https://github.com/matrix-org/matrix-doc/issues/586) with lower/upper case
+usernames, set `enforceLowercase` in the REST config to `true`.
+
+**IMPORTANT**: if this is a new installation, it is highly recommended to enforce lowercase, as it is not possible to
+workaround the bug at a later date and will cause issues with invites, searches, authentication.
+
+Restart synapse and login on the Homeserver using credentials present in your backend.  
 
 ## Profile auto-fill
-To be documented
+Auto-filling user profile depends on two conditions:
+- The REST auth module is configured for it, which is the case by default
+- Your Identity store is configured to provide profile data. See your Identity store [documentation](../backends/) on
+how to enable the feature.

docs/getting-started.md

@@ -0,0 +1,110 @@
+# Getting started
+1. [Preparation](#preparation)
+2. [Install](#install)
+3. [Configure](#configure)
+4. [Integrate](#integrate)
+5. [Validate](#validate)
+6. [Next steps](#next-steps)
+
+Following these quick start instructions, you will have a basic setup that can perform recursive/federated lookups and
+talk to the central Matrix.org Identity service.  
+This will be a good ground work for further integration with your existing Identity stores.
+
+## Preparation
+You will need:
+- Homeserver
+- Reverse proxy with regular TLS/SSL certificate (Let's encrypt) for your mxisd domain
+
+As synapse requires an HTTPS connection when talking to an Identity service, a reverse proxy is required as mxisd does
+not support HTTPS listener at this time.
+
+For maximum integration, it is best to have your Homeserver and mxisd reachable via the same hostname.  
+You can also use a dedicated domain for mxisd, but will not have access to some features.
+
+Be aware of a [NAT/Reverse proxy gotcha](https://github.com/kamax-io/mxisd/wiki/Gotchas#nating) if you use the same
+hostname.
+
+The following Quick Start guide assumes you will host the Homeserver and mxisd under the same hostname.  
+If you would like a high-level view of the infrastructure and how each feature is integrated, see the
+[dedicated document](architecture.md)
+
+## Install
+Install via:
+- [Debian package](install/debian.md)
+- [Docker image](install/docker.md)
+- [Sources](build.md)
+
+See the [Latest release](https://github.com/kamax-io/mxisd/releases/latest) for links to each.
+
+## Configure
+Create/edit a minimal configuration (see installer doc for the location):
+```
+matrix.domain: 'MyMatrixDomain.org'
+key.path: '/path/to/signing.key.file'
+storage.provider.sqlite.database: '/path/to/mxisd.db'
+```  
+- `matrix.domain` should be set to your Homeserver domain
+- `key.path` will store the signing keys, which must be kept safe!
+- `storage.provider.sqlite.database` is the location of the SQLite Database file which will hold state (invites, etc.)
+
+If your HS/mxisd hostname is not the same as your Matrix domain, configure `server.name`.  
+Complete configuration guide is available [here](configure.md).
+
+## Integrate
+For an overview of a typical mxisd infrastructure, see the [dedicated document](architecture.md)
+### Reverse proxy
+#### Apache2
+In the VirtualHost handling the domain with SSL, add the following line and replace `0.0.0.0` by the right address/host.  
+**This line MUST be present before the one for the homeserver!**
+```
+ProxyPass /_matrix/identity/ http://0.0.0.0:8090/_matrix/identity/
+```
+
+Typical VirtualHost configuration would be:
+```
+<VirtualHost *:443>
+    ServerName example.org
+    
+    ...
+    
+    ProxyPreserveHost on
+    ProxyPass /_matrix/identity/ http://10.1.2.3:8090/_matrix/identity/
+    ProxyPass /_matrix/ http://10.1.2.3:8008/_matrix/
+</VirtualHost>
+```
+
+### Synapse
+Add your mxisd domain into the `homeserver.yaml` at `trusted_third_party_id_servers` and restart synapse.  
+In a typical configuration, you would end up with something similair to:
+```
+trusted_third_party_id_servers:
+    - matrix.org
+    - vector.im
+    - example.org
+```
+It is recommended to remove `matrix.org` and `vector.im` so only your own Identity server is allowed by synapse. 
+
+## Validate
+Log in using your Matrix client and set `https://example.org` as your Identity server URL, replacing `example.org` by
+the relevant hostname which you configured in your reverse proxy.  
+Invite `mxisd-lookup-test@kamax.io` to a room, which should be turned into a Matrix invite to `@mxisd-lookup-test:kamax.io`.  
+**NOTE:** you might not see a Matrix suggestion for the e-mail address, which is normal. Still proceed with the invite.
+  
+If it worked, it means you are up and running and can enjoy mxisd in its basic mode! Congratulations!  
+If it did not work, [get in touch](#support) and we'll do our best to get you started.
+
+You can now integrate mxisd further with your infrastructure using the various [features](README.md) guides.
+
+## Next steps
+Once your mxisd server is up and running, here are the next steps to further enhance and integrate your installation:
+
+Enable extra features:
+- [Federation](features/federation.md)
+- [Authenticate with synapse](features/authentication.md), profile auto-provisioning if you wish
+- [Directory search](features/directory-users.md)
+
+Use your Identity stores:
+- [LDAP / Samba / Active directory](backends/ldap.md)
+- [SQL Database](backends/sql.md)
+- [Website / Web service / Web app](backends/rest.md)
+- [Google Firebase](backends/firebase.md)

src/main/java/io/kamax/mxisd/backend/ldap/LdapAuthProvider.java

@@ -20,7 +20,11 @@
 
 package io.kamax.mxisd.backend.ldap;
 
+import com.google.i18n.phonenumbers.NumberParseException;
+import com.google.i18n.phonenumbers.PhoneNumberUtil;
+import io.kamax.matrix.ThreePidMedium;
 import io.kamax.matrix._MatrixID;
+import io.kamax.mxisd.ThreePid;
 import io.kamax.mxisd.UserIdType;
 import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
 import io.kamax.mxisd.auth.provider.BackendAuthResult;
@@ -41,12 +45,17 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
 import java.io.IOException;
+import java.util.HashSet;
+import java.util.Optional;
+import java.util.Set;
 
 @Component
 public class LdapAuthProvider extends LdapGenericBackend implements AuthenticatorProvider {
 
     private Logger log = LoggerFactory.getLogger(LdapAuthProvider.class);
 
+    private PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
+
     @Autowired
     public LdapAuthProvider(LdapConfig cfg, MatrixConfig mxCfg) {
         super(cfg, mxCfg);
@@ -57,6 +66,21 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
         return getCfg().isEnabled();
     }
 
+    private Optional<String> getMsisdn(String phoneNumber) {
+        try { // FIXME export into dedicated ThreePid class within SDK (copy from Firebase Auth)
+            return Optional.of(phoneUtil.format(
+                    phoneUtil.parse(
+                            phoneNumber,
+                            null // No default region
+                    ),
+                    PhoneNumberUtil.PhoneNumberFormat.E164
+            ).substring(1)); // We want without the leading +
+        } catch (NumberParseException e) {
+            log.warn("Invalid phone number: {}", phoneNumber);
+            return Optional.empty();
+        }
+    }
+
     @Override
     public BackendAuthResult authenticate(_MatrixID mxid, String password) {
         log.info("Performing auth for {}", mxid);
@@ -74,7 +98,15 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
 
             String userFilter = "(" + getUidAtt() + "=" + userFilterValue + ")";
             userFilter = buildWithFilter(userFilter, getCfg().getAuth().getFilter());
-            try (EntryCursor cursor = conn.search(getBaseDn(), userFilter, SearchScope.SUBTREE, getUidAtt(), getAt().getName())) {
+
+            Set<String> attributes = new HashSet<>();
+            attributes.add(getUidAtt());
+            attributes.add(getAt().getName());
+            getAt().getThreepid().forEach((k, v) -> attributes.addAll(v));
+            String[] attArray = new String[attributes.size()];
+            attributes.toArray(attArray);
+
+            try (EntryCursor cursor = conn.search(getBaseDn(), userFilter, SearchScope.SUBTREE, attArray)) {
                 while (cursor.next()) {
                     Entry entry = cursor.get();
                     String dn = entry.getDn().getName();
@@ -99,7 +131,18 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
                     log.info("DN {} is a valid match", dn);
 
                     // TODO should we canonicalize the MXID?
-                    return BackendAuthResult.success(mxid.getId(), UserIdType.MatrixID, name);
+                    BackendAuthResult result = BackendAuthResult.success(mxid.getId(), UserIdType.MatrixID, name);
+                    log.info("Processing 3PIDs for profile");
+                    getAt().getThreepid().forEach((k, v) -> v.forEach(attId -> {
+                        getAttribute(entry, attId).ifPresent(tpidValue -> {
+                            if (ThreePidMedium.PhoneNumber.is(k)) {
+                                tpidValue = getMsisdn(tpidValue).orElse(tpidValue);
+                            }
+                            result.withThreePid(new ThreePid(k, tpidValue));
+                        });
+                    }));
+                    log.info("Found {} 3PIDs", result.getProfile().getThreePids().size());
+                    return result;
                 }
             } catch (CursorLdapReferralException e) {
                 log.warn("Entity for {} is only available via referral, skipping", mxid);

src/main/java/io/kamax/mxisd/controller/DefaultExceptionHandler.java

@@ -55,20 +55,29 @@ public class DefaultExceptionHandler {
     }
 
     @ExceptionHandler(InternalServerError.class)
-    public String handle(HttpServletRequest req, InternalServerError e, HttpServletResponse response) {
+    public String handle(HttpServletRequest request, HttpServletResponse response, InternalServerError e) {
         if (StringUtils.isNotBlank(e.getInternalReason())) {
             log.error("Reference #{} - {}", e.getReference(), e.getInternalReason());
         } else {
             log.error("Reference #{}", e);
         }
 
-        return handleGeneric(req, e, response);
+        return handleGeneric(request, response, e);
+    }
+
+    @ExceptionHandler(FeatureNotAvailable.class)
+    public String handle(HttpServletRequest request, HttpServletResponse response, FeatureNotAvailable e) {
+        if (StringUtils.isNotBlank(e.getInternalReason())) {
+            log.error("Feature not available: {}", e.getInternalReason());
+        }
+
+        return handleGeneric(request, response, e);
     }
 
     @ExceptionHandler(MatrixException.class)
-    public String handleGeneric(HttpServletRequest req, MatrixException e, HttpServletResponse response) {
+    public String handleGeneric(HttpServletRequest request, HttpServletResponse response, MatrixException e) {
         response.setStatus(e.getStatus());
-        return handle(req, e.getErrorCode(), e.getError());
+        return handle(request, e.getErrorCode(), e.getError());
     }
 
     @ResponseStatus(HttpStatus.BAD_REQUEST)

src/main/java/io/kamax/mxisd/controller/identity/v1/SessionController.java

@@ -23,7 +23,6 @@ package io.kamax.mxisd.controller.identity.v1;
 import io.kamax.mxisd.config.ServerConfig;
 import io.kamax.mxisd.config.ViewConfig;
 import io.kamax.mxisd.controller.identity.v1.remote.RemoteIdentityAPIv1;
-import io.kamax.mxisd.exception.InternalServerError;
 import io.kamax.mxisd.session.SessionMananger;
 import io.kamax.mxisd.session.ValidationResult;
 import org.slf4j.Logger;
@@ -36,7 +35,8 @@ import org.springframework.web.bind.annotation.RequestParam;
 
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
-import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
 
 import static org.springframework.web.bind.annotation.RequestMethod.GET;
 
@@ -69,15 +69,15 @@ class SessionController {
         ValidationResult r = mgr.validate(sid, secret, token);
         log.info("Session {} was validated", sid);
         if (r.getNextUrl().isPresent()) {
-            String url = srvCfg.getPublicUrl() + r.getNextUrl().get();
+            String url = r.getNextUrl().get();
-            log.info("Session {} validation: next URL is present, redirecting to {}", sid, url);
             try {
-                response.sendRedirect(url);
+                url = new URL(url).toString();
-                return "";
+            } catch (MalformedURLException e) {
-            } catch (IOException e) {
+                log.info("Session next URL {} is not a valid one, will prepend public URL {}", url, srvCfg.getPublicUrl());
-                log.warn("Unable to redirect user to {}", url);
+                url = srvCfg.getPublicUrl() + r.getNextUrl().get();
-                throw new InternalServerError(e);
             }
+            log.info("Session {} validation: next URL is present, redirecting to {}", sid, url);
+            return "redirect:" + url;
         } else {
             if (r.isCanRemote()) {
                 String url = srvCfg.getPublicUrl() + RemoteIdentityAPIv1.getRequestToken(r.getSession().getId(), r.getSession().getSecret());

src/main/java/io/kamax/mxisd/directory/DirectoryManager.java

@@ -31,6 +31,7 @@ import io.kamax.mxisd.exception.MatrixException;
 import io.kamax.mxisd.util.GsonUtil;
 import io.kamax.mxisd.util.RestClientUtils;
 import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang.StringUtils;
 import org.apache.http.client.methods.CloseableHttpResponse;
 import org.apache.http.client.methods.HttpPost;
 import org.apache.http.client.utils.URIBuilder;
@@ -88,7 +89,12 @@ public class DirectoryManager {
 
             if (status != 200) {
                 MatrixErrorInfo info = gson.fromJson(body, MatrixErrorInfo.class);
-                throw new MatrixException(status, info.getErrcode(), info.getError());
+                if (StringUtils.equals("M_UNRECOGNIZED", info.getErrcode())) { // FIXME no hardcoding, use Enum
+                    log.warn("Homeserver does not support Directory feature, skipping");
+                } else {
+                    log.error("Homeserver returned an error while performing directory search");
+                    throw new MatrixException(status, info.getErrcode(), info.getError());
+                }
             }
 
             UserDirectorySearchResult resultHs = gson.fromJson(body, UserDirectorySearchResult.class);

src/main/java/io/kamax/mxisd/exception/FeatureNotAvailable.java

@@ -0,0 +1,43 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.exception;
+
+import org.apache.http.HttpStatus;
+
+public class FeatureNotAvailable extends MatrixException {
+
+    private String internalReason;
+
+    public FeatureNotAvailable(String internalReason) {
+        super(
+                HttpStatus.SC_INTERNAL_SERVER_ERROR,
+                "M_NOT_AVAILABLE",
+                "This action is currently not available. Contact your administrator to enable it."
+        );
+
+        this.internalReason = internalReason;
+    }
+
+    public String getInternalReason() {
+        return internalReason;
+    }
+
+}

src/main/java/io/kamax/mxisd/matrix/IdentityServerUtils.java

@@ -4,6 +4,7 @@ import com.google.gson.JsonElement;
 import com.google.gson.JsonParseException;
 import com.google.gson.JsonParser;
 import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.xbill.DNS.*;
@@ -28,6 +29,10 @@ public class IdentityServerUtils {
     private static JsonParser parser = new JsonParser();
 
     public static boolean isUsable(String remote) {
+        if (StringUtils.isBlank(remote)) {
+            return false;
+        }
+
         try {
             // FIXME use Apache HTTP client
             HttpURLConnection rootSrvConn = (HttpURLConnection) new URL(
@@ -54,7 +59,7 @@ public class IdentityServerUtils {
             }
 
             return true;
-        } catch (IOException | JsonParseException e) {
+        } catch (IllegalArgumentException | IOException | JsonParseException e) {
             log.info("{} is not a usable Identity Server: {}", remote, e.getMessage());
             return false;
         }

src/main/java/io/kamax/mxisd/session/SessionMananger.java

@@ -272,9 +272,13 @@ public class SessionMananger {
 
         List<String> servers = mxCfg.getIdentity().getServers(policy.getToRemote().getServer());
         if (servers.isEmpty()) {
-            throw new InternalServerError();
+            throw new FeatureNotAvailable("Remote 3PID sessions are enabled but server list is " +
+                    "misconstrued (invalid ID or empty list");
         }
-        String url = IdentityServerUtils.findIsUrlForDomain(servers.get(0)).orElseThrow(InternalServerError::new);
+
+        String is = servers.get(0);
+        String url = IdentityServerUtils.findIsUrlForDomain(is)
+                .orElseThrow(() -> new InternalServerError(is + " could not be resolved to an Identity server"));
         log.info("Will use IS endpoint {}", url);
 
         String remoteSecret = session.isRemote() ? session.getRemoteSecret() : RandomStringUtils.randomAlphanumeric(16);
@@ -283,13 +287,17 @@ public class SessionMananger {
         body.addProperty("client_secret", remoteSecret);
         body.addProperty(session.getThreePid().getMedium(), session.getThreePid().getAddress());
         body.addProperty("send_attempt", session.increaseAndGetRemoteAttempt());
-        try {
+        if (ThreePidMedium.PhoneNumber.is(session.getThreePid().getMedium())) {
-            Phonenumber.PhoneNumber msisdn = phoneUtil.parse("+" + session.getThreePid().getAddress(), null);
+            try {
-            String country = phoneUtil.getRegionCodeForNumber(msisdn).toUpperCase();
+                Phonenumber.PhoneNumber msisdn = phoneUtil.parse("+" + session.getThreePid().getAddress(), null);
-            body.addProperty("phone_number", phoneUtil.format(msisdn, PhoneNumberUtil.PhoneNumberFormat.NATIONAL));
+                String country = phoneUtil.getRegionCodeForNumber(msisdn).toUpperCase();
-            body.addProperty("country", country);
+                body.addProperty("phone_number", phoneUtil.format(msisdn, PhoneNumberUtil.PhoneNumberFormat.NATIONAL));
-        } catch (NumberParseException e) {
+                body.addProperty("country", country);
-            throw new InternalServerError(e);
+            } catch (NumberParseException e) {
+                throw new InternalServerError(e);
+            }
+        } else {
+            body.addProperty(session.getThreePid().getMedium(), session.getThreePid().getAddress());
         }
 
         log.info("Requesting remote session with attempt {}", session.getRemoteAttempt());

src/main/java/io/kamax/mxisd/storage/ormlite/OrmLiteSqliteStorage.java

@@ -146,8 +146,8 @@ public class OrmLiteSqliteStorage implements IStorage {
         return withCatcher(() -> {
             List<ThreePidSessionDao> daoList = sessionDao.queryForMatchingArgs(new ThreePidSessionDao(tpid, secret));
             if (daoList.size() > 1) {
-                log.error("Lookup for 3PID Session {}:{} returned more than one result");
+                throw new InternalServerError("Lookup for 3PID Session " +
-                throw new InternalServerError();
+                        tpid + " returned more than one result");
             }
 
             if (daoList.isEmpty()) {

src/main/java/io/kamax/mxisd/threepid/connector/email/EmailSendGridNotificationHandler.java

@@ -26,11 +26,13 @@ import io.kamax.matrix.ThreePidMedium;
 import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.config.ServerConfig;
 import io.kamax.mxisd.config.threepid.connector.EmailSendGridConfig;
+import io.kamax.mxisd.exception.FeatureNotAvailable;
 import io.kamax.mxisd.invitation.IThreePidInviteReply;
 import io.kamax.mxisd.notification.INotificationHandler;
 import io.kamax.mxisd.threepid.notification.PlaceholderNotificationGenerator;
 import io.kamax.mxisd.threepid.session.IThreePidSession;
 import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.springframework.beans.factory.annotation.Autowired;
@@ -118,6 +120,11 @@ public class EmailSendGridNotificationHandler extends PlaceholderNotificationGen
     }
 
     private void send(String recipient, Email email) {
+        if (StringUtils.isBlank(cfg.getIdentity().getFrom())) {
+            throw new FeatureNotAvailable("3PID Email identity: sender address is empty - " +
+                    "You must set a value for notifications to work");
+        }
+
         try {
             email.addTo(recipient);
             email.setFrom(cfg.getIdentity().getFrom());

src/main/java/io/kamax/mxisd/threepid/connector/email/EmailSmtpConnector.java

@@ -23,6 +23,7 @@ package io.kamax.mxisd.threepid.connector.email;
 import com.sun.mail.smtp.SMTPTransport;
 import io.kamax.matrix.ThreePidMedium;
 import io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig;
+import io.kamax.mxisd.exception.FeatureNotAvailable;
 import io.kamax.mxisd.exception.InternalServerError;
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang.StringUtils;
@@ -66,6 +67,11 @@ public class EmailSmtpConnector implements IEmailConnector {
 
     @Override
     public void send(String senderAddress, String senderName, String recipient, String content) {
+        if (StringUtils.isBlank(senderAddress)) {
+            throw new FeatureNotAvailable("3PID Email identity: sender address is empty - " +
+                    "You must set a value for notifications to work");
+        }
+
         if (StringUtils.isBlank(content)) {
             throw new InternalServerError("Notification content is empty");
         }

src/main/resources/application.yaml

@@ -1,3 +1,8 @@
+# DO NOT USE THIS FILE AS-IS FOR YOUR INITIAL CONFIGURATION
+# ONLY TAKE THE SPECIFIC SECTION YOU WANT TO CONFIGURE
+#
+# For more information about configuration, visit https://github.com/kamax-io/mxisd/blob/master/docs/configure.md
+
 spring:
   main:
     banner-mode: 'off'