Various error handling improvements and user feedback

Directory integration

README.md

@@ -2,220 +2,166 @@ mxisd - Federated Matrix Identity Server Daemon
 -----
 ![Travis-CI build status](https://travis-ci.org/kamax-io/mxisd.svg?branch=master)  
 
-[Overview](#overview) | [Features](#features) | [Lookup process](#lookup-process) | [Packages](#packages) | 
-[From source](#from-source) | [Configuration](#configuration) | [Network Discovery](#network-discovery) | 
-[Integration](#integration) | [Support](#support)
+- [Overview](#overview)
+- [Features](#features)
+- [Why use mxisd](#why-use-mxisd)
+- [Quick start](#quick-start)
+- [Support](#support)
+- [Contribute](#contribute)
+- [FAQ](#faq)
+- [Contact](#contact)
 
 # Overview
-mxisd is a Federated Matrix Identity server for self-hosted Matrix infrastructures.
+mxisd is a Federated Matrix Identity server for self-hosted Matrix infrastructures with enhanced features.
+  
+It is specifically designed to connect to an Identity store (AD/Samba/LDAP, SQL Database, Web services/application, ...)
+and ease the integration of the Matrix ecosystem with an existing infrastructure, or to build a new one using lasting
+tools.
 
-mxisd uses a cascading lookup model which performs lookup from a more authoritative to a less authoritative source, usually doing:
-- Local identity stores: LDAP, etc.
-- Federated identity stores: another Identity Server in charge of a specific domain, if applicable
-- Configured identity stores: another Identity Server specifically configured, if part of some sort of group trust
-- Root identity store: vector.im/matrix.org central Identity Servers
+The core principle of mxisd is to map between Matrix IDs and 3PIDs (Thrid-party Identifiers) for the Homeserver and its
+users. 3PIDs can be anything that identify a user, like:
+- Full name
+- Email address
+- Phone number
+- Employee number
+- Skype/Live ID
+- Twitter handle
+- Facebook ID
+- ...
 
-mxisd provides an alternative to [sydent](https://github.com/matrix-org/sydent), while still connecting to the vector.im and matrix.org Identity servers, 
-by implementing the [Matrix Identity Service specification](https://matrix.org/docs/spec/identity_service/unstable.html).
+mxisd is an enhanced Identity service, which implements the [Matrix Identity service API](https://matrix.org/docs/spec/identity_service/unstable.html)
+but also several other features that greatly enhance user experience within Matrix.
 
-mxisd only aims to support workflows that do NOT break federation or basic lookup processes of the Matrix ecosystem.
+mxisd is the one stop shop for anything regarding Authentication, Directory and Identity management in Matrix built as a
+single coherent product.
 
 # Features
-- Single lookup of 3PID (E-mail, phone number, etc.) by the Matrix Client or Homeserver.
-- Bulk lookups when trying to find possible matches within contacts in Android and iOS clients.
-- Bind of 3PID by a Matrix user within a Matrix client - See [documentation](docs/sessions/3pid.md)
-- Support of invitation to rooms by e-mail with e-mail notification to invitee.
-- Authentication support in [synapse](https://github.com/matrix-org/synapse) via the [REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth).
+As a [regular Matrix Identity service](docs/features/identity.md):
+- Search for people by 3PID using its own Identity stores (LDAP, SQL, etc.)
+- Invite people to rooms by 3PID using its own Identity stores, with [notifications](docs/README.md)
+to the invitee (Email, SMS, etc.)
+- Allow users to add 3PIDs to their settings/profile
 
-In the pipe:
-- Support to proxy 3PID bindings in user profile to the central Matrix.org servers
+As an enhanced Identity service:
+- Use a recursive lookup mechanism when searching and inviting people by 3PID, allowing to fetch data from:
+  - Own Identity store
+  - Federated Identity servers, if applicable to the 3PID
+  - Arbitrary Identity servers
+  - Central Matrix Identity servers
+- [Extensive control of where 3PIDs are transmited](docs/sessions/3pid.md), so they are not leaked publicly by users
+- [Authentication support](docs/features/authentication.md) for [synapse](https://github.com/matrix-org/synapse) via the
+[REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth)
+- [Directory integration](docs/features/directory-users.md) which allows you to search for users within your
+organisation, even without prior Matrix contact
+- [Auto-fill of user profile](docs/features/authentication.md) (Display name, 3PIDs) via the
+[REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth)
 
-# Lookup Process
-Default Lookup strategy will use a priority order and a configurable recursive/local type of request.
+# Why use mxisd
+- Use your existing Identity store, do not duplicate information
+- Auto-fill user profiles with relevant information
+- As an organisation, stay in control of 3PIDs so they are not published to the central Matrix.org servers where they
+currently **cannot be removed**
+- Users can directly find each other using whatever attribute is relevant within your Identity store
+- Federate your Identity lookups so you can discover others and/or others can discover you, all with extensive ACLs
 
-## E-mail
-Given the 3PID `john.doe@example.org`, the following will be performed until a mapping is found:
-- LDAP: lookup the Matrix ID (partial or complete) from a configurable attribute using a dedicated query.
-- DNS: lookup another Identity Server using the domain part of an e-mail and:
-  - Look for a SRV record under `_matrix-identity._tcp.example.org`
-  - Lookup using the base domain name `example.org`
-- Forwarder: Proxy the request to other configurable identity servers.
+# Quick Start
+1. [Preparation](#preparation)
+2. [Install](#install)
+3. [Configure](#configure)
+4. [Integrate](#integrate)
+5. [Validate](#validate)
 
-## Phone number
-Given the phone number `+123456789`, the following lookup logic will be performed:
-- LDAP: lookup the Matrix ID (partial or complete) from a configurable attribute using a dedicated query.
-- Forwarder: Proxy the request to other configurable identity servers.
+Following these quick start instructions, you will have a basic setup that can perform recursive/federated lookups and
+talk to the central Matrix.org Identity service.  
+This will be a good ground work for further integration with your existing Identity stores.
 
-# Packages
-See [releases]((https://github.com/kamax-io/mxisd/releases)) for native installers of supported systems.  
-If none is available, please use other packages or build from source.
+## Preparation
+You will need:
+- Homeserver
+- Reverse proxy with regular TLS/SSL certificate (Let's encrypt) for your mxisd domain
 
-## Debian
-### Download
-See the [releases section](https://github.com/kamax-io/mxisd/releases).
+As synapse requires an HTTPS connection when talking to an Identity service, a reverse proxy is required as mxisd does
+not support HTTPS listener at this time.
 
-### Configure and run
-After installation:
-1. Copy the sample config file `/etc/mxisd/mxisd-sample.yaml` to `/etc/mxisd/mxisd.yaml`
-2. [Configure](#configuration)
-3. Start the service: `sudo systemctl start mxisd`
+For maximum integration, it is best to have your Homeserver and mxisd reachable via the same hostname.  
+You can also use a dedicated domain for mxisd, but will not have access to some features.
 
-### From source
-Requirements:
-- fakeroot
-- dpkg-deb
+Be aware of a [NAT/Reverse proxy gotcha](https://github.com/kamax-io/mxisd/wiki/Gotchas#nating) if you use the same
+hostname.
 
-Run:
-```
-./gradlew buildDeb 
-```
-You will find the debian package in `build/dist`
-
-## Docker
-```
-docker pull kamax/mxisd
-```
-
-For more info, see [the public repository](https://hub.docker.com/r/kamax/mxisd/)
-### From source
-[Build mxisd](#build) then build the docker image:
-```
-./gradlew dockerBuild
-```
-You can run a container of the given image and test it with the following command (adapt volumes host paths):
-```
-docker run -v /data/mxisd/etc:/etc/mxisd -v /data/mxisd/var:/var/mxisd -p 8090:8090 -t kamax/mxisd:latest-dev
-```
-
-# From Source
-## Requirements
-- JDK 1.8
-
-## Build
-```
-git clone https://github.com/kamax-io/mxisd.git
-cd mxisd
-./gradlew build
-```
-then see the [Configuration](#configuration) section.
-
-## Test build
-Start the server in foreground to validate the build:
-```
-java -jar build/libs/mxisd.jar
-```
-
-Ensure the signing key is available:
-```
-$ curl http://localhost:8090/_matrix/identity/api/v1/pubkey/ed25519:0
-{"public_key":"..."}
-```
-
-Test basic recursive lookup (requires Internet connection with access to TCP 443):
-```
-$ curl 'http://localhost:8090/_matrix/identity/api/v1/lookup?medium=email&address=mxisd-lookup-test@kamax.io'
-{"address":"mxisd-lookup-test@kamax.io","medium":"email","mxid":"@mxisd-lookup-test:kamax.io",...}
-```
-
-
-If you enabled LDAP, you can also validate your config with a similar request after replacing the `address` value with something present within your LDAP
-```
-curl "http://localhost:8090/_matrix/identity/api/v1/lookup?medium=email&address=john.doe@example.org"
-```
-
-If you plan on testing the integration with a homeserver, you will need to run an HTTPS reverse proxy in front of it
-as the reference Home Server implementation [synapse](https://github.com/matrix-org/synapse) requires a HTTPS connection
-to an ID server.  
-See the [Integration section](#integration) for more details.
+The following Quick Start guide assumes you will host the Homeserver and mxisd under the same hostname.  
+If you would like a high-level view of the infrastructure and how each feature is integrated, see the
+[dedicated document](docs/architecture.md)
 
 ## Install
-After [building](#build) the software, run all the following commands as `root` or using `sudo`
-1. Prepare files and directories:
+Install via:
+- [Debian package](docs/install/debian.md)
+- [Docker image](docs/install/docker.md)
+- [Sources](docs/build.md)
+
+See the [Latest release](https://github.com/kamax-io/mxisd/releases/latest) for links to each.
+
+## Configure
+Create/edit a minimal configuration (see installer doc for the location):
 ```
-# Create a dedicated user
-useradd -r mxisd
+matrix.domain: 'MyMatrixDomain.org'
+key.path: '/path/to/signing.key.file'
+storage.provider.sqlite.database: '/path/to/mxisd.db'
+```  
+- `matrix.domain` should be set to your Homeserver domain
+- `key.path` will store the signing keys, which must be kept safe!
+- `storage.provider.sqlite.database` is the location of the SQLite Database file which will hold state (invites, etc.)
 
-# Create bin directory
-mkdir /opt/mxisd
+If your HS/mxisd hostname is not the same as your Matrix domain, configure `server.name`.  
+Complete configuration guide is available [here](docs/configure.md).
 
-# Create config directory and set ownership
-mkdir /etc/mxisd
-chown mxisd /etc/mxisd
-
-# Create data directory and set ownership
-mkdir /var/opt/mxisd
-chown mxisd /var/opt/mxisd
-
-# Copy <repo root>/build/libs/mxisd.jar to bin directory
-cp ./build/libs/mxisd.jar /opt/mxisd/
-chown mxisd /opt/mxisd/mxisd.jar
-chmod a+x /opt/mxisd/mxisd.jar
-
-# Create symlink for easy exec
-ln -s /opt/mxisd/mxisd.jar /usr/bin/mxisd
+## Integrate
+For an overview of a typical mxisd infrastructure, see the [dedicated document](docs/architecture.md)
+### Reverse proxy
+#### Apache2
+In the VirtualHost handling the domain with SSL, add the following line and replace `0.0.0.0` by the right address/host.  
+**This line MUST be present before the one for the homeserver!**
+```
+ProxyPass /_matrix/identity/ http://0.0.0.0:8090/_matrix/identity/
 ```
 
-2. Copy the config file created earlier `./application.example.yaml` to `/etc/mxisd/mxisd.yaml`
-3. [Configure](#configuration)
-4. Copy `<repo root>/src/systemd/mxisd.service` to `/etc/systemd/system/` and edit if needed
-5. Enable service for auto-startup
+Typical VirtualHost configuration would be:
 ```
-systemctl enable mxisd
-```
-6. Start mxisd
-```
-systemctl start mxisd
+<VirtualHost *:443>
+    ServerName example.org
+    
+    ...
+    
+    ProxyPreserveHost on
+    ProxyPass /_matrix/identity/ http://10.1.2.3:8090/_matrix/identity/
+    ProxyPass /_matrix/ http://10.1.2.3:8008/_matrix/
+</VirtualHost>
 ```
 
-# Configuration
-After following the specific instructions to create a config file from the sample:
-1. Set the `matrix.domain` value to the domain value used in your Home Server configuration
-2. Set an absolute location for the signing keys using `key.path`
-3. Configure the E-mail notification sender with items starting with:
-  - `threepid.medium.email.identity`
-  - `threepid.medium.email.connectors.smtp`
-4. If you would like to support Phone number validation, see the [Twilio configuration](docs/threepids/msisdn/twilio-connector.md)
-
-In case your IS public domain does not match your Matrix domain, see `server.name` and `server.publicUrl` 
-config items.
-
-
-## Backends
-### LDAP (AD, Samba, LDAP)
-If you want to use LDAP backend as an Identity store:
-1. Enable it with `ldap.enabled`
-2. Configure connection options using items starting in `ldap.connection`
-3. You may want to valid default values for `ldap.attribute` items
-
-### SQL (SQLite, PostgreSQL)
-If you want to connect to use a synapse DB (SQLite or PostgreSQL) as Identity store, follow the example config for `sql` config items.
-
-### REST (Webapps/websites integration)
-If you want to use the REST backend as an Identity store:
-1. Enable it with `rest.enabled`
-2. Configure options starting with `rest` and see the dedicated documentation in `docs/backends/rest.md`
-
-# Network Discovery
-To allow other federated Identity Server to reach yours, the same algorithm used for Homeservers takes place:
-1. Check for the appropriate DNS SRV record
-2. If not found, use the base domain
- 
-If your Identity Server public hostname does not match your Matrix domain, configure the following DNS SRV entry 
-and replace `matrix.example.com` by your Identity server public hostname - **Make sure to end with a final dot!**
+### Synapse
+Add your mxisd domain into the `homeserver.yaml` at `trusted_third_party_id_servers` and restart synapse.  
+In a typical configuration, you would end up with something similair to:
 ```
-_matrix-identity._tcp.example.com. 3600 IN SRV 10 0 443 matrix.example.com.
-``` 
-This would only apply for 3PID that are DNS-based, like e-mails. For anything else, like phone numbers, no federation 
-is currently possible.  
+trusted_third_party_id_servers:
+    - matrix.org
+    - vector.im
+    - example.org
+```
+It is recommended to remove `matrix.org` and `vector.im` so only your own Identity server is allowed by synapse. 
 
-The port must be HTTPS capable. Typically, the port `8090` of mxisd should be behind a reverse proxy which does HTTPS.
-See the [integration section](#integration) for more details.
+### Federation and network discovery
+See the [dedicated document](docs/features/federation.md).
 
-# Integration
-- [HTTPS and Reverse proxy](https://github.com/kamax-io/mxisd/wiki/HTTPS)
-- [synapse](https://github.com/kamax-io/mxisd/wiki/Homeserver-Integration) as Identity server
-- [synapse with REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth/blob/master/README.md) 
-as authentication module
+## Validate
+Log in using your Matrix client and set `https://example.org` as your Identity server URL, replacing `example.org` by
+the relevant hostname which you configured in your reverse proxy.  
+Invite `mxisd-lookup-test@kamax.io` to a room, which should be turned into a Matrix invite to `@mxisd-lookup-test:kamax.io`.  
+**NOTE:** you might not see a Matrix suggestion for the e-mail address, which is normal. Still proceed with the invite.
+  
+If it worked, it means you are up and running and can enjoy mxisd in its basic mode! Congratulations!  
+If it did not work, [get in touch](#support) and we'll do our best to get you started.
+
+You can now integrate mxisd further with your infrastructure using the various [features](docs/README.md) guides.
 
 # Support
 ## Community
@@ -227,3 +173,87 @@ For more high-level discussion about the Identity Server architecture/API, go to
 ## Professional
 If you would prefer professional support/custom development for mxisd and/or for Matrix in general, including other open source technologies/products, 
 please visit [our website](https://www.kamax.io/) to get in touch with us and get a quote.
+
+We offer affordable monthly/yearly support plans for mxisd, synapse or your full Matrix infrastructure.
+
+# Contribute
+First and foremost, the best way to contribute is to use mxisd and tell us about it!  
+We would love to hear about your experience and get your feedback on how to make it an awesome product. 
+
+You can contribute as a community member by:
+- Opening issues for any weird behaviour or bug. mxisd should feel natural, let us know if it does not!
+- Helping us improve the documentation: tell us what is good or not good (in an issue or in Matrix), or make a PR with
+changes you feel improve the doc.
+- Contribute code directly: we love contributors! All your contributions will be licensed under AGPLv3.
+- Donate! any donation is welcome, regardless how small or big. This will directly be used for the fixed costs and
+developer time.
+
+You can contribute as an organisation/corporation by:
+- Get a [support contract](#support-professional). This is the best way you can help us as it ensures mxisd is
+maintained regularly and you get direct access to the support team.
+- Sponsoring new features or bug fixes. [Get in touch](#contact) so we can discuss it further.
+
+# FAQ
+### Do I need to use mxisd if I run a Homeserver?
+No, but it is recommended, even if you don't use any backends or integration.
+
+mxisd in its default configuration will use federation and involve the central Matrix.org Identity servers when
+performing queries, giving you access to at least the same information as if you were not running it.
+
+It will also give your users a choice to make their 3PIDs available publicly, ensuring they are made aware of the
+privacy consequences, which is not the case with the central Matrix.org servers.
+
+So mxisd is like your gatekeeper and guardian angel. It does not change what you already know, just adds some nice
+simple features on top of it.
+
+### I already use the synapse LDAP3 auth provider, why should I care about mxisd?
+The [synapse LDAP3 auth provider](https://github.com/matrix-org/matrix-synapse-ldap3) only handles on specific flow:
+validate credentials at login.
+
+It does not:
+- Auto-provision user profiles
+- Integrate with Identity management
+- Integrate with Directory searches
+- Protect you against the username case sensitivites issues in synapse
+
+mxisd is a replacement and enhancement of it, and also offers coherent results in all areas, which LDAP3 auth provider
+does not.
+
+### I saw that sydent is the official Identity server implemenation of the Matrix team, I should use that!
+You can, but [sydent](https://github.com/matrix-org/sydent):
+- [should not be used and/or self-hosted](https://github.com/matrix-org/sydent/issues/22)
+- is not meant to be linked to a specific Homeserver / domain
+- cannot handle federation or proxy lookups, effectively isolating your users from the rest of the network
+- forces you to duplicate all your identity data, so people can be found by 3PIDs
+- forces users to enter all their emails and phone numbers manually in their profile
+
+So really, you should go with mxisd.
+
+### I'm not sure I understand what an "Identity server" is supposed to be or do
+The current Identity service API is more a placeholder, as the Matrix devs did not have time so far to really work on
+what they want to do with that part of the ecosystem. Therefore, "Identity" is a misleading word currently.  
+Given the scope of the current Identity Service API, it would be best called "Invitation service".
+
+Because the current scope is so limited and no integration is done with the Homeserver, there was a big lack of features
+for groups/corporations/organisation. This is where mxisd comes in.
+
+mxisd implements the Identity Service API and also a set of features which are expected by regular users, truly living
+up to its "Identity server" name.
+
+### So mxisd is just a big hack! I don't want to use non-official features!
+mxisd primary concern is to always be compatible with the Matrix ecosystem and the Identity service API.  
+Whenever the API will be updated and/or enhanced, mxisd will follow, remaining 100% compatible with the ecosystem.
+
+We also directly talk with the Matrix developers to ensure all features we implement have their approval, and that we
+are in line with their vision of Identity management within the Matrix ecosystem.
+
+Therefore, using mxisd is a safe choice. It will be like using the central Matrix.org Identity servers, yet not closing
+the door to very nice enhancements and integrations.
+
+### Should I use mxisd if I don't host my own Homeserver?
+No
+
+# Contact
+Get in touch via:
+- Matrix at [#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io)
+- Email, see our website: [Kamax.io](https://www.kamax.io)

application.example.yaml

@@ -1,11 +1,6 @@
-# Sample configuration file explaining all possible options, their default value and if they are required or not.
+# Sample configuration file explaining the minimum required keys to be set to run mxisd
 #
-# Any optional configuration item will be prefixed by # (comment character) with the configuration item following
-# directly without any whitespace character.
-# Default values for optional configuration item will also follow such item.
-#
-# Any mandatory configuration item will not be prefixed by # and will also contain a value as example that must be
-# changed. It is advised to re-create a clean config file with only the required configuration item.
+# For a complete list of options, see https://github.com/kamax-io/mxisd
 
 #######################
 # Matrix config items #
@@ -16,48 +11,9 @@
 matrix.domain: ''
 
 
-
-#######################
-# Server config items #
-#######################
-# Indicate on which port the Identity Server will listen.
-#
-# This is be default an unencrypted port.
-# HTTPS can be configured using Tomcat configuration properties.
-#
-#server.port: 8090
-
-
-# Public hostname of this identity server.
-#
-# This would be typically be the same as your Matrix domain.
-# In case it is not, set this value.
-#
-# This value is used in various signatures within the Matrix protocol and should be a reachable hostname.
-# You can validate by ensuring you see a JSON answer when calling (replace the domain):
-# https://example.org/_matrix/identity/status
-#
-#server.name: 'example.org'
-
-
-# Public URL to reach this identity server
-#
-# This is used with 3PID invites in room and other Homeserver key verification workflow.
-# If left unconfigured, it will be generated from the server name.
-#
-# You should typically set this value if you want to change the public port under which
-# this Identity server is reachable.
-#
-# %SERVER_NAME% placeholder is available to avoid configuration duplication.
-# e.g. 'https://%SERVER_NAME%:8443'
-#
-#server.publicUrl: 'https://example.org'
-
-
-
-#############################
-# Signing keys config items #
-#############################
+################
+# Signing keys #
+################
 # Absolute path for the Identity Server signing key.
 # During testing, /var/tmp/mxisd.key is a possible value
 #
@@ -65,278 +21,7 @@ matrix.domain: ''
 #   - /var/opt/mxisd/sign.key
 #   - /var/local/mxisd/sign.key
 #   - /var/lib/mxisd/sign.key
-key.path: '/path/to/sign.key'
-
-
-
-#################################
-# Recurisve lookup config items #
-#################################
-# Configuration items for recursion-type of lookup
-#
-# Lookup access are divided into two types:
-# - Local
-# - Remote
-#
-# This is similar to DNS lookup and recursion and is therefore prone to the same vulnerabilities.
-# By default, only non-public hosts are allowed to perform recursive lookup.
-#
-# This will also prevent very basic endless loops where host A ask host B, which in turn is configured to ask host A,
-# which would then ask host B again, etc.
-
-# Enable recursive lookup globally
-#
-#lookup.recursive.enabled: true
-
-
-# Whitelist of CIDR that will trigger a recursive lookup.
-# The default list includes all private IPv4 address and the IPv6 loopback.
-#
-#lookup.recursive.allowedCidr:
-#  - '127.0.0.0/8'
-#  - '10.0.0.0/8'
-#  - '172.16.0.0/12'
-#  - '192.168.0.0/16'
-#  - '::1/128'
-
-
-# In case no binding is found, query an application server which implements the single lookup end-point
-# to return bridge virtual user that would allow the user to be contacted directly by the said bridge.
-#
-# If a binding is returned, the application server is not expected to sign the message as it is not meant to be
-# reachable from the outside.
-# If a signature is provided, it will be discarded/replaced by this IS implementation (to be implemented).
-#
-# IMPORTANT: This bypass the regular Invite system of the Homeserver. It will be up to the Application Server
-# to handle such invite. Also, if the bridged user were to actually join Matrix later, or if a 3PID binding is found
-# room rights and history would not be transferred, as it would appear as a regular Matrix user to the Homeserver.
-#
-# This configuration is only helpful for Application Services that want to overwrite bridging for 3PID that are
-# handled by the Homeserver. Do not enable unless the Application Server specifically supports it!
-
-# Enable unknown 3PID bridging globally
-#
-#lookup.recursive.bridge.enabled: false
-
-
-# Enable unknown 3PID bridging for hosts that are allowed to perform recursive lookups.
-# Leaving this setting to true is highly recommended in a standard setup, unless this Identity Server
-# is meant to always return a virtual user MXID even for the outside world.
-#
-#lookup.recursive.bridge.recursiveOnly: true
-
-
-# This mechanism can handle the following scenarios:
-#
-# - Single Application Server for all 3PID types: only configure the server value, comment out the rest.
-#
-# - Specific Application Server for some 3PID types, default server for the rest: configure the server value and
-#          each specific 3PID type.
-#
-# - Only specific 3PID types: do not configure the server value or leave it empty/blank, configure each specific
-#          3PID type.
-
-# Default application server to use for all 3PID types. Remove config item or leave empty/blank to disable.
-#
-#lookup.recursive.bridge.server: ''
-
-
-# Configure each 3PID type with a specific application server. Remove config item or leave empty/blank to disable.
-#
-#lookup.recursive.bridge.mappings.email: 'http://localhost:8091'
-#lookup.recursive.bridge.mappings.msisdn: ''
-
-
-
-#####################
-# LDAP config items #
-#####################
-# Global enable/disable switch
-#
-#ldap.enabled: false
-
-
-#### Connection related config items
-# If the connection should be secure
-#
-#ldap.connection.tls: false
-
-
-# Host to connect to
-#
-#ldap.connection.host: 'localhost'
-
-
-# Port to connect to
-#
-#ldap.connection.port: 389
-
-
-# Bind DN for the connection.
-#
-# If Bind DN and password are empty, anonymous authentication is performed
-#
-#ldap.connection.bindDn: 'CN=Matrix Identity Server,CN=Users,DC=example,DC=org'
-
-
-# Bind password for the connection.
-#
-#ldap.connection.bindPassword: 'password'
-
-
-# Base DN used in all queries
-#
-#ldap.connection.baseDn: 'CN=Users,DC=example,DC=org'
-
-
-#### How to map Matrix attributes with LDAP attributes when performing lookup/auth
-#
-# How should we resolve the Matrix ID in case of a match using the attribute.
-#
-# The following type are supported:
-# - uid : the attribute only contains the UID part of the Matrix ID. e.g. 'john.doe' in @john.doe:example.org
-# - mxid : the attribute contains the full Matrix ID - e.g. '@john.doe:example.org'
-#
-#ldap.attribute.uid.type: 'uid'
-
-
-# The attribute containing the binding itself. This value will be used differently depending on the type.
-#
-# /!\ This should match the synapse LDAP Authenticator 'uid' configuration /!\
-#
-# Typical values:
-# - For type 'uid': 'userPrincipalName' or 'uid' or 'saMAccountName'
-# - For type 'mxid', regardless of the directory type, we recommend using 'pager' as it is a standard attribute and
-#   is typically not used.
-#
-#ldap.attribute.uid.value: 'userPrincipalName'
-
-
-# The display name of the user
-#
-#ldap.attribute.name: 'displayName'
-
-
-#### Configuration section relating the authentication of users performed via LDAP.
-#
-# This can be done using the REST Auth module for synapse and pointing it to the identity server.
-# See https://github.com/kamax-io/matrix-synapse-rest-auth
-#
-# During authentication, What to filter potential users by, typically by using a dedicated group.
-# If this value is not set, login check will be performed for all entities within the LDAP
-#
-# Example: (memberOf=CN=Matrix Users,CN=Users,DC=example,DC=org)
-#
-#ldap.auth.filter: ''
-
-
-#### Configuration section relating to identity lookups
-#
-# E-mail query
-#
-#ldap.identity.medium.email: "(|(mailPrimaryAddress=%3pid)(mail=%3pid)(otherMailbox=%3pid))"
-
-
-# Phone numbers query
-#
-# Phone numbers use the MSISDN format: https://en.wikipedia.org/wiki/MSISDN
-# This format does not include international prefix (+ or 00) and therefore has to be put in the query.
-# Adapt this to your needs for each attribute.
-#
-#ldap.identity.medium.msisdn: "(|(telephoneNumber=+%3pid)(mobile=+%3pid)(homePhone=+%3pid)(otherTelephone=+%3pid)(otherMobile=+%3pid)(otherHomePhone=+%3pid))"
-
-
-
-############################
-# SQL Provider config item #
-############################
-#
-# Example configuration to integrate with synapse SQLite DB (default configuration)
-#
-#sql.enabled: true
-#sql.type: 'sqlite'
-#sql.connection: '/var/lib/matrix-synapse/homeserver.db'
-
-
-#
-# Example configuration to integrate with synapse PostgreSQL DB
-#sql.enabled: true
-#sql.type: 'postgresql'
-#sql.connection: '//dnsOrIpToServer/dbName?user=synapseDbUser&password=synapseDbPassword'
-
-
-#
-# Configuration for an arbitrary server with arbitrary driver
-#
-# sql.identity.type possible values:
-# - uid       Returned value is the localpart of the Matrix ID
-# - mxid      Full Matrix ID, including domain
-#
-# sql.identity.query MUST contain a column with label 'uid'
-#
-# If you would like to overwrite the global lookup query for specific medium type,
-# add a config item (see below for example) in the following format
-# sql.identity.medium.theMediumIdYouWant: 'the query'
-
-#sql.enabled: true
-#sql.type: 'jdbcDriverName'
-#sql.connection: '//dnsOrIpToServer/dbName?user=synapseDbUser&password=synapseDbPassword'
-#sql.identity.type: 'mxid'
-#sql.identity.query: 'SELECT raw AS uid FROM table WHERE medium = ? AND address = ?'
-#sql.identity.medium.email: 'SELECT raw AS uid FROM emailTable WHERE address = ?'
-
-
-
-#######################################
-# Lookup queries forward config items #
-#######################################
-# List of forwarders to use to try to match a 3PID.
-#
-# Each server will be tried in the given order, going to the next if no binding was found or an error occurred.
-# These are the current root Identity Servers of the Matrix network.
-#
-#forward.servers:
-#  - "https://matrix.org"
-#  - "https://vector.im"
-
-
-
-###################################
-# 3PID notifications config items #
-###################################
-# If you would like to change the content, see https://github.com/kamax-io/mxisd/blob/master/docs/threepids/notifications/template-generator.md
-#
-#### E-mail invite sender
-#
-# SMTP host
-threepid.medium.email.connectors.smtp.host: "smtp.example.org"
-
-
-# SMTP port
-threepid.medium.email.connectors.smtp.port: 587
-
-
-# TLS mode for the connection.
-#
-# Possible values:
-#  0    Disable TLS entirely
-#  1    Enable TLS if supported by server
-#  2    Force TLS and fail if not available
-#
-#threepid.medium.email.connectors.smtp.tls: 1
-
-
-# Login for SMTP
-threepid.medium.email.connectors.smtp.login: "matrix-identity@example.org"
-
-
-# Password for the account
-threepid.medium.email.connectors.smtp.password: "ThePassword"
-
-
-# The e-mail to send as. If empty, will be the same as login
-threepid.medium.email.identity.from: "matrix-identity@example.org"
-
+key.path: ''
 
 
 ############################
@@ -349,10 +34,7 @@ threepid.medium.email.identity.from: "matrix-identity@example.org"
 #
 #storage.backend: 'sqlite'
 
-
-#### Generic SQLite provider config
-#
-# Path to the SQLite DB file, required if SQLite backend is chosen
+# Path to the SQLite DB file
 #
 # Examples:
 #  - /var/opt/mxisd/mxisd.db
@@ -362,22 +44,53 @@ threepid.medium.email.identity.from: "matrix-identity@example.org"
 storage.provider.sqlite.database: '/path/to/mxisd.db'
 
 
+################
+# LDAP Backend #
+################
+# If you would like to integrate with your AD/Samba/LDAP server,
+# see https://github.com/kamax-io/mxisd/blob/master/docs/backends/ldap.md
 
-######################
-# DNS-related config #
-######################
-# The domain to overwrite
+###############
+# SQL Backend #
+###############
+# If you would like to integrate with a MySQL/MariaDB/PostgreQL/SQLite DB,
+# see https://github.com/kamax-io/mxisd/blob/master/docs/backends/sql.md
+
+################
+# REST Backend #
+################
+# If you would like to integrate with an existing web service/webapp,
+# see https://github.com/kamax-io/mxisd/blob/master/docs/backends/rest.md
+
+
+#################################################
+# Notifications for invites/addition to profile #
+#################################################
+# If you would like to change the content,
+# see https://github.com/kamax-io/mxisd/blob/master/docs/threepids/notifications/template-generator.md
 #
-#dns.overwrite.homeserver.name: 'example.org'
-
-
-# - 'env' from environment variable specified by value
-# - any other value will use the value as-is as host
+#### E-mail invite sender
 #
-#dns.overwrite.homeserver.type: 'raw'
+# SMTP host
+threepid.medium.email.connectors.smtp.host: "smtp.example.org"
 
+# SMTP port
+threepid.medium.email.connectors.smtp.port: 587
 
-# The value to use, depending on the type.
-# Protocol will always be HTTPS
+# TLS mode for the connection.
 #
-#dns.overwrite.homeserver.value: 'localhost:8448'
+# Possible values:
+#  0    Disable TLS entirely
+#  1    Enable TLS if supported by server (default)
+#  2    Force TLS and fail if not available
+#
+#threepid.medium.email.connectors.smtp.tls: 1
+
+# Login for SMTP
+threepid.medium.email.connectors.smtp.login: "matrix-identity@example.org"
+
+# Password for the account
+threepid.medium.email.connectors.smtp.password: "ThePassword"
+
+# The e-mail to send as. If empty, will be the same as login
+threepid.medium.email.identity.from: "matrix-identity@example.org"

build.gradle

@@ -110,6 +110,9 @@ dependencies {
     // ORMLite
     compile 'com.j256.ormlite:ormlite-jdbc:5.0'
 
+    // Connection Pool
+    compile 'com.mchange:c3p0:0.9.5.2'
+
     // SQLite
     compile 'org.xerial:sqlite-jdbc:3.20.0'
 
@@ -119,6 +122,9 @@ dependencies {
     // Twilio SDK for SMS
     compile 'com.twilio.sdk:twilio:7.14.5'
 
+    // SendGrid SDK to send emails from GCE
+    compile 'com.sendgrid:sendgrid-java:2.2.2'
+
     testCompile 'junit:junit:4.12'
     testCompile 'com.github.tomakehurst:wiremock:2.8.0'
 }

docs/README.md

@@ -0,0 +1,25 @@
+# Table of Contents
+
+- Installation
+  - [Debian package](install/debian.md)
+  - [Docker](install/docker.md)
+- [Build from source](build.md)
+- [Architecture overview](architecture.md)
+- [Configuration](configure.md)
+- Features
+  - [Matrix Identity Service](features/identity.md)
+  - [Homeserver Authentication](features/authentication.md)
+  - [Directory seach](features/directory-users.md)
+  - [Identity server Federation](features/federation.md)
+  - [Bridge integration](features/bridge-integration.md)
+- Backends
+  - [LDAP](backends/ldap.md)
+  - [SQL](backends/sql.md)
+  - [REST](backends/rest.md)
+  - [Google Firebase](backends/firebase.md)
+- Notifications
+  - Handlers
+    - [Basic](threepids/notifications/basic-handler.md)
+    - [SendGrid](threepids/notifications/sendgrid-handler.md)
+- [Sessions](sessions/3pid.md)
+  - [Views](sessions/3pid-views.md)

docs/architecture.md

@@ -0,0 +1,43 @@
+# Architecture
+## Overview
+### Basic setup without integration or federation
+```
+ Client
+   |
+TCP 443
+   |   +---------------------+            +---------------------------+
+   +-> | Reverse proxy       |            | Homeserver                |
+       |                     | TCP 8008   |                           |
+       |  /_matrix/* -------------------> | - 3PID invite from client |
+       |                     |            |   |                       |
+       |  /_matrix/identity/ |            |   |                       |
+       +--|------------------+            +---|-----------------------+
+          |                                   |
+          +<---------------------------------<+
+          |                                          Backends
+          |   +-------------------+                  +------+    +--------+
+ TCP 8090 +-> | mxisd             |          +-----> | LDAP | -> | SQL DB |
+              |                   |          |       +------+    +--------+ ....
+              | - Profile's 3PIDs >----+     |
+              | - 3PID Invites    |    |     |
+              +-|-----------------+    +>----+
+                |                      |     |       +--------------------------+
+                |                      |     |       | Central Identity service |
+                +>-------------------->+     +-----> | Matrix.org / Vector.im   |
+                |                            TCP 443 +--------------------------+
+             TCP 443
+                |  +------------------------+
+                |  | Remote Federated       |
+                |  | mxisd servers          |
+                |  |                        |
+                +--> - 3PID Invites         |
+                   +------------------------+
+```
+### With Authentication
+See the [dedicated document](features/authentication.md).
+
+### With Directory
+See the [dedicated document](features/directory-users.md).
+
+### With Federation
+See the [dedicated document](features/federation.md).

docs/backends/firebase.md

@@ -0,0 +1,9 @@
+# Google Firebase
+## Configuration
+To be completed. For now, see default structure and values:
+```
+firebase:
+  enabled: false
+  credentials: '/path/to/firebase/credentials.json'
+  database: 'https://my-project.firebaseio.com/'
+```

docs/backends/ldap.md

@@ -0,0 +1,86 @@
+# AD/Samba/LDAP backend
+## Configuration
+### Structure and default values
+```
+ldap:
+  enabled: false
+  filter: ''
+  connection:
+    host: ''
+    tls: false
+    port: 389
+    bindDn: ''
+    bindPassword: ''
+    baseDn: ''
+  attribute:
+    uid:
+      type: 'uid'
+      value: 'userPrincipalName'
+    name: 'displayName'
+    threepid:
+      email:
+        - 'mailPrimaryAddress'
+        - 'mail'
+        - 'otherMailbox'
+      msisdn:
+        - 'telephoneNumber'
+        - 'mobile'
+        - 'homePhone'
+        - 'otherTelephone'
+        - 'otherMobile'
+        - 'otherHomePhone'
+  auth:
+    filter: ''
+  directory:
+    attribute:
+      other: []
+    filter: ''
+  identity:
+    filter: ''
+    medium:
+      email: ''
+      msisdn: ''
+```
+### General
+| Item      | Description                                                                               |
+|-----------|-------------------------------------------------------------------------------------------|
+| `enabled` | Globaly enable/disable the LDAP backend                                                   |
+| `filter`  | Global filter to apply on all LDAP queries. Can be overwritten in each applicable section |
+
+### Connection
+| Item           | Description                                          |
+|----------------|------------------------------------------------------|
+| `host`         | Host to connect to                                   |
+| `port`         | Port to use                                          |
+| `tls`          | boolean to use TLS or not (STARTLS is not supported) |
+| `bindDn`       | Bind DN for authentication                           |
+| `bindPassword` | Bind password                                        |
+| `baseDn`       | Base DN for queries                                  |
+
+### Attributes
+| Item        | Description                                                                                                            |
+|-------------|------------------------------------------------------------------------------------------------------------------------|
+| `uid.type`  | Indicate how to process the User ID (UID) attribute:                                                                   |
+|             |   - `uid` will consider the value as the [Localpart](https://matrix.org/docs/spec/intro.html#user-identifiers)         |
+|             |   - `mxid` will consider the value as a complete [Matrix ID](https://matrix.org/docs/spec/intro.html#user-identifiers) |
+| `uid.value` | Attribute name refering to the User ID. This is typically `userPrincipalName` on AD/Samba setups and `uid` in LDAP     |
+| `name`      | Attribute name that contains the [Display Name](https://matrix.org/docs/spec/intro.html#profiles) of the user          |
+| `threepid`  | Namespace where each key is a 3PID type and contains a list of attributes                                              |
+
+### Authentication
+| Item     | Description                                                                                      |
+|----------|--------------------------------------------------------------------------------------------------|
+| `filter` | Specific user filter applied during authentication. Global filter is used if empty/blank/not set |
+
+### Directory
+| Item              | Description                                                         |
+|-------------------|---------------------------------------------------------------------|
+| `attribute.other` | Additional attributes to be used when performing directory searches |
+| `filter`          | Specific user filter applied during directory search.               |
+|                   | Global filter is used if empty/blank/not set                        |
+
+### Identity
+| Item     | Description                                                                                       |
+|----------|---------------------------------------------------------------------------------------------------|
+| `filter` | Specific user filter applied during identity search. Global filter is used if empty/blank/not set | 
+| `medium` | Namespace to overwrite generated queries from the list of attributes for each 3PID medium         |

docs/backends/rest.md

@@ -6,33 +6,33 @@ The REST backend allows you to query identity data in existing webapps, like:
 - self-hosted clouds (Nextcloud, ownCloud, ...)
 
 It supports the following mxisd flows:
-- Identity lookup
-- Authentication
+- [Authentication](#authentication)
+- [Directory](#directory)
+- [Identity](#identity)
 
 To integrate this backend with your webapp, you will need to implement three specific REST endpoints detailed below.
 
 
 ## Configuration
-| Key                            | Default                               | Description                                          |
----------------------------------|---------------------------------------|------------------------------------------------------|
-| rest.enabled                   | false                                 | Globally enable/disable the REST backend             |
-| rest.host                      | *empty*                               | Default base URL to use for the different endpoints. |
-| rest.endpoints.auth            | /_mxisd/identity/api/v1/auth          | Endpoint to validate credentials                     |
-| rest.endpoints.identity.single | /_mxisd/identity/api/v1/lookup/single | Endpoint to query a single 3PID                      |
-| rest.endpoints.identity.bulk   | /_mxisd/identity/api/v1/lookup/bulk   | Endpoint to query a list of 3PID                     |
+| Key                            | Default                                      | Description                                          |
+---------------------------------|----------------------------------------------|------------------------------------------------------|
+| rest.enabled                   | false                                        | Globally enable/disable the REST backend             |
+| rest.host                      | *empty*                                      | Default base URL to use for the different endpoints. |
+| rest.endpoints.auth            | /_mxisd/backend/api/v1/auth/login            | Validate credentials and get user profile            |
+| rest.endpoints.directory       | /_mxisd/backend/api/v1/directory/user/search | Search for users by arbitrary input                  |
+| rest.endpoints.identity.single | /_mxisd/backend/api/v1/identity/single       | Endpoint to query a single 3PID                      |
+| rest.endpoints.identity.bulk   | /_mxisd/backend/api/v1/identity/bulk         | Endpoint to query a list of 3PID                     |
 
 Endpoint values can handle two formats:
 - URL Path starting with `/` that gets happened to the `rest.host`
 - Full URL, if you want each endpoint to go to a specific server/protocol/port
 
-`rest.host` is only mandatory if at least one endpoint is not a full URL.
+`rest.host` is mandatory if at least one endpoint is not a full URL.
 
 ## Endpoints
-### Authenticate
-Configured with `rest.endpoints.auth`
-
+### Authentication
 HTTP method: `POST`  
-Encoding: JSON UTF-8
+Content-type: JSON UTF-8
   
 #### Request Body
 ```
@@ -84,12 +84,51 @@ If the authentication succeed:
 }
 ```
 
-### Lookup
-#### Single
-Configured with `rest.endpoints.identity.single`
+### Directory
+HTTP method: `POST`
+Content-type: JSON UTF-8
 
+#### Request Body
+```
+{
+  "by": "<search type>",
+  "search_term": "doe"
+}
+```
+`by` can be:
+- `name`
+- `threepid`
+
+#### Response Body:
+If users found:
+```
+{
+  "limited": false,
+  "results": [
+    {
+      "avatar_url": "http://domain.tld/path/to/avatar.png",
+      "display_name": "John Doe",
+      "user_id": "UserIdLocalpart"
+    },
+    {
+      ...
+    }
+  ]
+}
+```
+
+If no user found:
+```
+{
+  "limited": false,
+  "results": []
+}
+```
+
+### Identity
+#### Single 3PID lookup
 HTTP method: `POST`  
-Encoding: JSON UTF-8  
+Content-type: JSON UTF-8  
   
 #### Request Body
 ```
@@ -122,11 +161,9 @@ If no match was found:
 {}
 ```
 
-#### Bulk
-Configured with `rest.endpoints.identity.bulk`
-
+#### Bulk 3PID lookup
 HTTP method: `POST`  
-Encoding: JSON UTF-8  
+Content-type: JSON UTF-8  
   
 #### Request Body
 ```
@@ -175,4 +212,4 @@ If no match was found:
 {
   "lookup": []
 }
-```
+```

docs/backends/sql.md

@@ -0,0 +1,23 @@
+# SQL Backend
+## Configuration
+To be completed. For now, see default structure and values:
+```
+sql:
+  enabled: false
+  type: 'sqlite' or 'postgresql'
+  connection: ''
+  auth:
+    enabled: false
+  directory:
+    enabled: false
+    query:
+      name:
+        type: 'localpart'
+        value: 'SELECT 1'
+      threepid:
+        type: 'localpart'
+        value: 'SELECT 1'
+  identity:
+    type: 'localpart'
+    query: 'SELECT user_id AS uid FROM user_threepids WHERE medium = ? AND address = ?'
+```

docs/build.md

@@ -0,0 +1,105 @@
+# From source
+- [Binaries](#binaries)
+- [Debian package](#debian-package)
+- [Docker image](#docker-image)
+
+## Binaries
+### Requirements
+- JDK 1.8
+
+### Build
+```
+git clone https://github.com/kamax-io/mxisd.git
+cd mxisd
+./gradlew build
+```
+
+Create a new configuration file by coping `application.example.yaml` to `application.yaml` and edit to your needs.  
+For advanced configuration, see the [Configure section](configure.md).
+
+Start the server in foreground to validate the build and configuration:
+```
+java -jar build/libs/mxisd.jar
+```
+
+Ensure the signing key is available:
+```
+$ curl 'http://localhost:8090/_matrix/identity/api/v1/pubkey/ed25519:0'
+{"public_key":"..."}
+```
+
+Test basic recursive lookup (requires Internet connection with access to TCP 443):
+```
+$ curl 'http://localhost:8090/_matrix/identity/api/v1/lookup?medium=email&address=mxisd-lookup-test@kamax.io'
+{"address":"mxisd-lookup-test@kamax.io","medium":"email","mxid":"@mxisd-lookup-test:kamax.io",...}
+```
+
+If you enabled LDAP, you can also validate your config with a similar request after replacing the `address` value with
+something present within your LDAP
+```
+curl 'http://localhost:8090/_matrix/identity/api/v1/lookup?medium=email&address=john.doe@example.org'
+```
+
+If you plan on testing the integration with a homeserver, you will need to run an HTTPS reverse proxy in front of it
+as the reference Home Server implementation [synapse](https://github.com/matrix-org/synapse) requires a HTTPS connection
+to an ID server.  
+See the [Integration section](#integration) for more details.
+
+### Install
+1. Prepare files and directories:
+```
+# Create a dedicated user
+useradd -r mxisd
+
+# Create bin directory
+mkdir /opt/mxisd
+
+# Create config directory and set ownership
+mkdir /etc/opt/mxisd
+chown mxisd /etc/opt/mxisd
+
+# Create data directory and set ownership
+mkdir /var/opt/mxisd
+chown mxisd /var/opt/mxisd
+
+# Copy <repo root>/build/libs/mxisd.jar to bin directory
+cp ./build/libs/mxisd.jar /opt/mxisd/
+chown mxisd /opt/mxisd/mxisd.jar
+chmod a+x /opt/mxisd/mxisd.jar
+
+# Create symlink for easy exec
+ln -s /opt/mxisd/mxisd.jar /usr/bin/mxisd
+```
+
+2. Copy the sample config file `./application.example.yaml` to `/etc/opt/mxisd/mxisd.yaml`, edit to your needs
+4. Copy `<repo root>/src/systemd/mxisd.service` to `/etc/systemd/system/` and edit if needed
+5. Enable service for auto-startup
+```
+systemctl enable mxisd
+```
+6. Start mxisd
+```
+systemctl start mxisd
+```
+
+## Debian package
+### Requirements
+- fakeroot
+- dpkg-deb
+
+### Build
+[Build mxisd](#build) then:
+```
+./gradlew buildDeb 
+```
+You will find the debian package in `build/dist`
+
+## Docker image
+[Build mxisd](#build) then:
+```
+./gradlew dockerBuild
+```
+You can run a container of the given image and test it with the following command (adapt volumes host paths):
+```
+docker run -v /data/mxisd/etc:/etc/mxisd -v /data/mxisd/var:/var/mxisd -p 8090:8090 -t kamax/mxisd:latest-dev
+```

docs/configure.md

@@ -0,0 +1,183 @@
+# Configuration
+- [Syntax](#syntax)
+- [Variables](#variables)
+- [Categories](#categories)
+
+## Syntax
+The configuration file is YAML based, allowing two types of syntax.
+
+Properties-like:
+```
+my.config.item: 'value'
+```
+
+Object-like:
+```
+my:
+  config:
+    item: 'value'
+
+```
+These can also be combined within the same file.  
+Both syntax will be used interchangeably in these documents.
+
+Default values for each possible option are documented [here](../src/main/resources/application.yaml)
+
+## Variables
+It is possible to copy the value of a configuration item into another using the syntax `${config.key.item}`.  
+Example that will copy the value of `matrix.domain` into `server.name`:
+```
+matrix:
+  domain: 'example.org'
+
+server:
+  name: '${matrix.domain}'
+```
+
+**WARNING:** mxisd might overwrite/adapt some values during launch. Those changes will not be reflected into copied keys.
+
+## Categories
+For each category below, the base configuration path will be given, which needs to be appened to every configuration
+item described.
+
+Example: if the base path was `basePath` and the following table was given:
+
+| Name | Purpose |
+|------|---------|
+| item1 | To give an example |
+| item2 | To give another example |
+
+The following configurations could be used, all being equivalent:
+```
+basePath.item1: 'myValue'
+basePath.item2: 'myOtherValue'
+```
+```
+basePath:
+  item1: 'myValue'
+  item2: 'myOtherValue'
+```
+```
+basePath.item1: 'myValue'
+basePath:
+  item2: 'myOtherValue'
+```
+
+---
+
+In case a relative base path is given, it is appended to the one above.
+
+Example: With base path `basePath`, the relative base `relativeBasePath` and the following table:
+  
+| Name | Purpose |
+|------|---------|
+| item1 | To give an example |
+| item2 | To give another example |
+
+The following configurations could be used, all being equivalent:
+```
+basePath.relativeBasePath.item1: 'myValue'
+basePath.relativeBasePath.item2: 'myOtherValue'
+```
+```
+basePath:
+  relativeBasePath:
+    item1: 'myValue'
+    item2: 'myOtherValue'
+```
+```
+basePath.relativeBasePath.item1: 'myValue'
+basePath:
+  relativeBasePath:
+    item2: 'myOtherValue'
+```
+
+### Matrix
+Base path: `matrix`
+
+| Name | Purpose |
+|------|---------|
+| `domain` | Matrix domain name, same as the Homeserver, used to build appropriate Matrix IDs |
+
+---
+
+Relative base path: `identity`
+
+| Name | Purpose |
+|------|---------|
+| `servers` | Namespace to create arbitrary list of Identity servers, usable in other parts of the configuration |
+
+Example:
+```
+matrix.identity.servers:
+  root:
+    - 'https://matrix.org'
+```
+Create a list under the label `root` containing a single Identity server, `https://matrix.org`
+### Server
+| Name | Purpose |
+|------|---------|
+| `name` | Public hostname of mxisd, if different from the Matrix domain |
+| `port` | HTTP port to listen on (unencrypted) |
+| `publicUrl` | Defaults to `https://${server.name}` |
+
+### Storage
+Base path: `storage`
+
+| Name | Purpose |
+|------|---------|
+| `backend` | Specify which SQL backend to use. only `sqlite` is currently supported. |
+
+---
+Relative base path: `provider.sqlite`
+
+| Name | Purpose |
+|------|---------|
+| `database` | Absolute location of the SQLite database |
+
+### Backends
+- [LDAP](backends/ldap.md)
+- [SQL](backends/sql.md)
+- [REST](backends/rest.md)
+- [Google Firebase](backends/firebase.md)
+
+### Lookups
+work in progress, should not be configured.
+
+### Sessions
+See the [dedicated document](sessions/3pid.md)
+
+### Notifications
+Base path: `notification`
+
+| Name | Purpose |
+|------|---------|
+| handler | Namespace to specify the handler to use for each 3PID |
+| handlers | Namespace used by individual handlers for their own configuration |
+
+Example:
+```
+notification:
+  handler:
+    email: 'sendgrid'
+    msisdn: 'raw'
+  handlers:
+    raw:
+      ...
+    sendgrid:
+      ...
+```
+- Emails notifications would use the `sendgrid` handler, which define its own configuration user `handlers.sendgrid`
+- Phone notification would use the `raw` handler, basic default built-in handler of mxisd
+#### Handlers
+Relative base path: `handlers`
+
+Built-in:
+- [Basic](threepids/notifications/basic-handler.md)
+- [SendGrid](threepids/notifications/sendgrid-handler.md)
+
+### Views
+See the [dedicated document](sessions/3pid-views.md)
+
+### DNS Overwite
+Specific to other features.

docs/features/authentication.md

@@ -0,0 +1,28 @@
+# Authentication
+Performed via [synapse with REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth/blob/master/README.md)  
+Point the `endpoint` to mxisd internal IP on port 8090
+
+## Overview
+```
+                                                                                    Backends
+ Client                                                                             +------+
+   |                                            +-------------------------+    +--> | LDAP |
+   |   +---------------+  /_matrix/identity     | mxisd                   |    |    +------+
+   +-> | Reverse proxy | >------------------+   |                         |    |
+       +--|------------+                    |   |                         |    |    +--------+
+          |                                 +-----> Check wiht backends >------+--> | SQL DB |
+     Login request                          |   |                         |    |    +--------+
+          |                                 |   |     |                   |    |
+          |   +--------------------------+  |   +-----|-------------------+    +-->  Others
+          +-> | Homeserver               |  |         |
+              |                          |  |         |
+              | - Validate credentials >----+         |
+              |   Using REST auth module |            |
+              |                          |            |
+              | - Auto-provision <-------------------<+
+              |   user profiles          |    If valid credentials and supported by backend
+              +--------------------------+
+```
+
+## Profile auto-fill
+To be documented

docs/features/bridge-integration.md

@@ -0,0 +1 @@
+To be documented

docs/features/directory-users.md

@@ -0,0 +1,167 @@
+# User Directory
+- [Description](#description)
+- [Overview](#overview)
+- [Requirements](#requirements)
+- [Configuration](#configuration)
+  - [Reverse Proxy](#reverse-proxy)
+  - [DNS Overwrite](#dns-overwrite)
+  - [Backends](#backends)
+    - [LDAP](#ldap)
+    - [SQL](#sql)
+    - [REST](#rest)
+
+## Description
+This feature allows you to search for existing and/or potential users that are already present in your Identity backend
+or that already share a room with you on the Homeserver.
+
+Without any integration, synapse:
+- Only search within the users **already** known to you
+- Only search on the Display Name and the Matrix ID
+
+With mxisd integration, you can:
+- Search on Matrix ID, Display name and 3PIDs (Email, phone numbers) of any users already in your configured backend
+- Search for users which you are not in contact with yet. Super useful for corporations who want to give Matrix access
+internally, so users can just find themselves **prior** to having any common room(s)
+- Use any attribute of your backend to extend the search!
+- Include your homeserver search results to those found by mxisd (default behaviour, no configuration required)
+
+By integrating mxisd, you get the default behaviour with all the extras, ensuring your users will always find each other.
+
+## Overview
+This is performed by intercepting the Homeserver endpoint `/_matrix/client/r0/user_directory/search` like so:
+```
+           +----------------------------------------------+
+Client --> | Reverse proxy                                                                         Step 2
+           |                                              Step 1    +-------------------------+
+           |   /_matrix/client/r0/user_directory/search ----------> |                         |  Search in   +---------+
+           |                        /\                              | mxisd - Identity server | -----------> | Backend |
+           |   /_matrix/*            \----------------------------- |                         |  all users   +---------+
+           |        |            Step 4: Send back merged results   +-------------------------+
+           +        |                                                            |
+                    |                                                          Step 3
+                    |                                                            |
+                    |    +------------+                                Search in known users
+                    \--> | Homeserver | <----------------------------------------/
+                         +------------+   /_matrix/client/r0/user_directory/search
+```
+Steps:
+1. The intercepted request is directly sent to mxisd instead of the Homeserver.
+2. Enabled backends are queried for any match on the search value sent by the client.
+3. The Homeserver, from which the request was intercepted, is queried using the same request as the client.
+Its address is resolved using the DNS Overwrite feature to reach its internal address on a non-encrypted port.
+4. Results from backends and the Homeserver are merged together and sent back to the client, believing it was the HS
+which directly answered the request.
+
+## Requirements
+- Reverse proxy setup, which you should already have in place if you use mxisd
+- Compatible backends:
+  - LDAP
+  - SQL
+  - REST
+  
+## Configuration
+### Reverse Proxy
+Apache2 configuration to put under the relevant virtual domain:
+```
+ProxyPreserveHost on
+ProxyPass /_matrix/identity/ http://mxisdInternalIpAddress:8090/_matrix/identity/
+ProxyPass /_matrix/client/r0/user_directory/ http://mxisdInternalIpAddress:8090/_matrix/client/r0/user_directory/
+ProxyPass /_matrix/ http://HomeserverInternalIpAddress:8008/_matrix/
+```
+`ProxyPreserveHost` or equivalent must be enabled to detect to which Homeserver mxisd should talk to when building
+results.
+
+### DNS Overwrite
+Just like you need to configure a reverse proxy to send client requests to mxisd, you also need to configure mxisd with
+the internal IP of the Homeserver so it can talk to it directly to integrate its directory search.
+
+To do so, use the following configuration:
+```
+dns.overwrite.homeserver.client:
+  - name: 'example.org'
+    value: 'http://localhost:8008'
+```
+`name` must be the hostname of the URL that clients use when connecting to the Homeserver.  
+In case the hostname is the same as your Matrix domain, you can use `${matrix.domain}` to auto-populate the value using
+the `matrix.domain` configuration option and avoid duplicating it.
+
+`value` is the base intenral URL of the Homeserver, without any `/_matrix/..` or trailing `/`.
+
+### Backends
+#### LDAP
+To ensure Directory feature works, here's how the LDAP configuration should look like:
+```
+ldap:
+  enabled: false
+  filter: '(memberOf=CN=Matrix Users,OU=Groups,DC=example,DC=org)'
+  connection:
+    host: 'ldapIpOrDomain'
+    bindDn: 'CN=Matrix Identity Server,OU=Accounts,DC=example,DC=org'
+    bindPassword: 'mxisd'
+    baseDn: 'OU=Accounts,DC=example,DC=org'
+  attribute:
+    uid:
+      type: 'uid'
+      value: 'userPrincipalName'
+    name: 'displayName'
+    threepid:
+      email:
+        - 'mailPrimaryAddress'
+        - 'mail'
+        - 'otherMailbox'
+      msisdn:
+        - 'telephoneNumber'
+        - 'mobile'
+        - 'homePhone'
+        - 'otherTelephone'
+        - 'otherMobile'
+        - 'otherHomePhone'
+  directory:
+    attribute:
+      other:
+        - 'employeeNumber'
+        - 'someOtherAttribute'
+```
+Only include the `attribute` sub-sections if you would like to set another value. Else, it is best not to include them
+to inherit the default values.
+
+If you would like to include an attribute which is not a display name or a 3PID, you can use the
+`directory.attribute.other` to list any extra attributes you want included in searches. If you do not want to include
+any extra attribute, that configuration section can be skipped. 
+
+#### SQL
+If you plan to integrate directory search directly with synapse, use the `synapseSql` provider, based on the following
+config:
+```
+synapseSql:
+  enabled: true
+  type: <database ID>
+  connection: '<connection info>'
+```
+`type` and `connection`, including any other configuration item, follow the same values as the regular [SQL backend](../backends/sql.md).
+
+---
+
+For the regular SQL backend, the following configuration items are available:
+```
+sql:
+  directory:
+    enabled: true
+    query:
+      name:
+        type: 'localpart'
+        value: 'SELECT idColumn, displayNameColumn FROM table WHERE displayNameColumn LIKE ?'
+      threepid:
+        type: 'localpart'
+        value: 'SELECT idColumn, displayNameColumn FROM table WHERE threepidColumn LIKE ?'
+```
+For each query, `type` can be used to tell mxisd how to process the ID column:
+- `localpart` will append the `matrix.domain` to it
+- `mxid` will use the ID as-is. If it is not a valid Matrix ID, the search will fail.
+
+`value` is the SQL query and must return two columns:
+- The first being the User ID
+- The second being its display name
+
+#### REST
+See the [dedicated document](../backends/rest.md)

docs/features/federation.md

@@ -0,0 +1,33 @@
+# Identity service Federation
+## Overview
+```
+              +-------------------+   +-------------> +----------+
+              | mxisd             |   |               | Backends |
+              |                   |   |      +------> +----------+
+              |                   |   |      |
+              | Invites / Lookups |   |      |
+ Federated    | +--------+        |   |      |        +-------------------+
+ Identity  ---->| Remote |>-----------+      +------> | Remote Federated  |
+ Server       | +--------+        |          |        | mxisd servers     |
+              |                   |          |        +-------------------+
+              | +--------+        |          |
+ Homeserver --->| Local  |>------------------+
+ and clients  | +--------+        |          |        +--------------------------+ 
+              +-------------------+          +------> | Central Identity service |
+                                                      | Matrix.org / Vector.im   |
+                                                      +--------------------------+
+```
+To allow other federated Identity Server to reach yours, the same algorithm used for Homeservers takes place:
+1. Check for the appropriate DNS SRV record
+2. If not found, use the base domain
+
+## Configuration
+If your Identity Server public hostname does not match your Matrix domain, configure the following DNS SRV entry 
+and replace `matrix.example.com` by your Identity server public hostname - **Make sure to end with a final dot!**
+```
+_matrix-identity._tcp.example.com. 3600 IN SRV 10 0 443 matrix.example.com.
+``` 
+This would only apply for 3PID that are DNS-based, like e-mails. For anything else, like phone numbers, no federation 
+is currently possible.  
+
+The port must be HTTPS capable which is what you get in a regular setup with a reverse proxy from 443 to TCP 8090 of mxisd.

docs/features/identity.md

@@ -0,0 +1,3 @@
+To be documented.
+
+Implementation of the [Matrix Identity service API](https://matrix.org/docs/spec/identity_service/unstable.html)

docs/install/debian.md

@@ -0,0 +1,39 @@
+# Debian package
+## Install
+1. Donwload the [latest release](https://github.com/kamax-io/mxisd/releases/latest)
+2. Run:
+```
+dpkg -i /path/to/downloaded/mxisd.deb
+```
+## Files
+| Location                          | Purpose                                      |
+|-----------------------------------|----------------------------------------------|
+| /etc/mxisd                        | Configuration directory                      |
+| /etc/mxisd/mxisd.yaml             | Main configuration file                      |
+| /etc/mxisd/signing.key            | Default location for mxisd signing keys      |
+| /etc/systemd/system/mxisd.service | Systemd configuration file for mxisd service |
+| /usr/lib/mxisd                    | Binairies                                    |
+| /var/lib/mxisd                    | Data                                         |
+
+## Control
+Start mxisd using:
+```
+sudo systemctl start mxisd
+```
+
+Stop mxisd using:
+```
+sudo systemctl stop mxisd
+```
+
+## Troubleshoot
+All logs are sent to `STDOUT` which are saved in `/var/log/syslog` by default.  
+You can:
+- grep & tail using `mxisd`:
+```
+tail -n 99 -f /var/log/syslog | grep mxisd
+```
+- use Systemd's journal:
+```
+journalctl -f n 99 -u mxisd
+```

docs/install/docker.md

@@ -0,0 +1,14 @@
+# Docker
+## Fetch
+Pull the latest stable image:
+```
+docker pull kamax/mxisd
+```
+
+## Run
+Run it (adapt volume paths to your host):
+```
+docker run --rm -v /data/mxisd/etc:/etc/mxisd -v /data/mxisd/var:/var/mxisd -p 8090:8090 -t kamax/mxisd
+```
+
+For more info, including the list of possible tags, see [the public repository](https://hub.docker.com/r/kamax/mxisd/)

docs/sessions/3pid.md

@@ -17,7 +17,7 @@
     - [Sessions disabled](#sessions-disabled)
 
 ## Overview
-When adding an email, a phone number or any other kind of 3PID (Third-Party Identifier), 
+When adding an email, a phone number or any other kind of 3PID (Third-Party Identifier) in a Matrix client, 
 the identity server is called to validate the 3PID.
 
 Once this 3PID is validated, the Homeserver will publish the user Matrix ID on the Identity Server and

docs/threepids/notifications/basic-handler.md

@@ -0,0 +1,66 @@
+# Basic Notification handler
+Basic notification handler which uses two components:
+- Content generator, to produce the notifications
+- Connectors to send the notification content
+
+This handler can be used with the 3PID types:
+- `email`
+- `msisdn` (Phone numbers)
+
+## Generators
+- [Template](template-generator.md)
+## Connectors
+- Email
+  - [SMTP](../medium/email/smtp-connector.md)
+- SMS
+  - [Twilio](../medium/msisdn/twilio-connector.md)
+
+## Configuration
+Enabled by default or with:
+```
+notification:
+  handler:
+    email: 'raw'
+```
+
+**WARNING:** Will be consolidated soon, prone to breaking changes.  
+Structure and default values:
+```
+threepid:
+  medium:
+    email:
+      identity:
+        from: ''
+        name: ''
+      connector: 'smtp'
+      generator: 'template'
+      connectors:
+        smtp:
+          host: ''
+          port: 587
+          tls: 1
+          login: ''
+          password: ''
+      generators:
+        template:
+          invite: 'classpath:threepids/email/invite-template.eml'
+          session:
+            validation:
+              local: 'classpath:threepids/email/validate-local-template.eml'
+              remote: 'classpath:threepids/email/validate-remote-template.eml'
+    msisdn:
+      connector: 'twilio'
+      generator: 'template'
+      connectors:
+        twilio:
+          accountSid: ''
+          authToken: ''
+          number: ''
+      generators:
+        template:
+          invite: 'classpath:threepids/sms/invite-template.txt'
+          session:
+            validation:
+              local: 'classpath:threepids/sms/validate-local-template.txt'
+              remote: 'classpath:threepids/sms/validate-remote-template.txt'
+```

docs/threepids/notifications/sendgrid-handler.md

@@ -0,0 +1,9 @@
+# SendGrid Notification handler
+To be completed. See [raw possible configuration items](https://github.com/kamax-io/mxisd/blob/master/src/main/resources/application.yaml#L172).
+
+Enabled with:
+```
+notification:
+  handler:
+    email: 'sendgrid'
+```

docs/threepids/notifications/template-generator.md

@@ -64,7 +64,7 @@ This template is used when to user which added their 3PID address to their profi
 allows remote sessions.
 
 **NOTE:** 3PID session always require local validation of a token, even if a remote session is enforced.  
-One cannot bind a MXID to the session until both local and remote sessions have been validated.
+One cannot bind a Matrix ID to the session until both local and remote sessions have been validated.
 
 #### Placeholders
 | Placeholder          | Purpose                                                |

src/main/java/io/kamax/mxisd/ThreePid.java

@@ -48,4 +48,22 @@ public class ThreePid {
         return getMedium() + ":" + getAddress();
     }
 
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+
+        ThreePid threePid = (ThreePid) o;
+
+        if (!medium.equals(threePid.medium)) return false;
+        return address.equals(threePid.address);
+    }
+
+    @Override
+    public int hashCode() {
+        int result = medium.hashCode();
+        result = 31 * result + address.hashCode();
+        return result;
+    }
+
 }

src/main/java/io/kamax/mxisd/UserIdType.java

@@ -28,7 +28,7 @@ public enum UserIdType {
     Localpart("localpart"),
     MatrixID("mxid"),
     EmailLocalpart("email_localpart"),
-    Email("threepids/email");
+    Email("email");
 
     private String id;
 

src/main/java/io/kamax/mxisd/auth/AuthManager.java

@@ -71,14 +71,14 @@ public class AuthManager {
                     continue;
                 }
 
-                UserAuthResult authResult = new UserAuthResult().success(mxId, result.getProfile().getDisplayName());
+                UserAuthResult authResult = new UserAuthResult().success(result.getProfile().getDisplayName());
                 for (ThreePid pid : result.getProfile().getThreePids()) {
                     authResult.withThreePid(pid.getMedium(), pid.getAddress());
                 }
                 log.info("{} was authenticated by {}, publishing 3PID mappings, if any", id, provider.getClass().getSimpleName());
                 for (ThreePid pid : authResult.getThreePids()) {
                     log.info("Processing {} for {}", pid, id);
-                    invMgr.publishMappingIfInvited(new ThreePidMapping(pid, authResult.getMxid()));
+                    invMgr.publishMappingIfInvited(new ThreePidMapping(pid, mxId));
                 }
 
                 invMgr.lookupMappingsForInvites();

src/main/java/io/kamax/mxisd/auth/UserAuthResult.java

@@ -20,31 +20,30 @@
 
 package io.kamax.mxisd.auth;
 
-import io.kamax.matrix.ThreePidMedium;
 import io.kamax.mxisd.ThreePid;
 
-import java.util.ArrayList;
 import java.util.Collections;
-import java.util.List;
+import java.util.HashSet;
+import java.util.Set;
 
 public class UserAuthResult {
 
     private boolean success;
-    private String mxid;
     private String displayName;
-    private List<ThreePid> threePids = new ArrayList<>();
+    private String photo;
+    private Set<ThreePid> threePids = new HashSet<>();
 
     public UserAuthResult failure() {
         success = false;
-        mxid = null;
         displayName = null;
+        photo = null;
+        threePids.clear();
 
         return this;
     }
 
-    public UserAuthResult success(String mxid, String displayName) {
+    public UserAuthResult success(String displayName) {
         setSuccess(true);
-        setMxid(mxid);
         setDisplayName(displayName);
 
         return this;
@@ -58,14 +57,6 @@ public class UserAuthResult {
         this.success = success;
     }
 
-    public String getMxid() {
-        return mxid;
-    }
-
-    public void setMxid(String mxid) {
-        this.mxid = mxid;
-    }
-
     public String getDisplayName() {
         return displayName;
     }
@@ -74,8 +65,12 @@ public class UserAuthResult {
         this.displayName = displayName;
     }
 
-    public UserAuthResult withThreePid(ThreePidMedium medium, String address) {
-        return withThreePid(medium.getId(), address);
+    public String getPhoto() {
+        return photo;
+    }
+
+    public void setPhoto(String photo) {
+        this.photo = photo;
     }
 
     public UserAuthResult withThreePid(String medium, String address) {
@@ -84,8 +79,8 @@ public class UserAuthResult {
         return this;
     }
 
-    public List<ThreePid> getThreePids() {
-        return Collections.unmodifiableList(threePids);
+    public Set<ThreePid> getThreePids() {
+        return Collections.unmodifiableSet(threePids);
     }
 
 }

src/main/java/io/kamax/mxisd/auth/provider/BackendAuthResult.java

@@ -24,21 +24,21 @@ import io.kamax.mxisd.ThreePid;
 import io.kamax.mxisd.UserID;
 import io.kamax.mxisd.UserIdType;
 
-import java.util.ArrayList;
-import java.util.List;
+import java.util.HashSet;
+import java.util.Set;
 
 public class BackendAuthResult {
 
     public static class BackendAuthProfile {
 
         private String displayName;
-        private List<ThreePid> threePids = new ArrayList<>();
+        private Set<ThreePid> threePids = new HashSet<>();
 
         public String getDisplayName() {
             return displayName;
         }
 
-        public List<ThreePid> getThreePids() {
+        public Set<ThreePid> getThreePids() {
             return threePids;
         }
     }

src/main/java/io/kamax/mxisd/backend/firebase/GoogleFirebaseAuthenticator.java

@@ -20,11 +20,9 @@
 
 package io.kamax.mxisd.backend.firebase;
 
-import com.google.firebase.FirebaseApp;
-import com.google.firebase.FirebaseOptions;
-import com.google.firebase.auth.FirebaseAuth;
-import com.google.firebase.auth.FirebaseCredential;
-import com.google.firebase.auth.FirebaseCredentials;
+import com.google.firebase.auth.UserInfo;
+import com.google.i18n.phonenumbers.NumberParseException;
+import com.google.i18n.phonenumbers.PhoneNumberUtil;
 import io.kamax.matrix.ThreePidMedium;
 import io.kamax.matrix._MatrixID;
 import io.kamax.mxisd.ThreePid;
@@ -35,18 +33,18 @@ import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import java.io.FileInputStream;
-import java.io.IOException;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 
-public class GoogleFirebaseAuthenticator implements AuthenticatorProvider {
+public class GoogleFirebaseAuthenticator extends GoogleFirebaseBackend implements AuthenticatorProvider {
 
     private Logger log = LoggerFactory.getLogger(GoogleFirebaseAuthenticator.class);
 
-    private boolean isEnabled;
-    private FirebaseApp fbApp;
-    private FirebaseAuth fbAuth;
+    private PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
+
+    public GoogleFirebaseAuthenticator(boolean isEnabled, String credsPath, String db) {
+        super(isEnabled, "AuthenticationProvider", credsPath, db);
+    }
 
     private void waitOnLatch(BackendAuthResult result, CountDownLatch l, String purpose) {
         try {
@@ -57,46 +55,33 @@ public class GoogleFirebaseAuthenticator implements AuthenticatorProvider {
         }
     }
 
-    public GoogleFirebaseAuthenticator(boolean isEnabled) {
-        this.isEnabled = isEnabled;
+    private void toEmail(BackendAuthResult result, String email) {
+        if (StringUtils.isBlank(email)) {
+            return;
+        }
+
+        result.withThreePid(new ThreePid(ThreePidMedium.Email.getId(), email));
     }
 
-    public GoogleFirebaseAuthenticator(String credsPath, String db) {
-        this(true);
+    private void toMsisdn(BackendAuthResult result, String phoneNumber) {
+        if (StringUtils.isBlank(phoneNumber)) {
+            return;
+        }
+
         try {
-            fbApp = FirebaseApp.initializeApp(getOpts(credsPath, db), "AuthenticationProvider");
-            fbAuth = FirebaseAuth.getInstance(fbApp);
-
-            log.info("Google Firebase Authentication is ready");
-        } catch (IOException e) {
-            throw new RuntimeException("Error when initializing Firebase", e);
+            String number = phoneUtil.format(
+                    phoneUtil.parse(
+                            phoneNumber,
+                            null // No default region
+                    ),
+                    PhoneNumberUtil.PhoneNumberFormat.E164
+            ).substring(1); // We want without the leading +
+            result.withThreePid(new ThreePid(ThreePidMedium.PhoneNumber.getId(), number));
+        } catch (NumberParseException e) {
+            log.warn("Invalid phone number: {}", phoneNumber);
         }
     }
 
-    private FirebaseCredential getCreds(String credsPath) throws IOException {
-        if (StringUtils.isNotBlank(credsPath)) {
-            return FirebaseCredentials.fromCertificate(new FileInputStream(credsPath));
-        } else {
-            return FirebaseCredentials.applicationDefault();
-        }
-    }
-
-    private FirebaseOptions getOpts(String credsPath, String db) throws IOException {
-        if (StringUtils.isBlank(db)) {
-            throw new IllegalArgumentException("Firebase database is not configured");
-        }
-
-        return new FirebaseOptions.Builder()
-                .setCredential(getCreds(credsPath))
-                .setDatabaseUrl(db)
-                .build();
-    }
-
-    @Override
-    public boolean isEnabled() {
-        return isEnabled;
-    }
-
     private void waitOnLatch(CountDownLatch l) {
         try {
             l.await(30, TimeUnit.SECONDS);
@@ -117,7 +102,7 @@ public class GoogleFirebaseAuthenticator implements AuthenticatorProvider {
 
         String localpart = mxid.getLocalPart();
         CountDownLatch l = new CountDownLatch(1);
-        fbAuth.verifyIdToken(password).addOnSuccessListener(token -> {
+        getFirebase().verifyIdToken(password).addOnSuccessListener(token -> {
             try {
                 if (!StringUtils.equals(localpart, token.getUid())) {
                     log.info("Failure to authenticate {}: Matrix ID localpart '{}' does not match Firebase UID '{}'", mxid, localpart, token.getUid());
@@ -129,16 +114,17 @@ public class GoogleFirebaseAuthenticator implements AuthenticatorProvider {
                 log.info("{} was successfully authenticated", mxid);
                 log.info("Fetching profile for {}", mxid);
                 CountDownLatch userRecordLatch = new CountDownLatch(1);
-                fbAuth.getUser(token.getUid()).addOnSuccessListener(user -> {
+                getFirebase().getUser(token.getUid()).addOnSuccessListener(user -> {
                     try {
-                        if (StringUtils.isNotBlank(user.getEmail())) {
-                            result.withThreePid(new ThreePid(ThreePidMedium.Email.getId(), user.getEmail()));
-                        }
-
-                        if (StringUtils.isNotBlank(user.getPhoneNumber())) {
-                            result.withThreePid(new ThreePid(ThreePidMedium.PhoneNumber.getId(), user.getPhoneNumber()));
+                        toEmail(result, user.getEmail());
+                        toMsisdn(result, user.getPhoneNumber());
+
+                        for (UserInfo info : user.getProviderData()) {
+                            toEmail(result, info.getEmail());
+                            toMsisdn(result, info.getPhoneNumber());
                         }
 
+                        log.info("Got {} 3PIDs in profile", result.getProfile().getThreePids().size());
                     } finally {
                         userRecordLatch.countDown();
                     }

src/main/java/io/kamax/mxisd/backend/firebase/GoogleFirebaseBackend.java

@@ -0,0 +1,88 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.firebase;
+
+import com.google.firebase.FirebaseApp;
+import com.google.firebase.FirebaseOptions;
+import com.google.firebase.auth.FirebaseAuth;
+import com.google.firebase.auth.FirebaseCredential;
+import com.google.firebase.auth.FirebaseCredentials;
+import com.google.firebase.database.FirebaseDatabase;
+import org.apache.commons.lang.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+
+public class GoogleFirebaseBackend {
+
+    private Logger log = LoggerFactory.getLogger(GoogleFirebaseBackend.class);
+
+    private boolean isEnabled;
+    private FirebaseAuth fbAuth;
+    protected FirebaseDatabase fbDb;
+
+    GoogleFirebaseBackend(boolean isEnabled, String name, String credsPath, String db) {
+        this.isEnabled = isEnabled;
+        if (!isEnabled) {
+            return;
+        }
+
+        try {
+            FirebaseApp fbApp = FirebaseApp.initializeApp(getOpts(credsPath, db), name);
+            fbAuth = FirebaseAuth.getInstance(fbApp);
+            FirebaseDatabase.getInstance(fbApp);
+
+            log.info("Google Firebase Authentication is ready");
+        } catch (IOException e) {
+            throw new RuntimeException("Error when initializing Firebase", e);
+        }
+    }
+
+    private FirebaseCredential getCreds(String credsPath) throws IOException {
+        if (StringUtils.isNotBlank(credsPath)) {
+            return FirebaseCredentials.fromCertificate(new FileInputStream(credsPath));
+        } else {
+            return FirebaseCredentials.applicationDefault();
+        }
+    }
+
+    private FirebaseOptions getOpts(String credsPath, String db) throws IOException {
+        if (StringUtils.isBlank(db)) {
+            throw new IllegalArgumentException("Firebase database is not configured");
+        }
+
+        return new FirebaseOptions.Builder()
+                .setCredential(getCreds(credsPath))
+                .setDatabaseUrl(db)
+                .build();
+    }
+
+    FirebaseAuth getFirebase() {
+        return fbAuth;
+    }
+
+    public boolean isEnabled() {
+        return isEnabled;
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/firebase/GoogleFirebaseProvider.java

@@ -20,11 +20,6 @@
 
 package io.kamax.mxisd.backend.firebase;
 
-import com.google.firebase.FirebaseApp;
-import com.google.firebase.FirebaseOptions;
-import com.google.firebase.auth.FirebaseAuth;
-import com.google.firebase.auth.FirebaseCredential;
-import com.google.firebase.auth.FirebaseCredentials;
 import com.google.firebase.auth.UserRecord;
 import com.google.firebase.tasks.OnFailureListener;
 import com.google.firebase.tasks.OnSuccessListener;
@@ -34,72 +29,29 @@ import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.SingleLookupRequest;
 import io.kamax.mxisd.lookup.ThreePidMapping;
 import io.kamax.mxisd.lookup.provider.IThreePidProvider;
-import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import java.io.FileInputStream;
-import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Optional;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 
-public class GoogleFirebaseProvider implements IThreePidProvider {
+public class GoogleFirebaseProvider extends GoogleFirebaseBackend implements IThreePidProvider {
 
     private Logger log = LoggerFactory.getLogger(GoogleFirebaseProvider.class);
-
-    private boolean isEnabled;
     private String domain;
-    private FirebaseAuth fbAuth;
 
-    public GoogleFirebaseProvider(boolean isEnabled) {
-        this.isEnabled = isEnabled;
-    }
-
-    public GoogleFirebaseProvider(String credsPath, String db, String domain) {
-        this(true);
+    public GoogleFirebaseProvider(boolean isEnabled, String credsPath, String db, String domain) {
+        super(isEnabled, "ThreePidProvider", credsPath, db);
         this.domain = domain;
-
-        try {
-            FirebaseApp fbApp = FirebaseApp.initializeApp(getOpts(credsPath, db), "ThreePidProvider");
-            fbAuth = FirebaseAuth.getInstance(fbApp);
-
-            log.info("Google Firebase Authentication is ready");
-        } catch (IOException e) {
-            throw new RuntimeException("Error when initializing Firebase", e);
-        }
-    }
-
-    private FirebaseCredential getCreds(String credsPath) throws IOException {
-        if (StringUtils.isNotBlank(credsPath)) {
-            return FirebaseCredentials.fromCertificate(new FileInputStream(credsPath));
-        } else {
-            return FirebaseCredentials.applicationDefault();
-        }
-    }
-
-    private FirebaseOptions getOpts(String credsPath, String db) throws IOException {
-        if (StringUtils.isBlank(db)) {
-            throw new IllegalArgumentException("Firebase database is not configured");
-        }
-
-        return new FirebaseOptions.Builder()
-                .setCredential(getCreds(credsPath))
-                .setDatabaseUrl(db)
-                .build();
     }
 
     private String getMxid(UserRecord record) {
         return new MatrixID(record.getUid(), domain).getId();
     }
 
-    @Override
-    public boolean isEnabled() {
-        return isEnabled;
-    }
-
     @Override
     public boolean isLocal() {
         return true;
@@ -136,13 +88,13 @@ public class GoogleFirebaseProvider implements IThreePidProvider {
 
         if (ThreePidMedium.Email.is(medium)) {
             log.info("Performing E-mail 3PID lookup for {}", address);
-            fbAuth.getUserByEmail(address)
+            getFirebase().getUserByEmail(address)
                     .addOnSuccessListener(success)
                     .addOnFailureListener(failure);
             waitOnLatch(l);
         } else if (ThreePidMedium.PhoneNumber.is(medium)) {
             log.info("Performing msisdn 3PID lookup for {}", address);
-            fbAuth.getUserByPhoneNumber(address)
+            getFirebase().getUserByPhoneNumber(address)
                     .addOnSuccessListener(success)
                     .addOnFailureListener(failure);
             waitOnLatch(l);

src/main/java/io/kamax/mxisd/backend/ldap/LdapAuthProvider.java

@@ -24,6 +24,8 @@ import io.kamax.matrix._MatrixID;
 import io.kamax.mxisd.UserIdType;
 import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
 import io.kamax.mxisd.auth.provider.BackendAuthResult;
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.ldap.LdapConfig;
 import org.apache.commons.lang.StringUtils;
 import org.apache.directory.api.ldap.model.cursor.CursorException;
 import org.apache.directory.api.ldap.model.cursor.CursorLdapReferralException;
@@ -35,6 +37,7 @@ import org.apache.directory.api.ldap.model.message.SearchScope;
 import org.apache.directory.ldap.client.api.LdapConnection;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
 import java.io.IOException;
@@ -44,8 +47,9 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
 
     private Logger log = LoggerFactory.getLogger(LdapAuthProvider.class);
 
-    private String getUidAttribute() {
-        return getCfg().getAttribute().getUid().getValue();
+    @Autowired
+    public LdapAuthProvider(LdapConfig cfg, MatrixConfig mxCfg) {
+        super(cfg, mxCfg);
     }
 
     @Override
@@ -57,37 +61,26 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
     public BackendAuthResult authenticate(_MatrixID mxid, String password) {
         log.info("Performing auth for {}", mxid);
 
-        LdapConnection conn = getConn();
-        try {
+
+        try (LdapConnection conn = getConn()) {
             bind(conn);
 
-            String uidType = getCfg().getAttribute().getUid().getType();
-            String userFilterValue = StringUtils.equals(LdapThreePidProvider.UID, uidType) ? mxid.getLocalPart() : mxid.getId();
+            String uidType = getAt().getUid().getType();
+            String userFilterValue = StringUtils.equals(LdapGenericBackend.UID, uidType) ? mxid.getLocalPart() : mxid.getId();
             if (StringUtils.isBlank(userFilterValue)) {
                 log.warn("Username is empty, failing auth");
                 return BackendAuthResult.failure();
             }
 
-            String userFilter = "(" + getCfg().getAttribute().getUid().getValue() + "=" + userFilterValue + ")";
-            if (!StringUtils.isBlank(getCfg().getAuth().getFilter())) {
-                userFilter = "(&" + getCfg().getAuth().getFilter() + userFilter + ")";
-            }
-            EntryCursor cursor = conn.search(getCfg().getConn().getBaseDn(), userFilter, SearchScope.SUBTREE, getUidAttribute(), getCfg().getAttribute().getName());
-            try {
+            String userFilter = "(" + getUidAtt() + "=" + userFilterValue + ")";
+            userFilter = buildWithFilter(userFilter, getCfg().getAuth().getFilter());
+            try (EntryCursor cursor = conn.search(getBaseDn(), userFilter, SearchScope.SUBTREE, getUidAtt(), getAt().getName())) {
                 while (cursor.next()) {
                     Entry entry = cursor.get();
                     String dn = entry.getDn().getName();
                     log.info("Checking possible match, DN: {}", dn);
 
-                    Attribute attribute = entry.get(getUidAttribute());
-                    if (attribute == null) {
-                        log.info("DN {}: no attribute {}, skpping", dn, getUidAttribute());
-                        continue;
-                    }
-
-                    String data = attribute.get().toString();
-                    if (data.length() < 1) {
-                        log.info("DN {}: empty attribute {}, skipping", getUidAttribute());
+                    if (!getAttribute(entry, getUidAtt()).isPresent()) {
                         continue;
                     }
 
@@ -99,7 +92,7 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
                         return BackendAuthResult.failure();
                     }
 
-                    Attribute nameAttribute = entry.get(getCfg().getAttribute().getName());
+                    Attribute nameAttribute = entry.get(getAt().getName());
                     String name = nameAttribute != null ? nameAttribute.get().toString() : null;
 
                     log.info("Authentication successful for {}", entry.getDn().getName());
@@ -110,20 +103,12 @@ public class LdapAuthProvider extends LdapGenericBackend implements Authenticato
                 }
             } catch (CursorLdapReferralException e) {
                 log.warn("Entity for {} is only available via referral, skipping", mxid);
-            } finally {
-                cursor.close();
             }
 
             log.info("No match were found for {}", mxid);
             return BackendAuthResult.failure();
         } catch (LdapException | IOException | CursorException e) {
             throw new RuntimeException(e);
-        } finally {
-            try {
-                conn.close();
-            } catch (IOException e) {
-                throw new RuntimeException(e);
-            }
         }
     }
 

src/main/java/io/kamax/mxisd/backend/ldap/LdapDirectoryProvider.java

@@ -0,0 +1,120 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.ldap;
+
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.ldap.LdapAttributeConfig;
+import io.kamax.mxisd.config.ldap.LdapConfig;
+import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult;
+import io.kamax.mxisd.directory.IDirectoryProvider;
+import io.kamax.mxisd.exception.InternalServerError;
+import org.apache.directory.api.ldap.model.cursor.CursorException;
+import org.apache.directory.api.ldap.model.cursor.CursorLdapReferralException;
+import org.apache.directory.api.ldap.model.cursor.EntryCursor;
+import org.apache.directory.api.ldap.model.entry.Entry;
+import org.apache.directory.api.ldap.model.exception.LdapException;
+import org.apache.directory.api.ldap.model.message.SearchScope;
+import org.apache.directory.ldap.client.api.LdapConnection;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+@Component
+public class LdapDirectoryProvider extends LdapGenericBackend implements IDirectoryProvider {
+
+    private Logger log = LoggerFactory.getLogger(LdapDirectoryProvider.class);
+
+    @Autowired
+    public LdapDirectoryProvider(LdapConfig cfg, MatrixConfig mxCfg) {
+        super(cfg, mxCfg);
+    }
+
+    @Override
+    public boolean isEnabled() {
+        return getCfg().isEnabled();
+    }
+
+    protected UserDirectorySearchResult search(String query, List<String> attributes) {
+        UserDirectorySearchResult result = new UserDirectorySearchResult();
+        result.setLimited(false);
+
+        try (LdapConnection conn = getConn()) {
+            bind(conn);
+
+            LdapAttributeConfig atCfg = getCfg().getAttribute();
+
+            attributes = new ArrayList<>(attributes);
+            attributes.add(getUidAtt());
+            String[] attArray = new String[attributes.size()];
+            attributes.toArray(attArray);
+
+            String searchQuery = buildOrQueryWithFilter(getCfg().getDirectory().getFilter(), "*" + query + "*", attArray);
+            log.debug("Query: {}", searchQuery);
+            try (EntryCursor cursor = conn.search(getBaseDn(), searchQuery, SearchScope.SUBTREE, attArray)) {
+                while (cursor.next()) {
+                    Entry entry = cursor.get();
+                    log.info("Found possible match, DN: {}", entry.getDn().getName());
+                    getAttribute(entry, getUidAtt()).ifPresent(uid -> {
+                        log.info("DN {} is a valid match", entry.getDn().getName());
+                        try {
+                            UserDirectorySearchResult.Result entryResult = new UserDirectorySearchResult.Result();
+                            entryResult.setUserId(buildMatrixIdFromUid(uid));
+                            getAttribute(entry, atCfg.getName()).ifPresent(entryResult::setDisplayName);
+                            result.addResult(entryResult);
+                        } catch (IllegalArgumentException e) {
+                            log.warn("Bind was found but type {} is not supported", atCfg.getUid().getType());
+                        }
+                    });
+                }
+            }
+        } catch (CursorLdapReferralException e) {
+            log.warn("An entry is only available via referral, skipping");
+        } catch (IOException | LdapException | CursorException e) {
+            throw new InternalServerError(e);
+        }
+
+        return result;
+    }
+
+    @Override
+    public UserDirectorySearchResult searchByDisplayName(String query) {
+        log.info("Performing LDAP directory search on display name using '{}'", query);
+        List<String> attributes = new ArrayList<>();
+        attributes.add(getAt().getName());
+        attributes.addAll(getCfg().getDirectory().getAttribute().getOther());
+        return search(query, attributes);
+    }
+
+    @Override
+    public UserDirectorySearchResult searchBy3pid(String query) {
+        log.info("Performing LDAP directory search on 3PIDs using '{}'", query);
+        List<String> attributes = new ArrayList<>();
+        attributes.add(getAt().getName());
+        getCfg().getAttribute().getThreepid().forEach((k, v) -> attributes.addAll(v));
+        return search(query, attributes);
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/ldap/LdapGenericBackend.java

@@ -20,38 +20,121 @@
 
 package io.kamax.mxisd.backend.ldap;
 
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.ldap.LdapAttributeConfig;
 import io.kamax.mxisd.config.ldap.LdapConfig;
 import org.apache.commons.lang.StringUtils;
+import org.apache.directory.api.ldap.model.entry.Attribute;
+import org.apache.directory.api.ldap.model.entry.Entry;
 import org.apache.directory.api.ldap.model.exception.LdapException;
 import org.apache.directory.ldap.client.api.LdapConnection;
 import org.apache.directory.ldap.client.api.LdapNetworkConnection;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Component;
 
-@Component
-public class LdapGenericBackend {
+import java.util.Arrays;
+import java.util.List;
+import java.util.Optional;
+
+public abstract class LdapGenericBackend {
+
+    public static final String UID = "uid";
+    public static final String MATRIX_ID = "mxid";
 
     private Logger log = LoggerFactory.getLogger(LdapGenericBackend.class);
 
-    @Autowired
-    private LdapConfig ldapCfg;
+    private LdapConfig cfg;
+    private MatrixConfig mxCfg;
 
-    protected LdapConnection getConn() {
-        return new LdapNetworkConnection(ldapCfg.getConn().getHost(), ldapCfg.getConn().getPort(), ldapCfg.getConn().isTls());
-    }
-
-    protected void bind(LdapConnection conn) throws LdapException {
-        if (StringUtils.isBlank(ldapCfg.getConn().getBindDn()) && StringUtils.isBlank(ldapCfg.getConn().getBindPassword())) {
-            conn.anonymousBind();
-        } else {
-            conn.bind(ldapCfg.getConn().getBindDn(), ldapCfg.getConn().getBindPassword());
-        }
+    public LdapGenericBackend(LdapConfig cfg, MatrixConfig mxCfg) {
+        this.cfg = cfg;
+        this.mxCfg = mxCfg;
     }
 
     protected LdapConfig getCfg() {
-        return ldapCfg;
+        return cfg;
+    }
+
+    protected String getBaseDn() {
+        return cfg.getConn().getBaseDn();
+    }
+
+    protected LdapAttributeConfig getAt() {
+        return cfg.getAttribute();
+    }
+
+    protected String getUidAtt() {
+        return getAt().getUid().getValue();
+    }
+
+    protected synchronized LdapConnection getConn() throws LdapException {
+        return new LdapNetworkConnection(cfg.getConn().getHost(), cfg.getConn().getPort(), cfg.getConn().isTls());
+    }
+
+    protected void bind(LdapConnection conn) throws LdapException {
+        if (StringUtils.isBlank(cfg.getConn().getBindDn()) && StringUtils.isBlank(cfg.getConn().getBindPassword())) {
+            conn.anonymousBind();
+        } else {
+            conn.bind(cfg.getConn().getBindDn(), cfg.getConn().getBindPassword());
+        }
+    }
+
+    protected String buildWithFilter(String base, String filter) {
+        if (StringUtils.isBlank(filter)) {
+            return base;
+        } else {
+            return "(&" + filter + base + ")";
+        }
+    }
+
+    public static String buildOrQuery(String value, List<String> attributes) {
+        if (attributes.size() < 1) {
+            throw new IllegalArgumentException();
+        }
+
+        StringBuilder builder = new StringBuilder();
+        builder.append("(|");
+        attributes.forEach(s -> {
+            builder.append("(");
+            builder.append(s).append("=").append(value).append(")");
+        });
+        builder.append(")");
+        return builder.toString();
+    }
+
+    public static String buildOrQuery(String value, String... attributes) {
+        return buildOrQuery(value, Arrays.asList(attributes));
+    }
+
+    public String buildOrQueryWithFilter(String filter, String value, String... attributes) {
+        return buildWithFilter(buildOrQuery(value, attributes), filter);
+    }
+
+    public String buildMatrixIdFromUid(String uid) {
+        String uidType = getCfg().getAttribute().getUid().getType();
+        if (StringUtils.equals(UID, uidType)) {
+            return "@" + uid + ":" + mxCfg.getDomain();
+        } else if (StringUtils.equals(MATRIX_ID, uidType)) {
+            return uid;
+        } else {
+            throw new IllegalArgumentException("Bind type " + uidType + " is not supported");
+        }
+    }
+
+    public Optional<String> getAttribute(Entry entry, String attName) {
+        Attribute attribute = entry.get(attName);
+        if (attribute == null) {
+            log.info("DN {}: no attribute {}, skipping", entry.getDn(), attName);
+            return Optional.empty();
+        }
+
+        String value = attribute.get().toString();
+        if (StringUtils.isBlank(value)) {
+            log.info("DN {}: empty attribute {}, skipping", attName);
+            return Optional.empty();
+        }
+
+        return Optional.of(value);
     }
 
 }

src/main/java/io/kamax/mxisd/backend/ldap/LdapThreePidProvider.java

@@ -21,23 +21,21 @@
 package io.kamax.mxisd.backend.ldap;
 
 import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.ldap.LdapConfig;
 import io.kamax.mxisd.exception.InternalServerError;
 import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.SingleLookupRequest;
 import io.kamax.mxisd.lookup.ThreePidMapping;
 import io.kamax.mxisd.lookup.provider.IThreePidProvider;
-import org.apache.commons.lang.StringUtils;
 import org.apache.directory.api.ldap.model.cursor.CursorException;
 import org.apache.directory.api.ldap.model.cursor.CursorLdapReferralException;
 import org.apache.directory.api.ldap.model.cursor.EntryCursor;
-import org.apache.directory.api.ldap.model.entry.Attribute;
 import org.apache.directory.api.ldap.model.entry.Entry;
 import org.apache.directory.api.ldap.model.exception.LdapException;
 import org.apache.directory.api.ldap.model.message.SearchScope;
 import org.apache.directory.ldap.client.api.LdapConnection;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
 import java.io.IOException;
@@ -48,23 +46,17 @@ import java.util.Optional;
 @Component
 public class LdapThreePidProvider extends LdapGenericBackend implements IThreePidProvider {
 
-    public static final String UID = "uid";
-    public static final String MATRIX_ID = "mxid";
-
     private Logger log = LoggerFactory.getLogger(LdapThreePidProvider.class);
 
-    @Autowired
-    private MatrixConfig mxCfg;
+    public LdapThreePidProvider(LdapConfig cfg, MatrixConfig mxCfg) {
+        super(cfg, mxCfg);
+    }
 
     @Override
     public boolean isEnabled() {
         return getCfg().isEnabled();
     }
 
-    private String getUidAttribute() {
-        return getCfg().getAttribute().getUid().getValue();
-    }
-
     @Override
     public boolean isLocal() {
         return true;
@@ -76,46 +68,25 @@ public class LdapThreePidProvider extends LdapGenericBackend implements IThreePi
     }
 
     private Optional<String> lookup(LdapConnection conn, String medium, String value) {
-        String uidAttribute = getUidAttribute();
-
         Optional<String> queryOpt = getCfg().getIdentity().getQuery(medium);
         if (!queryOpt.isPresent()) {
             log.warn("{} is not a configured 3PID type for LDAP lookup", medium);
             return Optional.empty();
         }
 
-        String searchQuery = queryOpt.get().replaceAll("%3pid", value);
-        try (EntryCursor cursor = conn.search(getCfg().getConn().getBaseDn(), searchQuery, SearchScope.SUBTREE, uidAttribute)) {
+        String searchQuery = queryOpt.get().replaceAll(getCfg().getIdentity().getToken(), value);
+        try (EntryCursor cursor = conn.search(getBaseDn(), searchQuery, SearchScope.SUBTREE, getUidAtt())) {
             while (cursor.next()) {
                 Entry entry = cursor.get();
                 log.info("Found possible match, DN: {}", entry.getDn().getName());
 
-                Attribute attribute = entry.get(uidAttribute);
-                if (attribute == null) {
-                    log.info("DN {}: no attribute {}, skpping", entry.getDn(), getCfg().getAttribute());
-                    continue;
-                }
-
-                String data = attribute.get().toString();
-                if (data.length() < 1) {
-                    log.info("DN {}: empty attribute {}, skipping", getCfg().getAttribute());
-                    continue;
-                }
-
-                StringBuilder matrixId = new StringBuilder();
-                // TODO Should we turn this block into a map of functions?
-                String uidType = getCfg().getAttribute().getUid().getType();
-                if (StringUtils.equals(UID, uidType)) {
-                    matrixId.append("@").append(data).append(":").append(mxCfg.getDomain());
-                } else if (StringUtils.equals(MATRIX_ID, uidType)) {
-                    matrixId.append(data);
-                } else {
-                    log.warn("Bind was found but type {} is not supported", uidType);
+                Optional<String> data = getAttribute(entry, getUidAtt());
+                if (!data.isPresent()) {
                     continue;
                 }
 
                 log.info("DN {} is a valid match", entry.getDn().getName());
-                return Optional.of(matrixId.toString());
+                return Optional.of(buildMatrixIdFromUid(data.get()));
             }
         } catch (CursorLdapReferralException e) {
             log.warn("3PID {} is only available via referral, skipping", value);
@@ -128,21 +99,14 @@ public class LdapThreePidProvider extends LdapGenericBackend implements IThreePi
 
     @Override
     public Optional<SingleLookupReply> find(SingleLookupRequest request) {
-        log.info("Performing LDAP lookup ${request.getThreePid()} of type ${request.getType()}");
+        log.info("Performing LDAP lookup {} of type {}", request.getThreePid(), request.getType());
 
         try (LdapConnection conn = getConn()) {
             bind(conn);
-
-            Optional<String> mxid = lookup(conn, request.getType(), request.getThreePid());
-            if (mxid.isPresent()) {
-                return Optional.of(new SingleLookupReply(request, mxid.get()));
-            }
+            return lookup(conn, request.getType(), request.getThreePid()).map(id -> new SingleLookupReply(request, id));
         } catch (LdapException | IOException e) {
             throw new InternalServerError(e);
         }
-
-        log.info("No match found");
-        return Optional.empty();
     }
 
     @Override
@@ -155,11 +119,10 @@ public class LdapThreePidProvider extends LdapGenericBackend implements IThreePi
 
             for (ThreePidMapping mapping : mappings) {
                 try {
-                    Optional<String> mxid = lookup(conn, mapping.getMedium(), mapping.getValue());
-                    if (mxid.isPresent()) {
-                        mapping.setMxid(mxid.get());
+                    lookup(conn, mapping.getMedium(), mapping.getValue()).ifPresent(id -> {
+                        mapping.setMxid(id);
                         mappingsFound.add(mapping);
-                    }
+                    });
                 } catch (IllegalArgumentException e) {
                     log.warn("{} is not a supported 3PID type for LDAP lookup", mapping.getMedium());
                 }

src/main/java/io/kamax/mxisd/backend/rest/RestDirectoryProvider.java

@@ -0,0 +1,82 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.rest;
+
+import io.kamax.matrix.MatrixID;
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.rest.RestBackendConfig;
+import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchRequest;
+import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult;
+import io.kamax.mxisd.directory.IDirectoryProvider;
+import io.kamax.mxisd.exception.InternalServerError;
+import io.kamax.mxisd.util.RestClientUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.client.methods.CloseableHttpResponse;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+
+public class RestDirectoryProvider extends RestProvider implements IDirectoryProvider {
+
+    private MatrixConfig mxCfg;
+
+    public RestDirectoryProvider(RestBackendConfig cfg, MatrixConfig mxCfg) {
+        super(cfg);
+        this.mxCfg = mxCfg;
+    }
+
+    @Override
+    public boolean isEnabled() {
+        return cfg.isEnabled() && StringUtils.isNotBlank(cfg.getEndpoints().getDirectory());
+    }
+
+    private UserDirectorySearchResult search(String by, String query) {
+        UserDirectorySearchRequest request = new UserDirectorySearchRequest(query);
+        request.setBy(by);
+        try (CloseableHttpResponse httpResponse = client.execute(RestClientUtils.post(cfg.getEndpoints().getDirectory(), request))) {
+            int status = httpResponse.getStatusLine().getStatusCode();
+            if (status < 200 || status >= 300) {
+                throw new InternalServerError("REST backend: Error: " + IOUtils.toString(httpResponse.getEntity().getContent(), StandardCharsets.UTF_8));
+            }
+
+            UserDirectorySearchResult response = parser.parse(httpResponse, UserDirectorySearchResult.class);
+            for (UserDirectorySearchResult.Result result : response.getResults()) {
+                result.setUserId(new MatrixID(result.getUserId(), mxCfg.getDomain()).getId());
+            }
+
+            return response;
+        } catch (IOException e) {
+            throw new InternalServerError("REST backend: I/O error: " + e.getMessage());
+        }
+    }
+
+    @Override
+    public UserDirectorySearchResult searchByDisplayName(String query) {
+        return search("name", query);
+    }
+
+    @Override
+    public UserDirectorySearchResult searchBy3pid(String query) {
+        return search("threepid", query);
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/sql/SqlConnectionPool.java

@@ -0,0 +1,45 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.sql;
+
+import com.mchange.v2.c3p0.ComboPooledDataSource;
+import io.kamax.mxisd.config.sql.SqlConfig;
+
+import java.sql.Connection;
+import java.sql.SQLException;
+
+public class SqlConnectionPool {
+
+    private ComboPooledDataSource ds;
+
+    public SqlConnectionPool(SqlConfig cfg) {
+        ds = new ComboPooledDataSource();
+        ds.setJdbcUrl("jdbc:" + cfg.getType() + ":" + cfg.getConnection());
+        ds.setMinPoolSize(1);
+        ds.setMaxPoolSize(10);
+        ds.setAcquireIncrement(2);
+    }
+
+    public Connection get() throws SQLException {
+        return ds.getConnection();
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/sql/SqlDirectoryProvider.java

@@ -0,0 +1,114 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.sql;
+
+import io.kamax.matrix.MatrixID;
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.sql.SqlConfig;
+import io.kamax.mxisd.config.sql.SqlProviderConfig;
+import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult;
+import io.kamax.mxisd.directory.IDirectoryProvider;
+import io.kamax.mxisd.exception.InternalServerError;
+import org.apache.commons.lang.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Optional;
+
+import static io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult.Result;
+
+public abstract class SqlDirectoryProvider implements IDirectoryProvider {
+
+    private Logger log = LoggerFactory.getLogger(SqlDirectoryProvider.class);
+
+    protected SqlConfig cfg;
+    private MatrixConfig mxCfg;
+
+    private SqlConnectionPool pool;
+
+    public SqlDirectoryProvider(SqlConfig cfg, MatrixConfig mxCfg) {
+        this.cfg = cfg;
+        this.pool = new SqlConnectionPool(cfg);
+        this.mxCfg = mxCfg;
+    }
+
+    @Override
+    public boolean isEnabled() {
+        return cfg.isEnabled();
+    }
+
+    protected void setParameters(PreparedStatement stmt, String searchTerm) throws SQLException {
+        for (int i = 1; i <= stmt.getParameterMetaData().getParameterCount(); i++) {
+            stmt.setString(i, searchTerm);
+        }
+    }
+
+    protected Optional<Result> processRow(ResultSet rSet) throws SQLException {
+        Result item = new Result();
+        item.setUserId(rSet.getString(1));
+        item.setDisplayName(rSet.getString(2));
+        return Optional.of(item);
+    }
+
+    public UserDirectorySearchResult search(String searchTerm, SqlProviderConfig.Query query) {
+        try (Connection conn = pool.get()) {
+            log.info("Will execute query: {}", query.getValue());
+            try (PreparedStatement stmt = conn.prepareStatement(query.getValue())) {
+                setParameters(stmt, searchTerm);
+
+                try (ResultSet rSet = stmt.executeQuery()) {
+                    UserDirectorySearchResult result = new UserDirectorySearchResult();
+                    result.setLimited(false);
+
+                    while (rSet.next()) {
+                        processRow(rSet).ifPresent(e -> {
+                            if (StringUtils.equalsIgnoreCase("localpart", query.getType())) {
+                                e.setUserId(new MatrixID(e.getUserId(), mxCfg.getDomain()).getId());
+                            }
+                            result.addResult(e);
+                        });
+                    }
+
+                    return result;
+                }
+            }
+        } catch (SQLException e) {
+            throw new InternalServerError(e);
+        }
+    }
+
+    @Override
+    public UserDirectorySearchResult searchByDisplayName(String searchTerm) {
+        log.info("Searching users by display name using '{}'", searchTerm);
+        return search(searchTerm, cfg.getDirectory().getQuery().getName());
+    }
+
+    @Override
+    public UserDirectorySearchResult searchBy3pid(String searchTerm) {
+        log.info("Searching users by 3PID using '{}'", searchTerm);
+        return search(searchTerm, cfg.getDirectory().getQuery().getThreepid());
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/sql/SqlThreePidProvider.java

@@ -33,7 +33,10 @@ import org.slf4j.LoggerFactory;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
-import java.sql.*;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Optional;
@@ -43,11 +46,17 @@ public class SqlThreePidProvider implements IThreePidProvider {
 
     private Logger log = LoggerFactory.getLogger(SqlThreePidProvider.class);
 
-    @Autowired
+    private SqlProviderConfig cfg;
     private MatrixConfig mxCfg;
 
+    private SqlConnectionPool pool;
+
     @Autowired
-    private SqlProviderConfig cfg;
+    public SqlThreePidProvider(SqlProviderConfig cfg, MatrixConfig mxCfg) {
+        this.cfg = cfg;
+        this.pool = new SqlConnectionPool(cfg);
+        this.mxCfg = mxCfg;
+    }
 
     @Override
     public boolean isEnabled() {
@@ -64,37 +73,36 @@ public class SqlThreePidProvider implements IThreePidProvider {
         return 20;
     }
 
-    private Connection getConn() throws SQLException {
-        return DriverManager.getConnection("jdbc:" + cfg.getType() + ":" + cfg.getConnection());
-    }
-
     @Override
     public Optional<SingleLookupReply> find(SingleLookupRequest request) {
         log.info("SQL lookup");
         String stmtSql = StringUtils.defaultIfBlank(cfg.getIdentity().getMedium().get(request.getType()), cfg.getIdentity().getQuery());
         log.info("SQL query: {}", stmtSql);
-        try (PreparedStatement stmt = getConn().prepareStatement(stmtSql)) {
-            stmt.setString(1, request.getType().toLowerCase());
-            stmt.setString(2, request.getThreePid().toLowerCase());
+        try (Connection conn = pool.get()) {
+            try (PreparedStatement stmt = conn.prepareStatement(stmtSql)) {
+                stmt.setString(1, request.getType().toLowerCase());
+                stmt.setString(2, request.getThreePid().toLowerCase());
 
-            ResultSet rSet = stmt.executeQuery();
-            while (rSet.next()) {
-                String uid = rSet.getString("uid");
-                log.info("Found match: {}", uid);
-                if (StringUtils.equals("uid", cfg.getIdentity().getType())) {
-                    log.info("Resolving as localpart");
-                    return Optional.of(new SingleLookupReply(request, new MatrixID(uid, mxCfg.getDomain())));
-                }
-                if (StringUtils.equals("mxid", cfg.getIdentity().getType())) {
-                    log.info("Resolving as MXID");
-                    return Optional.of(new SingleLookupReply(request, new MatrixID(uid)));
-                }
+                try (ResultSet rSet = stmt.executeQuery()) {
+                    while (rSet.next()) {
+                        String uid = rSet.getString("uid");
+                        log.info("Found match: {}", uid);
+                        if (StringUtils.equals("uid", cfg.getIdentity().getType())) {
+                            log.info("Resolving as localpart");
+                            return Optional.of(new SingleLookupReply(request, new MatrixID(uid, mxCfg.getDomain())));
+                        }
+                        if (StringUtils.equals("mxid", cfg.getIdentity().getType())) {
+                            log.info("Resolving as MXID");
+                            return Optional.of(new SingleLookupReply(request, new MatrixID(uid)));
+                        }
 
-                log.info("Identity type is unknown, skipping");
+                        log.info("Identity type is unknown, skipping");
+                    }
+
+                    log.info("No match found in SQL");
+                    return Optional.empty();
+                }
             }
-
-            log.info("No match found in SQL");
-            return Optional.empty();
         } catch (SQLException e) {
             throw new RuntimeException(e);
         }

src/main/java/io/kamax/mxisd/backend/sql/SynapseSqliteDirectoryProvider.java

@@ -0,0 +1,71 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.sql;
+
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.sql.SqlProviderConfig;
+import io.kamax.mxisd.config.sql.synapse.SynapseSqlProviderConfig;
+import io.kamax.mxisd.exception.ConfigurationException;
+import org.apache.commons.lang.StringUtils;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+
+@Component
+public class SynapseSqliteDirectoryProvider extends SqlDirectoryProvider {
+
+    private SynapseSqlProviderConfig cfg;
+
+    @Autowired
+    public SynapseSqliteDirectoryProvider(SynapseSqlProviderConfig cfg, MatrixConfig mxCfg) {
+        super(cfg, mxCfg);
+
+        if (StringUtils.equals("sqlite", cfg.getType())) {
+            String userId = "'@' || p.user_id || ':" + mxCfg.getDomain() + "'";
+            SqlProviderConfig.Type queries = cfg.getDirectory().getQuery();
+            queries.getName().setValue(
+                    "select " + userId + ", displayname from profiles p where displayname like ?");
+            queries.getThreepid().setValue(
+                    "select t.user_id, p.displayname " +
+                            "from user_threepids t JOIN profiles p on t.user_id = " + userId + " " +
+                            "where t.address like ?");
+        } else if (StringUtils.equals("postgresql", cfg.getType())) {
+            String userId = "concat('@',p.user_id,':" + mxCfg.getDomain() + "')";
+            SqlProviderConfig.Type queries = cfg.getDirectory().getQuery();
+            queries.getName().setValue(
+                    "select " + userId + ", displayname from profiles p where displayname ilike ?");
+            queries.getThreepid().setValue(
+                    "select t.user_id, p.displayname " +
+                            "from user_threepids t JOIN profiles p on t.user_id = " + userId + " " +
+                            "where t.address ilike ?");
+        } else {
+            throw new ConfigurationException("Invalid SQL type");
+        }
+    }
+
+    @Override
+    protected void setParameters(PreparedStatement stmt, String searchTerm) throws SQLException {
+        stmt.setString(1, "%" + searchTerm + "%");
+    }
+
+}

src/main/java/io/kamax/mxisd/config/DnsOverwriteConfig.java

@@ -0,0 +1,106 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.config;
+
+import com.google.gson.Gson;
+import io.kamax.mxisd.util.GsonUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.context.annotation.Configuration;
+
+import javax.annotation.PostConstruct;
+import java.util.ArrayList;
+import java.util.List;
+
+@Configuration
+@ConfigurationProperties("dns.overwrite")
+public class DnsOverwriteConfig {
+
+    private Logger log = LoggerFactory.getLogger(DnsOverwriteConfig.class);
+
+    public static class Entry {
+
+        private String name;
+        private String value;
+
+        public String getName() {
+            return name;
+        }
+
+        public void setName(String name) {
+            this.name = name;
+        }
+
+        public String getValue() {
+            return value;
+        }
+
+        public void setValue(String value) {
+            this.value = value;
+        }
+
+    }
+
+    public static class Type {
+
+        List<Entry> client = new ArrayList<>();
+        List<Entry> federation = new ArrayList<>();
+
+        public List<Entry> getClient() {
+            return client;
+        }
+
+        public void setClient(List<Entry> client) {
+            this.client = client;
+        }
+
+        public List<Entry> getFederation() {
+            return federation;
+        }
+
+        public void setFederation(List<Entry> federation) {
+            this.federation = federation;
+        }
+
+    }
+
+    private Type homeserver = new Type();
+
+    public Type getHomeserver() {
+        return homeserver;
+    }
+
+    public void setHomeserver(Type homeserver) {
+        this.homeserver = homeserver;
+    }
+
+    @PostConstruct
+    public void build() {
+        Gson gson = GsonUtil.build();
+        log.info("--- DNS Overwrite config ---");
+        log.info("Homeserver:");
+        log.info("\tClient: {}", gson.toJson(getHomeserver().getClient()));
+        log.info("\tFederation: {}", gson.toJson(getHomeserver().getFederation()));
+
+    }
+
+}

src/main/java/io/kamax/mxisd/config/DnsOverwriteEntry.java

@@ -1,67 +0,0 @@
-/*
- * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
- *
- * https://max.kamax.io/
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU Affero General Public License as
- * published by the Free Software Foundation, either version 3 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU Affero General Public License for more details.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- */
-
-package io.kamax.mxisd.config;
-
-import org.apache.commons.lang.StringUtils;
-import org.springframework.boot.context.properties.ConfigurationProperties;
-import org.springframework.context.annotation.Configuration;
-
-@Configuration
-@ConfigurationProperties("dns.overwrite.homeserver")
-public class DnsOverwriteEntry {
-
-    private String name;
-    private String type;
-    private String value;
-
-    public String getName() {
-        return name;
-    }
-
-    public void setName(String name) {
-        this.name = name;
-    }
-
-    public String getType() {
-        return type;
-    }
-
-    public void setType(String type) {
-        this.type = type;
-    }
-
-    public String getValue() {
-        return value;
-    }
-
-    public void setValue(String value) {
-        this.value = value;
-    }
-
-    public String getTarget() {
-        if (StringUtils.equals("env", getType())) {
-            return System.getenv(getValue());
-        } else {
-            return getValue();
-        }
-    }
-
-}

src/main/java/io/kamax/mxisd/config/FirebaseConfig.java

@@ -71,7 +71,7 @@ public class FirebaseConfig {
     }
 
     @PostConstruct
-    private void postConstruct() {
+    public void build() {
         log.info("--- Firebase configuration ---");
         log.info("Enabled: {}", isEnabled());
         if (isEnabled()) {
@@ -82,20 +82,12 @@ public class FirebaseConfig {
 
     @Bean
     public AuthenticatorProvider getAuthProvider() {
-        if (!enabled) {
-            return new GoogleFirebaseAuthenticator(false);
-        } else {
-            return new GoogleFirebaseAuthenticator(credentials, database);
-        }
+        return new GoogleFirebaseAuthenticator(enabled, credentials, database);
     }
 
     @Bean
     public IThreePidProvider getLookupProvider() {
-        if (!enabled) {
-            return new GoogleFirebaseProvider(false);
-        } else {
-            return new GoogleFirebaseProvider(credentials, database, mxCfg.getDomain());
-        }
+        return new GoogleFirebaseProvider(enabled, credentials, database, mxCfg.getDomain());
     }
 
 }

src/main/java/io/kamax/mxisd/config/ldap/LdapAttributeConfig.java

@@ -23,12 +23,17 @@ package io.kamax.mxisd.config.ldap;
 import org.springframework.boot.context.properties.ConfigurationProperties;
 import org.springframework.context.annotation.Configuration;
 
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
 @Configuration
 @ConfigurationProperties(prefix = "ldap.attribute")
 public class LdapAttributeConfig {
 
     private LdapAttributeUidConfig uid;
     private String name;
+    private Map<String, List<String>> threepid = new HashMap<>();
 
     public LdapAttributeUidConfig getUid() {
         return uid;
@@ -46,4 +51,12 @@ public class LdapAttributeConfig {
         this.name = name;
     }
 
+    public Map<String, List<String>> getThreepid() {
+        return threepid;
+    }
+
+    public void setThreepid(Map<String, List<String>> threepid) {
+        this.threepid = threepid;
+    }
+
 }

src/main/java/io/kamax/mxisd/config/ldap/LdapConfig.java

@@ -21,7 +21,9 @@
 package io.kamax.mxisd.config.ldap;
 
 import com.google.gson.Gson;
-import io.kamax.mxisd.backend.ldap.LdapThreePidProvider;
+import io.kamax.matrix.ThreePidMedium;
+import io.kamax.mxisd.backend.ldap.LdapGenericBackend;
+import io.kamax.mxisd.exception.ConfigurationException;
 import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -30,21 +32,61 @@ import org.springframework.boot.context.properties.ConfigurationProperties;
 import org.springframework.context.annotation.Configuration;
 
 import javax.annotation.PostConstruct;
+import java.util.ArrayList;
+import java.util.List;
 
 @Configuration
 @ConfigurationProperties(prefix = "ldap")
 public class LdapConfig {
 
+    private Logger log = LoggerFactory.getLogger(LdapConfig.class);
     private static Gson gson = new Gson();
 
-    private Logger log = LoggerFactory.getLogger(LdapConfig.class);
-
     private boolean enabled;
+    private String filter;
+
+    public static class Directory {
+
+        public static class Attribute {
+
+            private List<String> other = new ArrayList<>();
+
+            public List<String> getOther() {
+                return other;
+            }
+
+            public void setOther(List<String> other) {
+                this.other = other;
+            }
+
+        }
+
+        private Attribute attribute = new Attribute();
+        private String filter;
+
+        public Attribute getAttribute() {
+            return attribute;
+        }
+
+        public void setAttribute(Attribute attribute) {
+            this.attribute = attribute;
+        }
+
+        public String getFilter() {
+            return filter;
+        }
+
+        public void setFilter(String filter) {
+            this.filter = filter;
+        }
+
+    }
 
     @Autowired
     private LdapConnectionConfig conn;
     private LdapAttributeConfig attribute;
     private LdapAuthConfig auth;
+    private Directory directory;
     private LdapIdentityConfig identity;
 
     public boolean isEnabled() {
@@ -55,6 +97,14 @@ public class LdapConfig {
         this.enabled = enabled;
     }
 
+    public String getFilter() {
+        return filter;
+    }
+
+    public void setFilter(String filter) {
+        this.filter = filter;
+    }
+
     public LdapConnectionConfig getConn() {
         return conn;
     }
@@ -79,6 +129,14 @@ public class LdapConfig {
         this.auth = auth;
     }
 
+    public Directory getDirectory() {
+        return directory;
+    }
+
+    public void setDirectory(Directory directory) {
+        this.directory = directory;
+    }
+
     public LdapIdentityConfig getIdentity() {
         return identity;
     }
@@ -100,7 +158,7 @@ public class LdapConfig {
             throw new IllegalStateException("LDAP Host must be configured!");
         }
 
-        if (1 > conn.getPort() || 65535 < conn.getPort()) {
+        if (conn.getPort() < 1 || conn.getPort() > 65535) {
             throw new IllegalStateException("LDAP port is not valid");
         }
 
@@ -114,10 +172,29 @@ public class LdapConfig {
         }
 
         String uidType = attribute.getUid().getType();
-        if (!StringUtils.equals(LdapThreePidProvider.UID, uidType) && !StringUtils.equals(LdapThreePidProvider.MATRIX_ID, uidType)) {
+        if (!StringUtils.equals(LdapGenericBackend.UID, uidType) && !StringUtils.equals(LdapGenericBackend.MATRIX_ID, uidType)) {
             throw new IllegalArgumentException("Unsupported LDAP UID type: " + uidType);
         }
 
+        if (StringUtils.isBlank(identity.getToken())) {
+            throw new ConfigurationException("ldap.identity.token");
+        }
+
+        // Build queries
+        attribute.getThreepid().forEach((k, v) -> {
+            if (StringUtils.isBlank(identity.getMedium().get(k))) {
+                if (ThreePidMedium.PhoneNumber.is(k)) {
+                    identity.getMedium().put(k, LdapGenericBackend.buildOrQuery("+" + getIdentity().getToken(), v));
+                } else {
+                    identity.getMedium().put(k, LdapGenericBackend.buildOrQuery(getIdentity().getToken(), v));
+                }
+            }
+        });
+
+        getAuth().setFilter(StringUtils.defaultIfBlank(getAuth().getFilter(), getFilter()));
+        getDirectory().setFilter(StringUtils.defaultIfBlank(getDirectory().getFilter(), getFilter()));
+        getIdentity().setFilter(StringUtils.defaultIfBlank(getIdentity().getFilter(), getFilter()));
+
         log.info("Host: {}", conn.getHost());
         log.info("Port: {}", conn.getPort());
         log.info("Bind DN: {}", conn.getBindDn());
@@ -125,6 +202,7 @@ public class LdapConfig {
 
         log.info("Attribute: {}", gson.toJson(attribute));
         log.info("Auth: {}", gson.toJson(auth));
+        log.info("Directory: {}", gson.toJson(directory));
         log.info("Identity: {}", gson.toJson(identity));
     }
 

src/main/java/io/kamax/mxisd/config/ldap/LdapIdentityConfig.java

@@ -31,8 +31,26 @@ import java.util.Optional;
 @ConfigurationProperties(prefix = "ldap.identity")
 public class LdapIdentityConfig {
 
+    private String filter;
+    private String token;
     private Map<String, String> medium = new HashMap<>();
 
+    public String getFilter() {
+        return filter;
+    }
+
+    public void setFilter(String filter) {
+        this.filter = filter;
+    }
+
+    public String getToken() {
+        return token;
+    }
+
+    public void setToken(String token) {
+        this.token = token;
+    }
+
     public Map<String, String> getMedium() {
         return medium;
     }

src/main/java/io/kamax/mxisd/config/rest/RestBackendConfig.java

@@ -60,16 +60,9 @@ public class RestBackendConfig {
 
     public static class Endpoints {
 
-        private IdentityEndpoints identity = new IdentityEndpoints();
         private String auth;
-
-        public IdentityEndpoints getIdentity() {
-            return identity;
-        }
-
-        public void setIdentity(IdentityEndpoints identity) {
-            this.identity = identity;
-        }
+        private String directory;
+        private IdentityEndpoints identity = new IdentityEndpoints();
 
         public String getAuth() {
             return auth;
@@ -79,6 +72,22 @@ public class RestBackendConfig {
             this.auth = auth;
         }
 
+        public String getDirectory() {
+            return directory;
+        }
+
+        public void setDirectory(String directory) {
+            this.directory = directory;
+        }
+
+        public IdentityEndpoints getIdentity() {
+            return identity;
+        }
+
+        public void setIdentity(IdentityEndpoints identity) {
+            this.identity = identity;
+        }
+
     }
 
     private Logger log = LoggerFactory.getLogger(RestBackendConfig.class);
@@ -136,11 +145,13 @@ public class RestBackendConfig {
 
         if (isEnabled()) {
             endpoints.setAuth(buildEndpointUrl(endpoints.getAuth()));
+            endpoints.setDirectory(buildEndpointUrl(endpoints.getDirectory()));
             endpoints.identity.setSingle(buildEndpointUrl(endpoints.identity.getSingle()));
             endpoints.identity.setBulk(buildEndpointUrl(endpoints.identity.getBulk()));
 
             log.info("Host: {}", getHost());
             log.info("Auth endpoint: {}", endpoints.getAuth());
+            log.info("Directory endpoint: {}", endpoints.getDirectory());
             log.info("Identity Single endpoint: {}", endpoints.identity.getSingle());
             log.info("Identity Bulk endpoint: {}", endpoints.identity.getBulk());
         }

src/main/java/io/kamax/mxisd/config/sql/SqlConfig.java

@@ -0,0 +1,221 @@
+package io.kamax.mxisd.config.sql;
+
+import io.kamax.mxisd.util.GsonUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public abstract class SqlConfig {
+
+    private Logger log = LoggerFactory.getLogger(SqlConfig.class);
+
+    public static class Query {
+
+        private String type;
+        private String value;
+
+        public String getType() {
+            return type;
+        }
+
+        public void setType(String type) {
+            this.type = type;
+        }
+
+        public String getValue() {
+            return value;
+        }
+
+        public void setValue(String value) {
+            this.value = value;
+        }
+
+    }
+
+    public static class Type {
+
+        private SqlProviderConfig.Query name = new SqlProviderConfig.Query();
+        private SqlProviderConfig.Query threepid = new SqlProviderConfig.Query();
+
+        public SqlProviderConfig.Query getName() {
+            return name;
+        }
+
+        public void setName(SqlProviderConfig.Query name) {
+            this.name = name;
+        }
+
+        public SqlProviderConfig.Query getThreepid() {
+            return threepid;
+        }
+
+        public void setThreepid(SqlProviderConfig.Query threepid) {
+            this.threepid = threepid;
+        }
+
+    }
+
+    public static class Auth {
+
+        private Boolean enabled;
+
+        public Boolean isEnabled() {
+            return enabled;
+        }
+
+        public void setEnabled(Boolean enabled) {
+            this.enabled = enabled;
+        }
+
+    }
+
+    public static class Directory {
+
+        private Boolean enabled;
+        private SqlProviderConfig.Type query = new SqlProviderConfig.Type();
+
+        public Boolean isEnabled() {
+            return enabled;
+        }
+
+        public void setEnabled(Boolean enabled) {
+            this.enabled = enabled;
+        }
+
+        public SqlProviderConfig.Type getQuery() {
+            return query;
+        }
+
+        public void setQuery(SqlProviderConfig.Type query) {
+            this.query = query;
+        }
+
+    }
+
+    public static class Identity {
+
+        private Boolean enabled;
+        private String type;
+        private String query;
+        private Map<String, String> medium = new HashMap<>();
+
+        public Boolean isEnabled() {
+            return enabled;
+        }
+
+        public void setEnabled(Boolean enabled) {
+            this.enabled = enabled;
+        }
+
+        public String getType() {
+            return type;
+        }
+
+        public void setType(String type) {
+            this.type = type;
+        }
+
+        public String getQuery() {
+            return query;
+        }
+
+        public void setQuery(String query) {
+            this.query = query;
+        }
+
+        public Map<String, String> getMedium() {
+            return medium;
+        }
+
+        public void setMedium(Map<String, String> medium) {
+            this.medium = medium;
+        }
+
+    }
+
+    private boolean enabled;
+    private String type;
+    private String connection;
+    private SqlProviderConfig.Auth auth = new SqlProviderConfig.Auth();
+    private SqlProviderConfig.Directory directory = new SqlProviderConfig.Directory();
+    private SqlProviderConfig.Identity identity = new SqlProviderConfig.Identity();
+
+    public boolean isEnabled() {
+        return enabled;
+    }
+
+    public void setEnabled(boolean enabled) {
+        this.enabled = enabled;
+    }
+
+    public String getType() {
+        return type;
+    }
+
+    public void setType(String type) {
+        this.type = type;
+    }
+
+    public String getConnection() {
+        return connection;
+    }
+
+    public void setConnection(String connection) {
+        this.connection = connection;
+    }
+
+    public SqlProviderConfig.Auth getAuth() {
+        return auth;
+    }
+
+    public void setAuth(SqlProviderConfig.Auth auth) {
+        this.auth = auth;
+    }
+
+    public SqlProviderConfig.Directory getDirectory() {
+        return directory;
+    }
+
+    public void setDirectory(SqlProviderConfig.Directory directory) {
+        this.directory = directory;
+    }
+
+    public SqlProviderConfig.Identity getIdentity() {
+        return identity;
+    }
+
+    public void setIdentity(SqlProviderConfig.Identity identity) {
+        this.identity = identity;
+    }
+
+    protected abstract String getProviderName();
+
+    public void build() {
+        log.info("--- " + getProviderName() + " Provider config ---");
+
+        if (getAuth().isEnabled() == null) {
+            getAuth().setEnabled(isEnabled());
+        }
+
+        if (getDirectory().isEnabled() == null) {
+            getDirectory().setEnabled(isEnabled());
+        }
+
+        if (getIdentity().isEnabled() == null) {
+            getIdentity().setEnabled(isEnabled());
+        }
+
+        log.info("Enabled: {}", isEnabled());
+        if (isEnabled()) {
+            log.info("Type: {}", getType());
+            log.info("Connection: {}", getConnection());
+            log.info("Auth enabled: {}", getAuth().isEnabled());
+            log.info("Directory queries: {}", GsonUtil.build().toJson(getDirectory().getQuery()));
+            log.info("Identity type: {}", getIdentity().getType());
+            log.info("Identity medium queries: {}", GsonUtil.build().toJson(getIdentity().getMedium()));
+        }
+    }
+
+}

src/main/java/io/kamax/mxisd/config/sql/SqlProviderAuthConfig.java

@@ -1,21 +0,0 @@
-package io.kamax.mxisd.config.sql;
-
-import org.springframework.boot.context.properties.ConfigurationProperties;
-import org.springframework.context.annotation.Configuration;
-
-// Unused
-@Configuration
-@ConfigurationProperties("sql.auth")
-public class SqlProviderAuthConfig {
-
-    private boolean enabled;
-
-    public boolean isEnabled() {
-        return enabled;
-    }
-
-    public void setEnabled(boolean enabled) {
-        this.enabled = enabled;
-    }
-
-}

src/main/java/io/kamax/mxisd/config/sql/SqlProviderConfig.java

@@ -20,77 +20,25 @@
 
 package io.kamax.mxisd.config.sql;
 
-import com.google.gson.Gson;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 import org.springframework.boot.context.properties.ConfigurationProperties;
 import org.springframework.context.annotation.Configuration;
+import org.springframework.context.annotation.Primary;
 
 import javax.annotation.PostConstruct;
 
 @Configuration
 @ConfigurationProperties("sql")
-public class SqlProviderConfig {
+@Primary
+public class SqlProviderConfig extends SqlConfig {
 
-    private Logger log = LoggerFactory.getLogger(SqlProviderConfig.class);
-
-    private boolean enabled;
-    private String type;
-    private String connection;
-    private SqlProviderAuthConfig auth;
-    private SqlProviderIdentityConfig identity;
-
-    public boolean isEnabled() {
-        return enabled;
-    }
-
-    public void setEnabled(boolean enabled) {
-        this.enabled = enabled;
-    }
-
-    public String getType() {
-        return type;
-    }
-
-    public void setType(String type) {
-        this.type = type;
-    }
-
-    public String getConnection() {
-        return connection;
-    }
-
-    public void setConnection(String connection) {
-        this.connection = connection;
-    }
-
-    public SqlProviderAuthConfig getAuth() {
-        return auth;
-    }
-
-    public void setAuth(SqlProviderAuthConfig auth) {
-        this.auth = auth;
-    }
-
-    public SqlProviderIdentityConfig getIdentity() {
-        return identity;
-    }
-
-    public void setIdentity(SqlProviderIdentityConfig identity) {
-        this.identity = identity;
+    @Override
+    protected String getProviderName() {
+        return "Generic SQL";
     }
 
     @PostConstruct
-    private void postConstruct() {
-        log.info("--- SQL Provider config ---");
-        log.info("Enabled: {}", isEnabled());
-        if (isEnabled()) {
-            log.info("Type: {}", getType());
-            log.info("Connection: {}", getConnection());
-            log.info("Auth enabled: {}", getAuth().isEnabled());
-            log.info("Identy type: {}", getIdentity().getType());
-            log.info("Identity medium queries: {}", new Gson().toJson(getIdentity().getMedium()));
-        }
+    public void build() {
+        super.build();
     }
 
-}
+}

src/main/java/io/kamax/mxisd/config/sql/synapse/SynapseSqlProviderConfig.java

@@ -18,35 +18,26 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.config;
+package io.kamax.mxisd.config.sql.synapse;
 
-import org.apache.commons.lang.StringUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
+import io.kamax.mxisd.config.sql.SqlConfig;
 import org.springframework.boot.context.properties.ConfigurationProperties;
 import org.springframework.context.annotation.Configuration;
 
-import java.util.Optional;
+import javax.annotation.PostConstruct;
 
 @Configuration
-@ConfigurationProperties("dns.overwrite")
-public class DnsOverwrite {
+@ConfigurationProperties("synapseSql")
+public class SynapseSqlProviderConfig extends SqlConfig {
 
-    private Logger log = LoggerFactory.getLogger(DnsOverwrite.class);
+    @Override
+    protected String getProviderName() {
+        return "Synapse SQL";
+    }
 
-    @Autowired
-    private ServerConfig srvCfg;
-
-    @Autowired
-    private DnsOverwriteEntry homeserver;
-
-    public Optional<DnsOverwriteEntry> findHost(String lookup) {
-        if (homeserver != null && StringUtils.equalsIgnoreCase(lookup, homeserver.getName())) {
-            return Optional.of(homeserver);
-        }
-
-        return Optional.empty();
+    @PostConstruct
+    public void build() {
+        super.build();
     }
 
 }

src/main/java/io/kamax/mxisd/config/threepid/connector/EmailSendGridConfig.java

@@ -0,0 +1,202 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.config.threepid.connector;
+
+import io.kamax.mxisd.util.GsonUtil;
+import org.apache.commons.lang.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.context.annotation.Configuration;
+
+import javax.annotation.PostConstruct;
+
+@Configuration
+@ConfigurationProperties("notification.handlers.sendgrid")
+public class EmailSendGridConfig {
+
+    public static class EmailTemplate {
+
+        public static class EmailBody {
+
+            private String text;
+            private String html;
+
+            public String getText() {
+                return text;
+            }
+
+            public void setText(String text) {
+                this.text = text;
+            }
+
+            public String getHtml() {
+                return html;
+            }
+
+            public void setHtml(String html) {
+                this.html = html;
+            }
+
+        }
+
+        private String subject;
+        private EmailBody body = new EmailBody();
+
+        public String getSubject() {
+            return subject;
+        }
+
+        public void setSubject(String subject) {
+            this.subject = subject;
+        }
+
+        public EmailBody getBody() {
+            return body;
+        }
+
+        public void setBody(EmailBody body) {
+            this.body = body;
+        }
+
+    }
+
+    public static class Api {
+
+        private String key;
+
+        public String getKey() {
+            return key;
+        }
+
+        public void setKey(String key) {
+            this.key = key;
+        }
+
+    }
+
+    public static class Identity {
+
+        private String from;
+        private String name;
+
+        public String getFrom() {
+            return from;
+        }
+
+        public void setFrom(String from) {
+            this.from = from;
+        }
+
+        public String getName() {
+            return name;
+        }
+
+        public void setName(String name) {
+            this.name = name;
+        }
+
+    }
+
+    public static class Templates {
+
+        public static class TemplateSession {
+
+            private EmailTemplate local = new EmailTemplate();
+            private EmailTemplate remote = new EmailTemplate();
+
+            public EmailTemplate getLocal() {
+                return local;
+            }
+
+            public void setLocal(EmailTemplate local) {
+                this.local = local;
+            }
+
+            public EmailTemplate getRemote() {
+                return remote;
+            }
+
+            public void setRemote(EmailTemplate remote) {
+                this.remote = remote;
+            }
+        }
+
+        private EmailTemplate invite = new EmailTemplate();
+        private TemplateSession session = new TemplateSession();
+
+        public EmailTemplate getInvite() {
+            return invite;
+        }
+
+        public void setInvite(EmailTemplate invite) {
+            this.invite = invite;
+        }
+
+        public TemplateSession getSession() {
+            return session;
+        }
+
+        public void setSession(TemplateSession session) {
+            this.session = session;
+        }
+
+    }
+
+    private Logger log = LoggerFactory.getLogger(EmailSendGridConfig.class);
+
+    private Api api = new Api();
+    private Identity identity = new Identity();
+    private Templates templates = new Templates();
+
+    public Api getApi() {
+        return api;
+    }
+
+    public void setApi(Api api) {
+        this.api = api;
+    }
+
+    public Identity getIdentity() {
+        return identity;
+    }
+
+    public void setIdentity(Identity identity) {
+        this.identity = identity;
+    }
+
+    public Templates getTemplates() {
+        return templates;
+    }
+
+    public void setTemplates(Templates templates) {
+        this.templates = templates;
+    }
+
+    @PostConstruct
+    public void build() {
+        log.info("--- Email SendGrid connector config ---");
+        log.info("API key configured?: {}", StringUtils.isNotBlank(api.getKey()));
+        log.info("Identity: {}", GsonUtil.build().toJson(identity));
+        log.info("Templates: {}", GsonUtil.build().toJson(templates));
+    }
+
+}

src/main/java/io/kamax/mxisd/config/threepid/notification/NotificationConfig.java

@@ -18,44 +18,40 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.config.sql;
+package io.kamax.mxisd.config.threepid.notification;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 import org.springframework.boot.context.properties.ConfigurationProperties;
 import org.springframework.context.annotation.Configuration;
 
+import javax.annotation.PostConstruct;
 import java.util.HashMap;
 import java.util.Map;
 
 @Configuration
-@ConfigurationProperties("sql.identity")
-public class SqlProviderIdentityConfig {
+@ConfigurationProperties("notification")
+public class NotificationConfig {
 
-    private String type;
-    private String query;
-    private Map<String, String> medium = new HashMap<>();
+    private Logger log = LoggerFactory.getLogger(NotificationConfig.class);
 
-    public String getType() {
-        return type;
+    private Map<String, String> handler = new HashMap<>();
+
+    public Map<String, String> getHandler() {
+        return handler;
     }
 
-    public void setType(String type) {
-        this.type = type;
+    public void setHandler(Map<String, String> handler) {
+        this.handler = handler;
     }
 
-    public String getQuery() {
-        return query;
-    }
-
-    public void setQuery(String query) {
-        this.query = query;
-    }
-
-    public Map<String, String> getMedium() {
-        return medium;
-    }
-
-    public void setMedium(Map<String, String> medium) {
-        this.medium = medium;
+    @PostConstruct
+    public void build() {
+        log.info("--- Notification config ---");
+        log.info("Handlers:");
+        handler.forEach((k, v) -> {
+            log.info("\t{}: {}", k, v);
+        });
     }
 
 }

src/main/java/io/kamax/mxisd/controller/DefaultExceptionHandler.java

@@ -18,14 +18,12 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller;
 
 import com.google.gson.Gson;
 import com.google.gson.JsonObject;
-import io.kamax.mxisd.exception.BadRequestException;
-import io.kamax.mxisd.exception.InternalServerError;
-import io.kamax.mxisd.exception.MappingAlreadyExistsException;
-import io.kamax.mxisd.exception.MatrixException;
+import com.google.gson.JsonSyntaxException;
+import io.kamax.mxisd.exception.*;
 import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -47,47 +45,75 @@ public class DefaultExceptionHandler {
 
     private static Gson gson = new Gson();
 
-    static String handle(String erroCode, String error) {
+    private String handle(HttpServletRequest req, String erroCode, String error) {
         JsonObject obj = new JsonObject();
         obj.addProperty("errcode", erroCode);
         obj.addProperty("error", error);
         obj.addProperty("success", false);
+        log.info("Request {} {} - Error {}: {}", req.getMethod(), req.getRequestURL(), erroCode, error);
         return gson.toJson(obj);
     }
 
     @ExceptionHandler(InternalServerError.class)
-    public String handle(InternalServerError e, HttpServletResponse response) {
+    public String handle(HttpServletRequest request, HttpServletResponse response, InternalServerError e) {
         if (StringUtils.isNotBlank(e.getInternalReason())) {
             log.error("Reference #{} - {}", e.getReference(), e.getInternalReason());
         } else {
             log.error("Reference #{}", e);
         }
 
-        return handleGeneric(e, response);
+        return handleGeneric(request, response, e);
+    }
+
+    @ExceptionHandler(FeatureNotAvailable.class)
+    public String handle(HttpServletRequest request, HttpServletResponse response, FeatureNotAvailable e) {
+        if (StringUtils.isNotBlank(e.getInternalReason())) {
+            log.error("Feature not available: {}", e.getInternalReason());
+        }
+
+        return handleGeneric(request, response, e);
     }
 
     @ExceptionHandler(MatrixException.class)
-    public String handleGeneric(MatrixException e, HttpServletResponse response) {
+    public String handleGeneric(HttpServletRequest request, HttpServletResponse response, MatrixException e) {
         response.setStatus(e.getStatus());
-        return handle(e.getErrorCode(), e.getError());
+        return handle(request, e.getErrorCode(), e.getError());
     }
 
     @ResponseStatus(HttpStatus.BAD_REQUEST)
     @ExceptionHandler(MissingServletRequestParameterException.class)
-    public String handle(MissingServletRequestParameterException e) {
-        return handle("M_INVALID_BODY", e.getMessage());
+    public String handle(HttpServletRequest req, MissingServletRequestParameterException e) {
+        return handle(req, "M_INCOMPLETE_REQUEST", e.getMessage());
+    }
+
+    @ResponseStatus(HttpStatus.BAD_REQUEST)
+    @ExceptionHandler(InvalidResponseJsonException.class)
+    public String handle(HttpServletRequest req, InvalidResponseJsonException e) {
+        return handle(req, "M_INVALID_JSON", e.getMessage());
+    }
+
+    @ResponseStatus(HttpStatus.BAD_REQUEST)
+    @ExceptionHandler(JsonSyntaxException.class)
+    public String handle(HttpServletRequest req, JsonSyntaxException e) {
+        return handle(req, "M_INVALID_JSON", e.getMessage());
+    }
+
+    @ResponseStatus(HttpStatus.BAD_REQUEST)
+    @ExceptionHandler(JsonMemberNotFoundException.class)
+    public String handle(HttpServletRequest req, JsonMemberNotFoundException e) {
+        return handle(req, "M_JSON_MISSING_KEYS", e.getMessage());
     }
 
     @ResponseStatus(HttpStatus.BAD_REQUEST)
     @ExceptionHandler(MappingAlreadyExistsException.class)
-    public String handle(MappingAlreadyExistsException e) {
-        return handle("M_ALREADY_EXISTS", e.getMessage());
+    public String handle(HttpServletRequest req, MappingAlreadyExistsException e) {
+        return handle(req, "M_ALREADY_EXISTS", e.getMessage());
     }
 
     @ResponseStatus(HttpStatus.BAD_REQUEST)
     @ExceptionHandler(BadRequestException.class)
-    public String handle(BadRequestException e) {
-        return handle("M_BAD_REQUEST", e.getMessage());
+    public String handle(HttpServletRequest req, BadRequestException e) {
+        return handle(req, "M_BAD_REQUEST", e.getMessage());
     }
 
     @ResponseStatus(HttpStatus.INTERNAL_SERVER_ERROR)
@@ -95,10 +121,11 @@ public class DefaultExceptionHandler {
     public String handle(HttpServletRequest req, RuntimeException e) {
         log.error("Unknown error when handling {}", req.getRequestURL(), e);
         return handle(
+                req,
                 "M_UNKNOWN",
                 StringUtils.defaultIfBlank(
                         e.getMessage(),
-                        "An internal server error occured. If this error persists, please contact support with reference #" +
+                        "An internal server error occurred. If this error persists, please contact support with reference #" +
                                 Instant.now().toEpochMilli()
                 )
         );

src/main/java/io/kamax/mxisd/controller/auth/v1/AuthController.java

@@ -18,15 +18,17 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.auth.v1;
 
 import com.google.gson.Gson;
 import com.google.gson.JsonElement;
 import com.google.gson.JsonObject;
-import com.google.gson.JsonParser;
 import io.kamax.mxisd.auth.AuthManager;
 import io.kamax.mxisd.auth.UserAuthResult;
-import org.apache.commons.io.IOUtils;
+import io.kamax.mxisd.controller.auth.v1.io.CredentialsValidationResponse;
+import io.kamax.mxisd.exception.JsonMemberNotFoundException;
+import io.kamax.mxisd.util.GsonParser;
+import io.kamax.mxisd.util.GsonUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.springframework.beans.factory.annotation.Autowired;
@@ -38,7 +40,6 @@ import org.springframework.web.bind.annotation.RestController;
 
 import javax.servlet.http.HttpServletRequest;
 import java.io.IOException;
-import java.nio.charset.StandardCharsets;
 
 @RestController
 @CrossOrigin
@@ -47,7 +48,8 @@ public class AuthController {
 
     private Logger log = LoggerFactory.getLogger(AuthController.class);
 
-    private Gson gson = new Gson();
+    private Gson gson = GsonUtil.build();
+    private GsonParser parser = new GsonParser(gson);
 
     @Autowired
     private AuthManager mgr;
@@ -55,14 +57,9 @@ public class AuthController {
     @RequestMapping(value = "/_matrix-internal/identity/v1/check_credentials", method = RequestMethod.POST)
     public String checkCredentials(HttpServletRequest req) {
         try {
-            JsonElement el = new JsonParser().parse(IOUtils.toString(req.getInputStream(), StandardCharsets.UTF_8));
-            if (!el.isJsonObject() || !el.getAsJsonObject().has("user")) {
-                throw new IllegalArgumentException("Missing user key");
-            }
-
-            JsonObject authData = el.getAsJsonObject().get("user").getAsJsonObject();
+            JsonObject authData = parser.parse(req.getInputStream(), "user");
             if (!authData.has("id") || !authData.has("password")) {
-                throw new IllegalArgumentException("Missing id or password keys");
+                throw new JsonMemberNotFoundException("Missing id or password keys");
             }
 
             String id = authData.get("id").getAsString();
@@ -70,16 +67,17 @@ public class AuthController {
             String password = authData.get("password").getAsString();
 
             UserAuthResult result = mgr.authenticate(id, password);
+            CredentialsValidationResponse response = new CredentialsValidationResponse(result.isSuccess());
 
-            JsonObject authObj = new JsonObject();
-            authObj.addProperty("success", result.isSuccess());
             if (result.isSuccess()) {
-                authObj.addProperty("mxid", result.getMxid());
-                authObj.addProperty("display_name", result.getDisplayName());
+                response.setDisplayName(result.getDisplayName());
+                response.getProfile().setThreePids(result.getThreePids());
             }
-            JsonObject obj = new JsonObject();
+            JsonElement authObj = gson.toJsonTree(response);
 
-            obj.add("authentication", authObj);
+            JsonObject obj = new JsonObject();
+            obj.add("auth", authObj);
+            obj.add("authentication", authObj); // TODO remove later, legacy support
             return gson.toJson(obj);
         } catch (IOException e) {
             throw new RuntimeException(e);

src/main/java/io/kamax/mxisd/controller/auth/v1/io/CredentialsValidationResponse.java

@@ -0,0 +1,74 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.controller.auth.v1.io;
+
+import io.kamax.mxisd.ThreePid;
+
+import java.util.HashSet;
+import java.util.Set;
+
+public class CredentialsValidationResponse {
+
+    public static class Profile {
+
+        private String displayName;
+        private Set<ThreePid> threePids = new HashSet<>();
+
+        public String getDisplayName() {
+            return displayName;
+        }
+
+        public Set<ThreePid> getThreePids() {
+            return threePids;
+        }
+
+        public void setThreePids(Set<ThreePid> threePids) {
+            this.threePids = new HashSet<>(threePids);
+        }
+
+    }
+
+    private boolean success;
+    private String displayName; // TODO remove later, legacy support
+    private Profile profile = new Profile();
+
+    public CredentialsValidationResponse(boolean success) {
+        this.success = success;
+    }
+
+    public boolean isSuccess() {
+        return success;
+    }
+
+    public String getDisplayName() {
+        return displayName;
+    }
+
+    public void setDisplayName(String displayName) {
+        this.displayName = displayName;
+        this.profile.displayName = displayName;
+    }
+
+    public Profile getProfile() {
+        return profile;
+    }
+
+}

src/main/java/io/kamax/mxisd/controller/directory/v1/UserDirectoryController.java

@@ -0,0 +1,56 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.controller.directory.v1;
+
+import com.google.gson.Gson;
+import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchRequest;
+import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult;
+import io.kamax.mxisd.directory.DirectoryManager;
+import io.kamax.mxisd.util.GsonParser;
+import io.kamax.mxisd.util.GsonUtil;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.MediaType;
+import org.springframework.web.bind.annotation.*;
+
+import javax.servlet.http.HttpServletRequest;
+import java.io.IOException;
+import java.net.URI;
+
+@RestController
+@CrossOrigin
+@RequestMapping(path = "/_matrix/client/r0/user_directory", produces = MediaType.APPLICATION_JSON_UTF8_VALUE)
+public class UserDirectoryController {
+
+    private Gson gson = GsonUtil.build();
+    private GsonParser parser = new GsonParser(gson);
+
+    @Autowired
+    private DirectoryManager mgr;
+
+    @RequestMapping(path = "/search", method = RequestMethod.POST)
+    public String search(HttpServletRequest request, @RequestParam("access_token") String accessToken) throws IOException {
+        UserDirectorySearchRequest searchQuery = parser.parse(request, UserDirectorySearchRequest.class);
+        URI target = URI.create(request.getRequestURL().toString());
+        UserDirectorySearchResult result = mgr.search(target, accessToken, searchQuery.getSearchTerm());
+        return gson.toJson(result);
+    }
+
+}

src/main/java/io/kamax/mxisd/controller/directory/v1/io/UserDirectorySearchRequest.java

@@ -0,0 +1,48 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.controller.directory.v1.io;
+
+public class UserDirectorySearchRequest {
+
+    private String by;
+    private String searchTerm;
+
+    public UserDirectorySearchRequest(String searchTerm) {
+        setSearchTerm(searchTerm);
+    }
+
+    public String getBy() {
+        return by;
+    }
+
+    public void setBy(String by) {
+        this.by = by;
+    }
+
+    public String getSearchTerm() {
+        return searchTerm;
+    }
+
+    public void setSearchTerm(String searchTerm) {
+        this.searchTerm = searchTerm;
+    }
+
+}

src/main/java/io/kamax/mxisd/controller/directory/v1/io/UserDirectorySearchResult.java

@@ -0,0 +1,83 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.controller.directory.v1.io;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class UserDirectorySearchResult {
+
+    public static class Result {
+
+        private String displayName;
+        private String avatarUrl;
+        private String userId;
+
+        public String getDisplayName() {
+            return displayName;
+        }
+
+        public void setDisplayName(String displayName) {
+            this.displayName = displayName;
+        }
+
+        public String getAvatarUrl() {
+            return avatarUrl;
+        }
+
+        public void setAvatarUrl(String avatarUrl) {
+            this.avatarUrl = avatarUrl;
+        }
+
+        public String getUserId() {
+            return userId;
+        }
+
+        public void setUserId(String userId) {
+            this.userId = userId;
+        }
+
+    }
+
+    private boolean limited;
+    private List<Result> results = new ArrayList<>();
+
+    public boolean isLimited() {
+        return limited;
+    }
+
+    public void setLimited(boolean limited) {
+        this.limited = limited;
+    }
+
+    public List<Result> getResults() {
+        return results;
+    }
+
+    public void setResults(List<Result> results) {
+        this.results = results;
+    }
+
+    public void addResult(Result result) {
+        this.results.add(result);
+    }
+
+}

src/main/java/io/kamax/mxisd/controller/identity/v1/ClientBulkLookupAnswer.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.identity.v1;
 
 import io.kamax.mxisd.lookup.ThreePidMapping;
 

src/main/java/io/kamax/mxisd/controller/identity/v1/ClientBulkLookupRequest.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.identity.v1;
 
 import io.kamax.mxisd.lookup.ThreePidMapping;
 

src/main/java/io/kamax/mxisd/controller/identity/v1/IdentityAPIv1.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.identity.v1;
 
 public class IdentityAPIv1 {
 

src/main/java/io/kamax/mxisd/controller/identity/v1/InvitationController.java

@@ -18,12 +18,12 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.identity.v1;
 
 import com.google.gson.Gson;
 import io.kamax.matrix.MatrixID;
 import io.kamax.mxisd.config.ServerConfig;
-import io.kamax.mxisd.controller.v1.io.ThreePidInviteReplyIO;
+import io.kamax.mxisd.controller.identity.v1.io.ThreePidInviteReplyIO;
 import io.kamax.mxisd.invitation.IThreePidInvite;
 import io.kamax.mxisd.invitation.IThreePidInviteReply;
 import io.kamax.mxisd.invitation.InvitationManager;

src/main/java/io/kamax/mxisd/controller/identity/v1/KeyController.java

@@ -18,11 +18,11 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.identity.v1;
 
 import com.google.gson.Gson;
 import com.google.gson.JsonObject;
-import io.kamax.mxisd.controller.v1.io.KeyValidityJson;
+import io.kamax.mxisd.controller.identity.v1.io.KeyValidityJson;
 import io.kamax.mxisd.exception.BadRequestException;
 import io.kamax.mxisd.key.KeyManager;
 import org.apache.commons.lang.StringUtils;

src/main/java/io/kamax/mxisd/controller/identity/v1/MappingController.java

@@ -18,11 +18,11 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.identity.v1;
 
 import com.google.gson.Gson;
 import com.google.gson.JsonObject;
-import io.kamax.mxisd.controller.v1.io.SingeLookupReplyJson;
+import io.kamax.mxisd.controller.identity.v1.io.SingeLookupReplyJson;
 import io.kamax.mxisd.exception.InternalServerError;
 import io.kamax.mxisd.lookup.*;
 import io.kamax.mxisd.lookup.strategy.LookupStrategy;
@@ -66,9 +66,11 @@ public class MappingController {
     private void setRequesterInfo(ALookupRequest lookupReq, HttpServletRequest req) {
         lookupReq.setRequester(req.getRemoteAddr());
         String xff = req.getHeader("X-FORWARDED-FOR");
-        lookupReq.setRecursive(StringUtils.isNotBlank(xff));
-        if (lookupReq.isRecursive()) {
+        log.debug("XFF header: {}", xff);
+        lookupReq.setRecursive(StringUtils.isBlank(xff));
+        if (!lookupReq.isRecursive()) {
             lookupReq.setRecurseHosts(Arrays.asList(xff.split(",")));
+            lookupReq.setRequester(lookupReq.getRecurseHosts().get(lookupReq.getRecurseHosts().size() - 1));
         }
 
         lookupReq.setUserAgent(req.getHeader("USER-AGENT"));
@@ -106,7 +108,7 @@ public class MappingController {
     String bulkLookup(HttpServletRequest request) {
         BulkLookupRequest lookupRequest = new BulkLookupRequest();
         setRequesterInfo(lookupRequest, request);
-        log.info("Got single lookup request from {} with client {} - Is recursive? {}", lookupRequest.getRequester(), lookupRequest.getUserAgent(), lookupRequest.isRecursive());
+        log.info("Got bulk lookup request from {} with client {} - Is recursive? {}", lookupRequest.getRequester(), lookupRequest.getUserAgent(), lookupRequest.isRecursive());
 
         try {
             ClientBulkLookupRequest input = parser.parse(request, ClientBulkLookupRequest.class);

src/main/java/io/kamax/mxisd/controller/identity/v1/SessionController.java

@@ -18,12 +18,11 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.identity.v1;
 
 import io.kamax.mxisd.config.ServerConfig;
 import io.kamax.mxisd.config.ViewConfig;
-import io.kamax.mxisd.controller.v1.remote.RemoteIdentityAPIv1;
-import io.kamax.mxisd.exception.InternalServerError;
+import io.kamax.mxisd.controller.identity.v1.remote.RemoteIdentityAPIv1;
 import io.kamax.mxisd.session.SessionMananger;
 import io.kamax.mxisd.session.ValidationResult;
 import org.slf4j.Logger;
@@ -36,7 +35,8 @@ import org.springframework.web.bind.annotation.RequestParam;
 
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
-import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
 
 import static org.springframework.web.bind.annotation.RequestMethod.GET;
 
@@ -69,15 +69,15 @@ class SessionController {
         ValidationResult r = mgr.validate(sid, secret, token);
         log.info("Session {} was validated", sid);
         if (r.getNextUrl().isPresent()) {
-            String url = srvCfg.getPublicUrl() + r.getNextUrl().get();
-            log.info("Session {} validation: next URL is present, redirecting to {}", sid, url);
+            String url = r.getNextUrl().get();
             try {
-                response.sendRedirect(url);
-                return "";
-            } catch (IOException e) {
-                log.warn("Unable to redirect user to {}", url);
-                throw new InternalServerError(e);
+                url = new URL(url).toString();
+            } catch (MalformedURLException e) {
+                log.info("Session next URL {} is not a valid one, will prepend public URL {}", url, srvCfg.getPublicUrl());
+                url = srvCfg.getPublicUrl() + r.getNextUrl().get();
             }
+            log.info("Session {} validation: next URL is present, redirecting to {}", sid, url);
+            return "redirect:" + url;
         } else {
             if (r.isCanRemote()) {
                 String url = srvCfg.getPublicUrl() + RemoteIdentityAPIv1.getRequestToken(r.getSession().getId(), r.getSession().getSecret());

src/main/java/io/kamax/mxisd/controller/identity/v1/SessionRestController.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.identity.v1;
 
 import com.google.gson.Gson;
 import com.google.gson.JsonObject;
@@ -26,9 +26,9 @@ import io.kamax.matrix.ThreePidMedium;
 import io.kamax.mxisd.ThreePid;
 import io.kamax.mxisd.config.ServerConfig;
 import io.kamax.mxisd.config.ViewConfig;
-import io.kamax.mxisd.controller.v1.io.SessionEmailTokenRequestJson;
-import io.kamax.mxisd.controller.v1.io.SessionPhoneTokenRequestJson;
-import io.kamax.mxisd.controller.v1.io.SuccessStatusJson;
+import io.kamax.mxisd.controller.identity.v1.io.SessionEmailTokenRequestJson;
+import io.kamax.mxisd.controller.identity.v1.io.SessionPhoneTokenRequestJson;
+import io.kamax.mxisd.controller.identity.v1.io.SuccessStatusJson;
 import io.kamax.mxisd.exception.BadRequestException;
 import io.kamax.mxisd.exception.SessionNotValidatedException;
 import io.kamax.mxisd.invitation.InvitationManager;

src/main/java/io/kamax/mxisd/controller/identity/v1/StatusController.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1;
+package io.kamax.mxisd.controller.identity.v1;
 
 import com.google.gson.Gson;
 import com.google.gson.JsonObject;

src/main/java/io/kamax/mxisd/controller/identity/v1/io/GenericTokenRequestJson.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1.io;
+package io.kamax.mxisd.controller.identity.v1.io;
 
 public abstract class GenericTokenRequestJson {
 

src/main/java/io/kamax/mxisd/controller/identity/v1/io/KeyValidityJson.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1.io;
+package io.kamax.mxisd.controller.identity.v1.io;
 
 public class KeyValidityJson {
 

src/main/java/io/kamax/mxisd/controller/identity/v1/io/RequestTokenResponse.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1.io;
+package io.kamax.mxisd.controller.identity.v1.io;
 
 public class RequestTokenResponse {
 

src/main/java/io/kamax/mxisd/controller/identity/v1/io/SessionEmailTokenRequestJson.java

@@ -18,14 +18,16 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1.io;
+package io.kamax.mxisd.controller.identity.v1.io;
+
+import io.kamax.matrix.ThreePidMedium;
 
 public class SessionEmailTokenRequestJson extends GenericTokenRequestJson {
 
     private String email;
 
     public String getMedium() {
-        return "threepids/email";
+        return ThreePidMedium.Email.getId();
     }
 
     public String getValue() {

src/main/java/io/kamax/mxisd/controller/identity/v1/io/SessionPhoneTokenRequestJson.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1.io;
+package io.kamax.mxisd.controller.identity.v1.io;
 
 import com.google.i18n.phonenumbers.NumberParseException;
 import com.google.i18n.phonenumbers.PhoneNumberUtil;

src/main/java/io/kamax/mxisd/controller/identity/v1/io/SingeLookupReplyJson.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1.io;
+package io.kamax.mxisd.controller.identity.v1.io;
 
 import io.kamax.mxisd.lookup.SingleLookupReply;
 

src/main/java/io/kamax/mxisd/controller/identity/v1/io/SuccessStatusJson.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1.io;
+package io.kamax.mxisd.controller.identity.v1.io;
 
 public class SuccessStatusJson {
 

src/main/java/io/kamax/mxisd/controller/identity/v1/io/ThreePidInviteReplyIO.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1.io;
+package io.kamax.mxisd.controller.identity.v1.io;
 
 import io.kamax.mxisd.invitation.IThreePidInviteReply;
 

src/main/java/io/kamax/mxisd/controller/identity/v1/remote/RemoteIdentityAPIv1.java

@@ -18,7 +18,7 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.v1.remote;
+package io.kamax.mxisd.controller.identity.v1.remote;
 
 public class RemoteIdentityAPIv1 {
 

src/main/java/io/kamax/mxisd/controller/identity/v1/remote/RemoteSessionController.java

@@ -1,4 +1,4 @@
-package io.kamax.mxisd.controller.v1.remote;
+package io.kamax.mxisd.controller.identity.v1.remote;
 
 import io.kamax.mxisd.config.ViewConfig;
 import io.kamax.mxisd.exception.SessionNotValidatedException;
@@ -14,8 +14,8 @@ import org.springframework.web.bind.annotation.RequestParam;
 
 import javax.servlet.http.HttpServletRequest;
 
-import static io.kamax.mxisd.controller.v1.remote.RemoteIdentityAPIv1.SESSION_CHECK;
-import static io.kamax.mxisd.controller.v1.remote.RemoteIdentityAPIv1.SESSION_REQUEST_TOKEN;
+import static io.kamax.mxisd.controller.identity.v1.remote.RemoteIdentityAPIv1.SESSION_CHECK;
+import static io.kamax.mxisd.controller.identity.v1.remote.RemoteIdentityAPIv1.SESSION_REQUEST_TOKEN;
 
 @Controller
 public class RemoteSessionController {

src/main/java/io/kamax/mxisd/directory/DirectoryManager.java

@@ -0,0 +1,133 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.directory;
+
+import com.google.gson.Gson;
+import com.google.gson.JsonSyntaxException;
+import io.kamax.matrix.MatrixErrorInfo;
+import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchRequest;
+import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult;
+import io.kamax.mxisd.dns.ClientDnsOverwrite;
+import io.kamax.mxisd.exception.InternalServerError;
+import io.kamax.mxisd.exception.MatrixException;
+import io.kamax.mxisd.util.GsonUtil;
+import io.kamax.mxisd.util.RestClientUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.io.IOException;
+import java.net.URI;
+import java.nio.charset.Charset;
+import java.util.List;
+import java.util.stream.Collectors;
+
+@Component
+public class DirectoryManager {
+
+    private Logger log = LoggerFactory.getLogger(DirectoryManager.class);
+
+    private List<IDirectoryProvider> providers;
+
+    private ClientDnsOverwrite dns;
+    private CloseableHttpClient client;
+    private Gson gson;
+
+    @Autowired
+    public DirectoryManager(List<IDirectoryProvider> providers, ClientDnsOverwrite dns) {
+        this.dns = dns;
+        this.client = HttpClients.custom().setUserAgent("mxisd").build(); //FIXME centralize
+        this.gson = GsonUtil.build();
+        this.providers = providers.stream().filter(IDirectoryProvider::isEnabled).collect(Collectors.toList());
+
+        log.info("Directory providers:");
+        this.providers.forEach(p -> log.info("\t- {}", p.getClass().getName()));
+    }
+
+    public UserDirectorySearchResult search(URI target, String accessToken, String query) {
+        log.info("Performing search for '{}'", query);
+        log.info("Original request URL: {}", target);
+        UserDirectorySearchResult result = new UserDirectorySearchResult();
+
+        URIBuilder builder = dns.transform(target);
+        log.info("Querying HS at {}", builder);
+        builder.setParameter("access_token", accessToken);
+        HttpPost req = RestClientUtils.post(
+                builder.toString(),
+                new UserDirectorySearchRequest(query));
+        try (CloseableHttpResponse res = client.execute(req)) {
+            int status = res.getStatusLine().getStatusCode();
+            Charset charset = ContentType.getOrDefault(res.getEntity()).getCharset();
+            String body = IOUtils.toString(res.getEntity().getContent(), charset);
+
+            if (status != 200) {
+                MatrixErrorInfo info = gson.fromJson(body, MatrixErrorInfo.class);
+                if (StringUtils.equals("M_UNRECOGNIZED", info.getErrcode())) { // FIXME no hardcoding, use Enum
+                    log.warn("Homeserver does not support Directory feature, skipping");
+                } else {
+                    log.error("Homeserver returned an error while performing directory search");
+                    throw new MatrixException(status, info.getErrcode(), info.getError());
+                }
+            }
+
+            UserDirectorySearchResult resultHs = gson.fromJson(body, UserDirectorySearchResult.class);
+            log.info("Found {} match(es) in HS for '{}'", resultHs.getResults().size(), query);
+            result.getResults().addAll(resultHs.getResults());
+            if (resultHs.isLimited()) {
+                result.setLimited(true);
+            }
+        } catch (JsonSyntaxException e) {
+            throw new InternalServerError("Invalid JSON reply from the HS: " + e.getMessage());
+        } catch (IOException e) {
+            throw new InternalServerError("Unable to query the HS: I/O error: " + e.getMessage());
+        }
+
+        for (IDirectoryProvider provider : providers) {
+            log.info("Using Directory provider {}", provider.getClass().getSimpleName());
+            UserDirectorySearchResult resultProvider = provider.searchByDisplayName(query);
+            log.info("Display name: found {} match(es) for '{}'", resultProvider.getResults().size(), query);
+            result.getResults().addAll(resultProvider.getResults());
+            if (resultProvider.isLimited()) {
+                result.setLimited(true);
+            }
+
+            resultProvider = provider.searchBy3pid(query);
+            log.info("Threepid: found {} match(es) for '{}'", resultProvider.getResults().size(), query);
+            result.getResults().addAll(resultProvider.getResults());
+            if (resultProvider.isLimited()) {
+                result.setLimited(true);
+            }
+        }
+
+        log.info("Total matches: {} - limited? {}", result.getResults().size(), result.isLimited());
+        return result;
+    }
+
+}

src/main/java/io/kamax/mxisd/directory/IDirectoryProvider.java

@@ -0,0 +1,33 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.directory;
+
+import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult;
+
+public interface IDirectoryProvider {
+
+    boolean isEnabled();
+
+    UserDirectorySearchResult searchByDisplayName(String query);
+
+    UserDirectorySearchResult searchBy3pid(String query);
+
+}

src/main/java/io/kamax/mxisd/dns/ClientDnsOverwrite.java

@@ -0,0 +1,74 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.dns;
+
+import io.kamax.mxisd.config.DnsOverwriteConfig;
+import io.kamax.mxisd.exception.ConfigurationException;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.net.MalformedURLException;
+import java.net.URI;
+import java.net.URL;
+import java.util.HashMap;
+import java.util.Map;
+
+import static io.kamax.mxisd.config.DnsOverwriteConfig.Entry;
+
+@Component
+public class ClientDnsOverwrite {
+
+    private Logger log = LoggerFactory.getLogger(ClientDnsOverwrite.class);
+
+    private Map<String, Entry> mappings;
+
+    @Autowired
+    public ClientDnsOverwrite(DnsOverwriteConfig cfg) {
+        mappings = new HashMap<>();
+        cfg.getHomeserver().getClient().forEach(e -> mappings.put(e.getName(), e));
+    }
+
+    public URIBuilder transform(URI initial) {
+        URIBuilder builder = new URIBuilder(initial);
+        Entry mapping = mappings.get(initial.getHost());
+        if (mapping == null) {
+            return builder;
+        }
+
+        try {
+            URL target = new URL(mapping.getValue());
+            builder.setScheme(target.getProtocol());
+            builder.setHost(target.getHost());
+            if (target.getPort() != -1) {
+                builder.setPort(target.getPort());
+            }
+
+            return builder;
+        } catch (MalformedURLException e) {
+            log.warn("Skipping DNS overwrite entry {} due to invalid value [{}]: {}", mapping.getName(), mapping.getValue(), e.getMessage());
+            throw new ConfigurationException("Invalid DNS overwrite entry in homeserver client: " + mapping.getName(), e.getMessage());
+        }
+    }
+
+}

src/main/java/io/kamax/mxisd/dns/FederationDnsOverwrite.java

@@ -0,0 +1,61 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.dns;
+
+import io.kamax.mxisd.config.DnsOverwriteConfig;
+import io.kamax.mxisd.config.ServerConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+import static io.kamax.mxisd.config.DnsOverwriteConfig.Entry;
+
+@Component
+public class FederationDnsOverwrite {
+
+    private Logger log = LoggerFactory.getLogger(FederationDnsOverwrite.class);
+
+    private ServerConfig srvCfg;
+    private Map<String, Entry> mappings;
+
+    @Autowired
+    public FederationDnsOverwrite(DnsOverwriteConfig cfg, ServerConfig srvCfg) {
+        this.srvCfg = srvCfg;
+
+        mappings = new HashMap<>();
+        cfg.getHomeserver().getFederation().forEach(e -> mappings.put(e.getName(), e));
+    }
+
+    public Optional<String> findHost(String lookup) {
+        Entry mapping = mappings.get(lookup);
+        if (mapping == null) {
+            return Optional.empty();
+        }
+
+        return Optional.of(mapping.getValue());
+    }
+
+}

src/main/java/io/kamax/mxisd/exception/FeatureNotAvailable.java

@@ -0,0 +1,43 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.exception;
+
+import org.apache.http.HttpStatus;
+
+public class FeatureNotAvailable extends MatrixException {
+
+    private String internalReason;
+
+    public FeatureNotAvailable(String internalReason) {
+        super(
+                HttpStatus.SC_INTERNAL_SERVER_ERROR,
+                "M_NOT_AVAILABLE",
+                "This action is currently not available. Contact your administrator to enable it."
+        );
+
+        this.internalReason = internalReason;
+    }
+
+    public String getInternalReason() {
+        return internalReason;
+    }
+
+}

src/main/java/io/kamax/mxisd/exception/MatrixException.java

@@ -20,7 +20,7 @@
 
 package io.kamax.mxisd.exception;
 
-public abstract class MatrixException extends MxisdException {
+public class MatrixException extends MxisdException {
 
     private int status;
     private String errorCode;

src/main/java/io/kamax/mxisd/exception/RemoteHomeServerException.java

@@ -0,0 +1,11 @@
+package io.kamax.mxisd.exception;
+
+import org.apache.http.HttpStatus;
+
+public class RemoteHomeServerException extends MatrixException {
+
+    public RemoteHomeServerException(String error) {
+        super(HttpStatus.SC_SERVICE_UNAVAILABLE, "M_REMOTE_HS_ERROR", "Error from remote server: " + error);
+    }
+
+}

src/main/java/io/kamax/mxisd/invitation/InvitationManager.java

@@ -24,8 +24,7 @@ import com.google.gson.Gson;
 import com.google.gson.JsonArray;
 import com.google.gson.JsonObject;
 import io.kamax.matrix.MatrixID;
-import io.kamax.mxisd.config.DnsOverwrite;
-import io.kamax.mxisd.config.DnsOverwriteEntry;
+import io.kamax.mxisd.dns.FederationDnsOverwrite;
 import io.kamax.mxisd.exception.BadRequestException;
 import io.kamax.mxisd.exception.MappingAlreadyExistsException;
 import io.kamax.mxisd.lookup.SingleLookupReply;
@@ -58,6 +57,8 @@ import javax.annotation.PreDestroy;
 import javax.net.ssl.HostnameVerifier;
 import javax.net.ssl.SSLContext;
 import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
 import java.nio.charset.StandardCharsets;
 import java.util.*;
 import java.util.concurrent.ConcurrentHashMap;
@@ -81,7 +82,7 @@ public class InvitationManager {
     private SignatureManager signMgr;
 
     @Autowired
-    private DnsOverwrite dns;
+    private FederationDnsOverwrite dns;
 
     private NotificationManager notifMgr;
 
@@ -160,11 +161,15 @@ public class InvitationManager {
     // TODO use caching mechanism
     // TODO export in matrix-java-sdk
     private String findHomeserverForDomain(String domain) {
-        Optional<DnsOverwriteEntry> entryOpt = dns.findHost(domain);
+        Optional<String> entryOpt = dns.findHost(domain);
         if (entryOpt.isPresent()) {
-            DnsOverwriteEntry entry = entryOpt.get();
-            log.info("Found DNS overwrite for {} to {}", entry.getName(), entry.getTarget());
-            return "https://" + entry.getTarget();
+            String entry = entryOpt.get();
+            log.info("Found DNS overwrite for {} to {}", domain, entry);
+            try {
+                return new URL(entry).toString();
+            } catch (MalformedURLException e) {
+                log.warn("Skipping homeserver Federation DNS overwrite for {} - not a valid URL: {}", domain, entry);
+            }
         }
 
         log.debug("Performing SRV lookup for {}", domain);
@@ -206,9 +211,16 @@ public class InvitationManager {
 
         String invId = getId(invitation);
         log.info("Handling invite for {}:{} from {} in room {}", invitation.getMedium(), invitation.getAddress(), invitation.getSender(), invitation.getRoomId());
-        if (invitations.containsKey(invId)) {
+        IThreePidInviteReply reply = invitations.get(invId);
+        if (reply != null) {
             log.info("Invite is already pending for {}:{}, returning data", invitation.getMedium(), invitation.getAddress());
-            return invitations.get(invId);
+            if (!StringUtils.equals(invitation.getRoomId(), reply.getInvite().getRoomId())) {
+                log.info("Sending new notification as new invite room {} is different from the original {}", invitation.getRoomId(), reply.getInvite().getRoomId());
+                notifMgr.sendForInvite(new ThreePidInviteReply(reply.getId(), invitation, reply.getToken(), reply.getDisplayName()));
+            } else {
+                // FIXME we should check attempt and send if bigger
+            }
+            return reply;
         }
 
         Optional<?> result = lookupMgr.find(invitation.getMedium(), invitation.getAddress(), true);
@@ -220,7 +232,7 @@ public class InvitationManager {
         String token = RandomStringUtils.randomAlphanumeric(64);
         String displayName = invitation.getAddress().substring(0, 3) + "...";
 
-        IThreePidInviteReply reply = new ThreePidInviteReply(invId, invitation, token, displayName);
+        reply = new ThreePidInviteReply(invId, invitation, token, displayName);
 
         log.info("Performing invite to {}:{}", invitation.getMedium(), invitation.getAddress());
         notifMgr.sendForInvite(reply);

src/main/java/io/kamax/mxisd/lookup/SingleLookupReply.java

@@ -24,7 +24,7 @@ import com.google.gson.Gson;
 import com.google.gson.JsonSyntaxException;
 import io.kamax.matrix.MatrixID;
 import io.kamax.matrix._MatrixID;
-import io.kamax.mxisd.controller.v1.io.SingeLookupReplyJson;
+import io.kamax.mxisd.controller.identity.v1.io.SingeLookupReplyJson;
 
 import java.time.Instant;
 

src/main/java/io/kamax/mxisd/lookup/provider/DnsLookupProvider.java

@@ -20,6 +20,7 @@
 
 package io.kamax.mxisd.lookup.provider;
 
+import io.kamax.matrix.ThreePidMedium;
 import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.SingleLookupRequest;
@@ -83,7 +84,7 @@ class DnsLookupProvider implements IThreePidProvider {
 
     @Override
     public Optional<SingleLookupReply> find(SingleLookupRequest request) {
-        if (!StringUtils.equals("threepids/email", request.getType())) { // TODO use enum
+        if (!ThreePidMedium.Email.is(request.getType())) { // TODO use enum
             log.info("Skipping unsupported type {} for {}", request.getType(), request.getThreePid());
             return Optional.empty();
         }
@@ -106,7 +107,7 @@ class DnsLookupProvider implements IThreePidProvider {
         Map<String, List<ThreePidMapping>> domains = new HashMap<>();
 
         for (ThreePidMapping mapping : mappings) {
-            if (!StringUtils.equals("threepids/email", mapping.getMedium())) {
+            if (!ThreePidMedium.Email.is(mapping.getMedium())) {
                 log.info("Skipping unsupported type {} for {}", mapping.getMedium(), mapping.getValue());
                 continue;
             }

src/main/java/io/kamax/mxisd/lookup/provider/RemoteIdentityServerFetcher.java

@@ -23,7 +23,7 @@ package io.kamax.mxisd.lookup.provider;
 import com.google.gson.Gson;
 import com.google.gson.JsonObject;
 import com.google.gson.JsonParseException;
-import io.kamax.mxisd.controller.v1.ClientBulkLookupRequest;
+import io.kamax.mxisd.controller.identity.v1.ClientBulkLookupRequest;
 import io.kamax.mxisd.exception.InvalidResponseJsonException;
 import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.SingleLookupRequest;

src/main/java/io/kamax/mxisd/lookup/strategy/RecursivePriorityLookupStrategy.java

@@ -43,26 +43,28 @@ public class RecursivePriorityLookupStrategy implements LookupStrategy {
 
     private Logger log = LoggerFactory.getLogger(RecursivePriorityLookupStrategy.class);
 
-    @Autowired
-    private RecursiveLookupConfig recursiveCfg;
-
-    @Autowired
+    private RecursiveLookupConfig cfg;
     private List<IThreePidProvider> providers;
-
-    @Autowired
     private IBridgeFetcher bridge;
 
     private List<CIDRUtils> allowedCidr = new ArrayList<>();
 
+    @Autowired
+    public RecursivePriorityLookupStrategy(RecursiveLookupConfig cfg, List<IThreePidProvider> providers, IBridgeFetcher bridge) {
+        this.cfg = cfg;
+        this.bridge = bridge;
+        this.providers = providers.stream().filter(IThreePidProvider::isEnabled).collect(Collectors.toList());
+    }
+
     @PostConstruct
     private void build() throws UnknownHostException {
         try {
             log.info("Found {} providers", providers.size());
-
+            providers.forEach(p -> log.info("\t- {}", p.getClass().getName()));
             providers.sort((o1, o2) -> Integer.compare(o2.getPriority(), o1.getPriority()));
 
-            log.info("Recursive lookup enabled: {}", recursiveCfg.isEnabled());
-            for (String cidr : recursiveCfg.getAllowedCidr()) {
+            log.info("Recursive lookup enabled: {}", cfg.isEnabled());
+            for (String cidr : cfg.getAllowedCidr()) {
                 log.info("{} is allowed for recursion", cidr);
                 allowedCidr.add(new CIDRUtils(cidr));
             }
@@ -75,7 +77,7 @@ public class RecursivePriorityLookupStrategy implements LookupStrategy {
         boolean canRecurse = false;
 
         try {
-            if (recursiveCfg.isEnabled()) {
+            if (cfg.isEnabled()) {
                 log.debug("Checking {} CIDRs for recursion", allowedCidr.size());
                 for (CIDRUtils cidr : allowedCidr) {
                     if (cidr.isInRange(source)) {
@@ -106,7 +108,7 @@ public class RecursivePriorityLookupStrategy implements LookupStrategy {
 
         log.info("Host {} allowed for recursion: {}", request.getRequester(), canRecurse);
         for (IThreePidProvider provider : providers) {
-            if (provider.isEnabled() && (provider.isLocal() || canRecurse || forceRecursive)) {
+            if (provider.isLocal() || canRecurse || forceRecursive) {
                 usableProviders.add(provider);
             }
         }
@@ -159,9 +161,9 @@ public class RecursivePriorityLookupStrategy implements LookupStrategy {
         }
 
         if (
-                recursiveCfg.getBridge() != null &&
-                        recursiveCfg.getBridge().getEnabled() &&
-                        (!recursiveCfg.getBridge().getRecursiveOnly() || isAllowedForRecursive(request.getRequester()))
+                cfg.getBridge() != null &&
+                        cfg.getBridge().getEnabled() &&
+                        (!cfg.getBridge().getRecursiveOnly() || isAllowedForRecursive(request.getRequester()))
                 ) {
             log.info("Using bridge failover for lookup");
             return bridge.find(request);

src/main/java/io/kamax/mxisd/matrix/IdentityServerUtils.java

@@ -4,6 +4,7 @@ import com.google.gson.JsonElement;
 import com.google.gson.JsonParseException;
 import com.google.gson.JsonParser;
 import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.xbill.DNS.*;
@@ -21,13 +22,17 @@ import java.util.Optional;
 // FIXME placeholder, this must go in matrix-java-sdk for 1.0
 public class IdentityServerUtils {
 
-    public static final String THREEPID_TEST_MEDIUM = "threepids/email";
+    public static final String THREEPID_TEST_MEDIUM = "email";
     public static final String THREEPID_TEST_ADDRESS = "mxisd-email-forever-unknown@forever-invalid.kamax.io";
 
     private static Logger log = LoggerFactory.getLogger(IdentityServerUtils.class);
     private static JsonParser parser = new JsonParser();
 
     public static boolean isUsable(String remote) {
+        if (StringUtils.isBlank(remote)) {
+            return false;
+        }
+
         try {
             // FIXME use Apache HTTP client
             HttpURLConnection rootSrvConn = (HttpURLConnection) new URL(
@@ -54,7 +59,7 @@ public class IdentityServerUtils {
             }
 
             return true;
-        } catch (IOException | JsonParseException e) {
+        } catch (IllegalArgumentException | IOException | JsonParseException e) {
             log.info("{} is not a usable Identity Server: {}", remote, e.getMessage());
             return false;
         }

src/main/java/io/kamax/mxisd/notification/INotificationHandler.java

@@ -25,6 +25,8 @@ import io.kamax.mxisd.threepid.session.IThreePidSession;
 
 public interface INotificationHandler {
 
+    String getId();
+
     String getMedium();
 
     void sendForInvite(IThreePidInviteReply invite);

src/main/java/io/kamax/mxisd/notification/NotificationManager.java

@@ -20,9 +20,13 @@
 
 package io.kamax.mxisd.notification;
 
+import io.kamax.mxisd.config.threepid.notification.NotificationConfig;
 import io.kamax.mxisd.exception.NotImplementedException;
 import io.kamax.mxisd.invitation.IThreePidInviteReply;
 import io.kamax.mxisd.threepid.session.IThreePidSession;
+import org.apache.commons.lang.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
@@ -33,12 +37,25 @@ import java.util.Map;
 @Component
 public class NotificationManager {
 
+    private Logger log = LoggerFactory.getLogger(NotificationManager.class);
+
     private Map<String, INotificationHandler> handlers;
 
     @Autowired
-    public NotificationManager(List<INotificationHandler> handlers) {
+    public NotificationManager(NotificationConfig cfg, List<INotificationHandler> handlers) {
         this.handlers = new HashMap<>();
-        handlers.forEach(h -> this.handlers.put(h.getMedium(), h));
+        handlers.forEach(h -> {
+            log.info("Found handler {} for medium {}", h.getId(), h.getMedium());
+            String handlerId = cfg.getHandler().get(h.getMedium());
+            if (StringUtils.isBlank(handlerId) || StringUtils.equals(handlerId, h.getId())) {
+                this.handlers.put(h.getMedium(), h);
+            }
+        });
+
+        log.info("--- Notification handler ---");
+        this.handlers.forEach((k, v) -> {
+            log.info("\tHandler for {}: {}", k, v.getId());
+        });
     }
 
     private INotificationHandler ensureMedium(String medium) {
@@ -46,7 +63,6 @@ public class NotificationManager {
         if (handler == null) {
             throw new NotImplementedException(medium + " is not a supported 3PID medium type");
         }
-
         return handler;
     }
 

src/main/java/io/kamax/mxisd/session/SessionMananger.java

@@ -30,8 +30,8 @@ import io.kamax.matrix._MatrixID;
 import io.kamax.mxisd.ThreePid;
 import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.config.SessionConfig;
-import io.kamax.mxisd.controller.v1.io.RequestTokenResponse;
-import io.kamax.mxisd.controller.v1.remote.RemoteIdentityAPIv1;
+import io.kamax.mxisd.controller.identity.v1.io.RequestTokenResponse;
+import io.kamax.mxisd.controller.identity.v1.remote.RemoteIdentityAPIv1;
 import io.kamax.mxisd.exception.*;
 import io.kamax.mxisd.lookup.ThreePidValidation;
 import io.kamax.mxisd.matrix.IdentityServerUtils;
@@ -272,9 +272,13 @@ public class SessionMananger {
 
         List<String> servers = mxCfg.getIdentity().getServers(policy.getToRemote().getServer());
         if (servers.isEmpty()) {
-            throw new InternalServerError();
+            throw new FeatureNotAvailable("Remote 3PID sessions are enabled but server list is " +
+                    "misconstrued (invalid ID or empty list");
         }
-        String url = IdentityServerUtils.findIsUrlForDomain(servers.get(0)).orElseThrow(InternalServerError::new);
+
+        String is = servers.get(0);
+        String url = IdentityServerUtils.findIsUrlForDomain(is)
+                .orElseThrow(() -> new InternalServerError(is + " could not be resolved to an Identity server"));
         log.info("Will use IS endpoint {}", url);
 
         String remoteSecret = session.isRemote() ? session.getRemoteSecret() : RandomStringUtils.randomAlphanumeric(16);
@@ -283,13 +287,17 @@ public class SessionMananger {
         body.addProperty("client_secret", remoteSecret);
         body.addProperty(session.getThreePid().getMedium(), session.getThreePid().getAddress());
         body.addProperty("send_attempt", session.increaseAndGetRemoteAttempt());
-        try {
-            Phonenumber.PhoneNumber msisdn = phoneUtil.parse("+" + session.getThreePid().getAddress(), null);
-            String country = phoneUtil.getRegionCodeForNumber(msisdn).toUpperCase();
-            body.addProperty("phone_number", phoneUtil.format(msisdn, PhoneNumberUtil.PhoneNumberFormat.NATIONAL));
-            body.addProperty("country", country);
-        } catch (NumberParseException e) {
-            throw new InternalServerError(e);
+        if (ThreePidMedium.PhoneNumber.is(session.getThreePid().getMedium())) {
+            try {
+                Phonenumber.PhoneNumber msisdn = phoneUtil.parse("+" + session.getThreePid().getAddress(), null);
+                String country = phoneUtil.getRegionCodeForNumber(msisdn).toUpperCase();
+                body.addProperty("phone_number", phoneUtil.format(msisdn, PhoneNumberUtil.PhoneNumberFormat.NATIONAL));
+                body.addProperty("country", country);
+            } catch (NumberParseException e) {
+                throw new InternalServerError(e);
+            }
+        } else {
+            body.addProperty(session.getThreePid().getMedium(), session.getThreePid().getAddress());
         }
 
         log.info("Requesting remote session with attempt {}", session.getRemoteAttempt());

src/main/java/io/kamax/mxisd/storage/ormlite/OrmLiteSqliteStorage.java

@@ -146,8 +146,8 @@ public class OrmLiteSqliteStorage implements IStorage {
         return withCatcher(() -> {
             List<ThreePidSessionDao> daoList = sessionDao.queryForMatchingArgs(new ThreePidSessionDao(tpid, secret));
             if (daoList.size() > 1) {
-                log.error("Lookup for 3PID Session {}:{} returned more than one result");
-                throw new InternalServerError();
+                throw new InternalServerError("Lookup for 3PID Session " +
+                        tpid + " returned more than one result");
             }
 
             if (daoList.isEmpty()) {

src/main/java/io/kamax/mxisd/threepid/connector/email/EmailSendGridNotificationHandler.java

@@ -0,0 +1,143 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.threepid.connector.email;
+
+import com.sendgrid.SendGrid;
+import com.sendgrid.SendGridException;
+import io.kamax.matrix.ThreePidMedium;
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.ServerConfig;
+import io.kamax.mxisd.config.threepid.connector.EmailSendGridConfig;
+import io.kamax.mxisd.exception.FeatureNotAvailable;
+import io.kamax.mxisd.invitation.IThreePidInviteReply;
+import io.kamax.mxisd.notification.INotificationHandler;
+import io.kamax.mxisd.threepid.notification.PlaceholderNotificationGenerator;
+import io.kamax.mxisd.threepid.session.IThreePidSession;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+
+import static com.sendgrid.SendGrid.Email;
+import static com.sendgrid.SendGrid.Response;
+import static io.kamax.mxisd.config.threepid.connector.EmailSendGridConfig.EmailTemplate;
+
+@Component
+public class EmailSendGridNotificationHandler extends PlaceholderNotificationGenerator implements INotificationHandler {
+
+    private Logger log = LoggerFactory.getLogger(EmailSendGridNotificationHandler.class);
+
+    private EmailSendGridConfig cfg;
+    private SendGrid sendgrid;
+
+    @Autowired
+    public EmailSendGridNotificationHandler(MatrixConfig mxCfg, ServerConfig srvCfg, EmailSendGridConfig cfg) {
+        super(mxCfg, srvCfg);
+        this.cfg = cfg;
+        this.sendgrid = new SendGrid(cfg.getApi().getKey());
+    }
+
+    @Override
+    public String getId() {
+        return "sendgrid";
+    }
+
+    @Override
+    public String getMedium() {
+        return ThreePidMedium.Email.getId();
+    }
+
+    protected Email getEmail() {
+        Email email = new Email();
+        email.setFrom(cfg.getIdentity().getFrom());
+        email.setFromName(cfg.getIdentity().getName());
+        return email;
+    }
+
+    private String getFromFile(String path) {
+        try {
+            return IOUtils.toString(new FileInputStream(path), StandardCharsets.UTF_8);
+        } catch (IOException e) {
+            throw new RuntimeException("Couldn't create notification content using file " + path, e);
+        }
+    }
+
+    @Override
+    public void sendForInvite(IThreePidInviteReply invite) {
+        EmailTemplate template = cfg.getTemplates().getInvite();
+        Email email = getEmail();
+        email.setSubject(populateForInvite(invite, template.getSubject()));
+        email.setText(populateForInvite(invite, getFromFile(template.getBody().getText())));
+        email.setHtml(populateForInvite(invite, getFromFile(template.getBody().getHtml())));
+
+        send(invite.getInvite().getAddress(), email);
+    }
+
+    @Override
+    public void sendForValidation(IThreePidSession session) {
+        EmailTemplate template = cfg.getTemplates().getSession().getLocal();
+        Email email = getEmail();
+        email.setSubject(populateForValidation(session, template.getSubject()));
+        email.setText(populateForValidation(session, getFromFile(template.getBody().getText())));
+        email.setHtml(populateForValidation(session, getFromFile(template.getBody().getHtml())));
+
+        send(session.getThreePid().getAddress(), email);
+    }
+
+    @Override
+    public void sendForRemoteValidation(IThreePidSession session) {
+        EmailTemplate template = cfg.getTemplates().getSession().getLocal();
+        Email email = getEmail();
+        email.setSubject(populateForRemoteValidation(session, template.getSubject()));
+        email.setText(populateForRemoteValidation(session, getFromFile(template.getBody().getText())));
+        email.setHtml(populateForRemoteValidation(session, getFromFile(template.getBody().getHtml())));
+
+        send(session.getThreePid().getAddress(), email);
+    }
+
+    private void send(String recipient, Email email) {
+        if (StringUtils.isBlank(cfg.getIdentity().getFrom())) {
+            throw new FeatureNotAvailable("3PID Email identity: sender address is empty - " +
+                    "You must set a value for notifications to work");
+        }
+
+        try {
+            email.addTo(recipient);
+            email.setFrom(cfg.getIdentity().getFrom());
+            email.setFromName(cfg.getIdentity().getName());
+            Response response = sendgrid.send(email);
+            if (response.getStatus()) {
+                log.info("Successfully sent email to {} using SendGrid", recipient);
+            } else {
+                throw new RuntimeException("Error sending via SendGrid to " + recipient + ": " + response.getMessage());
+            }
+        } catch (SendGridException e) {
+            throw new RuntimeException("Unable to send e-mail invite via SendGrid to " + recipient, e);
+        }
+    }
+
+}

src/main/java/io/kamax/mxisd/threepid/connector/email/EmailSmtpConnector.java

@@ -23,6 +23,7 @@ package io.kamax.mxisd.threepid.connector.email;
 import com.sun.mail.smtp.SMTPTransport;
 import io.kamax.matrix.ThreePidMedium;
 import io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig;
+import io.kamax.mxisd.exception.FeatureNotAvailable;
 import io.kamax.mxisd.exception.InternalServerError;
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang.StringUtils;
@@ -66,6 +67,11 @@ public class EmailSmtpConnector implements IEmailConnector {
 
     @Override
     public void send(String senderAddress, String senderName, String recipient, String content) {
+        if (StringUtils.isBlank(senderAddress)) {
+            throw new FeatureNotAvailable("3PID Email identity: sender address is empty - " +
+                    "You must set a value for notifications to work");
+        }
+
         if (StringUtils.isBlank(content)) {
             throw new InternalServerError("Notification content is empty");
         }

src/main/java/io/kamax/mxisd/threepid/notification/GenericTemplateNotificationGenerator.java

@@ -20,17 +20,14 @@
 
 package io.kamax.mxisd.threepid.notification;
 
-import io.kamax.mxisd.ThreePid;
 import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.config.ServerConfig;
 import io.kamax.mxisd.config.threepid.medium.GenericTemplateConfig;
-import io.kamax.mxisd.controller.v1.IdentityAPIv1;
 import io.kamax.mxisd.exception.InternalServerError;
 import io.kamax.mxisd.invitation.IThreePidInviteReply;
 import io.kamax.mxisd.threepid.session.IThreePidSession;
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang.StringUtils;
-import org.apache.commons.lang.WordUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.springframework.beans.factory.annotation.Autowired;
@@ -43,39 +40,20 @@ import java.io.InputStream;
 import java.nio.charset.StandardCharsets;
 
 @Component
-public abstract class GenericTemplateNotificationGenerator implements INotificationGenerator {
+public abstract class GenericTemplateNotificationGenerator extends PlaceholderNotificationGenerator implements INotificationGenerator {
 
     private Logger log = LoggerFactory.getLogger(GenericTemplateNotificationGenerator.class);
 
-    private MatrixConfig mxCfg;
-    private ServerConfig srvCfg;
     private GenericTemplateConfig cfg;
 
     @Autowired
     private ApplicationContext app;
 
     public GenericTemplateNotificationGenerator(MatrixConfig mxCfg, ServerConfig srvCfg, GenericTemplateConfig cfg) {
-        this.mxCfg = mxCfg;
-        this.srvCfg = srvCfg;
+        super(mxCfg, srvCfg);
         this.cfg = cfg;
     }
 
-    protected String populateForCommon(String body, ThreePid recipient) {
-        return body;
-    }
-
-    private String populateCommon(String body, ThreePid recipient) {
-        body = populateForCommon(body, recipient);
-
-        String domainPretty = WordUtils.capitalizeFully(mxCfg.getDomain());
-        body = body.replace("%DOMAIN%", mxCfg.getDomain());
-        body = body.replace("%DOMAIN_PRETTY%", domainPretty);
-        body = body.replace("%RECIPIENT_MEDIUM%", recipient.getMedium());
-        body = body.replace("%RECIPIENT_ADDRESS%", recipient.getAddress());
-
-        return body;
-    }
-
     private String getTemplateContent(String location) {
         try {
             InputStream is = StringUtils.startsWith(location, "classpath:") ?
@@ -86,65 +64,22 @@ public abstract class GenericTemplateNotificationGenerator implements INotificat
         }
     }
 
-    private String getTemplateAndPopulate(String location, ThreePid recipient) {
-        return populateCommon(getTemplateContent(location), recipient);
-    }
-
     @Override
     public String getForInvite(IThreePidInviteReply invite) {
-        ThreePid tpid = new ThreePid(invite.getInvite().getMedium(), invite.getInvite().getAddress());
-        String templateBody = getTemplateAndPopulate(cfg.getInvite(), tpid);
-
-        String senderName = invite.getInvite().getProperties().getOrDefault("sender_display_name", "");
-        String senderNameOrId = StringUtils.defaultIfBlank(senderName, invite.getInvite().getSender().getId());
-        String roomName = invite.getInvite().getProperties().getOrDefault("room_name", "");
-        String roomNameOrId = StringUtils.defaultIfBlank(roomName, invite.getInvite().getRoomId());
-
-        templateBody = templateBody.replace("%SENDER_ID%", invite.getInvite().getSender().getId());
-        templateBody = templateBody.replace("%SENDER_NAME%", senderName);
-        templateBody = templateBody.replace("%SENDER_NAME_OR_ID%", senderNameOrId);
-        templateBody = templateBody.replace("%INVITE_MEDIUM%", tpid.getMedium());
-        templateBody = templateBody.replace("%INVITE_ADDRESS%", tpid.getAddress());
-        templateBody = templateBody.replace("%ROOM_ID%", invite.getInvite().getRoomId());
-        templateBody = templateBody.replace("%ROOM_NAME%", roomName);
-        templateBody = templateBody.replace("%ROOM_NAME_OR_ID%", roomNameOrId);
-
-        return templateBody;
+        log.info("Generating notification content for 3PID invite");
+        return populateForInvite(invite, getTemplateContent(cfg.getInvite()));
     }
 
     @Override
     public String getForValidation(IThreePidSession session) {
         log.info("Generating notification content for 3PID Session validation");
-        String templateBody = getTemplateAndPopulate(cfg.getSession().getValidation().getLocal(), session.getThreePid());
-
-        String validationLink = srvCfg.getPublicUrl() + IdentityAPIv1.getValidate(
-                session.getThreePid().getMedium(),
-                session.getId(),
-                session.getSecret(),
-                session.getToken());
-
-        templateBody = templateBody.replace("%VALIDATION_LINK%", validationLink);
-        templateBody = templateBody.replace("%VALIDATION_TOKEN%", session.getToken());
-
-        return templateBody;
+        return populateForValidation(session, getTemplateContent(cfg.getSession().getValidation().getLocal()));
     }
 
     @Override
     public String getForRemoteValidation(IThreePidSession session) {
         log.info("Generating notification content for remote-only 3PID session");
-        String templateBody = getTemplateAndPopulate(cfg.getSession().getValidation().getRemote(), session.getThreePid());
-
-        String validationLink = srvCfg.getPublicUrl() + IdentityAPIv1.getValidate(
-                session.getThreePid().getMedium(),
-                session.getId(),
-                session.getSecret(),
-                session.getToken());
-
-        templateBody = templateBody.replace("%VALIDATION_LINK%", validationLink);
-        templateBody = templateBody.replace("%VALIDATION_TOKEN%", session.getToken());
-        templateBody = templateBody.replace("%NEXT_URL%", validationLink);
-
-        return templateBody;
+        return populateForRemoteValidation(session, getTemplateContent(cfg.getSession().getValidation().getRemote()));
     }
 
 }

src/main/java/io/kamax/mxisd/threepid/notification/PlaceholderNotificationGenerator.java

@@ -0,0 +1,89 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2017 Maxime Dor
+ *
+ * https://max.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.threepid.notification;
+
+import io.kamax.mxisd.ThreePid;
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.ServerConfig;
+import io.kamax.mxisd.controller.identity.v1.IdentityAPIv1;
+import io.kamax.mxisd.invitation.IThreePidInviteReply;
+import io.kamax.mxisd.threepid.session.IThreePidSession;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang.WordUtils;
+
+public abstract class PlaceholderNotificationGenerator {
+
+    private MatrixConfig mxCfg;
+    private ServerConfig srvCfg;
+
+    public PlaceholderNotificationGenerator(MatrixConfig mxCfg, ServerConfig srvCfg) {
+        this.mxCfg = mxCfg;
+        this.srvCfg = srvCfg;
+    }
+
+    protected String populateForCommon(String input, ThreePid recipient) {
+        String domainPretty = WordUtils.capitalizeFully(mxCfg.getDomain());
+
+        return input
+                .replace("%DOMAIN%", mxCfg.getDomain())
+                .replace("%DOMAIN_PRETTY%", domainPretty)
+                .replace("%RECIPIENT_MEDIUM%", recipient.getMedium())
+                .replace("%RECIPIENT_ADDRESS%", recipient.getAddress());
+    }
+
+    protected String populateForInvite(IThreePidInviteReply invite, String input) {
+        ThreePid tpid = new ThreePid(invite.getInvite().getMedium(), invite.getInvite().getAddress());
+
+        String senderName = invite.getInvite().getProperties().getOrDefault("sender_display_name", "");
+        String senderNameOrId = StringUtils.defaultIfBlank(senderName, invite.getInvite().getSender().getId());
+        String roomName = invite.getInvite().getProperties().getOrDefault("room_name", "");
+        String roomNameOrId = StringUtils.defaultIfBlank(roomName, invite.getInvite().getRoomId());
+
+        return populateForCommon(input, tpid)
+                .replace("%SENDER_ID%", invite.getInvite().getSender().getId())
+                .replace("%SENDER_NAME%", senderName)
+                .replace("%SENDER_NAME_OR_ID%", senderNameOrId)
+                .replace("%INVITE_MEDIUM%", tpid.getMedium())
+                .replace("%INVITE_ADDRESS%", tpid.getAddress())
+                .replace("%ROOM_ID%", invite.getInvite().getRoomId())
+                .replace("%ROOM_NAME%", roomName)
+                .replace("%ROOM_NAME_OR_ID%", roomNameOrId);
+    }
+
+    protected String populateForValidation(IThreePidSession session, String input) {
+        String validationLink = srvCfg.getPublicUrl() + IdentityAPIv1.getValidate(
+                session.getThreePid().getMedium(),
+                session.getId(),
+                session.getSecret(),
+                session.getToken()
+        );
+
+        return populateForCommon(input, session.getThreePid())
+                .replace("%VALIDATION_LINK%", validationLink)
+                .replace("%VALIDATION_TOKEN%", session.getToken())
+                .replace("%NEXT_URL%", validationLink);
+    }
+
+    protected String populateForRemoteValidation(IThreePidSession session, String input) {
+        return populateForValidation(session, input);
+    }
+
+}

src/main/java/io/kamax/mxisd/threepid/notification/email/EmailNotificationGenerator.java

@@ -47,6 +47,7 @@ public class EmailNotificationGenerator extends GenericTemplateNotificationGener
 
     @Override
     protected String populateForCommon(String body, ThreePid recipient) {
+        body = super.populateForCommon(body, recipient);
         body = body.replace("%FROM_EMAIL%", cfg.getIdentity().getFrom());
         body = body.replace("%FROM_NAME%", cfg.getIdentity().getName());
         return body;

src/main/java/io/kamax/mxisd/threepid/notification/email/EmailRawNotificationHandler.java

@@ -30,16 +30,21 @@ import org.springframework.stereotype.Component;
 import java.util.List;
 
 @Component
-public class EmailNotificationHandler extends GenericNotificationHandler<IEmailConnector, IEmailNotificationGenerator> {
+public class EmailRawNotificationHandler extends GenericNotificationHandler<IEmailConnector, IEmailNotificationGenerator> {
 
     private EmailConfig cfg;
 
     @Autowired
-    public EmailNotificationHandler(EmailConfig cfg, List<IEmailNotificationGenerator> generators, List<IEmailConnector> connectors) {
+    public EmailRawNotificationHandler(EmailConfig cfg, List<IEmailNotificationGenerator> generators, List<IEmailConnector> connectors) {
         this.cfg = cfg;
         process(connectors, generators);
     }
 
+    @Override
+    public String getId() {
+        return "raw";
+    }
+
     @Override
     public String getMedium() {
         return ThreePidMedium.Email.getId();