Restart mxisd service on Debian package upgrade/install if possible

- Support 3PID unbind via 3PID sessions

.gitignore

@@ -7,6 +7,7 @@ out/
 .idea/
 
 # Local dev config
+/mxisd.yaml
 /application.yaml
 
 # Local dev storage

Dockerfile

@@ -14,4 +14,5 @@ ENV SQLITE_DATABASE_PATH="/var/mxisd/mxisd.db"
 CMD [ "/start.sh" ]
 
 ADD src/docker/start.sh /start.sh
-ADD build/libs/mxisd.jar /mxisd.jar
+ADD src/script/mxisd /app/mxisd
+ADD build/libs/mxisd.jar /app/mxisd.jar

README.md

@@ -1,92 +1,95 @@
-mxisd - Federated Matrix Identity Server Daemon
------
-![Travis-CI build status](https://travis-ci.org/kamax-io/mxisd.svg?branch=master)  
+mxisd - Federated Matrix Identity Server
+----------------------------------------
+![Travis-CI build status](https://travis-ci.org/kamax-matrix/mxisd.svg?branch=master)  
 
 - [Overview](#overview)
 - [Features](#features)
-- [Why use mxisd](#why-use-mxisd)
+- [Use cases](#use-cases)
 - [Getting Started](#getting-started)
 - [Support](#support)
 - [Contribute](#contribute)
+- [Powered by mxisd](#powered-by-mxisd)
 - [FAQ](#faq)
 - [Contact](#contact)
 
 # Overview
 mxisd is a Federated Matrix Identity server for self-hosted Matrix infrastructures with [enhanced features](#features).
+As an enhanced Identity service, it implements the [Identity service API](https://matrix.org/docs/spec/identity_service/r0.2.0.html)
+and several [extra features](#features) that greatly enhance user experience within Matrix.
+It is the one stop shop for anything regarding Authentication, Directory and Identity management in Matrix built in a
+single coherent product.
   
-It is specifically designed to connect to an Identity store (AD/Samba/LDAP, SQL Database, Web services/application, ...)
-and ease the integration of the Matrix ecosystem with an existing infrastructure, or to build a new one using lasting
-tools.
+mxisd is specifically designed to connect to an existing on-premise Identity store (AD/Samba/LDAP, SQL Database,
+Web services/app, etc.) and ease the integration of a Matrix infrastructure within an existing one.  
+Check [our FAQ entry](docs/faq.md#what-kind-of-setup-is-mxisd-really-designed-for) to know if mxisd is a good fit for you.
 
-The core principle of mxisd is to map between Matrix IDs and 3PIDs (Third-party Identifiers) for the Homeserver and its
-users. 3PIDs can be anything that identify a user, like:
-- Full name
+The core principle of mxisd is to map between Matrix IDs and 3PIDs (Third-Party IDentifiers) for the Homeserver and its
+users. 3PIDs can be anything that uniquely and globally identify a user, like:
 - Email address
 - Phone number
-- Employee number
 - Skype/Live ID
 - Twitter handle
 - Facebook ID
-- ...
 
-mxisd is an enhanced Identity service, which implements the
-[Matrix Identity service API](https://matrix.org/docs/spec/identity_service/unstable.html) but also several
-[other features](#features) that greatly enhance user experience within Matrix.
-
-mxisd is the one stop shop for anything regarding Authentication, Directory and Identity management in Matrix built as a
-single coherent product.
+If you are unfamiliar with the Identity vocabulary and concepts in Matrix, **please read this [introduction](docs/concepts.md)**.  
 
 # Features
-As a [regular Matrix Identity service](docs/features/identity.md):
-- Search for people by 3PID using its own Identity stores (LDAP, SQL, etc.)
-- Invite people to rooms by 3PID using its own Identity stores, with [notifications](docs/README.md)
-to the invitee (Email, SMS, etc.)
-- Allow users to add 3PIDs to their settings/profile
+[Identity](docs/features/identity.md): As a [regular Matrix Identity service](https://matrix.org/docs/spec/identity_service/r0.2.0.html#general-principles):
+- Search for people by 3PID using its own Identity stores
+  ([Spec](https://matrix.org/docs/spec/identity_service/r0.2.0.html#association-lookup))
+- Invite people to rooms by 3PID using its own Identity stores, with notifications to the invitee (Email, SMS, etc.)
+  ([Spec](https://matrix.org/docs/spec/identity_service/r0.2.0.html#invitation-storage))
+- Allow users to add/remove 3PIDs to their settings/profile via 3PID sessions
+  ([Spec](https://matrix.org/docs/spec/identity_service/r0.2.0.html#establishing-associations))
+- Register accounts on your Homeserver with 3PIDs
+  ([Spec](https://matrix.org/docs/spec/identity_service/r0.2.0.html#establishing-associations))
 
 As an enhanced Identity service:
-- Use a recursive lookup mechanism when searching and inviting people by 3PID, allowing to fetch data from:
-  - Own Identity store
+- [Federation](docs/features/federation.md): Use a recursive lookup mechanism when searching and inviting people by 3PID,
+  allowing to fetch data from:
+  - Own Identity store(s)
   - Federated Identity servers, if applicable to the 3PID
   - Arbitrary Identity servers
   - Central Matrix Identity servers
-- [Extensive control of where 3PIDs are transmited](docs/sessions/3pid.md), so they are not leaked publicly by users
-- [Authentication support](docs/features/authentication.md) for [synapse](https://github.com/matrix-org/synapse) via the
-[REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth)
-- [Directory integration](docs/features/directory-users.md) which allows you to search for users within your
-organisation, even without prior Matrix contact
-- [Auto-fill of user profile](docs/features/authentication.md) (Display name, 3PIDs) via the
-[REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth)
+- [Session Control](docs/threepids/session/session.md): Extensive control of where 3PIDs are transmitted so they are not
+  leaked publicly by users
+- [Registration control](docs/features/registration.md): Control and restrict user registration based on 3PID patterns or criterias, like a pending invite
+- [Authentication](docs/features/authentication.md): Use your Identity stores to perform authentication in [synapse](https://github.com/matrix-org/synapse)
+  via the [REST password provider](https://github.com/kamax-io/matrix-synapse-rest-auth)
+- [Directory search](docs/features/directory.md) which allows you to search for users within your organisation,
+  even without prior contact within Matrix using arbitrary search terms
+- [Auto-fill of user profile](docs/features/authentication.md#profile-auto-fill) (Display name, 3PIDs)
+- [Bridge Integration](docs/features/bridge-integration.md): Automatically bridge users without a published Matrix ID
 
-# Why use mxisd
-- Use your existing Identity store, do not duplicate information
+# Use cases
+- Use your existing Identity stores, do not duplicate your users information
 - Auto-fill user profiles with relevant information
-- As an organisation, stay in control of 3PIDs so they are not published to the central Matrix.org servers where they
+- As an organisation, stay in control of your data so it is not published to other servers by default where they
   currently **cannot be removed**
 - Users can directly find each other using whatever attribute is relevant within your Identity store
-- Federate your Identity lookups so you can discover others and/or others can discover you, all with extensive ACLs
+- Federate your Identity server so you can discover others and/or others can discover you
+
+Also, check [our FAQ entry](docs/faq.md#what-kind-of-setup-is-mxisd-really-designed-for) to know if mxisd is a good fit for you.
 
 # Getting started
 See the [dedicated document](docs/getting-started.md)
 
 # Support
+## Troubleshooting
+A basic troubleshooting guide is available [here](docs/troubleshooting.md).
+
 ## Community
-If you need help, want to report a bug or just say hi, you can reach us on Matrix at 
-[#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io) or
-[directly peek anonymously](https://view.matrix.org/room/!NPRUEisLjcaMtHIzDr:kamax.io/).
-For more high-level discussion about the Identity Server architecture/API, go to 
-[#matrix-identity:matrix.org](https://matrix.to/#/#matrix-identity:matrix.org)
+Over Matrix: [#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io) ([Preview](https://view.matrix.org/room/!NPRUEisLjcaMtHIzDr:kamax.io/))
 
-## Professional
+## Commercial
 If you would prefer professional support/custom development for mxisd and/or for Matrix in general, including other open
-source technologies/products, please visit [our website](https://www.kamax.io/) to get in touch with us and get a quote.
-
-We offer affordable monthly/yearly support plans for mxisd, synapse or your full Matrix infrastructure.
+source technologies/products:
+- Visit our [website](https://www.kamax.io/) to get in touch with us and get a quote.
+- Come in our general Matrix room: [#kamax-matrix:kamax.io](https://matrix.to/#/#kamax-matrix:kamax.io)
 
 # Contribute 
-First and foremost, the best way to contribute is to use mxisd and tell us about it!  
-We would love to hear about your experience and get your feedback on how to make it an awesome product. 
-
 You can contribute as a community member by:
+- Giving us feedback about your usage of mxisd, even if it seems unimportant or if all is working well!
 - Opening issues for any weird behaviour or bug. mxisd should feel natural, let us know if it does not!
 - Helping us improve the documentation: tell us what is good or not good (in an issue or in Matrix), or make a PR with
 changes you feel improve the doc.
@@ -95,14 +98,19 @@ changes you feel improve the doc.
 be used for the fixed costs and developer time of mxisd.
 
 You can contribute as an organisation/corporation by:
-- Get a [support contract](#support-professional). This is the best way you can help us as it ensures mxisd is
+- Get a [support contract](#commercial). This is the best way you can help us as it ensures mxisd is
 maintained regularly and you get direct access to the support team.
 - Sponsoring new features or bug fixes. [Get in touch](#contact) so we can discuss it further.
 
+# Powered by mxisd
+The following projects use mxisd under the hood for some or all their features. Check them out!
+- [matrix-docker-ansible-deploy](https://github.com/spantaleev/matrix-docker-ansible-deploy)
+- [matrix-register-bot](https://github.com/krombel/matrix-register-bot)
+
 # FAQ
 See the [dedicated document](docs/faq.md)
 
 # Contact
 Get in touch via:
-- Matrix at [#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io)
-- Email, see our website: [Kamax.io](https://www.kamax.io)
+- Matrix: [#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io)
+- Email: see our website: [Kamax.io](https://www.kamax.io)

application.example.yaml

@@ -1,99 +0,0 @@
-# Sample configuration file explaining the minimum required keys to be set to run mxisd
-#
-# For a complete list of options, see https://github.com/kamax-io/mxisd
-
-#######################
-# Matrix config items #
-#######################
-# Matrix domain, same as the domain configure in your Homeserver configuration.
-# (note: in Synapse Homeserver, the Matrix domain may be defined as 'server_name' in configuration file).
-#
-# This is used to build the various identifiers for identity, auth and directory.
-matrix.domain: ''
-
-
-################
-# Signing keys #
-################
-# Absolute path for the Identity Server signing key.
-# During testing, /var/tmp/mxisd.key is a possible value
-#
-# For production, recommended location shall be one of the following:
-#   - /var/opt/mxisd/sign.key
-#   - /var/local/mxisd/sign.key
-#   - /var/lib/mxisd/sign.key
-#
-# The signing key is auto-generated during execution time if not present.
-key.path: ''
-
-
-############################
-# Persistence config items #
-############################
-
-# Configure the storage backend, usually a DB
-# Possible built-in values:
-#   sqlite                      SQLite backend, default
-#
-#storage.backend: 'sqlite'
-
-# Path to the SQLite DB file
-#
-# Examples:
-#  - /var/opt/mxisd/mxisd.db
-#  - /var/local/mxisd/mxisd.db
-#  - /var/lib/mxisd/mxisd.db
-#
-storage.provider.sqlite.database: '/path/to/mxisd.db'
-
-
-################
-# LDAP Backend #
-################
-# If you would like to integrate with your AD/Samba/LDAP server,
-# see https://github.com/kamax-io/mxisd/blob/master/docs/backends/ldap.md
-
-###############
-# SQL Backend #
-###############
-# If you would like to integrate with a MySQL/MariaDB/PostgreQL/SQLite DB,
-# see https://github.com/kamax-io/mxisd/blob/master/docs/backends/sql.md
-
-################
-# REST Backend #
-################
-# If you would like to integrate with an existing web service/webapp,
-# see https://github.com/kamax-io/mxisd/blob/master/docs/backends/rest.md
-
-
-#################################################
-# Notifications for invites/addition to profile #
-#################################################
-# If you would like to change the content,
-# see https://github.com/kamax-io/mxisd/blob/master/docs/threepids/notifications/template-generator.md
-#
-#### E-mail invite sender
-#
-# SMTP host
-threepid.medium.email.connectors.smtp.host: "smtp.example.org"
-
-# SMTP port
-threepid.medium.email.connectors.smtp.port: 587
-
-# TLS mode for the connection.
-#
-# Possible values:
-#  0    Disable TLS entirely
-#  1    Enable TLS if supported by server (default)
-#  2    Force TLS and fail if not available
-#
-#threepid.medium.email.connectors.smtp.tls: 1
-
-# Login for SMTP
-threepid.medium.email.connectors.smtp.login: "matrix-identity@example.org"
-
-# Password for the account
-threepid.medium.email.connectors.smtp.password: "ThePassword"
-
-# The e-mail to send as.
-threepid.medium.email.identity.from: "matrix-identity@example.org"

build.gradle

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -21,9 +21,11 @@
 import java.util.regex.Pattern
 
 apply plugin: 'java'
-apply plugin: 'org.springframework.boot'
+apply plugin: 'application'
+apply plugin: 'com.github.johnrengelman.shadow'
+apply plugin: 'idea'
 
-def confFileName = "application.example.yaml"
+def confFileName = "mxisd.example.yaml"
 def distDir = "${project.buildDir}/dist"
 
 def debBinPath = "/usr/lib/mxisd"
@@ -31,7 +33,8 @@ def debConfPath = "/etc/mxisd"
 def debDataPath = "/var/lib/mxisd"
 def debSystemdPath = "/etc/systemd/system"
 
-def debConfFileName = "mxisd-sample.yaml"
+def debConfFileName = confFileName
+def debStartScriptFilename = "mxisd"
 
 def debBuildBasePath = "${project.buildDir}/tmp/debian"
 def debBuildDebianPath = "${debBuildBasePath}/DEBIAN"
@@ -41,46 +44,63 @@ def debBuildDataPath = "${debBuildBasePath}${debDataPath}"
 def debBuildSystemdPath = "${debBuildBasePath}${debSystemdPath}"
 
 def dockerImageName = "kamax/mxisd"
-def dockerImageTag = "${dockerImageName}:${gitVersion()}"
+def dockerImageTag = "${dockerImageName}:${mxisdVersion()}"
+
+group = 'io.kamax'
+mainClassName = 'io.kamax.mxisd.MxisdStandaloneExec'
+sourceCompatibility = '1.8'
+targetCompatibility = '1.8'
+
+String mxisdVersion() {
+    def versionPattern = Pattern.compile("v(\\d+\\.)?(\\d+\\.)?(\\d+)(-.*)?")
+
+    String version = System.getenv('MXISD_BUILD_VERSION')
+    if (version == null || version.size() == 0) {
+        version = gitVersion()
+    }
+    return versionPattern.matcher(version).matches() ? version.substring(1) : version
+}
 
 String gitVersion() {
-    def versionPattern = Pattern.compile("v(\\d+\\.)?(\\d+\\.)?(\\d+)(-.*)?")
     ByteArrayOutputStream out = new ByteArrayOutputStream()
     exec {
         commandLine = ['git', 'describe', '--tags', '--always', '--dirty']
         standardOutput = out
     }
-    def v = out.toString().replace(System.lineSeparator(), '')
-    return versionPattern.matcher(v).matches() ? v.substring(1) : v
+    return out.toString().replace(System.lineSeparator(), '')
 }
 
 buildscript {
     repositories {
-        mavenCentral()
+        jcenter()
     }
 
     dependencies {
-        classpath 'org.springframework.boot:spring-boot-gradle-plugin:1.5.3.RELEASE'
+        classpath 'com.github.jengelman.gradle.plugins:shadow:4.0.3'
     }
 }
 
 repositories {
+    jcenter()
     maven { url "https://kamax.io/maven/releases/" }
-    mavenCentral()
+    maven { url "https://kamax.io/maven/snapshots/" }
 }
 
 dependencies {
+    // Logging
+    compile 'org.slf4j:slf4j-simple:1.7.25'
+
     // Easy file management
     compile 'commons-io:commons-io:2.5'
 
-    // Spring Boot - standalone app
-    compile 'org.springframework.boot:spring-boot-starter-web:1.5.10.RELEASE'
-
-    // Thymeleaf for HTML templates
-    compile "org.springframework.boot:spring-boot-starter-thymeleaf:1.5.10.RELEASE"
+    // Config management
+    compile 'org.yaml:snakeyaml:1.23'
 
     // Matrix Java SDK
-    compile 'io.kamax:matrix-java-sdk:0.0.8'
+    compile 'io.kamax:matrix-java-sdk:0.0.14-8-g0e57ec6'
+
+    // ORMLite
+    compile 'com.j256.ormlite:ormlite-jdbc:5.0'
 
     // ed25519 handling
     compile 'net.i2p.crypto:eddsa:0.1.0'
@@ -94,22 +114,16 @@ dependencies {
     // HTTP connections
     compile 'org.apache.httpcomponents:httpclient:4.5.3'
 
-    // JSON
-    compile 'com.google.code.gson:gson:2.8.1'
-
     // Phone numbers validation
     compile 'com.googlecode.libphonenumber:libphonenumber:8.7.1'
 
     // E-mail sending
-    compile 'com.sun.mail:javax.mail:1.5.6'
-    compile 'javax.mail:javax.mail-api:1.5.6'
+    compile 'javax.mail:javax.mail-api:1.6.2'
+    compile 'com.sun.mail:javax.mail:1.6.2'
 
     // Google Firebase Authentication backend
     compile 'com.google.firebase:firebase-admin:5.3.0'
 
-    // ORMLite
-    compile 'com.j256.ormlite:ormlite-jdbc:5.0'
-
     // Connection Pool
     compile 'com.mchange:c3p0:0.9.5.2'
 
@@ -117,7 +131,7 @@ dependencies {
     compile 'org.xerial:sqlite-jdbc:3.20.0'
 
     // PostgreSQL
-    compile 'org.postgresql:postgresql:42.1.4'
+    compile 'org.postgresql:postgresql:42.2.5'
 
     // MariaDB/MySQL
     compile 'org.mariadb.jdbc:mariadb-java-client:2.1.2'
@@ -128,36 +142,42 @@ dependencies {
     // SendGrid SDK to send emails from GCE
     compile 'com.sendgrid:sendgrid-java:2.2.2'
 
+    // ZT-Exec for exec identity store
+    compile 'org.zeroturnaround:zt-exec:1.10'
+
+    // HTTP server
+    compile 'io.undertow:undertow-core:2.0.16.Final'
+
+    // Command parser for AS interface
+    implementation 'commons-cli:commons-cli:1.4'
+
     testCompile 'junit:junit:4.12'
     testCompile 'com.github.tomakehurst:wiremock:2.8.0'
+    testCompile 'com.unboundid:unboundid-ldapsdk:4.0.9'
+    testCompile 'com.icegreen:greenmail:1.5.9'
 }
 
-springBoot {
-    executable = true
-
-    embeddedLaunchScriptProperties = [
-            confFolder: "/etc/default"
-    ]
+jar {
+    manifest {
+        attributes(
+                'Implementation-Version': mxisdVersion()
+        )
+    }
 }
 
-processResources {
+shadowJar {
+    baseName = project.name
+    classifier = null
+    version = null
+}
+
+task debBuild(dependsOn: shadowJar) {
     doLast {
-        copy {
-            from('build/resources/main/application.yaml') {
-                rename 'application.yaml', 'mxisd.yaml'
-            }
-            into 'build/resources/main'
-        }
-    }
-}
-
-task buildDeb(dependsOn: build) {
-    doLast {
-        def v = gitVersion()
-        println "Version for package: ${v}"
+        String debVersion = mxisdVersion()
+        println "Version for package: ${debVersion}"
         mkdir distDir
         mkdir debBuildBasePath
-        mkdir "${debBuildBasePath}/DEBIAN"
+        mkdir debBuildDebianPath
         mkdir debBuildBinPath
         mkdir debBuildConfPath
         mkdir debBuildDataPath
@@ -168,10 +188,10 @@ task buildDeb(dependsOn: build) {
             into debBuildBinPath
         }
 
-        ant.chmod(
-                file: "${debBuildBinPath}/mxisd.jar",
-                perm: 'a+x'
-        )
+        copy {
+            from "${project.file("src/script/" + debStartScriptFilename)}"
+            into debBuildBinPath
+        }
 
         copy {
             from(project.file(confFileName)) {
@@ -180,16 +200,16 @@ task buildDeb(dependsOn: build) {
             into debBuildConfPath
         }
 
-        ant.replaceregexp(
+        ant.replaceregexp( // FIXME adapt to new config format
                 file: "${debBuildConfPath}/${debConfFileName}",
-                match: "key.path:(.*)",
-                replace: "key.path: '${debDataPath}/signing.key'"
+                match: "key:\\R  path:(.*)",
+                replace: "key:\n  path: '${debDataPath}/keys'"
         )
 
-        ant.replaceregexp(
+        ant.replaceregexp( // FIXME adapt to new config format
                 file: "${debBuildConfPath}/${debConfFileName}",
-                match: "storage.provider.sqlite.database:(.*)",
-                replace: "storage.provider.sqlite.database: '${debDataPath}/mxisd.db'"
+                match: "storage:\\R  provider:\\R    sqlite:\\R      database:(.*)",
+                replace: "storage:\n  provider:\n    sqlite:\n      database: '${debDataPath}/store.db'"
         )
 
         copy {
@@ -200,7 +220,7 @@ task buildDeb(dependsOn: build) {
         ant.replace(
                 file: "${debBuildDebianPath}/control",
                 token: 'Version: 0',
-                value: "Version: ${v}"
+                value: "Version: ${debVersion}"
         )
 
         ant.replace(
@@ -209,6 +229,12 @@ task buildDeb(dependsOn: build) {
                 value: debDataPath
         )
 
+        ant.replace(
+                file: "${debBuildDebianPath}/postinst",
+                token: '%DEB_CONF_FILE%',
+                value: "${debConfPath}/mxisd.yaml"
+        )
+
         ant.chmod(
                 file: "${debBuildDebianPath}/postinst",
                 perm: 'a+x'
@@ -236,7 +262,7 @@ task buildDeb(dependsOn: build) {
     }
 }
 
-task dockerBuild(type: Exec, dependsOn: build) {
+task dockerBuild(type: Exec, dependsOn: shadowJar) {
     commandLine 'docker', 'build', '-t', dockerImageTag, project.rootDir
 
     doLast {

docs/README.md

@@ -1,26 +1,26 @@
 # Table of Contents
-
+- [Identity Concepts in Matrix](concepts.md)
+- [Getting Started](getting-started.md)
+- [Build from sources](build.md) (Optional)
 - Installation
   - [Debian package](install/debian.md)
+  - [ArchLinux](install/archlinux.md)
+  - [NixOS](install/nixos.md)
   - [Docker](install/docker.md)
-- [Build from source](build.md)
+  - [From source](install/source.md)
 - [Architecture overview](architecture.md)
 - [Configuration](configure.md)
 - Features
-  - [Matrix Identity Service](features/identity.md)
-  - [Homeserver Authentication](features/authentication.md)
-  - [Directory seach](features/directory-users.md)
-  - [Identity server Federation](features/federation.md)
+  - [Authentication](features/authentication.md)
+  - [Directory search](features/directory.md)
+  - [Identity](features/identity.md)
+  - [Federation](features/federation.md)
   - [Bridge integration](features/bridge-integration.md)
-- Backends
-  - [LDAP](backends/ldap.md)
-  - [SQL](backends/sql.md)
-  - [REST](backends/rest.md)
-  - [Google Firebase](backends/firebase.md)
-  - [Wordpress](backends/wordpress.md)
+- [Identity Stores](stores/README.md)
 - Notifications
   - Handlers
-    - [Basic](threepids/notifications/basic-handler.md)
-    - [SendGrid](threepids/notifications/sendgrid-handler.md)
-- [Sessions](sessions/3pid.md)
-  - [Views](sessions/3pid-views.md)
+    - [Basic](threepids/notification/basic-handler.md)
+    - [SendGrid](threepids/notification/sendgrid-handler.md)
+- [Sessions](threepids/session/session.md)
+  - [Views](threepids/session/session-views.md)
+- [FAQ](faq.md)

docs/_config.yml

@@ -1 +1 @@
-theme: jekyll-theme-cayman
+theme: jekyll-theme-hacker

docs/architecture.md

@@ -1,6 +1,6 @@
 # Architecture
 ## Overview
-### Basic setup without integration or incoming federation
+### Basic setup with default settings
 ```
  Client
    |
@@ -14,17 +14,14 @@ TCP 443
        +--|------------------+            +---|-----------------------+
           |                                   |
           +<---------------------------------<+
-          |                                          Backends
-          |   +-------------------+                  +------+    +--------+
- TCP 8090 +-> | mxisd             |          +-----> | LDAP | -> | SQL DB |
-              |                   |          |       +------+    +--------+ ....
-              | - Profile's 3PIDs >----+     |
-              | - 3PID Invites    |    |     |
-              +-|-----------------+    +>----+
-                |                      |     |       +--------------------------+
-                |                      |     |       | Central Identity service |
-                +>-------------------->+     +-----> | Matrix.org / Vector.im   |
-                |                            TCP 443 +--------------------------+
+          |
+          |   +-------------------+
+ TCP 8090 +-> | mxisd             |
+              |                   |
+              | - Profile's 3PIDs |
+              | - 3PID Invites    |
+              +-|-----------------+
+                |
              TCP 443
                 |  +------------------------+
                 |  | Remote Federated       |
@@ -37,7 +34,7 @@ TCP 443
 See the [dedicated document](features/authentication.md).
 
 ### With Directory
-See the [dedicated document](features/directory-users.md).
+See the [dedicated document](features/directory.md).
 
 ### With Federation
 See the [dedicated document](features/federation.md).

docs/backends/README.md

@@ -1,6 +0,0 @@
-# Identity Stores (Backends)
-- [Samba / Active Directory / LDAP](ldap.md)
-- [SQL Databases](sql.md)
-- [Website / Web service / Web app](rest.md)
-- [Google Firebase](firebase.md)
-- [Wordpress](wordpress.md)

docs/backends/firebase.md

@@ -1,19 +0,0 @@
-# Google Firebase
-https://firebase.google.com/
-
-## Requirements
-This backend requires a suitable Matrix client capable of performing Firebase authentication and passing the following
-information:
-- Firebase User ID as Matrix username
-- Firebase token as Matrix password
-
-If your client is Riot, you will need a custom version.
-
-## Configuration
-To be completed. For now, see default structure and values:
-```
-firebase:
-  enabled: false
-  credentials: '/path/to/firebase/credentials.json'
-  database: 'https://my-project.firebaseio.com/'
-```

docs/backends/ldap.md

@@ -1,140 +0,0 @@
-# LDAP (Samba / Active Directory / OpenLDAP)
-## Getting started
-To use your LDAP backend, add the bare minimum configuration in mxisd config file:
-```
-ldap.enabled: true
-ldap.connection.host: 'ldapHostnameOrIp'
-ldap.connection.bindDn: 'CN=My Mxisd User,OU=Users,DC=example,DC=org'
-ldap.connection.bindPassword: 'TheUserPassword'
-ldap.connection.baseDn: 'OU=Users,DC=example,DC=org'
-```
-These are standard LDAP connection configuration. mxisd will try to connect on port default port 389 without encryption.
-
----
-
-If you would like to use a TLS/SSL connection, use the following configuration options (STARTLS not supported):
-```
-ldap.connection.tls: true
-ldap.connection.port: 12345
-```
-
----
-
-You can also set a default global filter on any LDAP queries:
-```
-ldap.filter: '(memberOf=CN=My Matrix Users,OU=Groups,DC=example,DC=org)'
-```
-This example would only return users part of the group called `My Matrix Users`.
-This can be overwritten or append in each specific flow describe below.
-
----
-
-LDAP features are based on mapping LDAP attributes to Matrix concepts, like a Matrix ID, its localpart, the user display
-name, their email(s) and/or phone number(s).
-     
-Default attributes are well suited for Active Directory/Samba. In case you are using a native LDAP backend, you will
-most certainly configure those mappings.
-
-The following example would set the `uid` attribute as localpart and the Matrix display name to `cn`
-```
-ldap.attribute.uid.type: 'uid'
-ldap.attribute.uid.value: 'uid'
-ldap.attribute.name: 'cn'
-```
-
-You can also change the attribute lists for 3PID, like email or phone numbers.  
-The following example would overwrite the [default list of attributes](../../src/main/resources/application.yaml#L67)
-for emails and phone number:
-```
-ldap.attribute.threepid.email:
-  - 'mail'
-  - 'otherMailAttribute'
-
-ldap.attribute.threepid.msisdn:
-  - 'phone'
-  - 'otherPhoneAttribute'
-```
-
-## Identity
-Identity features (related to 3PID invites or searches) are enabled and configured using default values and no specific
-configuration item is needed to get started.
-
-If you would like to overwrite some global configuration relative to filter and/or attributes, see the Identity section
-of the Configuration below.
-
-## Authentication
-No further configuration is needed to enable authentication with LDAP once globally enabled and configured.  
-You have the possiblity to use a different query filter if you wish, see Configuration below.
-
-Profile auto-fill is enabled by default. It will use the `name` and `threepid` configuration options to get a lit of
-attributes to be used to build the user profile to pass on to synapse during authentication.
-
-## Directory
-No further configuration is needed to enable directory with LDAP once globally enabled and configured.
-
-If you would like to use extra attributes in search that are not 3PIDs, like nicknames, group names, employee number:
-```
-ldap.directory.attribute.other:
-  - 'myNicknameAttribute'
-  - 'memberOf'
-  - 'employeeNumberAttribute'
-```
-
-## Configuration
-Please read the [Configuration](../configure.md) explanatory note if you are not familiar with the terms used below.
- 
-### General
-Base path: `ldap`
-
-| Item      | Description                                                                               |
-|-----------|-------------------------------------------------------------------------------------------|
-| `enabled` | Globaly enable/disable the LDAP backend                                                   |
-| `filter`  | Global filter to apply on all LDAP queries. Can be overwritten in each applicable section |
-
-### Connection
-Base path: `ldap.connection`
-
-| Item           | Description                                          |
-|----------------|------------------------------------------------------|
-| `host`         | Host to connect to                                   |
-| `port`         | Port to use                                          |
-| `tls`          | boolean to use TLS or not (STARTLS is not supported) |
-| `bindDn`       | Bind DN for authentication                           |
-| `bindPassword` | Bind password                                        |
-| `baseDn`       | Base DN for queries                                  |
-
-### Attributes
-Base path: `ldap.attribute`
-
-| Item        | Description                                                                                                            |
-|-------------|------------------------------------------------------------------------------------------------------------------------|
-| `uid.type`  | Indicate how to process the User ID (UID) attribute:                                                                   |
-|             |   - `uid` will consider the value as the [Localpart](https://matrix.org/docs/spec/intro.html#user-identifiers)         |
-|             |   - `mxid` will consider the value as a complete [Matrix ID](https://matrix.org/docs/spec/intro.html#user-identifiers) |
-| `uid.value` | Attribute name refering to the User ID. This is typically `userPrincipalName` on AD/Samba setups and `uid` in LDAP     |
-| `name`      | Attribute name that contains the [Display Name](https://matrix.org/docs/spec/intro.html#profiles) of the user          |
-| `threepid`  | Namespace where each key is a 3PID type and contains a list of attributes                                              |
-
-### Authentication
-Base path: `ldap.auth`
-
-| Item     | Description                                                                                      |
-|----------|--------------------------------------------------------------------------------------------------|
-| `filter` | Specific user filter applied during authentication. Global filter is used if empty/blank/not set |
-
-### Directory
-Base path: `ldap.directory`
-
-| Item              | Description                                                         |
-|-------------------|---------------------------------------------------------------------|
-| `attribute.other` | Additional attributes to be used when performing directory searches |
-| `filter`          | Specific user filter applied during directory search.               |
-|                   | Global filter is used if empty/blank/not set                        |
-
-### Identity
-Base path: `ldap.identity`
-
-| Item     | Description                                                                                       |
-|----------|---------------------------------------------------------------------------------------------------|
-| `filter` | Specific user filter applied during identity search. Global filter is used if empty/blank/not set | 
-| `medium` | Namespace to overwrite generated queries from the list of attributes for each 3PID medium         |

docs/backends/rest.md

@@ -1,215 +0,0 @@
-# REST backend
-The REST backend allows you to query identity data in existing webapps, like:
-- Forums (phpBB, Discourse, etc.)
-- Custom Identity stores (Keycloak, ...)
-- CRMs (Wordpress, ...)
-- self-hosted clouds (Nextcloud, ownCloud, ...)
-
-It supports the following mxisd flows:
-- [Authentication](#authentication)
-- [Directory](#directory)
-- [Identity](#identity)
-
-To integrate this backend with your webapp, you will need to implement three specific REST endpoints detailed below.
-
-
-## Configuration
-| Key                            | Default                                      | Description                                          |
----------------------------------|----------------------------------------------|------------------------------------------------------|
-| rest.enabled                   | false                                        | Globally enable/disable the REST backend             |
-| rest.host                      | *empty*                                      | Default base URL to use for the different endpoints. |
-| rest.endpoints.auth            | /_mxisd/backend/api/v1/auth/login            | Validate credentials and get user profile            |
-| rest.endpoints.directory       | /_mxisd/backend/api/v1/directory/user/search | Search for users by arbitrary input                  |
-| rest.endpoints.identity.single | /_mxisd/backend/api/v1/identity/single       | Endpoint to query a single 3PID                      |
-| rest.endpoints.identity.bulk   | /_mxisd/backend/api/v1/identity/bulk         | Endpoint to query a list of 3PID                     |
-
-Endpoint values can handle two formats:
-- URL Path starting with `/` that gets happened to the `rest.host`
-- Full URL, if you want each endpoint to go to a specific server/protocol/port
-
-`rest.host` is mandatory if at least one endpoint is not a full URL.
-
-## Endpoints
-### Authentication
-HTTP method: `POST`  
-Content-type: JSON UTF-8
-  
-#### Request Body
-```
-{
-  "auth": {
-    "mxid": "@john.doe:example.org",
-    "localpart": "john.doe",
-    "domain": "example.org",
-    "password": "passwordOfTheUser"
-  }
-}
-```
-
-#### Response Body
-If the authentication fails:
-```
-{
-  "auth": {
-    "success": false
-  }
-}
-```
-
-If the authentication succeed:
-- `auth.id` supported values: `localpart`, `mxid`
-- `auth.profile` and any sub-member are all optional
-```
-{
-  "auth": {
-    "success": true,
-    "id": {
-      "type": "localpart",
-      "value": "john"
-    },
-    "profile": {
-      "display_name": "John Doe",
-      "three_pids": [
-        {
-          "medium": "email",
-          "address": "john.doe@example.org"
-        },
-        {
-          "medium": "msisdn",
-          "address": "123456789"
-        }
-      ]
-    }
-  }
-}
-```
-
-### Directory
-HTTP method: `POST`
-Content-type: JSON UTF-8
-
-#### Request Body
-```
-{
-  "by": "<search type>",
-  "search_term": "doe"
-}
-```
-`by` can be:
-- `name`
-- `threepid`
-
-#### Response Body:
-If users found:
-```
-{
-  "limited": false,
-  "results": [
-    {
-      "avatar_url": "http://domain.tld/path/to/avatar.png",
-      "display_name": "John Doe",
-      "user_id": "UserIdLocalpart"
-    },
-    {
-      ...
-    }
-  ]
-}
-```
-
-If no user found:
-```
-{
-  "limited": false,
-  "results": []
-}
-```
-
-### Identity
-#### Single 3PID lookup
-HTTP method: `POST`  
-Content-type: JSON UTF-8  
-  
-#### Request Body
-```
-{
-  "lookup": {
-    "medium": "email",
-    "address": "john.doe@example.org"
-  }
-}
-```
-
-#### Response Body
-If a match was found:
-- `lookup.id.type` supported values: `localpart`, `mxid`
-```
-{
-  "lookup": {
-    "medium": "email",
-    "address": "john.doe@example.org",
-    "id": {
-      "type": "mxid",
-      "value": "@john:example.org"
-    }
-  }
-}
-```
-
-If no match was found:
-```
-{}
-```
-
-#### Bulk 3PID lookup
-HTTP method: `POST`  
-Content-type: JSON UTF-8  
-  
-#### Request Body
-```
-{
-  "lookup": [
-    {
-      "medium": "email",
-      "address": "john.doe@example.org"
-    },
-    {
-      "medium": "msisdn",
-      "address": "123456789"
-    }
-  ]
-}
-```
-
-#### Response Body
-For all entries where a match was found:
-- `lookup[].id.type` supported values: `localpart`, `mxid`
-```
-{
-  "lookup": [
-    {
-      "medium": "email",
-      "address": "john.doe@example.org",
-      "id": {
-        "type": "localpart",
-        "value": "john"
-      }
-    },
-    {
-      "medium": "msisdn",
-      "address": "123456789",
-      "id": {
-        "type": "mxid",
-        "value": "@jane:example.org"
-      }
-    }
-  ]
-}
-```
-
-If no match was found:
-```
-{
-  "lookup": []
-}
-```

docs/backends/sql.md

@@ -1,23 +0,0 @@
-# SQL Backend
-## Configuration
-To be completed. For now, see default structure and values:
-```
-sql:
-  enabled: false
-  type: 'sqlite' or 'postgresql'
-  connection: ''
-  auth:
-    enabled: false
-  directory:
-    enabled: false
-    query:
-      name:
-        type: 'localpart'
-        value: 'SELECT 1'
-      threepid:
-        type: 'localpart'
-        value: 'SELECT 1'
-  identity:
-    type: 'localpart'
-    query: 'SELECT user_id AS uid FROM user_threepids WHERE medium = ? AND address = ?'
-```

docs/build.md

@@ -1,105 +1,74 @@
 # From source
 - [Binaries](#binaries)
+  - [Requirements](#requirements)
+  - [Build](#build)
 - [Debian package](#debian-package)
 - [Docker image](#docker-image)
+- [Next steps](#next-steps)
 
 ## Binaries
 ### Requirements
 - JDK 1.8
 
 ### Build
-```
-git clone https://github.com/kamax-io/mxisd.git
+```bash
+git clone https://github.com/kamax-matrix/mxisd.git
 cd mxisd
 ./gradlew build
 ```
 
-Create a new configuration file by coping `application.example.yaml` to `application.yaml` and edit to your needs.  
+Create a new configuration file by coping `mxisd.example.yaml` to `mxisd.yaml` and edit to your needs.  
 For advanced configuration, see the [Configure section](configure.md).
 
 Start the server in foreground to validate the build and configuration:
-```
+```bash
 java -jar build/libs/mxisd.jar
 ```
 
 Ensure the signing key is available:
-```
+```bash
 $ curl 'http://localhost:8090/_matrix/identity/api/v1/pubkey/ed25519:0'
+
 {"public_key":"..."}
 ```
 
 Test basic recursive lookup (requires Internet connection with access to TCP 443):
-```
-$ curl 'http://localhost:8090/_matrix/identity/api/v1/lookup?medium=email&address=mxisd-lookup-test@kamax.io'
-{"address":"mxisd-lookup-test@kamax.io","medium":"email","mxid":"@mxisd-lookup-test:kamax.io",...}
+```bash
+$ curl 'http://localhost:8090/_matrix/identity/api/v1/lookup?medium=email&address=mxisd-federation-test@kamax.io'
+
+{"address":"mxisd-federation-test@kamax.io","medium":"email","mxid":"@mxisd-lookup-test:kamax.io",...}
 ```
 
 If you enabled LDAP, you can also validate your config with a similar request after replacing the `address` value with
 something present within your LDAP
-```
+```bash
 curl 'http://localhost:8090/_matrix/identity/api/v1/lookup?medium=email&address=john.doe@example.org'
 ```
 
 If you plan on testing the integration with a homeserver, you will need to run an HTTPS reverse proxy in front of it
 as the reference Home Server implementation [synapse](https://github.com/matrix-org/synapse) requires a HTTPS connection
 to an ID server.  
-See the [Integration section](#integration) for more details.
 
-### Install
-1. Prepare files and directories:
-```
-# Create a dedicated user
-useradd -r mxisd
-
-# Create bin directory
-mkdir /opt/mxisd
-
-# Create config directory and set ownership
-mkdir /etc/opt/mxisd
-chown mxisd /etc/opt/mxisd
-
-# Create data directory and set ownership
-mkdir /var/opt/mxisd
-chown mxisd /var/opt/mxisd
-
-# Copy <repo root>/build/libs/mxisd.jar to bin directory
-cp ./build/libs/mxisd.jar /opt/mxisd/
-chown mxisd /opt/mxisd/mxisd.jar
-chmod a+x /opt/mxisd/mxisd.jar
-
-# Create symlink for easy exec
-ln -s /opt/mxisd/mxisd.jar /usr/bin/mxisd
-```
-
-2. Copy the sample config file `./application.example.yaml` to `/etc/opt/mxisd/mxisd.yaml`, edit to your needs
-4. Copy `<repo root>/src/systemd/mxisd.service` to `/etc/systemd/system/` and edit if needed
-5. Enable service for auto-startup
-```
-systemctl enable mxisd
-```
-6. Start mxisd
-```
-systemctl start mxisd
-```
+Next step: [Install your compiled binaries](install/source.md)
 
 ## Debian package
-### Requirements
+Requirements:
 - fakeroot
 - dpkg-deb
 
-### Build
 [Build mxisd](#build) then:
+```bash
+./gradlew debBuild
 ```
-./gradlew buildDeb 
-```
-You will find the debian package in `build/dist`
+You will find the debian package in `build/dist`.  
+Then follow the instruction in the [Debian package](install/debian.md) document.
 
 ## Docker image
 [Build mxisd](#build) then:
-```
+```bash
 ./gradlew dockerBuild
 ```
-You can run a container of the given image and test it with the following command (adapt volumes host paths):
-```
-docker run -v /data/mxisd/etc:/etc/mxisd -v /data/mxisd/var:/var/mxisd -p 8090:8090 -t kamax/mxisd:latest-dev
-```
+Then follow the instructions in the [Docker install](install/docker.md#configure) document.
+
+## Next steps
+- [Integrate with your infrastructure](getting-started.md#integrate)

docs/concepts.md

@@ -0,0 +1,43 @@
+# Concepts
+- [Matrix](#matrix)
+- [mxisd](#mxisd)
+
+## Matrix
+The following concepts are part of the Matrix ecosystem and specification.
+
+### 3PID
+`3PID` stands for Third-Party Identifier.  
+It is also commonly written:
+- `3pid`
+- `tpid`
+
+A 3PID is a globally unique canonical identifier which is made of:
+- Medium, which describes what network it belongs to (Email, Phone, Twitter, Discord, etc.)
+- Address, the actual value people typically use on a daily basis.
+
+mxisd core mission is to map those identifiers to Matrix User IDs.
+
+### Homeserver
+Where a user **account and data** are stored.
+
+### Identity server
+An Identity server:
+- Does lookup of 3PIDs to User Matrix IDs.
+- Does validate 3PIDs ownership, typically by sending a code that the user has to enter in an application/on a website.
+- Does send notifications about room invites where no Matrix User ID could be found for the invitee.
+
+An Identity server:
+- **DOES NOT** store user accounts.
+- **DOES NOT** store user data.
+- **DOES NOT** allow migration of user account and/or data between homeservers. 
+
+### 3PID session
+The fact to validate a 3PID (email, phone number, etc.) via the introduction of a token which was sent to the 3PID address.
+
+## mxisd
+The following concepts are specific to mxisd.
+
+### Identity store
+Where your user accounts and 3PID mappings are stored.
+
+mxisd itself **DOES NOT STORE** user accounts or 3PID mappings.

docs/configure.md

@@ -1,183 +1,86 @@
 # Configuration
+- [Concepts](#concepts)
   - [Syntax](#syntax)
-- [Variables](#variables)
-- [Categories](#categories)
+- [Matrix](#matrix)
+- [Server](#server)
+- [Storage](#storage)
+- [Identity stores](#identity-stores)
+- [3PID Validation sessions](#3pid-validation-sessions)
+- [Notifications](#notifications)
 
-## Syntax
-The configuration file is YAML based, allowing two types of syntax.
-
-Properties-like:
-```
-my.config.item: 'value'
-```
-
-Object-like:
-```
+## Concepts
+### Syntax
+The configuration file is [YAML](http://yaml.org/) based:
+```yaml
 my:
   config:
     item: 'value'
 
 ```
-These can also be combined within the same file.  
-Both syntax will be used interchangeably in these documents.
 
-Default values for each possible option are documented [here](../src/main/resources/application.yaml)
+When referencing keys in all documents, a property-like shorthand will be used. The shorthand for the above example would be `my.config.item`
 
-## Variables
-It is possible to copy the value of a configuration item into another using the syntax `${config.key.item}`.  
-Example that will copy the value of `matrix.domain` into `server.name`:
-```
+## Matrix
+`matrix.domain`
+Matrix domain name, same as the Homeserver, used to build appropriate Matrix IDs |
+
+---
+
+`matrix.identity.servers`
+Namespace to create arbitrary list of Identity servers, usable in other parts of the configuration |
+
+Example:
+```yaml
 matrix:
-  domain: 'example.org'
-
-server:
-  name: '${matrix.domain}'
+  identity:
+    servers:
+      myOtherServers:
+        - 'https://other1.example.org'
+        - 'https://other2.example.org'
 ```
+Create a list under the label `myOtherServers` containing two Identity servers: `https://other1.example.org` and `https://other2.example.org`.
 
-**WARNING:** mxisd might overwrite/adapt some values during launch. Those changes will not be reflected into copied keys.
+## Server
+- `server.name`: Public hostname of mxisd, if different from the Matrix domain.
+- `server.port`: HTTP port to listen on (unencrypted)
+- `server.publicUrl`: Defaults to `https://{server.name}`
 
-## Categories
-For each category below, the base configuration path will be given, which needs to be appended to every configuration
-item described.
+## Storage
+### SQLite
+`storage.provider.sqlite.database`: Absolute location of the SQLite database
 
-Example: if the base path was `basePath` and the following table was given:
+## Identity stores
+See the [Identity stores](stores/README.md) for specific configuration
 
-| Name | Purpose |
-|------|---------|
-| item1 | To give an example |
-| item2 | To give another example |
+## 3PID Validation sessions
+See the dedicated documents:
+- [Flow](threepids/session/session.md)
+- [Branding](threepids/session/session-views.md)
 
-The following configurations could be used, all being equivalent:
-```
-basePath.item1: 'myValue'
-basePath.item2: 'myOtherValue'
-```
-```
-basePath:
-  item1: 'myValue'
-  item2: 'myOtherValue'
-```
-```
-basePath.item1: 'myValue'
-basePath:
-  item2: 'myOtherValue'
-```
-
----
-
-In case a relative base path is given, it is appended to the one above.
-
-Example: With base path `basePath`, the relative base `relativeBasePath` and the following table:
-  
-| Name | Purpose |
-|------|---------|
-| item1 | To give an example |
-| item2 | To give another example |
-
-The following configurations could be used, all being equivalent:
-```
-basePath.relativeBasePath.item1: 'myValue'
-basePath.relativeBasePath.item2: 'myOtherValue'
-```
-```
-basePath:
-  relativeBasePath:
-    item1: 'myValue'
-    item2: 'myOtherValue'
-```
-```
-basePath.relativeBasePath.item1: 'myValue'
-basePath:
-  relativeBasePath:
-    item2: 'myOtherValue'
-```
-
-### Matrix
-Base path: `matrix`
-
-| Name | Purpose |
-|------|---------|
-| `domain` | Matrix domain name, same as the Homeserver, used to build appropriate Matrix IDs |
-
----
-
-Relative base path: `identity`
-
-| Name | Purpose |
-|------|---------|
-| `servers` | Namespace to create arbitrary list of Identity servers, usable in other parts of the configuration |
+## Notifications
+- `notification.handler.<3PID medium>`: Handler to use for the given 3PID medium. Repeatable.
 
 Example:
-```
-matrix.identity.servers:
-  root:
-    - 'https://matrix.org'
-```
-Create a list under the label `root` containing a single Identity server, `https://matrix.org`
-### Server
-| Name | Purpose |
-|------|---------|
-| `name` | Public hostname of mxisd, if different from the Matrix domain |
-| `port` | HTTP port to listen on (unencrypted) |
-| `publicUrl` | Defaults to `https://${server.name}` |
-
-### Storage
-Base path: `storage`
-
-| Name | Purpose |
-|------|---------|
-| `backend` | Specify which SQL backend to use. only `sqlite` is currently supported. |
-
----
-Relative base path: `provider.sqlite`
-
-| Name | Purpose |
-|------|---------|
-| `database` | Absolute location of the SQLite database |
-
-### Backends
-- [LDAP](backends/ldap.md)
-- [SQL](backends/sql.md)
-- [REST](backends/rest.md)
-- [Google Firebase](backends/firebase.md)
-
-### Lookups
-work in progress, should not be configured.
-
-### Sessions
-See the [dedicated document](sessions/3pid.md)
-
-### Notifications
-Base path: `notification`
-
-| Name | Purpose |
-|------|---------|
-| handler | Namespace to specify the handler to use for each 3PID |
-| handlers | Namespace used by individual handlers for their own configuration |
-
-Example:
-```
+```yaml
 notification:
   handler:
     email: 'sendgrid'
     msisdn: 'raw'
-  handlers:
-    raw:
-      ...
-    sendgrid:
-      ...
 ```
-- Emails notifications would use the `sendgrid` handler, which define its own configuration user `handlers.sendgrid`
-- Phone notification would use the `raw` handler, basic default built-in handler of mxisd
-#### Handlers
-Relative base path: `handlers`
+- Emails notifications would use the `sendgrid` handler, which define its own configuration under `notification.handlers.sendgrid`
+- Phone notification would use the `raw` handler, basic default built-in handler in mxisd
+
+### Handlers
+- `notification.handers.<handler ID>`: Handler-specific configuration for the given handler ID. Repeatable.
+
+Example:
+```yaml
+notification:
+  handlers:
+    raw: ...
+    sendgrid: ...
+```
 
 Built-in:
-- [Basic](threepids/notifications/basic-handler.md)
-- [SendGrid](threepids/notifications/sendgrid-handler.md)
-
-### Views
-See the [dedicated document](sessions/3pid-views.md)
-
-### DNS Overwite
-Specific to other features.
+- [Raw](threepids/notification/basic-handler.md)
+- [SendGrid](threepids/notification/sendgrid-handler.md)

docs/faq.md

@@ -1,30 +1,74 @@
-# FAQ
+# Frequently Asked Questions
+### This is all very complicated and I'm getting confused with all the words, concepts and diagrams - Help!
+Matrix is still a very young protocol and there are a whole lot of rough edges.  
+Identity in Matrix is one of the most difficult topic, mainly as it has not received much love in the past years.
+
+We have tried our best to put together documentation that requires almost no knowledge of Matrix inner workings to get a
+first basic setup running which relies on you reading the documentation in the right order:
+- [The Concepts](concepts.md) in few words.
+- [Getting Started](getting-started.md) step-by-step to a minimal working install.
+- [Identity stores](stores/README.md) you wish to fetch data from.
+- [Features](features) you are interested in that will use your Identity store(s) data.
+
+**IMPORTANT**: Be aware that mxisd tries to fit within the current protocol and existing products and basic understanding
+of the Matrix protocol is required for some advanced features.
+
+If all fails, come over to [the project room](https://matrix.to/#/#mxisd:kamax.io) and we'll do our best to get you
+started and answer questions you might have.
+
+### What kind of setup is mxisd really designed for?
+mxisd is primarily designed for setups that:
+- [Care for their privacy](https://github.com/kamax-matrix/mxisd/wiki/mxisd-and-your-privacy)
+- Have their own [domains](https://en.wikipedia.org/wiki/Domain_name)
+- Use those domains for their email addresses and all other services
+- Already have an [Identity store](stores/README.md), typically [LDAP-based](stores/ldap.md).
+
+If you meet all the conditions, then you are the prime use case we designed mxisd for. 
+
+If you meet some of the conditions, but not all, mxisd will still be a good fit for you but you won't fully enjoy all its
+features.
+
 ### Do I need to use mxisd if I run a Homeserver?
-No, but it is recommended, even if you don't use any backends or integration.
+No, but it is strongly recommended, even if you don't use any Identity store or integration.
 
-In its default configuration, mxisd will talk to the central Matrix Identity servers and use other federated public
-servers when performing queries, giving you access to at least the same information as if you were not running it.
-
-It will also give your users a choice to make their 3PIDs available publicly, ensuring they are made aware of the
-privacy consequences, which is not the case with the central Matrix.org servers.
+In its default configuration, mxisd uses other federated public servers when performing queries.  
+It can also [be configured](features/identity.md#lookups) to use the central matrix.org servers, giving you access to at
+least the same information as if you were not running it.
 
 So mxisd is like your gatekeeper and guardian angel. It does not change what you already know, just adds some nice
 simple features on top of it.
 
-### I already use the synapse LDAP3 auth provider, why should I care about mxisd?
-The [synapse LDAP3 auth provider](https://github.com/matrix-org/matrix-synapse-ldap3) is not longer maintained and
-only handles on specific flow: validate credentials at login.
+### I'm not sure I understand what an "Identity server" is supposed to be or do...
+The current Identity service API is more a placeholder, as the Matrix devs did not have time so far to really work on
+what they want to do with that part of the ecosystem. Therefore, "Identity" is currently a misleading word and concept.
+Given the scope of the current Identity Service API, it would be best called "Invitation service".
+
+Because the current scope is so limited and no integration is done with the Homeserver, there was a big lack of features
+for groups/corporations/organisation. This is where mxisd comes in.
+
+mxisd implements the Identity Service API and also a set of features which are expected by regular users, truly living
+up to its "Identity server" name.
+
+### Can I migrate my existing account on another Matrix server with mxisd?
+No.
+
+Accounts cannot currently migrate/move from one server to another.  
+See a [brief explanation document](concepts.md) about Matrix and mxisd concepts and vocabulary.
+
+### I already use the synapse LDAP3 auth provider. Why should I care about mxisd?
+The [synapse LDAP3 auth provider](https://github.com/matrix-org/matrix-synapse-ldap3) is not longer maintained despite
+saying so and only handles on specific flow: validate credentials at login.
 
 It does not:
 - Auto-provision user profiles
 - Integrate with Identity management
 - Integrate with Directory searches
-- Protect you against the username case sensitivites issues in synapse
+- Integrate with Profile data
 
-mxisd is a replacement and enhancement of it, offering coherent results in all areas, which LDAP3 auth provider
+mxisd is a replacement and enhancement of it, offering coherent results in all areas, which the LDAP3 auth provider
 does not.
 
-### Sydent is the official Identity server implementation of the Matrix team, why not use that?
+### Sydent is the official Identity server implementation of the Matrix team. Why not use that?
 You can, but [sydent](https://github.com/matrix-org/sydent):
 - [should not be used and/or self-hosted](https://github.com/matrix-org/sydent/issues/22)
 - is not meant to be linked to a specific Homeserver / domain
@@ -35,31 +79,19 @@ You can, but [sydent](https://github.com/matrix-org/sydent):
 So really, you should go with mxisd.
 
 ### Will I loose access to the central Matrix.org/Vector.im Identity data if I use mxisd?
-In its default configuration, mxisd act as a proxy to Matrix.org/Vector.im. You will have access to the same data and
-behaviour than if you were using them directly. There is no downside in using mxisd with the default configuration.
+No.
 
-mxisd can also be configured not to talk to the central Identity servers if you wish.
+In its default configuration, mxisd does not talk to the central Identity server matrix.org to avoid leaking your private
+data and those of people you might know.
 
-### I'm not sure I understand what an "Identity server" is supposed to be or do
-The current Identity service API is more a placeholder, as the Matrix devs did not have time so far to really work on
-what they want to do with that part of the ecosystem. Therefore, "Identity" is a misleading word currently.  
-Given the scope of the current Identity Service API, it would be best called "Invitation service".
-
-Because the current scope is so limited and no integration is done with the Homeserver, there was a big lack of features
-for groups/corporations/organisation. This is where mxisd comes in.
-
-mxisd implements the Identity Service API and also a set of features which are expected by regular users, truly living
-up to its "Identity server" name.
+[You can configure it](features/identity.md#lookups) to talk to the central Identity servers if you wish.
 
 ### So mxisd is just a big hack! I don't want to use non-official features!
-mxisd primary concern is to always be compatible with the Matrix ecosystem and the Identity service API.  
+mxisd primary concerns are your privacy and to always be compatible with the Matrix ecosystem and the Identity service API.  
 Whenever the API will be updated and/or enhanced, mxisd will follow, remaining 100% compatible with the ecosystem.
 
-We also directly talk with the Matrix developers to ensure all features we implement have their approval, and that we
-are in line with their vision of Identity management within the Matrix ecosystem.
-
-Therefore, using mxisd is a safe choice. It will be like using the central Matrix.org Identity servers, yet not closing
-the door to very nice enhancements and integrations.
-
 ### Should I use mxisd if I don't host my own Homeserver?
-No
+No.
+
+It is possible, but it is not supported and the scope of features will be extremely limited.
+Please consider hosting your own Homeserver and using mxisd alongside it.

docs/features/authentication.md

@@ -1,60 +1,59 @@
 # Authentication
-
 - [Description](#description)
+- [Basic](#basic)
   - [Overview](#overview)
-- [Getting started](#getting-started)
-  - [Synapse](#synapse)
+  - [synapse](#synapse)
   - [mxisd](#mxisd)
   - [Validate](#validate)
   - [Next steps](#next-steps)
     - [Profile auto-fil](#profile-auto-fill)
-- [Advanced Authentication](#advanced-authentication)
+- [Advanced](#advanced)
+  - [Overview](#overview-1)
   - [Requirements](#requirements)
   - [Configuration](#configuration)
     - [Reverse Proxy](#reverse-proxy)
       - [Apache2](#apache2)
     - [DNS Overwrite](#dns-overwrite)
-    - [Backends](#backends) 
 
 ## Description
-Authentication is an enhanced Identity feature of mxisd to ensure coherent and centralized identity management.
-
+Authentication is an enhanced feature of mxisd to ensure coherent and centralized identity management.  
 It allows to use Identity stores configured in mxisd to authenticate users on your Homeserver.
 
-This feature can also provide the ability to users to login on the Homeserver using their third party identities (3PIDs) provided by an Identity store.
+Authentication is divided into two parts:
+- [Basic](#basic): authenticate with a regular username.
+- [Advanced](#advanced): same as basic with extra abilities like authenticate using a 3PID or do username rewrite.
 
-## Overview
-An overview of the Authentication process is depicted below: 
+## Basic
+Authentication by username is possible by linking synapse and mxisd together using a specific module for synapse, also
+known as password provider.
 
+### Overview
+An overview of the Basic Authentication process:
 ```
-                                                                                    Backends
+                                                                                    Identity stores
  Client                                                                             +------+
    |                                            +-------------------------+    +--> | LDAP |
    |   +---------------+  /_matrix/identity     | mxisd                   |    |    +------+
    +-> | Reverse proxy | >------------------+   |                         |    |
        +--|------------+                    |   |                         |    |    +--------+
-          |                                 +-----> Check with backends >------+--> | SQL DB |
+          |                                 +-----> Check ID stores     >------+--> | SQL DB |
      Login request                          |   |                         |    |    +--------+
           |                                 |   |     |                   |    |
-          |   +--------------------------+  |   +-----|-------------------+    +-->  Others
+          |   +--------------------------+  |   +-----|-------------------+    +-->  ...
           +-> | Homeserver               |  |         |
               |                          |  |         |
               | - Validate credentials >----+         |
               |   Using REST auth module |            |
               |                          |            |
               | - Auto-provision <-------------------<+
-              |   user profiles          |    If valid credentials and supported by backend
+              |   user profiles          |    If valid credentials and supported by Identity store(s)
               +--------------------------+
 ```
 Performed on [synapse with REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth/blob/master/README.md)
 
-## Getting started
-Authentication is possible by linking synapse and mxisd together using the REST auth module
-(also known as password provider).
-
 ### Synapse
-- Install the [REST auth module](https://github.com/kamax-io/matrix-synapse-rest-auth).
-- Edit your synapse configuration:
+- Install the [password provider](https://github.com/kamax-io/matrix-synapse-rest-auth)
+- Edit your **synapse** configuration:
   - As described by the auth module documentation
   - Set `endpoint` to `http://mxisdAddress:8090` - Replace `mxisdAddress` by an IP/host name that provides a direct
   connection to mxisd.  
@@ -62,25 +61,31 @@ Authentication is possible by linking synapse and mxisd together using the REST
 - Restart synapse
 
 ### mxisd
-- Configure and enable at least one [Identity store](../backends/)
+- Configure and enable at least one [Identity store](../stores/README.md)
 - Restart mxisd
 
 ### Validate
-Login on the Homeserver using credentials present in your backend.
+Login on the Homeserver using credentials present in one of your Identity stores.
 
 ## Next steps
 ### Profile auto-fill
-Auto-filling user profile depends on two conditions:
-- The REST auth module is configured for it, which is the case by default
-- Your Identity store is configured to provide profile data. See your Identity store [documentation](../backends/) on
-how to enable the feature.
+Auto-filling user profile depends on its support by your configured Identity stores.  
+See your Identity store [documentation](../stores/README.md) on how to enable the feature.
 
 
-## Advanced Authentication
-The Authentication feature allows users to login to their Homeserver by using their 3PIDs registered in an available Identity store.
+## Advanced
+The Authentication feature allows users to:
+- Rewrite usernames matching a pattern to be mapped to another username via a 3PID.
+- login to their Homeserver by using their 3PIDs in a configured Identity store.
 
+This feature also allows to work around the following issues:
+- Lowercase all usernames for synapse, allowing case-insensitive login
+- Unable to login on synapse if username is numerical
+- Any generic transformation of username prior to sending to synapse, bypassing the restriction that password providers
+cannot change the localpart being authenticated.
+
+### Overview
 This is performed by intercepting the Homeserver endpoint `/_matrix/client/r0/login` as depicted below:
-
 ```
             +----------------------------+
             |  Reverse Proxy             |
@@ -106,31 +111,28 @@ Client+---->| /_matrix/client/r0/login +---------------->|
 
 Steps of user authentication using a 3PID:
 1. The intercepted login request is directly sent to mxisd instead of the Homeserver.
-2. Enabled backends are queried for a matching user identity in order to modify the request to use the user name.
-3. The Homeserver, from which the request was intercepted, is queried using the request at previous step. Its address is resolved using the DNS Overwrite feature to reach its internal address on a non-encrypted port.
+2. Identity stores are queried for a matching user identity in order to modify the request to use the user name.
+3. The Homeserver, from which the request was intercepted, is queried using the request at previous step.
+   Its address is resolved using the DNS Overwrite feature to reach its internal address on a non-encrypted port.
 4. The response from the Homeserver is sent back to the client, believing it was the HS which directly answered.
 
 ### Requirements
+- Compatible [Identity store](../stores/README.md)
+- [Basic Authentication configured and working](#basic)
+- Client and Homeserver using the [C2S API r0.4.x](https://matrix.org/docs/spec/client_server/r0.4.0.html) or later
 - Reverse proxy setup
-- Homeserver
-- Compatible Identity backends:
-	- LDAP
-	- REST
-	- Wordpress
 
 ### Configuration
-
 #### Reverse Proxy
-
 ##### Apache2
 The specific configuration to put under the relevant `VirtualHost`:
-```
+```apache
 ProxyPass /_matrix/client/r0/login http://localhost:8090/_matrix/client/r0/login
 ```
-`ProxyPreserveHost` or equivalent must be enabled to detect to which Homeserver mxisd should talk to when building results.
+`ProxyPreserveHost` or equivalent **must** be enabled to detect to which Homeserver mxisd should talk to when building results.
 
-Your VirtualHost should now look like this:
-```
+Your VirtualHost should now look similar to:
+```apache
 <VirtualHost *:443>
     ServerName example.org
     
@@ -138,25 +140,112 @@ Your VirtualHost should now look like this:
     
     ProxyPreserveHost on
     ProxyPass /_matrix/client/r0/login http://localhost:8090/_matrix/client/r0/login
-    ProxyPass /_matrix/identity/ http://localhost:8090/_matrix/identity/
-    ProxyPass /_matrix/ http://localhost:8008/_matrix/
+    ProxyPass /_matrix/identity http://localhost:8090/_matrix/identity
+    ProxyPass /_matrix http://localhost:8008/_matrix
 </VirtualHost>
 ```
 
-#### DNS Overwrite
-Just like you need to configure a reverse proxy to send client requests to mxisd, you also need to configure mxisd with the internal IP of the Homeserver so it can talk to it directly to integrate its directory search.
+##### nginx
 
+The specific configuration to add under the relevant `server`:
 
-To do so, put the following configuration in your `application.yaml`:
+```nginx
+location /_matrix/client/r0/login {
+    proxy_pass http://localhost:8090;
+    proxy_set_header Host $host;
+    proxy_set_header X-Forwarded-For $remote_addr;
+}
 ```
-dns.overwrite.homeserver.client:
+
+Your `server` section should now look similar to:
+
+```nginx
+server {
+    listen 443 ssl;
+    server_name matrix.example.org;
+    
+    # ...
+    
+    location /_matrix/client/r0/login {
+        proxy_pass http://localhost:8090;
+        proxy_set_header Host $host;
+        proxy_set_header X-Forwarded-For $remote_addr;
+    }
+    
+    location /_matrix/identity {
+        proxy_pass http://localhost:8090/_matrix/identity;
+        proxy_set_header Host $host;
+        proxy_set_header X-Forwarded-For $remote_addr;
+    }
+    
+    location /_matrix {
+        proxy_pass http://localhost:8008/_matrix;
+        proxy_set_header Host $host;
+        proxy_set_header X-Forwarded-For $remote_addr;
+    }
+}
+```
+
+#### DNS Overwrite
+
+Just like you need to configure a reverse proxy to send client requests to mxisd, you also need to configure mxisd with
+the internal IP of the Homeserver so it can talk to it directly to integrate its directory search.
+
+To do so, put the following configuration in your mxisd configuration:
+```yaml
+dns:
+  overwrite:
+    homeserver:
+      client:
         - name: 'example.org'
           value: 'http://localhost:8008'
 ```
 `name` must be the hostname of the URL that clients use when connecting to the Homeserver.
-In case the hostname is the same as your Matrix domain, you can use `${matrix.domain}` to auto-populate the `value` using the `matrix.domain` configuration option and avoid duplicating it.
+You can use `${server.name}` to auto-populate the `value` using the `server.name` configuration option and avoid duplicating it.
+In case the hostname is the same as your Matrix domain and `server.name` is not explicitely set in the config, `server.name` will default to
+`matrix.domain` and will still probably have the correct value.
 
-value is the base internal URL of the Homeserver, without any /_matrix/.. or trailing /.
+`value` is the base internal URL of the Homeserver, without any `/_matrix/..` or trailing `/`.
 
-#### Backends
-The Backends should be configured as described in the documentation of the [Directory User](directory-users.md) feature. 
+### Optional features
+
+The following features are available after you have a working Advanced setup:
+
+- Username rewrite: Allows you to rewrite the username of a regular login/pass authentication to a 3PID, that then gets resolved using the regular lookup process. Most common use case is to allow login with numerical usernames on synapse, which is not possible out of the box.
+
+#### Username rewrite
+In mxisd config:
+```yaml
+auth:
+  rewrite:
+    user:
+      rules:
+        - regex: <your regexp>
+          medium: 'your.custom.medium.type'
+```
+`rules` takes a list of rules. Rules have two properties:
+- `regexp`: The regex pattern to match. This **MUST** match the full string. See [Java regex](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html) for syntax.
+- `medium`: Custom 3PID type that will be used in the 3PID lookup. This can be anything you want and needs to be supported
+by your Identity store config and/or code.
+
+Rules are matched in listed order.
+
+Common regexp patterns:
+- Numerical usernames: `[0-9]+`
+
+##### LDAP Example
+If your users use their numerical employee IDs, which cannot be used with synapse, you can make it work with (relevant config only):
+```yaml
+auth:
+  rewrite:
+    user:
+      rules:
+        - regex: '[0-9]+'
+          medium: 'kmx.employee.id'
+          
+ldap:
+  attribute:
+    threepid:
+      kmx.employee.id:
+        - 'ldapAttributeForEmployeeId'
+```

docs/features/bridge-integration.md

@@ -1 +1,31 @@
-To be documented
+# Bridge Integration
+To help natural bridge integration into the regular usage of a Matrix client, mxisd provides a way for bridge to reply
+to 3PID queries if no mapping was found, allowing seamless bridging to a network.
+
+This is performed by implementing a specific endpoint on the bridge to map a 3PID lookup to a virtual user.
+
+**NOTE**: This document is incomplete and might be misleading. In doubt, come in our Matrix room.  
+You can also look at our [Email Bridge README](https://github.com/kamax-matrix/matrix-appservice-email#mxisd) for an example
+of working configuration.
+
+## Configuration
+```yaml
+lookup:
+  recursive:
+    bridge:
+      enabled: <boolean>
+      recursiveOnly: <boolean>
+      server: <URL to the bridge endpoint for all 3PID medium>
+      mappings:
+        <3PID MEDIUM HERE>: <URL to dedicated bridge for that medium>
+
+```
+
+## Integration
+Implement a simplified version of the [Identity service single lookup endpoint](https://kamax.io/matrix/api/identity_service/unstable.html#get-matrix-identity-api-v1-lookup)
+with only the following parameters needed:
+- `address`
+- `medium`
+- `mxid`
+
+Or an empty object if no resolution exists or desired.

docs/features/directory.md

@@ -7,10 +7,6 @@
     - [Apache2](#apache2)
     - [nginx](#nginx)
   - [DNS Overwrite](#dns-overwrite)
-  - [Backends](#backends)
-    - [LDAP](#ldap)
-    - [SQL](#sql)
-    - [REST](#rest)
 - [Next steps](#next-steps)
 
 ## Description
@@ -18,17 +14,17 @@ This feature allows you to search for existing and/or potential users that are a
 or that already share a room with you on the Homeserver.
 
 Without any integration, synapse:
-- Only search within the users **already** known to you
+- Only search within the users **already** known to you or in public rooms
 - Only search on the Display Name and the Matrix ID
 
-With mxisd integration, you can:
+By enabling this feature, you can by default:
 - Search on Matrix ID, Display name and 3PIDs (Email, phone numbers) of any users already in your configured backend
 - Search for users which you are not in contact with yet. Super useful for corporations who want to give Matrix access
 internally, so users can just find themselves **prior** to having any common room(s)
-- Use any attribute of your backend to extend the search!
-- Include your homeserver search results to those found by mxisd (default behaviour, no configuration required)
+- Add extra attributes of your backend to extend the search
+- Include your homeserver search results to those found by mxisd
 
-By integrating mxisd, you get the default behaviour with all the extras, ensuring your users will always find each other.
+By integrating mxisd, you get the default behaviour and a bunch of extras, ensuring your users will always find each other.
 
 ## Overview
 This is performed by intercepting the Homeserver endpoint `/_matrix/client/r0/user_directory/search` like so:
@@ -49,31 +45,28 @@ Client --> | Reverse proxy
 ```
 Steps:
 1. The intercepted request is directly sent to mxisd instead of the Homeserver.
-2. Enabled backends are queried for any match on the search value sent by the client.
+2. Identity stores are queried for any match on the search value sent by the client.
 3. The Homeserver, from which the request was intercepted, is queried using the same request as the client.
    Its address is resolved using the DNS Overwrite feature to reach its internal address on a non-encrypted port.
-4. Results from backends and the Homeserver are merged together and sent back to the client, believing it was the HS
+4. Results from Identity stores and the Homeserver are merged together and sent back to the client, believing it was the HS
 which directly answered the request.
 
 ## Requirements
 - Reverse proxy setup, which you should already have in place if you use mxisd
-- Compatible backends:
-  - LDAP
-  - SQL
-  - REST
+- At least one compatible [Identity store](../stores/README.md) enabled
   
 ## Configuration
 ### Reverse Proxy
 #### Apache2
 The specific configuration to put under the relevant `VirtualHost`:
-```
+```apache
 ProxyPass /_matrix/client/r0/user_directory/ http://0.0.0.0:8090/_matrix/client/r0/user_directory/
 ```
 `ProxyPreserveHost` or equivalent must be enabled to detect to which Homeserver mxisd should talk to when building
 results.
 
 Your `VirtualHost` should now look like this:
-```
+```apache
 <VirtualHost *:443>
     ServerName example.org
     
@@ -81,14 +74,14 @@ Your `VirtualHost` should now look like this:
     
     ProxyPreserveHost on
     ProxyPass /_matrix/client/r0/user_directory/ http://localhost:8090/_matrix/client/r0/user_directory/
-    ProxyPass /_matrix/identity/ http://localhost:8090/_matrix/identity/
-    ProxyPass /_matrix/ http://localhost:8008/_matrix/
+    ProxyPass /_matrix/identity http://localhost:8090/_matrix/identity
+    ProxyPass /_matrix http://localhost:8008/_matrix
 </VirtualHost>
 ```
 
 #### nginx
 The specific configuration to add under your `server` section is:
-```
+```nginx
 location /_matrix/client/r0/user_directory {
     proxy_pass http://0.0.0.0:8090/_matrix/client/r0/user_directory;
     proxy_set_header Host $host;
@@ -97,7 +90,7 @@ location /_matrix/client/r0/user_directory {
 ```
 
 Your `server` section should now look like this:
-```
+```nginx
 server {
     listen 443 ssl;
     server_name example.org;
@@ -129,103 +122,32 @@ Just like you need to configure a reverse proxy to send client requests to mxisd
 the internal IP of the Homeserver so it can talk to it directly to integrate its directory search.
 
 To do so, use the following configuration:
-```
-dns.overwrite.homeserver.client:
+```yaml
+dns:
+  overwrite:
+    homeserver:
+      client:
         - name: 'example.org'
           value: 'http://localhost:8008'
 ```
-`name` must be the hostname of the URL that clients use when connecting to the Homeserver.  
-In case the hostname is the same as your Matrix domain, you can use `${matrix.domain}` to auto-populate the value using
-the `matrix.domain` configuration option and avoid duplicating it.
-
-`value` is the base intenral URL of the Homeserver, without any `/_matrix/..` or trailing `/`.
-
-### Backends
-#### LDAP
-To ensure Directory feature works, here's how the LDAP configuration should look like:
-```
-ldap:
-  enabled: false
-  filter: '(memberOf=CN=Matrix Users,OU=Groups,DC=example,DC=org)'
-  connection:
-    host: 'ldapIpOrDomain'
-    bindDn: 'CN=Matrix Identity Server,OU=Accounts,DC=example,DC=org'
-    bindPassword: 'mxisd'
-    baseDn: 'OU=Accounts,DC=example,DC=org'
-  attribute:
-    uid:
-      type: 'uid'
-      value: 'userPrincipalName'
-    name: 'displayName'
-    threepid:
-      email:
-        - 'mailPrimaryAddress'
-        - 'mail'
-        - 'otherMailbox'
-      msisdn:
-        - 'telephoneNumber'
-        - 'mobile'
-        - 'homePhone'
-        - 'otherTelephone'
-        - 'otherMobile'
-        - 'otherHomePhone'
-  directory:
-    attribute:
-      other:
-        - 'employeeNumber'
-        - 'someOtherAttribute'
-```
-Only include the `attribute` sub-sections if you would like to set another value. Else, it is best not to include them
-to inherit the default values.
-
-If you would like to include an attribute which is not a display name or a 3PID, you can use the
-`directory.attribute.other` to list any extra attributes you want included in searches. If you do not want to include
-any extra attribute, that configuration section can be skipped. 
-
-#### SQL
-If you plan to integrate directory search directly with synapse, use the `synapseSql` provider, based on the following
-config:
-```
-synapseSql:
-  enabled: true
-  type: <database ID>
-  connection: '<connection info>'
-```
-`type` and `connection`, including any other configuration item, follow the same values as the regular [SQL backend](../backends/sql.md).
-
----
-
-For the regular SQL backend, the following configuration items are available:
-```
-sql:
-  directory:
-    enabled: true
-    query:
-      name:
-        type: 'localpart'
-        value: 'SELECT idColumn, displayNameColumn FROM table WHERE displayNameColumn LIKE ?'
-      threepid:
-        type: 'localpart'
-        value: 'SELECT idColumn, displayNameColumn FROM table WHERE threepidColumn LIKE ?'
-```
-For each query, `type` can be used to tell mxisd how to process the ID column:
-- `localpart` will append the `matrix.domain` to it
-- `mxid` will use the ID as-is. If it is not a valid Matrix ID, the search will fail.
-
-`value` is the SQL query and must return two columns:
-- The first being the User ID
-- The second being its display name
-
-#### REST
-See the [dedicated document](../backends/rest.md)
-
-#### Wordpress
-See the [dedicated document](../backends/wordpress.md)
+- `name` must be the hostname of the URL that clients use when connecting to the Homeserver.
+- `value` is the base internal URL of the Homeserver, without any `/_matrix/..` or trailing `/`.
 
 ## Next steps
 ### Homeserver results
 You can configure if the Homeserver should be queried at all when doing a directory search.  
-To disable Homeserver results, set the following in mxisd config file:
+To disable Homeserver results, set the following in mxisd configuration file:
+```yaml
+directory:
+  exclude:
+    homeserver: true
 ```
-directory.exclude.homeserever: true
+
+### 3PID exclusion in search
+You can configure if the 3PID should also be included when doing a directory search.
+By default, a search is performed on the 3PIDs. If you would like to not include them:
+```yaml
+directory:
+  exclude:
+    threepid: true
 ```

docs/features/experimental/application-service.md

@@ -0,0 +1,122 @@
+# Application Service
+**WARNING:** These features are currently highly experimental. They can be removed or modified without notice.  
+All the features requires a Homeserver capable of connecting [Application Services](https://matrix.org/docs/spec/application_service/r0.1.1.html).
+
+The following capabilities are provided in this feature:
+- [Admin commands](#admin-commands)
+- [Email Notification about room invites by Matrix IDs](#email-notification-about-room-invites-by-matrix-ids)
+- [Auto-reject of expired 3PID invites](#auto-reject-of-expired-3pid-invites)
+
+## Setup
+> **NOTE:** Make sure you are familiar with [configuration format and rules](../../configure.md).
+
+Integration as an Application service is a three steps process:
+1. Create the baseline mxisd configuration to allow integration.
+2. Integrate with the homeserver.
+3. Configure the specific capabilities, if applicable.
+
+### Configuration
+#### Variables
+Under the `appsvc` namespace:
+
+| Key                   | Type    | Required | Default | Purpose                                                        |
+|-----------------------|---------|----------|---------|----------------------------------------------------------------|
+| `enabled`             | boolean | No       | `false` | Globally enable/disable the feature                            |
+| `user.main`           | string  | No       | `mxisd` | Localpart for the main appservice user                         |
+| `endpoint.toHS.url`   | string  | Yes      | *None*  | Base URL to the Homeserver                                     |
+| `endpoint.toHS.token` | string  | Yes      | *None*  | Token to use when sending requests to the Homeserver           |
+| `endpoint.toAS.url`   | string  | Yes      | *None*  | Base URL to mxisd from the Homeserver                          |
+| `endpoint.toAS.token` | string  | Yes      | *None*  | Token for the Homeserver to use when sending requests to mxisd |
+
+#### Example
+```yaml
+appsvc:
+  enabled: true
+  endpoint:
+    toHS:
+      url: 'http://localhost:8008'
+      token: 'ExampleTokenToHS-ChangeMe!'
+    toAS:
+      url: 'http://localhost:8090'
+      token: 'ExampleTokenToAS-ChangeMe!'
+```
+### Integration
+#### Synapse
+Under the `appsvc.registration.synapse` namespace:
+
+| Key    | Type   | Required | Default            | Purpose                                                                  |
+|--------|--------|----------|--------------------|--------------------------------------------------------------------------|
+| `id`   | string | No       | `appservice-mxisd` | The unique, user-defined ID of this application service. See spec.       |
+| `file` | string | Yes      | *None*             | If defined, the synapse registration file that should be created/updated |
+
+##### Example 
+```yaml
+appsvc:
+  registration:
+    synapse:
+      file: '/etc/matrix-synapse/mxisd-appservice-registration.yaml'
+```
+
+Edit your `homeserver.yaml` and add a new entry to the appservice config file, which should look something like this:
+```yaml
+app_service_config_files:
+  - '/etc/matrix-synapse/mxisd-appservice-registration.yaml'
+  - ...
+```
+
+Restart synapse when done to register mxisd.
+
+#### Others
+See your Homeserver documentation on how to integrate.
+
+## Capabilities
+### Admin commands
+#### Setup
+Min config:
+```yaml
+appsvc:
+  feature:
+    admin:
+      allowedRoles:
+        - '+aMatrixCommunity:example.org'
+        - 'SomeLdapGroup'
+        - 'AnyOtherArbitraryRoleFromIdentityStores'
+```
+
+#### Use
+The following steps assume:
+- `matrix.domain` set to `example.org`
+- `appsvc.user.main` set to `mxisd` or not set
+
+1. Invite `@mxisd:example.org` to a new direct chat
+2. Type `!help` to get all available commands
+
+### Email Notification about room invites by Matrix IDs
+This feature allows for users found in Identity stores to be instantly notified about Room Invites, regardless if their
+account was already provisioned on the Homeserver.
+
+#### Requirements
+- [Identity store(s)](../../stores/README.md) supporting the Profile feature
+- At least one email entry in the identity store for each user that could be invited.
+
+#### Configuration
+In your mxisd config file:
+```yaml
+synapseSql:
+  enabled: false ## Do not use this line if Synapse is used as an Identity Store
+  type: '<DB TYPE>'
+  connection: '<DB CONNECTION URL>'
+```
+
+The `synapseSql` section is optional. It is used to retrieve display names which are not directly accessible in this mode.
+For details about `type` and `connection`, see the [relevant documentation](../../stores/synapse.md).
+If you do not configure it, some placeholders will not be available in the notification, like the Room name.
+
+You can also change the default template of the notification using the `generic.matrixId` template option.  
+See [the Template generator documentation](../../threepids/notification/template-generator.md) for more info.
+
+#### Test
+Invite a user which is part of your domain while an appropriate Identity store is used.
+
+### Auto-reject of expired 3PID invites
+*TBC*

docs/features/experimental/profile.md

@@ -0,0 +1,16 @@
+# Profile
+**WARNING**: The following sub-features are considered experimental and not officially supported. Use at your own peril.
+
+## Public Profile enhancement
+This feature allows to enhance a public profile query with more info than just Matrix ID and Display name, allowing for
+custom applications to retrieve custom data not currently provided by synapse, per example.
+
+**WARNING**: This information can be queried without authentication as per the specification. Do not enable unless in a
+controlled environment.
+
+### Configuration
+#### Reverse proxy
+##### Apache
+```apache
+ProxyPassMatch "^/_matrix/client/r0/profile/([^/]+)$" "http://127.0.0.1:8090/_matrix/client/r0/profile/$1"
+```

docs/features/federation.md

@@ -1,4 +1,15 @@
-# Identity service Federation
+# Federation
+Federation is the process by which domain owners can make compatible 3PIDs mapping auto-discoverable by looking for another
+Federated Identity server using the DNS domain part of the 3PID.
+
+Emails are the best candidate for this kind of resolution which are DNS domain based already.  
+On the other hand, Phone numbers cannot be resolved this way.
+
+For 3PIDs which are not compatible with the DNS system, mxisd can be configured to talk to fallback Identity servers like
+the central matrix.org one. See the [Identity feature](identity.md#lookups) for instructions on how to enable it.
+
+Outbound federation is enabled by default while inbound federation is opt-in and require a specific DNS record.
+
 ## Overview
 ```
               +-------------------+   +-------------> +----------+
@@ -6,28 +17,35 @@
               |                   |   |      +------> +----------+
               |                   |   |      |
               | Invites / Lookups |   |      |
- Federated    | +--------+        |   |      |        +-------------------+
- Identity  ---->| Remote |>-----------+      +------> | Remote Federated  |
- Server       | +--------+        |          |        | mxisd servers     |
-              |                   |          |        +-------------------+
-              | +--------+        |          |
- Homeserver --->| Local  |>------------------+
- and clients  | +--------+        |          |        +--------------------------+ 
-              +-------------------+          +------> | Central Identity service |
-                                                      | Matrix.org / Vector.im   |
-                                                      +--------------------------+
+ Federated    | +--------+        |   |      |
+ Identity  ---->| Remote |>-----------+      |
+ Server       | +--------+        |          |
+              |                   |          |
+              | +--------+        |          |        +-------------------+
+ Homeserver --->| Local  |>------------------+------> | Remote Federated  |
+ and clients  | +--------+        |                   | mxisd servers     |
+              +-------------------+                   +-------------------+
 ```
-To allow other federated Identity Server to reach yours, the same algorithm used for Homeservers takes place:
-1. Check for the appropriate DNS SRV record
-2. If not found, use the base domain
 
-## Configuration
-If your Identity Server public hostname does not match your Matrix domain, configure the following DNS SRV entry 
-and replace `matrix.example.com` by your Identity server public hostname - **Make sure to end with a final dot!**
+## Inbound
+If you would like to be reachable for lookups over federation, create the following DNS SRV entry and replace
+`matrix.example.com` by your Identity server public hostname:
 ```
 _matrix-identity._tcp.example.com. 3600 IN SRV 10 0 443 matrix.example.com.
 ``` 
-This would only apply for 3PID that are DNS-based, like e-mails. For anything else, like phone numbers, no federation 
-is currently possible.  
 
 The port must be HTTPS capable which is what you get in a regular setup with a reverse proxy from 443 to TCP 8090 of mxisd.
+
+## Outbound
+If you would like to disable outbound federation and isolate your identity server from the rest of the Matrix network,
+use the following mxisd configuration options:
+```yaml
+lookup:
+  recursive:
+    enabled: false
+invite:
+  resolution:
+    recursive: false
+``` 
+
+There is currently no way to selectively disable federation towards specific servers, but this feature is planned.

docs/features/identity.md

@@ -1,10 +1,96 @@
-# Matrix Identity Service
-**WARNING**: This document is incomplete and can be missleading.
+# Identity
+Implementation of the [Identity Service API r0.2.0](https://matrix.org/docs/spec/identity_service/r0.2.0.html).
 
-Implementation of the [Unofficial Matrix Identity Service API](https://kamax.io/matrix/api/identity_service/unstable.html).
+- [Lookups](#lookups)
+- [Invitations](#invitations)
+  - [Expiration](#expiration)
+  - [Policies](#policies)
+  - [Resolution](#resolution)
+- [3PIDs Management](#3pids-management)
 
-## Invitation
-Resolution can be customized using the following configuration:
+## Lookups
+If you would like to use the central matrix.org Identity server to ensure maximum discovery at the cost of potentially
+leaking all your contacts information, add the following to your configuration:
+```yaml
+forward:
+  servers:
+    - 'matrix-org'
+```
+**NOTE:** You should carefully consider enabling this option, which is discouraged.  
+For more info, see the [relevant issue](https://github.com/kamax-matrix/mxisd/issues/76).
+
+## Invitations
+### Expiration
+#### Overview
+Matrix does not provide a mean to remove/cancel pending 3PID invitations with the APIs. The current reference
+implementations also do not provide any mean to do so. This leads to 3PID invites forever stuck in rooms.
+
+To provide this functionality, mxisd uses a workaround: resolve the invite to a dedicated User ID, which can be
+controlled by mxisd or a bot/service that will then reject the invite.
+
+If this dedicated User ID is to be controlled by mxisd, the [Application Service](experimental/application-service.md)
+feature must be configured and integrated with your Homeserver, as well as the *Auto-reject 3PID invite capability*.
+
+#### Configuration
+```yaml
+invite:
+  expiration:
+    enabled: true/false
+    after: 5
+    resolveTo: '@john.doe:example.org'
+```
+`enabled`
+- Purpose: Enable or disable the invite expiration feature.
+- Default: `true`
+
+`after` 
+- Purpose: Amount of minutes before an invitation expires.
+- Default: `10080` (7 days)
+
+`resolveTo`
+- Purpose: Matrix User ID to resolve the expired invitations to.
+- Default: Computed from `appsvc.user.inviteExpired` and `matrix.domain`
+
+### Policies
+3PID invite policies are the companion feature of [Registration](registration.md). While the Registration feature acts on
+requirements for the invitee/register, this feature acts on requirement for the one(s) performing 3PID invites, ensuring
+a coherent system.
+
+It relies on only allowing people with specific [Roles](profile.md) to perform 3PID invites. This would typically allow
+a tight-control on a server setup with is "invite-only" or semi-open (relying on trusted people to invite new members).
+
+It's a middle ground between a closed server, where every user must be created or already exists in an Identity store,
+and an open server, where anyone can register.
+ 
+#### Integration
+Because Identity Servers do not control 3PID invites as per Matrix spec, mxisd needs to intercept a set of Homeserver
+endpoints to apply the policies.
+
+##### Reverse Proxy
+###### nginx
+**IMPORTANT**: Must be placed before your global `/_matrix` entry:
+```nginx
+location ~* ^/_matrix/client/r0/rooms/([^/]+)/invite$ {
+    proxy_pass		    http://127.0.0.1:8090;
+    proxy_set_header	Host $host;
+    proxy_set_header	X-Forwarded-For $remote_addr;
+}
+```
+
+#### Configuration
+The only policy currently available is to restrict 3PID invite to users having a specific (set of) role(s), like so:
+
+```yaml
+invite:
+  policy:
+    ifSender:
+      hasRole:
+        - '<THIS_ROLE>'
+        - '<OR_THIS_ROLE>'
+```
+
+### Resolution
+Resolution of 3PID invitations can be customized using the following configuration:
 
 `invite.resolution.recursive`  
 - Default value: `true`  
@@ -16,3 +102,6 @@ Resolution can be customized using the following configuration:
 `invite.resolution.timer`  
 - Default value: `1`  
 - Description: How often, in minutes, mxisd should try to resolve pending invites.
+
+## 3PIDs Management
+See the [3PID session documents](../threepids/session)

docs/features/profile.md

@@ -1,12 +1,10 @@
-# Profile enhancement
+# Profile
+The profile feature does not do anything on its own and acts as a support feature for others, allowing to retrieve
+information about a user based on its Matrix ID by querying enabled [Identity stores](../stores/README.md).
 
-## Configuration
+Currently supported:
+- Display name
+- 3PIDs
+- Roles/Groups
 
-### Reverse proxy
-
-#### Apache
-```
-ProxyPassMatch "^/_matrix/client/r0/profile/([^/]+)$" "http://127.0.0.1:8090/_matrix/client/r0/profile/$1"
-ProxyPassMatch "^/_matrix/client/r0/profile/([^/]+)/(.+)" "http://127.0.0.1:8008/_matrix/client/r0/profile/$1/$2"
-
-```
+Experimental sub-features are also available. See [the dedicated document](experimental/profile.md).

docs/features/registration.md

@@ -0,0 +1,111 @@
+# Registration
+- [Overview](#overview)
+- [Integration](#integration)
+  - [Reverse Proxy](#reverse-proxy)
+    - [nginx](#nginx)
+    - [Apache2](#apache2)
+  - [Homeserver](#homeserver)
+    - [synapse](#synapse)
+- [Configuration](#configuration)
+  - [Example](#example)
+- [Usage](#usage)
+
+## Overview
+**NOTE**: This feature is beta: it is considered stable enough for production but is incomplete and may contain bugs.
+
+Registration is an enhanced feature of mxisd to control registrations involving 3PIDs on a Homeserver based on policies:
+- Match pending 3PID invites on the server
+- Match 3PID pattern, like a specific set of domains for emails
+- In further releases, use 3PIDs found in Identity stores
+
+It aims to help open or invite-only registration servers control what is possible to do and ensure only approved people
+can register on a given server in a implementation-agnostic manner.
+
+**IMPORTANT:** This feature does not control registration in general. It only acts on endpoints related to 3PIDs during
+the registration process.  
+As such, it relies on the homeserver to require 3PIDs with the registration flows.
+
+This feature is not part of the Matrix Identity Server spec.
+
+## Integration
+mxisd needs to be integrated at several levels for this feature to work:
+- Reverse proxy: intercept the 3PID register endpoints and act on them
+- Homeserver: require 3PID to be part of the registration data
+
+Later version(s) of this feature may directly control registration itself to create a coherent experience
+### Reverse Proxy
+#### nginx
+```nginx
+location ~* ^/_matrix/client/r0/register/[^/]+/requestToken$ {
+	proxy_pass		http://localhost:8090;
+	proxy_set_header	Host $host;
+	proxy_set_header	X-Forwarded-For $remote_addr;
+}
+```
+
+#### Apache2
+> TBC
+
+### Homeserver
+#### Synapse
+```yaml
+enable_registration: true
+registrations_require_3pid:
+  - email
+```
+
+## Configuration
+See the [Configuration](../configure.md) introduction doc on how to read the configuration keys.  
+An example of working configuration is available at the end of this section.
+### Enable/Disable
+`register.allowed`, taking a boolean, can be used to enable/disable registration if the attempt is not 3PID-based.  
+`false` is the default value to prevent open registration, as you must allow it on the homeserver side.
+
+### For invites
+`register.invite`, taking a boolean, controls if registration can be made using a 3PID which matches a pending 3PID invite.  
+`true` is the default value.
+
+### 3PID-specific
+At this time, only `email` is supported with 3PID specific configuration with this feature.
+
+#### Email
+**Base key**: `register.threepid.email`
+
+##### Domain whitelist/blacklist
+If you would like to control which domains are allowed to be used when registering with an email, the following sub-keys
+are available:
+- `domain.whitelist`
+- `domain.blacklist`
+
+The value format is an hybrid between glob patterns and postfix configuration files with the following syntax:
+- `*<domain>` will match the domain and any sub-domain(s)
+- `.<domain>` will only match sub-domain(s)
+- `<domain>` will only match the exact domain
+
+The following table illustrates pattern and matching status against example values:
+
+| Config value   | Matches `example.org` | Matches `sub.example.org` |
+|--------------- |-----------------------|---------------------------|
+| `*example.org` | Yes                   | Yes                       |
+| `.example.org` | No                    | Yes                       |
+| `example.org`  | Yes                   | No                        |
+
+### Example
+For the following example configuration:
+```yaml
+register:
+  policy:
+    threepid:
+      email:
+        domain:
+          whitelist:
+            - '*example.org'
+            - '.example.net'
+            - 'example.com'
+```
+- Users can register using 3PIDs of pending invites, being allowed by default.
+- Users can register using an email from `example.org` and any sub-domain, only sub-domains of `example.net` and `example.com` but not its sub-domains.
+- Otherwise, user registration will be denied.
+
+## Usage
+Nothing special is needed. Register using a regular Matrix client.

docs/getting-started.md

@@ -6,23 +6,34 @@
 5. [Validate](#validate)
 6. [Next steps](#next-steps)
 
-Following these quick start instructions, you will have a basic setup that can perform recursive/federated lookups and
-talk to the central Matrix.org Identity service.  
-This will be a good ground work for further integration with your existing Identity stores.
+Following these quick start instructions, you will have a basic setup that can perform recursive/federated lookups.  
+This will be a good ground work for further integration with features and your existing Identity stores.
+
+---
+
+If you would like a more fully integrated setup out of the box, the [matrix-docker-ansible-deploy](https://github.com/spantaleev/matrix-docker-ansible-deploy)
+project provides a turn-key full-stack solution, including LDAP and the various mxisd features enabled and ready.  
+We work closely with the project owner so the latest mxisd version is always supported.
+
+If you choose to use it, this Getting Started guide is not applicable - See the project documentation. You may then
+directly go to the [Next steps](#next-steps).
 
 ## Preparation
 You will need:
-- Homeserver
+- Working Homeserver, ideally with working federation
 - Reverse proxy with regular TLS/SSL certificate (Let's encrypt) for your mxisd domain
 
-As synapse requires an HTTPS connection when talking to an Identity service, a reverse proxy is required as mxisd does
+If you use synapse:
+- It requires an HTTPS connection when talking to an Identity service, **a reverse proxy is required** as mxisd does
   not support HTTPS listener at this time.
+- HTTPS is hardcoded when talking to the Identity server. If your Identity server URL in your client is `https://matrix.example.org/`,
+  then you need to ensure `https://matrix.example.org/_matrix/identity/api/v1/...` will reach mxisd if called from the synapse host.
+  In doubt, test with `curl` or similar. 
 
-For maximum integration, it is best to have your Homeserver and mxisd reachable via the same hostname.  
-You can also use a dedicated domain for mxisd, but will not have access to some features.
+For maximum integration, it is best to have your Homeserver and mxisd reachable via the same public hostname.
 
-Be aware of a [NAT/Reverse proxy gotcha](https://github.com/kamax-io/mxisd/wiki/Gotchas#nating) if you use the same
-hostname.
+Be aware of a [NAT/Reverse proxy gotcha](https://github.com/kamax-matrix/mxisd/wiki/Gotchas#nating) if you use the same
+host.
 
 The following Quick Start guide assumes you will host the Homeserver and mxisd under the same hostname.  
 If you would like a high-level view of the infrastructure and how each feature is integrated, see the
@@ -30,23 +41,25 @@ If you would like a high-level view of the infrastructure and how each feature i
 
 ## Install
 Install via:
-- [Debian package](install/debian.md)
 - [Docker image](install/docker.md)
+- [Debian package](install/debian.md)
+- [ArchLinux](install/archlinux.md)
+- [NixOS](install/nixos.md)
 - [Sources](build.md)
 
-See the [Latest release](https://github.com/kamax-io/mxisd/releases/latest) for links to each.
+See the [Latest release](https://github.com/kamax-matrix/mxisd/releases/latest) for links to each.
 
 ## Configure
-**NOTE**: please view the install instruction for your platform, as this step might be optional/handled for you.
+> **NOTE**: Please view the install instruction for your platform, as this step might be optional or already handled for you.
   
-Create/edit a minimal configuration (see installer doc for the location):
-```
-matrix.domain: 'MyMatrixDomain.org'
-key.path: '/path/to/signing.key.file'
-storage.provider.sqlite.database: '/path/to/mxisd.db'
-```  
-- `matrix.domain` should be set to your Homeserver domain
-- `key.path` will store the signing keys, which must be kept safe!
+> **NOTE**: Details about configuration syntax and format are described [here](configure.md)
+
+If you haven't created a configuration file yet, copy `mxisd.example.yaml` to where the configuration file is stored given
+your installation method and edit to your needs.
+
+The following items must be at least configured:
+- `matrix.domain` should be set to your Homeserver domain (`server_name` in synapse configuration)
+- `key.path` will store the signing keys, which must be kept safe! If the file does not exist, keys will be generated for you.
 - `storage.provider.sqlite.database` is the location of the SQLite Database file which will hold state (invites, etc.)
 
 If your HS/mxisd hostname is not the same as your Matrix domain, configure `server.name`.  
@@ -59,20 +72,20 @@ For an overview of a typical mxisd infrastructure, see the [dedicated document](
 In the `VirtualHost` section handling the domain with SSL, add the following and replace `0.0.0.0` by the internal
 hostname/IP pointing to mxisd.  
 **This line MUST be present before the one for the homeserver!**
-```
-ProxyPass /_matrix/identity/ http://0.0.0.0:8090/_matrix/identity/
+```apache
+ProxyPass /_matrix/identity http://0.0.0.0:8090/_matrix/identity
 ```
 
 Typical configuration would look like:
-```
+```apache
 <VirtualHost *:443>
-    ServerName example.org
+    ServerName matrix.example.org
     
-    ...
+    # ...
     
     ProxyPreserveHost on
-    ProxyPass /_matrix/identity/ http://localhost:8090/_matrix/identity/
-    ProxyPass /_matrix/ http://localhost:8008/_matrix/
+    ProxyPass /_matrix/identity http://localhost:8090/_matrix/identity
+    ProxyPass /_matrix http://localhost:8008/_matrix
 </VirtualHost>
 ```
 
@@ -80,19 +93,19 @@ Typical configuration would look like:
 In the `server` section handling the domain with SSL, add the following and replace `0.0.0.0` with the internal
 hostname/IP pointing to mxisd.
 **This line MUST be present before the one for the homeserver!**
-```
+```nginx
 location /_matrix/identity {
     proxy_pass http://0.0.0.0:8090/_matrix/identity;
 }
 ```
 
 Typical configuration would look like:
-```
+```nginx
 server {
     listen 443 ssl;
-    server_name example.org;
+    server_name matrix.example.org;
     
-    ...
+    # ...
     
     location /_matrix/identity {
         proxy_pass http://localhost:8090/_matrix/identity;
@@ -110,37 +123,35 @@ server {
 
 ### Synapse
 Add your mxisd domain into the `homeserver.yaml` at `trusted_third_party_id_servers` and restart synapse.  
-In a typical configuration, you would end up with something similair to:
-```
+In a typical configuration, you would end up with something similar to:
+```yaml
 trusted_third_party_id_servers:
-    - matrix.org
-    - vector.im
-    - example.org
+    - matrix.example.org
 ```
-It is recommended to remove `matrix.org` and `vector.im` so only your own Identity server is authoritative for your HS.
+It is **highly recommended** to remove `matrix.org` and `vector.im` (or any other default entry) from your configuration
+so only your own Identity server is authoritative for your HS.
 
 ## Validate
-Log in using your Matrix client and set `https://example.org` as your Identity server URL, replacing `example.org` by
-the relevant hostname which you configured in your reverse proxy.  
-Invite `mxisd-lookup-test@kamax.io` to a room, which should be turned into a Matrix invite to `@mxisd-lookup-test:kamax.io`.  
-**NOTE:** you might not see a Matrix suggestion for the e-mail address, which is normal. Still proceed with the invite.
+**NOTE:** In case your homeserver has no working federation, step 5 will not happen. If step 4 took place, consider
+your installation validated.
+
+1. Log in using your Matrix client and set `https://matrix.example.org` as your Identity server URL, replacing `matrix.example.org`
+by the relevant hostname which you configured in your reverse proxy.
+2. Create a new empty room. All further actions will take place in this room.
+3. Invite `mxisd-federation-test@kamax.io`
+4. The 3PID invite should be turned into a Matrix invite to `@mxisd-lookup-test:kamax.io`.
+5. The invited test user will join the room, send a congratulation message and leave.
+**NOTE:** You might not see a suggestion for the e-mail address, which is normal. Still proceed with the invite.
   
 If it worked, it means you are up and running and can enjoy mxisd in its basic mode! Congratulations!  
-If it did not work, [get in touch](#support) and we'll do our best to get you started.
-
-You can now integrate mxisd further with your infrastructure using the various [features](README.md) guides.
+If it did not work, read the basic [troubleshooting guide](troubleshooting.md), [get in touch](../README.md#support) and
+we'll do our best to get you started.
 
 ## Next steps
-Once your mxisd server is up and running, here are the next steps to further enhance and integrate your installation:
+Once your mxisd server is up and running, there are several ways you can enhance and integrate further with your
+infrastructure:
 
-Enable extra features:
-- [Federation](features/federation.md)
-- [Authenticate with synapse](features/authentication.md), profile auto-provisioning if you wish
-- [Directory search](features/directory-users.md)
-
-Use your Identity stores:
-- [LDAP / Samba / Active directory](backends/ldap.md)
-- [SQL Database](backends/sql.md)
-- [Website / Web service / Web app](backends/rest.md)
-- [Google Firebase](backends/firebase.md)
-- [Wordpress](backends/wordpress.md)
+- [Enable extra features](features/)
+- [Use your own Identity stores](stores/README.md)
+- [Hardening your mxisd installation](install/security.md)
+- [Learn about day-to-day operations](operations.md)

docs/install/archlinux.md

@@ -0,0 +1,3 @@
+# Arch Linux package
+An Arch Linux package in the AUR repos is maintained by [r3pek](https://matrix.to/#/@r3pek:r3pek.org), a community member.  
+See https://aur.archlinux.org/packages/mxisd/

docs/install/debian.md

@@ -1,28 +1,31 @@
 # Debian package
+## Requirements
+- Any distribution that supports Java 8
+
 ## Install
-1. Donwload the [latest release](https://github.com/kamax-io/mxisd/releases/latest)
+1. Download the [latest release](https://github.com/kamax-matrix/mxisd/releases/latest)
 2. Run:
-```
+```bash
 dpkg -i /path/to/downloaded/mxisd.deb
 ```
 ## Files
 | Location                            | Purpose                                      |
-|-----------------------------------|----------------------------------------------|
-| /etc/mxisd                        | Configuration directory                      |
-| /etc/mxisd/mxisd.yaml             | Main configuration file                      |
-| /etc/mxisd/signing.key            | Default location for mxisd signing keys      |
-| /etc/systemd/system/mxisd.service | Systemd configuration file for mxisd service |
-| /usr/lib/mxisd                    | Binairies                                    |
-| /var/lib/mxisd                    | Data                                         |
+|-------------------------------------|----------------------------------------------|
+| `/etc/mxisd`                        | Configuration directory                      |
+| `/etc/mxisd/mxisd.yaml`             | Main configuration file                      |
+| `/etc/systemd/system/mxisd.service` | Systemd configuration file for mxisd service |
+| `/usr/lib/mxisd`                    | Binaries                                     |
+| `/var/lib/mxisd`                    | Data                                         |
+| `/var/lib/mxisd/signing.key`        | Default location for mxisd signing keys      |
 
 ## Control
 Start mxisd using:
-```
+```bash
 sudo systemctl start mxisd
 ```
 
 Stop mxisd using:
-```
+```bash
 sudo systemctl stop mxisd
 ```
 
@@ -35,5 +38,5 @@ tail -n 99 -f /var/log/syslog | grep mxisd
 ```
 - use Systemd's journal:
 ```
-journalctl -f n 99 -u mxisd
+journalctl -f -n 99 -u mxisd
 ```

docs/install/docker.md

@@ -1,7 +1,7 @@
 # Docker
 ## Fetch
 Pull the latest stable image:
-```
+```bash
 docker pull kamax/mxisd
 ```
 
@@ -15,8 +15,11 @@ Use the following command after adapting to your needs:
 - The `MATRIX_DOMAIN` environment variable to yours
 - The volumes host paths
 
-```
+```bash
 docker run --rm -e MATRIX_DOMAIN=example.org -v /data/mxisd/etc:/etc/mxisd -v /data/mxisd/var:/var/mxisd -p 8090:8090 -t kamax/mxisd
 ```
 
 For more info, including the list of possible tags, see [the public repository](https://hub.docker.com/r/kamax/mxisd/)
+
+## Troubleshoot
+Please read the [Troubleshooting guide](../troubleshooting.md).

docs/install/nixos.md

@@ -0,0 +1,8 @@
+# NixOS package
+mxisd is available as a NixOS package in the official repos.
+
+It is maintained by [maximilian](https://matrix.to/#/@maximilian:transformierende-gesellschaft.org), a community member.  
+
+Related resources:
+- [NixOS](https://nixos.org/)
+- [The module definition](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/networking/mxisd.nix)

docs/install/security.md

@@ -0,0 +1,30 @@
+# Security hardening
+## Overview
+This document outlines the various operations you may want to perform to increase the security of your installation and
+avoid leak of credentials/key pairs
+
+## Configuration
+Your config file should have the following ownership:
+- Dedicated user for mxisd, used to run the software
+- Dedicated group for mxisd, used by other applications to access and read configuration files
+
+Your config file should have the following access:
+- Read and write for the mxisd user
+- Read for the mxisd group
+- Nothing for others
+
+This translates into `640` and be applied with `chmod 640 /path/to/config/file.yaml`.
+
+## Data
+The only sensible place is the key store where mxisd's signing keys are stored. You should therefore limit access to only
+the mxisd user, and deny access to anything else.
+
+Your key store should have the following access:
+- Read and write for the mxisd user
+- Nothing for the mxisd group
+- Nothing for others
+
+The identity store can either be a file or a directory, depending on your version. v1.4 and higher are using a directory,
+everything before is using a file.
+- If your version is directory-based, you will want to apply chmod `700` on it.
+- If your version is file-based, you will want to apply chmod `600` on it.

docs/install/source.md

@@ -0,0 +1,44 @@
+# Install from sources
+## Instructions
+Follow the [build instructions](../build.md) then:
+
+### Prepare files and directories:
+```bash
+# Create a dedicated user
+useradd -r mxisd
+
+# Create config directory
+mkdir -p /etc/mxisd
+
+# Create data directory and set ownership
+mkdir -p /var/lib/mxisd
+chown -R mxisd /var/lib/mxisd
+
+# Create bin directory, copy the jar and launch scriot to bin directory
+mkdir /usr/lib/mxisd
+cp ./build/libs/mxisd.jar /usr/lib/mxisd/
+cp ./src/script/mxisd /usr/lib/mxisd
+chown -R mxisd /usr/lib/mxisd
+chmod a+x /usr/lib/mxisd/mxisd
+
+# Create symlink for easy exec
+ln -s /usr/lib/mxisd/mxisd /usr/bin/mxisd
+```
+
+### Prepare config file
+Copy the configuration file you've created following the build instructions to `/etc/mxisd/mxisd.yaml`
+
+### Prepare Systemd
+1. Copy `src/systemd/mxisd.service` to `/etc/systemd/system/` and edit if needed
+2. Enable service for auto-startup
+```bash
+systemctl enable mxisd
+```
+
+### Run
+```bash
+systemctl start mxisd
+```
+
+## Debug
+mxisd logs to stdout, which is normally sent to `/var/log/syslog` or `/var/log/messages`.

docs/operations.md

@@ -0,0 +1,21 @@
+# Operations Guide
+- [Overview](#overview)
+- [Maintenance](#maintenance)
+- [Backuo](#backup)
+
+## Overview
+This document gives various information for the day-to-day management and operations of mxisd.
+
+## Maintenance
+mxisd does not require any maintenance task to run at optimal performance.
+
+## Backup
+### Run
+mxisd requires all file in its configuration and data directory to be backed up.  
+They are usually located at:
+- `/etc/mxisd`
+- `/var/lib/mxisd`
+
+### Restore
+Reinstall mxisd, restore the two folders above in the appropriate location (depending on your install method) and you
+will be good to go. Simply start mxisd to restore functionality.

docs/sessions/3pid-views.md

@@ -1,82 +0,0 @@
-# Web pages for the 3PID session processes
-You can customize the various pages used during a 3PID validation using [Thymeleaf templates](http://www.thymeleaf.org/).
-
-## Configuration
-```
-view:
-  session:
-    local:
-      onTokenSubmit:
-        success: '/path/to/session/local/tokenSubmitSuccess-page.html'
-        failure: '/path/to/session/local/tokenSubmitFailure-page.html'
-    localRemote:
-      onTokenSubmit:
-        success: '/path/to/session/localRemote/tokenSubmitSuccess-page.html'
-        failure: '/path/to/session/local/tokenSubmitFailure-page.html'
-    remote:
-      onRequest:
-        success: '/path/to/session/remote/requestSuccess-page.html'
-        failure: '/path/to/session/remote/requestFailure-page.html'
-      onCheck:
-        success: '/path/to/session/remote/checkSuccess-page.html'
-        failure: '/path/to/session/remote/checkFailure-page.html'
-```
-3PID session are divided into three config sections:
-- `local` for local-only 3PID sessions
-- `localRemote` for local 3PID sessions that can also be turned into remote sessions, if the user so desires
-- `remote` for remote-only 3PID sessions
-
-Each section contains a sub-key per support event. Finally, a `success` and `failure` key is available depending on the
-outcome of the request.
-
-## Local
-### onTokenSubmit
-This is triggered when a user submit a validation token for a 3PID session. It is typically visited when clicking the
-link in a validation email.
-
-The template should typically inform the user that the validation was successful and to go back in their Matrix client
-to finish the validation process.
-
-#### Placeholders
-No object/placeholder are currently available.
-
-## Local & Remote
-### onTokenSubmit
-This is triggered when a user submit a validation token for a 3PID session. It is typically visited when clicking the
-link in a validation email.
-
-The template should typically inform the user that their 3PID address will not yet be publicly/globally usable. In case
-they want to make it, they should start a Remote 3PID session with a given link or that they can go back to their Matrix
-client if they do not wish to proceed any further.
-
-#### Placeholders
-##### Success
-`<a th:href="${remoteSessionLink}">text</a>` can be used to display the link to start a Remote 3PID session.
-
-##### Failure
-No object/placeholder are currently available.
-
-## Remote
-### onRequest
-This is triggered when a user starts a Remote 3PID session, usually from a link produced in the `local.onTokenSubmit`
-view or in a remote-only 3PID notification.
-
-The template should typically inform the user that the remote creation was successful, followed the instructions sent by
-the remote Identity server and, once that is done, click a link to validate the session.
-
-#### Placeholders
-##### Success
-`<a th:href="${checkLink}">text</a>` can be used to display the link to validate the Remote 3PID session.
-
-##### Failure
-No object/placeholder are currently available.
-
-### onCheck
-This is triggered when a user attempts to inform the Identity server that the Remote 3PID session has been validated
-with the remote Identity server.
-
-The template should typically inform the user that the validation was successful and to go back in their Matrix client
-to finish the validation process.
-
-#### Placeholders
-No object/placeholder are currently available.

docs/sessions/3pid.md

@@ -1,338 +0,0 @@
-# 3PID Sessions
-- [Overview](#overview)
-- [Purpose](#purpose)
-- [Federation](#federation)
-  - [3PID scope](#3pid-scope)
-  - [Session scope](#session-scope)
-- [Notifications](#notifications)
-  - [Email](#email)
-  - [Phone numbers](#msisdn-phone-numbers)
-- [Usage](#usage)
-  - [Configuration](#configuration)
-  - [Web views](#web-views)
-  - [Scenarios](#scenarios)
-    - [Default](#default)
-    - [Local sessions only](#local-sessions-only)
-    - [Remote sessions only](#remote-sessions-only)
-    - [Sessions disabled](#sessions-disabled)
-
-## Overview
-When adding an email, a phone number or any other kind of 3PID (Third-Party Identifier) in a Matrix client, 
-the identity server is called to validate the 3PID.
-
-Once this 3PID is validated, the Homeserver will publish the user Matrix ID on the Identity Server and
-add this 3PID to the Matrix account which initiated the request.
-
-## Purpose
-This serves two purposes:
-- Add the 3PID as an administrative/login info for the Homeserver directly
-- Publish, or *Bind*, the 3PID so it can be queried from Homeservers and clients when inviting someone in a room
-by a 3PID, allowing it to be resolved to a Matrix ID.
-
-## Federation
-Federation is based on the principle that one can get a domain name and serve services and information within that
-domain namespace in a way which can be discovered following a specific protocol or specification.
-
-In the Matrix eco-system, some 3PID can be federated (e.g. emails) while some others cannot (phone numbers).
-Also, Matrix users might add 3PIDs that would not point to the Identity server that actually holds the 3PID binding.  
-
-Example: a user from Homeserver `example.org` adds an email `john@gmail.com`.  
-If a federated lookup was performed, Identity servers would try to find the 3PID bind at the `gmail.com` server, and
-not `example.org`.
-
-To allow global publishing of 3PID bindings to be found anywhere within the current protocol specification, one would
-perform a *Remote session* and *Remote bind*, effectively starting a new 3PID session with another Identity server on
-behalf of the user.  
-To ensure lookup works consistency within the current Matrix network, the central Matrix.org Identity Server should be
-used to store *remote* sessions and binds.
-
-On the flip side, at the time of writing, the Matrix specification and the central Matrix.org servers do not allow to
-remote a 3PID bind. This means that once a 3PID is published (email, phone number, etc.), it cannot be easily remove
-and would require contacting the Matrix.org administrators for each bind individually.  
-This poses a privacy, control and security concern, especially for groups/corporations that want to keep a tight control
-on where such identifiers can be made publicly visible.
-
-To ensure full control, validation management rely on two concepts:
-- The scope of 3PID being validated
-- The scope of 3PID sessions that should be possible/offered
-
-### 3PID scope
-3PID can either be scoped as local or remote.
-
-Local means that they can looked up using federation and that such federation call would end up on the local
-Identity Server.  
-Remote means that they cannot be lookup using federation or that a federation call would not end up on the local
-Identity Server.
-
-Email addresses can either be local or remote 3PID, depending on the domain. If the address is one from the configured
-domain in the Identity server, it will be scoped as local. If it is from another domain, it will be as remote.
-
-Phone number can only be scoped as remote, since there is currently no way to perform DNS queries that would lead back
-to the Identity server who validated the phone number.
-
-### Session scope
-Sessions can be scoped as:
-- Local only - validate 3PIDs directly, do not allow the creation of 3PID sessions on a remote Identity server.
-- Local and Remote - validate 3PIDs directly, offer users to option to also validate and bind 3PID on another server.
-- Remote only - validate and bind 3PIDs on another server, no validation or bind done locally.
-
----
-
-**IMPORTANT NOTE:** mxisd does not store bindings directly. While a user can see its email, phone number or any other
-3PID in its settings/profile, it does **NOT** mean it is published anywhere and can be used to invite/search the user.
-Identity backends (LDAP, REST, SQL) are the ones holding such data.  
-If you still want added arbitrary 3PIDs to be discoverable on your local server, you will need to link mxisd to your
-synapse DB to make it an Identity backend.
-
-See the [Scenarios](#scenarios) for more info on how and why.
-
-## Notifications
-3PIDs are validated by sending a pre-formatted message containing a token to that 3PID address, which must be given to the
-Identity server that received the request. This is usually done by means of a URL to visit for email or a short number
-received by SMS for phone numbers.
-
-mxisd use two components for this:
-- Generator which produces the message to be sent with the necessary information the user needs to validate their session.
-- Connector which actually send the notification (e.g. SMTP for email).
-
-Built-in generators and connectors for supported 3PID types:
-
-### Email
-Generators:
-- [Template](../threepids/notifications/template-generator.md)
-
-Connectors:
-- [SMTP](../threepids/medium/email/smtp-connector.md)
-
-#### MSISDN (Phone numbers)
-Generators:
-- [Template](../threepids/notifications/template-generator.md)
-
-Connectors:
- - [Twilio](../threepids/medium/msisdn/twilio-connector.md) with SMS
-
-## Usage
-### Configuration
-The following example of configuration (incomplete extract) shows which items are relevant for 3PID sessions.
-
-**IMPORTANT:** Most configuration items shown have default values and should not be included in your own configuration
-file unless you want to specifically overwrite them.  
-Please refer to the full example config file to see which keys are mandatory and to be included in your configuration.
-```
-matrix:
-  identity:
-    servers:
-      configExample: # Not to be included in config! Already present in default config!
-        - 'https://example.org'
-
-
-threepid:
-  medium:
-    email:
-      connector: 'example1' # Not to be included in config! Already present in default config! 
-      generator: 'example2' # Not to be included in config! Already present in default config!
-      connectors:
-        example1:
-      generators:
-        example1:
-          key: "value"
-        example2:
-          key: "value"
-
-session:
-  policy:
-    validation:
-      enabled: true
-      forLocal:
-        enabled: true
-        toLocal: true
-        toRemote:
-          enabled: true
-          server: 'configExample'  # Not to be included in config! Already present in default config!
-      forRemote:
-        enabled: true
-        toLocal: false
-        toRemote:
-          enabled: true
-          server: 'configExample'  # Not to be included in config! Already present in default config!
-```
-
-`matrix.identity.servers` is the namespace to configure arbitrary list of Identity servers with a label as parent key.  
-In the above example, the list with label `configExample` contains a single server entry pointing to `https://example.org`.  
-
-**NOTE:** The server list is set to `root` by default and should typically NOT be included in your config.  
-
-Identity server entry can be of two format:
-- URL, bypassing any kind of domain and port discovery
-- Domain name as `string`, allowing federated discovery to take place.
-
-The label can be used in other places of the configuration, allowing you to only declare Identity servers once.
-
----
-
-`threepid.medium.<3PID>` is the namespace to configure 3PID specific items, not directly tied to any other component of
-mxisd.  
-In the above example, only `email` is defined as 3PID type.
-
-Each 3PID namespace comes with 4 configuration key allowing you to configure generators and connectors for notifications:
-- `connectors` is a configuration namespace to be used for any connector configuration. Child keys represent the unique
-ID for each connector.
-- `generators` is a configuration namespace to be used for any generator configuration. Child keys represent the unique
-ID for each generator.
-- `connector` is given the ID of the connector to be used at runtime.
-- `generator` is given the ID of the generator to be used at runtime.
-
-In the above example, emails notifications are generated by the `example2` module and sent with the `example1` module.  
-By default, `template` is used as generator and `smtp` as connector.
-
----
-
-`session.policy.validation` is the core configuration to control what users configured to use your Identity server
-are allowed to do in terms of 3PID sessions.
-
-The policy is divided contains a global on/off switch for 3PID sessions using `.enabled`  
-It is also divided into two sections: `forLocal` and `forRemote` which refers to the 3PID scopes.  
-
-Each scope is divided into three parts:
-- global on/off switch for 3PID sessions using `.enabled`
-- `toLocal` allowing or not local 3PID session validations
-- `toRemote` allowing or not remote 3PID session validations and to which server such sessions should be sent. 
-`.server` takes a Matrix Identity server list label. Only the first server in the list is currently used.
-
-If both `toLocal` and `toRemote` are enabled, the user will be offered to initiate a remote session once their 3PID
-locally validated.
-
-### Web views
-Once a user click on a validation link, it is taken to the Identity Server validation page where the token is submited.  
-If the session or token is invalid, an error page is displayed.  
-Workflow pages are also available for the remote 3PID session process.
-
-See [the dedicated document](3pid-views.md)
-on how to configure/customize/brand those pages to your liking.
-
-### Scenarios
-It is important to keep in mind that mxisd does not create bindings, irrelevant if a user added a 3PID to their profile.  
-Instead, when queried for bindings, mxisd will query Identity backends which are responsible to store this kind of information.
-
-This has the side effect that any 3PID added to a user profile which is NOT within a configured and enabled Identity backend
-will simply not be usable for search or invites, **even on the same Homeserver!**  
-mxisd does not store binds on purpose, as one of its primary goal is to ensure maximum compatibility with federation
-and the rest of the Matrix ecosystem is preserved.
-
-Nonetheless, because mxisd also aims at offering support for tight control over identity data, it is possible to have
-such 3PID bindings available for search and invite queries on the local Homeserver by using the `SQL` backend and
-configuring it to use the synapse database. Support for `SQLite` and `PostgreSQL` is available.
-
-See the [Local sessions only](#local-sessions-only) use case for more information on how to configure.
-
-#### Default
-By default, mxisd allows the following:
-
-|  | Local Session | Remote Session |
-|----------------|-------|--------|
-| **Local 3PID** | Yes | Yes, offered |
-| **Remote 3PID** | No, Remote forced | Yes |
-
-This is usually what people expect and will feel natural to users and does not involve further integration.
-
-This allows to stay in control for e-mail addresses which domain matches your Matrix environment, still making them
-discoverable with federation but not recorded in a 3rd party Identity server which is not under your control.  
-Users still get the possibility to publish globally their address if needed.
-
-Other e-mail addresses and phone number will be redirected to remote sessions to ensure full compatibility with the Matrix
-ecosystem and other federated servers.
-
-#### Local sessions only
-**NOTE:** This does not affect 3PID lookups (queries to find Matrix IDs) which will remain public due to limitation
-in the Matrix protocol.
-
-This configuration ensures maximum confidentiality and privacy.
-Typical use cases:
-- Private Homeserver, not federated
-- Internal Homeserver without direct Internet access
-- Custom product based on Matrix which does not federate
-
-No 3PID will be sent to a remote Identity server and all validation will be performed locally.  
-On the flip side, people with *Remote* 3PID scopes will not be found from other servers.
-
-Use the following values:
-```
-session:
-  policy:
-    validation:
-      enabled: true
-      forLocal:
-        enabled: true
-        toLocal: true
-        toRemote:
-          enabled: false
-      forRemote:
-        enabled: true
-        toLocal: true
-        toRemote:
-          enabled: false
-```
-
-**IMPORTANT**: When using local-only mode, you will also need to link mxisd to synapse if you want user searches and invites to work.
-To do so, add/edit the following configuration keys:
-```
-synapseSql:
-  enabled: true
-  type: 'SET TO PROPER VALUE'
-  connection: 'SET TO PROPER VALUE'
-```
-- `synapseSql.enabled` set to `true` to activate the SQL backend.
-- `synapseSql.type` can be set to `sqlite` or `postgresql`, depending on your synapse setup.
-- `synapseSql.connection` use a JDBC format which is appened after the `jdbc:type:` connection URI.
-Example values for each type:
-  - `sqlite`: `/path/to/homeserver.db`
-  - `postgresql`: `//localhost/database?user=synapse&password=synapse`
-
-#### Remote sessions only
-This configuration ensures all 3PID are made public for maximum compatibility and reach within the Matrix ecosystem, at
-the cost of confidentiality and privacy.  
-
-Typical use cases:
-- Public Homeserver
-- Homeserver with registration enabled
-
-Use the following values:
-```
-session:
-  policy:
-    validation:
-      enabled: true
-      forLocal:
-        enabled: true
-        toLocal: false
-        toRemote:
-          enabled: true
-      forRemote:
-        enabled: true
-        toLocal: false
-        toRemote:
-          enabled: true
-```
-
-#### Sessions disabled
-This configuration would disable 3PID session altogether, preventing users from adding emails and/or phone numbers to
-their profiles.  
-This would be used if mxisd is also performing authentication for the Homeserver, typically with synapse and the
-[REST Auth module](https://github.com/kamax-io/matrix-synapse-rest-auth).
-
-While this feature is not yet ready in the REST auth module, you would use this configuration mode to auto-populate 3PID
-at user login and prevent any further add.
-
-**This mode comes with several important restrictions:**
-- This does not prevent users from removing 3PID from their profile. They would be unable to add them back!
-- This prevents users from initiating remote session to make their 3PID binds globally visible
-
-It is therefore recommended to not fully disable sessions but instead restrict specific set of 3PID and Session scopes.
-
-Use the following values to enable this mode:
-```
-session:
-  policy:
-    validation:
-      enabled: false
-```

docs/stores/README.md

@@ -0,0 +1,8 @@
+# Identity Stores
+- [Synapse](synapse.md) - Turn your SynapseDB into a self-contained Identity store
+- [LDAP-based](ldap.md) - Any LDAP-based product like Active Directory, Samba, NetIQ, OpenLDAP
+- [SQL Databases](sql.md) - Most common databases like MariaDB, MySQL, PostgreSQL, SQLite
+- [Website / Web service / Web app](rest.md) - Arbitrary REST endpoints
+- [Executables](exec.md) - Run arbitrary executables with configurable stdin, arguments, environment and stdout
+- [Wordpress](wordpress.md) - Connect your Wordpress-powered website DB
+- [Google Firebase](firebase.md) - Use your Firebase users (with experimental SSO support!)

docs/stores/exec.md

@@ -0,0 +1,506 @@
+# Exec Identity Store
+- [Features](#features)
+- [Overview](#overview)
+- [Configuration](#configuration)
+  - [Global](#global)
+    - [Tokens](#tokens)
+  - [Executable](#executable)
+    - [Input](#input)
+    - [Output](#output)
+  - [Examples](#examples)
+  - [Per-Feature](#per-feature)
+- [Authentication](#authentication)
+  - [Tokens](#tokens-1)
+  - [Input](#input-1)
+  - [Output](#output-1)
+- [Directory](#directory)
+  - [Tokens](#tokens-2)
+  - [Input](#input-2)
+  - [Output](#output-2)
+- [Identity](#identity)
+  - [Single Lookup](#single-lookup)
+    - [Tokens](#tokens-3)
+    - [Input](#input-3)
+    - [Output](#output-3)
+  - [Bulk Lookup](#bulk-lookup)
+    - [Tokens](#tokens-4)
+    - [Input](#input-4)
+    - [Output](#output-4)
+- [Profile](#profile)
+  - [Tokens](#tokens-5)
+  - [Input](#input-5)
+  - [Output](#output-5)
+  
+---
+
+## Features
+|                       Name                      | Supported |
+|-------------------------------------------------|-----------|
+| [Authentication](../features/authentication.md) | Yes       |
+| [Directory](../features/directory.md)           | Yes       |
+| [Identity](../features/identity.md)             | Yes       |
+| [Profile](../features/profile.md)               | Yes       |
+
+This Identity Store lets you run arbitrary commands to handle the various requests in each support feature.  
+It is the most versatile Identity store of mxisd, allowing you to connect any kind of logic with any executable/script.
+
+## Overview
+Each request can be mapping to a fully customizable command configuration.  
+The various parameters can be provided via any combination of:
+- [Standard Input](https://en.wikipedia.org/wiki/Standard_streams#Standard_input_(stdin))
+- [Command-line arguments](https://en.wikipedia.org/wiki/Command-line_interface#Arguments)
+- [Environment variables](https://en.wikipedia.org/wiki/Environment_variable)
+
+Each of those supports a set of customizable token which will be replaced prior to running the command, allowing to
+provide the input values in any number of ways.
+
+Success and data will be provided via any combination of:
+- [Exit status](https://en.wikipedia.org/wiki/Exit_status)
+- [Standard Output](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout))
+
+Each of those supports a set of configuration item to decide how to process the value and/or in which format.
+
+All values, inputs and outputs are UTF-8 encoded.
+
+## Configuration
+Each feature comes with a set of possible lookup/action which is mapped to a generic configuration item block.  
+We will use the term `Executable` for each lookup/action and `Processor` for each configuration block.
+
+### Global
+```yaml
+exec:
+  enabled: <boolean>
+```
+Enable/disable the Identity store at a global/default level. Each feature can still be individually enabled/disabled.
+
+#### Tokens
+The following options allow to globally set tokens for value replacement across all features and processors config.  
+Not all features use all tokens, and each feature might also have its own specific tokens. See each feature documentation.
+  
+They can be set within the following scope:
+
+```yaml
+exec:
+  token:
+    <token>: '<value>'
+```
+
+---
+
+The following tokens and default values are available:
+```yaml
+localpart: '{localpart}'
+```
+Localpart of Matrix User IDs
+
+```yaml
+domain: '{domain}'
+```
+Domain of Matrix User IDs
+
+```yaml
+mxid: '{mxid}'
+```
+Full representation of Matrix User IDs
+
+```yaml
+medium: '{medium}'
+```
+Medium of 3PIDs
+
+```yaml
+address: '{address}'
+```
+Address of 3PIDs
+
+```yaml
+type: '{type}'
+```
+Type of query
+
+```yaml
+query: '{query}'
+```
+Query value
+
+### Executable
+*Executable*s have the following options:
+```yaml
+command: '/path/to/executableOrScript'
+
+```
+Set the executable (relative or absolute) path to be executed. If no command is given, the action will return a "neutral"
+result if possible or be skipped altogether.
+
+---
+
+Command line arguments can be given via a list via both YAML formats:
+```yaml
+args:
+ - '-t'
+ - '{token}'
+ - '-v'
+ - 'value'
+```
+or
+```yaml
+args: ['-t', '{token}', '-v', 'value]
+```
+Each argument will be processed for token replacement.
+
+---
+
+Environment variables can be given as key/value pairs:
+```yaml
+env:
+  ENV_VAR_1: 'value'
+  ENV_VAR_2: '{token}'
+``` 
+Each variable value will be processed for token replacement.
+
+#### Input
+Standard input can be configured in the namespaces `input` with:
+- `type`: The format to use
+- `template`: The full or partial template with tokens to be used when generating the input
+
+Not all features and *Executable*s allow for a template to be provided.  
+Templates for listed-based input are not supported at this time.  
+Default templates may be provided per *Executable*.
+
+The following types are available:
+- `json`: Use JSON format, shared with the [REST Identity Store](rest.md)
+- `plain`: Use a custom multi-lines, optionally tab-separated input
+
+#### Output
+Standard output can be configured in the namespaces `output` with:
+- `type`: The format to use
+- `template`: The full or partial template with tokens to be used when processing the output
+
+Not all features and *Executable*s allow for a template to be provided.  
+Templates for listed-based output are not supported at this time.  
+Default templates may be provided per *Executable*.
+
+The following types are available:
+- `json`: Use JSON format, shared with the [REST Identity Store](rest.md)
+- `plain`: Use a custom multi-lines, optionally tab-separated output
+
+### Examples
+#### Basic
+```yaml
+exec:
+  auth:
+    enabled: true
+    command: '/opt/mxisd-exec/auth.sh'
+    args: ['{localpart}']
+    input:
+      type: 'plain'
+      template: '{password}'
+  env:
+    DOMAIN: '{domain}'
+```
+With Authentication enabled, run `/opt/mxisd-exec/auth.sh` when validating credentials, providing:
+- A single command-line argument to provide the `localpart` as username 
+- A plain text string with the password token for standard input, which will be replaced by the password to check
+- A single environment variable `DOMAIN` containing Matrix ID domain, if given
+
+The command will use the default values for:
+- Success exit status of `0`
+- Failure exit status of `1`
+- Any other exit status considered as error
+- Standard output will not be processed
+
+#### Advanced
+Given the fictional `placeholder` feature:
+```yaml
+exec:
+  enabled: true
+  token:
+    mxid: '{matrixId}'
+  auth:
+    token:
+      localpart: '{username}'
+    command: '/path/to/executable'
+    args:
+      - '-u'
+      - '{username}'
+    env:
+      MATRIX_DOMAIN: '{domain}'
+      MATRIX_USER_ID: '{matrixId}'
+    output:
+      type: 'json'
+    exit:
+      success:
+        - 0
+        - 128
+      failure:
+        - 1
+        - 129
+```
+With:
+- The Identity store enabled for all features
+- A global specific token `{matrixId}` for Matrix User IDs, replacing the default `{mxid}`
+
+Running `/path/to/executable` providing:
+- A custom token for localpart, `{username}`, used as a 2nd command-line argument
+- An extracted Matrix User ID `localpart` provided as the second command line argument, the first one being `-u` 
+- A password, the extracted Matrix `domain` and the full User ID as arbitrary environment variables, respectively
+  `PASSWORD`, `MATRIX_DOMAIN` and `MATRIX_USER_ID`
+
+After execution:
+- Process stdout as [JSON](https://en.wikipedia.org/wiki/JSON)
+- Consider exit status `0` and `128` as success and try to process the stdout for data
+- Consider exit status `1` and `129` as failure and try to process the stdout for error code and message
+
+### Per Feature
+See each dedicated [Feature](#features) section.
+
+## Authentication
+The Authentication feature can be enabled/disabled using:
+```yaml
+exec:
+  auth:
+    enabled: <true/false>
+```
+
+---
+
+This feature provides a single *Executable* under the namespace:
+```yaml
+exec:
+  auth:
+  ...
+```
+
+### Tokens
+The following tokens/default values are specific to this feature:
+```yaml
+password: '{password}'
+```
+The provided password
+
+### Input
+Supported input types and default templates:
+
+#### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+#### Plain (`plain`)
+Default template:
+```
+{localpart}
+{domain}
+{mxid}
+{password}
+```
+
+### Output
+Supported output types and default templates:
+
+#### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+#### Plain (`plain`)
+**NOTE:** This has limited support. Use the JSON type for full support.
+
+Default template:
+```
+[success status, true or 1 are interpreted as success]
+[display name of the user]
+```
+
+## Directory
+The Directory feature can be enabled/disabled using:
+```yaml
+exec:
+  directory:
+    enabled: <true/false>
+```
+
+---
+
+Two search types configuration namespace are available, using the same input/output formats and templates:
+
+By name:
+```yaml
+exec:
+  directory:
+    search:
+      byName:
+        ...
+```
+By 3PID:
+```yaml
+exec:
+  directory:
+    search:
+      byThreepid:
+        ...
+```
+
+#### Tokens
+No specific tokens are available.
+
+#### Input
+Supported input types and default templates:
+
+##### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+##### Plain (`plain`)
+Default template:
+```
+[type of search, following the REST Identity store format]
+[query string]
+```
+
+#### Output
+Supported output types and default templates:
+
+##### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+##### Plain (`plain`)
+**Not supported at this time.** Use the JSON type.
+
+## Identity
+The Identity feature can be enabled/disabled using:
+```yaml
+exec.identity.enabled: <true/false>
+```
+
+### Single lookup
+Configuration namespace:
+```yaml
+exec.identity.lookup.single:
+  ...
+```
+
+#### Tokens
+No specific tokens are available.
+
+#### Input
+Supported input types and default templates:
+
+##### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+##### Plain (`plain`)
+Default template:
+```
+{medium}
+{address}
+```
+
+#### Output
+Supported output types and default templates:
+
+##### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+##### Plain (`plain`)
+Default template:
+```
+[User ID type, as documented in the REST Identity Store]
+[User ID value]
+```
+
+The User ID type will default to `localpart` if:
+- Only one line is returned
+- The first line is empty
+
+### Bulk lookup
+Configuration namespace:
+```yaml
+exec:
+  identity:
+    lookup:
+      bulk:
+        ...
+```
+
+#### Tokens
+No specific tokens are available.
+
+#### Input
+Supported input types and default templates:
+
+##### JSON (`json`)
+**NOTE:** Custom Templates are not supported.
+
+Same as the [REST Identity Store](rest.md).
+
+##### Plain (`plain`)
+**Not supported at this time.** Use the JSON type.
+
+#### Output
+Supported output types and default templates:
+
+##### JSON (`json`)
+**NOTE:** Custom Templates are not supported.
+
+Same as the [REST Identity Store](rest.md).
+
+##### Plain (`plain`)
+**Not supported at this time.** Use the JSON type.
+
+## Profile
+The Profile feature can be enabled/disabled using:
+```yaml
+exec:
+  profile:
+    enabled: <true/false>
+```
+
+---
+
+The following *Executable*s namespace are available, share the same input/output formats and templates:
+
+Get Display name:
+```yaml
+exec:
+  profile:
+    displayName:
+      ...
+```
+
+Get 3PIDs:
+```yaml
+exec:
+  profile:
+    threePid:
+      ...
+```
+
+Get Roles:
+```yaml
+exec:
+  profile:
+    role:
+      ...
+```
+
+
+### Tokens
+No specific tokens are available.
+
+### Input
+Supported input types and default templates:
+
+#### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+#### Plain (`plain`)
+Default template:
+```
+{localpart}
+{domain}
+{mxid}
+```
+### Output
+Supported output types and default templates:
+
+#### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+#### Plain (`plain`)
+**Not supported at this time.** Use the JSON type.

docs/stores/firebase.md

@@ -0,0 +1,59 @@
+# Google Firebase Identity store
+https://firebase.google.com/
+
+## Features
+|                       Name                      | Supported |
+|-------------------------------------------------|-----------|
+| [Authentication](../features/authentication.md) | Yes       |
+| [Directory](../features/directory.md)           | No        |
+| [Identity](../features/identity.md)             | Yes       |
+| [Profile](../features/profile.md)               | No        |
+
+## Requirements
+This backend requires a suitable Matrix client capable of performing Firebase authentication and passing the following
+information:
+- Firebase User ID as Matrix username
+- Firebase token as Matrix password
+
+If your client is Riot, you will need a custom version.
+
+## Configuration
+```yaml
+firebase:
+  enabled: <boolean>
+```
+Enable/disable this identity store.
+
+Example:
+```yaml
+firebase:
+  enabled: <boolean>
+```
+
+---
+
+```yaml
+firebase:
+  credentials: <string>
+```
+Path to the credentials file provided by Google Firebase to use with an external app.
+
+Example:
+```yaml
+firebase:
+  credentials: '/path/to/firebase/credentials.json'
+```
+
+---
+
+```yaml
+firebase:
+  database: <string>
+```
+URL to your Firebase database.
+
+Example:
+```yaml
+firebase:
+  database: 'https://my-project.firebaseio.com/'
+```

docs/stores/ldap.md

@@ -0,0 +1,141 @@
+# LDAP Identity store
+## Supported products:
+- Samba
+- Active Directory
+- OpenLDAP
+- NetIQ eDirectory
+
+For NetIQ, replace all the `ldap` prefix in the configuration by `netiq`.
+
+## Features
+|                       Name                      | Supported |
+|-------------------------------------------------|-----------|
+| [Authentication](../features/authentication.md) | Yes       |
+| [Directory](../features/directory.md)           | Yes       |
+| [Identity](../features/identity.md)             | Yes       |
+| [Profile](../features/profile.md)               | Yes       |
+
+## Getting started
+### Base
+To use your LDAP backend, add the bare minimum configuration in mxisd config file:
+```yaml
+ldap:
+  enabled: true
+  connection:
+    host: 'ldapHostnameOrIp'
+    port: 389
+    bindDn: 'CN=My Mxisd User,OU=Users,DC=example,DC=org'
+    bindPassword: 'TheUserPassword'
+    baseDNs:
+      - 'OU=Users,DC=example,DC=org'
+```
+These are standard LDAP connection configuration. mxisd will try to connect on port default port 389 without encryption.
+
+If you would like to use several Base DNs, simply add more entries under `baseDNs`.
+
+### TLS/SSL connection
+If you would like to use a TLS/SSL connection, use the following configuration options (STARTLS not supported):
+```yaml
+ldap:
+  connection:
+    tls: true
+    port: 12345
+```
+
+### Filter results
+You can also set a default global filter on any LDAP queries:
+```yaml
+ldap:
+  filter: '(memberOf=CN=My Matrix Users,OU=Groups,DC=example,DC=org)'
+```
+This example would only return users part of the group called `My Matrix Users`.
+This can be overwritten or append in each specific flow describe below.
+
+For supported syntax, see the [LDAP library documentation](http://directory.apache.org/api/user-guide/2.3-searching.html#filter).
+
+### Attribute mapping
+LDAP features are based on mapping LDAP attributes to Matrix concepts, like a Matrix ID, its localpart, the user display
+name, their email(s) and/or phone number(s).
+     
+Default attributes are well suited for Active Directory/Samba. In case you are using a native LDAP backend, you will
+most certainly configure those mappings.
+
+#### User ID
+`ldap.attribute.uid.type`: How to process the User ID (UID) attribute:
+- `uid` will consider the value as the [Localpart](https://matrix.org/docs/spec/intro.html#user-identifiers)
+- `mxid` will consider the value as a complete [Matrix ID](https://matrix.org/docs/spec/intro.html#user-identifiers)
+
+`ldap.attribute.uid.value`: Attribute to use to set the User ID value.
+
+The following example would set the `sAMAccountName` attribute as a Matrix User ID localpart:
+```yaml
+ldap:
+  attribute:
+    uid:
+      type: 'uid'
+      value: 'sAMAccountName'
+```
+
+#### Display name
+Use `ldap.attribute.name`.
+
+The following example would set the display name to the value of the `cn` attribute:
+```yaml
+ldap:
+  attribute:
+    name: 'cn'
+```
+
+#### 3PIDs
+You can also change the attribute lists for 3PID, like email or phone numbers.
+
+The following example would overwrite the [default list of attributes](../../src/main/java/io/kamax/mxisd/config/ldap/LdapConfig.java#L64)
+for emails and phone number:
+```yaml
+ldap:
+  attribute:
+    threepid:
+      email:
+        - 'mail'
+        - 'otherMailAttribute'
+      msisdn:
+        - 'phone'
+        - 'otherPhoneAttribute'
+```
+
+## Features
+### Identity
+Identity features (related to 3PID invites or searches) are enabled and configured using default values and no specific
+configuration item is needed to get started.
+
+#### Configuration
+- `ldap.identity.filter`: Specific user filter applied during identity search. Global filter is used if blank/not set.
+- `ldap.identity.medium`: Namespace to overwrite generated queries from the list of attributes for each 3PID medium.
+
+### Authentication
+After you have configured and enabled the [feature itself](../features/authentication.md), no further configuration is
+needed with this identity store to make it work.
+
+Profile auto-fill is enabled by default. It will use the `ldap.attribute.name` and `ldap.attribute.threepid` configuration
+options to get a lit of attributes to be used to build the user profile to pass on to synapse during authentication.
+
+#### Configuration
+- `ldap.auth.filter`: Specific user filter applied during username search. Global filter is used if blank/not set.
+
+### Directory
+After you have configured and enabled the [feature itself](../features/directory.md), no further configuration is
+needed with this identity store to make it work.
+
+#### Configuration
+To set a specific filter applied during directory search, use `ldap.directory.filter`
+
+If you would like to use extra attributes in search that are not 3PIDs, like nicknames, group names, employee number:
+```yaml
+ldap:
+  directory:
+    attribute:
+      other:
+        - 'myNicknameAttribute'
+        - 'memberOf'
+        - 'employeeNumberAttribute'
+```

docs/stores/rest.md

@@ -0,0 +1,277 @@
+# REST Identity store
+The REST backend allows you to query identity data in existing webapps, like:
+- Forums (phpBB, Discourse, etc.)
+- Custom Identity stores (Keycloak, ...)
+- CRMs (Wordpress, ...)
+- Self-hosted clouds (Nextcloud, ownCloud, ...)
+
+To integrate this backend with your webapp, you will need to implement the REST endpoints described below.
+
+## Features
+| Name                                            | Supported? |
+|-------------------------------------------------|------------|
+| [Authentication](../features/authentication.md) | Yes        |
+| [Directory](../features/directory.md)           | Yes        |
+| [Identity](../features/identity.md)             | Yes        |
+| [Profile](../features/profile.md)               | Yes        |
+
+## Configuration
+| Key                                  | Default                                        | Description                                          |
+|--------------------------------------|------------------------------------------------|------------------------------------------------------|
+| `rest.enabled`                       | `false`                                        | Globally enable/disable the REST backend             |
+| `rest.host`                          | *None*                                         | Default base URL to use for the different endpoints. |
+| `rest.endpoints.auth`                | `/_mxisd/backend/api/v1/auth/login`            | Validate credentials and get user profile            |
+| `rest.endpoints.directory`           | `/_mxisd/backend/api/v1/directory/user/search` | Search for users by arbitrary input                  |
+| `rest.endpoints.identity.single`     | `/_mxisd/backend/api/v1/identity/single`       | Endpoint to query a single 3PID                      |
+| `rest.endpoints.identity.bulk`       | `/_mxisd/backend/api/v1/identity/bulk`         | Endpoint to query a list of 3PID                     |
+| `rest.endpoints.profile.displayName` | `/_mxisd/backend/api/v1/profile/displayName`   | Query the display name for a Matrix ID
+| `rest.endpoints.profile.threepids`   | `/_mxisd/backend/api/v1/profile/threepids`     | Query the 3PIDs for a Matrix ID
+| `rest.endpoints.profile.roles`       | `/_mxisd/backend/api/v1/profile/roles`         | Query the Roles for a Matrix ID
+
+Endpoint values can handle two formats:
+- URL Path starting with `/` that gets happened to the `rest.host`
+- Full URL, if you want each endpoint to go to a specific server/protocol/port
+
+If an endpoint value is configured as an empty string, it will disable that specific feature, essentially bypassing the
+Identity store for that specific query.
+
+`rest.host` is mandatory if at least one endpoint is not a full URL.
+
+## Endpoints
+### Authentication
+- Method: `POST`
+- Content-Type: `application/json` (JSON)
+- Encoding: `UTF8`
+  
+#### Request Body
+```json
+{
+  "auth": {
+    "mxid": "@john.doe:example.org",
+    "localpart": "john.doe",
+    "domain": "example.org",
+    "password": "passwordOfTheUser"
+  }
+}
+```
+
+#### Response Body
+If the authentication fails:
+```json
+{
+  "auth": {
+    "success": false
+  }
+}
+```
+
+If the authentication succeed:
+- `auth.id` supported values: `localpart`, `mxid`
+- `auth.profile` and any sub-member are all optional
+```json
+{
+  "auth": {
+    "success": true,
+    "id": {
+      "type": "localpart",
+      "value": "john"
+    },
+    "profile": {
+      "display_name": "John Doe",
+      "three_pids": [
+        {
+          "medium": "email",
+          "address": "john.doe@example.org"
+        },
+        {
+          "medium": "msisdn",
+          "address": "123456789"
+        }
+      ]
+    }
+  }
+}
+```
+
+### Directory
+- Method: `POST`
+- Content-Type: `application/json` (JSON)
+- Encoding: `UTF8`
+
+#### Request Body
+```json
+{
+  "by": "<search type>",
+  "search_term": "doe"
+}
+```
+`by` can be:
+- `name`
+- `threepid`
+
+#### Response Body:
+If users found:
+```json
+{
+  "limited": false,
+  "results": [
+    {
+      "avatar_url": "http://domain.tld/path/to/avatar.png",
+      "display_name": "John Doe",
+      "user_id": "UserIdLocalpart"
+    },
+    {
+      "...": "..."
+    }
+  ]
+}
+```
+
+If no user found:
+```json
+{
+  "limited": false,
+  "results": []
+}
+```
+
+### Identity
+#### Single 3PID lookup
+- Method: `POST`
+- Content-Type: `application/json` (JSON)
+- Encoding: `UTF8`
+  
+##### Request Body
+```json
+{
+  "lookup": {
+    "medium": "email",
+    "address": "john.doe@example.org"
+  }
+}
+```
+
+##### Response Body
+If a match was found:
+- `lookup.id.type` supported values: `localpart`, `mxid`
+```json
+{
+  "lookup": {
+    "medium": "email",
+    "address": "john.doe@example.org",
+    "id": {
+      "type": "mxid",
+      "value": "@john:example.org"
+    }
+  }
+}
+```
+
+If no match was found:
+```json
+{}
+```
+
+#### Bulk 3PID lookup
+- Method: `POST`
+- Content-Type: `application/json` (JSON)
+- Encoding: `UTF8`
+  
+##### Request Body
+```json
+{
+  "lookup": [
+    {
+      "medium": "email",
+      "address": "john.doe@example.org"
+    },
+    {
+      "medium": "msisdn",
+      "address": "123456789"
+    }
+  ]
+}
+```
+
+##### Response Body
+For all entries where a match was found:
+- `lookup[].id.type` supported values: `localpart`, `mxid`
+```json
+{
+  "lookup": [
+    {
+      "medium": "email",
+      "address": "john.doe@example.org",
+      "id": {
+        "type": "localpart",
+        "value": "john"
+      }
+    },
+    {
+      "medium": "msisdn",
+      "address": "123456789",
+      "id": {
+        "type": "mxid",
+        "value": "@jane:example.org"
+      }
+    }
+  ]
+}
+```
+
+If no match was found:
+```json
+{
+  "lookup": []
+}
+```
+
+### Profile
+#### Request Body
+For all requests, the values are the same:
+- Method: `POST`
+- Content-Type: `application/json` (JSON)
+- Encoding: `UTF8`
+
+With body (example values):
+##### Request Body
+```json
+{
+    "mxid": "@john.doe:example.org",
+    "localpart": "john.doe",
+    "domain": "example.org"
+}
+```
+#### Response Body
+For all responses, the same object structure will be parsed, making the non-relevant fields as optional.
+
+Structure with example values:
+```json
+{
+  "profile": {
+    "display_name": "John Doe",
+    "threepids": [
+      {
+        "medium": "email",
+        "address": "john.doe@example.org"
+      },
+      {
+        "...": "..."
+      }
+    ],
+    "roles": [
+      "DomainUsers",
+      "SalesOrg",
+      "..."
+    ]
+  }
+}
+```
+The base `profile` key is mandatory. `display_name`, `threepids` and `roles` are only to be returned on the relevant request.
+
+If there is no profile, the following response is expected:
+```json
+{
+  "profile": {}
+}
+```

docs/stores/sql.md

@@ -0,0 +1,142 @@
+# SQL Identity store
+## Supported Databases
+- PostgreSQL
+- MariaDB
+- MySQL
+- SQLite
+
+## Features
+|                       Name                      | Supported |
+|-------------------------------------------------|-----------|
+| [Authentication](../features/authentication.md) | No        |
+| [Directory](../features/directory.md)           | Yes       |
+| [Identity](../features/identity.md)             | Yes       |
+| [Profile](../features/profile.md)               | Yes       |
+
+Due to the implementation complexity of supporting arbitrary hashing/encoding mechanisms or auth flow, Authentication
+will be out of scope of SQL Identity stores and should be done via one of the other identity stores, typically
+the [Exec Identity Store](exec.md) or the [REST Identity Store](rest.md).
+
+## Configuration
+### Basic
+```yaml
+sql:
+  enabled: <boolean>
+```
+Enable/disable the identity store
+
+---
+
+```yaml
+sql:
+  type: <string>
+```
+Set the SQL backend to use:
+- `sqlite`
+- `postgresql`
+- `mariadb`
+- `mysql`
+
+### Connection
+#### SQLite
+```yaml
+sql:
+  connection: <string>
+```
+Set the value to the absolute path to the Synapse SQLite DB file.
+Example: `/path/to/sqlite/file.db`
+
+#### Others
+```yaml
+sql:
+  connection: //<HOST[:PORT]/DB?user=USER&password=PASS
+```
+Set the connection info for the database by replacing the following values:
+- `HOST`: Hostname of the SQL server
+- `PORT`: Optional port value, if not default
+- `DB`: Database name
+- `USER`: Username for the connection
+- `PASS`: Password for the connection
+
+This follow the JDBC URI syntax. See [official website](https://docs.oracle.com/javase/tutorial/jdbc/basics/connecting.html#db_connection_url).
+
+### Directory
+```yaml
+sql:
+  directory:
+    enabled: false
+```
+
+---
+
+```yaml
+sql:
+  directory:
+    query:
+      name:
+        type: <string>
+        value: <string>
+      threepid:
+        type: <string>
+        value: <string>
+```
+For each query, `type` can be used to tell mxisd how to process the ID column:
+- `localpart` will append the `matrix.domain` to it
+- `mxid` will use the ID as-is. If it is not a valid Matrix ID, the search will fail.
+
+`value` is the SQL query and must return two columns:
+- The first being the User ID
+- The second being its display name
+
+Example:
+```yaml
+sql:
+  directory:
+    query:
+      name:
+        type: 'localpart'
+        value: 'SELECT idColumn, displayNameColumn FROM table WHERE displayNameColumn LIKE ?'
+      threepid:
+        type: 'localpart'
+        value: 'SELECT idColumn, displayNameColumn FROM table WHERE threepidColumn LIKE ?'
+```
+
+### Identity
+**NOTE**: Only single lookup is supported. Bulk lookup always returns no mapping. This is a restriction as the Matrix API
+does not allow paging or otherwise limit of results of the API, potentially leading to thousands and thousands 3PIDs at once.
+
+```yaml
+sql:
+  identity:
+    enabled: <boolean>
+    type: <string>
+    query: <string>
+    medium:
+     mediumTypeExample: <dedicated query>
+```
+`type` is used to tell mxisd how to process the returned `uid` column containing the User ID:
+- `localpart` will build a full Matrix ID using the `matrix.domain` value.
+- `mxid` will use the ID as-is. If it is not a valid Matrix ID, lookup(s) will fail.
+
+A specific query can also given per 3PID medium type.
+
+### Profile
+```yaml
+sql:
+  profile:
+    enabled: <boolean>
+    displayName:
+      query: <string>
+    threepid:
+      query: <string>
+    role:
+      type: <string>
+      query: <string>
+    
+
+```
+For the `role` query, `type` can be used to tell mxisd how to inject the User ID in the query:
+- `localpart` will extract and set only the localpart.
+- `mxid` will use the ID as-is.
+
+On each query, the first parameter `?` is set as a string with the corresponding ID format.

docs/stores/synapse.md

@@ -0,0 +1,55 @@
+# Synapse Identity Store
+Synapse's Database itself can be used as an Identity store. This identity store is a regular SQL store with
+built-in default queries that matches Synapse DB.
+
+## Features
+|                       Name                      | Supported |
+|-------------------------------------------------|-----------|
+| [Authentication](../features/authentication.md) | No        |
+| [Directory](../features/directory.md)           | Yes       |
+| [Identity](../features/identity.md)             | Yes       |
+| [Profile](../features/profile.md)               | Yes       |
+
+- Authentication is done by Synapse itself.
+- Roles are mapped to communities. The Role name/ID uses the community ID in the form `+id:domain.tld`
+
+## Configuration
+### Basic
+```yaml
+synapseSql:
+  enabled: <boolean>
+```
+Enable/disable the identity store
+
+---
+
+```yaml
+synapseSql:
+  type: <string>
+```
+Set the SQL backend to use which is configured in synapse:
+- `sqlite`
+- `postgresql`
+
+### SQLite
+```yaml
+synapseSql:
+  connection: <string>
+```
+Set the value to the absolute path to the Synapse SQLite DB file.
+Example: `/path/to/synapse/sqliteFile.db`
+
+### PostgreSQL
+```yaml
+synapseSql:
+  connection: //<HOST[:PORT]/DB?user=USER&password=PASS
+```
+Set the connection info for the database by replacing the following values:
+- `HOST`: Hostname of the SQL server
+- `PORT`: Optional port value, if not default
+- `DB`: Database name
+- `USER`: Username for the connection
+- `PASS`: Password for the connection
+
+### Query customization
+See the [SQL Identity store](sql.md)

docs/stores/wordpress.md

@@ -1,13 +1,16 @@
-# Wordpress
+# Wordpress Identity store
 This Identity store allows you to use user accounts registered on your Wordpress setup.  
 Two types of connections are required for full support:
 - [REST API](https://developer.wordpress.org/rest-api/) with JWT authentication
 - Direct SQL access
 
-This Identity store supports the following features:
-- [Authentication](../features/authentication.md)
-- [Directory](../features/directory-users.md)
-- [Identity](../features/identity.md)
+## Features
+|                       Name                      | Supported |
+|-------------------------------------------------|-----------|
+| [Authentication](../features/authentication.md) | Yes       |
+| [Directory](../features/directory.md)           | Yes       |
+| [Identity](../features/identity.md)             | Yes       |
+| [Profile](../features/profile.md)               | No        |
 
 ## Requirements
 - [Wordpress](https://wordpress.org/download/) >= 4.4
@@ -19,7 +22,7 @@ This Identity store supports the following features:
 ### Wordpress
 #### JWT Auth
 Set a JWT secret into `wp-config.php` like so:
-```
+```php
 define('JWT_AUTH_SECRET_KEY', 'your-top-secret-key');
 ```
 `your-top-secret-key` should be set to a randomly generated value which is kept secret.
@@ -30,26 +33,43 @@ If this is not the case for your installation, the mxisd URL will need to be app
 
 ### mxisd
 Enable in the configuration:
-```
-wordpress.enabled: true
+```yaml
+wordpress:
+  enabled: true
 ```
 Configure the URL to your Wordpress installation - see above about added `/index.php`:
-```
-wordpress.rest.base: 'http://localhost:8080'
+```yaml
+wordpress:
+  rest:
+    base: 'http://localhost:8080'
 ```
 Configure the SQL connection to your Wordpress database:
-```
-wordpress.sql.connection: '//127.0.0.1/wordpress?user=root&password=example'
+```yaml
+wordpress:
+  sql:
+    connection: '//127.0.0.1/wordpress?user=root&password=example'
 ```
 
 ---
 
 By default, MySQL database is expected. If you use another database, use:
-```
-wordpress.sql.type: 'jdbc-scheme'
+```yaml
+wordpress:
+  sql:
+    type: <string>
 ```
 With possible values:
 - `mysql`
 - `mariadb`
 - `postgresql`
 - `sqlite`
+
+---
+
+To configure the tables prefix for default queries, in case a custom value was set during Wordpress install:
+```yaml
+wordpress:
+  sql:
+    tablePrefix: <string>
+```
+By default, the value is set to `wp_`.

docs/threepids/medium/email/smtp-connector.md

@@ -1,8 +1,8 @@
 # Email notifications - SMTP connector
 Connector ID: `smtp`
 
-Example configuration:
-```
+## Configuration
+```yaml
 threepid:
   medium:
     email:
@@ -12,8 +12,8 @@ threepid:
       connectors:
         smtp:
           host: 'smtpHostname'
-          port: 587
-          tls: 1 # 0 = no STARTLS, 1 = try, 2 = force
+          tls: 1 # 0 = no STARTLS, 1 = try, 2 = force, 3 = TLS/SSL
+          port: 587 # Set appropriate value depending on your TLS setting
           login: 'smtpLogin'
           password: 'smtpPassword'
 ```

docs/threepids/medium/msisdn/twilio-connector.md

@@ -1,15 +1,14 @@
 # SMS notifications - Twilio connector
 Connector ID: `twilio`
 
-Example configuration:
-```
+## Configuration
+```yaml
 threepid:
   medium:
     msisdn:
       connectors:
         twilio:
-          accountSid: 'myAccountSid'
-          authToken: 'myAuthToken'
+          account_sid: 'myAccountSid'
+          auth_token: 'myAuthToken'
           number: '+123456789'
-
 ```

docs/threepids/notification/basic-handler.md

@@ -0,0 +1,40 @@
+# Basic Notification handler
+Basic notification handler which uses two components:
+- Content generator, to produce the notifications
+- Connectors to send the notification content
+
+This handler can be used with the 3PID types:
+- `email`
+- `msisdn` (Phone numbers)
+
+## Generators
+- [Template](template-generator.md)
+## Connectors
+- Email
+  - [SMTP](../medium/email/smtp-connector.md)
+- SMS
+  - [Twilio](../medium/msisdn/twilio-connector.md)
+
+## Configuration
+Enabled by default or with:
+```yaml
+notification:
+  handler:
+    email: 'raw'
+```
+
+**WARNING:** Will be consolidated soon, prone to breaking changes.  
+Structure and default values:
+```yaml
+threepid:
+  medium:
+    email:
+      identity:
+        from: ''
+        name: ''
+      connector: 'smtp'
+      generator: 'template'
+    msisdn:
+      connector: 'twilio'
+      generator: 'template'
+```

docs/threepids/notification/sendgrid-handler.md

@@ -0,0 +1,39 @@
+# SendGrid Notification handler
+> **WARNING:** This section is incomplete and may be misleading. Contact us if guidance is needed.
+
+Enable with:
+```yaml
+notification:
+  handler:
+    email: 'sendgrid'
+```
+
+Available Configuration keys:
+```yaml
+notification:
+  handlers:
+    sendgrid:
+      api:
+        key: <API key>
+      identity:
+        from: <Sender email address>
+        name: <Sender name>
+      templates:
+        invite:
+          subject: <Subject of the email notification sent for room invites>
+          body:
+            text: <Path to file containing the raw text part of the email. Do not set to not use one>
+            html: <Path to file containing the HTML part of the email. Do not set to not use one>
+        session:
+          validation:
+            subject: <Subject of the email notification sent for 3PID sessions>
+            body:
+              text: <Path to file containing the raw text part of the email. Do not set to not use one>
+              html: <Path to file containing the HTML part of the email. Do not set to not use one>
+          unbind:
+            fraudulent:
+              subject: <Subject of the email notification sent for potentially fraudulent 3PID unbinds>
+              body:
+                text: <Path to file containing the raw text part of the email. Do not set to not use one>
+                html: <Path to file containing the raw text part of the email. Do not set to not use one>
+``` 

docs/threepids/notification/template-generator.md

@@ -0,0 +1,113 @@
+# Notifications: Template generator
+Most of the Identity actions will trigger a notification of some kind, informing the user of some confirmation, next step
+or just informing them about the current state of things.
+
+Those notifications are by default generated from templates and by replacing placeholder tokens in them with the relevant
+values of the notification. It is possible to customize the value of some placeholders, making easy to set values in the builtin templates, and/or
+provide your own custom templates.
+
+Templates for the following events/actions are available:
+- [3PID invite](../../features/identity.md)
+- [3PID session: validation](../session/session.md)
+- [3PID session: fraudulent unbind](https://github.com/kamax-matrix/mxisd/wiki/mxisd-and-your-privacy#improving-your-privacy-one-commit-at-the-time)
+- [Matrix ID invite](../../features/experimental/application-service.md#email-notification-about-room-invites-by-matrix-ids)
+
+## Placeholders
+All placeholders **MUST** be surrounded with `%` in the template. Per example, the `DOMAIN` placeholder would become
+`%DOMAIN%` within the template. This ensures replacement doesn't happen on non-placeholder strings.
+
+### Global
+The following placeholders are available in every template:
+
+| Placeholder                     | Purpose                                                                      |
+|---------------------------------|------------------------------------------------------------------------------|
+| `DOMAIN`                        | Identity server authoritative domain, as configured in `matrix.domain`       |
+| `DOMAIN_PRETTY`                 | Same as `DOMAIN` with the first letter upper case and all other lower case   |
+| `FROM_EMAIL`                    | Email address configured in `threepid.medium.<3PID medium>.identity.from`    |
+| `FROM_NAME`                     | Name configured in `threepid.medium.<3PID medium>.identity.name`             |
+| `RECIPIENT_MEDIUM`              | The 3PID medium, like `email` or `msisdn`                                    |
+| `RECIPIENT_MEDIUM_URL_ENCODED`  | URL encoded value of `RECIPIENT_MEDIUM`                                      |
+| `RECIPIENT_ADDRESS`             | The address to which the notification is sent                                |
+| `RECIPIENT_ADDRESS_URL_ENCODED` | URL encoded value of `RECIPIENT_ADDRESS`                                     |
+
+### Room invitation
+Specific placeholders:
+
+| Placeholder                  | Purpose                                                                           |
+|------------------------------|-----------------------------------------------------------------------------------|
+| `SENDER_ID`                  | Matrix ID of the user who made the invite                                         |
+| `SENDER_NAME`                | Display name of the user who made the invite, if not available/set, empty         |
+| `SENDER_NAME_OR_ID`          | Display name of the user who made the invite. If not available/set, its Matrix ID |
+| `INVITE_MEDIUM`              | The 3PID medium for the invite.                                                   |
+| `INVITE_MEDIUM_URL_ENCODED`  | URL encoded value of `INVITE_MEDIUM`                                              |
+| `INVITE_ADDRESS`             | The 3PID address for the invite.                                                  |
+| `INVITE_ADDRESS_URL_ENCODED` | URL encoded value of `INVITE_ADDRESS`                                             |
+| `ROOM_ID`                    | The Matrix ID of the Room in which the invite took place                          |
+| `ROOM_NAME`                  | The Name of the room in which the invite took place. If not available/set, empty  |
+| `ROOM_NAME_OR_ID`            | The Name of the room in which the invite took place. If not available/set, its ID |
+| `REGISTER_URL`               | The URL to provide to the user allowing them to register their account, if needed |
+
+### Validation of 3PID Session
+Specific placeholders:
+
+| Placeholder        | Purpose                                                                              |
+|--------------------|--------------------------------------------------------------------------------------|
+| `VALIDATION_LINK`  | URL, including token, to validate the 3PID session.                                  |
+| `VALIDATION_TOKEN` | The token needed to validate the session, in case the user cannot use the link.      |
+| `NEXT_URL`         | URL to redirect to after the sessions has been validated.                            |
+
+## Templates
+mxisd comes with a set of builtin templates to easily get started. Those templates can be found
+[in the repository](https://github.com/kamax-matrix/mxisd/tree/master/src/main/resources/threepids). If you want to use
+customized templates, we recommend using the builtin templates as a starting point.
+
+> **NOTE**: The link above point to the latest version of the built-in templates. Those might be different from your
+version. Be sure to view the repo at the current tag.
+
+## Configuration
+All configuration is specific to [3PID mediums](https://matrix.org/docs/spec/appendices.html#pid-types) and happen
+under the namespace `threepid.medium.<medium>.generators.template`.
+
+Under such namespace, the following keys are available:
+- `invite`: Path to the 3PID invite notification template
+- `session.validation`: Path to the 3PID session validation notification template
+- `session.unbind.fraudulent`: Path to the 3PID session fraudulent unbind notification template
+- `generic.matrixId`: Path to the Matrix ID invite notification template
+- `placeholder`: Map of key/values to set static values for some placeholders.
+
+The `placeholder` map supports the following keys, mapped to their respective template placeholders:
+- `REGISTER_URL`
+
+### Example
+#### Simple
+```yaml
+threepid:
+  medium:
+    email:
+      generators:
+        template:
+          placeholder:
+            REGISTER_URL: 'https://matrix-client.example.org'
+```
+In this configuration, the builtin templates are used and a static value for the `REGISTER_URL` is set, allowing to point
+a newly invited user to a webapp allowing the creation of its account on the server.
+
+#### Advanced
+To configure paths to the various templates:
+```yaml
+threepid:
+  medium:
+    email:
+      generators:
+        template:
+          invite: '/path/to/invite-template.eml'
+          session:
+            validation: '/path/to/validate-template.eml'
+            unbind:
+              fraudulent: '/path/to/unbind-fraudulent-template.eml'
+          generic:
+            matrixId: '/path/to/mxid-invite-template.eml'
+          placeholder:
+            REGISTER_URL: 'https://matrix-client.example.org'
+```
+In this configuration, a custom template is used for each event and a static value for the `REGISTER_URL` is set.

docs/threepids/notifications/basic-handler.md

@@ -1,66 +0,0 @@
-# Basic Notification handler
-Basic notification handler which uses two components:
-- Content generator, to produce the notifications
-- Connectors to send the notification content
-
-This handler can be used with the 3PID types:
-- `email`
-- `msisdn` (Phone numbers)
-
-## Generators
-- [Template](template-generator.md)
-## Connectors
-- Email
-  - [SMTP](../medium/email/smtp-connector.md)
-- SMS
-  - [Twilio](../medium/msisdn/twilio-connector.md)
-
-## Configuration
-Enabled by default or with:
-```
-notification:
-  handler:
-    email: 'raw'
-```
-
-**WARNING:** Will be consolidated soon, prone to breaking changes.  
-Structure and default values:
-```
-threepid:
-  medium:
-    email:
-      identity:
-        from: ''
-        name: ''
-      connector: 'smtp'
-      generator: 'template'
-      connectors:
-        smtp:
-          host: ''
-          port: 587
-          tls: 1
-          login: ''
-          password: ''
-      generators:
-        template:
-          invite: 'classpath:threepids/email/invite-template.eml'
-          session:
-            validation:
-              local: 'classpath:threepids/email/validate-local-template.eml'
-              remote: 'classpath:threepids/email/validate-remote-template.eml'
-    msisdn:
-      connector: 'twilio'
-      generator: 'template'
-      connectors:
-        twilio:
-          accountSid: ''
-          authToken: ''
-          number: ''
-      generators:
-        template:
-          invite: 'classpath:threepids/sms/invite-template.txt'
-          session:
-            validation:
-              local: 'classpath:threepids/sms/validate-local-template.txt'
-              remote: 'classpath:threepids/sms/validate-remote-template.txt'
-```

docs/threepids/notifications/sendgrid-handler.md

@@ -1,9 +0,0 @@
-# SendGrid Notification handler
-To be completed. See [raw possible configuration items](https://github.com/kamax-io/mxisd/blob/master/src/main/resources/application.yaml#L172).
-
-Enabled with:
-```
-notification:
-  handler:
-    email: 'sendgrid'
-```

docs/threepids/notifications/template-generator.md

@@ -1,73 +0,0 @@
-# Notifications: Generate from templates
-To create notification content, you can use the `template` generator if supported for the 3PID medium which will read
-content from configured files.
-
-Placeholders can be integrated into the templates to dynamically populate such content with relevant information like
-the 3PID that was requested, the domain of your Identity server, etc.
-
-Templates can be configured for each event that would send a notification to the end user. Events share a set of common
-placeholders and also have their own individual set of placeholders.
-
-## Configuration
-To configure paths to the various templates:
-```
-threepid:
-  medium:
-    <YOUR 3PID MEDIUM HERE>:
-      generators:
-        template:
-          invite: '/path/to/invite-template.eml'
-          session:
-            validation:
-              local: '/path/to/validate-local-template.eml'
-              remote: 'path/to/validate-remote-template.eml'
-```
-The `template` generator is usually the default, so no further configuration is needed.
-
-##  Global placeholders
-| Placeholder           | Purpose                                                                      |
-|-----------------------|------------------------------------------------------------------------------|
-| `%DOMAIN%`            | Identity server authoritative domain, as configured in `matrix.domain`       |
-| `%DOMAIN_PRETTY%`     | Same as `%DOMAIN%` with the first letter upper case and all other lower case |
-| `%FROM_EMAIL%`        | Email address configured in `threepid.medium.<3PID medium>.identity.from`    |
-| `%FROM_NAME%`         | Name configured in `threepid.medium.<3PID medium>.identity.name`             |
-| `%RECIPIENT_MEDIUM%`  | The 3PID medium, like `email` or `msisdn`                                    |
-| `%RECIPIENT_ADDRESS%` | The address to which the notification is sent                                |
-
-## Events
-### Room invitation
-This template is used when someone is invited into a room using an email address which has no known bind to a Matrix ID.
-#### Placeholders
-| Placeholder           | Purpose                                                                                  |
-|-----------------------|------------------------------------------------------------------------------------------|
-| `%SENDER_ID%`         | Matrix ID of the user who made the invite                                                |
-| `%SENDER_NAME%`       | Display name of the user who made the invite, if not available/set, empty                |
-| `%SENDER_NAME_OR_ID%` | Display name of the user who made the invite. If not available/set, its Matrix ID        |
-| `%INVITE_MEDIUM%`     | The 3PID medium for the invite.                                                          |
-| `%INVITE_ADDRESS%`    | The 3PID address for the invite.                                                         |
-| `%ROOM_ID%`           | The Matrix ID of the Room in which the invite took place                                 |
-| `%ROOM_NAME%`         | The Name of the room in which the invite took place. If not available/set, empty         |
-| `%ROOM_NAME_OR_ID%`   | The Name of the room in which the invite took place. If not available/set, its Matrix ID |
-
-### Local validation of 3PID Session
-This template is used when to user which added their 3PID address to their profile/settings and the session policy
-allows at least local sessions.  
-
-#### Placeholders
-| Placeholder          | Purpose                                                                              |
-|----------------------|--------------------------------------------------------------------------------------|
-| `%VALIDATION_LINK%`  | URL, including token, to validate the 3PID session.                                  |
-| `%VALIDATION_TOKEN%` | The token needed to validate the local session, in case the user cannot use the link |
-
-### Remote validation of 3PID Session
-This template is used when to user which added their 3PID address to their profile/settings and the session policy only
-allows remote sessions.
-
-**NOTE:** 3PID session always require local validation of a token, even if a remote session is enforced.  
-One cannot bind a Matrix ID to the session until both local and remote sessions have been validated.
-
-#### Placeholders
-| Placeholder          | Purpose                                                |
-|----------------------|--------------------------------------------------------|
-| `%VALIDATION_TOKEN%` | The token needed to validate the session               |
-| `%NEXT_URL%`         | URL to continue with remote validation of the session. |

docs/threepids/session/session-views.md

@@ -0,0 +1,42 @@
+# Web pages for the 3PID sessions
+You can customize the various pages used during a 3PID validation using the options below.
+
+## Configuration
+Pseudo-configuration to illustrate the structure:
+```yaml
+# CONFIGURATION EXAMPLE
+# DO NOT COPY/PASTE THIS IN YOUR CONFIGURATION
+view:
+  session:
+    onTokenSubmit:
+      success: '/path/to/session/tokenSubmitSuccess-page.html'
+      failure: '/path/to/session/tokenSubmitFailure-page.html'
+# CONFIGURATION EXAMPLE
+# DO NOT COPY/PASTE THIS IN YOUR CONFIGURATION
+```
+
+`view.session`:
+This is triggered when a user submit a validation token for a 3PID session. It is typically visited when clicking the
+link in a validation email.
+
+The template should typically inform the user that the validation was successful and to go back in their Matrix client
+to finish the validation process, or that the validation failed.
+
+Two configuration keys are available that accept paths to HTML templates:
+- `success`
+- `failure`
+
+### Serving static assets
+mxisd will not serve any static asset (images, JS, CSS, etc.). If such are needed, you will need to serve them using the
+reverse proxy sitting in front of mxisd using a path outside of the `/_matrix/identity/` namespace. We advise using
+the base path `/static/` for such use cases, allowing to remain under the same hostname/origin.
+
+You can also serve such assets using absolute URL, possibly under other domains, but be aware of Cross-Origin restrictions
+in browsers which are out of scope of mxisd.
+
+## Placeholders
+### Success
+No object/placeholder are currently available.
+
+### Failure
+No object/placeholder are currently available.

docs/threepids/session/session.md

@@ -0,0 +1,147 @@
+# 3PID Sessions
+- [Overview](#overview)
+- [Restrictions](#restrictions)
+  - [Bindings](#bindings)
+  - [Federation](#federation)
+- [Notifications](#notifications)
+  - [Email](#email)
+  - [Phone numbers](#msisdn-(phone-numbers))
+- [Usage](#usage)
+  - [Configuration](#configuration)
+  - [Web views](#web-views)
+  - [Scenarios](#scenarios)
+    - [Sessions disabled](#sessions-disabled)
+
+## Overview
+When adding an email, a phone number or any other kind of 3PID (Third-Party Identifier) in a Matrix client,
+the identity server is contacted to validate the 3PID.
+
+To validate the 3PID, the identity server creates a session associated with a secret token. That token is sent via a message
+to the 3PID (e.g. an email) with a the necessary info so the user can submit them to the Identity Server, confirm ownership
+of the 3PID.
+
+Once this 3PID is validated, the Homeserver will request that the Identity Server links the provided user Matrix ID with
+the 3PID session and finally add the 3PID to its own data store.
+
+This serves two purposes:
+- Add the 3PID as an administrative/login info for the Homeserver directly
+- Links, called *Bind*, the 3PID so it can be queried from Homeservers and clients when inviting someone in a room
+by a 3PID, allowing it to be resolved to a Matrix ID.
+
+## Restrictions
+### Bindings
+mxisd does not store bindings directly. While a user can see its email, phone number or any other 3PID in its
+settings/profile, it does **NOT** mean it is published/saved anywhere or can be used to invite/search the user.
+
+Identity stores are the ones holding such data, irrelevant if a user added a 3PID to their profile. When queried for
+bindings, mxisd will query Identity stores which are responsible to store this kind of information.
+
+Therefore, by default, any 3PID added to a user profile which is NOT within a configured and enabled Identity backend
+will simply not be usable for search or invites, **even on the same Homeserver!**  
+
+To have such 3PID bindings available for search and invite queries on synapse, use its dedicated
+[Identity store](../../stores/synapse.md).
+
+### Federation
+In a federated set up, identity servers must cooperate to find the Matrix ID associated with a 3PID.
+
+Federation is based on the principle that each server is responsible for its own (dns) domain.
+Therefore only those 3PID can be federated that can be distinguished by their
+domain such as email addresses.
+
+Example: a user from Homeserver `example.org` adds an email `john@example.com`.  
+Federated identity servers would try to find the identity server at `example.com` and ask it for the Matrix ID of associated with `john@example.com`.
+
+Nevertheless, Matrix users might add 3PIDs that are not associated to a domain, for example telephone numbers.
+Or they might even add 3PIDs associated to a different domain (such as an email address hosted by Gmail).
+Such 3PIDs cannot be resolved in a federated way and will not be found from other servers.
+
+Example: a user from Homeserver `example.org` adds an email `john@gmail.com`.  
+If a federated lookup was performed, Identity servers would try to find the 3PID bind at the `gmail.com` server, and
+not `example.org`.
+
+As mxisd is built for self-hosted use cases, mainly for orgs/corps, this is usually not a problem for emails.  
+Sadly, there is currently no mechanism to make this work for phone numbers. 
+
+## Notifications
+3PIDs are validated by sending a pre-formatted message containing a token to that 3PID address, which must be given to the
+Identity server that received the request. This is usually done by means of a URL to visit for email or a short number
+received by SMS for phone numbers.
+
+mxisd use two components for this:
+- Generator which produces the message to be sent with the necessary information the user needs to validate their session.
+- Connector which actually send the notification (e.g. SMTP for email).
+
+Built-in generators and connectors for supported 3PID types:
+
+### Email
+Generators:
+- [Template](../notification/template-generator.md)
+
+Connectors:
+- [SMTP](../medium/email/smtp-connector.md)
+
+#### MSISDN (Phone numbers)
+Generators:
+- [Template](../notification/template-generator.md)
+
+Connectors:
+ - [Twilio](../medium/msisdn/twilio-connector.md) with SMS
+
+## Usage
+### Configuration
+The following example of configuration shows which items are relevant for 3PID sessions.
+
+**IMPORTANT:** Most configuration items shown have default values and should not be included in your own configuration
+file unless you want to specifically overwrite them.
+```yaml
+# CONFIGURATION EXAMPLE
+# DO NOT COPY/PASTE AS-IS IN YOUR CONFIGURATION
+
+session:
+  policy:
+    validation:
+      enabled: true
+    unbind:
+      fraudulent:
+        sendWarning: true
+
+# DO NOT COPY/PASTE AS-IS IN YOUR CONFIGURATION
+# CONFIGURATION EXAMPLE
+```
+
+`session.policy.validation` is the core configuration to control what users configured to use your Identity server
+are allowed to do in terms of 3PID sessions. The policy has a global on/off switch for 3PID sessions using `.enabled`  
+
+---
+
+`unbind.fraudulent` controls warning notifications if an illegal/fraudulent 3PID removal is attempted on the Identity server.  
+This is directly related to synapse disregard for privacy and new GDPR laws in Europe in an attempt to inform users about
+potential privacy leaks.
+
+For more information, see the corresponding [synapse issue](https://github.com/matrix-org/synapse/issues/4540).
+
+### Web views
+Once a user click on a validation link, it is taken to the Identity Server validation page where the token is submitted.  
+If the session or token is invalid, an error page is displayed.  
+Workflow pages are also available for the remote 3PID session process.
+
+See [the dedicated document](session-views.md)
+on how to configure/customize/brand those pages to your liking.
+
+### Scenarios
+#### Sessions disabled
+This configuration would disable 3PID sessions altogether, preventing users from validating emails and/or phone numbers
+and any subsequent actions that requires them, like adding them to their profiles.
+  
+This would be used if mxisd is also performing authentication for the Homeserver, typically with synapse and the
+[REST password provider](https://github.com/kamax-matrix/matrix-synapse-rest-auth), where 3PID mappings would be
+auto-populated.
+
+Use the following values to enable this mode:
+```yaml
+session:
+  policy:
+    validation:
+      enabled: false
+```

docs/troubleshooting.md

@@ -0,0 +1,58 @@
+# Troubleshooting
+- [Purpose](#purpose)
+- [Logs](#logs)
+  - [Locations](#locations)
+  - [Reading Them](#reading-them)
+  - [Common issues](#common-issues)
+- [Submit an issue](#submit-an-issue)
+
+## Purpose
+This document describes basic troubleshooting steps for mxisd.
+
+## Logs
+### Locations
+mxisd logs to `STDOUT` (Standard Output) and `STDERR` (Standard Error) only, which gets redirected
+to log file(s) depending on your system.
+
+If you use the [Debian package](install/debian.md), this goes to `syslog`.  
+If you use the [Docker image](install/docker.md), this goes to the container logs.  
+
+For any other platform, please refer to your package maintainer.
+
+### Increase verbosity
+To increase log verbosity and better track issues, the following means are available:
+- Add the `-v` command line parameter
+- Use the environment variable and value `MXISD_LOG_LEVEL=debug`
+
+### Reading them
+Before reporting an issue, it is important to produce clean and complete logs so they can be understood.
+
+It is usually useless to try to troubleshoot an issue based on a single log line. Any action or API request
+in mxisd would trigger more than one log lines, and those would be considered necessary context to
+understand what happened.
+
+You may also find things called *stacktraces*. Those are important to pin-point bugs and the likes and should
+always be included in any report. They also tend to be very specific about the issue at hand.
+
+Example of a stacktrace:
+```
+Exception in thread "main" java.lang.NullPointerException
+        at com.example.myproject.Book.getTitle(Book.java:16)
+        at com.example.myproject.Author.getBookTitles(Author.java:25)
+        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
+```
+
+### Common issues
+#### Internal Server Error
+`Contact your administrator with reference Transaction #123456789`
+
+This is a generic message produced in case of an unknown error. The transaction reference allows to easily find
+the location in the logs to look for an error.
+
+**IMPORTANT:** That line alone does not tell you anything about the error. You'll need the log lines before and after,
+usually including a stacktrace, to know what happened. Please take the time to read the surround output to get
+context about the issue at hand.
+
+## Submit an issue
+In case the logs do not allow you to understand the issue at hand, please submit clean and complete logs
+as explained [here](#reading-them) in a new issue on the repository, or [get in touch](../README.md#contact).

gradle/wrapper/gradle-wrapper.properties

@@ -1,6 +1,5 @@
-#Fri Aug 11 17:19:02 CEST 2017
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
+distributionUrl=https\://services.gradle.org/distributions/gradle-5.3.1-bin.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-4.0.2-bin.zip

gradlew

@@ -1,5 +1,21 @@
 #!/usr/bin/env sh
 
+#
+# Copyright 2015 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
 ##############################################################################
 ##
 ##  Gradle start up script for UN*X
@@ -28,7 +44,7 @@ APP_NAME="Gradle"
 APP_BASE_NAME=`basename "$0"`
 
 # Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
-DEFAULT_JVM_OPTS=""
+DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
 
 # Use the maximum available, or set MAX_FD != -1 to use that value.
 MAX_FD="maximum"

gradlew.bat

@@ -1,3 +1,19 @@
+@rem
+@rem Copyright 2015 the original author or authors.
+@rem
+@rem Licensed under the Apache License, Version 2.0 (the "License");
+@rem you may not use this file except in compliance with the License.
+@rem You may obtain a copy of the License at
+@rem
+@rem      http://www.apache.org/licenses/LICENSE-2.0
+@rem
+@rem Unless required by applicable law or agreed to in writing, software
+@rem distributed under the License is distributed on an "AS IS" BASIS,
+@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+@rem See the License for the specific language governing permissions and
+@rem limitations under the License.
+@rem
+
 @if "%DEBUG%" == "" @echo off
 @rem ##########################################################################
 @rem
@@ -14,7 +30,7 @@ set APP_BASE_NAME=%~n0
 set APP_HOME=%DIRNAME%
 
 @rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
-set DEFAULT_JVM_OPTS=
+set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
 
 @rem Find java.exe
 if defined JAVA_HOME goto findJavaFromJavaHome

mxisd.example.yaml

@@ -0,0 +1,111 @@
+# Sample configuration file explaining the minimum required keys to be set to run mxisd
+#
+# For a complete list of options, see https://github.com/kamax-matrix/mxisd/docs/README.md
+#
+# Please follow the Getting Started guide if this is your first time using/configuring mxisd
+#
+#  -- https://github.com/kamax-matrix/mxisd/blob/master/docs/getting-started.md#getting-started
+#
+
+#######################
+# Matrix config items #
+#######################
+# Matrix domain, same as the domain configure in your Homeserver configuration.
+# NOTE: in Synapse Homeserver, the Matrix domain is defined as 'server_name' in configuration file.
+#
+# This is used to build the various identifiers in all the features.
+#
+# If the hostname of the public URL used to reach your Matrix services is different from your Matrix domain,
+# per example matrix.domain.tld vs domain.tld, then use the server.name configuration option.
+# See the "Configure" section of the Getting Started guide for more info.
+#
+matrix:
+  domain: ''
+
+
+################
+# Signing keys #
+################
+# Absolute path for the Identity Server signing keys database.
+# /!\ THIS MUST **NOT** BE YOUR HOMESERVER KEYS FILE /!\
+# If this path does not exist, it will be auto-generated.
+#
+# During testing, /var/tmp/mxisd/keys is a possible value
+# For production, recommended location shall be one of the following:
+#   - /var/lib/mxisd/keys
+#   - /var/opt/mxisd/keys
+#   - /var/local/mxisd/keys
+#
+key:
+  path: ''
+
+
+# Path to the SQLite DB file for mxisd internal storage
+# /!\ THIS MUST **NOT** BE YOUR HOMESERVER DATABASE /!\
+#
+# Examples:
+#  - /var/opt/mxisd/store.db
+#  - /var/local/mxisd/store.db
+#  - /var/lib/mxisd/store.db
+#
+storage:
+  provider:
+    sqlite:
+      database: '/path/to/mxisd.db'
+
+
+###################
+# Identity Stores #
+###################
+# If you are using synapse standalone and do not have an Identity store,
+# see https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/synapse.md#synapse-identity-store
+#
+# If you would like to integrate with your AD/Samba/LDAP server,
+# see https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/ldap.md
+#
+# For any other Identity store, or to simply discover them,
+# see https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/README.md
+
+
+#################################################
+# Notifications for invites/addition to profile #
+#################################################
+# This is mandatory to deal with anything e-mail related.
+#
+# For an introduction to sessions, invites and 3PIDs in general,
+# see https://github.com/kamax-matrix/mxisd/blob/master/docs/threepids/session/session.md#3pid-sessions
+#
+# If you would like to change the content of the notifications,
+# see https://github.com/kamax-matrix/mxisd/blob/master/docs/threepids/notification/template-generator.md
+#
+#### E-mail connector
+threepid:
+  medium:
+    email:
+      identity:
+        # The e-mail to send as.
+        from: "matrix-identity@example.org"
+
+      connectors:
+        smtp:
+          # SMTP host
+          host: "smtp.example.org"
+
+          # TLS mode for the connection
+          # Possible values:
+          #  0    Disable any kind of TLS entirely
+          #  1    Enable STARTLS if supported by server (default)
+          #  2    Force STARTLS and fail if not available
+          #  3    Use full TLS/SSL instead of STARTLS
+          #
+          tls: 1
+
+          # SMTP port
+          # Be sure to adapt depending on your TLS choice, if changed from default
+          port: 587
+
+          # Login for SMTP
+          login: "matrix-identity@example.org"
+
+          # Password for the account
+          password: "ThePassword"

src/debian/control

@@ -1,7 +1,9 @@
 Package: mxisd
 Maintainer: Kamax.io <foss@kamax.io>
-Homepage: https://github.com/kamax-io/mxisd
+Homepage: https://github.com/kamax-matrix/mxisd
 Description: Federated Matrix Identity Server
 Architecture: all
+Section: net
+Priority: optional
 Depends: openjdk-8-jre | openjdk-8-jre-headless | openjdk-8-jdk | openjdk-8-jdk-headless
 Version: 0

src/debian/postinst

@@ -6,8 +6,14 @@ useradd -r mxisd || true
 # Set permissions for data directory
 chown -R mxisd:mxisd %DEB_DATA_DIR%
 
-# Create symlink to mxusd
-ln -sfT /usr/lib/mxisd/mxisd.jar /usr/bin/mxisd
+# Create symlink to mxisd run script
+ln -sfT /usr/lib/mxisd/mxisd /usr/bin/mxisd
 
 # Enable systemd service
 systemctl enable mxisd.service
+
+# If we already have a config file setup, we attempt to run mxisd automatically
+# Specifically targeted at upgrades where the service needs to be restarted
+if [ -f "%DEB_CONF_FILE%" ]; then
+    systemctl restart mxisd.service
+fi

src/docker/start.sh

@@ -1,25 +1,34 @@
-#!/usr/bin/env bash
+#!/bin/bash
+
 if [[ -n "$CONF_FILE_PATH" ]] && [ ! -f "$CONF_FILE_PATH" ]; then
     echo "Generating config file $CONF_FILE_PATH"
     touch "CONF_FILE_PATH"
 
     if [[ -n "$MATRIX_DOMAIN" ]]; then
         echo "Setting matrix domain to $MATRIX_DOMAIN"
-        echo "matrix.domain: $MATRIX_DOMAIN" >> "$CONF_FILE_PATH"
+        echo "matrix:" >> "$CONF_FILE_PATH"
+        echo "  domain: '$MATRIX_DOMAIN'" >> "$CONF_FILE_PATH"
+        echo >> "$CONF_FILE_PATH"
     fi
 
     if [[ -n "$SIGN_KEY_PATH" ]]; then
         echo "Setting signing key path to $SIGN_KEY_PATH"
-        echo "key.path: $SIGN_KEY_PATH" >> "$CONF_FILE_PATH"
+        echo "key:" >> "$CONF_FILE_PATH"
+        echo "  path: '$SIGN_KEY_PATH'" >> "$CONF_FILE_PATH"
+        echo >> "$CONF_FILE_PATH"
     fi
 
     if [[ -n "$SQLITE_DATABASE_PATH" ]]; then
         echo "Setting SQLite DB path to $SQLITE_DATABASE_PATH"
-        echo "storage.provider.sqlite.database: $SQLITE_DATABASE_PATH" >> "$CONF_FILE_PATH"
+        echo "storage:" >> "$CONF_FILE_PATH"
+        echo "  provider:" >> "$CONF_FILE_PATH"
+        echo "    sqlite:" >> "$CONF_FILE_PATH"
+        echo "      database: '$SQLITE_DATABASE_PATH'" >> "$CONF_FILE_PATH"
+        echo >> "$CONF_FILE_PATH"
     fi
 
     echo "Starting mxisd..."
     echo
 fi
 
-exec java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -Dspring.config.location=/etc/mxisd/ -Dspring.config.name=mxisd -jar /mxisd.jar
+exec java -jar /app/mxisd.jar -c /etc/mxisd/mxisd.yaml

src/main/java/edazdarevic/commons/net/CIDRUtils.java

@@ -20,8 +20,7 @@
  * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
  * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
  * THE SOFTWARE.
-*
-* */
+ */
 
 package edazdarevic.commons.net;
 
@@ -37,6 +36,7 @@ import java.util.List;
  * both IPv4 and IPv6.
  */
 public class CIDRUtils {
+
     private final String cidr;
 
     private InetAddress inetAddress;
@@ -44,7 +44,6 @@ public class CIDRUtils {
     private InetAddress endAddress;
     private final int prefixLength;
 
-
     public CIDRUtils(String cidr) throws UnknownHostException {
 
         this.cidr = cidr;
@@ -66,7 +65,6 @@ public class CIDRUtils {
 
 
     private void calculate() throws UnknownHostException {
-
         ByteBuffer maskBuffer;
         int targetSize;
         if (inetAddress.getAddress().length == 4) {
@@ -120,14 +118,9 @@ public class CIDRUtils {
     }
 
     public String getNetworkAddress() {
-
         return this.startAddress.getHostAddress();
     }
 
-    public String getBroadcastAddress() {
-        return this.endAddress.getHostAddress();
-    }
-
     public boolean isInRange(String ipAddress) throws UnknownHostException {
         InetAddress address = InetAddress.getByName(ipAddress);
         BigInteger start = new BigInteger(1, this.startAddress.getAddress());
@@ -139,4 +132,5 @@ public class CIDRUtils {
 
         return (st == -1 || st == 0) && (te == -1 || te == 0);
     }
+
 }

src/main/java/io/kamax/mxisd/HttpMxisd.java

@@ -0,0 +1,146 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd;
+
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.http.undertow.handler.InternalInfoHandler;
+import io.kamax.mxisd.http.undertow.handler.OptionsHandler;
+import io.kamax.mxisd.http.undertow.handler.SaneHandler;
+import io.kamax.mxisd.http.undertow.handler.as.v1.AsNotFoundHandler;
+import io.kamax.mxisd.http.undertow.handler.as.v1.AsTransactionHandler;
+import io.kamax.mxisd.http.undertow.handler.as.v1.AsUserHandler;
+import io.kamax.mxisd.http.undertow.handler.auth.RestAuthHandler;
+import io.kamax.mxisd.http.undertow.handler.auth.v1.LoginGetHandler;
+import io.kamax.mxisd.http.undertow.handler.auth.v1.LoginHandler;
+import io.kamax.mxisd.http.undertow.handler.auth.v1.LoginPostHandler;
+import io.kamax.mxisd.http.undertow.handler.directory.v1.UserDirectorySearchHandler;
+import io.kamax.mxisd.http.undertow.handler.identity.v1.*;
+import io.kamax.mxisd.http.undertow.handler.invite.v1.RoomInviteHandler;
+import io.kamax.mxisd.http.undertow.handler.profile.v1.InternalProfileHandler;
+import io.kamax.mxisd.http.undertow.handler.profile.v1.ProfileHandler;
+import io.kamax.mxisd.http.undertow.handler.register.v1.Register3pidRequestTokenHandler;
+import io.kamax.mxisd.http.undertow.handler.status.StatusHandler;
+import io.kamax.mxisd.http.undertow.handler.status.VersionHandler;
+import io.undertow.Handlers;
+import io.undertow.Undertow;
+import io.undertow.server.HttpHandler;
+
+import java.util.Objects;
+
+public class HttpMxisd {
+
+    // Core
+    private Mxisd m;
+
+    // I/O
+    private Undertow httpSrv;
+
+    static {
+        // Used in XNIO package, dependency of Undertow
+        // We switch to slf4j
+        System.setProperty("org.jboss.logging.provider", "slf4j");
+    }
+
+    public HttpMxisd(MxisdConfig cfg) {
+        m = new Mxisd(cfg);
+    }
+
+    public void start() {
+        m.start();
+
+        HttpHandler helloHandler = SaneHandler.around(new HelloHandler());
+
+        HttpHandler asUserHandler = SaneHandler.around(new AsUserHandler(m.getAs()));
+        HttpHandler asTxnHandler = SaneHandler.around(new AsTransactionHandler(m.getAs()));
+        HttpHandler asNotFoundHandler = SaneHandler.around(new AsNotFoundHandler(m.getAs()));
+
+        HttpHandler storeInvHandler = SaneHandler.around(new StoreInviteHandler(m.getConfig().getServer(), m.getInvite(), m.getKeyManager()));
+
+        httpSrv = Undertow.builder().addHttpListener(m.getConfig().getServer().getPort(), "0.0.0.0").setHandler(Handlers.routing()
+
+                .add("OPTIONS", "/**", SaneHandler.around(new OptionsHandler()))
+
+                // Status endpoints
+                .get(StatusHandler.Path, SaneHandler.around(new StatusHandler()))
+                .get(VersionHandler.Path, SaneHandler.around(new VersionHandler()))
+
+                // Authentication endpoints
+                .get(LoginHandler.Path, SaneHandler.around(new LoginGetHandler(m.getAuth(), m.getHttpClient())))
+                .post(LoginHandler.Path, SaneHandler.around(new LoginPostHandler(m.getAuth())))
+                .post(RestAuthHandler.Path, SaneHandler.around(new RestAuthHandler(m.getAuth())))
+
+                // Directory endpoints
+                .post(UserDirectorySearchHandler.Path, SaneHandler.around(new UserDirectorySearchHandler(m.getDirectory())))
+
+                // Key endpoints
+                .get(KeyGetHandler.Path, SaneHandler.around(new KeyGetHandler(m.getKeyManager())))
+                .get(RegularKeyIsValidHandler.Path, SaneHandler.around(new RegularKeyIsValidHandler(m.getKeyManager())))
+                .get(EphemeralKeyIsValidHandler.Path, SaneHandler.around(new EphemeralKeyIsValidHandler(m.getKeyManager())))
+
+                // Identity endpoints
+                .get(HelloHandler.Path, helloHandler)
+                .get(HelloHandler.Path + "/", helloHandler) // Be lax with possibly trailing slash
+                .get(SingleLookupHandler.Path, SaneHandler.around(new SingleLookupHandler(m.getConfig(), m.getIdentity(), m.getSign())))
+                .post(BulkLookupHandler.Path, SaneHandler.around(new BulkLookupHandler(m.getIdentity())))
+                .post(StoreInviteHandler.Path, storeInvHandler)
+                .post(SessionStartHandler.Path, SaneHandler.around(new SessionStartHandler(m.getSession())))
+                .get(SessionValidateHandler.Path, SaneHandler.around(new SessionValidationGetHandler(m.getSession(), m.getConfig())))
+                .post(SessionValidateHandler.Path, SaneHandler.around(new SessionValidationPostHandler(m.getSession())))
+                .get(SessionTpidGetValidatedHandler.Path, SaneHandler.around(new SessionTpidGetValidatedHandler(m.getSession())))
+                .post(SessionTpidBindHandler.Path, SaneHandler.around(new SessionTpidBindHandler(m.getSession(), m.getInvite(), m.getSign())))
+                .post(SessionTpidUnbindHandler.Path, SaneHandler.around(new SessionTpidUnbindHandler(m.getSession())))
+                .post(SignEd25519Handler.Path, SaneHandler.around(new SignEd25519Handler(m.getConfig(), m.getInvite(), m.getSign())))
+
+                // Profile endpoints
+                .get(ProfileHandler.Path, SaneHandler.around(new ProfileHandler(m.getProfile())))
+                .get(InternalProfileHandler.Path, SaneHandler.around(new InternalProfileHandler(m.getProfile())))
+
+                // Registration endpoints
+                .post(Register3pidRequestTokenHandler.Path, SaneHandler.around(new Register3pidRequestTokenHandler(m.getReg(), m.getClientDns(), m.getHttpClient())))
+
+                // Invite endpoints
+                .post(RoomInviteHandler.Path, SaneHandler.around(new RoomInviteHandler(m.getHttpClient(), m.getClientDns(), m.getInvite())))
+
+                // Application Service endpoints
+                .get(AsUserHandler.Path, asUserHandler)
+                .get("/_matrix/app/v1/rooms/**", asNotFoundHandler)
+                .put(AsTransactionHandler.Path, asTxnHandler)
+
+                .get("/users/{" + AsUserHandler.ID + "}", asUserHandler) // Legacy endpoint
+                .get("/rooms/**", asNotFoundHandler) // Legacy endpoint
+                .put("/transactions/{" + AsTransactionHandler.ID + "}", asTxnHandler) // Legacy endpoint
+
+                // Banned endpoints
+                .get(InternalInfoHandler.Path, SaneHandler.around(new InternalInfoHandler()))
+
+        ).build();
+
+        httpSrv.start();
+    }
+
+    public void stop() {
+        // Because it might have never been initialized if an exception is thrown early
+        if (Objects.nonNull(httpSrv)) httpSrv.stop();
+
+        m.stop();
+    }
+
+}

src/main/java/io/kamax/mxisd/Mxisd.java

@@ -0,0 +1,205 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd;
+
+import io.kamax.mxisd.as.AppSvcManager;
+import io.kamax.mxisd.auth.AuthManager;
+import io.kamax.mxisd.auth.AuthProviders;
+import io.kamax.mxisd.backend.IdentityStoreSupplier;
+import io.kamax.mxisd.backend.sql.synapse.Synapse;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.crypto.CryptoFactory;
+import io.kamax.mxisd.crypto.KeyManager;
+import io.kamax.mxisd.crypto.SignatureManager;
+import io.kamax.mxisd.crypto.ed25519.Ed25519KeyManager;
+import io.kamax.mxisd.directory.DirectoryManager;
+import io.kamax.mxisd.directory.DirectoryProviders;
+import io.kamax.mxisd.dns.ClientDnsOverwrite;
+import io.kamax.mxisd.dns.FederationDnsOverwrite;
+import io.kamax.mxisd.invitation.InvitationManager;
+import io.kamax.mxisd.lookup.ThreePidProviders;
+import io.kamax.mxisd.lookup.fetcher.IRemoteIdentityServerFetcher;
+import io.kamax.mxisd.lookup.provider.BridgeFetcher;
+import io.kamax.mxisd.lookup.provider.RemoteIdentityServerFetcher;
+import io.kamax.mxisd.lookup.strategy.LookupStrategy;
+import io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy;
+import io.kamax.mxisd.matrix.HomeserverFederationResolver;
+import io.kamax.mxisd.matrix.IdentityServerUtils;
+import io.kamax.mxisd.notification.NotificationHandlerSupplier;
+import io.kamax.mxisd.notification.NotificationHandlers;
+import io.kamax.mxisd.notification.NotificationManager;
+import io.kamax.mxisd.profile.ProfileManager;
+import io.kamax.mxisd.profile.ProfileProviders;
+import io.kamax.mxisd.registration.RegistrationManager;
+import io.kamax.mxisd.session.SessionManager;
+import io.kamax.mxisd.storage.IStorage;
+import io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+
+import java.util.ServiceLoader;
+
+public class Mxisd {
+
+    public static final String Name = StringUtils.defaultIfBlank(Mxisd.class.getPackage().getImplementationTitle(), "mxisd");
+    public static final String Version = StringUtils.defaultIfBlank(Mxisd.class.getPackage().getImplementationVersion(), "UNKNOWN");
+    public static final String Agent = Name + "/" + Version;
+
+    private MxisdConfig cfg;
+
+    private CloseableHttpClient httpClient;
+    private IRemoteIdentityServerFetcher srvFetcher;
+
+    private IStorage store;
+
+    private Ed25519KeyManager keyMgr;
+    private SignatureManager signMgr;
+    private ClientDnsOverwrite clientDns;
+
+    // Features
+    private AuthManager authMgr;
+    private DirectoryManager dirMgr;
+    private LookupStrategy idStrategy;
+    private InvitationManager invMgr;
+    private ProfileManager pMgr;
+    private AppSvcManager asHander;
+    private SessionManager sessMgr;
+    private NotificationManager notifMgr;
+    private RegistrationManager regMgr;
+
+    // HS-specific classes
+    private Synapse synapse;
+
+    public Mxisd(MxisdConfig cfg) {
+        this.cfg = cfg.build();
+    }
+
+    private void build() {
+        httpClient = HttpClients.custom()
+                .setUserAgent(Agent)
+                .setMaxConnPerRoute(Integer.MAX_VALUE)
+                .setMaxConnTotal(Integer.MAX_VALUE)
+                .build();
+
+        FederationDnsOverwrite fedDns = new FederationDnsOverwrite(cfg.getDns().getOverwrite());
+        HomeserverFederationResolver resolver = new HomeserverFederationResolver(fedDns, httpClient);
+        IdentityServerUtils.setHttpClient(httpClient);
+        srvFetcher = new RemoteIdentityServerFetcher(httpClient);
+
+        store = new OrmLiteSqlStorage(cfg);
+        keyMgr = CryptoFactory.getKeyManager(cfg.getKey());
+        signMgr = CryptoFactory.getSignatureManager(cfg, keyMgr);
+        clientDns = new ClientDnsOverwrite(cfg.getDns().getOverwrite());
+
+        synapse = new Synapse(cfg.getSynapseSql());
+        BridgeFetcher bridgeFetcher = new BridgeFetcher(cfg.getLookup().getRecursive().getBridge(), srvFetcher);
+        ServiceLoader.load(IdentityStoreSupplier.class).iterator().forEachRemaining(p -> p.accept(this));
+        ServiceLoader.load(NotificationHandlerSupplier.class).iterator().forEachRemaining(p -> p.accept(this));
+
+        idStrategy = new RecursivePriorityLookupStrategy(cfg.getLookup(), ThreePidProviders.get(), bridgeFetcher);
+        pMgr = new ProfileManager(ProfileProviders.get(), clientDns, httpClient);
+        notifMgr = new NotificationManager(cfg.getNotification(), NotificationHandlers.get());
+        sessMgr = new SessionManager(cfg.getSession(), cfg.getMatrix(), store, notifMgr);
+        invMgr = new InvitationManager(cfg, store, idStrategy, keyMgr, signMgr, resolver, notifMgr, pMgr);
+        authMgr = new AuthManager(cfg, AuthProviders.get(), idStrategy, invMgr, clientDns, httpClient);
+        dirMgr = new DirectoryManager(cfg.getDirectory(), clientDns, httpClient, DirectoryProviders.get());
+        regMgr = new RegistrationManager(cfg.getRegister(), httpClient, clientDns, invMgr);
+        asHander = new AppSvcManager(this);
+    }
+
+    public MxisdConfig getConfig() {
+        return cfg;
+    }
+
+    public CloseableHttpClient getHttpClient() {
+        return httpClient;
+    }
+
+    public ClientDnsOverwrite getClientDns() {
+        return clientDns;
+    }
+
+    public IRemoteIdentityServerFetcher getServerFetcher() {
+        return srvFetcher;
+    }
+
+    public KeyManager getKeyManager() {
+        return keyMgr;
+    }
+
+    public InvitationManager getInvite() {
+        return invMgr;
+    }
+
+    public LookupStrategy getIdentity() {
+        return idStrategy;
+    }
+
+    public AuthManager getAuth() {
+        return authMgr;
+    }
+
+    public SessionManager getSession() {
+        return sessMgr;
+    }
+
+    public DirectoryManager getDirectory() {
+        return dirMgr;
+    }
+
+    public ProfileManager getProfile() {
+        return pMgr;
+    }
+
+    public SignatureManager getSign() {
+        return signMgr;
+    }
+
+    public RegistrationManager getReg() {
+        return regMgr;
+    }
+
+    public AppSvcManager getAs() {
+        return asHander;
+    }
+
+    public NotificationManager getNotif() {
+        return notifMgr;
+    }
+
+    public IStorage getStore() {
+        return store;
+    }
+
+    public Synapse getSynapse() {
+        return synapse;
+    }
+
+    public void start() {
+        build();
+    }
+
+    public void stop() {
+        // no-op
+    }
+
+}

src/main/java/io/kamax/mxisd/MxisdStandaloneExec.java

@@ -0,0 +1,100 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd;
+
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.config.YamlConfigLoader;
+import io.kamax.mxisd.exception.ConfigurationException;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.Objects;
+
+public class MxisdStandaloneExec {
+
+    private static final Logger log = LoggerFactory.getLogger("App");
+
+    public static void main(String[] args) {
+        String logLevel = System.getenv("MXISD_LOG_LEVEL");
+        if (StringUtils.isNotBlank(logLevel)) {
+            System.setProperty("org.slf4j.simpleLogger.log.io.kamax.mxisd", logLevel);
+        }
+
+        try {
+            MxisdConfig cfg = null;
+            Iterator<String> argsIt = Arrays.asList(args).iterator();
+            while (argsIt.hasNext()) {
+                String arg = argsIt.next();
+                if (StringUtils.equalsAny(arg, "-h", "--help", "-?", "--usage")) {
+                    System.out.println("Available arguments:" + System.lineSeparator());
+                    System.out.println("  -h, --help       Show this help message");
+                    System.out.println("  --version        Print the version then exit");
+                    System.out.println("  -c, --config     Set the configuration file location");
+                    System.out.println("  -v               Increase log level (log more info)");
+                    System.out.println("  -vv              Further increase log level");
+                    System.out.println(" ");
+                    System.exit(0);
+                } else if (StringUtils.equals(arg, "-v")) {
+                    System.setProperty("org.slf4j.simpleLogger.log.io.kamax.mxisd", "debug");
+                } else if (StringUtils.equals(arg, "-vv")) {
+                    System.setProperty("org.slf4j.simpleLogger.log.io.kamax.mxisd", "trace");
+                } else if (StringUtils.equalsAny(arg, "-c", "--config")) {
+                    String cfgFile = argsIt.next();
+                    cfg = YamlConfigLoader.loadFromFile(cfgFile);
+                } else if (StringUtils.equals("--version", arg)) {
+                    System.out.println(Mxisd.Version);
+                    System.exit(0);
+                } else {
+                    System.err.println("Invalid argument: " + arg);
+                    System.err.println("Try '--help' for available arguments");
+                    System.exit(1);
+                }
+            }
+
+            if (Objects.isNull(cfg)) {
+                cfg = YamlConfigLoader.tryLoadFromFile("mxisd.yaml").orElseGet(MxisdConfig::new);
+            }
+
+            log.info("mxisd starting");
+            log.info("Version: {}", Mxisd.Version);
+
+            HttpMxisd mxisd = new HttpMxisd(cfg);
+            Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+                mxisd.stop();
+                log.info("mxisd stopped");
+            }));
+            mxisd.start();
+
+            log.info("mxisd started");
+        } catch (ConfigurationException e) {
+            log.error(e.getDetailedMessage());
+            log.error(e.getMessage());
+            System.exit(2);
+        } catch (Throwable t) {
+            t.printStackTrace();
+            System.exit(1);
+        }
+    }
+
+}

src/main/java/io/kamax/mxisd/UserID.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -30,6 +30,10 @@ public class UserID {
         // stub for (de)serialization
     }
 
+    public UserID(UserIdType type, String value) {
+        this(type.getId(), value);
+    }
+
     public UserID(String type, String value) {
         this.type = type;
         this.value = value;

src/main/java/io/kamax/mxisd/UserIdType.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as

src/main/java/io/kamax/mxisd/as/AppSvcManager.java

@@ -0,0 +1,292 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.as;
+
+import com.google.gson.JsonObject;
+import io.kamax.matrix.MatrixID;
+import io.kamax.matrix._MatrixID;
+import io.kamax.matrix.client.MatrixClientContext;
+import io.kamax.matrix.client.as.MatrixApplicationServiceClient;
+import io.kamax.matrix.event.EventKey;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.as.processor.event.EventTypeProcessor;
+import io.kamax.mxisd.as.processor.event.MembershipEventProcessor;
+import io.kamax.mxisd.as.processor.event.MessageEventProcessor;
+import io.kamax.mxisd.as.registration.SynapseRegistrationYaml;
+import io.kamax.mxisd.config.AppServiceConfig;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.exception.ConfigurationException;
+import io.kamax.mxisd.exception.HttpMatrixException;
+import io.kamax.mxisd.exception.NotAllowedException;
+import io.kamax.mxisd.storage.IStorage;
+import io.kamax.mxisd.storage.ormlite.dao.ASTransactionDao;
+import io.kamax.mxisd.util.GsonParser;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.ObjectUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.introspector.BeanAccess;
+import org.yaml.snakeyaml.representer.Representer;
+
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.util.*;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+
+public class AppSvcManager {
+
+    private static final Logger log = LoggerFactory.getLogger(AppSvcManager.class);
+
+    private final AppServiceConfig cfg;
+    private final IStorage store;
+    private final GsonParser parser = new GsonParser();
+
+    private MatrixApplicationServiceClient client;
+    private Map<String, EventTypeProcessor> processors = new HashMap<>();
+    private Map<String, CompletableFuture<String>> transactionsInProgress = new ConcurrentHashMap<>();
+
+    public AppSvcManager(Mxisd m) {
+        this.cfg = m.getConfig().getAppsvc();
+        this.store = m.getStore();
+
+        /*
+        We process the configuration to make sure all is fine and setting default values if needed
+         */
+
+        // By default, the feature is enabled
+        cfg.setEnabled(ObjectUtils.defaultIfNull(cfg.isEnabled(), false));
+
+        if (!cfg.isEnabled()) {
+            return;
+        }
+
+        if (Objects.isNull(cfg.getEndpoint().getToAS().getUrl())) {
+            throw new ConfigurationException("App Service: Endpoint: To AS: URL");
+        }
+
+        if (Objects.isNull(cfg.getEndpoint().getToAS().getToken())) {
+            throw new ConfigurationException("App Service: Endpoint: To AS: Token", "Must be set, even if to an empty string");
+        }
+
+        if (Objects.isNull(cfg.getEndpoint().getToHS().getUrl())) {
+            throw new ConfigurationException("App Service: Endpoint: To HS: URL");
+        }
+
+        if (Objects.isNull(cfg.getEndpoint().getToHS().getToken())) {
+            throw new ConfigurationException("App Service: Endpoint: To HS: Token", "Must be set, even if to an empty string");
+        }
+
+        // We set a default status for each feature individually
+        cfg.getFeature().getAdmin().setEnabled(ObjectUtils.defaultIfNull(cfg.getFeature().getAdmin().getEnabled(), cfg.isEnabled()));
+        cfg.getFeature().setCleanExpiredInvite(ObjectUtils.defaultIfNull(cfg.getFeature().getCleanExpiredInvite(), cfg.isEnabled()));
+        cfg.getFeature().setInviteById(ObjectUtils.defaultIfNull(cfg.getFeature().getInviteById(), false));
+
+        if (cfg.getFeature().getAdmin().getEnabled()) {
+            if (StringUtils.isBlank(cfg.getUser().getMain())) {
+                throw new ConfigurationException("App Service admin feature is enabled, but no main user configured");
+            }
+
+            if (cfg.getUser().getMain().startsWith("@") || cfg.getUser().getMain().contains(":")) {
+                throw new ConfigurationException("App Service: Users: Main ID: Is not a localpart");
+            }
+        }
+
+        if (cfg.getFeature().getCleanExpiredInvite()) {
+            if (StringUtils.isBlank(cfg.getUser().getInviteExpired())) {
+                throw new ConfigurationException("App Service user for Expired Invite is not set");
+            }
+
+            if (cfg.getUser().getMain().startsWith("@") || cfg.getUser().getMain().contains(":")) {
+                throw new ConfigurationException("App Service: Users: Expired Invite ID: Is not a localpart");
+            }
+        }
+
+        MatrixClientContext mxContext = new MatrixClientContext();
+        mxContext.setDomain(m.getConfig().getMatrix().getDomain());
+        mxContext.setToken(cfg.getEndpoint().getToHS().getToken());
+        mxContext.setHsBaseUrl(cfg.getEndpoint().getToHS().getUrl());
+        client = new MatrixApplicationServiceClient(mxContext);
+
+        processors.put("m.room.member", new MembershipEventProcessor(client, m));
+        processors.put("m.room.message", new MessageEventProcessor(m, client));
+
+        processSynapseConfig(m.getConfig());
+    }
+
+    private void processSynapseConfig(MxisdConfig cfg) {
+        String synapseRegFile = cfg.getAppsvc().getRegistration().getSynapse().getFile();
+
+        if (StringUtils.isBlank(synapseRegFile)) {
+            log.info("No synapse registration file path given - skipping generation...");
+            return;
+        }
+
+        SynapseRegistrationYaml syncCfg = SynapseRegistrationYaml.parse(cfg.getAppsvc(), cfg.getMatrix().getDomain());
+
+        Representer rep = new Representer();
+        rep.getPropertyUtils().setBeanAccess(BeanAccess.FIELD);
+        Yaml yaml = new Yaml(rep);
+
+        // SnakeYAML set the type of object on the first line, which can fail to be parsed on synapse
+        // We therefore need to split the resulting string, remove the first line, and then write it
+        List<String> lines = new ArrayList<>(Arrays.asList(yaml.dump(syncCfg).split("\\R+")));
+        if (StringUtils.equals(lines.get(0), "!!" + SynapseRegistrationYaml.class.getCanonicalName())) {
+            lines.remove(0);
+        }
+
+        try (FileOutputStream os = new FileOutputStream(synapseRegFile)) {
+            IOUtils.writeLines(lines, System.lineSeparator(), os, StandardCharsets.UTF_8);
+        } catch (IOException e) {
+            throw new RuntimeException("Unable to write synapse appservice registration file", e);
+        }
+    }
+
+    private void ensureEnabled() {
+        if (!cfg.isEnabled()) {
+            throw new HttpMatrixException(503, "M_NOT_AVAILABLE", "This feature is disabled");
+        }
+    }
+
+    public AppSvcManager withToken(String token) {
+        ensureEnabled();
+
+        if (StringUtils.isBlank(token)) {
+            log.info("Denying request without a HS token");
+            throw new HttpMatrixException(401, "M_UNAUTHORIZED", "No HS token");
+        }
+
+        if (!StringUtils.equals(cfg.getEndpoint().getToAS().getToken(), token)) {
+            log.info("Denying request with an invalid HS token");
+            throw new NotAllowedException("Invalid HS token");
+        }
+
+        return this;
+    }
+
+    public void processUser(String userId) {
+        client.createUser(MatrixID.asAcceptable(userId).getLocalPart());
+    }
+
+    public CompletableFuture<String> processTransaction(String txnId, InputStream is) {
+        ensureEnabled();
+
+        if (StringUtils.isEmpty(txnId)) {
+            throw new IllegalArgumentException("Transaction ID cannot be empty");
+        }
+
+        synchronized (this) {
+            Optional<ASTransactionDao> dao = store.getTransactionResult(cfg.getUser().getMain(), txnId);
+            if (dao.isPresent()) {
+                log.info("AS Transaction {} already processed - returning computed result", txnId);
+                return CompletableFuture.completedFuture(dao.get().getResult());
+            }
+
+            CompletableFuture<String> f = transactionsInProgress.get(txnId);
+            if (Objects.nonNull(f)) {
+                log.info("Returning future for transaction {}", txnId);
+                return f;
+            }
+
+            transactionsInProgress.put(txnId, new CompletableFuture<>());
+        }
+
+        CompletableFuture<String> future = transactionsInProgress.get(txnId);
+
+        Instant start = Instant.now();
+        log.info("Processing AS Transaction {}: start", txnId);
+        try {
+            List<JsonObject> events = GsonUtil.asList(GsonUtil.getArray(parser.parse(is), "events"), JsonObject.class);
+            is.close();
+            log.debug("{} event(s) parsed", events.size());
+
+            processTransaction(events);
+
+            Instant end = Instant.now();
+            String result = "{}";
+
+            try {
+                log.info("Saving transaction details to store");
+                store.insertTransactionResult(cfg.getUser().getMain(), txnId, end, result);
+            } finally {
+                log.debug("Removing CompletedFuture from transaction map");
+                transactionsInProgress.remove(txnId);
+            }
+
+            log.info("Processed AS transaction {} in {} ms", txnId, (Instant.now().toEpochMilli() - start.toEpochMilli()));
+            future.complete(result);
+        } catch (Exception e) {
+            log.error("Unable to properly process transaction {}", txnId, e);
+            future.completeExceptionally(e);
+        }
+
+        log.info("Processing AS Transaction {}: end", txnId);
+        return future;
+    }
+
+    private void processTransaction(List<JsonObject> eventsJson) {
+        log.info("Processing transaction events: start");
+
+        eventsJson.forEach(ev -> {
+            String evId = EventKey.Id.getStringOrNull(ev);
+            if (StringUtils.isBlank(evId)) {
+                log.warn("Event has no ID, skipping");
+                log.debug("Event:\n{}", GsonUtil.getPrettyForLog(ev));
+                return;
+            }
+            log.debug("Event {}: processing start", evId);
+
+            String roomId = EventKey.RoomId.getStringOrNull(ev);
+            if (StringUtils.isBlank(roomId)) {
+                log.debug("Event has no room ID, skipping");
+                return;
+            }
+
+            String senderId = EventKey.Sender.getStringOrNull(ev);
+            if (StringUtils.isBlank(senderId)) {
+                log.debug("Event has no sender ID, skipping");
+                return;
+            }
+            _MatrixID sender = MatrixID.asAcceptable(senderId);
+            log.debug("Sender: {}", senderId);
+
+            String evType = StringUtils.defaultIfBlank(EventKey.Type.getStringOrNull(ev), "<EMPTY/MISSING>");
+            EventTypeProcessor p = processors.get(evType);
+            if (Objects.isNull(p)) {
+                log.debug("No event processor for type {}, skipping", evType);
+                return;
+            }
+
+            p.process(ev, sender, roomId);
+
+            log.debug("Event {}: processing end", evId);
+        });
+
+        log.info("Processing transaction events: end");
+    }
+
+}

src/main/java/io/kamax/mxisd/as/processor/command/CommandProcessor.java

@@ -0,0 +1,32 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.as.processor.command;
+
+import io.kamax.matrix.client._MatrixClient;
+import io.kamax.matrix.hs._MatrixRoom;
+import io.kamax.mxisd.Mxisd;
+import org.apache.commons.cli.CommandLine;
+
+public interface CommandProcessor {
+
+    void process(Mxisd m, _MatrixClient client, _MatrixRoom room, CommandLine cmdLine);
+
+}

src/main/java/io/kamax/mxisd/as/processor/command/InviteCommandProcessor.java

@@ -0,0 +1,117 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.as.processor.command;
+
+import io.kamax.matrix.client._MatrixClient;
+import io.kamax.matrix.hs._MatrixRoom;
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.invitation.IThreePidInviteReply;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang.text.StrBuilder;
+
+import java.util.List;
+
+public class InviteCommandProcessor implements CommandProcessor {
+
+    public static final String Command = "invite";
+
+    @Override
+    public void process(Mxisd m, _MatrixClient client, _MatrixRoom room, CommandLine cmdLine) {
+        if (cmdLine.getArgs().length < 2) {
+            room.sendNotice(buildHelp());
+        } else {
+            String arg = cmdLine.getArgList().get(1);
+            String response;
+            if (StringUtils.equals("list", arg)) {
+
+                StrBuilder b = new StrBuilder();
+
+                List<IThreePidInviteReply> invites = m.getInvite().listInvites();
+                if (invites.isEmpty()) {
+                    b.appendln("No invites!");
+                    response = b.toString();
+                } else {
+                    b.appendln("Invites:");
+
+
+                    for (IThreePidInviteReply invite : invites) {
+                        b.appendNewLine().append("ID: ").append(invite.getId());
+                        b.appendNewLine().append("Room: ").append(invite.getInvite().getRoomId());
+                        b.appendNewLine().append("Medium: ").append(invite.getInvite().getMedium());
+                        b.appendNewLine().append("Address: ").append(invite.getInvite().getAddress());
+                        b.appendNewLine();
+                    }
+
+                    response = b.appendNewLine().append("Total: " + invites.size()).toString();
+                }
+            } else if (StringUtils.equals("show", arg)) {
+                if (cmdLine.getArgList().size() < 3) {
+                    response = buildHelp();
+                } else {
+                    String id = cmdLine.getArgList().get(2);
+                    IThreePidInviteReply invite = m.getInvite().getInvite(id);
+                    StrBuilder b = new StrBuilder();
+                    b.appendln("Details for Invitation #" + id);
+                    b.appendNewLine().append("Room: ").append(invite.getInvite().getRoomId());
+                    b.appendNewLine().append("Sender: ").append(invite.getInvite().getSender().toString());
+                    b.appendNewLine().append("Medium: ").append(invite.getInvite().getMedium());
+                    b.appendNewLine().append("Address: ").append(invite.getInvite().getAddress());
+                    b.appendNewLine().append("Display name: ").append(invite.getDisplayName());
+                    b.appendNewLine().appendNewLine().append("Properties:");
+                    invite.getInvite().getProperties().forEach((k, v) -> {
+                        b.appendNewLine().append("\t").append(k).append("=").append(v);
+                    });
+                    b.appendNewLine();
+
+                    response = b.toString();
+                }
+            } else if (StringUtils.equals("revoke", arg)) {
+                if (cmdLine.getArgList().size() < 3) {
+                    response = buildHelp();
+                } else {
+                    m.getInvite().expireInvite(cmdLine.getArgList().get(2));
+                    response = "OK";
+                }
+            } else {
+                response = buildError("Unknown invite action: " + arg, true);
+            }
+
+            room.sendNotice(response);
+        }
+    }
+
+    private String buildError(String message, boolean showHelp) {
+        if (showHelp) {
+            message = message + "\n\n" + buildHelp();
+        }
+
+        return message;
+    }
+
+    private String buildHelp() {
+        return "Available actions:\n\n" +
+                "list - List invites\n" +
+                "show ID - Show detailed info about a specific invite\n" +
+                "revoke ID - Revoke a pending invite by resolving it to the configured Expiration user\n";
+    }
+
+}

src/main/java/io/kamax/mxisd/as/processor/command/LookupCommandProcessor.java

@@ -0,0 +1,78 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.as.processor.command;
+
+import io.kamax.matrix.client._MatrixClient;
+import io.kamax.matrix.hs._MatrixRoom;
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.lookup.SingleLookupReply;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.lang.text.StrBuilder;
+import org.apache.commons.lang3.StringUtils;
+
+import java.util.Optional;
+
+public class LookupCommandProcessor implements CommandProcessor {
+
+    public static final String Command = "lookup";
+
+    @Override
+    public void process(Mxisd m, _MatrixClient client, _MatrixRoom room, CommandLine cmdLine) {
+        if (cmdLine.getArgList().size() != 3) {
+            room.sendNotice(getUsage());
+            return;
+        }
+
+        String medium = cmdLine.getArgList().get(1);
+        String address = cmdLine.getArgList().get(2);
+        if (StringUtils.isAnyBlank(medium, address)) {
+            room.sendNotice(getUsage());
+            return;
+        }
+
+        room.sendNotice("Processing...");
+        Optional<SingleLookupReply> r = m.getIdentity().find(medium, address, true);
+        if (!r.isPresent()) {
+            room.sendNotice("No result");
+            return;
+        }
+
+        SingleLookupReply lookup = r.get();
+        StrBuilder b = new StrBuilder();
+        b.append("Result for 3PID lookup of ").append(medium).append(" ").appendln(address).appendNewLine();
+        b.append("Matrix ID: ").appendln(lookup.getMxid().getId());
+        b.appendln("Validity:")
+                .append("  Not Before: ").appendln(lookup.getNotBefore())
+                .append("  Not After: ").appendln(lookup.getNotAfter());
+        b.appendln("Signatures:");
+        lookup.getSignatures().forEach((host, signs) -> {
+            b.append("  ").append(host).appendln(":");
+            signs.forEach((key, sign) -> b.append("    ").append(key).append(" -> ").appendln("OK"));
+        });
+
+        room.sendNotice(b.toString());
+    }
+
+    public String getUsage() {
+        return "lookup MEDIUM ADDRESS";
+    }
+
+}

src/main/java/io/kamax/mxisd/as/processor/command/PingCommandProcessor.java

@@ -0,0 +1,37 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.as.processor.command;
+
+import io.kamax.matrix.client._MatrixClient;
+import io.kamax.matrix.hs._MatrixRoom;
+import io.kamax.mxisd.Mxisd;
+import org.apache.commons.cli.CommandLine;
+
+public class PingCommandProcessor implements CommandProcessor {
+
+    public static final String Command = "ping";
+
+    @Override
+    public void process(Mxisd m, _MatrixClient client, _MatrixRoom room, CommandLine cmdLine) {
+        room.sendNotice("Pong!");
+    }
+
+}

src/main/java/io/kamax/mxisd/as/processor/event/EventTypeProcessor.java

@@ -0,0 +1,30 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.as.processor.event;
+
+import com.google.gson.JsonObject;
+import io.kamax.matrix._MatrixID;
+
+public interface EventTypeProcessor {
+
+    void process(JsonObject ev, _MatrixID sender, String roomId);
+
+}

src/main/java/io/kamax/mxisd/as/processor/event/MembershipEventProcessor.java

@@ -0,0 +1,167 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.as.processor.event;
+
+import com.google.gson.JsonObject;
+import io.kamax.matrix.MatrixID;
+import io.kamax.matrix.ThreePidMedium;
+import io.kamax.matrix._MatrixID;
+import io.kamax.matrix._ThreePid;
+import io.kamax.matrix.client.as.MatrixApplicationServiceClient;
+import io.kamax.matrix.event.EventKey;
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.backend.sql.synapse.Synapse;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.invitation.IMatrixIdInvite;
+import io.kamax.mxisd.invitation.MatrixIdInvite;
+import io.kamax.mxisd.notification.NotificationManager;
+import io.kamax.mxisd.profile.ProfileManager;
+import org.apache.commons.lang.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class MembershipEventProcessor implements EventTypeProcessor {
+
+    private final static Logger log = LoggerFactory.getLogger(MembershipEventProcessor.class);
+
+    private MatrixApplicationServiceClient client;
+
+    private final MxisdConfig cfg;
+    private ProfileManager profiler;
+    private NotificationManager notif;
+    private Synapse synapse;
+
+    public MembershipEventProcessor(
+            MatrixApplicationServiceClient client,
+            Mxisd m
+    ) {
+        this.client = client;
+        this.cfg = m.getConfig();
+        this.profiler = m.getProfile();
+        this.notif = m.getNotif();
+        this.synapse = m.getSynapse();
+    }
+
+    @Override
+    public void process(JsonObject ev, _MatrixID sender, String roomId) {
+        JsonObject content = EventKey.Content.findObj(ev).orElseGet(() -> {
+            log.debug("No content found, falling back to full object");
+            return ev;
+        });
+
+        String targetId = EventKey.StateKey.getStringOrNull(ev);
+        if (StringUtils.isBlank(targetId)) {
+            log.warn("Invalid event: No invitee ID, skipping");
+            return;
+        }
+
+        _MatrixID target = MatrixID.asAcceptable(targetId);
+        if (!StringUtils.equals(target.getDomain(), cfg.getMatrix().getDomain())) {
+            log.debug("Ignoring invite for {}: not a local user", targetId);
+            return;
+        }
+
+        log.info("Got membership event from {} to {} for room {}", sender.getId(), targetId, roomId);
+
+        boolean isForMainUser = StringUtils.equals(target.getLocalPart(), cfg.getAppsvc().getUser().getMain());
+        boolean isForExpInvUser = StringUtils.equals(target.getLocalPart(), cfg.getAppsvc().getUser().getInviteExpired());
+
+        if (StringUtils.equals("join", EventKey.Membership.getStringOrNull(content))) {
+            if (isForExpInvUser) {
+                log.warn("We joined the room {} for another identity as the main user, which is not supported. Leaving...", roomId);
+
+                client.getUser(target.getLocalPart()).getRoom(roomId).tryLeave().ifPresent(err -> {
+                    log.warn("Could not decline invite to room {}: {} - {}", roomId, err.getErrcode(), err.getError());
+                });
+            }
+        } else if (StringUtils.equals("invite", EventKey.Membership.getStringOrNull(content))) {
+            if (isForMainUser) {
+                processForMainUser(roomId, sender);
+            } else if (isForExpInvUser) {
+                processForExpiredInviteUser(roomId, target);
+            } else {
+                processForUserIdInvite(roomId, sender, target);
+            }
+        } else if (StringUtils.equals("leave", EventKey.Membership.getStringOrNull(content))) {
+            // TODO we need to find out if this is only us remaining and leave the room if so, using the right client for it
+        } else {
+            log.debug("This is not an supported type of membership event, skipping");
+        }
+    }
+
+    private void processForMainUser(String roomId, _MatrixID sender) {
+        boolean isAllowed = profiler.hasAnyRole(sender, cfg.getAppsvc().getFeature().getAdmin().getAllowedRoles());
+        if (!isAllowed) {
+            log.info("Sender does not have any of the required roles, denying");
+            client.getRoom(roomId).tryLeave().ifPresent(err -> {
+                log.warn("Could not decline invite to room {}: {} - {}", roomId, err.getErrcode(), err.getError());
+            });
+        } else {
+            client.getRoom(roomId).tryJoin().ifPresent(err -> {
+                log.warn("Could not join room {}: {} - {}", roomId, err.getErrcode(), err.getError());
+                client.getRoom(roomId).tryLeave().ifPresent(err1 -> {
+                    log.warn("Could not decline invite to room {} after failed join: {} - {}", roomId, err1.getErrcode(), err1.getError());
+                });
+            });
+        }
+    }
+
+    private void processForExpiredInviteUser(String roomId, _MatrixID invitee) {
+        client.getUser(invitee.getLocalPart()).getRoom(roomId).tryLeave().ifPresent(err -> {
+            log.warn("Could not decline invite to room {}: {} - {}", roomId, err.getErrcode(), err.getError());
+        });
+    }
+
+    private void processForUserIdInvite(String roomId, _MatrixID sender, _MatrixID invitee) {
+        String inviteeId = invitee.getId();
+
+        boolean wasSent = false;
+        List<_ThreePid> tpids = profiler.getThreepids(invitee).stream()
+                .filter(tpid -> ThreePidMedium.Email.is(tpid.getMedium()))
+                .collect(Collectors.toList());
+        log.info("Found {} email(s) in identity store for {}", tpids.size(), inviteeId);
+
+        for (_ThreePid tpid : tpids) {
+            log.info("Found Email to notify about room invitation: {}", tpid.getAddress());
+            Map<String, String> properties = new HashMap<>();
+            profiler.getDisplayName(sender).ifPresent(name -> properties.put("sender_display_name", name));
+            try {
+                synapse.getRoomName(roomId).ifPresent(name -> properties.put("room_name", name));
+            } catch (RuntimeException e) {
+                log.warn("Could not fetch room name", e);
+                log.info("Unable to fetch room name: Did you integrate your Homeserver as documented?");
+            }
+
+            IMatrixIdInvite inv = new MatrixIdInvite(roomId, sender, invitee, tpid.getMedium(), tpid.getAddress(), properties);
+            notif.sendForInvite(inv);
+            log.info("Notification for invite of {} sent to {}", inviteeId, tpid.getAddress());
+            wasSent = true;
+        }
+
+        log.info("Was notification sent? {}", wasSent);
+    }
+
+}

src/main/java/io/kamax/mxisd/as/processor/event/MessageEventProcessor.java

@@ -0,0 +1,127 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.as.processor.event;
+
+import com.google.gson.JsonObject;
+import io.kamax.matrix._MatrixID;
+import io.kamax.matrix._MatrixUserProfile;
+import io.kamax.matrix.client.as.MatrixApplicationServiceClient;
+import io.kamax.matrix.hs._MatrixRoom;
+import io.kamax.matrix.json.event.MatrixJsonRoomMessageEvent;
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.as.processor.command.CommandProcessor;
+import io.kamax.mxisd.as.processor.command.InviteCommandProcessor;
+import io.kamax.mxisd.as.processor.command.LookupCommandProcessor;
+import io.kamax.mxisd.as.processor.command.PingCommandProcessor;
+import org.apache.commons.cli.*;
+import org.apache.commons.lang.text.StrBuilder;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+public class MessageEventProcessor implements EventTypeProcessor {
+
+    private static final Logger log = LoggerFactory.getLogger(MessageEventProcessor.class);
+
+    private final Mxisd m;
+    private final MatrixApplicationServiceClient client;
+    private Map<String, CommandProcessor> processors;
+
+    public MessageEventProcessor(Mxisd m, MatrixApplicationServiceClient client) {
+        this.m = m;
+        this.client = client;
+
+        processors = new HashMap<>();
+        processors.put("?", (m1, client1, room, cmdLine) -> room.sendNotice(getHelp()));
+        processors.put("help", (m1, client1, room, cmdLine) -> room.sendNotice(getHelp()));
+        processors.put(PingCommandProcessor.Command, new PingCommandProcessor());
+        processors.put(InviteCommandProcessor.Command, new InviteCommandProcessor());
+        processors.put(LookupCommandProcessor.Command, new LookupCommandProcessor());
+    }
+
+    @Override
+    public void process(JsonObject ev, _MatrixID sender, String roomId) {
+        MatrixJsonRoomMessageEvent msgEv = new MatrixJsonRoomMessageEvent(ev);
+        if (StringUtils.equals("m.notice", msgEv.getBodyType())) {
+            log.info("Ignoring automated message");
+            return;
+        }
+
+        _MatrixRoom room = client.getRoom(roomId);
+
+        if (!m.getProfile().hasAnyRole(sender, m.getConfig().getAppsvc().getFeature().getAdmin().getAllowedRoles())) {
+            room.sendNotice("You are not allowed to interact with me.");
+            return;
+        }
+
+        List<_MatrixID> joinedUsers = room.getJoinedUsers().stream().map(_MatrixUserProfile::getId).collect(Collectors.toList());
+        boolean joinedWithMainUser = joinedUsers.contains(client.getWhoAmI());
+        boolean isAdminPrivate = joinedWithMainUser && joinedUsers.size() == 2;
+
+        if (!StringUtils.equals("m.text", msgEv.getBodyType())) {
+            log.info("Unsupported message event type: {}", msgEv.getBodyType());
+            return;
+        }
+
+        String command = msgEv.getBody();
+        if (!isAdminPrivate) {
+            if (!StringUtils.startsWith(command, "!" + Mxisd.Name + " ")) {
+                // Not for us
+                return;
+            }
+
+            command = command.substring(("!" + Mxisd.Name + " ").length());
+        }
+
+        try {
+            CommandLineParser p = new DefaultParser();
+            CommandLine cmdLine = p.parse(new Options(), command.split(" ", 0));
+            String cmd = cmdLine.getArgList().get(0);
+
+            CommandProcessor cp = processors.get(cmd);
+            if (Objects.isNull(cp)) {
+                room.sendNotice("Unknown command: " + command + "\n\n" + getHelp());
+            } else {
+                cp.process(m, client, room, cmdLine);
+            }
+        } catch (ParseException e) {
+            room.sendNotice("Invalid input" + "\n\n" + getHelp());
+        } catch (RuntimeException e) {
+            room.sendNotice("Error when running command: " + e.getMessage());
+        }
+    }
+
+    public String getHelp() {
+        StrBuilder builder = new StrBuilder();
+        builder.appendln("Available commands:");
+        for (String cmd : processors.keySet()) {
+            builder.append("\t").appendln(cmd);
+        }
+        return builder.toString();
+    }
+
+}

src/main/java/io/kamax/mxisd/as/registration/SynapseRegistrationYaml.java

@@ -0,0 +1,176 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.as.registration;
+
+import io.kamax.mxisd.config.AppServiceConfig;
+
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+public class SynapseRegistrationYaml {
+
+    public static SynapseRegistrationYaml parse(AppServiceConfig cfg, String domain) {
+        SynapseRegistrationYaml yaml = new SynapseRegistrationYaml();
+
+        yaml.setId(cfg.getRegistration().getSynapse().getId());
+        yaml.setUrl(cfg.getEndpoint().getToAS().getUrl());
+        yaml.setAsToken(cfg.getEndpoint().getToHS().getToken());
+        yaml.setHsToken(cfg.getEndpoint().getToAS().getToken());
+        yaml.setSenderLocalpart(cfg.getUser().getMain());
+
+        if (cfg.getFeature().getCleanExpiredInvite()) {
+            Namespace ns = new Namespace();
+            ns.setExclusive(true);
+            ns.setRegex("@" + cfg.getUser().getInviteExpired() + ":" + domain);
+            yaml.getNamespaces().getUsers().add(ns);
+        }
+
+        if (cfg.getFeature().getInviteById()) {
+            Namespace ns = new Namespace();
+            ns.setExclusive(false);
+            ns.setRegex("@*:" + domain);
+            yaml.getNamespaces().getUsers().add(ns);
+        }
+
+        return yaml;
+    }
+
+    public static class Namespace {
+
+        private String regex;
+        private boolean exclusive;
+
+        public String getRegex() {
+            return regex;
+        }
+
+        public void setRegex(String regex) {
+            this.regex = regex;
+        }
+
+        public boolean isExclusive() {
+            return exclusive;
+        }
+
+        public void setExclusive(boolean exclusive) {
+            this.exclusive = exclusive;
+        }
+
+    }
+
+    public static class Namespaces {
+
+        private List<Namespace> users = new ArrayList<>();
+        private List<Namespace> aliases = new ArrayList<>();
+        private List<Namespace> rooms = new ArrayList<>();
+
+        public List<Namespace> getUsers() {
+            return users;
+        }
+
+        public void setUsers(List<Namespace> users) {
+            this.users = users;
+        }
+
+        public List<Namespace> getAliases() {
+            return aliases;
+        }
+
+        public void setAliases(List<Namespace> aliases) {
+            this.aliases = aliases;
+        }
+
+        public List<Namespace> getRooms() {
+            return rooms;
+        }
+
+        public void setRooms(List<Namespace> rooms) {
+            this.rooms = rooms;
+        }
+
+    }
+
+    private String id;
+    private String url;
+    private String as_token;
+    private String hs_token;
+    private String sender_localpart;
+    private Namespaces namespaces = new Namespaces();
+
+    public String getId() {
+        return id;
+    }
+
+    public void setId(String id) {
+        this.id = id;
+    }
+
+    public String getUrl() {
+        return url;
+    }
+
+    public void setUrl(String url) {
+        this.url = url;
+    }
+
+    public void setUrl(URL url) {
+        if (Objects.isNull(url)) {
+            this.url = null;
+        } else {
+            this.url = url.toString();
+        }
+    }
+
+    public String getAsToken() {
+        return as_token;
+    }
+
+    public void setAsToken(String as_token) {
+        this.as_token = as_token;
+    }
+
+    public String getHsToken() {
+        return hs_token;
+    }
+
+    public void setHsToken(String hs_token) {
+        this.hs_token = hs_token;
+    }
+
+    public String getSenderLocalpart() {
+        return sender_localpart;
+    }
+
+    public void setSenderLocalpart(String sender_localpart) {
+        this.sender_localpart = sender_localpart;
+    }
+
+    public Namespaces getNamespaces() {
+        return namespaces;
+    }
+
+    public void setNamespaces(Namespaces namespaces) {
+        this.namespaces = namespaces;
+    }
+
+}

src/main/java/io/kamax/mxisd/auth/AuthManager.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -20,37 +20,93 @@
 
 package io.kamax.mxisd.auth;
 
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import com.google.gson.JsonSyntaxException;
+import com.google.i18n.phonenumbers.NumberParseException;
+import com.google.i18n.phonenumbers.PhoneNumberUtil;
+import com.google.i18n.phonenumbers.Phonenumber;
 import io.kamax.matrix.MatrixID;
 import io.kamax.matrix.ThreePid;
 import io.kamax.matrix._MatrixID;
 import io.kamax.matrix._ThreePid;
+import io.kamax.matrix.json.GsonUtil;
 import io.kamax.mxisd.UserIdType;
 import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
 import io.kamax.mxisd.auth.provider.BackendAuthResult;
+import io.kamax.mxisd.config.AuthenticationConfig;
 import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.dns.ClientDnsOverwrite;
+import io.kamax.mxisd.exception.RemoteLoginException;
 import io.kamax.mxisd.invitation.InvitationManager;
 import io.kamax.mxisd.lookup.ThreePidMapping;
+import io.kamax.mxisd.lookup.strategy.LookupStrategy;
+import io.kamax.mxisd.util.RestClientUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.util.EntityUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Service;
 
+import java.io.IOException;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Objects;
 
-@Service
 public class AuthManager {
 
-    private Logger log = LoggerFactory.getLogger(AuthManager.class);
+    private static final Logger log = LoggerFactory.getLogger(AuthManager.class);
 
-    @Autowired
-    private List<AuthenticatorProvider> providers = new ArrayList<>();
+    private static final String TypeKey = "type";
+    private static final String UserKey = "user";
+    private static final String IdentifierKey = "identifier";
+    private static final String ThreepidMediumKey = "medium";
+    private static final String ThreepidAddressKey = "address";
+    private static final String UserIdTypeValue = "m.id.user";
+    private static final String ThreepidTypeValue = "m.id.thirdparty";
 
-    @Autowired
+    private final Gson gson = GsonUtil.get(); // FIXME replace
+
+    private List<AuthenticatorProvider> providers;
     private MatrixConfig mxCfg;
-
-    @Autowired
+    private AuthenticationConfig cfg;
     private InvitationManager invMgr;
+    private ClientDnsOverwrite dns;
+    private LookupStrategy strategy;
+    private CloseableHttpClient client;
+
+    public AuthManager(
+            MxisdConfig cfg,
+            List<? extends AuthenticatorProvider> providers,
+            LookupStrategy strategy,
+            InvitationManager invMgr,
+            ClientDnsOverwrite dns,
+            CloseableHttpClient client
+    ) {
+        this.cfg = cfg.getAuth();
+        this.mxCfg = cfg.getMatrix();
+        this.providers = new ArrayList<>(providers);
+        this.strategy = strategy;
+        this.invMgr = invMgr;
+        this.dns = dns;
+        this.client = client;
+    }
+
+    public String resolveProxyUrl(URI target) {
+        URIBuilder builder = dns.transform(target);
+        String urlToLogin = builder.toString();
+        log.info("Proxy resolution: {} to {}", target.toString(), urlToLogin);
+        return urlToLogin;
+    }
 
     public UserAuthResult authenticate(String id, String password) {
         _MatrixID mxid = MatrixID.asAcceptable(id);
@@ -59,9 +115,10 @@ public class AuthManager {
                 continue;
             }
 
+            log.info("Attempting authentication with store {}", provider.getClass().getSimpleName());
+
             BackendAuthResult result = provider.authenticate(mxid, password);
             if (result.isSuccess()) {
-
                 String mxId;
                 if (UserIdType.Localpart.is(result.getId().getType())) {
                     mxId = MatrixID.from(result.getId().getValue(), mxCfg.getDomain()).acceptable().getId();
@@ -82,6 +139,12 @@ public class AuthManager {
                     invMgr.publishMappingIfInvited(new ThreePidMapping(pid, mxId));
                 }
 
+                try {
+                    MatrixID.asValid(mxId);
+                } catch (IllegalArgumentException e) {
+                    log.warn("The returned User ID {} is not a valid Matrix ID. Login might fail at the Homeserver level", mxId);
+                }
+
                 invMgr.lookupMappingsForInvites();
 
                 return authResult;
@@ -91,4 +154,128 @@ public class AuthManager {
         return new UserAuthResult().failure();
     }
 
+    public String proxyLogin(URI target, String body) {
+        JsonObject reqJsonObject = io.kamax.matrix.json.GsonUtil.parseObj(body);
+
+        GsonUtil.findObj(reqJsonObject, IdentifierKey).ifPresent(obj -> {
+            GsonUtil.findString(obj, TypeKey).ifPresent(type -> {
+                if (StringUtils.equals(type, UserIdTypeValue)) {
+                    log.info("Login request is User ID type");
+
+                    if (cfg.getRewrite().getUser().getRules().isEmpty()) {
+                        log.info("No User ID rewrite rules to apply");
+                    } else {
+                        log.info("User ID rewrite rules: checking for a match");
+
+                        String userId = GsonUtil.getStringOrThrow(obj, UserKey);
+                        for (AuthenticationConfig.Rule m : cfg.getRewrite().getUser().getRules()) {
+                            if (m.getPattern().matcher(userId).matches()) {
+                                log.info("Found matching pattern, resolving to 3PID with medium {}", m.getMedium());
+
+                                // Remove deprecated login info on the top object if exists to avoid duplication
+                                reqJsonObject.remove(UserKey);
+                                obj.addProperty(TypeKey, ThreepidTypeValue);
+                                obj.addProperty(ThreepidMediumKey, m.getMedium());
+                                obj.addProperty(ThreepidAddressKey, userId);
+
+                                log.info("Rewrite to 3PID done");
+                            }
+                        }
+
+                        log.info("User ID rewrite rules: done checking rules");
+                    }
+                }
+            });
+        });
+
+        GsonUtil.findObj(reqJsonObject, IdentifierKey).ifPresent(obj -> {
+            GsonUtil.findString(obj, TypeKey).ifPresent(type -> {
+                if (StringUtils.equals(type, ThreepidTypeValue)) {
+                    // Remove deprecated login info if exists to avoid duplication
+                    reqJsonObject.remove(ThreepidMediumKey);
+                    reqJsonObject.remove(ThreepidAddressKey);
+
+                    GsonUtil.findPrimitive(obj, ThreepidMediumKey).ifPresent(medium -> {
+                        GsonUtil.findPrimitive(obj, ThreepidAddressKey).ifPresent(address -> {
+                            log.info("Login request with medium '{}' and address '{}'", medium.getAsString(), address.getAsString());
+                            strategy.findLocal(medium.getAsString(), address.getAsString()).ifPresent(lookupDataOpt -> {
+                                obj.remove(ThreepidMediumKey);
+                                obj.remove(ThreepidAddressKey);
+                                obj.addProperty(TypeKey, UserIdTypeValue);
+                                obj.addProperty(UserKey, lookupDataOpt.getMxid().getLocalPart());
+                            });
+                        });
+                    });
+                }
+
+                if (StringUtils.equals(type, "m.id.phone")) {
+                    // Remove deprecated login info if exists to avoid duplication
+                    reqJsonObject.remove(ThreepidMediumKey);
+                    reqJsonObject.remove(ThreepidAddressKey);
+
+                    GsonUtil.findPrimitive(obj, "number").ifPresent(number -> {
+                        GsonUtil.findPrimitive(obj, "country").ifPresent(country -> {
+                            log.info("Login request with phone '{}'-'{}'", country.getAsString(), number.getAsString());
+                            try {
+                                PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
+                                Phonenumber.PhoneNumber phoneNumber = phoneUtil.parse(number.getAsString(), country.getAsString());
+                                String msisdn = phoneUtil.format(phoneNumber, PhoneNumberUtil.PhoneNumberFormat.E164).replace("+", "");
+                                String medium = "msisdn";
+                                strategy.findLocal(medium, msisdn).ifPresent(lookupDataOpt -> {
+                                    obj.remove("country");
+                                    obj.remove("number");
+                                    obj.addProperty(TypeKey, UserIdTypeValue);
+                                    obj.addProperty(UserKey, lookupDataOpt.getMxid().getLocalPart());
+                                });
+                            } catch (NumberParseException e) {
+                                log.error("Not a valid phone number");
+                                throw new RuntimeException(e);
+                            }
+                        });
+                    });
+                }
+            });
+        });
+
+        // invoke 'login' on homeserver
+        HttpPost httpPost = RestClientUtils.post(resolveProxyUrl(target), gson, reqJsonObject);
+        try (CloseableHttpResponse httpResponse = client.execute(httpPost)) {
+            // check http status
+            int status = httpResponse.getStatusLine().getStatusCode();
+            log.info("http status = {}", status);
+            if (status != 200) {
+                // try to get possible json error message from response
+                // otherwise just get returned plain error message
+                String errcode = String.valueOf(httpResponse.getStatusLine().getStatusCode());
+                String error = EntityUtils.toString(httpResponse.getEntity());
+                if (httpResponse.getEntity() != null) {
+                    try {
+                        JsonObject bodyJson = new JsonParser().parse(error).getAsJsonObject();
+                        if (bodyJson.has("errcode")) {
+                            errcode = bodyJson.get("errcode").getAsString();
+                        }
+                        if (bodyJson.has("error")) {
+                            error = bodyJson.get("error").getAsString();
+                        }
+                        throw new RemoteLoginException(status, errcode, error, bodyJson);
+                    } catch (JsonSyntaxException e) {
+                        log.warn("Response body is not JSON");
+                    }
+                }
+                throw new RemoteLoginException(status, errcode, error);
+            }
+
+            // return response
+            HttpEntity entity = httpResponse.getEntity();
+            if (Objects.isNull(entity)) {
+                log.warn("Expected HS to return data but got nothing");
+                return "";
+            } else {
+                return IOUtils.toString(httpResponse.getEntity().getContent(), StandardCharsets.UTF_8);
+            }
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
 }

src/main/java/io/kamax/mxisd/auth/AuthProviders.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -18,20 +18,25 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.controller.identity.v1.remote;
+package io.kamax.mxisd.auth;
 
-public class RemoteIdentityAPIv1 {
+import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
 
-    public static final String BASE = "/_matrix/identity/remote/api/v1";
-    public static final String SESSION_REQUEST_TOKEN = BASE + "/validate/requestToken";
-    public static final String SESSION_CHECK = BASE + "/validate/check";
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
 
-    public static String getRequestToken(String id, String secret) {
-        return SESSION_REQUEST_TOKEN + "?sid=" + id + "&client_secret=" + secret;
+public class AuthProviders {
+
+    private static final List<Supplier<? extends AuthenticatorProvider>> suppliers = new ArrayList<>();
+
+    public static void register(Supplier<? extends AuthenticatorProvider> supplier) {
+        suppliers.add(supplier);
     }
 
-    public static String getSessionCheck(String id, String secret) {
-        return SESSION_CHECK + "?sid=" + id + "&client_secret=" + secret;
+    public static List<? extends AuthenticatorProvider> get() {
+        return suppliers.stream().map(Supplier::get).collect(Collectors.toList());
     }
 
 }

src/main/java/io/kamax/mxisd/auth/UserAuthResult.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as

src/main/java/io/kamax/mxisd/auth/provider/AuthenticatorProvider.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as

src/main/java/io/kamax/mxisd/auth/provider/BackendAuthResult.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -38,6 +38,10 @@ public class BackendAuthResult {
             return displayName;
         }
 
+        public void setDisplayName(String displayName) {
+            this.displayName = displayName;
+        }
+
         public Set<ThreePid> getThreePids() {
             return threePids;
         }
@@ -73,6 +77,10 @@ public class BackendAuthResult {
     private UserID id;
     private BackendAuthProfile profile = new BackendAuthProfile();
 
+    public void setSuccess(boolean success) {
+        this.success = success;
+    }
+
     public Boolean isSuccess() {
         return success;
     }
@@ -81,6 +89,10 @@ public class BackendAuthResult {
         return id;
     }
 
+    public void setId(UserID id) {
+        this.id = id;
+    }
+
     public BackendAuthProfile getProfile() {
         return profile;
     }

src/main/java/io/kamax/mxisd/backend/IdentityStoreSupplier.java

@@ -0,0 +1,29 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend;
+
+import io.kamax.mxisd.Mxisd;
+
+import java.util.function.Consumer;
+
+public interface IdentityStoreSupplier extends Consumer<Mxisd> {
+
+}

src/main/java/io/kamax/mxisd/backend/exec/ExecAuthResult.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2018 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -18,16 +18,20 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd;
+package io.kamax.mxisd.backend.exec;
 
-import org.springframework.boot.SpringApplication;
-import org.springframework.boot.autoconfigure.SpringBootApplication;
+import io.kamax.mxisd.auth.provider.BackendAuthResult;
 
-@SpringBootApplication
-public class MatrixIdentityServerApplication {
+public class ExecAuthResult extends BackendAuthResult {
 
-    public static void main(String[] args) {
-        SpringApplication.run(MatrixIdentityServerApplication.class, args);
+    private int exitStatus;
+
+    public int getExitStatus() {
+        return exitStatus;
+    }
+
+    public void setExitStatus(int exitStatus) {
+        this.exitStatus = exitStatus;
     }
 
 }

src/main/java/io/kamax/mxisd/backend/exec/ExecAuthStore.java

@@ -0,0 +1,126 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.exec;
+
+import com.google.gson.JsonObject;
+import com.google.gson.JsonPrimitive;
+import io.kamax.matrix._MatrixID;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.UserID;
+import io.kamax.mxisd.UserIdType;
+import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
+import io.kamax.mxisd.backend.rest.RestAuthRequestJson;
+import io.kamax.mxisd.config.ExecConfig;
+import io.kamax.mxisd.exception.InternalServerError;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Objects;
+import java.util.Optional;
+
+public class ExecAuthStore extends ExecStore implements AuthenticatorProvider {
+
+    private transient final Logger log = LoggerFactory.getLogger(ExecAuthStore.class);
+
+    private ExecConfig.Auth cfg;
+
+    public ExecAuthStore(ExecConfig cfg) {
+        super(cfg);
+        this.cfg = Objects.requireNonNull(cfg.getAuth());
+    }
+
+    @Override
+    public boolean isEnabled() {
+        return cfg.isEnabled();
+    }
+
+    @Override
+    public ExecAuthResult authenticate(_MatrixID uId, String password) {
+        Objects.requireNonNull(uId);
+        Objects.requireNonNull(password);
+
+        log.info("Performing authentication for {}", uId.getId());
+
+        ExecAuthResult result = new ExecAuthResult();
+        result.setId(new UserID(UserIdType.Localpart, uId.getLocalPart()));
+
+        Processor<ExecAuthResult> p = new Processor<>(cfg);
+
+        p.addTokenMapper(cfg.getToken().getLocalpart(), uId::getLocalPart);
+        p.addTokenMapper(cfg.getToken().getDomain(), uId::getDomain);
+        p.addTokenMapper(cfg.getToken().getMxid(), uId::getId);
+        p.addTokenMapper(cfg.getToken().getPassword(), () -> password);
+
+        p.addJsonInputTemplate(tokens -> {
+            RestAuthRequestJson json = new RestAuthRequestJson();
+            json.setLocalpart(tokens.getLocalpart());
+            json.setDomain(tokens.getDomain());
+            json.setMxid(tokens.getMxid());
+            json.setPassword(tokens.getPassword());
+            return json;
+        });
+        p.addInputTemplate(PlainType, tokens -> tokens.getLocalpart() + System.lineSeparator() +
+                tokens.getDomain() + System.lineSeparator() +
+                tokens.getMxid() + System.lineSeparator() +
+                tokens.getPassword() + System.lineSeparator()
+        );
+
+        p.withExitHandler(pr -> result.setExitStatus(pr.getExitValue()));
+
+        p.withSuccessHandler(pr -> result.setSuccess(true));
+        p.withSuccessDefault(o -> result);
+        p.addSuccessMapper(JsonType, output -> {
+            JsonObject data = GsonUtil.getObj(GsonUtil.parseObj(output), "auth");
+            GsonUtil.findPrimitive(data, "success")
+                    .map(JsonPrimitive::getAsBoolean)
+                    .ifPresent(result::setSuccess);
+            GsonUtil.findObj(data, "profile")
+                    .flatMap(profile -> GsonUtil.findString(profile, "display_name"))
+                    .ifPresent(v -> result.getProfile().setDisplayName(v));
+
+            return result;
+        });
+        p.addSuccessMapper(PlainType, output -> {
+            String[] lines = output.split("\\R");
+            if (lines.length > 2) {
+                throw new InternalServerError("Exec auth command returned more than 2 lines (" + lines.length + ")");
+            }
+
+            result.setSuccess(Optional.ofNullable(StringUtils.isEmpty(lines[0]) ? null : lines[0])
+                    .map(v -> StringUtils.equalsAnyIgnoreCase(v, "true", "1"))
+                    .orElse(result.isSuccess()));
+
+            if (lines.length == 2) {
+                Optional.ofNullable(StringUtils.isEmpty(lines[1]) ? null : lines[1])
+                        .ifPresent(v -> result.getProfile().setDisplayName(v));
+            }
+
+            return result;
+        });
+
+        p.withFailureHandler(pr -> result.setSuccess(false));
+        p.withFailureDefault(o -> result);
+
+        return p.execute();
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/exec/ExecDirectoryStore.java

@@ -0,0 +1,87 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.exec;
+
+import io.kamax.matrix.MatrixID;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.config.ExecConfig;
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.directory.DirectoryProvider;
+import io.kamax.mxisd.http.io.UserDirectorySearchRequest;
+import io.kamax.mxisd.http.io.UserDirectorySearchResult;
+import org.apache.commons.lang3.StringUtils;
+
+public class ExecDirectoryStore extends ExecStore implements DirectoryProvider {
+
+    private ExecConfig.Directory cfg;
+    private MatrixConfig mxCfg;
+
+    public ExecDirectoryStore(MxisdConfig cfg) {
+        this(cfg.getExec(), cfg.getMatrix());
+    }
+
+    public ExecDirectoryStore(ExecConfig cfg, MatrixConfig mxCfg) {
+        super(cfg);
+        this.cfg = cfg.getDirectory();
+        this.mxCfg = mxCfg;
+    }
+
+    private UserDirectorySearchResult search(ExecConfig.Process cfg, UserDirectorySearchRequest request) {
+        if (StringUtils.isEmpty(cfg.getCommand())) {
+            return UserDirectorySearchResult.empty();
+        }
+
+        Processor<UserDirectorySearchResult> p = new Processor<>(cfg);
+
+        p.addJsonInputTemplate(tokens -> new UserDirectorySearchRequest(tokens.getType(), tokens.getQuery()));
+        p.addInputTemplate(PlainType, tokens -> tokens.getType() + System.lineSeparator() + tokens.getQuery());
+
+        p.addTokenMapper(cfg.getToken().getType(), request::getBy);
+        p.addTokenMapper(cfg.getToken().getQuery(), request::getSearchTerm);
+
+        p.addSuccessMapper(JsonType, output -> {
+            if (StringUtils.isBlank(output)) {
+                return UserDirectorySearchResult.empty();
+            }
+
+            UserDirectorySearchResult response = GsonUtil.get().fromJson(output, UserDirectorySearchResult.class);
+            for (UserDirectorySearchResult.Result result : response.getResults()) {
+                result.setUserId(MatrixID.asAcceptable(result.getUserId(), mxCfg.getDomain()).getId());
+            }
+            return response;
+        });
+        p.withFailureDefault(output -> new UserDirectorySearchResult());
+
+        return p.execute();
+    }
+
+    @Override
+    public UserDirectorySearchResult searchByDisplayName(String query) {
+        return search(cfg.getSearch().getByName(), new UserDirectorySearchRequest("name", query));
+    }
+
+    @Override
+    public UserDirectorySearchResult searchBy3pid(String query) {
+        return search(cfg.getSearch().getByName(), new UserDirectorySearchRequest("threepid", query));
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/exec/ExecIdentityStore.java

@@ -0,0 +1,197 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.exec;
+
+import com.google.gson.JsonArray;
+import com.google.gson.JsonParseException;
+import io.kamax.matrix.MatrixID;
+import io.kamax.matrix.ThreePid;
+import io.kamax.matrix._MatrixID;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.UserID;
+import io.kamax.mxisd.UserIdType;
+import io.kamax.mxisd.backend.rest.LookupBulkResponseJson;
+import io.kamax.mxisd.backend.rest.LookupSingleResponseJson;
+import io.kamax.mxisd.config.ExecConfig;
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.exception.InternalServerError;
+import io.kamax.mxisd.lookup.SingleLookupReply;
+import io.kamax.mxisd.lookup.SingleLookupRequest;
+import io.kamax.mxisd.lookup.ThreePidMapping;
+import io.kamax.mxisd.lookup.provider.IThreePidProvider;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+public class ExecIdentityStore extends ExecStore implements IThreePidProvider {
+
+    private transient final Logger log = LoggerFactory.getLogger(ExecIdentityStore.class);
+
+    private final ExecConfig.Identity cfg;
+    private final MatrixConfig mxCfg;
+
+    public ExecIdentityStore(ExecConfig cfg, MatrixConfig mxCfg) {
+        super(cfg);
+        this.cfg = cfg.getIdentity();
+        this.mxCfg = mxCfg;
+    }
+
+    @Override
+    public boolean isLocal() {
+        return true;
+    }
+
+    @Override
+    public int getPriority() {
+        return cfg.getPriority();
+    }
+
+    private ExecConfig.Process getSingleCfg() {
+        return cfg.getLookup().getSingle();
+    }
+
+    private _MatrixID getUserId(UserID id) {
+        if (Objects.isNull(id)) {
+            throw new JsonParseException("User id key is not present");
+        }
+
+        if (UserIdType.Localpart.is(id.getType())) {
+            return MatrixID.asAcceptable(id.getValue(), mxCfg.getDomain());
+        }
+
+        if (UserIdType.MatrixID.is(id.getType())) {
+            return MatrixID.asAcceptable(id.getValue());
+        }
+
+        throw new InternalServerError("Unknown user type: " + id.getType());
+    }
+
+    @Override
+    public Optional<SingleLookupReply> find(SingleLookupRequest request) {
+        Processor<Optional<SingleLookupReply>> p = new Processor<>();
+        p.withConfig(cfg.getLookup().getSingle());
+
+        p.addTokenMapper(getSingleCfg().getToken().getMedium(), request::getType);
+        p.addTokenMapper(getSingleCfg().getToken().getAddress(), request::getThreePid);
+
+        p.addJsonInputTemplate(tokens -> new ThreePid(tokens.getMedium(), tokens.getAddress()));
+        p.addInputTemplate(PlainType, tokens -> tokens.getMedium()
+                + System.lineSeparator()
+                + tokens.getAddress()
+        );
+
+        p.addSuccessMapper(JsonType, output -> {
+            if (StringUtils.isBlank(output)) {
+                return Optional.empty();
+            }
+
+            return GsonUtil.findObj(GsonUtil.parseObj(output), "lookup")
+                    .filter(obj -> !obj.entrySet().isEmpty())
+                    .map(json -> GsonUtil.get().fromJson(json, LookupSingleResponseJson.class))
+                    .map(lookup -> getUserId(lookup.getId()))
+                    .map(mxId -> new SingleLookupReply(request, mxId));
+        });
+
+        p.addSuccessMapper(PlainType, output -> {
+            String[] lines = output.split("\\R");
+            if (lines.length > 2) {
+                throw new InternalServerError("Exec auth command returned more than 2 lines (" + lines.length + ")");
+            }
+
+            if (lines.length == 1 && StringUtils.isBlank(lines[0])) {
+                return Optional.empty();
+            }
+
+            String type = StringUtils.trimToEmpty(lines.length == 1 ? UserIdType.Localpart.getId() : lines[0]);
+            String value = StringUtils.trimToEmpty(lines.length == 2 ? lines[1] : lines[0]);
+
+            if (UserIdType.Localpart.is(type)) {
+                return Optional.of(new SingleLookupReply(request, MatrixID.asAcceptable(value, mxCfg.getDomain())));
+            }
+
+            if (UserIdType.MatrixID.is(type)) {
+                return Optional.of(new SingleLookupReply(request, MatrixID.asAcceptable(value)));
+            }
+
+            throw new InternalServerError("Invalid user type: " + type);
+        });
+
+        p.withFailureDefault(o -> Optional.empty());
+
+        return p.execute();
+    }
+
+    @Override
+    public List<ThreePidMapping> populate(List<ThreePidMapping> mappings) {
+        Processor<List<ThreePidMapping>> p = new Processor<>();
+        p.withConfig(cfg.getLookup().getBulk());
+
+        p.addInput(JsonType, () -> {
+            JsonArray tpids = GsonUtil.asArray(mappings.stream()
+                    .map(mapping -> GsonUtil.get().toJsonTree(new ThreePid(mapping.getMedium(), mapping.getValue())))
+                    .collect(Collectors.toList()));
+            return GsonUtil.get().toJson(GsonUtil.makeObj("lookup", tpids));
+        });
+        p.addInput(PlainType, () -> {
+            StringBuilder input = new StringBuilder();
+            for (ThreePidMapping mapping : mappings) {
+                input.append(mapping.getMedium()).append("\t").append(mapping.getValue()).append(System.lineSeparator());
+            }
+            return input.toString();
+        });
+
+        p.addSuccessMapper(JsonType, output -> {
+            if (StringUtils.isBlank(output)) {
+                return Collections.emptyList();
+            }
+
+            LookupBulkResponseJson response = GsonUtil.get().fromJson(output, LookupBulkResponseJson.class);
+            return response.getLookup().stream().map(item -> {
+                ThreePidMapping mapping = new ThreePidMapping();
+                mapping.setMedium(item.getMedium());
+                mapping.setValue(item.getAddress());
+
+                if (UserIdType.Localpart.is(item.getId().getType())) {
+                    mapping.setValue(MatrixID.asAcceptable(item.getId().getValue(), mxCfg.getDomain()).getId());
+                    return mapping;
+                }
+
+                if (UserIdType.MatrixID.is(item.getId().getType())) {
+                    mapping.setValue(MatrixID.asAcceptable(item.getId().getValue()).getId());
+                    return mapping;
+                }
+
+                throw new InternalServerError("Invalid user type: " + item.getId().getType());
+            }).collect(Collectors.toList());
+        });
+
+        p.withFailureDefault(output -> Collections.emptyList());
+
+        return p.execute();
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/exec/ExecIdentityStoreSupplier.java

@@ -0,0 +1,56 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.exec;
+
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.auth.AuthProviders;
+import io.kamax.mxisd.backend.IdentityStoreSupplier;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.directory.DirectoryProviders;
+import io.kamax.mxisd.lookup.ThreePidProviders;
+import io.kamax.mxisd.profile.ProfileProviders;
+
+public class ExecIdentityStoreSupplier implements IdentityStoreSupplier {
+
+    @Override
+    public void accept(Mxisd mxisd) {
+        accept(mxisd.getConfig());
+    }
+
+    public void accept(MxisdConfig cfg) {
+        if (cfg.getExec().getAuth().isEnabled()) {
+            AuthProviders.register(() -> new ExecAuthStore(cfg.getExec()));
+        }
+
+        if (cfg.getExec().getDirectory().isEnabled()) {
+            DirectoryProviders.register(() -> new ExecDirectoryStore(cfg));
+        }
+
+        if (cfg.getExec().getIdentity().isEnabled()) {
+            ThreePidProviders.register(() -> new ExecIdentityStore(cfg.getExec(), cfg.getMatrix()));
+        }
+
+        if (cfg.getExec().getProfile().isEnabled()) {
+            ProfileProviders.register(() -> new ExecProfileStore(cfg.getExec()));
+        }
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/exec/ExecProfileStore.java

@@ -0,0 +1,91 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.exec;
+
+import io.kamax.matrix._MatrixID;
+import io.kamax.matrix._ThreePid;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.config.ExecConfig;
+import io.kamax.mxisd.profile.JsonProfileRequest;
+import io.kamax.mxisd.profile.JsonProfileResult;
+import io.kamax.mxisd.profile.ProfileProvider;
+import org.apache.commons.lang.StringUtils;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+public class ExecProfileStore extends ExecStore implements ProfileProvider {
+
+    private ExecConfig.Profile cfg;
+
+    public ExecProfileStore(ExecConfig cfg) {
+        super(cfg);
+        this.cfg = cfg.getProfile();
+    }
+
+    private Optional<JsonProfileResult> getFull(_MatrixID userId, ExecConfig.Process cfg) {
+        Processor<Optional<JsonProfileResult>> p = new Processor<>(cfg);
+
+        p.addJsonInputTemplate(tokens -> new JsonProfileRequest(tokens.getLocalpart(), tokens.getDomain(), tokens.getMxid()));
+        p.addInputTemplate(PlainType, tokens -> tokens.getLocalpart() + System.lineSeparator()
+                + tokens.getDomain() + System.lineSeparator()
+                + tokens.getMxid() + System.lineSeparator()
+        );
+
+        p.addTokenMapper(cfg.getToken().getLocalpart(), userId::getLocalPart);
+        p.addTokenMapper(cfg.getToken().getDomain(), userId::getDomain);
+        p.addTokenMapper(cfg.getToken().getMxid(), userId::getId);
+
+        p.withFailureDefault(v -> Optional.empty());
+
+        p.addSuccessMapper(JsonType, output -> {
+            if (StringUtils.isBlank(output)) {
+                return Optional.empty();
+            }
+
+            return GsonUtil.findObj(GsonUtil.parseObj(output), "profile")
+                    .map(obj -> GsonUtil.get().fromJson(obj, JsonProfileResult.class));
+        });
+
+        return p.execute();
+    }
+
+    @Override
+    public Optional<String> getDisplayName(_MatrixID userId) {
+        return getFull(userId, cfg.getDisplayName()).map(JsonProfileResult::getDisplayName);
+    }
+
+    @Override
+    public List<_ThreePid> getThreepids(_MatrixID userId) {
+        return getFull(userId, cfg.getThreePid())
+                .map(p -> Collections.<_ThreePid>unmodifiableList(p.getThreepids()))
+                .orElseGet(Collections::emptyList);
+    }
+
+    @Override
+    public List<String> getRoles(_MatrixID userId) {
+        return getFull(userId, cfg.getRole())
+                .map(JsonProfileResult::getRoles)
+                .orElseGet(Collections::emptyList);
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/exec/ExecStore.java

@@ -0,0 +1,257 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.exec;
+
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.config.ExecConfig;
+import io.kamax.mxisd.exception.InternalServerError;
+import org.apache.commons.io.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.zeroturnaround.exec.ProcessExecutor;
+import org.zeroturnaround.exec.ProcessResult;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+public class ExecStore {
+
+    public static final String JsonType = "json";
+    public static final String PlainType = "plain";
+
+    private static final Logger log = LoggerFactory.getLogger(ExecStore.class);
+
+    protected static String toJson(Object o) {
+        return GsonUtil.get().toJson(o);
+    }
+
+    private final ExecConfig cfg;
+    private Supplier<ProcessExecutor> executorSupplier = () -> new ProcessExecutor().readOutput(true);
+
+    public ExecStore(ExecConfig cfg) {
+        this.cfg = cfg;
+    }
+
+    public void setExecutorSupplier(Supplier<ProcessExecutor> supplier) {
+        executorSupplier = supplier;
+    }
+
+    public class Processor<V> {
+
+        private ExecConfig.Process cfg;
+
+        private Supplier<Optional<String>> inputSupplier;
+        private Function<String, String> inputTypeMapper;
+        private Function<String, String> inputUnknownTypeMapper;
+        private Map<String, Supplier<String>> inputTypeSuppliers;
+
+        private Map<String, Function<ExecConfig.Token, String>> inputTypeTemplates;
+        private Supplier<String> inputTypeNoTemplateHandler;
+        private Map<String, Supplier<String>> tokenMappers;
+        private Function<String, String> tokenHandler;
+
+        private Consumer<ProcessResult> onExitHandler;
+        private Consumer<ProcessResult> successHandler;
+        private Map<String, Function<String, V>> successMappers;
+        private Function<String, V> successDefault;
+        private Consumer<ProcessResult> failureHandler;
+        private Map<String, Function<String, V>> failureMappers;
+        private Function<String, V> failureDefault;
+        private Consumer<ProcessResult> unknownHandler;
+        private Map<String, Function<String, V>> unknownMappers;
+        private Function<String, V> unknownDefault;
+
+        public Processor(ExecConfig.Process cfg) {
+            this();
+            withConfig(cfg);
+        }
+
+        public Processor() {
+            tokenMappers = new HashMap<>();
+            inputTypeSuppliers = new HashMap<>();
+            inputTypeTemplates = new HashMap<>();
+
+            withTokenHandler(tokenHandler = input -> {
+                for (Map.Entry<String, Supplier<String>> entry : tokenMappers.entrySet()) {
+                    input = input.replace(entry.getKey(), entry.getValue().get());
+                }
+                return input;
+            });
+
+            inputTypeNoTemplateHandler = () -> cfg.getInput().getType()
+                    .map(type -> inputTypeTemplates.get(type).apply(cfg.getToken()))
+                    .orElse("");
+
+            inputUnknownTypeMapper = type -> tokenHandler.apply(cfg.getInput().getTemplate().orElseGet(inputTypeNoTemplateHandler));
+
+            inputTypeMapper = type -> {
+                if (!inputTypeSuppliers.containsKey(type)) {
+                    return inputUnknownTypeMapper.apply(type);
+                }
+
+                return inputTypeSuppliers.get(type).get();
+            };
+
+            inputSupplier = () -> cfg.getInput().getType().map(type -> inputTypeMapper.apply(type));
+
+            withExitHandler(pr -> {
+            });
+
+            successHandler = pr -> {
+            };
+            successMappers = new HashMap<>();
+            successDefault = output -> {
+                log.info("{} stdout: {}{}", cfg.getCommand(), System.lineSeparator(), output);
+                throw new InternalServerError("Exec command has no success handler configured. This is a bug. Please report.");
+            };
+
+            failureHandler = pr -> {
+            };
+            failureMappers = new HashMap<>();
+            failureDefault = output -> {
+                log.info("{} stdout: {}{}", cfg.getCommand(), System.lineSeparator(), output);
+                throw new InternalServerError("Exec command has no failure handler configured. This is a bug. Please report.");
+            };
+
+            unknownHandler = pr -> log.warn("Unexpected exit status: {}", pr.getExitValue());
+            unknownMappers = new HashMap<>();
+            withUnknownDefault(output -> {
+                log.error("{} stdout:{}{}", cfg.getCommand(), System.lineSeparator(), output);
+                throw new InternalServerError("Exec command returned with unexpected exit status");
+            });
+        }
+
+        public void withConfig(ExecConfig.Process cfg) {
+            this.cfg = cfg;
+        }
+
+        public void addTokenMapper(String token, Supplier<String> data) {
+            tokenMappers.put(token, data);
+        }
+
+        public void withTokenHandler(Function<String, String> handler) {
+            tokenHandler = handler;
+        }
+
+        public void addInput(String type, Supplier<String> handler) {
+            inputTypeSuppliers.put(type, handler);
+        }
+
+        protected void addInputTemplate(String type, Function<ExecConfig.Token, String> template) {
+            inputTypeTemplates.put(type, template);
+        }
+
+        public void addJsonInputTemplate(Function<ExecConfig.Token, Object> template) {
+            inputTypeTemplates.put(JsonType, token -> GsonUtil.get().toJson(template.apply(token)));
+        }
+
+        public void withExitHandler(Consumer<ProcessResult> handler) {
+            onExitHandler = handler;
+        }
+
+        public void withSuccessHandler(Consumer<ProcessResult> handler) {
+            successHandler = handler;
+        }
+
+        public void addSuccessMapper(String type, Function<String, V> mapper) {
+            successMappers.put(type, mapper);
+        }
+
+        public void withSuccessDefault(Function<String, V> mapper) {
+            successDefault = mapper;
+        }
+
+        public void withFailureHandler(Consumer<ProcessResult> handler) {
+            failureHandler = handler;
+        }
+
+        public void addFailureMapper(String type, Function<String, V> mapper) {
+            failureMappers.put(type, mapper);
+        }
+
+        public void withFailureDefault(Function<String, V> mapper) {
+            failureDefault = mapper;
+        }
+
+        public void addUnknownMapper(String type, Function<String, V> mapper) {
+            unknownMappers.put(type, mapper);
+        }
+
+        public void withUnknownDefault(Function<String, V> mapper) {
+            unknownDefault = mapper;
+        }
+
+        public V execute() {
+            log.info("Executing {}", cfg.getCommand());
+
+            try {
+                ProcessExecutor psExec = executorSupplier.get();
+
+                List<String> args = new ArrayList<>();
+                args.add(tokenHandler.apply(cfg.getCommand()));
+                args.addAll(cfg.getArgs().stream().map(arg -> tokenHandler.apply(arg)).collect(Collectors.toList()));
+                psExec.command(args);
+
+                psExec.environment(new HashMap<>(cfg.getEnv()).entrySet().stream()
+                        .peek(e -> e.setValue(tokenHandler.apply(e.getValue())))
+                        .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)));
+
+                inputSupplier.get().ifPresent(input -> psExec.redirectInput(IOUtils.toInputStream(input, StandardCharsets.UTF_8)));
+
+                ProcessResult psResult = psExec.execute();
+                String output = psResult.outputUTF8();
+                onExitHandler.accept(psResult);
+
+                if (cfg.getExit().getSuccess().contains(psResult.getExitValue())) {
+                    successHandler.accept(psResult);
+
+                    return cfg.getOutput().getType()
+                            .map(type -> successMappers.getOrDefault(type, successDefault).apply(output))
+                            .orElseGet(() -> successDefault.apply(output));
+                } else if (cfg.getExit().getFailure().contains(psResult.getExitValue())) {
+                    failureHandler.accept(psResult);
+
+                    return cfg.getOutput().getType()
+                            .map(type -> failureMappers.getOrDefault(type, failureDefault).apply(output))
+                            .orElseGet(() -> failureDefault.apply(output));
+                } else {
+                    unknownHandler.accept(psResult);
+
+                    return cfg.getOutput().getType()
+                            .map(type -> unknownMappers.getOrDefault(type, unknownDefault).apply(output))
+                            .orElseGet(() -> unknownDefault.apply(output));
+                }
+            } catch (RuntimeException | IOException | InterruptedException | TimeoutException e) {
+                log.error("Failed to execute {}", cfg.getCommand());
+                log.debug("Internal exception:", e);
+                throw new InternalServerError(e);
+            }
+        }
+
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/firebase/GoogleFirebaseAuthenticator.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -29,6 +29,7 @@ import io.kamax.matrix._MatrixID;
 import io.kamax.mxisd.UserIdType;
 import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
 import io.kamax.mxisd.auth.provider.BackendAuthResult;
+import io.kamax.mxisd.config.FirebaseConfig;
 import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -38,10 +39,14 @@ import java.util.concurrent.TimeUnit;
 
 public class GoogleFirebaseAuthenticator extends GoogleFirebaseBackend implements AuthenticatorProvider {
 
-    private Logger log = LoggerFactory.getLogger(GoogleFirebaseAuthenticator.class);
+    private transient final Logger log = LoggerFactory.getLogger(GoogleFirebaseAuthenticator.class);
 
     private PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
 
+    public GoogleFirebaseAuthenticator(FirebaseConfig cfg) {
+        this(cfg.isEnabled(), cfg.getCredentials(), cfg.getDatabase());
+    }
+
     public GoogleFirebaseAuthenticator(boolean isEnabled, String credsPath, String db) {
         super(isEnabled, "AuthenticationProvider", credsPath, db);
     }

src/main/java/io/kamax/mxisd/backend/firebase/GoogleFirebaseBackend.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -35,7 +35,7 @@ import java.io.IOException;
 
 public class GoogleFirebaseBackend {
 
-    private Logger log = LoggerFactory.getLogger(GoogleFirebaseBackend.class);
+    private transient final Logger log = LoggerFactory.getLogger(GoogleFirebaseBackend.class);
 
     private boolean isEnabled;
     private FirebaseAuth fbAuth;
@@ -60,7 +60,9 @@ public class GoogleFirebaseBackend {
 
     private FirebaseCredential getCreds(String credsPath) throws IOException {
         if (StringUtils.isNotBlank(credsPath)) {
-            return FirebaseCredentials.fromCertificate(new FileInputStream(credsPath));
+            try (FileInputStream is = new FileInputStream(credsPath)) {
+                return FirebaseCredentials.fromCertificate(is);
+            }
         } else {
             return FirebaseCredentials.applicationDefault();
         }

src/main/java/io/kamax/mxisd/backend/firebase/GoogleFirebaseProvider.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -25,6 +25,7 @@ import com.google.firebase.tasks.OnFailureListener;
 import com.google.firebase.tasks.OnSuccessListener;
 import io.kamax.matrix.MatrixID;
 import io.kamax.matrix.ThreePidMedium;
+import io.kamax.mxisd.config.MxisdConfig;
 import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.SingleLookupRequest;
 import io.kamax.mxisd.lookup.ThreePidMapping;
@@ -40,16 +41,20 @@ import java.util.concurrent.TimeUnit;
 
 public class GoogleFirebaseProvider extends GoogleFirebaseBackend implements IThreePidProvider {
 
-    private Logger log = LoggerFactory.getLogger(GoogleFirebaseProvider.class);
+    private transient final Logger log = LoggerFactory.getLogger(GoogleFirebaseProvider.class);
     private String domain;
 
+    public GoogleFirebaseProvider(MxisdConfig cfg) {
+        this(cfg.getFirebase().isEnabled(), cfg.getFirebase().getCredentials(), cfg.getFirebase().getDatabase(), cfg.getMatrix().getDomain());
+    }
+
     public GoogleFirebaseProvider(boolean isEnabled, String credsPath, String db, String domain) {
         super(isEnabled, "ThreePidProvider", credsPath, db);
         this.domain = domain;
     }
 
     private String getMxid(UserRecord record) {
-        return new MatrixID(record.getUid(), domain).getId();
+        return MatrixID.asAcceptable(record.getUid(), domain).getId();
     }
 
     @Override

src/main/java/io/kamax/mxisd/backend/firebase/GoogleFirebaseStoreSupplier.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2018 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -18,21 +18,26 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */
 
-package io.kamax.mxisd.spring;
+package io.kamax.mxisd.backend.firebase;
 
-import io.kamax.mxisd.exception.ConfigurationException;
-import org.springframework.boot.diagnostics.AbstractFailureAnalyzer;
-import org.springframework.boot.diagnostics.FailureAnalysis;
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.auth.AuthProviders;
+import io.kamax.mxisd.backend.IdentityStoreSupplier;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.lookup.ThreePidProviders;
 
-public class ConfigurationFailureAnalyzer extends AbstractFailureAnalyzer<ConfigurationException> {
+public class GoogleFirebaseStoreSupplier implements IdentityStoreSupplier {
 
     @Override
-    protected FailureAnalysis analyze(Throwable rootFailure, ConfigurationException cause) {
-        String message = cause.getMessage();
-        if (cause.getDetailedMessage().isPresent()) {
-            message += " - " + cause.getDetailedMessage().get();
+    public void accept(Mxisd mxisd) {
+        accept(mxisd.getConfig());
+    }
+
+    public void accept(MxisdConfig cfg) {
+        if (cfg.getFirebase().isEnabled()) {
+            AuthProviders.register(() -> new GoogleFirebaseAuthenticator(cfg.getFirebase()));
+            ThreePidProviders.register(() -> new GoogleFirebaseProvider(cfg));
         }
-        return new FailureAnalysis(message, "Double check the key value", cause);
     }
 
 }

src/main/java/io/kamax/mxisd/backend/ldap/LdapAuthProvider.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -30,6 +30,7 @@ import io.kamax.mxisd.auth.provider.AuthenticatorProvider;
 import io.kamax.mxisd.auth.provider.BackendAuthResult;
 import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.config.ldap.LdapConfig;
+import io.kamax.mxisd.exception.InternalServerError;
 import io.kamax.mxisd.util.GsonUtil;
 import org.apache.commons.lang.StringUtils;
 import org.apache.directory.api.ldap.model.cursor.CursorException;
@@ -42,8 +43,6 @@ import org.apache.directory.api.ldap.model.message.SearchScope;
 import org.apache.directory.ldap.client.api.LdapConnection;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Component;
 
 import java.io.IOException;
 import java.util.HashSet;
@@ -51,14 +50,12 @@ import java.util.List;
 import java.util.Optional;
 import java.util.Set;
 
-@Component
 public class LdapAuthProvider extends LdapBackend implements AuthenticatorProvider {
 
-    private Logger log = LoggerFactory.getLogger(LdapAuthProvider.class);
+    private transient final Logger log = LoggerFactory.getLogger(LdapAuthProvider.class);
 
     private PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();
 
-    @Autowired
     public LdapAuthProvider(LdapConfig cfg, MatrixConfig mxCfg) {
         super(cfg, mxCfg);
     }
@@ -87,7 +84,6 @@ public class LdapAuthProvider extends LdapBackend implements AuthenticatorProvid
     public BackendAuthResult authenticate(_MatrixID mxid, String password) {
         log.info("Performing auth for {}", mxid);
 
-
         try (LdapConnection conn = getConn()) {
             bind(conn);
 
@@ -108,11 +104,13 @@ public class LdapAuthProvider extends LdapBackend implements AuthenticatorProvid
             String[] attArray = new String[attributes.size()];
             attributes.toArray(attArray);
 
-            log.debug("Base DN: {}", getBaseDn());
             log.debug("Query: {}", userFilter);
             log.debug("Attributes: {}", GsonUtil.build().toJson(attArray));
 
-            try (EntryCursor cursor = conn.search(getBaseDn(), userFilter, SearchScope.SUBTREE, attArray)) {
+            for (String baseDN : getBaseDNs()) {
+                log.debug("Base DN: {}", baseDN);
+
+                try (EntryCursor cursor = conn.search(baseDN, userFilter, SearchScope.SUBTREE, attArray)) {
                     while (cursor.next()) {
                         Entry entry = cursor.get();
                         String dn = entry.getDn().getName();
@@ -159,11 +157,12 @@ public class LdapAuthProvider extends LdapBackend implements AuthenticatorProvid
                 } catch (CursorLdapReferralException e) {
                     log.warn("Entity for {} is only available via referral, skipping", mxid);
                 }
+            }
 
             log.info("No match were found for {}", mxid);
             return BackendAuthResult.failure();
         } catch (LdapException | IOException | CursorException e) {
-            throw new RuntimeException(e);
+            throw new InternalServerError(e);
         }
     }
 

src/main/java/io/kamax/mxisd/backend/ldap/LdapBackend.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -20,6 +20,7 @@
 
 package io.kamax.mxisd.backend.ldap;
 
+import io.kamax.matrix._MatrixID;
 import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.config.ldap.LdapConfig;
 import org.apache.commons.lang.StringUtils;
@@ -44,7 +45,7 @@ public abstract class LdapBackend {
     public static final String UID = "uid";
     public static final String MATRIX_ID = "mxid";
 
-    private Logger log = LoggerFactory.getLogger(LdapBackend.class);
+    private transient final Logger log = LoggerFactory.getLogger(LdapBackend.class);
 
     private LdapConfig cfg;
     private MatrixConfig mxCfg;
@@ -58,8 +59,8 @@ public abstract class LdapBackend {
         return cfg;
     }
 
-    protected String getBaseDn() {
-        return cfg.getConnection().getBaseDn();
+    protected List<String> getBaseDNs() {
+        return cfg.getConnection().getBaseDNs();
     }
 
     protected LdapConfig.Attribute getAt() {
@@ -70,7 +71,7 @@ public abstract class LdapBackend {
         return getAt().getUid().getValue();
     }
 
-    protected synchronized LdapConnection getConn() throws LdapException {
+    protected synchronized LdapConnection getConn() {
         return new LdapNetworkConnection(cfg.getConnection().getHost(), cfg.getConnection().getPort(), cfg.getConnection().isTls());
     }
 
@@ -124,6 +125,17 @@ public abstract class LdapBackend {
         }
     }
 
+    public String buildUidFromMatrixId(_MatrixID mxId) {
+        String uidType = getCfg().getAttribute().getUid().getType();
+        if (StringUtils.equals(UID, uidType)) {
+            return mxId.getLocalPart();
+        } else if (StringUtils.equals(MATRIX_ID, uidType)) {
+            return mxId.getId();
+        } else {
+            throw new IllegalArgumentException("Bind type " + uidType + " is not supported");
+        }
+    }
+
     public Optional<String> getAttribute(Entry entry, String attName) {
         Attribute attribute = entry.get(attName);
         if (attribute == null) {

src/main/java/io/kamax/mxisd/backend/ldap/LdapDirectoryProvider.java

@@ -1,8 +1,8 @@
 /*
  * mxisd - Matrix Identity Server Daemon
- * Copyright (C) 2017 Maxime Dor
+ * Copyright (C) 2017 Kamax Sarl
  *
- * https://max.kamax.io/
+ * https://www.kamax.io/
  *
  * This program is free software: you can redistribute it and/or modify
  * it under the terms of the GNU Affero General Public License as
@@ -22,9 +22,9 @@ package io.kamax.mxisd.backend.ldap;
 
 import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.config.ldap.LdapConfig;
-import io.kamax.mxisd.controller.directory.v1.io.UserDirectorySearchResult;
-import io.kamax.mxisd.directory.IDirectoryProvider;
+import io.kamax.mxisd.directory.DirectoryProvider;
 import io.kamax.mxisd.exception.InternalServerError;
+import io.kamax.mxisd.http.io.UserDirectorySearchResult;
 import io.kamax.mxisd.util.GsonUtil;
 import org.apache.directory.api.ldap.model.cursor.CursorException;
 import org.apache.directory.api.ldap.model.cursor.CursorLdapReferralException;
@@ -35,28 +35,19 @@ import org.apache.directory.api.ldap.model.message.SearchScope;
 import org.apache.directory.ldap.client.api.LdapConnection;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Component;
 
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
 
-@Component
-public class LdapDirectoryProvider extends LdapBackend implements IDirectoryProvider {
+public class LdapDirectoryProvider extends LdapBackend implements DirectoryProvider {
 
-    private Logger log = LoggerFactory.getLogger(LdapDirectoryProvider.class);
+    private transient final Logger log = LoggerFactory.getLogger(LdapDirectoryProvider.class);
 
-    @Autowired
     public LdapDirectoryProvider(LdapConfig cfg, MatrixConfig mxCfg) {
         super(cfg, mxCfg);
     }
 
-    @Override
-    public boolean isEnabled() {
-        return getCfg().isEnabled();
-    }
-
     protected UserDirectorySearchResult search(String query, List<String> attributes) {
         UserDirectorySearchResult result = new UserDirectorySearchResult();
         result.setLimited(false);
@@ -65,18 +56,19 @@ public class LdapDirectoryProvider extends LdapBackend implements IDirectoryProv
             bind(conn);
 
             LdapConfig.Attribute atCfg = getCfg().getAttribute();
-
             attributes = new ArrayList<>(attributes);
             attributes.add(getUidAtt());
             String[] attArray = new String[attributes.size()];
             attributes.toArray(attArray);
             String searchQuery = buildOrQueryWithFilter(getCfg().getDirectory().getFilter(), "*" + query + "*", attArray);
 
-            log.debug("Base DN: {}", getBaseDn());
             log.debug("Query: {}", searchQuery);
             log.debug("Attributes: {}", GsonUtil.build().toJson(attArray));
 
-            try (EntryCursor cursor = conn.search(getBaseDn(), searchQuery, SearchScope.SUBTREE, attArray)) {
+            for (String baseDN : getBaseDNs()) {
+                log.debug("Base DN: {}", baseDN);
+
+                try (EntryCursor cursor = conn.search(baseDN, searchQuery, SearchScope.SUBTREE, attArray)) {
                     while (cursor.next()) {
                         Entry entry = cursor.get();
                         log.info("Found possible match, DN: {}", entry.getDn().getName());
@@ -93,6 +85,7 @@ public class LdapDirectoryProvider extends LdapBackend implements IDirectoryProv
                         });
                     }
                 }
+            }
         } catch (CursorLdapReferralException e) {
             log.warn("An entry is only available via referral, skipping");
         } catch (IOException | LdapException | CursorException e) {

src/main/java/io/kamax/mxisd/backend/ldap/LdapProfileProvider.java

@@ -0,0 +1,150 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.ldap;
+
+import io.kamax.matrix.ThreePid;
+import io.kamax.matrix._MatrixID;
+import io.kamax.matrix._ThreePid;
+import io.kamax.mxisd.config.MatrixConfig;
+import io.kamax.mxisd.config.ldap.LdapConfig;
+import io.kamax.mxisd.exception.InternalServerError;
+import io.kamax.mxisd.profile.ProfileProvider;
+import org.apache.directory.api.ldap.model.cursor.CursorException;
+import org.apache.directory.api.ldap.model.cursor.CursorLdapReferralException;
+import org.apache.directory.api.ldap.model.cursor.EntryCursor;
+import org.apache.directory.api.ldap.model.entry.Entry;
+import org.apache.directory.api.ldap.model.exception.LdapException;
+import org.apache.directory.api.ldap.model.message.SearchScope;
+import org.apache.directory.ldap.client.api.LdapConnection;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+public class LdapProfileProvider extends LdapBackend implements ProfileProvider {
+
+    private transient final Logger log = LoggerFactory.getLogger(LdapProfileProvider.class);
+
+    public LdapProfileProvider(LdapConfig cfg, MatrixConfig mxCfg) {
+        super(cfg, mxCfg);
+    }
+
+    @Override
+    public Optional<String> getDisplayName(_MatrixID userId) {
+        String uid = buildUidFromMatrixId(userId);
+        log.info("Searching for display name of {}:", uid);
+
+        try (LdapConnection conn = getConn()) {
+            bind(conn);
+
+            String searchQuery = buildOrQueryWithFilter(getCfg().getProfile().getFilter(), uid, getUidAtt());
+            log.debug("Query: {}", searchQuery);
+
+            for (String baseDN : getBaseDNs()) {
+                log.debug("Base DN: {}", baseDN);
+                try (EntryCursor cursor = conn.search(baseDN, searchQuery, SearchScope.SUBTREE, getAt().getName())) {
+                    while (cursor.next()) {
+                        Entry entry = cursor.get();
+                        log.info("Found possible match, DN: {}", entry.getDn().getName());
+                        Optional<String> v = getAttribute(entry, getAt().getName()).flatMap(id -> {
+                            log.info("DN {} is a valid match", entry.getDn().getName());
+                            try {
+                                return getAttribute(entry, getAt().getName());
+                            } catch (IllegalArgumentException e) {
+                                log.warn("Bind was found but type {} is not supported", getAt().getUid().getType());
+                                return Optional.empty();
+                            }
+                        });
+
+                        if (v.isPresent()) {
+                            log.info("DN {} is the final match", entry.getDn().getName());
+                            return v;
+                        }
+                    }
+                } catch (CursorLdapReferralException e) {
+                    log.warn("An entry is only available via referral, skipping");
+                }
+            }
+        } catch (IOException | LdapException | CursorException e) {
+            throw new InternalServerError(e);
+        }
+
+        return Optional.empty();
+    }
+
+    @Override
+    public List<_ThreePid> getThreepids(_MatrixID userId) {
+        String uid = buildUidFromMatrixId(userId);
+        log.info("Searching for 3PIDs of {}:", uid);
+
+        List<_ThreePid> threePids = new ArrayList<>();
+        try (LdapConnection conn = getConn()) {
+            bind(conn);
+
+            getCfg().getAttribute().getThreepid().forEach((medium, attributes) -> {
+                String[] attArray = new String[attributes.size()];
+                attributes.toArray(attArray);
+
+                String searchQuery = buildOrQueryWithFilter(getCfg().getProfile().getFilter(), uid, getUidAtt());
+
+                log.debug("Query for 3PID {}: {}", medium, searchQuery);
+
+                for (String baseDN : getBaseDNs()) {
+                    log.debug("Base DN: {}", baseDN);
+                    try (EntryCursor cursor = conn.search(baseDN, searchQuery, SearchScope.SUBTREE, attArray)) {
+                        while (cursor.next()) {
+                            Entry entry = cursor.get();
+                            log.info("Found possible match, DN: {}", entry.getDn().getName());
+                            try {
+                                attributes.stream()
+                                        .flatMap(at -> getAttributes(entry, at).stream())
+                                        .forEach(address -> {
+                                            log.info("Found 3PID: {} - {}", medium, address);
+                                            threePids.add(new ThreePid(medium, address));
+                                        });
+                            } catch (IllegalArgumentException e) {
+                                log.warn("Bind was found but type {} is not supported", getAt().getUid().getType());
+                            }
+                        }
+                    } catch (CursorLdapReferralException e) {
+                        log.warn("An entry is only available via referral, skipping");
+                    } catch (LdapException | IOException | CursorException e) {
+                        throw new InternalServerError(e);
+                    }
+                }
+            });
+        } catch (IOException | LdapException e) {
+            throw new InternalServerError(e);
+        }
+
+        return threePids;
+    }
+
+    @Override
+    public List<String> getRoles(_MatrixID userId) {
+        return Collections.emptyList();
+    }
+
+}

src/main/java/io/kamax/mxisd/backend/ldap/LdapStoreSupplier.java

@@ -0,0 +1,47 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2018 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.backend.ldap;
+
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.auth.AuthProviders;
+import io.kamax.mxisd.backend.IdentityStoreSupplier;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.directory.DirectoryProviders;
+import io.kamax.mxisd.lookup.ThreePidProviders;
+import io.kamax.mxisd.profile.ProfileProviders;
+
+public class LdapStoreSupplier implements IdentityStoreSupplier {
+
+    @Override
+    public void accept(Mxisd mxisd) {
+        accept(mxisd.getConfig());
+    }
+
+    public void accept(MxisdConfig cfg) {
+        if (cfg.getLdap().isEnabled()) {
+            AuthProviders.register(() -> new LdapAuthProvider(cfg.getLdap(), cfg.getMatrix()));
+            DirectoryProviders.register(() -> new LdapDirectoryProvider(cfg.getLdap(), cfg.getMatrix()));
+            ThreePidProviders.register(() -> new LdapThreePidProvider(cfg.getLdap(), cfg.getMatrix()));
+            ProfileProviders.register(() -> new LdapProfileProvider(cfg.getLdap(), cfg.getMatrix()));
+        }
+    }
+
+}