Restart mxisd service on Debian package upgrade/install if possible

- Support 3PID unbind via 3PID sessions

README.md

@@ -14,7 +14,7 @@ mxisd - Federated Matrix Identity Server
 
 # Overview
 mxisd is a Federated Matrix Identity server for self-hosted Matrix infrastructures with [enhanced features](#features).
-As an enhanced Identity service, it implements the [Identity service API](https://matrix.org/docs/spec/identity_service/r0.1.0.html)
+As an enhanced Identity service, it implements the [Identity service API](https://matrix.org/docs/spec/identity_service/r0.2.0.html)
 and several [extra features](#features) that greatly enhance user experience within Matrix.
 It is the one stop shop for anything regarding Authentication, Directory and Identity management in Matrix built in a
 single coherent product.
@@ -34,15 +34,15 @@ users. 3PIDs can be anything that uniquely and globally identify a user, like:
 If you are unfamiliar with the Identity vocabulary and concepts in Matrix, **please read this [introduction](docs/concepts.md)**.  
 
 # Features
-[Identity](docs/features/identity.md): As a [regular Matrix Identity service](https://matrix.org/docs/spec/identity_service/r0.1.0.html#general-principles):
+[Identity](docs/features/identity.md): As a [regular Matrix Identity service](https://matrix.org/docs/spec/identity_service/r0.2.0.html#general-principles):
 - Search for people by 3PID using its own Identity stores
-  ([Spec](https://matrix.org/docs/spec/identity_service/r0.1.0.html#association-lookup))
+  ([Spec](https://matrix.org/docs/spec/identity_service/r0.2.0.html#association-lookup))
 - Invite people to rooms by 3PID using its own Identity stores, with notifications to the invitee (Email, SMS, etc.)
-  ([Spec](https://matrix.org/docs/spec/identity_service/r0.1.0.html#post-matrix-identity-api-v1-store-invite))
-- Allow users to add 3PIDs to their settings/profile
-  ([Spec](https://matrix.org/docs/spec/identity_service/r0.1.0.html#establishing-associations))
+  ([Spec](https://matrix.org/docs/spec/identity_service/r0.2.0.html#invitation-storage))
+- Allow users to add/remove 3PIDs to their settings/profile via 3PID sessions
+  ([Spec](https://matrix.org/docs/spec/identity_service/r0.2.0.html#establishing-associations))
 - Register accounts on your Homeserver with 3PIDs
-  ([Spec](https://matrix.org/docs/spec/identity_service/r0.1.0.html#establishing-associations))
+  ([Spec](https://matrix.org/docs/spec/identity_service/r0.2.0.html#establishing-associations))
 
 As an enhanced Identity service:
 - [Federation](docs/features/federation.md): Use a recursive lookup mechanism when searching and inviting people by 3PID,

build.gradle

@@ -48,6 +48,8 @@ def dockerImageTag = "${dockerImageName}:${mxisdVersion()}"
 
 group = 'io.kamax'
 mainClassName = 'io.kamax.mxisd.MxisdStandaloneExec'
+sourceCompatibility = '1.8'
+targetCompatibility = '1.8'
 
 String mxisdVersion() {
     def versionPattern = Pattern.compile("v(\\d+\\.)?(\\d+\\.)?(\\d+)(-.*)?")
@@ -227,6 +229,12 @@ task debBuild(dependsOn: shadowJar) {
                 value: debDataPath
         )
 
+        ant.replace(
+                file: "${debBuildDebianPath}/postinst",
+                token: '%DEB_CONF_FILE%',
+                value: "${debConfPath}/mxisd.yaml"
+        )
+
         ant.chmod(
                 file: "${debBuildDebianPath}/postinst",
                 perm: 'a+x'

docs/features/experimental/application-service.md

@@ -1,6 +1,6 @@
 # Application Service
 **WARNING:** These features are currently highly experimental. They can be removed or modified without notice.  
-All the features requires a Homeserver capable of connecting [Application Services](https://matrix.org/docs/spec/application_service/r0.1.0.html).
+All the features requires a Homeserver capable of connecting [Application Services](https://matrix.org/docs/spec/application_service/r0.1.1.html).
 
 The following capabilities are provided in this feature:
 - [Admin commands](#admin-commands)

docs/features/identity.md

@@ -1,5 +1,5 @@
 # Identity
-Implementation of the [Identity Service API r0.1.0](https://matrix.org/docs/spec/identity_service/r0.1.0.html).
+Implementation of the [Identity Service API r0.2.0](https://matrix.org/docs/spec/identity_service/r0.2.0.html).
 
 - [Lookups](#lookups)
 - [Invitations](#invitations)

docs/features/registration.md

@@ -3,7 +3,7 @@
 - [Integration](#integration)
   - [Reverse Proxy](#reverse-proxy)
     - [nginx](#nginx)
-    - [Apache](#apache)
+    - [Apache2](#apache2)
   - [Homeserver](#homeserver)
     - [synapse](#synapse)
 - [Configuration](#configuration)
@@ -16,7 +16,7 @@
 Registration is an enhanced feature of mxisd to control registrations involving 3PIDs on a Homeserver based on policies:
 - Match pending 3PID invites on the server
 - Match 3PID pattern, like a specific set of domains for emails
-- In futher releases, use 3PIDs found in Identity stores
+- In further releases, use 3PIDs found in Identity stores
 
 It aims to help open or invite-only registration servers control what is possible to do and ensure only approved people
 can register on a given server in a implementation-agnostic manner.
@@ -36,14 +36,14 @@ Later version(s) of this feature may directly control registration itself to cre
 ### Reverse Proxy
 #### nginx
 ```nginx
-location ^/_matrix/client/r0/register/[^/]/?$ {
-	proxy_pass		http://127.0.0.1:8090;
+location ~* ^/_matrix/client/r0/register/[^/]+/requestToken$ {
+	proxy_pass		http://localhost:8090;
 	proxy_set_header	Host $host;
 	proxy_set_header	X-Forwarded-For $remote_addr;
 }
 ```
 
-#### apache
+#### Apache2
 > TBC
 
 ### Homeserver
@@ -55,8 +55,8 @@ registrations_require_3pid:
 ```
 
 ## Configuration
-See the [Configuration](../configuration.md) introduction doc on how to read the configuration keys.  
-An example of working configuration is avaiable at the end of this section.
+See the [Configuration](../configure.md) introduction doc on how to read the configuration keys.  
+An example of working configuration is available at the end of this section.
 ### Enable/Disable
 `register.allowed`, taking a boolean, can be used to enable/disable registration if the attempt is not 3PID-based.  
 `false` is the default value to prevent open registration, as you must allow it on the homeserver side.
@@ -72,7 +72,7 @@ At this time, only `email` is supported with 3PID specific configuration with th
 **Base key**: `register.threepid.email`
 
 ##### Domain whitelist/blacklist
-If you would like to control which domains are allowed to be used when registrating with an email, the following sub-keys
+If you would like to control which domains are allowed to be used when registering with an email, the following sub-keys
 are available:
 - `domain.whitelist`
 - `domain.blacklist`
@@ -82,7 +82,7 @@ The value format is an hybrid between glob patterns and postfix configuration fi
 - `.<domain>` will only match sub-domain(s)
 - `<domain>` will only match the exact domain
 
-The following table illustrates pattern and maching status against example values:
+The following table illustrates pattern and matching status against example values:
 
 | Config value   | Matches `example.org` | Matches `sub.example.org` |
 |--------------- |-----------------------|---------------------------|

docs/threepids/notification/template-generator.md

@@ -20,27 +20,31 @@ All placeholders **MUST** be surrounded with `%` in the template. Per example, t
 The following placeholders are available in every template:
 
 | Placeholder                     | Purpose                                                                      |
-|---------------------|------------------------------------------------------------------------------|
+|---------------------------------|------------------------------------------------------------------------------|
 | `DOMAIN`                        | Identity server authoritative domain, as configured in `matrix.domain`       |
 | `DOMAIN_PRETTY`                 | Same as `DOMAIN` with the first letter upper case and all other lower case   |
 | `FROM_EMAIL`                    | Email address configured in `threepid.medium.<3PID medium>.identity.from`    |
 | `FROM_NAME`                     | Name configured in `threepid.medium.<3PID medium>.identity.name`             |
 | `RECIPIENT_MEDIUM`              | The 3PID medium, like `email` or `msisdn`                                    |
+| `RECIPIENT_MEDIUM_URL_ENCODED`  | URL encoded value of `RECIPIENT_MEDIUM`                                      |
 | `RECIPIENT_ADDRESS`             | The address to which the notification is sent                                |
+| `RECIPIENT_ADDRESS_URL_ENCODED` | URL encoded value of `RECIPIENT_ADDRESS`                                     |
 
 ### Room invitation
 Specific placeholders:
 
 | Placeholder                  | Purpose                                                                           |
-|---------------------|------------------------------------------------------------------------------------------|
+|------------------------------|-----------------------------------------------------------------------------------|
 | `SENDER_ID`                  | Matrix ID of the user who made the invite                                         |
 | `SENDER_NAME`                | Display name of the user who made the invite, if not available/set, empty         |
 | `SENDER_NAME_OR_ID`          | Display name of the user who made the invite. If not available/set, its Matrix ID |
 | `INVITE_MEDIUM`              | The 3PID medium for the invite.                                                   |
+| `INVITE_MEDIUM_URL_ENCODED`  | URL encoded value of `INVITE_MEDIUM`                                              |
 | `INVITE_ADDRESS`             | The 3PID address for the invite.                                                  |
+| `INVITE_ADDRESS_URL_ENCODED` | URL encoded value of `INVITE_ADDRESS`                                             |
 | `ROOM_ID`                    | The Matrix ID of the Room in which the invite took place                          |
 | `ROOM_NAME`                  | The Name of the room in which the invite took place. If not available/set, empty  |
-| `ROOM_NAME_OR_ID`   | The Name of the room in which the invite took place. If not available/set, its Matrix ID |
+| `ROOM_NAME_OR_ID`            | The Name of the room in which the invite took place. If not available/set, its ID |
 | `REGISTER_URL`               | The URL to provide to the user allowing them to register their account, if needed |
 
 ### Validation of 3PID Session

docs/threepids/session/session-views.md

@@ -26,6 +26,14 @@ Two configuration keys are available that accept paths to HTML templates:
 - `success`
 - `failure`
 
+### Serving static assets
+mxisd will not serve any static asset (images, JS, CSS, etc.). If such are needed, you will need to serve them using the
+reverse proxy sitting in front of mxisd using a path outside of the `/_matrix/identity/` namespace. We advise using
+the base path `/static/` for such use cases, allowing to remain under the same hostname/origin.
+
+You can also serve such assets using absolute URL, possibly under other domains, but be aware of Cross-Origin restrictions
+in browsers which are out of scope of mxisd.
+
 ## Placeholders
 ### Success
 No object/placeholder are currently available.

gradle/wrapper/gradle-wrapper.properties

@@ -1,6 +1,5 @@
-#Fri Aug 11 17:19:02 CEST 2017
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
+distributionUrl=https\://services.gradle.org/distributions/gradle-5.3.1-bin.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-4.0.2-bin.zip

gradlew

@@ -1,5 +1,21 @@
 #!/usr/bin/env sh
 
+#
+# Copyright 2015 the original author or authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
 ##############################################################################
 ##
 ##  Gradle start up script for UN*X
@@ -28,7 +44,7 @@ APP_NAME="Gradle"
 APP_BASE_NAME=`basename "$0"`
 
 # Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
-DEFAULT_JVM_OPTS=""
+DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
 
 # Use the maximum available, or set MAX_FD != -1 to use that value.
 MAX_FD="maximum"

gradlew.bat

@@ -1,3 +1,19 @@
+@rem
+@rem Copyright 2015 the original author or authors.
+@rem
+@rem Licensed under the Apache License, Version 2.0 (the "License");
+@rem you may not use this file except in compliance with the License.
+@rem You may obtain a copy of the License at
+@rem
+@rem      http://www.apache.org/licenses/LICENSE-2.0
+@rem
+@rem Unless required by applicable law or agreed to in writing, software
+@rem distributed under the License is distributed on an "AS IS" BASIS,
+@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+@rem See the License for the specific language governing permissions and
+@rem limitations under the License.
+@rem
+
 @if "%DEBUG%" == "" @echo off
 @rem ##########################################################################
 @rem
@@ -14,7 +30,7 @@ set APP_BASE_NAME=%~n0
 set APP_HOME=%DIRNAME%
 
 @rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
-set DEFAULT_JVM_OPTS=
+set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
 
 @rem Find java.exe
 if defined JAVA_HOME goto findJavaFromJavaHome

src/debian/postinst

@@ -11,3 +11,9 @@ ln -sfT /usr/lib/mxisd/mxisd /usr/bin/mxisd
 
 # Enable systemd service
 systemctl enable mxisd.service
+
+# If we already have a config file setup, we attempt to run mxisd automatically
+# Specifically targeted at upgrades where the service needs to be restarted
+if [ -f "%DEB_CONF_FILE%" ]; then
+    systemctl restart mxisd.service
+fi

src/main/java/io/kamax/mxisd/HttpMxisd.java

@@ -105,7 +105,7 @@ public class HttpMxisd {
                 .get(SessionValidateHandler.Path, SaneHandler.around(new SessionValidationGetHandler(m.getSession(), m.getConfig())))
                 .post(SessionValidateHandler.Path, SaneHandler.around(new SessionValidationPostHandler(m.getSession())))
                 .get(SessionTpidGetValidatedHandler.Path, SaneHandler.around(new SessionTpidGetValidatedHandler(m.getSession())))
-                .post(SessionTpidBindHandler.Path, SaneHandler.around(new SessionTpidBindHandler(m.getSession(), m.getInvite())))
+                .post(SessionTpidBindHandler.Path, SaneHandler.around(new SessionTpidBindHandler(m.getSession(), m.getInvite(), m.getSign())))
                 .post(SessionTpidUnbindHandler.Path, SaneHandler.around(new SessionTpidUnbindHandler(m.getSession())))
                 .post(SignEd25519Handler.Path, SaneHandler.around(new SignEd25519Handler(m.getConfig(), m.getInvite(), m.getSign())))
 

src/main/java/io/kamax/mxisd/Mxisd.java

@@ -107,7 +107,7 @@ public class Mxisd {
 
         store = new OrmLiteSqlStorage(cfg);
         keyMgr = CryptoFactory.getKeyManager(cfg.getKey());
-        signMgr = CryptoFactory.getSignatureManager(keyMgr);
+        signMgr = CryptoFactory.getSignatureManager(cfg, keyMgr);
         clientDns = new ClientDnsOverwrite(cfg.getDns().getOverwrite());
 
         synapse = new Synapse(cfg.getSynapseSql());
@@ -118,7 +118,7 @@ public class Mxisd {
         idStrategy = new RecursivePriorityLookupStrategy(cfg.getLookup(), ThreePidProviders.get(), bridgeFetcher);
         pMgr = new ProfileManager(ProfileProviders.get(), clientDns, httpClient);
         notifMgr = new NotificationManager(cfg.getNotification(), NotificationHandlers.get());
-        sessMgr = new SessionManager(cfg.getSession(), cfg.getMatrix(), store, notifMgr, idStrategy, httpClient);
+        sessMgr = new SessionManager(cfg.getSession(), cfg.getMatrix(), store, notifMgr);
         invMgr = new InvitationManager(cfg, store, idStrategy, keyMgr, signMgr, resolver, notifMgr, pMgr);
         authMgr = new AuthManager(cfg, AuthProviders.get(), idStrategy, invMgr, clientDns, httpClient);
         dirMgr = new DirectoryManager(cfg.getDirectory(), clientDns, httpClient, DirectoryProviders.get());

src/main/java/io/kamax/mxisd/as/AppSvcManager.java

@@ -176,10 +176,12 @@ public class AppSvcManager {
         ensureEnabled();
 
         if (StringUtils.isBlank(token)) {
+            log.info("Denying request without a HS token");
             throw new HttpMatrixException(401, "M_UNAUTHORIZED", "No HS token");
         }
 
         if (!StringUtils.equals(cfg.getEndpoint().getToAS().getToken(), token)) {
+            log.info("Denying request with an invalid HS token");
             throw new NotAllowedException("Invalid HS token");
         }
 

src/main/java/io/kamax/mxisd/as/processor/event/MembershipEventProcessor.java

@@ -27,7 +27,6 @@ import io.kamax.matrix._MatrixID;
 import io.kamax.matrix._ThreePid;
 import io.kamax.matrix.client.as.MatrixApplicationServiceClient;
 import io.kamax.matrix.event.EventKey;
-import io.kamax.matrix.hs._MatrixRoom;
 import io.kamax.mxisd.Mxisd;
 import io.kamax.mxisd.backend.sql.synapse.Synapse;
 import io.kamax.mxisd.config.MxisdConfig;
@@ -81,7 +80,7 @@ public class MembershipEventProcessor implements EventTypeProcessor {
 
         _MatrixID target = MatrixID.asAcceptable(targetId);
         if (!StringUtils.equals(target.getDomain(), cfg.getMatrix().getDomain())) {
-            log.debug("Ignoring invite for {}: not a local user");
+            log.debug("Ignoring invite for {}: not a local user", targetId);
             return;
         }
 
@@ -89,10 +88,9 @@ public class MembershipEventProcessor implements EventTypeProcessor {
 
         boolean isForMainUser = StringUtils.equals(target.getLocalPart(), cfg.getAppsvc().getUser().getMain());
         boolean isForExpInvUser = StringUtils.equals(target.getLocalPart(), cfg.getAppsvc().getUser().getInviteExpired());
-        boolean isUs = isForMainUser || isForExpInvUser;
 
         if (StringUtils.equals("join", EventKey.Membership.getStringOrNull(content))) {
-            if (!isForMainUser) {
+            if (isForExpInvUser) {
                 log.warn("We joined the room {} for another identity as the main user, which is not supported. Leaving...", roomId);
 
                 client.getUser(target.getLocalPart()).getRoom(roomId).tryLeave().ifPresent(err -> {
@@ -108,10 +106,7 @@ public class MembershipEventProcessor implements EventTypeProcessor {
                 processForUserIdInvite(roomId, sender, target);
             }
         } else if (StringUtils.equals("leave", EventKey.Membership.getStringOrNull(content))) {
-            _MatrixRoom room = client.getRoom(roomId);
-            if (!isUs && room.getJoinedUsers().size() == 1) {
             // TODO we need to find out if this is only us remaining and leave the room if so, using the right client for it
-            }
         } else {
             log.debug("This is not an supported type of membership event, skipping");
         }

src/main/java/io/kamax/mxisd/auth/AuthManager.java

@@ -64,6 +64,8 @@ import java.util.Objects;
 
 public class AuthManager {
 
+    private static final Logger log = LoggerFactory.getLogger(AuthManager.class);
+
     private static final String TypeKey = "type";
     private static final String UserKey = "user";
     private static final String IdentifierKey = "identifier";
@@ -72,7 +74,6 @@ public class AuthManager {
     private static final String UserIdTypeValue = "m.id.user";
     private static final String ThreepidTypeValue = "m.id.thirdparty";
 
-    private transient final Logger log = LoggerFactory.getLogger(AuthManager.class);
     private final Gson gson = GsonUtil.get(); // FIXME replace
 
     private List<AuthenticatorProvider> providers;
@@ -138,6 +139,12 @@ public class AuthManager {
                     invMgr.publishMappingIfInvited(new ThreePidMapping(pid, mxId));
                 }
 
+                try {
+                    MatrixID.asValid(mxId);
+                } catch (IllegalArgumentException e) {
+                    log.warn("The returned User ID {} is not a valid Matrix ID. Login might fail at the Homeserver level", mxId);
+                }
+
                 invMgr.lookupMappingsForInvites();
 
                 return authResult;

src/main/java/io/kamax/mxisd/config/MxisdConfig.java

@@ -83,6 +83,12 @@ public class MxisdConfig {
 
     }
 
+    public static MxisdConfig forDomain(String domain) {
+        MxisdConfig cfg = new MxisdConfig();
+        cfg.getMatrix().setDomain(domain);
+        return cfg;
+    }
+
     private AppServiceConfig appsvc = new AppServiceConfig();
     private AuthenticationConfig auth = new AuthenticationConfig();
     private DirectoryConfig directory = new DirectoryConfig();
@@ -309,6 +315,13 @@ public class MxisdConfig {
         this.wordpress = wordpress;
     }
 
+    public MxisdConfig inMemory() {
+        getKey().setPath(":memory:");
+        getStorage().getProvider().getSqlite().setDatabase(":memory:");
+
+        return this;
+    }
+
     public MxisdConfig build() {
         if (StringUtils.isBlank(getServer().getName())) {
             getServer().setName(getMatrix().getDomain());

src/main/java/io/kamax/mxisd/config/threepid/notification/NotificationConfig.java

@@ -20,7 +20,6 @@
 
 package io.kamax.mxisd.config.threepid.notification;
 
-import com.google.gson.JsonObject;
 import io.kamax.matrix.ThreePidMedium;
 import io.kamax.mxisd.threepid.notification.email.EmailRawNotificationHandler;
 import io.kamax.mxisd.threepid.notification.phone.PhoneNotificationHandler;
@@ -35,7 +34,7 @@ public class NotificationConfig {
     private transient final Logger log = LoggerFactory.getLogger(NotificationConfig.class);
 
     private Map<String, String> handler = new HashMap<>();
-    private Map<String, JsonObject> handlers = new HashMap<>();
+    private Map<String, Object> handlers = new HashMap<>();
 
     public NotificationConfig() {
         handler.put(ThreePidMedium.Email.getId(), EmailRawNotificationHandler.ID);
@@ -50,11 +49,11 @@ public class NotificationConfig {
         this.handler = handler;
     }
 
-    public Map<String, JsonObject> getHandlers() {
+    public Map<String, Object> getHandlers() {
         return handlers;
     }
 
-    public void setHandlers(Map<String, JsonObject> handlers) {
+    public void setHandlers(Map<String, Object> handlers) {
         this.handlers = handlers;
     }
 

src/main/java/io/kamax/mxisd/crypto/CryptoFactory.java

@@ -21,6 +21,7 @@
 package io.kamax.mxisd.crypto;
 
 import io.kamax.mxisd.config.KeyConfig;
+import io.kamax.mxisd.config.MxisdConfig;
 import io.kamax.mxisd.crypto.ed25519.Ed25519KeyManager;
 import io.kamax.mxisd.crypto.ed25519.Ed25519SignatureManager;
 import io.kamax.mxisd.storage.crypto.FileKeyStore;
@@ -54,8 +55,8 @@ public class CryptoFactory {
         return new Ed25519KeyManager(store);
     }
 
-    public static SignatureManager getSignatureManager(Ed25519KeyManager keyMgr) {
-        return new Ed25519SignatureManager(keyMgr);
+    public static SignatureManager getSignatureManager(MxisdConfig cfg, Ed25519KeyManager keyMgr) {
+        return new Ed25519SignatureManager(cfg, keyMgr);
     }
 
 }

src/main/java/io/kamax/mxisd/crypto/KeyType.java

@@ -22,7 +22,7 @@ package io.kamax.mxisd.crypto;
 
 /**
  * Types of keys used by an Identity server.
- * See https://matrix.org/docs/spec/identity_service/r0.1.0.html#key-management
+ * See https://matrix.org/docs/spec/identity_service/r0.2.0.html#key-management
  */
 public enum KeyType {
 

src/main/java/io/kamax/mxisd/crypto/SignatureManager.java

@@ -20,12 +20,57 @@
 
 package io.kamax.mxisd.crypto;
 
+import com.google.gson.JsonElement;
 import com.google.gson.JsonObject;
+import io.kamax.matrix.event.EventKey;
+import io.kamax.matrix.json.MatrixJson;
 
 import java.nio.charset.StandardCharsets;
+import java.util.Objects;
 
 public interface SignatureManager {
 
+    /**
+     * Sign the message with the default domain and add the signature to the <code>signatures</code> key.
+     * <p>
+     * If the key does not exist yet, it is created. If the key exist, the produced signature will be merged with any
+     * existing ones.
+     *
+     * @param message The message to sign with the default domain and add the produced signature to
+     * @return The provided message with the new signature
+     * @throws IllegalArgumentException If the <code>signatures</code> key exists and its value is not a JSON object
+     */
+    JsonObject signMessageGson(JsonObject message) throws IllegalArgumentException;
+
+    /**
+     * Sign the message and add the signature to the <code>signatures</code> key.
+     * <p>
+     * If the key does not exist yet, it is created. If the key exist, the produced signature will be merged with any
+     * existing ones.
+     *
+     * @param domain  The domain under which the signature should be added
+     * @param message The message to sign and add the produced signature to
+     * @return The provided message with the new signature
+     * @throws IllegalArgumentException If the <code>signatures</code> key exists and its value is not a JSON object
+     */
+    default JsonObject signMessageGson(String domain, JsonObject message) throws IllegalArgumentException {
+        JsonElement signEl = message.remove(EventKey.Signatures.get());
+        JsonObject oldSigns = new JsonObject();
+        if (!Objects.isNull(signEl)) {
+            if (!signEl.isJsonObject()) {
+                throw new IllegalArgumentException("Message contains a signatures key that is not a JSON object value");
+            }
+
+            oldSigns = signEl.getAsJsonObject();
+        }
+
+        JsonObject newSigns = signMessageGson(domain, MatrixJson.encodeCanonical(message));
+        oldSigns.entrySet().forEach(entry -> newSigns.add(entry.getKey(), entry.getValue()));
+        message.add(EventKey.Signatures.get(), newSigns);
+
+        return message;
+    }
+
     /**
      * Sign the message and produce a <code>signatures</code> object that can directly be added to the object being signed.
      *

src/main/java/io/kamax/mxisd/crypto/ed25519/Ed25519SignatureManager.java

@@ -23,6 +23,8 @@ package io.kamax.mxisd.crypto.ed25519;
 import com.google.gson.JsonObject;
 import io.kamax.matrix.codec.MxBase64;
 import io.kamax.matrix.json.MatrixJson;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.config.ServerConfig;
 import io.kamax.mxisd.crypto.KeyIdentifier;
 import io.kamax.mxisd.crypto.Signature;
 import io.kamax.mxisd.crypto.SignatureManager;
@@ -35,12 +37,19 @@ import java.security.SignatureException;
 
 public class Ed25519SignatureManager implements SignatureManager {
 
+    private final ServerConfig cfg;
     private final Ed25519KeyManager keyMgr;
 
-    public Ed25519SignatureManager(Ed25519KeyManager keyMgr) {
+    public Ed25519SignatureManager(MxisdConfig cfg, Ed25519KeyManager keyMgr) {
+        this.cfg = cfg.getServer();
         this.keyMgr = keyMgr;
     }
 
+    @Override
+    public JsonObject signMessageGson(JsonObject message) throws IllegalArgumentException {
+        return signMessageGson(cfg.getName(), message);
+    }
+
     @Override
     public JsonObject signMessageGson(String domain, String message) {
         Signature sign = sign(message);

src/main/java/io/kamax/mxisd/exception/InternalServerError.java

@@ -33,8 +33,7 @@ public class InternalServerError extends HttpMatrixException {
         super(
                 HttpStatus.SC_INTERNAL_SERVER_ERROR,
                 "M_UNKNOWN",
-                "An internal server error occured. If this error persists, please contact support with reference #" +
-                        Instant.now().toEpochMilli()
+                "An internal server error occurred. Contact your administrator with reference Transaction #" + Instant.now().toEpochMilli()
         );
     }
 

src/main/java/io/kamax/mxisd/http/undertow/handler/SaneHandler.java

@@ -97,7 +97,7 @@ public class SaneHandler extends BasicHttpHandler {
                 if (StringUtils.isNotBlank(e.getInternalReason())) {
                     log.error("Transaction #{} - {}", e.getReference(), e.getInternalReason());
                 } else {
-                    log.error("Transaction #{}", e);
+                    log.error("Transaction #{}", e.getReference(), e);
                 }
 
                 handleException(exchange, e);

src/main/java/io/kamax/mxisd/http/undertow/handler/auth/RestAuthHandler.java

@@ -36,7 +36,7 @@ public class RestAuthHandler extends BasicHttpHandler {
 
     public static final String Path = "/_matrix-internal/identity/v1/check_credentials";
 
-    private transient final Logger log = LoggerFactory.getLogger(RestAuthHandler.class);
+    private static final Logger log = LoggerFactory.getLogger(RestAuthHandler.class);
 
     private AuthManager mgr;
 
@@ -45,7 +45,7 @@ public class RestAuthHandler extends BasicHttpHandler {
     }
 
     @Override
-    public void handleRequest(HttpServerExchange exchange) throws Exception {
+    public void handleRequest(HttpServerExchange exchange) {
         JsonObject authData = parseJsonObject(exchange, "user");
         if (!authData.has("id") || !authData.has("password")) {
             throw new JsonMemberNotFoundException("Missing id or password keys");

src/main/java/io/kamax/mxisd/http/undertow/handler/directory/v1/UserDirectorySearchHandler.java

@@ -40,7 +40,7 @@ public class UserDirectorySearchHandler extends HomeserverProxyHandler {
     }
 
     @Override
-    public void handleRequest(HttpServerExchange exchange) throws Exception {
+    public void handleRequest(HttpServerExchange exchange) {
         String accessToken = getAccessToken(exchange);
         UserDirectorySearchRequest searchQuery = parseJsonTo(exchange, UserDirectorySearchRequest.class);
         URI target = URI.create(exchange.getRequestURL());

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/BulkLookupHandler.java

@@ -37,7 +37,7 @@ public class BulkLookupHandler extends LookupHandler {
 
     public static final String Path = IsAPIv1.Base + "/bulk_lookup";
 
-    private transient final Logger log = LoggerFactory.getLogger(SingleLookupHandler.class);
+    private static final Logger log = LoggerFactory.getLogger(SingleLookupHandler.class);
 
     private LookupStrategy strategy;
 

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/EphemeralKeyIsValidHandler.java

@@ -31,7 +31,7 @@ public class EphemeralKeyIsValidHandler extends KeyIsValidHandler {
 
     public static final String Path = IsAPIv1.Base + "/pubkey/ephemeral/isvalid";
 
-    private transient final Logger log = LoggerFactory.getLogger(EphemeralKeyIsValidHandler.class);
+    private static final Logger log = LoggerFactory.getLogger(EphemeralKeyIsValidHandler.class);
 
     private KeyManager mgr;
 

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/SessionTpidBindHandler.java

@@ -21,11 +21,15 @@
 package io.kamax.mxisd.http.undertow.handler.identity.v1;
 
 import com.google.gson.JsonObject;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.crypto.SignatureManager;
 import io.kamax.mxisd.exception.BadRequestException;
 import io.kamax.mxisd.http.IsAPIv1;
 import io.kamax.mxisd.http.io.identity.BindRequest;
+import io.kamax.mxisd.http.io.identity.SingeLookupReplyJson;
 import io.kamax.mxisd.http.undertow.handler.BasicHttpHandler;
 import io.kamax.mxisd.invitation.InvitationManager;
+import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.session.SessionManager;
 import io.undertow.server.HttpServerExchange;
 import io.undertow.util.QueryParameterUtils;
@@ -42,14 +46,16 @@ public class SessionTpidBindHandler extends BasicHttpHandler {
 
     public static final String Path = IsAPIv1.Base + "/3pid/bind";
 
-    private transient final Logger log = LoggerFactory.getLogger(SessionTpidBindHandler.class);
+    private static final Logger log = LoggerFactory.getLogger(SessionTpidBindHandler.class);
 
     private SessionManager mgr;
     private InvitationManager invMgr;
+    private SignatureManager signMgr;
 
-    public SessionTpidBindHandler(SessionManager mgr, InvitationManager invMgr) {
+    public SessionTpidBindHandler(SessionManager mgr, InvitationManager invMgr, SignatureManager signMgr) {
         this.mgr = mgr;
         this.invMgr = invMgr;
+        this.signMgr = signMgr;
     }
 
     @Override
@@ -74,8 +80,9 @@ public class SessionTpidBindHandler extends BasicHttpHandler {
         }
 
         try {
-            mgr.bind(bindReq.getSid(), bindReq.getSecret(), bindReq.getUserId());
-            respond(exchange, new JsonObject());
+            SingleLookupReply lookup = mgr.bind(bindReq.getSid(), bindReq.getSecret(), bindReq.getUserId());
+            JsonObject response = signMgr.signMessageGson(GsonUtil.makeObj(new SingeLookupReplyJson(lookup)));
+            respond(exchange, response);
         } catch (BadRequestException e) {
             log.info("requested session was not validated");
 

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/SessionTpidUnbindHandler.java

@@ -21,15 +21,22 @@
 package io.kamax.mxisd.http.undertow.handler.identity.v1;
 
 import com.google.gson.JsonObject;
+import io.kamax.mxisd.exception.BadRequestException;
+import io.kamax.mxisd.exception.NotAllowedException;
 import io.kamax.mxisd.http.IsAPIv1;
 import io.kamax.mxisd.http.undertow.handler.BasicHttpHandler;
 import io.kamax.mxisd.session.SessionManager;
 import io.undertow.server.HttpServerExchange;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 public class SessionTpidUnbindHandler extends BasicHttpHandler {
 
     public static final String Path = IsAPIv1.Base + "/3pid/unbind";
 
+    private static final Logger log = LoggerFactory.getLogger(SessionTpidUnbindHandler.class);
+
     private final SessionManager sessionMgr;
 
     public SessionTpidUnbindHandler(SessionManager sessionMgr) {
@@ -38,6 +45,18 @@ public class SessionTpidUnbindHandler extends BasicHttpHandler {
 
     @Override
     public void handleRequest(HttpServerExchange exchange) {
+        String auth = exchange.getRequestHeaders().getFirst("Authorization");
+        if (StringUtils.isNotEmpty(auth)) {
+            // We have a auth header to process
+            if (StringUtils.startsWith(auth, "X-Matrix ")) {
+                log.warn("A remote host attempted to unbind without proper authorization. Request was denied");
+                log.warn("See https://github.com/kamax-matrix/mxisd/wiki/mxisd-and-your-privacy for more info");
+                throw new NotAllowedException("3PID can only be removed via 3PID sessions, not via Homeserver signature");
+            } else {
+                throw new BadRequestException("Illegal authorization type");
+            }
+        }
+
         JsonObject body = parseJsonObject(exchange);
         sessionMgr.unbind(body);
         writeBodyAsUtf8(exchange, "{}");

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/SingleLookupHandler.java

@@ -21,9 +21,7 @@
 package io.kamax.mxisd.http.undertow.handler.identity.v1;
 
 import com.google.gson.JsonObject;
-import io.kamax.matrix.event.EventKey;
 import io.kamax.matrix.json.GsonUtil;
-import io.kamax.matrix.json.MatrixJson;
 import io.kamax.mxisd.config.MxisdConfig;
 import io.kamax.mxisd.config.ServerConfig;
 import io.kamax.mxisd.crypto.SignatureManager;
@@ -73,11 +71,8 @@ public class SingleLookupHandler extends LookupHandler {
             respondJson(exchange, "{}");
         } else {
             SingleLookupReply lookup = lookupOpt.get();
-
-            // FIXME signing should be done in the business model, not in the controller
             JsonObject obj = GsonUtil.makeObj(new SingeLookupReplyJson(lookup));
-            obj.add(EventKey.Signatures.get(), signMgr.signMessageGson(cfg.getName(), MatrixJson.encodeCanonical(obj)));
-
+            signMgr.signMessageGson(cfg.getName(), obj);
             respondJson(exchange, obj);
         }
     }

src/main/java/io/kamax/mxisd/invitation/InvitationManager.java

@@ -111,11 +111,11 @@ public class InvitationManager {
         this.notifMgr = notifMgr;
         this.profileMgr = profileMgr;
 
-        log.info("Loading saved invites");
+        log.debug("Loading saved invites");
         Collection<ThreePidInviteIO> ioList = storage.getInvites();
         ioList.forEach(io -> {
             io.getProperties().putIfAbsent(CreatedAtPropertyKey, defaultCreateTs);
-            log.info("Processing invite {}", GsonUtil.get().toJson(io));
+            log.debug("Processing invite {}", GsonUtil.get().toJson(io));
             ThreePidInvite invite = new ThreePidInvite(
                     MatrixID.asAcceptable(io.getSender()),
                     io.getMedium(),
@@ -127,6 +127,7 @@ public class InvitationManager {
             ThreePidInviteReply reply = new ThreePidInviteReply(io.getId(), invite, io.getToken(), "", Collections.emptyList());
             invitations.put(reply.getId(), reply);
         });
+        log.info("Loaded saved invites");
 
         // FIXME export such madness into matrix-java-sdk with a nice wrapper to talk to a homeserver
         try {
@@ -511,6 +512,9 @@ public class InvitationManager {
                     publishMapping(reply, lookup.getMxid().getId());
                 } else {
                     log.info("No mapping for pending invite {}", getIdForLog(reply));
+                    if (lookupMgr.getLocalProviders().isEmpty()) {
+                        log.warn("No Identity store has been configured, this invite may never resolve");
+                    }
                 }
             } catch (Throwable t) {
                 log.error("Unable to process invite", t);

src/main/java/io/kamax/mxisd/lookup/SingleLookupReply.java

@@ -72,7 +72,7 @@ public class SingleLookupReply {
     }
 
     public SingleLookupReply(SingleLookupRequest request, _MatrixID mxid) {
-        this(request, mxid, Instant.now(), Instant.ofEpochMilli(0), Instant.ofEpochMilli(253402300799000L));
+        this(request, mxid, Instant.now(), Instant.now().minusSeconds(60), Instant.now().plusSeconds(5 * 60));
     }
 
     public SingleLookupReply(SingleLookupRequest request, _MatrixID mxid, Instant timestamp, Instant notBefore, Instant notAfter) {

src/main/java/io/kamax/mxisd/matrix/IdentityServerUtils.java

@@ -79,7 +79,7 @@ public class IdentityServerUtils {
 
             JsonElement el = parser.parse(IOUtils.toString(res.getEntity().getContent(), StandardCharsets.UTF_8));
             if (!el.isJsonObject()) {
-                log.debug("IS {} did not send back an empty JSON object as per spec, not a valid IS");
+                log.debug("IS {} did not send back an empty JSON object as per spec, not a valid IS", remote);
                 return false;
             }
 
@@ -90,7 +90,7 @@ public class IdentityServerUtils {
         }
     }
 
-    public static String getSrvRecordName(String domain) {
+    private static String getSrvRecordName(String domain) {
         return "_matrix-identity._tcp." + domain;
     }
 

src/main/java/io/kamax/mxisd/session/SessionManager.java

@@ -27,20 +27,19 @@ import io.kamax.matrix._MatrixID;
 import io.kamax.matrix.json.GsonUtil;
 import io.kamax.mxisd.config.MatrixConfig;
 import io.kamax.mxisd.config.SessionConfig;
+import io.kamax.mxisd.exception.BadRequestException;
 import io.kamax.mxisd.exception.NotAllowedException;
-import io.kamax.mxisd.exception.NotImplementedException;
 import io.kamax.mxisd.exception.SessionNotValidatedException;
 import io.kamax.mxisd.exception.SessionUnknownException;
 import io.kamax.mxisd.lookup.SingleLookupReply;
+import io.kamax.mxisd.lookup.SingleLookupRequest;
 import io.kamax.mxisd.lookup.ThreePidValidation;
-import io.kamax.mxisd.lookup.strategy.LookupStrategy;
 import io.kamax.mxisd.notification.NotificationManager;
 import io.kamax.mxisd.storage.IStorage;
 import io.kamax.mxisd.storage.dao.IThreePidSessionDao;
 import io.kamax.mxisd.threepid.session.ThreePidSession;
 import org.apache.commons.lang3.RandomStringUtils;
 import org.apache.commons.lang3.StringUtils;
-import org.apache.http.impl.client.CloseableHttpClient;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -56,25 +55,17 @@ public class SessionManager {
     private MatrixConfig mxCfg;
     private IStorage storage;
     private NotificationManager notifMgr;
-    private LookupStrategy lookupMgr;
-
-    // FIXME export into central class, set version
-    private CloseableHttpClient client;
 
     public SessionManager(
             SessionConfig cfg,
             MatrixConfig mxCfg,
             IStorage storage,
-            NotificationManager notifMgr,
-            LookupStrategy lookupMgr,
-            CloseableHttpClient client
+            NotificationManager notifMgr
     ) {
         this.cfg = cfg;
         this.mxCfg = mxCfg;
         this.storage = storage;
         this.notifMgr = notifMgr;
-        this.lookupMgr = lookupMgr;
-        this.client = client;
     }
 
     private ThreePidSession getSession(String sid, String secret) {
@@ -128,7 +119,7 @@ public class SessionManager {
                 log.info("Generated new session {} to validate {} from server {}", sessionId, tpid, server);
 
                 storage.insertThreePidSession(session.getDao());
-                log.info("Stored session {}", sessionId, tpid, server);
+                log.info("Stored session {}", sessionId);
 
                 log.info("Session {} for {}: sending validation notification", sessionId, tpid);
                 notifMgr.sendForValidation(session);
@@ -157,7 +148,7 @@ public class SessionManager {
         return new ThreePidValidation(session.getThreePid(), session.getValidationTime());
     }
 
-    public void bind(String sid, String secret, String mxidRaw) {
+    public SingleLookupReply bind(String sid, String secret, String mxidRaw) {
         // We make sure we have an acceptable User ID
         if (StringUtils.isEmpty(mxidRaw)) {
             throw new IllegalArgumentException("No Matrix User ID provided");
@@ -171,60 +162,45 @@ public class SessionManager {
 
         // Only accept binds if the domain matches our own
         if (!StringUtils.equalsIgnoreCase(mxCfg.getDomain(), mxid.getDomain())) {
-            throw new NotAllowedException("Only Matrix IDs from domain " + mxCfg + " can be bound");
+            throw new NotAllowedException("Only Matrix IDs from domain " + mxCfg.getDomain() + " can be bound");
         }
 
         log.info("Session {}: Binding of {}:{} to Matrix ID {} is accepted",
                 session.getId(), session.getThreePid().getMedium(), session.getThreePid().getAddress(), mxid.getId());
+
+        SingleLookupRequest request = new SingleLookupRequest();
+        request.setType(session.getThreePid().getMedium());
+        request.setThreePid(session.getThreePid().getAddress());
+        return new SingleLookupReply(request, mxid);
     }
 
     public void unbind(JsonObject reqData) {
-        if (reqData.entrySet().size() == 2 && reqData.has("mxid") && reqData.has("threepid")) {
-            /* This is a HS request to remove a 3PID and is considered:
-             * - An attack on user privacy
-             * - A baffling spec breakage requiring IS and HS 3PID info to be independent [1]
-             * - A baffling spec breakage that 3PID (un)bind is only one way [2]
-             *
-             * Given the lack of response on our extensive feedback on the proposal [3] which has not landed in the spec yet [4],
-             * We'll be denying such unbind requests and will inform users using their 3PID that a fraudulent attempt of
-             * removing their 3PID binding has been attempted and blocked.
-             *
-             * [1]: https://matrix.org/docs/spec/client_server/r0.4.0.html#adding-account-administrative-contact-information
-             * [2]: https://matrix.org/docs/spec/identity_service/r0.1.0.html#privacy
-             * [3]: https://docs.google.com/document/d/135g2muVxmuml0iUnLoTZxk8M2ZSt3kJzg81chGh51yg/edit
-             * [4]: https://github.com/matrix-org/matrix-doc/issues/1194
-             */
-
-            log.warn("A remote host attempted to unbind without proper authorization. Request was denied");
-
-            if (!cfg.getPolicy().getUnbind().getFraudulent().getSendWarning()) {
-                log.info("Not sending notification to 3PID owner as per configuration");
-            } else {
-                log.info("Sending notification to 3PID owner as per configuration");
-
-                ThreePid tpid = GsonUtil.get().fromJson(GsonUtil.getObj(reqData, "threepid"), ThreePid.class);
-                Optional<SingleLookupReply> lookup = lookupMgr.findLocal(tpid.getMedium(), tpid.getAddress());
-                if (!lookup.isPresent()) {
-                    log.info("No 3PID owner found, not sending any notification");
-                } else {
-                    log.info("3PID owner found, sending notification");
+        _MatrixID mxid;
         try {
-                        notifMgr.sendForFraudulentUnbind(tpid);
-                        log.info("Notification sent");
-                    } catch (NotImplementedException e) {
-                        log.warn("Unable to send notification: {}", e.getMessage());
-                    } catch (RuntimeException e) {
-                        log.warn("Unable to send notification due to unknown error. See stacktrace below", e);
-                    }
-                }
+            mxid = MatrixID.asAcceptable(GsonUtil.getStringOrThrow(reqData, "mxid"));
+        } catch (IllegalArgumentException e) {
+            throw new BadRequestException(e.getMessage());
         }
 
-            throw new NotAllowedException("You have attempted to alter 3PID bindings, which can only be done by the 3PID owner directly. " +
-                    "We have informed the 3PID owner of your fraudulent attempt.");
+        String sid = GsonUtil.getStringOrNull(reqData, "sid");
+        String secret = GsonUtil.getStringOrNull(reqData, "client_secret");
+        ThreePid tpid = GsonUtil.get().fromJson(GsonUtil.getObj(reqData, "threepid"), ThreePid.class);
+
+        // We ensure the session was validated
+        ThreePidSession session = getSessionIfValidated(sid, secret);
+
+        // As per spec, we can only allow if the provided 3PID matches the validated session
+        if (!session.getThreePid().equals(tpid)) {
+            throw new BadRequestException("3PID to unbind does not match the one from the validated session");
         }
 
-        log.info("Denying unbind request as the endpoint is not defined in the spec.");
-        throw new NotAllowedException(499, "This endpoint does not exist in the spec and therefore is not supported.");
+        // We only allow unbind for the domain we manage, mirroring bind
+        if (!StringUtils.equalsIgnoreCase(mxCfg.getDomain(), mxid.getDomain())) {
+            throw new NotAllowedException("Only Matrix IDs from domain " + mxCfg.getDomain() + " can be unbound");
+        }
+
+        log.info("Session {}: Unbinding of {}:{} to Matrix ID {} is accepted",
+                session.getId(), session.getThreePid().getMedium(), session.getThreePid().getAddress(), mxid.getId());
     }
 
 }

src/main/java/io/kamax/mxisd/threepid/connector/phone/PhoneSmsTwilioConnector.java

@@ -21,10 +21,12 @@
 package io.kamax.mxisd.threepid.connector.phone;
 
 import com.twilio.Twilio;
+import com.twilio.exception.ApiException;
 import com.twilio.rest.api.v2010.account.Message;
 import com.twilio.type.PhoneNumber;
 import io.kamax.mxisd.config.threepid.connector.PhoneTwilioConfig;
-import io.kamax.mxisd.exception.BadRequestException;
+import io.kamax.mxisd.exception.InternalServerError;
+import io.kamax.mxisd.exception.NotImplementedException;
 import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -52,12 +54,17 @@ public class PhoneSmsTwilioConnector implements PhoneConnector {
     @Override
     public void send(String recipient, String content) {
         if (StringUtils.isBlank(cfg.getAccountSid()) || StringUtils.isBlank(cfg.getAuthToken()) || StringUtils.isBlank(cfg.getNumber())) {
-            throw new BadRequestException("Phone numbers cannot be validated at this time. Contact your administrator.");
+            log.error("Twilio connector in not fully configured and is missing mandatory configuration values.");
+            throw new NotImplementedException("Phone numbers cannot be validated at this time. Contact your administrator.");
         }
 
         recipient = "+" + recipient;
         log.info("Sending SMS notification from {} to {} with {} characters", cfg.getNumber(), recipient, content.length());
+        try {
             Message.creator(new PhoneNumber("+" + recipient), new PhoneNumber(cfg.getNumber()), content).create();
+        } catch (ApiException e) {
+            throw new InternalServerError(e);
+        }
     }
 
 }

src/main/java/io/kamax/mxisd/threepid/generator/PlaceholderNotificationGenerator.java

@@ -27,8 +27,9 @@ import io.kamax.mxisd.http.IsAPIv1;
 import io.kamax.mxisd.invitation.IMatrixIdInvite;
 import io.kamax.mxisd.invitation.IThreePidInviteReply;
 import io.kamax.mxisd.threepid.session.IThreePidSession;
-import org.apache.commons.lang.StringUtils;
+import io.kamax.mxisd.util.RestClientUtils;
 import org.apache.commons.lang.WordUtils;
+import org.apache.commons.lang3.StringUtils;
 
 import static io.kamax.mxisd.http.io.identity.StoreInviteRequest.Keys.RoomName;
 import static io.kamax.mxisd.http.io.identity.StoreInviteRequest.Keys.SenderDisplayName;
@@ -46,16 +47,26 @@ public abstract class PlaceholderNotificationGenerator {
     }
 
     protected String populateForCommon(ThreePid recipient, String input) {
+        if (StringUtils.isBlank(input)) {
+            return input;
+        }
+
         String domainPretty = WordUtils.capitalizeFully(mxCfg.getDomain());
 
         return input
                 .replace("%DOMAIN%", mxCfg.getDomain())
                 .replace("%DOMAIN_PRETTY%", domainPretty)
                 .replace("%RECIPIENT_MEDIUM%", recipient.getMedium())
-                .replace("%RECIPIENT_ADDRESS%", recipient.getAddress());
+                .replace("%RECIPIENT_MEDIUM_URL_ENCODED%", RestClientUtils.urlEncode(recipient.getMedium()))
+                .replace("%RECIPIENT_ADDRESS%", recipient.getAddress())
+                .replace("%RECIPIENT_ADDRESS_URL_ENCODED%", RestClientUtils.urlEncode(recipient.getAddress()));
     }
 
     protected String populateForInvite(IMatrixIdInvite invite, String input) {
+        if (StringUtils.isBlank(input)) {
+            return input;
+        }
+
         String senderName = invite.getProperties().getOrDefault(SenderDisplayName, "");
         String senderNameOrId = StringUtils.defaultIfBlank(senderName, invite.getSender().getId());
         String roomName = invite.getProperties().getOrDefault(RoomName, "");
@@ -72,6 +83,10 @@ public abstract class PlaceholderNotificationGenerator {
     }
 
     protected String populateForReply(IThreePidInviteReply invite, String input) {
+        if (StringUtils.isBlank(input)) {
+            return input;
+        }
+
         ThreePid tpid = new ThreePid(invite.getInvite().getMedium(), invite.getInvite().getAddress());
 
         String senderName = invite.getInvite().getProperties().getOrDefault(SenderDisplayName, "");
@@ -86,13 +101,19 @@ public abstract class PlaceholderNotificationGenerator {
                 .replace("%SENDER_NAME%", senderName)
                 .replace("%SENDER_NAME_OR_ID%", senderNameOrId)
                 .replace("%INVITE_MEDIUM%", tpid.getMedium())
+                .replace("%INVITE_MEDIUM_URL_ENCODED%", RestClientUtils.urlEncode(tpid.getMedium()))
                 .replace("%INVITE_ADDRESS%", tpid.getAddress())
+                .replace("%INVITE_ADDRESS_URL_ENCODED%", RestClientUtils.urlEncode(tpid.getAddress()))
                 .replace("%ROOM_ID%", invite.getInvite().getRoomId())
                 .replace("%ROOM_NAME%", roomName)
                 .replace("%ROOM_NAME_OR_ID%", roomNameOrId);
     }
 
     protected String populateForValidation(IThreePidSession session, String input) {
+        if (StringUtils.isBlank(input)) {
+            return input;
+        }
+
         String validationLink = srvCfg.getPublicUrl() + IsAPIv1.getValidate(
                 session.getThreePid().getMedium(),
                 session.getId(),

src/main/java/io/kamax/mxisd/threepid/notification/BuiltInNotificationHandlerSupplier.java

@@ -20,7 +20,7 @@
 
 package io.kamax.mxisd.threepid.notification;
 
-import com.google.gson.JsonObject;
+import com.google.gson.JsonSyntaxException;
 import io.kamax.matrix.ThreePidMedium;
 import io.kamax.matrix.json.GsonUtil;
 import io.kamax.mxisd.Mxisd;
@@ -65,13 +65,18 @@ public class BuiltInNotificationHandlerSupplier implements NotificationHandlerSu
         if (StringUtils.equals(EmailRawNotificationHandler.ID, handler)) {
             Object o = mxisd.getConfig().getThreepid().getMedium().get(ThreePidMedium.Email.getId());
             if (Objects.nonNull(o)) {
-                EmailConfig emailCfg = GsonUtil.get().fromJson(GsonUtil.makeObj(o), EmailConfig.class);
+                EmailConfig emailCfg;
+                try {
+                    emailCfg = GsonUtil.get().fromJson(GsonUtil.makeObj(o), EmailConfig.class);
+                } catch (JsonSyntaxException e) {
+                    throw new ConfigurationException("Invalid configuration for threepid email notification");
+                }
 
-                if (org.apache.commons.lang.StringUtils.isBlank(emailCfg.getGenerator())) {
+                if (StringUtils.isBlank(emailCfg.getGenerator())) {
                     throw new ConfigurationException("notification.email.generator");
                 }
 
-                if (org.apache.commons.lang.StringUtils.isBlank(emailCfg.getConnector())) {
+                if (StringUtils.isBlank(emailCfg.getConnector())) {
                     throw new ConfigurationException("notification.email.connector");
                 }
 
@@ -94,9 +99,15 @@ public class BuiltInNotificationHandlerSupplier implements NotificationHandlerSu
         }
 
         if (StringUtils.equals(EmailSendGridNotificationHandler.ID, handler)) {
-            JsonObject cfgJson = mxisd.getConfig().getNotification().getHandlers().get(EmailSendGridNotificationHandler.ID);
+            Object cfgJson = mxisd.getConfig().getNotification().getHandlers().get(EmailSendGridNotificationHandler.ID);
             if (Objects.nonNull(cfgJson)) {
-                EmailSendGridConfig cfg = GsonUtil.get().fromJson(cfgJson, EmailSendGridConfig.class);
+                EmailSendGridConfig cfg;
+                try {
+                    cfg = GsonUtil.get().fromJson(GsonUtil.get().toJson(cfgJson), EmailSendGridConfig.class);
+                } catch (JsonSyntaxException e) {
+                    throw new ConfigurationException("Invalid configuration for threepid email sendgrid handler");
+                }
+
                 NotificationHandlers.register(() -> new EmailSendGridNotificationHandler(mxisd.getConfig(), cfg));
             }
         }
@@ -107,7 +118,12 @@ public class BuiltInNotificationHandlerSupplier implements NotificationHandlerSu
         if (StringUtils.equals(PhoneNotificationHandler.ID, handler)) {
             Object o = mxisd.getConfig().getThreepid().getMedium().get(ThreePidMedium.PhoneNumber.getId());
             if (Objects.nonNull(o)) {
-                PhoneConfig cfg = GsonUtil.get().fromJson(GsonUtil.makeObj(o), PhoneConfig.class);
+                PhoneConfig cfg;
+                try {
+                    cfg = GsonUtil.get().fromJson(GsonUtil.makeObj(o), PhoneConfig.class);
+                } catch (JsonSyntaxException e) {
+                    throw new ConfigurationException("Invalid configuration for threepid msisdn notification");
+                }
 
                 List<PhoneGenerator> generators = StreamSupport
                         .stream(ServiceLoader.load(PhoneGeneratorSupplier.class).spliterator(), false)

src/main/java/io/kamax/mxisd/threepid/notification/email/EmailSendGridNotificationHandler.java

@@ -33,7 +33,7 @@ import io.kamax.mxisd.notification.NotificationHandler;
 import io.kamax.mxisd.threepid.generator.PlaceholderNotificationGenerator;
 import io.kamax.mxisd.threepid.session.IThreePidSession;
 import io.kamax.mxisd.util.FileUtil;
-import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -86,6 +86,9 @@ public class EmailSendGridNotificationHandler extends PlaceholderNotificationGen
     @Override
     public void sendForInvite(IMatrixIdInvite invite) {
         EmailTemplate template = cfg.getTemplates().getGeneric().get("matrixId");
+        if (StringUtils.isAllBlank(template.getBody().getText(), template.getBody().getHtml())) {
+            throw new FeatureNotAvailable("No template has been configured for Matrix ID invite notifications");
+        }
 
         Email email = getEmail();
         email.setSubject(populateForInvite(invite, template.getSubject()));
@@ -98,6 +101,10 @@ public class EmailSendGridNotificationHandler extends PlaceholderNotificationGen
     @Override
     public void sendForReply(IThreePidInviteReply invite) {
         EmailTemplate template = cfg.getTemplates().getInvite();
+        if (StringUtils.isAllBlank(template.getBody().getText(), template.getBody().getHtml())) {
+            throw new FeatureNotAvailable("No template has been configured for 3PID invite notifications");
+        }
+
         Email email = getEmail();
         email.setSubject(populateForReply(invite, template.getSubject()));
         email.setText(populateForReply(invite, getFromFile(template.getBody().getText())));
@@ -109,6 +116,10 @@ public class EmailSendGridNotificationHandler extends PlaceholderNotificationGen
     @Override
     public void sendForValidation(IThreePidSession session) {
         EmailTemplate template = cfg.getTemplates().getSession().getValidation();
+        if (StringUtils.isAllBlank(template.getBody().getText(), template.getBody().getHtml())) {
+            throw new FeatureNotAvailable("No template has been configured for validation notifications");
+        }
+
         Email email = getEmail();
         email.setSubject(populateForValidation(session, template.getSubject()));
         email.setText(populateForValidation(session, getFromFile(template.getBody().getText())));
@@ -120,6 +131,10 @@ public class EmailSendGridNotificationHandler extends PlaceholderNotificationGen
     @Override
     public void sendForFraudulentUnbind(ThreePid tpid) {
         EmailTemplate template = cfg.getTemplates().getSession().getUnbind().getFraudulent();
+        if (StringUtils.isAllBlank(template.getBody().getText(), template.getBody().getHtml())) {
+            throw new FeatureNotAvailable("No template has been configured for fraudulent unbind notifications");
+        }
+
         Email email = getEmail();
         email.setSubject(populateForCommon(tpid, template.getSubject()));
         email.setText(populateForCommon(tpid, getFromFile(template.getBody().getText())));

src/main/java/io/kamax/mxisd/util/RestClientUtils.java

@@ -25,12 +25,22 @@ import org.apache.http.client.methods.HttpPost;
 import org.apache.http.entity.ContentType;
 import org.apache.http.entity.StringEntity;
 
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
 import java.nio.charset.StandardCharsets;
 
 public class RestClientUtils {
 
     private static Gson gson = GsonUtil.build();
 
+    public static String urlEncode(String value) {
+        try {
+            return URLEncoder.encode(value, StandardCharsets.UTF_8.name());
+        } catch (UnsupportedEncodingException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
     public static HttpPost post(String url, String body) {
         StringEntity entity = new StringEntity(body, StandardCharsets.UTF_8);
         entity.setContentType(ContentType.APPLICATION_JSON.toString());

src/main/resources/threepids/email/invite-template.eml

@@ -9,19 +9,12 @@ Content-Disposition: inline
 
 Hi,
 
-%SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on
-Matrix. To join the conversation, register an account on %REGISTER_URL%
+%SENDER_NAME_OR_ID% has invited you to the Matrix room [%ROOM_NAME_OR_ID%].
+To join the conversation, register an account using %REGISTER_URL%
 
+You may be required to provide the same email used for this invite during registration.
 You can also register an account on a public server and get in touch with them.
 
-
-About Matrix:
-
-Matrix is an open standard for interoperable, decentralised, real-time communication
-over IP, supporting group chat, file transfer, voice and video calling, integrations to
-other apps, bridges to other communication systems and much more. It can be used to power
-Instant Messaging, VoIP/WebRTC signalling, Internet of Things communication.
-
 Thanks,
 
 %DOMAIN_PRETTY% Admins
@@ -68,18 +61,11 @@ pre, code {
                 <td id="inner">
                     <p>Hi,</p>
 
-<p>%SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on
-Matrix. To join the conversation, register an account on <a href="%REGISTER_URL%">%DOMAIN%</a>.</p>
+                    <p>%SENDER_NAME_OR_ID% has invited you to the Matrix room [%ROOM_NAME_OR_ID%].<br/>
+                    To join the conversation, register an account on <a href="%REGISTER_URL%">%DOMAIN%</a>.</p>
 
-<pYou can also register an account on a public server and get in touch with them.</p>
-
-<br>
-<p>About Matrix:</p>
-
-<p>Matrix is an open standard for interoperable, decentralised, real-time communication
-   over IP, supporting group chat, file transfer, voice and video calling, integrations to
-   other apps, bridges to other communication systems and much more. It can be used to power
-   Instant Messaging, VoIP/WebRTC signalling, Internet of Things communication.</p>
+                    <p>You may be required to provide the same email used for this invite during registration.<br/>
+                    You can also register an account on a public server and get in touch with them.</p>
 
                     <p>Thanks,</p>
 

src/main/resources/threepids/email/mxid-template.eml

@@ -9,7 +9,7 @@ Content-Disposition: inline
 
 Hi,
 
-%SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on Matrix.
+%SENDER_NAME_OR_ID% has invited you to the Matrix room [%ROOM_NAME_OR_ID%].
 
 Thanks,
 
@@ -57,7 +57,7 @@ pre, code {
                 <td id="inner">
                     <p>Hi,</p>
 
-<p>%SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on Matrix.</p>
+                    <p>%SENDER_NAME_OR_ID% has invited you to the Matrix room [%ROOM_NAME_OR_ID%].</p>
 
                     <p>Thanks,</p>
 

src/test/java/io/kamax/mxisd/test/MxisdDefaultTest.java

@@ -33,11 +33,7 @@ public class MxisdDefaultTest {
 
     @Test
     public void defaultConfig() {
-        MxisdConfig cfg = new MxisdConfig();
-        cfg.getMatrix().setDomain(domain);
-        cfg.getKey().setPath(":memory:");
-        cfg.getStorage().getProvider().getSqlite().setDatabase(":memory:");
-
+        MxisdConfig cfg = MxisdConfig.forDomain(domain).inMemory();
         Mxisd m = new Mxisd(cfg);
         m.start();
 

src/test/java/io/kamax/mxisd/test/MxisdTest.java

@@ -41,10 +41,7 @@ public class MxisdTest {
 
     @Before
     public void before() {
-        MxisdConfig cfg = new MxisdConfig();
-        cfg.getMatrix().setDomain("localhost");
-        cfg.getKey().setPath(":memory:");
-        cfg.getStorage().getProvider().getSqlite().setDatabase(":memory:");
+        MxisdConfig cfg = MxisdConfig.forDomain("localhost").inMemory();
 
         MemoryThreePid mem3pid = new MemoryThreePid();
         mem3pid.setMedium("email");

src/test/java/io/kamax/mxisd/test/auth/AuthManagerTest.java

@@ -0,0 +1,77 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.test.auth;
+
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.auth.AuthManager;
+import io.kamax.mxisd.auth.UserAuthResult;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.config.memory.MemoryIdentityConfig;
+import io.kamax.mxisd.config.memory.MemoryThreePid;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+public class AuthManagerTest {
+
+    private static AuthManager mgr;
+
+    // FIXME we should be able to easily build the class ourselves
+    // FIXME use constants
+    @BeforeClass
+    public static void beforeClass() {
+        MxisdConfig cfg = new MxisdConfig();
+        cfg.getMatrix().setDomain("localhost");
+        cfg.getKey().setPath(":memory:");
+        cfg.getStorage().getProvider().getSqlite().setDatabase(":memory:");
+
+        MemoryThreePid mem3pid = new MemoryThreePid();
+        mem3pid.setMedium("email");
+        mem3pid.setAddress("john@localhost");
+        MemoryIdentityConfig validCfg = new MemoryIdentityConfig();
+        validCfg.setUsername("john");
+        validCfg.setPassword("doe");
+        validCfg.getThreepids().add(mem3pid);
+        MemoryIdentityConfig illegalUser = new MemoryIdentityConfig();
+        illegalUser.setUsername("JANE");
+        illegalUser.setPassword("doe");
+        cfg.getMemory().setEnabled(true);
+        cfg.getMemory().getIdentities().add(validCfg);
+        cfg.getMemory().getIdentities().add(illegalUser);
+
+        Mxisd m = new Mxisd(cfg);
+        m.start();
+        mgr = m.getAuth();
+    }
+
+    @Test
+    public void basic() {
+        UserAuthResult result = mgr.authenticate("@john:localhost", "doe");
+        assertTrue(result.isSuccess());
+
+        // For backward-compatibility as per instructed by the spec, we do not fail on an illegal username
+        // This makes sure we don't break it
+        result = mgr.authenticate("@JANE:localhost", "doe");
+        assertTrue(result.isSuccess());
+    }
+
+}

src/test/java/io/kamax/mxisd/test/crypto/SignatureManagerTest.java

@@ -21,8 +21,10 @@
 package io.kamax.mxisd.test.crypto;
 
 import com.google.gson.JsonObject;
+import io.kamax.matrix.event.EventKey;
 import io.kamax.matrix.json.GsonUtil;
 import io.kamax.matrix.json.MatrixJson;
+import io.kamax.mxisd.config.MxisdConfig;
 import io.kamax.mxisd.crypto.Signature;
 import io.kamax.mxisd.crypto.SignatureManager;
 import io.kamax.mxisd.crypto.ed25519.Ed25519Key;
@@ -36,10 +38,14 @@ import org.junit.Test;
 
 import static org.hamcrest.core.Is.is;
 import static org.hamcrest.core.IsEqual.equalTo;
+import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertThat;
 
 public class SignatureManagerTest {
 
+    private static final String lookupData = "{\n" + "  \"not_before\": 0,\n" + "  \"address\": \"mxisd-federation-test@kamax.io\",\n"
+            + "  \"medium\": \"email\",\n" + "  \"mxid\": \"@mxisd-lookup-test:kamax.io\",\n"
+            + "  \"not_after\": 253402300799000,\n" + "  \"ts\": 1523482030147\n" + "}";
     private static SignatureManager signMgr;
 
     private static SignatureManager build(String keySeed) {
@@ -47,7 +53,7 @@ public class SignatureManagerTest {
         KeyStore store = new MemoryKeyStore();
         store.add(key);
 
-        return new Ed25519SignatureManager(new Ed25519KeyManager(store));
+        return new Ed25519SignatureManager(MxisdConfig.forDomain("localhost").inMemory().build(), new Ed25519KeyManager(store));
     }
 
     @BeforeClass
@@ -98,12 +104,19 @@ public class SignatureManagerTest {
 
     @Test
     public void onIdentityLookup() {
-        String value = MatrixJson.encodeCanonical("{\n" + "  \"address\": \"mxisd-federation-test@kamax.io\",\n"
-                + "  \"medium\": \"email\",\n" + "  \"mxid\": \"@mxisd-lookup-test:kamax.io\",\n"
-                + "  \"not_after\": 253402300799000,\n" + "  \"not_before\": 0,\n" + "  \"ts\": 1523482030147\n" + "}");
-
+        String value = MatrixJson.encodeCanonical(lookupData);
         String sign = "ObKA4PNQh2g6c7Yo2QcTcuDgIwhknG7ZfqmNYzbhrbLBOqZomU22xX9raufN2Y3ke1FXsDqsGs7WBDodmzZJCg";
         testSign(value, sign);
     }
 
+    @Test
+    public void onIdentityLookupFull() {
+        JsonObject data = GsonUtil.parseObj(lookupData);
+        signMgr.signMessageGson("localhost", data);
+        JsonObject signatures = EventKey.Signatures.getObj(data);
+        JsonObject domainSign = GsonUtil.getObj(signatures, "localhost");
+        String sign = GsonUtil.getStringOrThrow(domainSign, "ed25519:0");
+        assertEquals(sign, "ObKA4PNQh2g6c7Yo2QcTcuDgIwhknG7ZfqmNYzbhrbLBOqZomU22xX9raufN2Y3ke1FXsDqsGs7WBDodmzZJCg");
+    }
+
 }

src/test/java/io/kamax/mxisd/test/util/RestClientUtilsTest.java

@@ -0,0 +1,36 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.test.util;
+
+import io.kamax.mxisd.util.RestClientUtils;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+public class RestClientUtilsTest {
+
+    @Test
+    public void urlEncode() {
+        String encoded = RestClientUtils.urlEncode("john+doe@example.org");
+        assertEquals("john%2Bdoe%40example.org", encoded);
+    }
+
+}