Document mxisd does not require any maintenance task for day-to-day operations

- Give means to increase logging verbosity
- Explain how to do in the troubleshooting guide

docs/features/experimental/application-service.md

@@ -21,7 +21,7 @@ Under the `appsvc` namespace:
 
 | Key                   | Type    | Required | Default | Purpose                                                        |
 |-----------------------|---------|----------|---------|----------------------------------------------------------------|
-| `enabled`             | boolean | No       | `true`  | Globally enable/disable the feature                            |
+| `enabled`             | boolean | No       | `false` | Globally enable/disable the feature                            |
 | `user.main`           | string  | No       | `mxisd` | Localpart for the main appservice user                         |
 | `endpoint.toHS.url`   | string  | Yes      | *None*  | Base URL to the Homeserver                                     |
 | `endpoint.toHS.token` | string  | Yes      | *None*  | Token to use when sending requests to the Homeserver           |
@@ -31,6 +31,7 @@ Under the `appsvc` namespace:
 #### Example
 ```yaml
 appsvc:
+  enabled: true
   endpoint:
     toHS:
       url: 'http://localhost:8008'

docs/getting-started.md

@@ -153,3 +153,5 @@ infrastructure:
 
 - [Enable extra features](features/)
 - [Use your own Identity stores](stores/README.md)
+- [Hardening your mxisd installation](install/security.md)
+- [Learn about day-to-day operations](operations.md)

docs/install/docker.md

@@ -20,3 +20,6 @@ docker run --rm -e MATRIX_DOMAIN=example.org -v /data/mxisd/etc:/etc/mxisd -v /d
 ```
 
 For more info, including the list of possible tags, see [the public repository](https://hub.docker.com/r/kamax/mxisd/)
+
+## Troubleshoot
+Please read the [Troubleshooting guide](../troubleshooting.md).

docs/install/security.md

@@ -0,0 +1,30 @@
+# Security hardening
+## Overview
+This document outlines the various operations you may want to perform to increase the security of your installation and
+avoid leak of credentials/key pairs
+
+## Configuration
+Your config file should have the following ownership:
+- Dedicated user for mxisd, used to run the software
+- Dedicated group for mxisd, used by other applications to access and read configuration files
+
+Your config file should have the following access:
+- Read and write for the mxisd user
+- Read for the mxisd group
+- Nothing for others
+
+This translates into `640` and be applied with `chmod 640 /path/to/config/file.yaml`.
+
+## Data
+The only sensible place is the key store where mxisd's signing keys are stored. You should therefore limit access to only
+the mxisd user, and deny access to anything else.
+
+Your key store should have the following access:
+- Read and write for the mxisd user
+- Nothing for the mxisd group
+- Nothing for others
+
+The identity store can either be a file or a directory, depending on your version. v1.4 and higher are using a directory,
+everything before is using a file.
+- If your version is directory-based, you will want to apply chmod `700` on it.
+- If your version is file-based, you will want to apply chmod `600` on it.

docs/operations.md

@@ -0,0 +1,21 @@
+# Operations Guide
+- [Overview](#overview)
+- [Maintenance](#maintenance)
+- [Backuo](#backup)
+
+## Overview
+This document gives various information for the day-to-day management and operations of mxisd.
+
+## Maintenance
+mxisd does not require any maintenance task to run at optimal performance.
+
+## Backup
+### Run
+mxisd requires all file in its configuration and data directory to be backed up.  
+They are usually located at:
+- `/etc/mxisd`
+- `/var/lib/mxisd`
+
+### Restore
+Reinstall mxisd, restore the two folders above in the appropriate location (depending on your install method) and you
+will be good to go. Simply start mxisd to restore functionality.

docs/stores/ldap.md

@@ -89,7 +89,7 @@ ldap:
 #### 3PIDs
 You can also change the attribute lists for 3PID, like email or phone numbers.
 
-The following example would overwrite the [default list of attributes](../../src/main/resources/application.yaml#L67)
+The following example would overwrite the [default list of attributes](../../src/main/java/io/kamax/mxisd/config/ldap/LdapConfig.java#L64)
 for emails and phone number:
 ```yaml
 ldap:

docs/stores/sql.md

@@ -102,9 +102,41 @@ sql:
 ```
 
 ### Identity
+**NOTE**: Only single lookup is supported. Bulk lookup always returns no mapping. This is a restriction as the Matrix API
+does not allow paging or otherwise limit of results of the API, potentially leading to thousands and thousands 3PIDs at once.
+
 ```yaml
 sql:
   identity:
+    enabled: <boolean>
     type: <string>
     query: <string>
+    medium:
+     mediumTypeExample: <dedicated query>
 ```
+`type` is used to tell mxisd how to process the returned `uid` column containing the User ID:
+- `localpart` will build a full Matrix ID using the `matrix.domain` value.
+- `mxid` will use the ID as-is. If it is not a valid Matrix ID, lookup(s) will fail.
+
+A specific query can also given per 3PID medium type.
+
+### Profile
+```yaml
+sql:
+  profile:
+    enabled: <boolean>
+    displayName:
+      query: <string>
+    threepid:
+      query: <string>
+    role:
+      type: <string>
+      query: <string>
+    
+
+```
+For the `role` query, `type` can be used to tell mxisd how to inject the User ID in the query:
+- `localpart` will extract and set only the localpart.
+- `mxid` will use the ID as-is.
+
+On each query, the first parameter `?` is set as a string with the corresponding ID format.

docs/threepids/medium/email/smtp-connector.md

@@ -12,8 +12,8 @@ threepid:
       connectors:
         smtp:
           host: 'smtpHostname'
-          port: 587
+          tls: 1 # 0 = no STARTLS, 1 = try, 2 = force, 3 = TLS/SSL
-          tls: 1 # 0 = no STARTLS, 1 = try, 2 = force
+          port: 587 # Set appropriate value depending on your TLS setting
           login: 'smtpLogin'
           password: 'smtpPassword'
 ```

docs/threepids/medium/msisdn/twilio-connector.md

@@ -8,7 +8,7 @@ threepid:
     msisdn:
       connectors:
         twilio:
-          accountSid: 'myAccountSid'
+          account_sid: 'myAccountSid'
-          authToken: 'myAuthToken'
+          auth_token: 'myAuthToken'
           number: '+123456789'
 ```

docs/threepids/notification/template-generator.md

@@ -1,19 +1,99 @@
-# Notifications: Generate from templates
+# Notifications: Template generator
-To create notification content, you can use the `template` generator if supported for the 3PID medium which will read
+Most of the Identity actions will trigger a notification of some kind, informing the user of some confirmation, next step
-content from configured files.
+or just informing them about the current state of things.
 
-Placeholders can be integrated into the templates to dynamically populate such content with relevant information like
+Those notifications are by default generated from templates and by replacing placeholder tokens in them with the relevant
-the 3PID that was requested, the domain of your Identity server, etc.
+values of the notification. It is possible to customize the value of some placeholders, making easy to set values in the builtin templates, and/or
+provide your own custom templates.
 
-Templates can be configured for each event that would send a notification to the end user. Events share a set of common
+Templates for the following events/actions are available:
-placeholders and also have their own individual set of placeholders.
+- [3PID invite](../../features/identity.md)
+- [3PID session: validation](../session/session.md)
+- [3PID session: fraudulent unbind](https://github.com/kamax-matrix/mxisd/wiki/mxisd-and-your-privacy#improving-your-privacy-one-commit-at-the-time)
+- [Matrix ID invite](../../features/experimental/application-service.md#email-notification-about-room-invites-by-matrix-ids)
+
+## Placeholders
+All placeholders **MUST** be surrounded with `%` in the template. Per example, the `DOMAIN` placeholder would become
+`%DOMAIN%` within the template. This ensures replacement doesn't happen on non-placeholder strings.
+
+### Global
+The following placeholders are available in every template:
+
+| Placeholder         | Purpose                                                                      |
+|---------------------|------------------------------------------------------------------------------|
+| `DOMAIN`            | Identity server authoritative domain, as configured in `matrix.domain`       |
+| `DOMAIN_PRETTY`     | Same as `DOMAIN` with the first letter upper case and all other lower case   |
+| `FROM_EMAIL`        | Email address configured in `threepid.medium.<3PID medium>.identity.from`    |
+| `FROM_NAME`         | Name configured in `threepid.medium.<3PID medium>.identity.name`             |
+| `RECIPIENT_MEDIUM`  | The 3PID medium, like `email` or `msisdn`                                    |
+| `RECIPIENT_ADDRESS` | The address to which the notification is sent                                |
+
+### Room invitation
+Specific placeholders:
+
+| Placeholder         | Purpose                                                                                  |
+|---------------------|------------------------------------------------------------------------------------------|
+| `SENDER_ID`         | Matrix ID of the user who made the invite                                                |
+| `SENDER_NAME`       | Display name of the user who made the invite, if not available/set, empty                |
+| `SENDER_NAME_OR_ID` | Display name of the user who made the invite. If not available/set, its Matrix ID        |
+| `INVITE_MEDIUM`     | The 3PID medium for the invite.                                                          |
+| `INVITE_ADDRESS`    | The 3PID address for the invite.                                                         |
+| `ROOM_ID`           | The Matrix ID of the Room in which the invite took place                                 |
+| `ROOM_NAME`         | The Name of the room in which the invite took place. If not available/set, empty         |
+| `ROOM_NAME_OR_ID`   | The Name of the room in which the invite took place. If not available/set, its Matrix ID |
+| `REGISTER_URL`      | The URL to provide to the user allowing them to register their account, if needed        |
+
+### Validation of 3PID Session
+Specific placeholders:
+
+| Placeholder        | Purpose                                                                              |
+|--------------------|--------------------------------------------------------------------------------------|
+| `VALIDATION_LINK`  | URL, including token, to validate the 3PID session.                                  |
+| `VALIDATION_TOKEN` | The token needed to validate the session, in case the user cannot use the link.      |
+| `NEXT_URL`         | URL to redirect to after the sessions has been validated.                            |
+
+## Templates
+mxisd comes with a set of builtin templates to easily get started. Those templates can be found
+[in the repository](https://github.com/kamax-matrix/mxisd/tree/master/src/main/resources/threepids). If you want to use
+customized templates, we recommend using the builtin templates as a starting point.
+
+> **NOTE**: The link above point to the latest version of the built-in templates. Those might be different from your
+version. Be sure to view the repo at the current tag.
 
 ## Configuration
+All configuration is specific to [3PID mediums](https://matrix.org/docs/spec/appendices.html#pid-types) and happen
+under the namespace `threepid.medium.<medium>.generators.template`.
+
+Under such namespace, the following keys are available:
+- `invite`: Path to the 3PID invite notification template
+- `session.validation`: Path to the 3PID session validation notification template
+- `session.unbind.fraudulent`: Path to the 3PID session fraudulent unbind notification template
+- `generic.matrixId`: Path to the Matrix ID invite notification template
+- `placeholder`: Map of key/values to set static values for some placeholders.
+
+The `placeholder` map supports the following keys, mapped to their respective template placeholders:
+- `REGISTER_URL`
+
+### Example
+#### Simple
+```yaml
+threepid:
+  medium:
+    email:
+      generators:
+        template:
+          placeholder:
+            REGISTER_URL: 'https://matrix-client.example.org'
+```
+In this configuration, the builtin templates are used and a static value for the `REGISTER_URL` is set, allowing to point
+a newly invited user to a webapp allowing the creation of its account on the server.
+
+#### Advanced
 To configure paths to the various templates:
 ```yaml
 threepid:
   medium:
-    <YOUR 3PID MEDIUM HERE>:
+    email:
       generators:
         template:
           invite: '/path/to/invite-template.eml'
@@ -23,41 +103,7 @@ threepid:
               fraudulent: '/path/to/unbind-fraudulent-template.eml'
           generic:
             matrixId: '/path/to/mxid-invite-template.eml'
+          placeholder:
+            REGISTER_URL: 'https://matrix-client.example.org'
 ```
-The `template` generator is usually the default, so no further configuration is needed.
+In this configuration, a custom template is used for each event and a static value for the `REGISTER_URL` is set.
-
-##  Global placeholders
-| Placeholder           | Purpose                                                                      |
-|-----------------------|------------------------------------------------------------------------------|
-| `%DOMAIN%`            | Identity server authoritative domain, as configured in `matrix.domain`       |
-| `%DOMAIN_PRETTY%`     | Same as `%DOMAIN%` with the first letter upper case and all other lower case |
-| `%FROM_EMAIL%`        | Email address configured in `threepid.medium.<3PID medium>.identity.from`    |
-| `%FROM_NAME%`         | Name configured in `threepid.medium.<3PID medium>.identity.name`             |
-| `%RECIPIENT_MEDIUM%`  | The 3PID medium, like `email` or `msisdn`                                    |
-| `%RECIPIENT_ADDRESS%` | The address to which the notification is sent                                |
-
-## Events
-### Room invitation
-This template is used when someone is invited into a room using an email address which has no known bind to a Matrix ID.
-#### Placeholders
-| Placeholder           | Purpose                                                                                  |
-|-----------------------|------------------------------------------------------------------------------------------|
-| `%SENDER_ID%`         | Matrix ID of the user who made the invite                                                |
-| `%SENDER_NAME%`       | Display name of the user who made the invite, if not available/set, empty                |
-| `%SENDER_NAME_OR_ID%` | Display name of the user who made the invite. If not available/set, its Matrix ID        |
-| `%INVITE_MEDIUM%`     | The 3PID medium for the invite.                                                          |
-| `%INVITE_ADDRESS%`    | The 3PID address for the invite.                                                         |
-| `%ROOM_ID%`           | The Matrix ID of the Room in which the invite took place                                 |
-| `%ROOM_NAME%`         | The Name of the room in which the invite took place. If not available/set, empty         |
-| `%ROOM_NAME_OR_ID%`   | The Name of the room in which the invite took place. If not available/set, its Matrix ID |
-
-### Validation of 3PID Session
-This template is used when to user which added their 3PID address to their profile/settings and the session policy
-allows at least local sessions.  
-
-#### Placeholders
-| Placeholder          | Purpose                                                                              |
-|----------------------|--------------------------------------------------------------------------------------|
-| `%VALIDATION_LINK%`  | URL, including token, to validate the 3PID session.                                  |
-| `%VALIDATION_TOKEN%` | The token needed to validate the session, in case the user cannot use the link.      |
-| `%NEXT_URL%`         | URL to redirect to after the sessions has been validated.                            |

docs/troubleshooting.md

@@ -19,6 +19,11 @@ If you use the [Docker image](install/docker.md), this goes to the container log
 
 For any other platform, please refer to your package maintainer.
 
+### Increase verbosity
+To increase log verbosity and better track issues, the following means are available:
+- Add the `-v` command line parameter
+- Use the environment variable and value `MXISD_LOG_LEVEL=debug`
+
 ### Reading them
 Before reporting an issue, it is important to produce clean and complete logs so they can be understood.
 

mxisd.example.yaml

@@ -91,19 +91,19 @@ threepid:
           # SMTP host
           host: "smtp.example.org"
 
-          # SMTP port
+          # TLS mode for the connection
-          port: 587
-
-          # STARTLS mode for the connection.
-          # SSL/TLS is currently not supported. See https://github.com/kamax-matrix/mxisd/issues/125
-          #
           # Possible values:
           #  0    Disable any kind of TLS entirely
           #  1    Enable STARTLS if supported by server (default)
           #  2    Force STARTLS and fail if not available
+          #  3    Use full TLS/SSL instead of STARTLS
           #
           tls: 1
 
+          # SMTP port
+          # Be sure to adapt depending on your TLS choice, if changed from default
+          port: 587
+
           # Login for SMTP
           login: "matrix-identity@example.org"
 

src/main/java/io/kamax/mxisd/HttpMxisd.java

@@ -73,7 +73,6 @@ public class HttpMxisd {
         HttpHandler asNotFoundHandler = SaneHandler.around(new AsNotFoundHandler(m.getAs()));
 
         HttpHandler storeInvHandler = SaneHandler.around(new StoreInviteHandler(m.getConfig().getServer(), m.getInvite(), m.getKeyManager()));
-        HttpHandler sessValidateHandler = SaneHandler.around(new SessionValidateHandler(m.getSession(), m.getConfig().getServer(), m.getConfig().getView()));
 
         httpSrv = Undertow.builder().addHttpListener(m.getConfig().getServer().getPort(), "0.0.0.0").setHandler(Handlers.routing()
 
@@ -103,8 +102,8 @@ public class HttpMxisd {
                 .post(BulkLookupHandler.Path, SaneHandler.around(new BulkLookupHandler(m.getIdentity())))
                 .post(StoreInviteHandler.Path, storeInvHandler)
                 .post(SessionStartHandler.Path, SaneHandler.around(new SessionStartHandler(m.getSession())))
-                .get(SessionValidateHandler.Path, sessValidateHandler)
+                .get(SessionValidateHandler.Path, SaneHandler.around(new SessionValidationGetHandler(m.getSession(), m.getConfig())))
-                .post(SessionValidateHandler.Path, sessValidateHandler)
+                .post(SessionValidateHandler.Path, SaneHandler.around(new SessionValidationPostHandler(m.getSession())))
                 .get(SessionTpidGetValidatedHandler.Path, SaneHandler.around(new SessionTpidGetValidatedHandler(m.getSession())))
                 .post(SessionTpidBindHandler.Path, SaneHandler.around(new SessionTpidBindHandler(m.getSession(), m.getInvite())))
                 .post(SessionTpidUnbindHandler.Path, SaneHandler.around(new SessionTpidUnbindHandler(m.getSession())))

src/main/java/io/kamax/mxisd/Mxisd.java

@@ -41,6 +41,7 @@ import io.kamax.mxisd.lookup.provider.BridgeFetcher;
 import io.kamax.mxisd.lookup.provider.RemoteIdentityServerFetcher;
 import io.kamax.mxisd.lookup.strategy.LookupStrategy;
 import io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy;
+import io.kamax.mxisd.matrix.HomeserverFederationResolver;
 import io.kamax.mxisd.matrix.IdentityServerUtils;
 import io.kamax.mxisd.notification.NotificationHandlerSupplier;
 import io.kamax.mxisd.notification.NotificationHandlers;
@@ -99,6 +100,8 @@ public class Mxisd {
                 .setMaxConnTotal(Integer.MAX_VALUE)
                 .build();
 
+        FederationDnsOverwrite fedDns = new FederationDnsOverwrite(cfg.getDns().getOverwrite());
+        HomeserverFederationResolver resolver = new HomeserverFederationResolver(fedDns, httpClient);
         IdentityServerUtils.setHttpClient(httpClient);
         srvFetcher = new RemoteIdentityServerFetcher(httpClient);
 
@@ -106,10 +109,9 @@ public class Mxisd {
         keyMgr = CryptoFactory.getKeyManager(cfg.getKey());
         signMgr = CryptoFactory.getSignatureManager(keyMgr);
         clientDns = new ClientDnsOverwrite(cfg.getDns().getOverwrite());
-        FederationDnsOverwrite fedDns = new FederationDnsOverwrite(cfg.getDns().getOverwrite());
+
         synapse = new Synapse(cfg.getSynapseSql());
         BridgeFetcher bridgeFetcher = new BridgeFetcher(cfg.getLookup().getRecursive().getBridge(), srvFetcher);
-
         ServiceLoader.load(IdentityStoreSupplier.class).iterator().forEachRemaining(p -> p.accept(this));
         ServiceLoader.load(NotificationHandlerSupplier.class).iterator().forEachRemaining(p -> p.accept(this));
 
@@ -117,7 +119,7 @@ public class Mxisd {
         pMgr = new ProfileManager(ProfileProviders.get(), clientDns, httpClient);
         notifMgr = new NotificationManager(cfg.getNotification(), NotificationHandlers.get());
         sessMgr = new SessionManager(cfg.getSession(), cfg.getMatrix(), store, notifMgr, idStrategy, httpClient);
-        invMgr = new InvitationManager(cfg, store, idStrategy, keyMgr, signMgr, fedDns, notifMgr, pMgr);
+        invMgr = new InvitationManager(cfg, store, idStrategy, keyMgr, signMgr, resolver, notifMgr, pMgr);
         authMgr = new AuthManager(cfg, AuthProviders.get(), idStrategy, invMgr, clientDns, httpClient);
         dirMgr = new DirectoryManager(cfg.getDirectory(), clientDns, httpClient, DirectoryProviders.get());
         regMgr = new RegistrationManager(cfg.getRegister(), httpClient, clientDns, invMgr);

src/main/java/io/kamax/mxisd/MxisdStandaloneExec.java

@@ -36,6 +36,11 @@ public class MxisdStandaloneExec {
     private static final Logger log = LoggerFactory.getLogger("App");
 
     public static void main(String[] args) {
+        String logLevel = System.getenv("MXISD_LOG_LEVEL");
+        if (StringUtils.isNotBlank(logLevel)) {
+            System.setProperty("org.slf4j.simpleLogger.log.io.kamax.mxisd", logLevel);
+        }
+
         try {
             MxisdConfig cfg = null;
             Iterator<String> argsIt = Arrays.asList(args).iterator();
@@ -46,8 +51,14 @@ public class MxisdStandaloneExec {
                     System.out.println("  -h, --help       Show this help message");
                     System.out.println("  --version        Print the version then exit");
                     System.out.println("  -c, --config     Set the configuration file location");
+                    System.out.println("  -v               Increase log level (log more info)");
+                    System.out.println("  -vv              Further increase log level");
                     System.out.println(" ");
                     System.exit(0);
+                } else if (StringUtils.equals(arg, "-v")) {
+                    System.setProperty("org.slf4j.simpleLogger.log.io.kamax.mxisd", "debug");
+                } else if (StringUtils.equals(arg, "-vv")) {
+                    System.setProperty("org.slf4j.simpleLogger.log.io.kamax.mxisd", "trace");
                 } else if (StringUtils.equalsAny(arg, "-c", "--config")) {
                     String cfgFile = argsIt.next();
                     cfg = YamlConfigLoader.loadFromFile(cfgFile);
@@ -61,13 +72,13 @@ public class MxisdStandaloneExec {
                 }
             }
 
-            log.info("mxisd starting");
-            log.info("Version: {}", Mxisd.Version);
-
             if (Objects.isNull(cfg)) {
                 cfg = YamlConfigLoader.tryLoadFromFile("mxisd.yaml").orElseGet(MxisdConfig::new);
             }
 
+            log.info("mxisd starting");
+            log.info("Version: {}", Mxisd.Version);
+
             HttpMxisd mxisd = new HttpMxisd(cfg);
             Runtime.getRuntime().addShutdownHook(new Thread(() -> {
                 mxisd.stop();

src/main/java/io/kamax/mxisd/config/RegisterConfig.java

@@ -69,16 +69,16 @@ public class RegisterConfig {
         }
 
         private List<String> buildPatterns(List<String> domains) {
-            log.info("Building email policy");
+            log.debug("Building email policy");
             return domains.stream().map(d -> {
                 if (StringUtils.startsWith(d, "*")) {
-                    log.info("Found domain and subdomain policy");
+                    log.debug("Found domain and subdomain policy");
                     d = "(.*)" + d.substring(1);
                 } else if (StringUtils.startsWith(d, ".")) {
-                    log.info("Found subdomain-only policy");
+                    log.debug("Found subdomain-only policy");
                     d = "(.*)" + d;
                 } else {
-                    log.info("Found domain-only policy");
+                    log.debug("Found domain-only policy");
                 }
 
                 return "([^@]+)@" + d.replace(".", "\\.");
@@ -175,10 +175,10 @@ public class RegisterConfig {
     }
 
     public void build() {
-        log.info("--- Registration config ---");
+        log.debug("--- Registration config ---");
 
-        log.info("Before Build");
+        log.debug("Before Build");
-        log.info(GsonUtil.getPrettyForLog(this));
+        log.debug(GsonUtil.getPrettyForLog(this));
 
         new HashMap<>(getPolicy().getThreepid()).forEach((medium, policy) -> {
             if (ThreePidMedium.Email.is(medium)) {
@@ -194,8 +194,8 @@ public class RegisterConfig {
             getPolicy().getThreepid().put(medium, policy);
         });
 
-        log.info("After Build");
+        log.debug("After Build");
-        log.info(GsonUtil.getPrettyForLog(this));
+        log.debug(GsonUtil.getPrettyForLog(this));
     }
 
 }

src/main/java/io/kamax/mxisd/config/threepid/medium/GenericTemplateConfig.java

@@ -77,6 +77,7 @@ public class GenericTemplateConfig {
     private String invite;
     private Session session = new Session();
     private Map<String, String> generic = new HashMap<>();
+    private Map<String, String> placeholder = new HashMap<>();
 
     public String getInvite() {
         return invite;
@@ -98,4 +99,12 @@ public class GenericTemplateConfig {
         this.generic = generic;
     }
 
+    public Map<String, String> getPlaceholder() {
+        return placeholder;
+    }
+
+    public void setPlaceholder(Map<String, String> placeholder) {
+        this.placeholder = placeholder;
+    }
+
 }

src/main/java/io/kamax/mxisd/http/undertow/handler/BasicHttpHandler.java

@@ -31,6 +31,7 @@ import io.kamax.mxisd.proxy.Response;
 import io.kamax.mxisd.util.RestClientUtils;
 import io.undertow.server.HttpHandler;
 import io.undertow.server.HttpServerExchange;
+import io.undertow.server.handlers.form.FormData;
 import io.undertow.util.HttpString;
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang3.StringUtils;
@@ -48,10 +49,7 @@ import java.net.InetSocketAddress;
 import java.net.URI;
 import java.net.URLDecoder;
 import java.nio.charset.StandardCharsets;
-import java.util.Deque;
+import java.util.*;
-import java.util.LinkedList;
-import java.util.Map;
-import java.util.Optional;
 
 public abstract class BasicHttpHandler implements HttpHandler {
 
@@ -122,6 +120,20 @@ public abstract class BasicHttpHandler implements HttpHandler {
         return GsonUtil.parseObj(getBodyUtf8(exchange));
     }
 
+    protected String getOrThrow(FormData data, String key) {
+        FormData.FormValue value = data.getFirst(key);
+        if (Objects.isNull(value)) {
+            throw new IllegalArgumentException("Form key " + key + " is missing");
+        }
+
+        String object = value.getValue();
+        if (Objects.isNull(object)) {
+            throw new IllegalArgumentException("Form key " + key + " does not have a value");
+        }
+
+        return object;
+    }
+
     protected void putHeader(HttpServerExchange ex, String name, String value) {
         ex.getResponseHeaders().put(HttpString.tryFromString(name), value);
     }

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/SessionValidateHandler.java

@@ -20,94 +20,36 @@
 
 package io.kamax.mxisd.http.undertow.handler.identity.v1;
 
-import io.kamax.mxisd.config.ServerConfig;
-import io.kamax.mxisd.config.ViewConfig;
 import io.kamax.mxisd.http.IsAPIv1;
-import io.kamax.mxisd.http.io.identity.SuccessStatusJson;
 import io.kamax.mxisd.http.undertow.handler.BasicHttpHandler;
 import io.kamax.mxisd.session.SessionManager;
 import io.kamax.mxisd.session.ValidationResult;
-import io.kamax.mxisd.util.FileUtil;
+import org.apache.commons.lang3.StringUtils;
-import io.undertow.server.HttpServerExchange;
-import io.undertow.util.HttpString;
-import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import java.io.IOException;
+public abstract class SessionValidateHandler extends BasicHttpHandler {
-import java.net.MalformedURLException;
-import java.net.URL;
-
-public class SessionValidateHandler extends BasicHttpHandler {
 
     public static final String Path = IsAPIv1.Base + "/validate/{medium}/submitToken";
 
     private transient final Logger log = LoggerFactory.getLogger(SessionValidateHandler.class);
 
     private SessionManager mgr;
-    private ServerConfig srvCfg;
-    private ViewConfig viewCfg;
 
-    public SessionValidateHandler(SessionManager mgr, ServerConfig srvCfg, ViewConfig viewCfg) {
+    public SessionValidateHandler(SessionManager mgr) {
         this.mgr = mgr;
-        this.srvCfg = srvCfg;
-        this.viewCfg = viewCfg;
     }
 
-    @Override
+    protected ValidationResult handleRequest(String sid, String secret, String token) {
-    public void handleRequest(HttpServerExchange exchange) {
+        if (StringUtils.isEmpty(sid)) {
-        String medium = getQueryParameter(exchange, "medium");
+            throw new IllegalArgumentException("sid is not set or is empty");
-        String sid = getQueryParameter(exchange, "sid");
-        String secret = getQueryParameter(exchange, "client_secret");
-        String token = getQueryParameter(exchange, "token");
-
-        boolean isHtmlRequest = false;
-        for (String v : exchange.getRequestHeaders().get("Accept")) {
-            if (StringUtils.startsWithIgnoreCase(v, "text/html")) {
-                isHtmlRequest = true;
-                break;
-            }
         }
 
-        if (isHtmlRequest) {
+        if (StringUtils.isEmpty(secret)) {
-            handleHtmlRequest(exchange, medium, sid, secret, token);
+            throw new IllegalArgumentException("client secret is not set or is empty");
-        } else {
-            handleJsonRequest(exchange, sid, secret, token);
-        }
         }
 
-    private void handleHtmlRequest(HttpServerExchange exchange, String medium, String sid, String secret, String token) {
+        return mgr.validate(sid, secret, token);
-        log.info("Validating session {} for medium {}", sid, medium);
-        ValidationResult r = mgr.validate(sid, secret, token);
-        log.info("Session {} was validated", sid);
-        if (r.getNextUrl().isPresent()) {
-            String url = r.getNextUrl().get();
-            try {
-                url = new URL(url).toString();
-            } catch (MalformedURLException e) {
-                log.info("Session next URL {} is not a valid one, will prepend public URL {}", url, srvCfg.getPublicUrl());
-                url = srvCfg.getPublicUrl() + r.getNextUrl().get();
-            }
-            log.info("Session {} validation: next URL is present, redirecting to {}", sid, url);
-            exchange.setStatusCode(302);
-            exchange.getResponseHeaders().add(HttpString.tryFromString("Location"), url);
-        } else {
-            try {
-                String data = FileUtil.load(viewCfg.getSession().getOnTokenSubmit().getSuccess());
-                writeBodyAsUtf8(exchange, data);
-            } catch (IOException e) {
-                throw new RuntimeException(e);
-            }
-        }
-    }
-
-    private void handleJsonRequest(HttpServerExchange exchange, String sid, String secret, String token) {
-        log.info("Requested: {}", exchange.getRequestURL());
-
-        mgr.validate(sid, secret, token);
-        log.info("Session {} was validated", sid);
-
-        respondJson(exchange, new SuccessStatusJson(true));
     }
 
 }

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/SessionValidationGetHandler.java

@@ -0,0 +1,82 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.http.undertow.handler.identity.v1;
+
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.config.ServerConfig;
+import io.kamax.mxisd.config.ViewConfig;
+import io.kamax.mxisd.session.SessionManager;
+import io.kamax.mxisd.session.ValidationResult;
+import io.kamax.mxisd.util.FileUtil;
+import io.undertow.server.HttpServerExchange;
+import io.undertow.util.HttpString;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+
+public class SessionValidationGetHandler extends SessionValidateHandler {
+
+    private transient final Logger log = LoggerFactory.getLogger(SessionValidationGetHandler.class);
+
+    private ServerConfig srvCfg;
+    private ViewConfig viewCfg;
+
+    public SessionValidationGetHandler(SessionManager mgr, MxisdConfig cfg) {
+        super(mgr);
+        this.srvCfg = cfg.getServer();
+        this.viewCfg = cfg.getView();
+    }
+
+    @Override
+    public void handleRequest(HttpServerExchange exchange) {
+        log.info("Handling GET request to validate session");
+
+        String sid = getQueryParameter(exchange, "sid");
+        String secret = getQueryParameter(exchange, "client_secret");
+        String token = getQueryParameter(exchange, "token");
+
+        ValidationResult r = handleRequest(sid, secret, token);
+        log.info("Session {} was validated", sid);
+        if (r.getNextUrl().isPresent()) {
+            String url = r.getNextUrl().get();
+            try {
+                url = new URL(url).toString();
+            } catch (MalformedURLException e) {
+                log.info("Session next URL {} is not a valid one, will prepend public URL {}", url, srvCfg.getPublicUrl());
+                url = srvCfg.getPublicUrl() + r.getNextUrl().get();
+            }
+            log.info("Session {} validation: next URL is present, redirecting to {}", sid, url);
+            exchange.setStatusCode(302);
+            exchange.getResponseHeaders().add(HttpString.tryFromString("Location"), url);
+        } else {
+            try {
+                String data = FileUtil.load(viewCfg.getSession().getOnTokenSubmit().getSuccess());
+                writeBodyAsUtf8(exchange, data);
+            } catch (IOException e) {
+                throw new RuntimeException(e);
+            }
+        }
+    }
+
+}

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/SessionValidationPostHandler.java

@@ -0,0 +1,80 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.http.undertow.handler.identity.v1;
+
+import com.google.gson.JsonObject;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.http.io.identity.SuccessStatusJson;
+import io.kamax.mxisd.session.SessionManager;
+import io.undertow.server.HttpServerExchange;
+import io.undertow.server.handlers.form.FormData;
+import io.undertow.server.handlers.form.FormParserFactory;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+public class SessionValidationPostHandler extends SessionValidateHandler {
+
+    private transient final Logger log = LoggerFactory.getLogger(SessionValidationPostHandler.class);
+
+    private FormParserFactory factory;
+
+    public SessionValidationPostHandler(SessionManager mgr) {
+        super(mgr);
+        factory = FormParserFactory.builder().build();
+    }
+
+
+    @Override
+    public void handleRequest(HttpServerExchange exchange) throws IOException {
+        log.info("Handling POST request to validate session");
+
+        String sid;
+        String secret;
+        String token;
+
+        String contentType = getContentType(exchange).orElseThrow(() -> new IllegalArgumentException("Content type header is not set"));
+        if (StringUtils.equals(contentType, "application/json")) { // FIXME use MIME parsing tools
+            log.info("Parsing as JSON data");
+
+            JsonObject body = parseJsonObject(exchange);
+            sid = GsonUtil.getStringOrThrow(body, "sid");
+            secret = GsonUtil.getStringOrThrow(body, "client_secret");
+            token = GsonUtil.getStringOrThrow(body, "token");
+        } else if (StringUtils.equals(contentType, "application/x-www-form-urlencoded")) { // FIXME use MIME parsing tools
+            log.info("Parsing as Form data");
+
+            FormData data = factory.createParser(exchange).parseBlocking();
+            sid = getOrThrow(data, "sid");
+            secret = getOrThrow(data, "client_secret");
+            token = getOrThrow(data, "token");
+        } else {
+            log.info("Unsupported Content type: {}", contentType);
+            throw new IllegalArgumentException("Unsupported Content type: " + contentType);
+        }
+
+        handleRequest(sid, secret, token);
+        respondJson(exchange, new SuccessStatusJson(true));
+    }
+
+}

src/main/java/io/kamax/mxisd/invitation/InvitationManager.java

@@ -30,7 +30,6 @@ import io.kamax.mxisd.config.InvitationConfig;
 import io.kamax.mxisd.config.MxisdConfig;
 import io.kamax.mxisd.config.ServerConfig;
 import io.kamax.mxisd.crypto.*;
-import io.kamax.mxisd.dns.FederationDnsOverwrite;
 import io.kamax.mxisd.exception.BadRequestException;
 import io.kamax.mxisd.exception.ConfigurationException;
 import io.kamax.mxisd.exception.MappingAlreadyExistsException;
@@ -38,6 +37,7 @@ import io.kamax.mxisd.exception.ObjectNotFoundException;
 import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.ThreePidMapping;
 import io.kamax.mxisd.lookup.strategy.LookupStrategy;
+import io.kamax.mxisd.matrix.HomeserverFederationResolver;
 import io.kamax.mxisd.notification.NotificationManager;
 import io.kamax.mxisd.profile.ProfileManager;
 import io.kamax.mxisd.storage.IStorage;
@@ -57,13 +57,10 @@ import org.apache.http.impl.client.HttpClients;
 import org.apache.http.ssl.SSLContextBuilder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.xbill.DNS.*;
 
 import javax.net.ssl.HostnameVerifier;
 import javax.net.ssl.SSLContext;
 import java.io.IOException;
-import java.net.MalformedURLException;
-import java.net.URL;
 import java.nio.charset.StandardCharsets;
 import java.time.DateTimeException;
 import java.time.Instant;
@@ -85,7 +82,7 @@ public class InvitationManager {
     private LookupStrategy lookupMgr;
     private KeyManager keyMgr;
     private SignatureManager signMgr;
-    private FederationDnsOverwrite dns;
+    private HomeserverFederationResolver resolver;
     private NotificationManager notifMgr;
     private ProfileManager profileMgr;
 
@@ -100,7 +97,7 @@ public class InvitationManager {
             LookupStrategy lookupMgr,
             KeyManager keyMgr,
             SignatureManager signMgr,
-            FederationDnsOverwrite dns,
+            HomeserverFederationResolver resolver,
             NotificationManager notifMgr,
             ProfileManager profileMgr
     ) {
@@ -110,7 +107,7 @@ public class InvitationManager {
         this.lookupMgr = lookupMgr;
         this.keyMgr = keyMgr;
         this.signMgr = signMgr;
-        this.dns = dns;
+        this.resolver = resolver;
         this.notifMgr = notifMgr;
         this.profileMgr = profileMgr;
 
@@ -207,56 +204,6 @@ public class InvitationManager {
         return reply.getInvite().getSender().getId() + ":" + reply.getInvite().getRoomId() + ":" + reply.getInvite().getMedium() + ":" + reply.getInvite().getAddress();
     }
 
-    private String getSrvRecordName(String domain) {
-        return "_matrix._tcp." + domain;
-    }
-
-    // TODO use caching mechanism
-    // TODO export in matrix-java-sdk
-    private String findHomeserverForDomain(String domain) {
-        Optional<String> entryOpt = dns.findHost(domain);
-        if (entryOpt.isPresent()) {
-            String entry = entryOpt.get();
-            log.info("Found DNS overwrite for {} to {}", domain, entry);
-            try {
-                return new URL(entry).toString();
-            } catch (MalformedURLException e) {
-                log.warn("Skipping homeserver Federation DNS overwrite for {} - not a valid URL: {}", domain, entry);
-            }
-        }
-
-        log.debug("Performing SRV lookup for {}", domain);
-        String lookupDns = getSrvRecordName(domain);
-        log.info("Lookup name: {}", lookupDns);
-
-        try {
-            List<SRVRecord> srvRecords = new ArrayList<>();
-            Record[] rawRecords = new Lookup(lookupDns, Type.SRV).run();
-            if (rawRecords != null && rawRecords.length > 0) {
-                for (Record record : rawRecords) {
-                    if (Type.SRV == record.getType()) {
-                        srvRecords.add((SRVRecord) record);
-                    } else {
-                        log.info("Got non-SRV record: {}", record.toString());
-                    }
-                }
-
-                srvRecords.sort(Comparator.comparingInt(SRVRecord::getPriority));
-                for (SRVRecord record : srvRecords) {
-                    log.info("Found SRV record: {}", record.toString());
-                    return "https://" + record.getTarget().toString(true) + ":" + record.getPort();
-                }
-            } else {
-                log.info("No SRV record for {}", lookupDns);
-            }
-        } catch (TextParseException e) {
-            log.warn("Unable to perform DNS SRV query for {}: {}", lookupDns, e.getMessage());
-        }
-
-        log.info("Performing basic lookup using domain name {}", domain);
-        return "https://" + domain + ":8448";
-    }
-
     private Optional<SingleLookupReply> lookup3pid(String medium, String address) {
         if (!cfg.getResolution().isRecursive()) {
             log.warn("/!\\ /!\\ --- RECURSIVE INVITE RESOLUTION HAS BEEN DISABLED --- /!\\ /!\\");
@@ -413,7 +360,7 @@ public class InvitationManager {
                     continue;
                 }
 
-                log.info("Invite {} has expired at TS {} - Expiring and resolving to {}", targetTs, targetMxid);
+                log.info("Invite {} has expired at TS {} - Expiring and resolving to {}", reply.getId(), targetTs, targetMxid);
                 publishMapping(reply, targetMxid);
             } catch (NumberFormatException | DateTimeException e) {
                 log.warn("Invite {} has an invalid creation TS, setting to default value of {}", reply.getId(), defaultCreateTs);
@@ -475,7 +422,7 @@ public class InvitationManager {
         String address = reply.getInvite().getAddress();
         String domain = reply.getInvite().getSender().getDomain();
         log.info("Discovering HS for domain {}", domain);
-        String hsUrlOpt = findHomeserverForDomain(domain);
+        String hsUrlOpt = resolver.resolve(domain).toString();
 
         // TODO this is needed as this will block if called during authentication cycle due to synapse implementation
         new Thread(() -> { // FIXME need to make this retry-able and within a general background working pool

src/main/java/io/kamax/mxisd/matrix/HomeserverFederationResolver.java

@@ -0,0 +1,215 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.matrix;
+
+import com.google.gson.JsonObject;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.matrix.json.InvalidJsonException;
+import io.kamax.mxisd.dns.FederationDnsOverwrite;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.util.EntityUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.xbill.DNS.*;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.*;
+
+public class HomeserverFederationResolver {
+
+    private static final Logger log = LoggerFactory.getLogger(HomeserverFederationResolver.class);
+
+    private FederationDnsOverwrite dns;
+    private CloseableHttpClient client;
+
+    public HomeserverFederationResolver(FederationDnsOverwrite dns, CloseableHttpClient client) {
+        this.dns = dns;
+        this.client = client;
+    }
+
+    private String getDefaultScheme() {
+        return "https";
+    }
+
+    private int getDefaultPort() {
+        return 8448;
+    }
+
+    private String getDnsSrvPrefix() {
+        return "_matrix._tcp.";
+    }
+
+    private String buildSrvRecordName(String domain) {
+        return getDnsSrvPrefix() + domain;
+    }
+
+    private Optional<URL> resolveOverwrite(String domain) {
+        Optional<String> entryOpt = dns.findHost(domain);
+        if (!entryOpt.isPresent()) {
+            log.info("No DNS overwrite for {}", domain);
+            return Optional.empty();
+        }
+
+        try {
+            return Optional.of(new URL(entryOpt.get()));
+        } catch (MalformedURLException e) {
+            log.warn("Skipping homeserver Federation DNS overwrite for {} - not a valid URL: {}", domain, entryOpt.get());
+            return Optional.empty();
+        }
+    }
+
+    private Optional<URL> resolveLiteral(String domain) {
+        if (domain.contains("[") && domain.contains("]")) {
+            // This is an IPv6
+            if (domain.contains("]:")) {
+                // With a custom port, we return as is
+                return Optional.of(build(domain));
+            } else {
+                return Optional.of(build(domain + ":" + getDefaultPort()));
+            }
+        }
+
+        if (domain.contains(":")) {
+            // This is a domain or IPv4 with an explicit port, we return as is
+            return Optional.of(build(domain));
+        }
+
+        // At this point, we do not account for the provided string to be an IPv4 without a port. We will therefore
+        // perform well-known lookup and SRV record. While this is not needed, we don't expect the SRV to return anything
+        // and the well-known shouldn't either, but it might, leading to a wrong destination potentially.
+        //
+        // We accept this risk as mxisd is not meant to be used without DNS domain as per FAQ. We also provide resolution
+        // override facilities. Therefore, we accept to not handle this case until we get report of such unwanted behaviour
+        // that still fix mxisd use case and can't be resolved via override.
+
+        return Optional.empty();
+    }
+
+    private Optional<URL> resolveWellKnown(String domain) {
+        log.debug("Performing Well-known lookup for {}", domain);
+        HttpGet wnReq = new HttpGet("https://" + domain + "/.well-known/matrix/server");
+        try (CloseableHttpResponse wnRes = client.execute(wnReq)) {
+            int status = wnRes.getStatusLine().getStatusCode();
+            if (status == 200) {
+                try {
+                    JsonObject body = GsonUtil.parseObj(EntityUtils.toString(wnRes.getEntity()));
+                    String server = GsonUtil.getStringOrNull(body, "m.server");
+                    if (StringUtils.isNotBlank(server)) {
+                        log.debug("Found well-known entry: {}", server);
+                        return Optional.of(build(server));
+                    }
+                } catch (InvalidJsonException e) {
+                    log.info("Could not parse well-known resource: {}", e.getMessage());
+                }
+            } else {
+                log.info("Well-known did not return status code 200 but {}, ignoring", status);
+            }
+
+            return Optional.empty();
+        } catch (IOException e) {
+            throw new RuntimeException("Error while trying to lookup well-known for " + domain, e);
+        }
+    }
+
+    private Optional<URL> resolveDnsSrv(String domain) {
+        log.debug("Performing SRV lookup for {}", domain);
+        String lookupDns = buildSrvRecordName(domain);
+        log.debug("Lookup name: {}", lookupDns);
+
+        try {
+            List<SRVRecord> srvRecords = new ArrayList<>();
+            Record[] rawRecords = new Lookup(lookupDns, Type.SRV).run();
+            if (Objects.isNull(rawRecords) || rawRecords.length == 0) {
+                log.debug("No SRV record for {}", domain);
+                return Optional.empty();
+            }
+
+            for (Record record : rawRecords) {
+                if (Type.SRV == record.getType()) {
+                    srvRecords.add((SRVRecord) record);
+                } else {
+                    log.debug("Ignoring non-SRV record: {}", record.toString());
+                }
+            }
+
+            if (srvRecords.size() < 1) {
+                log.warn("DNS SRV records were found for {} but none is usable", lookupDns);
+                return Optional.empty();
+            }
+
+            srvRecords.sort(Comparator.comparingInt(SRVRecord::getPriority));
+            SRVRecord record = srvRecords.get(0);
+            return Optional.of(build(record.getTarget().toString(true) + ":" + record.getPort()));
+        } catch (TextParseException e) {
+            log.warn("Unable to perform DNS SRV query for {}: {}", lookupDns, e.getMessage());
+        }
+
+        return Optional.empty();
+    }
+
+    public URL build(String authority) {
+        try {
+            return new URL(getDefaultScheme() + "://" + authority);
+        } catch (MalformedURLException e) {
+            throw new RuntimeException("Could not build URL for " + authority, e);
+        }
+    }
+
+    public URL resolve(String domain) {
+        Optional<URL> s1 = resolveOverwrite(domain);
+        if (s1.isPresent()) {
+            URL dest = s1.get();
+            log.info("Resolution of {} via DNS overwrite to {}", domain, dest);
+            return dest;
+        }
+
+        Optional<URL> s2 = resolveLiteral(domain);
+        if (s2.isPresent()) {
+            URL dest = s2.get();
+            log.info("Resolution of {} as IP literal or IP/hostname with explicit port to {}", domain, dest);
+            return dest;
+        }
+
+        Optional<URL> s3 = resolveWellKnown(domain);
+        if (s3.isPresent()) {
+            URL dest = s3.get();
+            log.info("Resolution of {} via well-known to {}", domain, dest);
+            return dest;
+        }
+        // The domain needs to be resolved
+
+        Optional<URL> s4 = resolveDnsSrv(domain);
+        if (s4.isPresent()) {
+            URL dest = s4.get();
+            log.info("Resolution of {} via DNS SRV record to {}", domain, dest);
+        }
+
+        URL dest = build(domain + ":" + getDefaultPort());
+        log.info("Resolution of {} to {}", domain, dest);
+        return dest;
+    }
+
+}

src/main/java/io/kamax/mxisd/session/SessionManager.java

@@ -38,8 +38,8 @@ import io.kamax.mxisd.notification.NotificationManager;
 import io.kamax.mxisd.storage.IStorage;
 import io.kamax.mxisd.storage.dao.IThreePidSessionDao;
 import io.kamax.mxisd.threepid.session.ThreePidSession;
-import org.apache.commons.lang.RandomStringUtils;
+import org.apache.commons.lang3.RandomStringUtils;
-import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.http.impl.client.CloseableHttpClient;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -139,12 +139,13 @@ public class SessionManager {
     }
 
     public ValidationResult validate(String sid, String secret, String token) {
+        log.info("Validating session {}", sid);
         ThreePidSession session = getSession(sid, secret);
-        log.info("Attempting validation for session {} from {}", session.getId(), session.getServer());
+        log.info("Session {} is from {}", session.getId(), session.getServer());
 
         session.validate(token);
         storage.updateThreePidSession(session.getDao());
-        log.info("Session {} has been validated locally", session.getId());
+        log.info("Session {} has been validated", session.getId());
 
         ValidationResult r = new ValidationResult(session);
         session.getNextLink().ifPresent(r::setNextUrl);

src/main/java/io/kamax/mxisd/threepid/connector/email/BlackholeEmailConnector.java

@@ -0,0 +1,37 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.threepid.connector.email;
+
+public class BlackholeEmailConnector implements EmailConnector {
+
+    public static final String ID = "none";
+
+    @Override
+    public String getId() {
+        return ID;
+    }
+
+    @Override
+    public void send(String senderAddress, String senderName, String recipient, String content) {
+        //dev/null
+    }
+
+}

src/main/java/io/kamax/mxisd/threepid/connector/email/BuiltInEmailConnectorSupplier.java

@@ -33,6 +33,10 @@ public class BuiltInEmailConnectorSupplier implements EmailConnectorSupplier {
 
     @Override
     public Optional<EmailConnector> apply(EmailConfig cfg, Mxisd mxisd) {
+        if (StringUtils.equals(BlackholeEmailConnector.ID, cfg.getConnector())) {
+            return Optional.of(new BlackholeEmailConnector());
+        }
+
         if (StringUtils.equals(EmailSmtpConnector.ID, cfg.getConnector())) {
             EmailSmtpConfig smtpCfg = GsonUtil.get().fromJson(cfg.getConnectors().getOrDefault(EmailSmtpConnector.ID, new JsonObject()), EmailSmtpConfig.class);
             return Optional.of(new EmailSmtpConnector(smtpCfg));

src/main/java/io/kamax/mxisd/threepid/connector/email/EmailSmtpConnector.java

@@ -31,14 +31,17 @@ import org.apache.commons.lang3.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import javax.mail.Header;
 import javax.mail.Message;
 import javax.mail.MessagingException;
 import javax.mail.Session;
 import javax.mail.internet.InternetAddress;
 import javax.mail.internet.MimeMessage;
+import javax.mail.internet.MimeUtility;
 import java.io.UnsupportedEncodingException;
 import java.nio.charset.StandardCharsets;
 import java.util.Date;
+import java.util.Enumeration;
 import java.util.Properties;
 
 public class EmailSmtpConnector implements EmailConnector {
@@ -66,6 +69,10 @@ public class EmailSmtpConnector implements EmailConnector {
             sCfg.setProperty("mail.smtp.auth", "true");
         }
 
+        if (cfg.getTls() == 3) {
+            sCfg.setProperty("mail.smtp.ssl.enable", "true");
+        }
+
         session = Session.getInstance(sCfg);
     }
 
@@ -93,7 +100,16 @@ public class EmailSmtpConnector implements EmailConnector {
         try {
             InternetAddress sender = new InternetAddress(senderAddress, senderName);
             MimeMessage msg = new MimeMessage(session, IOUtils.toInputStream(content, StandardCharsets.UTF_8));
-            msg.setHeader("X-Mailer", Mxisd.Agent);
+
+            // We must encode our headers ourselves as we have no guarantee that they were in the provided data.
+            // This is required to support UTF-8 characters from user display names or room names in the subject header per example
+            Enumeration<Header> headers = msg.getAllHeaders();
+            while (headers.hasMoreElements()) {
+                Header header = headers.nextElement();
+                msg.setHeader(header.getName(), MimeUtility.encodeText(header.getValue()));
+            }
+
+            msg.setHeader("X-Mailer", MimeUtility.encodeText(Mxisd.Agent));
             msg.setSentDate(new Date());
             msg.setFrom(sender);
             msg.setRecipients(Message.RecipientType.TO, recipient);
@@ -101,8 +117,11 @@ public class EmailSmtpConnector implements EmailConnector {
 
             log.info("Sending invite to {} via SMTP using {}:{}", recipient, cfg.getHost(), cfg.getPort());
             SMTPTransport transport = (SMTPTransport) session.getTransport("smtp");
+
+            if (cfg.getTls() < 3) {
                 transport.setStartTLS(cfg.getTls() > 0);
                 transport.setRequireStartTLS(cfg.getTls() > 1);
+            }
 
             log.info("Connecting to {}:{}", cfg.getHost(), cfg.getPort());
             if (StringUtils.isAllEmpty(cfg.getLogin(), cfg.getPassword())) {

src/main/java/io/kamax/mxisd/threepid/connector/phone/BlackholePhoneConnector.java

@@ -24,14 +24,14 @@ public class BlackholePhoneConnector implements PhoneConnector {
 
     public static final String ID = "none";
 
-    @Override
-    public void send(String recipient, String content) {
-        //dev/null
-    }
-
     @Override
     public String getId() {
         return ID;
     }
 
+    @Override
+    public void send(String recipient, String content) {
+        //dev/null
+    }
+
 }

src/main/java/io/kamax/mxisd/threepid/connector/phone/BuiltInPhoneConnectorSupplier.java

@@ -33,15 +33,15 @@ public class BuiltInPhoneConnectorSupplier implements PhoneConnectorSupplier {
 
     @Override
     public Optional<PhoneConnector> apply(PhoneConfig cfg, Mxisd mxisd) {
+        if (StringUtils.equals(BlackholePhoneConnector.ID, cfg.getConnector())) {
+            return Optional.of(new BlackholePhoneConnector());
+        }
+
         if (StringUtils.equals(PhoneSmsTwilioConnector.ID, cfg.getConnector())) {
             PhoneTwilioConfig cCfg = GsonUtil.get().fromJson(cfg.getConnectors().getOrDefault(PhoneSmsTwilioConnector.ID, new JsonObject()), PhoneTwilioConfig.class);
             return Optional.of(new PhoneSmsTwilioConnector(cCfg));
         }
 
-        if (StringUtils.equals(BlackholePhoneConnector.ID, cfg.getConnector())) {
-            return Optional.of(new BlackholePhoneConnector());
-        }
-
         return Optional.empty();
     }
 

src/main/java/io/kamax/mxisd/threepid/generator/GenericTemplateNotificationGenerator.java

@@ -68,6 +68,7 @@ public abstract class GenericTemplateNotificationGenerator extends PlaceholderNo
     @Override
     public String getForReply(IThreePidInviteReply invite) {
         log.info("Generating notification content for 3PID invite");
+        invite.getInvite().getProperties().putAll(cfg.getPlaceholder());
         return populateForReply(invite, getTemplateContent(cfg.getInvite()));
     }
 

src/main/java/io/kamax/mxisd/threepid/generator/PlaceholderNotificationGenerator.java

@@ -35,6 +35,8 @@ import static io.kamax.mxisd.http.io.identity.StoreInviteRequest.Keys.SenderDisp
 
 public abstract class PlaceholderNotificationGenerator {
 
+    public static final String RegisterUrl = "REGISTER_URL";
+
     private MatrixConfig mxCfg;
     private ServerConfig srvCfg;
 
@@ -76,8 +78,10 @@ public abstract class PlaceholderNotificationGenerator {
         String senderNameOrId = StringUtils.defaultIfBlank(senderName, invite.getInvite().getSender().getId());
         String roomName = invite.getInvite().getProperties().getOrDefault(RoomName, "");
         String roomNameOrId = StringUtils.defaultIfBlank(roomName, invite.getInvite().getRoomId());
+        String registerUrl = StringUtils.defaultIfBlank(invite.getInvite().getProperties().get(RegisterUrl), "https://" + mxCfg.getDomain());
 
         return populateForCommon(tpid, input)
+                .replace("%" + RegisterUrl + "%", registerUrl)
                 .replace("%SENDER_ID%", invite.getInvite().getSender().getId())
                 .replace("%SENDER_NAME%", senderName)
                 .replace("%SENDER_NAME_OR_ID%", senderNameOrId)
@@ -102,10 +106,6 @@ public abstract class PlaceholderNotificationGenerator {
                 .replace("%NEXT_URL%", validationLink);
     }
 
-    protected String populateForRemoteValidation(IThreePidSession session, String input) {
-        return populateForValidation(session, input);
-    }
-
     protected String populateForFraudulentUndind(ThreePid tpid, String input) {
         return populateForCommon(tpid, input);
     }

src/main/resources/simplelogger.properties

@@ -1,2 +1,4 @@
 org.slf4j.simpleLogger.logFile=System.out
 org.slf4j.simpleLogger.log.org.xnio=warn
+org.slf4j.simpleLogger.log.com.j256.ormlite.table.TableUtils=warn
+org.slf4j.simpleLogger.log.com.mchange.v2.log.MLog=warn

src/main/resources/threepids/email/invite-template.eml

@@ -10,7 +10,7 @@ Content-Disposition: inline
 Hi,
 
 %SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on
-Matrix. To join the conversation, register an account on https://%DOMAIN%
+Matrix. To join the conversation, register an account on %REGISTER_URL%
 
 You can also register an account on a public server and get in touch with them.
 
@@ -69,7 +69,7 @@ pre, code {
 <p>Hi,</p>
 
 <p>%SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on
-Matrix. To join the conversation, register an account on <a href="https://%DOMAIN%">%DOMAIN%</a>.</p>
+Matrix. To join the conversation, register an account on <a href="%REGISTER_URL%">%DOMAIN%</a>.</p>
 
 <pYou can also register an account on a public server and get in touch with them.</p>
 

src/test/java/io/kamax/mxisd/test/matrix/HomeserverFederationResolverTest.java

@@ -0,0 +1,64 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.test.matrix;
+
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.dns.FederationDnsOverwrite;
+import io.kamax.mxisd.matrix.HomeserverFederationResolver;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.net.URL;
+
+import static org.junit.Assert.assertEquals;
+
+public class HomeserverFederationResolverTest {
+
+    private static HomeserverFederationResolver resolver;
+
+    @BeforeClass
+    public static void beforeClass() {
+        CloseableHttpClient client = HttpClients.custom()
+                .setUserAgent(Mxisd.Agent)
+                .setMaxConnPerRoute(Integer.MAX_VALUE)
+                .setMaxConnTotal(Integer.MAX_VALUE)
+                .build();
+
+        FederationDnsOverwrite fedDns = new FederationDnsOverwrite(new MxisdConfig().getDns().getOverwrite());
+        resolver = new HomeserverFederationResolver(fedDns, client);
+    }
+
+    @Test
+    public void hostnameWithoutPort() {
+        URL url = resolver.resolve("example.org");
+        assertEquals("https://example.org:8448", url.toString());
+    }
+
+    @Test
+    public void hostnameWithPort() {
+        URL url = resolver.resolve("example.org:443");
+        assertEquals("https://example.org:443", url.toString());
+    }
+
+}

src/test/java/io/kamax/mxisd/test/notification/EmailNotificationTest.java

@@ -32,7 +32,10 @@ import io.kamax.mxisd.config.MxisdConfig;
 import io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig;
 import io.kamax.mxisd.config.threepid.medium.EmailConfig;
 import io.kamax.mxisd.invitation.MatrixIdInvite;
+import io.kamax.mxisd.invitation.ThreePidInvite;
+import io.kamax.mxisd.invitation.ThreePidInviteReply;
 import io.kamax.mxisd.threepid.connector.email.EmailSmtpConnector;
+import io.kamax.mxisd.threepid.generator.PlaceholderNotificationGenerator;
 import io.kamax.mxisd.threepid.session.ThreePidSession;
 import org.apache.commons.lang.RandomStringUtils;
 import org.junit.After;
@@ -45,6 +48,7 @@ import javax.mail.internet.MimeBodyPart;
 import javax.mail.internet.MimeMessage;
 import javax.mail.internet.MimeMultipart;
 import java.io.IOException;
+import java.util.ArrayList;
 import java.util.Collections;
 
 import static junit.framework.TestCase.assertEquals;
@@ -56,7 +60,8 @@ public class EmailNotificationTest {
     private final String user = "mxisd";
     private final String notifiee = "john";
     private final String sender = user + "@" + domain;
-    private final String senderEmail = "\"Mxisd Server (Unit Test)\" <" + sender + ">";
+    private final String senderName = "\"Mxisd Server あ (Unit Test)\" <" + sender + ">";
+    private final String senderNameEncoded = "=?UTF-8?Q?=22Mxisd_Server_=E3=81=82_=28Unit_T?= =?UTF-8?Q?est=29=22_=3Cmxisd=40localhost=3E?= <mxisd@localhost>";
     private final String target = notifiee + "@" + domain;
 
     private Mxisd m;
@@ -72,7 +77,7 @@ public class EmailNotificationTest {
         EmailConfig eCfg = new EmailConfig();
         eCfg.setConnector(EmailSmtpConnector.ID);
         eCfg.getIdentity().setFrom(sender);
-        eCfg.getIdentity().setName("Mxisd Server (Unit Test)");
+        eCfg.getIdentity().setName(senderName);
         eCfg.getConnectors().put(EmailSmtpConnector.ID, GsonUtil.makeObj(smtpCfg));
 
         MxisdConfig cfg = new MxisdConfig();
@@ -114,10 +119,33 @@ public class EmailNotificationTest {
         assertEquals(1, gm.getReceivedMessages().length);
         MimeMessage msg = gm.getReceivedMessages()[0];
         assertEquals(1, msg.getFrom().length);
-        assertEquals(senderEmail, msg.getFrom()[0].toString());
+        assertEquals(senderNameEncoded, msg.getFrom()[0].toString());
         assertEquals(1, msg.getRecipients(Message.RecipientType.TO).length);
     }
 
+    @Test
+    public void forThreepidInvite() throws MessagingException, IOException {
+        String registerUrl = "https://" + RandomStringUtils.randomAlphanumeric(20) + ".example.org/register";
+        gm.setUser(user, user);
+
+        _MatrixID sender = MatrixID.asAcceptable(user, domain);
+        ThreePidInvite inv = new ThreePidInvite(sender, ThreePidMedium.Email.getId(), target, "!rid:" + domain);
+        inv.getProperties().put(PlaceholderNotificationGenerator.RegisterUrl, registerUrl);
+        m.getNotif().sendForReply(new ThreePidInviteReply("a", inv, "b", "c", new ArrayList<>()));
+
+        assertEquals(1, gm.getReceivedMessages().length);
+        MimeMessage msg = gm.getReceivedMessages()[0];
+        assertEquals(1, msg.getFrom().length);
+        assertEquals(senderNameEncoded, msg.getFrom()[0].toString());
+        assertEquals(1, msg.getRecipients(Message.RecipientType.TO).length);
+
+        // We just check on the text/plain one. HTML is multipart and it's difficult so we skip
+        MimeMultipart content = (MimeMultipart) msg.getContent();
+        MimeBodyPart mbp = (MimeBodyPart) content.getBodyPart(0);
+        String mbpContent = mbp.getContent().toString();
+        assertTrue(mbpContent.contains(registerUrl));
+    }
+
     @Test
     public void forValidation() throws MessagingException, IOException {
         gm.setUser(user, user);
@@ -138,7 +166,7 @@ public class EmailNotificationTest {
         assertEquals(1, gm.getReceivedMessages().length);
         MimeMessage msg = gm.getReceivedMessages()[0];
         assertEquals(1, msg.getFrom().length);
-        assertEquals(senderEmail, msg.getFrom()[0].toString());
+        assertEquals(senderNameEncoded, msg.getFrom()[0].toString());
         assertEquals(1, msg.getRecipients(Message.RecipientType.TO).length);
 
         // We just check on the text/plain one. HTML is multipart and it's difficult so we skip