Fix missing return in Homeserver endpoint discovery, skipping DNS SRV

- Give means to increase logging verbosity
- Explain how to do in the troubleshooting guide

README.md

@@ -31,7 +31,7 @@ users. 3PIDs can be anything that uniquely and globally identify a user, like:
 - Twitter handle
 - Facebook ID
 
-If you are unfamiliar with the Identity vocabulary and concepts in Matrix, **please read this [introduction](docs/concepts.md)**.
+If you are unfamiliar with the Identity vocabulary and concepts in Matrix, **please read this [introduction](docs/concepts.md)**.  
 
 # Features
 [Identity](docs/features/identity.md): As a [regular Matrix Identity service](https://matrix.org/docs/spec/identity_service/r0.1.0.html#general-principles):
@@ -53,6 +53,7 @@ As an enhanced Identity service:
   - Central Matrix Identity servers
 - [Session Control](docs/threepids/session/session.md): Extensive control of where 3PIDs are transmitted so they are not
   leaked publicly by users
+- [Registration control](docs/features/registration.md): Control and restrict user registration based on 3PID patterns or criterias, like a pending invite
 - [Authentication](docs/features/authentication.md): Use your Identity stores to perform authentication in [synapse](https://github.com/matrix-org/synapse)
   via the [REST password provider](https://github.com/kamax-io/matrix-synapse-rest-auth)
 - [Directory search](docs/features/directory.md) which allows you to search for users within your organisation,
@@ -80,8 +81,6 @@ A basic troubleshooting guide is available [here](docs/troubleshooting.md).
 ## Community
 Over Matrix: [#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io) ([Preview](https://view.matrix.org/room/!NPRUEisLjcaMtHIzDr:kamax.io/))
 
-For more high-level discussion about the Identity Server architecture/API, go to  [#matrix-identity:kamax.io](https://matrix.to/#/#matrix-identity:kamax.io)
-
 ## Commercial
 If you would prefer professional support/custom development for mxisd and/or for Matrix in general, including other open
 source technologies/products:

docs/features/experimental/application-service.md

@@ -1,25 +1,107 @@
-# Integration as an Application Service
+# Application Service
 **WARNING:** These features are currently highly experimental. They can be removed or modified without notice.  
-All the features requires a Homeserver capable of connecting Application Services.
+All the features requires a Homeserver capable of connecting [Application Services](https://matrix.org/docs/spec/application_service/r0.1.0.html).
 
-## Email notification for Room invites by Matrix ID
+The following capabilities are provided in this feature:
+- [Admin commands](#admin-commands)
+- [Email Notification about room invites by Matrix IDs](#email-notification-about-room-invites-by-matrix-ids)
+- [Auto-reject of expired 3PID invites](#auto-reject-of-expired-3pid-invites)
+
+## Setup
+> **NOTE:** Make sure you are familiar with [configuration format and rules](../../configure.md).
+
+Integration as an Application service is a three steps process:
+1. Create the baseline mxisd configuration to allow integration.
+2. Integrate with the homeserver.
+3. Configure the specific capabilities, if applicable.
+
+### Configuration
+#### Variables
+Under the `appsvc` namespace:
+
+| Key                   | Type    | Required | Default | Purpose                                                        |
+|-----------------------|---------|----------|---------|----------------------------------------------------------------|
+| `enabled`             | boolean | No       | `false` | Globally enable/disable the feature                            |
+| `user.main`           | string  | No       | `mxisd` | Localpart for the main appservice user                         |
+| `endpoint.toHS.url`   | string  | Yes      | *None*  | Base URL to the Homeserver                                     |
+| `endpoint.toHS.token` | string  | Yes      | *None*  | Token to use when sending requests to the Homeserver           |
+| `endpoint.toAS.url`   | string  | Yes      | *None*  | Base URL to mxisd from the Homeserver                          |
+| `endpoint.toAS.token` | string  | Yes      | *None*  | Token for the Homeserver to use when sending requests to mxisd |
+
+#### Example
+```yaml
+appsvc:
+  enabled: true
+  endpoint:
+    toHS:
+      url: 'http://localhost:8008'
+      token: 'ExampleTokenToHS-ChangeMe!'
+    toAS:
+      url: 'http://localhost:8090'
+      token: 'ExampleTokenToAS-ChangeMe!'
+```
+### Integration
+#### Synapse
+Under the `appsvc.registration.synapse` namespace:
+
+| Key    | Type   | Required | Default            | Purpose                                                                  |
+|--------|--------|----------|--------------------|--------------------------------------------------------------------------|
+| `id`   | string | No       | `appservice-mxisd` | The unique, user-defined ID of this application service. See spec.       |
+| `file` | string | Yes      | *None*             | If defined, the synapse registration file that should be created/updated |
+
+##### Example 
+```yaml
+appsvc:
+  registration:
+    synapse:
+      file: '/etc/matrix-synapse/mxisd-appservice-registration.yaml'
+```
+
+Edit your `homeserver.yaml` and add a new entry to the appservice config file, which should look something like this:
+```yaml
+app_service_config_files:
+  - '/etc/matrix-synapse/mxisd-appservice-registration.yaml'
+  - ...
+```
+
+Restart synapse when done to register mxisd.
+
+#### Others
+See your Homeserver documentation on how to integrate.
+
+## Capabilities
+### Admin commands
+#### Setup
+Min config:
+```yaml
+appsvc:
+  feature:
+    admin:
+      allowedRoles:
+        - '+aMatrixCommunity:example.org'
+        - 'SomeLdapGroup'
+        - 'AnyOtherArbitraryRoleFromIdentityStores'
+```
+
+#### Use
+The following steps assume:
+- `matrix.domain` set to `example.org`
+- `appsvc.user.main` set to `mxisd` or not set
+
+1. Invite `@mxisd:example.org` to a new direct chat
+2. Type `!help` to get all available commands
+
+### Email Notification about room invites by Matrix IDs
 This feature allows for users found in Identity stores to be instantly notified about Room Invites, regardless if their
 account was already provisioned on the Homeserver.
 
-### Requirements
+#### Requirements
 - [Identity store(s)](../../stores/README.md) supporting the Profile feature
 - At least one email entry in the identity store for each user that could be invited.
 
-### Configuration
+#### Configuration
 In your mxisd config file:
 ```yaml
-matrix:
-  listener:
-    url:  '<URL TO THE CS API OF THE HOMESERVER>'
-    localpart: 'appservice-mxisd'
-    token:
-      hs: 'HS_TOKEN_CHANGE_ME'
-
 synapseSql:
   enabled: false ## Do not use this line if Synapse is used as an Identity Store
   type: '<DB TYPE>'
@@ -33,40 +115,8 @@ If you do not configure it, some placeholders will not be available in the notif
 You can also change the default template of the notification using the `generic.matrixId` template option.  
 See [the Template generator documentation](../../threepids/notification/template-generator.md) for more info.
 
-### Homeserver integration
+#### Test
-#### Synapse
-Create a new appservice registration file. Futher config will assume it is in `/etc/matrix-synapse/appservice-mxisd.yaml`
-```yaml
-id: "appservice-mxisd"
-url: "http://127.0.0.1:8090"
-as_token: "AS_TOKEN_CHANGE_ME"
-hs_token: "HS_TOKEN_CHANGE_ME"
-sender_localpart: "appservice-mxisd"
-namespaces:
-  users:
-    - regex: "@*"
-      exclusive: false
-  aliases: []
-  rooms: []
-```
-`id`: An arbitrary unique string to identify the AS.  
-`url`: mxisd to reach mxisd. This ideally should be HTTP and not going through any reverse proxy.  
-`as_token`: Arbitrary value used by mxisd when talking to the HS. Not currently used.  
-`hs_token`: Arbitrary value used by synapse when talking to mxisd. Must match `token.hs` in mxisd config.
-`sender_localpart`: Username for the mxisd itself on the HS. Default configuration should be kept.  
-`namespaces`: To be kept as is.  
-
-Edit your `homeserver.yaml` and add a new entry to the appservice config file, which should look something like this:
-```yaml
-app_service_config_files:
-  - '/etc/matrix-synapse/appservice-mxisd.yaml'
-  - ...
-```
-
-Restart synapse when done to register mxisd.
-
-#### Others
-See your Homeserver documentation on how to integrate.
-
-### Test
 Invite a user which is part of your domain while an appropriate Identity store is used.
+
+### Auto-reject of expired 3PID invites
+*TBC*

docs/features/identity.md

@@ -1,6 +1,13 @@
 # Identity
 Implementation of the [Identity Service API r0.1.0](https://matrix.org/docs/spec/identity_service/r0.1.0.html).
 
+- [Lookups](#lookups)
+- [Invitations](#invitations)
+  - [Expiration](#expiration)
+  - [Policies](#policies)
+  - [Resolution](#resolution)
+- [3PIDs Management](#3pids-management)
+
 ## Lookups
 If you would like to use the central matrix.org Identity server to ensure maximum discovery at the cost of potentially
 leaking all your contacts information, add the following to your configuration:
@@ -12,8 +19,78 @@ forward:
 **NOTE:** You should carefully consider enabling this option, which is discouraged.  
 For more info, see the [relevant issue](https://github.com/kamax-matrix/mxisd/issues/76).
 
-## Room Invitations
+## Invitations
-Resolution can be customized using the following configuration:
+### Expiration
+#### Overview
+Matrix does not provide a mean to remove/cancel pending 3PID invitations with the APIs. The current reference
+implementations also do not provide any mean to do so. This leads to 3PID invites forever stuck in rooms.
+
+To provide this functionality, mxisd uses a workaround: resolve the invite to a dedicated User ID, which can be
+controlled by mxisd or a bot/service that will then reject the invite.
+
+If this dedicated User ID is to be controlled by mxisd, the [Application Service](experimental/application-service.md)
+feature must be configured and integrated with your Homeserver, as well as the *Auto-reject 3PID invite capability*.
+
+#### Configuration
+```yaml
+invite:
+  expiration:
+    enabled: true/false
+    after: 5
+    resolveTo: '@john.doe:example.org'
+```
+`enabled`
+- Purpose: Enable or disable the invite expiration feature.
+- Default: `true`
+
+`after` 
+- Purpose: Amount of minutes before an invitation expires.
+- Default: `10080` (7 days)
+
+`resolveTo`
+- Purpose: Matrix User ID to resolve the expired invitations to.
+- Default: Computed from `appsvc.user.inviteExpired` and `matrix.domain`
+
+### Policies
+3PID invite policies are the companion feature of [Registration](registration.md). While the Registration feature acts on
+requirements for the invitee/register, this feature acts on requirement for the one(s) performing 3PID invites, ensuring
+a coherent system.
+
+It relies on only allowing people with specific [Roles](profile.md) to perform 3PID invites. This would typically allow
+a tight-control on a server setup with is "invite-only" or semi-open (relying on trusted people to invite new members).
+
+It's a middle ground between a closed server, where every user must be created or already exists in an Identity store,
+and an open server, where anyone can register.
+ 
+#### Integration
+Because Identity Servers do not control 3PID invites as per Matrix spec, mxisd needs to intercept a set of Homeserver
+endpoints to apply the policies.
+
+##### Reverse Proxy
+###### nginx
+**IMPORTANT**: Must be placed before your global `/_matrix` entry:
+```nginx
+location ~* ^/_matrix/client/r0/rooms/([^/]+)/invite$ {
+    proxy_pass		    http://127.0.0.1:8090;
+    proxy_set_header	Host $host;
+    proxy_set_header	X-Forwarded-For $remote_addr;
+}
+```
+
+#### Configuration
+The only policy currently available is to restrict 3PID invite to users having a specific (set of) role(s), like so:
+
+```yaml
+invite:
+  policy:
+    ifSender:
+      hasRole:
+        - '<THIS_ROLE>'
+        - '<OR_THIS_ROLE>'
+```
+
+### Resolution
+Resolution of 3PID invitations can be customized using the following configuration:
 
 `invite.resolution.recursive`  
 - Default value: `true`  
@@ -26,5 +103,5 @@ Resolution can be customized using the following configuration:
 - Default value: `1`  
 - Description: How often, in minutes, mxisd should try to resolve pending invites.
 
-## 3PID addition to user profile
+## 3PIDs Management
 See the [3PID session documents](../threepids/session)

docs/features/registration.md

@@ -0,0 +1,111 @@
+# Registration
+- [Overview](#overview)
+- [Integration](#integration)
+  - [Reverse Proxy](#reverse-proxy)
+    - [nginx](#nginx)
+    - [Apache](#apache)
+  - [Homeserver](#homeserver)
+    - [synapse](#synapse)
+- [Configuration](#configuration)
+  - [Example](#example)
+- [Usage](#usage)
+
+## Overview
+**NOTE**: This feature is beta: it is considered stable enough for production but is incomplete and may contain bugs.
+
+Registration is an enhanced feature of mxisd to control registrations involving 3PIDs on a Homeserver based on policies:
+- Match pending 3PID invites on the server
+- Match 3PID pattern, like a specific set of domains for emails
+- In futher releases, use 3PIDs found in Identity stores
+
+It aims to help open or invite-only registration servers control what is possible to do and ensure only approved people
+can register on a given server in a implementation-agnostic manner.
+
+**IMPORTANT:** This feature does not control registration in general. It only acts on endpoints related to 3PIDs during
+the registration process.  
+As such, it relies on the homeserver to require 3PIDs with the registration flows.
+
+This feature is not part of the Matrix Identity Server spec.
+
+## Integration
+mxisd needs to be integrated at several levels for this feature to work:
+- Reverse proxy: intercept the 3PID register endpoints and act on them
+- Homeserver: require 3PID to be part of the registration data
+
+Later version(s) of this feature may directly control registration itself to create a coherent experience
+### Reverse Proxy
+#### nginx
+```nginx
+location ^/_matrix/client/r0/register/[^/]/?$ {
+	proxy_pass		http://127.0.0.1:8090;
+	proxy_set_header	Host $host;
+	proxy_set_header	X-Forwarded-For $remote_addr;
+}
+```
+
+#### apache
+> TBC
+
+### Homeserver
+#### Synapse
+```yaml
+enable_registration: true
+registrations_require_3pid:
+  - email
+```
+
+## Configuration
+See the [Configuration](../configuration.md) introduction doc on how to read the configuration keys.  
+An example of working configuration is avaiable at the end of this section.
+### Enable/Disable
+`register.allowed`, taking a boolean, can be used to enable/disable registration if the attempt is not 3PID-based.  
+`false` is the default value to prevent open registration, as you must allow it on the homeserver side.
+
+### For invites
+`register.invite`, taking a boolean, controls if registration can be made using a 3PID which matches a pending 3PID invite.  
+`true` is the default value.
+
+### 3PID-specific
+At this time, only `email` is supported with 3PID specific configuration with this feature.
+
+#### Email
+**Base key**: `register.threepid.email`
+
+##### Domain whitelist/blacklist
+If you would like to control which domains are allowed to be used when registrating with an email, the following sub-keys
+are available:
+- `domain.whitelist`
+- `domain.blacklist`
+
+The value format is an hybrid between glob patterns and postfix configuration files with the following syntax:
+- `*<domain>` will match the domain and any sub-domain(s)
+- `.<domain>` will only match sub-domain(s)
+- `<domain>` will only match the exact domain
+
+The following table illustrates pattern and maching status against example values:
+
+| Config value   | Matches `example.org` | Matches `sub.example.org` |
+|--------------- |-----------------------|---------------------------|
+| `*example.org` | Yes                   | Yes                       |
+| `.example.org` | No                    | Yes                       |
+| `example.org`  | Yes                   | No                        |
+
+### Example
+For the following example configuration:
+```yaml
+register:
+  policy:
+    threepid:
+      email:
+        domain:
+          whitelist:
+            - '*example.org'
+            - '.example.net'
+            - 'example.com'
+```
+- Users can register using 3PIDs of pending invites, being allowed by default.
+- Users can register using an email from `example.org` and any sub-domain, only sub-domains of `example.net` and `example.com` but not its sub-domains.
+- Otherwise, user registration will be denied.
+
+## Usage
+Nothing special is needed. Register using a regular Matrix client.

docs/getting-started.md

@@ -153,3 +153,5 @@ infrastructure:
 
 - [Enable extra features](features/)
 - [Use your own Identity stores](stores/README.md)
+- [Hardening your mxisd installation](install/security.md)
+- [Learn about day-to-day operations](operations.md)

docs/install/docker.md

@@ -20,3 +20,6 @@ docker run --rm -e MATRIX_DOMAIN=example.org -v /data/mxisd/etc:/etc/mxisd -v /d
 ```
 
 For more info, including the list of possible tags, see [the public repository](https://hub.docker.com/r/kamax/mxisd/)
+
+## Troubleshoot
+Please read the [Troubleshooting guide](../troubleshooting.md).

docs/install/security.md

@@ -0,0 +1,30 @@
+# Security hardening
+## Overview
+This document outlines the various operations you may want to perform to increase the security of your installation and
+avoid leak of credentials/key pairs
+
+## Configuration
+Your config file should have the following ownership:
+- Dedicated user for mxisd, used to run the software
+- Dedicated group for mxisd, used by other applications to access and read configuration files
+
+Your config file should have the following access:
+- Read and write for the mxisd user
+- Read for the mxisd group
+- Nothing for others
+
+This translates into `640` and be applied with `chmod 640 /path/to/config/file.yaml`.
+
+## Data
+The only sensible place is the key store where mxisd's signing keys are stored. You should therefore limit access to only
+the mxisd user, and deny access to anything else.
+
+Your key store should have the following access:
+- Read and write for the mxisd user
+- Nothing for the mxisd group
+- Nothing for others
+
+The identity store can either be a file or a directory, depending on your version. v1.4 and higher are using a directory,
+everything before is using a file.
+- If your version is directory-based, you will want to apply chmod `700` on it.
+- If your version is file-based, you will want to apply chmod `600` on it.

docs/operations.md

@@ -0,0 +1,21 @@
+# Operations Guide
+- [Overview](#overview)
+- [Maintenance](#maintenance)
+- [Backuo](#backup)
+
+## Overview
+This document gives various information for the day-to-day management and operations of mxisd.
+
+## Maintenance
+mxisd does not require any maintenance task to run at optimal performance.
+
+## Backup
+### Run
+mxisd requires all file in its configuration and data directory to be backed up.  
+They are usually located at:
+- `/etc/mxisd`
+- `/var/lib/mxisd`
+
+### Restore
+Reinstall mxisd, restore the two folders above in the appropriate location (depending on your install method) and you
+will be good to go. Simply start mxisd to restore functionality.

docs/stores/ldap.md

@@ -89,7 +89,7 @@ ldap:
 #### 3PIDs
 You can also change the attribute lists for 3PID, like email or phone numbers.
 
-The following example would overwrite the [default list of attributes](../../src/main/resources/application.yaml#L67)
+The following example would overwrite the [default list of attributes](../../src/main/java/io/kamax/mxisd/config/ldap/LdapConfig.java#L64)
 for emails and phone number:
 ```yaml
 ldap:

docs/stores/sql.md

@@ -102,9 +102,41 @@ sql:
 ```
 
 ### Identity
+**NOTE**: Only single lookup is supported. Bulk lookup always returns no mapping. This is a restriction as the Matrix API
+does not allow paging or otherwise limit of results of the API, potentially leading to thousands and thousands 3PIDs at once.
+
 ```yaml
 sql:
   identity:
+    enabled: <boolean>
     type: <string>
     query: <string>
+    medium:
+     mediumTypeExample: <dedicated query>
 ```
+`type` is used to tell mxisd how to process the returned `uid` column containing the User ID:
+- `localpart` will build a full Matrix ID using the `matrix.domain` value.
+- `mxid` will use the ID as-is. If it is not a valid Matrix ID, lookup(s) will fail.
+
+A specific query can also given per 3PID medium type.
+
+### Profile
+```yaml
+sql:
+  profile:
+    enabled: <boolean>
+    displayName:
+      query: <string>
+    threepid:
+      query: <string>
+    role:
+      type: <string>
+      query: <string>
+    
+
+```
+For the `role` query, `type` can be used to tell mxisd how to inject the User ID in the query:
+- `localpart` will extract and set only the localpart.
+- `mxid` will use the ID as-is.
+
+On each query, the first parameter `?` is set as a string with the corresponding ID format.

docs/stores/synapse.md

@@ -1,5 +1,6 @@
 # Synapse Identity Store
-Synapse's Database itself can be used as an Identity store.
+Synapse's Database itself can be used as an Identity store. This identity store is a regular SQL store with
+built-in default queries that matches Synapse DB.
 
 ## Features
 |                       Name                      | Supported |
@@ -9,7 +10,8 @@ Synapse's Database itself can be used as an Identity store.
 | [Identity](../features/identity.md)             | Yes       |
 | [Profile](../features/profile.md)               | Yes       |
 
-Authentication is done by Synapse itself.
+- Authentication is done by Synapse itself.
+- Roles are mapped to communities. The Role name/ID uses the community ID in the form `+id:domain.tld`
 
 ## Configuration
 ### Basic

docs/threepids/medium/email/smtp-connector.md

@@ -12,8 +12,8 @@ threepid:
       connectors:
         smtp:
           host: 'smtpHostname'
-          port: 587
+          tls: 1 # 0 = no STARTLS, 1 = try, 2 = force, 3 = TLS/SSL
-          tls: 1 # 0 = no STARTLS, 1 = try, 2 = force
+          port: 587 # Set appropriate value depending on your TLS setting
           login: 'smtpLogin'
           password: 'smtpPassword'
 ```

docs/threepids/medium/msisdn/twilio-connector.md

@@ -8,7 +8,7 @@ threepid:
     msisdn:
       connectors:
         twilio:
-          accountSid: 'myAccountSid'
+          account_sid: 'myAccountSid'
-          authToken: 'myAuthToken'
+          auth_token: 'myAuthToken'
           number: '+123456789'
 ```

docs/threepids/notification/template-generator.md

@@ -1,63 +1,109 @@
-# Notifications: Generate from templates
+# Notifications: Template generator
-To create notification content, you can use the `template` generator if supported for the 3PID medium which will read
+Most of the Identity actions will trigger a notification of some kind, informing the user of some confirmation, next step
-content from configured files.
+or just informing them about the current state of things.
 
-Placeholders can be integrated into the templates to dynamically populate such content with relevant information like
+Those notifications are by default generated from templates and by replacing placeholder tokens in them with the relevant
-the 3PID that was requested, the domain of your Identity server, etc.
+values of the notification. It is possible to customize the value of some placeholders, making easy to set values in the builtin templates, and/or
+provide your own custom templates.
 
-Templates can be configured for each event that would send a notification to the end user. Events share a set of common
+Templates for the following events/actions are available:
-placeholders and also have their own individual set of placeholders.
+- [3PID invite](../../features/identity.md)
+- [3PID session: validation](../session/session.md)
+- [3PID session: fraudulent unbind](https://github.com/kamax-matrix/mxisd/wiki/mxisd-and-your-privacy#improving-your-privacy-one-commit-at-the-time)
+- [Matrix ID invite](../../features/experimental/application-service.md#email-notification-about-room-invites-by-matrix-ids)
+
+## Placeholders
+All placeholders **MUST** be surrounded with `%` in the template. Per example, the `DOMAIN` placeholder would become
+`%DOMAIN%` within the template. This ensures replacement doesn't happen on non-placeholder strings.
+
+### Global
+The following placeholders are available in every template:
+
+| Placeholder         | Purpose                                                                      |
+|---------------------|------------------------------------------------------------------------------|
+| `DOMAIN`            | Identity server authoritative domain, as configured in `matrix.domain`       |
+| `DOMAIN_PRETTY`     | Same as `DOMAIN` with the first letter upper case and all other lower case   |
+| `FROM_EMAIL`        | Email address configured in `threepid.medium.<3PID medium>.identity.from`    |
+| `FROM_NAME`         | Name configured in `threepid.medium.<3PID medium>.identity.name`             |
+| `RECIPIENT_MEDIUM`  | The 3PID medium, like `email` or `msisdn`                                    |
+| `RECIPIENT_ADDRESS` | The address to which the notification is sent                                |
+
+### Room invitation
+Specific placeholders:
+
+| Placeholder         | Purpose                                                                                  |
+|---------------------|------------------------------------------------------------------------------------------|
+| `SENDER_ID`         | Matrix ID of the user who made the invite                                                |
+| `SENDER_NAME`       | Display name of the user who made the invite, if not available/set, empty                |
+| `SENDER_NAME_OR_ID` | Display name of the user who made the invite. If not available/set, its Matrix ID        |
+| `INVITE_MEDIUM`     | The 3PID medium for the invite.                                                          |
+| `INVITE_ADDRESS`    | The 3PID address for the invite.                                                         |
+| `ROOM_ID`           | The Matrix ID of the Room in which the invite took place                                 |
+| `ROOM_NAME`         | The Name of the room in which the invite took place. If not available/set, empty         |
+| `ROOM_NAME_OR_ID`   | The Name of the room in which the invite took place. If not available/set, its Matrix ID |
+| `REGISTER_URL`      | The URL to provide to the user allowing them to register their account, if needed        |
+
+### Validation of 3PID Session
+Specific placeholders:
+
+| Placeholder        | Purpose                                                                              |
+|--------------------|--------------------------------------------------------------------------------------|
+| `VALIDATION_LINK`  | URL, including token, to validate the 3PID session.                                  |
+| `VALIDATION_TOKEN` | The token needed to validate the session, in case the user cannot use the link.      |
+| `NEXT_URL`         | URL to redirect to after the sessions has been validated.                            |
+
+## Templates
+mxisd comes with a set of builtin templates to easily get started. Those templates can be found
+[in the repository](https://github.com/kamax-matrix/mxisd/tree/master/src/main/resources/threepids). If you want to use
+customized templates, we recommend using the builtin templates as a starting point.
+
+> **NOTE**: The link above point to the latest version of the built-in templates. Those might be different from your
+version. Be sure to view the repo at the current tag.
 
 ## Configuration
+All configuration is specific to [3PID mediums](https://matrix.org/docs/spec/appendices.html#pid-types) and happen
+under the namespace `threepid.medium.<medium>.generators.template`.
+
+Under such namespace, the following keys are available:
+- `invite`: Path to the 3PID invite notification template
+- `session.validation`: Path to the 3PID session validation notification template
+- `session.unbind.fraudulent`: Path to the 3PID session fraudulent unbind notification template
+- `generic.matrixId`: Path to the Matrix ID invite notification template
+- `placeholder`: Map of key/values to set static values for some placeholders.
+
+The `placeholder` map supports the following keys, mapped to their respective template placeholders:
+- `REGISTER_URL`
+
+### Example
+#### Simple
+```yaml
+threepid:
+  medium:
+    email:
+      generators:
+        template:
+          placeholder:
+            REGISTER_URL: 'https://matrix-client.example.org'
+```
+In this configuration, the builtin templates are used and a static value for the `REGISTER_URL` is set, allowing to point
+a newly invited user to a webapp allowing the creation of its account on the server.
+
+#### Advanced
 To configure paths to the various templates:
 ```yaml
 threepid:
   medium:
-    <YOUR 3PID MEDIUM HERE>:
+    email:
       generators:
         template:
           invite: '/path/to/invite-template.eml'
           session:
             validation: '/path/to/validate-template.eml'
             unbind:
-              frandulent: '/path/to/unbind-fraudulent-template.eml'
+              fraudulent: '/path/to/unbind-fraudulent-template.eml'
           generic:
             matrixId: '/path/to/mxid-invite-template.eml'
+          placeholder:
+            REGISTER_URL: 'https://matrix-client.example.org'
 ```
-The `template` generator is usually the default, so no further configuration is needed.
+In this configuration, a custom template is used for each event and a static value for the `REGISTER_URL` is set.
-
-##  Global placeholders
-| Placeholder           | Purpose                                                                      |
-|-----------------------|------------------------------------------------------------------------------|
-| `%DOMAIN%`            | Identity server authoritative domain, as configured in `matrix.domain`       |
-| `%DOMAIN_PRETTY%`     | Same as `%DOMAIN%` with the first letter upper case and all other lower case |
-| `%FROM_EMAIL%`        | Email address configured in `threepid.medium.<3PID medium>.identity.from`    |
-| `%FROM_NAME%`         | Name configured in `threepid.medium.<3PID medium>.identity.name`             |
-| `%RECIPIENT_MEDIUM%`  | The 3PID medium, like `email` or `msisdn`                                    |
-| `%RECIPIENT_ADDRESS%` | The address to which the notification is sent                                |
-
-## Events
-### Room invitation
-This template is used when someone is invited into a room using an email address which has no known bind to a Matrix ID.
-#### Placeholders
-| Placeholder           | Purpose                                                                                  |
-|-----------------------|------------------------------------------------------------------------------------------|
-| `%SENDER_ID%`         | Matrix ID of the user who made the invite                                                |
-| `%SENDER_NAME%`       | Display name of the user who made the invite, if not available/set, empty                |
-| `%SENDER_NAME_OR_ID%` | Display name of the user who made the invite. If not available/set, its Matrix ID        |
-| `%INVITE_MEDIUM%`     | The 3PID medium for the invite.                                                          |
-| `%INVITE_ADDRESS%`    | The 3PID address for the invite.                                                         |
-| `%ROOM_ID%`           | The Matrix ID of the Room in which the invite took place                                 |
-| `%ROOM_NAME%`         | The Name of the room in which the invite took place. If not available/set, empty         |
-| `%ROOM_NAME_OR_ID%`   | The Name of the room in which the invite took place. If not available/set, its Matrix ID |
-
-### Validation of 3PID Session
-This template is used when to user which added their 3PID address to their profile/settings and the session policy
-allows at least local sessions.  
-
-#### Placeholders
-| Placeholder          | Purpose                                                                              |
-|----------------------|--------------------------------------------------------------------------------------|
-| `%VALIDATION_LINK%`  | URL, including token, to validate the 3PID session.                                  |
-| `%VALIDATION_TOKEN%` | The token needed to validate the session, in case the user cannot use the link.      |
-| `%NEXT_URL%`         | URL to redirect to after the sessions has been validated.                            |

docs/troubleshooting.md

@@ -19,6 +19,11 @@ If you use the [Docker image](install/docker.md), this goes to the container log
 
 For any other platform, please refer to your package maintainer.
 
+### Increase verbosity
+To increase log verbosity and better track issues, the following means are available:
+- Add the `-v` command line parameter
+- Use the environment variable and value `MXISD_LOG_LEVEL=debug`
+
 ### Reading them
 Before reporting an issue, it is important to produce clean and complete logs so they can be understood.
 

mxisd.example.yaml

@@ -91,19 +91,19 @@ threepid:
           # SMTP host
           host: "smtp.example.org"
 
-          # SMTP port
+          # TLS mode for the connection
-          port: 587
-
-          # STARTLS mode for the connection.
-          # SSL/TLS is currently not supported. See https://github.com/kamax-matrix/mxisd/issues/125
-          #
           # Possible values:
           #  0    Disable any kind of TLS entirely
           #  1    Enable STARTLS if supported by server (default)
           #  2    Force STARTLS and fail if not available
+          #  3    Use full TLS/SSL instead of STARTLS
           #
           tls: 1
 
+          # SMTP port
+          # Be sure to adapt depending on your TLS choice, if changed from default
+          port: 587
+
           # Login for SMTP
           login: "matrix-identity@example.org"
 

src/debian/control

@@ -3,5 +3,7 @@ Maintainer: Kamax.io <foss@kamax.io>
 Homepage: https://github.com/kamax-matrix/mxisd
 Description: Federated Matrix Identity Server
 Architecture: all
+Section: net
+Priority: optional
 Depends: openjdk-8-jre | openjdk-8-jre-headless | openjdk-8-jdk | openjdk-8-jdk-headless
 Version: 0

src/main/java/io/kamax/mxisd/HttpMxisd.java

@@ -73,7 +73,6 @@ public class HttpMxisd {
         HttpHandler asNotFoundHandler = SaneHandler.around(new AsNotFoundHandler(m.getAs()));
 
         HttpHandler storeInvHandler = SaneHandler.around(new StoreInviteHandler(m.getConfig().getServer(), m.getInvite(), m.getKeyManager()));
-        HttpHandler sessValidateHandler = SaneHandler.around(new SessionValidateHandler(m.getSession(), m.getConfig().getServer(), m.getConfig().getView()));
 
         httpSrv = Undertow.builder().addHttpListener(m.getConfig().getServer().getPort(), "0.0.0.0").setHandler(Handlers.routing()
 
@@ -103,8 +102,8 @@ public class HttpMxisd {
                 .post(BulkLookupHandler.Path, SaneHandler.around(new BulkLookupHandler(m.getIdentity())))
                 .post(StoreInviteHandler.Path, storeInvHandler)
                 .post(SessionStartHandler.Path, SaneHandler.around(new SessionStartHandler(m.getSession())))
-                .get(SessionValidateHandler.Path, sessValidateHandler)
+                .get(SessionValidateHandler.Path, SaneHandler.around(new SessionValidationGetHandler(m.getSession(), m.getConfig())))
-                .post(SessionValidateHandler.Path, sessValidateHandler)
+                .post(SessionValidateHandler.Path, SaneHandler.around(new SessionValidationPostHandler(m.getSession())))
                 .get(SessionTpidGetValidatedHandler.Path, SaneHandler.around(new SessionTpidGetValidatedHandler(m.getSession())))
                 .post(SessionTpidBindHandler.Path, SaneHandler.around(new SessionTpidBindHandler(m.getSession(), m.getInvite())))
                 .post(SessionTpidUnbindHandler.Path, SaneHandler.around(new SessionTpidUnbindHandler(m.getSession())))

src/main/java/io/kamax/mxisd/Mxisd.java

@@ -41,6 +41,7 @@ import io.kamax.mxisd.lookup.provider.BridgeFetcher;
 import io.kamax.mxisd.lookup.provider.RemoteIdentityServerFetcher;
 import io.kamax.mxisd.lookup.strategy.LookupStrategy;
 import io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy;
+import io.kamax.mxisd.matrix.HomeserverFederationResolver;
 import io.kamax.mxisd.matrix.IdentityServerUtils;
 import io.kamax.mxisd.notification.NotificationHandlerSupplier;
 import io.kamax.mxisd.notification.NotificationHandlers;
@@ -99,6 +100,8 @@ public class Mxisd {
                 .setMaxConnTotal(Integer.MAX_VALUE)
                 .build();
 
+        FederationDnsOverwrite fedDns = new FederationDnsOverwrite(cfg.getDns().getOverwrite());
+        HomeserverFederationResolver resolver = new HomeserverFederationResolver(fedDns, httpClient);
         IdentityServerUtils.setHttpClient(httpClient);
         srvFetcher = new RemoteIdentityServerFetcher(httpClient);
 
@@ -106,10 +109,9 @@ public class Mxisd {
         keyMgr = CryptoFactory.getKeyManager(cfg.getKey());
         signMgr = CryptoFactory.getSignatureManager(keyMgr);
         clientDns = new ClientDnsOverwrite(cfg.getDns().getOverwrite());
-        FederationDnsOverwrite fedDns = new FederationDnsOverwrite(cfg.getDns().getOverwrite());
+
         synapse = new Synapse(cfg.getSynapseSql());
         BridgeFetcher bridgeFetcher = new BridgeFetcher(cfg.getLookup().getRecursive().getBridge(), srvFetcher);
-
         ServiceLoader.load(IdentityStoreSupplier.class).iterator().forEachRemaining(p -> p.accept(this));
         ServiceLoader.load(NotificationHandlerSupplier.class).iterator().forEachRemaining(p -> p.accept(this));
 
@@ -117,7 +119,7 @@ public class Mxisd {
         pMgr = new ProfileManager(ProfileProviders.get(), clientDns, httpClient);
         notifMgr = new NotificationManager(cfg.getNotification(), NotificationHandlers.get());
         sessMgr = new SessionManager(cfg.getSession(), cfg.getMatrix(), store, notifMgr, idStrategy, httpClient);
-        invMgr = new InvitationManager(cfg, store, idStrategy, keyMgr, signMgr, fedDns, notifMgr, pMgr);
+        invMgr = new InvitationManager(cfg, store, idStrategy, keyMgr, signMgr, resolver, notifMgr, pMgr);
         authMgr = new AuthManager(cfg, AuthProviders.get(), idStrategy, invMgr, clientDns, httpClient);
         dirMgr = new DirectoryManager(cfg.getDirectory(), clientDns, httpClient, DirectoryProviders.get());
         regMgr = new RegistrationManager(cfg.getRegister(), httpClient, clientDns, invMgr);

src/main/java/io/kamax/mxisd/MxisdStandaloneExec.java

@@ -36,6 +36,11 @@ public class MxisdStandaloneExec {
     private static final Logger log = LoggerFactory.getLogger("App");
 
     public static void main(String[] args) {
+        String logLevel = System.getenv("MXISD_LOG_LEVEL");
+        if (StringUtils.isNotBlank(logLevel)) {
+            System.setProperty("org.slf4j.simpleLogger.log.io.kamax.mxisd", logLevel);
+        }
+
         try {
             MxisdConfig cfg = null;
             Iterator<String> argsIt = Arrays.asList(args).iterator();
@@ -46,8 +51,14 @@ public class MxisdStandaloneExec {
                     System.out.println("  -h, --help       Show this help message");
                     System.out.println("  --version        Print the version then exit");
                     System.out.println("  -c, --config     Set the configuration file location");
+                    System.out.println("  -v               Increase log level (log more info)");
+                    System.out.println("  -vv              Further increase log level");
                     System.out.println(" ");
                     System.exit(0);
+                } else if (StringUtils.equals(arg, "-v")) {
+                    System.setProperty("org.slf4j.simpleLogger.log.io.kamax.mxisd", "debug");
+                } else if (StringUtils.equals(arg, "-vv")) {
+                    System.setProperty("org.slf4j.simpleLogger.log.io.kamax.mxisd", "trace");
                 } else if (StringUtils.equalsAny(arg, "-c", "--config")) {
                     String cfgFile = argsIt.next();
                     cfg = YamlConfigLoader.loadFromFile(cfgFile);
@@ -61,13 +72,13 @@ public class MxisdStandaloneExec {
                 }
             }
 
-            log.info("mxisd starting");
-            log.info("Version: {}", Mxisd.Version);
-
             if (Objects.isNull(cfg)) {
                 cfg = YamlConfigLoader.tryLoadFromFile("mxisd.yaml").orElseGet(MxisdConfig::new);
             }
 
+            log.info("mxisd starting");
+            log.info("Version: {}", Mxisd.Version);
+
             HttpMxisd mxisd = new HttpMxisd(cfg);
             Runtime.getRuntime().addShutdownHook(new Thread(() -> {
                 mxisd.stop();

src/main/java/io/kamax/mxisd/config/InvitationConfig.java

@@ -34,7 +34,7 @@ public class InvitationConfig {
     public static class Expiration {
 
         private Boolean enabled;
-        private long after;
+        private long after = 60 * 24 * 7; // One calendar week (60min/1h * 24 = 1d * 7 = 1w)
         private String resolveTo;
 
         public Boolean isEnabled() {

src/main/java/io/kamax/mxisd/config/RegisterConfig.java

@@ -69,16 +69,16 @@ public class RegisterConfig {
         }
 
         private List<String> buildPatterns(List<String> domains) {
-            log.info("Building email policy");
+            log.debug("Building email policy");
             return domains.stream().map(d -> {
                 if (StringUtils.startsWith(d, "*")) {
-                    log.info("Found domain and subdomain policy");
+                    log.debug("Found domain and subdomain policy");
                     d = "(.*)" + d.substring(1);
                 } else if (StringUtils.startsWith(d, ".")) {
-                    log.info("Found subdomain-only policy");
+                    log.debug("Found subdomain-only policy");
                     d = "(.*)" + d;
                 } else {
-                    log.info("Found domain-only policy");
+                    log.debug("Found domain-only policy");
                 }
 
                 return "([^@]+)@" + d.replace(".", "\\.");
@@ -175,10 +175,10 @@ public class RegisterConfig {
     }
 
     public void build() {
-        log.info("--- Registration config ---");
+        log.debug("--- Registration config ---");
 
-        log.info("Before Build");
+        log.debug("Before Build");
-        log.info(GsonUtil.getPrettyForLog(this));
+        log.debug(GsonUtil.getPrettyForLog(this));
 
         new HashMap<>(getPolicy().getThreepid()).forEach((medium, policy) -> {
             if (ThreePidMedium.Email.is(medium)) {
@@ -194,8 +194,8 @@ public class RegisterConfig {
             getPolicy().getThreepid().put(medium, policy);
         });
 
-        log.info("After Build");
+        log.debug("After Build");
-        log.info(GsonUtil.getPrettyForLog(this));
+        log.debug(GsonUtil.getPrettyForLog(this));
     }
 
 }

src/main/java/io/kamax/mxisd/config/threepid/medium/GenericTemplateConfig.java

@@ -77,6 +77,7 @@ public class GenericTemplateConfig {
     private String invite;
     private Session session = new Session();
     private Map<String, String> generic = new HashMap<>();
+    private Map<String, String> placeholder = new HashMap<>();
 
     public String getInvite() {
         return invite;
@@ -98,4 +99,12 @@ public class GenericTemplateConfig {
         this.generic = generic;
     }
 
+    public Map<String, String> getPlaceholder() {
+        return placeholder;
+    }
+
+    public void setPlaceholder(Map<String, String> placeholder) {
+        this.placeholder = placeholder;
+    }
+
 }

src/main/java/io/kamax/mxisd/crypto/GenericKeyIdentifier.java

@@ -36,7 +36,7 @@ public class GenericKeyIdentifier implements KeyIdentifier {
 
     public GenericKeyIdentifier(KeyType type, String algo, String serial) {
         if (StringUtils.isAnyBlank(algo, serial)) {
-            throw new IllegalArgumentException("Aglorith and/or Serial cannot be blank");
+            throw new IllegalArgumentException("Algorithm and/or Serial cannot be blank");
         }
 
         this.type = Objects.requireNonNull(type);

src/main/java/io/kamax/mxisd/http/undertow/handler/BasicHttpHandler.java

@@ -31,6 +31,7 @@ import io.kamax.mxisd.proxy.Response;
 import io.kamax.mxisd.util.RestClientUtils;
 import io.undertow.server.HttpHandler;
 import io.undertow.server.HttpServerExchange;
+import io.undertow.server.handlers.form.FormData;
 import io.undertow.util.HttpString;
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang3.StringUtils;
@@ -48,10 +49,7 @@ import java.net.InetSocketAddress;
 import java.net.URI;
 import java.net.URLDecoder;
 import java.nio.charset.StandardCharsets;
-import java.util.Deque;
+import java.util.*;
-import java.util.LinkedList;
-import java.util.Map;
-import java.util.Optional;
 
 public abstract class BasicHttpHandler implements HttpHandler {
 
@@ -122,6 +120,20 @@ public abstract class BasicHttpHandler implements HttpHandler {
         return GsonUtil.parseObj(getBodyUtf8(exchange));
     }
 
+    protected String getOrThrow(FormData data, String key) {
+        FormData.FormValue value = data.getFirst(key);
+        if (Objects.isNull(value)) {
+            throw new IllegalArgumentException("Form key " + key + " is missing");
+        }
+
+        String object = value.getValue();
+        if (Objects.isNull(object)) {
+            throw new IllegalArgumentException("Form key " + key + " does not have a value");
+        }
+
+        return object;
+    }
+
     protected void putHeader(HttpServerExchange ex, String name, String value) {
         ex.getResponseHeaders().put(HttpString.tryFromString(name), value);
     }

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/SessionValidateHandler.java

@@ -20,94 +20,36 @@
 
 package io.kamax.mxisd.http.undertow.handler.identity.v1;
 
-import io.kamax.mxisd.config.ServerConfig;
-import io.kamax.mxisd.config.ViewConfig;
 import io.kamax.mxisd.http.IsAPIv1;
-import io.kamax.mxisd.http.io.identity.SuccessStatusJson;
 import io.kamax.mxisd.http.undertow.handler.BasicHttpHandler;
 import io.kamax.mxisd.session.SessionManager;
 import io.kamax.mxisd.session.ValidationResult;
-import io.kamax.mxisd.util.FileUtil;
+import org.apache.commons.lang3.StringUtils;
-import io.undertow.server.HttpServerExchange;
-import io.undertow.util.HttpString;
-import org.apache.commons.lang.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import java.io.IOException;
+public abstract class SessionValidateHandler extends BasicHttpHandler {
-import java.net.MalformedURLException;
-import java.net.URL;
-
-public class SessionValidateHandler extends BasicHttpHandler {
 
     public static final String Path = IsAPIv1.Base + "/validate/{medium}/submitToken";
 
     private transient final Logger log = LoggerFactory.getLogger(SessionValidateHandler.class);
 
     private SessionManager mgr;
-    private ServerConfig srvCfg;
-    private ViewConfig viewCfg;
 
-    public SessionValidateHandler(SessionManager mgr, ServerConfig srvCfg, ViewConfig viewCfg) {
+    public SessionValidateHandler(SessionManager mgr) {
         this.mgr = mgr;
-        this.srvCfg = srvCfg;
-        this.viewCfg = viewCfg;
     }
 
-    @Override
+    protected ValidationResult handleRequest(String sid, String secret, String token) {
-    public void handleRequest(HttpServerExchange exchange) {
+        if (StringUtils.isEmpty(sid)) {
-        String medium = getQueryParameter(exchange, "medium");
+            throw new IllegalArgumentException("sid is not set or is empty");
-        String sid = getQueryParameter(exchange, "sid");
-        String secret = getQueryParameter(exchange, "client_secret");
-        String token = getQueryParameter(exchange, "token");
-
-        boolean isHtmlRequest = false;
-        for (String v : exchange.getRequestHeaders().get("Accept")) {
-            if (StringUtils.startsWithIgnoreCase(v, "text/html")) {
-                isHtmlRequest = true;
-                break;
-            }
         }
 
-        if (isHtmlRequest) {
+        if (StringUtils.isEmpty(secret)) {
-            handleHtmlRequest(exchange, medium, sid, secret, token);
+            throw new IllegalArgumentException("client secret is not set or is empty");
-        } else {
-            handleJsonRequest(exchange, sid, secret, token);
         }
-    }
 
-    private void handleHtmlRequest(HttpServerExchange exchange, String medium, String sid, String secret, String token) {
+        return mgr.validate(sid, secret, token);
-        log.info("Validating session {} for medium {}", sid, medium);
-        ValidationResult r = mgr.validate(sid, secret, token);
-        log.info("Session {} was validated", sid);
-        if (r.getNextUrl().isPresent()) {
-            String url = r.getNextUrl().get();
-            try {
-                url = new URL(url).toString();
-            } catch (MalformedURLException e) {
-                log.info("Session next URL {} is not a valid one, will prepend public URL {}", url, srvCfg.getPublicUrl());
-                url = srvCfg.getPublicUrl() + r.getNextUrl().get();
-            }
-            log.info("Session {} validation: next URL is present, redirecting to {}", sid, url);
-            exchange.setStatusCode(302);
-            exchange.getResponseHeaders().add(HttpString.tryFromString("Location"), url);
-        } else {
-            try {
-                String data = FileUtil.load(viewCfg.getSession().getOnTokenSubmit().getSuccess());
-                writeBodyAsUtf8(exchange, data);
-            } catch (IOException e) {
-                throw new RuntimeException(e);
-            }
-        }
-    }
-
-    private void handleJsonRequest(HttpServerExchange exchange, String sid, String secret, String token) {
-        log.info("Requested: {}", exchange.getRequestURL());
-
-        mgr.validate(sid, secret, token);
-        log.info("Session {} was validated", sid);
-
-        respondJson(exchange, new SuccessStatusJson(true));
     }
 
 }

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/SessionValidationGetHandler.java

@@ -0,0 +1,82 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.http.undertow.handler.identity.v1;
+
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.config.ServerConfig;
+import io.kamax.mxisd.config.ViewConfig;
+import io.kamax.mxisd.session.SessionManager;
+import io.kamax.mxisd.session.ValidationResult;
+import io.kamax.mxisd.util.FileUtil;
+import io.undertow.server.HttpServerExchange;
+import io.undertow.util.HttpString;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+
+public class SessionValidationGetHandler extends SessionValidateHandler {
+
+    private transient final Logger log = LoggerFactory.getLogger(SessionValidationGetHandler.class);
+
+    private ServerConfig srvCfg;
+    private ViewConfig viewCfg;
+
+    public SessionValidationGetHandler(SessionManager mgr, MxisdConfig cfg) {
+        super(mgr);
+        this.srvCfg = cfg.getServer();
+        this.viewCfg = cfg.getView();
+    }
+
+    @Override
+    public void handleRequest(HttpServerExchange exchange) {
+        log.info("Handling GET request to validate session");
+
+        String sid = getQueryParameter(exchange, "sid");
+        String secret = getQueryParameter(exchange, "client_secret");
+        String token = getQueryParameter(exchange, "token");
+
+        ValidationResult r = handleRequest(sid, secret, token);
+        log.info("Session {} was validated", sid);
+        if (r.getNextUrl().isPresent()) {
+            String url = r.getNextUrl().get();
+            try {
+                url = new URL(url).toString();
+            } catch (MalformedURLException e) {
+                log.info("Session next URL {} is not a valid one, will prepend public URL {}", url, srvCfg.getPublicUrl());
+                url = srvCfg.getPublicUrl() + r.getNextUrl().get();
+            }
+            log.info("Session {} validation: next URL is present, redirecting to {}", sid, url);
+            exchange.setStatusCode(302);
+            exchange.getResponseHeaders().add(HttpString.tryFromString("Location"), url);
+        } else {
+            try {
+                String data = FileUtil.load(viewCfg.getSession().getOnTokenSubmit().getSuccess());
+                writeBodyAsUtf8(exchange, data);
+            } catch (IOException e) {
+                throw new RuntimeException(e);
+            }
+        }
+    }
+
+}

src/main/java/io/kamax/mxisd/http/undertow/handler/identity/v1/SessionValidationPostHandler.java

@@ -0,0 +1,80 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.http.undertow.handler.identity.v1;
+
+import com.google.gson.JsonObject;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.http.io.identity.SuccessStatusJson;
+import io.kamax.mxisd.session.SessionManager;
+import io.undertow.server.HttpServerExchange;
+import io.undertow.server.handlers.form.FormData;
+import io.undertow.server.handlers.form.FormParserFactory;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+public class SessionValidationPostHandler extends SessionValidateHandler {
+
+    private transient final Logger log = LoggerFactory.getLogger(SessionValidationPostHandler.class);
+
+    private FormParserFactory factory;
+
+    public SessionValidationPostHandler(SessionManager mgr) {
+        super(mgr);
+        factory = FormParserFactory.builder().build();
+    }
+
+
+    @Override
+    public void handleRequest(HttpServerExchange exchange) throws IOException {
+        log.info("Handling POST request to validate session");
+
+        String sid;
+        String secret;
+        String token;
+
+        String contentType = getContentType(exchange).orElseThrow(() -> new IllegalArgumentException("Content type header is not set"));
+        if (StringUtils.equals(contentType, "application/json")) { // FIXME use MIME parsing tools
+            log.info("Parsing as JSON data");
+
+            JsonObject body = parseJsonObject(exchange);
+            sid = GsonUtil.getStringOrThrow(body, "sid");
+            secret = GsonUtil.getStringOrThrow(body, "client_secret");
+            token = GsonUtil.getStringOrThrow(body, "token");
+        } else if (StringUtils.equals(contentType, "application/x-www-form-urlencoded")) { // FIXME use MIME parsing tools
+            log.info("Parsing as Form data");
+
+            FormData data = factory.createParser(exchange).parseBlocking();
+            sid = getOrThrow(data, "sid");
+            secret = getOrThrow(data, "client_secret");
+            token = getOrThrow(data, "token");
+        } else {
+            log.info("Unsupported Content type: {}", contentType);
+            throw new IllegalArgumentException("Unsupported Content type: " + contentType);
+        }
+
+        handleRequest(sid, secret, token);
+        respondJson(exchange, new SuccessStatusJson(true));
+    }
+
+}

src/main/java/io/kamax/mxisd/invitation/InvitationManager.java

@@ -30,7 +30,6 @@ import io.kamax.mxisd.config.InvitationConfig;
 import io.kamax.mxisd.config.MxisdConfig;
 import io.kamax.mxisd.config.ServerConfig;
 import io.kamax.mxisd.crypto.*;
-import io.kamax.mxisd.dns.FederationDnsOverwrite;
 import io.kamax.mxisd.exception.BadRequestException;
 import io.kamax.mxisd.exception.ConfigurationException;
 import io.kamax.mxisd.exception.MappingAlreadyExistsException;
@@ -38,6 +37,7 @@ import io.kamax.mxisd.exception.ObjectNotFoundException;
 import io.kamax.mxisd.lookup.SingleLookupReply;
 import io.kamax.mxisd.lookup.ThreePidMapping;
 import io.kamax.mxisd.lookup.strategy.LookupStrategy;
+import io.kamax.mxisd.matrix.HomeserverFederationResolver;
 import io.kamax.mxisd.notification.NotificationManager;
 import io.kamax.mxisd.profile.ProfileManager;
 import io.kamax.mxisd.storage.IStorage;
@@ -57,13 +57,10 @@ import org.apache.http.impl.client.HttpClients;
 import org.apache.http.ssl.SSLContextBuilder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.xbill.DNS.*;
 
 import javax.net.ssl.HostnameVerifier;
 import javax.net.ssl.SSLContext;
 import java.io.IOException;
-import java.net.MalformedURLException;
-import java.net.URL;
 import java.nio.charset.StandardCharsets;
 import java.time.DateTimeException;
 import java.time.Instant;
@@ -85,7 +82,7 @@ public class InvitationManager {
     private LookupStrategy lookupMgr;
     private KeyManager keyMgr;
     private SignatureManager signMgr;
-    private FederationDnsOverwrite dns;
+    private HomeserverFederationResolver resolver;
     private NotificationManager notifMgr;
     private ProfileManager profileMgr;
 
@@ -100,7 +97,7 @@ public class InvitationManager {
             LookupStrategy lookupMgr,
             KeyManager keyMgr,
             SignatureManager signMgr,
-            FederationDnsOverwrite dns,
+            HomeserverFederationResolver resolver,
             NotificationManager notifMgr,
             ProfileManager profileMgr
     ) {
@@ -110,7 +107,7 @@ public class InvitationManager {
         this.lookupMgr = lookupMgr;
         this.keyMgr = keyMgr;
         this.signMgr = signMgr;
-        this.dns = dns;
+        this.resolver = resolver;
         this.notifMgr = notifMgr;
         this.profileMgr = profileMgr;
 
@@ -172,12 +169,6 @@ public class InvitationManager {
 
             // Enabled by default
             cfg.getInvite().getExpiration().setEnabled(true);
-
-            // We'll resolve to our computed User ID
-            cfg.getInvite().getExpiration().setResolveTo(mxId);
-
-            // One calendar week (60min/1h * 24 = 1d * 7 = 1w)
-            cfg.getInvite().getExpiration().setAfter(60 * 24 * 7);
         }
 
         if (cfg.getInvite().getExpiration().isEnabled()) {
@@ -213,56 +204,6 @@ public class InvitationManager {
         return reply.getInvite().getSender().getId() + ":" + reply.getInvite().getRoomId() + ":" + reply.getInvite().getMedium() + ":" + reply.getInvite().getAddress();
     }
 
-    private String getSrvRecordName(String domain) {
-        return "_matrix._tcp." + domain;
-    }
-
-    // TODO use caching mechanism
-    // TODO export in matrix-java-sdk
-    private String findHomeserverForDomain(String domain) {
-        Optional<String> entryOpt = dns.findHost(domain);
-        if (entryOpt.isPresent()) {
-            String entry = entryOpt.get();
-            log.info("Found DNS overwrite for {} to {}", domain, entry);
-            try {
-                return new URL(entry).toString();
-            } catch (MalformedURLException e) {
-                log.warn("Skipping homeserver Federation DNS overwrite for {} - not a valid URL: {}", domain, entry);
-            }
-        }
-
-        log.debug("Performing SRV lookup for {}", domain);
-        String lookupDns = getSrvRecordName(domain);
-        log.info("Lookup name: {}", lookupDns);
-
-        try {
-            List<SRVRecord> srvRecords = new ArrayList<>();
-            Record[] rawRecords = new Lookup(lookupDns, Type.SRV).run();
-            if (rawRecords != null && rawRecords.length > 0) {
-                for (Record record : rawRecords) {
-                    if (Type.SRV == record.getType()) {
-                        srvRecords.add((SRVRecord) record);
-                    } else {
-                        log.info("Got non-SRV record: {}", record.toString());
-                    }
-                }
-
-                srvRecords.sort(Comparator.comparingInt(SRVRecord::getPriority));
-                for (SRVRecord record : srvRecords) {
-                    log.info("Found SRV record: {}", record.toString());
-                    return "https://" + record.getTarget().toString(true) + ":" + record.getPort();
-                }
-            } else {
-                log.info("No SRV record for {}", lookupDns);
-            }
-        } catch (TextParseException e) {
-            log.warn("Unable to perform DNS SRV query for {}: {}", lookupDns, e.getMessage());
-        }
-
-        log.info("Performing basic lookup using domain name {}", domain);
-        return "https://" + domain + ":8448";
-    }
-
     private Optional<SingleLookupReply> lookup3pid(String medium, String address) {
         if (!cfg.getResolution().isRecursive()) {
             log.warn("/!\\ /!\\ --- RECURSIVE INVITE RESOLUTION HAS BEEN DISABLED --- /!\\ /!\\");
@@ -419,7 +360,7 @@ public class InvitationManager {
                     continue;
                 }
 
-                log.info("Invite {} has expired at TS {} - Expiring and resolving to {}", targetTs, targetMxid);
+                log.info("Invite {} has expired at TS {} - Expiring and resolving to {}", reply.getId(), targetTs, targetMxid);
                 publishMapping(reply, targetMxid);
             } catch (NumberFormatException | DateTimeException e) {
                 log.warn("Invite {} has an invalid creation TS, setting to default value of {}", reply.getId(), defaultCreateTs);
@@ -481,7 +422,7 @@ public class InvitationManager {
         String address = reply.getInvite().getAddress();
         String domain = reply.getInvite().getSender().getDomain();
         log.info("Discovering HS for domain {}", domain);
-        String hsUrlOpt = findHomeserverForDomain(domain);
+        String hsUrlOpt = resolver.resolve(domain).toString();
 
         // TODO this is needed as this will block if called during authentication cycle due to synapse implementation
         new Thread(() -> { // FIXME need to make this retry-able and within a general background working pool

src/main/java/io/kamax/mxisd/matrix/HomeserverFederationResolver.java

@@ -0,0 +1,216 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.matrix;
+
+import com.google.gson.JsonObject;
+import io.kamax.matrix.json.GsonUtil;
+import io.kamax.matrix.json.InvalidJsonException;
+import io.kamax.mxisd.dns.FederationDnsOverwrite;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.util.EntityUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.xbill.DNS.*;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.*;
+
+public class HomeserverFederationResolver {
+
+    private static final Logger log = LoggerFactory.getLogger(HomeserverFederationResolver.class);
+
+    private FederationDnsOverwrite dns;
+    private CloseableHttpClient client;
+
+    public HomeserverFederationResolver(FederationDnsOverwrite dns, CloseableHttpClient client) {
+        this.dns = dns;
+        this.client = client;
+    }
+
+    private String getDefaultScheme() {
+        return "https";
+    }
+
+    private int getDefaultPort() {
+        return 8448;
+    }
+
+    private String getDnsSrvPrefix() {
+        return "_matrix._tcp.";
+    }
+
+    private String buildSrvRecordName(String domain) {
+        return getDnsSrvPrefix() + domain;
+    }
+
+    private Optional<URL> resolveOverwrite(String domain) {
+        Optional<String> entryOpt = dns.findHost(domain);
+        if (!entryOpt.isPresent()) {
+            log.info("No DNS overwrite for {}", domain);
+            return Optional.empty();
+        }
+
+        try {
+            return Optional.of(new URL(entryOpt.get()));
+        } catch (MalformedURLException e) {
+            log.warn("Skipping homeserver Federation DNS overwrite for {} - not a valid URL: {}", domain, entryOpt.get());
+            return Optional.empty();
+        }
+    }
+
+    private Optional<URL> resolveLiteral(String domain) {
+        if (domain.contains("[") && domain.contains("]")) {
+            // This is an IPv6
+            if (domain.contains("]:")) {
+                // With a custom port, we return as is
+                return Optional.of(build(domain));
+            } else {
+                return Optional.of(build(domain + ":" + getDefaultPort()));
+            }
+        }
+
+        if (domain.contains(":")) {
+            // This is a domain or IPv4 with an explicit port, we return as is
+            return Optional.of(build(domain));
+        }
+
+        // At this point, we do not account for the provided string to be an IPv4 without a port. We will therefore
+        // perform well-known lookup and SRV record. While this is not needed, we don't expect the SRV to return anything
+        // and the well-known shouldn't either, but it might, leading to a wrong destination potentially.
+        //
+        // We accept this risk as mxisd is not meant to be used without DNS domain as per FAQ. We also provide resolution
+        // override facilities. Therefore, we accept to not handle this case until we get report of such unwanted behaviour
+        // that still fix mxisd use case and can't be resolved via override.
+
+        return Optional.empty();
+    }
+
+    private Optional<URL> resolveWellKnown(String domain) {
+        log.debug("Performing Well-known lookup for {}", domain);
+        HttpGet wnReq = new HttpGet("https://" + domain + "/.well-known/matrix/server");
+        try (CloseableHttpResponse wnRes = client.execute(wnReq)) {
+            int status = wnRes.getStatusLine().getStatusCode();
+            if (status == 200) {
+                try {
+                    JsonObject body = GsonUtil.parseObj(EntityUtils.toString(wnRes.getEntity()));
+                    String server = GsonUtil.getStringOrNull(body, "m.server");
+                    if (StringUtils.isNotBlank(server)) {
+                        log.debug("Found well-known entry: {}", server);
+                        return Optional.of(build(server));
+                    }
+                } catch (InvalidJsonException e) {
+                    log.info("Could not parse well-known resource: {}", e.getMessage());
+                }
+            } else {
+                log.info("Well-known did not return status code 200 but {}, ignoring", status);
+            }
+
+            return Optional.empty();
+        } catch (IOException e) {
+            throw new RuntimeException("Error while trying to lookup well-known for " + domain, e);
+        }
+    }
+
+    private Optional<URL> resolveDnsSrv(String domain) {
+        log.debug("Performing SRV lookup for {}", domain);
+        String lookupDns = buildSrvRecordName(domain);
+        log.debug("Lookup name: {}", lookupDns);
+
+        try {
+            List<SRVRecord> srvRecords = new ArrayList<>();
+            Record[] rawRecords = new Lookup(lookupDns, Type.SRV).run();
+            if (Objects.isNull(rawRecords) || rawRecords.length == 0) {
+                log.debug("No SRV record for {}", domain);
+                return Optional.empty();
+            }
+
+            for (Record record : rawRecords) {
+                if (Type.SRV == record.getType()) {
+                    srvRecords.add((SRVRecord) record);
+                } else {
+                    log.debug("Ignoring non-SRV record: {}", record.toString());
+                }
+            }
+
+            if (srvRecords.size() < 1) {
+                log.warn("DNS SRV records were found for {} but none is usable", lookupDns);
+                return Optional.empty();
+            }
+
+            srvRecords.sort(Comparator.comparingInt(SRVRecord::getPriority));
+            SRVRecord record = srvRecords.get(0);
+            return Optional.of(build(record.getTarget().toString(true) + ":" + record.getPort()));
+        } catch (TextParseException e) {
+            log.warn("Unable to perform DNS SRV query for {}: {}", lookupDns, e.getMessage());
+        }
+
+        return Optional.empty();
+    }
+
+    public URL build(String authority) {
+        try {
+            return new URL(getDefaultScheme() + "://" + authority);
+        } catch (MalformedURLException e) {
+            throw new RuntimeException("Could not build URL for " + authority, e);
+        }
+    }
+
+    public URL resolve(String domain) {
+        Optional<URL> s1 = resolveOverwrite(domain);
+        if (s1.isPresent()) {
+            URL dest = s1.get();
+            log.info("Resolution of {} via DNS overwrite to {}", domain, dest);
+            return dest;
+        }
+
+        Optional<URL> s2 = resolveLiteral(domain);
+        if (s2.isPresent()) {
+            URL dest = s2.get();
+            log.info("Resolution of {} as IP literal or IP/hostname with explicit port to {}", domain, dest);
+            return dest;
+        }
+
+        Optional<URL> s3 = resolveWellKnown(domain);
+        if (s3.isPresent()) {
+            URL dest = s3.get();
+            log.info("Resolution of {} via well-known to {}", domain, dest);
+            return dest;
+        }
+        // The domain needs to be resolved
+
+        Optional<URL> s4 = resolveDnsSrv(domain);
+        if (s4.isPresent()) {
+            URL dest = s4.get();
+            log.info("Resolution of {} via DNS SRV record to {}", domain, dest);
+            return dest;
+        }
+
+        URL dest = build(domain + ":" + getDefaultPort());
+        log.info("Resolution of {} to {}", domain, dest);
+        return dest;
+    }
+
+}

src/main/java/io/kamax/mxisd/session/SessionManager.java

@@ -38,8 +38,8 @@ import io.kamax.mxisd.notification.NotificationManager;
 import io.kamax.mxisd.storage.IStorage;
 import io.kamax.mxisd.storage.dao.IThreePidSessionDao;
 import io.kamax.mxisd.threepid.session.ThreePidSession;
-import org.apache.commons.lang.RandomStringUtils;
+import org.apache.commons.lang3.RandomStringUtils;
-import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.http.impl.client.CloseableHttpClient;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -139,12 +139,13 @@ public class SessionManager {
     }
 
     public ValidationResult validate(String sid, String secret, String token) {
+        log.info("Validating session {}", sid);
         ThreePidSession session = getSession(sid, secret);
-        log.info("Attempting validation for session {} from {}", session.getId(), session.getServer());
+        log.info("Session {} is from {}", session.getId(), session.getServer());
 
         session.validate(token);
         storage.updateThreePidSession(session.getDao());
-        log.info("Session {} has been validated locally", session.getId());
+        log.info("Session {} has been validated", session.getId());
 
         ValidationResult r = new ValidationResult(session);
         session.getNextLink().ifPresent(r::setNextUrl);

src/main/java/io/kamax/mxisd/threepid/connector/email/BlackholeEmailConnector.java

@@ -0,0 +1,37 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.threepid.connector.email;
+
+public class BlackholeEmailConnector implements EmailConnector {
+
+    public static final String ID = "none";
+
+    @Override
+    public String getId() {
+        return ID;
+    }
+
+    @Override
+    public void send(String senderAddress, String senderName, String recipient, String content) {
+        //dev/null
+    }
+
+}

src/main/java/io/kamax/mxisd/threepid/connector/email/BuiltInEmailConnectorSupplier.java

@@ -33,6 +33,10 @@ public class BuiltInEmailConnectorSupplier implements EmailConnectorSupplier {
 
     @Override
     public Optional<EmailConnector> apply(EmailConfig cfg, Mxisd mxisd) {
+        if (StringUtils.equals(BlackholeEmailConnector.ID, cfg.getConnector())) {
+            return Optional.of(new BlackholeEmailConnector());
+        }
+
         if (StringUtils.equals(EmailSmtpConnector.ID, cfg.getConnector())) {
             EmailSmtpConfig smtpCfg = GsonUtil.get().fromJson(cfg.getConnectors().getOrDefault(EmailSmtpConnector.ID, new JsonObject()), EmailSmtpConfig.class);
             return Optional.of(new EmailSmtpConnector(smtpCfg));

src/main/java/io/kamax/mxisd/threepid/connector/email/EmailSmtpConnector.java

@@ -31,14 +31,17 @@ import org.apache.commons.lang3.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import javax.mail.Header;
 import javax.mail.Message;
 import javax.mail.MessagingException;
 import javax.mail.Session;
 import javax.mail.internet.InternetAddress;
 import javax.mail.internet.MimeMessage;
+import javax.mail.internet.MimeUtility;
 import java.io.UnsupportedEncodingException;
 import java.nio.charset.StandardCharsets;
 import java.util.Date;
+import java.util.Enumeration;
 import java.util.Properties;
 
 public class EmailSmtpConnector implements EmailConnector {
@@ -66,6 +69,10 @@ public class EmailSmtpConnector implements EmailConnector {
             sCfg.setProperty("mail.smtp.auth", "true");
         }
 
+        if (cfg.getTls() == 3) {
+            sCfg.setProperty("mail.smtp.ssl.enable", "true");
+        }
+
         session = Session.getInstance(sCfg);
     }
 
@@ -93,7 +100,16 @@ public class EmailSmtpConnector implements EmailConnector {
         try {
             InternetAddress sender = new InternetAddress(senderAddress, senderName);
             MimeMessage msg = new MimeMessage(session, IOUtils.toInputStream(content, StandardCharsets.UTF_8));
-            msg.setHeader("X-Mailer", Mxisd.Agent);
+
+            // We must encode our headers ourselves as we have no guarantee that they were in the provided data.
+            // This is required to support UTF-8 characters from user display names or room names in the subject header per example
+            Enumeration<Header> headers = msg.getAllHeaders();
+            while (headers.hasMoreElements()) {
+                Header header = headers.nextElement();
+                msg.setHeader(header.getName(), MimeUtility.encodeText(header.getValue()));
+            }
+
+            msg.setHeader("X-Mailer", MimeUtility.encodeText(Mxisd.Agent));
             msg.setSentDate(new Date());
             msg.setFrom(sender);
             msg.setRecipients(Message.RecipientType.TO, recipient);
@@ -101,8 +117,11 @@ public class EmailSmtpConnector implements EmailConnector {
 
             log.info("Sending invite to {} via SMTP using {}:{}", recipient, cfg.getHost(), cfg.getPort());
             SMTPTransport transport = (SMTPTransport) session.getTransport("smtp");
-            transport.setStartTLS(cfg.getTls() > 0);
+
-            transport.setRequireStartTLS(cfg.getTls() > 1);
+            if (cfg.getTls() < 3) {
+                transport.setStartTLS(cfg.getTls() > 0);
+                transport.setRequireStartTLS(cfg.getTls() > 1);
+            }
 
             log.info("Connecting to {}:{}", cfg.getHost(), cfg.getPort());
             if (StringUtils.isAllEmpty(cfg.getLogin(), cfg.getPassword())) {

src/main/java/io/kamax/mxisd/threepid/connector/phone/BlackholePhoneConnector.java

@@ -24,14 +24,14 @@ public class BlackholePhoneConnector implements PhoneConnector {
 
     public static final String ID = "none";
 
-    @Override
-    public void send(String recipient, String content) {
-        //dev/null
-    }
-
     @Override
     public String getId() {
         return ID;
     }
 
+    @Override
+    public void send(String recipient, String content) {
+        //dev/null
+    }
+
 }

src/main/java/io/kamax/mxisd/threepid/connector/phone/BuiltInPhoneConnectorSupplier.java

@@ -33,15 +33,15 @@ public class BuiltInPhoneConnectorSupplier implements PhoneConnectorSupplier {
 
     @Override
     public Optional<PhoneConnector> apply(PhoneConfig cfg, Mxisd mxisd) {
+        if (StringUtils.equals(BlackholePhoneConnector.ID, cfg.getConnector())) {
+            return Optional.of(new BlackholePhoneConnector());
+        }
+
         if (StringUtils.equals(PhoneSmsTwilioConnector.ID, cfg.getConnector())) {
             PhoneTwilioConfig cCfg = GsonUtil.get().fromJson(cfg.getConnectors().getOrDefault(PhoneSmsTwilioConnector.ID, new JsonObject()), PhoneTwilioConfig.class);
             return Optional.of(new PhoneSmsTwilioConnector(cCfg));
         }
 
-        if (StringUtils.equals(BlackholePhoneConnector.ID, cfg.getConnector())) {
-            return Optional.of(new BlackholePhoneConnector());
-        }
-
         return Optional.empty();
     }
 

src/main/java/io/kamax/mxisd/threepid/generator/GenericTemplateNotificationGenerator.java

@@ -68,6 +68,7 @@ public abstract class GenericTemplateNotificationGenerator extends PlaceholderNo
     @Override
     public String getForReply(IThreePidInviteReply invite) {
         log.info("Generating notification content for 3PID invite");
+        invite.getInvite().getProperties().putAll(cfg.getPlaceholder());
         return populateForReply(invite, getTemplateContent(cfg.getInvite()));
     }
 

src/main/java/io/kamax/mxisd/threepid/generator/PlaceholderNotificationGenerator.java

@@ -35,6 +35,8 @@ import static io.kamax.mxisd.http.io.identity.StoreInviteRequest.Keys.SenderDisp
 
 public abstract class PlaceholderNotificationGenerator {
 
+    public static final String RegisterUrl = "REGISTER_URL";
+
     private MatrixConfig mxCfg;
     private ServerConfig srvCfg;
 
@@ -76,8 +78,10 @@ public abstract class PlaceholderNotificationGenerator {
         String senderNameOrId = StringUtils.defaultIfBlank(senderName, invite.getInvite().getSender().getId());
         String roomName = invite.getInvite().getProperties().getOrDefault(RoomName, "");
         String roomNameOrId = StringUtils.defaultIfBlank(roomName, invite.getInvite().getRoomId());
+        String registerUrl = StringUtils.defaultIfBlank(invite.getInvite().getProperties().get(RegisterUrl), "https://" + mxCfg.getDomain());
 
         return populateForCommon(tpid, input)
+                .replace("%" + RegisterUrl + "%", registerUrl)
                 .replace("%SENDER_ID%", invite.getInvite().getSender().getId())
                 .replace("%SENDER_NAME%", senderName)
                 .replace("%SENDER_NAME_OR_ID%", senderNameOrId)
@@ -102,10 +106,6 @@ public abstract class PlaceholderNotificationGenerator {
                 .replace("%NEXT_URL%", validationLink);
     }
 
-    protected String populateForRemoteValidation(IThreePidSession session, String input) {
-        return populateForValidation(session, input);
-    }
-
     protected String populateForFraudulentUndind(ThreePid tpid, String input) {
         return populateForCommon(tpid, input);
     }

src/main/resources/simplelogger.properties

@@ -1,2 +1,4 @@
 org.slf4j.simpleLogger.logFile=System.out
 org.slf4j.simpleLogger.log.org.xnio=warn
+org.slf4j.simpleLogger.log.com.j256.ormlite.table.TableUtils=warn
+org.slf4j.simpleLogger.log.com.mchange.v2.log.MLog=warn

src/main/resources/threepids/email/invite-template.eml

@@ -10,7 +10,7 @@ Content-Disposition: inline
 Hi,
 
 %SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on
-Matrix. To join the conversation, register an account on https://%DOMAIN%
+Matrix. To join the conversation, register an account on %REGISTER_URL%
 
 You can also register an account on a public server and get in touch with them.
 
@@ -69,7 +69,7 @@ pre, code {
 <p>Hi,</p>
 
 <p>%SENDER_NAME_OR_ID% has invited you into a room [%ROOM_NAME_OR_ID%] on
-Matrix. To join the conversation, register an account on <a href="https://%DOMAIN%">%DOMAIN%</a>.</p>
+Matrix. To join the conversation, register an account on <a href="%REGISTER_URL%">%DOMAIN%</a>.</p>
 
 <pYou can also register an account on a public server and get in touch with them.</p>
 

src/test/java/io/kamax/mxisd/test/matrix/HomeserverFederationResolverTest.java

@@ -0,0 +1,64 @@
+/*
+ * mxisd - Matrix Identity Server Daemon
+ * Copyright (C) 2019 Kamax Sarl
+ *
+ * https://www.kamax.io/
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+package io.kamax.mxisd.test.matrix;
+
+import io.kamax.mxisd.Mxisd;
+import io.kamax.mxisd.config.MxisdConfig;
+import io.kamax.mxisd.dns.FederationDnsOverwrite;
+import io.kamax.mxisd.matrix.HomeserverFederationResolver;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.net.URL;
+
+import static org.junit.Assert.assertEquals;
+
+public class HomeserverFederationResolverTest {
+
+    private static HomeserverFederationResolver resolver;
+
+    @BeforeClass
+    public static void beforeClass() {
+        CloseableHttpClient client = HttpClients.custom()
+                .setUserAgent(Mxisd.Agent)
+                .setMaxConnPerRoute(Integer.MAX_VALUE)
+                .setMaxConnTotal(Integer.MAX_VALUE)
+                .build();
+
+        FederationDnsOverwrite fedDns = new FederationDnsOverwrite(new MxisdConfig().getDns().getOverwrite());
+        resolver = new HomeserverFederationResolver(fedDns, client);
+    }
+
+    @Test
+    public void hostnameWithoutPort() {
+        URL url = resolver.resolve("example.org");
+        assertEquals("https://example.org:8448", url.toString());
+    }
+
+    @Test
+    public void hostnameWithPort() {
+        URL url = resolver.resolve("example.org:443");
+        assertEquals("https://example.org:443", url.toString());
+    }
+
+}

src/test/java/io/kamax/mxisd/test/notification/EmailNotificationTest.java

@@ -32,7 +32,10 @@ import io.kamax.mxisd.config.MxisdConfig;
 import io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig;
 import io.kamax.mxisd.config.threepid.medium.EmailConfig;
 import io.kamax.mxisd.invitation.MatrixIdInvite;
+import io.kamax.mxisd.invitation.ThreePidInvite;
+import io.kamax.mxisd.invitation.ThreePidInviteReply;
 import io.kamax.mxisd.threepid.connector.email.EmailSmtpConnector;
+import io.kamax.mxisd.threepid.generator.PlaceholderNotificationGenerator;
 import io.kamax.mxisd.threepid.session.ThreePidSession;
 import org.apache.commons.lang.RandomStringUtils;
 import org.junit.After;
@@ -45,6 +48,7 @@ import javax.mail.internet.MimeBodyPart;
 import javax.mail.internet.MimeMessage;
 import javax.mail.internet.MimeMultipart;
 import java.io.IOException;
+import java.util.ArrayList;
 import java.util.Collections;
 
 import static junit.framework.TestCase.assertEquals;
@@ -56,7 +60,8 @@ public class EmailNotificationTest {
     private final String user = "mxisd";
     private final String notifiee = "john";
     private final String sender = user + "@" + domain;
-    private final String senderEmail = "\"Mxisd Server (Unit Test)\" <" + sender + ">";
+    private final String senderName = "\"Mxisd Server あ (Unit Test)\" <" + sender + ">";
+    private final String senderNameEncoded = "=?UTF-8?Q?=22Mxisd_Server_=E3=81=82_=28Unit_T?= =?UTF-8?Q?est=29=22_=3Cmxisd=40localhost=3E?= <mxisd@localhost>";
     private final String target = notifiee + "@" + domain;
 
     private Mxisd m;
@@ -72,7 +77,7 @@ public class EmailNotificationTest {
         EmailConfig eCfg = new EmailConfig();
         eCfg.setConnector(EmailSmtpConnector.ID);
         eCfg.getIdentity().setFrom(sender);
-        eCfg.getIdentity().setName("Mxisd Server (Unit Test)");
+        eCfg.getIdentity().setName(senderName);
         eCfg.getConnectors().put(EmailSmtpConnector.ID, GsonUtil.makeObj(smtpCfg));
 
         MxisdConfig cfg = new MxisdConfig();
@@ -114,10 +119,33 @@ public class EmailNotificationTest {
         assertEquals(1, gm.getReceivedMessages().length);
         MimeMessage msg = gm.getReceivedMessages()[0];
         assertEquals(1, msg.getFrom().length);
-        assertEquals(senderEmail, msg.getFrom()[0].toString());
+        assertEquals(senderNameEncoded, msg.getFrom()[0].toString());
         assertEquals(1, msg.getRecipients(Message.RecipientType.TO).length);
     }
 
+    @Test
+    public void forThreepidInvite() throws MessagingException, IOException {
+        String registerUrl = "https://" + RandomStringUtils.randomAlphanumeric(20) + ".example.org/register";
+        gm.setUser(user, user);
+
+        _MatrixID sender = MatrixID.asAcceptable(user, domain);
+        ThreePidInvite inv = new ThreePidInvite(sender, ThreePidMedium.Email.getId(), target, "!rid:" + domain);
+        inv.getProperties().put(PlaceholderNotificationGenerator.RegisterUrl, registerUrl);
+        m.getNotif().sendForReply(new ThreePidInviteReply("a", inv, "b", "c", new ArrayList<>()));
+
+        assertEquals(1, gm.getReceivedMessages().length);
+        MimeMessage msg = gm.getReceivedMessages()[0];
+        assertEquals(1, msg.getFrom().length);
+        assertEquals(senderNameEncoded, msg.getFrom()[0].toString());
+        assertEquals(1, msg.getRecipients(Message.RecipientType.TO).length);
+
+        // We just check on the text/plain one. HTML is multipart and it's difficult so we skip
+        MimeMultipart content = (MimeMultipart) msg.getContent();
+        MimeBodyPart mbp = (MimeBodyPart) content.getBodyPart(0);
+        String mbpContent = mbp.getContent().toString();
+        assertTrue(mbpContent.contains(registerUrl));
+    }
+
     @Test
     public void forValidation() throws MessagingException, IOException {
         gm.setUser(user, user);
@@ -138,7 +166,7 @@ public class EmailNotificationTest {
         assertEquals(1, gm.getReceivedMessages().length);
         MimeMessage msg = gm.getReceivedMessages()[0];
         assertEquals(1, msg.getFrom().length);
-        assertEquals(senderEmail, msg.getFrom()[0].toString());
+        assertEquals(senderNameEncoded, msg.getFrom()[0].toString());
         assertEquals(1, msg.getRecipients(Message.RecipientType.TO).length);
 
         // We just check on the text/plain one. HTML is multipart and it's difficult so we skip