Merge branch 'to-v1.4'

Fix typo in example configuration

README.md

@@ -31,7 +31,7 @@ users. 3PIDs can be anything that uniquely and globally identify a user, like:
 - Twitter handle
 - Facebook ID
 
-If you are unfamiliar with the Identity vocabulary and concepts in Matrix, **please read this [introduction](docs/concepts.md)**.
+If you are unfamiliar with the Identity vocabulary and concepts in Matrix, **please read this [introduction](docs/concepts.md)**.  
 
 # Features
 [Identity](docs/features/identity.md): As a [regular Matrix Identity service](https://matrix.org/docs/spec/identity_service/r0.1.0.html#general-principles):
@@ -53,6 +53,7 @@ As an enhanced Identity service:
   - Central Matrix Identity servers
 - [Session Control](docs/threepids/session/session.md): Extensive control of where 3PIDs are transmitted so they are not
   leaked publicly by users
+- [Registration control](docs/features/registration.md): Control and restrict user registration based on 3PID patterns or criterias, like a pending invite
 - [Authentication](docs/features/authentication.md): Use your Identity stores to perform authentication in [synapse](https://github.com/matrix-org/synapse)
   via the [REST password provider](https://github.com/kamax-io/matrix-synapse-rest-auth)
 - [Directory search](docs/features/directory.md) which allows you to search for users within your organisation,
@@ -80,8 +81,6 @@ A basic troubleshooting guide is available [here](docs/troubleshooting.md).
 ## Community
 Over Matrix: [#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io) ([Preview](https://view.matrix.org/room/!NPRUEisLjcaMtHIzDr:kamax.io/))
 
-For more high-level discussion about the Identity Server architecture/API, go to  [#matrix-identity:kamax.io](https://matrix.to/#/#matrix-identity:kamax.io)
-
 ## Commercial
 If you would prefer professional support/custom development for mxisd and/or for Matrix in general, including other open
 source technologies/products:

docs/features/experimental/application-service.md

@@ -1,25 +1,106 @@
-# Integration as an Application Service
+# Application Service
 **WARNING:** These features are currently highly experimental. They can be removed or modified without notice.  
-All the features requires a Homeserver capable of connecting Application Services.
+All the features requires a Homeserver capable of connecting [Application Services](https://matrix.org/docs/spec/application_service/r0.1.0.html).
 
-## Email notification for Room invites by Matrix ID
+The following capabilities are provided in this feature:
+- [Admin commands](#admin-commands)
+- [Email Notification about room invites by Matrix IDs](#email-notification-about-room-invites-by-matrix-ids)
+- [Auto-reject of expired 3PID invites](#auto-reject-of-expired-3pid-invites)
+
+## Setup
+> **NOTE:** Make sure you are familiar with [configuration format and rules](../../configure.md).
+
+Integration as an Application service is a three steps process:
+1. Create the baseline mxisd configuration to allow integration.
+2. Integrate with the homeserver.
+3. Configure the specific capabilities, if applicable.
+
+### Configuration
+#### Variables
+Under the `appsvc` namespace:
+
+| Key                   | Type    | Required | Default | Purpose                                                        |
+|-----------------------|---------|----------|---------|----------------------------------------------------------------|
+| `enabled`             | boolean | No       | `true`  | Globally enable/disable the feature                            |
+| `user.main`           | string  | No       | `mxisd` | Localpart for the main appservice user                         |
+| `endpoint.toHS.url`   | string  | Yes      | *None*  | Base URL to the Homeserver                                     |
+| `endpoint.toHS.token` | string  | Yes      | *None*  | Token to use when sending requests to the Homeserver           |
+| `endpoint.toAS.url`   | string  | Yes      | *None*  | Base URL to mxisd from the Homeserver                          |
+| `endpoint.toAS.token` | string  | Yes      | *None*  | Token for the Homeserver to use when sending requests to mxisd |
+
+#### Example
+```yaml
+appsvc:
+  endpoint:
+    toHS:
+      url: 'http://localhost:8008'
+      token: 'ExampleTokenToHS-ChangeMe!'
+    toAS:
+      url: 'http://localhost:8090'
+      token: 'ExampleTokenToAS-ChangeMe!'
+```
+### Integration
+#### Synapse
+Under the `appsvc.registration.synapse` namespace:
+
+| Key    | Type   | Required | Default            | Purpose                                                                  |
+|--------|--------|----------|--------------------|--------------------------------------------------------------------------|
+| `id`   | string | No       | `appservice-mxisd` | The unique, user-defined ID of this application service. See spec.       |
+| `file` | string | Yes      | *None*             | If defined, the synapse registration file that should be created/updated |
+
+##### Example 
+```yaml
+appsvc:
+  registration:
+    synapse:
+      file: '/etc/matrix-synapse/mxisd-appservice-registration.yaml'
+```
+
+Edit your `homeserver.yaml` and add a new entry to the appservice config file, which should look something like this:
+```yaml
+app_service_config_files:
+  - '/etc/matrix-synapse/mxisd-appservice-registration.yaml'
+  - ...
+```
+
+Restart synapse when done to register mxisd.
+
+#### Others
+See your Homeserver documentation on how to integrate.
+
+## Capabilities
+### Admin commands
+#### Setup
+Min config:
+```yaml
+appsvc:
+  feature:
+    admin:
+      allowedRoles:
+        - '+aMatrixCommunity:example.org'
+        - 'SomeLdapGroup'
+        - 'AnyOtherArbitraryRoleFromIdentityStores'
+```
+
+#### Use
+The following steps assume:
+- `matrix.domain` set to `example.org`
+- `appsvc.user.main` set to `mxisd` or not set
+
+1. Invite `@mxisd:example.org` to a new direct chat
+2. Type `!help` to get all available commands
+
+### Email Notification about room invites by Matrix IDs
 This feature allows for users found in Identity stores to be instantly notified about Room Invites, regardless if their
 account was already provisioned on the Homeserver.
 
-### Requirements
+#### Requirements
 - [Identity store(s)](../../stores/README.md) supporting the Profile feature
 - At least one email entry in the identity store for each user that could be invited.
 
-### Configuration
+#### Configuration
 In your mxisd config file:
 ```yaml
-matrix:
-  listener:
-    url:  '<URL TO THE CS API OF THE HOMESERVER>'
-    localpart: 'appservice-mxisd'
-    token:
-      hs: 'HS_TOKEN_CHANGE_ME'
-
 synapseSql:
   enabled: false ## Do not use this line if Synapse is used as an Identity Store
   type: '<DB TYPE>'
@@ -33,40 +114,8 @@ If you do not configure it, some placeholders will not be available in the notif
 You can also change the default template of the notification using the `generic.matrixId` template option.  
 See [the Template generator documentation](../../threepids/notification/template-generator.md) for more info.
 
-### Homeserver integration
-#### Synapse
-Create a new appservice registration file. Futher config will assume it is in `/etc/matrix-synapse/appservice-mxisd.yaml`
-```yaml
-id: "appservice-mxisd"
-url: "http://127.0.0.1:8090"
-as_token: "AS_TOKEN_CHANGE_ME"
-hs_token: "HS_TOKEN_CHANGE_ME"
-sender_localpart: "appservice-mxisd"
-namespaces:
-  users:
-    - regex: "@*"
-      exclusive: false
-  aliases: []
-  rooms: []
-```
-`id`: An arbitrary unique string to identify the AS.  
-`url`: mxisd to reach mxisd. This ideally should be HTTP and not going through any reverse proxy.  
-`as_token`: Arbitrary value used by mxisd when talking to the HS. Not currently used.  
-`hs_token`: Arbitrary value used by synapse when talking to mxisd. Must match `token.hs` in mxisd config.
-`sender_localpart`: Username for the mxisd itself on the HS. Default configuration should be kept.  
-`namespaces`: To be kept as is.  
-
-Edit your `homeserver.yaml` and add a new entry to the appservice config file, which should look something like this:
-```yaml
-app_service_config_files:
-  - '/etc/matrix-synapse/appservice-mxisd.yaml'
-  - ...
-```
-
-Restart synapse when done to register mxisd.
-
-#### Others
-See your Homeserver documentation on how to integrate.
-
-### Test
+#### Test
 Invite a user which is part of your domain while an appropriate Identity store is used.
+
+### Auto-reject of expired 3PID invites
+*TBC*

docs/features/identity.md

@@ -1,6 +1,13 @@
 # Identity
 Implementation of the [Identity Service API r0.1.0](https://matrix.org/docs/spec/identity_service/r0.1.0.html).
 
+- [Lookups](#lookups)
+- [Invitations](#invitations)
+  - [Expiration](#expiration)
+  - [Policies](#policies)
+  - [Resolution](#resolution)
+- [3PIDs Management](#3pids-management)
+
 ## Lookups
 If you would like to use the central matrix.org Identity server to ensure maximum discovery at the cost of potentially
 leaking all your contacts information, add the following to your configuration:
@@ -12,8 +19,78 @@ forward:
 **NOTE:** You should carefully consider enabling this option, which is discouraged.  
 For more info, see the [relevant issue](https://github.com/kamax-matrix/mxisd/issues/76).
 
-## Room Invitations
-Resolution can be customized using the following configuration:
+## Invitations
+### Expiration
+#### Overview
+Matrix does not provide a mean to remove/cancel pending 3PID invitations with the APIs. The current reference
+implementations also do not provide any mean to do so. This leads to 3PID invites forever stuck in rooms.
+
+To provide this functionality, mxisd uses a workaround: resolve the invite to a dedicated User ID, which can be
+controlled by mxisd or a bot/service that will then reject the invite.
+
+If this dedicated User ID is to be controlled by mxisd, the [Application Service](experimental/application-service.md)
+feature must be configured and integrated with your Homeserver, as well as the *Auto-reject 3PID invite capability*.
+
+#### Configuration
+```yaml
+invite:
+  expiration:
+    enabled: true/false
+    after: 5
+    resolveTo: '@john.doe:example.org'
+```
+`enabled`
+- Purpose: Enable or disable the invite expiration feature.
+- Default: `true`
+
+`after` 
+- Purpose: Amount of minutes before an invitation expires.
+- Default: `10080` (7 days)
+
+`resolveTo`
+- Purpose: Matrix User ID to resolve the expired invitations to.
+- Default: Computed from `appsvc.user.inviteExpired` and `matrix.domain`
+
+### Policies
+3PID invite policies are the companion feature of [Registration](registration.md). While the Registration feature acts on
+requirements for the invitee/register, this feature acts on requirement for the one(s) performing 3PID invites, ensuring
+a coherent system.
+
+It relies on only allowing people with specific [Roles](profile.md) to perform 3PID invites. This would typically allow
+a tight-control on a server setup with is "invite-only" or semi-open (relying on trusted people to invite new members).
+
+It's a middle ground between a closed server, where every user must be created or already exists in an Identity store,
+and an open server, where anyone can register.
+ 
+#### Integration
+Because Identity Servers do not control 3PID invites as per Matrix spec, mxisd needs to intercept a set of Homeserver
+endpoints to apply the policies.
+
+##### Reverse Proxy
+###### nginx
+**IMPORTANT**: Must be placed before your global `/_matrix` entry:
+```nginx
+location ~* ^/_matrix/client/r0/rooms/([^/]+)/invite$ {
+    proxy_pass		    http://127.0.0.1:8090;
+    proxy_set_header	Host $host;
+    proxy_set_header	X-Forwarded-For $remote_addr;
+}
+```
+
+#### Configuration
+The only policy currently available is to restrict 3PID invite to users having a specific (set of) role(s), like so:
+
+```yaml
+invite:
+  policy:
+    ifSender:
+      hasRole:
+        - '<THIS_ROLE>'
+        - '<OR_THIS_ROLE>'
+```
+
+### Resolution
+Resolution of 3PID invitations can be customized using the following configuration:
 
 `invite.resolution.recursive`  
 - Default value: `true`  
@@ -26,5 +103,5 @@ Resolution can be customized using the following configuration:
 - Default value: `1`  
 - Description: How often, in minutes, mxisd should try to resolve pending invites.
 
-## 3PID addition to user profile
+## 3PIDs Management
 See the [3PID session documents](../threepids/session)

docs/features/registration.md

@@ -0,0 +1,111 @@
+# Registration
+- [Overview](#overview)
+- [Integration](#integration)
+  - [Reverse Proxy](#reverse-proxy)
+    - [nginx](#nginx)
+    - [Apache](#apache)
+  - [Homeserver](#homeserver)
+    - [synapse](#synapse)
+- [Configuration](#configuration)
+  - [Example](#example)
+- [Usage](#usage)
+
+## Overview
+**NOTE**: This feature is beta: it is considered stable enough for production but is incomplete and may contain bugs.
+
+Registration is an enhanced feature of mxisd to control registrations involving 3PIDs on a Homeserver based on policies:
+- Match pending 3PID invites on the server
+- Match 3PID pattern, like a specific set of domains for emails
+- In futher releases, use 3PIDs found in Identity stores
+
+It aims to help open or invite-only registration servers control what is possible to do and ensure only approved people
+can register on a given server in a implementation-agnostic manner.
+
+**IMPORTANT:** This feature does not control registration in general. It only acts on endpoints related to 3PIDs during
+the registration process.  
+As such, it relies on the homeserver to require 3PIDs with the registration flows.
+
+This feature is not part of the Matrix Identity Server spec.
+
+## Integration
+mxisd needs to be integrated at several levels for this feature to work:
+- Reverse proxy: intercept the 3PID register endpoints and act on them
+- Homeserver: require 3PID to be part of the registration data
+
+Later version(s) of this feature may directly control registration itself to create a coherent experience
+### Reverse Proxy
+#### nginx
+```nginx
+location ^/_matrix/client/r0/register/[^/]/?$ {
+	proxy_pass		http://127.0.0.1:8090;
+	proxy_set_header	Host $host;
+	proxy_set_header	X-Forwarded-For $remote_addr;
+}
+```
+
+#### apache
+> TBC
+
+### Homeserver
+#### Synapse
+```yaml
+enable_registration: true
+registrations_require_3pid:
+  - email
+```
+
+## Configuration
+See the [Configuration](../configuration.md) introduction doc on how to read the configuration keys.  
+An example of working configuration is avaiable at the end of this section.
+### Enable/Disable
+`register.allowed`, taking a boolean, can be used to enable/disable registration if the attempt is not 3PID-based.  
+`false` is the default value to prevent open registration, as you must allow it on the homeserver side.
+
+### For invites
+`register.invite`, taking a boolean, controls if registration can be made using a 3PID which matches a pending 3PID invite.  
+`true` is the default value.
+
+### 3PID-specific
+At this time, only `email` is supported with 3PID specific configuration with this feature.
+
+#### Email
+**Base key**: `register.threepid.email`
+
+##### Domain whitelist/blacklist
+If you would like to control which domains are allowed to be used when registrating with an email, the following sub-keys
+are available:
+- `domain.whitelist`
+- `domain.blacklist`
+
+The value format is an hybrid between glob patterns and postfix configuration files with the following syntax:
+- `*<domain>` will match the domain and any sub-domain(s)
+- `.<domain>` will only match sub-domain(s)
+- `<domain>` will only match the exact domain
+
+The following table illustrates pattern and maching status against example values:
+
+| Config value   | Matches `example.org` | Matches `sub.example.org` |
+|--------------- |-----------------------|---------------------------|
+| `*example.org` | Yes                   | Yes                       |
+| `.example.org` | No                    | Yes                       |
+| `example.org`  | Yes                   | No                        |
+
+### Example
+For the following example configuration:
+```yaml
+register:
+  policy:
+    threepid:
+      email:
+        domain:
+          whitelist:
+            - '*example.org'
+            - '.example.net'
+            - 'example.com'
+```
+- Users can register using 3PIDs of pending invites, being allowed by default.
+- Users can register using an email from `example.org` and any sub-domain, only sub-domains of `example.net` and `example.com` but not its sub-domains.
+- Otherwise, user registration will be denied.
+
+## Usage
+Nothing special is needed. Register using a regular Matrix client.

docs/stores/synapse.md

@@ -1,5 +1,6 @@
 # Synapse Identity Store
-Synapse's Database itself can be used as an Identity store.
+Synapse's Database itself can be used as an Identity store. This identity store is a regular SQL store with
+built-in default queries that matches Synapse DB.
 
 ## Features
 |                       Name                      | Supported |
@@ -9,7 +10,8 @@ Synapse's Database itself can be used as an Identity store.
 | [Identity](../features/identity.md)             | Yes       |
 | [Profile](../features/profile.md)               | Yes       |
 
-Authentication is done by Synapse itself.
+- Authentication is done by Synapse itself.
+- Roles are mapped to communities. The Role name/ID uses the community ID in the form `+id:domain.tld`
 
 ## Configuration
 ### Basic

docs/threepids/notification/template-generator.md

@@ -20,7 +20,7 @@ threepid:
           session:
             validation: '/path/to/validate-template.eml'
             unbind:
-              frandulent: '/path/to/unbind-fraudulent-template.eml'
+              fraudulent: '/path/to/unbind-fraudulent-template.eml'
           generic:
             matrixId: '/path/to/mxid-invite-template.eml'
 ```

src/debian/control

@@ -3,5 +3,7 @@ Maintainer: Kamax.io <foss@kamax.io>
 Homepage: https://github.com/kamax-matrix/mxisd
 Description: Federated Matrix Identity Server
 Architecture: all
+Section: net
+Priority: optional
 Depends: openjdk-8-jre | openjdk-8-jre-headless | openjdk-8-jdk | openjdk-8-jdk-headless
 Version: 0

src/main/java/io/kamax/mxisd/config/InvitationConfig.java

@@ -34,7 +34,7 @@ public class InvitationConfig {
     public static class Expiration {
 
         private Boolean enabled;
-        private long after;
+        private long after = 60 * 24 * 7; // One calendar week (60min/1h * 24 = 1d * 7 = 1w)
         private String resolveTo;
 
         public Boolean isEnabled() {

src/main/java/io/kamax/mxisd/crypto/GenericKeyIdentifier.java

@@ -36,7 +36,7 @@ public class GenericKeyIdentifier implements KeyIdentifier {
 
     public GenericKeyIdentifier(KeyType type, String algo, String serial) {
         if (StringUtils.isAnyBlank(algo, serial)) {
-            throw new IllegalArgumentException("Aglorith and/or Serial cannot be blank");
+            throw new IllegalArgumentException("Algorithm and/or Serial cannot be blank");
         }
 
         this.type = Objects.requireNonNull(type);

src/main/java/io/kamax/mxisd/invitation/InvitationManager.java

@@ -172,12 +172,6 @@ public class InvitationManager {
 
             // Enabled by default
             cfg.getInvite().getExpiration().setEnabled(true);
-
-            // We'll resolve to our computed User ID
-            cfg.getInvite().getExpiration().setResolveTo(mxId);
-
-            // One calendar week (60min/1h * 24 = 1d * 7 = 1w)
-            cfg.getInvite().getExpiration().setAfter(60 * 24 * 7);
         }
 
         if (cfg.getInvite().getExpiration().isEnabled()) {