Improve troubleshooting doc/flows

- Use better wording for unknown server error
- Add basic troubleshooting doc

README.md

@@ -74,6 +74,9 @@ Also, check [our FAQ entry](docs/faq.md#what-kind-of-setup-is-mxisd-really-desig
 See the [dedicated document](docs/getting-started.md)
 
 # Support
+## Troubleshooting
+A basic troubleshooting guide is available [here](docs/troubleshooting.md).
+
 ## Community
 Over Matrix: [#mxisd:kamax.io](https://matrix.to/#/#mxisd:kamax.io) ([Preview](https://view.matrix.org/room/!NPRUEisLjcaMtHIzDr:kamax.io/))
 

build.gradle

@@ -190,13 +190,13 @@ task debBuild(dependsOn: shadowJar) {
         ant.replaceregexp( // FIXME adapt to new config format
                 file: "${debBuildConfPath}/${debConfFileName}",
                 match: "key:\\R  path:(.*)",
-                replace: "key:\n  path: '${debDataPath}/signing.key'"
+                replace: "key:\n  path: '${debDataPath}/keys'"
         )
 
         ant.replaceregexp( // FIXME adapt to new config format
                 file: "${debBuildConfPath}/${debConfFileName}",
                 match: "storage:\\R  provider:\\R    sqlite:\\R      database:(.*)",
-                replace: "storage:\n  provider:\n    sqlite:\n      database: '${debDataPath}/mxisd.db'"
+                replace: "storage:\n  provider:\n    sqlite:\n      database: '${debDataPath}/store.db'"
         )
 
         copy {

docs/features/experimental/application-service.md

@@ -26,7 +26,7 @@ synapseSql:
   connection: '<DB CONNECTION URL>'
 ```
 
-The `synapseSql` section is used to retrieve display names which are not directly accessible in this mode.
+The `synapseSql` section is optional. It is used to retrieve display names which are not directly accessible in this mode.
 For details about `type` and `connection`, see the [relevant documentation](../../stores/synapse.md).
 If you do not configure it, some placeholders will not be available in the notification, like the Room name.
 

docs/features/federation.md

@@ -46,15 +46,6 @@ lookup:
 invite:
   resolution:
     recursive: false
-session:
-  policy:
-    validation:
-      forLocal:
-        toRemote:
-          enabled: false
-      forRemote:
-        toRemote:
-          enabled: false
 ``` 
 
 There is currently no way to selectively disable federation towards specific servers, but this feature is planned.

docs/features/identity.md

@@ -1,6 +1,4 @@
 # Identity
-**WARNING**: This document is incomplete and can be misleading.
-
 Implementation of the [Identity Service API r0.1.0](https://matrix.org/docs/spec/identity_service/r0.1.0.html).
 
 ## Lookups

docs/getting-started.md

@@ -144,7 +144,8 @@ by the relevant hostname which you configured in your reverse proxy.
 **NOTE:** You might not see a suggestion for the e-mail address, which is normal. Still proceed with the invite.
   
 If it worked, it means you are up and running and can enjoy mxisd in its basic mode! Congratulations!  
-If it did not work, [get in touch](../README.md#support) and we'll do our best to get you started.
+If it did not work, read the basic [troubleshooting guide](troubleshooting.md), [get in touch](../README.md#support) and
+we'll do our best to get you started.
 
 ## Next steps
 Once your mxisd server is up and running, there are several ways you can enhance and integrate further with your

docs/stores/exec.md

@@ -212,21 +212,29 @@ The command will use the default values for:
 #### Advanced
 Given the fictional `placeholder` feature:
 ```yaml
-exec.enabled: true
+exec:
-exec.token.mxid: '{matrixId}'
+  enabled: true
-
+  token:
-exec.placeholder.token.localpart: '{username}'
+    mxid: '{matrixId}'
-exec.placeholder.command: '/path/to/executable'
+  auth:
-exec.placeholder.args:
+    token:
+      localpart: '{username}'
+    command: '/path/to/executable'
+    args:
       - '-u'
       - '{username}'
-exec.placeholder.env:
+    env:
       MATRIX_DOMAIN: '{domain}'
       MATRIX_USER_ID: '{matrixId}'
-  
+    output:
-exec.placeholder.output.type: 'json'
+      type: 'json'
-exec.placeholder.exit.success: [0, 128]
+    exit:
-exec.placeholder.exit.failure: [1, 129]
+      success:
+        - 0
+        - 128
+      failure:
+        - 1
+        - 129
 ```
 With:
 - The Identity store enabled for all features

docs/threepids/medium/email/smtp-connector.md

@@ -1,6 +1,4 @@
 # Email notifications - SMTP connector
-Enabled by default.
-
 Connector ID: `smtp`
 
 ## Configuration

docs/threepids/medium/msisdn/twilio-connector.md

@@ -1,6 +1,4 @@
 # SMS notifications - Twilio connector
-Enabled by default.
-
 Connector ID: `twilio`
 
 ## Configuration

docs/threepids/notification/template-generator.md

@@ -51,7 +51,7 @@ This template is used when someone is invited into a room using an email address
 | `%ROOM_NAME%`         | The Name of the room in which the invite took place. If not available/set, empty         |
 | `%ROOM_NAME_OR_ID%`   | The Name of the room in which the invite took place. If not available/set, its Matrix ID |
 
-### Local validation of 3PID Session
+### Validation of 3PID Session
 This template is used when to user which added their 3PID address to their profile/settings and the session policy
 allows at least local sessions.  
 
@@ -59,17 +59,5 @@ allows at least local sessions.
 | Placeholder          | Purpose                                                                              |
 |----------------------|--------------------------------------------------------------------------------------|
 | `%VALIDATION_LINK%`  | URL, including token, to validate the 3PID session.                                  |
-| `%VALIDATION_TOKEN%` | The token needed to validate the local session, in case the user cannot use the link |
+| `%VALIDATION_TOKEN%` | The token needed to validate the session, in case the user cannot use the link.      |
-
+| `%NEXT_URL%`         | URL to redirect to after the sessions has been validated.                            |
-### Remote validation of 3PID Session
-This template is used when to user which added their 3PID address to their profile/settings and the session policy only
-allows remote sessions.
-
-**NOTE:** 3PID session always require local validation of a token, even if a remote session is enforced.  
-One cannot bind a Matrix ID to the session until both local and remote sessions have been validated.
-
-#### Placeholders
-| Placeholder          | Purpose                                                |
-|----------------------|--------------------------------------------------------|
-| `%VALIDATION_TOKEN%` | The token needed to validate the session               |
-| `%NEXT_URL%`         | URL to continue with remote validation of the session. |

docs/troubleshooting.md

@@ -0,0 +1,53 @@
+# Troubleshooting
+- [Purpose](#purpose)
+- [Logs](#logs)
+  - [Locations](#locations)
+  - [Reading Them](#reading-them)
+  - [Common issues](#common-issues)
+- [Submit an issue](#submit-an-issue)
+
+## Purpose
+This document describes basic troubleshooting steps for mxisd.
+
+## Logs
+### Locations
+mxisd logs to `STDOUT` (Standard Output) and `STDERR` (Standard Error) only, which gets redirected
+to log file(s) depending on your system.
+
+If you use the [Debian package](install/debian.md), this goes to `syslog`.  
+If you use the [Docker image](install/docker.md), this goes to the container logs.  
+
+For any other platform, please refer to your package maintainer.
+
+### Reading them
+Before reporting an issue, it is important to produce clean and complete logs so they can be understood.
+
+It is usually useless to try to troubleshoot an issue based on a single log line. Any action or API request
+in mxisd would trigger more than one log lines, and those would be considered necessary context to
+understand what happened.
+
+You may also find things called *stacktraces*. Those are important to pin-point bugs and the likes and should
+always be included in any report. They also tend to be very specific about the issue at hand.
+
+Example of a stacktrace:
+```
+Exception in thread "main" java.lang.NullPointerException
+        at com.example.myproject.Book.getTitle(Book.java:16)
+        at com.example.myproject.Author.getBookTitles(Author.java:25)
+        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
+```
+
+### Common issues
+#### Internal Server Error
+`Contact your administrator with reference Transaction #123456789`
+
+This is a generic message produced in case of an unknown error. The transaction reference allows to easily find
+the location in the logs to look for an error.
+
+**IMPORTANT:** That line alone does not tell you anything about the error. You'll need the log lines before and after,
+usually including a stacktrace, to know what happened. Please take the time to read the surround output to get
+context about the issue at hand.
+
+## Submit an issue
+In case the logs do not allow you to understand the issue at hand, please submit clean and complete logs
+as explained [here](#reading-them) in a new issue on the repository, or [get in touch](../README.md#contact).

mxisd.example.yaml

@@ -1,6 +1,11 @@
 # Sample configuration file explaining the minimum required keys to be set to run mxisd
 #
 # For a complete list of options, see https://github.com/kamax-matrix/mxisd/docs/README.md
+#
+# Please follow the Getting Started guide if this is your first time using/configuring mxisd
+#
+#  -- https://github.com/kamax-matrix/mxisd/blob/master/docs/getting-started.md#getting-started
+#
 
 #######################
 # Matrix config items #
@@ -16,26 +21,27 @@ matrix:
 ################
 # Signing keys #
 ################
-# Absolute path for the Identity Server signing key.
+# Absolute path for the Identity Server signing keys database.
-# This is **NOT** your homeserver key.
+# /!\ THIS MUST **NOT** BE YOUR HOMESERVER KEYS FILE /!\
-# The signing key is auto-generated during execution time if not present.
+# If this path does not exist, it will be auto-generated.
 #
-# During testing, /var/tmp/mxisd.key is a possible value
+# During testing, /var/tmp/mxisd/keys is a possible value
 # For production, recommended location shall be one of the following:
-#   - /var/opt/mxisd/sign.key
+#   - /var/lib/mxisd/keys
-#   - /var/local/mxisd/sign.key
+#   - /var/opt/mxisd/keys
-#   - /var/lib/mxisd/sign.key
+#   - /var/local/mxisd/keys
 #
 key:
   path: ''
 
 
 # Path to the SQLite DB file for mxisd internal storage
+# /!\ THIS MUST **NOT** BE YOUR HOMESERVER DATABASE /!\
 #
 # Examples:
-#  - /var/opt/mxisd/mxisd.db
+#  - /var/opt/mxisd/store.db
-#  - /var/local/mxisd/mxisd.db
+#  - /var/local/mxisd/store.db
-#  - /var/lib/mxisd/mxisd.db
+#  - /var/lib/mxisd/store.db
 #
 storage:
   provider:
@@ -43,48 +49,31 @@ storage:
       database: '/path/to/mxisd.db'
 
 
-####################
+###################
-# Fallback servers #
+# Identity Stores #
-####################
+###################
+# If you are using synapse standalone and do not have an Identity store,
+# see https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/synapse.md#synapse-identity-store
 #
-# Root/Central servers to be used as final fallback when performing lookups.
-# By default, for privacy reasons, matrix.org servers are not enabled.
-# See the following issue: https://github.com/kamax-matrix/mxisd/issues/76
-#
-# If you would like to use them and trade away your privacy for convenience, uncomment the following option:
-#
-#forward:
-#  servers: ['matrix-org']
-
-
-################
-# LDAP Backend #
-################
 # If you would like to integrate with your AD/Samba/LDAP server,
 # see https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/ldap.md
-
+#
-
+# For any other Identity store, or to simply discover them,
-###############
+# see https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/README.md
-# SQL Backend #
-###############
-# If you would like to integrate with a MySQL/MariaDB/PostgreQL/SQLite DB,
-# see https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/sql.md
-
-
-################
-# REST Backend #
-################
-# If you would like to integrate with an existing web service/webapp,
-# see https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/rest.md
 
 
 #################################################
 # Notifications for invites/addition to profile #
 #################################################
-# If you would like to change the content,
+# This is mandatory to deal with anything e-mail related.
+#
+# For an introduction to sessions, invites and 3PIDs in general,
+# see https://github.com/kamax-matrix/mxisd/blob/master/docs/threepids/session/session.md#3pid-sessions
+#
+# If you would like to change the content of the notifications,
 # see https://github.com/kamax-matrix/mxisd/blob/master/docs/threepids/notification/template-generator.md
 #
-#### E-mail invite sender
+#### E-mail connector
 threepid:
   medium:
     email:
@@ -100,12 +89,13 @@ threepid:
           # SMTP port
           port: 587
 
-          # TLS mode for the connection.
+          # STARTLS mode for the connection.
+          # SSL/TLS is currently not supported. See https://github.com/kamax-matrix/mxisd/issues/125
           #
           # Possible values:
-          #  0    Disable TLS entirely
+          #  0    Disable any kind of TLS entirely
-          #  1    Enable TLS if supported by server (default)
+          #  1    Enable STARTLS if supported by server (default)
-          #  2    Force TLS and fail if not available
+          #  2    Force STARTLS and fail if not available
           #
           tls: 1
 

src/main/java/io/kamax/mxisd/MxisdStandaloneExec.java

@@ -22,20 +22,21 @@ package io.kamax.mxisd;
 
 import io.kamax.mxisd.config.MxisdConfig;
 import io.kamax.mxisd.config.YamlConfigLoader;
+import io.kamax.mxisd.exception.ConfigurationException;
 import org.apache.commons.lang3.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-import java.io.IOException;
 import java.util.Arrays;
 import java.util.Iterator;
 import java.util.Objects;
 
 public class MxisdStandaloneExec {
 
-    private static final Logger log = LoggerFactory.getLogger("");
+    private static final Logger log = LoggerFactory.getLogger("App");
 
-    public static void main(String[] args) throws IOException {
+    public static void main(String[] args) {
+        try {
             log.info("------------- mxisd starting -------------");
             MxisdConfig cfg = null;
 
@@ -55,7 +56,6 @@ public class MxisdStandaloneExec {
                 cfg = YamlConfigLoader.tryLoadFromFile("mxisd.yaml").orElseGet(MxisdConfig::new);
             }
 
-        try {
             HttpMxisd mxisd = new HttpMxisd(cfg);
             Runtime.getRuntime().addShutdownHook(new Thread(() -> {
                 mxisd.stop();
@@ -64,6 +64,10 @@ public class MxisdStandaloneExec {
             mxisd.start();
 
             log.info("------------- mxisd started -------------");
+        } catch (ConfigurationException e) {
+            log.error(e.getDetailedMessage());
+            log.error(e.getMessage());
+            System.exit(2);
         } catch (Throwable t) {
             t.printStackTrace();
             System.exit(1);

src/main/java/io/kamax/mxisd/backend/exec/ExecAuthStore.java

@@ -44,6 +44,7 @@ public class ExecAuthStore extends ExecStore implements AuthenticatorProvider {
     private ExecConfig.Auth cfg;
 
     public ExecAuthStore(ExecConfig cfg) {
+        super(cfg);
         this.cfg = Objects.requireNonNull(cfg.getAuth());
     }
 

src/main/java/io/kamax/mxisd/backend/exec/ExecDirectoryStore.java

@@ -36,11 +36,12 @@ public class ExecDirectoryStore extends ExecStore implements DirectoryProvider {
     private MatrixConfig mxCfg;
 
     public ExecDirectoryStore(MxisdConfig cfg) {
-        this(cfg.getExec().getDirectory(), cfg.getMatrix());
+        this(cfg.getExec(), cfg.getMatrix());
     }
 
-    public ExecDirectoryStore(ExecConfig.Directory cfg, MatrixConfig mxCfg) {
+    public ExecDirectoryStore(ExecConfig cfg, MatrixConfig mxCfg) {
-        this.cfg = cfg;
+        super(cfg);
+        this.cfg = cfg.getDirectory();
         this.mxCfg = mxCfg;
     }
 

src/main/java/io/kamax/mxisd/backend/exec/ExecIdentityStore.java

@@ -55,11 +55,8 @@ public class ExecIdentityStore extends ExecStore implements IThreePidProvider {
     private final MatrixConfig mxCfg;
 
     public ExecIdentityStore(ExecConfig cfg, MatrixConfig mxCfg) {
-        this(cfg.getIdentity(), mxCfg);
+        super(cfg);
-    }
+        this.cfg = cfg.getIdentity();
-
-    public ExecIdentityStore(ExecConfig.Identity cfg, MatrixConfig mxCfg) {
-        this.cfg = cfg;
         this.mxCfg = mxCfg;
     }
 

src/main/java/io/kamax/mxisd/backend/exec/ExecProfileStore.java

@@ -38,11 +38,8 @@ public class ExecProfileStore extends ExecStore implements ProfileProvider {
     private ExecConfig.Profile cfg;
 
     public ExecProfileStore(ExecConfig cfg) {
-        this(cfg.getProfile());
+        super(cfg);
-    }
+        this.cfg = cfg.getProfile();
-
-    public ExecProfileStore(ExecConfig.Profile cfg) {
-        this.cfg = cfg;
     }
 
     private Optional<JsonProfileResult> getFull(_MatrixID userId, ExecConfig.Process cfg) {

src/main/java/io/kamax/mxisd/backend/exec/ExecStore.java

@@ -43,14 +43,19 @@ public class ExecStore {
     public static final String JsonType = "json";
     public static final String PlainType = "plain";
 
+    private static final Logger log = LoggerFactory.getLogger(ExecStore.class);
+
     protected static String toJson(Object o) {
         return GsonUtil.get().toJson(o);
     }
 
-    private transient final Logger log = LoggerFactory.getLogger(ExecStore.class);
+    private final ExecConfig cfg;
-
     private Supplier<ProcessExecutor> executorSupplier = () -> new ProcessExecutor().readOutput(true);
 
+    public ExecStore(ExecConfig cfg) {
+        this.cfg = cfg;
+    }
+
     public void setExecutorSupplier(Supplier<ProcessExecutor> supplier) {
         executorSupplier = supplier;
     }
@@ -64,7 +69,7 @@ public class ExecStore {
         private Function<String, String> inputUnknownTypeMapper;
         private Map<String, Supplier<String>> inputTypeSuppliers;
 
-        private Map<String, Function<ExecConfig.TokenOverride, String>> inputTypeTemplates;
+        private Map<String, Function<ExecConfig.Token, String>> inputTypeTemplates;
         private Supplier<String> inputTypeNoTemplateHandler;
         private Map<String, Supplier<String>> tokenMappers;
         private Function<String, String> tokenHandler;
@@ -156,11 +161,11 @@ public class ExecStore {
             inputTypeSuppliers.put(type, handler);
         }
 
-        protected void addInputTemplate(String type, Function<ExecConfig.TokenOverride, String> template) {
+        protected void addInputTemplate(String type, Function<ExecConfig.Token, String> template) {
             inputTypeTemplates.put(type, template);
         }
 
-        public void addJsonInputTemplate(Function<ExecConfig.TokenOverride, Object> template) {
+        public void addJsonInputTemplate(Function<ExecConfig.Token, Object> template) {
             inputTypeTemplates.put(JsonType, token -> GsonUtil.get().toJson(template.apply(token)));
         }
 

src/main/java/io/kamax/mxisd/backend/rest/RestProfileProvider.java

@@ -32,7 +32,7 @@ import io.kamax.mxisd.profile.JsonProfileRequest;
 import io.kamax.mxisd.profile.JsonProfileResult;
 import io.kamax.mxisd.profile.ProfileProvider;
 import org.apache.commons.io.IOUtils;
-import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.http.client.methods.CloseableHttpResponse;
 import org.apache.http.client.methods.HttpPost;
 import org.apache.http.client.utils.URIBuilder;
@@ -49,7 +49,7 @@ import java.util.function.Function;
 
 public class RestProfileProvider extends RestProvider implements ProfileProvider {
 
-    private transient final Logger log = LoggerFactory.getLogger(RestProfileProvider.class);
+    private static final Logger log = LoggerFactory.getLogger(RestProfileProvider.class);
 
     public RestProfileProvider(RestBackendConfig cfg) {
         super(cfg);
@@ -60,15 +60,13 @@ public class RestProfileProvider extends RestProvider implements ProfileProvider
             Function<RestBackendConfig.ProfileEndpoints, Optional<String>> endpoint,
             Function<JsonProfileResult, Optional<T>> value
     ) {
-        return cfg.getEndpoints().getProfile()
+        Optional<String> url = endpoint.apply(cfg.getEndpoints().getProfile());
-                // We get the endpoint
+        if (!url.isPresent()) {
-                .flatMap(endpoint)
+            return Optional.empty();
-                // We only continue if there is a value
+        }
-                .filter(StringUtils::isNotBlank)
+
-                // We use the endpoint
-                .flatMap(url -> {
         try {
-                        URIBuilder builder = new URIBuilder(url);
+            URIBuilder builder = new URIBuilder(url.get());
             HttpPost req = new HttpPost(builder.build());
             req.setEntity(new StringEntity(GsonUtil.get().toJson(new JsonProfileRequest(userId)), ContentType.APPLICATION_JSON));
             try (CloseableHttpResponse res = client.execute(req)) {
@@ -107,17 +105,26 @@ public class RestProfileProvider extends RestProvider implements ProfileProvider
             log.error("I/O Error during backend request", e);
             throw new InternalServerError();
         }
-                });
     }
 
     @Override
     public Optional<String> getDisplayName(_MatrixID userId) {
-        return doRequest(userId, p -> Optional.ofNullable(p.getDisplayName()), profile -> Optional.ofNullable(profile.getDisplayName()));
+        return doRequest(userId, p -> {
+            if (StringUtils.isBlank(p.getDisplayName())) {
+                return Optional.empty();
+            }
+            return Optional.ofNullable(p.getDisplayName());
+        }, profile -> Optional.ofNullable(profile.getDisplayName()));
     }
 
     @Override
     public List<_ThreePid> getThreepids(_MatrixID userId) {
-        return doRequest(userId, p -> Optional.ofNullable(p.getThreepids()), profile -> {
+        return doRequest(userId, p -> {
+            if (StringUtils.isBlank(p.getThreepids())) {
+                return Optional.empty();
+            }
+            return Optional.ofNullable(p.getThreepids());
+        }, profile -> {
             List<_ThreePid> t = new ArrayList<>();
             if (Objects.nonNull(profile.getThreepids())) {
                 t.addAll(profile.getThreepids());
@@ -128,7 +135,12 @@ public class RestProfileProvider extends RestProvider implements ProfileProvider
 
     @Override
     public List<String> getRoles(_MatrixID userId) {
-        return doRequest(userId, p -> Optional.ofNullable(p.getRoles()), profile -> {
+        return doRequest(userId, p -> {
+            if (StringUtils.isBlank(p.getRoles())) {
+                return Optional.empty();
+            }
+            return Optional.ofNullable(p.getRoles());
+        }, profile -> {
             List<String> t = new ArrayList<>();
             if (Objects.nonNull(profile.getRoles())) {
                 t.addAll(profile.getRoles());

src/main/java/io/kamax/mxisd/config/DirectoryConfig.java

@@ -36,9 +36,8 @@ public class DirectoryConfig {
             return homeserver;
         }
 
-        public Exclude setHomeserver(boolean homeserver) {
+        public void setHomeserver(boolean homeserver) {
             this.homeserver = homeserver;
-            return this;
         }
 
         public boolean getThreepid() {

src/main/java/io/kamax/mxisd/config/ExecConfig.java

@@ -20,13 +20,11 @@
 
 package io.kamax.mxisd.config;
 
-import org.apache.commons.lang3.StringUtils;
-
 import java.util.*;
 
 public class ExecConfig {
 
-    public class IO {
+    public static class IO {
 
         private String type;
         private String template;
@@ -49,7 +47,7 @@ public class ExecConfig {
 
     }
 
-    public class Exit {
+    public static class Exit {
 
         private List<Integer> success = Collections.singletonList(0);
         private List<Integer> failure = Collections.singletonList(1);
@@ -72,84 +70,7 @@ public class ExecConfig {
 
     }
 
-    public class TokenOverride {
+    public static class Token {
-
-        private String localpart;
-        private String domain;
-        private String mxid;
-        private String password;
-        private String medium;
-        private String address;
-        private String type;
-        private String query;
-
-        public String getLocalpart() {
-            return StringUtils.defaultIfEmpty(localpart, getToken().getLocalpart());
-        }
-
-        public void setLocalpart(String localpart) {
-            this.localpart = localpart;
-        }
-
-        public String getDomain() {
-            return StringUtils.defaultIfEmpty(domain, getToken().getDomain());
-        }
-
-        public void setDomain(String domain) {
-            this.domain = domain;
-        }
-
-        public String getMxid() {
-            return StringUtils.defaultIfEmpty(mxid, getToken().getMxid());
-        }
-
-        public void setMxid(String mxid) {
-            this.mxid = mxid;
-        }
-
-        public String getPassword() {
-            return StringUtils.defaultIfEmpty(password, getToken().getPassword());
-        }
-
-        public void setPassword(String password) {
-            this.password = password;
-        }
-
-        public String getMedium() {
-            return StringUtils.defaultIfEmpty(medium, getToken().getMedium());
-        }
-
-        public void setMedium(String medium) {
-            this.medium = medium;
-        }
-
-        public String getAddress() {
-            return StringUtils.defaultIfEmpty(address, getToken().getAddress());
-        }
-
-        public void setAddress(String address) {
-            this.address = address;
-        }
-
-        public String getType() {
-            return StringUtils.defaultIfEmpty(type, getToken().getType());
-        }
-
-        public void setType(String type) {
-            this.type = type;
-        }
-
-        public String getQuery() {
-            return StringUtils.defaultIfEmpty(query, getToken().getQuery());
-        }
-
-        public void setQuery(String query) {
-            this.query = query;
-        }
-
-    }
-
-    public class Token {
 
         private String localpart = "{localpart}";
         private String domain = "{domain}";
@@ -226,9 +147,9 @@ public class ExecConfig {
 
     }
 
-    public class Process {
+    public static class Process {
 
-        private TokenOverride token = new TokenOverride();
+        private Token token = new Token();
         private String command;
 
         private List<String> args = new ArrayList<>();
@@ -238,11 +159,11 @@ public class ExecConfig {
         private Exit exit = new Exit();
         private IO output = new IO();
 
-        public TokenOverride getToken() {
+        public Token getToken() {
             return token;
         }
 
-        public void setToken(TokenOverride token) {
+        public void setToken(Token token) {
             this.token = token;
         }
 
@@ -300,7 +221,7 @@ public class ExecConfig {
 
     }
 
-    public class Auth extends Process {
+    public static class Auth extends Process {
 
         private Boolean enabled;
 
@@ -314,9 +235,9 @@ public class ExecConfig {
 
     }
 
-    public class Directory {
+    public static class Directory {
 
-        public class Search {
+        public static class Search {
 
             private Process byName = new Process();
             private Process byThreepid = new Process();
@@ -360,7 +281,7 @@ public class ExecConfig {
 
     }
 
-    public class Lookup {
+    public static class Lookup {
 
         private Process single = new Process();
         private Process bulk = new Process();
@@ -383,7 +304,7 @@ public class ExecConfig {
 
     }
 
-    public class Identity {
+    public static class Identity {
 
         private Boolean enabled;
         private int priority;
@@ -415,7 +336,7 @@ public class ExecConfig {
 
     }
 
-    public class Profile {
+    public static class Profile {
 
         private Boolean enabled;
         private Process displayName = new Process();

src/main/java/io/kamax/mxisd/config/YamlConfigLoader.java

@@ -21,12 +21,16 @@
 package io.kamax.mxisd.config;
 
 import io.kamax.matrix.json.GsonUtil;
+import io.kamax.mxisd.exception.ConfigurationException;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.yaml.snakeyaml.Yaml;
 import org.yaml.snakeyaml.constructor.Constructor;
+import org.yaml.snakeyaml.introspector.BeanAccess;
+import org.yaml.snakeyaml.parser.ParserException;
 import org.yaml.snakeyaml.representer.Representer;
 
+import java.io.File;
 import java.io.FileInputStream;
 import java.io.FileNotFoundException;
 import java.io.IOException;
@@ -37,17 +41,26 @@ public class YamlConfigLoader {
     private static final Logger log = LoggerFactory.getLogger(YamlConfigLoader.class);
 
     public static MxisdConfig loadFromFile(String path) throws IOException {
-        log.debug("Reading config from {}", path);
+        File f = new File(path).getAbsoluteFile();
+        log.info("Reading config from {}", f.toString());
         Representer rep = new Representer();
+        rep.getPropertyUtils().setBeanAccess(BeanAccess.FIELD);
         rep.getPropertyUtils().setAllowReadOnlyProperties(true);
         rep.getPropertyUtils().setSkipMissingProperties(true);
         Yaml yaml = new Yaml(new Constructor(MxisdConfig.class), rep);
-        try (FileInputStream is = new FileInputStream(path)) {
+        try (FileInputStream is = new FileInputStream(f)) {
-            Object o = yaml.load(is);
+            MxisdConfig raw = yaml.load(is);
             log.debug("Read config in memory from {}", path);
-            MxisdConfig cfg = GsonUtil.get().fromJson(GsonUtil.get().toJson(o), MxisdConfig.class);
+
+            // SnakeYaml set objects to null when there is no value set in the config, even a full sub-tree.
+            // This is problematic for default config values and objects, to avoid NPEs.
+            // Therefore, we'll use Gson to re-parse the data in a way that avoids us checking the whole config for nulls.
+            MxisdConfig cfg = GsonUtil.get().fromJson(GsonUtil.get().toJson(raw), MxisdConfig.class);
+
             log.info("Loaded config from {}", path);
             return cfg;
+        } catch (ParserException t) {
+            throw new ConfigurationException(t.getMessage(), "Could not parse YAML config file - Please check indentation and that the configuration options exist");
         }
     }
 

src/main/java/io/kamax/mxisd/config/rest/RestBackendConfig.java

@@ -28,7 +28,6 @@ import org.slf4j.LoggerFactory;
 import java.net.MalformedURLException;
 import java.net.URL;
 import java.util.Objects;
-import java.util.Optional;
 
 public class RestBackendConfig {
 
@@ -118,8 +117,8 @@ public class RestBackendConfig {
             this.identity = identity;
         }
 
-        public Optional<ProfileEndpoints> getProfile() {
+        public ProfileEndpoints getProfile() {
-            return Optional.ofNullable(profile);
+            return profile;
         }
 
         public void setProfile(ProfileEndpoints profile) {
@@ -128,7 +127,7 @@ public class RestBackendConfig {
 
     }
 
-    private transient final Logger log = LoggerFactory.getLogger(RestBackendConfig.class);
+    private static final Logger log = LoggerFactory.getLogger(RestBackendConfig.class);
 
     private boolean enabled;
     private String host;
@@ -197,6 +196,11 @@ public class RestBackendConfig {
             log.info("Directory endpoint: {}", endpoints.getDirectory());
             log.info("Identity Single endpoint: {}", endpoints.identity.getSingle());
             log.info("Identity Bulk endpoint: {}", endpoints.identity.getBulk());
+
+            log.info("Profile endpoints:");
+            log.info("  - Display name: {}", getEndpoints().getProfile().getDisplayName());
+            log.info("  - 3PIDs: {}", getEndpoints().getProfile().getThreepids());
+            log.info("  - Roles: {}", getEndpoints().getProfile().getRoles());
         }
     }
 

src/main/java/io/kamax/mxisd/config/sql/SqlConfig.java

@@ -21,6 +21,7 @@
 package io.kamax.mxisd.config.sql;
 
 import io.kamax.mxisd.util.GsonUtil;
+import org.apache.commons.lang3.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -314,7 +315,8 @@ public abstract class SqlConfig {
         log.info("Enabled: {}", isEnabled());
         if (isEnabled()) {
             log.info("Type: {}", getType());
-            log.info("Connection: {}", getConnection());
+            log.info("Has connection info? {}", !StringUtils.isEmpty(getConnection()));
+            log.debug("Connection: {}", getConnection());
             log.info("Auth enabled: {}", getAuth().isEnabled());
             log.info("Directory queries: {}", GsonUtil.build().toJson(getDirectory().getQuery()));
             log.info("Identity type: {}", getIdentity().getType());

src/main/java/io/kamax/mxisd/exception/ConfigurationException.java

@@ -20,11 +20,8 @@
 
 package io.kamax.mxisd.exception;
 
-import java.util.Optional;
-
 public class ConfigurationException extends RuntimeException {
 
-    private String key;
     private String detailedMsg;
 
     public ConfigurationException(String key) {
@@ -40,8 +37,8 @@ public class ConfigurationException extends RuntimeException {
         this.detailedMsg = detailedMsg;
     }
 
-    public Optional<String> getDetailedMessage() {
+    public String getDetailedMessage() {
-        return Optional.ofNullable(detailedMsg);
+        return detailedMsg;
     }
 
 }

src/main/java/io/kamax/mxisd/http/undertow/handler/BasicHttpHandler.java

@@ -101,6 +101,10 @@ public abstract class BasicHttpHandler implements HttpHandler {
         return GsonUtil.parseObj(getBodyUtf8(exchange));
     }
 
+    protected void putHeader(HttpServerExchange ex, String name, String value) {
+        ex.getResponseHeaders().put(HttpString.tryFromString(name), value);
+    }
+
     protected void respond(HttpServerExchange ex, int statusCode, JsonElement bodyJson) {
         respondJson(ex, statusCode, GsonUtil.get().toJson(bodyJson));
     }

src/main/java/io/kamax/mxisd/http/undertow/handler/SaneHandler.java

@@ -27,8 +27,7 @@ import io.kamax.matrix.json.InvalidJsonException;
 import io.kamax.mxisd.exception.*;
 import io.undertow.server.HttpHandler;
 import io.undertow.server.HttpServerExchange;
-import io.undertow.util.HttpString;
+import org.apache.commons.lang3.StringUtils;
-import org.apache.commons.lang.StringUtils;
 import org.apache.http.HttpStatus;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -37,15 +36,22 @@ import java.time.Instant;
 
 public class SaneHandler extends BasicHttpHandler {
 
+    private static final Logger log = LoggerFactory.getLogger(SaneHandler.class);
+
+    private static final String CorsOriginName = "Access-Control-Allow-Origin";
+    private static final String CorsOriginValue = "*";
+    private static final String CorsMethodsName = "Access-Control-Allow-Methods";
+    private static final String CorsMethodsValue = "GET, POST, PUT, DELETE, OPTIONS";
+    private static final String CorsHeadersName = "Access-Control-Allow-Headers";
+    private static final String CorsHeadersValue = "Origin, X-Requested-With, Content-Type, Accept, Authorization";
+
     public static SaneHandler around(HttpHandler h) {
         return new SaneHandler(h);
     }
 
-    private transient final Logger log = LoggerFactory.getLogger(SaneHandler.class);
+    private final HttpHandler child;
 
-    private HttpHandler child;
+    private SaneHandler(HttpHandler child) {
-
-    public SaneHandler(HttpHandler child) {
         this.child = child;
     }
 
@@ -58,9 +64,9 @@ public class SaneHandler extends BasicHttpHandler {
         } else {
             try {
                 // CORS headers as per spec
-                exchange.getResponseHeaders().put(HttpString.tryFromString("Access-Control-Allow-Origin"), "*");
+                putHeader(exchange, CorsOriginName, CorsOriginValue);
-                exchange.getResponseHeaders().put(HttpString.tryFromString("Access-Control-Allow-Methods"), "GET, POST, PUT, DELETE, OPTIONS");
+                putHeader(exchange, CorsMethodsName, CorsMethodsValue);
-                exchange.getResponseHeaders().put(HttpString.tryFromString("Access-Control-Allow-Headers"), "Origin, X-Requested-With, Content-Type, Accept, Authorization");
+                putHeader(exchange, CorsHeadersName, CorsHeadersValue);
 
                 child.handleRequest(exchange);
             } catch (IllegalArgumentException e) {
@@ -89,9 +95,9 @@ public class SaneHandler extends BasicHttpHandler {
                 handleException(exchange, e);
             } catch (InternalServerError e) {
                 if (StringUtils.isNotBlank(e.getInternalReason())) {
-                    log.error("Reference #{} - {}", e.getReference(), e.getInternalReason());
+                    log.error("Transaction #{} - {}", e.getReference(), e.getInternalReason());
                 } else {
-                    log.error("Reference #{}", e);
+                    log.error("Transaction #{}", e);
                 }
 
                 handleException(exchange, e);
@@ -105,14 +111,11 @@ public class SaneHandler extends BasicHttpHandler {
                 respond(exchange, e.getStatus(), buildErrorBody(exchange, e.getErrorCode(), e.getError()));
             } catch (RuntimeException e) {
                 log.error("Unknown error when handling {}", exchange.getRequestURL(), e);
-                respond(exchange, HttpStatus.SC_INTERNAL_SERVER_ERROR, buildErrorBody(exchange,
+                String message = e.getMessage();
-                        "M_UNKNOWN",
+                if (StringUtils.isBlank(message)) {
-                        StringUtils.defaultIfBlank(
+                    message = "An internal server error occurred. Contact your administrator with reference Transaction #" + Instant.now().toEpochMilli();
-                                e.getMessage(),
+                }
-                                "An internal server error occurred. If this error persists, please contact support with reference #" +
+                respond(exchange, HttpStatus.SC_INTERNAL_SERVER_ERROR, buildErrorBody(exchange, "M_UNKNOWN", message));
-                                        Instant.now().toEpochMilli()
-                        )
-                ));
             } finally {
                 exchange.endExchange();
             }

src/test/java/io/kamax/mxisd/test/backend/exec/ExecDirectoryStoreTest.java

@@ -56,32 +56,32 @@ public class ExecDirectoryStoreTest extends ExecStoreTest {
         }));
     }
 
-    private ExecConfig.Directory getCfg() {
+    private ExecConfig getCfg() {
-        ExecConfig.Directory cfg = new ExecConfig().build().getDirectory();
+        ExecConfig cfg = new ExecConfig().build();
         assertFalse(cfg.isEnabled());
         cfg.setEnabled(true);
         assertTrue(cfg.isEnabled());
-        cfg.getSearch().getByName().getOutput().setType(ExecStore.JsonType);
+        cfg.getDirectory().getSearch().getByName().getOutput().setType(ExecStore.JsonType);
         return cfg;
     }
 
-    private ExecDirectoryStore getStore(ExecConfig.Directory cfg) {
+    private ExecDirectoryStore getStore(ExecConfig cfg) {
         ExecDirectoryStore store = new ExecDirectoryStore(cfg, getMatrixCfg());
         store.setExecutorSupplier(this::build);
         return store;
     }
 
     private ExecDirectoryStore getStore(String command) {
-        ExecConfig.Directory cfg = getCfg();
+        ExecConfig cfg = getCfg();
-        cfg.getSearch().getByName().setCommand(command);
+        cfg.getDirectory().getSearch().getByName().setCommand(command);
-        cfg.getSearch().getByThreepid().setCommand(command);
+        cfg.getDirectory().getSearch().getByThreepid().setCommand(command);
         return getStore(cfg);
     }
 
     @Test
     public void byNameNoCommandDefined() {
-        ExecConfig.Directory cfg = getCfg();
+        ExecConfig cfg = getCfg();
-        assertTrue(StringUtils.isEmpty(cfg.getSearch().getByName().getCommand()));
+        assertTrue(StringUtils.isEmpty(cfg.getDirectory().getSearch().getByName().getCommand()));
         ExecDirectoryStore store = getStore(cfg);
 
         UserDirectorySearchResult result = store.searchByDisplayName("user");

src/test/java/io/kamax/mxisd/test/backend/exec/ExecIdentityStoreTest.java

@@ -62,17 +62,17 @@ public class ExecIdentityStoreTest extends ExecStoreTest {
         }));
     }
 
-    private ExecConfig.Identity getCfg() {
+    private ExecConfig getCfg() {
-        ExecConfig.Identity cfg = new ExecConfig().build().getIdentity();
+        ExecConfig cfg = new ExecConfig().build();
         assertFalse(cfg.isEnabled());
         cfg.setEnabled(true);
         assertTrue(cfg.isEnabled());
-        cfg.getLookup().getSingle().getOutput().setType(ExecStore.JsonType);
+        cfg.getIdentity().getLookup().getSingle().getOutput().setType(ExecStore.JsonType);
-        cfg.getLookup().getBulk().getOutput().setType(ExecStore.JsonType);
+        cfg.getIdentity().getLookup().getBulk().getOutput().setType(ExecStore.JsonType);
         return cfg;
     }
 
-    private ExecIdentityStore getStore(ExecConfig.Identity cfg) {
+    private ExecIdentityStore getStore(ExecConfig cfg) {
         ExecIdentityStore store = new ExecIdentityStore(cfg, getMatrixCfg());
         store.setExecutorSupplier(this::build);
         assertTrue(store.isLocal());
@@ -80,9 +80,9 @@ public class ExecIdentityStoreTest extends ExecStoreTest {
     }
 
     private ExecIdentityStore getStore(String command) {
-        ExecConfig.Identity cfg = getCfg();
+        ExecConfig cfg = getCfg();
-        cfg.getLookup().getSingle().setCommand(command);
+        cfg.getIdentity().getLookup().getSingle().setCommand(command);
-        cfg.getLookup().getBulk().setCommand(command);
+        cfg.getIdentity().getLookup().getBulk().setCommand(command);
         return getStore(cfg);
     }
 

src/test/java/io/kamax/mxisd/test/backend/exec/ExecProfileStoreTest.java

@@ -70,28 +70,28 @@ public class ExecProfileStoreTest extends ExecStoreTest {
 
     }
 
-    private ExecConfig.Profile getCfg() {
+    private ExecConfig getCfg() {
-        ExecConfig.Profile cfg = new ExecConfig().build().getProfile();
+        ExecConfig cfg = new ExecConfig().build();
         assertFalse(cfg.isEnabled());
         cfg.setEnabled(true);
         assertTrue(cfg.isEnabled());
-        cfg.getDisplayName().getOutput().setType(ExecStore.JsonType);
+        cfg.getProfile().getDisplayName().getOutput().setType(ExecStore.JsonType);
-        cfg.getThreePid().getOutput().setType(ExecStore.JsonType);
+        cfg.getProfile().getThreePid().getOutput().setType(ExecStore.JsonType);
-        cfg.getRole().getOutput().setType(ExecStore.JsonType);
+        cfg.getProfile().getRole().getOutput().setType(ExecStore.JsonType);
         return cfg;
     }
 
-    private ExecProfileStore getStore(ExecConfig.Profile cfg) {
+    private ExecProfileStore getStore(ExecConfig cfg) {
         ExecProfileStore store = new ExecProfileStore(cfg);
         store.setExecutorSupplier(this::build);
         return store;
     }
 
     private ExecProfileStore getStore(String command) {
-        ExecConfig.Profile cfg = getCfg();
+        ExecConfig cfg = getCfg();
-        cfg.getDisplayName().setCommand(command);
+        cfg.getProfile().getDisplayName().setCommand(command);
-        cfg.getThreePid().setCommand(command);
+        cfg.getProfile().getThreePid().setCommand(command);
-        cfg.getRole().setCommand(command);
+        cfg.getProfile().getRole().setCommand(command);
         return getStore(cfg);
     }
 

src/test/java/io/kamax/mxisd/test/backend/rest/RestProfileProviderTest.java

@@ -23,6 +23,7 @@ package io.kamax.mxisd.test.backend.rest;
 import com.github.tomakehurst.wiremock.junit.WireMockRule;
 import io.kamax.matrix.MatrixID;
 import io.kamax.matrix._MatrixID;
+import io.kamax.matrix._ThreePid;
 import io.kamax.matrix.json.GsonUtil;
 import io.kamax.mxisd.backend.rest.RestProfileProvider;
 import io.kamax.mxisd.config.rest.RestBackendConfig;
@@ -34,6 +35,7 @@ import org.junit.Before;
 import org.junit.Rule;
 import org.junit.Test;
 
+import java.util.List;
 import java.util.Optional;
 
 import static com.github.tomakehurst.wiremock.client.WireMock.*;
@@ -42,27 +44,40 @@ import static org.junit.Assert.*;
 
 public class RestProfileProviderTest {
 
+    private static final int MockHttpPort = 65000;
+    private static final String MockHttpHost = "localhost";
+
     @Rule
-    public WireMockRule wireMockRule = new WireMockRule(65000);
+    public WireMockRule wireMockRule = new WireMockRule(MockHttpPort);
 
     private final String displayNameEndpoint = "/displayName";
 
-    private final _MatrixID userId = MatrixID.from("john", "matrix.localhost").valid();
+    private final _MatrixID userId = MatrixID.from("john", "matrix." + MockHttpHost).valid();
 
     private RestProfileProvider p;
 
-    @Before
+    private RestBackendConfig getCfg(RestBackendConfig.Endpoints endpoints) {
-    public void before() {
-        ProfileEndpoints endpoints = new ProfileEndpoints();
-        endpoints.setDisplayName(displayNameEndpoint);
-
         RestBackendConfig cfg = new RestBackendConfig();
         cfg.setEnabled(true);
-        cfg.setHost("http://localhost:65000");
+        cfg.setHost("http://" + MockHttpHost + ":" + MockHttpPort);
-        cfg.getEndpoints().setProfile(endpoints);
+        cfg.setEndpoints(endpoints);
         cfg.build();
 
-        p = new RestProfileProvider(cfg);
+        return cfg;
+    }
+
+    private RestProfileProvider get(RestBackendConfig cfg) {
+        return new RestProfileProvider(cfg);
+    }
+
+    @Before
+    public void before() {
+        ProfileEndpoints pEndpoints = new ProfileEndpoints();
+        pEndpoints.setDisplayName(displayNameEndpoint);
+        RestBackendConfig.Endpoints endpoints = new RestBackendConfig.Endpoints();
+        endpoints.setProfile(pEndpoints);
+
+        p = get(getCfg(endpoints));
     }
 
     @Test
@@ -144,4 +159,26 @@ public class RestProfileProviderTest {
         }
     }
 
+    @Test
+    public void forEmptyEndpoints() {
+        ProfileEndpoints pEndpoints = new ProfileEndpoints();
+        pEndpoints.setDisplayName("");
+        pEndpoints.setThreepids("");
+        pEndpoints.setRoles("");
+
+        RestBackendConfig.Endpoints endpoints = new RestBackendConfig.Endpoints();
+        endpoints.setProfile(pEndpoints);
+
+        RestProfileProvider p = get(getCfg(endpoints));
+
+        Optional<String> dn = p.getDisplayName(userId);
+        assertFalse(dn.isPresent());
+
+        List<String> roles = p.getRoles(userId);
+        assertTrue(roles.isEmpty());
+
+        List<_ThreePid> tpids = p.getThreepids(userId);
+        assertTrue(tpids.isEmpty());
+    }
+
 }