Complete the documentation and polish the code

docs/stores/exec.md

@@ -1,60 +1,470 @@
 # Exec Identity Store
-This Identity Store lets you run arbitrary commands to handle the various requests in each support feature.
+- [Features](#features)
-
+- [Overview](#overview)
-This is the most versatile Identity store of mxisd, allowing you to connect any kind of logic in any language/scripting.
+- [Configuration](#configuration)
+  - [Global](#global)
+    - [Tokens](#tokens)
+  - [Executable](#executable)
+    - [Input](#input)
+    - [Output](#output)
+  - [Examples](#examples)
+  - [Per-Feature](#per-feature)
+- [Authentication](#authentication)
+  - [Tokens](#tokens-1)
+  - [Input](#input-1)
+  - [Output](#output-1)
+- [Directory](#directory)
+  - [Tokens](#tokens-2)
+  - [Input](#input-2)
+  - [Output](#output-2)
+- [Identity](#identity)
+  - [Single Lookup](#single-lookup)
+    - [Tokens](#tokens-3)
+    - [Input](#input-3)
+    - [Output](#output-3)
+  - [Bulk Lookup](#bulk-lookup)
+    - [Tokens](#tokens-4)
+    - [Input](#input-4)
+    - [Output](#output-4)
+- [Profile](#profile)
+  - [Tokens](#tokens-5)
+  - [Input](#input-5)
+  - [Output](#output-5)
+  
+---
 
 ## Features
-|      Name      |   Supported?  |
+|                       Name                      | Supported |
-|----------------|---------------|
+|-------------------------------------------------|-----------|
-| Authentication | Yes           |
+| [Authentication](../features/authentication.md) | Yes       |
-| Directory      | *In Progress* |
+| [Directory](../features/directory.md)           | Yes       |
-| Identity       | *In Progress* |
+| [Identity](../features/identity.md)             | Yes       |
-| Profile        | *In Progress* |
+| [Profile](#profile)                             | Yes       |
+
+This Identity Store lets you run arbitrary commands to handle the various requests in each support feature.  
+It is the most versatile Identity store of mxisd, allowing you to connect any kind of logic with any executable/script.
 
 ## Overview
 Each request can be mapping to a fully customizable command configuration.  
 The various parameters can be provided via any combination of:
-- Standard Input
+- [Standard Input](https://en.wikipedia.org/wiki/Standard_streams#Standard_input_(stdin))
-- Command line arguments
+- [Command-line arguments](https://en.wikipedia.org/wiki/Command-line_interface#Arguments)
-- Environment variables
+- [Environment variables](https://en.wikipedia.org/wiki/Environment_variable)
 
 Each of those supports a set of customizable token which will be replaced prior to running the command, allowing to
 provide the input values in any number of ways.
 
-Success and data will be provided via [Exit status](https://en.wikipedia.org/wiki/Exit_status) and Standard Output, both
+Success and data will be provided via any combination of:
-supporting a set of options. 
+- [Exit status](https://en.wikipedia.org/wiki/Exit_status)
+- [Standard Output](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout))
+
+Each of those supports a set of configuration item to decide how to process the value and/or in which format.
+
+All values, inputs and outputs are UTF-8 encoded.
 
 ## Configuration
+Each feature comes with a set of possible lookup/action which is mapped to a generic configuration item block.  
+We will use the term `Executable` for each lookup/action and `Processor` for each configuration block.
+
+### Global
 ```yaml
 exec.enabled: <boolean>
 ```
-Enable/disable the Identity store at a global/default level. Each feature can still be enabled/disabled specifically.
+Enable/disable the Identity store at a global/default level. Each feature can still be individually enabled/disabled.
 
-*TBC*
+#### Tokens
+The following options allow to globally set tokens for value replacement across all features and processors config.  
+Not all features use all tokens, and each feature might also have its own specific tokens. See each feature documentation.
+  
+They can be set within the following scope:
 
-## Use-case examples
+```yaml
+exec.token.<token>: '<value>'
+```
+
+---
+
+The following tokens and default values are available:
+```yaml
+localpart: '{localpart}'
+```
+Localpart of Matrix User IDs
+
+```yaml
+domain: '{domain}'
+```
+Domain of Matrix User IDs
+
+```yaml
+mxid: '{mxid}'
+```
+Full representation of Matrix User IDs
+
+```yaml
+medium: '{medium}'
+```
+Medium of 3PIDs
+
+```yaml
+address: '{address}'
+```
+Address of 3PIDs
+
+```yaml
+type: '{type}'
+```
+Type of query
+
+```yaml
+query: '{query}'
+```
+Query value
+
+### Executable
+*Executable*s have the following options:
+```yaml
+command: '/path/to/executableOrScript'
+
+```
+Set the executable (relative or absolute) path to be executed. If no command is given, the action will return a "neutral"
+result if possible or be skipped altogether.
+
+---
+
+Command line arguments can be given via a list via both YAML formats:
+```yaml
+args:
+ - '-t'
+ - '{token}'
+ - '-v'
+ - 'value'
+```
+or
+```yaml
+args: ['-t', '{token}', '-v', 'value]
+```
+Each argument will be processed for token replacement.
+
+---
+
+Environment variables can be given as key/value pairs:
+```yaml
+env:
+  ENV_VAR_1: 'value'
+  ENV_VAR_2: '{token}'
+``` 
+Each variable value will be processed for token replacement.
+
+#### Input
+Standard input can be configured in the namespaces `input` with:
+- `type`: The format to use
+- `template`: The full or partial template with tokens to be used when generating the input
+
+Not all features and *Executable*s allow for a template to be provided.  
+Templates for listed-based input are not supported at this time.  
+Default templates may be provided per *Executable*.
+
+The following types are available:
+- `json`: Use JSON format, shared with the [REST Identity Store](rest.md)
+- `plain`: Use a custom multi-lines, optionally tab-separated input
+
+#### Output
+Standard output can be configured in the namespaces `output` with:
+- `type`: The format to use
+- `template`: The full or partial template with tokens to be used when processing the output
+
+Not all features and *Executable*s allow for a template to be provided.  
+Templates for listed-based output are not supported at this time.  
+Default templates may be provided per *Executable*.
+
+The following types are available:
+- `json`: Use JSON format, shared with the [REST Identity Store](rest.md)
+- `plain`: Use a custom multi-lines, optionally tab-separated output
+
+### Examples
+#### Basic
+```yaml
+exec.auth.enabled: true
+exec.auth.command: '/opt/mxisd-exec/auth.sh'
+exec.auth.args: ['{localpart}']
+exec.auth.input.type: 'plain'
+exec.auth.input.template: '{password}'
+exec.auth.env:
+  DOMAIN: '{domain}'
+```
+With Authentication enabled, run `/opt/mxisd-exec/auth.sh` when validating credentials, providing:
+- A single command-line argument to provide the `localoart` as username 
+- A plain text string with the password token for standard input, which will be replaced by the password to check
+- A single environment variable `DOMAIN` containing Matrix ID domain, if given
+
+The command will use the default values for:
+- Success exit status of `0`
+- Failure exit status of `1`
+- Any other exit status considered as error
+- The standard output processing as not processed
+
+#### Advanced
+Given the fictional `placeholder` feature:
 ```yaml
 exec.enabled: true
+exec.token.mxid: '{matrixId}'
 
-exec.auth.command: '/path/to/auth/executable'
+exec.placeholder.token.localpart: '{username}'
-exec.auth.args: ['-u', '{localpart}']
+exec.placeholder.command: '/path/to/executable'
-exec.auth.env:
+exec.placeholder.args:
-  PASSWORD: '{password}'
+  - '-u'
+  - '{username}'
+exec.placeholder.env:
   MATRIX_DOMAIN: '{domain}'
-  MATRIX_USER_ID: '{mxid}'
+  MATRIX_USER_ID: '{matrixId}'
+  
+exec.placeholder.output.type: 'json'
+exec.placeholder.exit.success: [0, 128]
+exec.placeholder.exit.failure: [1, 129]
 ```
-This will run `/path/to/auth/executable` with:
+With:
-- The extracted Matrix User ID `localpart` provided as the second command line argument, the first one being `-u` 
+- The Identity store enabled for all features
-- The password, the extract Matrix `domain` and the full User ID as arbitrary environment variables, respectively `PASSWORD`, `MATRIX_DOMAIN` and `MATRIX_USER_ID`
+- A global specific token `{matrixId}` for Matrix User IDs, replacing the default `{mxid}`
 
+Running `/path/to/executable` providing:
+- A custom token for localpart, `{username}`, used as a 2nd command-line argument
+- An extracted Matrix User ID `localpart` provided as the second command line argument, the first one being `-u` 
+- A password, the extracted Matrix `domain` and the full User ID as arbitrary environment variables, respectively
+  `PASSWORD`, `MATRIX_DOMAIN` and `MATRIX_USER_ID`
+
+After execution:
+- Process stdout as [JSON](https://en.wikipedia.org/wiki/JSON)
+- Consider exit status `0` and `128` as success and try to process the stdout for data
+- Consider exit status `1` and `129` as failure and try to process the stdout for error code and message
+
+### Per Feature
+See each dedicated [Feature](#features) section.
+
+## Authentication
+The Authentication feature can be enabled/disabled using:
 ```yaml
-## Few more available config items
+exec.auth.enabled: <true/false>
-#
-# exec.token.domain: '{matrixDomain}' # This sets the default replacement token for the Matrix Domain of the User ID, across all features.
-# exec.auth.token.domain: '{matrixDomainForAuth}' # We can also set another token specific to a feature.
-# exec.auth.input: 'json' # This is not supported yet.
-# exec.auth.exit.success: [0] # Exit status that will consider the request successful. This is already the default.
-# exec.auth.exit.failure: [1,2,3] # Exist status that will consider the request failed. Anything else than success or failure statuses will throw an exception.
-# exec.auth.output: 'json' # Required if stdout should be read on success. This uses the same output as the REST Identity store for Auth.
 ```
-*TBC*
+
+---
+
+This feature provides a single *Executable* under the namespace:
+```yaml
+exec.auth:
+  ...
+```
+
+### Tokens
+The following tokens/default values are specific to this feature:
+```yaml
+password: '{password}'
+```
+The provided password
+
+### Input
+Supported input types and default templates:
+
+#### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+#### Plain (`plain`)
+Default template:
+```
+{localpart}
+{domain}
+{mxid}
+{password}
+```
+
+### Output
+Supported output types and default templates:
+
+#### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+#### Plain (`plain`)
+**NOTE:** This has limited support. Use the JSON type for full support.
+
+Default template:
+```
+[success status, true or 1 are interpreted as success]
+[display name of the user]
+```
+
+## Directory
+The Directory feature can be enabled/disabled using:
+```yaml
+exec.directory.enabled: <true/false>
+```
+
+---
+
+Two search types configuration namespace are available, using the same input/output formats and templates:
+
+By name:
+```yaml
+exec.directory.search.byName:
+  ...
+```
+By 3PID:
+```yaml
+exec.directory.search.byThreepid:
+  ...
+```
+
+#### Tokens
+No specific tokens are available.
+
+#### Input
+Supported input types and default templates:
+
+##### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+##### Plain (`plain`)
+Default template:
+```
+[type of search, following the REST Identity store format]
+[query string]
+```
+
+#### Output
+Supported output types and default templates:
+
+##### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+##### Plain (`plain`)
+**Not supported at this time.** Use the JSON type.
+
+## Identity
+The Identity feature can be enabled/disabled using:
+```yaml
+exec.identity.enabled: <true/false>
+```
+
+### Single lookup
+Configuration namespace:
+```yaml
+exec.identity.lookup.single:
+  ...
+```
+
+#### Tokens
+No specific tokens are available.
+
+#### Input
+Supported input types and default templates:
+
+##### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+##### Plain (`plain`)
+Default template:
+```
+{medium}
+{address}
+```
+
+#### Output
+Supported output types and default templates:
+
+##### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+##### Plain (`plain`)
+Default template:
+```
+[User ID type, as documented in the REST Identity Store]
+[User ID value]
+```
+
+The User ID type will default to `localpart` if:
+- Only one line is returned
+- The first line is empty
+
+### Bulk lookup
+Configuration namespace:
+```yaml
+exec.identity.lookup.bulk:
+  ...
+```
+
+#### Tokens
+No specific tokens are available.
+
+#### Input
+Supported input types and default templates:
+
+##### JSON (`json`)
+**NOTE:** Custom Templates are not supported.
+
+Same as the [REST Identity Store](rest.md).
+
+##### Plain (`plain`)
+**Not supported at this time.** Use the JSON type.
+
+#### Output
+Supported output types and default templates:
+
+##### JSON (`json`)
+**NOTE:** Custom Templates are not supported.
+
+Same as the [REST Identity Store](rest.md).
+
+##### Plain (`plain`)
+**Not supported at this time.** Use the JSON type.
+
+## Profile
+The Profile feature can be enabled/disabled using:
+```yaml
+exec.profile.enabled: <true/false>
+```
+
+---
+
+The following *Executable*s namespace are available, share the same input/output formats and templates:
+
+Get Display name:
+```yaml
+exec.profile.displayName:
+  ...
+```
+
+Get 3PIDs:
+```yaml
+exec.profile.threePid:
+  ...
+```
+
+Get Roles:
+```yaml
+exec.profile.role:
+  ...
+```
+
+
+### Tokens
+No specific tokens are available.
+
+### Input
+Supported input types and default templates:
+
+#### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+#### Plain (`plain`)
+Default template:
+```
+{localpart}
+{domain}
+{mxid}
+```
+### Output
+Supported output types and default templates:
+
+#### JSON (`json`)
+Same as the [REST Identity Store](rest.md);
+
+#### Plain (`plain`)
+**Not supported at this time.** Use the JSON type.

src/main/java/io/kamax/mxisd/backend/exec/ExecAuthStore.java

@@ -81,7 +81,7 @@ public class ExecAuthStore extends ExecStore implements AuthenticatorProvider {
             json.setPassword(tokens.getPassword());
             return json;
         });
-        p.addInputTemplate(MultilinesType, tokens -> tokens.getLocalpart() + System.lineSeparator() +
+        p.addInputTemplate(PlainType, tokens -> tokens.getLocalpart() + System.lineSeparator() +
                 tokens.getDomain() + System.lineSeparator() +
                 tokens.getMxid() + System.lineSeparator() +
                 tokens.getPassword() + System.lineSeparator()
@@ -102,7 +102,7 @@ public class ExecAuthStore extends ExecStore implements AuthenticatorProvider {
 
             return result;
         });
-        p.addSuccessMapper(MultilinesType, output -> {
+        p.addSuccessMapper(PlainType, output -> {
             String[] lines = output.split("\\R");
             if (lines.length > 2) {
                 throw new InternalServerError("Exec auth command returned more than 2 lines (" + lines.length + ")");

src/main/java/io/kamax/mxisd/backend/exec/ExecDirectoryStore.java

@@ -60,7 +60,7 @@ public class ExecDirectoryStore extends ExecStore implements IDirectoryProvider
         Processor<UserDirectorySearchResult> p = new Processor<>(cfg);
 
         p.addJsonInputTemplate(tokens -> new UserDirectorySearchRequest(tokens.getType(), tokens.getQuery()));
-        p.addInputTemplate(MultilinesType, tokens -> tokens.getType() + System.lineSeparator() + tokens.getQuery());
+        p.addInputTemplate(PlainType, tokens -> tokens.getType() + System.lineSeparator() + tokens.getQuery());
 
         p.addTokenMapper(cfg.getToken().getType(), request::getBy);
         p.addTokenMapper(cfg.getToken().getQuery(), request::getSearchTerm);

src/main/java/io/kamax/mxisd/backend/exec/ExecIdentityStore.java

@@ -111,7 +111,7 @@ public class ExecIdentityStore extends ExecStore implements IThreePidProvider {
         p.addTokenMapper(getSingleCfg().getToken().getAddress(), request::getThreePid);
 
         p.addJsonInputTemplate(tokens -> new ThreePid(tokens.getMedium(), tokens.getAddress()));
-        p.addInputTemplate(MultilinesType, tokens -> tokens.getMedium()
+        p.addInputTemplate(PlainType, tokens -> tokens.getMedium()
                 + System.lineSeparator()
                 + tokens.getAddress()
         );
@@ -128,7 +128,7 @@ public class ExecIdentityStore extends ExecStore implements IThreePidProvider {
                     .map(mxId -> new SingleLookupReply(request, mxId));
         });
 
-        p.addSuccessMapper(MultilinesType, output -> {
+        p.addSuccessMapper(PlainType, output -> {
             String[] lines = output.split("\\R");
             if (lines.length > 2) {
                 throw new InternalServerError("Exec auth command returned more than 2 lines (" + lines.length + ")");
@@ -138,7 +138,7 @@ public class ExecIdentityStore extends ExecStore implements IThreePidProvider {
                 return Optional.empty();
             }
 
-            String type = StringUtils.trimToEmpty(lines.length == 1 ? "uid" : lines[0]);
+            String type = StringUtils.trimToEmpty(lines.length == 1 ? UserIdType.Localpart.getId() : lines[0]);
             String value = StringUtils.trimToEmpty(lines.length == 2 ? lines[1] : lines[0]);
 
             if (UserIdType.Localpart.is(type)) {
@@ -168,7 +168,7 @@ public class ExecIdentityStore extends ExecStore implements IThreePidProvider {
                     .collect(Collectors.toList()));
             return GsonUtil.get().toJson(GsonUtil.makeObj("lookup", tpids));
         });
-        p.addInput(MultilinesType, () -> {
+        p.addInput(PlainType, () -> {
             StringBuilder input = new StringBuilder();
             for (ThreePidMapping mapping : mappings) {
                 input.append(mapping.getMedium()).append("\t").append(mapping.getValue()).append(System.lineSeparator());

src/main/java/io/kamax/mxisd/backend/exec/ExecProfileStore.java

@@ -58,7 +58,7 @@ public class ExecProfileStore extends ExecStore implements ProfileProvider {
         Processor<Optional<JsonProfileResult>> p = new Processor<>(cfg);
 
         p.addJsonInputTemplate(tokens -> new JsonProfileRequest(tokens.getLocalpart(), tokens.getDomain(), tokens.getMxid()));
-        p.addInputTemplate(MultilinesType, tokens -> tokens.getLocalpart() + System.lineSeparator()
+        p.addInputTemplate(PlainType, tokens -> tokens.getLocalpart() + System.lineSeparator()
                 + tokens.getDomain() + System.lineSeparator()
                 + tokens.getMxid() + System.lineSeparator()
         );

src/main/java/io/kamax/mxisd/backend/exec/ExecStore.java

@@ -41,7 +41,7 @@ import java.util.stream.Collectors;
 public class ExecStore {
 
     public static final String JsonType = "json";
-    public static final String MultilinesType = "multilines";
+    public static final String PlainType = "plain";
 
     protected static String toJson(Object o) {
         return GsonUtil.get().toJson(o);

src/test/java/io/kamax/mxisd/backend/exec/auth/input/ExecAuthInputMultilinesTest.java

@@ -32,7 +32,7 @@ public class ExecAuthInputMultilinesTest extends ExecAuthStoreTest {
 
     @Override
     protected void setValidInput() {
-        cfg.getAuth().getInput().setType(ExecStore.MultilinesType);
+        cfg.getAuth().getInput().setType(ExecStore.PlainType);
         cfg.getAuth().getInput().setTemplate(null);
     }