Fix hash generation.

docs/MSC2140_MSC2134.md

@@ -17,6 +17,7 @@ NOTE: the v1 is deprecated, therefore recommend to use only v2 and disable v1 (d
 matrix:
   v1: false
 ```
+NOTE: Riot Web version 1.5.5 and below checks the v1 for backward compatibility.
 
 ## Terms
 
@@ -35,13 +36,95 @@ policy:
           url: https://ma1sd.host.tld/term_fr.html  # localized url
       regexp:
         - '/_matrix/identity/v2/account.*'
-        - '/_matrix/identity/v2/hash_lookup'
+        - '/_matrix/identity/v2/hash_details'
+        - '/_matrix/identity/v2/lookup'
 ```
 Where:
 
 - `term_name` -- name of the terms.
-- `regexp` -- regexp patterns for API.
+- `version` -- the terms version.
+- `lang` -- the term language.
+- `name` -- the name of the term.
+- `url` -- the url of the term.
+- `regexp` -- regexp patterns for API which should be available only after accepting the terms.
+
+API will be checks for accepted terms only with authorization.
+There are the next API:
+- [`GET /_matrix/identity/v2/account`](https://matrix.org/docs/spec/identity_service/r0.3.0#get-matrix-identity-v2-account) - Gets information about what user owns the access token used in the request.
+- [`POST /_matrix/identity/v2/account/logout`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-account-logout) - Logs out the access token, preventing it from being used to authenticate future requests to the server.
+- [`GET /_matrix/identity/v2/hash_details`](https://matrix.org/docs/spec/identity_service/r0.3.0#get-matrix-identity-v2-hash-details) - Gets parameters for hashing identifiers from the server. This can include any of the algorithms defined in this specification.
+- [`POST /_matrix/identity/v2/lookup`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-lookup) - Looks up the set of Matrix User IDs which have bound the 3PIDs given, if bindings are available. Note that the format of the addresses is defined later in this specification.
+- [`POST /_matrix/identity/v2/validate/email/requestToken`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-validate-email-requesttoken) - Create a session for validating an email address.
+- [`POST /_matrix/identity/v2/validate/email/submitToken`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-validate-email-submittoken) - Validate ownership of an email address.
+- [`GET /_matrix/identity/v2/validate/email/submitToken`](https://matrix.org/docs/spec/identity_service/r0.3.0#get-matrix-identity-v2-validate-email-submittoken) - Validate ownership of an email address.
+- [`POST /_matrix/identity/v2/validate/msisdn/requestToken`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-validate-msisdn-requesttoken) - Create a session for validating a phone number.
+- [`POST /_matrix/identity/v2/validate/msisdn/submitToken`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-validate-msisdn-submittoken) - Validate ownership of a phone number.
+- [`GET /_matrix/identity/v2/validate/msisdn/submitToken`](https://matrix.org/docs/spec/identity_service/r0.3.0#get-matrix-identity-v2-validate-msisdn-submittoken) - Validate ownership of a phone number.
+- [`GET /_matrix/identity/v2/3pid/getValidated3pid`](https://matrix.org/docs/spec/identity_service/r0.3.0#get-matrix-identity-v2-3pid-getvalidated3pid) - Determines if a given 3pid has been validated by a user.
+- [`POST /_matrix/identity/v2/3pid/bind`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-3pid-bind) - Publish an association between a session and a Matrix user ID.
+- [`POST /_matrix/identity/v2/3pid/unbind`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-3pid-unbind) - Remove an association between a session and a Matrix user ID.
+- [`POST /_matrix/identity/v2/store-invite`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-store-invite) - Store pending invitations to a user's 3pid.
+- [`POST /_matrix/identity/v2/sign-ed25519`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-sign-ed25519) - Sign invitation details.
+
+There is only one exception: [`POST /_matrix/identity/v2/terms`](https://matrix.org/docs/spec/identity_service/r0.3.0#post-matrix-identity-v2-terms) which uses for accepting the terms and requires the authorization.
+
+## [Hash lookup](https://github.com/matrix-org/matrix-doc/blob/hs/hash-identity/proposals/2134-identity-hash-lookup.md)
+
+Hashes and the pepper updates together according to the `rotationPolicy`.
+
+```.yaml
+hashing:
+  enabled: true # enable or disable the hash lookup MSC2140 (default is false)
+  pepperLength: 20 # length of the pepper value (default is 20)
+  rotationPolicy: per_requests # or `per_seconds` how often the hashes will be updating
+  hashStorageType: sql # or `in_memory` where the hashes will be stored
+  algorithms:
+    - none   # the same as v1 bulk lookup
+    - sha256 # hash the 3PID and pepper.
+  delay: 2m # how often hashes will be updated if rotation policy = per_seconds (default is 10s)
+  requests: 10 # how many lookup requests will be performed before updating hashes if rotation policy = per_requests (default is 10)
+```
+
+When enabled and client requests the `none` algorithms then hash lookups works as v1 bulk lookup.
+
+Delay specified in the format: `2d 4h 12m 34s` - this means 2 days 4 hours 12 minutes and 34 seconds. Zero units may be omitted. For example:
+
+- 12s - 12 seconds
+- 3m - 3 minutes
+- 5m 6s - 5 minutes and 6 seconds
+- 6h 3s - 6 hours and 3 seconds
 
 
-## Hash lookup
+Sha256 algorithm supports only sql, memory and exec 3PID providers.
+For sql provider (i.e. for the `synapseSql`):
+```.yaml
+synapseSql:
+  lookup:
+    query: 'select user_id as mxid, medium, address from user_threepids' # query for retrive 3PIDs for hashes.
+```
+
+For general sql provider:
+```.yaml
+sql:
+  lookup:
+    query: 'select user as mxid, field1 as medium, field2 as address from some_table' # query for retrive 3PIDs for hashes.
+```
+
+Each query should return the `mxid`, `medium` and `address` fields.
+
+
+For memory providers:
+```.yaml
+memory:
+  hashEnabled: true # enable the hash lookup (defaults is false)
+```
+
+For exec providers:
+```.yaml
+exec:
+  identity:
+    hashEnabled: true # enable the hash lookup (defaults is false)
+```
+
+NOTE: Federation requests work only with `none` algorithms.
 

gradle/wrapper/gradle-wrapper.properties

@@ -1,5 +1,6 @@
+#Thu Dec 05 22:39:36 MSK 2019
+distributionUrl=https\://services.gradle.org/distributions/gradle-6.0-all.zip
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-6.0-bin.zip
-zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
+zipStoreBase=GRADLE_USER_HOME

ma1sd.example.yaml

@@ -21,6 +21,8 @@
 #
 matrix:
   domain: ''
+  v1: true   # deprecated
+  v2: true   # MSC2140 API v2
 
 
 ################
@@ -109,3 +111,40 @@ threepid:
 
           # Password for the account
           password: "ThePassword"
+
+
+#### MSC2134 (hash lookup)
+
+hashing:
+  enabled: false # enable or disable the hash lookup MSC2140 (default is false)
+  pepperLength: 20 # length of the pepper value (default is 20)
+  rotationPolicy: per_requests # or `per_seconds` how often the hashes will be updating
+  hashStorageType: sql # or `in_memory` where the hashes will be stored
+  algorithms:
+    - none   # the same as v1 bulk lookup
+    - sha256 # hash the 3PID and pepper.
+  delay: 2m # how often hashes will be updated if rotation policy = per_seconds (default is 10s)
+  requests: 10 # how many lookup requests will be performed before updating hashes if rotation policy = per_requests (default is 10)
+
+### hash lookup for synapseSql provider.
+# synapseSql:
+#   lookup:
+#     query: 'select user_id as mxid, medium, address from user_threepids' # query for retrive 3PIDs for hashes.
+
+#### MSC2140 (Terms)
+policy:
+  policies:
+    term_name: # term name
+      version: 1.0 # version
+      terms:
+        en:  # lang
+          name: term name en  # localized name
+          url: https://ma1sd.host.tld/term_en.html  # localized url
+        fe:  # lang
+          name: term name fr  # localized name
+          url: https://ma1sd.host.tld/term_fr.html  # localized url
+      regexp:
+        - '/_matrix/identity/v2/account.*'
+        - '/_matrix/identity/v2/hash_details'
+        - '/_matrix/identity/v2/lookup'
+

src/main/java/io/kamax/mxisd/HttpMxisd.java

@@ -198,8 +198,7 @@ public class HttpMxisd {
 
     private void termsEndpoints(RoutingHandler routingHandler) {
         routingHandler.get(GetTermsHandler.PATH, new GetTermsHandler(m.getConfig().getPolicy()));
-        wrapWithTokenAndAuthorizationHandlers(routingHandler, Methods.POST, sane(new AcceptTermsHandler(m.getAccMgr())),
-            AcceptTermsHandler.PATH, true);
+        routingHandler.post(AcceptTermsHandler.PATH, sane(new AcceptTermsHandler(m.getAccMgr())));
     }
 
     private void hashEndpoints(RoutingHandler routingHandler) {

src/main/java/io/kamax/mxisd/backend/exec/ExecIdentityStore.java

@@ -173,6 +173,10 @@ public class ExecIdentityStore extends ExecStore implements IThreePidProvider {
 
     @Override
     public Iterable<ThreePidMapping> populateHashes() {
+        if (!cfg.isHashLookup()) {
+            return Collections.emptyList();
+        }
+
         Processor<List<ThreePidMapping>> p = new Processor<>();
         p.withConfig(cfg.getLookup().getBulk());
 

src/main/java/io/kamax/mxisd/backend/memory/MemoryIdentityStore.java

@@ -174,6 +174,10 @@ public class MemoryIdentityStore implements AuthenticatorProvider, DirectoryProv
 
     @Override
     public Iterable<ThreePidMapping> populateHashes() {
+        if (!cfg.isHashEnabled()) {
+            return Collections.emptyList();
+        }
+
         return cfg.getIdentities().stream()
             .map(mic -> mic.getThreepids().stream().map(mtp -> new ThreePidMapping(mtp.getMedium(), mtp.getAddress(), mic.getUsername())))
             .flatMap(s -> s).collect(

src/main/java/io/kamax/mxisd/config/DurationDeserializer.java

@@ -0,0 +1,30 @@
+package io.kamax.mxisd.config;
+
+public class DurationDeserializer {
+
+    public long deserialize(String argument) {
+        long duration = 0L;
+        for (String part : argument.split(" ")) {
+            String unit = part.substring(part.length() - 1);
+            long value = Long.parseLong(part.substring(0, part.length() - 1));
+            switch (unit) {
+                case "s":
+                    duration += value;
+                    break;
+                case "m":
+                    duration += value * 60;
+                    break;
+                case "h":
+                    duration += value * 60 * 60;
+                    break;
+                case "d":
+                    duration += value * 60 * 60 * 24;
+                    break;
+                default:
+                    throw new IllegalArgumentException(String.format("Unknown duration unit: %s", unit));
+            }
+        }
+
+        return duration;
+    }
+}

src/main/java/io/kamax/mxisd/config/ExecConfig.java

@@ -309,6 +309,7 @@ public class ExecConfig {
         private Boolean enabled;
         private int priority;
         private Lookup lookup = new Lookup();
+        private boolean hashLookup = false;
 
         public Boolean isEnabled() {
             return enabled;
@@ -334,6 +335,13 @@ public class ExecConfig {
             this.lookup = lookup;
         }
 
+        public boolean isHashLookup() {
+            return hashLookup;
+        }
+
+        public void setHashLookup(boolean hashLookup) {
+            this.hashLookup = hashLookup;
+        }
     }
 
     public static class Profile {

src/main/java/io/kamax/mxisd/config/HashingConfig.java

@@ -14,7 +14,9 @@ public class HashingConfig {
     private int pepperLength = 20;
     private RotationPolicyEnum rotationPolicy;
     private HashStorageEnum hashStorageType;
-    private long delay = 10;
+    private String delay = "10s";
+    private transient long delayInSeconds = 10;
+    private int requests = 10;
     private List<Algorithm> algorithms = new ArrayList<>();
 
     public void build() {
@@ -23,10 +25,15 @@ public class HashingConfig {
             LOGGER.info("   Pepper length: {}", getPepperLength());
             LOGGER.info("   Rotation policy: {}", getRotationPolicy());
             LOGGER.info("   Hash storage type: {}", getHashStorageType());
-            if (RotationPolicyEnum.per_seconds == rotationPolicy) {
-                LOGGER.info("   Rotation delay: {}", delay);
+            if (RotationPolicyEnum.per_seconds == getRotationPolicy()) {
+                setDelayInSeconds(new DurationDeserializer().deserialize(getDelay()));
+                LOGGER.info("   Rotation delay: {}", getDelay());
+                LOGGER.info("   Rotation delay in seconds: {}", getDelayInSeconds());
             }
-            LOGGER.info("   Algorithms: {}", algorithms);
+            if (RotationPolicyEnum.per_requests == getRotationPolicy()) {
+                LOGGER.info("   Rotation after requests: {}", getRequests());
+            }
+            LOGGER.info("   Algorithms: {}", getAlgorithms());
         } else {
             LOGGER.info("Hash configuration disabled, used only `none` pepper.");
         }
@@ -79,14 +86,30 @@ public class HashingConfig {
         this.hashStorageType = hashStorageType;
     }
 
-    public long getDelay() {
+    public String getDelay() {
         return delay;
     }
 
-    public void setDelay(long delay) {
+    public void setDelay(String delay) {
         this.delay = delay;
     }
 
+    public long getDelayInSeconds() {
+        return delayInSeconds;
+    }
+
+    public void setDelayInSeconds(long delayInSeconds) {
+        this.delayInSeconds = delayInSeconds;
+    }
+
+    public int getRequests() {
+        return requests;
+    }
+
+    public void setRequests(int requests) {
+        this.requests = requests;
+    }
+
     public List<Algorithm> getAlgorithms() {
         return algorithms;
     }

src/main/java/io/kamax/mxisd/config/memory/MemoryStoreConfig.java

@@ -27,6 +27,7 @@ public class MemoryStoreConfig {
 
     private boolean enabled;
     private List<MemoryIdentityConfig> identities = new ArrayList<>();
+    private boolean hashEnabled = false;
 
     public boolean isEnabled() {
         return enabled;
@@ -44,6 +45,14 @@ public class MemoryStoreConfig {
         this.identities = identities;
     }
 
+    public boolean isHashEnabled() {
+        return hashEnabled;
+    }
+
+    public void setHashEnabled(boolean hashEnabled) {
+        this.hashEnabled = hashEnabled;
+    }
+
     public void build() {
         // no-op
     }

src/main/java/io/kamax/mxisd/hash/HashEngine.java

@@ -9,6 +9,7 @@ import org.apache.commons.lang3.RandomStringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.util.Base64;
 import java.util.List;
 
 public class HashEngine {
@@ -18,6 +19,7 @@ public class HashEngine {
     private final List<? extends IThreePidProvider> providers;
     private final HashStorage hashStorage;
     private final HashingConfig config;
+    private final Base64.Encoder base64 = Base64.getUrlEncoder().withoutPadding();
     private String pepper;
 
     public HashEngine(List<? extends IThreePidProvider> providers, HashStorage hashStorage, HashingConfig config) {
@@ -51,7 +53,7 @@ public class HashEngine {
     }
 
     protected String hash(ThreePidMapping pidMapping) {
-        return DigestUtils.sha256Hex(pidMapping.getValue() + " " + pidMapping.getMedium() + " " + getPepper());
+        return base64.encodeToString(DigestUtils.sha256(pidMapping.getValue() + " " + pidMapping.getMedium() + " " + getPepper()));
     }
 
     protected String newPepper() {

src/main/java/io/kamax/mxisd/hash/HashManager.java

@@ -58,10 +58,10 @@ public class HashManager {
         if (config.isEnabled()) {
             switch (config.getRotationPolicy()) {
                 case per_requests:
-                    this.rotationStrategy = new RotationPerRequests();
+                    this.rotationStrategy = new RotationPerRequests(config.getRequests());
                     break;
                 case per_seconds:
-                    this.rotationStrategy = new TimeBasedRotation(config.getDelay());
+                    this.rotationStrategy = new TimeBasedRotation(config.getDelayInSeconds());
                     break;
                 default:
                     throw new IllegalArgumentException("Unknown rotation type: " + config.getHashStorageType());

src/main/java/io/kamax/mxisd/hash/rotation/RotationPerRequests.java

@@ -8,6 +8,11 @@ public class RotationPerRequests implements HashRotationStrategy {
 
     private HashEngine hashEngine;
     private final AtomicInteger counter = new AtomicInteger(0);
+    private final int barrier;
+
+    public RotationPerRequests(int barrier) {
+        this.barrier = barrier;
+    }
 
     @Override
     public void register(HashEngine hashEngine) {
@@ -23,7 +28,7 @@ public class RotationPerRequests implements HashRotationStrategy {
     @Override
     public synchronized void newRequest() {
         int newValue = counter.incrementAndGet();
-        if (newValue >= 10) {
+        if (newValue >= barrier) {
             counter.set(0);
             trigger();
         }

src/main/java/io/kamax/mxisd/hash/storage/SqlHashStorage.java

@@ -12,6 +12,7 @@ public class SqlHashStorage implements HashStorage {
 
     public SqlHashStorage(IStorage storage) {
         this.storage = storage;
+        Runtime.getRuntime().addShutdownHook(new Thread(storage::clearHashes));
     }
 
     @Override

src/test/java/io/kamax/mxisd/test/config/DurationDeserializerTest.java

@@ -0,0 +1,20 @@
+package io.kamax.mxisd.test.config;
+
+import static org.junit.Assert.assertEquals;
+
+import io.kamax.mxisd.config.DurationDeserializer;
+import org.junit.Test;
+
+public class DurationDeserializerTest {
+
+    @Test
+    public void durationLoadTest() {
+        DurationDeserializer deserializer = new DurationDeserializer();
+
+        assertEquals(4, deserializer.deserialize("4s"));
+        assertEquals((60 * 60) + 4, deserializer.deserialize("1h 4s"));
+        assertEquals((2 * 60) + 4, deserializer.deserialize("2m 4s"));
+        assertEquals((2 * 60 * 60) + (7 * 60) + 4, deserializer.deserialize("2h 7m 4s"));
+        assertEquals((60 * 60 * 24) + (2 * 60 * 60) + (7 * 60) + 4, deserializer.deserialize("1d 2h 7m 4s"));
+    }
+}

src/test/java/io/kamax/mxisd/test/hash/HashEngineTest.java

@@ -5,11 +5,14 @@ import static org.junit.Assert.assertEquals;
 import org.apache.commons.codec.digest.DigestUtils;
 import org.junit.Test;
 
+import java.util.Base64;
+
 public class HashEngineTest {
 
     @Test
     public void sha256test() {
-        assertEquals("a26de61ae3055f84b33ac1a179b9ad5301f9109024f4db1ae653ea525d2136f4",
-            DigestUtils.sha256Hex("user2@mail.homeserver.tld email I9x4vpcWjqp9X8iiOY4a"));
+        Base64.Encoder encoder = Base64.getUrlEncoder().withoutPadding();
+        assertEquals("rujYzy1w0JxulN_rVlErGUmkdXT5znL0sjSF_IWreko",
+            encoder.encodeToString(DigestUtils.sha256("user@mail.homeserver.tld email I9x4vpcWjqp9X8iiOY4a")));
     }
 }