filtermail: use single SMTP connection to reinject mails

Services are distinguished based on ALPN.
For example,
    openssl s_client -connect example.org:443 -alpn smtp
gives SMTP connection and
    openssl s_client -connect example.org:443 -alpn imap
gives IMAP connection.

.github/workflows/test-and-deploy.yaml

@@ -15,10 +15,14 @@ jobs:
   deploy:
     name: deploy on staging2.testrun.org, and run tests
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     concurrency:
-      group: staging-deploy
-      cancel-in-progress: true
+      group: ci-${{ github.workflow }}-${{ github.ref }}
+      cancel-in-progress: ${{ !contains(github.ref, '$GITHUB_REF') }}
     steps:
+      - uses: jsok/serialize-workflow-action@v1
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
       - uses: actions/checkout@v4
 
       - name: prepare SSH
@@ -72,12 +76,12 @@ jobs:
 
       - run: cmdeploy init staging2.testrun.org
 
-      - run: cmdeploy run
+      - run: cmdeploy run --verbose
 
       - name: set DNS entries
         run: |
-          ssh -o StrictHostKeyChecking=accept-new -v root@staging2.testrun.org chown opendkim:opendkim -R /etc/dkimkeys
-          cmdeploy dns --zonefile staging-generated.zone
+          ssh -o StrictHostKeyChecking=accept-new root@staging2.testrun.org chown opendkim:opendkim -R /etc/dkimkeys
+          cmdeploy dns --zonefile staging-generated.zone --verbose
           cat staging-generated.zone >> .github/workflows/staging.testrun.org-default.zone
           cat .github/workflows/staging.testrun.org-default.zone
           scp .github/workflows/staging.testrun.org-default.zone root@ns.testrun.org:/etc/nsd/staging2.testrun.org.zone
@@ -88,5 +92,5 @@ jobs:
         run: CHATMAIL_DOMAIN2=nine.testrun.org cmdeploy test --slow
 
       - name: cmdeploy dns (try 3 times)
-        run: cmdeploy dns || cmdeploy dns || cmdeploy dns
+        run: cmdeploy dns -v || cmdeploy dns -v || cmdeploy dns -v
 

CHANGELOG.md

@@ -2,6 +2,36 @@
 
 ## untagged
 
+- BREAKING: new required chatmail.ini values:
+
+  mailboxes_dir = /home/vmail/mail/{mail_domain}
+  passdb = /home/vmail/passdb.sqlite
+  
+  reducing hardcoding these two paths all over the files, also improving testability. 
+  ([#351](https://github.com/deltachat/chatmail/pull/351))
+
+- BREAKING: new required chatmail.ini value 'delete_inactive_users_after = 100'
+  which removes users from database and mails after 100 days without any login. 
+  ([#350](https://github.com/deltachat/chatmail/pull/350))
+
+- reload nginx in the acmetool cronjob
+  ([#360](https://github.com/deltachat/chatmail/pull/360))
+
+- remove checking of reverse-DNS PTR records.  Chatmail-servers don't
+  depend on it and even in the wider e-mail system it's not common anymore. 
+  If it's an issue, a chatmail operator can still care to properly set reverse DNS. 
+  ([#348](https://github.com/deltachat/chatmail/pull/348))
+
+- Make DNS-checking faster and more interactive, run it fully during "cmdeploy run",
+  also introducing a generic mechanism for rapid remote ssh-based python function execution. 
+  ([#346](https://github.com/deltachat/chatmail/pull/346))
+
+- Don't fix file owner ship of /home/vmail 
+  ([#345](https://github.com/deltachat/chatmail/pull/345))
+
+- Support iterating over all users with doveadm commands 
+  ([#344](https://github.com/deltachat/chatmail/pull/344))
+
 - Test and fix for attempts to create inadmissible accounts 
   ([#333](https://github.com/deltachat/chatmail/pull/321))
 
@@ -21,6 +51,24 @@
 - filtermail: do not allow ASCII armor without actual payload
   ([#325](https://github.com/deltachat/chatmail/pull/325))
 
+- Remove sieve to enable hardlink deduplication in LMTP
+  ([#343](https://github.com/deltachat/chatmail/pull/343))
+
+- dovecot: enable gzip compression on disk
+  ([#341](https://github.com/deltachat/chatmail/pull/341))
+
+- DKIM-sign Content-Type and oversign all signed headers
+  ([#296](https://github.com/deltachat/chatmail/pull/296))
+
+- Add nonci_accounts metric
+  ([#347](https://github.com/deltachat/chatmail/pull/347))
+
+- doveauth: log when a new account is created
+  ([#349](https://github.com/deltachat/chatmail/pull/349))
+
+- Multiplex HTTPS, IMAP and SMTP on port 443
+  ([#357](https://github.com/deltachat/chatmail/pull/357))
+
 ## 1.3.0 - 2024-06-06
 
 - don't check necessary DNS records on cmdeploy init anymore

README.md

@@ -155,7 +155,8 @@ While this file is present, account creation will be blocked.
 
 [Postfix](http://www.postfix.org/) listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
 [Dovecot](https://www.dovecot.org/) listens on ports 143 (imap) and 993 (imaps).
-[nginx](https://www.nginx.com/) listens on port 443 (https).
+[nginx](https://www.nginx.com/) listens on port 8443 (https-alt) and 443 (https).
+Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
 [acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (http).
 
 Delta Chat apps will, however, discover all ports and configurations

chatmaild/pyproject.toml

@@ -26,6 +26,7 @@ chatmail-metadata = "chatmaild.metadata:main"
 filtermail = "chatmaild.filtermail:main"
 echobot = "chatmaild.echo:main"
 chatmail-metrics = "chatmaild.metrics:main"
+delete_inactive_users = "chatmaild.delete_inactive_users:main"
 
 [project.entry-points.pytest11]
 "chatmaild.testplugin" = "chatmaild.tests.plugin"

chatmaild/src/chatmaild/config.py

@@ -1,9 +1,13 @@
+from pathlib import Path
+
 import iniconfig
 
 
 def read_config(inipath):
+    assert Path(inipath).exists(), inipath
     cfg = iniconfig.IniConfig(inipath)
-    return Config(inipath, params=cfg.sections["params"])
+    params = cfg.sections["params"]
+    return Config(inipath, params=params)
 
 
 class Config:
@@ -13,11 +17,14 @@ class Config:
         self.max_user_send_per_minute = int(params["max_user_send_per_minute"])
         self.max_mailbox_size = params["max_mailbox_size"]
         self.delete_mails_after = params["delete_mails_after"]
+        self.delete_inactive_users_after = int(params["delete_inactive_users_after"])
         self.username_min_length = int(params["username_min_length"])
         self.username_max_length = int(params["username_max_length"])
         self.password_min_length = int(params["password_min_length"])
         self.passthrough_senders = params["passthrough_senders"].split()
         self.passthrough_recipients = params["passthrough_recipients"].split()
+        self.mailboxes_dir = Path(params["mailboxes_dir"].strip())
+        self.passdb_path = Path(params["passdb_path"].strip())
         self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
         self.postfix_reinject_port = int(params["postfix_reinject_port"])
         self.iroh_relay = params.get("iroh_relay")
@@ -29,14 +36,36 @@ class Config:
     def _getbytefile(self):
         return open(self._inipath, "rb")
 
+    def get_user_maildir(self, addr):
+        if addr and addr != "." and "/" not in addr:
+            res = self.mailboxes_dir.joinpath(addr).resolve()
+            if res.is_relative_to(self.mailboxes_dir):
+                return res
+        raise ValueError(f"invalid address {addr!r}")
 
-def write_initial_config(inipath, mail_domain):
+
+def write_initial_config(inipath, mail_domain, overrides):
+    """Write out default config file, using the specified config value overrides."""
     from importlib.resources import files
 
     inidir = files(__package__).joinpath("ini")
-    content = (
-        inidir.joinpath("chatmail.ini.f").read_text().format(mail_domain=mail_domain)
-    )
+    source_inipath = inidir.joinpath("chatmail.ini.f")
+    content = source_inipath.read_text().format(mail_domain=mail_domain)
+
+    # apply config overrides
+    new_lines = []
+    for line in content.split("\n"):
+        new_line = line.strip()
+        if new_line and new_line[0] not in "#[":
+            name, value = map(str.strip, new_line.split("=", maxsplit=1))
+            value = overrides.get(name, value)
+            new_line = f"{name} = {value}"
+        new_lines.append(new_line)
+
+    content = "\n".join(new_lines)
+
+    # apply testrun privacy overrides
+
     if mail_domain.endswith(".testrun.org"):
         override_inipath = inidir.joinpath("override-testrun.ini")
         privacy = iniconfig.IniConfig(override_inipath)["privacy"]

chatmaild/src/chatmaild/delete_inactive_users.py

@@ -0,0 +1,33 @@
+"""
+Remove inactive users
+"""
+
+import shutil
+import sys
+import time
+
+from .config import read_config
+from .database import Database
+from .doveauth import iter_userdb_lastlogin_before
+
+
+def delete_inactive_users(db, config, CHUNK=100):
+    cutoff_date = time.time() - config.delete_inactive_users_after * 86400
+
+    old_users = iter_userdb_lastlogin_before(db, cutoff_date)
+    chunks = (old_users[i : i + CHUNK] for i in range(0, len(old_users), CHUNK))
+    for sublist in chunks:
+        for user in sublist:
+            user_mail_dir = config.get_user_maildir(user)
+            shutil.rmtree(user_mail_dir, ignore_errors=True)
+
+        with db.write_transaction() as conn:
+            for user in sublist:
+                conn.execute("DELETE FROM users WHERE addr = ?", (user,))
+
+
+def main():
+    (cfgpath,) = sys.argv[1:]
+    config = read_config(cfgpath)
+    db = Database(config.passdb_path)
+    delete_inactive_users(db, config)

chatmaild/src/chatmaild/doveauth.py

@@ -68,7 +68,7 @@ def is_allowed_to_create(config: Config, user, cleartext_password) -> bool:
 def get_user_data(db, config: Config, user):
     if user == f"echo@{config.mail_domain}":
         return dict(
-            home=f"/home/vmail/mail/{config.mail_domain}/echo@{config.mail_domain}",
+            home=str(config.get_user_maildir(user)),
             uid="vmail",
             gid="vmail",
         )
@@ -76,7 +76,7 @@ def get_user_data(db, config: Config, user):
     with db.read_connection() as conn:
         result = conn.get_user(user)
     if result:
-        result["home"] = f"/home/vmail/mail/{config.mail_domain}/{user}"
+        result["home"] = str(config.get_user_maildir(user))
         result["uid"] = "vmail"
         result["gid"] = "vmail"
     return result
@@ -86,7 +86,7 @@ def lookup_userdb(db, config: Config, user):
     return get_user_data(db, config, user)
 
 
-def lookup_passdb(db, config: Config, user, cleartext_password):
+def lookup_passdb(db, config: Config, user, cleartext_password, last_login=None):
     if user == f"echo@{config.mail_domain}":
         # Echobot writes password it wants to log in with into /run/echobot/password
         try:
@@ -96,21 +96,25 @@ def lookup_passdb(db, config: Config, user, cleartext_password):
             return None
 
         return dict(
-            home=f"/home/vmail/mail/{config.mail_domain}/echo@{config.mail_domain}",
+            home=str(config.get_user_maildir(user)),
             uid="vmail",
             gid="vmail",
             password=encrypt_password(password),
         )
 
+    if last_login is None:
+        last_login = time.time()
+    last_login = int(last_login)
+
     with db.write_transaction() as conn:
         userdata = conn.get_user(user)
         if userdata:
             # Update last login time.
             conn.execute(
-                "UPDATE users SET last_login=? WHERE addr=?", (int(time.time()), user)
+                "UPDATE users SET last_login=? WHERE addr=?", (last_login, user)
             )
 
-            userdata["home"] = f"/home/vmail/mail/{config.mail_domain}/{user}"
+            userdata["home"] = str(config.get_user_maildir(user))
             userdata["uid"] = "vmail"
             userdata["gid"] = "vmail"
             return userdata
@@ -120,15 +124,34 @@ def lookup_passdb(db, config: Config, user, cleartext_password):
         encrypted_password = encrypt_password(cleartext_password)
         q = """INSERT INTO users (addr, password, last_login)
                VALUES (?, ?, ?)"""
-        conn.execute(q, (user, encrypted_password, int(time.time())))
+        conn.execute(q, (user, encrypted_password, last_login))
+        print(f"Created address: {user}", file=sys.stderr)
         return dict(
-            home=f"/home/vmail/mail/{config.mail_domain}/{user}",
+            home=str(config.get_user_maildir(user)),
             uid="vmail",
             gid="vmail",
             password=encrypted_password,
         )
 
 
+def iter_userdb(db) -> list:
+    """Get a list of all user addresses."""
+    with db.read_connection() as conn:
+        rows = conn.execute(
+            "SELECT addr from users",
+        ).fetchall()
+    return [x[0] for x in rows]
+
+
+def iter_userdb_lastlogin_before(db, cutoff_date):
+    """Get a list of users where last login was before cutoff_date."""
+    with db.read_connection() as conn:
+        rows = conn.execute(
+            "SELECT addr FROM users WHERE last_login < ?", (cutoff_date,)
+        ).fetchall()
+    return [x[0] for x in rows]
+
+
 def split_and_unescape(s):
     """Split strings using double quote as a separator and backslash as escape character
     into parts."""
@@ -192,6 +215,13 @@ def handle_dovecot_request(msg, db, config: Config):
                     reply_command = "N"
         json_res = json.dumps(res) if res else ""
         return f"{reply_command}{json_res}\n"
+    elif short_command == "I":  # ITERATE
+        # example: I0\t0\tshared/userdb/
+        parts = msg[1:].split("\t")
+        if parts[2] == "shared/userdb/":
+            result = "".join(f"Oshared/userdb/{user}\t\n" for user in iter_userdb(db))
+            return f"{result}\n"
+
     raise UnknownCommand(msg)
 
 
@@ -215,9 +245,9 @@ class ThreadedUnixStreamServer(ThreadingMixIn, UnixStreamServer):
 
 
 def main():
-    socket = sys.argv[1]
-    db = Database(sys.argv[2])
-    config = read_config(sys.argv[3])
+    socket, cfgpath = sys.argv[1:]
+    config = read_config(cfgpath)
+    db = Database(config.passdb_path)
 
     class Handler(StreamRequestHandler):
         def handle(self):

chatmaild/src/chatmaild/filtermail.py

@@ -143,12 +143,18 @@ def check_encrypted(message):
 
 async def asyncmain_beforequeue(config):
     port = config.filtermail_smtp_port
-    Controller(BeforeQueueHandler(config), hostname="127.0.0.1", port=port).start()
+    smtp_client = SMTPClient("localhost", config.postfix_reinject_port)
+    Controller(
+        BeforeQueueHandler(config, smtp_client=smtp_client),
+        hostname="127.0.0.1",
+        port=port,
+    ).start()
 
 
 class BeforeQueueHandler:
-    def __init__(self, config):
+    def __init__(self, config, smtp_client):
         self.config = config
+        self.smtp_client = smtp_client
         self.send_rate_limiter = SendRateLimiter()
 
     async def handle_MAIL(self, server, session, envelope, address, mail_options):
@@ -170,8 +176,9 @@ class BeforeQueueHandler:
         if error:
             return error
         logging.info("re-injecting the mail that passed checks")
-        client = SMTPClient("localhost", self.config.postfix_reinject_port)
-        client.sendmail(envelope.mail_from, envelope.rcpt_tos, envelope.content)
+        self.smtp_client.sendmail(
+            envelope.mail_from, envelope.rcpt_tos, envelope.content
+        )
         return "250 OK"
 
     def check_DATA(self, envelope):

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -8,18 +8,21 @@ mail_domain = {mail_domain}
 #
 
 #
-# Account Restrictions
+# Restrictions on user addresses
 #
 
 # how many mails a user can send out per minute
 max_user_send_per_minute = 60
 
-# maximum mailbox size of a chatmail account
+# maximum mailbox size of a chatmail address
 max_mailbox_size = 100M
 
 # days after which mails are unconditionally deleted
 delete_mails_after = 20
 
+# days after which users without a login are deleted (database and mails)
+delete_inactive_users_after = 100
+
 # minimum length a username must have
 username_min_length = 9
 
@@ -29,7 +32,7 @@ username_max_length = 9
 # minimum length a password must have
 password_min_length = 9
 
-# list of chatmail accounts which can send outbound un-encrypted mail
+# list of chatmail addresses which can send outbound un-encrypted mail
 passthrough_senders =
 
 # list of e-mail recipients for which to accept outbound un-encrypted mails
@@ -39,6 +42,12 @@ passthrough_recipients = xstore@testrun.org groupsbot@hispanilandia.net
 # Deployment Details
 #
 
+# Directory where user mailboxes are stored
+mailboxes_dir = /home/vmail/mail/{mail_domain}
+
+# user address sqlite database path
+passdb_path = /home/vmail/passdb.sqlite
+
 # where the filtermail SMTP service listens
 filtermail_smtp_port = 10080
 
@@ -60,4 +69,3 @@ privacy_pdo =
 
 # postal address of the privacy supervisor
 privacy_supervisor =
-

chatmaild/src/chatmaild/metadata.py

@@ -1,7 +1,6 @@
 import logging
 import os
 import sys
-from pathlib import Path
 from socketserver import (
     StreamRequestHandler,
     ThreadingMixIn,
@@ -128,12 +127,12 @@ class ThreadedUnixStreamServer(ThreadingMixIn, UnixStreamServer):
 
 
 def main():
-    socket, vmail_dir, config_path = sys.argv[1:]
+    socket, config_path = sys.argv[1:]
 
     config = read_config(config_path)
     iroh_relay = config.iroh_relay
 
-    vmail_dir = Path(vmail_dir)
+    vmail_dir = config.mailboxes_dir
     if not vmail_dir.exists():
         logging.error("vmail dir does not exist: %r", vmail_dir)
         return 1

chatmaild/src/chatmaild/metrics.py

@@ -1,6 +1,5 @@
 #!/usr/bin/env python3
 import sys
-import time
 from pathlib import Path
 
 
@@ -16,9 +15,15 @@ def main(vmail_dir=None):
         if path.name[:3] in ("ci-", "ac_"):
             ci_accounts += 1
 
-    timestamp = int(time.time() * 1000)
-    print(f"accounts {accounts} {timestamp}")
-    print(f"ci_accounts {ci_accounts} {timestamp}")
+    print("# HELP total number of accounts")
+    print("# TYPE accounts gauge")
+    print(f"accounts {accounts}")
+    print("# HELP number of CI accounts")
+    print("# TYPE ci_accounts gauge")
+    print(f"ci_accounts {ci_accounts}")
+    print("# HELP number of non-CI accounts")
+    print("# TYPE nonci_accounts gauge")
+    print(f"nonci_accounts {accounts - ci_accounts}")
 
 
 if __name__ == "__main__":

chatmaild/src/chatmaild/tests/plugin.py

@@ -16,7 +16,11 @@ def make_config(tmp_path):
     inipath = tmp_path.joinpath("chatmail.ini")
 
     def make_conf(mail_domain):
-        write_initial_config(inipath, mail_domain=mail_domain)
+        basedir = tmp_path.joinpath(f"vmail/{mail_domain}")
+        basedir.mkdir(parents=True, exist_ok=True)
+        passdb = tmp_path.joinpath("vmail/passdb.sqlite")
+        overrides = dict(mailboxes_dir=str(basedir), passdb_path=str(passdb))
+        write_initial_config(inipath, mail_domain, overrides=overrides)
         return read_config(inipath)
 
     return make_conf

chatmaild/src/chatmaild/tests/test_config.py

@@ -1,3 +1,4 @@
+import pytest
 from chatmaild.config import read_config
 
 
@@ -30,3 +31,31 @@ def test_read_config_testrun(make_config):
     assert config.password_min_length == 9
     assert "privacy@testrun.org" in config.passthrough_recipients
     assert config.passthrough_senders == []
+
+
+def test_config_userstate_paths(make_config, tmp_path):
+    config = make_config("something.testrun.org")
+    mailboxes_dir = config.mailboxes_dir
+    passdb_path = config.passdb_path
+    assert mailboxes_dir.name == "something.testrun.org"
+    assert passdb_path.name == "passdb.sqlite"
+    assert passdb_path.is_relative_to(tmp_path)
+    assert config.mail_domain == "something.testrun.org"
+    path = config.get_user_maildir("user1@something.testrun.org")
+    assert not path.exists()
+    assert path == mailboxes_dir.joinpath("user1@something.testrun.org")
+
+    with pytest.raises(ValueError):
+        config.get_user_maildir("")
+
+    with pytest.raises(ValueError):
+        config.get_user_maildir(None)
+
+    with pytest.raises(ValueError):
+        config.get_user_maildir("../some@something.testrun.org")
+
+    with pytest.raises(ValueError):
+        config.get_user_maildir("..")
+
+    with pytest.raises(ValueError):
+        config.get_user_maildir(".")

chatmaild/src/chatmaild/tests/test_delete_inactive_users.py

@@ -0,0 +1,51 @@
+import time
+
+from chatmaild.delete_inactive_users import delete_inactive_users
+from chatmaild.doveauth import lookup_passdb
+
+
+def test_remove_stale_users(db, example_config):
+    new = time.time()
+    old = new - (example_config.delete_inactive_users_after * 86400) - 1
+
+    def create_user(addr, last_login):
+        lookup_passdb(db, example_config, addr, "q9mr3faue", last_login=last_login)
+        md = example_config.get_user_maildir(addr)
+        md.mkdir(parents=True)
+        md.joinpath("cur").mkdir()
+        md.joinpath("cur", "something").mkdir()
+
+    # create some stale and some new accounts
+    to_remove = []
+    for i in range(150):
+        addr = f"oldold{i:03}@chat.example.org"
+        create_user(addr, last_login=old)
+        with db.read_connection() as conn:
+            assert conn.get_user(addr)
+        to_remove.append(addr)
+
+    remain = []
+    for i in range(5):
+        addr = f"newnew{i:03}@chat.example.org"
+        create_user(addr, last_login=new)
+        remain.append(addr)
+
+    # check pre and post-conditions for delete_inactive_users()
+
+    for addr in to_remove:
+        assert example_config.get_user_maildir(addr).exists()
+
+    delete_inactive_users(db, example_config)
+
+    for p in example_config.mailboxes_dir.iterdir():
+        assert not p.name.startswith("old")
+
+    for addr in to_remove:
+        assert not example_config.get_user_maildir(addr).exists()
+        with db.read_connection() as conn:
+            assert not conn.get_user(addr)
+
+    for addr in remain:
+        assert example_config.get_user_maildir(addr).exists()
+        with db.read_connection() as conn:
+            assert conn.get_user(addr)

chatmaild/src/chatmaild/tests/test_doveauth.py

@@ -12,6 +12,8 @@ from chatmaild.doveauth import (
     handle_dovecot_protocol,
     handle_dovecot_request,
     is_allowed_to_create,
+    iter_userdb,
+    iter_userdb_lastlogin_before,
     lookup_passdb,
 )
 from chatmaild.newemail import create_newemail_dict
@@ -27,6 +29,35 @@ def test_basic(db, example_config):
     assert data == data2
 
 
+def test_iterate_addresses(db, example_config):
+    addresses = []
+
+    for i in range(10):
+        addresses.append(f"asdf1234{i}@chat.example.org")
+        lookup_passdb(db, example_config, addresses[-1], "q9mr3faue")
+    res = iter_userdb(db)
+    assert res == addresses
+
+
+def test_iterate_addresses_lastlogin_before(db, example_config):
+    addresses = []
+
+    cutoff_date = 1000
+    for i in range(10):
+        addr = f"oldold{i:03}@chat.example.org"
+        lookup_passdb(
+            db, example_config, addr, "q9mr3faue", last_login=cutoff_date - 10
+        )
+        addresses.append(addr)
+
+    for i in range(5):
+        addr = f"newnew{i:03}@chat.example.org"
+        lookup_passdb(db, example_config, addr, "q9mr3faue", last_login=cutoff_date + i)
+
+    res = iter_userdb_lastlogin_before(db, cutoff_date)
+    assert sorted(res) == sorted(addresses)
+
+
 def test_invalid_username_length(example_config):
     config = example_config
     config.username_min_length = 6
@@ -83,10 +114,7 @@ def test_handle_dovecot_request(db, example_config):
     assert res
     assert res[0] == "O" and res.endswith("\n")
     userdata = json.loads(res[1:].strip())
-    assert (
-        userdata["home"]
-        == "/home/vmail/mail/chat.example.org/some42123@chat.example.org"
-    )
+    assert userdata["home"].endswith("chat.example.org/some42123@chat.example.org")
     assert userdata["uid"] == userdata["gid"] == "vmail"
     assert userdata["password"].startswith("{SHA512-CRYPT}")
 
@@ -108,6 +136,18 @@ def test_handle_dovecot_protocol(db, example_config):
     assert wfile.getvalue() == b"N\n"
 
 
+def test_handle_dovecot_protocol_iterate(db, gencreds, example_config):
+    lookup_passdb(db, example_config, "asdf00000@chat.example.org", "q9mr3faue")
+    lookup_passdb(db, example_config, "asdf11111@chat.example.org", "q9mr3faue")
+    rfile = io.BytesIO(b"H3\t2\t0\t\tauth\nI0\t0\tshared/userdb/")
+    wfile = io.BytesIO()
+    handle_dovecot_protocol(rfile, wfile, db, example_config)
+    lines = wfile.getvalue().decode("ascii").split("\n")
+    assert lines[0] == "Oshared/userdb/asdf00000@chat.example.org\t"
+    assert lines[1] == "Oshared/userdb/asdf11111@chat.example.org\t"
+    assert not lines[2]
+
+
 def test_50_concurrent_lookups_different_accounts(db, gencreds, example_config):
     num_threads = 50
     req_per_thread = 5

chatmaild/src/chatmaild/tests/test_metrics.py

@@ -8,9 +8,10 @@ def test_main(tmp_path, capsys):
     out, _ = capsys.readouterr()
     d = {}
     for line in out.split("\n"):
-        if line.strip():
-            name, num, _ = line.split()
+        if line.strip() and not line.startswith("#"):
+            name, num = line.split()
             d[name] = int(num)
 
     assert d["accounts"] == 4
     assert d["ci_accounts"] == 3
+    assert d["nonci_accounts"] == 1

cmdeploy/pyproject.toml

@@ -18,6 +18,7 @@ dependencies = [
   "ruff",
   "pytest",
   "pytest-xdist", 
+  "execnet",
   "imap_tools",
 ]
 

cmdeploy/src/cmdeploy/__init__.py

@@ -92,7 +92,7 @@ def _install_remote_venv_with_chatmaild(config) -> None:
         group="root",
         mode="644",
         config={
-            "mail_domain": config.mail_domain,
+            "mailboxes_dir": config.mailboxes_dir,
             "execpath": f"{remote_venv_dir}/bin/chatmail-metrics",
         },
     )
@@ -338,20 +338,6 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
     )
     need_restart |= lua_push_notification_script.changed
 
-    sieve_script = files.put(
-        src=importlib.resources.files(__package__).joinpath("dovecot/default.sieve"),
-        dest="/etc/dovecot/default.sieve",
-        user="root",
-        group="root",
-        mode="644",
-    )
-    need_restart |= sieve_script.changed
-    if sieve_script.changed:
-        server.shell(
-            name="compile sieve script",
-            commands=["/usr/bin/sievec /etc/dovecot/default.sieve"],
-        )
-
     files.template(
         src=importlib.resources.files(__package__).joinpath("dovecot/expunge.cron.j2"),
         dest="/etc/cron.d/expunge",
@@ -361,14 +347,6 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
         config=config,
     )
 
-    files.put(
-            src=importlib.resources.files(__package__).joinpath("dovecot/remove-seen.py"),
-            dest="/usr/local/bin/remove-seen.py",
-            user="root",
-            group="root",
-            mode="755"
-    )
-
     # as per https://doc.dovecot.org/configuration_manual/os/
     # it is recommended to set the following inotify limits
     for name in ("max_user_instances", "max_user_watches"):
@@ -487,11 +465,6 @@ def deploy_chatmail(config_path: Path) -> None:
     )
     server.user(name="Create echobot user", user="echobot", system=True)
 
-    server.shell(
-        name="Fix file owner in /home/vmail",
-        commands=["test -d /home/vmail && chown -R vmail:vmail /home/vmail"],
-    )
-
     # Add our OBS repository for dovecot_no_delay
     files.put(
         name="Add Deltachat OBS GPG key to apt keyring",
@@ -556,12 +529,12 @@ def deploy_chatmail(config_path: Path) -> None:
 
     apt.packages(
         name="Install Dovecot",
-        packages=["dovecot-imapd", "dovecot-lmtpd", "dovecot-sieve"],
+        packages=["dovecot-imapd", "dovecot-lmtpd"],
     )
 
     apt.packages(
         name="Install nginx",
-        packages=["nginx"],
+        packages=["nginx", "libnginx-mod-stream"],
     )
 
     apt.packages(
@@ -657,5 +630,3 @@ def deploy_chatmail(config_path: Path) -> None:
         name="Ensure cron is installed",
         packages=["cron"],
     )
-
-

cmdeploy/src/cmdeploy/acmetool/acmetool.cron

@@ -1,4 +1,4 @@
 SHELL=/bin/sh
 PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
 MAILTO=root
-20 16 * * * root /usr/bin/acmetool --batch reconcile && systemctl reload dovecot && systemctl reload postfix
+20 16 * * * root /usr/bin/acmetool --batch reconcile && systemctl reload dovecot && systemctl reload postfix && systemctl reload nginx

cmdeploy/src/cmdeploy/cmdeploy.py

@@ -15,7 +15,9 @@ from pathlib import Path
 from chatmaild.config import read_config, write_initial_config
 from termcolor import colored
 
-from cmdeploy.dns import check_necessary_dns, show_dns
+from . import remote_funcs
+from .dns import show_dns
+from .sshexec import SSHExec
 
 #
 # cmdeploy sub commands and options
@@ -35,8 +37,9 @@ def init_cmd(args, out):
     mail_domain = args.chatmail_domain
     if args.inipath.exists():
         print(f"Path exists, not modifying: {args.inipath}")
+        return 1
     else:
-        write_initial_config(args.inipath, mail_domain)
+        write_initial_config(args.inipath, mail_domain, overrides={})
         out.green(f"created config file for {mail_domain} in {args.inipath}")
 
 
@@ -51,12 +54,7 @@ def run_cmd_options(parser):
 
 def run_cmd(args, out):
     """Deploy chatmail services on the remote server."""
-    mail_domain = args.config.mail_domain
-    if not check_necessary_dns(
-        out,
-        mail_domain,
-    ):
-        sys.exit(1)
+    retcode, remote_data = show_dns(args, out)
 
     env = os.environ.copy()
     env["CHATMAIL_INI"] = args.inipath
@@ -65,7 +63,15 @@ def run_cmd(args, out):
     cmd = f"{pyinf} --ssh-user root {args.config.mail_domain} {deploy_path}"
 
     out.check_call(cmd, env=env)
-    print("Deploy completed, call `cmdeploy dns` next.")
+    if retcode == 0:
+        out.green("Deploy completed, call `cmdeploy test` next.")
+    elif not remote_data["acme_account_url"]:
+        out.red("Deploy completed but letsencrypt not configured")
+        out.red("Run 'cmdeploy dns' or 'cmdeploy run' again")
+        retcode = 0
+    else:
+        out.red("Deploy failed")
+    return retcode
 
 
 def dns_cmd_options(parser):
@@ -77,15 +83,15 @@ def dns_cmd_options(parser):
 
 
 def dns_cmd(args, out):
-    """Generate dns zone file."""
-    exit_code = show_dns(args, out)
-    exit(exit_code)
+    """Check DNS entries and optionally generate dns zone file."""
+    retcode, remote_data = show_dns(args, out)
+    return retcode
 
 
 def status_cmd(args, out):
     """Display status for online chatmail instance."""
 
-    ssh = f"ssh root@{args.config.mail_domain}"
+    sshexec = args.get_sshexec()
 
     out.green(f"chatmail domain: {args.config.mail_domain}")
     if args.config.privacy_mail:
@@ -93,10 +99,8 @@ def status_cmd(args, out):
     else:
         out.red("no privacy settings")
 
-    s1 = "systemctl --type=service --state=running"
-    for line in out.shell_output(f"{ssh} -- {s1}").split("\n"):
-        if line.startswith("  "):
-            print(line)
+    for line in sshexec(remote_funcs.get_systemd_running):
+        print(line)
 
 
 def test_cmd_options(parser):
@@ -125,7 +129,7 @@ def test_cmd(args, out):
         "-n4",
         "-rs",
         "-x",
-        "-vrx",
+        "-v",
         "--durations=5",
     ]
     if args.slow:
@@ -135,14 +139,6 @@ def test_cmd(args, out):
 
 
 def fmt_cmd_options(parser):
-    parser.add_argument(
-        "--verbose",
-        "-v",
-        dest="verbose",
-        action="store_true",
-        help="provide information on invocations",
-    )
-
     parser.add_argument(
         "--check",
         "-c",
@@ -172,7 +168,6 @@ def fmt_cmd(args, out):
 
     out.check_call(" ".join(format_args), quiet=not args.verbose)
     out.check_call(" ".join(check_args), quiet=not args.verbose)
-    return 0
 
 
 def bench_cmd(args, out):
@@ -208,16 +203,6 @@ class Out:
         color = "red" if red else ("green" if green else None)
         print(colored(msg, color), file=file)
 
-    def shell_output(self, arg, no_print=False, timeout=10):
-        if not no_print:
-            self(f"[$ {arg}]", file=sys.stderr)
-            output = subprocess.STDOUT
-        else:
-            output = subprocess.DEVNULL
-        return subprocess.check_output(
-            arg, shell=True, timeout=timeout, stderr=output
-        ).decode()
-
     def check_call(self, arg, env=None, quiet=False):
         if not quiet:
             self(f"[$ {arg}]", file=sys.stderr)
@@ -240,6 +225,14 @@ def add_config_option(parser):
         type=Path,
         help="path to the chatmail.ini file",
     )
+    parser.add_argument(
+        "--verbose",
+        "-v",
+        dest="verbose",
+        action="store_true",
+        default=False,
+        help="provide verbose logging",
+    )
 
 
 def add_subcommand(subparsers, func):
@@ -279,11 +272,18 @@ def get_parser():
 
 
 def main(args=None):
-    """Provide main entry point for 'xdcget' CLI invocation."""
+    """Provide main entry point for 'cmdeploy' CLI invocation."""
     parser = get_parser()
     args = parser.parse_args(args=args)
     if not hasattr(args, "func"):
         return parser.parse_args(["-h"])
+
+    def get_sshexec(log=None):
+        print(f"[ssh] login to {args.config.mail_domain}")
+        return SSHExec(args.config.mail_domain, remote_funcs, log=log)
+
+    args.get_sshexec = get_sshexec
+
     out = Out()
     kwargs = {}
     if args.func.__name__ not in ("init_cmd", "fmt_cmd"):

cmdeploy/src/cmdeploy/dns.py

@@ -1,209 +1,59 @@
 import datetime
 import importlib
-import subprocess
 import sys
 
-import requests
-
-
-class DNS:
-    def __init__(self, out, mail_domain):
-        self.session = requests.Session()
-        self.out = out
-        self.ssh = f"ssh root@{mail_domain} -- "
-        self.out.shell_output(
-            f"{ self.ssh }'apt-get update && apt-get install -y dnsutils'",
-            timeout=60,
-            no_print=True,
-        )
-        try:
-            self.shell(f"unbound-control flush_zone {mail_domain}")
-        except subprocess.CalledProcessError:
-            pass
-
-    def shell(self, cmd):
-        try:
-            return self.out.shell_output(f"{self.ssh}{cmd}", no_print=True)
-        except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
-            if "exit status 255" in str(e) or "timed out" in str(e):
-                self.out.red(f"Error: can't reach the server with: {self.ssh[:-4]}")
-                sys.exit(1)
-            else:
-                raise
-
-    def get_ipv4(self):
-        cmd = "ip a | grep 'inet ' | grep 'scope global' | grep -oE '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}' | head -1"
-        return self.shell(cmd).strip()
-
-    def get_ipv6(self):
-        cmd = "ip a | grep inet6 | grep 'scope global' | sed -e 's#/64 scope global##' | sed -e 's#inet6##'"
-        return self.shell(cmd).strip()
-
-    def get(self, typ: str, domain: str) -> str:
-        """Get a DNS entry or empty string if there is none."""
-        dig_result = self.shell(f"dig -r -q {domain} -t {typ} +short")
-        line = dig_result.partition("\n")[0]
-        return line
-
-    def check_ptr_record(self, ip: str, mail_domain) -> bool:
-        """Check the PTR record for an IPv4 or IPv6 address."""
-        result = self.shell(f"dig -r -x {ip} +short").rstrip()
-        return result == f"{mail_domain}."
+from . import remote_funcs
 
 
 def show_dns(args, out) -> int:
-    """Check existing DNS records, optionally write them to zone file, return exit code 0 or 1."""
+    """Check existing DNS records, optionally write them to zone file
+    and return (exitcode, remote_data) tuple."""
     template = importlib.resources.files(__package__).joinpath("chatmail.zone.f")
     mail_domain = args.config.mail_domain
-    ssh = f"ssh root@{mail_domain}"
-    dns = DNS(out, mail_domain)
 
-    print("Checking your DKIM keys and DNS entries...")
-    try:
-        acme_account_url = out.shell_output(f"{ssh} -- acmetool account-url")
-    except subprocess.CalledProcessError:
-        print("Please run `cmdeploy run` first.")
-        return 1
+    def log_progress(data):
+        sys.stdout.write(".")
+        sys.stdout.flush()
 
-    dkim_selector = "opendkim"
-    dkim_pubkey = out.shell_output(
-        ssh + f" -- openssl rsa -in /etc/dkimkeys/{dkim_selector}.private"
-        " -pubout 2>/dev/null | awk '/-/{next}{printf(\"%s\",$0)}'"
-    )
-    dkim_entry_value = f"v=DKIM1;k=rsa;p={dkim_pubkey};s=email;t=s"
-    dkim_entry_str = ""
-    while len(dkim_entry_value) >= 255:
-        dkim_entry_str += '"' + dkim_entry_value[:255] + '" '
-        dkim_entry_value = dkim_entry_value[255:]
-    dkim_entry_str += '"' + dkim_entry_value + '"'
-    dkim_entry = f"{dkim_selector}._domainkey.{mail_domain}. TXT {dkim_entry_str}"
+    sshexec = args.get_sshexec(log=print if args.verbose else log_progress)
+    print("Checking DNS entries ", end="\n" if args.verbose else "")
 
-    ipv6 = dns.get_ipv6()
-    reverse_ipv6 = dns.check_ptr_record(ipv6, mail_domain)
-    ipv4 = dns.get_ipv4()
-    reverse_ipv4 = dns.check_ptr_record(ipv4, mail_domain)
-    to_print = []
+    remote_data = sshexec(remote_funcs.perform_initial_checks, mail_domain=mail_domain)
+
+    assert remote_data["ipv4"] or remote_data["ipv6"]
 
     with open(template, "r") as f:
-        zonefile = (
-            f.read()
-            .format(
-                acme_account_url=acme_account_url,
-                sts_id=datetime.datetime.now().strftime("%Y%m%d%H%M"),
-                chatmail_domain=args.config.mail_domain,
-                dkim_entry=dkim_entry,
-                ipv6=ipv6,
-                ipv4=ipv4,
-            )
-            .strip()
+        zonefile = f.read().format(
+            acme_account_url=remote_data["acme_account_url"],
+            dkim_entry=remote_data["dkim_entry"],
+            ipv6=remote_data["ipv6"],
+            ipv4=remote_data["ipv4"],
+            sts_id=datetime.datetime.now().strftime("%Y%m%d%H%M"),
+            chatmail_domain=args.config.mail_domain,
         )
-        try:
-            with open(args.zonefile, "w+") as zf:
-                zf.write(zonefile)
-                print(f"DNS records successfully written to: {args.zonefile}")
-            return 0
-        except TypeError:
-            pass
-        for raw_line in zonefile.splitlines():
-            line = raw_line.format(
-                acme_account_url=acme_account_url,
-                sts_id=datetime.datetime.now().strftime("%Y%m%d%H%M"),
-                chatmail_domain=args.config.mail_domain,
-                dkim_entry=dkim_entry,
-                ipv6=ipv6,
-            ).strip()
-            for typ in ["A", "AAAA", "CNAME", "CAA"]:
-                if f" {typ} " in line:
-                    domain, value = line.split(f" {typ} ")
-                    current = dns.get(typ, domain.strip()[:-1])
-                    if current != value.strip():
-                        to_print.append(line)
-            if " MX " in line:
-                domain, typ, prio, value = line.split()
-                current = dns.get(typ, domain[:-1])
-                if not current:
-                    to_print.append(line)
-                elif current.split()[1] != value:
-                    print(line.replace(prio, str(int(current[0]) + 1)))
-            if " SRV " in line:
-                domain, typ, prio, weight, port, value = line.split()
-                current = dns.get("SRV", domain[:-1])
-                if current != f"{prio} {weight} {port} {value}":
-                    to_print.append(line)
-            if " TXT " in line:
-                domain, value = line.split(" TXT ")
-                current = dns.get("TXT", domain.strip()[:-1])
-                if domain.startswith("_mta-sts."):
-                    if current:
-                        if current.split("id=")[0] == value.split("id=")[0]:
-                            continue
 
-                # TXT records longer than 255 bytes
-                # are split into multiple <character-string>s.
-                # This typically happens with DKIM record
-                # which contains long RSA key.
-                #
-                # Removing `" "` before comparison
-                # to get back a single string.
-                if current.replace('" "', "") != value.replace('" "', ""):
-                    to_print.append(line)
+    to_print = sshexec(remote_funcs.check_zonefile, zonefile=zonefile)
+    if not args.verbose:
+        print()
+
+    if getattr(args, "zonefile", None):
+        with open(args.zonefile, "w+") as zf:
+            zf.write(zonefile)
+        out.green(f"DNS records successfully written to: {args.zonefile}")
+        return 0, remote_data
 
-    exit_code = 0
     if to_print:
         to_print.insert(
-            0, "You should configure the following DNS entries at your provider:\n"
+            0, "You should configure the following entries at your DNS provider:\n"
         )
         to_print.append(
-            "\nIf you already configured the DNS entries, wait a bit until the DNS entries propagate to the Internet."
+            "\nIf you already configured the DNS entries, "
+            "wait a bit until the DNS entries propagate to the Internet."
         )
-        print("\n".join(to_print))
+        out.red("\n".join(to_print))
         exit_code = 1
     else:
-        out.green("Great! All your DNS entries are correct.")
+        out.green("Great! All your DNS entries are verified and correct.")
+        exit_code = 0
 
-    to_print = []
-    if not reverse_ipv4:
-        to_print.append(f"\tIPv4:\t{ipv4}\t{args.config.mail_domain}")
-    if not reverse_ipv6:
-        to_print.append(f"\tIPv6:\t{ipv6}\t{args.config.mail_domain}")
-    if len(to_print) > 0:
-        if len(to_print) == 1:
-            warning = "You should add the following PTR/reverse DNS entry:"
-        else:
-            warning = "You should add the following PTR/reverse DNS entries:"
-        out.red(warning)
-        for entry in to_print:
-            print(entry)
-        print(
-            "You can do so at your hosting provider (maybe this isn't your DNS provider)."
-        )
-        exit_code = 1
-    return exit_code
-
-
-def check_necessary_dns(out, mail_domain):
-    """Check whether $mail_domain and mta-sts.$mail_domain resolve."""
-    print("Checking necessary DNS records... ")
-    dns = DNS(out, mail_domain)
-    ipv4 = dns.get("A", mail_domain)
-    ipv6 = dns.get("AAAA", mail_domain)
-    mta_entry = dns.get("CNAME", "mta-sts." + mail_domain)
-    www_entry = dns.get("CNAME", "www." + mail_domain)
-    to_print = []
-    if not (ipv4 or ipv6):
-        to_print.append(f"\t{mail_domain}.\t\t\tA<your server's IPv4 address>")
-    if mta_entry != mail_domain + ".":
-        to_print.append(f"\tmta-sts.{mail_domain}.\tCNAME\t{mail_domain}.")
-    if www_entry != mail_domain + ".":
-        to_print.append(f"\twww.{mail_domain}.\tCNAME\t{mail_domain}.")
-    if to_print:
-        to_print.insert(
-            0,
-            "\nFor chatmail to work, you need to configure this at your DNS provider:\n",
-        )
-        for line in to_print:
-            print(line)
-        print()
-    else:
-        dns.out.green("All necessary DNS records seem to be set.")
-        return True
+    return exit_code, remote_data

cmdeploy/src/cmdeploy/dovecot/auth.conf

@@ -1,5 +1,7 @@
 uri = proxy:/run/doveauth/doveauth.socket:auth
-iterate_disable = yes
+iterate_disable = no
+iterate_prefix = userdb/
+
 default_pass_scheme = plain
 # %E escapes characters " (double quote), ' (single quote) and \ (backslash) with \ (backslash).
 # See <https://doc.dovecot.org/configuration_manual/config_file/config_variables/#modifiers>

cmdeploy/src/cmdeploy/dovecot/default.sieve

@@ -1,7 +0,0 @@
-require ["imap4flags"];
-
-# flag the message so it doesn't cause a push notification
-
-if header :is ["Auto-Submitted"] ["auto-replied", "auto-generated"] {
-	addflag "$Auto";
-}

cmdeploy/src/cmdeploy/dovecot/dovecot.conf.j2

@@ -38,7 +38,14 @@ service imap {
 mail_server_admin = mailto:root@{{ config.mail_domain }}
 mail_server_comment = Chatmail server
 
-mail_plugins = quota
+# `zlib` enables compressing messages stored in the maildir.
+# See
+# <https://doc.dovecot.org/configuration_manual/zlib_plugin/>
+# for documentation.
+#
+# quota plugin documentation:
+# <https://doc.dovecot.org/configuration_manual/quota_plugin/>
+mail_plugins = zlib quota
 
 # these are the capabilities Delta Chat cares about actually 
 # so let's keep the network overhead per login small
@@ -60,7 +67,7 @@ userdb {
 ##
 
 # Mailboxes are stored in the "mail" directory of the vmail user home.
-mail_location = maildir:/home/vmail/mail/%d/%u
+mail_location = maildir:{{ config.mailboxes_dir }}/%u
 
 namespace inbox {
   inbox = yes
@@ -96,7 +103,7 @@ mail_privileged_group = vmail
 # Pass all IMAP METADATA requests to the server implementing Dovecot's dict protocol.
 mail_attribute_dict = proxy:/run/chatmail-metadata/metadata.socket:metadata
 
-# Enable IMAP COMPRESS (RFC 4978).
+# `imap_zlib` enables IMAP COMPRESS (RFC 4978).
 # <https://datatracker.ietf.org/doc/html/rfc4978.html>
 protocol imap {
   mail_plugins = $mail_plugins imap_zlib imap_quota
@@ -104,9 +111,6 @@ protocol imap {
 }
 
 protocol lmtp {
-  # quota plugin documentation:
-  # <https://doc.dovecot.org/configuration_manual/quota_plugin/>
-  #
   # notify plugin is a dependency of push_notification plugin:
   # <https://doc.dovecot.org/settings/plugin/notify-plugin/>
   #
@@ -115,10 +119,11 @@ protocol lmtp {
   #
   # mail_lua and push_notification_lua are needed for Lua push notification handler.
   # <https://doc.dovecot.org/configuration_manual/push_notification/#configuration>
-  #
-  # Sieve to mark messages that should not be notified as \Seen
-  # <https://doc.dovecot.org/configuration_manual/sieve/configuration/>
-  mail_plugins = $mail_plugins quota mail_lua notify push_notification push_notification_lua sieve
+  mail_plugins = $mail_plugins mail_lua notify push_notification push_notification_lua
+}
+
+plugin {
+  zlib_save = gz
 }
 
 plugin {
@@ -140,10 +145,6 @@ plugin {
   push_notification_driver = lua:file=/etc/dovecot/push_notification.lua
 }
 
-plugin {
-  sieve_default = file:/etc/dovecot/default.sieve
-}
-
 service lmtp {
   user=vmail
 

cmdeploy/src/cmdeploy/dovecot/expunge.cron.j2

@@ -1,12 +1,12 @@
 # delete all mails after {{ config.delete_mails_after }} days, in the Inbox
-2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
 # or in any IMAP subfolder
-2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/.*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
 # even if they are unseen
-2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
-2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/.*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
 # or only temporary (but then they shouldn't be around after {{ config.delete_mails_after }} days anyway).
-2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
-2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/.*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
-3 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -name 'maildirsize' -type f -delete
-4 0 * * * vmail /usr/local/bin/remove-seen.py /home/vmail/mail/{{ config.mail_domain }}
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+3 0 * * * vmail find {{ config.mailboxes_dir }} -name 'maildirsize' -type f -delete
+4 0 * * * vmail /usr/local/lib/chatmaild/venv/bin/delete_inactive_users /usr/local/lib/chatmaild/chatmail.ini 

cmdeploy/src/cmdeploy/dovecot/push_notification.lua

@@ -17,12 +17,8 @@ function dovecot_lua_notify_event_message_new(user, event)
 
   if user.username ~= event.from_address then
     -- Incoming message
-    if not contains(event.keywords, "$Auto") then
-      -- Not an Auto-Submitted message, notifying.
-
-      -- Notify METADATA server about new message.
-      mbox:metadata_set("/private/messagenew", "")
-    end
+    -- Notify METADATA server about new message.
+    mbox:metadata_set("/private/messagenew", "")
   end
 
   mbox:free()

cmdeploy/src/cmdeploy/dovecot/remove-seen.py

@@ -1,41 +0,0 @@
-#!/usr/bin/env python3
-"""Remove seen messages that are older than two days
-if maildir has more than 80 MB of messages."""
-import sys
-import time
-from pathlib import Path
-
-
-def getdirsize(path):
-    return sum(f.stat().st_size for f in path.glob("**/*") if f.is_file())
-
-
-def parse_dovecot_seen(path):
-    return "S" in path.name.split(":2,")[-1]
-
-
-def main():
-    now = time.time()
-
-    mailhome = Path(sys.argv[1])
-
-    for p in mailhome.iterdir():
-        dirsize = getdirsize(p / "cur") + getdirsize(p / "new")
-        if dirsize < 80000000:
-            continue
-
-        removed_bytes = 0
-        for mailpath in (p / "cur").iterdir():
-            seen = parse_dovecot_seen(mailpath)
-            stat = mailpath.stat()
-            size = stat.st_size
-            if seen and now > stat.st_mtime + 2 * 24 * 3600:
-                removed_bytes += size
-                mailpath.unlink(missing_ok=True)
-
-        if removed_bytes > 0:
-            (p / "maildirsize").unlink(missing_ok=True)
-
-
-if __name__ == "__main__":
-    main()

cmdeploy/src/cmdeploy/metrics.cron.j2

@@ -1 +1 @@
-*/5 * * * * root {{ config.execpath }} /home/vmail/mail/{{ config.mail_domain }} >/var/www/html/metrics
+*/5 * * * * root {{ config.execpath }} {{ config.mailboxes_dir }} >/var/www/html/metrics

cmdeploy/src/cmdeploy/nginx/autoconfig.xml.j2

@@ -19,6 +19,13 @@
       <authentication>password-cleartext</authentication>
       <username>%EMAILADDRESS%</username>
     </incomingServer>
+    <incomingServer type="imap">
+      <hostname>{{ config.domain_name }}</hostname>
+      <port>443</port>
+      <socketType>SSL</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+    </incomingServer>
     <outgoingServer type="smtp">
       <hostname>{{ config.domain_name }}</hostname>
       <port>465</port>
@@ -33,5 +40,12 @@
       <authentication>password-cleartext</authentication>
       <username>%EMAILADDRESS%</username>
     </outgoingServer>
+    <outgoingServer type="smtp">
+      <hostname>{{ config.domain_name }}</hostname>
+      <port>443</port>
+      <socketType>SSL</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+    </outgoingServer>
   </emailProvider>
 </clientConfig>

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -1,3 +1,5 @@
+load_module modules/ngx_stream_module.so;
+
 user www-data;
 worker_processes auto;
 pid /run/nginx.pid;
@@ -8,6 +10,21 @@ events {
 	# multi_accept on;
 }
 
+stream {
+        map $ssl_preread_alpn_protocols $proxy {
+            default 127.0.0.1:8443;
+            ~\bsmtp\b 127.0.0.1:submissions;
+            ~\bimap\b 127.0.0.1:imaps;
+        }
+
+        server {
+                listen 443;
+                listen [::]:443;
+                proxy_pass $proxy;
+                ssl_preread on;
+        }
+}
+
 http {
 	sendfile on;
 	tcp_nopush on;
@@ -26,8 +43,8 @@ http {
 	gzip on;
 
 	server {
-		listen 443 ssl default_server;
-		listen [::]:443 ssl default_server;
+		listen 8443 ssl default_server;
+		listen [::]:8443 ssl default_server;
 
 		root /var/www/html;
 
@@ -78,8 +95,8 @@ http {
 
 	# Redirect www. to non-www
 	server {
-		listen 443 ssl;
-		listen [::]:443 ssl;
+		listen 8443 ssl;
+		listen [::]:8443 ssl;
 		server_name www.{{ config.domain_name }};
 		return 301 $scheme://{{ config.domain_name }}$request_uri;
 		access_log syslog:server=unix:/dev/log,facility=local7;

cmdeploy/src/cmdeploy/opendkim/opendkim.conf

@@ -25,7 +25,24 @@ KeyTable          /etc/dkimkeys/KeyTable
 SigningTable      refile:/etc/dkimkeys/SigningTable
 
 # Sign Autocrypt header in addition to the default specified in RFC 6376.
-SignHeaders *,+autocrypt
+#
+# Default list is here:
+# <https://github.com/trusteddomainproject/OpenDKIM/blob/5c539587561785a66c1f67f720f2fb741f320785/libopendkim/dkim.c#L221-L245>
+SignHeaders *,+autocrypt,+content-type
+
+# Prevent addition of second Content-Type header
+# and other important headers that should not be added
+# after signing the message.
+# See
+# <https://www.zone.eu/blog/2024/05/17/bimi-and-dmarc-cant-save-you/>
+# and RFC 6376 (page 41) for reference.
+#
+# We don't use "l=" body length so the problem described in RFC 6376
+# is not applicable, but adding e.g. a second "From" header
+# or second "Autocrypt" header is better prevented in any case.
+#
+# Default is empty.
+OversignHeaders from,reply-to,subject,date,to,cc,resent-date,resent-from,resent-sender,resent-to,resent-cc,in-reply-to,references,list-id,list-help,list-unsubscribe,list-subscribe,list-post,list-owner,list-archive,autocrypt
 
 # Script to ignore signatures that do not correspond to the From: domain.
 ScreenPolicyScript /etc/opendkim/screen.lua

cmdeploy/src/cmdeploy/remote_funcs.py

@@ -0,0 +1,105 @@
+"""
+Functions to be executed on an ssh-connected host.
+
+All functions of this module need to work with Python builtin types
+and standard library dependencies only.
+
+When a remote function executes remotely, it runs in a system python interpreter
+without any installed dependencies.
+
+"""
+
+import re
+import socket
+from subprocess import CalledProcessError, check_output
+
+
+def shell(command, fail_ok=False):
+    log(f"$ {command}")
+    try:
+        return check_output(command, shell=True).decode().rstrip()
+    except CalledProcessError:
+        if not fail_ok:
+            raise
+        return ""
+
+
+def get_systemd_running():
+    lines = shell("systemctl --type=service --state=running").split("\n")
+    return [line for line in lines if line.startswith("  ")]
+
+
+def perform_initial_checks(mail_domain):
+    res = {}
+
+    res["acme_account_url"] = shell("acmetool account-url", fail_ok=True)
+    if not shell("dig", fail_ok=True):
+        shell("apt-get install -y dnsutils")
+    shell(f"unbound-control flush_zone {mail_domain}", fail_ok=True)
+
+    res["dkim_entry"] = get_dkim_entry(mail_domain, dkim_selector="opendkim")
+    res["ipv4"] = get_ip_address(socket.AF_INET)
+    res["ipv6"] = get_ip_address(socket.AF_INET6)
+    return res
+
+
+def get_dkim_entry(mail_domain, dkim_selector):
+    dkim_pubkey = shell(
+        f"openssl rsa -in /etc/dkimkeys/{dkim_selector}.private "
+        "-pubout 2>/dev/null | awk '/-/{next}{printf(\"%s\",$0)}'"
+    )
+    dkim_value_raw = f"v=DKIM1;k=rsa;p={dkim_pubkey};s=email;t=s"
+    dkim_value = '" "'.join(re.findall(".{1,255}", dkim_value_raw))
+    return f'{dkim_selector}._domainkey.{mail_domain}. TXT "{dkim_value}"'
+
+
+def get_ip_address(typ):
+    sock = socket.socket(typ, socket.SOCK_DGRAM)
+    sock.settimeout(0)
+    sock.connect(("notifications.delta.chat", 1))
+    return sock.getsockname()[0]
+
+
+def query_dns(typ, domain):
+    res = shell(f"dig -r -q {domain} -t {typ} +short")
+    return set(filter(None, res.split("\n")))
+
+
+def check_zonefile(zonefile):
+    diff = []
+
+    for zf_line in zonefile.splitlines():
+        zf_domain, zf_typ, zf_value = zf_line.split(maxsplit=2)
+        zf_domain = zf_domain.rstrip(".")
+        zf_value = zf_value.strip()
+        query_values = query_dns(zf_typ, zf_domain)
+        if zf_value in query_values:
+            continue
+
+        if zf_typ == "CAA" and zf_value.endswith('accounturi="'):
+            # this is an initial run where acmetool did not work yet
+            continue
+
+        if query_values and zf_typ == "TXT" and zf_domain.startswith("_mta-sts."):
+            (query_value,) = query_values
+            if query_value.split("id=")[0] == zf_value.split("id=")[0]:
+                continue
+
+        assert zf_typ in ("A", "AAAA", "CNAME", "CAA", "SRV", "MX", "TXT"), zf_line
+        diff.append(zf_line)
+
+    return diff
+
+
+# check if this module is executed remotely
+# and setup a simple serialized function-execution loop
+
+if __name__ == "__channelexec__":
+
+    def log(item):
+        channel.send(("log", item))  # noqa
+
+    while 1:
+        func_name, kwargs = channel.receive()  # noqa
+        res = globals()[func_name](**kwargs)  # noqa
+        channel.send(("finish", res))  # noqa

cmdeploy/src/cmdeploy/service/chatmail-metadata.service.f

@@ -2,7 +2,7 @@
 Description=Chatmail dict proxy for IMAP METADATA
 
 [Service]
-ExecStart={execpath} /run/chatmail-metadata/metadata.socket /home/vmail/mail/{mail_domain} {config_path}
+ExecStart={execpath} /run/chatmail-metadata/metadata.socket {config_path}
 Restart=always
 RestartSec=30
 User=vmail

cmdeploy/src/cmdeploy/service/doveauth.service.f

@@ -2,7 +2,7 @@
 Description=Chatmail dict authentication proxy for dovecot
 
 [Service]
-ExecStart={execpath} /run/doveauth/doveauth.socket /home/vmail/passdb.sqlite {config_path}
+ExecStart={execpath} /run/doveauth/doveauth.socket {config_path}
 Restart=always
 RestartSec=30
 User=vmail

cmdeploy/src/cmdeploy/sshexec.py

@@ -0,0 +1,20 @@
+import execnet
+
+
+class SSHExec:
+    RemoteError = execnet.RemoteError
+
+    def __init__(self, host, remote_funcs, log=None, python="python3", timeout=60):
+        self.gateway = execnet.makegateway(f"ssh=root@{host}//python={python}")
+        self._remote_cmdloop_channel = self.gateway.remote_exec(remote_funcs)
+        self.log = log
+        self.timeout = timeout
+
+    def __call__(self, func, **kwargs):
+        self._remote_cmdloop_channel.send((func.__name__, kwargs))
+        while 1:
+            code, data = self._remote_cmdloop_channel.receive(timeout=self.timeout)
+            if code == "log" and self.log:
+                self.log(data)
+            elif code == "finish":
+                return data

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -2,6 +2,24 @@ import smtplib
 
 import pytest
 
+from cmdeploy import remote_funcs
+from cmdeploy.sshexec import SSHExec
+
+
+class TestSSHExecutor:
+    @pytest.fixture
+    def sshexec(self, sshdomain):
+        return SSHExec(sshdomain, remote_funcs)
+
+    def test_ls(self, sshexec):
+        out = sshexec(remote_funcs.shell, command="ls")
+        out2 = sshexec(remote_funcs.shell, command="ls")
+        assert out == out2
+
+    def test_perform_initial(self, sshexec, maildomain):
+        res = sshexec(remote_funcs.perform_initial_checks, mail_domain=maildomain)
+        assert res["ipv4"] or res["ipv6"]
+
 
 def test_remote(remote, imap_or_smtp):
     lineproducer = remote.iter_output(imap_or_smtp.logcmd)
@@ -90,12 +108,12 @@ def test_exceed_rate_limit(cmsetup, gencreds, maildata, chatmail_config):
 def test_expunged(remote, chatmail_config):
     outdated_days = int(chatmail_config.delete_mails_after) + 1
     find_cmds = [
-        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/cur/*' -mtime +{outdated_days} -type f",
-        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/.*/cur/*' -mtime +{outdated_days} -type f",
-        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/new/*' -mtime +{outdated_days} -type f",
-        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/.*/new/*' -mtime +{outdated_days} -type f",
-        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/tmp/*' -mtime +{outdated_days} -type f",
-        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/.*/tmp/*' -mtime +{outdated_days} -type f",
+        f"find {chatmail_config.mailboxes_dir} -path '*/cur/*' -mtime +{outdated_days} -type f",
+        f"find {chatmail_config.mailboxes_dir} -path '*/.*/cur/*' -mtime +{outdated_days} -type f",
+        f"find {chatmail_config.mailboxes_dir} -path '*/new/*' -mtime +{outdated_days} -type f",
+        f"find {chatmail_config.mailboxes_dir} -path '*/.*/new/*' -mtime +{outdated_days} -type f",
+        f"find {chatmail_config.mailboxes_dir} -path '*/tmp/*' -mtime +{outdated_days} -type f",
+        f"find {chatmail_config.mailboxes_dir} -path '*/.*/tmp/*' -mtime +{outdated_days} -type f",
     ]
     for cmd in find_cmds:
         for line in remote.iter_output(cmd):

cmdeploy/src/cmdeploy/tests/test_cmdeploy.py

@@ -21,8 +21,9 @@ class TestCmdline:
         run = parser.parse_args(["run"])
         assert init and run
 
-    @pytest.mark.xfail(reason="init doesn't exit anymore, check for CLI output instead")
-    def test_init_not_overwrite(self):
-        main(["init", "chat.example.org"])
-        with pytest.raises(SystemExit):
-            main(["init", "chat.example.org"])
+    def test_init_not_overwrite(self, capsys):
+        assert main(["init", "chat.example.org"]) == 0
+        capsys.readouterr()
+        assert main(["init", "chat.example.org"]) == 1
+        out, err = capsys.readouterr()
+        assert "path exists" in out.lower()