Document how to migrate the server

I wanted to add `COMPRESS=DEFLATE`,
but it should be added only for sessions
that are logged in because `COMPRESS`
command does not work before logging in.

Dovecot already does it correctly
if we don't overwrite the capability string.

CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog for chatmail deployment 
 
+## untagged
+
+- fix checking for required DNS records
+  ([#412](https://github.com/deltachat/chatmail/pull/412))
+
+- add a paragraph about "account deletion" to info page 
+  ([#405](https://github.com/deltachat/chatmail/pull/405))
+
+- avoid nginx listening on ipv6 if v6 is dsiabled 
+  ([#402](https://github.com/deltachat/chatmail/pull/402))
+
+- refactor ssh-based execution to allow organizing remote functions in
+  modules. 
+  ([#396](https://github.com/deltachat/chatmail/pull/396))
+
+- trigger "apt upgrade" during "cmdeploy run" 
+  ([#398](https://github.com/deltachat/chatmail/pull/398))
+
+- drop hispanilandia passthrough address
+  ([#401](https://github.com/deltachat/chatmail/pull/401))
+
+- set CAA record flags to 0
+
+- add IMAP capabilities instead of overwriting them
+  ([#413](https://github.com/deltachat/chatmail/pull/413))
+
+
+## 1.4.1 2024-07-31
+
+- fix metadata dictproxy which would confuse transactions
+  resulting in missed notifications and other issues. 
+  ([#393](https://github.com/deltachat/chatmail/pull/393))
+  ([#394](https://github.com/deltachat/chatmail/pull/394))
+
+- add optional "imap_rawlog" config option. If true, 
+  .in/.out files are created in user home dirs 
+  containing the imap protocol messages. 
+  ([#389](https://github.com/deltachat/chatmail/pull/389))
+
 ## 1.4.0 2024-07-28
 
 - Add `disable_ipv6` config option to chatmail.ini.

README.md

@@ -34,8 +34,8 @@ Please substitute it with your own domain.
     scripts/cmdeploy init chat.example.org  # <-- use your domain 
    ```
 
-3. Setup first DNS records for your chatmail domain, 
+3. Point your domain to the server's IP address,
-   according to the hints provided by `cmdeploy init`.
+   if you haven't done so already.
    Verify that SSH root login works:
 
    ```
@@ -47,7 +47,8 @@ Please substitute it with your own domain.
    ```
     scripts/cmdeploy run
    ```
-   This script will also show you additional DNS records
+   This script will check that you have all necessary DNS records.
+   If DNS records are missing, it will recommend
    which you should configure at your DNS provider
    (it can take some time until they are public).
 
@@ -59,7 +60,7 @@ To check the status of your remotely running chatmail service:
 scripts/cmdeploy status
 ```
 
-To check whether your DNS records are correct:
+To display and check all recommended DNS records:
 
 ```
 scripts/cmdeploy dns
@@ -186,3 +187,113 @@ to MAIL FROM with
 and rejects incorrectly authenticated emails with [`reject_sender_login_mismatch`](reject_sender_login_mismatch) policy.
 `From:` header must correspond to envelope MAIL FROM,
 this is ensured by `filtermail` proxy.
+
+## Migrating chatmail server to a new host
+
+If you want to migrate your chatmail server to a new host,
+follow these steps:
+
+1. Block all ports except 80 and 22 with firewall on a new server.
+
+   To do this, add the following config to `/etc/nftables.conf`:
+```
+#!/usr/sbin/nft -f
+
+flush ruleset
+
+table inet filter {
+    chain input {
+        type filter hook input priority filter; policy drop;
+
+        # Accept ICMP.
+        # It is especially important to accept ICMPv6 ND messages,
+        # otherwise IPv6 connectivity breaks.
+        icmp type { echo-request } accept
+        icmpv6 type { echo-request, nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
+
+        tcp dport { ssh, http } accept
+
+        ct state established accept
+    }
+    chain forward {
+        type filter hook forward priority filter;
+    }
+    chain output {
+        type filter hook output priority filter;
+    }
+}
+```
+   Then execute `nft -f /etc/nftables.conf` as root.
+
+   This will ensure users will not connect to the new server
+   and mails will not be delivered to the new server
+   before you finish the setup.
+
+   Port 22 is needed for SSH access
+   and port 80 is needed to get a TLS certificate.
+   They are not used by Delta Chat
+   or by other email servers trying to deliver the messages.
+
+2. Point DNS to the new IP addresses.
+
+   You can already remove the old IP addresses from DNS.
+   Existing Delta Chat users will still be able to connect
+   to the old server, send and receive messages,
+   but new users will fail to create new profiles
+   with your chatmail server.
+
+3. Setup the new server with `cmdeploy`.
+
+   This step is similar to initial setup.
+   However, because ports Delta Chat uses are blocked,
+   new server will not become usable immediately.
+   If other servers try to deliver messages to your new server they will fail,
+   but normally email servers will retry delivering messages
+   for at least a week, so messages will not be lost.
+
+4. Firewall all ports except `ssh` (22) on the old server.
+   Existing users will not be able to connect from now on
+   and no more messages will be delivered to your old chatmail server.
+
+   Blocking users from connecting to the new server
+   until mailboxes are migrated is needed to avoid UID validity change.
+   If Delta Chat connects to the new server before it is fully set up,
+   it will lose track of the IMAP message UID
+   and miss messages that arrived during migration.
+
+   Same for SMTP port 25, you want it blocked during migration so no new mails arrive
+   while the server is moving.
+
+5. Use `rsync -avz` over SSH to copy /home/vmail/mail from the old server to the new one
+   preserving file permissions and timestamps.
+
+6. Unblock ports used by Delta Chat and SMTP message exchange.
+   For that you can modify `/etc/nftables.conf` as follows:
+```
+#!/usr/sbin/nft -f
+
+flush ruleset
+
+table inet filter {
+    chain input {
+        type filter hook input priority filter; policy drop;
+
+        # Accept ICMP.
+        # It is especially important to accept ICMPv6 ND messages,
+        # otherwise IPv6 connectivity breaks.
+        icmp type { echo-request } accept
+        icmpv6 type { echo-request, nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
+
+        tcp dport { ssh, smtp, http, https, imap, imaps, submission, submissions } accept
+
+        ct state established accept
+    }
+    chain forward {
+        type filter hook forward priority filter;
+    }
+    chain output {
+        type filter hook output priority filter;
+    }
+}
+```
+Execute `nft -f /etc/nftables.conf` as root to apply the changes.

chatmaild/src/chatmaild/__init__.py

@@ -0,0 +1 @@
+

chatmaild/src/chatmaild/config.py

@@ -31,6 +31,7 @@ class Config:
         self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
         self.postfix_reinject_port = int(params["postfix_reinject_port"])
         self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
+        self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
         self.iroh_relay = params.get("iroh_relay")
         self.privacy_postal = params.get("privacy_postal")
         self.privacy_mail = params.get("privacy_mail")

chatmaild/src/chatmaild/dictproxy.py

@@ -4,21 +4,24 @@ from socketserver import StreamRequestHandler, ThreadingUnixStreamServer
 
 
 class DictProxy:
-    def __init__(self):
-        self.transactions = {}
-
     def loop_forever(self, rfile, wfile):
+        # Transaction storage is local to each handler loop.
+        # Dovecot reuses transaction IDs across connections,
+        # starting transaction with the name `1`
+        # on two different connections to the same proxy sometimes.
+        transactions = {}
+
         while True:
             msg = rfile.readline().strip().decode()
             if not msg:
                 break
 
-            res = self.handle_dovecot_request(msg)
+            res = self.handle_dovecot_request(msg, transactions)
             if res:
                 wfile.write(res.encode("ascii"))
                 wfile.flush()
 
-    def handle_dovecot_request(self, msg):
+    def handle_dovecot_request(self, msg, transactions):
         # see https://doc.dovecot.org/developer_manual/design/dict_protocol/#dovecot-dict-protocol
         short_command = msg[0]
         parts = msg[1:].split("\t")
@@ -37,11 +40,14 @@ class DictProxy:
         transaction_id = parts[0]
 
         if short_command == "B":
-            return self.handle_begin_transaction(transaction_id, parts)
+            return self.handle_begin_transaction(transaction_id, parts, transactions)
         elif short_command == "C":
-            return self.handle_commit_transaction(transaction_id, parts)
+            return self.handle_commit_transaction(transaction_id, parts, transactions)
         elif short_command == "S":
-            return self.handle_set(transaction_id, parts)
+            addr = transactions[transaction_id]["addr"]
+            if not self.handle_set(addr, parts):
+                transactions[transaction_id]["res"] = "F\n"
+                logging.error(f"dictproxy-set failed for {addr!r}: {msg!r}")
 
     def handle_lookup(self, parts):
         logging.warning(f"lookup ignored: {parts!r}")
@@ -52,19 +58,18 @@ class DictProxy:
         # If we don't return empty line Dovecot will timeout.
         return "\n"
 
-    def handle_begin_transaction(self, transaction_id, parts):
+    def handle_begin_transaction(self, transaction_id, parts, transactions):
         addr = parts[1]
-        self.transactions[transaction_id] = dict(addr=addr, res="O\n")
+        transactions[transaction_id] = dict(addr=addr, res="O\n")
 
-    def handle_set(self, transaction_id, parts):
+    def handle_set(self, addr, parts):
         # For documentation on key structure see
         # https://github.com/dovecot/core/blob/main/src/lib-storage/mailbox-attribute.h
+        return False
 
-        self.transactions[transaction_id]["res"] = "F\n"
+    def handle_commit_transaction(self, transaction_id, parts, transactions):
-
-    def handle_commit_transaction(self, transaction_id, parts):
         # return whatever "set" command(s) set as result.
-        return self.transactions.pop(transaction_id)["res"]
+        return transactions.pop(transaction_id)["res"]
 
     def serve_forever_from_socket(self, socket):
         dictproxy = self

chatmaild/src/chatmaild/doveauth.py

@@ -141,7 +141,7 @@ class AuthDictProxy(DictProxy):
             return
 
         user.set_password(encrypt_password(cleartext_password))
-        print(f"Created address: {user}", file=sys.stderr)
+        print(f"Created address: {addr}", file=sys.stderr)
         return user.get_userdb_dict()
 
 

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -23,8 +23,8 @@ max_message_size = 31457280
 # days after which mails are unconditionally deleted
 delete_mails_after = 20
 
-# days after which users without a login are deleted (database and mails)
+# days after which users without a successful login are deleted (database and mails)
-delete_inactive_users_after = 100
+delete_inactive_users_after = 90
 
 # minimum length a username must have
 username_min_length = 9
@@ -39,7 +39,8 @@ password_min_length = 9
 passthrough_senders =
 
 # list of e-mail recipients for which to accept outbound un-encrypted mails
-passthrough_recipients = xstore@testrun.org groupsbot@hispanilandia.net
+# (space-separated)
+passthrough_recipients = xstore@testrun.org 
 
 #
 # Deployment Details
@@ -54,6 +55,17 @@ postfix_reinject_port = 10025
 # if set to "True" IPv6 is disabled
 disable_ipv6 = False
 
+#
+# Debugging options 
+#
+
+# set to True if you want to track imap protocol execution
+# in per-maildir ".in/.out" files. 
+# Note that you need to manually cleanup these files
+# so use this option with caution on production servers. 
+imap_rawlog = false 
+
+
 #
 # Privacy Policy
 #

chatmaild/src/chatmaild/ini/override-testrun.ini

@@ -1,7 +1,7 @@
 
 [privacy]
 
-passthrough_recipients = privacy@testrun.org xstore@testrun.org groupsbot@hispanilandia.net
+passthrough_recipients = privacy@testrun.org xstore@testrun.org
 
 privacy_postal =
     Merlinux GmbH, Represented by the managing director H. Krekel,

chatmaild/src/chatmaild/lastlogin.py

@@ -9,20 +9,19 @@ class LastLoginDictProxy(DictProxy):
         super().__init__()
         self.config = config
 
-    def handle_set(self, transaction_id, parts):
+    def handle_set(self, addr, parts):
         keyname = parts[1].split("/")
         value = parts[2] if len(parts) > 2 else ""
-        addr = self.transactions[transaction_id]["addr"]
         if keyname[0] == "shared" and keyname[1] == "last-login":
             if addr.startswith("echo@"):
-                return
+                return True
             addr = keyname[2]
             timestamp = int(value)
             user = self.config.get_user(addr)
             user.set_last_login_timestamp(timestamp)
-        else:
+            return True
-            # Transaction failed.
+
-            self.transactions[transaction_id]["res"] = "F\n"
+        return False
 
 
 def main():

chatmaild/src/chatmaild/metadata.py

@@ -62,24 +62,19 @@ class MetadataDictProxy(DictProxy):
         logging.warning(f"lookup ignored: {parts!r}")
         return "N\n"
 
-    def handle_set(self, transaction_id, parts):
+    def handle_set(self, addr, parts):
         # For documentation on key structure see
         # https://github.com/dovecot/core/blob/main/src/lib-storage/mailbox-attribute.h
         keyname = parts[1].split("/")
         value = parts[2] if len(parts) > 2 else ""
-        addr = self.transactions[transaction_id]["addr"]
         if keyname[0] == "priv" and keyname[2] == self.metadata.DEVICETOKEN_KEY:
             self.metadata.add_token_to_addr(addr, value)
+            return True
         elif keyname[0] == "priv" and keyname[2] == "messagenew":
             self.notifier.new_message_for_addr(addr, self.metadata)
-        else:
+            return True
-            # Transaction failed.
+
-            try:
+        return False
-                self.transactions[transaction_id]["res"] = "F\n"
-            except KeyError:
-                logging.error(
-                    f"could not mark tx as failed: {transaction_id} {self.transactions}"
-                )
 
 
 def main():

chatmaild/src/chatmaild/tests/plugin.py

@@ -7,6 +7,7 @@ from email.parser import BytesParser
 from pathlib import Path
 
 import pytest
+
 from chatmaild.config import read_config, write_initial_config
 
 

chatmaild/src/chatmaild/tests/test_config.py

@@ -1,4 +1,5 @@
 import pytest
+
 from chatmaild.config import read_config
 
 

chatmaild/src/chatmaild/tests/test_doveauth.py

@@ -4,8 +4,9 @@ import queue
 import threading
 import traceback
 
-import chatmaild.doveauth
 import pytest
+
+import chatmaild.doveauth
 from chatmaild.doveauth import (
     AuthDictProxy,
     is_allowed_to_create,
@@ -72,12 +73,13 @@ def test_nocreate_file(monkeypatch, tmpdir, dictproxy):
 
 
 def test_handle_dovecot_request(dictproxy):
+    transactions = {}
     # Test that password can contain ", ', \ and /
     msg = (
         'Lshared/passdb/laksjdlaksjdlak\\\\sjdlk\\"12j\\\'3l1/k2j3123"'
         "some42123@chat.example.org\tsome42123@chat.example.org"
     )
-    res = dictproxy.handle_dovecot_request(msg)
+    res = dictproxy.handle_dovecot_request(msg, transactions)
     assert res
     assert res[0] == "O" and res.endswith("\n")
     userdata = json.loads(res[1:].strip())

chatmaild/src/chatmaild/tests/test_filtermail.py

@@ -1,4 +1,5 @@
 import pytest
+
 from chatmaild.filtermail import (
     BeforeQueueHandler,
     SendRateLimiter,

chatmaild/src/chatmaild/tests/test_lastlogin.py

@@ -12,28 +12,30 @@ def test_handle_dovecot_request_last_login(testaddr, example_config):
     authproxy = AuthDictProxy(config=example_config)
     authproxy.lookup_passdb(testaddr, "1l2k3j1l2k3jl123")
 
+    dictproxy_transactions = {}
+
     # Begin transaction
     tx = "1111"
     msg = f"B{tx}\t{testaddr}"
-    res = dictproxy.handle_dovecot_request(msg)
+    res = dictproxy.handle_dovecot_request(msg, dictproxy_transactions)
     assert not res
-    assert dictproxy.transactions == {tx: dict(addr=testaddr, res="O\n")}
+    assert dictproxy_transactions == {tx: dict(addr=testaddr, res="O\n")}
 
     # set last-login info for user
     user = dictproxy.config.get_user(testaddr)
     timestamp = int(time.time())
     msg = f"S{tx}\tshared/last-login/{testaddr}\t{timestamp}"
-    res = dictproxy.handle_dovecot_request(msg)
+    res = dictproxy.handle_dovecot_request(msg, dictproxy_transactions)
     assert not res
-    assert len(dictproxy.transactions) == 1
+    assert len(dictproxy_transactions) == 1
     read_timestamp = user.get_last_login_timestamp()
     assert read_timestamp == timestamp // 86400 * 86400
 
     # finish transaction
     msg = f"C{tx}"
-    res = dictproxy.handle_dovecot_request(msg)
+    res = dictproxy.handle_dovecot_request(msg, dictproxy_transactions)
     assert res == "O\n"
-    assert len(dictproxy.transactions) == 0
+    assert len(dictproxy_transactions) == 0
 
 
 def test_handle_dovecot_request_last_login_echobot(example_config):
@@ -44,17 +46,19 @@ def test_handle_dovecot_request_last_login_echobot(example_config):
     authproxy.lookup_passdb(testaddr, "ignore")
     user = dictproxy.config.get_user(testaddr)
 
+    transactions = {}
+
     # set last-login info for user
     tx = "1111"
     msg = f"B{tx}\t{testaddr}"
-    res = dictproxy.handle_dovecot_request(msg)
+    res = dictproxy.handle_dovecot_request(msg, transactions)
     assert not res
-    assert dictproxy.transactions == {tx: dict(addr=testaddr, res="O\n")}
+    assert transactions == {tx: dict(addr=testaddr, res="O\n")}
 
     timestamp = int(time.time())
     msg = f"S{tx}\tshared/last-login/{testaddr}\t{timestamp}"
-    res = dictproxy.handle_dovecot_request(msg)
+    res = dictproxy.handle_dovecot_request(msg, transactions)
     assert not res
-    assert len(dictproxy.transactions) == 1
+    assert len(transactions) == 1
     read_timestamp = user.get_last_login_timestamp()
     assert read_timestamp is None

chatmaild/src/chatmaild/tests/test_metadata.py

@@ -3,6 +3,7 @@ import time
 
 import pytest
 import requests
+
 from chatmaild.metadata import (
     Metadata,
     MetadataDictProxy,
@@ -88,42 +89,45 @@ def test_notifier_remove_without_set(metadata, testaddr):
 
 
 def test_handle_dovecot_request_lookup_fails(dictproxy, testaddr):
-    res = dictproxy.handle_dovecot_request(f"Lpriv/123/chatmail\t{testaddr}")
+    transactions = {}
+    res = dictproxy.handle_dovecot_request(
+        f"Lpriv/123/chatmail\t{testaddr}", transactions
+    )
     assert res == "N\n"
 
 
 def test_handle_dovecot_request_happy_path(dictproxy, testaddr, token):
     metadata = dictproxy.metadata
-    transactions = dictproxy.transactions
+    transactions = {}
     notifier = dictproxy.notifier
 
     # set device token in a transaction
     tx = "1111"
     msg = f"B{tx}\t{testaddr}"
-    res = dictproxy.handle_dovecot_request(msg)
+    res = dictproxy.handle_dovecot_request(msg, transactions)
     assert not res and not metadata.get_tokens_for_addr(testaddr)
     assert transactions == {tx: dict(addr=testaddr, res="O\n")}
 
     msg = f"S{tx}\tpriv/guid00/devicetoken\t{token}"
-    res = dictproxy.handle_dovecot_request(msg)
+    res = dictproxy.handle_dovecot_request(msg, transactions)
     assert not res
     assert len(transactions) == 1
     assert metadata.get_tokens_for_addr(testaddr) == [token]
 
     msg = f"C{tx}"
-    res = dictproxy.handle_dovecot_request(msg)
+    res = dictproxy.handle_dovecot_request(msg, transactions)
     assert res == "O\n"
     assert len(transactions) == 0
     assert metadata.get_tokens_for_addr(testaddr) == [token]
 
     # trigger notification for incoming message
     tx2 = "2222"
-    assert dictproxy.handle_dovecot_request(f"B{tx2}\t{testaddr}") is None
+    assert dictproxy.handle_dovecot_request(f"B{tx2}\t{testaddr}", transactions) is None
     msg = f"S{tx2}\tpriv/guid00/messagenew"
-    assert dictproxy.handle_dovecot_request(msg) is None
+    assert dictproxy.handle_dovecot_request(msg, transactions) is None
     queue_item = notifier.retry_queues[0].get()[1]
     assert queue_item.token == token
-    assert dictproxy.handle_dovecot_request(f"C{tx2}") == "O\n"
+    assert dictproxy.handle_dovecot_request(f"C{tx2}", transactions) == "O\n"
     assert not transactions
     assert queue_item.path.exists()
 

chatmaild/src/chatmaild/tests/test_user.py

@@ -12,11 +12,11 @@ def test_get_user_dict_not_set(testaddr, example_config, caplog):
     user = example_config.get_user(testaddr)
     assert not caplog.records
     assert user.get_userdb_dict() == {}
-    assert len(caplog.records) == 1
+    assert len(caplog.records) == 0
 
     user.set_password("")
     assert user.get_userdb_dict() == {}
-    assert len(caplog.records) == 2
+    assert len(caplog.records) == 1
 
 
 def test_get_user_dict(make_config, tmp_path):

chatmaild/src/chatmaild/user.py

@@ -26,7 +26,6 @@ class User:
         try:
             pw = self.password_path.read_text()
         except FileNotFoundError:
-            logging.error(f"password not set for: {self.addr}")
             return {}
 
         if not pw:

cmdeploy/src/cmdeploy/__init__.py

@@ -489,6 +489,7 @@ def deploy_chatmail(config_path: Path) -> None:
     )
 
     apt.update(name="apt update", cache_time=24 * 3600)
+    apt.upgrade(name="upgrade apt packages", auto_remove=True)
 
     apt.packages(
         name="Install rsync",

cmdeploy/src/cmdeploy/chatmail.zone.j2

@@ -20,7 +20,7 @@ www.{{ mail_domain }}.               CNAME {{ mail_domain }}.
 _dmarc.{{ mail_domain }}.            TXT "v=DMARC1;p=reject;adkim=s;aspf=s"
 
 {% if acme_account_url %}
-{{ mail_domain }}.                   CAA 128 issue "letsencrypt.org;accounturi={{ acme_account_url }}"
+{{ mail_domain }}.                   CAA 0 issue "letsencrypt.org;accounturi={{ acme_account_url }}"
 {% endif %}
 _adsp._domainkey.{{ mail_domain }}.  TXT "dkim=discardable"
 

cmdeploy/src/cmdeploy/cmdeploy.py

@@ -12,14 +12,13 @@ import shutil
 import subprocess
 import sys
 from pathlib import Path
-from packaging import version
 
 import pyinfra
-
 from chatmaild.config import read_config, write_initial_config
+from packaging import version
 from termcolor import colored
 
-from . import dns, remote_funcs
+from . import dns, remote
 from .sshexec import SSHExec
 
 #
@@ -133,7 +132,7 @@ def status_cmd(args, out):
     else:
         out.red("no privacy settings")
 
-    for line in sshexec(remote_funcs.get_systemd_running):
+    for line in sshexec(remote.rshell.get_systemd_running):
         print(line)
 
 
@@ -314,7 +313,7 @@ def main(args=None):
 
     def get_sshexec():
         print(f"[ssh] login to {args.config.mail_domain}")
-        return SSHExec(args.config.mail_domain, remote_funcs, verbose=args.verbose)
+        return SSHExec(args.config.mail_domain, verbose=args.verbose)
 
     args.get_sshexec = get_sshexec
 

cmdeploy/src/cmdeploy/dns.py

@@ -3,12 +3,12 @@ import importlib
 
 from jinja2 import Template
 
-from . import remote_funcs
+from . import remote
 
 
 def get_initial_remote_data(sshexec, mail_domain):
     return sshexec.logged(
-        call=remote_funcs.perform_initial_checks, kwargs=dict(mail_domain=mail_domain)
+        call=remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=mail_domain)
     )
 
 
@@ -16,9 +16,12 @@ def check_initial_remote_data(remote_data, print=print):
     mail_domain = remote_data["mail_domain"]
     if not remote_data["A"] and not remote_data["AAAA"]:
         print(f"Missing A and/or AAAA DNS records for {mail_domain}!")
-    elif not remote_data["MTA_STS"]:
+    elif remote_data["MTA_STS"] != f"{mail_domain}.":
         print("Missing MTA-STS CNAME record:")
-        print(f"mta-sts.{mail_domain}.   CNAME  {mail_domain}")
+        print(f"mta-sts.{mail_domain}.   CNAME  {mail_domain}.")
+    elif remote_data["WWW"] != f"{mail_domain}.":
+        print("Missing www CNAME record:")
+        print(f"www.{mail_domain}.   CNAME  {mail_domain}.")
     else:
         return remote_data
 
@@ -42,7 +45,8 @@ def check_full_zone(sshexec, remote_data, out, zonefile) -> int:
     and return (exitcode, remote_data) tuple."""
 
     required_diff, recommended_diff = sshexec.logged(
-        remote_funcs.check_zonefile, kwargs=dict(zonefile=zonefile)
+        remote.rdns.check_zonefile,
+        kwargs=dict(zonefile=zonefile, mail_domain=remote_data["mail_domain"]),
     )
 
     if required_diff:

cmdeploy/src/cmdeploy/dovecot/dovecot.conf.j2

@@ -51,10 +51,7 @@ mail_server_comment = Chatmail server
 # <https://doc.dovecot.org/configuration_manual/quota_plugin/>
 mail_plugins = zlib quota
 
-# these are the capabilities Delta Chat cares about actually 
+imap_capability = +XDELTAPUSH XCHATMAIL
-# so let's keep the network overhead per login small
-# https://github.com/deltachat/deltachat-core-rust/blob/master/src/imap/capabilities.rs
-imap_capability = IMAP4rev1 IDLE MOVE QUOTA CONDSTORE NOTIFY METADATA XDELTAPUSH XCHATMAIL
 
 
 # Authentication for system users.
@@ -203,3 +200,24 @@ ssl_key = </var/lib/acme/live/{{ config.mail_domain }}/privkey
 ssl_dh = </usr/share/dovecot/dh.pem
 ssl_min_protocol = TLSv1.2
 ssl_prefer_server_ciphers = yes
+
+
+{% if config.imap_rawlog %}
+service postlogin {
+  executable = script-login -d rawlog
+  unix_listener postlogin {
+ }  
+} 
+service imap {
+  executable = imap postlogin 
+} 
+  
+protocol imap { 
+  #rawlog_dir = /tmp/rawlog/%u
+  # Put .in and .out imap protocol logging files into per-user homedir 
+  # You can use a command like this to combine into one protocol stream:
+  # sort -sn <(sed 's/ / C: /' *.in) <(sed 's/ / S: /' cat *.out)
+
+  rawlog_dir = %h 
+} 
+{% endif %}

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -19,7 +19,9 @@ stream {
 
         server {
                 listen 443;
+                {% if not disable_ipv6 %}
                 listen [::]:443;
+                {% endif %}
                 proxy_pass $proxy;
                 ssl_preread on;
         }

cmdeploy/src/cmdeploy/remote/__init__.py

@@ -0,0 +1,12 @@
+"""
+
+The 'cmdeploy.remote' sub package contains modules with remotely executing functions.
+
+Its "_sshexec_bootstrap" module is executed remotely through `SSHExec`
+and its main() loop there stays connected via a command channel,
+ready to receive function invocations ("command") and return results.
+"""
+
+from . import rdns, rshell
+
+__all__ = ["rdns", "rshell"]

cmdeploy/src/cmdeploy/remote/_sshexec_bootstrap.py

@@ -0,0 +1,30 @@
+import builtins
+import importlib
+import traceback
+
+## Function Execution server
+
+
+def _run_loop(cmd_channel):
+    while cmd := cmd_channel.receive():
+        cmd_channel.send(_handle_one_request(cmd))
+
+
+def _handle_one_request(cmd):
+    pymod_path, func_name, kwargs = cmd
+    try:
+        mod = importlib.import_module(pymod_path)
+        func = getattr(mod, func_name)
+        res = func(**kwargs)
+        return ("finish", res)
+    except:
+        data = traceback.format_exc()
+        return ("error", data)
+
+
+def main(channel):
+    # enable simple "print" logging
+
+    builtins.print = lambda x="": channel.send(("log", x))
+
+    _run_loop(channel)

cmdeploy/src/cmdeploy/remote/rdns.py

@@ -11,40 +11,26 @@ All functions of this module
 """
 
 import re
-import traceback
-from subprocess import CalledProcessError, check_output
 
-
+from .rshell import CalledProcessError, shell
-def shell(command, fail_ok=False):
-    print(f"$ {command}")
-    try:
-        return check_output(command, shell=True).decode().rstrip()
-    except CalledProcessError:
-        if not fail_ok:
-            raise
-        return ""
-
-
-def get_systemd_running():
-    lines = shell("systemctl --type=service --state=running").split("\n")
-    return [line for line in lines if line.startswith("  ")]
 
 
 def perform_initial_checks(mail_domain):
     """Collecting initial DNS settings."""
     assert mail_domain
-    A = query_dns("A", mail_domain)
-    AAAA = query_dns("AAAA", mail_domain)
-    MTA_STS = query_dns("CNAME", f"mta-sts.{mail_domain}")
-
-    res = dict(mail_domain=mail_domain, A=A, AAAA=AAAA, MTA_STS=MTA_STS)
-    if not MTA_STS or (not A and not AAAA):
-        return res
-
-    res["acme_account_url"] = shell("acmetool account-url", fail_ok=True)
     if not shell("dig", fail_ok=True):
         shell("apt-get install -y dnsutils")
     shell(f"unbound-control flush_zone {mail_domain}", fail_ok=True)
+    A = query_dns("A", mail_domain)
+    AAAA = query_dns("AAAA", mail_domain)
+    MTA_STS = query_dns("CNAME", f"mta-sts.{mail_domain}")
+    WWW = query_dns("CNAME", f"www.{mail_domain}")
+
+    res = dict(mail_domain=mail_domain, A=A, AAAA=AAAA, MTA_STS=MTA_STS, WWW=WWW)
+    if not MTA_STS or not WWW or (not A and not AAAA):
+        return res
+
+    res["acme_account_url"] = shell("acmetool account-url", fail_ok=True)
     res["dkim_entry"] = get_dkim_entry(mail_domain, dkim_selector="opendkim")
 
     # parse out sts-id if exists, example: "v=STSv1; id=2090123"
@@ -74,8 +60,9 @@ def query_dns(typ, domain):
     return ""
 
 
-def check_zonefile(zonefile):
+def check_zonefile(zonefile, mail_domain):
     """Check expected zone file entries."""
+    shell(f"unbound-control flush_zone {mail_domain}", fail_ok=True)
     required = True
     required_diff = []
     recommended_diff = []
@@ -99,37 +86,3 @@ def check_zonefile(zonefile):
                 recommended_diff.append(zf_line)
 
     return required_diff, recommended_diff
-
-
-## Function Execution server
-
-
-def _run_loop(cmd_channel):
-    while 1:
-        cmd = cmd_channel.receive()
-        if cmd is None:
-            break
-
-        cmd_channel.send(_handle_one_request(cmd))
-
-
-def _handle_one_request(cmd):
-    func_name, kwargs = cmd
-    try:
-        res = globals()[func_name](**kwargs)
-        return ("finish", res)
-    except:
-        data = traceback.format_exc()
-        return ("error", data)
-
-
-# check if this module is executed remotely
-# and setup a simple serialized function-execution loop
-
-if __name__ == "__channelexec__":
-    channel = channel  # noqa (channel object gets injected)
-
-    # enable simple "print" logging for anyone changing this module
-    globals()["print"] = lambda x="": channel.send(("log", x))
-
-    _run_loop(channel)

cmdeploy/src/cmdeploy/remote/rshell.py

@@ -0,0 +1,16 @@
+from subprocess import CalledProcessError, check_output
+
+
+def shell(command, fail_ok=False):
+    print(f"$ {command}")
+    try:
+        return check_output(command, shell=True).decode().rstrip()
+    except CalledProcessError:
+        if not fail_ok:
+            raise
+        return ""
+
+
+def get_systemd_running():
+    lines = shell("systemctl --type=service --state=running").split("\n")
+    return [line for line in lines if line.startswith("  ")]

cmdeploy/src/cmdeploy/sshexec.py

@@ -1,12 +1,45 @@
+import inspect
+import os
 import sys
+from queue import Queue
 
 import execnet
 
+from . import remote
+
 
 class FuncError(Exception):
     pass
 
 
+def bootstrap_remote(gateway, remote=remote):
+    """Return a command channel which can execute remote functions."""
+    source_init_path = inspect.getfile(remote)
+    basedir = os.path.dirname(source_init_path)
+    name = os.path.basename(basedir)
+
+    # rsync sourcedir to remote host
+    remote_pkg_path = f"/root/from-cmdeploy/{name}"
+    q = Queue()
+    finish = lambda: q.put(None)
+    rsync = execnet.RSync(sourcedir=basedir, verbose=False)
+    rsync.add_target(gateway, remote_pkg_path, finishedcallback=finish, delete=True)
+    rsync.send()
+    q.get()
+
+    # start sshexec bootstrap and return its command channel
+    remote_sys_path = os.path.dirname(remote_pkg_path)
+    channel = gateway.remote_exec(
+        f"""
+        import sys
+        sys.path.insert(0, {remote_sys_path!r})
+        from remote._sshexec_bootstrap import main
+        main(channel)
+    """
+    )
+    return channel
+
+
 def print_stderr(item="", end="\n"):
     print(item, file=sys.stderr, end=end)
 
@@ -15,16 +48,18 @@ class SSHExec:
     RemoteError = execnet.RemoteError
     FuncError = FuncError
 
-    def __init__(self, host, remote_funcs, verbose=False, python="python3", timeout=60):
+    def __init__(self, host, verbose=False, python="python3", timeout=60):
         self.gateway = execnet.makegateway(f"ssh=root@{host}//python={python}")
-        self._remote_cmdloop_channel = self.gateway.remote_exec(remote_funcs)
+        self._remote_cmdloop_channel = bootstrap_remote(self.gateway, remote)
         self.timeout = timeout
         self.verbose = verbose
 
     def __call__(self, call, kwargs=None, log_callback=None):
         if kwargs is None:
             kwargs = {}
-        self._remote_cmdloop_channel.send((call.__name__, kwargs))
+        assert call.__module__.startswith("cmdeploy.remote")
+        modname = call.__module__.replace("cmdeploy.", "")
+        self._remote_cmdloop_channel.send((modname, call.__name__, kwargs))
         while 1:
             code, data = self._remote_cmdloop_channel.receive(timeout=self.timeout)
             if log_callback is not None and code == "log":

cmdeploy/src/cmdeploy/tests/data/zftest.zone

@@ -11,7 +11,7 @@ _submission._tcp.zftest.testrun.org.  SRV 0 1 587 zftest.testrun.org.
 _submissions._tcp.zftest.testrun.org. SRV 0 1 465 zftest.testrun.org.
 _imap._tcp.zftest.testrun.org.        SRV 0 1 143 zftest.testrun.org.
 _imaps._tcp.zftest.testrun.org.       SRV 0 1 993 zftest.testrun.org.
-zftest.testrun.org.                   CAA 128 issue "letsencrypt.org;accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1371472956"
+zftest.testrun.org.                   CAA 0 issue "letsencrypt.org;accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1371472956"
 zftest.testrun.org.                   TXT "v=spf1 a:zftest.testrun.org ~all"
 _dmarc.zftest.testrun.org.            TXT "v=DMARC1;p=reject;adkim=s;aspf=s"
 _adsp._domainkey.zftest.testrun.org.  TXT "dkim=discardable"

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -2,29 +2,29 @@ import smtplib
 
 import pytest
 
-from cmdeploy import remote_funcs
+from cmdeploy import remote
 from cmdeploy.sshexec import SSHExec
 
 
 class TestSSHExecutor:
     @pytest.fixture(scope="class")
     def sshexec(self, sshdomain):
-        return SSHExec(sshdomain, remote_funcs)
+        return SSHExec(sshdomain)
 
     def test_ls(self, sshexec):
-        out = sshexec(call=remote_funcs.shell, kwargs=dict(command="ls"))
+        out = sshexec(call=remote.rdns.shell, kwargs=dict(command="ls"))
-        out2 = sshexec(call=remote_funcs.shell, kwargs=dict(command="ls"))
+        out2 = sshexec(call=remote.rdns.shell, kwargs=dict(command="ls"))
         assert out == out2
 
     def test_perform_initial(self, sshexec, maildomain):
         res = sshexec(
-            remote_funcs.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
+            remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
         )
         assert res["A"] or res["AAAA"]
 
     def test_logged(self, sshexec, maildomain, capsys):
         sshexec.logged(
-            remote_funcs.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
+            remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
         )
         out, err = capsys.readouterr()
         assert err.startswith("Collecting")
@@ -33,21 +33,21 @@ class TestSSHExecutor:
 
         sshexec.verbose = True
         sshexec.logged(
-            remote_funcs.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
+            remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
         )
         out, err = capsys.readouterr()
         lines = err.split("\n")
         assert len(lines) > 4
-        assert remote_funcs.perform_initial_checks.__doc__ in lines[0]
+        assert remote.rdns.perform_initial_checks.__doc__ in lines[0]
 
     def test_exception(self, sshexec, capsys):
         try:
             sshexec.logged(
-                remote_funcs.perform_initial_checks,
+                remote.rdns.perform_initial_checks,
                 kwargs=dict(mail_domain=None),
             )
         except sshexec.FuncError as e:
-            assert "remote_funcs.py" in str(e)
+            assert "rdns.py" in str(e)
             assert "AssertionError" in str(e)
         else:
             pytest.fail("didn't raise exception")

cmdeploy/src/cmdeploy/tests/test_dns.py

@@ -1,6 +1,6 @@
 import pytest
 
-from cmdeploy import remote_funcs
+from cmdeploy import remote
 from cmdeploy.dns import check_full_zone, check_initial_remote_data
 
 
@@ -14,7 +14,7 @@ def mockdns_base(monkeypatch):
         except KeyError:
             return ""
 
-    monkeypatch.setattr(remote_funcs, query_dns.__name__, query_dns)
+    monkeypatch.setattr(remote.rdns, query_dns.__name__, query_dns)
     return qdict
 
 
@@ -24,7 +24,10 @@ def mockdns(mockdns_base):
         {
             "A": {"some.domain": "1.1.1.1"},
             "AAAA": {"some.domain": "fde5:cd7a:9e1c:3240:5a99:936f:cdac:53ae"},
-            "CNAME": {"mta-sts.some.domain": "some.domain"},
+            "CNAME": {
+                "mta-sts.some.domain": "some.domain.",
+                "www.some.domain": "some.domain.",
+            },
         }
     )
     return mockdns_base
@@ -32,14 +35,16 @@ def mockdns(mockdns_base):
 
 class TestPerformInitialChecks:
     def test_perform_initial_checks_ok1(self, mockdns):
-        remote_data = remote_funcs.perform_initial_checks("some.domain")
+        remote_data = remote.rdns.perform_initial_checks("some.domain")
-        assert len(remote_data) == 7
+        assert remote_data["A"] == mockdns["A"]["some.domain"]
+        assert remote_data["AAAA"] == mockdns["AAAA"]["some.domain"]
+        assert remote_data["MTA_STS"] == mockdns["CNAME"]["mta-sts.some.domain"]
+        assert remote_data["WWW"] == mockdns["CNAME"]["www.some.domain"]
 
     @pytest.mark.parametrize("drop", ["A", "AAAA"])
     def test_perform_initial_checks_with_one_of_A_AAAA(self, mockdns, drop):
         del mockdns[drop]
-        remote_data = remote_funcs.perform_initial_checks("some.domain")
+        remote_data = remote.rdns.perform_initial_checks("some.domain")
-        assert len(remote_data) == 7
         assert not remote_data[drop]
 
         l = []
@@ -48,9 +53,8 @@ class TestPerformInitialChecks:
         assert not l
 
     def test_perform_initial_checks_no_mta_sts(self, mockdns):
-        del mockdns["CNAME"]
+        del mockdns["CNAME"]["mta-sts.some.domain"]
-        remote_data = remote_funcs.perform_initial_checks("some.domain")
+        remote_data = remote.rdns.perform_initial_checks("some.domain")
-        assert len(remote_data) == 4
         assert not remote_data["MTA_STS"]
 
         l = []
@@ -85,14 +89,18 @@ class TestZonefileChecks:
     def test_check_zonefile_all_ok(self, cm_data, mockdns_base):
         zonefile = cm_data.get("zftest.zone")
         parse_zonefile_into_dict(zonefile, mockdns_base)
-        required_diff, recommended_diff = remote_funcs.check_zonefile(zonefile)
+        required_diff, recommended_diff = remote.rdns.check_zonefile(
+            zonefile, "some.domain"
+        )
         assert not required_diff and not recommended_diff
 
     def test_check_zonefile_recommended_not_set(self, cm_data, mockdns_base):
         zonefile = cm_data.get("zftest.zone")
         zonefile_mocked = zonefile.split("; Recommended")[0]
         parse_zonefile_into_dict(zonefile_mocked, mockdns_base)
-        required_diff, recommended_diff = remote_funcs.check_zonefile(zonefile)
+        required_diff, recommended_diff = remote.rdns.check_zonefile(
+            zonefile, "some.domain"
+        )
         assert not required_diff
         assert len(recommended_diff) == 8
 
@@ -101,6 +109,7 @@ class TestZonefileChecks:
         zonefile_mocked = zonefile.split("; Recommended")[0]
         parse_zonefile_into_dict(zonefile_mocked, mockdns_base, only_required=True)
         mssh = MockSSHExec()
+        mockdns_base["mail_domain"] = "some.domain"
         res = check_full_zone(mssh, mockdns_base, out=mockout, zonefile=zonefile)
         assert res == 0
         assert "WARNING" in mockout.captured_plain[0]
@@ -110,6 +119,7 @@ class TestZonefileChecks:
         zonefile = cm_data.get("zftest.zone")
         parse_zonefile_into_dict(zonefile, mockdns_base)
         mssh = MockSSHExec()
+        mockdns_base["mail_domain"] = "some.domain"
         res = check_full_zone(mssh, mockdns_base, out=mockout, zonefile=zonefile)
         assert res == 0
         assert not mockout.captured_red

www/src/index.md

@@ -11,7 +11,11 @@ for Delta Chat users.  For details how it avoids storing personal information
 please see our [privacy policy](privacy.html). 
 {% endif %}
 
-👉 **Tap** or scan this QR code to get a `@{{config.mail_domain}}` chat profile
+<a class="cta-button" href="DCACCOUNT:https://{{ config.mail_domain }}/new">Get a {{config.mail_domain}} chat profile</a>
+
+If you are viewing this page on a different device
+without a Delta Chat app,
+you can also **scan this QR code** with Delta Chat:
 
 <a href="DCACCOUNT:https://{{ config.mail_domain }}/new">
     <img width=300 style="float: none;" src="qr-chatmail-invite-{{config.mail_domain}}.png" /></a>

www/src/info.md

@@ -8,11 +8,9 @@ for the usage in chats, especially DeltaChat.
 
 ### Choosing a chatmail address instead of using a random one
 
-In the Delta Chat account setup 
+In the Delta Chat account setup you may tap `Create a profile` then `Use other server` and choose `Classic e-mail login`. Here fill the two fields like this: 
-you may tap `I already have a profile`
-and fill the two fields like this: 
 
-- `Address`: invent a word with
+- `E-Mail Address`: invent a word with
 {% if username_min_length == username_max_length %}
   *exactly* {{ username_min_length }}
 {% else %}
@@ -26,7 +24,7 @@ and fill the two fields like this:
   characters
   and append `@{{config.mail_domain}}` to it.
 
-- `Password`: invent at least {{ password_min_length }} characters.
+- `Existing Password`: invent at least {{ password_min_length }} characters.
 
 If the e-mail address is not yet taken, you'll get that account. 
 The first login sets your password. 
@@ -45,6 +43,20 @@ The first login sets your password.
 - You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
 
 
+### <a name="account-deletion"></a> Account deletion 
+
+If you remove a {{ config.mail_domain }} profile from within the Delta Chat app, 
+then the according account on the server, along with all associated data,
+is automatically deleted {{ config.delete_inactive_users_after }} days afterwards. 
+
+If you use multiple devices 
+then you need to remove the according chat profile from each device
+in order for all account data to be removed on the server side. 
+
+If you have any further questions or requests regarding account deletion
+please send a message from your account to {{ config.privacy_mail }}. 
+
+
 ### Who are the operators? Which software is running? 
 
 This chatmail provider is run by a small voluntary group of devs and sysadmins,

www/src/main.css

@@ -72,3 +72,15 @@ code {
   color: red;
   font-weight: bold;
 }
+
+.cta-button, .cta-button:hover, .cta-button:visited {
+    border: 1.5px solid #a4c2d0;
+    border-radius: 5px;
+    padding: 10px;
+    display: inline-block;
+    margin: 10px 0;
+
+    background: linear-gradient(120deg, #77888f, #364e59);
+    color: white !important;
+    font-weight: bold;
+}