doc: cmdeploy command makes manual configuration obsolete

fix #604
related to #629
pulled out of https://github.com/Keonik1/relay/pull/3

.github/ISSUE_TEMPLATE/config.yml

@@ -1 +1,5 @@
 blank_issues_enabled: true
+contact_links:
+  - name: Mutual Help Chat Group
+    url: https://i.delta.chat/#6CBFF8FFD505C0FDEA20A66674F2916EA8FBEE99&a=invitebot%40nine.testrun.org&g=Chatmail%20Mutual%20Help&x=7sFF7Ik50pWv6J1z7RVC5527&i=X69wTFfvCfs3d-JzqP0kVA3i&s=ibp-447dU-wUq-52QanwAtWc
+    about: If you have troubles setting up the relay server, feel free to ask here.

.github/workflows/test-and-deploy-ipv4only.yaml

@@ -77,7 +77,7 @@ jobs:
           cmdeploy init staging-ipv4.testrun.org
           sed -i 's#disable_ipv6 = False#disable_ipv6 = True#' chatmail.ini
 
-      - run: cmdeploy run --verbose --skip-dns-check
+      - run: cmdeploy run
 
       - name: set DNS entries
         run: |

.github/workflows/test-and-deploy.yaml

@@ -75,7 +75,7 @@ jobs:
 
       - run: cmdeploy init staging2.testrun.org
 
-      - run: cmdeploy run --verbose --skip-dns-check
+      - run: cmdeploy run --verbose
 
       - name: set DNS entries
         run: |

ARCHITECTURE.md

@@ -1,6 +1,4 @@
-## Chatmail Relay
-
-This diagram shows components of the chatmail relay; this is a draft
+This diagram shows components of the chatmail server; this is a draft
 overview as of mid-August 2025:
 
 ```mermaid
@@ -21,6 +19,7 @@ graph LR;
     /var/lib/acme`")] --> nginx-internal;
     cron --- chatmail-metrics;
     cron --- acmetool;
+    cron --- expunge;
     chatmail-metrics --- website;
     acmetool --> certs[("`TLS certs
     /var/lib/acme`")];
@@ -36,8 +35,7 @@ graph LR;
     dovecot --- users;
     dovecot --- |metadata.socket|chatmail-metadata;
     doveauth --- users;
-    chatmail-expire-daily --- users;
-    chatmail-fsreport-daily --- users;
+    expunge --- users;
     chatmail-metadata --- iroh-relay;
     certs-nginx --> postfix;
     certs-nginx --> dovecot;
@@ -50,66 +48,3 @@ graph LR;
 The edges in this graph should not be taken too literally; they
 reflect some sort of communication path or dependency relationship
 between components of the chatmail server.
-
-## cmdeploy
-
-cmdeploy is a Python program that uses the pyinfra library to deploy
-chatmail servers, with all the necessary software, configuration, and
-services.  The deployment process performs three primary types of operation:
-
-1. Installation of software, universal across all deployments.
-2. Configuration of software, with deploy-specific variations.
-3. Activation of services.
-
-The process is implemented through a family of "deployer" objects
-which all derive from a common `Deployer` base class, defined in
-[deployer.py](cmdeploy/src/cmdeploy/deployer.py).  Each object
-provides implementation methods for the three stages -- install,
-configure, and activate.  The top-level procedure in
-`deploy_chatmail()` calls these methods for all the deployer objects,
-first calling all the install methods, then the configure methods,
-then the activate methods.
-
-The base class also implements support for a CMDEPLOY_STAGES
-environment variable, which allows limiting the process to specific
-stages.  Note that some deployers are stateful between the stages
-(this is one reason why they are implemented as objects), and that
-state will not get propagated between stages when run in separate
-invocations of cmdeploy.  This environment variable is intended for
-use in future revisions to support building Docker images with
-software pre-installed, and configuration of containers at run time
-from environmnet variables.
-
-The `install_impl()` method for the deployer classes is static, to
-ensure that it does not rely on any object state, in particular, the
-configuration details of the deployment.  This helps ensure that all
-install methods are suitable for running as part of a container image
-build.
-
-Operations that start services for systemd-based deployments should
-only be called from the `activate_impl()` methods.  These methods will
-not be called in non-systemd container environments.
-
-### Deployer objects
-
-One might ask why the deployers are implemented as object classes, as
-opposed to callable functions or the like.  There are various reasons
-why objects are a good fit for the deployment process.
-
-1. Objects provide a way to organize the install, configure, and
-deploy operations for each component that is installed, supporting a
-"driver" type of pattern.  This could be implemented in other ways
-without objects, such as function jump tables, but objects provide a
-clean and formalized way to do essentially the same thing.
-
-2. Class inheritance provides a natural way to define
-component-specific operations for the various stages of deployment, by
-overriding the no-op stub methods in the base class.  The base class
-handles policy decisions about which stages are to be executed,
-ensuring consistent handling of the stages in a central location.
-
-3. Some of the components track state between stages, basing decisions
-like whether to restart a service on whether the software or
-configuration of that service was changed in an earlier stage.
-Keeping track of state between method calls is an ideal use case for
-objects.

CHANGELOG.md

@@ -2,76 +2,24 @@
 
 ## untagged
 
-- Organized cmdeploy into install, configure, and activate stages
-  ([#695](https://github.com/chatmail/relay/pull/695))
-
-- don't deploy the website if there are merge conflicts in the www folder
-  ([#714](https://github.com/chatmail/relay/pull/714))
-
-- acmetool: use ECDSA keys instead of RSA
-  ([#689](https://github.com/chatmail/relay/pull/689))
-
-- Require TLS 1.2 for outgoing SMTP connections
-  ([#685](https://github.com/chatmail/relay/pull/685))
-
-- require STARTTLS for incoming port 25 connections
-  ([#684](https://github.com/chatmail/relay/pull/684))
-
-- filtermail: run CPU-intensive handle_DATA in a thread pool executor
-  ([#676](https://github.com/chatmail/relay/pull/676))
-
-- don't use the complicated logging module in filtermail to exclude a potential source of errors. 
-  ([#674](https://github.com/chatmail/relay/pull/674))
-
-- Specify nginx.conf to only handle `mail_domain`, www, and mta-sts domains
-  ([#636](https://github.com/chatmail/relay/pull/636))
-
-- Setup TURN server
-  ([#621](https://github.com/chatmail/relay/pull/621))
-
 - cmdeploy: make --ssh-host work with localhost
   ([#659](https://github.com/chatmail/relay/pull/659))
 
 - Update iroh-relay to 0.35.0
   ([#650](https://github.com/chatmail/relay/pull/650))
 
-- filtermail: accept mails from Protonmail
-  ([#616](https://github.com/chatmail/relay/pull/616))
-
 - Ignore all RCPT TO: parameters
   ([#651](https://github.com/chatmail/relay/pull/651))
 
-- Increase opendkim DNS Timeout from 5 to 60 seconds
-  ([#672](https://github.com/chatmail/relay/pull/672))
-
-- Add config parameter for Let's Encrypt ACME email
-  ([#663](https://github.com/chatmail/relay/pull/663))
-
 - Use max username length in newemail.py, not min
   ([#648](https://github.com/chatmail/relay/pull/648))
 
-- Add startup for `fcgiwrap.service` because sometimes it did not start automatically.
-  ([#657](https://github.com/chatmail/relay/pull/657))
-
-- Add `cmdeploy init --force` command for recreating chatmail.ini
-  ([#656](https://github.com/chatmail/relay/pull/656))
-
 - Increase maxproc for reinjecting ports from 10 to 100
   ([#646](https://github.com/chatmail/relay/pull/646))
 
 - Allow ports 143 and 993 to be used by `dovecot` process
   ([#639](https://github.com/chatmail/relay/pull/639))
 
-- Add `--skip-dns-check` argument to `cmdeploy run` command, which disables DNS record checking before installation.
-  ([#661](https://github.com/chatmail/relay/pull/661))
-
-- Rework expiry of message files and mailboxes in Python 
-  to only do a single iteration over sometimes millions of messages
-  instead of doing "find" commands that iterate 9 times over the messages. 
-  Provide an "fsreport" CLI for more fine grained analysis of message files. 
-  ([#637](https://github.com/chatmail/relay/pull/637))
-
-
 ## 1.7.0 2025-09-11
 
 - Make www upload path configurable

README.md

@@ -180,10 +180,6 @@ The components of chatmail are:
 - [Iroh relay](https://www.iroh.computer/docs/concepts/relay) 
   which helps client devices to establish Peer-to-Peer connections 
 
-- [TURN](https://github.com/chatmail/chatmail-turn)
-  to enable relay users to start webRTC calls
-  even if a p2p connection can't be established
-
 - and the chatmaild services, explained in the next section:
 
 ### chatmaild
@@ -308,8 +304,6 @@ Chatmail address creation will be denied while this file is present.
 [Nginx](https://www.nginx.com/) listens on port 8443 (HTTPS-ALT) and 443 (HTTPS).
 Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
 [acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (HTTP).
-[chatmail-turn](https://github.com/chatmail/chatmail-turn) listens on UDP port 3478 (STUN/TURN),
-and temporarily opens UDP ports when users request them. UDP port range is not restricted, any free port may be allocated.
 
 chatmail-core based apps will, however, discover all ports and configurations
 automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail relay server.
@@ -462,94 +456,15 @@ to send messages outside.
 
 To setup a reverse proxy
 (or rather Destination NAT, DNAT)
-for your chatmail relay,
-put the following configuration in `/etc/nftables.conf`:
-```
-#!/usr/sbin/nft -f
-
-flush ruleset
-
-define wan = eth0
-
-# Which ports to proxy.
-#
-# Note that SSH is not proxied
-# so it is possible to log into the proxy server 
-# and not the original one.
-define ports = { smtp, http, https, imap, imaps, submission, submissions }
-
-# The host we want to proxy to.
-define ipv4_address = AAA.BBB.CCC.DDD
-define ipv6_address = [XXX::1]
-
-table ip nat {
-        chain prerouting {
-                type nat hook prerouting priority dstnat; policy accept;
-                iif $wan tcp dport $ports dnat to $ipv4_address
-        }
-
-        chain postrouting {
-                type nat hook postrouting priority 0;
-
-                oifname $wan masquerade
-        }
-}
-
-table ip6 nat {
-        chain prerouting {
-                type nat hook prerouting priority dstnat; policy accept;
-                iif $wan tcp dport $ports dnat to $ipv6_address
-        }
-
-        chain postrouting {
-                type nat hook postrouting priority 0;
-
-                oifname $wan masquerade
-        }
-}
-
-table inet filter {
-        chain input {
-                type filter hook input priority filter; policy drop;
-
-                # Accept ICMP.
-                # It is especially important to accept ICMPv6 ND messages,
-                # otherwise IPv6 connectivity breaks.
-                icmp type { echo-request } accept
-                icmpv6 type { echo-request, nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
-
-                # Allow incoming SSH connections.
-                tcp dport { ssh } accept
-
-                ct state established accept
-        }
-        chain forward {
-                type filter hook forward priority filter; policy drop;
-
-                ct state established accept
-                ip daddr $ipv4_address counter accept
-                ip6 daddr $ipv6_address counter accept
-        }
-        chain output {
-                type filter hook output priority filter;
-        }
-}
-```
-
-Run `systemctl enable nftables.service`
-to ensure configuration is reloaded when the proxy relay reboots.
-
-Uncomment in `/etc/sysctl.conf` the following two lines:
+for your chatmail relay, run:
 
 ```
-net.ipv4.ip_forward=1
-net.ipv6.conf.all.forwarding=1
+scripts/cmdeploy proxy <proxy_ip_address> --relay-ipv4 <relay_ipv4_address> --relay-ipv6 <relay_ipv6_address>
 ```
 
-Then reboot the relay or do `sysctl -p` and `nft -f /etc/nftables.conf`.
-
 Once proxy relay is set up,
-you can add its IP address to the DNS.
+you can add its IP address to the DNS,
+or distribute it as you wish.
 
 ## Neighbors and Acquaintances
 

chatmaild/pyproject.toml

@@ -27,10 +27,8 @@ chatmail-metadata = "chatmaild.metadata:main"
 filtermail = "chatmaild.filtermail:main"
 echobot = "chatmaild.echo:main"
 chatmail-metrics = "chatmaild.metrics:main"
-chatmail-expire = "chatmaild.expire:main"
-chatmail-fsreport = "chatmaild.fsreport:main"
+delete_inactive_users = "chatmaild.delete_inactive_users:main"
 lastlogin = "chatmaild.lastlogin:main"
-turnserver = "chatmaild.turnserver:main"
 
 [project.entry-points.pytest11]
 "chatmaild.testplugin" = "chatmaild.tests.plugin"
@@ -72,6 +70,5 @@ commands =
 [testenv]
 deps = pytest 
        pdbpp 
-       pytest-localserver
 commands = pytest -v -rsXx {posargs}
 """

chatmaild/src/chatmaild/config.py

@@ -44,7 +44,6 @@ class Config:
         )
         self.mtail_address = params.get("mtail_address")
         self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
-        self.acme_email = params.get("acme_email", "")
         self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
         if "iroh_relay" not in params:
             self.iroh_relay = "https://" + params["mail_domain"]

chatmaild/src/chatmaild/delete_inactive_users.py

@@ -0,0 +1,31 @@
+"""
+Remove inactive users
+"""
+
+import os
+import shutil
+import sys
+import time
+
+from .config import read_config
+
+
+def delete_inactive_users(config):
+    cutoff_date = time.time() - config.delete_inactive_users_after * 86400
+    for addr in os.listdir(config.mailboxes_dir):
+        try:
+            user = config.get_user(addr)
+        except ValueError:
+            continue
+
+        read_timestamp = user.get_last_login_timestamp()
+        if read_timestamp and read_timestamp < cutoff_date:
+            path = config.mailboxes_dir.joinpath(addr)
+            assert path == user.maildir
+            shutil.rmtree(path, ignore_errors=True)
+
+
+def main():
+    (cfgpath,) = sys.argv[1:]
+    config = read_config(cfgpath)
+    delete_inactive_users(config)

chatmaild/src/chatmaild/expire.py

@@ -1,218 +0,0 @@
-"""
-Expire old messages and addresses.
-
-"""
-
-import os
-import shutil
-import sys
-import time
-from argparse import ArgumentParser
-from collections import namedtuple
-from datetime import datetime
-from stat import S_ISREG
-
-from chatmaild.config import read_config
-
-FileEntry = namedtuple("FileEntry", ("relpath", "mtime", "size"))
-
-
-def iter_mailboxes(basedir, maxnum):
-    if not os.path.exists(basedir):
-        print_info(f"no mailboxes found at: {basedir}")
-        return
-
-    for name in os_listdir_if_exists(basedir)[:maxnum]:
-        if "@" in name:
-            yield MailboxStat(basedir + "/" + name)
-
-
-def get_file_entry(path):
-    """return a FileEntry or None if the path does not exist or is not a regular file."""
-    try:
-        st = os.stat(path)
-    except FileNotFoundError:
-        return None
-    if not S_ISREG(st.st_mode):
-        return None
-    return FileEntry(path, st.st_mtime, st.st_size)
-
-
-def os_listdir_if_exists(path):
-    """return a list of names obtained from os.listdir or an empty list if the path does not exist."""
-    try:
-        return os.listdir(path)
-    except FileNotFoundError:
-        return []
-
-
-class MailboxStat:
-    last_login = None
-
-    def __init__(self, basedir):
-        self.basedir = str(basedir)
-        # all detected messages in cur/new/tmp folders
-        self.messages = []
-
-        # all detected files in mailbox top dir
-        self.extrafiles = []
-
-        # scan all relevant files (without recursion)
-        old_cwd = os.getcwd()
-        try:
-            os.chdir(self.basedir)
-        except FileNotFoundError:
-            return
-        for name in os_listdir_if_exists("."):
-            if name in ("cur", "new", "tmp"):
-                for msg_name in os_listdir_if_exists(name):
-                    entry = get_file_entry(f"{name}/{msg_name}")
-                    if entry is not None:
-                        self.messages.append(entry)
-
-            else:
-                entry = get_file_entry(name)
-                if entry is not None:
-                    self.extrafiles.append(entry)
-                    if name == "password":
-                        self.last_login = entry.mtime
-        self.extrafiles.sort(key=lambda x: -x.size)
-        os.chdir(old_cwd)
-
-
-def print_info(msg):
-    print(msg, file=sys.stderr)
-
-
-class Expiry:
-    def __init__(self, config, dry, now, verbose):
-        self.config = config
-        self.dry = dry
-        self.now = now
-        self.verbose = verbose
-        self.del_mboxes = 0
-        self.all_mboxes = 0
-        self.del_files = 0
-        self.all_files = 0
-        self.start = time.time()
-
-    def remove_mailbox(self, mboxdir):
-        if self.verbose:
-            print_info(f"removing {mboxdir}")
-        if not self.dry:
-            shutil.rmtree(mboxdir)
-        self.del_mboxes += 1
-
-    def remove_file(self, path, mtime=None):
-        if self.verbose:
-            if mtime is not None:
-                date = datetime.fromtimestamp(mtime).strftime("%b %d")
-                print_info(f"removing {date} {path}")
-            else:
-                print_info(f"removing {path}")
-        if not self.dry:
-            try:
-                os.unlink(path)
-            except FileNotFoundError:
-                print_info(f"file not found/vanished {path}")
-        self.del_files += 1
-
-    def process_mailbox_stat(self, mbox):
-        cutoff_without_login = (
-            self.now - int(self.config.delete_inactive_users_after) * 86400
-        )
-        cutoff_mails = self.now - int(self.config.delete_mails_after) * 86400
-        cutoff_large_mails = self.now - int(self.config.delete_large_after) * 86400
-
-        self.all_mboxes += 1
-        changed = False
-        if mbox.last_login and mbox.last_login < cutoff_without_login:
-            self.remove_mailbox(mbox.basedir)
-            return
-
-        # all to-be-removed files are relative to the mailbox basedir
-        try:
-            os.chdir(mbox.basedir)
-        except FileNotFoundError:
-            print_info(f"mailbox not found/vanished {mbox.basedir}")
-            return
-
-        mboxname = os.path.basename(mbox.basedir)
-        if self.verbose:
-            date = datetime.fromtimestamp(mbox.last_login) if mbox.last_login else None
-            if date:
-                print_info(f"checking mailbox {date.strftime('%b %d')} {mboxname}")
-            else:
-                print_info(f"checking mailbox (no last_login) {mboxname}")
-        self.all_files += len(mbox.messages)
-        for message in mbox.messages:
-            if message.mtime < cutoff_mails:
-                self.remove_file(message.relpath, mtime=message.mtime)
-            elif message.size > 200000 and message.mtime < cutoff_large_mails:
-                # we only remove noticed large files (not unnoticed ones in new/)
-                if message.relpath.startswith("cur/"):
-                    self.remove_file(message.relpath, mtime=message.mtime)
-            else:
-                continue
-            changed = True
-        if changed:
-            self.remove_file("maildirsize")
-
-    def get_summary(self):
-        return (
-            f"Removed {self.del_mboxes} out of {self.all_mboxes} mailboxes "
-            f"and {self.del_files} out of {self.all_files} files in existing mailboxes "
-            f"in {time.time() - self.start:2.2f} seconds"
-        )
-
-
-def main(args=None):
-    """Expire mailboxes and messages according to chatmail config"""
-    parser = ArgumentParser(description=main.__doc__)
-    ini = "/usr/local/lib/chatmaild/chatmail.ini"
-    parser.add_argument(
-        "chatmail_ini",
-        action="store",
-        nargs="?",
-        help=f"path pointing to chatmail.ini file, default: {ini}",
-        default=ini,
-    )
-    parser.add_argument(
-        "--days", action="store", help="assume date to be days older than now"
-    )
-
-    parser.add_argument(
-        "--maxnum",
-        default=None,
-        action="store",
-        help="maximum number of mailboxes to iterate on",
-    )
-    parser.add_argument(
-        "-v",
-        dest="verbose",
-        action="store_true",
-        help="print out removed files and mailboxes",
-    )
-
-    parser.add_argument(
-        "--remove",
-        dest="remove",
-        action="store_true",
-        help="actually remove all expired files and dirs",
-    )
-    args = parser.parse_args(args)
-
-    config = read_config(args.chatmail_ini)
-    now = datetime.utcnow().timestamp()
-    if args.days:
-        now = now - 86400 * int(args.days)
-
-    maxnum = int(args.maxnum) if args.maxnum else None
-    exp = Expiry(config, dry=not args.remove, now=now, verbose=args.verbose)
-    for mailbox in iter_mailboxes(str(config.mailboxes_dir), maxnum=maxnum):
-        exp.process_mailbox_stat(mailbox)
-    print(exp.get_summary())
-
-
-if __name__ == "__main__":
-    main(sys.argv[1:])

chatmaild/src/chatmaild/filtermail.py

@@ -2,6 +2,7 @@
 import asyncio
 import base64
 import binascii
+import logging
 import sys
 import time
 from email import policy
@@ -82,14 +83,8 @@ def check_openpgp_payload(payload: bytes):
     return False
 
 
-def check_armored_payload(payload: str, outgoing: bool):
-    """Check the armored PGP message for invalid content.
-
-    :param payload: the armored PGP message
-    :param outgoing: whether the message is outgoing or incoming
-    :return: whether the message is a valid PGP message
-    """
-    prefix = "-----BEGIN PGP MESSAGE-----\r\n"
+def check_armored_payload(payload: str):
+    prefix = "-----BEGIN PGP MESSAGE-----\r\n\r\n"
     if not payload.startswith(prefix):
         return False
     payload = payload.removeprefix(prefix)
@@ -101,16 +96,6 @@ def check_armored_payload(payload: str, outgoing: bool):
         return False
     payload = payload.removesuffix(suffix)
 
-    version_comment = "Version: "
-    if payload.startswith(version_comment):
-        if outgoing:  # Disallow comments in outgoing messages
-            return False
-        # Remove comments from incoming messages
-        payload = payload.partition("\r\n")[2]
-
-    while payload.startswith("\r\n"):
-        payload = payload.removeprefix("\r\n")
-
     # Remove CRC24.
     payload = payload.rpartition("=")[0]
 
@@ -146,7 +131,7 @@ def is_securejoin(message):
     return True
 
 
-def check_encrypted(message, outgoing=True):
+def check_encrypted(message):
     """Check that the message is an OpenPGP-encrypted message.
 
     MIME structure of the message must correspond to <https://www.rfc-editor.org/rfc/rfc3156>.
@@ -173,7 +158,7 @@ def check_encrypted(message, outgoing=True):
             if part.get_content_type() != "application/octet-stream":
                 return False
 
-            if not check_armored_payload(part.get_payload(), outgoing=outgoing):
+            if not check_armored_payload(part.get_payload()):
                 return False
         else:
             return False
@@ -227,7 +212,7 @@ class OutgoingBeforeQueueHandler:
         self.send_rate_limiter = SendRateLimiter()
 
     async def handle_MAIL(self, server, session, envelope, address, mail_options):
-        log_info(f"handle_MAIL from {address}")
+        logging.info(f"handle_MAIL from {address}")
         envelope.mail_from = address
         max_sent = self.config.max_user_send_per_minute
         if not self.send_rate_limiter.is_sending_allowed(address, max_sent):
@@ -240,15 +225,11 @@ class OutgoingBeforeQueueHandler:
         return "250 OK"
 
     async def handle_DATA(self, server, session, envelope):
-        loop = asyncio.get_running_loop()
-        return await loop.run_in_executor(None, self.sync_handle_DATA, envelope)
-
-    def sync_handle_DATA(self, envelope):
-        log_info("handle_DATA before-queue")
+        logging.info("handle_DATA before-queue")
         error = self.check_DATA(envelope)
         if error:
             return error
-        log_info("re-injecting the mail that passed checks")
+        logging.info("re-injecting the mail that passed checks")
         client = SMTPClient("localhost", self.config.postfix_reinject_port)
         client.sendmail(
             envelope.mail_from, envelope.rcpt_tos, envelope.original_content
@@ -257,10 +238,10 @@ class OutgoingBeforeQueueHandler:
 
     def check_DATA(self, envelope):
         """the central filtering function for e-mails."""
-        log_info(f"Processing DATA message from {envelope.mail_from}")
+        logging.info(f"Processing DATA message from {envelope.mail_from}")
 
         message = BytesParser(policy=policy.default).parsebytes(envelope.content)
-        mail_encrypted = check_encrypted(message, outgoing=True)
+        mail_encrypted = check_encrypted(message)
 
         _, from_addr = parseaddr(message.get("from").strip())
 
@@ -297,15 +278,11 @@ class IncomingBeforeQueueHandler:
         self.config = config
 
     async def handle_DATA(self, server, session, envelope):
-        loop = asyncio.get_running_loop()
-        return await loop.run_in_executor(None, self.sync_handle_DATA, envelope)
-
-    def sync_handle_DATA(self, envelope):
-        log_info("handle_DATA before-queue")
+        logging.info("handle_DATA before-queue")
         error = self.check_DATA(envelope)
         if error:
             return error
-        log_info("re-injecting the mail that passed checks")
+        logging.info("re-injecting the mail that passed checks")
 
         # the smtp daemon on reinject_port_incoming gives it to dkim milter
         # which looks at source address to determine whether to verify or sign
@@ -321,10 +298,10 @@ class IncomingBeforeQueueHandler:
 
     def check_DATA(self, envelope):
         """the central filtering function for e-mails."""
-        log_info(f"Processing DATA message from {envelope.mail_from}")
+        logging.info(f"Processing DATA message from {envelope.mail_from}")
 
         message = BytesParser(policy=policy.default).parsebytes(envelope.content)
-        mail_encrypted = check_encrypted(message, outgoing=False)
+        mail_encrypted = check_encrypted(message)
 
         if mail_encrypted or is_securejoin(message):
             print("Incoming: Filtering encrypted mail.", file=sys.stderr)
@@ -363,19 +340,16 @@ class SendRateLimiter:
         return False
 
 
-def log_info(msg):
-    print(msg, file=sys.stderr)
-
-
 def main():
     args = sys.argv[1:]
     assert len(args) == 2
     config = read_config(args[0])
     mode = args[1]
+    logging.basicConfig(level=logging.WARN)
     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
     assert mode in ["incoming", "outgoing"]
     task = asyncmain_beforequeue(config, mode)
     loop.create_task(task)
-    log_info("entering serving loop")
+    logging.info("entering serving loop")
     loop.run_forever()

chatmaild/src/chatmaild/fsreport.py

@@ -1,168 +0,0 @@
-"""
-command line tool to analyze mailbox message storage
-
-example invocation:
-
-    python -m chatmaild.fsreport /path/to/chatmail.ini
-
-to show storage summaries for all "cur" folders
-
-    python -m chatmaild.fsreport /path/to/chatmail.ini --mdir cur
-
-to show storage summaries only for first 1000 mailboxes
-
-    python -m chatmaild.fsreport /path/to/chatmail.ini --maxnum 1000
-
-"""
-
-import os
-from argparse import ArgumentParser
-from datetime import datetime
-
-from chatmaild.config import read_config
-from chatmaild.expire import iter_mailboxes
-
-DAYSECONDS = 24 * 60 * 60
-MONTHSECONDS = DAYSECONDS * 30
-
-
-def HSize(size: int):
-    """Format a size integer as a Human-readable string Kilobyte, Megabyte or Gigabyte"""
-    if size < 10000:
-        return f"{size / 1000:5.2f}K"
-    if size < 1000 * 1000:
-        return f"{size / 1000:5.0f}K"
-    if size < 1000 * 1000 * 1000:
-        return f"{int(size / 1000000):5.0f}M"
-    return f"{size / 1000000000:5.2f}G"
-
-
-class Report:
-    def __init__(self, now, min_login_age, mdir):
-        self.size_extra = 0
-        self.size_messages = 0
-        self.now = now
-        self.min_login_age = min_login_age
-        self.mdir = mdir
-
-        self.num_ci_logins = self.num_all_logins = 0
-        self.login_buckets = {x: 0 for x in (1, 10, 30, 40, 80, 100, 150)}
-
-        self.message_buckets = {x: 0 for x in (0, 160000, 500000, 2000000)}
-
-    def process_mailbox_stat(self, mailbox):
-        # categorize login times
-        last_login = mailbox.last_login
-        if last_login:
-            self.num_all_logins += 1
-            if os.path.basename(mailbox.basedir)[:3] == "ci-":
-                self.num_ci_logins += 1
-            else:
-                for days in self.login_buckets:
-                    if last_login >= self.now - days * DAYSECONDS:
-                        self.login_buckets[days] += 1
-
-        cutoff_login_date = self.now - self.min_login_age * DAYSECONDS
-        if last_login and last_login <= cutoff_login_date:
-            # categorize message sizes
-            for size in self.message_buckets:
-                for msg in mailbox.messages:
-                    if msg.size >= size:
-                        if self.mdir and not msg.relpath.startswith(self.mdir):
-                            continue
-                        self.message_buckets[size] += msg.size
-
-        self.size_messages += sum(entry.size for entry in mailbox.messages)
-        self.size_extra += sum(entry.size for entry in mailbox.extrafiles)
-
-    def dump_summary(self):
-        all_messages = self.size_messages
-        print()
-        print("## Mailbox storage use analysis")
-        print(f"Mailbox data total size: {HSize(self.size_extra + all_messages)}")
-        print(f"Messages total size    : {HSize(all_messages)}")
-        try:
-            percent = self.size_extra / (self.size_extra + all_messages) * 100
-        except ZeroDivisionError:
-            percent = 100
-        print(f"Extra files : {HSize(self.size_extra)} ({percent:.2f}%)")
-
-        print()
-        if self.min_login_age:
-            print(f"### Message storage for {self.min_login_age} days old logins")
-
-        pref = f"[{self.mdir}] " if self.mdir else ""
-        for minsize, sumsize in self.message_buckets.items():
-            percent = (sumsize / all_messages * 100) if all_messages else 0
-            print(
-                f"{pref}larger than {HSize(minsize)}: {HSize(sumsize)} ({percent:.2f}%)"
-            )
-
-        user_logins = self.num_all_logins - self.num_ci_logins
-
-        def p(num):
-            return f"({num / user_logins * 100:2.2f}%)" if user_logins else "100%"
-
-        print()
-        print(f"## Login stats, from date reference {datetime.fromtimestamp(self.now)}")
-        print(f"all:     {HSize(self.num_all_logins)}")
-        print(f"non-ci:  {HSize(user_logins)}")
-        print(f"ci:      {HSize(self.num_ci_logins)}")
-        for days, active in self.login_buckets.items():
-            print(f"last {days:3} days: {HSize(active)} {p(active)}")
-
-
-def main(args=None):
-    """Report about filesystem storage usage of all mailboxes and messages"""
-    parser = ArgumentParser(description=main.__doc__)
-    ini = "/usr/local/lib/chatmaild/chatmail.ini"
-    parser.add_argument(
-        "chatmail_ini",
-        action="store",
-        nargs="?",
-        help=f"path pointing to chatmail.ini file, default: {ini}",
-        default=ini,
-    )
-    parser.add_argument(
-        "--days",
-        default=0,
-        action="store",
-        help="assume date to be days older than now",
-    )
-    parser.add_argument(
-        "--min-login-age",
-        default=0,
-        dest="min_login_age",
-        action="store",
-        help="only sum up message size if last login is at least min-login-age days old",
-    )
-    parser.add_argument(
-        "--mdir",
-        action="store",
-        help="only consider 'cur' or 'new' or 'tmp' messages for summary",
-    )
-
-    parser.add_argument(
-        "--maxnum",
-        default=None,
-        action="store",
-        help="maximum number of mailboxes to iterate on",
-    )
-
-    args = parser.parse_args(args)
-
-    config = read_config(args.chatmail_ini)
-
-    now = datetime.utcnow().timestamp()
-    if args.days:
-        now = now - 86400 * int(args.days)
-
-    maxnum = int(args.maxnum) if args.maxnum else None
-    rep = Report(now=now, min_login_age=int(args.min_login_age), mdir=args.mdir)
-    for mbox in iter_mailboxes(str(config.mailboxes_dir), maxnum=maxnum):
-        rep.process_mailbox_stat(mbox)
-    rep.dump_summary()
-
-
-if __name__ == "__main__":
-    main()

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -45,9 +45,6 @@ passthrough_senders =
 # (space-separated, item may start with "@" to whitelist whole recipient domains)
 passthrough_recipients = xstore@testrun.org echo@{mail_domain}
 
-# path to www directory - documented here: https://github.com/chatmail/relay/#custom-web-pages
-#www_folder = www
-
 #
 # Deployment Details
 #
@@ -63,9 +60,6 @@ postfix_reinject_port_incoming = 10026
 # if set to "True" IPv6 is disabled
 disable_ipv6 = False
 
-# Your email adress, which will be used in acmetool to manage Let's Encrypt SSL certificates
-acme_email = 
-
 # Defaults to https://iroh.{{mail_domain}} and running `iroh-relay` on the chatmail
 # service.
 # If you set it to anything else, the service will be disabled

chatmaild/src/chatmaild/metadata.py

@@ -7,7 +7,6 @@ from .config import read_config
 from .dictproxy import DictProxy
 from .filedict import FileDict
 from .notifier import Notifier
-from .turnserver import turn_credentials
 
 
 def _is_valid_token_timestamp(timestamp, now):
@@ -76,12 +75,11 @@ class Metadata:
 
 
 class MetadataDictProxy(DictProxy):
-    def __init__(self, notifier, metadata, iroh_relay=None, turn_hostname=None):
+    def __init__(self, notifier, metadata, iroh_relay=None):
         super().__init__()
         self.notifier = notifier
         self.metadata = metadata
         self.iroh_relay = iroh_relay
-        self.turn_hostname = turn_hostname
 
     def handle_lookup(self, parts):
         # Lpriv/43f5f508a7ea0366dff30200c15250e3/devicetoken\tlkj123poi@c2.testrun.org
@@ -100,11 +98,6 @@ class MetadataDictProxy(DictProxy):
             ):
                 # Handle `GETMETADATA "" /shared/vendor/deltachat/irohrelay`
                 return f"O{self.iroh_relay}\n"
-            elif keyname == "vendor/vendor.dovecot/pvt/server/vendor/deltachat/turn":
-                res = turn_credentials()
-                port = 3478
-                return f"O{self.turn_hostname}:{port}:{res}\n"
-
         logging.warning(f"lookup ignored: {parts!r}")
         return "N\n"
 
@@ -128,7 +121,6 @@ def main():
 
     config = read_config(config_path)
     iroh_relay = config.iroh_relay
-    mail_domain = config.mail_domain
 
     vmail_dir = config.mailboxes_dir
     if not vmail_dir.exists():
@@ -142,10 +134,7 @@ def main():
     notifier.start_notification_threads(metadata.remove_token_from_addr)
 
     dictproxy = MetadataDictProxy(
-        notifier=notifier,
-        metadata=metadata,
-        iroh_relay=iroh_relay,
-        turn_hostname=mail_domain,
+        notifier=notifier, metadata=metadata, iroh_relay=iroh_relay
     )
 
     dictproxy.serve_forever_from_socket(socket)

chatmaild/src/chatmaild/tests/test_delete_inactive_users.py

@@ -1,7 +1,7 @@
 import time
 
+from chatmaild.delete_inactive_users import delete_inactive_users
 from chatmaild.doveauth import AuthDictProxy
-from chatmaild.expire import main as main_expire
 
 
 def test_login_timestamps(example_config):
@@ -45,12 +45,7 @@ def test_delete_inactive_users(example_config):
     for addr in to_remove:
         assert example_config.get_user(addr).maildir.exists()
 
-    main_expire(
-        args=[
-            "--remove",
-            str(example_config._inipath),
-        ]
-    )
+    delete_inactive_users(example_config)
 
     for p in example_config.mailboxes_dir.iterdir():
         assert not p.name.startswith("old")

chatmaild/src/chatmaild/tests/test_expire.py

@@ -1,150 +0,0 @@
-import os
-import random
-from datetime import datetime
-from fnmatch import fnmatch
-from pathlib import Path
-
-import pytest
-
-from chatmaild.expire import (
-    FileEntry,
-    MailboxStat,
-    get_file_entry,
-    iter_mailboxes,
-    os_listdir_if_exists,
-)
-from chatmaild.expire import main as expiry_main
-from chatmaild.fsreport import main as report_main
-
-
-def fill_mbox(basedir):
-    basedir1 = basedir.joinpath("mailbox1@example.org")
-    basedir1.mkdir()
-    password = basedir1.joinpath("password")
-    password.write_text("xxx")
-    basedir1.joinpath("maildirsize").write_text("xxx")
-
-    garbagedir = basedir1.joinpath("garbagedir")
-    garbagedir.mkdir()
-
-    create_new_messages(basedir1, ["cur/msg1"], size=500)
-    create_new_messages(basedir1, ["new/msg2"], size=600)
-    return basedir1
-
-
-def create_new_messages(basedir, relpaths, size=1000, days=0):
-    now = datetime.utcnow().timestamp()
-
-    for relpath in relpaths:
-        msg_path = Path(basedir).joinpath(relpath)
-        msg_path.parent.mkdir(parents=True, exist_ok=True)
-        msg_path.write_text("x" * size)
-        # accessed now, modified N days ago
-        os.utime(msg_path, (now, now - days * 86400))
-
-
-@pytest.fixture
-def mbox1(example_config):
-    basedir1 = fill_mbox(example_config.mailboxes_dir)
-    return MailboxStat(basedir1)
-
-
-def test_filentry_ordering(tmp_path):
-    l = [FileEntry(f"x{i}", size=i + 10, mtime=1000 - i) for i in range(10)]
-    sorted = list(l)
-    random.shuffle(l)
-    l.sort(key=lambda x: x.size)
-    assert l == sorted
-
-
-def test_no_mailbxoes(tmp_path, capsys):
-    assert [] == list(iter_mailboxes(str(tmp_path.joinpath("notexists")), maxnum=10))
-    out, err = capsys.readouterr()
-    assert "no mailboxes" in err
-
-
-def test_stats_mailbox(mbox1):
-    password = Path(mbox1.basedir).joinpath("password")
-    assert mbox1.last_login == password.stat().st_mtime
-    assert len(mbox1.messages) == 2
-
-    msgs = list(sorted(mbox1.messages, key=lambda x: x.size))
-    assert len(msgs) == 2
-    assert msgs[0].size == 500  # cur
-    assert msgs[1].size == 600  # new
-
-    create_new_messages(mbox1.basedir, ["large-extra"], size=1000)
-    create_new_messages(mbox1.basedir, ["index-something"], size=3)
-    mbox2 = MailboxStat(mbox1.basedir)
-    assert len(mbox2.extrafiles) == 4
-    assert mbox2.extrafiles[0].size == 1000
-
-    # cope well with mailbox dirs that have no password (for whatever reason)
-    Path(mbox1.basedir).joinpath("password").unlink()
-    mbox3 = MailboxStat(mbox1.basedir)
-    assert mbox3.last_login is None
-
-
-def test_report_no_mailboxes(example_config):
-    args = (str(example_config._inipath),)
-    report_main(args)
-
-
-def test_report(mbox1, example_config):
-    args = (str(example_config._inipath),)
-    report_main(args)
-    args = list(args) + "--days 1".split()
-    report_main(args)
-    args = list(args) + "--min-login-age 1".split()
-    report_main(args)
-    args = list(args) + "--mdir cur".split()
-    report_main(args)
-
-
-def test_expiry_cli_basic(example_config, mbox1):
-    args = (str(example_config._inipath),)
-    expiry_main(args)
-
-
-def test_expiry_cli_old_files(capsys, example_config, mbox1):
-    relpaths_old = ["cur/msg_old1", "cur/msg_old1"]
-    cutoff_days = int(example_config.delete_mails_after) + 1
-    create_new_messages(mbox1.basedir, relpaths_old, size=1000, days=cutoff_days)
-
-    relpaths_large = ["cur/msg_old_large1", "new/msg_old_large2"]
-    cutoff_days = int(example_config.delete_large_after) + 1
-    create_new_messages(
-        mbox1.basedir, relpaths_large, size=1000 * 300, days=cutoff_days
-    )
-
-    create_new_messages(mbox1.basedir, ["cur/shouldstay"], size=1000 * 300, days=1)
-
-    args = str(example_config._inipath), "--remove", "-v"
-    expiry_main(args)
-    out, err = capsys.readouterr()
-
-    allpaths = relpaths_old + relpaths_large + ["maildirsize"]
-    for path in allpaths:
-        for line in err.split("\n"):
-            if fnmatch(line, f"removing*{path}"):
-                break
-        else:
-            if path != "new/msg_old_large2":
-                pytest.fail(f"failed to remove {path}\n{err}")
-
-    assert "shouldstay" not in err
-
-
-def test_get_file_entry(tmp_path):
-    assert get_file_entry(str(tmp_path.joinpath("123123"))) is None
-    p = tmp_path.joinpath("x")
-    p.write_text("hello")
-    entry = get_file_entry(str(p))
-    assert entry.size == 5
-    assert entry.mtime
-
-
-def test_os_listdir_if_exists(tmp_path):
-    tmp_path.joinpath("x").write_text("hello")
-    assert len(os_listdir_if_exists(str(tmp_path))) == 1
-    assert len(os_listdir_if_exists(str(tmp_path.joinpath("123123")))) == 0

chatmaild/src/chatmaild/tests/test_filtermail.py

@@ -241,9 +241,8 @@ def test_cleartext_passthrough_senders(gencreds, handler, maildata):
 
 
 def test_check_armored_payload():
-    prefix = "-----BEGIN PGP MESSAGE-----\r\n"
-    comment = "Version: ProtonMail\r\n"
-    payload = """\r
+    payload = """-----BEGIN PGP MESSAGE-----\r
+\r
 wU4DSqFx0d1yqAoSAQdAYkX/ZN/Az4B0k7X47zKyWrXxlDEdS3WOy0Yf2+GJTFgg\r
 Zk5ql0mLG8Ze+ZifCS0XMO4otlemSyJ0K1ZPdFMGzUDBTgNqzkFabxXoXRIBB0AM\r
 755wlX41X6Ay3KhnwBq7yEqSykVH6F3x11iHPKraLCAGZoaS8bKKNy/zg5slda1X\r
@@ -279,25 +278,16 @@ UN4fiB0KR9JyG2ayUdNJVkXZSZLnHyRgiaadlpUo16LVvw==\r
 \r
 """
 
-    commented_payload = prefix + comment + payload
-    assert check_armored_payload(commented_payload, outgoing=False) == True
-    assert check_armored_payload(commented_payload, outgoing=True) == False
-
-    payload = prefix + payload
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
+    assert check_armored_payload(payload) == True
 
     payload = payload.removesuffix("\r\n")
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
+    assert check_armored_payload(payload) == True
 
     payload = payload.removesuffix("\r\n")
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
+    assert check_armored_payload(payload) == True
 
     payload = payload.removesuffix("\r\n")
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
+    assert check_armored_payload(payload) == True
 
     payload = """-----BEGIN PGP MESSAGE-----\r
 \r
@@ -305,8 +295,7 @@ HELLOWORLD
 -----END PGP MESSAGE-----\r
 \r
 """
-    assert check_armored_payload(payload, outgoing=False) == False
-    assert check_armored_payload(payload, outgoing=True) == False
+    assert check_armored_payload(payload) == False
 
     payload = """-----BEGIN PGP MESSAGE-----\r
 \r
@@ -314,8 +303,7 @@ HELLOWORLD
 -----END PGP MESSAGE-----\r
 \r
 """
-    assert check_armored_payload(payload, outgoing=False) == False
-    assert check_armored_payload(payload, outgoing=True) == False
+    assert check_armored_payload(payload) == False
 
     # Test payload using partial body length
     # as generated by GopenPGP.
@@ -357,5 +345,4 @@ myLbG7cJB787QjplEyVe2P/JBO6xYvbkJLf9Q+HaviTO25rugRSrYsoKMDfO8VlQ\r
 =6iHb\r
 -----END PGP MESSAGE-----\r
 """
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
+    assert check_armored_payload(payload) == True

chatmaild/src/chatmaild/tests/test_filtermail_blackbox.py

@@ -1,78 +0,0 @@
-import smtplib
-import subprocess
-import sys
-
-import pytest
-
-
-@pytest.fixture
-def smtpserver():
-    from pytest_localserver import smtp
-
-    server = smtp.Server("127.0.0.1")
-    server.start()
-    yield server
-    server.stop()
-
-
-@pytest.fixture
-def make_popen(request):
-    def popen(cmdargs, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kw):
-        p = subprocess.Popen(
-            cmdargs,
-            stdout=subprocess.PIPE,
-            stderr=subprocess.PIPE,
-        )
-
-        def fin():
-            p.terminate()
-            out, err = p.communicate()
-            print(out.decode("ascii"))
-            print(err.decode("ascii"), file=sys.stderr)
-
-        request.addfinalizer(fin)
-        return p
-
-    return popen
-
-
-@pytest.mark.parametrize("filtermail_mode", ["outgoing", "incoming"])
-def test_one_mail(
-    make_config, make_popen, smtpserver, maildata, filtermail_mode, monkeypatch
-):
-    monkeypatch.setenv("PYTHONUNBUFFERED", "1")
-    smtp_inject_port = 20025
-    if filtermail_mode == "outgoing":
-        settings = dict(
-            postfix_reinject_port=smtpserver.port,
-            filtermail_smtp_port=smtp_inject_port,
-        )
-    else:
-        settings = dict(
-            postfix_reinject_port_incoming=smtpserver.port,
-            filtermail_smtp_port_incoming=smtp_inject_port,
-        )
-
-    config = make_config("example.org", settings=settings)
-    path = str(config._inipath)
-
-    popen = make_popen(["filtermail", path, filtermail_mode])
-    line = popen.stderr.readline().strip()
-    if b"loop" not in line:
-        print(line.decode("ascii"), file=sys.stderr)
-        pytest.fail("starting filtermail failed")
-
-    addr = f"user1@{config.mail_domain}"
-    config.get_user(addr).set_password("l1k2j3l1k2j3l")
-
-    # send encrypted mail
-    data = str(maildata("encrypted.eml", from_addr=addr, to_addr=addr))
-    client = smtplib.SMTP("localhost", smtp_inject_port)
-    client.sendmail(addr, [addr], data)
-    assert len(smtpserver.outbox) == 1
-
-    # send un-encrypted mail that errors
-    data = str(maildata("fake-encrypted.eml", from_addr=addr, to_addr=addr))
-    with pytest.raises(smtplib.SMTPDataError) as e:
-        client.sendmail(addr, [addr], data)
-    assert e.value.smtp_code == 523

chatmaild/src/chatmaild/turnserver.py

@@ -1,9 +0,0 @@
-#!/usr/bin/env python3
-import socket
-
-
-def turn_credentials() -> str:
-    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_socket:
-        client_socket.connect("/run/chatmail-turn/turn.socket")
-        with client_socket.makefile("rb") as file:
-            return file.readline().decode("utf-8").strip()

cmdeploy/src/cmdeploy/acmetool/__init__.py

@@ -2,79 +2,66 @@ import importlib.resources
 
 from pyinfra.operations import apt, files, server, systemd
 
-from ..deployer import Deployer
 
+def deploy_acmetool(email="", domains=[]):
+    """Deploy acmetool."""
+    apt.packages(
+        name="Install acmetool",
+        packages=["acmetool"],
+    )
 
-class AcmetoolDeployer(Deployer):
-    def __init__(self, *, email, domains, **kwargs):
-        super().__init__(**kwargs)
-        self.domains = domains
-        self.email = email
-        self.need_restart = False
+    files.put(
+        src=importlib.resources.files(__package__).joinpath("acmetool.cron").open("rb"),
+        dest="/etc/cron.d/acmetool",
+        user="root",
+        group="root",
+        mode="644",
+    )
 
-    @staticmethod
-    def install_impl():
-        apt.packages(
-            name="Install acmetool",
-            packages=["acmetool"],
-        )
+    files.put(
+        src=importlib.resources.files(__package__).joinpath("acmetool.hook").open("rb"),
+        dest="/usr/lib/acme/hooks/nginx",
+        user="root",
+        group="root",
+        mode="744",
+    )
 
-    def configure_impl(self):
-        files.put(
-            src=importlib.resources.files(__package__).joinpath("acmetool.cron").open("rb"),
-            dest="/etc/cron.d/acmetool",
-            user="root",
-            group="root",
-            mode="644",
-        )
+    files.template(
+        src=importlib.resources.files(__package__).joinpath("response-file.yaml.j2"),
+        dest="/var/lib/acme/conf/responses",
+        user="root",
+        group="root",
+        mode="644",
+        email=email,
+    )
 
-        files.put(
-            src=importlib.resources.files(__package__).joinpath("acmetool.hook").open("rb"),
-            dest="/usr/lib/acme/hooks/nginx",
-            user="root",
-            group="root",
-            mode="744",
-        )
+    files.template(
+        src=importlib.resources.files(__package__).joinpath("target.yaml.j2"),
+        dest="/var/lib/acme/conf/target",
+        user="root",
+        group="root",
+        mode="644",
+    )
 
-        files.template(
-            src=importlib.resources.files(__package__).joinpath("response-file.yaml.j2"),
-            dest="/var/lib/acme/conf/responses",
-            user="root",
-            group="root",
-            mode="644",
-            email=self.email,
-        )
+    service_file = files.put(
+        src=importlib.resources.files(__package__).joinpath(
+            "acmetool-redirector.service"
+        ),
+        dest="/etc/systemd/system/acmetool-redirector.service",
+        user="root",
+        group="root",
+        mode="644",
+    )
 
-        files.template(
-            src=importlib.resources.files(__package__).joinpath("target.yaml.j2"),
-            dest="/var/lib/acme/conf/target",
-            user="root",
-            group="root",
-            mode="644",
-        )
+    systemd.service(
+        name="Setup acmetool-redirector service",
+        service="acmetool-redirector.service",
+        running=True,
+        enabled=True,
+        restarted=service_file.changed,
+    )
 
-        service_file = files.put(
-            src=importlib.resources.files(__package__).joinpath(
-                "acmetool-redirector.service"
-            ),
-            dest="/etc/systemd/system/acmetool-redirector.service",
-            user="root",
-            group="root",
-            mode="644",
-        )
-        self.need_restart = service_file.changed
-
-    def activate_impl(self):
-        systemd.service(
-            name="Setup acmetool-redirector service",
-            service="acmetool-redirector.service",
-            running=True,
-            enabled=True,
-            restarted=self.need_restart,
-        )
-        self.need_restart = False
-
-        server.shell(
-            name=f"Request certificate for: {', '.join(self.domains)}",
-            commands=[f"acmetool want --xlog.severity=debug {' '.join(self.domains)}"],
-        )
+    server.shell(
+        name=f"Request certificate for: {', '.join(domains)}",
+        commands=[f"acmetool want --xlog.severity=debug {' '.join(domains)}"],
+    )

cmdeploy/src/cmdeploy/acmetool/target.yaml.j2

@@ -1,8 +1,7 @@
 request:
   provider: https://acme-v02.api.letsencrypt.org/directory
   key:
-    type: ecdsa
-    ecdsa-curve: nistp256
+    type: rsa
   challenge:
     webroot-paths:
     - /var/www/html/.well-known/acme-challenge

cmdeploy/src/cmdeploy/cmdeploy.py

@@ -19,7 +19,7 @@ from packaging import version
 from termcolor import colored
 
 from . import dns, remote
-from .sshexec import SSHExec, LocalExec
+from .sshexec import SSHExec
 
 #
 # cmdeploy sub commands and options
@@ -32,30 +32,17 @@ def init_cmd_options(parser):
         action="store",
         help="fully qualified DNS domain name for your chatmail instance",
     )
-    parser.add_argument(
-        "--force",
-        dest="recreate_ini",
-        action="store_true",
-        help="force reacreate ini file",
-    )
 
 
 def init_cmd(args, out):
     """Initialize chatmail config file."""
     mail_domain = args.chatmail_domain
-    inipath = args.inipath
     if args.inipath.exists():
-        if not args.recreate_ini:
-            print(f"[WARNING] Path exists, not modifying: {inipath}")
-            return 1
-        else:
-            print(
-                f"[WARNING] Force argument was provided, deleting config file: {inipath}"
-            )
-            inipath.unlink()
-
-    write_initial_config(inipath, mail_domain, overrides={})
-    out.green(f"created config file for {mail_domain} in {inipath}")
+        print(f"Path exists, not modifying: {args.inipath}")
+        return 1
+    else:
+        write_initial_config(args.inipath, mail_domain, overrides={})
+        out.green(f"created config file for {mail_domain} in {args.inipath}")
 
 
 def run_cmd_options(parser):
@@ -72,12 +59,10 @@ def run_cmd_options(parser):
         help="install/upgrade the server, but disable postfix & dovecot for now",
     )
     parser.add_argument(
-        "--skip-dns-check",
-        dest="dns_check_disabled",
-        action="store_true",
-        help="disable checks nslookup for dns",
+        "--ssh-host",
+        dest="ssh_host",
+        help="Deploy to 'localhost' or to a specific SSH host",
     )
-    add_ssh_host_option(parser)
 
 
 def run_cmd(args, out):
@@ -86,10 +71,9 @@ def run_cmd(args, out):
     ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
     sshexec = get_sshexec(ssh_host)
     require_iroh = args.config.enable_iroh_relay
-    if not args.dns_check_disabled:
-        remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
-        if not dns.check_initial_remote_data(remote_data, print=out.red):
-            return 1
+    remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
+    if not dns.check_initial_remote_data(remote_data, print=out.red):
+        return 1
 
     env = os.environ.copy()
     env["CHATMAIL_INI"] = args.inipath
@@ -99,7 +83,7 @@ def run_cmd(args, out):
     pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
 
     cmd = f"{pyinf} --ssh-user root {ssh_host} {deploy_path} -y"
-    if ssh_host in ["localhost", "@docker"]:
+    if ssh_host == "localhost":
         cmd = f"{pyinf} @local {deploy_path} -y"
 
     if version.parse(pyinfra.__version__) < version.parse("3"):
@@ -109,15 +93,14 @@ def run_cmd(args, out):
     try:
         retcode = out.check_call(cmd, env=env)
         if retcode == 0:
-            if not args.disable_mail:
-                print("\nYou can try out the relay by talking to this echo bot: ")
-                sshexec = SSHExec(args.config.mail_domain, verbose=args.verbose)
-                print(
-                    sshexec(
-                        call=remote.rshell.shell,
-                        kwargs=dict(command="cat /var/lib/echobot/invite-link.txt"),
-                    )
+            print("\nYou can try out the relay by talking to this echo bot: ")
+            sshexec = SSHExec(args.config.mail_domain, verbose=args.verbose)
+            print(
+                sshexec(
+                    call=remote.rshell.shell,
+                    kwargs=dict(command="cat /var/lib/echobot/invite-link.txt"),
                 )
+            )
             out.green("Deploy completed, call `cmdeploy dns` next.")
         elif not remote_data["acme_account_url"]:
             out.red("Deploy completed but letsencrypt not configured")
@@ -139,7 +122,11 @@ def dns_cmd_options(parser):
         default=None,
         help="write out a zonefile",
     )
-    add_ssh_host_option(parser)
+    parser.add_argument(
+        "--ssh-host",
+        dest="ssh_host",
+        help="Run the DNS queries on 'localhost' or on a specific SSH host",
+    )
 
 
 def dns_cmd(args, out):
@@ -221,6 +208,61 @@ def test_cmd(args, out):
     return ret
 
 
+def proxy_cmd_options(parser: argparse.ArgumentParser):
+    parser.add_argument(
+        "ip_address",
+        help="specify a server to deploy to; can also be an inventory.py file",
+    )
+    parser.add_argument(
+        "--relay-ipv4",
+        dest="relay_ipv4",
+        help="The ipv4 address of the relay you want to forward traffic to",
+    )
+    parser.add_argument(
+        "--relay-ipv6",
+        dest="relay_ipv6",
+        help="The ipv6 address of the relay you want to forward traffic to",
+    )
+    parser.add_argument(
+        "--dry-run",
+        dest="dry_run",
+        action="store_true",
+        help="don't actually modify the server",
+    )
+
+
+def proxy_cmd(args, out):
+    """Deploy reverse proxy on a second server."""
+    env = os.environ.copy()
+    env["RELAY_IPV4"] = args.relay_ipv4
+    env["RELAY_IPV6"] = args.relay_ipv6
+    deploy_path = importlib.resources.files(__package__).joinpath("proxy-deploy.py").resolve()
+    pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
+
+    sshexec = args.get_sshexec()
+    # :todo make sure relay is deployed to args.relay_ipv4 and args.relay_ipv6
+
+    # abort if IP address == the chatmail relay itself: if port 22 is open AND /etc/chatmail-version exists
+    if sshexec.logged(call=remote.rshell.get_port_service, args=[22]):
+        if sshexec.logged(call=remote.rshell.chatmail_version):
+            out.red("Can not deploy proxy on the chatmail relay itself, use another server")
+            return 1
+        cmd = f"{pyinf} --ssh-user root {args.ip_address} {deploy_path} -y"
+        out.check_call(cmd, env=env)  # during first try, only set SSH port to 2222
+
+    cmd = f"{pyinf} --ssh-port 2222 --ssh-user root {args.ip_address} {deploy_path} -y"
+    try:
+        retcode = out.check_call(cmd, env=env)
+        if retcode == 0:
+            out.green("Reverse proxy deployed - you can distribute the IP address now.")
+        else:
+            out.red("Deploying reverse proxy failed")
+    except subprocess.CalledProcessError:
+        out.red("Deploying reverse proxy failed")
+        retcode = 1
+    return retcode
+
+
 def fmt_cmd_options(parser):
     parser.add_argument(
         "--check",
@@ -299,15 +341,6 @@ class Out:
         return proc.returncode
 
 
-def add_ssh_host_option(parser):
-    parser.add_argument(
-        "--ssh-host",
-        dest="ssh_host",
-        help="Run commands on 'localhost', via '@docker', or on a specific SSH host "
-             "instead of chatmail.ini's mail_domain.",
-    )
-
-
 def add_config_option(parser):
     parser.add_argument(
         "--config",
@@ -365,9 +398,7 @@ def get_parser():
 
 def get_sshexec(ssh_host: str, verbose=True):
     if ssh_host in ["localhost", "@local"]:
-        return LocalExec(verbose, docker=False)
-    elif ssh_host == "@docker":
-        return LocalExec(verbose, docker=True)
+        return "localhost"
     if verbose:
         print(f"[ssh] login to {ssh_host}")
     return SSHExec(ssh_host, verbose=verbose)

cmdeploy/src/cmdeploy/deployer.py

@@ -1,59 +0,0 @@
-import os
-
-from pyinfra.operations import server
-
-
-class Deployment:
-    def install(self, deployer):
-        # optional 'required_users' contains a list of (user, group, secondary-group-list) tuples.
-        # If the group is None, no group is created corresponding to that user.
-        # If the secondary group list is not None, all listed groups are created as well.
-        required_users = getattr(deployer, "required_users", [])
-        for user, group, groups in required_users:
-            if group is not None:
-                server.group(
-                    name="Create {} group".format(group), group=group, system=True
-                )
-            if groups is not None:
-                for group2 in groups:
-                    server.group(
-                        name="Create {} group".format(group2), group=group2, system=True
-                    )
-            server.user(
-                name="Create {} user".format(user),
-                user=user,
-                group=group,
-                groups=groups,
-                system=True,
-            )
-
-        ret = bool(deployer.install())
-        if ret:
-            deployer.need_restart = True
-
-    def configure(self, deployer):
-        deployer.configure()
-
-    def activate(self, deployer):
-        deployer.activate()
-
-    def perform_stages(self, deployers):
-        default_stages = "install,configure,activate"
-        stages = os.getenv("CMDEPLOY_STAGES", default_stages).split(",")
-
-        for stage in stages:
-            for deployer in deployers:
-                getattr(self, stage)(deployer)
-
-
-class Deployer:
-    need_restart = False
-
-    def install(self):
-        pass
-
-    def configure(self):
-        pass
-
-    def activate(self):
-        pass

cmdeploy/src/cmdeploy/dns.py

@@ -7,9 +7,13 @@ from . import remote
 
 
 def get_initial_remote_data(sshexec, mail_domain):
-    return sshexec.logged(
-        call=remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=mail_domain)
-    )
+    if sshexec == "localhost":
+        result = remote.rdns.perform_initial_checks(mail_domain)
+    else:
+        result = sshexec.logged(
+            call=remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=mail_domain)
+        )
+    return result
 
 
 def check_initial_remote_data(remote_data, *, print=print):
@@ -44,9 +48,14 @@ def check_full_zone(sshexec, remote_data, out, zonefile) -> int:
     """Check existing DNS records, optionally write them to zone file
     and return (exitcode, remote_data) tuple."""
 
-    required_diff, recommended_diff = sshexec.logged(
-        remote.rdns.check_zonefile, kwargs=dict(zonefile=zonefile, verbose=False),
-    )
+    if sshexec == "localhost":
+        required_diff, recommended_diff = remote.rdns.check_zonefile(
+            zonefile=zonefile, verbose=False
+        )
+    else:
+        required_diff, recommended_diff = sshexec.logged(
+            remote.rdns.check_zonefile, kwargs=dict(zonefile=zonefile, verbose=False),
+        )
 
     returncode = 0
     if required_diff:

cmdeploy/src/cmdeploy/dovecot/dovecot.conf.j2

@@ -70,12 +70,6 @@ userdb {
 # Mailboxes are stored in the "mail" directory of the vmail user home.
 mail_location = maildir:{{ config.mailboxes_dir }}/%u
 
-# index/cache files are not very useful for chatmail relay operations 
-# but it's not clear how to disable them completely. 
-# According to https://doc.dovecot.org/2.3/settings/advanced/#core_setting-mail_cache_max_size
-# if the cache file becomes larger than the specified size, it is truncated by dovecot 
-mail_cache_max_size = 500K
-
 namespace inbox {
   inbox = yes
 

cmdeploy/src/cmdeploy/dovecot/expunge.cron.j2

@@ -0,0 +1,14 @@
+# delete already seen big mails after 7 days, in the INBOX
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_large_after }} -size +200k -type f -delete
+# delete all mails after {{ config.delete_mails_after }} days, in the Inbox
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+# or in any IMAP subfolder
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+# even if they are unseen
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+# or only temporary (but then they shouldn't be around after {{ config.delete_mails_after }} days anyway).
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+3 0 * * * vmail find {{ config.mailboxes_dir }} -name 'maildirsize' -type f -delete
+4 0 * * * vmail /usr/local/lib/chatmaild/venv/bin/delete_inactive_users /usr/local/lib/chatmaild/chatmail.ini 

cmdeploy/src/cmdeploy/iroh-relay.toml

@@ -1,11 +1,5 @@
 enable_relay = true
 http_bind_addr = "[::]:3340"
-
-# Disable built-in STUN server in iroh-relay 0.35
-# as we deploy our own TURN server instead.
-# STUN server is going to be removed in iroh-relay 1.0
-# and this line can be removed after upgrade.
-enable_stun = false
-
+enable_stun = true
 enable_metrics = false
 metrics_bind_addr = "127.0.0.1:9092"

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -66,7 +66,7 @@ http {
 
 		index index.html index.htm;
 
-		server_name {{ config.domain_name }} www.{{ config.domain_name }} mta-sts.{{ config.domain_name }};
+		server_name _;
 
 		access_log syslog:server=unix:/dev/log,facility=local7;
 

cmdeploy/src/cmdeploy/opendkim/opendkim.conf

@@ -13,7 +13,6 @@ OversignHeaders		From
 On-BadSignature         reject
 On-KeyNotFound          reject
 On-NoSignature          reject
-DNSTimeout          60
 
 # Signing domain, selector, and key (required). For example, perform signing
 # for domain "example.com" with selector "2020" (2020._domainkey.example.com),

cmdeploy/src/cmdeploy/policy-rc.d

@@ -1,3 +0,0 @@
-#!/bin/sh
-echo "All runlevel operations denied by policy" >&2
-exit 101

cmdeploy/src/cmdeploy/postfix/main.cf.j2

@@ -26,7 +26,6 @@ smtp_tls_security_level=verify
 smtp_tls_servername = hostname
 smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
 smtp_tls_policy_maps = inline:{nauta.cu=may}
-smtp_tls_protocols = >=TLSv1.2
 smtpd_tls_protocols = >=TLSv1.2
 
 # Disable anonymous cipher suites

cmdeploy/src/cmdeploy/postfix/master.cf.j2

@@ -14,7 +14,6 @@ smtp      inet  n       -       y       -       -       smtpd -v
 {%- else %}
 smtp      inet  n       -       y       -       -       smtpd 
 {%- endif %}
-  -o smtpd_tls_security_level=encrypt
   -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
 submission inet n       -       y       -       5000    smtpd
   -o syslog_name=postfix/submission

cmdeploy/src/cmdeploy/proxy-deploy.py

@@ -0,0 +1,19 @@
+import os
+
+import pyinfra
+from pyinfra import host
+
+from proxy import configure_ssh, configure_proxy
+
+
+def main():
+    ipv4_relay = os.getenv("IPV4_RELAY")
+    ipv6_relay = os.getenv("IPV6_RELAY")
+
+    configure_ssh()
+    if host.data.get("ssh_port") not in (None, 22):
+        configure_proxy(ipv4_relay, ipv6_relay)
+
+
+if pyinfra.is_cli:
+    main()

cmdeploy/src/cmdeploy/proxy.py

@@ -0,0 +1,63 @@
+import importlib
+
+from pyinfra import host
+from pyinfra.operations import files, server, apt, systemd
+
+def configure_ssh():
+    files.replace(
+        name="Configure sshd to use port 2222",
+        path="/etc/ssh/sshd_config",
+        text="Port 22\n",
+        replace="Port 2222\n",
+    )
+    systemd.service(
+        name="apply SSH config",
+        service="ssh",
+        reloaded=True,
+    )
+    apt.update()
+
+
+def configure_proxy(ipv4_relay, ipv6_relay):
+    files.put(
+        name="Configure nftables",
+        src=importlib.resources.files(__package__).joinpath("proxy_files/nftables.conf.j2"),
+        dest="/etc/nftables.conf",
+        ipv4_address=ipv4_relay,  # :todo what if only one of them is specified?
+        ipv6_address=ipv6_relay,
+    )
+
+    server.sysctl(name="enable IPv4 forwarding", key="net.ipv4.ip_forward", value=1, persist=True)
+
+    server.sysctl(
+        name="enable IPv6 forwarding",
+        key="net.ipv6.conf.all.forwarding",
+        value=1,
+        persist=True,
+    )
+
+    server.shell(
+        name="apply forwarding configuration",
+        commands=[
+            "sysctl -p",
+            "nft -f /etc/nftables.conf",
+        ],
+    )
+
+    if host.data.get("floating_ips"):
+        i = 0
+        for floating_ip in host.data.get("floating_ips"):
+            i += 1
+            files.template(
+                name="Add floating IPs",
+                src="servers/proxy-nine/files/60-floating.ip.cfg.j2",
+                dest=f"/etc/network/interfaces.d/{59 + i}-floating.ip.cfg",
+                ip_address=floating_ip,
+                i=i,
+            )
+
+        systemd.service(
+            name="apply floating IPs",
+            service="networking",
+            restarted=True,
+        )

cmdeploy/src/cmdeploy/proxy_files/60-floating.ip.cfg.j2

@@ -0,0 +1,4 @@
+auto eth0:{{ i }}
+iface eth0:{{ i }} inet static
+    address {{ ip_address }}
+    netmask 32

cmdeploy/src/cmdeploy/proxy_files/nftables.conf.j2

@@ -0,0 +1,67 @@
+#!/usr/sbin/nft -f
+
+flush ruleset
+
+define wan = eth0
+
+# which ports to proxy
+define ports = { smtp, http, https, imap, imaps, submission, submissions }
+
+# the host we want to proxy to
+define ipv4_address = {{ ipv4_address }}
+define ipv6_address = [{{ ipv6_address }}]
+
+table ip nat {
+        chain prerouting {
+                type nat hook prerouting priority dstnat; policy accept;
+                iif $wan tcp dport $ports dnat to $ipv4_address
+        }
+
+        chain postrouting {
+                type nat hook postrouting priority 0;
+
+                oifname $wan masquerade
+        }
+}
+
+table ip6 nat {
+        chain prerouting {
+                type nat hook prerouting priority dstnat; policy accept;
+                iif $wan tcp dport $ports dnat to $ipv6_address
+        }
+
+        chain postrouting {
+                type nat hook postrouting priority 0;
+
+                oifname $wan masquerade
+        }
+}
+
+table inet filter {
+        chain input {
+                type filter hook input priority filter; policy drop;
+
+                # Accept ICMP.
+                # It is especially important to accept ICMPv6 ND messages,
+                # otherwise IPv6 connectivity breaks.
+                icmp type { echo-request } accept
+                icmpv6 type { echo-request, nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
+
+                # Allow incoming SSH connections.
+                tcp dport { 22, 2222 } accept
+                # Allow incoming shadowsocks connections.
+                tcp dport { 8388 } accept
+
+                ct state established accept
+        }
+        chain forward {
+                type filter hook forward priority filter; policy drop;
+
+                ct state established accept
+                ip daddr $ipv4_address counter accept
+                ip6 daddr $ipv6_address counter accept
+        }
+        chain output {
+                type filter hook output priority filter;
+        }
+}

cmdeploy/src/cmdeploy/remote/rdns.py

@@ -73,7 +73,9 @@ def query_dns(typ, domain):
 
     # Query authoritative nameserver directly to bypass DNS cache.
     res = shell(f"dig @{ns} -r -q {domain} -t {typ} +short", print=log_progress)
-    return next((line for line in res.split("\n") if not line.startswith(';')), '')
+    if res:
+        return res.split("\n")[0]
+    return ""
 
 
 def check_zonefile(zonefile, verbose=True):

cmdeploy/src/cmdeploy/remote/rshell.py

@@ -21,6 +21,20 @@ def shell(command, fail_ok=False, print=print):
         return ""
 
 
+def get_port_service(port: int) -> str:
+    return shell(
+        "ss -lptn 'src :%d' | awk 'NR>1 {print $6,$7}' | sed 's/users:((\"//;s/\".*//'"
+        % (port,)
+    )
+
+
+def chatmail_version():
+    version = shell("cat /etc/chatmail-version")
+    if "cat: /etc/chatmail-version:" in version:
+        version = None
+    return version
+
+
 def get_systemd_running():
     lines = shell("systemctl --type=service --state=running").split("\n")
     return [line for line in lines if line.startswith("  ")]

cmdeploy/src/cmdeploy/service/chatmail-expire.service.f

@@ -1,9 +0,0 @@
-[Unit]
-Description=chatmail mail storage expiration job
-After=network.target
-
-[Service]
-Type=oneshot
-User=vmail
-ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-expire /usr/local/lib/chatmaild/chatmail.ini -v --remove
-

cmdeploy/src/cmdeploy/service/chatmail-expire.timer.f

@@ -1,8 +0,0 @@
-[Unit]
-Description=Run Daily chatmail-expire job
-
-[Timer]
-OnCalendar=*-*-* 00:02:00
-
-[Install]
-WantedBy=timers.target

cmdeploy/src/cmdeploy/service/chatmail-fsreport.service.f

@@ -1,9 +0,0 @@
-[Unit]
-Description=chatmail file system storage reporting job 
-After=network.target
-
-[Service]
-Type=oneshot
-User=vmail
-ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-fsreport /usr/local/lib/chatmaild/chatmail.ini 
-

cmdeploy/src/cmdeploy/service/chatmail-fsreport.timer.f

@@ -1,9 +0,0 @@
-[Unit]
-Description=Run Daily Chatmail fsreport Job
-
-[Timer]
-OnCalendar=*-*-* 08:02:00
-Persistent=true
-
-[Install]
-WantedBy=timers.target

cmdeploy/src/cmdeploy/service/turnserver.service.f

@@ -1,16 +0,0 @@
-[Unit]
-Description=A wrapper for the TURN server
-After=network.target
-
-[Service]
-Type=simple
-Restart=always
-ExecStart=/usr/local/bin/chatmail-turn --realm {mail_domain} --socket /run/chatmail-turn/turn.socket
-
-# Create /run/chatmail-turn
-RuntimeDirectory=chatmail-turn
-User=vmail
-Group=vmail
-
-[Install]
-WantedBy=multi-user.target

cmdeploy/src/cmdeploy/sshexec.py

@@ -82,19 +82,3 @@ class SSHExec:
             res = self(call, kwargs, log_callback=remote.rshell.log_progress)
             print_stderr()
             return res
-
-
-class LocalExec:
-    def __init__(self, verbose=False, docker=False):
-        self.verbose = verbose
-        self.docker = docker
-
-    def logged(self, call, kwargs: dict):
-        where = "locally"
-        if self.docker:
-            if call == remote.rdns.perform_initial_checks:
-                kwargs['pre_command'] = "docker exec chatmail "
-                where = "in docker"
-        if self.verbose:
-            print(f"Running {where}: {call.__name__}(**{kwargs})")
-        return call(**kwargs)

cmdeploy/src/cmdeploy/tests/online/benchmark.py

@@ -37,7 +37,7 @@ class TestDC:
 
     def test_ping_pong(self, benchmark, cmfactory):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_accepted_chat(ac1, ac2)
+        chat = cmfactory.get_protected_chat(ac1, ac2)
 
         def dc_ping_pong():
             chat.send_text("ping")
@@ -49,7 +49,7 @@ class TestDC:
 
     def test_send_10_receive_10(self, benchmark, cmfactory, lp):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_accepted_chat(ac1, ac2)
+        chat = cmfactory.get_protected_chat(ac1, ac2)
 
         def dc_send_10_receive_10():
             for i in range(10):

cmdeploy/src/cmdeploy/tests/online/test_0_login.py

@@ -1,5 +1,5 @@
 import queue
-import smtplib
+import socket
 import threading
 
 import pytest
@@ -91,23 +91,25 @@ def test_concurrent_logins_same_account(
 
 def test_no_vrfy(chatmail_config):
     domain = chatmail_config.mail_domain
-
-    s = smtplib.SMTP(domain)
-    s.starttls()
-
-    s.putcmd("vrfy", f"wrongaddress@{chatmail_config.mail_domain}")
-    result = s.getreply()
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    sock.settimeout(10)
+    try:
+        sock.connect((domain, 25))
+    except socket.timeout:
+        pytest.skip(f"port 25 not reachable for {domain}")
+    banner = sock.recv(1024)
+    print(banner)
+    sock.send(b"VRFY wrongaddress@%s\r\n" % (chatmail_config.mail_domain.encode(),))
+    result = sock.recv(1024)
     print(result)
-    s.putcmd("vrfy", f"echo@{chatmail_config.mail_domain}")
-    result2 = s.getreply()
+    sock.send(b"VRFY echo@%s\r\n" % (chatmail_config.mail_domain.encode(),))
+    result2 = sock.recv(1024)
     print(result2)
-    assert result[0] == result2[0] == 252
-    assert result[1][0:6] == result2[1][0:6] == b"2.0.0 "
-    s.putcmd("vrfy", "wrongaddress")
-    result = s.getreply()
+    assert result[0:10] == result2[0:10]
+    sock.send(b"VRFY wrongaddress\r\n")
+    result = sock.recv(1024)
     print(result)
-    s.putcmd("vrfy", "echo")
-    result2 = s.getreply()
+    sock.send(b"VRFY echo\r\n")
+    result2 = sock.recv(1024)
     print(result2)
-    assert result[0] == result2[0] == 252
-    assert result[1][0:6] == result2[1][0:6] == b"2.0.0 "
+    assert result[0:10] == result2[0:10] == b"252 2.0.0 "

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -2,7 +2,6 @@ import datetime
 import smtplib
 import socket
 import subprocess
-import time
 
 import pytest
 
@@ -32,7 +31,6 @@ class TestSSHExecutor:
         )
         out, err = capsys.readouterr()
         assert err.startswith("Collecting")
-        # XXX could not figure out how capturing can be made to work properly
         #assert err.endswith("....\n")
         assert err.count("\n") == 1
 
@@ -42,7 +40,6 @@ class TestSSHExecutor:
         )
         out, err = capsys.readouterr()
         lines = err.split("\n")
-        # XXX could not figure out how capturing can be made to work properly
         #assert len(lines) > 4
         assert remote.rdns.perform_initial_checks.__doc__ in lines[0]
 
@@ -72,7 +69,7 @@ def test_timezone_env(remote):
     for line in remote.iter_output("env"):
         print(line)
         if line == "tz=:/etc/localtime":
-            return
+            return True
     pytest.fail("TZ is not set")
 
 
@@ -143,23 +140,12 @@ def test_reject_missing_dkim(cmsetup, maildata, from_addr):
         "encrypted.eml", from_addr=from_addr, to_addr=recipient.addr
     ).as_string()
     conn = smtplib.SMTP(cmsetup.maildomain, 25, timeout=10)
-    conn.starttls()
 
     with conn as s:
         with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):
             s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)
 
 
-def try_n_times(n, f):
-    for _ in range(n - 1):
-        try:
-            return f()
-        except Exception:
-            time.sleep(1)
-
-    return f()
-
-
 def test_rewrite_subject(cmsetup, maildata):
     """Test that subject gets replaced with [...]."""
     user1, user2 = cmsetup.gen_users(2)
@@ -172,8 +158,7 @@ def test_rewrite_subject(cmsetup, maildata):
     ).as_string()
     user1.smtp.sendmail(from_addr=user1.addr, to_addrs=[user2.addr], msg=sent_msg)
 
-    # The message may need some time to get delivered by postfix.
-    messages = try_n_times(5, user2.imap.fetch_all_messages)
+    messages = user2.imap.fetch_all_messages()
     assert len(messages) == 1
     rcvd_msg = messages[0]
     assert "Subject: [...]" not in sent_msg
@@ -224,14 +209,8 @@ def test_expunged(remote, chatmail_config):
 
 
 def test_deployed_state(remote):
-    try:
-        git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
-    except Exception:
-        git_hash = "unknown\n"
-    try:
-        git_diff = subprocess.check_output(["git", "diff"]).decode()
-    except Exception:
-        git_diff = ""
+    git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
+    git_diff = subprocess.check_output(["git", "diff"]).decode()
     git_status = [git_hash.strip()]
     for line in git_diff.splitlines():
         git_status.append(line.strip().lower())

cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py

@@ -56,7 +56,7 @@ class TestEndToEndDeltaChat:
         """Test that a DC account can send a message to a second DC account
         on the same chat-mail instance."""
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_accepted_chat(ac1, ac2)
+        chat = cmfactory.get_protected_chat(ac1, ac2)
         chat.send_text("message0")
 
         lp.sec("wait for ac2 to receive message")
@@ -70,7 +70,7 @@ class TestEndToEndDeltaChat:
         before quota is exceeded, and thus depends on the speed of the upload.
         """
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_accepted_chat(ac1, ac2)
+        chat = cmfactory.get_protected_chat(ac1, ac2)
 
         user = ac2.get_config("configured_addr")
 
@@ -153,7 +153,7 @@ def test_hide_senders_ip_address(cmfactory):
     assert ipaddress.ip_address(public_ip)
 
     user1, user2 = cmfactory.get_online_accounts(2)
-    chat = cmfactory.get_accepted_chat(user1, user2)
+    chat = cmfactory.get_protected_chat(user1, user2)
 
     chat.send_text("testing submission header cleanup")
     user2._evtracker.wait_next_incoming_message()

cmdeploy/src/cmdeploy/tests/test_cmdeploy.py

@@ -26,15 +26,10 @@ class TestCmdline:
     def test_init_not_overwrite(self, capsys):
         assert main(["init", "chat.example.org"]) == 0
         capsys.readouterr()
-
         assert main(["init", "chat.example.org"]) == 1
         out, err = capsys.readouterr()
         assert "path exists" in out.lower()
 
-        assert main(["init", "chat.example.org", "--force"]) == 0
-        out, err = capsys.readouterr()
-        assert "deleting config file" in out.lower()
-
 
 def test_www_folder(example_config, tmp_path):
     reporoot = importlib.resources.files(__package__).joinpath("../../../../").resolve()

cmdeploy/src/cmdeploy/tests/test_dns.py

@@ -1,5 +1,3 @@
-from copy import deepcopy
-
 import pytest
 
 from cmdeploy import remote
@@ -10,63 +8,38 @@ from cmdeploy.dns import check_full_zone, check_initial_remote_data
 def mockdns_base(monkeypatch):
     qdict = {}
 
-    def shell(command, fail_ok=False, print=print):
-        if command.startswith("dig"):
-            if command == "dig":
-                return "."
-            if "SOA" in command:
-                return (
-                    "delta.chat. 21600 IN SOA ns1.first-ns.de. dns.hetzner.com."
-                    " 2025102800 14400 1800 604800 3600"
-                )
-            command_chunks = command.split()
-            domain, typ = command_chunks[4], command_chunks[6]
-            try:
-                return qdict[typ][domain]
-            except KeyError:
-                return ""
-        return remote.rshell.shell(command=command, fail_ok=fail_ok, print=print)
+    def query_dns(typ, domain):
+        try:
+            return qdict[typ][domain]
+        except KeyError:
+            return ""
 
-    monkeypatch.setattr(remote.rdns, shell.__name__, shell)
+    monkeypatch.setattr(remote.rdns, query_dns.__name__, query_dns)
     return qdict
 
 
 @pytest.fixture
-def mockdns_expected():
-    return {
-        "A": {"some.domain": "1.1.1.1"},
-        "AAAA": {"some.domain": "fde5:cd7a:9e1c:3240:5a99:936f:cdac:53ae"},
-        "CNAME": {
-            "mta-sts.some.domain": "some.domain.",
-            "www.some.domain": "some.domain.",
-        },
-    }
-
-
-@pytest.fixture(params=["plain", "with-dns-comments"])
-def mockdns(request, mockdns_base, mockdns_expected):
-    mockdns_base.update(deepcopy(mockdns_expected))
-    match request.param:
-        case "plain":
-            pass
-        case "with-dns-comments":
-            for typ, data in mockdns_base.items():
-                for host, result in data.items():
-                    mockdns_base[typ][host] = (
-                        ";; some unsuccessful attempt result\n"
-                        "; and another with a single semicolon\n"
-                        f"{result}"
-                    )
+def mockdns(mockdns_base):
+    mockdns_base.update(
+        {
+            "A": {"some.domain": "1.1.1.1"},
+            "AAAA": {"some.domain": "fde5:cd7a:9e1c:3240:5a99:936f:cdac:53ae"},
+            "CNAME": {
+                "mta-sts.some.domain": "some.domain.",
+                "www.some.domain": "some.domain.",
+            },
+        }
+    )
     return mockdns_base
 
 
 class TestPerformInitialChecks:
-    def test_perform_initial_checks_ok1(self, mockdns, mockdns_expected):
+    def test_perform_initial_checks_ok1(self, mockdns):
         remote_data = remote.rdns.perform_initial_checks("some.domain")
-        assert remote_data["A"] == mockdns_expected["A"]["some.domain"]
-        assert remote_data["AAAA"] == mockdns_expected["AAAA"]["some.domain"]
-        assert remote_data["MTA_STS"] == mockdns_expected["CNAME"]["mta-sts.some.domain"]
-        assert remote_data["WWW"] == mockdns_expected["CNAME"]["www.some.domain"]
+        assert remote_data["A"] == mockdns["A"]["some.domain"]
+        assert remote_data["AAAA"] == mockdns["AAAA"]["some.domain"]
+        assert remote_data["MTA_STS"] == mockdns["CNAME"]["mta-sts.some.domain"]
+        assert remote_data["WWW"] == mockdns["CNAME"]["www.some.domain"]
 
     @pytest.mark.parametrize("drop", ["A", "AAAA"])
     def test_perform_initial_checks_with_one_of_A_AAAA(self, mockdns, drop):

cmdeploy/src/cmdeploy/www.py

@@ -4,7 +4,6 @@ import time
 import traceback
 import webbrowser
 from pathlib import Path
-import re
 
 import markdown
 from chatmaild.config import read_config
@@ -13,9 +12,6 @@ from jinja2 import Template
 from .genqr import gen_qr_png_data
 
 
-_MERGE_CONFLICT_RE = re.compile(r"^<<<<<<<.+^=======.+^>>>>>>>", re.DOTALL | re.MULTILINE)
-
-
 def snapshot_dir_stats(somedir):
     d = {}
     for path in somedir.iterdir():
@@ -120,17 +116,6 @@ def _build_webpages(src_dir, build_dir, config):
     return build_dir
 
 
-def find_merge_conflict(src_dir) -> Path:
-    assert src_dir.exists(), src_dir
-    result = None
-    for path in src_dir.iterdir():
-        if path.suffix in [".css", ".html", ".md"]:
-            if _MERGE_CONFLICT_RE.search(path.read_text()):
-                result = path
-                break
-    return result
-
-
 def main():
     path = importlib.resources.files(__package__)
     reporoot = path.joinpath("../../../").resolve()