strike this weird CHATMAIL_DOMAIN variable

This expands the character set used for passwords generated for new
accounts.  The set it taken from the set used by the pass tool.  The
special characters is the full GNU grep [:punct:] set.

.github/CODE_OF_CONDUCT.md

@@ -0,0 +1,4 @@
+
+Please refer to 
+[Delta Chat community standards and practices](https://delta.chat/en/community-standards)
+which also apply for all chatmail developments. 

.github/workflows/ci.yaml

@@ -0,0 +1,40 @@
+name: CI
+
+on:
+  pull_request:
+  push:
+
+jobs:
+  tox:
+    name: isolated chatmaild tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: run chatmaild tests 
+        working-directory: chatmaild
+        run: pipx run tox
+
+  scripts:
+    name: deploy-chatmail tests 
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: initenv 
+        run: scripts/initenv.sh
+
+      - name: append venv/bin to PATH
+        run: echo venv/bin >>$GITHUB_PATH
+
+      - name: run formatting checks 
+        run: cmdeploy fmt -v 
+
+      - name: run deploy-chatmail offline tests 
+        run: pytest --pyargs cmdeploy 
+
+      - name: initialize with chatmail domain
+        run: cmdeploy init chat.example.org  
+
+      # all other cmdeploy commands require a staging server 
+      # see https://github.com/deltachat/chatmail/issues/100

.gitignore

@@ -3,6 +3,9 @@ __pycache__/
 *.py[cod]
 *$py.class
 *.swp
+*qr-*.png
+chatmail.ini
+
 
 # C extensions
 *.so
@@ -10,7 +13,6 @@ __pycache__/
 # Distribution / packaging
 .Python
 build/
-doveauth/dist/
 develop-eggs/
 dist/
 downloads/
@@ -160,3 +162,5 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+chatmail.zone

LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2023, chatmail and delta chat teams
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,65 +1,158 @@
-# Chat Mail server configuration
 
-This package deploys Postfix and Dovecot servers, including OpenDKIM for DKIM signing.
+<img width="800px" src="www/src/collage-top.png"/>
 
-Postfix uses Dovecot for authentication as described in <https://www.postfix.org/SASL_README.html#server_dovecot>
+# Chatmail services optimized for Delta Chat apps 
 
-## Getting started 
+This repository helps to setup a ready-to-use chatmail server
+comprised of a minimal setup of the battle-tested 
+[postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services. 
 
-prepare:
+The setup is designed and optimized for providing chatmail accounts 
+for use by [Delta Chat apps](https://delta.chat).
 
-    pip install -e chatmail-infra
+Chatmail accounts are automatically created by a first login, 
+after which the initially specified password is required for using them. 
 
+## Deploying your own chatmail server 
 
-then run with pyinfra command line tool: 
+We use `chat.example.org` as the chatmail domain in the following steps. 
+Please substitute it with your own domain. 
 
-    CHATMAIL_DOMAIN=c1.testrun.org pyinfra --ssh-user root c1.testrun.org deploy.py
+1. Install the `cmdeploy` command in a virtualenv
 
+   ```
+    git clone https://github.com/deltachat/chatmail
+    cd chatmail
+    scripts/initenv.sh
+   ```
 
-## Structure (wip)
-```
+2. Create chatmail configuration file `chatmail.ini`:
 
-# package doveauth tool and deploy chatmail server to a envvar-specified ssh-reachable host 
-deploy.py 
+   ```
+    scripts/cmdeploy init chat.example.org  # <-- use your domain 
+   ```
 
-# chatmail pyinfra deploy package 
-chatmail-pyinfra 
-    pyproject.toml
-    chatmail/__init__ ...
+3. Setup first DNS records for your chatmail domain, 
+   according to the hints provided by `cmdeploy init`.
+   Verify that SSH root login works:
 
-    # tests against the deployed system 
-    tests/test_online_test.py
+   ```
+    ssh root@chat.example.org   # <-- use your domain 
+   ```
 
-# doveauth tool used by dovecot's auth mechanism on the host system 
-doveauth
-    README.md
-    pyproject.toml
-    doveauth.py
-    doveauth.lua
-    test_doveauth.py
+4. Deploy to the remote chatmail server:
 
-# lmtp server to block (outgoing) unencrypted messages 
-filtermail 
-    README.md
-    pyproject.toml
-    .... 
+   ```
+    scripts/cmdeploy run
+   ```
 
+5. To output a DNS zone file from which you can transfer DNS records 
+   to your DNS provider:
 
-# scripts for setup/development/deployment 
+   ```
+    scripts/cmdeploy dns
+   ```
 
-scripts/
-    init.sh  # create venv/other perequires
-    deploy.sh  # run pyinfra based deploy of everything
+### Other helpful commands:
+
+To check the status of your remotely running chatmail service:
 
+```
+scripts/cmdeploy status
 ```
 
-## Dovecot/Postfix configuration
+To test whether your chatmail service is working correctly:
+
+```
+scripts/cmdeploy test
+```
+
+To measure the performance of your chatmail service:
+
+```
+scripts/cmdeploy bench
+```
+
+## Overview of this repository
+
+This repository drives the development of chatmail services, 
+comprised of minimal setups of
+
+- [postfix smtp server](https://www.postfix.org)
+- [dovecot imap server](https://www.dovecot.org)
+
+as well as custom services that are integrated with these two:
+
+- `chatmaild/src/chatmaild/doveauth.py` implements
+  create-on-login account creation semantics and is used
+  by Dovecot during login authentication and by Postfix
+  which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
+  to authenticate users
+  to send mails for them.
+
+- `chatmaild/src/chatmaild/filtermail.py` prevents
+  unencrypted e-mail from leaving the chatmail service
+  and is integrated into postfix's outbound mail pipelines.
+
+There is also the `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
+which helps with setting up and managing the chatmail service.
+`cmdeploy run` uses [pyinfra-based scripting](https://pyinfra.com/)
+in `cmdeploy/src/cmdeploy/__init__.py`
+to automatically install all chatmail components on a server.
+
+
+### Home page and getting started for users
+
+`cmdeploy run` also creates default static Web pages and deploys them
+to a nginx web server with: 
+
+- a default `index.html` along with a QR code that users can click to
+  create accounts on your chatmail provider,
+
+- a default `info.html` that is linked from the home page,
+
+- a default `policy.html` that is linked from the home page.
+
+All `.html` files are generated
+by the according markdown `.md` file in the `www/src` directory.
+
+
+### Refining the web pages
+
+
+```
+scripts/cmdeploy webdev
+```
+
+This starts a local live development cycle for chatmail Web pages:
+
+- uses the `www/src/page-layout.html` file for producing static
+  HTML pages from `www/src/*.md` files
+
+- continously builds the web presence reading files from `www/src` directory
+  and generating html files and copying assets to the `www/build` directory.
+
+- Starts a browser window automatically where you can "refresh" as needed.
+
+
+## Emergency Commands to disable automatic account creation
+
+If you need to stop account creation,
+e.g. because some script is wildly creating accounts,
+login to the server with ssh and run:
+
+```
+    touch /etc/chatmail-nocreate
+```
+
+While this file is present, account creation will be blocked.
 
 ### Ports
 
 Postfix listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
 Dovecot listens on ports 143(imap) and 993 (imaps).
 
-## DNS
+Delta Chat apps will, however, discover all ports and configurations
+automatically by reading the `autoconfig.xml` file from the chatmail service.
+
 
-For DKIM you must add a DNS entry as in /etc/opendkim/selector.txt (where selector is the opendkim_selector configured in the chatmail inventory).

chatmail-pyinfra/pyproject.toml

@@ -1,30 +0,0 @@
-[build-system]
-requires = ["setuptools>=45"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "chatmail"
-version = "0.1"
-dependencies = [
-  "pyinfra",
-]
-
-[tool.pytest.ini_options]
-addopts = "-v -ra --strict-markers"
-
-[tool.tox]
-legacy_tox_ini = """
-[tox]
-isolated_build = true
-envlist = lint
-
-[testenv:lint]
-skipdist = True
-skip_install = True
-deps =
-  ruff
-  black
-commands =
-  black --quiet --check --diff src/
-  ruff src/
-"""

chatmail-pyinfra/src/chatmail/__init__.py

@@ -1,213 +0,0 @@
-"""
-Chat Mail pyinfra deploy.
-"""
-import importlib.resources
-from pathlib import Path
-
-from pyinfra import host, logger
-from pyinfra.operations import apt, files, server, systemd, python
-from pyinfra.facts.files import File
-from .acmetool import deploy_acmetool
-
-
-def _install_doveauth() -> None:
-    """Setup chatctl."""
-    doveauth_filename = "doveauth-0.1.tar.gz"
-    doveauth_path = importlib.resources.files(__package__).joinpath(
-        f"../../../doveauth/dist/{doveauth_filename}"
-    )
-    remote_path = f"/tmp/{doveauth_filename}"
-    if Path(str(doveauth_path)).exists():
-        files.put(
-            name="upload local doveauth build",
-            src=doveauth_path.open("rb"),
-            dest=remote_path,
-        )
-        apt.packages(
-            name="apt install python3-pip",
-            packages="python3-pip",
-        )
-        # Maybe if we introduce dependencies to the doveauth package at some point, we should not install doveauth
-        # system-wide anymore. For now it's fine though.
-        server.shell(
-            name="install local doveauth build with pip",
-            commands=[f"pip install --break-system-packages {remote_path}"],
-        )
-
-
-def _configure_opendkim(domain: str, dkim_selector: str) -> bool:
-    """Configures OpenDKIM"""
-    need_restart = False
-
-    main_config = files.template(
-        src=importlib.resources.files(__package__).joinpath("opendkim/opendkim.conf"),
-        dest="/etc/opendkim.conf",
-        user="root",
-        group="root",
-        mode="644",
-        config={"domain_name": domain, "opendkim_selector": dkim_selector},
-    )
-
-    files.directory(
-        name="Add opendkim socket directory to /var/spool/postfix",
-        path="/var/spool/postfix/opendkim",
-        user="opendkim",
-        group="opendkim",
-        mode="750",
-        present=True,
-    )
-
-    if not host.get_fact(File, f"/etc/dkimkeys/{dkim_selector}.private"):
-        server.shell(
-            name="Generate OpenDKIM domain keys",
-            commands=[
-                f"opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
-            ],
-            _sudo=True,
-            _sudo_user="opendkim",
-        )
-
-    need_restart |= main_config.changed
-
-    return need_restart
-
-
-def _configure_postfix(domain: str) -> bool:
-    """Configures Postfix SMTP server."""
-    need_restart = False
-
-    main_config = files.template(
-        src=importlib.resources.files(__package__).joinpath("postfix/main.cf.j2"),
-        dest="/etc/postfix/main.cf",
-        user="root",
-        group="root",
-        mode="644",
-        config={"domain_name": domain},
-    )
-    need_restart |= main_config.changed
-
-    master_config = files.put(
-        src=importlib.resources.files(__package__)
-        .joinpath("postfix/master.cf")
-        .open("rb"),
-        dest="/etc/postfix/master.cf",
-        user="root",
-        group="root",
-        mode="644",
-    )
-    need_restart |= master_config.changed
-
-    return need_restart
-
-
-def _configure_dovecot(mail_server: str) -> bool:
-    """Configures Dovecot IMAP server."""
-    need_restart = False
-
-    main_config = files.template(
-        src=importlib.resources.files(__package__).joinpath("dovecot/dovecot.conf.j2"),
-        dest="/etc/dovecot/dovecot.conf",
-        user="root",
-        group="root",
-        mode="644",
-        config={"hostname": mail_server},
-    )
-    need_restart |= main_config.changed
-
-    # luarocks install http lpeg_patterns fifo
-    auth_script = files.put(
-        src=importlib.resources.files("doveauth").joinpath("doveauth.lua"),
-        dest="/etc/dovecot/doveauth.lua",
-        user="root",
-        group="root",
-        mode="644",
-    )
-    need_restart |= auth_script.changed
-
-    return need_restart
-
-
-def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> None:
-    """Deploy a chat-mail instance.
-
-    :param mail_domain: domain part of your future email addresses
-    :param mail_server: the DNS name under which your mail server is reachable
-    :param dkim_selector:
-    """
-
-    apt.update(name="apt update")
-    server.group(name="Create vmail group", group="vmail", system=True)
-    server.user(name="Create vmail user", user="vmail", group="vmail", system=True)
-
-    server.group(name="Create opendkim group", group="opendkim", system=True)
-    server.user(
-        name="Add postfix user to opendkim group for socket access",
-        user="postfix",
-        groups=["opendkim"],
-        system=True,
-    )
-
-    # Deploy acmetool to have TLS certificates.
-    deploy_acmetool(domains=[mail_server])
-
-    apt.packages(
-        name="Install Postfix",
-        packages="postfix",
-    )
-
-    apt.packages(
-        name="Install Dovecot",
-        packages=[
-            "dovecot-imapd",
-            "dovecot-lmtpd",
-            "dovecot-auth-lua",
-        ],
-    )
-
-    apt.packages(
-        name="Install OpenDKIM",
-        packages=[
-            "opendkim",
-            "opendkim-tools",
-        ],
-    )
-
-    _install_doveauth()
-    dovecot_need_restart = _configure_dovecot(mail_server)
-    postfix_need_restart = _configure_postfix(mail_domain)
-    opendkim_need_restart = _configure_opendkim(mail_domain, dkim_selector)
-
-    systemd.service(
-        name="Start and enable OpenDKIM",
-        service="opendkim.service",
-        running=True,
-        enabled=True,
-        restarted=opendkim_need_restart,
-    )
-
-    systemd.service(
-        name="Start and enable Postfix",
-        service="postfix.service",
-        running=True,
-        enabled=True,
-        restarted=postfix_need_restart,
-    )
-
-    systemd.service(
-        name="Start and enable Dovecot",
-        service="dovecot.service",
-        running=True,
-        enabled=True,
-        restarted=dovecot_need_restart,
-    )
-
-    def callback():
-        result = server.shell(
-            commands=[
-                f"""sed 's/\tIN/ 600 IN/;s/\t(//;s/\"$//;s/^\t  \"//g; s/ ).*//' """
-                f"""/etc/dkimkeys/{dkim_selector}.txt | tr --delete '\n'"""
-            ]
-        )
-        logger.info(f"Add this TXT entry into DNS zone: {result.stdout}")
-
-    python.call(name="Print TXT entry for DKIM", function=callback)

chatmail-pyinfra/src/chatmail/acmetool/acmetool-redirector.service

@@ -1,11 +0,0 @@
-[Unit]
-Description=acmetool HTTP redirector
-
-[Service]
-Type=notify
-ExecStart=/usr/bin/acmetool redirector --service.uid=daemon
-Restart=always
-RestartSec=30
-
-[Install]
-WantedBy=multi-user.target

chatmail-pyinfra/src/chatmail/dovecot/dovecot.conf.j2

@@ -1,99 +0,0 @@
-## Dovecot configuration file
-
-protocols = imap lmtp
-
-auth_mechanisms = plain
-
-auth_verbose = yes
-auth_debug = yes
-auth_debug_passwords = yes
-auth_verbose_passwords = plain
-
-# Authentication for system users.
-passdb {
-  driver = lua
-  args = file=/etc/dovecot/doveauth.lua
-}
-userdb {
-  driver = lua
-  args = file=/etc/dovecot/doveauth.lua
-}
-
-##
-## Mailbox locations and namespaces
-##
-
-# Mailboxes are stored in the "mail" directory of the vmail user home.
-mail_location = maildir:/home/vmail/mail/%d/%u
-
-namespace inbox {
-  inbox = yes
-
-  mailbox Drafts {
-    special_use = \Drafts
-  }
-  mailbox Junk {
-    special_use = \Junk
-  }
-  mailbox Trash {
-    special_use = \Trash
-  }
-
-  # For \Sent mailboxes there are two widely used names. We'll mark both of
-  # them as \Sent. User typically deletes one of them if duplicates are created.
-  mailbox Sent {
-    special_use = \Sent
-  }
-  mailbox "Sent Messages" {
-    special_use = \Sent
-  }
-}
-
-mail_uid = vmail
-mail_gid = vmail
-mail_privileged_group = vmail
-
-##
-## Mail processes
-##
-
-# Enable IMAP COMPRESS (RFC 4978).
-# <https://datatracker.ietf.org/doc/html/rfc4978.html>
-protocol imap {
-  mail_plugins = $mail_plugins imap_zlib
-}
-
-plugin {
-  imap_compress_deflate_level = 6
-}
-
-service lmtp {
-  user=vmail
-
-  unix_listener /var/spool/postfix/private/dovecot-lmtp {
-    group = postfix
-    mode = 0600
-    user = postfix
-  }
-}
-
-service auth {
-  unix_listener /var/spool/postfix/private/auth {
-    mode = 0660
-    user = postfix
-    group = postfix
-  }
-}
-
-service auth-worker {
-  # Default is root.
-  # Drop privileges we don't need.
-  user = vmail
-}
-
-ssl = required
-ssl_cert = </var/lib/acme/live/{{ config.hostname }}/fullchain
-ssl_key = </var/lib/acme/live/{{ config.hostname }}/privkey
-ssl_dh = </usr/share/dovecot/dh.pem
-ssl_min_protocol = TLSv1.2
-ssl_prefer_server_ciphers = yes

chatmail-pyinfra/tests/test_online_login.py

@@ -1,28 +0,0 @@
-import pytest
-import imaplib
-
-
-@pytest.fixture
-def conn():
-    return connect("c1.testrun.org")
-
-
-def login(conn, user, password):
-    print("trying to login", user, password)
-    conn.login(user, password)
-
-
-def connect(host):
-    print(f"connecting to {host}")
-    conn = imaplib.IMAP4_SSL(host)
-    return conn
-
-
-def test_login_ok(conn):
-    login(conn, "link2xt@c1.testrun.org", "Ahyei6ie")
-
-
-def test_login_fail(conn):
-    with pytest.raises(imaplib.IMAP4.error) as excinfo:
-        login(conn, "link2xt@c1.testrun.org", "qweqwe")
-    assert "AUTHENTICATIONFAILED" in str(excinfo)

chatmaild/MANIFEST.in

@@ -0,0 +1,4 @@
+include src/chatmaild/*.f 
+include src/chatmaild/ini/*.ini.f
+include src/chatmaild/ini/*.ini
+include src/chatmaild/tests/mail-data/*

chatmaild/README.md

@@ -0,0 +1,5 @@
+# chatmaild
+
+chatmaild provides dovecot autentication
+to create dovecot users on login
+and mail filtering.

chatmaild/pyproject.toml

@@ -0,0 +1,55 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "chatmaild"
+version = "0.2"
+dependencies = [
+  "aiosmtpd",
+  "iniconfig",
+  "deltachat-rpc-server",
+  "deltachat-rpc-client",
+]
+
+[tool.setuptools]
+include-package-data = true
+
+[tool.setuptools.packages.find]
+where = ['src']
+
+[project.scripts]
+doveauth = "chatmaild.doveauth:main"
+filtermail = "chatmaild.filtermail:main"
+echobot = "chatmaild.echo:main"
+
+[project.entry-points.pytest11]
+"chatmaild.testplugin" = "chatmaild.tests.plugin"
+
+[tool.pytest.ini_options]
+addopts = "-v -ra --strict-markers"
+log_format = "%(asctime)s %(levelname)s %(message)s"
+log_date_format = "%Y-%m-%d %H:%M:%S"
+log_level = "INFO"
+
+[tool.tox]
+legacy_tox_ini = """
+[tox]
+isolated_build = true
+envlist = lint,py
+
+[testenv:lint]
+skipdist = True
+skip_install = True
+deps =
+  ruff
+  black
+commands =
+  black --quiet --check --diff src/
+  ruff src/
+
+[testenv]
+deps = pytest 
+       pdbpp 
+commands = pytest -v -rsXx {posargs}
+"""

chatmaild/src/chatmaild/config.py

@@ -0,0 +1,59 @@
+import iniconfig
+
+
+def read_config(inipath):
+    cfg = iniconfig.IniConfig(inipath)
+    return Config(inipath, params=cfg.sections["params"])
+
+
+class Config:
+    def __init__(self, inipath, params):
+        self._inipath = inipath
+        self.mail_domain = params["mail_domain"]
+        self.max_user_send_per_minute = int(params["max_user_send_per_minute"])
+        self.max_mailbox_size = params["max_mailbox_size"]
+        self.delete_mails_after = params["delete_mails_after"]
+        self.username_min_length = int(params["username_min_length"])
+        self.username_max_length = int(params["username_max_length"])
+        self.password_min_length = int(params["password_min_length"])
+        self.passthrough_senders = params["passthrough_senders"].split()
+        self.passthrough_recipients = params["passthrough_recipients"].split()
+        self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
+        self.postfix_reinject_port = int(params["postfix_reinject_port"])
+        self.privacy_postal = params.get("privacy_postal")
+        self.privacy_mail = params.get("privacy_mail")
+        self.privacy_pdo = params.get("privacy_pdo")
+        self.privacy_supervisor = params.get("privacy_supervisor")
+
+    def _getbytefile(self):
+        return open(self._inipath, "rb")
+
+
+def write_initial_config(inipath, mail_domain):
+    from importlib.resources import files
+
+    inidir = files(__package__).joinpath("ini")
+    content = (
+        inidir.joinpath("chatmail.ini.f").read_text().format(mail_domain=mail_domain)
+    )
+    if mail_domain.endswith(".testrun.org"):
+        override_inipath = inidir.joinpath("override-testrun.ini")
+        privacy = iniconfig.IniConfig(override_inipath)["privacy"]
+        lines = []
+        for line in content.split("\n"):
+            for key, value in privacy.items():
+                value_lines = value.strip().split("\n")
+                if not line.startswith(f"{key} =") or not value_lines:
+                    continue
+                if len(value_lines) == 1:
+                    lines.append(f"{key} = {value}")
+                else:
+                    lines.append(f"{key} =")
+                    for vl in value_lines:
+                        lines.append(f"    {vl}")
+                break
+            else:
+                lines.append(line)
+        content = "\n".join(lines)
+
+    inipath.write_text(content)

chatmaild/src/chatmaild/database.py

@@ -33,13 +33,6 @@ class Connection:
     def cursor(self):
         return self._sqlconn.cursor()
 
-    def create_user(self, addr: str, password: str):
-        """Create a row in the users table."""
-        self.execute("PRAGMA foreign_keys=on;")
-        q = """INSERT INTO users (addr, password, last_login)
-               VALUES (?, ?, ?)"""
-        self.execute(q, (addr, password, int(time.time())))
-
     def get_user(self, addr: str) -> {}:
         """Get a row from the users table."""
         q = "SELECT addr, password, last_login from users WHERE addr = ?"
@@ -53,30 +46,15 @@ class Connection:
             )
         return result
 
-    def set_config(self, name: str, value: str) -> str:
-        ok = [
-            "dbversion",
-        ]
-        assert name in ok, name
-        q = "INSERT OR REPLACE INTO config (key, value) VALUES (?, ?)"
-        self.cursor().execute(q, (name, value)).fetchone()
-        return value
-
-    def get_config(self, key: str) -> str:
-        q = "SELECT key, value from config WHERE name = ?"
-        c = self._sqlconn.cursor()
-        try:
-            return c.execute(q, key).fetchone()
-        except sqlite3.OperationalError:
-            return None
-
 
 class Database:
     def __init__(self, path: str):
         self.path = Path(path)
         self.ensure_tables()
 
-    def _get_connection(self, write=False, transaction=False, closing=False) -> Connection:
+    def _get_connection(
+        self, write=False, transaction=False, closing=False
+    ) -> Connection:
         # we let the database serialize all writers at connection time
         # to play it very safe (we don't have massive amounts of writes).
         mode = "ro"
@@ -129,10 +107,20 @@ class Database:
     def read_connection(self, closing=True) -> Connection:
         return self._get_connection(closing=closing, write=False)
 
+    def get_schema_version(self) -> int:
+        with self.read_connection() as conn:
+            dbversion = conn.execute("PRAGMA user_version").fetchone()[0]
+        return dbversion
+
     CURRENT_DBVERSION = 1
 
     def ensure_tables(self):
         with self.write_transaction() as conn:
+            if self.get_schema_version() > 1:
+                raise DBError(
+                    "version is %s; downgrading schema is not supported"
+                    % (self.get_schema_version(),)
+                )
             conn.execute(
                 """
                 CREATE TABLE IF NOT EXISTS users (
@@ -142,12 +130,4 @@ class Database:
                 )
             """,
             )
-            conn.execute(
-                """
-                CREATE TABLE IF NOT EXISTS config (
-                    key TEXT PRIMARY KEY,
-                    value TEXT
-                )
-            """,
-            )
-            conn.set_config("dbversion", self.CURRENT_DBVERSION)
+            conn.execute("PRAGMA user_version=%s" % (self.CURRENT_DBVERSION,))

chatmaild/src/chatmaild/doveauth.py

@@ -0,0 +1,166 @@
+import logging
+import os
+import time
+import sys
+import json
+import crypt
+from socketserver import (
+    UnixStreamServer,
+    StreamRequestHandler,
+    ThreadingMixIn,
+)
+import pwd
+
+from .database import Database
+from .config import read_config, Config
+
+NOCREATE_FILE = "/etc/chatmail-nocreate"
+
+
+def encrypt_password(password: str):
+    # https://doc.dovecot.org/configuration_manual/authentication/password_schemes/
+    passhash = crypt.crypt(password, crypt.METHOD_SHA512)
+    return "{SHA512-CRYPT}" + passhash
+
+
+def is_allowed_to_create(config: Config, user, cleartext_password) -> bool:
+    """Return True if user and password are admissable."""
+    if os.path.exists(NOCREATE_FILE):
+        logging.warning(f"blocked account creation because {NOCREATE_FILE!r} exists.")
+        return False
+
+    if len(cleartext_password) < config.password_min_length:
+        logging.warning(
+            "Password needs to be at least %s characters long",
+            config.password_min_length,
+        )
+        return False
+
+    parts = user.split("@")
+    if len(parts) != 2:
+        logging.warning(f"user {user!r} is not a proper e-mail address")
+        return False
+    localpart, domain = parts
+
+    if (
+        len(localpart) > config.username_max_length
+        or len(localpart) < config.username_min_length
+    ):
+        if localpart != "echo":
+            logging.warning(
+                "localpart %s has to be between %s and %s chars long",
+                localpart,
+                config.username_min_length,
+                config.username_max_length,
+            )
+            return False
+
+    return True
+
+
+def get_user_data(db, user):
+    with db.read_connection() as conn:
+        result = conn.get_user(user)
+    if result:
+        result["uid"] = "vmail"
+        result["gid"] = "vmail"
+    return result
+
+
+def lookup_userdb(db, user):
+    return get_user_data(db, user)
+
+
+def lookup_passdb(db, config: Config, user, cleartext_password):
+    with db.write_transaction() as conn:
+        userdata = conn.get_user(user)
+        if userdata:
+            # Update last login time.
+            conn.execute(
+                "UPDATE users SET last_login=? WHERE addr=?", (int(time.time()), user)
+            )
+
+            userdata["uid"] = "vmail"
+            userdata["gid"] = "vmail"
+            return userdata
+        if not is_allowed_to_create(config, user, cleartext_password):
+            return
+
+        encrypted_password = encrypt_password(cleartext_password)
+        q = """INSERT INTO users (addr, password, last_login)
+               VALUES (?, ?, ?)"""
+        conn.execute(q, (user, encrypted_password, int(time.time())))
+        return dict(
+            home=f"/home/vmail/{user}",
+            uid="vmail",
+            gid="vmail",
+            password=encrypted_password,
+        )
+
+
+def handle_dovecot_request(msg, db, config: Config):
+    short_command = msg[0]
+    if short_command == "L":  # LOOKUP
+        parts = msg[1:].split("\t")
+        keyname, user = parts[:2]
+        namespace, type, *args = keyname.split("/")
+        reply_command = "F"
+        res = ""
+        if namespace == "shared":
+            if type == "userdb":
+                if user.endswith(f"@{config.mail_domain}"):
+                    res = lookup_userdb(db, user)
+                if res:
+                    reply_command = "O"
+                else:
+                    reply_command = "N"
+            elif type == "passdb":
+                if user.endswith(f"@{config.mail_domain}"):
+                    res = lookup_passdb(db, config, user, cleartext_password=args[0])
+                if res:
+                    reply_command = "O"
+                else:
+                    reply_command = "N"
+        json_res = json.dumps(res) if res else ""
+        return f"{reply_command}{json_res}\n"
+    return None
+
+
+class ThreadedUnixStreamServer(ThreadingMixIn, UnixStreamServer):
+    request_queue_size = 100
+
+
+def main():
+    socket = sys.argv[1]
+    passwd_entry = pwd.getpwnam(sys.argv[2])
+    db = Database(sys.argv[3])
+    config = read_config(sys.argv[4])
+
+    class Handler(StreamRequestHandler):
+        def handle(self):
+            try:
+                while True:
+                    msg = self.rfile.readline().strip().decode()
+                    if not msg:
+                        break
+                    res = handle_dovecot_request(msg, db, config)
+                    if res:
+                        self.wfile.write(res.encode("ascii"))
+                        self.wfile.flush()
+                    else:
+                        logging.warn("request had no answer: %r", msg)
+            except Exception:
+                logging.exception("Exception in the handler")
+                raise
+
+    try:
+        os.unlink(socket)
+    except FileNotFoundError:
+        pass
+
+    with ThreadedUnixStreamServer(socket, Handler) as server:
+        os.chown(socket, uid=passwd_entry.pw_uid, gid=passwd_entry.pw_gid)
+        try:
+            server.serve_forever()
+        except KeyboardInterrupt:
+            pass

chatmaild/src/chatmaild/doveauth.service.f

@@ -0,0 +1,10 @@
+[Unit]
+Description=Chatmail dict authentication proxy for dovecot
+
+[Service]
+ExecStart={execpath} /run/dovecot/doveauth.socket vmail /home/vmail/passdb.sqlite {config_path}
+Restart=always
+RestartSec=30
+
+[Install]
+WantedBy=multi-user.target

chatmaild/src/chatmaild/echo.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""Advanced echo bot example.
+
+it will echo back any message that has non-empty text and also supports the /help command.
+"""
+import logging
+import os
+import sys
+from threading import Thread
+
+from deltachat_rpc_client import Bot, DeltaChat, EventType, Rpc, events
+
+from chatmaild.newemail import create_newemail_dict
+from chatmaild.config import read_config
+
+hooks = events.HookCollection()
+
+
+@hooks.on(events.RawEvent)
+def log_event(event):
+    if event.kind == EventType.INFO:
+        logging.info(event.msg)
+    elif event.kind == EventType.WARNING:
+        logging.warning(event.msg)
+
+
+@hooks.on(events.RawEvent(EventType.ERROR))
+def log_error(event):
+    logging.error(event.msg)
+
+
+@hooks.on(events.MemberListChanged)
+def on_memberlist_changed(event):
+    logging.info(
+        "member %s was %s", event.member, "added" if event.member_added else "removed"
+    )
+
+
+@hooks.on(events.GroupImageChanged)
+def on_group_image_changed(event):
+    logging.info("group image %s", "deleted" if event.image_deleted else "changed")
+
+
+@hooks.on(events.GroupNameChanged)
+def on_group_name_changed(event):
+    logging.info("group name changed, old name: %s", event.old_name)
+
+
+@hooks.on(events.NewMessage(func=lambda e: not e.command))
+def echo(event):
+    snapshot = event.message_snapshot
+    if snapshot.text or snapshot.file:
+        snapshot.chat.send_message(text=snapshot.text, file=snapshot.file)
+
+
+@hooks.on(events.NewMessage(command="/help"))
+def help_command(event):
+    snapshot = event.message_snapshot
+    snapshot.chat.send_text("Send me any message and I will echo it back")
+
+
+def main():
+    path = os.environ.get("PATH")
+    venv_path = sys.argv[0].strip("echobot")
+    os.environ["PATH"] = path + ":" + venv_path
+    with Rpc() as rpc:
+        deltachat = DeltaChat(rpc)
+        system_info = deltachat.get_system_info()
+        logging.info("Running deltachat core %s", system_info.deltachat_core_version)
+
+        accounts = deltachat.get_all_accounts()
+        account = accounts[0] if accounts else deltachat.add_account()
+
+        bot = Bot(account, hooks)
+        if not bot.is_configured():
+            config = read_config(sys.argv[1])
+            password = create_newemail_dict(config).get("password")
+            email = "echo@" + config.mail_domain
+            configure_thread = Thread(
+                target=bot.configure, kwargs={"email": email, "password": password}
+            )
+            configure_thread.start()
+        bot.run_forever()
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO)
+    main()

chatmaild/src/chatmaild/echobot.service.f

@@ -0,0 +1,11 @@
+[Unit]
+Description=Chatmail echo bot for testing it works
+
+[Service]
+ExecStart={execpath} {config_path}
+Environment="PATH={remote_venv_dir}:$PATH"
+Restart=always
+RestartSec=30
+
+[Install]
+WantedBy=multi-user.target

chatmaild/src/chatmaild/filtermail.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+import asyncio
+import logging
+import time
+import sys
+from email.parser import BytesParser
+from email import policy
+from email.utils import parseaddr
+
+from aiosmtpd.controller import Controller
+from smtplib import SMTP as SMTPClient
+
+from .config import read_config
+
+
+def check_encrypted(message):
+    """Check that the message is an OpenPGP-encrypted message."""
+    if not message.is_multipart():
+        return False
+    if message.get("subject") != "...":
+        return False
+    if message.get_content_type() != "multipart/encrypted":
+        return False
+    parts_count = 0
+    for part in message.iter_parts():
+        if parts_count == 0:
+            if part.get_content_type() != "application/pgp-encrypted":
+                return False
+        elif parts_count == 1:
+            if part.get_content_type() != "application/octet-stream":
+                return False
+        else:
+            return False
+        parts_count += 1
+    return True
+
+
+def check_mdn(message, envelope):
+    if len(envelope.rcpt_tos) != 1:
+        return False
+
+    for name in ["auto-submitted", "chat-version"]:
+        if not message.get(name):
+            return False
+
+    if message.get_content_type() != "multipart/report":
+        return False
+
+    body = message.get_body()
+    if body.get_content_type() != "text/plain":
+        return False
+
+    if list(body.iter_attachments()) or list(body.iter_parts()):
+        return False
+
+    # even with all mime-structural checks an attacker
+    # could try to abuse the subject or body to contain links or other
+    # annoyance -- we skip on checking subject/body for now as Delta Chat
+    # should evolve to create E2E-encrypted read receipts anyway.
+    # and then MDNs are just encrypted mail and can pass the border
+    # to other instances.
+
+    return True
+
+
+async def asyncmain_beforequeue(config):
+    port = config.filtermail_smtp_port
+    Controller(BeforeQueueHandler(config), hostname="127.0.0.1", port=port).start()
+
+
+class BeforeQueueHandler:
+    def __init__(self, config):
+        self.config = config
+        self.send_rate_limiter = SendRateLimiter()
+
+    async def handle_MAIL(self, server, session, envelope, address, mail_options):
+        logging.info(f"handle_MAIL from {address}")
+        envelope.mail_from = address
+        max_sent = self.config.max_user_send_per_minute
+        if not self.send_rate_limiter.is_sending_allowed(address, max_sent):
+            return f"450 4.7.1: Too much mail from {address}"
+
+        parts = envelope.mail_from.split("@")
+        if len(parts) != 2:
+            return f"500 Invalid from address <{envelope.mail_from!r}>"
+
+        return "250 OK"
+
+    async def handle_DATA(self, server, session, envelope):
+        logging.info("handle_DATA before-queue")
+        error = self.check_DATA(envelope)
+        if error:
+            return error
+        logging.info("re-injecting the mail that passed checks")
+        client = SMTPClient("localhost", self.config.postfix_reinject_port)
+        client.sendmail(envelope.mail_from, envelope.rcpt_tos, envelope.content)
+        return "250 OK"
+
+    def check_DATA(self, envelope):
+        """the central filtering function for e-mails."""
+        logging.info(f"Processing DATA message from {envelope.mail_from}")
+
+        message = BytesParser(policy=policy.default).parsebytes(envelope.content)
+        mail_encrypted = check_encrypted(message)
+
+        _, from_addr = parseaddr(message.get("from").strip())
+        logging.info(f"mime-from: {from_addr} envelope-from: {envelope.mail_from!r}")
+        if envelope.mail_from.lower() != from_addr.lower():
+            return f"500 Invalid FROM <{from_addr!r}> for <{envelope.mail_from!r}>"
+
+        if not mail_encrypted and check_mdn(message, envelope):
+            return
+
+        if envelope.mail_from in self.config.passthrough_senders:
+            return
+
+        passthrough_recipients = self.config.passthrough_recipients
+        envelope_from_domain = from_addr.split("@").pop()
+        for recipient in envelope.rcpt_tos:
+            if envelope.mail_from == recipient:
+                # Always allow sending emails to self.
+                continue
+            if recipient in passthrough_recipients:
+                continue
+            res = recipient.split("@")
+            if len(res) != 2:
+                return f"500 Invalid address <{recipient}>"
+            _recipient_addr, recipient_domain = res
+
+            is_outgoing = recipient_domain != envelope_from_domain
+            if is_outgoing and not mail_encrypted:
+                is_securejoin = message.get("secure-join") in [
+                    "vc-request",
+                    "vg-request",
+                ]
+                if not is_securejoin:
+                    return f"500 Invalid unencrypted mail to <{recipient}>"
+
+
+class SendRateLimiter:
+    def __init__(self):
+        self.addr2timestamps = {}
+
+    def is_sending_allowed(self, mail_from, max_send_per_minute):
+        last = self.addr2timestamps.setdefault(mail_from, [])
+        now = time.time()
+        last[:] = [ts for ts in last if ts >= (now - 60)]
+        if len(last) <= max_send_per_minute:
+            last.append(now)
+            return True
+        return False
+
+
+def main():
+    args = sys.argv[1:]
+    assert len(args) == 1
+    config = read_config(args[0])
+    logging.basicConfig(level=logging.WARN)
+    loop = asyncio.new_event_loop()
+    asyncio.set_event_loop(loop)
+    task = asyncmain_beforequeue(config)
+    loop.create_task(task)
+    loop.run_forever()

chatmaild/src/chatmaild/filtermail.service.f

@@ -0,0 +1,10 @@
+[Unit]
+Description=Chatmail Postfix BeforeQeue filter 
+
+[Service]
+ExecStart={execpath} {config_path}
+Restart=always
+RestartSec=30
+
+[Install]
+WantedBy=multi-user.target

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -0,0 +1,63 @@
+[params]
+
+# mail domain (MUST be set to fully qualified chat mail domain)
+mail_domain = {mail_domain}
+
+#
+# If you only do private test deploys, you don't need to modify any settings below
+#
+
+#
+# Account Restrictions
+#
+
+# how many mails a user can send out per minute
+max_user_send_per_minute = 60
+
+# maximum mailbox size of a chatmail account
+max_mailbox_size = 100M
+
+# time after which seen mails are deleted
+delete_mails_after = 40d
+
+# minimum length a username must have
+username_min_length = 9
+
+# maximum length a username can have
+username_max_length = 9
+
+# minimum length a password must have
+password_min_length = 9
+
+# list of chatmail accounts which can send outbound un-encrypted mail
+passthrough_senders =
+
+# list of e-mail recipients for which to accept outbound un-encrypted mails
+passthrough_recipients =
+
+#
+# Deployment Details
+#
+
+# where the filtermail SMTP service listens
+filtermail_smtp_port = 10080
+
+# postfix accepts on the localhost reinject SMTP port
+postfix_reinject_port = 10025
+
+#
+# Privacy Policy
+#
+
+# postal address of privacy contact
+privacy_postal =
+
+# email address of privacy contact
+privacy_mail =
+
+# postal address of the privacy data officer
+privacy_pdo =
+
+# postal address of the privacy supervisor
+privacy_supervisor =
+

chatmaild/src/chatmaild/ini/override-testrun.ini

@@ -0,0 +1,16 @@
+
+[privacy]
+
+passthrough_recipients = privacy@testrun.org
+
+privacy_postal =
+    Merlinux GmbH, Represented by the managing director H. Krekel,
+    Reichgrafen Str. 20, 79102 Freiburg, Germany
+
+privacy_mail = privacy@testrun.org
+privacy_pdo =
+    Prof. Dr. Fabian Schmieder, lexICT UG (limited), Ostfeldstr. 49, 30559 Hannover.
+    You can contact him at *delta-privacy@merlinux.eu* (Keyword: DPO)
+privacy_supervisor =
+    State Commissioner for Data Protection and Freedom of Information of
+    Baden-Württemberg in 70173 Stuttgart, Germany.

chatmaild/src/chatmaild/newemail.py

@@ -0,0 +1,36 @@
+#!/usr/local/lib/chatmaild/venv/bin/python3
+
+""" CGI script for creating new accounts. """
+
+import json
+import random
+import secrets
+import string
+
+from chatmaild.config import read_config, Config
+
+CONFIG_PATH = "/usr/local/lib/chatmaild/chatmail.ini"
+ALPHANUMERIC = string.ascii_lowercase + string.digits
+ALPHANUMERIC_PUNCT = string.ascii_letters + string.digits + string.punctuation
+
+
+def create_newemail_dict(config: Config):
+    user = "".join(random.choices(ALPHANUMERIC, k=config.username_min_length))
+    password = "".join(
+        secrets.choice(ALPHANUMERIC_PUNCT)
+        for _ in range(config.password_min_length + 3)
+    )
+    return dict(email=f"{user}@{config.mail_domain}", password=f"{password}")
+
+
+def print_new_account():
+    config = read_config(CONFIG_PATH)
+    creds = create_newemail_dict(config)
+
+    print("Content-Type: application/json")
+    print("")
+    print(json.dumps(creds))
+
+
+if __name__ == "__main__":
+    print_new_account()

chatmaild/src/chatmaild/tests/mail-data/encrypted.eml

@@ -0,0 +1,66 @@
+From: {from_addr}
+To: {to_addr}
+Subject: ...
+Date: Sun, 15 Oct 2023 16:43:21 +0000
+Message-ID: <Mr.UVyJWZmkCKM.hGzNc6glBE_@c2.testrun.org>
+In-Reply-To: <Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>
+References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
+	<Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>
+Chat-Version: 1.0
+Autocrypt: addr={from_addr}; prefer-encrypt=mutual;
+	keydata=xjMEZSwWjhYJKwYBBAHaRw8BAQdAQBEhqeJh0GueHB6kF/DUQqYCxARNBVokg/AzT+7LqH
+	rNFzxiYXJiYXpAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUsFo4CGwMECwkIBwYVCAkKCwID
+	FgIBFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX9A4AEAnHWHp49eBCMHK5t66gYPiW
+	XQuB1mwUjzGfYWB+0RXUoA/0xcQ3FbUNlGKW7Blp6eMFfViv6Mv2d3kNSXACB6nmcMzjgEZSwWjhIK
+	KwYBBAGXVQEFAQEHQBpY5L2M1XHo0uxf8SX1wNLBp/OVvidoWHQF2Jz+kJsUAwEIB8J4BBgWCAAgBQ
+	JlLBaOAhsMFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX/INgEA37AJaNvruYsJVanP
+	IXnYw4CKd55UAwl8Zcy+M2diAbkA/0fHHcGV4r78hpbbL1Os52DPOdqYQRauIeJUeG+G6bQO
+MIME-Version: 1.0
+Content-Type: multipart/encrypted; protocol="application/pgp-encrypted";
+	boundary="YFrteb74qSXmggbOxZL9dRnhymywAi"
+
+
+--YFrteb74qSXmggbOxZL9dRnhymywAi
+Content-Description: PGP/MIME version identification
+Content-Type: application/pgp-encrypted
+
+Version: 1
+
+
+--YFrteb74qSXmggbOxZL9dRnhymywAi
+Content-Description: OpenPGP encrypted message
+Content-Disposition: inline; filename="encrypted.asc";
+Content-Type: application/octet-stream; name="encrypted.asc"
+
+-----BEGIN PGP MESSAGE-----
+
+wU4DhW3gBZ/VvCYSAQdA8bMs2spwbKdGjVsL1ByPkNrqD7frpB73maeL6I6SzDYg
+O5G53tv339RdKq3WRcCtEEvxjHlUx2XNwXzC04BpmfvBTgNfPUyLDzjXnxIBB0Ae
+8ymwGvXMCCimHXN0Dg8Ui62KOi03h0UgheoHWovJSCDF4CKre/xtFr3nL7lq/PKI
+JsjVNz7/RK9FSXF6WwfONtLCyQGEuVAsB/KXfCBEyfKhaMwGHvhujRidGW5uV1no
+lMGl3ODmo29Lgeu2uSE7EpJRZoe6hU6ddmBkqxax61ZtkaFlGFFpdo2K8balNNdz
+ZsJ/9mmI9x3oOJ4/l1nhQbUO9ADbs7gJhFdV5Qkp30b5fCI7bU+aoe1ccBbLe/WM
+YUty1PqcuQT7XjA+XmYuL261tvW8pBetT+i33/E2d8PzzYt2IuK9qeevyS+yxdwA
+kfwejFWzzsUlJaDxs1x4XOxkMgSj+jo+g12dFOb7fyClsAnq23iDb8AuaT/BScAI
++lO+gher69+6LmM7VGHLG5k762J1jTaQCaKt1s8TAWV99Eo4491vL6fyvk3l/Cfg
+RXSwiWFgj19Pn0Rq7CD9v22UE2vdUMBTcV4aw79mClk1YQ23jbF0y5DCjPdJ62Zo
+tskBgFt3NoWV80jZ76zIBLrrjLwCCll8JjJtFwSkt2GX5RFBsVa4A8IDht9RtEk7
+rrHgbSZQfkauEi/mH3/6CDZoLqSHudUZ7d4MaJwun1TkFYGe2ORwGJd4OBj3oGJp
+H8YBwCpk///L/fKjX0Gg3M8nrpM4wrRFhPKidAgO/kcm25X4+ZHlVkWBTCt5RWKI
+fHh6oLDZCqCfcgMkE1KKmwfIHaUkhq5BPRigwy6i5dh1DM4+1UCLh3dxzVbqE9b9
+61NB19nXdRtDA2sOUnj9ve6m/wEPyCb6/zBQZqvCBYb1/AjdXpUrFT+DbpfyxaXN
+XfhDVb5mNqNM/IVj0V5fvTc6vOfYbzQtPm10H+FdWWfb+rJRfyC3MA2w2IqstFe3
+w3bu2iE6CQvSqRvge+ZqLKt/NqYwOURiUmpuklbl3kPJ97+mfKWoiqk8Iz1VY+bb
+NMUC7aoGv+jcoj+WS6PYO8N6BeRVUUB3ZJSf8nzjgxm1/BcM+UD3BPrlhT11ODRs
+baifGbprMWwt3dhb8cQgRT8GPdpO1OsDkzL6iikMjLHWWiA99GV6ruiHsIPw6boW
+A6/uSOskbDHOROotKmddGTBd0iiHXAoQsJFt1ZjUkt6EHrgWs+GAvrvKpXs1mrz8
+uj3GwEFrHS+Xuf2UDgpszYT3hI2cL/kUtGakVR7m7vVMZqXBUbZdGAEb1PZNPwsI
+E4aMK02+EVB+tSN4Fzj99N2YD0inVYt+oPjr2tHhUS6aSGBNS/48Ki47DOg4Sxkn
+lkOWnEbCD+XTnbDd
+=agR5
+-----END PGP MESSAGE-----
+
+
+--YFrteb74qSXmggbOxZL9dRnhymywAi--
+
+

chatmaild/src/chatmaild/tests/mail-data/fake-encrypted.eml

@@ -0,0 +1,25 @@
+Subject: =?utf-8?q?Message_from_foobar=40c2=2Etestrun=2Eorg?=
+Chat-Disposition-Notification-To: foobar@c2.testrun.org
+Chat-User-Avatar: 0
+From: <{from_addr}>
+To: <{to_addr}>
+Date: Sun, 15 Oct 2023 16:41:44 +0000
+Message-ID: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
+References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
+Chat-Version: 1.0
+Autocrypt: addr={from_addr}; prefer-encrypt=mutual;
+	keydata=xjMEZSrw3hYJKwYBBAHaRw8BAQdAiEKNQFU28c6qsx4vo/JHdt73RXdjMOmByf/XsGiJ7m
+	nNFzxmb29iYXJAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUq8N4CGwMECwkIBwYVCAkKCwID
+	FgIBFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJCX3gEAhm0MehE5byBBU1avPczr/I
+	HjNLht7Qf6++mAhlJmtDcA/0C8VYJhsUpmiDjuZaMDWNv4FO2BJG6LH7gSm6n7ClMJzjgEZSrw3hIK
+	KwYBBAGXVQEFAQEHQAxGG/QW0owCfMp1A+vXEMwgzWcBpNFr58kX2eXuPpM6AwEIB8J4BBgWCAAgBQ
+	JlKvDeAhsMFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJDg1gEAwLf8KDoAAKyYgjyI
+	vYvO9VEgBni1C4Xx1VjcaEmlDK8BALoFuUCK+enw76TtDcAUKhlhUiM6SDRExkS4Nskp/BcK
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
+
+-----BEGIN PGP MESSAGE-----
+Hi!
+-----END PGP MESSAGE-----
+
+

chatmaild/src/chatmaild/tests/mail-data/mdn.eml

@@ -0,0 +1,33 @@
+Subject: Message opened
+From: <{from_addr}>
+To: <{to_addr}>
+Date: Sun, 15 Oct 2023 16:43:25 +0000
+Message-ID: <Mr.78MWtlV7RAi.goCFzBhCYfy@c2.testrun.org>
+Auto-Submitted: auto-replied
+Chat-Version: 1.0
+MIME-Version: 1.0
+Content-Type: multipart/report; report-type=disposition-notification;
+	boundary="Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi"
+
+
+--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi
+Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
+
+The "Hi!" message you sent was displayed on the screen of the recipient.
+
+This is no guarantee the content was read.
+
+
+--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi
+Content-Type: message/disposition-notification
+
+Reporting-UA: Delta Chat 1.124.1
+Original-Recipient: rfc822;barbaz@c2.testrun.org
+Final-Recipient: rfc822;barbaz@c2.testrun.org
+Original-Message-ID: <Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>
+Disposition: manual-action/MDN-sent-automatically; displayed
+
+
+--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi--
+
+

chatmaild/src/chatmaild/tests/mail-data/plain.eml

@@ -0,0 +1,23 @@
+Subject: =?utf-8?q?Message_from_foobar=40c2=2Etestrun=2Eorg?=
+Chat-Disposition-Notification-To: foobar@c2.testrun.org
+Chat-User-Avatar: 0
+From: <{from_addr}>
+To: <{to_addr}>
+Date: Sun, 15 Oct 2023 16:41:44 +0000
+Message-ID: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
+References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
+Chat-Version: 1.0
+Autocrypt: addr=foobar@c2.testrun.org; prefer-encrypt=mutual;
+	keydata=xjMEZSrw3hYJKwYBBAHaRw8BAQdAiEKNQFU28c6qsx4vo/JHdt73RXdjMOmByf/XsGiJ7m
+	nNFzxmb29iYXJAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUq8N4CGwMECwkIBwYVCAkKCwID
+	FgIBFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJCX3gEAhm0MehE5byBBU1avPczr/I
+	HjNLht7Qf6++mAhlJmtDcA/0C8VYJhsUpmiDjuZaMDWNv4FO2BJG6LH7gSm6n7ClMJzjgEZSrw3hIK
+	KwYBBAGXVQEFAQEHQAxGG/QW0owCfMp1A+vXEMwgzWcBpNFr58kX2eXuPpM6AwEIB8J4BBgWCAAgBQ
+	JlKvDeAhsMFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJDg1gEAwLf8KDoAAKyYgjyI
+	vYvO9VEgBni1C4Xx1VjcaEmlDK8BALoFuUCK+enw76TtDcAUKhlhUiM6SDRExkS4Nskp/BcK
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
+
+Hi!
+
+

chatmaild/src/chatmaild/tests/plugin.py

@@ -0,0 +1,68 @@
+import random
+import importlib.resources
+import itertools
+from email.parser import BytesParser
+from email import policy
+import pytest
+
+from chatmaild.database import Database
+from chatmaild.config import read_config, write_initial_config
+
+
+@pytest.fixture
+def make_config(tmp_path):
+    inipath = tmp_path.joinpath("chatmail.ini")
+
+    def make_conf(mail_domain):
+        write_initial_config(inipath, mail_domain=mail_domain)
+        return read_config(inipath)
+
+    return make_conf
+
+
+@pytest.fixture
+def example_config(make_config):
+    return make_config("chat.example.org")
+
+
+@pytest.fixture
+def maildomain(example_config):
+    return example_config.mail_domain
+
+
+@pytest.fixture
+def gencreds(maildomain):
+    count = itertools.count()
+    next(count)
+
+    def gen(domain=None):
+        domain = domain if domain else maildomain
+        while 1:
+            num = next(count)
+            alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
+            user = "".join(random.choices(alphanumeric, k=10))
+            user = f"ac{num}_{user}"[:9]
+            password = "".join(random.choices(alphanumeric, k=12))
+            yield f"{user}@{domain}", f"{password}"
+
+    return lambda domain=None: next(gen(domain))
+
+
+@pytest.fixture()
+def db(tmpdir):
+    db_path = tmpdir / "passdb.sqlite"
+    print("database path:", db_path)
+    return Database(db_path)
+
+
+@pytest.fixture
+def maildata(request):
+    datadir = importlib.resources.files(__package__).joinpath("mail-data")
+    assert datadir.exists(), datadir
+
+    def maildata(name, from_addr, to_addr):
+        data = datadir.joinpath(name).read_text()
+        text = data.format(from_addr=from_addr, to_addr=to_addr)
+        return BytesParser(policy=policy.default).parsebytes(text.encode())
+
+    return maildata

chatmaild/src/chatmaild/tests/test_config.py

@@ -0,0 +1,32 @@
+from chatmaild.config import read_config
+
+
+def test_read_config_basic(example_config):
+    assert example_config.mail_domain == "chat.example.org"
+    assert not example_config.privacy_supervisor and not example_config.privacy_mail
+    assert not example_config.privacy_pdo and not example_config.privacy_postal
+
+    inipath = example_config._inipath
+    inipath.write_text(inipath.read_text().replace("60", "37"))
+    example_config = read_config(inipath)
+    assert example_config.max_user_send_per_minute == 37
+    assert example_config.mail_domain == "chat.example.org"
+
+
+def test_read_config_testrun(make_config):
+    config = make_config("something.testrun.org")
+    assert config.mail_domain == "something.testrun.org"
+    assert len(config.privacy_postal.split("\n")) > 1
+    assert len(config.privacy_supervisor.split("\n")) > 1
+    assert len(config.privacy_pdo.split("\n")) > 1
+    assert config.privacy_mail == "privacy@testrun.org"
+    assert config.filtermail_smtp_port == 10080
+    assert config.postfix_reinject_port == 10025
+    assert config.max_user_send_per_minute == 60
+    assert config.max_mailbox_size == "100M"
+    assert config.delete_mails_after == "40d"
+    assert config.username_min_length == 9
+    assert config.username_max_length == 9
+    assert config.password_min_length == 9
+    assert config.passthrough_recipients == ["privacy@testrun.org"]
+    assert config.passthrough_senders == []

chatmaild/src/chatmaild/tests/test_doveauth.py

@@ -0,0 +1,95 @@
+import json
+import pytest
+import threading
+import queue
+import traceback
+
+import chatmaild.doveauth
+from chatmaild.doveauth import get_user_data, lookup_passdb, handle_dovecot_request
+from chatmaild.database import DBError
+
+
+def test_basic(db, example_config):
+    lookup_passdb(db, example_config, "asdf12345@chat.example.org", "q9mr3faue")
+    data = get_user_data(db, "asdf12345@chat.example.org")
+    assert data
+    data2 = lookup_passdb(
+        db, example_config, "asdf12345@chat.example.org", "q9mr3jewvadsfaue"
+    )
+    assert data == data2
+
+
+def test_dont_overwrite_password_on_wrong_login(db, example_config):
+    """Test that logging in with a different password doesn't create a new user"""
+    res = lookup_passdb(
+        db, example_config, "newuser12@chat.example.org", "kajdlkajsldk12l3kj1983"
+    )
+    assert res["password"]
+    res2 = lookup_passdb(db, example_config, "newuser12@chat.example.org", "kajdslqwe")
+    # this function always returns a password hash, which is actually compared by dovecot.
+    assert res["password"] == res2["password"]
+
+
+def test_nocreate_file(db, monkeypatch, tmpdir, example_config):
+    p = tmpdir.join("nocreate")
+    p.write("")
+    monkeypatch.setattr(chatmaild.doveauth, "NOCREATE_FILE", str(p))
+    lookup_passdb(
+        db, example_config, "newuser12@chat.example.org", "zequ0Aimuchoodaechik"
+    )
+    assert not get_user_data(db, "newuser12@chat.example.org")
+
+
+def test_db_version(db):
+    assert db.get_schema_version() == 1
+
+
+def test_too_high_db_version(db):
+    with db.write_transaction() as conn:
+        conn.execute("PRAGMA user_version=%s;" % (999,))
+    with pytest.raises(DBError):
+        db.ensure_tables()
+
+
+def test_handle_dovecot_request(db, example_config):
+    msg = (
+        "Lshared/passdb/laksjdlaksjdlaksjdlk12j3l1k2j3123/"
+        "some42123@chat.example.org\tsome42123@chat.example.org"
+    )
+    res = handle_dovecot_request(msg, db, example_config)
+    assert res
+    assert res[0] == "O" and res.endswith("\n")
+    userdata = json.loads(res[1:].strip())
+    assert userdata["home"] == "/home/vmail/some42123@chat.example.org"
+    assert userdata["uid"] == userdata["gid"] == "vmail"
+    assert userdata["password"].startswith("{SHA512-CRYPT}")
+
+
+def test_50_concurrent_lookups_different_accounts(db, gencreds, example_config):
+    num_threads = 50
+    req_per_thread = 5
+    results = queue.Queue()
+
+    def lookup(db):
+        for i in range(req_per_thread):
+            addr, password = gencreds()
+            try:
+                lookup_passdb(db, example_config, addr, password)
+            except Exception:
+                results.put(traceback.format_exc())
+            else:
+                results.put(None)
+
+    threads = []
+    for i in range(num_threads):
+        thread = threading.Thread(target=lookup, args=(db,), daemon=True)
+        threads.append(thread)
+
+    print(f"created {num_threads} threads, starting them and waiting for results")
+    for thread in threads:
+        thread.start()
+
+    for i in range(num_threads * req_per_thread):
+        res = results.get()
+        if res is not None:
+            pytest.fail(f"concurrent lookup failed\n{res}")

chatmaild/src/chatmaild/tests/test_filtermail.py

@@ -0,0 +1,145 @@
+from chatmaild.filtermail import (
+    check_encrypted,
+    BeforeQueueHandler,
+    SendRateLimiter,
+    check_mdn,
+)
+
+import pytest
+
+
+@pytest.fixture
+def maildomain():
+    # let's not depend on a real chatmail instance for the offline tests below
+    return "chatmail.example.org"
+
+
+@pytest.fixture
+def handler(make_config, maildomain):
+    config = make_config(maildomain)
+    return BeforeQueueHandler(config)
+
+
+def test_reject_forged_from(maildata, gencreds, handler):
+    class env:
+        mail_from = gencreds()[0]
+        rcpt_tos = [gencreds()[0]]
+
+    # test that the filter lets good mail through
+    to_addr = gencreds()[0]
+    env.content = maildata(
+        "plain.eml", from_addr=env.mail_from, to_addr=to_addr
+    ).as_bytes()
+
+    assert not handler.check_DATA(envelope=env)
+
+    # test that the filter rejects forged mail
+    env.content = maildata(
+        "plain.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
+    ).as_bytes()
+    error = handler.check_DATA(envelope=env)
+    assert "500" in error
+
+
+def test_filtermail_no_encryption_detection(maildata):
+    msg = maildata(
+        "plain.eml", from_addr="some@example.org", to_addr="other@example.org"
+    )
+    assert not check_encrypted(msg)
+
+    # https://xkcd.com/1181/
+    msg = maildata(
+        "fake-encrypted.eml", from_addr="some@example.org", to_addr="other@example.org"
+    )
+    assert not check_encrypted(msg)
+
+
+def test_filtermail_encryption_detection(maildata):
+    msg = maildata("encrypted.eml", from_addr="1@example.org", to_addr="2@example.org")
+    assert check_encrypted(msg)
+
+    # if the subject is not "..." it is not considered ac-encrypted
+    msg.replace_header("Subject", "Click this link")
+    assert not check_encrypted(msg)
+
+
+def test_filtermail_is_mdn(maildata, gencreds, handler):
+    from_addr = gencreds()[0]
+    to_addr = gencreds()[0] + ".other"
+    msg = maildata("mdn.eml", from_addr, to_addr)
+
+    class env:
+        mail_from = from_addr
+        rcpt_tos = [to_addr]
+        content = msg.as_bytes()
+
+    assert check_mdn(msg, env)
+    print(msg.as_string())
+
+    assert not handler.check_DATA(env)
+
+
+def test_filtermail_to_multiple_recipients_no_mdn(maildata, gencreds):
+    from_addr = gencreds()[0]
+    to_addr = gencreds()[0] + ".other"
+    thirdaddr = gencreds()[0]
+    msg = maildata("mdn.eml", from_addr, to_addr)
+
+    class env:
+        mail_from = from_addr
+        rcpt_tos = [to_addr, thirdaddr]
+        content = msg.as_bytes()
+
+    assert not check_mdn(msg, env)
+
+
+def test_send_rate_limiter():
+    limiter = SendRateLimiter()
+    for i in range(100):
+        if limiter.is_sending_allowed("some@example.org", 10):
+            if i <= 10:
+                continue
+            pytest.fail("limiter didn't work")
+        else:
+            assert i == 11
+            break
+
+
+def test_excempt_privacy(maildata, gencreds, handler):
+    from_addr = gencreds()[0]
+    to_addr = "privacy@testrun.org"
+    handler.config.passthrough_recipients = [to_addr]
+    false_to = "privacy@something.org"
+
+    msg = maildata("plain.eml", from_addr, to_addr)
+
+    class env:
+        mail_from = from_addr
+        rcpt_tos = [to_addr]
+        content = msg.as_bytes()
+
+    # assert that None/no error is returned
+    assert not handler.check_DATA(envelope=env)
+
+    class env2:
+        mail_from = from_addr
+        rcpt_tos = [to_addr, false_to]
+        content = msg.as_bytes()
+
+    assert "500" in handler.check_DATA(envelope=env2)
+
+
+def test_passthrough_senders(gencreds, handler, maildata):
+    acc1 = gencreds()[0]
+    to_addr = "recipient@something.org"
+    handler.config.passthrough_senders = [acc1]
+
+    msg = maildata("plain.eml", acc1, to_addr)
+
+    class env:
+        mail_from = acc1
+        rcpt_tos = to_addr
+        content = msg.as_bytes()
+
+    # assert that None/no error is returned
+    assert not handler.check_DATA(envelope=env)

chatmaild/src/chatmaild/tests/test_newmail.py

@@ -0,0 +1,27 @@
+import json
+
+import chatmaild
+from chatmaild.newemail import create_newemail_dict, print_new_account
+
+
+def test_create_newemail_dict(example_config):
+    ac1 = create_newemail_dict(example_config)
+    assert "@" in ac1["email"]
+    assert len(ac1["password"]) >= 10
+
+    ac2 = create_newemail_dict(example_config)
+
+    assert ac1["email"] != ac2["email"]
+    assert ac1["password"] != ac2["password"]
+
+
+def test_print_new_account(capsys, monkeypatch, maildomain, tmpdir, example_config):
+    monkeypatch.setattr(chatmaild.newemail, "CONFIG_PATH", str(example_config._inipath))
+    print_new_account()
+    out, err = capsys.readouterr()
+    lines = out.split("\n")
+    assert lines[0] == "Content-Type: application/json"
+    assert not lines[1]
+    dic = json.loads(lines[2])
+    assert dic["email"].endswith(f"@{example_config.mail_domain}")
+    assert len(dic["password"]) >= 10

cmdeploy/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "cmdeploy"
+version = "0.2"
+dependencies = [
+  "pyinfra",
+  "pillow",
+  "qrcode",
+  "markdown",
+  "pytest",
+  "setuptools>=68",
+  "termcolor",
+  "build",
+  "tox",
+  "ruff",
+  "black",
+  "pytest",
+  "pytest-xdist", 
+]
+
+[project.scripts]
+cmdeploy = "cmdeploy.cmdeploy:main"
+
+[project.entry-points.pytest11]
+"chatmaild.testplugin" = "chatmaild.tests.plugin"
+"cmdeploy.testplugin" = "cmdeploy.tests.plugin"
+
+[tool.pytest.ini_options]
+addopts = "-v -ra --strict-markers"

cmdeploy/src/cmdeploy/__init__.py

@@ -0,0 +1,509 @@
+"""
+Chat Mail pyinfra deploy.
+"""
+import sys
+import importlib.resources
+import subprocess
+import shutil
+import io
+from pathlib import Path
+
+from pyinfra import host
+from pyinfra.operations import apt, files, server, systemd, pip
+from pyinfra.facts.files import File
+from pyinfra.facts.systemd import SystemdEnabled
+from .acmetool import deploy_acmetool
+
+from chatmaild.config import read_config, Config
+
+
+def _build_chatmaild(dist_dir) -> None:
+    dist_dir = Path(dist_dir).resolve()
+    if dist_dir.exists():
+        shutil.rmtree(dist_dir)
+    dist_dir.mkdir()
+    subprocess.check_output(
+        [sys.executable, "-m", "build", "-n"]
+        + ["--sdist", "chatmaild", "--outdir", str(dist_dir)]
+    )
+    entries = list(dist_dir.iterdir())
+    assert len(entries) == 1
+    return entries[0]
+
+
+def remove_legacy_artifacts():
+    # disable legacy doveauth-dictproxy.service
+    if host.get_fact(SystemdEnabled).get("doveauth-dictproxy.service"):
+        systemd.service(
+            name="Disable legacy doveauth-dictproxy.service",
+            service="doveauth-dictproxy.service",
+            running=False,
+            enabled=False,
+        )
+
+
+def _install_remote_venv_with_chatmaild(config) -> None:
+    remove_legacy_artifacts()
+    dist_file = _build_chatmaild(dist_dir=Path("chatmaild/dist"))
+    remote_base_dir = "/usr/local/lib/chatmaild"
+    remote_dist_file = f"{remote_base_dir}/dist/{dist_file.name}"
+    remote_venv_dir = f"{remote_base_dir}/venv"
+    remote_chatmail_inipath = f"{remote_base_dir}/chatmail.ini"
+    root_owned = dict(user="root", group="root", mode="644")
+
+    apt.packages(
+        name="apt install python3-virtualenv",
+        packages=["python3-virtualenv"],
+    )
+
+    files.put(
+        name="Upload chatmaild source package",
+        src=dist_file.open("rb"),
+        dest=remote_dist_file,
+        create_remote_dir=True,
+        **root_owned,
+    )
+
+    files.put(
+        name=f"Upload {remote_chatmail_inipath}",
+        src=config._getbytefile(),
+        dest=remote_chatmail_inipath,
+        **root_owned,
+    )
+
+    pip.virtualenv(
+        name=f"chatmaild virtualenv {remote_venv_dir}",
+        path=remote_venv_dir,
+        always_copy=True,
+    )
+
+    server.shell(
+        name=f"forced pip-install {dist_file.name}",
+        commands=[
+            f"{remote_venv_dir}/bin/pip install --force-reinstall {remote_dist_file}"
+        ],
+    )
+
+    # install systemd units
+    for fn in (
+        "doveauth",
+        "filtermail",
+        "echobot",
+    ):
+        params = dict(
+            execpath=f"{remote_venv_dir}/bin/{fn}",
+            config_path=remote_chatmail_inipath,
+            remote_venv_dir=remote_venv_dir,
+        )
+        source_path = importlib.resources.files("chatmaild").joinpath(f"{fn}.service.f")
+        content = source_path.read_text().format(**params).encode()
+
+        files.put(
+            name=f"Upload {fn}.service",
+            src=io.BytesIO(content),
+            dest=f"/etc/systemd/system/{fn}.service",
+            **root_owned,
+        )
+        systemd.service(
+            name=f"Setup {fn} service",
+            service=f"{fn}.service",
+            running=True,
+            enabled=True,
+            restarted=True,
+            daemon_reload=True,
+        )
+
+
+def _configure_opendkim(domain: str, dkim_selector: str) -> bool:
+    """Configures OpenDKIM"""
+    need_restart = False
+
+    main_config = files.template(
+        src=importlib.resources.files(__package__).joinpath("opendkim/opendkim.conf"),
+        dest="/etc/opendkim.conf",
+        user="root",
+        group="root",
+        mode="644",
+        config={"domain_name": domain, "opendkim_selector": dkim_selector},
+    )
+    need_restart |= main_config.changed
+
+    files.directory(
+        name="Add opendkim directory to /etc",
+        path="/etc/opendkim",
+        user="opendkim",
+        group="opendkim",
+        mode="750",
+        present=True,
+    )
+
+    keytable = files.template(
+        src=importlib.resources.files(__package__).joinpath("opendkim/KeyTable"),
+        dest="/etc/dkimkeys/KeyTable",
+        user="opendkim",
+        group="opendkim",
+        mode="644",
+        config={"domain_name": domain, "opendkim_selector": dkim_selector},
+    )
+    need_restart |= keytable.changed
+
+    signing_table = files.template(
+        src=importlib.resources.files(__package__).joinpath("opendkim/SigningTable"),
+        dest="/etc/dkimkeys/SigningTable",
+        user="opendkim",
+        group="opendkim",
+        mode="644",
+        config={"domain_name": domain, "opendkim_selector": dkim_selector},
+    )
+    need_restart |= signing_table.changed
+
+    files.directory(
+        name="Add opendkim socket directory to /var/spool/postfix",
+        path="/var/spool/postfix/opendkim",
+        user="opendkim",
+        group="opendkim",
+        mode="750",
+        present=True,
+    )
+
+    if not host.get_fact(File, f"/etc/dkimkeys/{dkim_selector}.private"):
+        server.shell(
+            name="Generate OpenDKIM domain keys",
+            commands=[
+                f"opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
+            ],
+            _sudo=True,
+            _sudo_user="opendkim",
+        )
+
+    return need_restart
+
+
+def _install_mta_sts_daemon() -> bool:
+    need_restart = False
+
+    config = files.put(
+        name="upload postfix-mta-sts-resolver config",
+        src=importlib.resources.files(__package__).joinpath(
+            "postfix/mta-sts-daemon.yml"
+        ),
+        dest="/etc/mta-sts-daemon.yml",
+        user="root",
+        group="root",
+        mode="644",
+    )
+    need_restart |= config.changed
+
+    server.shell(
+        name="install postfix-mta-sts-resolver with pip",
+        commands=[
+            "python3 -m virtualenv /usr/local/lib/postfix-mta-sts-resolver",
+            "/usr/local/lib/postfix-mta-sts-resolver/bin/pip install postfix-mta-sts-resolver",
+        ],
+    )
+
+    systemd_unit = files.put(
+        name="upload mta-sts-daemon systemd unit",
+        src=importlib.resources.files(__package__).joinpath(
+            "postfix/mta-sts-daemon.service"
+        ),
+        dest="/etc/systemd/system/mta-sts-daemon.service",
+        user="root",
+        group="root",
+        mode="644",
+    )
+    need_restart |= systemd_unit.changed
+
+    return need_restart
+
+
+def _configure_postfix(config: Config, debug: bool = False) -> bool:
+    """Configures Postfix SMTP server."""
+    need_restart = False
+
+    main_config = files.template(
+        src=importlib.resources.files(__package__).joinpath("postfix/main.cf.j2"),
+        dest="/etc/postfix/main.cf",
+        user="root",
+        group="root",
+        mode="644",
+        config=config,
+    )
+    need_restart |= main_config.changed
+
+    master_config = files.template(
+        src=importlib.resources.files(__package__).joinpath("postfix/master.cf.j2"),
+        dest="/etc/postfix/master.cf",
+        user="root",
+        group="root",
+        mode="644",
+        debug=debug,
+        config=config,
+    )
+    need_restart |= master_config.changed
+
+    return need_restart
+
+
+def _configure_dovecot(config: Config, debug: bool = False) -> bool:
+    """Configures Dovecot IMAP server."""
+    need_restart = False
+
+    main_config = files.template(
+        src=importlib.resources.files(__package__).joinpath("dovecot/dovecot.conf.j2"),
+        dest="/etc/dovecot/dovecot.conf",
+        user="root",
+        group="root",
+        mode="644",
+        config=config,
+        debug=debug,
+    )
+    need_restart |= main_config.changed
+    auth_config = files.put(
+        src=importlib.resources.files(__package__).joinpath("dovecot/auth.conf"),
+        dest="/etc/dovecot/auth.conf",
+        user="root",
+        group="root",
+        mode="644",
+    )
+    need_restart |= auth_config.changed
+
+    files.template(
+        src=importlib.resources.files(__package__).joinpath("dovecot/expunge.cron.j2"),
+        dest="/etc/cron.d/expunge",
+        user="root",
+        group="root",
+        mode="644",
+        config=config,
+    )
+
+    # as per https://doc.dovecot.org/configuration_manual/os/
+    # it is recommended to set the following inotify limits
+    for name in ("max_user_instances", "max_user_watches"):
+        key = f"fs.inotify.{name}"
+        server.sysctl(
+            name=f"Change {key}",
+            key=key,
+            value=65535,
+            persist=True,
+        )
+
+    return need_restart
+
+
+def _configure_nginx(domain: str, debug: bool = False) -> bool:
+    """Configures nginx HTTP server."""
+    need_restart = False
+
+    main_config = files.template(
+        src=importlib.resources.files(__package__).joinpath("nginx/nginx.conf.j2"),
+        dest="/etc/nginx/nginx.conf",
+        user="root",
+        group="root",
+        mode="644",
+        config={"domain_name": domain},
+    )
+    need_restart |= main_config.changed
+
+    autoconfig = files.template(
+        src=importlib.resources.files(__package__).joinpath("nginx/autoconfig.xml.j2"),
+        dest="/var/www/html/.well-known/autoconfig/mail/config-v1.1.xml",
+        user="root",
+        group="root",
+        mode="644",
+        config={"domain_name": domain},
+    )
+    need_restart |= autoconfig.changed
+
+    mta_sts_config = files.template(
+        src=importlib.resources.files(__package__).joinpath("nginx/mta-sts.txt.j2"),
+        dest="/var/www/html/.well-known/mta-sts.txt",
+        user="root",
+        group="root",
+        mode="644",
+        config={"domain_name": domain},
+    )
+    need_restart |= mta_sts_config.changed
+
+    # install CGI newemail script
+    #
+    cgi_dir = "/usr/lib/cgi-bin"
+    files.directory(
+        name=f"Ensure {cgi_dir} exists",
+        path=cgi_dir,
+        user="root",
+        group="root",
+    )
+
+    files.put(
+        name="Upload cgi newemail.py script",
+        src=importlib.resources.files("chatmaild").joinpath("newemail.py").open("rb"),
+        dest=f"{cgi_dir}/newemail.py",
+        user="root",
+        group="root",
+        mode="755",
+    )
+
+    return need_restart
+
+
+def check_config(config):
+    mail_domain = config.mail_domain
+    if mail_domain != "testrun.org" and not mail_domain.endswith(".testrun.org"):
+        blocked_words = "merlinux schmieder testrun.org".split()
+        for value in config.__dict__.values():
+            if any(x in value for x in blocked_words):
+                raise ValueError(
+                    f"please set your own privacy contacts/addresses in {config._inipath}"
+                )
+    return config
+
+
+def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> None:
+    """Deploy a chat-mail instance.
+
+    :param mail_domain: domain part of your future email addresses
+    :param mail_server: the DNS name under which your mail server is reachable
+    :param dkim_selector:
+    """
+    from .www import build_webpages
+
+    apt.update(name="apt update", cache_time=24 * 3600)
+    server.group(name="Create vmail group", group="vmail", system=True)
+    server.user(name="Create vmail user", user="vmail", group="vmail", system=True)
+
+    server.group(name="Create opendkim group", group="opendkim", system=True)
+    server.user(
+        name="Add postfix user to opendkim group for socket access",
+        user="postfix",
+        groups=["opendkim"],
+        system=True,
+    )
+
+    # Run local DNS resolver `unbound`.
+    # `resolvconf` takes care of setting up /etc/resolv.conf
+    # to use 127.0.0.1 as the resolver.
+    apt.packages(
+        name="Install unbound",
+        packages="unbound",
+    )
+    systemd.service(
+        name="Start and enable unbound",
+        service="unbound.service",
+        running=True,
+        enabled=True,
+    )
+
+    # Deploy acmetool to have TLS certificates.
+    deploy_acmetool(nginx_hook=True, domains=[mail_server, f"mta-sts.{mail_server}"])
+
+    apt.packages(
+        name="Install Postfix",
+        packages="postfix",
+    )
+
+    apt.packages(
+        name="Install Dovecot",
+        packages=["dovecot-imapd", "dovecot-lmtpd"],
+    )
+
+    apt.packages(
+        name="Install OpenDKIM",
+        packages=[
+            "opendkim",
+            "opendkim-tools",
+        ],
+    )
+
+    apt.packages(
+        name="Install nginx",
+        packages=["nginx"],
+    )
+
+    apt.packages(
+        name="Install fcgiwrap",
+        packages=["fcgiwrap"],
+    )
+
+    pkg_root = importlib.resources.files(__package__)
+    chatmail_ini = pkg_root.joinpath("../../../chatmail.ini").resolve()
+    config = read_config(chatmail_ini)
+    check_config(config)
+    www_path = pkg_root.joinpath("../../../www").resolve()
+
+    build_dir = www_path.joinpath("build")
+    src_dir = www_path.joinpath("src")
+    build_webpages(src_dir, build_dir, config)
+    files.rsync(f"{build_dir}/", "/var/www/html", flags=["-avz"])
+
+    _install_remote_venv_with_chatmaild(config)
+    debug = False
+    dovecot_need_restart = _configure_dovecot(config, debug=debug)
+    postfix_need_restart = _configure_postfix(config, debug=debug)
+    opendkim_need_restart = _configure_opendkim(mail_domain, dkim_selector)
+    mta_sts_need_restart = _install_mta_sts_daemon()
+    nginx_need_restart = _configure_nginx(mail_domain)
+
+    systemd.service(
+        name="Start and enable OpenDKIM",
+        service="opendkim.service",
+        running=True,
+        enabled=True,
+        restarted=opendkim_need_restart,
+    )
+
+    systemd.service(
+        name="Start and enable MTA-STS daemon",
+        service="mta-sts-daemon.service",
+        daemon_reload=True,
+        running=True,
+        enabled=True,
+        restarted=mta_sts_need_restart,
+    )
+
+    systemd.service(
+        name="Start and enable Postfix",
+        service="postfix.service",
+        running=True,
+        enabled=True,
+        restarted=postfix_need_restart,
+    )
+
+    systemd.service(
+        name="Start and enable Dovecot",
+        service="dovecot.service",
+        running=True,
+        enabled=True,
+        restarted=dovecot_need_restart,
+    )
+
+    systemd.service(
+        name="Start and enable nginx",
+        service="nginx.service",
+        running=True,
+        enabled=True,
+        restarted=nginx_need_restart,
+    )
+
+    # This file is used by auth proxy.
+    # https://wiki.debian.org/EtcMailName
+    server.shell(
+        name="Setup /etc/mailname",
+        commands=[f"echo {mail_domain} >/etc/mailname; chmod 644 /etc/mailname"],
+    )
+
+    journald_conf = files.put(
+        name="Configure journald",
+        src=importlib.resources.files(__package__).joinpath("journald.conf"),
+        dest="/etc/systemd/journald.conf",
+        user="root",
+        group="root",
+        mode="644",
+    )
+    systemd.service(
+        name="Start and enable journald",
+        service="systemd-journald.service",
+        running=True,
+        enabled=True,
+        restarted=journald_conf,
+    )

cmdeploy/src/cmdeploy/acmetool/__init__.py

@@ -1,6 +1,6 @@
 import importlib.resources
 
-from pyinfra.operations import apt, files, systemd, server
+from pyinfra.operations import apt, files, server
 
 
 def deploy_acmetool(nginx_hook=False, email="", domains=[]):
@@ -38,25 +38,15 @@ def deploy_acmetool(nginx_hook=False, email="", domains=[]):
         email=email,
     )
 
-    service_file = files.put(
-        src=importlib.resources.files(__package__)
-        .joinpath("acmetool-redirector.service")
-        .open("rb"),
-        dest="/etc/systemd/system/acmetool-redirector.service",
+    files.template(
+        src=importlib.resources.files(__package__).joinpath("target.yaml.j2"),
+        dest="/var/lib/acme/conf/target",
         user="root",
         group="root",
         mode="644",
     )
-    systemd.service(
-        name="Setup acmetool-redirector service",
-        service="acmetool-redirector.service",
-        running=True,
-        enabled=True,
-        restarted=service_file.changed,
-    )
 
-    for domain in domains:
-        server.shell(
-            name=f"Request certificate for {domain}",
-            commands=[f"acmetool want {domain}"],
-        )
+    server.shell(
+        name=f"Request certificate for: { ', '.join(domains) }",
+        commands=[f"acmetool want { ' '.join(domains)}"],
+    )

cmdeploy/src/cmdeploy/acmetool/target.yaml.j2

@@ -0,0 +1,7 @@
+request:
+  provider: https://acme-v02.api.letsencrypt.org/directory
+  key:
+    type: rsa
+  challenge:
+    webroot-paths:
+    - /var/www/html/.well-known/acme-challenge

cmdeploy/src/cmdeploy/chatmail.zone.f

@@ -0,0 +1,12 @@
+{chatmail_domain}.                   MX 10 {chatmail_domain}.
+_submission._tcp.{chatmail_domain}.  SRV 0 1 587 {chatmail_domain}.
+_submissions._tcp.{chatmail_domain}. SRV 0 1 465 {chatmail_domain}.
+_imap._tcp.{chatmail_domain}.        SRV 0 1 143 {chatmail_domain}.
+_imaps._tcp.{chatmail_domain}.       SRV 0 1 993 {chatmail_domain}.
+{chatmail_domain}.                IN CAA 128 issue "letsencrypt.org;accounturi={acme_account_url}"
+{chatmail_domain}.                   TXT "v=spf1 a:{chatmail_domain} -all"
+_dmarc.{chatmail_domain}.            TXT "v=DMARC1;p=reject;rua=mailto:{email};ruf=mailto:{email};fo=1;adkim=r;aspf=r"
+_mta-sts.{chatmail_domain}.       IN TXT "v=STSv1; id={sts_id}"
+mta-sts.{chatmail_domain}.        IN CNAME {chatmail_domain}.
+_smtp._tls.{chatmail_domain}.     IN TXT "v=TLSRPTv1;rua=mailto:{email}"
+{dkim_entry}

cmdeploy/src/cmdeploy/cmdeploy.py

@@ -0,0 +1,316 @@
+"""
+Provides the `cmdeploy` entry point function,
+along with command line option and subcommand parsing.
+"""
+import argparse
+import datetime
+import shutil
+import subprocess
+import importlib.resources
+import importlib.util
+import os
+import sys
+from pathlib import Path
+
+
+from termcolor import colored
+from chatmaild.config import read_config, write_initial_config
+
+
+#
+# cmdeploy sub commands and options
+#
+
+
+def init_cmd_options(parser):
+    parser.add_argument(
+        "chatmail_domain",
+        action="store",
+        help="fully qualified DNS domain name for your chatmail instance",
+    )
+
+
+def init_cmd(args, out):
+    """Initialize chatmail config file."""
+    if args.inipath.exists():
+        out.red(f"Path exists, not modifying: {args.inipath}")
+        raise SystemExit(1)
+    write_initial_config(args.inipath, args.chatmail_domain)
+    out.green(f"created config file for {args.chatmail_domain} in {args.inipath}")
+
+
+def run_cmd_options(parser):
+    parser.add_argument(
+        "--dry-run",
+        dest="dry_run",
+        action="store_true",
+        help="don't actually modify the server",
+    )
+
+
+def run_cmd(args, out):
+    """Deploy chatmail services on the remote server."""
+
+    env = os.environ.copy()
+    env["CHATMAIL_DOMAIN"] = args.config.mail_domain
+    deploy_path = "cmdeploy/src/cmdeploy/deploy.py"
+    pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
+    cmd = f"{pyinf} --ssh-user root {args.config.mail_domain} {deploy_path}"
+    out.check_call(cmd, env=env)
+
+
+def dns_cmd(args, out):
+    """Generate dns zone file."""
+    template = importlib.resources.files(__package__).joinpath("chatmail.zone.f")
+    ssh = f"ssh root@{args.config.mail_domain}"
+
+    def read_dkim_entries(entry):
+        lines = []
+        for line in entry.split("\n"):
+            if line.startswith(";") or not line.strip():
+                continue
+            line = line.replace("\t", " ")
+            lines.append(line)
+        return "\n".join(lines)
+
+    acme_account_url = out.shell_output(f"{ssh} -- acmetool account-url")
+    dkim_entry = read_dkim_entries(out.shell_output(f"{ssh} -- opendkim-genzone -F"))
+
+    out(
+        f"[writing {args.config.mail_domain} zone data (using space as separator) to stdout output]",
+        green=True,
+    )
+    print(
+        template.read_text()
+        .format(
+            acme_account_url=acme_account_url,
+            email=f"root@{args.config.mail_domain}",
+            sts_id=datetime.datetime.now().strftime("%Y%m%d%H%M"),
+            chatmail_domain=args.config.mail_domain,
+            dkim_entry=dkim_entry,
+        )
+        .strip()
+    )
+
+
+def status_cmd(args, out):
+    """Display status for online chatmail instance."""
+
+    ssh = f"ssh root@{args.config.mail_domain}"
+
+    out.green(f"chatmail domain: {args.config.mail_domain}")
+    if args.config.privacy_mail:
+        out.green("privacy settings: present")
+    else:
+        out.red("no privacy settings")
+
+    s1 = "systemctl --type=service --state=running"
+    for line in out.shell_output(f"{ssh} -- {s1}").split("\n"):
+        if line.startswith("  "):
+            print(line)
+
+
+def test_cmd_options(parser):
+    parser.add_argument(
+        "--slow",
+        dest="slow",
+        action="store_true",
+        help="also run slow tests",
+    )
+
+
+def test_cmd(args, out):
+    """Run local and online tests for chatmail deployment.
+
+    This will automatically pip-install 'deltachat' if it's not available.
+    """
+
+    x = importlib.util.find_spec("deltachat")
+    if x is None:
+        out.check_call(f"{sys.executable} -m pip install deltachat")
+
+    pytest_path = shutil.which("pytest")
+    pytest_args = [
+        pytest_path,
+        "cmdeploy/src/",
+        "-n4",
+        "-rs",
+        "-x",
+        "-vrx",
+        "--durations=5",
+    ]
+    if args.slow:
+        pytest_args.append("--slow")
+    ret = out.run_ret(pytest_args)
+    return ret
+
+
+def fmt_cmd_options(parser):
+    parser.add_argument(
+        "--verbose",
+        "-v",
+        dest="verbose",
+        action="store_true",
+        help="provide information on invocations",
+    )
+
+    parser.add_argument(
+        "--check",
+        "-c",
+        action="store_true",
+        help="only check but don't fix problems",
+    )
+
+
+def fmt_cmd(args, out):
+    """Run formattting fixes (ruff and black) on all chatmail source code."""
+
+    sources = [str(importlib.resources.files(x)) for x in ("chatmaild", "cmdeploy")]
+    black_args = [shutil.which("black")]
+    ruff_args = [shutil.which("ruff")]
+
+    if args.check:
+        black_args.append("--check")
+    else:
+        ruff_args.append("--fix")
+
+    if not args.verbose:
+        black_args.append("-q")
+        ruff_args.append("-q")
+
+    black_args.extend(sources)
+    ruff_args.extend(sources)
+
+    out.check_call(" ".join(black_args), quiet=not args.verbose)
+    out.check_call(" ".join(ruff_args), quiet=not args.verbose)
+    return 0
+
+
+def bench_cmd(args, out):
+    """Run benchmarks against an online chatmail instance."""
+    args = ["pytest", "--pyargs", "cmdeploy.tests.online.benchmark", "-vrx"]
+    cmdstring = " ".join(args)
+    out.green(f"[$ {cmdstring}]")
+    subprocess.check_call(args)
+
+
+def webdev_cmd(args, out):
+    """Run local web development loop for static web pages."""
+    from .www import main
+
+    main()
+
+
+#
+# Parsing command line options and starting commands
+#
+
+
+class Out:
+    """Convenience output printer providing coloring."""
+
+    def red(self, msg, file=sys.stderr):
+        print(colored(msg, "red"), file=file)
+
+    def green(self, msg, file=sys.stderr):
+        print(colored(msg, "green"), file=file)
+
+    def __call__(self, msg, red=False, green=False, file=sys.stdout):
+        color = "red" if red else ("green" if green else None)
+        print(colored(msg, color), file=file)
+
+    def shell_output(self, arg):
+        self(f"[$ {arg}]", file=sys.stderr)
+        return subprocess.check_output(arg, shell=True).decode()
+
+    def check_call(self, arg, env=None, quiet=False):
+        if not quiet:
+            self(f"[$ {arg}]", file=sys.stderr)
+        return subprocess.check_call(arg, shell=True, env=env)
+
+    def run_ret(self, args, env=None, quiet=False):
+        if not quiet:
+            cmdstring = " ".join(args)
+            self(f"[$ {cmdstring}]", file=sys.stderr)
+        proc = subprocess.run(args, env=env)
+        return proc.returncode
+
+
+def add_config_option(parser):
+    parser.add_argument(
+        "--config",
+        dest="inipath",
+        action="store",
+        default=Path("chatmail.ini"),
+        type=Path,
+        help="path to the chatmail.ini file",
+    )
+
+
+def add_subcommand(subparsers, func):
+    name = func.__name__
+    assert name.endswith("_cmd")
+    name = name[:-4]
+    doc = func.__doc__.strip()
+    help = doc.split("\n")[0].strip(".")
+    p = subparsers.add_parser(name, description=doc, help=help)
+    p.set_defaults(func=func)
+    add_config_option(p)
+    return p
+
+
+description = """
+Setup your chatmail server configuration and
+deploy it via SSH to your remote location.
+"""
+
+
+def get_parser():
+    """Return an ArgumentParser for the 'cmdeploy' CLI"""
+
+    parser = argparse.ArgumentParser(description=description.strip())
+    subparsers = parser.add_subparsers(title="subcommands")
+
+    # find all subcommands in the module namespace
+    glob = globals()
+    for name, func in glob.items():
+        if name.endswith("_cmd"):
+            subparser = add_subcommand(subparsers, func)
+            addopts = glob.get(name + "_options")
+            if addopts is not None:
+                addopts(subparser)
+
+    return parser
+
+
+def main(args=None):
+    """Provide main entry point for 'xdcget' CLI invocation."""
+    parser = get_parser()
+    args = parser.parse_args(args=args)
+    if not hasattr(args, "func"):
+        return parser.parse_args(["-h"])
+    out = Out()
+    kwargs = {}
+    if args.func.__name__ not in ("init_cmd", "fmt_cmd"):
+        if not args.inipath.exists():
+            out.red(f"expecting {args.inipath} to exist, run init first?")
+            raise SystemExit(1)
+        try:
+            args.config = read_config(args.inipath)
+        except Exception as ex:
+            out.red(ex)
+            raise SystemExit(1)
+
+    try:
+        res = args.func(args, out, **kwargs)
+        if res is None:
+            res = 0
+        return res
+
+    except KeyboardInterrupt:
+        out.red("KeyboardInterrupt")
+        sys.exit(130)
+
+
+if __name__ == "__main__":
+    main()

cmdeploy/src/cmdeploy/deploy.py

@@ -1,12 +1,12 @@
 import os
 import pyinfra
-from chatmail import deploy_chatmail
+from cmdeploy import deploy_chatmail
 
 
 def main():
     mail_domain = os.getenv("CHATMAIL_DOMAIN")
     mail_server = os.getenv("CHATMAIL_SERVER", mail_domain)
-    dkim_selector = os.getenv("CHATMAIL_DKIM_SELECTOR", "2023")
+    dkim_selector = os.getenv("CHATMAIL_DKIM_SELECTOR", "dkim")
 
     assert mail_domain
     assert mail_server

cmdeploy/src/cmdeploy/dovecot/auth.conf

@@ -0,0 +1,5 @@
+uri = proxy:/run/dovecot/doveauth.socket:auth
+iterate_disable = yes
+default_pass_scheme = plain
+password_key = passdb/%w/%u
+user_key = userdb/%u

cmdeploy/src/cmdeploy/dovecot/dovecot.conf.j2

@@ -0,0 +1,144 @@
+## Dovecot configuration file
+
+protocols = imap lmtp
+
+auth_mechanisms = plain
+
+{% if debug == true %}
+auth_verbose = yes
+auth_debug = yes
+auth_debug_passwords = yes
+auth_verbose_passwords = plain
+auth_cache_size = 100M
+mail_debug = yes
+{% endif %}
+
+
+mail_plugins = quota
+
+# these are the capabilities Delta Chat cares about actually 
+# so let's keep the network overhead per login small
+# https://github.com/deltachat/deltachat-core-rust/blob/master/src/imap/capabilities.rs
+imap_capability = IMAP4rev1 IDLE MOVE QUOTA CONDSTORE NOTIFY
+
+
+# Authentication for system users.
+passdb {
+  driver = dict
+  args = /etc/dovecot/auth.conf
+}
+userdb {
+  driver = dict
+  args = /etc/dovecot/auth.conf
+}
+##
+## Mailbox locations and namespaces
+##
+
+# Mailboxes are stored in the "mail" directory of the vmail user home.
+mail_location = maildir:/home/vmail/mail/%d/%u
+
+namespace inbox {
+  inbox = yes
+
+  mailbox Drafts {
+    special_use = \Drafts
+  }
+  mailbox Junk {
+    special_use = \Junk
+  }
+  mailbox Trash {
+    special_use = \Trash
+  }
+
+  # For \Sent mailboxes there are two widely used names. We'll mark both of
+  # them as \Sent. User typically deletes one of them if duplicates are created.
+  mailbox Sent {
+    special_use = \Sent
+  }
+  mailbox "Sent Messages" {
+    special_use = \Sent
+  }
+}
+
+mail_uid = vmail
+mail_gid = vmail
+mail_privileged_group = vmail
+
+##
+## Mail processes
+##
+
+# Enable IMAP COMPRESS (RFC 4978).
+# <https://datatracker.ietf.org/doc/html/rfc4978.html>
+protocol imap {
+  mail_plugins = $mail_plugins imap_zlib imap_quota
+}
+
+protocol lmtp {
+  mail_plugins = $mail_plugins quota
+}
+
+plugin {
+  imap_compress_deflate_level = 6
+}
+
+plugin {
+  # for now we define static quota-rules for all users 
+  quota = maildir:User quota
+  quota_rule = *:storage={{ config.max_mailbox_size }}
+  quota_max_mail_size=30M
+  quota_grace = 0
+  # quota_over_flag_value = TRUE
+}
+
+
+
+service lmtp {
+  user=vmail
+
+  unix_listener /var/spool/postfix/private/dovecot-lmtp {
+    group = postfix
+    mode = 0600
+    user = postfix
+  }
+}
+
+service auth {
+  unix_listener /var/spool/postfix/private/auth {
+    mode = 0660
+    user = postfix
+    group = postfix
+  }
+}
+
+service auth-worker {
+  # Default is root.
+  # Drop privileges we don't need.
+  user = vmail
+}
+
+service imap-login {
+    # High-security mode.
+    # Each process serves a single connection and exits afterwards.
+    # This is the default, but we set it explicitly to be sure.
+    # See <https://doc.dovecot.org/admin_manual/login_processes/#high-security-mode> for details.
+    service_count = 1
+
+    # Inrease the number of simultaneous connections.
+    #
+    # As of Dovecot 2.3.19.1 the default is 100 processes.
+    # Combined with `service_count = 1` it means only 100 connections
+    # can be handled simultaneously.
+    process_limit = 10000
+
+    # Avoid startup latency for new connections.
+    process_min_avail = 10
+}
+
+ssl = required
+ssl_cert = </var/lib/acme/live/{{ config.mail_domain }}/fullchain
+ssl_key = </var/lib/acme/live/{{ config.mail_domain }}/privkey
+ssl_dh = </usr/share/dovecot/dh.pem
+ssl_min_protocol = TLSv1.2
+ssl_prefer_server_ciphers = yes

cmdeploy/src/cmdeploy/dovecot/expunge.cron.j2

@@ -0,0 +1,4 @@
+2 0 * * * dovecot doveadm expunge -A SEEN BEFORE {{ config.delete_mails_after }} INBOX
+2 0 * * * dovecot doveadm expunge -A SEEN BEFORE {{ config.delete_mails_after }} Deltachat
+2 0 * * * dovecot doveadm expunge -A SEEN BEFORE {{ config.delete_mails_after }} Trash
+2 30 * * * dovecot doveadm purge -A

cmdeploy/src/cmdeploy/genqr.py

@@ -0,0 +1,87 @@
+import importlib
+import qrcode
+import os
+from PIL import ImageFont, ImageDraw, Image
+import io
+
+
+def gen_qr_png_data(maildomain):
+    url = f"DCACCOUNT:https://{maildomain}/cgi-bin/newemail.py"
+    image = gen_qr(maildomain, url)
+    temp = io.BytesIO()
+    image.save(temp, format="png")
+    temp.seek(0)
+    return temp
+
+
+def gen_qr(maildomain, url):
+    # taken and modified from
+    # https://github.com/deltachat/mailadm/blob/master/src/mailadm/gen_qr.py
+
+    # info = f"{maildomain} invite code"
+    info = ""
+
+    # load QR code
+    qr = qrcode.QRCode(
+        version=1,
+        error_correction=qrcode.constants.ERROR_CORRECT_H,
+        box_size=1,
+        border=1,
+    )
+    qr.add_data(url)
+    qr.make(fit=True)
+    qr_img = qr.make_image(fill_color="black", back_color="white")
+
+    # paint all elements
+    ttf_path = str(
+        importlib.resources.files(__package__).joinpath("data/opensans-regular.ttf")
+    )
+    logo_red_path = str(
+        importlib.resources.files(__package__).joinpath("data/delta-chat-bw.png")
+    )
+
+    assert os.path.exists(ttf_path), ttf_path
+    font_size = 16
+    font = ImageFont.truetype(font=ttf_path, size=font_size)
+
+    num_lines = ((info).count("\n") + 1) if info else 0
+
+    size = width = 384
+    qr_padding = 6
+    text_height = font_size * num_lines
+    height = size + text_height
+
+    image = Image.new("RGBA", (width, height), "white")
+    qr_final_size = width - (qr_padding * 2)
+
+    if num_lines:
+        draw = ImageDraw.Draw(image)
+
+        # draw text
+        if hasattr(font, "getsize"):
+            info_pos = (width - font.getsize(info.strip())[0]) // 2
+        else:
+            info_pos = (width - font.getbbox(info.strip())[3]) // 2
+
+        draw.multiline_text(
+            (info_pos, size - qr_padding // 2),
+            info,
+            font=font,
+            fill="black",
+            align="right",
+        )
+
+    # paste QR code
+    image.paste(
+        qr_img.resize((qr_final_size, qr_final_size), resample=Image.NEAREST),
+        (qr_padding, qr_padding),
+    )
+
+    # background delta logo
+    logo2_img = Image.open(logo_red_path)
+    logo2_width = int(size / 6)
+    logo2 = logo2_img.resize((logo2_width, logo2_width), resample=Image.NEAREST)
+    pos = int((size / 2) - (logo2_width / 2))
+    image.paste(logo2, (pos, pos), mask=logo2)
+
+    return image

cmdeploy/src/cmdeploy/journald.conf

@@ -0,0 +1,2 @@
+[Journal]
+MaxRetentionSec=3d

cmdeploy/src/cmdeploy/nginx/autoconfig.xml.j2

@@ -0,0 +1,37 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<clientConfig version="1.1">
+  <emailProvider id="{{ config.domain_name }}">
+    <domain>{{ config.domain_name }}</domain>
+    <displayName>{{ config.domain_name }} chatmail</displayName>
+    <displayShortName>{{ config.domain_name }}</displayShortName>
+    <incomingServer type="imap">
+      <hostname>{{ config.domain_name }}</hostname>
+      <port>993</port>
+      <socketType>SSL</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+    </incomingServer>
+    <incomingServer type="imap">
+      <hostname>{{ config.domain_name }}</hostname>
+      <port>143</port>
+      <socketType>STARTTLS</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+    </incomingServer>
+    <outgoingServer type="smtp">
+      <hostname>{{ config.domain_name }}</hostname>
+      <port>465</port>
+      <socketType>SSL</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+    </outgoingServer>
+    <outgoingServer type="smtp">
+      <hostname>{{ config.domain_name }}</hostname>
+      <port>587</port>
+      <socketType>STARTTLS</socketType>
+      <authentication>password-cleartext</authentication>
+      <username>%EMAILADDRESS%</username>
+    </outgoingServer>
+  </emailProvider>
+</clientConfig>

cmdeploy/src/cmdeploy/nginx/mta-sts.txt.j2

@@ -0,0 +1,4 @@
+version: STSv1
+mode: enforce
+mx: {{ config.domain_name }}
+max_age: 2419200

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -0,0 +1,55 @@
+user www-data;
+worker_processes auto;
+pid /run/nginx.pid;
+error_log /var/log/nginx/error.log;
+
+events {
+	worker_connections 768;
+	# multi_accept on;
+}
+
+http {
+	sendfile on;
+	tcp_nopush on;
+
+	# Do not emit nginx version on error pages.
+	server_tokens off;
+
+	include /etc/nginx/mime.types;
+	default_type application/octet-stream;
+
+	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
+	ssl_prefer_server_ciphers on;
+	ssl_certificate /var/lib/acme/live/{{ config.domain_name }}/fullchain;
+	ssl_certificate_key /var/lib/acme/live/{{ config.domain_name }}/privkey;
+
+	gzip on;
+
+	server {
+		listen 443 ssl default_server;
+		listen [::]:443 ssl default_server;
+
+		root /var/www/html;
+
+		index index.html index.htm;
+
+		server_name _;
+
+		location / {
+			# First attempt to serve request as file, then
+			# as directory, then fall back to displaying a 404.
+			try_files $uri $uri/ =404;
+		}
+
+        # add cgi-bin support
+        include /usr/share/doc/fcgiwrap/examples/nginx.conf;
+	}
+    server {
+        listen 80 default_server;
+        listen [::]:80 default_server;
+        server_name _;
+
+        return 301 https://$host$request_uri;
+    }
+}
+

cmdeploy/src/cmdeploy/opendkim/KeyTable

@@ -0,0 +1 @@
+dkim._domainkey.{{ config.domain_name }} {{ config.domain_name }}:{{ config.opendkim_selector }}:/etc/dkimkeys/dkim.private

cmdeploy/src/cmdeploy/opendkim/SigningTable

@@ -0,0 +1 @@
+*@{{ config.domain_name }} {{ config.opendkim_selector }}._domainkey.{{ config.domain_name }}

cmdeploy/src/cmdeploy/opendkim/opendkim.conf

@@ -1,7 +1,4 @@
-# This is a basic configuration for signing and verifying. It can easily be
-# adapted to suit a basic installation. See opendkim.conf(5) and
-# /usr/share/doc/opendkim/examples/opendkim.conf.sample for complete
-# documentation of available configuration parameters.
+# OpenDKIM configuration.
 
 Syslog			yes
 SyslogSuccess		yes
@@ -21,7 +18,9 @@ OversignHeaders		From
 # setup options can be found in /usr/share/doc/opendkim/README.opendkim.
 Domain			{{ config.domain_name }}
 Selector		{{ config.opendkim_selector }}
-KeyFile		/etc/dkimkeys/{{ config.opendkim_selector }}.private
+KeyFile		  /etc/dkimkeys/{{ config.opendkim_selector }}.private
+KeyTable          /etc/dkimkeys/KeyTable
+SigningTable      refile:/etc/dkimkeys/SigningTable
 
 # In Debian, opendkim runs as user "opendkim". A umask of 007 is required when
 # using a local socket with MTAs that access the socket as a non-privileged

cmdeploy/src/cmdeploy/postfix/main.cf.j2

@@ -1,4 +1,4 @@
-myorigin = {{ config.domain_name }}
+myorigin = {{ config.mail_domain }}
 
 smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
 biff = no
@@ -16,16 +16,17 @@ readme_directory = no
 compatibility_level = 2
 
 # TLS parameters
-smtpd_tls_cert_file=/var/lib/acme/live/{{ config.domain_name }}/fullchain
-smtpd_tls_key_file=/var/lib/acme/live/{{ config.domain_name }}/privkey
+smtpd_tls_cert_file=/var/lib/acme/live/{{ config.mail_domain }}/fullchain
+smtpd_tls_key_file=/var/lib/acme/live/{{ config.mail_domain }}/privkey
 smtpd_tls_security_level=may
 
 smtp_tls_CApath=/etc/ssl/certs
 smtp_tls_security_level=may
 smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
+smtp_tls_policy_maps = socketmap:inet:127.0.0.1:8461:postfix
 
 smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
-myhostname = {{ config.domain_name }}
+myhostname = {{ config.mail_domain }}
 alias_maps = hash:/etc/aliases
 alias_database = hash:/etc/aliases
 
@@ -37,12 +38,14 @@ mydestination =
 relayhost = 
 mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
 mailbox_size_limit = 0
+# maximum 30MB sized messages
+message_size_limit = 31457280
 recipient_delimiter = +
 inet_interfaces = all
 inet_protocols = all
 
 virtual_transport = lmtp:unix:private/dovecot-lmtp
-virtual_mailbox_domains = {{ config.domain_name }}
+virtual_mailbox_domains = {{ config.mail_domain }}
 
 smtpd_milters = unix:opendkim/opendkim.sock
 non_smtpd_milters = $smtpd_milters

cmdeploy/src/cmdeploy/postfix/master.cf.j2

@@ -9,7 +9,11 @@
 # service type  private unpriv  chroot  wakeup  maxproc command + args
 #               (yes)   (yes)   (no)    (never) (100)
 # ==========================================================================
-smtp      inet  n       -       y       -       -       smtpd
+{% if debug == true %}
+smtp      inet  n       -       y       -       -       smtpd -v
+{% else %}
+smtp      inet  n       -       y       -       -       smtpd 
+{% endif %}
 #smtp      inet  n       -       y       -       1       postscreen
 #smtpd     pass  -       -       y       -       -       smtpd
 #dnsblog   unix  -       -       y       -       0       dnsblog
@@ -28,6 +32,8 @@ submission inet n       -       y       -       -       smtpd
   -o smtpd_recipient_restrictions=
   -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
   -o milter_macro_daemon_name=ORIGINATING
+  -o smtpd_client_connection_count_limit=1000
+  -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
 smtps     inet  n       -       y       -       -       smtpd
   -o syslog_name=postfix/smtps
   -o smtpd_tls_wrappermode=yes
@@ -41,7 +47,9 @@ smtps     inet  n       -       y       -       -       smtpd
   -o smtpd_sender_restrictions=$mua_sender_restrictions
   -o smtpd_recipient_restrictions=
   -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
+  -o smtpd_client_connection_count_limit=1000
   -o milter_macro_daemon_name=ORIGINATING
+  -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
 #628       inet  n       -       y       -       -       qmqpd
 pickup    unix  n       -       y       60      1       pickup
 cleanup   unix  n       -       y       -       0       cleanup
@@ -64,9 +72,11 @@ showq     unix  n       -       y       -       -       showq
 error     unix  -       -       y       -       -       error
 retry     unix  -       -       y       -       -       error
 discard   unix  -       -       y       -       -       discard
-local     unix  -       n       n       -       -       local
-virtual   unix  -       n       n       -       -       virtual
 lmtp      unix  -       -       y       -       -       lmtp
 anvil     unix  -       -       y       -       1       anvil
 scache    unix  -       -       y       -       1       scache
 postlog   unix-dgram n  -       n       -       1       postlogd
+filter    unix -        n       n       -       -       lmtp
+# Local SMTP server for reinjecting filered mail.
+localhost:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
+  -o syslog_name=postfix/reinject

cmdeploy/src/cmdeploy/postfix/mta-sts-daemon.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=Postfix MTA-STS resolver daemon
+
+[Service]
+ExecStart=/usr/local/lib/postfix-mta-sts-resolver/bin/mta-sts-daemon
+Restart=always
+RestartSec=30
+
+[Install]
+WantedBy=multi-user.target

cmdeploy/src/cmdeploy/postfix/mta-sts-daemon.yml

@@ -0,0 +1,13 @@
+host: 127.0.0.1
+port: 8461
+reuse_port: true
+shutdown_timeout: 20
+cache:
+  type: internal
+  options:
+    cache_size: 10000
+proactive_policy_fetching:
+  enabled: true
+default_zone:
+  strict_testing: false
+  timeout: 4

cmdeploy/src/cmdeploy/tests/online/benchmark.py

@@ -0,0 +1,60 @@
+def test_tls_imap(benchmark, imap):
+    def imap_connect():
+        imap.connect()
+
+    benchmark(imap_connect, 10)
+
+
+def test_login_imap(benchmark, imap, gencreds):
+    def imap_connect_and_login():
+        imap.connect()
+        imap.login(*gencreds())
+
+    benchmark(imap_connect_and_login, 10)
+
+
+def test_tls_smtp(benchmark, smtp):
+    def smtp_connect():
+        smtp.connect()
+
+    benchmark(smtp_connect, 10)
+
+
+def test_login_smtp(benchmark, smtp, gencreds):
+    def smtp_connect_and_login():
+        smtp.connect()
+        smtp.login(*gencreds())
+
+    benchmark(smtp_connect_and_login, 10)
+
+
+class TestDC:
+    def test_autoconfigure(self, benchmark, cmfactory):
+        def dc_autoconfig_and_idle_ready():
+            cmfactory.get_online_accounts(1)
+
+        benchmark(dc_autoconfig_and_idle_ready, 5)
+
+    def test_ping_pong(self, benchmark, cmfactory):
+        ac1, ac2 = cmfactory.get_online_accounts(2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
+
+        def dc_ping_pong():
+            chat.send_text("ping")
+            msg = ac2.wait_next_incoming_message()
+            msg.chat.send_text("pong")
+            ac1.wait_next_incoming_message()
+
+        benchmark(dc_ping_pong, 5)
+
+    def test_send_10_receive_10(self, benchmark, cmfactory, lp):
+        ac1, ac2 = cmfactory.get_online_accounts(2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
+
+        def dc_send_10_receive_10():
+            for i in range(10):
+                chat.send_text(f"hello {i}")
+            for i in range(10):
+                ac2.wait_next_incoming_message()
+
+        benchmark(dc_send_10_receive_10, 5)

cmdeploy/src/cmdeploy/tests/online/test_0_login.py

@@ -0,0 +1,70 @@
+import pytest
+import threading
+import queue
+
+
+def test_login_basic_functioning(imap_or_smtp, gencreds, lp):
+    """Test a) that an initial login creates a user automatically
+    and b) verify we can also login a second time with the same password
+    and c) that using a different password fails the login."""
+    user, password = gencreds()
+    lp.sec(f"login first time with {user} {password}")
+    imap_or_smtp.connect()
+    imap_or_smtp.login(user, password)
+    lp.indent("success")
+
+    lp.sec(f"reconnect and login second time {user} {password}")
+    imap_or_smtp.connect()
+    imap_or_smtp.login(user, password)
+    imap_or_smtp.connect()
+    lp.sec("success")
+
+    lp.sec(f"reconnect and verify wrong password fails {user} ")
+    imap_or_smtp.connect()
+    with pytest.raises(imap_or_smtp.AuthError):
+        imap_or_smtp.login(user, password + "wrong")
+
+    lp.sec("creating users with a short password is not allowed")
+    user, _password = gencreds()
+    with pytest.raises(imap_or_smtp.AuthError):
+        imap_or_smtp.login(user, "admin")
+
+
+def test_login_same_password(imap_or_smtp, gencreds):
+    """Test two different users logging in with the same password
+    to ensure that authentication process does not confuse the users
+    by using only the password hash as a key.
+    """
+    user1, password1 = gencreds()
+    user2, _ = gencreds()
+    imap_or_smtp.connect()
+    imap_or_smtp.login(user1, password1)
+    imap_or_smtp.connect()
+    imap_or_smtp.login(user2, password1)
+
+
+def test_concurrent_logins_same_account(
+    make_imap_connection, make_smtp_connection, gencreds
+):
+    """Test concurrent smtp and imap logins
+    and check remote server succeeds on each connection.
+    """
+    user1, password1 = gencreds()
+    login_results = queue.Queue()
+
+    def login_smtp_imap(smtp, imap):
+        try:
+            imap.login(user1, password1)
+        except Exception:
+            login_results.put(False)
+        else:
+            login_results.put(True)
+
+    conns = [(make_smtp_connection(), make_imap_connection()) for i in range(10)]
+
+    for args in conns:
+        thread = threading.Thread(target=login_smtp_imap, args=args, daemon=True)
+        thread.start()
+
+    for _ in conns:
+        assert login_results.get()

cmdeploy/src/cmdeploy/tests/online/test_0_qr.py

@@ -0,0 +1,16 @@
+import requests
+
+from cmdeploy.genqr import gen_qr_png_data
+
+
+def test_gen_qr_png_data(maildomain):
+    data = gen_qr_png_data(maildomain)
+    assert data
+
+
+def test_fastcgi_working(maildomain, chatmail_config):
+    url = f"https://{maildomain}/cgi-bin/newemail.py"
+    print(url)
+    res = requests.post(url)
+    assert maildomain in res.json().get("email")
+    assert len(res.json().get("password")) > chatmail_config.password_min_length

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -0,0 +1,63 @@
+import smtplib
+import pytest
+
+
+def test_remote(remote, imap_or_smtp):
+    lineproducer = remote.iter_output(imap_or_smtp.logcmd)
+    imap_or_smtp.connect()
+    assert imap_or_smtp.name in next(lineproducer)
+
+
+def test_use_two_chatmailservers(cmfactory, maildomain2):
+    ac1 = cmfactory.new_online_configuring_account(cache=False)
+    cmfactory.switch_maildomain(maildomain2)
+    ac2 = cmfactory.new_online_configuring_account(cache=False)
+    cmfactory.bring_accounts_online()
+    cmfactory.get_accepted_chat(ac1, ac2)
+    domain1 = ac1.get_config("addr").split("@")[1]
+    domain2 = ac2.get_config("addr").split("@")[1]
+    assert domain1 != domain2
+
+
+@pytest.mark.parametrize("forgeaddr", ["internal", "someone@example.org"])
+def test_reject_forged_from(cmsetup, maildata, gencreds, lp, forgeaddr):
+    user1, user3 = cmsetup.gen_users(2)
+
+    lp.sec("send encrypted message with forged from")
+    print("envelope_from", user1.addr)
+    if forgeaddr == "internal":
+        addr_to_forge = cmsetup.gen_users(1)[0].addr
+    else:
+        addr_to_forge = "someone@example.org"
+
+    print("message to inject:")
+    msg = maildata("encrypted.eml", from_addr=addr_to_forge, to_addr=user3.addr)
+    msg = msg.as_string()
+    for line in msg.split("\n")[:4]:
+        print(f"   {line}")
+
+    lp.sec("Send forged mail and check remote postfix lmtp processing result")
+    with pytest.raises(smtplib.SMTPException) as e:
+        user1.smtp.sendmail(from_addr=user1.addr, to_addrs=[user3.addr], msg=msg)
+    assert "500" in str(e.value)
+
+
+@pytest.mark.slow
+def test_exceed_rate_limit(cmsetup, gencreds, maildata, chatmail_config):
+    """Test that the per-account send-mail limit is exceeded."""
+    user1, user2 = cmsetup.gen_users(2)
+    mail = maildata(
+        "encrypted.eml", from_addr=user1.addr, to_addr=user2.addr
+    ).as_string()
+    for i in range(chatmail_config.max_user_send_per_minute + 5):
+        print("Sending mail", str(i))
+        try:
+            user1.smtp.sendmail(user1.addr, [user2.addr], mail)
+        except smtplib.SMTPException as e:
+            if i < chatmail_config.max_user_send_per_minute:
+                pytest.fail(f"rate limit was exceeded too early with msg {i}")
+            outcome = e.recipients[user2.addr]
+            assert outcome[0] == 450
+            assert b"4.7.1: Too much mail from" in outcome[1]
+            return
+    pytest.fail("Rate limit was not exceeded")

cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py

@@ -0,0 +1,121 @@
+import time
+import re
+import random
+import pytest
+
+
+class TestEndToEndDeltaChat:
+    "Tests that use Delta Chat accounts on the chat mail instance."
+
+    def test_one_on_one(self, cmfactory, lp):
+        """Test that a DC account can send a message to a second DC account
+        on the same chat-mail instance."""
+        ac1, ac2 = cmfactory.get_online_accounts(2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
+
+        lp.sec("ac1: prepare and send text message to ac2")
+        chat.send_text("message0")
+
+        lp.sec("wait for ac2 to receive message")
+        msg2 = ac2._evtracker.wait_next_incoming_message()
+        assert msg2.text == "message0"
+
+    @pytest.mark.slow
+    def test_exceed_quota(self, cmfactory, lp, tmpdir, remote, chatmail_config):
+        """This is a very slow test as it needs to upload >100MB of mail data
+        before quota is exceeded, and thus depends on the speed of the upload.
+        """
+        ac1, ac2 = cmfactory.get_online_accounts(2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
+
+        def parse_size_limit(limit: str) -> int:
+            """Parse a size limit and return the number of bytes as integer.
+
+            Example input: 100M, 2.4T, 500 K
+            """
+            units = {"B": 1, "K": 2**10, "M": 2**20, "G": 2**30, "T": 2**40}
+            size = re.sub(r"([KMGT])", r" \1", limit.upper())
+            number, unit = [string.strip() for string in size.split()]
+            return int(float(number) * units[unit])
+
+        quota = parse_size_limit(chatmail_config.max_mailbox_size)
+        attachsize = 1 * 1024 * 1024
+        num_to_send = quota // attachsize + 2
+        lp.sec(f"ac1: send {num_to_send} large files to ac2")
+        lp.indent(f"per-user quota is assumed to be: {quota/(1024*1024)}MB")
+        alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
+        msgs = []
+        for i in range(num_to_send):
+            attachment = tmpdir / f"attachment{i}"
+            data = "".join(random.choice(alphanumeric) for i in range(1024))
+            with open(attachment, "w+") as f:
+                for j in range(attachsize // len(data)):
+                    f.write(data)
+
+            msg = chat.send_file(str(attachment))
+            msgs.append(msg)
+            lp.indent(f"Sent out msg {i}, size {attachsize/(1024*1024)}MB")
+
+        lp.sec("ac2: check messages are arriving until quota is reached")
+
+        addr = ac2.get_config("addr").lower()
+        saved_ok = 0
+        for line in remote.iter_output("journalctl -f -u dovecot"):
+            if addr not in line:
+                # print(line)
+                continue
+            if "quota" in line:
+                if "quota exceeded" in line:
+                    if saved_ok < num_to_send // 2:
+                        pytest.fail(
+                            f"quota exceeded too early: after {saved_ok} messages already"
+                        )
+                    lp.indent("good, message sending failed because quota was exceeded")
+                    return
+            if "saved mail to inbox" in line:
+                saved_ok += 1
+                print(f"{saved_ok}: {line}")
+                if saved_ok >= num_to_send:
+                    break
+
+        pytest.fail("sending succeeded although messages should exceed quota")
+
+    def test_securejoin(self, cmfactory, lp, maildomain2):
+        ac1 = cmfactory.new_online_configuring_account(cache=False)
+        cmfactory.switch_maildomain(maildomain2)
+        ac2 = cmfactory.new_online_configuring_account(cache=False)
+        cmfactory.bring_accounts_online()
+
+        lp.sec("ac1: create QR code and let ac2 scan it, starting the securejoin")
+        qr = ac1.get_setup_contact_qr()
+
+        lp.sec("ac2: start QR-code based setup contact protocol")
+        ch = ac2.qr_setup_contact(qr)
+        assert ch.id >= 10
+        ac1._evtracker.wait_securejoin_inviter_progress(1000)
+
+    def test_read_receipts_between_instances(self, cmfactory, lp, maildomain2):
+        ac1 = cmfactory.new_online_configuring_account(cache=False)
+        cmfactory.switch_maildomain(maildomain2)
+        ac2 = cmfactory.new_online_configuring_account(cache=False)
+        cmfactory.bring_accounts_online()
+
+        lp.sec("setup encrypted comms between ac1 and ac2 on different instances")
+        qr = ac1.get_setup_contact_qr()
+        ch = ac2.qr_setup_contact(qr)
+        assert ch.id >= 10
+        ac1._evtracker.wait_securejoin_inviter_progress(1000)
+
+        lp.sec("ac1 sends a message and ac2 marks it as seen")
+        chat = ac1.create_chat(ac2)
+        msg = chat.send_text("hi")
+        m = ac2.wait_next_incoming_message()
+        m.mark_seen()
+        # we can only indirectly wait for mark-seen to cause an smtp-error
+        lp.sec("try to wait for markseen to complete and check error states")
+        deadline = time.time() + 3.1
+        while time.time() < deadline:
+            msgs = m.chat.get_messages()
+            for msg in msgs:
+                assert "error" not in m.get_message_info()
+            time.sleep(1)

cmdeploy/src/cmdeploy/tests/plugin.py

@@ -0,0 +1,405 @@
+import os
+import io
+import time
+import random
+import subprocess
+import imaplib
+import smtplib
+import itertools
+from pathlib import Path
+import pytest
+
+from chatmaild.database import Database
+from chatmaild.config import read_config
+
+
+conftestdir = Path(__file__).parent
+
+
+def pytest_addoption(parser):
+    parser.addoption(
+        "--slow", action="store_true", default=False, help="also run slow tests"
+    )
+
+
+def pytest_configure(config):
+    config._benchresults = {}
+    config.addinivalue_line(
+        "markers", "slow: mark test to require --slow option to run"
+    )
+
+
+def pytest_runtest_setup(item):
+    markers = list(item.iter_markers(name="slow"))
+    if markers:
+        if not item.config.getoption("--slow"):
+            pytest.skip("skipping slow test, use --slow to run")
+
+
+@pytest.fixture
+def chatmail_config(pytestconfig):
+    current = basedir = Path().resolve()
+    while 1:
+        path = current.joinpath("chatmail.ini").resolve()
+        if path.exists():
+            return read_config(path)
+        if current == current.parent:
+            break
+        current = current.parent
+
+    pytest.skip(f"no chatmail.ini file found in {basedir} or parent dirs")
+
+
+@pytest.fixture
+def maildomain(chatmail_config):
+    return chatmail_config.mail_domain
+
+
+@pytest.fixture
+def sshdomain(maildomain):
+    return os.environ.get("CHATMAIL_SSH", maildomain)
+
+
+@pytest.fixture
+def maildomain2():
+    domain = os.environ.get("CHATMAIL_DOMAIN2")
+    if not domain:
+        pytest.skip("set CHATMAIL_DOMAIN2 to a ssh-reachable chatmail instance")
+    return domain
+
+
+@pytest.fixture
+def sshdomain2(maildomain2):
+    return os.environ.get("CHATMAIL_SSH2", maildomain2)
+
+
+def pytest_report_header():
+    domain = os.environ.get("CHATMAIL_DOMAIN")
+    if domain:
+        text = f"chatmail test instance: {domain}"
+        return ["-" * len(text), text, "-" * len(text)]
+
+
+@pytest.fixture
+def benchmark(request):
+    def bench(func, num, name=None, reportfunc=None):
+        if name is None:
+            name = func.__name__
+        durations = []
+        for i in range(num):
+            now = time.time()
+            func()
+            durations.append(time.time() - now)
+        durations.sort()
+        request.config._benchresults[name] = (reportfunc, durations)
+
+    return bench
+
+
+def pytest_terminal_summary(terminalreporter):
+    tr = terminalreporter
+    results = tr.config._benchresults
+    if not results:
+        return
+
+    tr.section("benchmark results")
+    float_names = "median min max".split()
+    width = max(map(len, float_names))
+
+    def fcol(parts):
+        return " ".join(part.rjust(width) for part in parts)
+
+    headers = f"{'benchmark name': <30} " + fcol(float_names)
+    tr.write_line(headers)
+    tr.write_line("-" * len(headers))
+    summary_lines = []
+
+    for name, (reportfunc, durations) in results.items():
+        measures = [
+            sorted(durations)[len(durations) // 2],
+            min(durations),
+            max(durations),
+        ]
+        line = f"{name: <30} "
+        line += fcol(f"{float: 2.2f}" for float in measures)
+        tr.write_line(line)
+        vmedian, vmin, vmax = measures
+        if reportfunc:
+            for line in reportfunc(vmin=vmin, vmedian=vmedian, vmax=vmax):
+                summary_lines.append(line)
+
+    if summary_lines:
+        tr.write_line("")
+        tr.section("benchmark summary measures")
+        for line in summary_lines:
+            tr.write_line(line)
+
+
+@pytest.fixture
+def imap(maildomain):
+    return ImapConn(maildomain)
+
+
+@pytest.fixture
+def make_imap_connection(maildomain):
+    def make_imap_connection():
+        conn = ImapConn(maildomain)
+        conn.connect()
+        return conn
+
+    return make_imap_connection
+
+
+class ImapConn:
+    AuthError = imaplib.IMAP4.error
+    logcmd = "journalctl -f -u dovecot"
+    name = "dovecot"
+
+    def __init__(self, host):
+        self.host = host
+
+    def connect(self):
+        print(f"imap-connect {self.host}")
+        self.conn = imaplib.IMAP4_SSL(self.host)
+
+    def login(self, user, password):
+        print(f"imap-login {user!r} {password!r}")
+        self.conn.login(user, password)
+
+    def fetch_all(self):
+        print("imap-fetch all")
+        status, res = self.conn.select()
+        if int(res[0]) == 0:
+            raise ValueError("no messages in imap folder")
+        status, results = self.conn.fetch("1:*", "(RFC822)")
+        assert status == "OK"
+        return results
+
+    def fetch_all_messages(self):
+        print("imap-fetch all messages")
+        results = self.fetch_all()
+        messages = []
+        for item in results:
+            if len(item) == 2:
+                messages.append(item[1].decode())
+        return messages
+
+
+@pytest.fixture
+def smtp(maildomain):
+    return SmtpConn(maildomain)
+
+
+@pytest.fixture
+def make_smtp_connection(maildomain):
+    def make_smtp_connection():
+        conn = SmtpConn(maildomain)
+        conn.connect()
+        return conn
+
+    return make_smtp_connection
+
+
+class SmtpConn:
+    AuthError = smtplib.SMTPAuthenticationError
+    logcmd = "journalctl -f -t postfix/smtpd -t postfix/smtp -t postfix/lmtp"
+    name = "postfix"
+
+    def __init__(self, host):
+        self.host = host
+
+    def connect(self):
+        print(f"smtp-connect {self.host}")
+        self.conn = smtplib.SMTP_SSL(self.host)
+
+    def login(self, user, password):
+        print(f"smtp-login {user!r} {password!r}")
+        self.conn.login(user, password)
+
+    def sendmail(self, from_addr, to_addrs, msg):
+        print(f"smtp-sendmail from={from_addr!r} to_addrs={to_addrs!r}")
+        print(f"smtp-sendmail message size: {len(msg)}")
+        return self.conn.sendmail(from_addr=from_addr, to_addrs=to_addrs, msg=msg)
+
+
+@pytest.fixture(params=["imap", "smtp"])
+def imap_or_smtp(request):
+    return request.getfixturevalue(request.param)
+
+
+@pytest.fixture
+def gencreds(chatmail_config):
+    count = itertools.count()
+    next(count)
+
+    def gen(domain=None):
+        domain = domain if domain else chatmail_config.mail_domain
+        while 1:
+            num = next(count)
+            alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
+            user = "".join(
+                random.choices(alphanumeric, k=chatmail_config.username_max_length)
+            )
+            if domain == "nine.testrun.org":
+                user = f"ac{num}_{user}"[:9]
+            else:
+                user = f"ac{num}_{user}"[: chatmail_config.username_max_length]
+            password = "".join(
+                random.choices(alphanumeric, k=chatmail_config.password_min_length)
+            )
+            yield f"{user}@{domain}", f"{password}"
+
+    return lambda domain=None: next(gen(domain))
+
+
+@pytest.fixture()
+def db(tmpdir):
+    db_path = tmpdir / "passdb.sqlite"
+    print("database path:", db_path)
+    return Database(db_path)
+
+
+#
+# Delta Chat testplugin re-use
+# use the cmfactory fixture to get chatmail instance accounts
+#
+
+
+class ChatmailTestProcess:
+    """Provider for chatmail instance accounts as used by deltachat.testplugin.acfactory"""
+
+    def __init__(self, pytestconfig, maildomain, gencreds):
+        self.pytestconfig = pytestconfig
+        self.maildomain = maildomain
+        assert "." in self.maildomain, maildomain
+        self.gencreds = gencreds
+        self._addr2files = {}
+
+    def get_liveconfig_producer(self):
+        while 1:
+            user, password = self.gencreds(self.maildomain)
+            config = {
+                "addr": user,
+                "mail_pw": password,
+            }
+            # speed up account configuration
+            config["mail_server"] = self.maildomain
+            config["send_server"] = self.maildomain
+            yield config
+
+    def cache_maybe_retrieve_configured_db_files(self, cache_addr, db_target_path):
+        pass
+
+    def cache_maybe_store_configured_db_files(self, acc):
+        pass
+
+
+@pytest.fixture
+def cmfactory(request, gencreds, tmpdir, maildomain):
+    # cloned from deltachat.testplugin.amfactory
+    pytest.importorskip("deltachat")
+    from deltachat.testplugin import ACFactory
+
+    data = request.getfixturevalue("data")
+
+    testproc = ChatmailTestProcess(request.config, maildomain, gencreds)
+    am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=data)
+
+    # nb. a bit hacky
+    # would probably be better if deltachat's test machinery grows native support
+    def switch_maildomain(maildomain2):
+        am.testprocess.maildomain = maildomain2
+
+    am.switch_maildomain = switch_maildomain
+
+    yield am
+    if hasattr(request.node, "rep_call") and request.node.rep_call.failed:
+        if testproc.pytestconfig.getoption("--extra-info"):
+            logfile = io.StringIO()
+            am.dump_imap_summary(logfile=logfile)
+            print(logfile.getvalue())
+            # request.node.add_report_section("call", "imap-server-state", s)
+
+
+@pytest.fixture
+def remote(sshdomain):
+    return Remote(sshdomain)
+
+
+class Remote:
+    def __init__(self, sshdomain):
+        self.sshdomain = sshdomain
+
+    def iter_output(self, logcmd=""):
+        getjournal = "journalctl -f" if not logcmd else logcmd
+        self.popen = subprocess.Popen(
+            ["ssh", f"root@{self.sshdomain}", getjournal],
+            stdout=subprocess.PIPE,
+        )
+        while 1:
+            line = self.popen.stdout.readline()
+            res = line.decode().strip().lower()
+            if res:
+                yield res
+            else:
+                break
+
+
+@pytest.fixture
+def lp(request):
+    class LP:
+        def sec(self, msg):
+            print(f"---- {msg} ----")
+
+        def indent(self, msg):
+            print(f"     {msg}")
+
+    return LP()
+
+
+@pytest.fixture
+def cmsetup(maildomain, gencreds):
+    return CMSetup(maildomain, gencreds)
+
+
+class CMSetup:
+    def __init__(self, maildomain, gencreds):
+        self.maildomain = maildomain
+        self.gencreds = gencreds
+
+    def gen_users(self, num):
+        print(f"Creating {num} online users")
+        users = []
+        for i in range(num):
+            addr, password = self.gencreds()
+            user = CMUser(self.maildomain, addr, password)
+            assert user.smtp
+            users.append(user)
+        return users
+
+
+class CMUser:
+    def __init__(self, maildomain, addr, password):
+        self.maildomain = maildomain
+        self.addr = addr
+        self.password = password
+        self._smtp = None
+        self._imap = None
+
+    @property
+    def smtp(self):
+        if not self._smtp:
+            handle = SmtpConn(self.maildomain)
+            handle.connect()
+            handle.login(self.addr, self.password)
+            self._smtp = handle
+        return self._smtp
+
+    @property
+    def imap(self):
+        if not self._imap:
+            imap = ImapConn(self.maildomain)
+            imap.connect()
+            imap.login(self.addr, self.password)
+            self._imap = imap
+        return self._imap

cmdeploy/src/cmdeploy/tests/pytest.ini

@@ -0,0 +1,2 @@
+[pytest]
+addopts = -vrsx --strict-markers

cmdeploy/src/cmdeploy/tests/test_cmdeploy.py

@@ -0,0 +1,33 @@
+import os
+
+import pytest
+from cmdeploy.cmdeploy import get_parser, main
+from chatmaild.config import read_config
+
+
+@pytest.fixture(autouse=True)
+def _chdir(tmp_path):
+    old = os.getcwd()
+    os.chdir(tmp_path)
+    yield
+    os.chdir(old)
+
+
+class TestCmdline:
+    def test_parser(self, capsys):
+        parser = get_parser()
+        parser.parse_args([])
+        init = parser.parse_args(["init", "chat.example.org"])
+        run = parser.parse_args(["run"])
+        assert init and run
+
+    def test_init(self, tmp_path):
+        main(["init", "chat.example.org"])
+        inipath = tmp_path.joinpath("chatmail.ini")
+        config = read_config(inipath)
+        assert config.mail_domain == "chat.example.org"
+
+    def test_init_not_overwrite(self):
+        main(["init", "chat.example.org"])
+        with pytest.raises(SystemExit):
+            main(["init", "chat.example.org"])

cmdeploy/src/cmdeploy/tests/test_helpers.py

@@ -0,0 +1,13 @@
+import importlib.resources
+
+from cmdeploy.www import build_webpages
+
+
+def test_build_webpages(tmp_path, make_config):
+    pkgroot = importlib.resources.files("cmdeploy")
+    src_dir = pkgroot.joinpath("../../../www/src").resolve()
+    assert src_dir.exists(), src_dir
+    config = make_config("chat.example.org")
+    build_dir = tmp_path.joinpath("build")
+    build_webpages(src_dir, build_dir, config)
+    assert len([x for x in build_dir.iterdir() if x.suffix == ".html"]) >= 3

cmdeploy/src/cmdeploy/www.py

@@ -0,0 +1,169 @@
+import importlib.resources
+import webbrowser
+import hashlib
+import time
+import traceback
+
+import markdown
+from jinja2 import Template
+from .genqr import gen_qr_png_data
+from chatmaild.config import read_config
+
+
+def snapshot_dir_stats(somedir):
+    d = {}
+    for path in somedir.iterdir():
+        if path.is_file() and path.name[0] != "." and path.suffix != ".swp":
+            mtime = path.stat().st_mtime
+            hash = hashlib.md5(path.read_bytes()).hexdigest()
+            d[path] = (mtime, hash)
+    return d
+
+
+def prepare_template(source):
+    assert source.exists(), source
+    render_vars = {}
+    render_vars["pagename"] = "home" if source.stem == "index" else source.stem
+    render_vars["markdown_html"] = markdown.markdown(source.read_text())
+    page_layout = source.with_name("page-layout.html").read_text()
+    return render_vars, page_layout
+
+
+def build_webpages(src_dir, build_dir, config):
+    try:
+        _build_webpages(src_dir, build_dir, config)
+    except Exception:
+        print(traceback.format_exc())
+
+
+def timespan_to_english(timespan):
+    val = int(timespan[:-1])
+    c = timespan[-1].lower()
+    match c:
+        case "y":
+            return f"{val} years"
+        case "m":
+            return f"{val} months"
+        case "w":
+            return f"{val} weeks"
+        case "d":
+            return f"{val} days"
+        case "h":
+            return f"{val} hours"
+        case "c":
+            return f"{val} seconds"
+        case _:
+            raise ValueError(
+                c
+                + " is not a valid time unit. Try [y]ears, [w]eeks, [d]ays, or [h]ours"
+            )
+
+
+def int_to_english(number):
+    if number >= 0 and number <= 12:
+        a = [
+            "zero",
+            "one",
+            "two",
+            "three",
+            "four",
+            "five",
+            "six",
+            "seven",
+            "eight",
+            "nine",
+            "ten",
+            "eleven",
+            "twelve",
+        ]
+        return a[number]
+    elif number <= 50:
+        return str(number)
+    if number > 50:
+        return "more"
+
+
+def _build_webpages(src_dir, build_dir, config):
+    mail_domain = config.mail_domain
+    assert src_dir.exists(), src_dir
+    if not build_dir.exists():
+        build_dir.mkdir()
+
+    qr_path = build_dir.joinpath(f"qr-chatmail-invite-{mail_domain}.png")
+    qr_path.write_bytes(gen_qr_png_data(mail_domain).read())
+
+    for path in src_dir.iterdir():
+        if path.suffix == ".md":
+            render_vars, content = prepare_template(path)
+            render_vars["username_min_length"] = int_to_english(
+                config.username_min_length
+            )
+            render_vars["username_max_length"] = int_to_english(
+                config.username_max_length
+            )
+            render_vars["password_min_length"] = int_to_english(
+                config.password_min_length
+            )
+            render_vars["delete_mails_after"] = timespan_to_english(
+                config.delete_mails_after
+            )
+            target = build_dir.joinpath(path.stem + ".html")
+
+            # recursive jinja2 rendering
+            while 1:
+                new = Template(content).render(config=config, **render_vars)
+                if new == content:
+                    break
+                content = new
+
+            with target.open("w") as f:
+                f.write(content)
+        elif path.name != "page-layout.html":
+            target = build_dir.joinpath(path.name)
+            target.write_bytes(path.read_bytes())
+    return build_dir
+
+
+def main():
+    path = importlib.resources.files(__package__)
+    reporoot = path.joinpath("../../../").resolve()
+    inipath = reporoot.joinpath("chatmail.ini")
+    config = read_config(inipath)
+    config.webdev = True
+    assert config.mail_domain
+    www_path = reporoot.joinpath("www")
+    src_path = www_path.joinpath("src")
+    stats = None
+    build_dir = www_path.joinpath("build")
+    src_dir = www_path.joinpath("src")
+    index_path = build_dir.joinpath("index.html")
+
+    # start web page generation, open a browser and wait for changes
+    build_webpages(src_dir, build_dir, config)
+    webbrowser.open(str(index_path))
+    stats = snapshot_dir_stats(src_path)
+    print(f"\nOpened URL: file://{index_path.resolve()}\n")
+    print(f"watching {src_path} directory for changes")
+
+    changenum = 0
+    for count in range(0, 1000000):
+        newstats = snapshot_dir_stats(src_path)
+        if newstats == stats and count % 60 != 0:
+            count += 1
+            time.sleep(1.0)
+            continue
+
+        for key in newstats:
+            if stats[key] != newstats[key]:
+                print(f"*** CHANGED: {key}")
+                changenum += 1
+
+        stats = newstats
+        build_webpages(src_dir, build_dir, config)
+        print(f"[{changenum}] regenerated web pages at: {index_path}")
+        print(f"URL: file://{index_path.resolve()}\n\n")
+        count = 0
+
+
+if __name__ == "__main__":
+    main()

doveauth/README.md

@@ -1,7 +0,0 @@
-# doveauth
-
-doveauth is a python tool
-to create dovecot users on login.
-It is called by the
-[dovecot lua authentication module](https://doc.dovecot.org/configuration_manual/authentication/lua_based_authentication/)
-

doveauth/pyproject.toml

@@ -1,30 +0,0 @@
-[build-system]
-requires = ["setuptools>=45"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "doveauth"
-version = "0.1"
-
-[project.scripts]
-doveauth = "doveauth.doveauth:main"
-
-[tool.pytest.ini_options]
-addopts = "-v -ra --strict-markers"
-
-[tool.tox]
-legacy_tox_ini = """
-[tox]
-isolated_build = true
-envlist = lint
-
-[testenv:lint]
-skipdist = True
-skip_install = True
-deps =
-  ruff
-  black
-commands =
-  black --quiet --check --diff src/
-  ruff src/
-"""

doveauth/src/doveauth/doveauth.lua

@@ -1,59 +0,0 @@
-
--- Escape shell argument by hex encoding it and wrapping in quotes.
-function escape(data)
-   b16 = data:gsub(".", function(char) return string.format("%2X", char:byte()) end)
-   return ("'"..b16.."'")
-end
-
--- call out to python program to actually manage authentication for dovecot
-
-function chatctl_verify(user, password)
-    local cmd = "doveauth hexauth "..escape(user).." "..escape(password)
-    print("executing: "..cmd)
-    local handle = io.popen(cmd)
-    local result = handle:read("*a")
-    handle:close()
-    return split_chatctl(result)
-end
-
-function chatctl_lookup(user) 
-    local cmd = "doveauth hexlookup "..escape(user)
-    assert(user)
-    print("executing: "..cmd)
-    local handle = io.popen(cmd)
-    local result = handle:read("*a")
-    handle:close()
-    return split_chatctl(result)
-end
-
-function get_extra_dovecot_output(res)
-    return {home=res.home, uid=res.uid, gid=res.gid}
-end
-
-
-function auth_password_verify(request, password)
-    local res = chatctl_verify(request.user, password)
-    -- request:log_error("auth_password_verify "..request.user.." "..password)
-    if res.status == "ok" then 
-        local extra = get_extra_dovecot_output(res)
-        return dovecot.auth.PASSDB_RESULT_OK, get_extra_dovecot_output(res)
-    end
-    return dovecot.auth.PASSDB_RESULT_PASSWORD_MISMATCH, ""
-end
-
-
-function auth_userdb_lookup(request)
-    local res = chatctl_lookup(request.user) 
-    if res.status == "ok" then
-        return dovecot.auth.USERDB_RESULT_OK, get_extra_dovecot_output(res)
-    end
-    return dovecot.auth.USERDB_RESULT_USER_UNKNOWN, "no such user"
-end
-
-function split_chatctl(output) 
-    local ret = {}
-    for key, value in output:gmatch "(%w+)%s*=%s*(%w+)" do
-        ret[key] = value
-    end
-    return ret
-end

doveauth/src/doveauth/doveauth.py

@@ -1,65 +0,0 @@
-#!/usr/bin/env python3
-import base64
-import sys
-
-from .database import Database
-
-
-def get_user_data(db, user):
-    with db.read_connection() as conn:
-        result = conn.get_user(user)
-    if result:
-        result['uid'] = "vmail"
-        result['gid'] = "vmail"
-    return result
-
-
-def create_user(db, user, password):
-    with db.write_transaction() as conn:
-        conn.create_user(user, password)
-    return dict(home=f"/home/vmail/{user}", uid="vmail", gid="vmail", password=password)
-
-
-def verify_user(db, user, password):
-    userdata = get_user_data(db, user)
-    if userdata:
-        if userdata.get("password") == password:
-            userdata["status"] = "ok"
-        else:
-            userdata["status"] = "fail"
-    else:
-        userdata = create_user(db, user, password)
-        userdata["status"] = "ok"
-
-    return userdata
-
-
-def lookup_user(db, user):
-    userdata = get_user_data(db, user)
-    if userdata:
-        userdata["status"] = "ok"
-    else:
-        userdata["status"] = "fail"
-    return userdata
-
-
-def dump_result(res):
-    for key, value in res.items():
-        print(f"{key}={value}")
-
-
-def main():
-    db = Database("/home/vmail/passdb.sqlite")
-    if sys.argv[1] == "hexauth":
-        login = base64.b16decode(sys.argv[2]).decode()
-        password = base64.b16decode(sys.argv[3]).decode()
-        res = verify_user(db, login, password)
-        dump_result(res)
-    elif sys.argv[1] == "hexlookup":
-        login = base64.b16decode(sys.argv[2]).decode()
-        res = lookup_user(db, login)
-        dump_result(res)
-
-
-if __name__ == "__main__":
-    main()

doveauth/src/doveauth/test_doveauth.lua

@@ -1,78 +0,0 @@
-
-require "doveauth"
-
--- simulate dovecot defined result codes 
-
-dovecot = {
-    auth = {
-        PASSDB_RESULT_OK="PASSWORD-OK",
-        PASSDB_RESULT_PASSWORD_MISMATCH="PASSWORD-MISMATCH",
-        USERDB_RESULT_OK="USERDB-OK",
-        USERDB_RESULT_USER_UNKNOWN="USERDB-UNKNOWN"
-    }
-}
-
-
--- Tests for testing the lua<->python interaction 
-
-function test_password_verify_ok(user, password) 
-    local res, extra = auth_password_verify({user=user}, password)
-    assert(res==dovecot.auth.PASSDB_RESULT_OK)
-    assert(extra.uid == "vmail")
-    assert(extra.gid == "vmail")
-    -- assert(extra.homedir == "/home/vmail/link2xt")
-    print("OK test_password_verify_ok "..user.." "..password)
-end
-
-function test_password_verify_mismatch(user, password) 
-    local res = auth_password_verify({user=user}, password)
-    assert(res == dovecot.auth.PASSDB_RESULT_PASSWORD_MISMATCH)
-    print("OK test_password_verify_mismatch "..user.." "..password)
-end
-
-function test_userdb_lookup_ok(user)
-    local res, extra = auth_userdb_lookup({user=user})
-    assert(extra.uid == "vmail")
-    assert(extra.gid == "vmail")
-    assert(res == dovecot.auth.USERDB_RESULT_OK)
-    print("OK test_userdb_lookup_ok "..user)
-end
-
-function test_userdb_lookup_mismatch(user)
-    local res, extra = auth_userdb_lookup({user=user})
-    assert(res == dovecot.auth.USERDB_RESULT_USER_UNKNOWN)
-    print("OK test_userdb_lookup_mismatch "..user)
-end
-
-function test_passdb_lookup_ok(user)
-    local res, extra = auth_passdb_lookup({user=user})
-    assert(extra.uid == "vmail")
-    assert(extra.gid == "vmail")
-    assert(res == dovecot.auth.PASSDB_RESULT_OK)
-    print("OK test_passdb_lookup_ok "..user)
-end
-
-function test_passdb_lookup_mismatch(user)
-    local res, extra = auth_passdb_lookup({user=user})
-    assert(res == dovecot.auth.PASSDB_RESULT_USER_UNKNOWN)
-    print("OK test_passdb_lookup_mismatch "..user)
-end
-
-function test_split_chatctl()
-    local res = split_chatctl("a=3 b=4\nc=5")
-    assert(res["a"] == "3")
-    assert(res["b"] == "4")
-    assert(res["c"] == "5")
-    print("OK test_split_chatctl")
-end 
-
-test_split_chatctl()
-test_password_verify_ok("link2xt@c1.testrun.org", "Ahyei6ie")
-test_password_verify_mismatch("link2xt@c1.testrun.org", "Aqwlek")
-test_userdb_lookup_ok("link2xt@c1.testrun.org")
-test_userdb_lookup_mismatch("wlekqjlew@xyz.org")
-
--- probably not needed by dovecot? 
--- test_passdb_lookup_ok("link2xt@c1.testrun.org")
--- test_passdb_lookup_mismatch("llqkwjelqwe@xyz.org")
-

doveauth/src/doveauth/test_doveauth.py

@@ -1,25 +0,0 @@
-import subprocess
-import pytest
-
-from doveauth.doveauth import get_user_data, verify_user, Database
-
-
-def test_basic(tmpdir):
-    db = Database(tmpdir / "passdb.sqlite")
-    verify_user(db, "link2xt@c1.testrun.org", "asdf")
-    data = get_user_data(db, "link2xt@c1.testrun.org")
-    assert data
-
-
-def test_verify_or_create(tmpdir):
-    db = Database(tmpdir / "passdb.sqlite")
-    res = verify_user(db, "newuser1@something.org", "kajdlkajsldk12l3kj1983")
-    assert res["status"] == "ok"
-    res = verify_user(db, "newuser1@something.org", "kajdlqweqwe")
-    assert res["status"] == "fail"
-
-
-def test_lua_integration(request):
-    p = request.fspath.dirpath("test_doveauth.lua")
-    proc = subprocess.run(["lua", str(p)])
-    assert proc.returncode == 0

plan.txt

@@ -1,37 +0,0 @@
-# Chat-mail server development (up until Oct 18th)
-
-## Dovecot goals/steps
-
-1. create-user-on-login ("doveauth")
-
-2. per-user quota (adaptive)
-
-3. automatic expiry of messages older than M days
-
-4. automatic expiry of users that haven't logged in for N days
-
-
-## Postfix goals/steps
-
-1. block all outgoing mails with our own LMTP program
-
-2. only allow (outgoing) mails if secure-join or autocrypt-pgp-encrypted format
-   (probably via an lmtp service)
-
-3. basic outgoing send rate/limits (depending on "account-rating")
-
-
-## online tests (first with plain python/pytest)
-
-- write tests for dovecot login (exists)
-- write tests for postfix logins
-- write A<>B send/receive tests
-
-
-## Delta Chat
-
-1. qr code that defines access to a chatmail instance (like mailadm but without http etc.)
-
-2. support for creating username/password and verifying login works
-
-

scripts/cmdeploy

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+#
+# Wrapper for cmdelpoy to run it in activated virtualenv.
+set -e
+. venv/bin/activate
+cmdeploy "$@"

scripts/deploy.sh

@@ -1,7 +0,0 @@
-#!/usr/bin/env bash
-: ${CHATMAIL_DOMAIN:=c1.testrun.org}
-export CHATMAIL_DOMAIN
-cd doveauth
-venv/bin/python3 -m build
-../chatmail-pyinfra/venv/bin/pyinfra --ssh-user root "$CHATMAIL_DOMAIN" ../deploy.py
-rm -r dist/

scripts/init.sh

@@ -1,8 +0,0 @@
-#!/bin/sh
-python3 -m venv chatmail-pyinfra/venv
-chatmail-pyinfra/venv/bin/pip install pyinfra pytest
-chatmail-pyinfra/venv/bin/pip install -e chatmail-pyinfra
-chatmail-pyinfra/venv/bin/pip install -e doveauth
-python3 -m venv doveauth/venv
-doveauth/venv/bin/pip install pytest build
-doveauth/venv/bin/pip install -e doveauth

scripts/initenv.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+set -e
+python3 -m venv venv
+
+venv/bin/pip install -e chatmaild 
+venv/bin/pip install -e cmdeploy

scripts/test.sh

@@ -1,4 +0,0 @@
-#!/bin/sh
-chatmail-pyinfra/venv/bin/pytest chatmail-pyinfra/tests
-cd doveauth/src/doveauth
-../../venv/bin/pytest

www/src/index.md

@@ -0,0 +1,19 @@
+
+<img class="banner" src="collage-top.png"/>
+
+## Dear [Delta Chat](https://get.delta.chat) users and newcomers, 
+
+Welcome to instant, interoperable and [privacy-preserving](privacy.html) messaging :) 
+
+👉 **Tap** or scan this QR code to get a random `@{{config.mail_domain}}` e-mail address
+
+<a href="DCACCOUNT:https://{{ config.mail_domain }}/cgi-bin/newemail.py">
+    <img width=300 style="float: none;" src="qr-chatmail-invite-{{config.mail_domain}}.png" /></a>
+
+🐣 **Choose** your Avatar and Name
+
+💬 **Start** chatting with any Delta Chat contacts using [QR invite codes](https://delta.chat/en/help#howtoe2ee)
+
+{% if config.mail_domain != "nine.testrun.org" %}
+<div class="experimental">Note: this is only a temporary development chatmail service</div>
+{% endif %}

www/src/info.md

@@ -0,0 +1,50 @@
+
+<img class="banner" src="collage-info.png"/>
+
+## More information 
+
+### Choosing a chatmail address instead of using a random one
+
+In the Delta Chat account setup 
+you may tap `LOG INTO YOUR E-MAIL ACCOUNT` 
+and fill the two fields like this: 
+
+- `Address`: invent a word with
+{% if username_min_length == username_max_length %}
+  *exactly* {{ username_min_length }}
+{% else %}
+  {{ username_min_length}}
+  {% if username_max_length == "more" %}
+    or more
+  {% else %}
+    to {{ username_max_length }}
+  {% endif %}
+{% endif %}
+  characters
+  and append `@{{config.mail_domain}}` to it.
+
+- `Password`: invent at least {{ password_min_length }} characters.
+
+If the e-mail address is not yet taken, you'll get that account. 
+The first login sets your password. 
+
+
+### Rate and storage limits 
+
+- Un-encrypted messages are blocked to recipients outside
+  {{config.mail_domain}} but setting up contact via [QR invite codes](https://delta.chat/en/help#howtoe2ee) 
+  allows your messages to pass freely to any outside recipients.
+
+- You may send up to {{ config.max_user_send_per_minute }} messages per minute.
+
+- Seen messages are removed {{ delete_mails_after }} after arriving on the server.
+
+- You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
+
+
+### Who are the operators? Which software is running? 
+
+This chatmail provider is run by a small voluntary group of devs and sysadmins,
+who [publically develop chatmail provider setups](https://github.com/deltachat/chatmail).
+Chatmail setups aim to be very low-maintenance, resource efficient and 
+interoperable with any other standards-compliant e-mail service. 

www/src/logo.svg

@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   width="145"
+   height="145"
+   version="1.1"
+   id="svg4"
+   sodipodi:docname="At_sign.svg"
+   inkscape:version="1.2.2 (b0a84865, 2022-12-01)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg">
+  <defs
+     id="defs8" />
+  <sodipodi:namedview
+     id="namedview6"
+     pagecolor="#ffffff"
+     bordercolor="#000000"
+     borderopacity="0.25"
+     inkscape:showpageshadow="2"
+     inkscape:pageopacity="0.0"
+     inkscape:pagecheckerboard="0"
+     inkscape:deskcolor="#d1d1d1"
+     showgrid="false"
+     inkscape:zoom="3.0241379"
+     inkscape:cx="67.622577"
+     inkscape:cy="72.913341"
+     inkscape:window-width="1390"
+     inkscape:window-height="1027"
+     inkscape:window-x="55"
+     inkscape:window-y="25"
+     inkscape:window-maximized="0"
+     inkscape:current-layer="svg4" />
+  <g
+     aria-label="@"
+     id="text2"
+     style="font-size:144px;font-family:Arial">
+    <path
+       d="m 79.927878,94.422406 c -2.704286,3.120332 -5.741407,5.637394 -9.111364,7.551194 -3.328352,1.87221 -6.677506,2.80831 -10.047463,2.80831 -3.702792,0 -7.301573,-1.08172 -10.796342,-3.24515 -3.49477,-2.163426 -6.344671,-5.491779 -8.549704,-9.985058 -2.163429,-4.493275 -3.245144,-9.423397 -3.245144,-14.790365 0,-6.615099 1.684978,-13.230199 5.054935,-19.845299 3.411561,-6.656705 7.634407,-11.649233 12.66854,-14.977585 5.034133,-3.328352 9.92265,-4.992528 14.665552,-4.992528 3.619583,0 7.072748,0.956901 10.359496,2.870704 3.286748,1.872198 6.115847,4.742902 8.487297,8.612111 l 2.121825,-9.673023 h 11.170784 l -8.986557,41.87483 c -1.248129,5.824616 -1.872194,9.048957 -1.872194,9.673023 0,1.123319 0.416044,2.101022 1.248132,2.93311 0.873692,0.790484 1.913802,1.185726 3.120332,1.185726 2.20503,0 5.096537,-1.268934 8.674517,-3.806803 4.7429,-3.328352 8.4873,-7.780023 11.23319,-13.355013 2.78749,-5.616594 4.18124,-11.399606 4.18124,-17.349035 0,-6.947935 -1.78899,-13.438222 -5.36697,-19.47086 -3.53637,-6.032638 -8.84094,-10.858749 -15.913687,-14.478332 -7.03114,-3.619583 -14.811161,-5.429374 -23.340064,-5.429374 -9.73543,0 -18.638772,2.288242 -26.710026,6.864726 -8.029649,4.534879 -14.27031,11.06677 -18.721981,19.595673 -4.410066,8.487298 -6.615099,17.598662 -6.615099,27.334092 0,10.193078 2.205033,18.971607 6.615099,26.33559 2.290454,3.78888 -7.136335,18.96983 -3.810585,21.73443 3.138096,2.60861 18.971963,-7.14297 23.031819,-5.44631 8.404089,3.53637 17.702673,5.30456 27.895752,5.30456 10.90035,0 20.032515,-1.83059 27.396492,-5.49178 7.36399,-3.66119 12.87657,-8.11286 16.53776,-13.35501 l 9.29559,4 c -2.12183,4.36846 -3.76221,4.82013 -8.92116,9.35501 -5.15895,4.53488 -11.2956,8.11286 -18.40995,10.73393 -7.114346,2.66268 -15.684851,3.99402 -25.711512,3.99402 -9.236177,0 -17.76508,-1.18572 -25.586707,-3.55717 -7.780023,-2.37145 -29.296198,9.26152 -34.78798,4.47701 -5.49178,-4.7429 5.248856,-25.42482 2.461361,-31.62388 -3.49477,-7.863231 -5.242155,-16.350531 -5.242155,-25.461894 0,-10.151474 2.08022,-19.824498 6.240661,-29.019071 5.075736,-11.274793 12.273297,-19.907706 21.592683,-25.898739 9.360991,-5.991034 20.69819,-8.986551 34.011599,-8.986551 10.317891,0 19.574873,2.121824 27.77093,6.365473 8.23767,4.202045 14.72796,10.484309 19.47086,18.846794 4.03563,7.197561 6.05344,15.019189 6.05344,23.464883 0,12.065277 -4.24365,22.77841 -12.73094,32.1394 -7.572,8.404095 -15.85128,12.606135 -24.837827,12.606135 -2.870704,0 -5.200551,-0.43684 -6.98954,-1.31053 -1.747385,-0.8737 -3.037121,-2.12183 -3.869209,-3.744402 -0.540857,-1.040114 -0.936099,-2.829105 -1.185726,-5.366972 z M 49.723082,77.510217 c 0,5.699803 1.352143,10.130671 4.05643,13.292606 2.704286,3.161935 5.803814,4.742902 9.298583,4.742902 2.329847,0 4.784506,-0.686473 7.363979,-2.059418 2.579473,-1.41455 5.034133,-3.49477 7.363979,-6.240661 2.371451,-2.74589 4.306056,-6.219857 5.803815,-10.421902 1.497759,-4.243649 2.246638,-8.487298 2.246638,-12.730947 0,-5.658198 -1.41455,-10.047462 -4.243649,-13.167793 -2.787495,-3.12033 -6.199056,-4.680495 -10.234683,-4.680495 -2.662682,0 -5.179749,0.686473 -7.5512,2.059418 -2.329846,1.331341 -4.597286,3.494769 -6.802319,6.490286 -2.205033,2.995517 -3.97322,6.635903 -5.304561,10.921156 -1.331341,4.285253 -1.997012,8.216869 -1.997012,11.794848 z"
+       id="path347"
+       style="stroke-width:0.887561"
+       sodipodi:nodetypes="ccsscscsscccccscsccsccsccscscssccscscccsccsccscscccssscccscscsss" />
+  </g>
+</svg>

www/src/main.css

@@ -0,0 +1,74 @@
+
+body {
+  font-family: system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen', 'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue', 'Segoe UI Emoji', 'Apple Color Emoji', 'Noto Color Emoji', sans-serif;
+  line-height: 1.4;
+  font-size: 1.2em;
+  max-width: 800px;
+  margin: 20px auto;
+  padding: 0 10px;
+  color: #363636;
+  background: #fff;
+}
+
+h1 {
+  font-size: 2.2em;
+  margin-top: 0;
+}
+
+h1, h2, h3, h4, h5, h6 {
+  color: #000;
+  margin-bottom: 12px;
+  margin-top: 24px;
+  font-weight: 600;
+}
+
+a {
+  text-decoration: none;
+  color: #0076d1;
+}
+
+a:hover {
+  text-decoration: underline;
+}
+
+img, video {
+  max-width: 100%;
+  height: auto;
+}
+
+code {
+  background: #efefef;
+  padding: 2.5px 5px;
+  border-radius: 6px;
+}
+
+#menu {
+  display: flex;
+  flex-wrap: wrap;
+  padding: 0;
+}
+
+#menu li {
+  display: inline-block;
+  padding-right: 0.5em;
+}
+
+#domain {
+  margin-left: auto;
+}
+
+#domain a {
+  color: #888;
+}
+
+.banner {
+  width: 100%;
+}
+
+.experimental {
+  margin: 3em 0;
+  padding: 1em;
+  border: 4px dashed red;
+  color: red;
+  font-weight: bold;
+}

www/src/page-layout.html

@@ -0,0 +1,27 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <meta charset="utf-8" />
+    {% if config.webdev %}
+    <meta http-equiv="refresh" content="3">
+    {% endif %}
+    <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0">
+    <title>{{ config.mail_domain }} {{ pagename }}</title>
+    <link rel="stylesheet" href="./main.css">
+    <link rel="icon" href="/logo.svg">
+    <link rel=”mask-icon” href=”/logo.svg” color=”#000000">
+</head>
+<body>
+
+<ul id="menu">
+    <li><a href="index.html">home</a></li>
+    <li><a href="info.html">info</a></li>
+    <li><a href="privacy.html">privacy</a></li>
+    <li><a href="https://github.com/deltachat/chatmail">public code ↗</a></li>
+    <li id="domain"><a href="index.html">{{ config.mail_domain }}</a></li>
+</ul>
+
+{{ markdown_html }}
+
+</body>
+</html>

www/src/privacy.md

@@ -0,0 +1,240 @@
+<img class="banner" src="collage-privacy.png"/>
+
+# Privacy Policy for {{ config.mail_domain }} 
+
+We want to show you in a fair and transparent way
+what personal data is processed by us.
+We follow a strict privacy-by-design approach
+and try to avoid processing your data in the first place,
+but as you may know,
+the internet, 
+and in particular sending e-mail messages,
+does not work without data.
+Still,
+it's only fair that you know at all times
+what personal data is processed
+when you use our service.
+
+If you have any remaining questions about data protection, please contact us. 
+
+## 1. Name and contact information 
+
+Responsible for the processing of your personal data is:
+```
+{{ config.privacy_postal }}
+```
+
+E-mail: {{ config.privacy_mail }}
+
+We have appointed a data protection officer:
+
+```
+{{ config.privacy_pdo }}
+```
+
+## 2. Processing when using chat e-mail services
+
+We provide e-mail services optimized for the use from [Delta Chat](https://delta.chat) apps 
+and process only the data necessary
+for the setup and technical execution of the e-mail dispatch.
+The purpose of the processing is to
+read, write, manage, delete, send, and receive emails.
+For this purpose,
+we operate server-side software
+that enables us to send and receive e-mail messages.
+Allowing the use of the e-mail service,
+we process the following data and details:
+
+- Outgoing and incoming messages (SMTP) are stored for transit 
+  on behalf of their users until the message can be delivered.
+
+- E-Mail-Messages are stored for the recipient and made accessible via IMAP protocols,
+  until explicitly deleted by the user or until a fixed time period is exceeded,
+  (*usually 4-8 weeks*).
+
+- IMAP and SMTP protocols are password protected with unique credentials for each account.
+
+- Users can retrieve or delete all stored messages 
+  without intervention from the operators using standard IMAP client tools.
+
+### 3.1 Account setup
+
+Creating an account happens in one of two ways on our mail servers: 
+
+- with a QR invitation token 
+  which is scanned using the Delta Chat app
+  and then the account is created.
+
+- by letting Delta Chat otherwise create an account 
+  and register it with a {{ config.mail_domain }} mail server. 
+
+In either case, we process the newly created email address.
+No phone numbers,
+other email addresses,
+or other identifiable data
+is currently required.
+The legal basis for the processing is
+Art. 6 (1) lit. b GDPR,
+as you have a usage contract with us
+by using our services.
+
+## 3.2 Processing of E-Mail-Messages
+
+In addition,
+we will process data
+to keep the server infrastructure operational
+for purposes of e-mail dispatch
+and abuse prevention.
+
+- Therefore,
+  it is necessary to process the content and/or metadata
+  (e.g., headers of the email as well as smtp chatter)
+  of E-Mail-Messages in transit. 
+
+- We will keep logs of messages in transit for a limited time.
+  These logs are used to debug delivery problems and software bugs.
+
+In addition,
+we process data to protect the systems from excessive use.
+Therefore, limits are enforced:
+
+- rate limits
+
+- storage limits
+
+- message size limits
+
+- any other limit neccessary for the whole server to function in a healthy way
+  and to prevent abuse.
+
+The processing and use of the above permissions
+are performed to provide the service.
+The data processing is necessary for the use of our services,
+therefore the legal basis of the processing is
+Art. 6 (1) lit. b GDPR,
+as you have a usage contract with us
+by using our services.
+The legal basis for the data processing
+for the purposes of security and abuse prevention is
+Art. 6 (1) lit. f GDPR.
+Our legitimate interest results
+from the aforementioned purposes.
+We will not use the collected data
+for the purpose of drawing conclusions
+about your person.
+
+
+## 3. Processing when using our Website
+
+When you visit our website,
+the browser used on your end device
+automatically sends information to the server of our website.
+This information is temporarily stored in a so-called log file.
+The following information is collected and stored
+until it is automatically deleted
+(*usually 7 days*):
+
+- used type of browser,
+
+- used operating system, 
+
+- access date and time as well as
+
+- country of origin and IP address, 
+
+- the requested file name or HTTP resource,
+
+- the amount of data transferred,
+
+- the access status (file transferred, file not found, etc.) and
+
+- the page from which the file was requested.
+
+This website is hosted by an external service provider (hoster).
+The personal data collected on this website is stored
+on the hoster's servers.
+Our hoster will process your data
+only to the extent necessary to fulfill its obligations
+to perform under our instructions.
+In order to ensure data protection-compliant processing,
+we have concluded a data processing agreement with our hoster.
+
+The aforementioned data is processed by us for the following purposes:  
+
+- Ensuring a reliable connection setup of the website,
+
+- ensuring a convenient use of our website,
+
+- checking and ensuring system security and stability, and
+
+- for other administrative purposes.
+
+The legal basis for the data processing is
+Art. 6 (1) lit. f GDPR.
+Our legitimate interest results
+from the aforementioned purposes of data collection.
+We will not use the collected data
+for the purpose of drawing conclusions about your person.
+
+## 4. Transfer of Data
+
+Your personal data
+will not be transferred to third parties
+for purposes other than those listed below:
+
+a) you have given your express consent
+in accordance with Art. 6 para. 1 sentence 1 lit. a GDPR,  
+
+b) the disclosure is necessary for the assertion, exercise or defence of legal claims
+pursuant to Art. 6 (1) sentence 1 lit. f GDPR
+and there is no reason to assume that you have
+an overriding interest worthy of protection
+in the non-disclosure of your data,  
+
+c) in the event that there is a legal obligation to disclose your data
+pursuant to Art. 6 para. 1 sentence 1 lit. c GDPR,
+as well as  
+
+d) this is legally permissible and necessary
+in accordance with Art. 6 Para. 1 S. 1 lit. b GDPR
+for the processing of contractual relationships with you,  
+
+e) this is carried out by a service provider
+acting on our behalf and on our exclusive instructions,
+whom we have carefully selected (Art. 28 (1) GDPR)
+and with whom we have concluded a corresponding contract on commissioned processing (Art. 28 (3) GDPR),
+which obliges our contractor,
+among other things,
+to implement appropriate security measures
+and grants us comprehensive control powers.
+
+## 5. Rights of the data subject
+
+The rights arise from Articles 12 to 23 GDPR.
+Since no personal data is stored on our servers,
+even in encrypted form,
+there is no need to provide information
+on these or possible objections.
+A deletion can be made
+directly in the Delta Chat email messenger.
+
+If you have any questions or complaints, 
+please feel free to contact us by email:  
+{{ config.privacy_mail }}
+
+As a rule, you can contact the supervisory authority of your usual place of residence
+or workplace
+or our registered office for this purpose.
+The supervisory authority responsible for our place of business
+is the `{{ config.privacy_supervisor }}`.
+
+
+## 6. Validity of this privacy policy 
+
+This data protection declaration is valid
+as of *December 2023*.
+Due to the further development of our service and offers
+or due to changed legal or official requirements,
+it may become necessary to revise this data protection declaration from time to time.
+
+