Add script to remove old seen messages

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,33 +0,0 @@
----
-name: Bug report
-about: Report something that isn't working.
-title: ''
-assignees: ''
-
----
-
-<!--
-Please fill out as much of this form as you can (leaving out stuff that is not applicable is ok).
--->
-
-- Server OS (Operating System) - preferably Debian 12: 
-- On which OS you run cmdeploy:
-- chatmail/relay version: `git rev-parse HEAD`
-
-## Expected behavior
-
-*What did you try to achieve?*
-
-## Actual behavior
-
-*What happened instead?*
-
-### Steps to reproduce the problem:
-
-1. 
-2.
-
-### Screenshots
-
-### Logs
-

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +0,0 @@
-blank_issues_enabled: true
-contact_links:
-  - name: Mutual Help Chat Group
-    url: https://i.delta.chat/#6CBFF8FFD505C0FDEA20A66674F2916EA8FBEE99&a=invitebot%40nine.testrun.org&g=Chatmail%20Mutual%20Help&x=7sFF7Ik50pWv6J1z7RVC5527&i=X69wTFfvCfs3d-JzqP0kVA3i&s=ibp-447dU-wUq-52QanwAtWc
-    about: If you have troubles setting up the relay server, feel free to ask here.

.github/workflows/ci.yaml

@@ -10,10 +10,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-        # Checkout pull request HEAD commit instead of merge commit
-        # Otherwise `test_deployed_state` will be unhappy.
-        with:
-          ref: ${{ github.event.pull_request.head.sha }}
 
       - name: run chatmaild tests 
         working-directory: chatmaild

.github/workflows/staging-ipv4.testrun.org-default.zone

@@ -1,20 +0,0 @@
-;; Zone file for staging-ipv4.testrun.org
-
-$ORIGIN staging-ipv4.testrun.org.
-$TTL 300
-
-@  IN  SOA     ns.testrun.org. root.nine.testrun.org (
-  2023010101 ; Serial
-  7200       ; Refresh
-  3600       ; Retry
-  1209600    ; Expire
-  3600       ; Negative response caching TTL
-)
-
-;; Nameservers.
-@  IN NS ns.testrun.org.
-
-;; DNS records.
-@       IN      A       37.27.95.249
-mta-sts.staging-ipv4.testrun.org.    CNAME   staging-ipv4.testrun.org.
-www.staging-ipv4.testrun.org.        CNAME   staging-ipv4.testrun.org.

.github/workflows/test-and-deploy-ipv4only.yaml

@@ -1,97 +0,0 @@
-name: deploy on staging-ipv4.testrun.org, and run tests
-
-on:
-  push:
-    branches:
-      - main
-  pull_request:
-    paths-ignore:
-      - 'scripts/**'
-      - '**/README.md'
-      - 'CHANGELOG.md'
-      - 'LICENSE'
-
-jobs:
-  deploy:
-    name: deploy on staging-ipv4.testrun.org, and run tests
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    concurrency:
-      group: ci-ipv4-${{ github.workflow }}-${{ github.ref }}
-      cancel-in-progress: ${{ !contains(github.ref, '$GITHUB_REF') }}
-    steps:
-      - uses: jsok/serialize-workflow-action@v1
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-      - uses: actions/checkout@v4
-
-      - name: prepare SSH
-        run: |
-          mkdir ~/.ssh
-          echo "${{ secrets.STAGING_SSH_KEY }}" >> ~/.ssh/id_ed25519
-          chmod 600 ~/.ssh/id_ed25519
-          ssh-keyscan staging-ipv4.testrun.org > ~/.ssh/known_hosts
-          # save previous acme & dkim state
-          rsync -avz root@staging-ipv4.testrun.org:/var/lib/acme acme-ipv4 || true
-          rsync -avz root@staging-ipv4.testrun.org:/etc/dkimkeys dkimkeys-ipv4 || true
-          # store previous acme & dkim state on ns.testrun.org, if it contains useful certs
-          if [ -f dkimkeys-ipv4/dkimkeys/opendkim.private ]; then rsync -avz -e "ssh -o StrictHostKeyChecking=accept-new" dkimkeys-ipv4 root@ns.testrun.org:/tmp/ || true; fi
-          if [ "$(ls -A acme-ipv4/acme/certs)" ]; then rsync -avz -e "ssh -o StrictHostKeyChecking=accept-new" acme-ipv4 root@ns.testrun.org:/tmp/ || true; fi
-          # make sure CAA record isn't set
-          scp -o StrictHostKeyChecking=accept-new .github/workflows/staging-ipv4.testrun.org-default.zone root@ns.testrun.org:/etc/nsd/staging-ipv4.testrun.org.zone
-          ssh root@ns.testrun.org sed -i '/CAA/d' /etc/nsd/staging-ipv4.testrun.org.zone
-          ssh root@ns.testrun.org nsd-checkzone staging-ipv4.testrun.org /etc/nsd/staging-ipv4.testrun.org.zone
-          ssh root@ns.testrun.org systemctl reload nsd
-
-      - name: rebuild staging-ipv4.testrun.org to have a clean VPS
-        run: |
-            curl -X POST \
-            -H "Authorization: Bearer ${{ secrets.HETZNER_API_TOKEN }}" \
-            -H "Content-Type: application/json" \
-            -d '{"image":"debian-12"}' \
-            "https://api.hetzner.cloud/v1/servers/${{ secrets.STAGING_IPV4_SERVER_ID }}/actions/rebuild"
-
-      - run: scripts/initenv.sh
-
-      - name: append venv/bin to PATH
-        run: echo venv/bin >>$GITHUB_PATH
-
-      - name: upload TLS cert after rebuilding
-        run: |
-          echo " --- wait until staging-ipv4.testrun.org VPS is rebuilt --- "
-          rm ~/.ssh/known_hosts
-          while ! ssh -o ConnectTimeout=180 -o StrictHostKeyChecking=accept-new -v root@staging-ipv4.testrun.org id -u ; do sleep 1 ; done
-          ssh -o StrictHostKeyChecking=accept-new -v root@staging-ipv4.testrun.org id -u
-          # download acme & dkim state from ns.testrun.org
-          rsync -e "ssh -o StrictHostKeyChecking=accept-new" -avz root@ns.testrun.org:/tmp/acme-ipv4/acme acme-restore || true
-          rsync -avz root@ns.testrun.org:/tmp/dkimkeys-ipv4/dkimkeys dkimkeys-restore || true
-          # restore acme & dkim state to staging2.testrun.org
-          rsync -avz acme-restore/acme root@staging-ipv4.testrun.org:/var/lib/ || true
-          rsync -avz dkimkeys-restore/dkimkeys root@staging-ipv4.testrun.org:/etc/ || true
-          ssh -o StrictHostKeyChecking=accept-new -v root@staging-ipv4.testrun.org chown root:root -R /var/lib/acme || true
-
-      - name: run deploy-chatmail offline tests 
-        run: pytest --pyargs cmdeploy 
-
-      - run: |
-          cmdeploy init staging-ipv4.testrun.org
-          sed -i 's#disable_ipv6 = False#disable_ipv6 = True#' chatmail.ini
-
-      - run: cmdeploy run --verbose --skip-dns-check
-
-      - name: set DNS entries
-        run: |
-          ssh -o StrictHostKeyChecking=accept-new -v root@staging-ipv4.testrun.org chown opendkim:opendkim -R /etc/dkimkeys
-          cmdeploy dns --zonefile staging-generated.zone
-          cat staging-generated.zone >> .github/workflows/staging-ipv4.testrun.org-default.zone
-          cat .github/workflows/staging-ipv4.testrun.org-default.zone
-          scp .github/workflows/staging-ipv4.testrun.org-default.zone root@ns.testrun.org:/etc/nsd/staging-ipv4.testrun.org.zone
-          ssh root@ns.testrun.org nsd-checkzone staging-ipv4.testrun.org /etc/nsd/staging-ipv4.testrun.org.zone
-          ssh root@ns.testrun.org systemctl reload nsd
-
-      - name: cmdeploy test
-        run: CHATMAIL_DOMAIN2=nine.testrun.org cmdeploy test --slow
-
-      - name: cmdeploy dns
-        run: cmdeploy dns -v
-

.github/workflows/test-and-deploy.yaml

@@ -15,14 +15,10 @@ jobs:
   deploy:
     name: deploy on staging2.testrun.org, and run tests
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     concurrency:
-      group: ci-${{ github.workflow }}-${{ github.ref }}
-      cancel-in-progress: ${{ !contains(github.ref, '$GITHUB_REF') }}
+      group: staging-deploy
+      cancel-in-progress: true
     steps:
-      - uses: jsok/serialize-workflow-action@v1
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
       - uses: actions/checkout@v4
 
       - name: prepare SSH
@@ -38,9 +34,7 @@ jobs:
           if [ -f dkimkeys/opendkim.private ]; then rsync -avz -e "ssh -o StrictHostKeyChecking=accept-new" dkimkeys root@ns.testrun.org:/tmp/ || true; fi
           if [ "$(ls -A acme/certs)" ]; then rsync -avz -e "ssh -o StrictHostKeyChecking=accept-new" acme root@ns.testrun.org:/tmp/ || true; fi
           # make sure CAA record isn't set
-          scp -o StrictHostKeyChecking=accept-new .github/workflows/staging.testrun.org-default.zone root@ns.testrun.org:/etc/nsd/staging2.testrun.org.zone
-          ssh root@ns.testrun.org sed -i '/CAA/d' /etc/nsd/staging2.testrun.org.zone
-          ssh root@ns.testrun.org nsd-checkzone staging2.testrun.org /etc/nsd/staging2.testrun.org.zone
+          ssh -o StrictHostKeyChecking=accept-new root@ns.testrun.org sed -i '/CAA/d' /etc/nsd/staging2.testrun.org.zone
           ssh root@ns.testrun.org systemctl reload nsd
 
       - name: rebuild staging2.testrun.org to have a clean VPS
@@ -66,21 +60,24 @@ jobs:
           rsync -e "ssh -o StrictHostKeyChecking=accept-new" -avz root@ns.testrun.org:/tmp/acme acme-restore || true
           rsync -avz root@ns.testrun.org:/tmp/dkimkeys dkimkeys-restore || true
           # restore acme & dkim state to staging2.testrun.org
-          rsync -avz acme-restore/acme root@staging2.testrun.org:/var/lib/ || true
-          rsync -avz dkimkeys-restore/dkimkeys root@staging2.testrun.org:/etc/ || true
+          rsync -avz acme-restore/acme/ root@staging2.testrun.org:/var/lib/acme || true
+          rsync -avz dkimkeys-restore/dkimkeys/ root@staging2.testrun.org:/etc/dkimkeys || true
           ssh -o StrictHostKeyChecking=accept-new -v root@staging2.testrun.org chown root:root -R /var/lib/acme || true
 
+      - name: run formatting checks 
+        run: cmdeploy fmt -v 
+
       - name: run deploy-chatmail offline tests 
         run: pytest --pyargs cmdeploy 
 
       - run: cmdeploy init staging2.testrun.org
 
-      - run: cmdeploy run --verbose --skip-dns-check
+      - run: cmdeploy run
 
       - name: set DNS entries
         run: |
-          ssh -o StrictHostKeyChecking=accept-new root@staging2.testrun.org chown opendkim:opendkim -R /etc/dkimkeys
-          cmdeploy dns --zonefile staging-generated.zone --verbose
+          ssh -o StrictHostKeyChecking=accept-new -v root@staging2.testrun.org chown opendkim:opendkim -R /etc/dkimkeys
+          cmdeploy dns --zonefile staging-generated.zone
           cat staging-generated.zone >> .github/workflows/staging.testrun.org-default.zone
           cat .github/workflows/staging.testrun.org-default.zone
           scp .github/workflows/staging.testrun.org-default.zone root@ns.testrun.org:/etc/nsd/staging2.testrun.org.zone
@@ -90,6 +87,6 @@ jobs:
       - name: cmdeploy test
         run: CHATMAIL_DOMAIN2=nine.testrun.org cmdeploy test --slow
 
-      - name: cmdeploy dns
-        run: cmdeploy dns -v
+      - name: cmdeploy dns (try 3 times)
+        run: cmdeploy dns || cmdeploy dns || cmdeploy dns
 

ARCHITECTURE.md

@@ -1,50 +0,0 @@
-This diagram shows components of the chatmail server; this is a draft
-overview as of mid-August 2025:
-
-```mermaid
-graph LR;
-    cmdeploy --- sshd;
-    letsencrypt --- |80|acmetool-redirector;
-    acmetool-redirector --- |443|nginx-right(["`nginx
-    (external)`"]);
-    nginx-external --- |465|postfix;
-    nginx-external(["`nginx
-    (external)`"]) --- |8443|nginx-internal["`nginx
-    (internal)`"];
-    nginx-internal --- website["`Website
-    /var/www/html`"];
-    nginx-internal --- newemail.py;
-    nginx-internal --- autoconfig.xml;
-    certs-nginx[("`TLS certs
-    /var/lib/acme`")] --> nginx-internal;
-    cron --- chatmail-metrics;
-    cron --- acmetool;
-    cron --- expunge;
-    chatmail-metrics --- website;
-    acmetool --> certs[("`TLS certs
-    /var/lib/acme`")];
-    nginx-external --- |993|dovecot;
-    autoconfig.xml --- postfix;
-    autoconfig.xml --- dovecot;
-    postfix --- echobot;
-    postfix --- |10080,10081|filtermail;
-    postfix --- users["`User data
-    home/vmail/mail`"];
-    postfix --- |doveauth.socket|doveauth;
-    dovecot --- |doveauth.socket|doveauth;
-    dovecot --- users;
-    dovecot --- |metadata.socket|chatmail-metadata;
-    doveauth --- users;
-    expunge --- users;
-    chatmail-metadata --- iroh-relay;
-    certs-nginx --> postfix;
-    certs-nginx --> dovecot;
-    style certs fill:#ff6;
-    style certs-nginx fill:#ff6;
-    style nginx-external fill:#fc9;
-    style nginx-right fill:#fc9;
-```
-
-The edges in this graph should not be taken too literally; they
-reflect some sort of communication path or dependency relationship
-between components of the chatmail server.

CHANGELOG.md

@@ -2,418 +2,78 @@
 
 ## untagged
 
-- filtermail: run CPU-intensive handle_DATA in a thread pool executor
-  ([#676](https://github.com/chatmail/relay/pull/676))
-
-- don't use the complicated logging module in filtermail to exclude a potential source of errors. 
-  ([#674](https://github.com/chatmail/relay/pull/674))
-
-- Specify nginx.conf to only handle `mail_domain`, www, and mta-sts domains
-  ([#636](https://github.com/chatmail/relay/pull/636))
-
-- Setup TURN server
-  ([#621](https://github.com/chatmail/relay/pull/621))
-
-- cmdeploy: make --ssh-host work with localhost
-  ([#659](https://github.com/chatmail/relay/pull/659))
-
-- Update iroh-relay to 0.35.0
-  ([#650](https://github.com/chatmail/relay/pull/650))
-
-- filtermail: accept mails from Protonmail
-  ([#616](https://github.com/chatmail/relay/pull/655))
-
-- Ignore all RCPT TO: parameters
-  ([#651](https://github.com/chatmail/relay/pull/651))
-
-- Increase opendkim DNS Timeout from 5 to 60 seconds
-  ([#672](https://github.com/chatmail/relay/pull/672))
-
-- Add config parameter for Let's Encrypt ACME email
-  ([#663](https://github.com/chatmail/relay/pull/663))
-
-- Use max username length in newemail.py, not min
-  ([#648](https://github.com/chatmail/relay/pull/648))
-
-- Add startup for `fcgiwrap.service` because sometimes it did not start automatically.
-  ([#657](https://github.com/chatmail/relay/pull/657))
-
-- Add `cmdeploy init --force` command for recreating chatmail.ini
-  ([#656](https://github.com/chatmail/relay/pull/656))
-
-- Increase maxproc for reinjecting ports from 10 to 100
-  ([#646](https://github.com/chatmail/relay/pull/646))
-
-- Allow ports 143 and 993 to be used by `dovecot` process
-  ([#639](https://github.com/chatmail/relay/pull/639))
-
-- Add `--skip-dns-check` argument to `cmdeploy run` command, which disables DNS record checking before installation.
-  ([#661](https://github.com/chatmail/relay/pull/661))
-
-## 1.7.0 2025-09-11
-
-- Make www upload path configurable
-  ([#618](https://github.com/chatmail/relay/pull/618))
-
-- Check whether GCC is installed in initenv.sh
-  ([#608](https://github.com/chatmail/relay/pull/608))
-
-- Expire push notification tokens after 90 days
-  ([#583](https://github.com/chatmail/relay/pull/583))
-
-- Use official `mtail` binary instead of `mtail` package
-  ([#581](https://github.com/chatmail/relay/pull/581))
-
-- dovecot: install from download.delta.chat instead of openSUSE Build Service
-  ([#590](https://github.com/chatmail/relay/pull/590))
-
-- Reconfigure Dovecot imap-login service to high-performance mode
-  ([#578](https://github.com/chatmail/relay/pull/578))
-
-- Set timezone to improve dovecot performance
-  ([#584](https://github.com/chatmail/relay/pull/584))
-
-- Increase nginx connection limits
-  ([#576](https://github.com/chatmail/relay/pull/576))
-
-- If `dns-utils` needs to be installed before cmdeploy run, apt update to make sure it works
-  ([#560](https://github.com/chatmail/relay/pull/560))
-
-- filtermail: respect config message size limit
-  ([#572](https://github.com/chatmail/relay/pull/572))
-
-- Don't deploy if one of the ports used for chatmail relay services is occupied by an unexpected process
-  ([#568](https://github.com/chatmail/relay/pull/568))
-
-- Add config value after how many days large files are deleted
-  ([#555](https://github.com/chatmail/relay/pull/555))
-
-- cmdeploy: push relay version to /etc/chatmail-version
-  ([#573](https://github.com/chatmail/relay/pull/573))
-
-- filtermail: allow partial body length in OpenPGP payloads
-  ([#570](https://github.com/chatmail/relay/pull/570))
-
-- chatmaild: allow echobot to receive unencrypted messages by default
-  ([#556](https://github.com/chatmail/relay/pull/556))
-
-
-## 1.6.0 2025-04-11
-
-- Handle Port-25 connect errors more gracefully (common with VPNs)
-  ([#552](https://github.com/chatmail/relay/pull/552))
-
-- Avoid "acmetool not found" during initial run
-  ([#550](https://github.com/chatmail/relay/pull/550))
-
-- Fix timezone handling such that client/servers do not need to use
-  same timezone. 
-  ([#553](https://github.com/chatmail/relay/pull/553))
-
-- Enforce end-to-end encryption for incoming messages. 
-  New user address mailboxes now get a `enforceE2EEincoming` file 
-  which prohibits incoming cleartext messages from other domains. 
-  An outside MTA trying to submit a cleartext message will 
-  get a "523 Encryption Needed" response, see RFC5248. 
-  If the file does not exist (as it the case for all existing accounts) 
-  incoming cleartext messages are accepted. 
-  ([#538](https://github.com/chatmail/server/pull/538))
-
-- Enforce end-to-end encryption between local addresses 
-  ([#535](https://github.com/chatmail/server/pull/535))
-
-- unbound: check that port 53 is not occupied by a different process
-  ([#537](https://github.com/chatmail/server/pull/537))
-
-- unbound: before unbound is there, use 9.9.9.9 for resolving
-  ([#518](https://github.com/chatmail/relay/pull/518))
-
-- Limit the bind for the HTTPS server on 8443 to 127.0.0.1 
-  ([#522](https://github.com/chatmail/server/pull/522))
-  ([#532](https://github.com/chatmail/server/pull/532))
-
-- Send SNI when connecting to outside servers
-  ([#524](https://github.com/chatmail/server/pull/524))
-
-- postfix master.cf: use 127.0.0.1 for consistency
-  ([#544](https://github.com/chatmail/relay/pull/544))
-
-- Pass through `original_content` instead of `content` in filtermail
-  ([#509](https://github.com/chatmail/server/pull/509))
-
-- Document TLS requirements in the readme
-  ([#514](https://github.com/chatmail/server/pull/514))
-
-- Remove cleanup service from submission ports
-  ([#512](https://github.com/chatmail/server/pull/512))
-
-- cmdeploy dovecot: delete big messages after 7 days
-  ([#504](https://github.com/chatmail/server/pull/504))
-
-- mtail: fix getting logs from STDIN
-  ([#502](https://github.com/chatmail/server/pull/502))
-
-- filtermail: don't require exactly 2 lines after openPGP payload
-  ([#497](https://github.com/chatmail/server/pull/497))
-
-- cmdeploy dns: offer alternative DKIM record format for some web interfaces
-  ([#470](https://github.com/chatmail/server/pull/470))
-
-- journald: remove old logs from disk
-  ([#490](https://github.com/chatmail/server/pull/490))
-
-- opendkim: restart once every day to mend RAM leaks
-  ([#498](https://github.com/chatmail/server/pull/498)
-
-- migration guide: let opendkim own the DKIM keys directory
-  ([#468](https://github.com/chatmail/server/pull/468))
-
-- improve secure-join message detection
-  ([#473](https://github.com/chatmail/server/pull/473))
-
-- use old crypt lib in python < 3.11
-  ([#483](https://github.com/chatmail/server/pull/483))
-
-- chatmaild: set umask to 0700 for doveauth + metadata
-  ([#490](https://github.com/chatmail/server/pull/492))
-
-- remove MTA-STS daemon
-  ([#488](https://github.com/chatmail/server/pull/488))
-
-- replace `Subject` with `[...]` for all outgoing mails.
-  ([#481](https://github.com/chatmail/server/pull/481))
-
-- opendkim: use su instead of sudo
-  ([#491](https://github.com/chatmail/server/pull/491))
-
-## 1.5.0 2024-12-20
-
-- cmdeploy dns: always show recommended DNS records
-  ([#463](https://github.com/chatmail/server/pull/463))
-
-- add `--all` to `cmdeploy dns`
-  ([#462](https://github.com/chatmail/server/pull/462))
-
-- fix `_mta-sts` TXT DNS record
-  ([#461](https://github.com/chatmail/server/pull/461)
-
-- deploy `iroh-relay` and also update "realtime relay services" in privacy policy. 
-  ([#434](https://github.com/chatmail/server/pull/434))
-  ([#451](https://github.com/chatmail/server/pull/451))
-
-- add guide to migrate chatmail to a new server
-  ([#429](https://github.com/chatmail/server/pull/429))
-
-- disable anvil authentication penalty
-  ([#414](https://github.com/chatmail/server/pull/444)
-
-- increase `request_queue_size` for UNIX sockets to 1000.
-  ([#437](https://github.com/chatmail/server/pull/437))
-
-- add argument to `cmdeploy run` for specifying
-  a different SSH host than `mail_domain`
-  ([#439](https://github.com/chatmail/server/pull/439))
-
-- query autoritative nameserver to bypass DNS cache
-  ([#424](https://github.com/chatmail/server/pull/424))
-
-- add mtail support (new optional `mtail_address` ini value)
-  This defines the address on which [`mtail`](https://google.github.io/mtail/)
-  exposes its metrics collected from the logs.
-  If you want to collect the metrics with Prometheus,
-  setup a private network (e.g. WireGuard interface)
-  and assign an IP address from this network to the host.
-  If you do not plan to collect metrics,
-  keep this setting unset.
-  ([#388](https://github.com/chatmail/server/pull/388))
-
-- fix checking for required DNS records
-  ([#412](https://github.com/chatmail/server/pull/412))
-
-- add support for specifying whole domains for recipient passthrough list
-  ([#408](https://github.com/chatmail/server/pull/408))
-
-- add a paragraph about "account deletion" to info page 
-  ([#405](https://github.com/chatmail/server/pull/405))
-
-- avoid nginx listening on ipv6 if v6 is dsiabled 
-  ([#402](https://github.com/chatmail/server/pull/402))
-
-- refactor ssh-based execution to allow organizing remote functions in
-  modules. 
-  ([#396](https://github.com/chatmail/server/pull/396))
-
-- trigger "apt upgrade" during "cmdeploy run" 
-  ([#398](https://github.com/chatmail/server/pull/398))
-
-- drop hispanilandia passthrough address
-  ([#401](https://github.com/chatmail/server/pull/401))
-
-- set CAA record flags to 0
-
-- add IMAP capabilities instead of overwriting them
-  ([#413](https://github.com/chatmail/server/pull/413))
-
-- fix OpenPGP payload check
-  ([#435](https://github.com/chatmail/server/pull/435))
-
-- fix Dovecot quota_max_mail_size to use max_message_size config value
-  ([#438](https://github.com/chatmail/server/pull/438))
-
-
-## 1.4.1 2024-07-31
-
-- fix metadata dictproxy which would confuse transactions
-  resulting in missed notifications and other issues. 
-  ([#393](https://github.com/chatmail/server/pull/393))
-  ([#394](https://github.com/chatmail/server/pull/394))
-
-- add optional "imap_rawlog" config option. If true, 
-  .in/.out files are created in user home dirs 
-  containing the imap protocol messages. 
-  ([#389](https://github.com/chatmail/server/pull/389))
-
-## 1.4.0 2024-07-28
-
-- Add `disable_ipv6` config option to chatmail.ini.
-  Required if the server doesn't have IPv6 connectivity.
-  ([#312](https://github.com/chatmail/server/pull/312))
-
-- allow current K9/Thunderbird-mail releases to send encrypted messages
-  outside by accepting their localized "encrypted subject" strings. 
-  ([#370](https://github.com/chatmail/server/pull/370))
-
-- Migrate and remove sqlite database in favor of password/lastlogin tracking 
-  in a user's maildir.  
-  ([#379](https://github.com/chatmail/server/pull/379))
-
-- Require pyinfra V3 installed on the client side,
-  run `./scripts/initenv.sh` to upgrade locally.
-  ([#378](https://github.com/chatmail/server/pull/378))
-
-- don't hardcode "/home/vmail" paths but rather set them 
-  once in the config object and use it everywhere else, 
-  thereby also improving testability.  
-  ([#351](https://github.com/chatmail/server/pull/351))
-  temporarily introduced obligatory "passdb_path" and "mailboxes_dir" 
-  settings but they were removed/obsoleted in 
-  ([#380](https://github.com/chatmail/server/pull/380))
-
-- BREAKING: new required chatmail.ini value 'delete_inactive_users_after = 100'
-  which removes users from database and mails after 100 days without any login. 
-  ([#350](https://github.com/chatmail/server/pull/350))
-
-- Refine DNS checking to distinguish between "required" and "recommended" settings 
-  ([#372](https://github.com/chatmail/server/pull/372))
-
-- reload nginx in the acmetool cronjob
-  ([#360](https://github.com/chatmail/server/pull/360))
-
-- remove checking of reverse-DNS PTR records.  Chatmail-servers don't
-  depend on it and even in the wider e-mail system it's not common anymore. 
-  If it's an issue, a chatmail operator can still care to properly set reverse DNS. 
-  ([#348](https://github.com/chatmail/server/pull/348))
-
-- Make DNS-checking faster and more interactive, run it fully during "cmdeploy run",
-  also introducing a generic mechanism for rapid remote ssh-based python function execution. 
-  ([#346](https://github.com/chatmail/server/pull/346))
-
-- Don't fix file owner ship of /home/vmail 
-  ([#345](https://github.com/chatmail/server/pull/345))
-
-- Support iterating over all users with doveadm commands 
-  ([#344](https://github.com/chatmail/server/pull/344))
-
 - Test and fix for attempts to create inadmissible accounts 
-  ([#333](https://github.com/chatmail/server/pull/321))
+  ([#333](https://github.com/deltachat/chatmail/pull/321))
 
 - check that OpenPGP has only PKESK, SKESK and SEIPD packets
-  ([#323](https://github.com/chatmail/server/pull/323),
-   [#324](https://github.com/chatmail/server/pull/324))
+  ([#323](https://github.com/deltachat/chatmail/pull/323),
+   [#324](https://github.com/deltachat/chatmail/pull/324))
 
 - improve filtermail checks for encrypted messages and drop support for unencrypted MDNs
-  ([#320](https://github.com/chatmail/server/pull/320))
+  ([#320](https://github.com/deltachat/chatmail/pull/320))
 
 - replace `bash` with `/bin/sh`
-  ([#334](https://github.com/chatmail/server/pull/334))
+  ([#334](https://github.com/deltachat/chatmail/pull/334))
 
 - Increase number of logged in IMAP sessions to 50000
-  ([#335](https://github.com/chatmail/server/pull/335))
+  ([#335](https://github.com/deltachat/chatmail/pull/335))
 
 - filtermail: do not allow ASCII armor without actual payload
-  ([#325](https://github.com/chatmail/server/pull/325))
-
-- Remove sieve to enable hardlink deduplication in LMTP
-  ([#343](https://github.com/chatmail/server/pull/343))
-
-- dovecot: enable gzip compression on disk
-  ([#341](https://github.com/chatmail/server/pull/341))
-
-- DKIM-sign Content-Type and oversign all signed headers
-  ([#296](https://github.com/chatmail/server/pull/296))
-
-- Add nonci_accounts metric
-  ([#347](https://github.com/chatmail/server/pull/347))
-
-- doveauth: log when a new account is created
-  ([#349](https://github.com/chatmail/server/pull/349))
-
-- Multiplex HTTPS, IMAP and SMTP on port 443
-  ([#357](https://github.com/chatmail/server/pull/357))
+  ([#325](https://github.com/deltachat/chatmail/pull/325))
 
 ## 1.3.0 - 2024-06-06
 
 - don't check necessary DNS records on cmdeploy init anymore
-  ([#316](https://github.com/chatmail/server/pull/316))
+  ([#316](https://github.com/deltachat/chatmail/pull/316))
 
 - ensure cron and acl are installed
-  ([#293](https://github.com/chatmail/server/pull/293),
-  [#310](https://github.com/chatmail/server/pull/310))
+  ([#293](https://github.com/deltachat/chatmail/pull/293),
+  [#310](https://github.com/deltachat/chatmail/pull/310))
 
 - change default for delete_mails_after from 40 to 20 days
-  ([#300](https://github.com/chatmail/server/pull/300))
+  ([#300](https://github.com/deltachat/chatmail/pull/300))
 
 - save journald logs only to memory and save nginx logs to journald instead of file
-  ([#299](https://github.com/chatmail/server/pull/299))
+  ([#299](https://github.com/deltachat/chatmail/pull/299))
 
 - fix writing of multiple obs repositories in `/etc/apt/sources.list`
-  ([#290](https://github.com/chatmail/server/pull/290))
+  ([#290](https://github.com/deltachat/chatmail/pull/290))
 
 - metadata: add support for `/shared/vendor/deltachat/irohrelay`
-  ([#284](https://github.com/chatmail/server/pull/284))
+  ([#284](https://github.com/deltachat/chatmail/pull/284))
 
 - Emit "XCHATMAIL" capability from IMAP server 
-  ([#278](https://github.com/chatmail/server/pull/278))
+  ([#278](https://github.com/deltachat/chatmail/pull/278))
 
 - Move echobot `into /var/lib/echobot`
-  ([#281](https://github.com/chatmail/server/pull/281))
+  ([#281](https://github.com/deltachat/chatmail/pull/281))
 
 - Accept Let's Encrypt's new Terms of Services
-  ([#275](https://github.com/chatmail/server/pull/276))
+  ([#275](https://github.com/deltachat/chatmail/pull/276))
 
 - Reload Dovecot and Postfix when TLS certificate updates
-  ([#271](https://github.com/chatmail/server/pull/271))
+  ([#271](https://github.com/deltachat/chatmail/pull/271))
 
 - Use forked version of dovecot without hardcoded delays
-  ([#270](https://github.com/chatmail/server/pull/270))
+  ([#270](https://github.com/deltachat/chatmail/pull/270))
 
 ## 1.2.0 - 2024-04-04
 
 - Install dig on the server to resolve DNS records
-  ([#267](https://github.com/chatmail/server/pull/267))
+  ([#267](https://github.com/deltachat/chatmail/pull/267))
 
 - preserve notification order and exponentially backoff with 
   retries for tokens where we didn't get a successful return
-  ([#265](https://github.com/chatmail/server/pull/263))
+  ([#265](https://github.com/deltachat/chatmail/pull/263))
 
 - Run chatmail-metadata and doveauth as vmail
-  ([#261](https://github.com/chatmail/server/pull/261))
+  ([#261](https://github.com/deltachat/chatmail/pull/261))
 
 - Apply systemd restrictions to echobot
-  ([#259](https://github.com/chatmail/server/pull/259))
+  ([#259](https://github.com/deltachat/chatmail/pull/259))
 
 - re-enable running the CI in pull requests, but not concurrently 
-  ([#258](https://github.com/chatmail/server/pull/258))
+  ([#258](https://github.com/deltachat/chatmail/pull/258))
 
 
 ## 1.1.0 - 2024-03-28
@@ -421,27 +81,27 @@
 ### The changelog starts to record changes from March 15th, 2024 
 
 - Move systemd unit templates to cmdeploy package 
-  ([#255](https://github.com/chatmail/server/pull/255))
+  ([#255](https://github.com/deltachat/chatmail/pull/255))
 
 - Persist push tokens and support multiple device per address 
-  ([#254](https://github.com/chatmail/server/pull/254))
+  ([#254](https://github.com/deltachat/chatmail/pull/254))
 
 - Avoid warning for regular doveauth protocol's hello message. 
-  ([#250](https://github.com/chatmail/server/pull/250))
+  ([#250](https://github.com/deltachat/chatmail/pull/250))
 
 - Fix various tests to pass again with "cmdeploy test". 
-  ([#245](https://github.com/chatmail/server/pull/245),
-  [#242](https://github.com/chatmail/server/pull/242)
+  ([#245](https://github.com/deltachat/chatmail/pull/245),
+  [#242](https://github.com/deltachat/chatmail/pull/242)
 
 - Ensure lets-encrypt certificates are reloaded after renewal 
-  ([#244]) https://github.com/chatmail/server/pull/244
+  ([#244]) https://github.com/deltachat/chatmail/pull/244
 
 - Persist tokens to avoid iOS users loosing push-notifications when the
   chatmail metadata service is restarted (happens regularly during deploys)
-  ([#238](https://github.com/chatmail/server/pull/239)
+  ([#238](https://github.com/deltachat/chatmail/pull/239)
 
 - Fix failing sieve-script compile errors on incoming messages
-  ([#237](https://github.com/chatmail/server/pull/239)
+  ([#237](https://github.com/deltachat/chatmail/pull/239)
 
 - Fix quota reporting after expunging of old mails
-  ([#233](https://github.com/chatmail/server/pull/239)
+  ([#233](https://github.com/deltachat/chatmail/pull/239)

README.md

@@ -1,105 +1,57 @@
 
 <img width="800px" src="www/src/collage-top.png"/>
 
-# Chatmail relays for end-to-end encrypted e-mail
+# Chatmail services optimized for Delta Chat apps 
 
-Chatmail relay servers are interoperable Mail Transport Agents (MTAs) designed for: 
-
-- **Convenience:** Low friction instant onboarding
-
-- **Privacy:** No name, phone numbers, email required or collected 
-
-- **End-to-End Encryption enforced**: only OpenPGP messages with metadata minimization allowed 
-
-- **Instant:** Privacy-preserving Push Notifications for Apple, Google, and Huawei
-
-- **Speed:** Message delivery in half a second, with optional P2P realtime connections 
-
-- **Transport Security:** Strict TLS and DKIM enforced 
-
-- **Reliability:** No spam or IP reputation checks; rate-limits are suitable for realtime chats
-
-- **Efficiency:** Messages are only stored for transit and removed automatically
-
-This repository contains everything needed to setup a ready-to-use chatmail relay
+This repository helps to setup a ready-to-use chatmail server
 comprised of a minimal setup of the battle-tested 
-[Postfix SMTP](https://www.postfix.org) and [Dovecot IMAP](https://www.dovecot.org) MTAs/MDAs.
+[postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services. 
 
-The automated setup is designed and optimized for providing chatmail addresses
-for immediate permission-free onboarding through chat apps and bots.
-Chatmail addresses are automatically created at first login,
-after which the initially specified password is required
-for sending and receiving messages through them.
+The setup is designed and optimized for providing chatmail accounts 
+for use by [Delta Chat apps](https://delta.chat).
 
-Please see [this list of known apps and client projects](https://chatmail.at/clients.html) 
-and [this list of known public 3rd party chatmail relay servers](https://chatmail.at/relays).
+Chatmail accounts are automatically created by a first login, 
+after which the initially specified password is required for using them. 
 
+## Deploying your own chatmail server 
 
-## Minimal requirements, Prerequisites 
+To deploy chatmail on your own server, you must have set-up ssh authentication and need to use an ed25519 key, due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191). You also need to add your private key to the local ssh-agent, because you can't type in your password during deployment.
 
-You will need the following: 
-
-- Control over a domain through a DNS provider of your choice.
-
-- A Debian 12 server with reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports.
-  IPv6 is encouraged if available.
-  Chatmail relay servers only require 1GB RAM, one CPU, and perhaps 10GB storage for a
-  few thousand active chatmail addresses.
-
-- Key-based SSH authentication to the root user.
-  You must add a passphrase-protected private key to your local ssh-agent
-  because you can't type in your passphrase during deployment.
-  (An ed25519 private key is required due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191))
-
-
-## Getting started 
-
-We use `chat.example.org` as the chatmail domain in the following steps.
+We use `chat.example.org` as the chatmail domain in the following steps. 
 Please substitute it with your own domain. 
 
-1. Setup the initial DNS records.
-   The following is an example in the familiar BIND zone file format with
-   a TTL of 1 hour (3600 seconds).
-   Please substitute your domain and IP addresses.
+1. Install the `cmdeploy` command in a virtualenv
 
    ```
-    chat.example.com. 3600 IN A 198.51.100.5
-    chat.example.com. 3600 IN AAAA 2001:db8::5
-    www.chat.example.com. 3600 IN CNAME chat.example.com.
-    mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
-   ```
-
-2. On your local PC, clone the repository and bootstrap the Python virtualenv.
-
-   ```
-    git clone https://github.com/chatmail/relay
-    cd relay
+    git clone https://github.com/deltachat/chatmail
+    cd chatmail
     scripts/initenv.sh
    ```
 
-3. On your local PC, create chatmail configuration file `chatmail.ini`:
+2. Create chatmail configuration file `chatmail.ini`:
 
    ```
     scripts/cmdeploy init chat.example.org  # <-- use your domain 
    ```
 
-4. Verify that SSH root login to your remote server works:
+3. Setup first DNS records for your chatmail domain, 
+   according to the hints provided by `cmdeploy init`.
+   Verify that SSH root login works:
 
    ```
-    ssh root@chat.example.org  # <-- use your domain 
+    ssh root@chat.example.org   # <-- use your domain 
    ```
 
-5. From your local PC, deploy the remote chatmail relay server:
+4. Deploy to the remote chatmail server:
 
    ```
     scripts/cmdeploy run
    ```
-   This script will also check that you have all necessary DNS records.
-   If DNS records are missing, it will recommend
+   This script will also show you additional DNS records
    which you should configure at your DNS provider
    (it can take some time until they are public).
 
-### Other helpful commands
+### Other helpful commands:
 
 To check the status of your remotely running chatmail service:
 
@@ -107,7 +59,7 @@ To check the status of your remotely running chatmail service:
 scripts/cmdeploy status
 ```
 
-To display and check all recommended DNS records:
+To check whether your DNS records are correct:
 
 ```
 scripts/cmdeploy dns
@@ -127,113 +79,43 @@ scripts/cmdeploy bench
 
 ## Overview of this repository
 
-This repository has four directories:
+This repository drives the development of chatmail services, 
+comprised of minimal setups of
 
-- [cmdeploy](https://github.com/chatmail/relay/tree/main/cmdeploy)
-  is a collection of configuration files
-  and a [pyinfra](https://pyinfra.com)-based deployment script.
+- [postfix smtp server](https://www.postfix.org)
+- [dovecot imap server](https://www.dovecot.org)
 
-- [chatmaild](https://github.com/chatmail/relay/tree/main/chatmaild)
-  is a Python package containing several small services
-  which handle authentication,
-  trigger push notifications on new messages,
-  ensure that outbound mails are encrypted,
-  delete inactive users,
-  and some other minor things.
-  chatmaild can also be installed as a stand-alone Python package.
+as well as custom services that are integrated with these two:
 
-- [www](https://github.com/chatmail/relay/tree/main/www)
-  contains the html, css, and markdown files
-  which make up a chatmail relay's web page.
-  Edit them before deploying to make your chatmail relay stand out.
-
-- [scripts](https://github.com/chatmail/relay/tree/main/scripts)
-  offers two convenience tools for beginners;
-  `initenv.sh` installs the necessary dependencies to a local virtual environment,
-  and the `scripts/cmdeploy` script enables you
-  to run the `cmdeploy` command line tool in the local virtual environment.
-
-### cmdeploy
-
-The `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
-helps with setting up and managing the chatmail service.
-`cmdeploy init` creates the `chatmail.ini` config file.
-`cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [`script`](cmdeploy/src/cmdeploy/__init__.py)
-to automatically install or upgrade all chatmail components on a relay,
-according to the `chatmail.ini` config.
-
-The components of chatmail are:
-
-- [Postfix SMTP MTA](https://www.postfix.org) accepts and relays messages
-  (both from your users and from the wider e-mail MTA network)
-
-- [Dovecot IMAP MDA](https://www.dovecot.org) stores messages for your users until they download them
-
-- [Nginx](https://nginx.org/) shows the web page with your privacy policy and additional information
-
-- [acmetool](https://hlandau.github.io/acmetool/) manages TLS certificates for Dovecot, Postfix, and Nginx
-
-- [OpenDKIM](http://www.opendkim.org/) for signing messages with DKIM and rejecting inbound messages without DKIM
-
-- [mtail](https://google.github.io/mtail/) for collecting anonymized metrics in case you have monitoring
-
-- [Iroh relay](https://www.iroh.computer/docs/concepts/relay) 
-  which helps client devices to establish Peer-to-Peer connections 
-
-- and the chatmaild services, explained in the next section:
-
-### chatmaild
-
-`chatmaild` implements various systemd-controlled services  
-that integrate with Dovecot and Postfix to achieve instant-onboarding and 
-only relaying OpenPGP end-to-end messages encrypted messages. 
-A short overview of `chatmaild` services: 
-
-- [`doveauth`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/doveauth.py) 
-  implements create-on-login address semantics and is used 
-  by Dovecot during IMAP login and by Postfix during SMTP/SUBMISSION login
+- `chatmaild/src/chatmaild/doveauth.py` implements
+  create-on-login account creation semantics and is used
+  by Dovecot during login authentication and by Postfix
   which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
-  to authenticate logins. 
+  to authenticate users
+  to send mails for them.
 
-- [`filtermail`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/filtermail.py) 
-  prevents unencrypted email from leaving or entering the chatmail service
-  and is integrated into Postfix's outbound and inbound mail pipelines.
+- `chatmaild/src/chatmaild/filtermail.py` prevents
+  unencrypted e-mail from leaving the chatmail service
+  and is integrated into postfix's outbound mail pipelines.
 
-- [`chatmail-metadata`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
-  [Dovecot lua script](https://github.com/chatmail/relay/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
-  to store user-specific relay-side config.
-  On new messages,
-  it [passes the user's push notification token](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/notifier.py)
-  to [notifications.delta.chat](https://delta.chat/help#instant-delivery)
-  so the push notifications on the user's phone can be triggered
-  by Apple/Google/Huawei.
+There is also the `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
+which helps with setting up and managing the chatmail service.
+`cmdeploy run` uses [pyinfra-based scripting](https://pyinfra.com/)
+in `cmdeploy/src/cmdeploy/__init__.py`
+to automatically install all chatmail components on a server.
 
-- [`delete_inactive_users`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
-  deletes users if they have not logged in for a very long time.
-  The timeframe can be configured in `chatmail.ini`.
-
-- [`lastlogin`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/lastlogin.py)
-  is contacted by Dovecot when a user logs in
-  and stores the date of the login.
-
-- [`echobot`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/echo.py)
-  is a small bot for test purposes.
-  It simply echoes back messages from users.
-
-- [`chatmail-metrics`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metrics.py)
-  collects some metrics and displays them at `https://example.org/metrics`.
 
 ### Home page and getting started for users
 
-`cmdeploy run` also creates default static web pages and deploys them
-to a Nginx web server with:
+`cmdeploy run` also creates default static Web pages and deploys them
+to a nginx web server with: 
 
 - a default `index.html` along with a QR code that users can click to
-  create an address on your chatmail relay 
+  create accounts on your chatmail provider,
 
-- a default `info.html` that is linked from the home page
+- a default `info.html` that is linked from the home page,
 
-- a default `policy.html` that is linked from the home page
+- a default `policy.html` that is linked from the home page.
 
 All `.html` files are generated
 by the according markdown `.md` file in the `www/src` directory.
@@ -241,76 +123,47 @@ by the according markdown `.md` file in the `www/src` directory.
 
 ### Refining the web pages
 
+
 ```
 scripts/cmdeploy webdev
 ```
 
-This starts a local live development cycle for chatmail web pages:
+This starts a local live development cycle for chatmail Web pages:
 
 - uses the `www/src/page-layout.html` file for producing static
   HTML pages from `www/src/*.md` files
 
 - continously builds the web presence reading files from `www/src` directory
-  and generating HTML files and copying assets to the `www/build` directory.
+  and generating html files and copying assets to the `www/build` directory.
 
 - Starts a browser window automatically where you can "refresh" as needed.
 
-#### Custom web pages
 
-You can skip uploading a web page
-by setting `www_folder=disabled` in `chatmail.ini`.
+## Emergency Commands to disable automatic account creation
 
-If you want to manage your web pages outside this git repository,
-you can set `www_folder` in `chatmail.ini` to a custom directory on your computer.
-`cmdeploy run` will upload it as the server's home page,
-and if it contains a `src/index.md` file,
-will build it with hugo.
-
-
-## Mailbox directory layout
-
-Fresh chatmail addresses have a mailbox directory that contains: 
-
-- a `password` file with the salted password required for authenticating
-  whether a login may use the address to send/receive messages. 
-  If you modify the password file manually, you effectively block the user. 
-
-- `enforceE2EEincoming` is a default-created file with each address. 
-  If present the file indicates that this chatmail address rejects incoming cleartext messages.
-  If absent the address accepts incoming cleartext messages. 
-
-- `dovecot*`, `cur`, `new` and `tmp` represent IMAP/mailbox state. 
-  If the address is only used by one device, the Maildir directories
-  will typically be empty unless the user of that address hasn't been online 
-  for a while. 
-
-
-## Emergency Commands to disable automatic address creation
-
-If you need to stop address creation,
-e.g. because some script is wildly creating addresses, 
-login with ssh and run:
+If you need to stop account creation,
+e.g. because some script is wildly creating accounts,
+login to the server with ssh and run:
 
 ```
     touch /etc/chatmail-nocreate
 ```
 
-Chatmail address creation will be denied while this file is present.
+While this file is present, account creation will be blocked.
 
 ### Ports
 
-[Postfix](http://www.postfix.org/) listens on ports 25 (SMTP) and 587 (SUBMISSION) and 465 (SUBMISSIONS).
-[Dovecot](https://www.dovecot.org/) listens on ports 143 (IMAP) and 993 (IMAPS).
-[Nginx](https://www.nginx.com/) listens on port 8443 (HTTPS-ALT) and 443 (HTTPS).
-Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
-[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (HTTP).
+[Postfix](http://www.postfix.org/) listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
+[Dovecot](https://www.dovecot.org/) listens on ports 143 (imap) and 993 (imaps).
+[nginx](https://www.nginx.com/) listens on port 443 (https).
+[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (http).
 
-chatmail-core based apps will, however, discover all ports and configurations
-automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail relay server.
+Delta Chat apps will, however, discover all ports and configurations
+automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail service.
 
 ## Email authentication
 
-Chatmail relays enforce [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
+chatmail servers rely on [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
 to authenticate incoming emails.
 Incoming emails must have a valid DKIM signature with
 Signing Domain Identifier (SDID, `d=` parameter in the DKIM-Signature header)
@@ -332,227 +185,3 @@ to MAIL FROM with
 and rejects incorrectly authenticated emails with [`reject_sender_login_mismatch`](reject_sender_login_mismatch) policy.
 `From:` header must correspond to envelope MAIL FROM,
 this is ensured by `filtermail` proxy.
-
-## TLS requirements
-
-Postfix is configured to require valid TLS
-by setting [`smtp_tls_security_level`](https://www.postfix.org/postconf.5.html#smtp_tls_security_level) to `verify`.
-If emails don't arrive at your chatmail relay server, 
-the problem is likely that your relay does not have a valid TLS certificate.
-
-You can test it by resolving `MX` records of your relay domain
-and then connecting to MX relays (e.g `mx.example.org`) with
-`openssl s_client -connect mx.example.org:25 -verify_hostname mx.example.org -verify_return_error -starttls smtp`
-from the host that has open port 25 to verify that certificate is valid.
-
-When providing a TLS certificate to your chatmail relay server, 
-make sure to provide the full certificate chain
-and not just the last certificate.
-
-If you are running an Exim server and don't see incoming connections
-from a chatmail relay server in the logs,
-make sure `smtp_no_mail` log item is enabled in the config
-with `log_selector = +smtp_no_mail`.
-By default Exim does not log sessions that are closed
-before sending the `MAIL` command.
-This happens if certificate is not recognized as valid by Postfix,
-so you might think that connection is not established
-while actually it is a problem with your TLS certificate.
-
-## Migrating a chatmail relay to a new host
-
-If you want to migrate chatmail relay from an old machine
-to a new machine,
-you can use these steps.
-They were tested with a Linux laptop;
-you might need to adjust some of the steps to your environment.
-
-Let's assume that your `mail_domain` is `mail.example.org`,
-all involved machines run Debian 12,
-your old site's IP address is `13.37.13.37`,
-and your new site's IP address is `13.12.23.42`.
-
-Note, you should lower the TTLs of your DNS records to a value
-such as 300 (5 minutes) so the migration happens as smoothly as possible.
-
-During the guide you might get a warning about changed SSH Host keys;
-in this case, just run `ssh-keygen -R "mail.example.org"` as recommended.
-
-1. First, disable mail services on the old site. 
-
-   ```
-    cmdeploy run --disable-mail --ssh-host 13.37.13.37
-   ```
-
-   Now your users will notice the migration
-   and will not be able to send or receive messages
-   until the migration is completed.
-
-2. Now we want to copy `/home/vmail`, `/var/lib/acme`, `/etc/dkimkeys`, `/run/echobot`, and `/var/spool/postfix` to the new site. 
-   Login to the old site while forwarding your SSH agent 
-   so you can copy directly from the old to the new site with your SSH key:
-   ```
-    ssh -A root@13.37.13.37
-    tar c - /home/vmail/mail /var/lib/acme /etc/dkimkeys /run/echobot /var/spool/postfix | ssh root@13.12.23.42 "tar x -C /"
-   ```
-
-   This transfers all addresses, the TLS certificate, DKIM keys (so DKIM DNS record remains valid), and the echobot's password so it continues to function.
-   It also preserves the Postfix mail spool so any messages pending delivery will still be delivered.
-
-3. Install chatmail on the new machine:
-
-   ```
-    cmdeploy run --disable-mail --ssh-host 13.12.23.42
-   ```
-   Postfix and Dovecot are disabled for now; we will enable them later. 
-   We first need to make the new site fully operational. 
-
-3. On the new site, run the following to ensure the ownership is correct in case UIDs/GIDs changed:
-
-   ```
-    chown root: -R /var/lib/acme
-    chown opendkim: -R /etc/dkimkeys
-    chown vmail: -R /home/vmail/mail
-    chown echobot: -R /run/echobot
-   ```
-
-4. Now, update DNS entries. 
-
-   If other MTAs try to deliver messages to your chatmail domain they may fail intermittently,
-   as DNS catches up with the new site settings 
-   but normally will retry delivering messages
-   for at least a week, so messages will not be lost.
-
-5. Finally, you can execute `cmdeploy run --ssh-host 13.12.23.42` to turn on chatmail on the new relay.
-   Your users will be able to use the chatmail relay as soon as the DNS changes have propagated.
-   Voilà!
-
-## Setting up a reverse proxy
-
-A chatmail relay MTA does not track or depend on the client IP address
-for its operation, so it can be run behind a reverse proxy.
-This will not even affect incoming mail authentication
-as DKIM only checks the cryptographic signature
-of the message and does not use the IP address as the input.
-
-For example, you may want to self-host your chatmail relay
-and only use hosted VPS to provide a public IP address
-for client connections and incoming mail.
-You can connect chatmail relay to VPS
-using a tunnel protocol
-such as [WireGuard](https://www.wireguard.com/)
-and setup a reverse proxy on a VPS
-to forward connections to the chatmail relay
-over the tunnel.
-You can also setup multiple reverse proxies
-for your chatmail relay in different networks
-to ensure your relay is reachable even when
-one of the IPs becomes inaccessible due to
-hosting or routing problems.
-
-Note that your chatmail relay still needs
-to be able to make outgoing connections on port 25
-to send messages outside.
-
-To setup a reverse proxy
-(or rather Destination NAT, DNAT)
-for your chatmail relay,
-put the following configuration in `/etc/nftables.conf`:
-```
-#!/usr/sbin/nft -f
-
-flush ruleset
-
-define wan = eth0
-
-# Which ports to proxy.
-#
-# Note that SSH is not proxied
-# so it is possible to log into the proxy server 
-# and not the original one.
-define ports = { smtp, http, https, imap, imaps, submission, submissions }
-
-# The host we want to proxy to.
-define ipv4_address = AAA.BBB.CCC.DDD
-define ipv6_address = [XXX::1]
-
-table ip nat {
-        chain prerouting {
-                type nat hook prerouting priority dstnat; policy accept;
-                iif $wan tcp dport $ports dnat to $ipv4_address
-        }
-
-        chain postrouting {
-                type nat hook postrouting priority 0;
-
-                oifname $wan masquerade
-        }
-}
-
-table ip6 nat {
-        chain prerouting {
-                type nat hook prerouting priority dstnat; policy accept;
-                iif $wan tcp dport $ports dnat to $ipv6_address
-        }
-
-        chain postrouting {
-                type nat hook postrouting priority 0;
-
-                oifname $wan masquerade
-        }
-}
-
-table inet filter {
-        chain input {
-                type filter hook input priority filter; policy drop;
-
-                # Accept ICMP.
-                # It is especially important to accept ICMPv6 ND messages,
-                # otherwise IPv6 connectivity breaks.
-                icmp type { echo-request } accept
-                icmpv6 type { echo-request, nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
-
-                # Allow incoming SSH connections.
-                tcp dport { ssh } accept
-
-                ct state established accept
-        }
-        chain forward {
-                type filter hook forward priority filter; policy drop;
-
-                ct state established accept
-                ip daddr $ipv4_address counter accept
-                ip6 daddr $ipv6_address counter accept
-        }
-        chain output {
-                type filter hook output priority filter;
-        }
-}
-```
-
-Run `systemctl enable nftables.service`
-to ensure configuration is reloaded when the proxy relay reboots.
-
-Uncomment in `/etc/sysctl.conf` the following two lines:
-
-```
-net.ipv4.ip_forward=1
-net.ipv6.conf.all.forwarding=1
-```
-
-Then reboot the relay or do `sysctl -p` and `nft -f /etc/nftables.conf`.
-
-Once proxy relay is set up,
-you can add its IP address to the DNS.
-
-## Neighbors and Acquaintances
-
-Here are some related projects that you may be interested in:
-
-- [Mox](https://github.com/mjl-/mox): A Golang email server.  [Work is in
-  progress](https://github.com/mjl-/mox/issues/251) to modify it to support all
-  of the features and configuration settings required to operate as a chatmail
-  relay.
-- [Maddy-Chatmail](https://github.com/sadraiiali/maddy_chatmail): a plugin for the
-  [Maddy email server](https://maddy.email/) which aims to implement the
-  chatmail relay features and configuration options.

chatmaild/pyproject.toml

@@ -12,7 +12,6 @@ dependencies = [
   "deltachat-rpc-client",
   "filelock",
   "requests",
-  "crypt-r >= 3.13.1 ; python_version >= '3.11'",
 ]
 
 [tool.setuptools]
@@ -27,9 +26,6 @@ chatmail-metadata = "chatmaild.metadata:main"
 filtermail = "chatmaild.filtermail:main"
 echobot = "chatmaild.echo:main"
 chatmail-metrics = "chatmaild.metrics:main"
-delete_inactive_users = "chatmaild.delete_inactive_users:main"
-lastlogin = "chatmaild.lastlogin:main"
-turnserver = "chatmaild.turnserver:main"
 
 [project.entry-points.pytest11]
 "chatmaild.testplugin" = "chatmaild.tests.plugin"
@@ -49,9 +45,6 @@ lint.select = [
   "PLE", # Pylint Error
   "PLW", # Pylint Warning
 ]
-lint.ignore = [
-  "PLC0415" # import-outside-top-level
-]
 
 [tool.tox]
 legacy_tox_ini = """

chatmaild/src/chatmaild/__init__.py

@@ -1 +0,0 @@
-

chatmaild/src/chatmaild/config.py

@@ -1,21 +1,9 @@
-from pathlib import Path
-
 import iniconfig
 
-from chatmaild.user import User
-
-echobot_password_path = Path("/run/echobot/password")
-
 
 def read_config(inipath):
-    assert Path(inipath).exists(), inipath
     cfg = iniconfig.IniConfig(inipath)
-    params = cfg.sections["params"]
-    default_config_content = get_default_config_content(params["mail_domain"])
-    df_params = iniconfig.IniConfig("ini", data=default_config_content)["params"]
-    new_params = dict(df_params.items())
-    new_params.update(params)
-    return Config(inipath, params=new_params)
+    return Config(inipath, params=cfg.sections["params"])
 
 
 class Config:
@@ -24,101 +12,38 @@ class Config:
         self.mail_domain = params["mail_domain"]
         self.max_user_send_per_minute = int(params["max_user_send_per_minute"])
         self.max_mailbox_size = params["max_mailbox_size"]
-        self.max_message_size = int(params.get("max_message_size", "31457280"))
         self.delete_mails_after = params["delete_mails_after"]
-        self.delete_large_after = params["delete_large_after"]
-        self.delete_inactive_users_after = int(params["delete_inactive_users_after"])
         self.username_min_length = int(params["username_min_length"])
         self.username_max_length = int(params["username_max_length"])
         self.password_min_length = int(params["password_min_length"])
         self.passthrough_senders = params["passthrough_senders"].split()
         self.passthrough_recipients = params["passthrough_recipients"].split()
-        self.www_folder = params.get("www_folder", "")
         self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
-        self.filtermail_smtp_port_incoming = int(
-            params["filtermail_smtp_port_incoming"]
-        )
         self.postfix_reinject_port = int(params["postfix_reinject_port"])
-        self.postfix_reinject_port_incoming = int(
-            params["postfix_reinject_port_incoming"]
-        )
-        self.mtail_address = params.get("mtail_address")
-        self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
-        self.acme_email = params.get("acme_email", "")
-        self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
-        if "iroh_relay" not in params:
-            self.iroh_relay = "https://" + params["mail_domain"]
-            self.enable_iroh_relay = True
-        else:
-            self.iroh_relay = params["iroh_relay"].strip()
-            self.enable_iroh_relay = False
+        self.iroh_relay = params.get("iroh_relay")
         self.privacy_postal = params.get("privacy_postal")
         self.privacy_mail = params.get("privacy_mail")
         self.privacy_pdo = params.get("privacy_pdo")
         self.privacy_supervisor = params.get("privacy_supervisor")
 
-        # deprecated option
-        mbdir = params.get("mailboxes_dir", f"/home/vmail/mail/{self.mail_domain}")
-        self.mailboxes_dir = Path(mbdir.strip())
-
-        # old unused option (except for first migration from sqlite to maildir store)
-        self.passdb_path = Path(params.get("passdb_path", "/home/vmail/passdb.sqlite"))
-
     def _getbytefile(self):
         return open(self._inipath, "rb")
 
-    def get_user(self, addr) -> User:
-        if not addr or "@" not in addr or "/" in addr:
-            raise ValueError(f"invalid address {addr!r}")
 
-        maildir = self.mailboxes_dir.joinpath(addr)
-        if addr.startswith("echo@"):
-            password_path = echobot_password_path
-        else:
-            password_path = maildir.joinpath("password")
-
-        return User(maildir, addr, password_path, uid="vmail", gid="vmail")
-
-
-def write_initial_config(inipath, mail_domain, overrides):
-    """Write out default config file, using the specified config value overrides."""
-    content = get_default_config_content(mail_domain, **overrides)
-    inipath.write_text(content)
-
-
-def get_default_config_content(mail_domain, **overrides):
+def write_initial_config(inipath, mail_domain):
     from importlib.resources import files
 
     inidir = files(__package__).joinpath("ini")
-    source_inipath = inidir.joinpath("chatmail.ini.f")
-    content = source_inipath.read_text().format(mail_domain=mail_domain)
-
-    # apply config overrides
-    new_lines = []
-    extra = overrides.copy()
-    for line in content.split("\n"):
-        new_line = line.strip()
-        if new_line and new_line[0] not in "#[":
-            name, value = map(str.strip, new_line.split("=", maxsplit=1))
-            value = extra.pop(name, value)
-            new_line = f"{name} = {value}"
-        new_lines.append(new_line)
-
-    for name, value in extra.items():
-        new_line = f"{name} = {value}"
-        new_lines.append(new_line)
-
-    content = "\n".join(new_lines)
-
-    # apply testrun privacy overrides
-
+    content = (
+        inidir.joinpath("chatmail.ini.f").read_text().format(mail_domain=mail_domain)
+    )
     if mail_domain.endswith(".testrun.org"):
         override_inipath = inidir.joinpath("override-testrun.ini")
         privacy = iniconfig.IniConfig(override_inipath)["privacy"]
         lines = []
         for line in content.split("\n"):
             for key, value in privacy.items():
-                value_lines = value.format(mail_domain=mail_domain).strip().split("\n")
+                value_lines = value.strip().split("\n")
                 if not line.startswith(f"{key} =") or not value_lines:
                     continue
                 if len(value_lines) == 1:
@@ -131,4 +56,5 @@ def get_default_config_content(mail_domain, **overrides):
             else:
                 lines.append(line)
         content = "\n".join(lines)
-    return content
+
+    inipath.write_text(content)

chatmaild/src/chatmaild/database.py

@@ -0,0 +1,133 @@
+import contextlib
+import sqlite3
+import time
+from pathlib import Path
+
+
+class DBError(Exception):
+    """error during an operation on the database."""
+
+
+class Connection:
+    def __init__(self, sqlconn, write):
+        self._sqlconn = sqlconn
+        self._write = write
+
+    def close(self):
+        self._sqlconn.close()
+
+    def commit(self):
+        self._sqlconn.commit()
+
+    def rollback(self):
+        self._sqlconn.rollback()
+
+    def execute(self, query, params=()):
+        cur = self.cursor()
+        try:
+            cur.execute(query, params)
+        except sqlite3.IntegrityError as e:
+            raise DBError(e)
+        return cur
+
+    def cursor(self):
+        return self._sqlconn.cursor()
+
+    def get_user(self, addr: str) -> {}:
+        """Get a row from the users table."""
+        q = "SELECT addr, password, last_login from users WHERE addr = ?"
+        row = self._sqlconn.execute(q, (addr,)).fetchone()
+        result = {}
+        if row:
+            result = dict(
+                user=row[0],
+                password=row[1],
+                last_login=row[2],
+            )
+        return result
+
+
+class Database:
+    def __init__(self, path: str):
+        self.path = Path(path)
+        self.ensure_tables()
+
+    def _get_connection(
+        self, write=False, transaction=False, closing=False
+    ) -> Connection:
+        # we let the database serialize all writers at connection time
+        # to play it very safe (we don't have massive amounts of writes).
+        mode = "ro"
+        if write:
+            mode = "rw"
+        if not self.path.exists():
+            mode = "rwc"
+        uri = "file:%s?mode=%s" % (self.path, mode)
+        sqlconn = sqlite3.connect(
+            uri,
+            timeout=60,
+            isolation_level=None if transaction else "DEFERRED",
+            uri=True,
+        )
+
+        # Enable Write-Ahead Logging to avoid readers blocking writers and vice versa.
+        if write:
+            sqlconn.execute("PRAGMA journal_mode=wal")
+
+        if transaction:
+            start_time = time.time()
+            while 1:
+                try:
+                    sqlconn.execute("begin immediate")
+                    break
+                except sqlite3.OperationalError:
+                    # another thread may be writing, give it a chance to finish
+                    time.sleep(0.1)
+                    if time.time() - start_time > 5:
+                        # if it takes this long, something is wrong
+                        raise
+        conn = Connection(sqlconn, write=write)
+        if closing:
+            conn = contextlib.closing(conn)
+        return conn
+
+    @contextlib.contextmanager
+    def write_transaction(self):
+        conn = self._get_connection(closing=False, write=True, transaction=True)
+        try:
+            yield conn
+        except Exception:
+            conn.rollback()
+            conn.close()
+            raise
+        else:
+            conn.commit()
+            conn.close()
+
+    def read_connection(self, closing=True) -> Connection:
+        return self._get_connection(closing=closing, write=False)
+
+    def get_schema_version(self) -> int:
+        with self.read_connection() as conn:
+            dbversion = conn.execute("PRAGMA user_version").fetchone()[0]
+        return dbversion
+
+    CURRENT_DBVERSION = 1
+
+    def ensure_tables(self):
+        with self.write_transaction() as conn:
+            if self.get_schema_version() > 1:
+                raise DBError(
+                    "version is %s; downgrading schema is not supported"
+                    % (self.get_schema_version(),)
+                )
+            conn.execute(
+                """
+                CREATE TABLE IF NOT EXISTS users (
+                    addr TEXT PRIMARY KEY,
+                    password TEXT,
+                    last_login INTEGER
+                )
+            """,
+            )
+            conn.execute("PRAGMA user_version=%s" % (self.CURRENT_DBVERSION,))

chatmaild/src/chatmaild/delete_inactive_users.py

@@ -1,31 +0,0 @@
-"""
-Remove inactive users
-"""
-
-import os
-import shutil
-import sys
-import time
-
-from .config import read_config
-
-
-def delete_inactive_users(config):
-    cutoff_date = time.time() - config.delete_inactive_users_after * 86400
-    for addr in os.listdir(config.mailboxes_dir):
-        try:
-            user = config.get_user(addr)
-        except ValueError:
-            continue
-
-        read_timestamp = user.get_last_login_timestamp()
-        if read_timestamp and read_timestamp < cutoff_date:
-            path = config.mailboxes_dir.joinpath(addr)
-            assert path == user.maildir
-            shutil.rmtree(path, ignore_errors=True)
-
-
-def main():
-    (cfgpath,) = sys.argv[1:]
-    config = read_config(cfgpath)
-    delete_inactive_users(config)

chatmaild/src/chatmaild/dictproxy.py

@@ -1,98 +0,0 @@
-import logging
-import os
-from socketserver import StreamRequestHandler, ThreadingUnixStreamServer
-
-
-class DictProxy:
-    def loop_forever(self, rfile, wfile):
-        # Transaction storage is local to each handler loop.
-        # Dovecot reuses transaction IDs across connections,
-        # starting transaction with the name `1`
-        # on two different connections to the same proxy sometimes.
-        transactions = {}
-
-        while True:
-            msg = rfile.readline().strip().decode()
-            if not msg:
-                break
-
-            res = self.handle_dovecot_request(msg, transactions)
-            if res:
-                wfile.write(res.encode("ascii"))
-                wfile.flush()
-
-    def handle_dovecot_request(self, msg, transactions):
-        # see https://doc.dovecot.org/developer_manual/design/dict_protocol/#dovecot-dict-protocol
-        short_command = msg[0]
-        parts = msg[1:].split("\t")
-
-        if short_command == "L":
-            return self.handle_lookup(parts)
-        elif short_command == "I":
-            return self.handle_iterate(parts)
-        elif short_command == "H":
-            return  # no version checking
-
-        if short_command not in ("BSC"):
-            logging.warning(f"unknown dictproxy request: {msg!r}")
-            return
-
-        transaction_id = parts[0]
-
-        if short_command == "B":
-            return self.handle_begin_transaction(transaction_id, parts, transactions)
-        elif short_command == "C":
-            return self.handle_commit_transaction(transaction_id, parts, transactions)
-        elif short_command == "S":
-            addr = transactions[transaction_id]["addr"]
-            if not self.handle_set(addr, parts):
-                transactions[transaction_id]["res"] = "F\n"
-                logging.error(f"dictproxy-set failed for {addr!r}: {msg!r}")
-
-    def handle_lookup(self, parts):
-        logging.warning(f"lookup ignored: {parts!r}")
-        return "N\n"
-
-    def handle_iterate(self, parts):
-        # Empty line means ITER_FINISHED.
-        # If we don't return empty line Dovecot will timeout.
-        return "\n"
-
-    def handle_begin_transaction(self, transaction_id, parts, transactions):
-        addr = parts[1]
-        transactions[transaction_id] = dict(addr=addr, res="O\n")
-
-    def handle_set(self, addr, parts):
-        # For documentation on key structure see
-        # https://github.com/dovecot/core/blob/main/src/lib-storage/mailbox-attribute.h
-        return False
-
-    def handle_commit_transaction(self, transaction_id, parts, transactions):
-        # return whatever "set" command(s) set as result.
-        return transactions.pop(transaction_id)["res"]
-
-    def serve_forever_from_socket(self, socket):
-        dictproxy = self
-
-        class Handler(StreamRequestHandler):
-            def handle(self):
-                try:
-                    dictproxy.loop_forever(self.rfile, self.wfile)
-                except Exception:
-                    logging.exception("Exception in the handler")
-                    raise
-
-        try:
-            os.unlink(socket)
-        except FileNotFoundError:
-            pass
-
-        with CustomThreadingUnixStreamServer(socket, Handler) as server:
-            try:
-                server.serve_forever()
-            except KeyboardInterrupt:
-                pass
-
-
-class CustomThreadingUnixStreamServer(ThreadingUnixStreamServer):
-    request_queue_size = 1000

chatmaild/src/chatmaild/doveauth.py

@@ -1,23 +1,29 @@
+import crypt
 import json
 import logging
 import os
 import sys
-
-try:
-    import crypt_r
-except ImportError:
-    import crypt as crypt_r
+import time
+from pathlib import Path
+from socketserver import (
+    StreamRequestHandler,
+    ThreadingMixIn,
+    UnixStreamServer,
+)
 
 from .config import Config, read_config
-from .dictproxy import DictProxy
-from .migrate_db import migrate_from_db_to_maildir
+from .database import Database
 
 NOCREATE_FILE = "/etc/chatmail-nocreate"
 
 
+class UnknownCommand(ValueError):
+    """dictproxy handler received an unkown command"""
+
+
 def encrypt_password(password: str):
     # https://doc.dovecot.org/configuration_manual/authentication/password_schemes/
-    passhash = crypt_r.crypt(password, crypt_r.METHOD_SHA512)
+    passhash = crypt.crypt(password, crypt.METHOD_SHA512)
     return "{SHA512-CRYPT}" + passhash
 
 
@@ -59,6 +65,70 @@ def is_allowed_to_create(config: Config, user, cleartext_password) -> bool:
     return True
 
 
+def get_user_data(db, config: Config, user):
+    if user == f"echo@{config.mail_domain}":
+        return dict(
+            home=f"/home/vmail/mail/{config.mail_domain}/echo@{config.mail_domain}",
+            uid="vmail",
+            gid="vmail",
+        )
+
+    with db.read_connection() as conn:
+        result = conn.get_user(user)
+    if result:
+        result["home"] = f"/home/vmail/mail/{config.mail_domain}/{user}"
+        result["uid"] = "vmail"
+        result["gid"] = "vmail"
+    return result
+
+
+def lookup_userdb(db, config: Config, user):
+    return get_user_data(db, config, user)
+
+
+def lookup_passdb(db, config: Config, user, cleartext_password):
+    if user == f"echo@{config.mail_domain}":
+        # Echobot writes password it wants to log in with into /run/echobot/password
+        try:
+            password = Path("/run/echobot/password").read_text()
+        except Exception:
+            logging.exception("Exception when trying to read /run/echobot/password")
+            return None
+
+        return dict(
+            home=f"/home/vmail/mail/{config.mail_domain}/echo@{config.mail_domain}",
+            uid="vmail",
+            gid="vmail",
+            password=encrypt_password(password),
+        )
+
+    with db.write_transaction() as conn:
+        userdata = conn.get_user(user)
+        if userdata:
+            # Update last login time.
+            conn.execute(
+                "UPDATE users SET last_login=? WHERE addr=?", (int(time.time()), user)
+            )
+
+            userdata["home"] = f"/home/vmail/mail/{config.mail_domain}/{user}"
+            userdata["uid"] = "vmail"
+            userdata["gid"] = "vmail"
+            return userdata
+        if not is_allowed_to_create(config, user, cleartext_password):
+            return
+
+        encrypted_password = encrypt_password(cleartext_password)
+        q = """INSERT INTO users (addr, password, last_login)
+               VALUES (?, ?, ?)"""
+        conn.execute(q, (user, encrypted_password, int(time.time())))
+        return dict(
+            home=f"/home/vmail/mail/{config.mail_domain}/{user}",
+            uid="vmail",
+            gid="vmail",
+            password=encrypted_password,
+        )
+
+
 def split_and_unescape(s):
     """Split strings using double quote as a separator and backslash as escape character
     into parts."""
@@ -85,12 +155,15 @@ def split_and_unescape(s):
     yield out
 
 
-class AuthDictProxy(DictProxy):
-    def __init__(self, config):
-        super().__init__()
-        self.config = config
+def handle_dovecot_request(msg, db, config: Config):
+    # see https://doc.dovecot.org/3.0/developer_manual/design/dict_protocol/
+    short_command = msg[0]
+    if short_command == "H":  # HELLO
+        # we don't do any checking on versions and just return
+        return
+    elif short_command == "L":  # LOOKUP
+        parts = msg[1:].split("\t")
 
-    def handle_lookup(self, parts):
         # Dovecot <2.3.17 has only one part,
         # do not attempt to read any other parts for compatibility.
         keyname = parts[0]
@@ -98,14 +171,13 @@ class AuthDictProxy(DictProxy):
         namespace, type, args = keyname.split("/", 2)
         args = list(split_and_unescape(args))
 
-        config = self.config
         reply_command = "F"
         res = ""
         if namespace == "shared":
             if type == "userdb":
                 user = args[0]
                 if user.endswith(f"@{config.mail_domain}"):
-                    res = self.lookup_userdb(user)
+                    res = lookup_userdb(db, config, user)
                 if res:
                     reply_command = "O"
                 else:
@@ -113,48 +185,55 @@ class AuthDictProxy(DictProxy):
             elif type == "passdb":
                 user = args[1]
                 if user.endswith(f"@{config.mail_domain}"):
-                    res = self.lookup_passdb(user, cleartext_password=args[0])
+                    res = lookup_passdb(db, config, user, cleartext_password=args[0])
                 if res:
                     reply_command = "O"
                 else:
                     reply_command = "N"
         json_res = json.dumps(res) if res else ""
         return f"{reply_command}{json_res}\n"
+    raise UnknownCommand(msg)
 
-    def handle_iterate(self, parts):
-        # example: I0\t0\tshared/userdb/
-        if parts[2] == "shared/userdb/":
-            result = "".join(
-                f"Oshared/userdb/{user}\t\n" for user in self.iter_userdb()
-            )
-            return f"{result}\n"
 
-    def iter_userdb(self) -> list:
-        """Get a list of all user addresses."""
-        return [x for x in os.listdir(self.config.mailboxes_dir) if "@" in x]
+def handle_dovecot_protocol(rfile, wfile, db: Database, config: Config):
+    while True:
+        msg = rfile.readline().strip().decode()
+        if not msg:
+            break
+        try:
+            res = handle_dovecot_request(msg, db, config)
+        except UnknownCommand:
+            logging.warning("unknown command: %r", msg)
+        else:
+            if res:
+                wfile.write(res.encode("ascii"))
+                wfile.flush()
 
-    def lookup_userdb(self, addr):
-        return self.config.get_user(addr).get_userdb_dict()
 
-    def lookup_passdb(self, addr, cleartext_password):
-        user = self.config.get_user(addr)
-        userdata = user.get_userdb_dict()
-        if userdata:
-            return userdata
-        if not is_allowed_to_create(self.config, addr, cleartext_password):
-            return
-
-        user.set_password(encrypt_password(cleartext_password))
-        print(f"Created address: {addr}", file=sys.stderr)
-        return user.get_userdb_dict()
+class ThreadedUnixStreamServer(ThreadingMixIn, UnixStreamServer):
+    request_queue_size = 100
 
 
 def main():
-    socket, cfgpath = sys.argv[1:]
-    config = read_config(cfgpath)
+    socket = sys.argv[1]
+    db = Database(sys.argv[2])
+    config = read_config(sys.argv[3])
 
-    migrate_from_db_to_maildir(config)
+    class Handler(StreamRequestHandler):
+        def handle(self):
+            try:
+                handle_dovecot_protocol(self.rfile, self.wfile, db, config)
+            except Exception:
+                logging.exception("Exception in the handler")
+                raise
 
-    dictproxy = AuthDictProxy(config=config)
+    try:
+        os.unlink(socket)
+    except FileNotFoundError:
+        pass
 
-    dictproxy.serve_forever_from_socket(socket)
+    with ThreadedUnixStreamServer(socket, Handler) as server:
+        try:
+            server.serve_forever()
+        except KeyboardInterrupt:
+            pass

chatmaild/src/chatmaild/echo.py

@@ -12,8 +12,7 @@ from pathlib import Path
 
 from deltachat_rpc_client import Bot, DeltaChat, EventType, Rpc, events
 
-from chatmaild.config import echobot_password_path, read_config
-from chatmaild.doveauth import encrypt_password
+from chatmaild.config import read_config
 from chatmaild.newemail import create_newemail_dict
 
 hooks = events.HookCollection()
@@ -22,9 +21,9 @@ hooks = events.HookCollection()
 @hooks.on(events.RawEvent)
 def log_event(event):
     if event.kind == EventType.INFO:
-        logging.info(event.msg)
+        logging.info("%s", event.msg)
     elif event.kind == EventType.WARNING:
-        logging.warning(event.msg)
+        logging.warning("%s", event.msg)
 
 
 @hooks.on(events.RawEvent(EventType.ERROR))
@@ -46,7 +45,7 @@ def on_group_image_changed(event):
 
 @hooks.on(events.GroupNameChanged)
 def on_group_name_changed(event):
-    logging.info(f"group name changed, old name: {event.old_name}")
+    logging.info("group name changed, old name: %s", event.old_name)
 
 
 @hooks.on(events.NewMessage(func=lambda e: not e.command))
@@ -73,7 +72,7 @@ def main():
     with Rpc() as rpc:
         deltachat = DeltaChat(rpc)
         system_info = deltachat.get_system_info()
-        logging.info(f"Running deltachat core {system_info.deltachat_core_version}")
+        logging.info("Running deltachat core %s", system_info.deltachat_core_version)
 
         accounts = deltachat.get_all_accounts()
         account = accounts[0] if accounts else deltachat.add_account()
@@ -81,27 +80,23 @@ def main():
         bot = Bot(account, hooks)
 
         config = read_config(sys.argv[1])
-        addr = "echo@" + config.mail_domain
 
         # Create password file
         if bot.is_configured():
             password = bot.account.get_config("mail_pw")
         else:
             password = create_newemail_dict(config)["password"]
+        Path("/run/echobot/password").write_text(password)
 
-        echobot_password_path.write_text(encrypt_password(password))
         # Give the user which doveauth runs as access to the password file.
-        subprocess.check_call(
-            ["/usr/bin/setfacl", "-m", "user:vmail:r", echobot_password_path],
+        subprocess.run(
+            ["/usr/bin/setfacl", "-m", "user:vmail:r", "/run/echobot/password"],
+            check=True,
         )
 
         if not bot.is_configured():
-            bot.configure(addr, password)
-
-        # write invite link to working directory
-        invitelink = bot.account.get_qr_code()
-        Path("invite-link.txt").write_text(invitelink)
-
+            email = "echo@" + config.mail_domain
+            bot.configure(email, password)
         bot.run_forever()
 
 

chatmaild/src/chatmaild/filedict.py

@@ -2,7 +2,6 @@ import json
 import logging
 import os
 from contextlib import contextmanager
-from random import randint
 
 import filelock
 
@@ -33,12 +32,5 @@ class FileDict:
         except FileNotFoundError:
             return {}
         except Exception:
-            logging.warning(f"corrupt serialization state at: {self.path!r}")
+            logging.warning("corrupt serialization state at: %r", self.path)
             return {}
-
-
-def write_bytes_atomic(path, content):
-    rint = randint(0, 10000000)
-    tmp = path.with_name(path.name + f".tmp-{rint}")
-    tmp.write_bytes(content)
-    os.rename(tmp, path)

chatmaild/src/chatmaild/filtermail.py

@@ -2,6 +2,7 @@
 import asyncio
 import base64
 import binascii
+import logging
 import sys
 import time
 from email import policy
@@ -10,12 +11,9 @@ from email.utils import parseaddr
 from smtplib import SMTP as SMTPClient
 
 from aiosmtpd.controller import Controller
-from aiosmtpd.smtp import SMTP
 
 from .config import read_config
 
-ENCRYPTION_NEEDED_523 = "523 Encryption Needed: Invalid Unencrypted Mail"
-
 
 def check_openpgp_payload(payload: bytes):
     """Checks the OpenPGP payload.
@@ -37,12 +35,6 @@ def check_openpgp_payload(payload: bytes):
 
         packet_type_id = payload[i] & 0x3F
         i += 1
-
-        while payload[i] >= 224 and payload[i] < 255:
-            # Partial body length.
-            partial_length = 1 << (payload[i] & 0x1F)
-            i += 1 + partial_length
-
         if payload[i] < 192:
             # One-octet length.
             body_len = payload[i]
@@ -61,17 +53,16 @@ def check_openpgp_payload(payload: bytes):
             )
             i += 5
         else:
-            # Impossible, partial body length was processed above.
+            # Partial body length is not allowed.
             return False
 
         i += body_len
 
         if i == len(payload):
-            # Last packet should be
-            # Symmetrically Encrypted and Integrity Protected Data Packet (SEIPD)
-            #
-            # This is the only place where this function may return `True`.
-            return packet_type_id == 18
+            if packet_type_id == 18:
+                # Last packet should be
+                # Symmetrically Encrypted and Integrity Protected Data Packet (SEIPD)
+                return True
         elif packet_type_id not in [1, 3]:
             # All packets except the last one must be either
             # Public-Key Encrypted Session Key Packet (PKESK)
@@ -79,39 +70,26 @@ def check_openpgp_payload(payload: bytes):
             # Symmetric-Key Encrypted Session Key Packet (SKESK)
             return False
 
-    return False
+    if i == 0:
+        return False
+
+    if i > len(payload):
+        # Payload is truncated.
+        return False
+    return True
 
 
-def check_armored_payload(payload: str, outgoing: bool):
-    """Check the armored PGP message for invalid content.
-
-    :param payload: the armored PGP message
-    :param outgoing: whether the message is outgoing or incoming
-    :return: whether the message is a valid PGP message
-    """
-    prefix = "-----BEGIN PGP MESSAGE-----\r\n"
+def check_armored_payload(payload: str):
+    prefix = "-----BEGIN PGP MESSAGE-----\r\n\r\n"
     if not payload.startswith(prefix):
         return False
     payload = payload.removeprefix(prefix)
 
-    while payload.endswith("\r\n"):
-        payload = payload.removesuffix("\r\n")
-    suffix = "-----END PGP MESSAGE-----"
+    suffix = "-----END PGP MESSAGE-----\r\n\r\n"
     if not payload.endswith(suffix):
         return False
     payload = payload.removesuffix(suffix)
 
-    # Disallow comments in outgoing messages
-    version_comment = "Version: "
-    if payload.startswith(version_comment):
-        splitindex = payload.find("\r\n") + 2
-        payload = payload[splitindex:]
-        if outgoing:
-            return False
-
-    while payload.startswith("\r\n"):
-        payload = payload.removeprefix("\r\n")
-
     # Remove CRC24.
     payload = payload.rpartition("=")[0]
 
@@ -126,34 +104,15 @@ def check_armored_payload(payload: str, outgoing: bool):
         return False
 
 
-def is_securejoin(message):
-    if message.get("secure-join") not in ["vc-request", "vg-request"]:
-        return False
-    if not message.is_multipart():
-        return False
-    parts_count = 0
-    for part in message.iter_parts():
-        parts_count += 1
-        if parts_count > 1:
-            return False
-        if part.is_multipart():
-            return False
-        if part.get_content_type() != "text/plain":
-            return False
-
-        payload = part.get_payload().strip().lower()
-        if payload not in ("secure-join: vc-request", "secure-join: vg-request"):
-            return False
-    return True
-
-
-def check_encrypted(message, outgoing=True):
+def check_encrypted(message):
     """Check that the message is an OpenPGP-encrypted message.
 
     MIME structure of the message must correspond to <https://www.rfc-editor.org/rfc/rfc3156>.
     """
     if not message.is_multipart():
         return False
+    if message.get("subject") != "...":
+        return False
     if message.get_content_type() != "multipart/encrypted":
         return False
     parts_count = 0
@@ -174,7 +133,7 @@ def check_encrypted(message, outgoing=True):
             if part.get_content_type() != "application/octet-stream":
                 return False
 
-            if not check_armored_payload(part.get_payload(), outgoing=outgoing):
+            if not check_armored_payload(part.get_payload()):
                 return False
         else:
             return False
@@ -182,53 +141,18 @@ def check_encrypted(message, outgoing=True):
     return True
 
 
-async def asyncmain_beforequeue(config, mode):
-    if mode == "outgoing":
-        port = config.filtermail_smtp_port
-        handler = OutgoingBeforeQueueHandler(config)
-    else:
-        port = config.filtermail_smtp_port_incoming
-        handler = IncomingBeforeQueueHandler(config)
-    HackedController(
-        handler,
-        hostname="127.0.0.1",
-        port=port,
-        data_size_limit=config.max_message_size,
-    ).start()
+async def asyncmain_beforequeue(config):
+    port = config.filtermail_smtp_port
+    Controller(BeforeQueueHandler(config), hostname="127.0.0.1", port=port).start()
 
 
-def recipient_matches_passthrough(recipient, passthrough_recipients):
-    for addr in passthrough_recipients:
-        if recipient == addr:
-            return True
-        if addr[0] == "@" and recipient.endswith(addr):
-            return True
-    return False
-
-
-class HackedController(Controller):
-    def factory(self):
-        return SMTPDiscardRCPTO_options(self.handler, **self.SMTP_kwargs)
-
-
-class SMTPDiscardRCPTO_options(SMTP):
-    def _getparams(self, params):
-        # Ignore RCPT TO parameters.
-        #
-        # Otherwise parameters such as `ORCPT=...`
-        # or `NOTIFY=DELAY,FAILURE` (generated by Stalwart)
-        # make aiosmtpd reject the message here:
-        # <https://github.com/aio-libs/aiosmtpd/blob/98f578389ae86e5345cc343fa4e5a17b21d9c96d/aiosmtpd/smtp.py#L1379-L1384>
-        return {}
-
-
-class OutgoingBeforeQueueHandler:
+class BeforeQueueHandler:
     def __init__(self, config):
         self.config = config
         self.send_rate_limiter = SendRateLimiter()
 
     async def handle_MAIL(self, server, session, envelope, address, mail_options):
-        log_info(f"handle_MAIL from {address}")
+        logging.info(f"handle_MAIL from {address}")
         envelope.mail_from = address
         max_sent = self.config.max_user_send_per_minute
         if not self.send_rate_limiter.is_sending_allowed(address, max_sent):
@@ -241,113 +165,51 @@ class OutgoingBeforeQueueHandler:
         return "250 OK"
 
     async def handle_DATA(self, server, session, envelope):
-        loop = asyncio.get_running_loop()
-        return await loop.run_in_executor(None, self.sync_handle_DATA, envelope)
-
-    def sync_handle_DATA(self, envelope):
-        log_info("handle_DATA before-queue")
+        logging.info("handle_DATA before-queue")
         error = self.check_DATA(envelope)
         if error:
             return error
-        log_info("re-injecting the mail that passed checks")
+        logging.info("re-injecting the mail that passed checks")
         client = SMTPClient("localhost", self.config.postfix_reinject_port)
-        client.sendmail(
-            envelope.mail_from, envelope.rcpt_tos, envelope.original_content
-        )
+        client.sendmail(envelope.mail_from, envelope.rcpt_tos, envelope.content)
         return "250 OK"
 
     def check_DATA(self, envelope):
         """the central filtering function for e-mails."""
-        log_info(f"Processing DATA message from {envelope.mail_from}")
+        logging.info(f"Processing DATA message from {envelope.mail_from}")
 
         message = BytesParser(policy=policy.default).parsebytes(envelope.content)
-        mail_encrypted = check_encrypted(message, outgoing=True)
+        mail_encrypted = check_encrypted(message)
 
         _, from_addr = parseaddr(message.get("from").strip())
-
+        logging.info(f"mime-from: {from_addr} envelope-from: {envelope.mail_from!r}")
         if envelope.mail_from.lower() != from_addr.lower():
             return f"500 Invalid FROM <{from_addr!r}> for <{envelope.mail_from!r}>"
 
-        if mail_encrypted or is_securejoin(message):
-            print("Outgoing: Filtering encrypted mail.", file=sys.stderr)
-            return
-
-        print("Outgoing: Filtering unencrypted mail.", file=sys.stderr)
-
         if envelope.mail_from in self.config.passthrough_senders:
             return
 
-        # allow self-sent Autocrypt Setup Message
-        if envelope.rcpt_tos == [from_addr]:
-            if message.get("subject") == "Autocrypt Setup Message":
-                if message.get_content_type() == "multipart/mixed":
-                    return
-
         passthrough_recipients = self.config.passthrough_recipients
-
+        envelope_from_domain = from_addr.split("@").pop()
         for recipient in envelope.rcpt_tos:
-            if recipient_matches_passthrough(recipient, passthrough_recipients):
+            if envelope.mail_from == recipient:
+                # Always allow sending emails to self.
                 continue
-
-            print("Rejected unencrypted mail.", file=sys.stderr)
-            return ENCRYPTION_NEEDED_523
-
-
-class IncomingBeforeQueueHandler:
-    def __init__(self, config):
-        self.config = config
-
-    async def handle_DATA(self, server, session, envelope):
-        loop = asyncio.get_running_loop()
-        return await loop.run_in_executor(None, self.sync_handle_DATA, envelope)
-
-    def sync_handle_DATA(self, envelope):
-        log_info("handle_DATA before-queue")
-        error = self.check_DATA(envelope)
-        if error:
-            return error
-        log_info("re-injecting the mail that passed checks")
-
-        # the smtp daemon on reinject_port_incoming gives it to dkim milter
-        # which looks at source address to determine whether to verify or sign
-        client = SMTPClient(
-            "localhost",
-            self.config.postfix_reinject_port_incoming,
-            source_address=("127.0.0.2", 0),
-        )
-        client.sendmail(
-            envelope.mail_from, envelope.rcpt_tos, envelope.original_content
-        )
-        return "250 OK"
-
-    def check_DATA(self, envelope):
-        """the central filtering function for e-mails."""
-        log_info(f"Processing DATA message from {envelope.mail_from}")
-
-        message = BytesParser(policy=policy.default).parsebytes(envelope.content)
-        mail_encrypted = check_encrypted(message, outgoing=False)
-
-        if mail_encrypted or is_securejoin(message):
-            print("Incoming: Filtering encrypted mail.", file=sys.stderr)
-            return
-
-        print("Incoming: Filtering unencrypted mail.", file=sys.stderr)
-
-        # we want cleartext mailer-daemon messages to pass through
-        # chatmail core will typically not display them as normal messages
-        if message.get("auto-submitted"):
-            _, from_addr = parseaddr(message.get("from").strip())
-            if from_addr.lower().startswith("mailer-daemon@"):
-                if message.get_content_type() == "multipart/report":
-                    return
-
-        for recipient in envelope.rcpt_tos:
-            user = self.config.get_user(recipient)
-            if user is None or user.is_incoming_cleartext_ok():
+            if recipient in passthrough_recipients:
                 continue
+            res = recipient.split("@")
+            if len(res) != 2:
+                return f"500 Invalid address <{recipient}>"
+            _recipient_addr, recipient_domain = res
 
-            print("Rejected unencrypted mail.", file=sys.stderr)
-            return ENCRYPTION_NEEDED_523
+            is_outgoing = recipient_domain != envelope_from_domain
+            if is_outgoing and not mail_encrypted:
+                is_securejoin = message.get("secure-join") in [
+                    "vc-request",
+                    "vg-request",
+                ]
+                if not is_securejoin:
+                    return f"500 Invalid unencrypted mail to <{recipient}>"
 
 
 class SendRateLimiter:
@@ -364,19 +226,13 @@ class SendRateLimiter:
         return False
 
 
-def log_info(msg):
-    print(msg, file=sys.stderr)
-
-
 def main():
     args = sys.argv[1:]
-    assert len(args) == 2
+    assert len(args) == 1
     config = read_config(args[0])
-    mode = args[1]
+    logging.basicConfig(level=logging.WARN)
     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
-    assert mode in ["incoming", "outgoing"]
-    task = asyncmain_beforequeue(config, mode)
+    task = asyncmain_beforequeue(config)
     loop.create_task(task)
-    log_info("entering serving loop")
     loop.run_forever()

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -8,27 +8,18 @@ mail_domain = {mail_domain}
 #
 
 #
-# Restrictions on user addresses
+# Account Restrictions
 #
 
 # how many mails a user can send out per minute
 max_user_send_per_minute = 60
 
-# maximum mailbox size of a chatmail address
+# maximum mailbox size of a chatmail account
 max_mailbox_size = 100M
 
-# maximum message size for an e-mail in bytes
-max_message_size = 31457280
-
 # days after which mails are unconditionally deleted
 delete_mails_after = 20
 
-# days after which large messages (>200k) are unconditionally deleted
-delete_large_after = 7
-
-# days after which users without a successful login are deleted (database and mails)
-delete_inactive_users_after = 90
-
 # minimum length a username must have
 username_min_length = 9
 
@@ -38,68 +29,22 @@ username_max_length = 9
 # minimum length a password must have
 password_min_length = 9
 
-# list of chatmail addresses which can send outbound un-encrypted mail
+# list of chatmail accounts which can send outbound un-encrypted mail
 passthrough_senders =
 
 # list of e-mail recipients for which to accept outbound un-encrypted mails
-# (space-separated, item may start with "@" to whitelist whole recipient domains)
-passthrough_recipients = xstore@testrun.org echo@{mail_domain}
-
-# path to www directory - documented here: https://github.com/chatmail/relay/#custom-web-pages
-#www_folder = www
+passthrough_recipients = xstore@testrun.org groupsbot@hispanilandia.net
 
 #
 # Deployment Details
 #
 
-# SMTP outgoing filtermail and reinjection 
+# where the filtermail SMTP service listens
 filtermail_smtp_port = 10080
+
+# postfix accepts on the localhost reinject SMTP port
 postfix_reinject_port = 10025
 
-# SMTP incoming filtermail and reinjection 
-filtermail_smtp_port_incoming = 10081
-postfix_reinject_port_incoming = 10026
-
-# if set to "True" IPv6 is disabled
-disable_ipv6 = False
-
-# Your email adress, which will be used in acmetool to manage Let's Encrypt SSL certificates
-acme_email = 
-
-# Defaults to https://iroh.{{mail_domain}} and running `iroh-relay` on the chatmail
-# service.
-# If you set it to anything else, the service will be disabled
-# and users will be directed to use the given iroh relay URL.
-# Set it to empty string if you want users to use their default iroh relay.
-# iroh_relay =
-
-# Address on which `mtail` listens,
-# e.g. 127.0.0.1 or some private network
-# address like 192.168.10.1.
-# You can point Prometheus
-# or some other OpenMetrics-compatible
-# collector to
-# http://{{mtail_address}}:3903/metrics
-# and display collected metrics with Grafana.
-#
-# WARNING: do not expose this service
-# to the public IP address.
-#
-# `mtail is not running if the setting is not set.
-
-# mtail_address = 127.0.0.1
-
-#
-# Debugging options 
-#
-
-# set to True if you want to track imap protocol execution
-# in per-maildir ".in/.out" files. 
-# Note that you need to manually cleanup these files
-# so use this option with caution on production servers. 
-imap_rawlog = false 
-
-
 #
 # Privacy Policy
 #
@@ -115,3 +60,4 @@ privacy_pdo =
 
 # postal address of the privacy supervisor
 privacy_supervisor =
+

chatmaild/src/chatmaild/ini/override-testrun.ini

@@ -1,7 +1,7 @@
 
 [privacy]
 
-passthrough_recipients = privacy@testrun.org xstore@testrun.org echo@{mail_domain}
+passthrough_recipients = privacy@testrun.org xstore@testrun.org groupsbot@hispanilandia.net
 
 privacy_postal =
     Merlinux GmbH, Represented by the managing director H. Krekel,

chatmaild/src/chatmaild/lastlogin.py

@@ -1,31 +0,0 @@
-import sys
-
-from .config import read_config
-from .dictproxy import DictProxy
-
-
-class LastLoginDictProxy(DictProxy):
-    def __init__(self, config):
-        super().__init__()
-        self.config = config
-
-    def handle_set(self, addr, parts):
-        keyname = parts[1].split("/")
-        value = parts[2] if len(parts) > 2 else ""
-        if keyname[0] == "shared" and keyname[1] == "last-login":
-            if addr.startswith("echo@"):
-                return True
-            addr = keyname[2]
-            timestamp = int(value)
-            user = self.config.get_user(addr)
-            user.set_last_login_timestamp(timestamp)
-            return True
-
-        return False
-
-
-def main():
-    socket, config_path = sys.argv[1:]
-    config = read_config(config_path)
-    dictproxy = LastLoginDictProxy(config=config)
-    dictproxy.serve_forever_from_socket(socket)

chatmaild/src/chatmaild/metadata.py

@@ -1,24 +1,28 @@
 import logging
+import os
 import sys
-import time
-from contextlib import contextmanager
+from pathlib import Path
+from socketserver import (
+    StreamRequestHandler,
+    ThreadingMixIn,
+    UnixStreamServer,
+)
 
 from .config import read_config
-from .dictproxy import DictProxy
 from .filedict import FileDict
 from .notifier import Notifier
-from .turnserver import turn_credentials
 
-
-def _is_valid_token_timestamp(timestamp, now):
-    # Token if invalid after 90 days
-    # or if the timestamp is in the future.
-    return timestamp > now - 3600 * 24 * 90 and timestamp < now + 60
+DICTPROXY_HELLO_CHAR = "H"
+DICTPROXY_LOOKUP_CHAR = "L"
+DICTPROXY_ITERATE_CHAR = "I"
+DICTPROXY_BEGIN_TRANSACTION_CHAR = "B"
+DICTPROXY_SET_CHAR = "S"
+DICTPROXY_COMMIT_TRANSACTION_CHAR = "C"
+DICTPROXY_TRANSACTION_CHARS = "BSC"
 
 
 class Metadata:
-    # each SETMETADATA on this key appends to dictionary
-    # mapping of unique device tokens
+    # each SETMETADATA on this key appends to a list of unique device tokens
     # which only ever get removed if the upstream indicates the token is invalid
     DEVICETOKEN_KEY = "devicetoken"
 
@@ -28,109 +32,108 @@ class Metadata:
     def get_metadata_dict(self, addr):
         return FileDict(self.vmail_dir / addr / "metadata.json")
 
-    @contextmanager
-    def _modify_tokens(self, addr):
-        with self.get_metadata_dict(addr).modify() as data:
-            tokens = data.setdefault(self.DEVICETOKEN_KEY, {})
-            now = int(time.time())
-            if isinstance(tokens, list):
-                data[self.DEVICETOKEN_KEY] = tokens = {t: now for t in tokens}
-
-            expired_tokens = [
-                token
-                for token, timestamp in tokens.items()
-                if not _is_valid_token_timestamp(tokens[token], now)
-            ]
-            for expired_token in expired_tokens:
-                del tokens[expired_token]
-
-            yield tokens
-
     def add_token_to_addr(self, addr, token):
-        with self._modify_tokens(addr) as tokens:
-            tokens[token] = int(time.time())
+        with self.get_metadata_dict(addr).modify() as data:
+            tokens = data.setdefault(self.DEVICETOKEN_KEY, [])
+            if token not in tokens:
+                tokens.append(token)
 
     def remove_token_from_addr(self, addr, token):
-        with self._modify_tokens(addr) as tokens:
+        with self.get_metadata_dict(addr).modify() as data:
+            tokens = data.get(self.DEVICETOKEN_KEY, [])
             if token in tokens:
-                del tokens[token]
+                tokens.remove(token)
 
     def get_tokens_for_addr(self, addr):
         mdict = self.get_metadata_dict(addr).read()
-        tokens = mdict.get(self.DEVICETOKEN_KEY, {})
-
-        now = int(time.time())
-        if isinstance(tokens, dict):
-            token_list = [
-                token
-                for token, timestamp in tokens.items()
-                if _is_valid_token_timestamp(timestamp, now)
-            ]
-            if len(token_list) < len(tokens):
-                # Some tokens have expired, remove them.
-                with self._modify_tokens(addr) as _tokens:
-                    pass
-        else:
-            token_list = []
-        return token_list
+        return mdict.get(self.DEVICETOKEN_KEY, [])
 
 
-class MetadataDictProxy(DictProxy):
-    def __init__(self, notifier, metadata, iroh_relay=None, turn_hostname=None):
-        super().__init__()
-        self.notifier = notifier
-        self.metadata = metadata
-        self.iroh_relay = iroh_relay
-        self.turn_hostname = turn_hostname
+def handle_dovecot_protocol(rfile, wfile, notifier, metadata, iroh_relay=None):
+    transactions = {}
+    while True:
+        msg = rfile.readline().strip().decode()
+        if not msg:
+            break
 
-    def handle_lookup(self, parts):
+        res = handle_dovecot_request(msg, transactions, notifier, metadata, iroh_relay)
+        if res:
+            wfile.write(res.encode("ascii"))
+            wfile.flush()
+
+
+def handle_dovecot_request(msg, transactions, notifier, metadata, iroh_relay=None):
+    # see https://doc.dovecot.org/3.0/developer_manual/design/dict_protocol/
+    short_command = msg[0]
+    parts = msg[1:].split("\t")
+    if short_command == DICTPROXY_LOOKUP_CHAR:
         # Lpriv/43f5f508a7ea0366dff30200c15250e3/devicetoken\tlkj123poi@c2.testrun.org
         keyparts = parts[0].split("/", 2)
         if keyparts[0] == "priv":
             keyname = keyparts[2]
             addr = parts[1]
-            if keyname == self.metadata.DEVICETOKEN_KEY:
-                res = " ".join(self.metadata.get_tokens_for_addr(addr))
+            if keyname == metadata.DEVICETOKEN_KEY:
+                res = " ".join(metadata.get_tokens_for_addr(addr))
                 return f"O{res}\n"
         elif keyparts[0] == "shared":
             keyname = keyparts[2]
             if (
                 keyname == "vendor/vendor.dovecot/pvt/server/vendor/deltachat/irohrelay"
-                and self.iroh_relay
+                and iroh_relay
             ):
                 # Handle `GETMETADATA "" /shared/vendor/deltachat/irohrelay`
-                return f"O{self.iroh_relay}\n"
-            elif keyname == "vendor/vendor.dovecot/pvt/server/vendor/deltachat/turn":
-                res = turn_credentials()
-                port = 3478
-                return f"O{self.turn_hostname}:{port}:{res}\n"
-
-        logging.warning(f"lookup ignored: {parts!r}")
+                return f"O{iroh_relay}\n"
+        logging.warning("lookup ignored: %r", msg)
         return "N\n"
+    elif short_command == DICTPROXY_ITERATE_CHAR:
+        # Empty line means ITER_FINISHED.
+        # If we don't return empty line Dovecot will timeout.
+        return "\n"
+    elif short_command == DICTPROXY_HELLO_CHAR:
+        return  # no version checking
 
-    def handle_set(self, addr, parts):
+    if short_command not in (DICTPROXY_TRANSACTION_CHARS):
+        logging.warning("unknown dictproxy request: %r", msg)
+        return
+
+    transaction_id = parts[0]
+
+    if short_command == DICTPROXY_BEGIN_TRANSACTION_CHAR:
+        addr = parts[1]
+        transactions[transaction_id] = dict(addr=addr, res="O\n")
+    elif short_command == DICTPROXY_COMMIT_TRANSACTION_CHAR:
+        # each set devicetoken operation persists directly
+        # and does not wait until a "commit" comes
+        # because our dovecot config does not involve
+        # multiple set-operations in a single commit
+        return transactions.pop(transaction_id)["res"]
+    elif short_command == DICTPROXY_SET_CHAR:
         # For documentation on key structure see
         # https://github.com/dovecot/core/blob/main/src/lib-storage/mailbox-attribute.h
+
         keyname = parts[1].split("/")
         value = parts[2] if len(parts) > 2 else ""
-        if keyname[0] == "priv" and keyname[2] == self.metadata.DEVICETOKEN_KEY:
-            self.metadata.add_token_to_addr(addr, value)
-            return True
+        addr = transactions[transaction_id]["addr"]
+        if keyname[0] == "priv" and keyname[2] == metadata.DEVICETOKEN_KEY:
+            metadata.add_token_to_addr(addr, value)
         elif keyname[0] == "priv" and keyname[2] == "messagenew":
-            self.notifier.new_message_for_addr(addr, self.metadata)
-            return True
+            notifier.new_message_for_addr(addr, metadata)
+        else:
+            # Transaction failed.
+            transactions[transaction_id]["res"] = "F\n"
 
-        return False
+
+class ThreadedUnixStreamServer(ThreadingMixIn, UnixStreamServer):
+    request_queue_size = 100
 
 
 def main():
-    socket, config_path = sys.argv[1:]
+    socket, vmail_dir, config_path = sys.argv[1:]
 
     config = read_config(config_path)
     iroh_relay = config.iroh_relay
-    mail_domain = config.mail_domain
 
-    vmail_dir = config.mailboxes_dir
+    vmail_dir = Path(vmail_dir)
     if not vmail_dir.exists():
         logging.error("vmail dir does not exist: %r", vmail_dir)
         return 1
@@ -141,11 +144,23 @@ def main():
     notifier = Notifier(queue_dir)
     notifier.start_notification_threads(metadata.remove_token_from_addr)
 
-    dictproxy = MetadataDictProxy(
-        notifier=notifier,
-        metadata=metadata,
-        iroh_relay=iroh_relay,
-        turn_hostname=mail_domain,
-    )
+    class Handler(StreamRequestHandler):
+        def handle(self):
+            try:
+                handle_dovecot_protocol(
+                    self.rfile, self.wfile, notifier, metadata, iroh_relay
+                )
+            except Exception:
+                logging.exception("Exception in the dovecot dictproxy handler")
+                raise
 
-    dictproxy.serve_forever_from_socket(socket)
+    try:
+        os.unlink(socket)
+    except FileNotFoundError:
+        pass
+
+    with ThreadedUnixStreamServer(socket, Handler) as server:
+        try:
+            server.serve_forever()
+        except KeyboardInterrupt:
+            pass

chatmaild/src/chatmaild/metrics.py

@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 import sys
+import time
 from pathlib import Path
 
 
@@ -11,21 +12,13 @@ def main(vmail_dir=None):
     ci_accounts = 0
 
     for path in Path(vmail_dir).iterdir():
-        if not path.joinpath("cur").is_dir():
-            continue
         accounts += 1
         if path.name[:3] in ("ci-", "ac_"):
             ci_accounts += 1
 
-    print("# HELP total number of accounts")
-    print("# TYPE accounts gauge")
-    print(f"accounts {accounts}")
-    print("# HELP number of CI accounts")
-    print("# TYPE ci_accounts gauge")
-    print(f"ci_accounts {ci_accounts}")
-    print("# HELP number of non-CI accounts")
-    print("# TYPE nonci_accounts gauge")
-    print(f"nonci_accounts {accounts - ci_accounts}")
+    timestamp = int(time.time() * 1000)
+    print(f"accounts {accounts} {timestamp}")
+    print(f"ci_accounts {ci_accounts} {timestamp}")
 
 
 if __name__ == "__main__":

chatmaild/src/chatmaild/migrate_db.py

@@ -1,63 +0,0 @@
-"""
-migration code from old sqlite databases into per-maildir "password" files
-where mtime reflects and is updated to be the "last-login" time.
-"""
-
-import logging
-import os
-import sqlite3
-import sys
-
-from chatmaild.config import read_config
-
-
-def get_all_rows(path):
-    assert path.exists()
-    uri = f"file:{path}?mode=ro"
-    sqlconn = sqlite3.connect(uri, timeout=60, isolation_level="DEFERRED", uri=True)
-    cur = sqlconn.cursor()
-    cur.execute("SELECT * from users")
-    rows = cur.fetchall()
-    sqlconn.close()
-    return rows
-
-
-def migrate_from_db_to_maildir(config, chunking=10000):
-    path = config.passdb_path
-    if not path.exists():
-        return
-
-    all_rows = get_all_rows(path)
-
-    # don't transfer special/CI accounts
-    rows = [row for row in all_rows if row[0][:3] not in ("ci-", "ac_")]
-
-    logging.info(f"ignoring {len(all_rows) - len(rows)} CI accounts")
-    logging.info(f"migrating {len(rows)} sqlite database passwords to user dirs")
-
-    for i, row in enumerate(rows):
-        addr = row[0]
-        enc_password = row[1]
-        user = config.get_user(addr)
-        user.set_password(enc_password)
-
-        if len(row) == 3 and row[2]:
-            timestamp = int(row[2])
-            user.set_last_login_timestamp(timestamp)
-
-        if i > 0 and i % chunking == 0:
-            logging.info(f"migration-progress: {i} passwords transferred")
-
-    logging.info("migration: all passwords migrated")
-    oldpath = config.passdb_path.with_suffix(config.passdb_path.suffix + ".old")
-    os.rename(config.passdb_path, oldpath)
-    for path in config.passdb_path.parent.iterdir():
-        if path.name.startswith(config.passdb_path.name + "-"):
-            path.unlink()
-    logging.info(f"migration: moved database to {oldpath!r}")
-
-
-if __name__ == "__main__":
-    config = read_config(sys.argv[1])
-    logging.basicConfig(level=logging.INFO)
-    migrate_from_db_to_maildir(config)

chatmaild/src/chatmaild/newemail.py

@@ -15,7 +15,7 @@ ALPHANUMERIC_PUNCT = string.ascii_letters + string.digits + string.punctuation
 
 
 def create_newemail_dict(config: Config):
-    user = "".join(random.choices(ALPHANUMERIC, k=config.username_max_length))
+    user = "".join(random.choices(ALPHANUMERIC, k=config.username_min_length))
     password = "".join(
         secrets.choice(ALPHANUMERIC_PUNCT)
         for _ in range(config.password_min_length + 3)

chatmaild/src/chatmaild/notifier.py

@@ -17,11 +17,11 @@ and which are scheduled for retry using exponential back-off timing.
 If a token notification would be scheduled more than DROP_DEADLINE seconds
 after its first attempt, it is dropped with a log error.
 
-Note that tokens are opaque to the notification machinery here
-and are encrypted foreclosing all ability to distinguish
+Note that tokens are completely opaque to the notification machinery here
+and will in the future be encrypted foreclosing all ability to distinguish
 which device token ultimately goes to which phone-provider notification service,
 or to understand the relation of "device tokens" and chatmail addresses.
-The meaning and format of tokens is basically a matter of chatmail Core and
+The meaning and format of tokens is basically a matter of Delta-Chat Core and
 the `notification.delta.chat` service.
 """
 
@@ -92,15 +92,10 @@ class Notifier:
     def requeue_persistent_queue_items(self):
         for queue_path in self.queue_dir.iterdir():
             if queue_path.name.endswith(".tmp"):
-                logging.warning(f"removing spurious queue item: {queue_path!r}")
-                queue_path.unlink()
-                continue
-            try:
-                queue_item = PersistentQueueItem.read_from_path(queue_path)
-            except ValueError:
-                logging.warning(f"removing spurious queue item: {queue_path!r}")
+                logging.warning("removing spurious queue item: %r", queue_path)
                 queue_path.unlink()
                 continue
+            queue_item = PersistentQueueItem.read_from_path(queue_path)
             self.queue_for_retry(queue_item)
 
     def queue_for_retry(self, queue_item, retry_num=0):
@@ -109,7 +104,7 @@ class Notifier:
         deadline = queue_item.start_ts + self.DROP_DEADLINE
         if retry_num >= len(self.retry_queues) or when > deadline:
             queue_item.delete()
-            logging.error(f"notification exceeded deadline: {queue_item.token!r}")
+            logging.error("notification exceeded deadline: %r", queue_item.token)
             return
 
         self.retry_queues[retry_num].put((when, queue_item))
@@ -167,5 +162,5 @@ class NotifyThread(Thread):
                 queue_item.delete()
                 return
 
-        logging.warning(f"Notification request failed: {res!r}")
+        logging.warning("Notification request failed: %r", res)
         self.notifier.queue_for_retry(queue_item, retry_num=self.retry_num + 1)

chatmaild/src/chatmaild/tests/mail-data/asm.eml

@@ -1,56 +0,0 @@
-From: {from_addr}
-To: {to_addr}
-Autocrypt-Setup-Message: v1
-Subject: Autocrypt Setup Message
-Date: Tue, 22 Jan 2019 12:56:29 +0100
-Content-type: multipart/mixed; boundary="Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ"
-
---Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
-Content-Type: text/plain
-
-This message contains all information to transfer your Autocrypt
-settings along with your secret key securely from your original
-device.
-
-To set up your new device for Autocrypt, please follow the
-instuctions that should be presented by your new device.
-
-You can keep this message and use it as a backup for your secret
-key. If you want to do this, you should write down the Setup Code
-and store it securely.
---Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
-Content-Type: application/autocrypt-setup
-Content-Disposition: attachment; filename="autocrypt-setup-message.html"
-
-<html><body>
-<p>
-This is the Autocrypt setup file used to transfer settings and
-keys between clients. You can decrypt it using the Setup Code
-presented on your old device, and then import the contained key
-into your keyring.
-</p>
-
-<pre>
------BEGIN PGP MESSAGE-----
-Passphrase-Format: numeric9x4
-Passphrase-Begin: 17
-
-jA0EBwMCFAxADoCdzeX/0ukBlqI5+pfpKb751qd/7nLNbkpy3gVcaf1QwRPZYt40
-Ynp08UqRQ2g48ZlnzHLSwlTGOPTuv2Jt8ka+pgZ45xzvJSG2gau03xP4VsC271kR
-VmCjdb0Y6Rk96mAwfGzrkbaRQ9Z7fIoL866GOv6h9neiVIkp+JYlTV6ISD0ZQJ4Q
-I6dOQkB/TWZyVjtiJDOQHdfNWliA6NtqaLq19wlu9L5xXjuNpY95KwR8EJXWe0+o
-Y3d2U/KxOAkXKghP2Qg1GtlPVeGC5T4p03TGI6pzKT+kHX6Rrm9wK6sM9aTquMmF
-Vok84Jg1DFnwivWC2RILR81rXi7k/+Y6MUbveFgJ9cQduqpxnmD7TjOblYu7M6zp
-YGAUxh8DRKlIMn2QsA++DBYQ6ACZvwuY8qTDLkqPDo4WqM313dsMJbyGjDdVE7EM
-PESS+RlABETpZXz8g/ycr6DIUNdlbPcmYlsBfHWDOuR2GFFTwmlv5slWS39dJv38
-E0eIe1CwdxI801Se7t7dUUS/ZF8wb6GlmxOcqGbF8eko1Z0S64IAm7/h13MRQCxI
-geQnHfGYVJ2FOimoCMEKwfa9x++RFTDW0u7spDC2uWvK/1viV8OfRppFhLr/kmKb
-18lWXuAz80DAjUDUsVqEq2MvJBJGoCJUEyjuRsLkHYRM5jYk4v50LyyR0Om73nWF
-nZBqmqNzdr7Xb9PHHdFhnEc0VvoYbrcM0RVYcEMW3YbmejM891j1d6Iv+/n/qND/
-NdebGrfWJMmFLf/iEkzTZ3/v5inW9LpWoRc94ioCjJTaEo8Rib6ARRFaJVIsmNXi
-YicFGO98D+zX+a2t9Yz6IpPajVslnOp6ScpmXgts/2XWD7oE+JgxSAqo/dLVsHgP
-Ufo=
-=pulM
------END PGP MESSAGE-----
-</pre></body></html>
---Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ--

chatmaild/src/chatmaild/tests/mail-data/encrypted.eml

@@ -1,6 +1,6 @@
 From: {from_addr}
 To: {to_addr}
-Subject: {subject}
+Subject: ...
 Date: Sun, 15 Oct 2023 16:43:21 +0000
 Message-ID: <Mr.UVyJWZmkCKM.hGzNc6glBE_@c2.testrun.org>
 In-Reply-To: <Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>

chatmaild/src/chatmaild/tests/mail-data/literal.eml

@@ -1,44 +1,44 @@
-From: {from_addr}
-To: {to_addr}
-Subject: ...
-Date: Sun, 15 Oct 2023 16:43:21 +0000
-Message-ID: <Mr.UVyJWZmkCKM.hGzNc6glBE_@c2.testrun.org>
-In-Reply-To: <Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>
-References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
-	<Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>
-Chat-Version: 1.0
-Autocrypt: addr={from_addr}; prefer-encrypt=mutual;
-	keydata=xjMEZSwWjhYJKwYBBAHaRw8BAQdAQBEhqeJh0GueHB6kF/DUQqYCxARNBVokg/AzT+7LqH
-	rNFzxiYXJiYXpAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUsFo4CGwMECwkIBwYVCAkKCwID
-	FgIBFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX9A4AEAnHWHp49eBCMHK5t66gYPiW
-	XQuB1mwUjzGfYWB+0RXUoA/0xcQ3FbUNlGKW7Blp6eMFfViv6Mv2d3kNSXACB6nmcMzjgEZSwWjhIK
-	KwYBBAGXVQEFAQEHQBpY5L2M1XHo0uxf8SX1wNLBp/OVvidoWHQF2Jz+kJsUAwEIB8J4BBgWCAAgBQ
-	JlLBaOAhsMFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX/INgEA37AJaNvruYsJVanP
-	IXnYw4CKd55UAwl8Zcy+M2diAbkA/0fHHcGV4r78hpbbL1Os52DPOdqYQRauIeJUeG+G6bQO
-MIME-Version: 1.0
-Content-Type: multipart/encrypted; protocol="application/pgp-encrypted";
-	boundary="YFrteb74qSXmggbOxZL9dRnhymywAi"
-
-
---YFrteb74qSXmggbOxZL9dRnhymywAi
-Content-Description: PGP/MIME version identification
-Content-Type: application/pgp-encrypted
-
-Version: 1
-
-
---YFrteb74qSXmggbOxZL9dRnhymywAi
-Content-Description: OpenPGP encrypted message
-Content-Disposition: inline; filename="encrypted.asc";
-Content-Type: application/octet-stream; name="encrypted.asc"
-
------BEGIN PGP MESSAGE-----
-
-yxJiAAAAAABIZWxsbyB3b3JsZCE=
-=1I/B
------END PGP MESSAGE-----
-
-
---YFrteb74qSXmggbOxZL9dRnhymywAi--
-
+From: {from_addr}
+
+To: {to_addr}
+
+Subject: ...
+
+Date: Sun, 15 Oct 2023 16:43:21 +0000
+
+Message-ID: <Mr.UVyJWZmkCKM.hGzNc6glBE_@c2.testrun.org>
+
+In-Reply-To: <Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>
+
+References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
+
+	<Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>
+
+Chat-Version: 1.0
+
+Autocrypt: addr={from_addr}; prefer-encrypt=mutual;
+
+	keydata=xjMEZSwWjhYJKwYBBAHaRw8BAQdAQBEhqeJh0GueHB6kF/DUQqYCxARNBVokg/AzT+7LqH
+
+	rNFzxiYXJiYXpAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUsFo4CGwMECwkIBwYVCAkKCwID
+
+	FgIBFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX9A4AEAnHWHp49eBCMHK5t66gYPiW
+
+	XQuB1mwUjzGfYWB+0RXUoA/0xcQ3FbUNlGKW7Blp6eMFfViv6Mv2d3kNSXACB6nmcMzjgEZSwWjhIK
+
+	KwYBBAGXVQEFAQEHQBpY5L2M1XHo0uxf8SX1wNLBp/OVvidoWHQF2Jz+kJsUAwEIB8J4BBgWCAAgBQ
+
+	JlLBaOAhsMFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX/INgEA37AJaNvruYsJVanP
+
+	IXnYw4CKd55UAwl8Zcy+M2diAbkA/0fHHcGV4r78hpbbL1Os52DPOdqYQRauIeJUeG+G6bQO
+
+MIME-Version: 1.0
+
+Content-Type: multipart/encrypted; protocol="application/pgp-encrypted";
+
+	boundary="YFrteb74qSXmggbOxZL9dRnhymywAi"
+
+
+
+
 

chatmaild/src/chatmaild/tests/mail-data/mailer-daemon.eml

@@ -1,46 +0,0 @@
-Date: Fri, 8 Jul 1994 09:21:47 -0400
-From: Mail Delivery Subsystem <MAILER-DAEMON@example.org>
-Subject: Returned mail: User unknown
-To: <owner-ups-mib@CS.UTK.EDU>
-Auto-Submitted: auto-replied 
-MIME-Version: 1.0
-Content-Type: multipart/report; report-type=delivery-status;
-      boundary="JAA13167.773673707/CS.UTK.EDU"
-
---JAA13167.773673707/CS.UTK.EDU
-content-type: text/plain; charset=us-ascii
-
-   ----- The following addresses had delivery problems -----
-<arathib@vnet.ibm.com>  (unrecoverable error)
-<wsnell@sdcc13.ucsd.edu>  (unrecoverable error)
-
---JAA13167.773673707/CS.UTK.EDU
-content-type: message/delivery-status
-
-Reporting-MTA: dns; cs.utk.edu
-
-Original-Recipient: rfc822;arathib@vnet.ibm.com
-Final-Recipient: rfc822;arathib@vnet.ibm.com
-Action: failed
-Status: 5.0.0 (permanent failure)
-Diagnostic-Code: smtp;
- 550 'arathib@vnet.IBM.COM' is not a registered gateway user
-Remote-MTA: dns; vnet.ibm.com
-
-Original-Recipient: rfc822;johnh@hpnjld.njd.hp.com
-Final-Recipient: rfc822;johnh@hpnjld.njd.hp.com
-Action: delayed
-Status: 4.0.0 (hpnjld.njd.jp.com: host name lookup failure)
-
-Original-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
-Final-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
-Action: failed
-Status: 5.0.0
-Diagnostic-Code: smtp; 550 user unknown
-Remote-MTA: dns; sdcc13.ucsd.edu
-
---JAA13167.773673707/CS.UTK.EDU
-content-type: message/rfc822
-
-[original message goes here]
---JAA13167.773673707/CS.UTK.EDU--

chatmaild/src/chatmaild/tests/mail-data/securejoin-vc-fake.eml

@@ -1,21 +0,0 @@
-Subject: Message from {from_addr}
-From: <{from_addr}>
-To: <{to_addr}>
-Date: Sun, 15 Oct 2023 16:43:25 +0000
-Message-ID: <Mr.78MWtlV7RAi.goCFzBhCYfy@c2.testrun.org>
-Chat-Version: 1.0
-Secure-Join: vc-request
-Secure-Join-Invitenumber: RANDOM-TOKEN
-MIME-Version: 1.0
-Content-Type: multipart/mixed; boundary="Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi"
-
-
---Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi
-Content-Type: text/plain; charset=utf-8
-
-Buy viagra!
-
-
---Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi--
-
-

chatmaild/src/chatmaild/tests/mail-data/securejoin-vc.eml

@@ -1,21 +0,0 @@
-Subject: Message from {from_addr}
-From: <{from_addr}>
-To: <{to_addr}>
-Date: Sun, 15 Oct 2023 16:43:25 +0000
-Message-ID: <Mr.78MWtlV7RAi.goCFzBhCYfy@c2.testrun.org>
-Chat-Version: 1.0
-Secure-Join: vc-request
-Secure-Join-Invitenumber: RANDOM-TOKEN
-MIME-Version: 1.0
-Content-Type: multipart/mixed; boundary="Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi"
-
-
---Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi
-Content-Type: text/plain; charset=utf-8
-
-Secure-Join: vc-request
-
-
---Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi--
-
-

chatmaild/src/chatmaild/tests/plugin.py

@@ -7,20 +7,16 @@ from email.parser import BytesParser
 from pathlib import Path
 
 import pytest
-
 from chatmaild.config import read_config, write_initial_config
+from chatmaild.database import Database
 
 
 @pytest.fixture
 def make_config(tmp_path):
     inipath = tmp_path.joinpath("chatmail.ini")
 
-    def make_conf(mail_domain, settings=None):
-        basedir = tmp_path.joinpath(f"vmail/{mail_domain}")
-        basedir.mkdir(parents=True, exist_ok=True)
-        overrides = settings.copy() if settings else {}
-        overrides["mailboxes_dir"] = str(basedir)
-        write_initial_config(inipath, mail_domain, overrides=overrides)
+    def make_conf(mail_domain):
+        write_initial_config(inipath, mail_domain=mail_domain)
         return read_config(inipath)
 
     return make_conf
@@ -36,11 +32,6 @@ def maildomain(example_config):
     return example_config.mail_domain
 
 
-@pytest.fixture
-def testaddr(maildomain):
-    return f"user.name@{maildomain}"
-
-
 @pytest.fixture
 def gencreds(maildomain):
     count = itertools.count()
@@ -59,6 +50,13 @@ def gencreds(maildomain):
     return lambda domain=None: next(gen(domain))
 
 
+@pytest.fixture()
+def db(tmpdir):
+    db_path = tmpdir / "passdb.sqlite"
+    print("database path:", db_path)
+    return Database(db_path)
+
+
 @pytest.fixture
 def maildata(request):
     try:
@@ -69,29 +67,11 @@ def maildata(request):
 
     assert datadir.exists(), datadir
 
-    def maildata(name, from_addr, to_addr, subject="[...]"):
+    def maildata(name, from_addr, to_addr):
         # Using `.read_bytes().decode()` instead of `.read_text()` to preserve newlines.
         data = datadir.joinpath(name).read_bytes().decode()
-        text = data.format(from_addr=from_addr, to_addr=to_addr, subject=subject)
-        return BytesParser(policy=policy.SMTP).parsebytes(text.encode())
+
+        text = data.format(from_addr=from_addr, to_addr=to_addr)
+        return BytesParser(policy=policy.default).parsebytes(text.encode())
 
     return maildata
-
-
-@pytest.fixture
-def mockout():
-    class MockOut:
-        captured_red = []
-        captured_green = []
-        captured_plain = []
-
-        def red(self, msg):
-            self.captured_red.append(msg)
-
-        def green(self, msg):
-            self.captured_green.append(msg)
-
-        def __call__(self, msg):
-            self.captured_plain.append(msg)
-
-    return MockOut()

chatmaild/src/chatmaild/tests/test_config.py

@@ -1,5 +1,3 @@
-import pytest
-
 from chatmaild.config import read_config
 
 
@@ -15,14 +13,6 @@ def test_read_config_basic(example_config):
     assert example_config.mail_domain == "chat.example.org"
 
 
-def test_read_config_basic_using_defaults(tmp_path, maildomain):
-    inipath = tmp_path.joinpath("chatmail.ini")
-    inipath.write_text(f"[params]\nmail_domain = {maildomain}")
-    example_config = read_config(inipath)
-    assert example_config.max_user_send_per_minute == 60
-    assert example_config.filtermail_smtp_port_incoming == 10081
-
-
 def test_read_config_testrun(make_config):
     config = make_config("something.testrun.org")
     assert config.mail_domain == "something.testrun.org"
@@ -35,41 +25,8 @@ def test_read_config_testrun(make_config):
     assert config.max_user_send_per_minute == 60
     assert config.max_mailbox_size == "100M"
     assert config.delete_mails_after == "20"
-    assert config.delete_large_after == "7"
     assert config.username_min_length == 9
     assert config.username_max_length == 9
     assert config.password_min_length == 9
     assert "privacy@testrun.org" in config.passthrough_recipients
     assert config.passthrough_senders == []
-
-
-def test_config_userstate_paths(make_config, tmp_path):
-    config = make_config("something.testrun.org")
-    mailboxes_dir = config.mailboxes_dir
-    passdb_path = config.passdb_path
-    assert mailboxes_dir.name == "something.testrun.org"
-    assert str(passdb_path) == "/home/vmail/passdb.sqlite"
-    assert config.mail_domain == "something.testrun.org"
-    path = config.get_user("user1@something.testrun.org").maildir
-    assert not path.exists()
-    assert path == mailboxes_dir.joinpath("user1@something.testrun.org")
-
-    with pytest.raises(ValueError):
-        config.get_user("")
-
-    with pytest.raises(ValueError):
-        config.get_user(None)
-
-    with pytest.raises(ValueError):
-        config.get_user("../some@something.testrun.org").maildir
-
-    with pytest.raises(ValueError):
-        config.get_user("..").maildir
-
-    with pytest.raises(ValueError):
-        config.get_user(".")
-
-
-def test_config_max_message_size(make_config, tmp_path):
-    config = make_config("something.testrun.org", dict(max_message_size="10000"))
-    assert config.max_message_size == 10000

chatmaild/src/chatmaild/tests/test_delete_inactive_users.py

@@ -1,59 +0,0 @@
-import time
-
-from chatmaild.delete_inactive_users import delete_inactive_users
-from chatmaild.doveauth import AuthDictProxy
-
-
-def test_login_timestamps(example_config):
-    testaddr = "someuser@chat.example.org"
-    user = example_config.get_user(testaddr)
-
-    # password file needs to be set because it's mtime tracks last-login time
-    user.set_password("1l2k3j1l2k3j123")
-    for i in range(10):
-        user.set_last_login_timestamp(86400 * 4 + i)
-        assert user.get_last_login_timestamp() == 86400 * 4
-
-
-def test_delete_inactive_users(example_config):
-    new = time.time()
-    old = new - (example_config.delete_inactive_users_after * 86400) - 1
-    dictproxy = AuthDictProxy(example_config)
-
-    def create_user(addr, last_login):
-        dictproxy.lookup_passdb(addr, "q9mr3faue")
-        user = example_config.get_user(addr)
-        user.maildir.joinpath("cur").mkdir()
-        user.maildir.joinpath("cur", "something").mkdir()
-        user.set_last_login_timestamp(timestamp=last_login)
-
-    # create some stale and some new accounts
-    to_remove = []
-    for i in range(150):
-        addr = f"oldold{i:03}@chat.example.org"
-        create_user(addr, last_login=old)
-        to_remove.append(addr)
-
-    remain = []
-    for i in range(5):
-        addr = f"newnew{i:03}@chat.example.org"
-        create_user(addr, last_login=new)
-        remain.append(addr)
-
-    # check pre and post-conditions for delete_inactive_users()
-
-    for addr in to_remove:
-        assert example_config.get_user(addr).maildir.exists()
-
-    delete_inactive_users(example_config)
-
-    for p in example_config.mailboxes_dir.iterdir():
-        assert not p.name.startswith("old")
-
-    for addr in to_remove:
-        assert not example_config.get_user(addr).maildir.exists()
-
-    for addr in remain:
-        userdir = example_config.get_user(addr).maildir
-        assert userdir.exists()
-        assert userdir.joinpath("password").read_text()

chatmaild/src/chatmaild/tests/test_doveauth.py

@@ -4,41 +4,29 @@ import queue
 import threading
 import traceback
 
-import pytest
-
 import chatmaild.doveauth
+import pytest
+from chatmaild.database import DBError
 from chatmaild.doveauth import (
-    AuthDictProxy,
+    get_user_data,
+    handle_dovecot_protocol,
+    handle_dovecot_request,
     is_allowed_to_create,
+    lookup_passdb,
 )
 from chatmaild.newemail import create_newemail_dict
 
 
-@pytest.fixture
-def dictproxy(example_config):
-    return AuthDictProxy(config=example_config)
-
-
-def test_basic(dictproxy, gencreds):
-    addr, password = gencreds()
-    dictproxy.lookup_passdb(addr, password)
-    data = dictproxy.lookup_userdb(addr)
+def test_basic(db, example_config):
+    lookup_passdb(db, example_config, "asdf12345@chat.example.org", "q9mr3faue")
+    data = get_user_data(db, example_config, "asdf12345@chat.example.org")
     assert data
-    data2 = dictproxy.lookup_passdb(addr, password)
+    data2 = lookup_passdb(
+        db, example_config, "asdf12345@chat.example.org", "q9mr3jewvadsfaue"
+    )
     assert data == data2
 
 
-def test_iterate_addresses(dictproxy):
-    addresses = []
-
-    for i in range(10):
-        addresses.append(f"asdf1234{i}@chat.example.org")
-        dictproxy.lookup_passdb(addresses[-1], "q9mr3faue")
-
-    res = dictproxy.iter_userdb()
-    assert set(res) == set(addresses)
-
-
 def test_invalid_username_length(example_config):
     config = example_config
     config.username_min_length = 6
@@ -53,83 +41,83 @@ def test_invalid_username_length(example_config):
     )
 
 
-def test_dont_overwrite_password_on_wrong_login(dictproxy):
+def test_dont_overwrite_password_on_wrong_login(db, example_config):
     """Test that logging in with a different password doesn't create a new user"""
-    res = dictproxy.lookup_passdb(
-        "newuser12@chat.example.org", "kajdlkajsldk12l3kj1983"
+    res = lookup_passdb(
+        db, example_config, "newuser12@chat.example.org", "kajdlkajsldk12l3kj1983"
     )
     assert res["password"]
-    res2 = dictproxy.lookup_passdb("newuser12@chat.example.org", "kajdslqwe")
+    res2 = lookup_passdb(db, example_config, "newuser12@chat.example.org", "kajdslqwe")
     # this function always returns a password hash, which is actually compared by dovecot.
     assert res["password"] == res2["password"]
 
 
-def test_nocreate_file(monkeypatch, tmpdir, dictproxy):
+def test_nocreate_file(db, monkeypatch, tmpdir, example_config):
     p = tmpdir.join("nocreate")
     p.write("")
     monkeypatch.setattr(chatmaild.doveauth, "NOCREATE_FILE", str(p))
-    dictproxy.lookup_passdb("newuser12@chat.example.org", "zequ0Aimuchoodaechik")
-    assert not dictproxy.lookup_userdb("newuser12@chat.example.org")
+    lookup_passdb(
+        db, example_config, "newuser12@chat.example.org", "zequ0Aimuchoodaechik"
+    )
+    assert not get_user_data(db, example_config, "newuser12@chat.example.org")
 
 
-def test_handle_dovecot_request(dictproxy):
-    transactions = {}
+def test_db_version(db):
+    assert db.get_schema_version() == 1
+
+
+def test_too_high_db_version(db):
+    with db.write_transaction() as conn:
+        conn.execute("PRAGMA user_version=%s;" % (999,))
+    with pytest.raises(DBError):
+        db.ensure_tables()
+
+
+def test_handle_dovecot_request(db, example_config):
     # Test that password can contain ", ', \ and /
     msg = (
         'Lshared/passdb/laksjdlaksjdlak\\\\sjdlk\\"12j\\\'3l1/k2j3123"'
         "some42123@chat.example.org\tsome42123@chat.example.org"
     )
-    res = dictproxy.handle_dovecot_request(msg, transactions)
+    res = handle_dovecot_request(msg, db, example_config)
     assert res
     assert res[0] == "O" and res.endswith("\n")
     userdata = json.loads(res[1:].strip())
-    assert userdata["home"].endswith("chat.example.org/some42123@chat.example.org")
+    assert (
+        userdata["home"]
+        == "/home/vmail/mail/chat.example.org/some42123@chat.example.org"
+    )
     assert userdata["uid"] == userdata["gid"] == "vmail"
     assert userdata["password"].startswith("{SHA512-CRYPT}")
 
 
-def test_handle_dovecot_protocol_hello_is_skipped(example_config, caplog):
-    dictproxy = AuthDictProxy(config=example_config)
+def test_handle_dovecot_protocol_hello_is_skipped(db, example_config, caplog):
     rfile = io.BytesIO(b"H3\t2\t0\t\tauth\n")
     wfile = io.BytesIO()
-    dictproxy.loop_forever(rfile, wfile)
+    handle_dovecot_protocol(rfile, wfile, db, example_config)
     assert wfile.getvalue() == b""
     assert not caplog.messages
 
 
-def test_handle_dovecot_protocol_user_not_exists(example_config):
-    dictproxy = AuthDictProxy(config=example_config)
+def test_handle_dovecot_protocol(db, example_config):
     rfile = io.BytesIO(
         b"H3\t2\t0\t\tauth\nLshared/userdb/foobar@chat.example.org\tfoobar@chat.example.org\n"
     )
     wfile = io.BytesIO()
-    dictproxy.loop_forever(rfile, wfile)
+    handle_dovecot_protocol(rfile, wfile, db, example_config)
     assert wfile.getvalue() == b"N\n"
 
 
-def test_handle_dovecot_protocol_iterate(gencreds, example_config):
-    dictproxy = AuthDictProxy(config=example_config)
-    dictproxy.lookup_passdb("asdf00000@chat.example.org", "q9mr3faue")
-    dictproxy.lookup_passdb("asdf11111@chat.example.org", "q9mr3faue")
-    rfile = io.BytesIO(b"H3\t2\t0\t\tauth\nI0\t0\tshared/userdb/")
-    wfile = io.BytesIO()
-    dictproxy.loop_forever(rfile, wfile)
-    lines = wfile.getvalue().decode("ascii").split("\n")
-    assert "Oshared/userdb/asdf00000@chat.example.org\t" in lines
-    assert "Oshared/userdb/asdf11111@chat.example.org\t" in lines
-    assert not lines[2]
-
-
-def test_50_concurrent_lookups_different_accounts(gencreds, dictproxy):
+def test_50_concurrent_lookups_different_accounts(db, gencreds, example_config):
     num_threads = 50
     req_per_thread = 5
     results = queue.Queue()
 
-    def lookup():
+    def lookup(db):
         for i in range(req_per_thread):
             addr, password = gencreds()
             try:
-                dictproxy.lookup_passdb(addr, password)
+                lookup_passdb(db, example_config, addr, password)
             except Exception:
                 results.put(traceback.format_exc())
             else:
@@ -137,7 +125,7 @@ def test_50_concurrent_lookups_different_accounts(gencreds, dictproxy):
 
     threads = []
     for i in range(num_threads):
-        thread = threading.Thread(target=lookup, daemon=True)
+        thread = threading.Thread(target=lookup, args=(db,), daemon=True)
         threads.append(thread)
 
     print(f"created {num_threads} threads, starting them and waiting for results")

chatmaild/src/chatmaild/tests/test_filedict.py

@@ -1,6 +1,4 @@
-import threading
-
-from chatmaild.filedict import FileDict, write_bytes_atomic
+from chatmaild.filedict import FileDict
 
 
 def test_basic(tmp_path):
@@ -19,21 +17,3 @@ def test_bad_marshal_file(tmp_path, caplog):
     fdict1.path.write_bytes(b"l12k3l12k3l")
     assert fdict1.read() == {}
     assert "corrupt" in caplog.records[0].msg
-
-
-def test_write_bytes_atomic_concurrent(tmp_path):
-    p = tmp_path.joinpath("somefile.ext")
-    write_bytes_atomic(p, b"hello")
-
-    threads = []
-    for i in range(30):
-        content = f"hello{i}".encode("ascii")
-        t = threading.Thread(target=lambda: write_bytes_atomic(p, content))
-        t.start()
-        threads.append(t)
-
-    for t in threads:
-        t.join()
-
-    assert p.read_text().strip() != "hello"
-    assert len(list(p.parent.iterdir())) == 1

chatmaild/src/chatmaild/tests/test_filtermail.py

@@ -1,12 +1,9 @@
 import pytest
-
 from chatmaild.filtermail import (
-    IncomingBeforeQueueHandler,
-    OutgoingBeforeQueueHandler,
+    BeforeQueueHandler,
     SendRateLimiter,
     check_armored_payload,
     check_encrypted,
-    is_securejoin,
 )
 
 
@@ -19,13 +16,7 @@ def maildomain():
 @pytest.fixture
 def handler(make_config, maildomain):
     config = make_config(maildomain)
-    return OutgoingBeforeQueueHandler(config)
-
-
-@pytest.fixture
-def inhandler(make_config, maildomain):
-    config = make_config(maildomain)
-    return IncomingBeforeQueueHandler(config)
+    return BeforeQueueHandler(config)
 
 
 def test_reject_forged_from(maildata, gencreds, handler):
@@ -36,14 +27,14 @@ def test_reject_forged_from(maildata, gencreds, handler):
     # test that the filter lets good mail through
     to_addr = gencreds()[0]
     env.content = maildata(
-        "encrypted.eml", from_addr=env.mail_from, to_addr=to_addr
+        "plain.eml", from_addr=env.mail_from, to_addr=to_addr
     ).as_bytes()
 
     assert not handler.check_DATA(envelope=env)
 
     # test that the filter rejects forged mail
     env.content = maildata(
-        "encrypted.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
+        "plain.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
     ).as_bytes()
     error = handler.check_DATA(envelope=env)
     assert "500" in error
@@ -62,29 +53,14 @@ def test_filtermail_no_encryption_detection(maildata):
     assert not check_encrypted(msg)
 
 
-def test_filtermail_securejoin_detection(maildata):
-    msg = maildata(
-        "securejoin-vc.eml", from_addr="some@example.org", to_addr="other@example.org"
-    )
-    assert is_securejoin(msg)
-
-    msg = maildata(
-        "securejoin-vc-fake.eml",
-        from_addr="some@example.org",
-        to_addr="other@example.org",
-    )
-    assert not is_securejoin(msg)
-
-
 def test_filtermail_encryption_detection(maildata):
-    msg = maildata(
-        "encrypted.eml",
-        from_addr="1@example.org",
-        to_addr="2@example.org",
-        subject="Subject does not matter, will be replaced anyway",
-    )
+    msg = maildata("encrypted.eml", from_addr="1@example.org", to_addr="2@example.org")
     assert check_encrypted(msg)
 
+    # if the subject is not "..." it is not considered ac-encrypted
+    msg.replace_header("Subject", "Click this link")
+    assert not check_encrypted(msg)
+
 
 def test_filtermail_no_literal_packets(maildata):
     """Test that literal OpenPGP packet is not considered an encrypted mail."""
@@ -96,7 +72,7 @@ def test_filtermail_unencrypted_mdn(maildata, gencreds):
     """Unencrypted MDNs should not pass."""
     from_addr = gencreds()[0]
     to_addr = gencreds()[0] + ".other"
-    msg = maildata("mdn.eml", from_addr=from_addr, to_addr=to_addr)
+    msg = maildata("mdn.eml", from_addr, to_addr)
 
     assert not check_encrypted(msg)
 
@@ -113,13 +89,13 @@ def test_send_rate_limiter():
             break
 
 
-def test_cleartext_excempt_privacy(maildata, gencreds, handler):
+def test_excempt_privacy(maildata, gencreds, handler):
     from_addr = gencreds()[0]
     to_addr = "privacy@testrun.org"
     handler.config.passthrough_recipients = [to_addr]
     false_to = "privacy@something.org"
 
-    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
+    msg = maildata("plain.eml", from_addr, to_addr)
 
     class env:
         mail_from = from_addr
@@ -134,102 +110,15 @@ def test_cleartext_excempt_privacy(maildata, gencreds, handler):
         rcpt_tos = [to_addr, false_to]
         content = msg.as_bytes()
 
-    assert "523" in handler.check_DATA(envelope=env2)
+    assert "500" in handler.check_DATA(envelope=env2)
 
 
-def test_cleartext_self_send_autocrypt_setup_message(maildata, gencreds, handler):
-    from_addr = gencreds()[0]
-    to_addr = from_addr
-
-    msg = maildata("asm.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    assert not handler.check_DATA(envelope=env)
-
-
-def test_cleartext_send_fails(maildata, gencreds, handler):
-    from_addr = gencreds()[0]
-    to_addr = gencreds()[0]
-
-    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    res = handler.check_DATA(envelope=env)
-    assert "523 Encryption Needed" in res
-
-
-def test_cleartext_incoming_fails(maildata, gencreds, inhandler):
-    from_addr = gencreds()[0]
-    to_addr, password = gencreds()
-
-    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    user = inhandler.config.get_user(to_addr)
-    user.set_password(password)
-    res = inhandler.check_DATA(envelope=env)
-    assert "523 Encryption Needed" in res
-
-    user.allow_incoming_cleartext()
-    assert not inhandler.check_DATA(envelope=env)
-
-
-def test_cleartext_incoming_mailer_daemon(maildata, gencreds, inhandler):
-    from_addr = "mailer-daemon@example.org"
-    to_addr = gencreds()[0]
-
-    msg = maildata("mailer-daemon.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    assert not inhandler.check_DATA(envelope=env)
-
-
-def test_cleartext_passthrough_domains(maildata, gencreds, handler):
-    from_addr = gencreds()[0]
-    to_addr = "privacy@x.y.z"
-    handler.config.passthrough_recipients = ["@x.y.z"]
-    false_to = "something@x.y"
-
-    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    # assert that None/no error is returned
-    assert not handler.check_DATA(envelope=env)
-
-    class env2:
-        mail_from = from_addr
-        rcpt_tos = [to_addr, false_to]
-        content = msg.as_bytes()
-
-    assert "523" in handler.check_DATA(envelope=env2)
-
-
-def test_cleartext_passthrough_senders(gencreds, handler, maildata):
+def test_passthrough_senders(gencreds, handler, maildata):
     acc1 = gencreds()[0]
     to_addr = "recipient@something.org"
     handler.config.passthrough_senders = [acc1]
 
-    msg = maildata("plain.eml", from_addr=acc1, to_addr=to_addr)
+    msg = maildata("plain.eml", acc1, to_addr)
 
     class env:
         mail_from = acc1
@@ -241,9 +130,8 @@ def test_cleartext_passthrough_senders(gencreds, handler, maildata):
 
 
 def test_check_armored_payload():
-    prefix = "-----BEGIN PGP MESSAGE-----\r\n"
-    comment = "Version: ProtonMail\r\n"
-    payload = """\r
+    payload = """-----BEGIN PGP MESSAGE-----\r
+\r
 wU4DSqFx0d1yqAoSAQdAYkX/ZN/Az4B0k7X47zKyWrXxlDEdS3WOy0Yf2+GJTFgg\r
 Zk5ql0mLG8Ze+ZifCS0XMO4otlemSyJ0K1ZPdFMGzUDBTgNqzkFabxXoXRIBB0AM\r
 755wlX41X6Ay3KhnwBq7yEqSykVH6F3x11iHPKraLCAGZoaS8bKKNy/zg5slda1X\r
@@ -276,28 +164,9 @@ UN4fiB0KR9JyG2ayUdNJVkXZSZLnHyRgiaadlpUo16LVvw==\r
 =b5Kp\r
 -----END PGP MESSAGE-----\r
 \r
-\r
 """
 
-    commented_payload = prefix + comment + payload
-    assert check_armored_payload(commented_payload, outgoing=False) == True
-    assert check_armored_payload(commented_payload, outgoing=True) == False
-
-    payload = prefix + payload
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
-
-    payload = payload.removesuffix("\r\n")
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
-
-    payload = payload.removesuffix("\r\n")
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
-
-    payload = payload.removesuffix("\r\n")
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
+    assert check_armored_payload(payload) == True
 
     payload = """-----BEGIN PGP MESSAGE-----\r
 \r
@@ -305,8 +174,7 @@ HELLOWORLD
 -----END PGP MESSAGE-----\r
 \r
 """
-    assert check_armored_payload(payload, outgoing=False) == False
-    assert check_armored_payload(payload, outgoing=True) == False
+    assert check_armored_payload(payload) == False
 
     payload = """-----BEGIN PGP MESSAGE-----\r
 \r
@@ -314,48 +182,4 @@ HELLOWORLD
 -----END PGP MESSAGE-----\r
 \r
 """
-    assert check_armored_payload(payload, outgoing=False) == False
-    assert check_armored_payload(payload, outgoing=True) == False
-
-    # Test payload using partial body length
-    # as generated by GopenPGP.
-    payload = """-----BEGIN PGP MESSAGE-----\r
-\r
-wV4DdCVjRfOT3TQSAQdAY5+pjT6mlCxPGdR3be4w7oJJRUGIPI/Vnh+mJxGSm34w\r
-LNlVc89S1g22uQYFif2sUJsQWbpoHpNkuWpkSgOaHmNvrZiY/YU5iv+cZ3LbmtUG\r
-0uoBisSHh9O1c+5sYZSbrvYZ1NOwlD7Fv/U5/Mw4E5+CjxfdgNGp5o3DDddzPK78\r
-jseDhdSXxnaiIJC93hxNX6R1RPt3G2gukyzx69wciPQShcF8zf3W3o75Ed7B8etV\r
-QEeB16xzdFhKa9JxdjTu3osgCs21IO7wpcFkjc7nZzlW6jPnELJJaNmv4yOOCjMp\r
-6YAkaN/BkL+jHTznHDuDsT5ilnTXpwHDU1Cm9PIx/KFcNCQnIB+2DcdIHPHUH1ci\r
-jvqoeXAVWjKXEjS7PqPFuP/xGbrWG2ugs+toXJOKbgRkExvKs1dwPFKrgghvCVbW\r
-AcKejQKAPArLwpkA7aD875TZQShvGt74fNs45XBlGOYOnNOAJ1KAmzrXLIDViyyB\r
-kDsmTBk785xofuCkjBpXSe6vsMprPzCteDfaUibh8FHeJjucxPerwuOPEmnogNaf\r
-YyL4+iy8H8I9/p7pmUqILprxTG0jTOtlk0bTVzeiF56W1xbtSEMuOo4oFbQTyOM2\r
-bKXaYo774Jm+rRtKAnnI2dtf9RpK19cog6YNzfYjesLKbXDsPZbN5rmwyFiCvvxC\r
-kQ6JLob+B2fPdY2gzy7LypxktS8Zi1HJcWDHJGVmQodaDLqKUObb4M26bXDe6oxI\r
-NS8PJz5exVbM3KhZnUOEn6PJRBBf5a/ZqxlhZPcQo/oBuhKpBRpO5kSDwPIUByu3\r
-UlXLSkpMqe9pUarAOEuQjfl2RVY7U+RrQYp4YP5keMO+i8NCefAFbowTTufO1JIq\r
-2nVgCi/QVnxZyEc9OYt/8AE3g4cdojE+vsSDifZLSWYIetpfrohHv3dT3StD1QRG\r
-0QE6qq6oKpg/IL0cjvuX4c7a7bslv2fXp8t75y37RU6253qdIebhxc/cRhPbc/yu\r
-p0YLyD4SrvKTLP2ZV95jT4IPEpqm4AN3QmiOzdtqR2gLyb62L8QfqI/FdwsIiRiM\r
-hqydwoqt/lfSqG1WKPh+6EkMkH+TDiCC1BQdbN1MNcyUtcjb35PR2c8Ld2TF3guA\r
-jLIqMt/Vb7hBoMb2FcsOYY25ka9oV62OwgKWLXnFzk+modMR5fzb4kxVVAYEqP+D\r
-T5KO1Vs76v1fyPGOq6BbBCvLwTqe/e6IZInJles4v5jrhnLcGKmNGivCUDe6X6NY\r
-UKNt5RsZllwDQpaAb5dMNhyrk8SgIE7TBI7rvqIdUCE52Vy+0JDxFg5olRpFUfO6\r
-/MyTW3Yo/ekk/npHr7iYYqJTCc21bDGLWQcIo/XO7WPxrKNWGBNPFnkRdw0MaKr4\r
-+cEM3V8NFnSEpC12xA+RX/CezuJtwXZK5MpG76eYqMO6qyC+c25YcFecEufDZDxx\r
-ZLqRszVRyxyWPtk/oIeQK2v9wOqY6N9/ff01gHz69vqYqN5bUw/QKZsmx1zW+gPw\r
-6x2tDK2BHeYl182gCbhlKISRFwCtbjqZSkiKWao/VtygHkw0fK34avJuyQ/X9YaN\r
-BRy+7Lf3VA53pnB5WJ1xwRXN8VDvmZeXzv2krHveCMemj0OjnRoCLu117xN0A5m9\r
-Fm/RoDix5PolDHtWTtr2m1n2hp2LHnj8at9lFEd0SKhAYHVL9KjzycwWODZRXt+x\r
-zGDDuooEeTvdY5NLyKcl4gETz1ZP4Ez5jGGjhPSwSpq1mU7UaJ9ZXXdr4KHyifW6\r
-ggNzNsGhXTap7IWZpTtqXABydfiBshmH2NjqtNDwBweJVSgP10+r0WhMWlaZs6xl\r
-V3o5yskJt6GlkwpJxZrTvN6Tiww/eW7HFV6NGf7IRSWY5tJc/iA7/92tOmkdvJ1q\r
-myLbG7cJB787QjplEyVe2P/JBO6xYvbkJLf9Q+HaviTO25rugRSrYsoKMDfO8VlQ\r
-1CcnTPVtApPZJEQzAWJEgVAM8uIlkqWJJMgyWT34sTkdBeCUFGloXQFs9Yxd0AGf\r
-/zHEkYZSTKpVSvAIGu4=\r
-=6iHb\r
------END PGP MESSAGE-----\r
-"""
-    assert check_armored_payload(payload, outgoing=False) == True
-    assert check_armored_payload(payload, outgoing=True) == True
+    assert check_armored_payload(payload) == False

chatmaild/src/chatmaild/tests/test_lastlogin.py

@@ -1,64 +0,0 @@
-import time
-
-from chatmaild.doveauth import AuthDictProxy
-from chatmaild.lastlogin import (
-    LastLoginDictProxy,
-)
-
-
-def test_handle_dovecot_request_last_login(testaddr, example_config):
-    dictproxy = LastLoginDictProxy(config=example_config)
-
-    authproxy = AuthDictProxy(config=example_config)
-    authproxy.lookup_passdb(testaddr, "1l2k3j1l2k3jl123")
-
-    dictproxy_transactions = {}
-
-    # Begin transaction
-    tx = "1111"
-    msg = f"B{tx}\t{testaddr}"
-    res = dictproxy.handle_dovecot_request(msg, dictproxy_transactions)
-    assert not res
-    assert dictproxy_transactions == {tx: dict(addr=testaddr, res="O\n")}
-
-    # set last-login info for user
-    user = dictproxy.config.get_user(testaddr)
-    timestamp = int(time.time())
-    msg = f"S{tx}\tshared/last-login/{testaddr}\t{timestamp}"
-    res = dictproxy.handle_dovecot_request(msg, dictproxy_transactions)
-    assert not res
-    assert len(dictproxy_transactions) == 1
-    read_timestamp = user.get_last_login_timestamp()
-    assert read_timestamp == timestamp // 86400 * 86400
-
-    # finish transaction
-    msg = f"C{tx}"
-    res = dictproxy.handle_dovecot_request(msg, dictproxy_transactions)
-    assert res == "O\n"
-    assert len(dictproxy_transactions) == 0
-
-
-def test_handle_dovecot_request_last_login_echobot(example_config):
-    dictproxy = LastLoginDictProxy(config=example_config)
-
-    authproxy = AuthDictProxy(config=example_config)
-    testaddr = f"echo@{example_config.mail_domain}"
-    authproxy.lookup_passdb(testaddr, "ignore")
-    user = dictproxy.config.get_user(testaddr)
-
-    transactions = {}
-
-    # set last-login info for user
-    tx = "1111"
-    msg = f"B{tx}\t{testaddr}"
-    res = dictproxy.handle_dovecot_request(msg, transactions)
-    assert not res
-    assert transactions == {tx: dict(addr=testaddr, res="O\n")}
-
-    timestamp = int(time.time())
-    msg = f"S{tx}\tshared/last-login/{testaddr}\t{timestamp}"
-    res = dictproxy.handle_dovecot_request(msg, transactions)
-    assert not res
-    assert len(transactions) == 1
-    read_timestamp = user.get_last_login_timestamp()
-    assert read_timestamp is None

chatmaild/src/chatmaild/tests/test_metadata.py

@@ -3,10 +3,10 @@ import time
 
 import pytest
 import requests
-
 from chatmaild.metadata import (
     Metadata,
-    MetadataDictProxy,
+    handle_dovecot_protocol,
+    handle_dovecot_request,
 )
 from chatmaild.notifier import (
     Notifier,
@@ -30,8 +30,8 @@ def metadata(tmp_path):
 
 
 @pytest.fixture
-def dictproxy(notifier, metadata):
-    return MetadataDictProxy(notifier=notifier, metadata=metadata)
+def testaddr():
+    return "user.name@example.org"
 
 
 @pytest.fixture
@@ -88,51 +88,51 @@ def test_notifier_remove_without_set(metadata, testaddr):
     assert not metadata.get_tokens_for_addr(testaddr)
 
 
-def test_handle_dovecot_request_lookup_fails(dictproxy, testaddr):
-    transactions = {}
-    res = dictproxy.handle_dovecot_request(
-        f"Lpriv/123/chatmail\t{testaddr}", transactions
+def test_handle_dovecot_request_lookup_fails(notifier, metadata, testaddr):
+    res = handle_dovecot_request(
+        f"Lpriv/123/chatmail\t{testaddr}", {}, notifier, metadata
     )
     assert res == "N\n"
 
 
-def test_handle_dovecot_request_happy_path(dictproxy, testaddr, token):
-    metadata = dictproxy.metadata
+def test_handle_dovecot_request_happy_path(notifier, metadata, testaddr, token):
     transactions = {}
-    notifier = dictproxy.notifier
 
     # set device token in a transaction
     tx = "1111"
     msg = f"B{tx}\t{testaddr}"
-    res = dictproxy.handle_dovecot_request(msg, transactions)
+    res = handle_dovecot_request(msg, transactions, notifier, metadata)
     assert not res and not metadata.get_tokens_for_addr(testaddr)
     assert transactions == {tx: dict(addr=testaddr, res="O\n")}
 
     msg = f"S{tx}\tpriv/guid00/devicetoken\t{token}"
-    res = dictproxy.handle_dovecot_request(msg, transactions)
+    res = handle_dovecot_request(msg, transactions, notifier, metadata)
     assert not res
     assert len(transactions) == 1
     assert metadata.get_tokens_for_addr(testaddr) == [token]
 
     msg = f"C{tx}"
-    res = dictproxy.handle_dovecot_request(msg, transactions)
+    res = handle_dovecot_request(msg, transactions, notifier, metadata)
     assert res == "O\n"
     assert len(transactions) == 0
     assert metadata.get_tokens_for_addr(testaddr) == [token]
 
     # trigger notification for incoming message
     tx2 = "2222"
-    assert dictproxy.handle_dovecot_request(f"B{tx2}\t{testaddr}", transactions) is None
+    assert (
+        handle_dovecot_request(f"B{tx2}\t{testaddr}", transactions, notifier, metadata)
+        is None
+    )
     msg = f"S{tx2}\tpriv/guid00/messagenew"
-    assert dictproxy.handle_dovecot_request(msg, transactions) is None
+    assert handle_dovecot_request(msg, transactions, notifier, metadata) is None
     queue_item = notifier.retry_queues[0].get()[1]
     assert queue_item.token == token
-    assert dictproxy.handle_dovecot_request(f"C{tx2}", transactions) == "O\n"
+    assert handle_dovecot_request(f"C{tx2}", transactions, notifier, metadata) == "O\n"
     assert not transactions
     assert queue_item.path.exists()
 
 
-def test_handle_dovecot_protocol_set_devicetoken(dictproxy):
+def test_handle_dovecot_protocol_set_devicetoken(metadata, notifier):
     rfile = io.BytesIO(
         b"\n".join(
             [
@@ -144,12 +144,12 @@ def test_handle_dovecot_protocol_set_devicetoken(dictproxy):
         )
     )
     wfile = io.BytesIO()
-    dictproxy.loop_forever(rfile, wfile)
+    handle_dovecot_protocol(rfile, wfile, notifier, metadata)
     assert wfile.getvalue() == b"O\n"
-    assert dictproxy.metadata.get_tokens_for_addr("user@example.org") == ["01234"]
+    assert metadata.get_tokens_for_addr("user@example.org") == ["01234"]
 
 
-def test_handle_dovecot_protocol_set_get_devicetoken(dictproxy):
+def test_handle_dovecot_protocol_set_get_devicetoken(metadata, notifier):
     rfile = io.BytesIO(
         b"\n".join(
             [
@@ -161,19 +161,19 @@ def test_handle_dovecot_protocol_set_get_devicetoken(dictproxy):
         )
     )
     wfile = io.BytesIO()
-    dictproxy.loop_forever(rfile, wfile)
-    assert dictproxy.metadata.get_tokens_for_addr("user@example.org") == ["01234"]
+    handle_dovecot_protocol(rfile, wfile, notifier, metadata)
+    assert metadata.get_tokens_for_addr("user@example.org") == ["01234"]
     assert wfile.getvalue() == b"O\n"
 
     rfile = io.BytesIO(
         b"\n".join([b"HELLO", b"Lpriv/0123/devicetoken\tuser@example.org"])
     )
     wfile = io.BytesIO()
-    dictproxy.loop_forever(rfile, wfile)
+    handle_dovecot_protocol(rfile, wfile, notifier, metadata)
     assert wfile.getvalue() == b"O01234\n"
 
 
-def test_handle_dovecot_protocol_iterate(dictproxy):
+def test_handle_dovecot_protocol_iterate(metadata, notifier):
     rfile = io.BytesIO(
         b"\n".join(
             [
@@ -183,7 +183,7 @@ def test_handle_dovecot_protocol_iterate(dictproxy):
         )
     )
     wfile = io.BytesIO()
-    dictproxy.loop_forever(rfile, wfile)
+    handle_dovecot_protocol(rfile, wfile, notifier, metadata)
     assert wfile.getvalue() == b"\n"
 
 
@@ -242,22 +242,6 @@ def test_requeue_removes_tmp_files(notifier, metadata, testaddr, caplog):
     assert queue_item.addr == testaddr
 
 
-def test_requeue_removes_invalid_files(notifier, metadata, testaddr, caplog):
-    metadata.add_token_to_addr(testaddr, "01234")
-    notifier.new_message_for_addr(testaddr, metadata)
-    # empty/invalid files should be ignored
-    p = notifier.queue_dir.joinpath("1203981203")
-    p.touch()
-    notifier2 = notifier.__class__(notifier.queue_dir)
-    notifier2.requeue_persistent_queue_items()
-    assert "spurious" in caplog.records[0].msg
-    assert not p.exists()
-    assert notifier2.retry_queues[0].qsize() == 1
-    when, queue_item = notifier2.retry_queues[0].get()
-    assert when <= int(time.time())
-    assert queue_item.addr == testaddr
-
-
 def test_start_and_stop_notification_threads(notifier, testaddr):
     threads = notifier.start_notification_threads(None)
     for retry_num, threadlist in threads.items():
@@ -314,7 +298,7 @@ def test_persistent_queue_items(tmp_path, testaddr, token):
     assert not queue_item < item2 and not item2 < queue_item
 
 
-def test_iroh_relay(dictproxy):
+def test_iroh_relay(metadata):
     rfile = io.BytesIO(
         b"\n".join(
             [
@@ -324,6 +308,5 @@ def test_iroh_relay(dictproxy):
         )
     )
     wfile = io.BytesIO()
-    dictproxy.iroh_relay = "https://example.org/"
-    dictproxy.loop_forever(rfile, wfile)
+    handle_dovecot_protocol(rfile, wfile, notifier, metadata, "https://example.org/")
     assert wfile.getvalue() == b"Ohttps://example.org/\n"

chatmaild/src/chatmaild/tests/test_metrics.py

@@ -2,23 +2,15 @@ from chatmaild.metrics import main
 
 
 def test_main(tmp_path, capsys):
-    paths = []
     for x in ("ci-asllkj", "ac_12l3kj", "qweqwe", "ci-l1k2j31l2k3"):
-        p = tmp_path.joinpath(x)
-        p.mkdir()
-        p.joinpath("cur").mkdir()
-        paths.append(p)
-
-    tmp_path.joinpath("nomailbox").mkdir()
-
+        tmp_path.joinpath(x).mkdir()
     main(tmp_path)
     out, _ = capsys.readouterr()
     d = {}
     for line in out.split("\n"):
-        if line.strip() and not line.startswith("#"):
-            name, num = line.split()
+        if line.strip():
+            name, num, _ = line.split()
             d[name] = int(num)
 
     assert d["accounts"] == 4
     assert d["ci_accounts"] == 3
-    assert d["nonci_accounts"] == 1

chatmaild/src/chatmaild/tests/test_migrate_db.py

@@ -1,67 +0,0 @@
-import sqlite3
-
-from chatmaild.migrate_db import migrate_from_db_to_maildir
-
-
-def test_migration_not_exists(tmp_path, example_config):
-    example_config.passdb_path = tmp_path.joinpath("sqlite")
-
-
-def test_migration(tmp_path, example_config, caplog):
-    passdb_path = tmp_path.joinpath("passdb.sqlite")
-    uri = f"file:{passdb_path}?mode=rwc"
-    sqlconn = sqlite3.connect(uri, timeout=60, uri=True)
-    sqlconn.execute(
-        """
-        CREATE TABLE users (
-            addr TEXT PRIMARY KEY,
-            password TEXT,
-            last_login INTEGER
-        )
-    """
-    )
-    all = {}
-
-    for i in range(500):
-        values = (f"somsom{i:03}@example.org", f"passwo{i:03}", i * 86400)
-        sqlconn.execute(
-            """
-            INSERT INTO users (addr, password, last_login)
-            VALUES (?, ?, ?)""",
-            values,
-        )
-        all[values[0]] = values[1:]
-
-    for i in range(500):
-        values = (f"pompom{i:03}@example.org", f"wopass{i:03}", "")
-        sqlconn.execute(
-            """
-            INSERT INTO users (addr, password, last_login)
-            VALUES (?, ?, ?)""",
-            values,
-        )
-        all[values[0]] = values[1:]
-
-    sqlconn.commit()
-    sqlconn.close()
-
-    assert passdb_path.stat().st_size > 10000
-
-    example_config.passdb_path = passdb_path
-
-    assert not caplog.records
-
-    migrate_from_db_to_maildir(example_config, chunking=500)
-    assert len(caplog.records) > 3
-
-    for path in example_config.mailboxes_dir.iterdir():
-        if "@" not in path.name:
-            continue
-        password, last_login = all.pop(path.name)
-        user = example_config.get_user(path.name)
-        if last_login:
-            assert user.get_last_login_timestamp() == last_login
-        assert password == user.get_userdb_dict()["password"]
-
-    assert not all
-    assert not example_config.passdb_path.exists()

chatmaild/src/chatmaild/tests/test_user.py

@@ -1,56 +0,0 @@
-def test_login_timestamp(testaddr, example_config):
-    user = example_config.get_user(testaddr)
-    user.set_password("someeqkjwelkqwjleqwe")
-    user.set_last_login_timestamp(100000)
-    assert user.get_last_login_timestamp() == 86400
-
-    user.set_last_login_timestamp(200000)
-    assert user.get_last_login_timestamp() == 86400 * 2
-
-
-def test_get_user_dict_not_set(testaddr, example_config, caplog):
-    user = example_config.get_user(testaddr)
-    assert not caplog.records
-    assert user.get_userdb_dict() == {}
-    assert len(caplog.records) == 0
-
-    user.set_password("")
-    assert user.get_userdb_dict() == {}
-    assert len(caplog.records) == 1
-
-
-def test_get_user_dict(make_config, tmp_path):
-    config = make_config("something.testrun.org")
-    addr = "user1@something.org"
-    user = config.get_user(addr)
-    enc_password = "l1k2j31lk2j3l1k23j123"
-    user.set_password(enc_password)
-    data = user.get_userdb_dict()
-    assert addr in str(data["home"])
-    assert data["uid"] == "vmail"
-    assert data["gid"] == "vmail"
-    assert data["password"] == enc_password
-
-
-def test_no_mailboxes_dir(testaddr, example_config, tmp_path):
-    p = tmp_path.joinpath("a", "mailboxes")
-    example_config.mailboxes_dir = p
-
-    user = example_config.get_user(testaddr)
-    user.set_password("someeqkjwelkqwjleqwe")
-    user.set_last_login_timestamp(100000)
-    assert user.get_last_login_timestamp() == 86400
-
-
-def test_set_get_cleartext_flag(testaddr, example_config, tmp_path):
-    p = tmp_path.joinpath("a", "mailboxes")
-    example_config.mailboxes_dir = p
-
-    user = example_config.get_user(testaddr)
-    user.set_password("someeqkjwelkqwjleqwe")
-    user.set_last_login_timestamp(100000)
-    assert user.get_last_login_timestamp() == 86400
-
-    assert not user.is_incoming_cleartext_ok()
-    user.allow_incoming_cleartext()
-    assert user.is_incoming_cleartext_ok()

chatmaild/src/chatmaild/turnserver.py

@@ -1,9 +0,0 @@
-#!/usr/bin/env python3
-import socket
-
-
-def turn_credentials() -> str:
-    with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_socket:
-        client_socket.connect("/run/chatmail-turn/turn.socket")
-        with client_socket.makefile("rb") as file:
-            return file.readline().decode("utf-8")

chatmaild/src/chatmaild/user.py

@@ -1,84 +0,0 @@
-import logging
-import os
-
-from chatmaild.filedict import write_bytes_atomic
-
-
-def get_daytimestamp(timestamp) -> int:
-    return int(timestamp) // 86400 * 86400
-
-
-class User:
-    def __init__(self, maildir, addr, password_path, uid, gid):
-        self.maildir = maildir
-        self.addr = addr
-        self.password_path = password_path
-        self.enforce_E2EE_path = maildir.joinpath("enforceE2EEincoming")
-        self.uid = uid
-        self.gid = gid
-
-    @property
-    def can_track(self):
-        return "@" in self.addr and not self.addr.startswith("echo@")
-
-    def get_userdb_dict(self):
-        """Return a non-empty dovecot 'userdb' style dict
-        if the user has an existing non-empty password"""
-        try:
-            pw = self.password_path.read_text()
-        except FileNotFoundError:
-            return {}
-
-        if not pw:
-            logging.error(f"password is empty for: {self.addr}")
-            return {}
-
-        home = str(self.maildir)
-        return dict(addr=self.addr, home=home, uid=self.uid, gid=self.gid, password=pw)
-
-    def is_incoming_cleartext_ok(self):
-        return not self.enforce_E2EE_path.exists()
-
-    def allow_incoming_cleartext(self):
-        if self.enforce_E2EE_path.exists():
-            self.enforce_E2EE_path.unlink()
-
-    def set_password(self, enc_password):
-        """Set the specified password for this user.
-
-        This method can be called concurrently
-        but there is no guarantee which of the password-set calls will win.
-        """
-        self.maildir.mkdir(exist_ok=True, parents=True)
-        password = enc_password.encode("ascii")
-
-        try:
-            write_bytes_atomic(self.password_path, password)
-        except PermissionError:
-            if not self.addr.startswith("echo@"):
-                logging.error(f"could not write password for: {self.addr}")
-                raise
-        if not self.addr.startswith("echo@"):
-            self.enforce_E2EE_path.touch()
-
-    def set_last_login_timestamp(self, timestamp):
-        """Track login time with daily granularity
-        to minimize touching files and to minimize metadata leakage."""
-        if not self.can_track:
-            return
-        try:
-            mtime = int(os.stat(self.password_path).st_mtime)
-        except FileNotFoundError:
-            logging.error(f"Can not get last login timestamp for {self.addr}")
-            return
-
-        timestamp = get_daytimestamp(timestamp)
-        if mtime != timestamp:
-            os.utime(self.password_path, (timestamp, timestamp))
-
-    def get_last_login_timestamp(self):
-        if self.can_track:
-            try:
-                return int(self.password_path.stat().st_mtime)
-            except FileNotFoundError:
-                pass

cmdeploy/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 name = "cmdeploy"
 version = "0.2"
 dependencies = [
-  "pyinfra>=3",
+  "pyinfra",
   "pillow",
   "qrcode",
   "markdown",
@@ -18,7 +18,6 @@ dependencies = [
   "ruff",
   "pytest",
   "pytest-xdist", 
-  "execnet",
   "imap_tools",
 ]
 
@@ -41,6 +40,3 @@ lint.select = [
   "PLE", # Pylint Error
   "PLW", # Pylint Warning
 ]
-lint.ignore = [
-  "PLC0415" # import-outside-top-level
-]

cmdeploy/src/cmdeploy/__init__.py

@@ -7,35 +7,17 @@ import io
 import shutil
 import subprocess
 import sys
-from io import StringIO
 from pathlib import Path
 
 from chatmaild.config import Config, read_config
-from pyinfra import facts, host, logger
-from pyinfra.api import FactBase
-from pyinfra.facts.files import File, Sha256File
-from pyinfra.facts.server import Sysctl
+from pyinfra import host
+from pyinfra.facts.files import File
 from pyinfra.facts.systemd import SystemdEnabled
 from pyinfra.operations import apt, files, pip, server, systemd
 
 from .acmetool import deploy_acmetool
 
 
-class Port(FactBase):
-    """
-    Returns the process occuping a port.
-    """
-
-    def command(self, port: int) -> str:
-        return (
-            "ss -lptn 'src :%d' | awk 'NR>1 {print $6,$7}' | sed 's/users:((\"//;s/\".*//'"
-            % (port,)
-        )
-
-    def process(self, output: [str]) -> str:
-        return output[0]
-
-
 def _build_chatmaild(dist_dir) -> None:
     dist_dir = Path(dist_dir).resolve()
     if dist_dir.exists():
@@ -96,11 +78,6 @@ def _install_remote_venv_with_chatmaild(config) -> None:
         always_copy=True,
     )
 
-    apt.packages(
-        name="install gcc and headers to build crypt_r source package",
-        packages=["gcc", "python3-dev"],
-    )
-
     server.shell(
         name=f"forced pip-install {dist_file.name}",
         commands=[
@@ -115,7 +92,7 @@ def _install_remote_venv_with_chatmaild(config) -> None:
         group="root",
         mode="644",
         config={
-            "mailboxes_dir": config.mailboxes_dir,
+            "mail_domain": config.mail_domain,
             "execpath": f"{remote_venv_dir}/bin/chatmail-metrics",
         },
     )
@@ -124,15 +101,11 @@ def _install_remote_venv_with_chatmaild(config) -> None:
     for fn in (
         "doveauth",
         "filtermail",
-        "filtermail-incoming",
         "echobot",
         "chatmail-metadata",
-        "lastlogin",
-        "turnserver",
     ):
-        execpath = fn if fn != "filtermail-incoming" else "filtermail"
         params = dict(
-            execpath=f"{remote_venv_dir}/bin/{execpath}",
+            execpath=f"{remote_venv_dir}/bin/{fn}",
             config_path=remote_chatmail_inipath,
             remote_venv_dir=remote_venv_dir,
             mail_domain=config.mail_domain,
@@ -236,37 +209,51 @@ def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
         server.shell(
             name="Generate OpenDKIM domain keys",
             commands=[
-                f"/usr/sbin/opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
+                f"opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
             ],
-            _use_su_login=True,
-            _su_user="opendkim",
+            _sudo=True,
+            _sudo_user="opendkim",
         )
 
-    service_file = files.put(
-        name="Configure opendkim to restart once a day",
-        src=importlib.resources.files(__package__).joinpath("opendkim/systemd.conf"),
-        dest="/etc/systemd/system/opendkim.service.d/10-prevent-memory-leak.conf",
-    )
-    need_restart |= service_file.changed
-
     return need_restart
 
 
-def _uninstall_mta_sts_daemon() -> None:
-    # Remove configuration.
-    files.file("/etc/mta-sts-daemon.yml", present=False)
+def _install_mta_sts_daemon() -> bool:
+    need_restart = False
 
-    files.directory("/usr/local/lib/postfix-mta-sts-resolver", present=False)
-
-    files.file("/etc/systemd/system/mta-sts-daemon.service", present=False)
-
-    systemd.service(
-        name="Stop MTA-STS daemon",
-        service="mta-sts-daemon.service",
-        daemon_reload=True,
-        running=False,
-        enabled=False,
+    config = files.put(
+        name="upload postfix-mta-sts-resolver config",
+        src=importlib.resources.files(__package__).joinpath(
+            "postfix/mta-sts-daemon.yml"
+        ),
+        dest="/etc/mta-sts-daemon.yml",
+        user="root",
+        group="root",
+        mode="644",
     )
+    need_restart |= config.changed
+
+    server.shell(
+        name="install postfix-mta-sts-resolver with pip",
+        commands=[
+            "python3 -m virtualenv /usr/local/lib/postfix-mta-sts-resolver",
+            "/usr/local/lib/postfix-mta-sts-resolver/bin/pip install postfix-mta-sts-resolver",
+        ],
+    )
+
+    systemd_unit = files.put(
+        name="upload mta-sts-daemon systemd unit",
+        src=importlib.resources.files(__package__).joinpath(
+            "postfix/mta-sts-daemon.service"
+        ),
+        dest="/etc/systemd/system/mta-sts-daemon.service",
+        user="root",
+        group="root",
+        mode="644",
+    )
+    need_restart |= systemd_unit.changed
+
+    return need_restart
 
 
 def _configure_postfix(config: Config, debug: bool = False) -> bool:
@@ -280,7 +267,6 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
         group="root",
         mode="644",
         config=config,
-        disable_ipv6=config.disable_ipv6,
     )
     need_restart |= main_config.changed
 
@@ -319,40 +305,6 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
     return need_restart
 
 
-def _install_dovecot_package(package: str, arch: str):
-    arch = "amd64" if arch == "x86_64" else arch
-    arch = "arm64" if arch == "aarch64" else arch
-    url = f"https://download.delta.chat/dovecot/dovecot-{package}_2.3.21%2Bdfsg1-3_{arch}.deb"
-    deb_filename = "/root/" + url.split("/")[-1]
-
-    match (package, arch):
-        case ("core", "amd64"):
-            sha256 = "43f593332e22ac7701c62d58b575d2ca409e0f64857a2803be886c22860f5587"
-        case ("core", "arm64"):
-            sha256 = "4d21eba1a83f51c100f08f2e49f0c9f8f52f721ebc34f75018e043306da993a7"
-        case ("imapd", "amd64"):
-            sha256 = "8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86"
-        case ("imapd", "arm64"):
-            sha256 = "178fa877ddd5df9930e8308b518f4b07df10e759050725f8217a0c1fb3fd707f"
-        case ("lmtpd", "amd64"):
-            sha256 = "2f69ba5e35363de50962d42cccbfe4ed8495265044e244007d7ccddad77513ab"
-        case ("lmtpd", "arm64"):
-            sha256 = "89f52fb36524f5877a177dff4a713ba771fd3f91f22ed0af7238d495e143b38f"
-        case _:
-            apt.packages(packages=[f"dovecot-{package}"])
-            return
-
-    files.download(
-        name=f"Download dovecot-{package}",
-        src=url,
-        dest=deb_filename,
-        sha256sum=sha256,
-        cache_time=60 * 60 * 24 * 365 * 10,  # never redownload the package
-    )
-
-    apt.deb(name=f"Install dovecot-{package}", src=deb_filename)
-
-
 def _configure_dovecot(config: Config, debug: bool = False) -> bool:
     """Configures Dovecot IMAP server."""
     need_restart = False
@@ -365,7 +317,6 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
         mode="644",
         config=config,
         debug=debug,
-        disable_ipv6=config.disable_ipv6,
     )
     need_restart |= main_config.changed
     auth_config = files.put(
@@ -387,6 +338,20 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
     )
     need_restart |= lua_push_notification_script.changed
 
+    sieve_script = files.put(
+        src=importlib.resources.files(__package__).joinpath("dovecot/default.sieve"),
+        dest="/etc/dovecot/default.sieve",
+        user="root",
+        group="root",
+        mode="644",
+    )
+    need_restart |= sieve_script.changed
+    if sieve_script.changed:
+        server.shell(
+            name="compile sieve script",
+            commands=["/usr/bin/sievec /etc/dovecot/default.sieve"],
+        )
+
     files.template(
         src=importlib.resources.files(__package__).joinpath("dovecot/expunge.cron.j2"),
         dest="/etc/cron.d/expunge",
@@ -396,14 +361,18 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
         config=config,
     )
 
+    files.put(
+            src=importlib.resources.files(__package__).joinpath("dovecot/remove-seen.py"),
+            dest="/usr/local/bin/remove-seen.py",
+            user="root",
+            group="root",
+            mode="755"
+    )
+
     # as per https://doc.dovecot.org/configuration_manual/os/
     # it is recommended to set the following inotify limits
     for name in ("max_user_instances", "max_user_watches"):
         key = f"fs.inotify.{name}"
-        if host.get_fact(Sysctl)[key] > 65535:
-            # Skip updating limits if already sufficient
-            # (enables running in incus containers where sysctl readonly)
-            continue
         server.sysctl(
             name=f"Change {key}",
             key=key,
@@ -411,17 +380,10 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
             persist=True,
         )
 
-    timezone_env = files.line(
-        name="Set TZ environment variable",
-        path="/etc/environment",
-        line="TZ=:/etc/localtime",
-    )
-    need_restart |= timezone_env.changed
-
     return need_restart
 
 
-def _configure_nginx(config: Config, debug: bool = False) -> bool:
+def _configure_nginx(domain: str, debug: bool = False) -> bool:
     """Configures nginx HTTP server."""
     need_restart = False
 
@@ -431,8 +393,7 @@ def _configure_nginx(config: Config, debug: bool = False) -> bool:
         user="root",
         group="root",
         mode="644",
-        config={"domain_name": config.mail_domain},
-        disable_ipv6=config.disable_ipv6,
+        config={"domain_name": domain},
     )
     need_restart |= main_config.changed
 
@@ -442,7 +403,7 @@ def _configure_nginx(config: Config, debug: bool = False) -> bool:
         user="root",
         group="root",
         mode="644",
-        config={"domain_name": config.mail_domain},
+        config={"domain_name": domain},
     )
     need_restart |= autoconfig.changed
 
@@ -452,7 +413,7 @@ def _configure_nginx(config: Config, debug: bool = False) -> bool:
         user="root",
         group="root",
         mode="644",
-        config={"domain_name": config.mail_domain},
+        config={"domain_name": domain},
     )
     need_restart |= mta_sts_config.changed
 
@@ -498,181 +459,16 @@ def check_config(config):
     return config
 
 
-def deploy_turn_server(config):
-    (url, sha256sum) = {
-        "x86_64": (
-            "https://github.com/chatmail/chatmail-turn/releases/download/v0.3/chatmail-turn-x86_64-linux",
-            "841e527c15fdc2940b0469e206188ea8f0af48533be12ecb8098520f813d41e4",
-        ),
-        "aarch64": (
-            "https://github.com/chatmail/chatmail-turn/releases/download/v0.3/chatmail-turn-aarch64-linux",
-            "a5fc2d06d937b56a34e098d2cd72a82d3e89967518d159bf246dc69b65e81b42",
-        ),
-    }[host.get_fact(facts.server.Arch)]
-
-    need_restart = False
-
-    existing_sha256sum = host.get_fact(Sha256File, "/usr/local/bin/chatmail-turn")
-    if existing_sha256sum != sha256sum:
-        server.shell(
-            name="Download chatmail-turn",
-            commands=[
-                f"(curl -L {url} >/usr/local/bin/chatmail-turn.new && (echo '{sha256sum} /usr/local/bin/chatmail-turn.new' | sha256sum -c) && mv /usr/local/bin/chatmail-turn.new /usr/local/bin/chatmail-turn)",
-                "chmod 755 /usr/local/bin/chatmail-turn",
-            ],
-        )
-        need_restart = True
-
-    source_path = importlib.resources.files(__package__).joinpath(
-        "service", "turnserver.service.f"
-    )
-    content = source_path.read_text().format(mail_domain=config.mail_domain).encode()
-
-    systemd_unit = files.put(
-        name="Upload turnserver.service",
-        src=io.BytesIO(content),
-        dest="/etc/systemd/system/turnserver.service",
-        user="root",
-        group="root",
-        mode="644",
-    )
-    need_restart |= systemd_unit.changed
-
-    systemd.service(
-        name="Setup turnserver service",
-        service="turnserver.service",
-        running=True,
-        enabled=True,
-        restarted=need_restart,
-        daemon_reload=systemd_unit.changed,
-    )
-
-
-def deploy_mtail(config):
-    # Uninstall mtail package, we are going to install a static binary.
-    apt.packages(name="Uninstall mtail", packages=["mtail"], present=False)
-
-    (url, sha256sum) = {
-        "x86_64": (
-            "https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_amd64.tar.gz",
-            "123c2ee5f48c3eff12ebccee38befd2233d715da736000ccde49e3d5607724e4",
-        ),
-        "aarch64": (
-            "https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_arm64.tar.gz",
-            "aa04811c0929b6754408676de520e050c45dddeb3401881888a092c9aea89cae",
-        ),
-    }[host.get_fact(facts.server.Arch)]
-
-    server.shell(
-        name="Download mtail",
-        commands=[
-            f"(echo '{sha256sum} /usr/local/bin/mtail' | sha256sum -c) || (curl -L {url} | gunzip | tar -x -f - mtail -O >/usr/local/bin/mtail.new && mv /usr/local/bin/mtail.new /usr/local/bin/mtail)",
-            "chmod 755 /usr/local/bin/mtail",
-        ],
-    )
-
-    # Using our own systemd unit instead of `/usr/lib/systemd/system/mtail.service`.
-    # This allows to read from journalctl instead of log files.
-    files.template(
-        src=importlib.resources.files(__package__).joinpath("mtail/mtail.service.j2"),
-        dest="/etc/systemd/system/mtail.service",
-        user="root",
-        group="root",
-        mode="644",
-        address=config.mtail_address or "127.0.0.1",
-        port=3903,
-    )
-
-    mtail_conf = files.put(
-        name="Mtail configuration",
-        src=importlib.resources.files(__package__).joinpath(
-            "mtail/delivered_mail.mtail"
-        ),
-        dest="/etc/mtail/delivered_mail.mtail",
-        user="root",
-        group="root",
-        mode="644",
-    )
-
-    systemd.service(
-        name="Start and enable mtail",
-        service="mtail.service",
-        running=bool(config.mtail_address),
-        enabled=bool(config.mtail_address),
-        restarted=mtail_conf.changed,
-    )
-
-
-def deploy_iroh_relay(config) -> None:
-    (url, sha256sum) = {
-        "x86_64": (
-            "https://github.com/n0-computer/iroh/releases/download/v0.35.0/iroh-relay-v0.35.0-x86_64-unknown-linux-musl.tar.gz",
-            "45c81199dbd70f8c4c30fef7f3b9727ca6e3cea8f2831333eeaf8aa71bf0fac1",
-        ),
-        "aarch64": (
-            "https://github.com/n0-computer/iroh/releases/download/v0.35.0/iroh-relay-v0.35.0-aarch64-unknown-linux-musl.tar.gz",
-            "f8ef27631fac213b3ef668d02acd5b3e215292746a3fc71d90c63115446008b1",
-        ),
-    }[host.get_fact(facts.server.Arch)]
-
-    apt.packages(
-        name="Install curl",
-        packages=["curl"],
-    )
-
-    need_restart = False
-
-    existing_sha256sum = host.get_fact(Sha256File, "/usr/local/bin/iroh-relay")
-    if existing_sha256sum != sha256sum:
-        server.shell(
-            name="Download iroh-relay",
-            commands=[
-                f"(curl -L {url} | gunzip | tar -x -f - ./iroh-relay -O >/usr/local/bin/iroh-relay.new && (echo '{sha256sum} /usr/local/bin/iroh-relay.new' | sha256sum -c) && mv /usr/local/bin/iroh-relay.new /usr/local/bin/iroh-relay)",
-                "chmod 755 /usr/local/bin/iroh-relay",
-            ],
-        )
-        need_restart = True
-
-    systemd_unit = files.put(
-        name="Upload iroh-relay systemd unit",
-        src=importlib.resources.files(__package__).joinpath("iroh-relay.service"),
-        dest="/etc/systemd/system/iroh-relay.service",
-        user="root",
-        group="root",
-        mode="644",
-    )
-    need_restart |= systemd_unit.changed
-
-    iroh_config = files.put(
-        name="Upload iroh-relay config",
-        src=importlib.resources.files(__package__).joinpath("iroh-relay.toml"),
-        dest="/etc/iroh-relay.toml",
-        user="root",
-        group="root",
-        mode="644",
-    )
-    need_restart |= iroh_config.changed
-
-    systemd.service(
-        name="Start and enable iroh-relay",
-        service="iroh-relay.service",
-        running=True,
-        enabled=config.enable_iroh_relay,
-        restarted=need_restart,
-    )
-
-
-def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
+def deploy_chatmail(config_path: Path) -> None:
     """Deploy a chat-mail instance.
 
     :param config_path: path to chatmail.ini
-    :param disable_mail: whether to disable postfix & dovecot
     """
     config = read_config(config_path)
     check_config(config)
     mail_domain = config.mail_domain
 
-    from .www import build_webpages, get_paths
+    from .www import build_webpages
 
     server.group(name="Create vmail group", group="vmail", system=True)
     server.user(name="Create vmail user", user="vmail", group="vmail", system=True)
@@ -690,7 +486,11 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         system=True,
     )
     server.user(name="Create echobot user", user="echobot", system=True)
-    server.user(name="Create iroh user", user="iroh", system=True)
+
+    server.shell(
+        name="Fix file owner in /home/vmail",
+        commands=["test -d /home/vmail && chown -R vmail:vmail /home/vmail"],
+    )
 
     # Add our OBS repository for dovecot_no_delay
     files.put(
@@ -707,56 +507,19 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         path="/etc/apt/sources.list",
         line="deb [signed-by=/etc/apt/keyrings/obs-home-deltachat.gpg] https://download.opensuse.org/repositories/home:/deltachat/Debian_12/ ./",
         escape_regex_characters=True,
-        present=False,
+        ensure_newline=True,
     )
 
-    if host.get_fact(Port, port=53) != "unbound":
-        files.line(
-            name="Add 9.9.9.9 to resolv.conf",
-            path="/etc/resolv.conf",
-            line="nameserver 9.9.9.9",
-        )
     apt.update(name="apt update", cache_time=24 * 3600)
-    apt.upgrade(name="upgrade apt packages", auto_remove=True)
 
     apt.packages(
         name="Install rsync",
         packages=["rsync"],
     )
 
-    deploy_turn_server(config)
-
     # Run local DNS resolver `unbound`.
     # `resolvconf` takes care of setting up /etc/resolv.conf
     # to use 127.0.0.1 as the resolver.
-    from cmdeploy.cmdeploy import Out
-
-    port_services = [
-        (["master", "smtpd"], 25),
-        ("unbound", 53),
-        ("acmetool", 80),
-        (["imap-login", "dovecot"], 143),
-        ("nginx", 443),
-        (["master", "smtpd"], 465),
-        (["master", "smtpd"], 587),
-        (["imap-login", "dovecot"], 993),
-        ("iroh-relay", 3340),
-        ("nginx", 8443),
-        (["master", "smtpd"], config.postfix_reinject_port),
-        (["master", "smtpd"], config.postfix_reinject_port_incoming),
-        ("filtermail", config.filtermail_smtp_port),
-        ("filtermail", config.filtermail_smtp_port_incoming),
-    ]
-    for service, port in port_services:
-        print(f"Checking if port {port} is available for {service}...")
-        running_service = host.get_fact(Port, port=port)
-        if running_service:
-            if running_service not in service:
-                Out().red(
-                    f"Deploy failed: port {port} is occupied by: {running_service}"
-                )
-                exit(1)
-
     apt.packages(
         name="Install unbound",
         packages=["unbound", "unbound-anchor", "dnsutils"],
@@ -775,13 +538,9 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         enabled=True,
     )
 
-    deploy_iroh_relay(config)
-
     # Deploy acmetool to have TLS certificates.
-    tls_domains = [mail_domain, f"mta-sts.{mail_domain}", f"www.{mail_domain}"]
     deploy_acmetool(
-        email=config.acme_email,
-        domains=tls_domains,
+        domains=[mail_domain, f"mta-sts.{mail_domain}", f"www.{mail_domain}"],
     )
 
     apt.packages(
@@ -795,14 +554,14 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         packages="postfix",
     )
 
-    if not "dovecot.service" in host.get_fact(SystemdEnabled):
-        _install_dovecot_package("core", host.get_fact(facts.server.Arch))
-        _install_dovecot_package("imapd", host.get_fact(facts.server.Arch))
-        _install_dovecot_package("lmtpd", host.get_fact(facts.server.Arch))
+    apt.packages(
+        name="Install Dovecot",
+        packages=["dovecot-imapd", "dovecot-lmtpd", "dovecot-sieve"],
+    )
 
     apt.packages(
         name="Install nginx",
-        packages=["nginx", "libnginx-mod-stream"],
+        packages=["nginx"],
     )
 
     apt.packages(
@@ -810,23 +569,19 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         packages=["fcgiwrap"],
     )
 
-    www_path, src_dir, build_dir = get_paths(config)
-    # if www_folder was set to a non-existing folder, skip upload
-    if not www_path.is_dir():
-        logger.warning("Building web pages is disabled in chatmail.ini, skipping")
-    else:
-        # if www_folder is a hugo page, build it
-        if build_dir:
-            www_path = build_webpages(src_dir, build_dir, config)
-        # if it is not a hugo page, upload it as is
-        files.rsync(f"{www_path}/", "/var/www/html", flags=["-avz", "--chown=www-data"])
+    www_path = importlib.resources.files(__package__).joinpath("../../../www").resolve()
+
+    build_dir = www_path.joinpath("build")
+    src_dir = www_path.joinpath("src")
+    build_webpages(src_dir, build_dir, config)
+    files.rsync(f"{build_dir}/", "/var/www/html", flags=["-avz"])
 
     _install_remote_venv_with_chatmaild(config)
     debug = False
     dovecot_need_restart = _configure_dovecot(config, debug=debug)
     postfix_need_restart = _configure_postfix(config, debug=debug)
-    nginx_need_restart = _configure_nginx(config)
-    _uninstall_mta_sts_daemon()
+    mta_sts_need_restart = _install_mta_sts_daemon()
+    nginx_need_restart = _configure_nginx(mail_domain)
 
     _remove_rspamd()
     opendkim_need_restart = _configure_opendkim(mail_domain, "opendkim")
@@ -836,27 +591,35 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         service="opendkim.service",
         running=True,
         enabled=True,
-        daemon_reload=opendkim_need_restart,
         restarted=opendkim_need_restart,
     )
 
+    systemd.service(
+        name="Start and enable MTA-STS daemon",
+        service="mta-sts-daemon.service",
+        daemon_reload=True,
+        running=True,
+        enabled=True,
+        restarted=mta_sts_need_restart,
+    )
+
     # Dovecot should be started before Postfix
     # because it creates authentication socket
     # required by Postfix.
     systemd.service(
-        name="disable dovecot for now" if disable_mail else "Start and enable Dovecot",
+        name="Start and enable Dovecot",
         service="dovecot.service",
-        running=False if disable_mail else True,
-        enabled=False if disable_mail else True,
-        restarted=dovecot_need_restart if not disable_mail else False,
+        running=True,
+        enabled=True,
+        restarted=dovecot_need_restart,
     )
 
     systemd.service(
-        name="disable postfix for now" if disable_mail else "Start and enable Postfix",
+        name="Start and enable Postfix",
         service="postfix.service",
-        running=False if disable_mail else True,
-        enabled=False if disable_mail else True,
-        restarted=postfix_need_restart if not disable_mail else False,
+        running=True,
+        enabled=True,
+        restarted=postfix_need_restart,
     )
 
     systemd.service(
@@ -867,19 +630,6 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         restarted=nginx_need_restart,
     )
 
-    systemd.service(
-        name="Start and enable fcgiwrap",
-        service="fcgiwrap.service",
-        running=True,
-        enabled=True,
-    )
-
-    systemd.service(
-        name="Restart echobot if postfix and dovecot were just started",
-        service="echobot.service",
-        restarted=postfix_need_restart and dovecot_need_restart,
-    )
-
     # This file is used by auth proxy.
     # https://wiki.debian.org/EtcMailName
     server.shell(
@@ -902,29 +652,10 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
         enabled=True,
         restarted=journald_conf.changed,
     )
-    files.directory(
-        name="Ensure old logs on disk are deleted",
-        path="/var/log/journal/",
-        present=False,
-    )
 
     apt.packages(
         name="Ensure cron is installed",
         packages=["cron"],
     )
-    try:
-        git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
-    except Exception:
-        git_hash = "unknown\n"
-    try:
-        git_diff = subprocess.check_output(["git", "diff"]).decode()
-    except Exception:
-        git_diff = ""
-    files.put(
-        name="Upload chatmail relay git commiit hash",
-        src=StringIO(git_hash + git_diff),
-        dest="/etc/chatmail-version",
-        mode="700",
-    )
 
-    deploy_mtail(config)
+

cmdeploy/src/cmdeploy/acmetool/__init__.py

@@ -1,5 +1,7 @@
 import importlib.resources
 
+from pyinfra import host
+from pyinfra.facts.systemd import SystemdStatus
 from pyinfra.operations import apt, files, server, systemd
 
 
@@ -52,6 +54,12 @@ def deploy_acmetool(email="", domains=[]):
         group="root",
         mode="644",
     )
+    if host.get_fact(SystemdStatus).get("nginx.service"):
+        systemd.service(
+            name="Stop nginx service to free port 80",
+            service="nginx",
+            running=False,
+        )
 
     systemd.service(
         name="Setup acmetool-redirector service",
@@ -62,6 +70,6 @@ def deploy_acmetool(email="", domains=[]):
     )
 
     server.shell(
-        name=f"Request certificate for: {', '.join(domains)}",
-        commands=[f"acmetool want --xlog.severity=debug {' '.join(domains)}"],
+        name=f"Request certificate for: { ', '.join(domains) }",
+        commands=[f"acmetool want --xlog.severity=debug { ' '.join(domains)}"],
     )

cmdeploy/src/cmdeploy/acmetool/acmetool.cron

@@ -1,4 +1,4 @@
 SHELL=/bin/sh
 PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
 MAILTO=root
-20 16 * * * root /usr/bin/acmetool --batch reconcile && systemctl reload dovecot && systemctl reload postfix && systemctl reload nginx
+20 16 * * * root /usr/bin/acmetool --batch reconcile && systemctl reload dovecot && systemctl reload postfix

cmdeploy/src/cmdeploy/acmetool/response-file.yaml.j2

@@ -1,2 +1,2 @@
 "acme-enter-email": "{{ email }}"
-"acme-agreement:https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf": true
+"acme-agreement:https://letsencrypt.org/documents/LE-SA-v1.4-April-3-2024.pdf": true

cmdeploy/src/cmdeploy/chatmail.zone.f

@@ -0,0 +1,15 @@
+{chatmail_domain}.                   A     {ipv4}
+{chatmail_domain}.                   AAAA  {ipv6}
+{chatmail_domain}.                   MX 10 {chatmail_domain}.
+_submission._tcp.{chatmail_domain}.  SRV 0 1 587 {chatmail_domain}.
+_submissions._tcp.{chatmail_domain}. SRV 0 1 465 {chatmail_domain}.
+_imap._tcp.{chatmail_domain}.        SRV 0 1 143 {chatmail_domain}.
+_imaps._tcp.{chatmail_domain}.       SRV 0 1 993 {chatmail_domain}.
+{chatmail_domain}.                   CAA 128 issue "letsencrypt.org;accounturi={acme_account_url}"
+{chatmail_domain}.                   TXT "v=spf1 a:{chatmail_domain} ~all"
+_dmarc.{chatmail_domain}.            TXT "v=DMARC1;p=reject;adkim=s;aspf=s"
+_mta-sts.{chatmail_domain}.          TXT "v=STSv1; id={sts_id}"
+mta-sts.{chatmail_domain}.           CNAME {chatmail_domain}.
+www.{chatmail_domain}.               CNAME {chatmail_domain}.
+{dkim_entry}
+_adsp._domainkey.{chatmail_domain}.  TXT "dkim=discardable"

cmdeploy/src/cmdeploy/chatmail.zone.j2

@@ -1,30 +0,0 @@
-;
-; Required DNS entries for chatmail servers 
-;
-{% if A %}
-{{ mail_domain }}.                   A     {{ A }}
-{% endif %}
-{% if AAAA %}
-{{ mail_domain }}.                   AAAA  {{ AAAA }}
-{% endif %}
-{{ mail_domain }}.                   MX 10 {{ mail_domain }}.
-_mta-sts.{{ mail_domain }}.          TXT "v=STSv1; id={{ sts_id }}"
-mta-sts.{{ mail_domain }}.           CNAME {{ mail_domain }}.
-www.{{ mail_domain }}.               CNAME {{ mail_domain }}.
-{{ dkim_entry }}
-
-;
-; Recommended DNS entries for interoperability and security-hardening
-;
-{{ mail_domain }}.                   TXT "v=spf1 a ~all"
-_dmarc.{{ mail_domain }}.            TXT "v=DMARC1;p=reject;adkim=s;aspf=s"
-
-{% if acme_account_url %}
-{{ mail_domain }}.                   CAA 0 issue "letsencrypt.org;accounturi={{ acme_account_url }}"
-{% endif %}
-_adsp._domainkey.{{ mail_domain }}.  TXT "dkim=discardable"
-
-_submission._tcp.{{ mail_domain }}.  SRV 0 1 587 {{ mail_domain }}.
-_submissions._tcp.{{ mail_domain }}. SRV 0 1 465 {{ mail_domain }}.
-_imap._tcp.{{ mail_domain }}.        SRV 0 1 143 {{ mail_domain }}.
-_imaps._tcp.{{ mail_domain }}.       SRV 0 1 993 {{ mail_domain }}.

cmdeploy/src/cmdeploy/cmdeploy.py

@@ -7,19 +7,15 @@ import argparse
 import importlib.resources
 import importlib.util
 import os
-import pathlib
 import shutil
 import subprocess
 import sys
 from pathlib import Path
 
-import pyinfra
 from chatmaild.config import read_config, write_initial_config
-from packaging import version
 from termcolor import colored
 
-from . import dns, remote
-from .sshexec import SSHExec, LocalExec
+from cmdeploy.dns import check_necessary_dns, show_dns
 
 #
 # cmdeploy sub commands and options
@@ -32,30 +28,16 @@ def init_cmd_options(parser):
         action="store",
         help="fully qualified DNS domain name for your chatmail instance",
     )
-    parser.add_argument(
-        "--force",
-        dest="recreate_ini",
-        action="store_true",
-        help="force reacreate ini file",
-    )
 
 
 def init_cmd(args, out):
     """Initialize chatmail config file."""
     mail_domain = args.chatmail_domain
-    inipath = args.inipath
     if args.inipath.exists():
-        if not args.recreate_ini:
-            print(f"[WARNING] Path exists, not modifying: {inipath}")
-            return 1
-        else:
-            print(
-                f"[WARNING] Force argument was provided, deleting config file: {inipath}"
-            )
-            inipath.unlink()
-
-    write_initial_config(inipath, mail_domain, overrides={})
-    out.green(f"created config file for {mail_domain} in {inipath}")
+        print(f"Path exists, not modifying: {args.inipath}")
+    else:
+        write_initial_config(args.inipath, mail_domain)
+        out.green(f"created config file for {mail_domain} in {args.inipath}")
 
 
 def run_cmd_options(parser):
@@ -65,116 +47,45 @@ def run_cmd_options(parser):
         action="store_true",
         help="don't actually modify the server",
     )
-    parser.add_argument(
-        "--disable-mail",
-        dest="disable_mail",
-        action="store_true",
-        help="install/upgrade the server, but disable postfix & dovecot for now",
-    )
-    parser.add_argument(
-        "--skip-dns-check",
-        dest="dns_check_disabled",
-        action="store_true",
-        help="disable checks nslookup for dns",
-    )
-    add_ssh_host_option(parser)
 
 
 def run_cmd(args, out):
     """Deploy chatmail services on the remote server."""
-
-    ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
-    sshexec = get_sshexec(ssh_host)
-    require_iroh = args.config.enable_iroh_relay
-    if not args.dns_check_disabled:
-        remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
-        if not dns.check_initial_remote_data(remote_data, print=out.red):
-            return 1
+    mail_domain = args.config.mail_domain
+    if not check_necessary_dns(
+        out,
+        mail_domain,
+    ):
+        sys.exit(1)
 
     env = os.environ.copy()
     env["CHATMAIL_INI"] = args.inipath
-    env["CHATMAIL_DISABLE_MAIL"] = "True" if args.disable_mail else ""
-    env["CHATMAIL_REQUIRE_IROH"] = "True" if require_iroh else ""
     deploy_path = importlib.resources.files(__package__).joinpath("deploy.py").resolve()
     pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
+    cmd = f"{pyinf} --ssh-user root {args.config.mail_domain} {deploy_path}"
 
-    cmd = f"{pyinf} --ssh-user root {ssh_host} {deploy_path} -y"
-    if ssh_host in ["localhost", "@docker"]:
-        cmd = f"{pyinf} @local {deploy_path} -y"
-
-    if version.parse(pyinfra.__version__) < version.parse("3"):
-        out.red("Please re-run scripts/initenv.sh to update pyinfra to version 3.")
-        return 1
-
-    try:
-        retcode = out.check_call(cmd, env=env)
-        if retcode == 0:
-            if not args.disable_mail:
-                print("\nYou can try out the relay by talking to this echo bot: ")
-                sshexec = SSHExec(args.config.mail_domain, verbose=args.verbose)
-                print(
-                    sshexec(
-                        call=remote.rshell.shell,
-                        kwargs=dict(command="cat /var/lib/echobot/invite-link.txt"),
-                    )
-                )
-            out.green("Deploy completed, call `cmdeploy dns` next.")
-        elif not remote_data["acme_account_url"]:
-            out.red("Deploy completed but letsencrypt not configured")
-            out.red("Run 'cmdeploy run' again")
-            retcode = 0
-        else:
-            out.red("Deploy failed")
-    except subprocess.CalledProcessError:
-        out.red("Deploy failed")
-        retcode = 1
-    return retcode
+    out.check_call(cmd, env=env)
+    print("Deploy completed, call `cmdeploy dns` next.")
 
 
 def dns_cmd_options(parser):
     parser.add_argument(
         "--zonefile",
         dest="zonefile",
-        type=pathlib.Path,
-        default=None,
-        help="write out a zonefile",
+        help="print the whole zonefile for deploying directly",
     )
-    add_ssh_host_option(parser)
 
 
 def dns_cmd(args, out):
-    """Check DNS entries and optionally generate dns zone file."""
-    ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
-    sshexec = get_sshexec(ssh_host, verbose=args.verbose)
-    remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
-    if not remote_data:
-        return 1
-
-    if not remote_data["acme_account_url"]:
-        out.red("could not get letsencrypt account url, please run 'cmdeploy run'")
-        return 1
-
-    if not remote_data["dkim_entry"]:
-        out.red("could not determine dkim_entry, please run 'cmdeploy run'")
-        return 1
-
-    zonefile = dns.get_filled_zone_file(remote_data)
-
-    if args.zonefile:
-        args.zonefile.write_text(zonefile)
-        out.green(f"DNS records successfully written to: {args.zonefile}")
-        return 0
-
-    retcode = dns.check_full_zone(
-        sshexec, remote_data=remote_data, zonefile=zonefile, out=out
-    )
-    return retcode
+    """Generate dns zone file."""
+    exit_code = show_dns(args, out)
+    exit(exit_code)
 
 
 def status_cmd(args, out):
     """Display status for online chatmail instance."""
 
-    sshexec = args.get_sshexec()
+    ssh = f"ssh root@{args.config.mail_domain}"
 
     out.green(f"chatmail domain: {args.config.mail_domain}")
     if args.config.privacy_mail:
@@ -182,8 +93,10 @@ def status_cmd(args, out):
     else:
         out.red("no privacy settings")
 
-    for line in sshexec(remote.rshell.get_systemd_running):
-        print(line)
+    s1 = "systemctl --type=service --state=running"
+    for line in out.shell_output(f"{ssh} -- {s1}").split("\n"):
+        if line.startswith("  "):
+            print(line)
 
 
 def test_cmd_options(parser):
@@ -212,7 +125,7 @@ def test_cmd(args, out):
         "-n4",
         "-rs",
         "-x",
-        "-v",
+        "-vrx",
         "--durations=5",
     ]
     if args.slow:
@@ -222,6 +135,14 @@ def test_cmd(args, out):
 
 
 def fmt_cmd_options(parser):
+    parser.add_argument(
+        "--verbose",
+        "-v",
+        dest="verbose",
+        action="store_true",
+        help="provide information on invocations",
+    )
+
     parser.add_argument(
         "--check",
         "-c",
@@ -251,6 +172,7 @@ def fmt_cmd(args, out):
 
     out.check_call(" ".join(format_args), quiet=not args.verbose)
     out.check_call(" ".join(check_args), quiet=not args.verbose)
+    return 0
 
 
 def bench_cmd(args, out):
@@ -286,6 +208,16 @@ class Out:
         color = "red" if red else ("green" if green else None)
         print(colored(msg, color), file=file)
 
+    def shell_output(self, arg, no_print=False, timeout=10):
+        if not no_print:
+            self(f"[$ {arg}]", file=sys.stderr)
+            output = subprocess.STDOUT
+        else:
+            output = subprocess.DEVNULL
+        return subprocess.check_output(
+            arg, shell=True, timeout=timeout, stderr=output
+        ).decode()
+
     def check_call(self, arg, env=None, quiet=False):
         if not quiet:
             self(f"[$ {arg}]", file=sys.stderr)
@@ -299,15 +231,6 @@ class Out:
         return proc.returncode
 
 
-def add_ssh_host_option(parser):
-    parser.add_argument(
-        "--ssh-host",
-        dest="ssh_host",
-        help="Run commands on 'localhost', via '@docker', or on a specific SSH host "
-             "instead of chatmail.ini's mail_domain.",
-    )
-
-
 def add_config_option(parser):
     parser.add_argument(
         "--config",
@@ -317,14 +240,6 @@ def add_config_option(parser):
         type=Path,
         help="path to the chatmail.ini file",
     )
-    parser.add_argument(
-        "--verbose",
-        "-v",
-        dest="verbose",
-        action="store_true",
-        default=False,
-        help="provide verbose logging",
-    )
 
 
 def add_subcommand(subparsers, func):
@@ -363,23 +278,12 @@ def get_parser():
     return parser
 
 
-def get_sshexec(ssh_host: str, verbose=True):
-    if ssh_host in ["localhost", "@local"]:
-        return LocalExec(verbose, docker=False)
-    elif ssh_host == "@docker":
-        return LocalExec(verbose, docker=True)
-    if verbose:
-        print(f"[ssh] login to {ssh_host}")
-    return SSHExec(ssh_host, verbose=verbose)
-
-
 def main(args=None):
-    """Provide main entry point for 'cmdeploy' CLI invocation."""
+    """Provide main entry point for 'xdcget' CLI invocation."""
     parser = get_parser()
     args = parser.parse_args(args=args)
     if not hasattr(args, "func"):
         return parser.parse_args(["-h"])
-
     out = Out()
     kwargs = {}
     if args.func.__name__ not in ("init_cmd", "fmt_cmd"):
@@ -397,6 +301,7 @@ def main(args=None):
         if res is None:
             res = 0
         return res
+
     except KeyboardInterrupt:
         out.red("KeyboardInterrupt")
         sys.exit(130)

cmdeploy/src/cmdeploy/deploy.py

@@ -11,9 +11,8 @@ def main():
         "CHATMAIL_INI",
         importlib.resources.files("cmdeploy").joinpath("../../../chatmail.ini"),
     )
-    disable_mail = bool(os.environ.get("CHATMAIL_DISABLE_MAIL"))
 
-    deploy_chatmail(config_path, disable_mail)
+    deploy_chatmail(config_path)
 
 
 if pyinfra.is_cli:

cmdeploy/src/cmdeploy/dns.py

@@ -1,70 +1,209 @@
 import datetime
 import importlib
+import subprocess
+import sys
 
-from jinja2 import Template
-
-from . import remote
+import requests
 
 
-def get_initial_remote_data(sshexec, mail_domain):
-    return sshexec.logged(
-        call=remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=mail_domain)
-    )
-
-
-def check_initial_remote_data(remote_data, *, print=print):
-    mail_domain = remote_data["mail_domain"]
-    if not remote_data["A"] and not remote_data["AAAA"]:
-        print(f"Missing A and/or AAAA DNS records for {mail_domain}!")
-    elif remote_data["MTA_STS"] != f"{mail_domain}.":
-        print("Missing MTA-STS CNAME record:")
-        print(f"mta-sts.{mail_domain}.   CNAME  {mail_domain}.")
-    elif remote_data["WWW"] != f"{mail_domain}.":
-        print("Missing www CNAME record:")
-        print(f"www.{mail_domain}.   CNAME  {mail_domain}.")
-    else:
-        return remote_data
-
-
-def get_filled_zone_file(remote_data):
-    sts_id = remote_data.get("sts_id")
-    if not sts_id:
-        remote_data["sts_id"] = datetime.datetime.now().strftime("%Y%m%d%H%M")
-
-    template = importlib.resources.files(__package__).joinpath("chatmail.zone.j2")
-    content = template.read_text()
-    zonefile = Template(content).render(**remote_data)
-    lines = [x.strip() for x in zonefile.split("\n") if x.strip()]
-    lines.append("")
-    zonefile = "\n".join(lines)
-    return zonefile
-
-
-def check_full_zone(sshexec, remote_data, out, zonefile) -> int:
-    """Check existing DNS records, optionally write them to zone file
-    and return (exitcode, remote_data) tuple."""
-
-    required_diff, recommended_diff = sshexec.logged(
-        remote.rdns.check_zonefile, kwargs=dict(zonefile=zonefile, verbose=False),
-    )
-
-    returncode = 0
-    if required_diff:
-        out.red("Please set required DNS entries at your DNS provider:\n")
-        for line in required_diff:
-            out(line)
-        out("")
-        returncode = 1
-    if remote_data.get("dkim_entry") in required_diff:
-        out(
-            "If the DKIM entry above does not work with your DNS provider, you can try this one:\n"
+class DNS:
+    def __init__(self, out, mail_domain):
+        self.session = requests.Session()
+        self.out = out
+        self.ssh = f"ssh root@{mail_domain} -- "
+        self.out.shell_output(
+            f"{ self.ssh }'apt-get update && apt-get install -y dnsutils'",
+            timeout=60,
+            no_print=True,
         )
-        out(remote_data.get("web_dkim_entry") + "\n")
-    if recommended_diff:
-        out("WARNING: these recommended DNS entries are not set:\n")
-        for line in recommended_diff:
-            out(line)
+        try:
+            self.shell(f"unbound-control flush_zone {mail_domain}")
+        except subprocess.CalledProcessError:
+            pass
 
-    if not (recommended_diff or required_diff):
-        out.green("Great! All your DNS entries are verified and correct.")
-    return returncode
+    def shell(self, cmd):
+        try:
+            return self.out.shell_output(f"{self.ssh}{cmd}", no_print=True)
+        except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
+            if "exit status 255" in str(e) or "timed out" in str(e):
+                self.out.red(f"Error: can't reach the server with: {self.ssh[:-4]}")
+                sys.exit(1)
+            else:
+                raise
+
+    def get_ipv4(self):
+        cmd = "ip a | grep 'inet ' | grep 'scope global' | grep -oE '[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}' | head -1"
+        return self.shell(cmd).strip()
+
+    def get_ipv6(self):
+        cmd = "ip a | grep inet6 | grep 'scope global' | sed -e 's#/64 scope global##' | sed -e 's#inet6##'"
+        return self.shell(cmd).strip()
+
+    def get(self, typ: str, domain: str) -> str:
+        """Get a DNS entry or empty string if there is none."""
+        dig_result = self.shell(f"dig -r -q {domain} -t {typ} +short")
+        line = dig_result.partition("\n")[0]
+        return line
+
+    def check_ptr_record(self, ip: str, mail_domain) -> bool:
+        """Check the PTR record for an IPv4 or IPv6 address."""
+        result = self.shell(f"dig -r -x {ip} +short").rstrip()
+        return result == f"{mail_domain}."
+
+
+def show_dns(args, out) -> int:
+    """Check existing DNS records, optionally write them to zone file, return exit code 0 or 1."""
+    template = importlib.resources.files(__package__).joinpath("chatmail.zone.f")
+    mail_domain = args.config.mail_domain
+    ssh = f"ssh root@{mail_domain}"
+    dns = DNS(out, mail_domain)
+
+    print("Checking your DKIM keys and DNS entries...")
+    try:
+        acme_account_url = out.shell_output(f"{ssh} -- acmetool account-url")
+    except subprocess.CalledProcessError:
+        print("Please run `cmdeploy run` first.")
+        return 1
+
+    dkim_selector = "opendkim"
+    dkim_pubkey = out.shell_output(
+        ssh + f" -- openssl rsa -in /etc/dkimkeys/{dkim_selector}.private"
+        " -pubout 2>/dev/null | awk '/-/{next}{printf(\"%s\",$0)}'"
+    )
+    dkim_entry_value = f"v=DKIM1;k=rsa;p={dkim_pubkey};s=email;t=s"
+    dkim_entry_str = ""
+    while len(dkim_entry_value) >= 255:
+        dkim_entry_str += '"' + dkim_entry_value[:255] + '" '
+        dkim_entry_value = dkim_entry_value[255:]
+    dkim_entry_str += '"' + dkim_entry_value + '"'
+    dkim_entry = f"{dkim_selector}._domainkey.{mail_domain}. TXT {dkim_entry_str}"
+
+    ipv6 = dns.get_ipv6()
+    reverse_ipv6 = dns.check_ptr_record(ipv6, mail_domain)
+    ipv4 = dns.get_ipv4()
+    reverse_ipv4 = dns.check_ptr_record(ipv4, mail_domain)
+    to_print = []
+
+    with open(template, "r") as f:
+        zonefile = (
+            f.read()
+            .format(
+                acme_account_url=acme_account_url,
+                sts_id=datetime.datetime.now().strftime("%Y%m%d%H%M"),
+                chatmail_domain=args.config.mail_domain,
+                dkim_entry=dkim_entry,
+                ipv6=ipv6,
+                ipv4=ipv4,
+            )
+            .strip()
+        )
+        try:
+            with open(args.zonefile, "w+") as zf:
+                zf.write(zonefile)
+                print(f"DNS records successfully written to: {args.zonefile}")
+            return 0
+        except TypeError:
+            pass
+        for raw_line in zonefile.splitlines():
+            line = raw_line.format(
+                acme_account_url=acme_account_url,
+                sts_id=datetime.datetime.now().strftime("%Y%m%d%H%M"),
+                chatmail_domain=args.config.mail_domain,
+                dkim_entry=dkim_entry,
+                ipv6=ipv6,
+            ).strip()
+            for typ in ["A", "AAAA", "CNAME", "CAA"]:
+                if f" {typ} " in line:
+                    domain, value = line.split(f" {typ} ")
+                    current = dns.get(typ, domain.strip()[:-1])
+                    if current != value.strip():
+                        to_print.append(line)
+            if " MX " in line:
+                domain, typ, prio, value = line.split()
+                current = dns.get(typ, domain[:-1])
+                if not current:
+                    to_print.append(line)
+                elif current.split()[1] != value:
+                    print(line.replace(prio, str(int(current[0]) + 1)))
+            if " SRV " in line:
+                domain, typ, prio, weight, port, value = line.split()
+                current = dns.get("SRV", domain[:-1])
+                if current != f"{prio} {weight} {port} {value}":
+                    to_print.append(line)
+            if " TXT " in line:
+                domain, value = line.split(" TXT ")
+                current = dns.get("TXT", domain.strip()[:-1])
+                if domain.startswith("_mta-sts."):
+                    if current:
+                        if current.split("id=")[0] == value.split("id=")[0]:
+                            continue
+
+                # TXT records longer than 255 bytes
+                # are split into multiple <character-string>s.
+                # This typically happens with DKIM record
+                # which contains long RSA key.
+                #
+                # Removing `" "` before comparison
+                # to get back a single string.
+                if current.replace('" "', "") != value.replace('" "', ""):
+                    to_print.append(line)
+
+    exit_code = 0
+    if to_print:
+        to_print.insert(
+            0, "You should configure the following DNS entries at your provider:\n"
+        )
+        to_print.append(
+            "\nIf you already configured the DNS entries, wait a bit until the DNS entries propagate to the Internet."
+        )
+        print("\n".join(to_print))
+        exit_code = 1
+    else:
+        out.green("Great! All your DNS entries are correct.")
+
+    to_print = []
+    if not reverse_ipv4:
+        to_print.append(f"\tIPv4:\t{ipv4}\t{args.config.mail_domain}")
+    if not reverse_ipv6:
+        to_print.append(f"\tIPv6:\t{ipv6}\t{args.config.mail_domain}")
+    if len(to_print) > 0:
+        if len(to_print) == 1:
+            warning = "You should add the following PTR/reverse DNS entry:"
+        else:
+            warning = "You should add the following PTR/reverse DNS entries:"
+        out.red(warning)
+        for entry in to_print:
+            print(entry)
+        print(
+            "You can do so at your hosting provider (maybe this isn't your DNS provider)."
+        )
+        exit_code = 1
+    return exit_code
+
+
+def check_necessary_dns(out, mail_domain):
+    """Check whether $mail_domain and mta-sts.$mail_domain resolve."""
+    print("Checking necessary DNS records... ")
+    dns = DNS(out, mail_domain)
+    ipv4 = dns.get("A", mail_domain)
+    ipv6 = dns.get("AAAA", mail_domain)
+    mta_entry = dns.get("CNAME", "mta-sts." + mail_domain)
+    www_entry = dns.get("CNAME", "www." + mail_domain)
+    to_print = []
+    if not (ipv4 or ipv6):
+        to_print.append(f"\t{mail_domain}.\t\t\tA<your server's IPv4 address>")
+    if mta_entry != mail_domain + ".":
+        to_print.append(f"\tmta-sts.{mail_domain}.\tCNAME\t{mail_domain}.")
+    if www_entry != mail_domain + ".":
+        to_print.append(f"\twww.{mail_domain}.\tCNAME\t{mail_domain}.")
+    if to_print:
+        to_print.insert(
+            0,
+            "\nFor chatmail to work, you need to configure this at your DNS provider:\n",
+        )
+        for line in to_print:
+            print(line)
+        print()
+    else:
+        dns.out.green("All necessary DNS records seem to be set.")
+        return True

cmdeploy/src/cmdeploy/dovecot/auth.conf

@@ -1,7 +1,5 @@
 uri = proxy:/run/doveauth/doveauth.socket:auth
-iterate_disable = no
-iterate_prefix = userdb/
-
+iterate_disable = yes
 default_pass_scheme = plain
 # %E escapes characters " (double quote), ' (single quote) and \ (backslash) with \ (backslash).
 # See <https://doc.dovecot.org/configuration_manual/config_file/config_variables/#modifiers>

cmdeploy/src/cmdeploy/dovecot/default.sieve

@@ -0,0 +1,7 @@
+require ["imap4flags"];
+
+# flag the message so it doesn't cause a push notification
+
+if header :is ["Auto-Submitted"] ["auto-replied", "auto-generated"] {
+	addflag "$Auto";
+}

cmdeploy/src/cmdeploy/dovecot/dovecot.conf.j2

@@ -1,9 +1,5 @@
 ## Dovecot configuration file
 
-{% if disable_ipv6 %}
-listen = *
-{% endif %}
-
 protocols = imap lmtp
 
 auth_mechanisms = plain
@@ -42,16 +38,12 @@ service imap {
 mail_server_admin = mailto:root@{{ config.mail_domain }}
 mail_server_comment = Chatmail server
 
-# `zlib` enables compressing messages stored in the maildir.
-# See
-# <https://doc.dovecot.org/configuration_manual/zlib_plugin/>
-# for documentation.
-#
-# quota plugin documentation:
-# <https://doc.dovecot.org/configuration_manual/quota_plugin/>
-mail_plugins = zlib quota
+mail_plugins = quota
 
-imap_capability = +XDELTAPUSH XCHATMAIL
+# these are the capabilities Delta Chat cares about actually 
+# so let's keep the network overhead per login small
+# https://github.com/deltachat/deltachat-core-rust/blob/master/src/imap/capabilities.rs
+imap_capability = IMAP4rev1 IDLE MOVE QUOTA CONDSTORE NOTIFY METADATA XDELTAPUSH XCHATMAIL
 
 
 # Authentication for system users.
@@ -68,7 +60,7 @@ userdb {
 ##
 
 # Mailboxes are stored in the "mail" directory of the vmail user home.
-mail_location = maildir:{{ config.mailboxes_dir }}/%u
+mail_location = maildir:/home/vmail/mail/%d/%u
 
 namespace inbox {
   inbox = yes
@@ -104,20 +96,17 @@ mail_privileged_group = vmail
 # Pass all IMAP METADATA requests to the server implementing Dovecot's dict protocol.
 mail_attribute_dict = proxy:/run/chatmail-metadata/metadata.socket:metadata
 
-# `imap_zlib` enables IMAP COMPRESS (RFC 4978).
+# Enable IMAP COMPRESS (RFC 4978).
 # <https://datatracker.ietf.org/doc/html/rfc4978.html>
 protocol imap {
-  mail_plugins = $mail_plugins imap_zlib imap_quota last_login
+  mail_plugins = $mail_plugins imap_zlib imap_quota
   imap_metadata = yes
 }
 
-plugin {
-  last_login_dict = proxy:/run/chatmail-lastlogin/lastlogin.socket:lastlogin
-  #last_login_key = last-login/%u # default
-  last_login_precision = s
-}
-
 protocol lmtp {
+  # quota plugin documentation:
+  # <https://doc.dovecot.org/configuration_manual/quota_plugin/>
+  #
   # notify plugin is a dependency of push_notification plugin:
   # <https://doc.dovecot.org/settings/plugin/notify-plugin/>
   #
@@ -126,11 +115,10 @@ protocol lmtp {
   #
   # mail_lua and push_notification_lua are needed for Lua push notification handler.
   # <https://doc.dovecot.org/configuration_manual/push_notification/#configuration>
-  mail_plugins = $mail_plugins mail_lua notify push_notification push_notification_lua
-}
-
-plugin {
-  zlib_save = gz
+  #
+  # Sieve to mark messages that should not be notified as \Seen
+  # <https://doc.dovecot.org/configuration_manual/sieve/configuration/>
+  mail_plugins = $mail_plugins quota mail_lua notify push_notification push_notification_lua sieve
 }
 
 plugin {
@@ -141,7 +129,7 @@ plugin {
   # for now we define static quota-rules for all users 
   quota = maildir:User quota
   quota_rule = *:storage={{ config.max_mailbox_size }}
-  quota_max_mail_size={{ config.max_message_size }}
+  quota_max_mail_size=30M
   quota_grace = 0
   # quota_over_flag_value = TRUE
 }
@@ -152,6 +140,10 @@ plugin {
   push_notification_driver = lua:file=/etc/dovecot/push_notification.lua
 }
 
+plugin {
+  sieve_default = file:/etc/dovecot/default.sieve
+}
+
 service lmtp {
   user=vmail
 
@@ -177,72 +169,26 @@ service auth-worker {
 }
 
 service imap-login {
-    # High-performance mode as described in
-    # <https://doc.dovecot.org/2.3/admin_manual/login_processes/#high-performance-mode>
-    #
-    # So-called high-security mode described in
-    # <https://doc.dovecot.org/2.3/admin_manual/login_processes/#high-security-mode>
-    # and enabled by default with `service_count = 1` starts one process per connection
-    # and has problems logging in thousands of users after Dovecot restart.
-    service_count = 0
+    # High-security mode.
+    # Each process serves a single connection and exits afterwards.
+    # This is the default, but we set it explicitly to be sure.
+    # See <https://doc.dovecot.org/admin_manual/login_processes/#high-security-mode> for details.
+    service_count = 1
 
-    # Increase virtual memory size limit.
-    # Since imap-login processes handle TLS connections
-    # even after logging users in
-    # and many connections are handled by each process,
-    # memory size limit should be increased.
+    # Inrease the number of simultaneous connections.
     #
-    # Otherwise the whole process eventually dies
-    # with an error similar to
-    #   imap-login: Fatal: master: service(imap-login):
-    #   child 1422951 returned error 83
-    #   (Out of memory (service imap-login { vsz_limit=256 MB },
-    #    you may need to increase it)
-    # and takes down all its TLS connections at once.
-    vsz_limit = 1G
+    # As of Dovecot 2.3.19.1 the default is 100 processes.
+    # Combined with `service_count = 1` it means only 100 connections
+    # can be handled simultaneously.
+    process_limit = 10000
 
     # Avoid startup latency for new connections.
-    #
-    # Should be set to at least the number of CPU cores
-    # according to the documentation.
     process_min_avail = 10
 }
 
-service anvil {
-    # We are disabling anvil penalty on failed login attempts
-    # because it can only detect brute forcing by IP address
-    # not by username. As the correct IP address is not handed
-    # to dovecot anyway, it is more of hindrance than of use.
-    # See <https://www.dovecot.org/list/dovecot/2012-May/135485.html> for details.
-    unix_listener anvil-auth-penalty {
-        mode = 0
-    }
-}
-
 ssl = required
 ssl_cert = </var/lib/acme/live/{{ config.mail_domain }}/fullchain
 ssl_key = </var/lib/acme/live/{{ config.mail_domain }}/privkey
 ssl_dh = </usr/share/dovecot/dh.pem
-ssl_min_protocol = TLSv1.3
+ssl_min_protocol = TLSv1.2
 ssl_prefer_server_ciphers = yes
-
-
-{% if config.imap_rawlog %}
-service postlogin {
-  executable = script-login -d rawlog
-  unix_listener postlogin {
- }  
-} 
-service imap {
-  executable = imap postlogin 
-} 
-  
-protocol imap { 
-  #rawlog_dir = /tmp/rawlog/%u
-  # Put .in and .out imap protocol logging files into per-user homedir 
-  # You can use a command like this to combine into one protocol stream:
-  # sort -sn <(sed 's/ / C: /' *.in) <(sed 's/ / S: /' cat *.out)
-
-  rawlog_dir = %h 
-} 
-{% endif %}

cmdeploy/src/cmdeploy/dovecot/expunge.cron.j2

@@ -1,14 +1,12 @@
-# delete already seen big mails after 7 days, in the INBOX
-2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_large_after }} -size +200k -type f -delete
 # delete all mails after {{ config.delete_mails_after }} days, in the Inbox
-2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
 # or in any IMAP subfolder
-2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/.*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
 # even if they are unseen
-2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
-2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/.*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
 # or only temporary (but then they shouldn't be around after {{ config.delete_mails_after }} days anyway).
-2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
-2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
-3 0 * * * vmail find {{ config.mailboxes_dir }} -name 'maildirsize' -type f -delete
-4 0 * * * vmail /usr/local/lib/chatmaild/venv/bin/delete_inactive_users /usr/local/lib/chatmaild/chatmail.ini 
+2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+2 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -path '*/.*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
+3 0 * * * vmail find /home/vmail/mail/{{ config.mail_domain }} -name 'maildirsize' -type f -delete
+4 0 * * * vmail /usr/local/bin/remove-seen.py /home/vmail/mail/{{ config.mail_domain }}

cmdeploy/src/cmdeploy/dovecot/push_notification.lua

@@ -2,14 +2,27 @@ function dovecot_lua_notify_begin_txn(user)
   return user
 end
 
+function contains(v, needle)
+  for _, keyword in ipairs(v) do
+    if keyword == needle then
+      return true
+    end
+  end
+  return false
+end
+
 function dovecot_lua_notify_event_message_new(user, event)
   local mbox = user:mailbox(event.mailbox)
   mbox:sync()
 
   if user.username ~= event.from_address then
     -- Incoming message
-    -- Notify METADATA server about new message.
-    mbox:metadata_set("/private/messagenew", "")
+    if not contains(event.keywords, "$Auto") then
+      -- Not an Auto-Submitted message, notifying.
+
+      -- Notify METADATA server about new message.
+      mbox:metadata_set("/private/messagenew", "")
+    end
   end
 
   mbox:free()

cmdeploy/src/cmdeploy/dovecot/remove-seen.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+"""Remove seen messages that are older than two days
+if maildir has more than 80 MB of messages."""
+import sys
+import time
+from pathlib import Path
+
+
+def getdirsize(path):
+    return sum(f.stat().st_size for f in path.glob("**/*") if f.is_file())
+
+
+def parse_dovecot_seen(path):
+    return "S" in path.name.split(":2,")[-1]
+
+
+def main():
+    now = time.time()
+
+    mailhome = Path(sys.argv[1])
+
+    for p in mailhome.iterdir():
+        dirsize = getdirsize(p / "cur") + getdirsize(p / "new")
+        if dirsize < 80000000:
+            continue
+
+        removed_bytes = 0
+        for mailpath in (p / "cur").iterdir():
+            seen = parse_dovecot_seen(mailpath)
+            stat = mailpath.stat()
+            size = stat.st_size
+            if seen and now > stat.st_mtime + 2 * 24 * 3600:
+                removed_bytes += size
+                mailpath.unlink(missing_ok=True)
+
+        if removed_bytes > 0:
+            (p / "maildirsize").unlink(missing_ok=True)
+
+
+if __name__ == "__main__":
+    main()

cmdeploy/src/cmdeploy/iroh-relay.service

@@ -1,12 +0,0 @@
-[Unit]
-Description=Iroh relay
-
-[Service]
-ExecStart=/usr/local/bin/iroh-relay --config-path /etc/iroh-relay.toml
-Restart=on-failure
-RestartSec=5s
-User=iroh
-Group=iroh
-
-[Install]
-WantedBy=multi-user.target

cmdeploy/src/cmdeploy/iroh-relay.toml

@@ -1,11 +0,0 @@
-enable_relay = true
-http_bind_addr = "[::]:3340"
-
-# Disable built-in STUN server in iroh-relay 0.35
-# as we deploy our own TURN server instead.
-# STUN server is going to be removed in iroh-relay 1.0
-# and this line can be removed after upgrade.
-enable_stun = false
-
-enable_metrics = false
-metrics_bind_addr = "127.0.0.1:9092"

cmdeploy/src/cmdeploy/metrics.cron.j2

@@ -1 +1 @@
-*/5 * * * * root {{ config.execpath }} {{ config.mailboxes_dir }} >/var/www/html/metrics
+*/5 * * * * root {{ config.execpath }} /home/vmail/mail/{{ config.mail_domain }} >/var/www/html/metrics

cmdeploy/src/cmdeploy/mtail/delivered_mail.mtail

@@ -1,64 +0,0 @@
-counter delivered_mail
-/saved mail to INBOX$/ {
-  delivered_mail++
-}
-
-counter quota_exceeded
-/Quota exceeded \(mailbox for user is full\)$/ {
-  quota_exceeded++
-}
-
-# Essentially the number of outgoing messages.
-counter dkim_signed
-/DKIM-Signature field added/ {
-  dkim_signed++
-}
-
-counter created_accounts
-counter created_ci_accounts
-counter created_nonci_accounts
-
-/: Created address: (?P<addr>.*)$/ {
-  created_accounts++
-
-  $addr =~ /ci-/ {
-    created_ci_accounts++
-  } else {
-    created_nonci_accounts++
-  }
-}
-
-counter postfix_timeouts
-/timeout after DATA/ {
-  postfix_timeouts++
-}
-
-counter postfix_noqueue
-/postfix\/.*NOQUEUE/ {
-  postfix_noqueue++
-}
-
-counter warning_count
-/warning/ {
-  warning_count++
-}
-
-
-counter filtered_mail_count
-
-counter encrypted_mail_count
-/Filtering encrypted mail\./ {
-  encrypted_mail_count++
-  filtered_mail_count++
-}
-
-counter unencrypted_mail_count
-/Filtering unencrypted mail\./ {
-  unencrypted_mail_count++
-  filtered_mail_count++
-}
-
-counter rejected_unencrypted_mail_count
-/Rejected unencrypted mail\./ {
-  rejected_unencrypted_mail_count++
-}

cmdeploy/src/cmdeploy/mtail/mtail.service.j2

@@ -1,10 +0,0 @@
-[Unit]
-Description=mtail
-
-[Service]
-Type=simple
-ExecStart=/bin/sh -c "journalctl -f -o short-iso -n 0 | /usr/local/bin/mtail --address={{ address }} --port={{ port }} --progs /etc/mtail --logtostderr --logs -"
-Restart=on-failure
-
-[Install]
-WantedBy=multi-user.target

cmdeploy/src/cmdeploy/nginx/autoconfig.xml.j2

@@ -19,13 +19,6 @@
       <authentication>password-cleartext</authentication>
       <username>%EMAILADDRESS%</username>
     </incomingServer>
-    <incomingServer type="imap">
-      <hostname>{{ config.domain_name }}</hostname>
-      <port>443</port>
-      <socketType>SSL</socketType>
-      <authentication>password-cleartext</authentication>
-      <username>%EMAILADDRESS%</username>
-    </incomingServer>
     <outgoingServer type="smtp">
       <hostname>{{ config.domain_name }}</hostname>
       <port>465</port>
@@ -40,12 +33,5 @@
       <authentication>password-cleartext</authentication>
       <username>%EMAILADDRESS%</username>
     </outgoingServer>
-    <outgoingServer type="smtp">
-      <hostname>{{ config.domain_name }}</hostname>
-      <port>443</port>
-      <socketType>SSL</socketType>
-      <authentication>password-cleartext</authentication>
-      <username>%EMAILADDRESS%</username>
-    </outgoingServer>
   </emailProvider>
 </clientConfig>

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -1,46 +1,13 @@
-load_module modules/ngx_stream_module.so;
-
 user www-data;
 worker_processes auto;
-
-# Increase the number of connections
-# that a worker process can open
-# to avoid errors such as
-#   accept4() failed (24: Too many open files)
-# and
-#   socket() failed (24: Too many open files) while connecting to upstream
-# in the logs.
-# <https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile>
-worker_rlimit_nofile 2048;
 pid /run/nginx.pid;
 error_log syslog:server=unix:/dev/log,facility=local3;
 
 events {
-        # Increase to avoid errors such as
-        #   768 worker_connections are not enough while connecting to upstream
-        # in the logs.
-        # <https://nginx.org/en/docs/ngx_core_module.html#worker_connections>
-	worker_connections 2048;
+	worker_connections 768;
 	# multi_accept on;
 }
 
-stream {
-        map $ssl_preread_alpn_protocols $proxy {
-            default 127.0.0.1:8443;
-            ~\bsmtp\b 127.0.0.1:465;
-            ~\bimap\b 127.0.0.1:993;
-        }
-
-        server {
-                listen 443;
-                {% if not disable_ipv6 %}
-                listen [::]:443;
-                {% endif %}
-                proxy_pass $proxy;
-                ssl_preread on;
-        }
-}
-
 http {
 	sendfile on;
 	tcp_nopush on;
@@ -59,14 +26,14 @@ http {
 	gzip on;
 
 	server {
-  
-		listen 127.0.0.1:8443 ssl default_server;
+		listen 443 ssl default_server;
+		listen [::]:443 ssl default_server;
 
 		root /var/www/html;
 
 		index index.html index.htm;
 
-		server_name {{ config.domain_name }} www.{{ config.domain_name }} mta-sts.{{ config.domain_name }};
+		server_name _;
 
 		access_log syslog:server=unix:/dev/log,facility=local7;
 
@@ -107,31 +74,12 @@ http {
 			include /etc/nginx/fastcgi_params;
 			fastcgi_param SCRIPT_FILENAME /usr/lib/cgi-bin/newemail.py;
 		}
-
-                # Proxy to iroh-relay service.
-                location /relay {
-                        proxy_pass http://127.0.0.1:3340;
-                        proxy_http_version 1.1;
-
-                        # Upgrade header is normally set to "iroh derp http" or "websocket".
-                        proxy_set_header Upgrade $http_upgrade;
-                        proxy_set_header Connection "upgrade";
-                }
-
-		location /relay/probe {
-                        proxy_pass http://127.0.0.1:3340;
-                        proxy_http_version 1.1;
-		}
-
-		location /generate_204 {
-                        proxy_pass http://127.0.0.1:3340;
-                        proxy_http_version 1.1;
-		}
 	}
 
 	# Redirect www. to non-www
 	server {
-		listen 127.0.0.1:8443 ssl;
+		listen 443 ssl;
+		listen [::]:443 ssl;
 		server_name www.{{ config.domain_name }};
 		return 301 $scheme://{{ config.domain_name }}$request_uri;
 		access_log syslog:server=unix:/dev/log,facility=local7;

cmdeploy/src/cmdeploy/opendkim/opendkim.conf

@@ -13,7 +13,6 @@ OversignHeaders		From
 On-BadSignature         reject
 On-KeyNotFound          reject
 On-NoSignature          reject
-DNSTimeout          60
 
 # Signing domain, selector, and key (required). For example, perform signing
 # for domain "example.com" with selector "2020" (2020._domainkey.example.com),
@@ -26,24 +25,7 @@ KeyTable          /etc/dkimkeys/KeyTable
 SigningTable      refile:/etc/dkimkeys/SigningTable
 
 # Sign Autocrypt header in addition to the default specified in RFC 6376.
-#
-# Default list is here:
-# <https://github.com/trusteddomainproject/OpenDKIM/blob/5c539587561785a66c1f67f720f2fb741f320785/libopendkim/dkim.c#L221-L245>
-SignHeaders *,+autocrypt,+content-type
-
-# Prevent addition of second Content-Type header
-# and other important headers that should not be added
-# after signing the message.
-# See
-# <https://www.zone.eu/blog/2024/05/17/bimi-and-dmarc-cant-save-you/>
-# and RFC 6376 (page 41) for reference.
-#
-# We don't use "l=" body length so the problem described in RFC 6376
-# is not applicable, but adding e.g. a second "From" header
-# or second "Autocrypt" header is better prevented in any case.
-#
-# Default is empty.
-OversignHeaders from,reply-to,subject,date,to,cc,resent-date,resent-from,resent-sender,resent-to,resent-cc,in-reply-to,references,list-id,list-help,list-unsubscribe,list-subscribe,list-post,list-owner,list-archive,autocrypt
+SignHeaders *,+autocrypt
 
 # Script to ignore signatures that do not correspond to the From: domain.
 ScreenPolicyScript /etc/opendkim/screen.lua

cmdeploy/src/cmdeploy/opendkim/systemd.conf

@@ -1,3 +0,0 @@
-[Service]
-Restart=always
-RuntimeMaxSec=1d

cmdeploy/src/cmdeploy/postfix/main.cf.j2

@@ -20,12 +20,9 @@ smtpd_tls_key_file=/var/lib/acme/live/{{ config.mail_domain }}/privkey
 smtpd_tls_security_level=may
 
 smtp_tls_CApath=/etc/ssl/certs
-smtp_tls_security_level=verify
-# Send SNI extension when connecting to other servers.
-# <https://www.postfix.org/postconf.5.html#smtp_tls_servername>
-smtp_tls_servername = hostname
+smtp_tls_security_level=may
 smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
-smtp_tls_policy_maps = inline:{nauta.cu=may}
+smtp_tls_policy_maps = socketmap:inet:127.0.0.1:8461:postfix
 smtpd_tls_protocols = >=TLSv1.2
 
 # Disable anonymous cipher suites
@@ -65,14 +62,11 @@ mydestination =
 relayhost = 
 mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
 mailbox_size_limit = 0
-message_size_limit = {{config.max_message_size}}
+# maximum 30MB sized messages
+message_size_limit = 31457280
 recipient_delimiter = +
 inet_interfaces = all
-{% if disable_ipv6 %}
-inet_protocols = ipv4
-{% else %}
 inet_protocols = all
-{% endif %}
 
 virtual_transport = lmtp:unix:private/dovecot-lmtp
 virtual_mailbox_domains = {{ config.mail_domain }}
@@ -83,7 +77,3 @@ mua_helo_restrictions = permit_mynetworks, reject_invalid_helo_hostname, reject_
 
 # 1:1 map MAIL FROM to SASL login name.
 smtpd_sender_login_maps = regexp:/etc/postfix/login_map
-
-# Do not lookup SMTP client hostnames to reduce delays
-# and avoid unnecessary DNS requests.
-smtpd_peername_lookup = no

cmdeploy/src/cmdeploy/postfix/master.cf.j2

@@ -14,11 +14,10 @@ smtp      inet  n       -       y       -       -       smtpd -v
 {%- else %}
 smtp      inet  n       -       y       -       -       smtpd 
 {%- endif %}
-  -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
-submission inet n       -       y       -       5000    smtpd
+  -o smtpd_milters=unix:opendkim/opendkim.sock
+submission inet n       -       y       -       -       smtpd
   -o syslog_name=postfix/submission
   -o smtpd_tls_security_level=encrypt
-  -o smtpd_tls_mandatory_protocols=>=TLSv1.3
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_sasl_type=dovecot
   -o smtpd_sasl_path=private/auth
@@ -32,11 +31,11 @@ submission inet n       -       y       -       5000    smtpd
   -o milter_macro_daemon_name=ORIGINATING
   -o smtpd_client_connection_count_limit=1000
   -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
-smtps     inet  n       -       y       -       5000    smtpd
+  -o cleanup_service_name=authclean
+smtps     inet  n       -       y       -       -       smtpd
   -o syslog_name=postfix/smtps
   -o smtpd_tls_wrappermode=yes
   -o smtpd_tls_security_level=encrypt
-  -o smtpd_tls_mandatory_protocols=>=TLSv1.3
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_sasl_type=dovecot
   -o smtpd_sasl_path=private/auth
@@ -49,6 +48,7 @@ smtps     inet  n       -       y       -       5000    smtpd
   -o smtpd_client_connection_count_limit=1000
   -o milter_macro_daemon_name=ORIGINATING
   -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
+  -o cleanup_service_name=authclean
 #628       inet  n       -       y       -       -       qmqpd
 pickup    unix  n       -       y       60      1       pickup
 cleanup   unix  n       -       y       -       0       cleanup
@@ -76,27 +76,17 @@ anvil     unix  -       -       y       -       1       anvil
 scache    unix  -       -       y       -       1       scache
 postlog   unix-dgram n  -       n       -       1       postlogd
 filter    unix -        n       n       -       -       lmtp
-# Local SMTP server for reinjecting outgoing filtered mail.
-127.0.0.1:{{ config.postfix_reinject_port }} inet  n       -       n       -       100      smtpd
+# Local SMTP server for reinjecting filered mail.
+localhost:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
   -o syslog_name=postfix/reinject
   -o smtpd_milters=unix:opendkim/opendkim.sock
   -o cleanup_service_name=authclean
 
-# Local SMTP server for reinjecting incoming filtered mail 
-127.0.0.1:{{ config.postfix_reinject_port_incoming }} inet  n       -       n       -       100      smtpd
-  -o syslog_name=postfix/reinject_incoming
-  -o smtpd_milters=unix:opendkim/opendkim.sock
-
 # Cleanup `Received` headers for authenticated mail
 # to avoid leaking client IP.
 #
 # We do not do this for received mails
 # as this will break DKIM signatures
 # if `Received` header is signed.
-#
-# This service also rewrites
-# Subject with `[...]`
-# to make sure the users
-# cannot send unprotected Subject.
 authclean unix  n       -       -       -       0       cleanup
   -o header_checks=regexp:/etc/postfix/submission_header_cleanup

cmdeploy/src/cmdeploy/postfix/submission_header_cleanup

@@ -2,4 +2,3 @@
 /^X-Originating-IP:/    IGNORE
 /^X-Mailer:/            IGNORE
 /^User-Agent:/          IGNORE
-/^Subject:/             REPLACE Subject: [...]

cmdeploy/src/cmdeploy/remote/__init__.py

@@ -1,12 +0,0 @@
-"""
-
-The 'cmdeploy.remote' sub package contains modules with remotely executing functions.
-
-Its "_sshexec_bootstrap" module is executed remotely through `SSHExec`
-and its main() loop there stays connected via a command channel,
-ready to receive function invocations ("command") and return results.
-"""
-
-from . import rdns, rshell
-
-__all__ = ["rdns", "rshell"]

cmdeploy/src/cmdeploy/remote/_sshexec_bootstrap.py

@@ -1,30 +0,0 @@
-import builtins
-import importlib
-import traceback
-
-## Function Execution server
-
-
-def _run_loop(cmd_channel):
-    while cmd := cmd_channel.receive():
-        cmd_channel.send(_handle_one_request(cmd))
-
-
-def _handle_one_request(cmd):
-    pymod_path, func_name, kwargs = cmd
-    try:
-        mod = importlib.import_module(pymod_path)
-        func = getattr(mod, func_name)
-        res = func(**kwargs)
-        return ("finish", res)
-    except:
-        data = traceback.format_exc()
-        return ("error", data)
-
-
-def main(channel):
-    # enable simple "print" logging
-
-    builtins.print = lambda x="": channel.send(("log", x))
-
-    _run_loop(channel)

cmdeploy/src/cmdeploy/remote/rdns.py

@@ -1,105 +0,0 @@
-"""
-Pure python functions which execute remotely in a system Python interpreter.
-
-All functions of this module
-
-- need to get and and return Python builtin data types only,
-
-- can only use standard library dependencies,
-
-- can freely call each other.
-"""
-
-import re
-
-from .rshell import CalledProcessError, shell, log_progress
-
-
-def perform_initial_checks(mail_domain, pre_command=""):
-    """Collecting initial DNS settings."""
-    assert mail_domain
-    if not shell("dig", fail_ok=True, print=log_progress):
-        shell("apt-get update && apt-get install -y dnsutils", print=log_progress)
-    A = query_dns("A", mail_domain)
-    AAAA = query_dns("AAAA", mail_domain)
-    MTA_STS = query_dns("CNAME", f"mta-sts.{mail_domain}")
-    WWW = query_dns("CNAME", f"www.{mail_domain}")
-
-    res = dict(mail_domain=mail_domain, A=A, AAAA=AAAA, MTA_STS=MTA_STS, WWW=WWW)
-    res["acme_account_url"] = shell(pre_command + "acmetool account-url", fail_ok=True, print=log_progress)
-    res["dkim_entry"], res["web_dkim_entry"] = get_dkim_entry(
-        mail_domain, pre_command, dkim_selector="opendkim"
-    )
-
-    if not MTA_STS or not WWW or (not A and not AAAA):
-        return res
-
-    # parse out sts-id if exists, example: "v=STSv1; id=2090123"
-    parts = query_dns("TXT", f"_mta-sts.{mail_domain}").split("id=")
-    res["sts_id"] = parts[1].rstrip('"') if len(parts) == 2 else ""
-    return res
-
-
-def get_dkim_entry(mail_domain, pre_command, dkim_selector):
-    try:
-        dkim_pubkey = shell(
-            f"{pre_command}openssl rsa -in /etc/dkimkeys/{dkim_selector}.private "
-            "-pubout 2>/dev/null | awk '/-/{next}{printf(\"%s\",$0)}'",
-            print=log_progress
-        )
-    except CalledProcessError:
-        return
-    dkim_value_raw = f"v=DKIM1;k=rsa;p={dkim_pubkey};s=email;t=s"
-    dkim_value = '" "'.join(re.findall(".{1,255}", dkim_value_raw))
-    web_dkim_value = "".join(re.findall(".{1,255}", dkim_value_raw))
-    return (
-        f'{dkim_selector}._domainkey.{mail_domain}. TXT "{dkim_value}"',
-        f'{dkim_selector}._domainkey.{mail_domain}. TXT "{web_dkim_value}"',
-    )
-
-
-def query_dns(typ, domain):
-    # Get autoritative nameserver from the SOA record.
-    soa_answers = [
-        x.split()
-        for x in shell(f"dig -r -q {domain} -t SOA +noall +authority +answer", print=log_progress).split(
-            "\n"
-        )
-    ]
-    soa = [a for a in soa_answers if len(a) >= 3 and a[3] == "SOA"]
-    if not soa:
-        return
-    ns = soa[0][4]
-
-    # Query authoritative nameserver directly to bypass DNS cache.
-    res = shell(f"dig @{ns} -r -q {domain} -t {typ} +short", print=log_progress)
-    if res:
-        return res.split("\n")[0]
-    return ""
-
-
-def check_zonefile(zonefile, verbose=True):
-    """Check expected zone file entries."""
-    required = True
-    required_diff = []
-    recommended_diff = []
-
-    for zf_line in zonefile.splitlines():
-        if "; Recommended" in zf_line:
-            required = False
-            continue
-        if not zf_line.strip() or zf_line.startswith(";"):
-            continue
-        print(f"dns-checking {zf_line!r}") if verbose else log_progress("")
-        zf_domain, zf_typ, zf_value = zf_line.split(maxsplit=2)
-        zf_domain = zf_domain.rstrip(".")
-        zf_value = zf_value.strip()
-        query_value = query_dns(zf_typ, zf_domain)
-        if zf_value != query_value:
-            assert zf_typ in ("A", "AAAA", "CNAME", "CAA", "SRV", "MX", "TXT"), zf_line
-            if required:
-                required_diff.append(zf_line)
-            else:
-                recommended_diff.append(zf_line)
-
-    return required_diff, recommended_diff

cmdeploy/src/cmdeploy/remote/rshell.py

@@ -1,45 +0,0 @@
-import sys
-
-from subprocess import DEVNULL, CalledProcessError, check_output
-
-
-def log_progress(data):
-    sys.stderr.write(".")
-    sys.stderr.flush()
-
-
-def shell(command, fail_ok=False, print=print):
-    print(f"$ {command}")
-    args = dict(shell=True)
-    if fail_ok:
-        args["stderr"] = DEVNULL
-    try:
-        return check_output(command, **args).decode().rstrip()
-    except CalledProcessError:
-        if not fail_ok:
-            raise
-        return ""
-
-
-def get_systemd_running():
-    lines = shell("systemctl --type=service --state=running").split("\n")
-    return [line for line in lines if line.startswith("  ")]
-
-
-def write_numbytes(path, num):
-    with open(path, "w") as f:
-        f.write("x" * num)
-
-
-def dovecot_recalc_quota(user):
-    shell(f"doveadm quota recalc -u {user}")
-    output = shell(f"doveadm quota get -u {user}")
-    #
-    # Quota name Type    Value  Limit                                              %
-    # User quota STORAGE     5 102400                                              0
-    # User quota MESSAGE     2      -                                              0
-    #
-    for line in output.split("\n"):
-        parts = line.split()
-        if parts[2] == "STORAGE":
-            return dict(value=int(parts[3]), limit=int(parts[4]), percent=int(parts[5]))

cmdeploy/src/cmdeploy/service/chatmail-metadata.service.f

@@ -2,12 +2,11 @@
 Description=Chatmail dict proxy for IMAP METADATA
 
 [Service]
-ExecStart={execpath} /run/chatmail-metadata/metadata.socket {config_path}
+ExecStart={execpath} /run/chatmail-metadata/metadata.socket /home/vmail/mail/{mail_domain} {config_path}
 Restart=always
 RestartSec=30
 User=vmail
 RuntimeDirectory=chatmail-metadata
-UMask=0077
 
 [Install]
 WantedBy=multi-user.target

cmdeploy/src/cmdeploy/service/doveauth.service.f

@@ -2,12 +2,11 @@
 Description=Chatmail dict authentication proxy for dovecot
 
 [Service]
-ExecStart={execpath} /run/doveauth/doveauth.socket {config_path}
+ExecStart={execpath} /run/doveauth/doveauth.socket /home/vmail/passdb.sqlite {config_path}
 Restart=always
 RestartSec=30
 User=vmail
 RuntimeDirectory=doveauth
-UMask=0077
 
 [Install]
 WantedBy=multi-user.target

cmdeploy/src/cmdeploy/service/filtermail-incoming.service.f

@@ -1,12 +0,0 @@
-[Unit]
-Description=Incoming Chatmail Postfix before queue filter 
-
-[Service]
-ExecStart={execpath} {config_path} incoming
-Restart=always
-RestartSec=30
-User=vmail
-
-[Install]
-WantedBy=multi-user.target
-

cmdeploy/src/cmdeploy/service/filtermail.service.f

@@ -1,11 +1,10 @@
 [Unit]
-Description=Outgoing Chatmail Postfix before queue filter
+Description=Chatmail Postfix before queue filter
 
 [Service]
-ExecStart={execpath} {config_path} outgoing
+ExecStart={execpath} {config_path}
 Restart=always
 RestartSec=30
-User=vmail
 
 [Install]
 WantedBy=multi-user.target

cmdeploy/src/cmdeploy/service/lastlogin.service.f

@@ -1,12 +0,0 @@
-[Unit]
-Description=Dict proxy for last-login tracking
-
-[Service]
-ExecStart={execpath} /run/chatmail-lastlogin/lastlogin.socket {config_path}
-Restart=always
-RestartSec=30
-User=vmail
-RuntimeDirectory=chatmail-lastlogin
-
-[Install]
-WantedBy=multi-user.target

cmdeploy/src/cmdeploy/service/turnserver.service.f

@@ -1,16 +0,0 @@
-[Unit]
-Description=A wrapper for the TURN server
-After=network.target
-
-[Service]
-Type=simple
-Restart=always
-ExecStart=/usr/local/bin/chatmail-turn --realm {mail_domain} --socket /run/chatmail-turn/turn.socket
-
-# Create /run/chatmail-turn
-RuntimeDirectory=chatmail-turn
-User=vmail
-Group=vmail
-
-[Install]
-WantedBy=multi-user.target

cmdeploy/src/cmdeploy/sshexec.py

@@ -1,100 +0,0 @@
-import inspect
-import os
-import sys
-from queue import Queue
-
-import execnet
-
-from . import remote
-
-
-class FuncError(Exception):
-    pass
-
-
-def bootstrap_remote(gateway, remote=remote):
-    """Return a command channel which can execute remote functions."""
-    source_init_path = inspect.getfile(remote)
-    basedir = os.path.dirname(source_init_path)
-    name = os.path.basename(basedir)
-
-    # rsync sourcedir to remote host
-    remote_pkg_path = f"/root/from-cmdeploy/{name}"
-    q = Queue()
-    finish = lambda: q.put(None)
-    rsync = execnet.RSync(sourcedir=basedir, verbose=False)
-    rsync.add_target(gateway, remote_pkg_path, finishedcallback=finish, delete=True)
-    rsync.send()
-    q.get()
-
-    # start sshexec bootstrap and return its command channel
-    remote_sys_path = os.path.dirname(remote_pkg_path)
-    channel = gateway.remote_exec(
-        f"""
-        import sys
-        sys.path.insert(0, {remote_sys_path!r})
-        from remote._sshexec_bootstrap import main
-        main(channel)
-    """
-    )
-    return channel
-
-
-def print_stderr(item="", end="\n"):
-    print(item, file=sys.stderr, end=end)
-    sys.stderr.flush()
-
-
-class SSHExec:
-    RemoteError = execnet.RemoteError
-    FuncError = FuncError
-
-    def __init__(self, host, verbose=False, python="python3", timeout=60):
-        self.gateway = execnet.makegateway(f"ssh=root@{host}//python={python}")
-        self._remote_cmdloop_channel = bootstrap_remote(self.gateway, remote)
-        self.timeout = timeout
-        self.verbose = verbose
-
-    def __call__(self, call, kwargs=None, log_callback=None):
-        if kwargs is None:
-            kwargs = {}
-        assert call.__module__.startswith("cmdeploy.remote")
-        modname = call.__module__.replace("cmdeploy.", "")
-        self._remote_cmdloop_channel.send((modname, call.__name__, kwargs))
-        while 1:
-            code, data = self._remote_cmdloop_channel.receive(timeout=self.timeout)
-            if log_callback is not None and code == "log":
-                log_callback(data)
-            elif code == "finish":
-                return data
-            elif code == "error":
-                raise self.FuncError(data)
-
-    def logged(self, call, kwargs):
-        title = call.__doc__
-        if not title:
-            title = call.__name__
-        if self.verbose:
-            print_stderr("[ssh] " + title)
-            return self(call, kwargs, log_callback=print_stderr)
-        else:
-            print_stderr(title, end="")
-            res = self(call, kwargs, log_callback=remote.rshell.log_progress)
-            print_stderr()
-            return res
-
-
-class LocalExec:
-    def __init__(self, verbose=False, docker=False):
-        self.verbose = verbose
-        self.docker = docker
-
-    def logged(self, call, kwargs: dict):
-        where = "locally"
-        if self.docker:
-            if call == remote.rdns.perform_initial_checks:
-                kwargs['pre_command'] = "docker exec chatmail "
-                where = "in docker"
-        if self.verbose:
-            print(f"Running {where}: {call.__name__}(**{kwargs})")
-        return call(**kwargs)

cmdeploy/src/cmdeploy/tests/data/zftest.zone

@@ -1,17 +0,0 @@
-; Required DNS entries for chatmail servers
-zftest.testrun.org.                   A     135.181.204.127
-zftest.testrun.org.                   AAAA  2a01:4f9:c012:52f4::1
-zftest.testrun.org.                   MX 10 zftest.testrun.org.
-_mta-sts.zftest.testrun.org.          TXT "v=STSv1; id=202403211706"
-mta-sts.zftest.testrun.org.           CNAME zftest.testrun.org.
-www.zftest.testrun.org.               CNAME zftest.testrun.org.
-opendkim._domainkey.zftest.testrun.org. TXT "v=DKIM1;k=rsa;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoYt82CVUyz2ouaqjX2kB+5J80knAyoOU3MGU5aWppmwUwwTvj/oSTSpkc5JMtVTRmKKr8NUDWAL1Yw7dfGqqPHdHfwwjS3BIvDzYx+hzgtz62RnfNgV+/2MAoNpfX7cAFIHdRzEHNtwugc3RDLquqPoupAE3Y2YRw2T5zG5fILh4vwIcJZL5Uq6B92j8wwJqOex" "33n+vm1NKQ9rxo/UsHAmZlJzpooXcG/4igTBxJyJlamVSRR6N7Nul1v//YJb7J6v2o0iPHW6uE0StzKaPPNC2IVosSRFbD9H2oqppltptFSNPlI0E+t0JBWHem6YK7xcugiO3ImMCaaU8g6Jt/wIDAQAB;s=email;t=s"
-; Recommended DNS entries
-_submission._tcp.zftest.testrun.org.  SRV 0 1 587 zftest.testrun.org.
-_submissions._tcp.zftest.testrun.org. SRV 0 1 465 zftest.testrun.org.
-_imap._tcp.zftest.testrun.org.        SRV 0 1 143 zftest.testrun.org.
-_imaps._tcp.zftest.testrun.org.       SRV 0 1 993 zftest.testrun.org.
-zftest.testrun.org.                   CAA 0 issue "letsencrypt.org;accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1371472956"
-zftest.testrun.org.                   TXT "v=spf1 a:zftest.testrun.org ~all"
-_dmarc.zftest.testrun.org.            TXT "v=DMARC1;p=reject;adkim=s;aspf=s"
-_adsp._domainkey.zftest.testrun.org.  TXT "dkim=discardable"

cmdeploy/src/cmdeploy/tests/online/benchmark.py

@@ -37,24 +37,24 @@ class TestDC:
 
     def test_ping_pong(self, benchmark, cmfactory):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def dc_ping_pong():
             chat.send_text("ping")
-            msg = ac2._evtracker.wait_next_incoming_message()
+            msg = ac2.wait_next_incoming_message()
             msg.chat.send_text("pong")
-            ac1._evtracker.wait_next_incoming_message()
+            ac1.wait_next_incoming_message()
 
         benchmark(dc_ping_pong, 5)
 
     def test_send_10_receive_10(self, benchmark, cmfactory, lp):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def dc_send_10_receive_10():
             for i in range(10):
                 chat.send_text(f"hello {i}")
             for i in range(10):
-                ac2._evtracker.wait_next_incoming_message()
+                ac2.wait_next_incoming_message()
 
         benchmark(dc_send_10_receive_10, 5)

cmdeploy/src/cmdeploy/tests/online/test_0_login.py

@@ -90,13 +90,8 @@ def test_concurrent_logins_same_account(
 
 
 def test_no_vrfy(chatmail_config):
-    domain = chatmail_config.mail_domain
     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    sock.settimeout(10)
-    try:
-        sock.connect((domain, 25))
-    except socket.timeout:
-        pytest.skip(f"port 25 not reachable for {domain}")
+    sock.connect((chatmail_config.mail_domain, 25))
     banner = sock.recv(1024)
     print(banner)
     sock.send(b"VRFY wrongaddress@%s\r\n" % (chatmail_config.mail_domain.encode(),))

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -1,80 +1,7 @@
-import datetime
 import smtplib
-import socket
-import subprocess
-import time
 
 import pytest
 
-from cmdeploy import remote
-from cmdeploy.sshexec import SSHExec
-
-
-class TestSSHExecutor:
-    @pytest.fixture(scope="class")
-    def sshexec(self, sshdomain):
-        return SSHExec(sshdomain)
-
-    def test_ls(self, sshexec):
-        out = sshexec(call=remote.rdns.shell, kwargs=dict(command="ls"))
-        out2 = sshexec(call=remote.rdns.shell, kwargs=dict(command="ls"))
-        assert out == out2
-
-    def test_perform_initial(self, sshexec, maildomain):
-        res = sshexec(
-            remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
-        )
-        assert res["A"] or res["AAAA"]
-
-    def test_logged(self, sshexec, maildomain, capsys):
-        sshexec.logged(
-            remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
-        )
-        out, err = capsys.readouterr()
-        assert err.startswith("Collecting")
-        # XXX could not figure out how capturing can be made to work properly
-        #assert err.endswith("....\n")
-        assert err.count("\n") == 1
-
-        sshexec.verbose = True
-        sshexec.logged(
-            remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=maildomain)
-        )
-        out, err = capsys.readouterr()
-        lines = err.split("\n")
-        # XXX could not figure out how capturing can be made to work properly
-        #assert len(lines) > 4
-        assert remote.rdns.perform_initial_checks.__doc__ in lines[0]
-
-    def test_exception(self, sshexec, capsys):
-        try:
-            sshexec.logged(
-                remote.rdns.perform_initial_checks,
-                kwargs=dict(mail_domain=None),
-            )
-        except sshexec.FuncError as e:
-            assert "rdns.py" in str(e)
-            assert "AssertionError" in str(e)
-        else:
-            pytest.fail("didn't raise exception")
-
-    def test_opendkim_restarted(self, sshexec):
-        """check that opendkim is not running for longer than a day."""
-        cmd = "systemctl show opendkim --timestamp=utc --property=ActiveEnterTimestamp"
-        out = sshexec(call=remote.rshell.shell, kwargs=dict(command=cmd))
-        datestring = out.split("=")[1]
-        since_date = datetime.datetime.strptime(datestring, "%a %Y-%m-%d %H:%M:%S %Z")
-        now = datetime.datetime.now(since_date.tzinfo)
-        assert (now - since_date).total_seconds() < 60 * 60 * 51
-
-
-def test_timezone_env(remote):
-    for line in remote.iter_output("env"):
-        print(line)
-        if line == "tz=:/etc/localtime":
-            return
-    pytest.fail("TZ is not set")
-
 
 def test_remote(remote, imap_or_smtp):
     lineproducer = remote.iter_output(imap_or_smtp.logcmd)
@@ -130,57 +57,14 @@ def test_authenticated_from(cmsetup, maildata):
 
 @pytest.mark.parametrize("from_addr", ["fake@example.org", "fake@testrun.org"])
 def test_reject_missing_dkim(cmsetup, maildata, from_addr):
-    domain = cmsetup.maildomain
-    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    sock.settimeout(10)
-    try:
-        sock.connect((domain, 25))
-    except socket.timeout:
-        pytest.skip(f"port 25 not reachable for {domain}")
-
+    """Test that emails with missing or wrong DMARC, DKIM, and SPF entries are rejected."""
     recipient = cmsetup.gen_users(1)[0]
-    msg = maildata(
-        "encrypted.eml", from_addr=from_addr, to_addr=recipient.addr
-    ).as_string()
-    conn = smtplib.SMTP(cmsetup.maildomain, 25, timeout=10)
-
-    with conn as s:
+    msg = maildata("plain.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
+    with smtplib.SMTP(cmsetup.maildomain, 25) as s:
         with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):
             s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)
 
 
-def try_n_times(n, f):
-    for _ in range(n - 1):
-        try:
-            return f()
-        except Exception:
-            time.sleep(1)
-
-    return f()
-
-
-def test_rewrite_subject(cmsetup, maildata):
-    """Test that subject gets replaced with [...]."""
-    user1, user2 = cmsetup.gen_users(2)
-
-    sent_msg = maildata(
-        "encrypted.eml",
-        from_addr=user1.addr,
-        to_addr=user2.addr,
-        subject="Unencrypted subject",
-    ).as_string()
-    user1.smtp.sendmail(from_addr=user1.addr, to_addrs=[user2.addr], msg=sent_msg)
-
-    # The message may need some time to get delivered by postfix.
-    messages = try_n_times(5, user2.imap.fetch_all_messages)
-    assert len(messages) == 1
-    rcvd_msg = messages[0]
-    assert "Subject: [...]" not in sent_msg
-    assert "Subject: [...]" in rcvd_msg
-    assert "Subject: Unencrypted subject" in sent_msg
-    assert "Subject: Unencrypted subject" not in rcvd_msg
-
-
 @pytest.mark.slow
 def test_exceed_rate_limit(cmsetup, gencreds, maildata, chatmail_config):
     """Test that the per-account send-mail limit is exceeded."""
@@ -206,38 +90,13 @@ def test_exceed_rate_limit(cmsetup, gencreds, maildata, chatmail_config):
 def test_expunged(remote, chatmail_config):
     outdated_days = int(chatmail_config.delete_mails_after) + 1
     find_cmds = [
-        f"find {chatmail_config.mailboxes_dir} -path '*/cur/*' -mtime +{outdated_days} -type f",
-        f"find {chatmail_config.mailboxes_dir} -path '*/.*/cur/*' -mtime +{outdated_days} -type f",
-        f"find {chatmail_config.mailboxes_dir} -path '*/new/*' -mtime +{outdated_days} -type f",
-        f"find {chatmail_config.mailboxes_dir} -path '*/.*/new/*' -mtime +{outdated_days} -type f",
-        f"find {chatmail_config.mailboxes_dir} -path '*/tmp/*' -mtime +{outdated_days} -type f",
-        f"find {chatmail_config.mailboxes_dir} -path '*/.*/tmp/*' -mtime +{outdated_days} -type f",
+        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/cur/*' -mtime +{outdated_days} -type f",
+        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/.*/cur/*' -mtime +{outdated_days} -type f",
+        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/new/*' -mtime +{outdated_days} -type f",
+        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/.*/new/*' -mtime +{outdated_days} -type f",
+        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/tmp/*' -mtime +{outdated_days} -type f",
+        f"find /home/vmail/mail/{chatmail_config.mail_domain} -path '*/.*/tmp/*' -mtime +{outdated_days} -type f",
     ]
-    outdated_days = int(chatmail_config.delete_large_after) + 1
-    find_cmds.append(
-        "find {chatmail_config.mailboxes_dir} -path '*/cur/*' -mtime +{outdated_days} -size +200k -type f"
-    )
     for cmd in find_cmds:
         for line in remote.iter_output(cmd):
             assert not line
-
-
-def test_deployed_state(remote):
-    try:
-        git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
-    except Exception:
-        git_hash = "unknown\n"
-    try:
-        git_diff = subprocess.check_output(["git", "diff"]).decode()
-    except Exception:
-        git_diff = ""
-    git_status = [git_hash.strip()]
-    for line in git_diff.splitlines():
-        git_status.append(line.strip().lower())
-    remote_version = []
-    for line in remote.iter_output("cat /etc/chatmail-version"):
-        print(line)
-        remote_version.append(line)
-    # assert len(git_status) == len(remote_version)  # for some reason, we only get 11 lines from remote.iter_output()
-    for i in range(len(remote_version)):
-        assert git_status[i] == remote_version[i], "You have undeployed changes."

cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py

@@ -1,4 +1,5 @@
 import ipaddress
+import random
 import re
 import time
 
@@ -6,9 +7,6 @@ import imap_tools
 import pytest
 import requests
 
-from cmdeploy.remote import rshell
-from cmdeploy.sshexec import SSHExec
-
 
 @pytest.fixture
 def imap_mailbox(cmfactory):
@@ -56,23 +54,22 @@ class TestEndToEndDeltaChat:
         """Test that a DC account can send a message to a second DC account
         on the same chat-mail instance."""
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
+
+        lp.sec("ac1: prepare and send text message to ac2")
         chat.send_text("message0")
 
         lp.sec("wait for ac2 to receive message")
         msg2 = ac2._evtracker.wait_next_incoming_message()
         assert msg2.text == "message0"
 
-    def test_exceed_quota(
-        self, cmfactory, lp, tmpdir, remote, chatmail_config, sshdomain
-    ):
+    @pytest.mark.slow
+    def test_exceed_quota(self, cmfactory, lp, tmpdir, remote, chatmail_config):
         """This is a very slow test as it needs to upload >100MB of mail data
         before quota is exceeded, and thus depends on the speed of the upload.
         """
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
-
-        user = ac2.get_config("configured_addr")
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def parse_size_limit(limit: str) -> int:
             """Parse a size limit and return the number of bytes as integer.
@@ -85,27 +82,49 @@ class TestEndToEndDeltaChat:
             return int(float(number) * units[unit])
 
         quota = parse_size_limit(chatmail_config.max_mailbox_size)
+        attachsize = 1 * 1024 * 1024
+        num_to_send = quota // attachsize + 2
+        lp.sec(f"ac1: send {num_to_send} large files to ac2")
+        lp.indent(f"per-user quota is assumed to be: {quota/(1024*1024)}MB")
+        alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
+        msgs = []
+        for i in range(num_to_send):
+            attachment = tmpdir / f"attachment{i}"
+            data = "".join(random.choice(alphanumeric) for i in range(1024))
+            with open(attachment, "w+") as f:
+                for j in range(attachsize // len(data)):
+                    f.write(data)
 
-        lp.sec(f"filling remote inbox for {user}")
-        fn = f"7743102289.M843172P2484002.c20,S={quota},W=2398:2,"
-        path = chatmail_config.mailboxes_dir.joinpath(user, "cur", fn)
-        sshexec = SSHExec(sshdomain)
-        sshexec(call=rshell.write_numbytes, kwargs=dict(path=str(path), num=120))
-        res = sshexec(call=rshell.dovecot_recalc_quota, kwargs=dict(user=user))
-        assert res["percent"] >= 100
+            msg = chat.send_file(str(attachment))
+            msgs.append(msg)
+            lp.indent(f"Sent out msg {i}, size {attachsize/(1024*1024)}MB")
 
-        lp.sec("ac2: check quota is triggered")
+        lp.sec("ac2: check messages are arriving until quota is reached")
 
-        starting = True
+        addr = ac2.get_config("addr").lower()
+        saved_ok = 0
         for line in remote.iter_output("journalctl -n0 -f -u dovecot"):
-            if starting:
-                chat.send_text("hello")
-                starting = False
-            if user not in line:
+            if addr not in line:
                 # print(line)
                 continue
-            if "quota exceeded" in line:
-                return
+            if "quota" in line:
+                if "quota exceeded" in line:
+                    if saved_ok < num_to_send // 2:
+                        pytest.fail(
+                            f"quota exceeded too early: after {saved_ok} messages already"
+                        )
+                    lp.indent("good, message sending failed because quota was exceeded")
+                    return
+            if (
+                "stored mail into mailbox 'inbox'" in line
+                or "saved mail to inbox" in line
+            ):
+                saved_ok += 1
+                print(f"{saved_ok}: {line}")
+                if saved_ok >= num_to_send:
+                    break
+
+        pytest.fail("sending succeeded although messages should exceed quota")
 
     def test_securejoin(self, cmfactory, lp, maildomain2):
         ac1 = cmfactory.new_online_configuring_account(cache=False)
@@ -153,7 +172,7 @@ def test_hide_senders_ip_address(cmfactory):
     assert ipaddress.ip_address(public_ip)
 
     user1, user2 = cmfactory.get_online_accounts(2)
-    chat = cmfactory.get_protected_chat(user1, user2)
+    chat = cmfactory.get_accepted_chat(user1, user2)
 
     chat.send_text("testing submission header cleanup")
     user2._evtracker.wait_next_incoming_message()
@@ -162,18 +181,11 @@ def test_hide_senders_ip_address(cmfactory):
     assert public_ip not in msg.obj.as_string()
 
 
-def test_echobot(cmfactory, chatmail_config, lp, sshdomain):
+def test_echobot(cmfactory, chatmail_config, lp):
     ac = cmfactory.get_online_accounts(1)[0]
 
-    # establish contact with echobot
-    sshexec = SSHExec(sshdomain)
-    command = "cat /var/lib/echobot/invite-link.txt"
-    echo_invite_link = sshexec(call=rshell.shell, kwargs=dict(command=command))
-    chat = ac.qr_setup_contact(echo_invite_link)
-    ac._evtracker.wait_securejoin_joiner_progress(1000)
-
-    # send message and check it gets replied back
-    lp.sec("Send message to echobot")
+    lp.sec(f"Send message to echo@{chatmail_config.mail_domain}")
+    chat = ac.create_chat(f"echo@{chatmail_config.mail_domain}")
     text = "hi, I hope you text me back"
     chat.send_text(text)
     lp.sec("Wait for reply from echobot")

cmdeploy/src/cmdeploy/tests/plugin.py

@@ -10,6 +10,7 @@ from pathlib import Path
 
 import pytest
 from chatmaild.config import read_config
+from chatmaild.database import Database
 
 conftestdir = Path(__file__).parent
 
@@ -34,7 +35,7 @@ def pytest_runtest_setup(item):
             pytest.skip("skipping slow test, use --slow to run")
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture
 def chatmail_config(pytestconfig):
     current = basedir = Path().resolve()
     while 1:
@@ -48,12 +49,12 @@ def chatmail_config(pytestconfig):
     pytest.skip(f"no chatmail.ini file found in {basedir} or parent dirs")
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture
 def maildomain(chatmail_config):
     return chatmail_config.mail_domain
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture
 def sshdomain(maildomain):
     return os.environ.get("CHATMAIL_SSH", maildomain)
 
@@ -62,7 +63,7 @@ def sshdomain(maildomain):
 def maildomain2():
     domain = os.environ.get("CHATMAIL_DOMAIN2")
     if not domain:
-        pytest.skip("set CHATMAIL_DOMAIN2 to a second chatmail server")
+        pytest.skip("set CHATMAIL_DOMAIN2 to a ssh-reachable chatmail instance")
     return domain
 
 
@@ -78,17 +79,6 @@ def pytest_report_header():
         return ["-" * len(text), text, "-" * len(text)]
 
 
-@pytest.fixture
-def cm_data(request):
-    datadir = request.fspath.dirpath("data")
-
-    class CMData:
-        def get(self, name):
-            return datadir.join(name).read()
-
-    return CMData()
-
-
 @pytest.fixture
 def benchmark(request):
     def bench(func, num, name=None, reportfunc=None):
@@ -261,6 +251,13 @@ def gencreds(chatmail_config):
     return lambda domain=None: next(gen(domain))
 
 
+@pytest.fixture()
+def db(tmpdir):
+    db_path = tmpdir / "passdb.sqlite"
+    print("database path:", db_path)
+    return Database(db_path)
+
+
 #
 # Delta Chat testplugin re-use
 # use the cmfactory fixture to get chatmail instance accounts
@@ -302,13 +299,10 @@ def cmfactory(request, gencreds, tmpdir, maildomain):
     pytest.importorskip("deltachat")
     from deltachat.testplugin import ACFactory
 
+    data = request.getfixturevalue("data")
+
     testproc = ChatmailTestProcess(request.config, maildomain, gencreds)
-
-    class Data:
-        def read_path(self, path):
-            return
-
-    am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=Data())
+    am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=data)
 
     # nb. a bit hacky
     # would probably be better if deltachat's test machinery grows native support

cmdeploy/src/cmdeploy/tests/test_cmdeploy.py

@@ -1,10 +1,8 @@
-import importlib
 import os
 
 import pytest
 
 from cmdeploy.cmdeploy import get_parser, main
-from cmdeploy.www import get_paths
 
 
 @pytest.fixture(autouse=True)
@@ -23,39 +21,8 @@ class TestCmdline:
         run = parser.parse_args(["run"])
         assert init and run
 
-    def test_init_not_overwrite(self, capsys):
-        assert main(["init", "chat.example.org"]) == 0
-        capsys.readouterr()
-
-        assert main(["init", "chat.example.org"]) == 1
-        out, err = capsys.readouterr()
-        assert "path exists" in out.lower()
-
-        assert main(["init", "chat.example.org", "--force"]) == 0
-        out, err = capsys.readouterr()
-        assert "deleting config file" in out.lower()
-
-
-def test_www_folder(example_config, tmp_path):
-    reporoot = importlib.resources.files(__package__).joinpath("../../../../").resolve()
-    assert not example_config.www_folder
-    www_path, src_dir, build_dir = get_paths(example_config)
-    assert www_path.absolute() == reporoot.joinpath("www").absolute()
-    assert src_dir == reporoot.joinpath("www").joinpath("src")
-    assert build_dir == reporoot.joinpath("www").joinpath("build")
-    example_config.www_folder = "disabled"
-    www_path, _, _ = get_paths(example_config)
-    assert not www_path.is_dir()
-    example_config.www_folder = str(tmp_path)
-    www_path, src_dir, build_dir = get_paths(example_config)
-    assert www_path == tmp_path
-    assert not src_dir.exists()
-    assert not build_dir
-    src_path = tmp_path.joinpath("src")
-    os.mkdir(src_path)
-    with open(src_path / "index.md", "w") as f:
-        f.write("# Test")
-    www_path, src_dir, build_dir = get_paths(example_config)
-    assert www_path == tmp_path
-    assert src_dir == src_path
-    assert build_dir == tmp_path.joinpath("build")
+    @pytest.mark.xfail(reason="init doesn't exit anymore, check for CLI output instead")
+    def test_init_not_overwrite(self):
+        main(["init", "chat.example.org"])
+        with pytest.raises(SystemExit):
+            main(["init", "chat.example.org"])

cmdeploy/src/cmdeploy/tests/test_dns.py

@@ -1,123 +0,0 @@
-import pytest
-
-from cmdeploy import remote
-from cmdeploy.dns import check_full_zone, check_initial_remote_data
-
-
-@pytest.fixture
-def mockdns_base(monkeypatch):
-    qdict = {}
-
-    def query_dns(typ, domain):
-        try:
-            return qdict[typ][domain]
-        except KeyError:
-            return ""
-
-    monkeypatch.setattr(remote.rdns, query_dns.__name__, query_dns)
-    return qdict
-
-
-@pytest.fixture
-def mockdns(mockdns_base):
-    mockdns_base.update(
-        {
-            "A": {"some.domain": "1.1.1.1"},
-            "AAAA": {"some.domain": "fde5:cd7a:9e1c:3240:5a99:936f:cdac:53ae"},
-            "CNAME": {
-                "mta-sts.some.domain": "some.domain.",
-                "www.some.domain": "some.domain.",
-            },
-        }
-    )
-    return mockdns_base
-
-
-class TestPerformInitialChecks:
-    def test_perform_initial_checks_ok1(self, mockdns):
-        remote_data = remote.rdns.perform_initial_checks("some.domain")
-        assert remote_data["A"] == mockdns["A"]["some.domain"]
-        assert remote_data["AAAA"] == mockdns["AAAA"]["some.domain"]
-        assert remote_data["MTA_STS"] == mockdns["CNAME"]["mta-sts.some.domain"]
-        assert remote_data["WWW"] == mockdns["CNAME"]["www.some.domain"]
-
-    @pytest.mark.parametrize("drop", ["A", "AAAA"])
-    def test_perform_initial_checks_with_one_of_A_AAAA(self, mockdns, drop):
-        del mockdns[drop]
-        remote_data = remote.rdns.perform_initial_checks("some.domain")
-        assert not remote_data[drop]
-
-        l = []
-        res = check_initial_remote_data(remote_data, print=l.append)
-        assert res
-        assert not l
-
-    def test_perform_initial_checks_no_mta_sts(self, mockdns):
-        del mockdns["CNAME"]["mta-sts.some.domain"]
-        remote_data = remote.rdns.perform_initial_checks("some.domain")
-        assert not remote_data["MTA_STS"]
-
-        l = []
-        res = check_initial_remote_data(remote_data, print=l.append)
-        assert not res
-        assert len(l) == 2
-
-
-def parse_zonefile_into_dict(zonefile, mockdns_base, only_required=False):
-    for zf_line in zonefile.split("\n"):
-        if zf_line.startswith("#"):
-            if "Recommended" in zf_line and only_required:
-                return
-            continue
-        if not zf_line.strip():
-            continue
-        zf_domain, zf_typ, zf_value = zf_line.split(maxsplit=2)
-        zf_domain = zf_domain.rstrip(".")
-        zf_value = zf_value.strip()
-        mockdns_base.setdefault(zf_typ, {})[zf_domain] = zf_value
-
-
-class MockSSHExec:
-    def logged(self, func, kwargs):
-        return func(**kwargs)
-
-    def call(self, func, kwargs):
-        return func(**kwargs)
-
-
-class TestZonefileChecks:
-    def test_check_zonefile_all_ok(self, cm_data, mockdns_base):
-        zonefile = cm_data.get("zftest.zone")
-        parse_zonefile_into_dict(zonefile, mockdns_base)
-        required_diff, recommended_diff = remote.rdns.check_zonefile(zonefile)
-        assert not required_diff and not recommended_diff
-
-    def test_check_zonefile_recommended_not_set(self, cm_data, mockdns_base):
-        zonefile = cm_data.get("zftest.zone")
-        zonefile_mocked = zonefile.split("; Recommended")[0]
-        parse_zonefile_into_dict(zonefile_mocked, mockdns_base)
-        required_diff, recommended_diff = remote.rdns.check_zonefile(zonefile)
-        assert not required_diff
-        assert len(recommended_diff) == 8
-
-    def test_check_zonefile_output_required_fine(self, cm_data, mockdns_base, mockout):
-        zonefile = cm_data.get("zftest.zone")
-        zonefile_mocked = zonefile.split("; Recommended")[0]
-        parse_zonefile_into_dict(zonefile_mocked, mockdns_base, only_required=True)
-        mssh = MockSSHExec()
-        mockdns_base["mail_domain"] = "some.domain"
-        res = check_full_zone(mssh, mockdns_base, out=mockout, zonefile=zonefile)
-        assert res == 0
-        assert "WARNING" in mockout.captured_plain[0]
-        assert len(mockout.captured_plain) == 9
-
-    def test_check_zonefile_output_full(self, cm_data, mockdns_base, mockout):
-        zonefile = cm_data.get("zftest.zone")
-        parse_zonefile_into_dict(zonefile, mockdns_base)
-        mssh = MockSSHExec()
-        mockdns_base["mail_domain"] = "some.domain"
-        res = check_full_zone(mssh, mockdns_base, out=mockout, zonefile=zonefile)
-        assert res == 0
-        assert not mockout.captured_red
-        assert "correct" in mockout.captured_green[0]
-        assert not mockout.captured_red

cmdeploy/src/cmdeploy/www.py

@@ -3,7 +3,6 @@ import importlib.resources
 import time
 import traceback
 import webbrowser
-from pathlib import Path
 
 import markdown
 from chatmaild.config import read_config
@@ -31,25 +30,9 @@ def prepare_template(source):
     return render_vars, page_layout
 
 
-def get_paths(config) -> (Path, Path, Path):
-    reporoot = importlib.resources.files(__package__).joinpath("../../../").resolve()
-    www_path = Path(config.www_folder)
-    # if www_folder was not set, use default directory
-    if config.www_folder == "":
-        www_path = reporoot.joinpath("www")
-    src_dir = www_path.joinpath("src")
-    # if www_folder is a hugo page, build it
-    if src_dir.joinpath("index.md").is_file():
-        build_dir = www_path.joinpath("build")
-    # if it is not a hugo page, upload it as is
-    else:
-        build_dir = None
-    return www_path, src_dir, build_dir
-
-
-def build_webpages(src_dir, build_dir, config) -> Path:
+def build_webpages(src_dir, build_dir, config):
     try:
-        return _build_webpages(src_dir, build_dir, config)
+        _build_webpages(src_dir, build_dir, config)
     except Exception:
         print(traceback.format_exc())
 
@@ -123,11 +106,15 @@ def main():
     config = read_config(inipath)
     config.webdev = True
     assert config.mail_domain
+    www_path = reporoot.joinpath("www")
+    src_path = www_path.joinpath("src")
+    stats = None
+    build_dir = www_path.joinpath("build")
+    src_dir = www_path.joinpath("src")
+    index_path = build_dir.joinpath("index.html")
 
     # start web page generation, open a browser and wait for changes
-    www_path, src_path, build_dir = get_paths(config)
-    build_dir = build_webpages(src_path, build_dir, config)
-    index_path = build_dir.joinpath("index.html")
+    build_webpages(src_dir, build_dir, config)
     webbrowser.open(str(index_path))
     stats = snapshot_dir_stats(src_path)
     print(f"\nOpened URL: file://{index_path.resolve()}\n")
@@ -148,7 +135,7 @@ def main():
                 changenum += 1
 
         stats = newstats
-        build_webpages(src_path, build_dir, config)
+        build_webpages(src_dir, build_dir, config)
         print(f"[{changenum}] regenerated web pages at: {index_path}")
         print(f"URL: file://{index_path.resolve()}\n\n")
         count = 0

scripts/initenv.sh

@@ -1,23 +1,5 @@
 #!/bin/sh
 set -e
-
-if command -v lsb_release 2>&1 >/dev/null; then
-  case "$(lsb_release -is)" in
-    Ubuntu | Debian )
-      if ! dpkg -l | grep python3-dev 2>&1 >/dev/null
-      then
-        echo "You need to install python3-dev for installing the other dependencies."
-        exit 1
-      fi
-      if ! gcc --version 2>&1 >/dev/null
-      then
-        echo "You need to install gcc for building Python dependencies."
-        exit 1
-      fi
-      ;;
-  esac
-fi
-
 python3 -m venv --upgrade-deps venv
 
 venv/bin/pip install -e chatmaild 

www/src/index.md

@@ -11,11 +11,7 @@ for Delta Chat users.  For details how it avoids storing personal information
 please see our [privacy policy](privacy.html). 
 {% endif %}
 
-<a class="cta-button" href="DCACCOUNT:https://{{ config.mail_domain }}/new">Get a {{config.mail_domain}} chat profile</a>
-
-If you are viewing this page on a different device
-without a Delta Chat app,
-you can also **scan this QR code** with Delta Chat:
+👉 **Tap** or scan this QR code to get a `@{{config.mail_domain}}` chat profile
 
 <a href="DCACCOUNT:https://{{ config.mail_domain }}/new">
     <img width=300 style="float: none;" src="qr-chatmail-invite-{{config.mail_domain}}.png" /></a>

www/src/info.md

@@ -6,6 +6,31 @@ interoperable e-mail service for everyone. What's behind a `chatmail` is
 effectively a normal e-mail address just like any other but optimized 
 for the usage in chats, especially DeltaChat.
 
+### Choosing a chatmail address instead of using a random one
+
+In the Delta Chat account setup 
+you may tap `I already have a profile`
+and fill the two fields like this: 
+
+- `Address`: invent a word with
+{% if username_min_length == username_max_length %}
+  *exactly* {{ username_min_length }}
+{% else %}
+  {{ username_min_length}}
+  {% if username_max_length == "more" %}
+    or more
+  {% else %}
+    to {{ username_max_length }}
+  {% endif %}
+{% endif %}
+  characters
+  and append `@{{config.mail_domain}}` to it.
+
+- `Password`: invent at least {{ password_min_length }} characters.
+
+If the e-mail address is not yet taken, you'll get that account. 
+The first login sets your password. 
+
 
 ### Rate and storage limits 
 
@@ -15,25 +40,10 @@ for the usage in chats, especially DeltaChat.
 
 - You may send up to {{ config.max_user_send_per_minute }} messages per minute.
 
+- Messages are unconditionally removed {{ config.delete_mails_after }} days after arriving on the server.
+
 - You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
 
-- Messages are unconditionally removed latest {{ config.delete_mails_after }} days after arriving on the server.
-  Earlier, if storage may exceed otherwise.
-
-
-### <a name="account-deletion"></a> Account deletion 
-
-If you remove a {{ config.mail_domain }} profile from within the Delta Chat app, 
-then the according account on the server, along with all associated data,
-is automatically deleted {{ config.delete_inactive_users_after }} days afterwards. 
-
-If you use multiple devices 
-then you need to remove the according chat profile from each device
-in order for all account data to be removed on the server side. 
-
-If you have any further questions or requests regarding account deletion
-please send a message from your account to {{ config.privacy_mail }}. 
-
 
 ### Who are the operators? Which software is running? 
 

www/src/main.css

@@ -72,15 +72,3 @@ code {
   color: red;
   font-weight: bold;
 }
-
-.cta-button, .cta-button:hover, .cta-button:visited {
-    border: 1.5px solid #a4c2d0;
-    border-radius: 5px;
-    padding: 10px;
-    display: inline-block;
-    margin: 10px 0;
-
-    background: linear-gradient(120deg, #77888f, #364e59);
-    color: white !important;
-    font-weight: bold;
-}

www/src/privacy.md

@@ -3,8 +3,8 @@
 
 {% if config.mail_domain == "nine.testrun.org" %}
 Welcome to `{{config.mail_domain}}`, the default chatmail onboarding server for Delta Chat users. 
-It is operated on the side by a small sysops team
-on a voluntary basis.
+It is operated on the side by a small sysops team employed by [merlinux](https://merlinux.eu),
+an open-source R&D company also acting as the fiscal sponsor of Delta Chat app developments. 
 See [other chatmail servers](https://delta.chat/en/chatmail) for alternative server operators. 
 {% endif %}
 
@@ -23,22 +23,18 @@ A chatmail server behaves more like the Signal messaging server
 but does not know about phone numbers and securely and automatically interoperates 
 with other chatmail and classic e-mail servers. 
 
-Unlike classic e-mail servers, this chatmail server 
+In particular, this chatmail server 
 
 - unconditionally removes messages after {{ config.delete_mails_after }} days,
 
 - prohibits sending out un-encrypted messages,
 
-- does not store Internet addresses ("IP addresses"), 
-
-- does not process IP addresses in relation to email addresses.
-
-Due to the resulting lack of personal data processing
-this chatmail server may not require a privacy policy.
-
-Nevertheless, we provide legal details below to make life easier
-for data protection specialists and lawyers scrutinizing chatmail operations.
+- only has temporary log files used for debugging purposes.
 
+Legally, authorities might still regard chatmail as a "classic e-mail" server
+which collects and retains personal data. 
+We do not agree on this interpretation. Nevertheless, we provide more legal details below
+to make life easier for data protection specialists and lawyers scrutinizing chatmail operations. 
 
 
 ## 1. Name and contact information 
@@ -58,18 +54,18 @@ We have appointed a data protection officer:
 
 ## 2. Processing when using chat e-mail services
 
-We provide services optimized for the use from [Delta Chat](https://delta.chat) apps
+We provide e-mail services optimized for the use from [Delta Chat](https://delta.chat) apps 
 and process only the data necessary
-for the setup and technical execution of message delivery.
-The purpose of the processing is that users can
-read, write, manage, delete, send, and receive chat messages.
+for the setup and technical execution of the e-mail dispatch.
+The purpose of the processing is to
+read, write, manage, delete, send, and receive emails.
 For this purpose,
 we operate server-side software
-that enables us to send and receive messages.
+that enables us to send and receive e-mail messages.
+Allowing the use of the e-mail service,
+we process the following data and details:
 
-We process the following data and details:
-
-- Outgoing and incoming messages (SMTP) are stored for transit
+- Outgoing and incoming messages (SMTP) are stored for transit 
   on behalf of their users until the message can be delivered.
 
 - E-Mail-Messages are stored for the recipient and made accessible via IMAP protocols,
@@ -78,15 +74,9 @@ We process the following data and details:
 
 - IMAP and SMTP protocols are password protected with unique credentials for each account.
 
-- Users can retrieve or delete all stored messages
+- Users can retrieve or delete all stored messages 
   without intervention from the operators using standard IMAP client tools.
 
-- Users can connect to a "realtime relay service"
-  to establish Peer-to-Peer connection between user devices,
-  allowing them to send and retrieve ephemeral messages
-  which are never stored on the chatmail server, also not in encrypted form.
-
-
 ### 2.1 Account setup
 
 Creating an account happens in one of two ways on our mail servers: 
@@ -263,7 +253,7 @@ is the `{{ config.privacy_supervisor }}`.
 ## 6. Validity of this privacy policy 
 
 This data protection declaration is valid
-as of *October 2024*.
+as of *December 2023*.
 Due to the further development of our service and offers
 or due to changed legal or official requirements,
 it may become necessary to revise this data protection declaration from time to time.