Remove DKIM-Signature from incoming mail after checking

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,32 +0,0 @@
----
-name: Bug report
-about: Report something that isn't working.
-title: ''
-assignees: ''
-
----
-
-<!--
-Please fill out as much of this form as you can (leaving out stuff that is not applicable is ok).
--->
-
-- Server OS (Operating System) - preferably Debian 12: 
-- On which OS you run cmdeploy:
-
-## Expected behavior
-
-*What did you try to achieve?*
-
-## Actual behavior
-
-*What happened instead?*
-
-### Steps to reproduce the problem:
-
-1. 
-2.
-
-### Screenshots
-
-### Logs
-

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +0,0 @@
-blank_issues_enabled: true
-contact_links:
-  - name: Mutual Help Chat Group
-    url: https://i.delta.chat/#C2846EB4C1CB8DF84B1818F5E3A638FC3FBDC981&a=stalebot1%40nine.testrun.org&g=Chatmail%20Mutual%20Help&x=7sFF7Ik50pWv6J1z7RVC5527&i=d7s1HvOsk5UrSf9AoqRZggg4&s=XmX_9BAW6-g5Ao5E8PyaeKNB
-    about: If you have troubles setting up the relay server, feel free to ask here.

CHANGELOG.md

@@ -2,21 +2,8 @@
 
 ## untagged
 
-- Enforce end-to-end encryption for incoming messages. 
-  New user address mailboxes now get a `enforceE2EEincoming` file 
-  which prohibits incoming cleartext messages from other domains. 
-  An outside MTA trying to submit a cleartext message will 
-  get a "523 Encryption Needed" response, see RFC5248. 
-  If the file does not exist (as it the case for all existing accounts) 
-  incoming cleartext messages are accepted. 
-  ([#538](https://github.com/chatmail/server/pull/538))
-
-- Enforce end-to-end encryption between local addresses 
-  ([#535](https://github.com/chatmail/server/pull/535))
-
-- Limit the bind for the HTTPS server on 8443 to 127.0.0.1 
-  ([#522](https://github.com/chatmail/server/pull/522))
-  ([#532](https://github.com/chatmail/server/pull/532))
+- Remove `DKIM-Signature` from incoming mails after verifying
+  ([#530](https://github.com/chatmail/server/pull/530))
 
 - Send SNI when connecting to outside servers
   ([#524](https://github.com/chatmail/server/pull/524))

README.md

@@ -1,96 +1,82 @@
 
 <img width="800px" src="www/src/collage-top.png"/>
 
-# Chatmail relays for end-to-end encrypted e-mail
+# Chatmail servers for secure instant messaging 
 
-Chatmail relay servers are interoperable Mail Transport Agents (MTAs) designed for: 
+Chatmail servers are minimal interoperable e-mail routing machines designed for:
 
-- **Convenience:** Low friction instant onboarding
+- **Convenience:** Instant onboarding, with optional Google/Apple/Huawei push notifications
 
-- **Privacy:** No name, phone numbers, email required or collected 
+- **Privacy:** Just login, no questions asked, no name, numbers or e-mail needed
 
-- **End-to-End Encryption enforced**: only OpenPGP messages with metadata minimization allowed 
+- **Speed:** End-to-End Message delivery in well under a second
 
-- **Instant:** Privacy-preserving Push Notifications for Apple, Google, and Huawei
+- **Security:** Strict TLS, DKIM and OpenPGP with metadata-minimization enforced. 
 
-- **Speed:** Message delivery in half a second, with optional P2P realtime connections 
+- **Relaxation:** No annoying spam-checking, IP reputation or rate limits
 
-- **Transport Security:** Strict TLS and DKIM enforced 
+- **Efficiency:** messages are only stored for transit and removed automatically. 
 
-- **Reliability:** No spam or IP reputation checks; rate-limits are suitable for realtime chats
-
-- **Efficiency:** Messages are only stored for transit and removed automatically
-
-This repository contains everything needed to setup a ready-to-use chatmail relay
+This repository contains everything needed to setup a ready-to-use chatmail server
 comprised of a minimal setup of the battle-tested 
-[Postfix SMTP](https://www.postfix.org) and [Dovecot IMAP](https://www.dovecot.org) MTAs/MDAs.
+[postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services. 
 
-The automated setup is designed and optimized for providing chatmail addresses
-for immediate permission-free onboarding through chat apps and bots.
-Chatmail addresses are automatically created at first login,
-after which the initially specified password is required
-for sending and receiving messages through them.
+The automated setup is designed and optimized for providing chatmail addresses 
+for immediate permission-free onboarding through chat apps and bots. 
+Chatmail addresses are automatically created by a first login, 
+after which the initially specified password is required 
+for sending and receiving messages through them. 
 
-Please see [this list of known apps and client projects](https://chatmail.at/clients.html) 
-and [this list of known public 3rd party chatmail relay servers](https://chatmail.at/relays).
+Please see [this list of known apps and client projects](https://chatmail.at/apps.html) which offer instant onboarding on chatmail servers,
+and [this list of known public 3rd party chatmail servers](https://delta.chat/en/chatmail). 
 
 
 ## Minimal requirements, Prerequisites 
 
 You will need the following: 
 
-- Control over a domain through a DNS provider of your choice.
+- control over a domain through a DNS provider of your choice, 
 
-- A Debian 12 server with reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports.
-  IPv6 is encouraged if available.
-  Chatmail relay servers only require 1GB RAM, one CPU, and perhaps 10GB storage for a
-  few thousand active chatmail addresses.
+- a remote Debian 12 machine with IPV4 and preferably also IPV6 addresses and
+  reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports. 
+  Machine needs 1GB RAM, one slow CPU and maybe 10GB storage for a
+  few thousand active chatmail addresses,
 
-- Key-based SSH authentication to the root user.
-  You must add a passphrase-protected private key to your local ssh-agent
-  because you can't type in your passphrase during deployment.
-  (An ed25519 private key is required due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191))
+- a terminal window with password-less ssh root login to the remote machine;
+  you must have set up ssh authentication and need to use an ed25519 key, 
+  due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191);
+  you also need to add your private key to the local ssh-agent, 
+  because you can't type in your password during deployment.
 
 
 ## Getting started 
 
-We use `chat.example.org` as the chatmail domain in the following steps.
+We use `chat.example.org` as the chatmail domain in the following steps. 
 Please substitute it with your own domain. 
 
-1. Setup the initial DNS records.
-   The following is an example in the familiar BIND zone file format with
-   a TTL of 1 hour (3600 seconds).
-   Please substitute your domain and IP addresses.
+1. Install the `cmdeploy` command in a virtualenv
 
    ```
-    chat.example.com. 3600 IN A 198.51.100.5
-    chat.example.com. 3600 IN AAAA 2001:db8::5
-    www.chat.example.com. 3600 IN CNAME chat.example.com.
-    mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
-   ```
-
-2. Clone the repository and bootstrap the Python virtualenv.
-
-   ```
-    git clone https://github.com/chatmail/relay
-    cd relay
+    git clone https://github.com/chatmail/server
+    cd chatmail
     scripts/initenv.sh
    ```
 
-3. Create chatmail configuration file `chatmail.ini`:
+2. Create chatmail configuration file `chatmail.ini`:
 
    ```
     scripts/cmdeploy init chat.example.org  # <-- use your domain 
    ```
 
-4. Verify that SSH root login works:
+3. Point your domain to the server's IP address,
+   if you haven't done so already.
+   Verify that SSH root login works:
 
    ```
     ssh root@chat.example.org   # <-- use your domain 
    ```
 
-
-5. Deploy the remote chatmail relay server:
+4. Deploy to the remote chatmail server:
 
    ```
     scripts/cmdeploy run
@@ -130,25 +116,25 @@ scripts/cmdeploy bench
 
 This repository has four directories:
 
-- [cmdeploy](https://github.com/chatmail/relay/tree/main/cmdeploy)
+- [cmdeploy](https://github.com/chatmail/server/tree/main/cmdeploy)
   is a collection of configuration files
   and a [pyinfra](https://pyinfra.com)-based deployment script.
 
-- [chatmaild](https://github.com/chatmail/relay/tree/main/chatmaild)
-  is a Python package containing several small services
+- [chatmaild](https://github.com/chatmail/server/tree/main/chatmaild)
+  is a python package containing several small services
   which handle authentication,
   trigger push notifications on new messages,
   ensure that outbound mails are encrypted,
   delete inactive users,
   and some other minor things.
-  chatmaild can also be installed as a stand-alone Python package.
+  chatmaild can also be installed as a stand-alone python package.
 
-- [www](https://github.com/chatmail/relay/tree/main/www)
+- [www](https://github.com/chatmail/server/tree/main/www)
   contains the html, css, and markdown files
-  which make up a chatmail relay's web page.
-  Edit them before deploying to make your chatmail relay stand out.
+  which make up a chatmail server's web page.
+  Edit them before deploying to make your chatmail server stand out.
 
-- [scripts](https://github.com/chatmail/relay/tree/main/scripts)
+- [scripts](https://github.com/chatmail/server/tree/main/scripts)
   offers two convenience tools for beginners;
   `initenv.sh` installs the necessary dependencies to a local virtual environment,
   and the `scripts/cmdeploy` script enables you
@@ -160,81 +146,79 @@ The `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
 helps with setting up and managing the chatmail service.
 `cmdeploy init` creates the `chatmail.ini` config file.
 `cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [script](`cmdeploy/src/cmdeploy/__init__.py`)
-to automatically install or upgrade all chatmail components on a relay,
+to automatically install or upgrade all chatmail components on a server,
 according to the `chatmail.ini` config.
 
 The components of chatmail are:
 
-- [Postfix SMTP MTA](https://www.postfix.org) accepts and relays messages
-  (both from your users and from the wider e-mail MTA network)
+- [postfix smtp server](https://www.postfix.org) accepts sent messages (both from your users and from other servers)
 
-- [Dovecot IMAP MDA](https://www.dovecot.org) stores messages for your users until they download them
+- [dovecot imap server](https://www.dovecot.org) stores messages for your users until they download them
 
-- [Nginx](https://nginx.org/) shows the web page with your privacy policy and additional information
+- [nginx](https://nginx.org/) shows the web page with your privacy policy and additional information
 
-- [acmetool](https://hlandau.github.io/acmetool/) manages TLS certificates for Dovecot, Postfix, and Nginx
+- [acmetool](https://hlandau.github.io/acmetool/) manages TLS certificates for dovecot, postfix, and nginx
 
-- [OpenDKIM](http://www.opendkim.org/) for signing messages with DKIM and rejecting inbound messages without DKIM
+- [opendkim](http://www.opendkim.org/) for signing messages with DKIM and rejecting inbound messages without DKIM
 
 - [mtail](https://google.github.io/mtail/) for collecting anonymized metrics in case you have monitoring
 
-- [Iroh relay](https://www.iroh.computer/docs/concepts/relay) 
-  which helps client devices to establish Peer-to-Peer connections 
-
 - and the chatmaild services, explained in the next section:
 
 ### chatmaild
 
-`chatmaild` implements various systemd-controlled services  
-that integrate with Dovecot and Postfix to achieve instant-onboarding and 
-only relaying OpenPGP end-to-end messages encrypted messages. 
-A short overview of `chatmaild` services: 
+chatmaild offers several commands
+which differentiate a *chatmail* server from a classic mail server.
+If you deploy them with cmdeploy,
+they are run by systemd services in the background.
+A short overview:
 
-- [`doveauth`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/doveauth.py) 
-  implements create-on-login address semantics and is used 
-  by Dovecot during IMAP login and by Postfix during SMTP/SUBMISSION login
+- [`doveauth`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/doveauth.py) implements
+  create-on-login account creation semantics and is used
+  by Dovecot during login authentication and by Postfix
   which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
-  to authenticate logins. 
+  to authenticate users
+  to send mails for them.
 
-- [`filtermail`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/filtermail.py) 
-  prevents unencrypted email from leaving or entering the chatmail service
-  and is integrated into Postfix's outbound and inbound mail pipelines.
+- [`filtermail`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/filtermail.py) prevents
+  unencrypted e-mail from leaving the chatmail service
+  and is integrated into postfix's outbound mail pipelines.
 
-- [`chatmail-metadata`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
-  [Dovecot lua script](https://github.com/chatmail/relay/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
-  to store user-specific relay-side config.
+- [`chatmail-metadata`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
+  [dovecot lua script](https://github.com/chatmail/server/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
+  to store user-specific server-side config.
   On new messages,
-  it [passes the user's push notification token](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/notifier.py)
+  it [passes the user's push notification token](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/notifier.py)
   to [notifications.delta.chat](https://delta.chat/help#instant-delivery)
   so the push notifications on the user's phone can be triggered
-  by Apple/Google/Huawei.
+  by Apple/Google.
 
-- [`delete_inactive_users`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
+- [`delete_inactive_users`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
   deletes users if they have not logged in for a very long time.
   The timeframe can be configured in `chatmail.ini`.
 
-- [`lastlogin`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/lastlogin.py)
-  is contacted by Dovecot when a user logs in
+- [`lastlogin`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/lastlogin.py)
+  is contacted by dovecot when a user logs in
   and stores the date of the login.
 
-- [`echobot`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/echo.py)
+- [`echobot`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/echo.py)
   is a small bot for test purposes.
   It simply echoes back messages from users.
 
-- [`chatmail-metrics`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metrics.py)
+- [`chatmail-metrics`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/metrics.py)
   collects some metrics and displays them at `https://example.org/metrics`.
 
 ### Home page and getting started for users
 
-`cmdeploy run` also creates default static web pages and deploys them
-to a Nginx web server with:
+`cmdeploy run` also creates default static Web pages and deploys them
+to a nginx web server with: 
 
 - a default `index.html` along with a QR code that users can click to
-  create an address on your chatmail relay 
+  create accounts on your chatmail provider,
 
-- a default `info.html` that is linked from the home page
+- a default `info.html` that is linked from the home page,
 
-- a default `policy.html` that is linked from the home page
+- a default `policy.html` that is linked from the home page.
 
 All `.html` files are generated
 by the according markdown `.md` file in the `www/src` directory.
@@ -242,64 +226,48 @@ by the according markdown `.md` file in the `www/src` directory.
 
 ### Refining the web pages
 
+
 ```
 scripts/cmdeploy webdev
 ```
 
-This starts a local live development cycle for chatmail web pages:
+This starts a local live development cycle for chatmail Web pages:
 
 - uses the `www/src/page-layout.html` file for producing static
   HTML pages from `www/src/*.md` files
 
 - continously builds the web presence reading files from `www/src` directory
-  and generating HTML files and copying assets to the `www/build` directory.
+  and generating html files and copying assets to the `www/build` directory.
 
 - Starts a browser window automatically where you can "refresh" as needed.
 
-## Mailbox directory layout
 
-Fresh chatmail addresses have a mailbox directory that contains: 
+## Emergency Commands to disable automatic account creation
 
-- a `password` file with the salted password required for authenticating
-  whether a login may use the address to send/receive messages. 
-  If you modify the password file manually, you effectively block the user. 
-
-- `enforceE2EEincoming` is a default-created file with each address. 
-  If present the file indicates that this chatmail address rejects incoming cleartext messages.
-  If absent the address accepts incoming cleartext messages. 
-
-- `dovecot*`, `cur`, `new` and `tmp` represent IMAP/mailbox state. 
-  If the address is only used by one device, the Maildir directories
-  will typically be empty unless the user of that address hasn't been online 
-  for a while. 
-
-
-## Emergency Commands to disable automatic address creation
-
-If you need to stop address creation,
-e.g. because some script is wildly creating addresses, 
-login with ssh and run:
+If you need to stop account creation,
+e.g. because some script is wildly creating accounts,
+login to the server with ssh and run:
 
 ```
     touch /etc/chatmail-nocreate
 ```
 
-Chatmail address creation will be denied while this file is present.
+While this file is present, account creation will be blocked.
 
 ### Ports
 
-[Postfix](http://www.postfix.org/) listens on ports 25 (SMTP) and 587 (SUBMISSION) and 465 (SUBMISSIONS).
-[Dovecot](https://www.dovecot.org/) listens on ports 143 (IMAP) and 993 (IMAPS).
-[Nginx](https://www.nginx.com/) listens on port 8443 (HTTPS-ALT) and 443 (HTTPS).
+[Postfix](http://www.postfix.org/) listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
+[Dovecot](https://www.dovecot.org/) listens on ports 143 (imap) and 993 (imaps).
+[nginx](https://www.nginx.com/) listens on port 8443 (https-alt) and 443 (https).
 Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
-[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (HTTP).
+[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (http).
 
-chatmail-core based apps will, however, discover all ports and configurations
-automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail relay server.
+Chatmail-core based apps will, however, discover all ports and configurations
+automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail server.
 
 ## Email authentication
 
-Chatmail relays enforce [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
+chatmail servers rely on [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
 to authenticate incoming emails.
 Incoming emails must have a valid DKIM signature with
 Signing Domain Identifier (SDID, `d=` parameter in the DKIM-Signature header)
@@ -326,20 +294,20 @@ this is ensured by `filtermail` proxy.
 
 Postfix is configured to require valid TLS
 by setting [`smtp_tls_security_level`](https://www.postfix.org/postconf.5.html#smtp_tls_security_level) to `verify`.
-If emails don't arrive at your chatmail relay server, 
-the problem is likely that your relay does not have a valid TLS certificate.
+If emails don't arrive from a chatmail server to your server,
+the problem is likely that your server does not have a valid TLS certificate.
 
-You can test it by resolving `MX` records of your relay domain
-and then connecting to MX relays (e.g `mx.example.org`) with
+You can test it by resolving `MX` records of your server domain
+and then connecting to MX servers (e.g `mx.example.org`) with
 `openssl s_client -connect mx.example.org:25 -verify_hostname mx.example.org -verify_return_error -starttls smtp`
 from the host that has open port 25 to verify that certificate is valid.
 
-When providing a TLS certificate to your chatmail relay server, 
+When providing a TLS certificate to your server,
 make sure to provide the full certificate chain
 and not just the last certificate.
 
-If you are running an Exim server and don't see incoming connections
-from a chatmail relay server in the logs,
+If you are running Exim server and don't see incoming connections
+from a chatmail server in the logs,
 make sure `smtp_no_mail` log item is enabled in the config
 with `log_selector = +smtp_no_mail`.
 By default Exim does not log sessions that are closed
@@ -348,104 +316,101 @@ This happens if certificate is not recognized as valid by Postfix,
 so you might think that connection is not established
 while actually it is a problem with your TLS certificate.
 
-## Migrating a chatmail relay to a new host
+## Migrating chatmail server to a new host
 
-If you want to migrate chatmail relay from an old machine
+If you want to migrate chatmail from an old machine
 to a new machine,
 you can use these steps.
-They were tested with a Linux laptop;
+They were tested with a linux laptop;
 you might need to adjust some of the steps to your environment.
 
 Let's assume that your `mail_domain` is `mail.example.org`,
 all involved machines run Debian 12,
-your old site's IP address is `13.37.13.37`,
-and your new site's IP address is `13.12.23.42`.
+your old server's IP address is `13.37.13.37`,
+and your new server's IP address is `13.12.23.42`.
 
-Note, you should lower the TTLs of your DNS records to a value
-such as 300 (5 minutes) so the migration happens as smoothly as possible.
+During the guide, you might get a warning about changed SSH Host keys;
+in this case, just run `ssh-keygen -R "mail.example.org"` as recommended
+to make sure you can connect with SSH.
 
-During the guide you might get a warning about changed SSH Host keys;
-in this case, just run `ssh-keygen -R "mail.example.org"` as recommended.
+1. First, copy `/var/lib/acme` to the new server with
+   `ssh root@13.37.13.37 tar c /var/lib/acme | ssh root@13.12.23.42 tar x -C /var/lib/`.
+   This transfers your TLS certificate.
 
-1. First, disable mail services on the old site. 
+2. You should also copy `/etc/dkimkeys` to the new server with
+   `ssh root@13.37.13.37 tar c /etc/dkimkeys | ssh root@13.12.23.42 tar x -C /etc/`
+   so the DKIM DNS record stays correct.
 
-   ```
-    cmdeploy run --disable-mail --ssh-host 13.37.13.37
-   ```
+3. On the new server, run `chown root: -R /var/lib/acme` and `chown opendkim: -R /etc/dkimkeys` to make sure the permissions are correct.
+
+4. Run `cmdeploy run --disable-mail --ssh-host 13.12.23.42` to install chatmail on the new machine.
+   postfix and dovecot are disabled for now,
+   we will enable them later.
+
+5. Now, point DNS to the new IP addresses.
+
+   You can already remove the old IP addresses from DNS.
+   Existing Chatmail app users or bots will still be able to connect
+   to the old server, send and receive messages,
+   but new ones will fail to create new profiles
+   with your chatmail server.
+
+   If other servers try to deliver messages to your new server they will fail,
+   but normally email servers will retry delivering messages
+   for at least a week, so messages will not be lost.
+
+6. Now you can run `cmdeploy run --disable-mail --ssh-host 13.37.13.37` to disable your old server.
 
    Now your users will notice the migration
    and will not be able to send or receive messages
    until the migration is completed.
 
-2. Now we want to copy `/home/vmail`, `/var/lib/acme`, `/etc/dkimkeys`, `/run/echobot`, and `/var/spool/postfix` to the new site. 
-   Login to the old site while forwarding your SSH agent 
-   so you can copy directly from the old to the new site with your SSH key:
-   ```
-    ssh -A root@13.37.13.37
-    tar c - /home/vmail/mail /var/lib/acme /etc/dkimkeys /run/echobot /var/spool/postfix | ssh root@13.12.23.42 "tar x -C /"
-   ```
+7. After everything is stopped,
+    you can copy the `/home/vmail/mail` directory to the new server.
+    It includes all user data, messages, password hashes, etc.
 
-   This transfers all addresses, the TLS certificate, DKIM keys (so DKIM DNS record remains valid), and the echobot's password so it continues to function.
-   It also preserves the Postfix mail spool so any messages pending delivery will still be delivered.
+    Just run: `ssh root@13.37.13.37 tar c /home/vmail/mail | ssh root@13.12.23.42 tar x -C /home/vmail/`
 
-3. Install chatmail on the new machine:
+    After this, your new server has all the necessary files to start operating :)
 
-   ```
-    cmdeploy run --disable-mail --ssh-host 13.12.23.42
-   ```
-   Postfix and Dovecot are disabled for now; we will enable them later. 
-   We first need to make the new site fully operational. 
+8. To be sure the permissions are still fine,
+    run `chown vmail: -R /home/vmail` on the new server.
 
-3. On the new site, run the following to ensure the ownership is correct in case UIDs/GIDs changed:
-
-   ```
-    chown root: -R /var/lib/acme
-    chown opendkim: -R /etc/dkimkeys
-    chown vmail: -R /home/vmail/mail
-    chown echobot: -R /run/echobot
-   ```
-
-4. Now, update DNS entries. 
-
-   If other MTAs try to deliver messages to your chatmail domain they may fail intermittently,
-   as DNS catches up with the new site settings 
-   but normally will retry delivering messages
-   for at least a week, so messages will not be lost.
-
-5. Finally, you can execute `cmdeploy run --ssh-host 13.12.23.42` to turn on chatmail on the new relay.
-   Your users will be able to use the chatmail relay as soon as the DNS changes have propagated.
-   Voilà!
+9. Finally, you can run `cmdeploy run` to turn on chatmail on the new server.
+    Your users can continue using the chatmail server,
+    and messages which were sent after step 6. should arrive now.
+    Voilà!
 
 ## Setting up a reverse proxy
 
-A chatmail relay MTA does not track or depend on the client IP address
+A chatmail server does not depend on the client IP address
 for its operation, so it can be run behind a reverse proxy.
 This will not even affect incoming mail authentication
 as DKIM only checks the cryptographic signature
 of the message and does not use the IP address as the input.
 
-For example, you may want to self-host your chatmail relay
+For example, you may want to self-host your chatmail server
 and only use hosted VPS to provide a public IP address
 for client connections and incoming mail.
-You can connect chatmail relay to VPS
+You can connect chatmail server to VPS
 using a tunnel protocol
 such as [WireGuard](https://www.wireguard.com/)
 and setup a reverse proxy on a VPS
-to forward connections to the chatmail relay
+to forward connections to the chatmail server
 over the tunnel.
 You can also setup multiple reverse proxies
-for your chatmail relay in different networks
-to ensure your relay is reachable even when
+for your chatmail server in different networks
+to ensure your server is reachable even when
 one of the IPs becomes inaccessible due to
 hosting or routing problems.
 
-Note that your chatmail relay still needs
+Note that your server still needs
 to be able to make outgoing connections on port 25
 to send messages outside.
 
 To setup a reverse proxy
 (or rather Destination NAT, DNAT)
-for your chatmail relay,
+for your chatmail server,
 put the following configuration in `/etc/nftables.conf`:
 ```
 #!/usr/sbin/nft -f
@@ -457,7 +422,7 @@ define wan = eth0
 # Which ports to proxy.
 #
 # Note that SSH is not proxied
-# so it is possible to log into the proxy server 
+# so it is possible to log into the proxy server
 # and not the original one.
 define ports = { smtp, http, https, imap, imaps, submission, submissions }
 
@@ -520,7 +485,7 @@ table inet filter {
 ```
 
 Run `systemctl enable nftables.service`
-to ensure configuration is reloaded when the proxy relay reboots.
+to ensure configuration is reloaded when the proxy server reboots.
 
 Uncomment in `/etc/sysctl.conf` the following two lines:
 
@@ -529,7 +494,7 @@ net.ipv4.ip_forward=1
 net.ipv6.conf.all.forwarding=1
 ```
 
-Then reboot the relay or do `sysctl -p` and `nft -f /etc/nftables.conf`.
+Then reboot the server or do `sysctl -p` and `nft -f /etc/nftables.conf`.
 
-Once proxy relay is set up,
+Once proxy server is set up,
 you can add its IP address to the DNS.

chatmaild/src/chatmaild/config.py

@@ -11,11 +11,7 @@ def read_config(inipath):
     assert Path(inipath).exists(), inipath
     cfg = iniconfig.IniConfig(inipath)
     params = cfg.sections["params"]
-    default_config_content = get_default_config_content(params["mail_domain"])
-    df_params = iniconfig.IniConfig("ini", data=default_config_content)["params"]
-    new_params = dict(df_params.items())
-    new_params.update(params)
-    return Config(inipath, params=new_params)
+    return Config(inipath, params=params)
 
 
 class Config:
@@ -33,13 +29,7 @@ class Config:
         self.passthrough_senders = params["passthrough_senders"].split()
         self.passthrough_recipients = params["passthrough_recipients"].split()
         self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
-        self.filtermail_smtp_port_incoming = int(
-            params["filtermail_smtp_port_incoming"]
-        )
         self.postfix_reinject_port = int(params["postfix_reinject_port"])
-        self.postfix_reinject_port_incoming = int(
-            params["postfix_reinject_port_incoming"]
-        )
         self.mtail_address = params.get("mtail_address")
         self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
         self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
@@ -79,11 +69,6 @@ class Config:
 
 def write_initial_config(inipath, mail_domain, overrides):
     """Write out default config file, using the specified config value overrides."""
-    content = get_default_config_content(mail_domain, **overrides)
-    inipath.write_text(content)
-
-
-def get_default_config_content(mail_domain, **overrides):
     from importlib.resources import files
 
     inidir = files(__package__).joinpath("ini")
@@ -128,4 +113,5 @@ def get_default_config_content(mail_domain, **overrides):
             else:
                 lines.append(line)
         content = "\n".join(lines)
-    return content
+
+    inipath.write_text(content)

chatmaild/src/chatmaild/echo.py

@@ -8,7 +8,6 @@ import logging
 import os
 import subprocess
 import sys
-from pathlib import Path
 
 from deltachat_rpc_client import Bot, DeltaChat, EventType, Rpc, events
 
@@ -98,10 +97,6 @@ def main():
         if not bot.is_configured():
             bot.configure(addr, password)
 
-        # write invite link to working directory
-        invitelink = bot.account.get_qr_code()
-        Path("invite-link.txt").write_text(invitelink)
-
         bot.run_forever()
 
 

chatmaild/src/chatmaild/filtermail.py

@@ -11,12 +11,9 @@ from email.utils import parseaddr
 from smtplib import SMTP as SMTPClient
 
 from aiosmtpd.controller import Controller
-from aiosmtpd.smtp import SMTP
 
 from .config import read_config
 
-ENCRYPTION_NEEDED_523 = "523 Encryption Needed: Invalid Unencrypted Mail"
-
 
 def check_openpgp_payload(payload: bytes):
     """Checks the OpenPGP payload.
@@ -160,14 +157,9 @@ def check_encrypted(message):
     return True
 
 
-async def asyncmain_beforequeue(config, mode):
-    if mode == "outgoing":
-        port = config.filtermail_smtp_port
-        handler = OutgoingBeforeQueueHandler(config)
-    else:
-        port = config.filtermail_smtp_port_incoming
-        handler = IncomingBeforeQueueHandler(config)
-    HackedController(handler, hostname="127.0.0.1", port=port).start()
+async def asyncmain_beforequeue(config):
+    port = config.filtermail_smtp_port
+    Controller(BeforeQueueHandler(config), hostname="127.0.0.1", port=port).start()
 
 
 def recipient_matches_passthrough(recipient, passthrough_recipients):
@@ -179,21 +171,7 @@ def recipient_matches_passthrough(recipient, passthrough_recipients):
     return False
 
 
-class HackedController(Controller):
-    def factory(self):
-        return SMTPDiscardRCPTO_options(self.handler, **self.SMTP_kwargs)
-
-
-class SMTPDiscardRCPTO_options(SMTP):
-    def _getparams(self, params):
-        # aiosmtpd's SMTP daemon fails to handle a request if there are RCPT TO options
-        # We just ignore them for our incoming filtermail purposes
-        if len(params) == 1 and params[0].startswith("ORCPT"):
-            return {}
-        return super()._getparams(params)
-
-
-class OutgoingBeforeQueueHandler:
+class BeforeQueueHandler:
     def __init__(self, config):
         self.config = config
         self.send_rate_limiter = SendRateLimiter()
@@ -231,86 +209,40 @@ class OutgoingBeforeQueueHandler:
         mail_encrypted = check_encrypted(message)
 
         _, from_addr = parseaddr(message.get("from").strip())
+        envelope_from_domain = from_addr.split("@").pop()
 
+        logging.info(f"mime-from: {from_addr} envelope-from: {envelope.mail_from!r}")
         if envelope.mail_from.lower() != from_addr.lower():
             return f"500 Invalid FROM <{from_addr!r}> for <{envelope.mail_from!r}>"
 
-        if mail_encrypted or is_securejoin(message):
-            print("Outgoing: Filtering encrypted mail.", file=sys.stderr)
-            return
-
-        print("Outgoing: Filtering unencrypted mail.", file=sys.stderr)
+        if mail_encrypted:
+            print("Filtering encrypted mail.", file=sys.stderr)
+        else:
+            print("Filtering unencrypted mail.", file=sys.stderr)
 
         if envelope.mail_from in self.config.passthrough_senders:
             return
 
-        # allow self-sent Autocrypt Setup Message
-        if envelope.rcpt_tos == [from_addr]:
-            if message.get("subject") == "Autocrypt Setup Message":
-                if message.get_content_type() == "multipart/mixed":
-                    return
-
         passthrough_recipients = self.config.passthrough_recipients
 
-        for recipient in envelope.rcpt_tos:
-            if recipient_matches_passthrough(recipient, passthrough_recipients):
-                continue
-
-            print("Rejected unencrypted mail.", file=sys.stderr)
-            return ENCRYPTION_NEEDED_523
-
-
-class IncomingBeforeQueueHandler:
-    def __init__(self, config):
-        self.config = config
-
-    async def handle_DATA(self, server, session, envelope):
-        logging.info("handle_DATA before-queue")
-        error = self.check_DATA(envelope)
-        if error:
-            return error
-        logging.info("re-injecting the mail that passed checks")
-
-        # the smtp daemon on reinject_port_incoming gives it to dkim milter
-        # which looks at source address to determine whether to verify or sign
-        client = SMTPClient(
-            "localhost",
-            self.config.postfix_reinject_port_incoming,
-            source_address=("127.0.0.2", 0),
-        )
-        client.sendmail(
-            envelope.mail_from, envelope.rcpt_tos, envelope.original_content
-        )
-        return "250 OK"
-
-    def check_DATA(self, envelope):
-        """the central filtering function for e-mails."""
-        logging.info(f"Processing DATA message from {envelope.mail_from}")
-
-        message = BytesParser(policy=policy.default).parsebytes(envelope.content)
-        mail_encrypted = check_encrypted(message)
-
         if mail_encrypted or is_securejoin(message):
-            print("Incoming: Filtering encrypted mail.", file=sys.stderr)
             return
 
-        print("Incoming: Filtering unencrypted mail.", file=sys.stderr)
-
-        # we want cleartext mailer-daemon messages to pass through
-        # chatmail core will typically not display them as normal messages
-        if message.get("auto-submitted"):
-            _, from_addr = parseaddr(message.get("from").strip())
-            if from_addr.lower().startswith("mailer-daemon@"):
-                if message.get_content_type() == "multipart/report":
-                    return
-
         for recipient in envelope.rcpt_tos:
-            user = self.config.get_user(recipient)
-            if user is None or user.is_incoming_cleartext_ok():
+            if envelope.mail_from == recipient:
+                # Always allow sending emails to self.
                 continue
+            if recipient_matches_passthrough(recipient, passthrough_recipients):
+                continue
+            res = recipient.split("@")
+            if len(res) != 2:
+                return f"500 Invalid address <{recipient}>"
+            _recipient_addr, recipient_domain = res
 
-            print("Rejected unencrypted mail.", file=sys.stderr)
-            return ENCRYPTION_NEEDED_523
+            is_outgoing = recipient_domain != envelope_from_domain
+            if is_outgoing:
+                print("Rejected unencrypted mail.", file=sys.stderr)
+                return f"500 Invalid unencrypted mail to <{recipient}>"
 
 
 class SendRateLimiter:
@@ -329,14 +261,11 @@ class SendRateLimiter:
 
 def main():
     args = sys.argv[1:]
-    assert len(args) == 2
+    assert len(args) == 1
     config = read_config(args[0])
-    mode = args[1]
     logging.basicConfig(level=logging.WARN)
     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
-    assert mode in ["incoming", "outgoing"]
-    task = asyncmain_beforequeue(config, mode)
+    task = asyncmain_beforequeue(config)
     loop.create_task(task)
-    logging.info("entering serving loop")
     loop.run_forever()

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -46,13 +46,11 @@ passthrough_recipients = xstore@testrun.org
 # Deployment Details
 #
 
-# SMTP outgoing filtermail and reinjection 
+# where the filtermail SMTP service listens
 filtermail_smtp_port = 10080
-postfix_reinject_port = 10025
 
-# SMTP incoming filtermail and reinjection 
-filtermail_smtp_port_incoming = 10081
-postfix_reinject_port_incoming = 10026
+# postfix accepts on the localhost reinject SMTP port
+postfix_reinject_port = 10025
 
 # if set to "True" IPv6 is disabled
 disable_ipv6 = False

chatmaild/src/chatmaild/metrics.py

@@ -11,8 +11,6 @@ def main(vmail_dir=None):
     ci_accounts = 0
 
     for path in Path(vmail_dir).iterdir():
-        if not path.joinpath("cur").is_dir():
-            continue
         accounts += 1
         if path.name[:3] in ("ci-", "ac_"):
             ci_accounts += 1

chatmaild/src/chatmaild/tests/mail-data/asm.eml

@@ -1,56 +0,0 @@
-From: {from_addr}
-To: {to_addr}
-Autocrypt-Setup-Message: v1
-Subject: Autocrypt Setup Message
-Date: Tue, 22 Jan 2019 12:56:29 +0100
-Content-type: multipart/mixed; boundary="Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ"
-
---Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
-Content-Type: text/plain
-
-This message contains all information to transfer your Autocrypt
-settings along with your secret key securely from your original
-device.
-
-To set up your new device for Autocrypt, please follow the
-instuctions that should be presented by your new device.
-
-You can keep this message and use it as a backup for your secret
-key. If you want to do this, you should write down the Setup Code
-and store it securely.
---Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
-Content-Type: application/autocrypt-setup
-Content-Disposition: attachment; filename="autocrypt-setup-message.html"
-
-<html><body>
-<p>
-This is the Autocrypt setup file used to transfer settings and
-keys between clients. You can decrypt it using the Setup Code
-presented on your old device, and then import the contained key
-into your keyring.
-</p>
-
-<pre>
------BEGIN PGP MESSAGE-----
-Passphrase-Format: numeric9x4
-Passphrase-Begin: 17
-
-jA0EBwMCFAxADoCdzeX/0ukBlqI5+pfpKb751qd/7nLNbkpy3gVcaf1QwRPZYt40
-Ynp08UqRQ2g48ZlnzHLSwlTGOPTuv2Jt8ka+pgZ45xzvJSG2gau03xP4VsC271kR
-VmCjdb0Y6Rk96mAwfGzrkbaRQ9Z7fIoL866GOv6h9neiVIkp+JYlTV6ISD0ZQJ4Q
-I6dOQkB/TWZyVjtiJDOQHdfNWliA6NtqaLq19wlu9L5xXjuNpY95KwR8EJXWe0+o
-Y3d2U/KxOAkXKghP2Qg1GtlPVeGC5T4p03TGI6pzKT+kHX6Rrm9wK6sM9aTquMmF
-Vok84Jg1DFnwivWC2RILR81rXi7k/+Y6MUbveFgJ9cQduqpxnmD7TjOblYu7M6zp
-YGAUxh8DRKlIMn2QsA++DBYQ6ACZvwuY8qTDLkqPDo4WqM313dsMJbyGjDdVE7EM
-PESS+RlABETpZXz8g/ycr6DIUNdlbPcmYlsBfHWDOuR2GFFTwmlv5slWS39dJv38
-E0eIe1CwdxI801Se7t7dUUS/ZF8wb6GlmxOcqGbF8eko1Z0S64IAm7/h13MRQCxI
-geQnHfGYVJ2FOimoCMEKwfa9x++RFTDW0u7spDC2uWvK/1viV8OfRppFhLr/kmKb
-18lWXuAz80DAjUDUsVqEq2MvJBJGoCJUEyjuRsLkHYRM5jYk4v50LyyR0Om73nWF
-nZBqmqNzdr7Xb9PHHdFhnEc0VvoYbrcM0RVYcEMW3YbmejM891j1d6Iv+/n/qND/
-NdebGrfWJMmFLf/iEkzTZ3/v5inW9LpWoRc94ioCjJTaEo8Rib6ARRFaJVIsmNXi
-YicFGO98D+zX+a2t9Yz6IpPajVslnOp6ScpmXgts/2XWD7oE+JgxSAqo/dLVsHgP
-Ufo=
-=pulM
------END PGP MESSAGE-----
-</pre></body></html>
---Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ--

chatmaild/src/chatmaild/tests/mail-data/mailer-daemon.eml

@@ -1,46 +0,0 @@
-Date: Fri, 8 Jul 1994 09:21:47 -0400
-From: Mail Delivery Subsystem <MAILER-DAEMON@example.org>
-Subject: Returned mail: User unknown
-To: <owner-ups-mib@CS.UTK.EDU>
-Auto-Submitted: auto-replied 
-MIME-Version: 1.0
-Content-Type: multipart/report; report-type=delivery-status;
-      boundary="JAA13167.773673707/CS.UTK.EDU"
-
---JAA13167.773673707/CS.UTK.EDU
-content-type: text/plain; charset=us-ascii
-
-   ----- The following addresses had delivery problems -----
-<arathib@vnet.ibm.com>  (unrecoverable error)
-<wsnell@sdcc13.ucsd.edu>  (unrecoverable error)
-
---JAA13167.773673707/CS.UTK.EDU
-content-type: message/delivery-status
-
-Reporting-MTA: dns; cs.utk.edu
-
-Original-Recipient: rfc822;arathib@vnet.ibm.com
-Final-Recipient: rfc822;arathib@vnet.ibm.com
-Action: failed
-Status: 5.0.0 (permanent failure)
-Diagnostic-Code: smtp;
- 550 'arathib@vnet.IBM.COM' is not a registered gateway user
-Remote-MTA: dns; vnet.ibm.com
-
-Original-Recipient: rfc822;johnh@hpnjld.njd.hp.com
-Final-Recipient: rfc822;johnh@hpnjld.njd.hp.com
-Action: delayed
-Status: 4.0.0 (hpnjld.njd.jp.com: host name lookup failure)
-
-Original-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
-Final-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
-Action: failed
-Status: 5.0.0
-Diagnostic-Code: smtp; 550 user unknown
-Remote-MTA: dns; sdcc13.ucsd.edu
-
---JAA13167.773673707/CS.UTK.EDU
-content-type: message/rfc822
-
-[original message goes here]
---JAA13167.773673707/CS.UTK.EDU--

chatmaild/src/chatmaild/tests/plugin.py

@@ -72,8 +72,9 @@ def maildata(request):
     def maildata(name, from_addr, to_addr, subject="[...]"):
         # Using `.read_bytes().decode()` instead of `.read_text()` to preserve newlines.
         data = datadir.joinpath(name).read_bytes().decode()
+
         text = data.format(from_addr=from_addr, to_addr=to_addr, subject=subject)
-        return BytesParser(policy=policy.SMTP).parsebytes(text.encode())
+        return BytesParser(policy=policy.default).parsebytes(text.encode())
 
     return maildata
 

chatmaild/src/chatmaild/tests/test_config.py

@@ -15,14 +15,6 @@ def test_read_config_basic(example_config):
     assert example_config.mail_domain == "chat.example.org"
 
 
-def test_read_config_basic_using_defaults(tmp_path, maildomain):
-    inipath = tmp_path.joinpath("chatmail.ini")
-    inipath.write_text(f"[params]\nmail_domain = {maildomain}")
-    example_config = read_config(inipath)
-    assert example_config.max_user_send_per_minute == 60
-    assert example_config.filtermail_smtp_port_incoming == 10081
-
-
 def test_read_config_testrun(make_config):
     config = make_config("something.testrun.org")
     assert config.mail_domain == "something.testrun.org"

chatmaild/src/chatmaild/tests/test_filtermail.py

@@ -1,8 +1,7 @@
 import pytest
 
 from chatmaild.filtermail import (
-    IncomingBeforeQueueHandler,
-    OutgoingBeforeQueueHandler,
+    BeforeQueueHandler,
     SendRateLimiter,
     check_armored_payload,
     check_encrypted,
@@ -19,13 +18,7 @@ def maildomain():
 @pytest.fixture
 def handler(make_config, maildomain):
     config = make_config(maildomain)
-    return OutgoingBeforeQueueHandler(config)
-
-
-@pytest.fixture
-def inhandler(make_config, maildomain):
-    config = make_config(maildomain)
-    return IncomingBeforeQueueHandler(config)
+    return BeforeQueueHandler(config)
 
 
 def test_reject_forged_from(maildata, gencreds, handler):
@@ -36,14 +29,14 @@ def test_reject_forged_from(maildata, gencreds, handler):
     # test that the filter lets good mail through
     to_addr = gencreds()[0]
     env.content = maildata(
-        "encrypted.eml", from_addr=env.mail_from, to_addr=to_addr
+        "plain.eml", from_addr=env.mail_from, to_addr=to_addr
     ).as_bytes()
 
     assert not handler.check_DATA(envelope=env)
 
     # test that the filter rejects forged mail
     env.content = maildata(
-        "encrypted.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
+        "plain.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
     ).as_bytes()
     error = handler.check_DATA(envelope=env)
     assert "500" in error
@@ -113,7 +106,7 @@ def test_send_rate_limiter():
             break
 
 
-def test_cleartext_excempt_privacy(maildata, gencreds, handler):
+def test_excempt_privacy(maildata, gencreds, handler):
     from_addr = gencreds()[0]
     to_addr = "privacy@testrun.org"
     handler.config.passthrough_recipients = [to_addr]
@@ -134,73 +127,10 @@ def test_cleartext_excempt_privacy(maildata, gencreds, handler):
         rcpt_tos = [to_addr, false_to]
         content = msg.as_bytes()
 
-    assert "523" in handler.check_DATA(envelope=env2)
+    assert "500" in handler.check_DATA(envelope=env2)
 
 
-def test_cleartext_self_send_autocrypt_setup_message(maildata, gencreds, handler):
-    from_addr = gencreds()[0]
-    to_addr = from_addr
-
-    msg = maildata("asm.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    assert not handler.check_DATA(envelope=env)
-
-
-def test_cleartext_send_fails(maildata, gencreds, handler):
-    from_addr = gencreds()[0]
-    to_addr = gencreds()[0]
-
-    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    res = handler.check_DATA(envelope=env)
-    assert "523 Encryption Needed" in res
-
-
-def test_cleartext_incoming_fails(maildata, gencreds, inhandler):
-    from_addr = gencreds()[0]
-    to_addr, password = gencreds()
-
-    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    user = inhandler.config.get_user(to_addr)
-    user.set_password(password)
-    res = inhandler.check_DATA(envelope=env)
-    assert "523 Encryption Needed" in res
-
-    user.allow_incoming_cleartext()
-    assert not inhandler.check_DATA(envelope=env)
-
-
-def test_cleartext_incoming_mailer_daemon(maildata, gencreds, inhandler):
-    from_addr = "mailer-daemon@example.org"
-    to_addr = gencreds()[0]
-
-    msg = maildata("mailer-daemon.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    assert not inhandler.check_DATA(envelope=env)
-
-
-def test_cleartext_passthrough_domains(maildata, gencreds, handler):
+def test_passthrough_domains(maildata, gencreds, handler):
     from_addr = gencreds()[0]
     to_addr = "privacy@x.y.z"
     handler.config.passthrough_recipients = ["@x.y.z"]
@@ -221,10 +151,10 @@ def test_cleartext_passthrough_domains(maildata, gencreds, handler):
         rcpt_tos = [to_addr, false_to]
         content = msg.as_bytes()
 
-    assert "523" in handler.check_DATA(envelope=env2)
+    assert "500" in handler.check_DATA(envelope=env2)
 
 
-def test_cleartext_passthrough_senders(gencreds, handler, maildata):
+def test_passthrough_senders(gencreds, handler, maildata):
     acc1 = gencreds()[0]
     to_addr = "recipient@something.org"
     handler.config.passthrough_senders = [acc1]

chatmaild/src/chatmaild/tests/test_metrics.py

@@ -2,15 +2,8 @@ from chatmaild.metrics import main
 
 
 def test_main(tmp_path, capsys):
-    paths = []
     for x in ("ci-asllkj", "ac_12l3kj", "qweqwe", "ci-l1k2j31l2k3"):
-        p = tmp_path.joinpath(x)
-        p.mkdir()
-        p.joinpath("cur").mkdir()
-        paths.append(p)
-
-    tmp_path.joinpath("nomailbox").mkdir()
-
+        tmp_path.joinpath(x).mkdir()
     main(tmp_path)
     out, _ = capsys.readouterr()
     d = {}

chatmaild/src/chatmaild/tests/test_user.py

@@ -40,17 +40,3 @@ def test_no_mailboxes_dir(testaddr, example_config, tmp_path):
     user.set_password("someeqkjwelkqwjleqwe")
     user.set_last_login_timestamp(100000)
     assert user.get_last_login_timestamp() == 86400
-
-
-def test_set_get_cleartext_flag(testaddr, example_config, tmp_path):
-    p = tmp_path.joinpath("a", "mailboxes")
-    example_config.mailboxes_dir = p
-
-    user = example_config.get_user(testaddr)
-    user.set_password("someeqkjwelkqwjleqwe")
-    user.set_last_login_timestamp(100000)
-    assert user.get_last_login_timestamp() == 86400
-
-    assert not user.is_incoming_cleartext_ok()
-    user.allow_incoming_cleartext()
-    assert user.is_incoming_cleartext_ok()

chatmaild/src/chatmaild/user.py

@@ -13,7 +13,6 @@ class User:
         self.maildir = maildir
         self.addr = addr
         self.password_path = password_path
-        self.enforce_E2EE_path = maildir.joinpath("enforceE2EEincoming")
         self.uid = uid
         self.gid = gid
 
@@ -36,13 +35,6 @@ class User:
         home = str(self.maildir)
         return dict(addr=self.addr, home=home, uid=self.uid, gid=self.gid, password=pw)
 
-    def is_incoming_cleartext_ok(self):
-        return not self.enforce_E2EE_path.exists()
-
-    def allow_incoming_cleartext(self):
-        if self.enforce_E2EE_path.exists():
-            self.enforce_E2EE_path.unlink()
-
     def set_password(self, enc_password):
         """Set the specified password for this user.
 
@@ -58,7 +50,6 @@ class User:
             if not self.addr.startswith("echo@"):
                 logging.error(f"could not write password for: {self.addr}")
                 raise
-        self.enforce_E2EE_path.touch()
 
     def set_last_login_timestamp(self, timestamp):
         """Track login time with daily granularity

cmdeploy/src/cmdeploy/__init__.py

@@ -106,14 +106,12 @@ def _install_remote_venv_with_chatmaild(config) -> None:
     for fn in (
         "doveauth",
         "filtermail",
-        "filtermail-incoming",
         "echobot",
         "chatmail-metadata",
         "lastlogin",
     ):
-        execpath = fn if fn != "filtermail-incoming" else "filtermail"
         params = dict(
-            execpath=f"{remote_venv_dir}/bin/{execpath}",
+            execpath=f"{remote_venv_dir}/bin/{fn}",
             config_path=remote_chatmail_inipath,
             remote_venv_dir=remote_venv_dir,
             mail_domain=config.mail_domain,
@@ -230,7 +228,6 @@ def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
     )
     need_restart |= service_file.changed
 
-
     return need_restart
 
 
@@ -277,7 +274,18 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
     )
     need_restart |= master_config.changed
 
-    header_cleanup = files.put(
+    incoming_header_cleanup = files.put(
+        src=importlib.resources.files(__package__).joinpath(
+            "postfix/incoming_header_cleanup"
+        ),
+        dest="/etc/postfix/incoming_header_cleanup",
+        user="root",
+        group="root",
+        mode="644",
+    )
+    need_restart |= incoming_header_cleanup.changed
+
+    submission_header_cleanup = files.put(
         src=importlib.resources.files(__package__).joinpath(
             "postfix/submission_header_cleanup"
         ),
@@ -286,7 +294,7 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
         group="root",
         mode="644",
     )
-    need_restart |= header_cleanup.changed
+    need_restart |= submission_header_cleanup.changed
 
     # Login map that 1:1 maps email address to login.
     login_map = files.put(
@@ -543,6 +551,7 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
 
     server.group(name="Create vmail group", group="vmail", system=True)
     server.user(name="Create vmail user", user="vmail", group="vmail", system=True)
+    server.user(name="Create filtermail user", user="filtermail", system=True)
     server.group(name="Create opendkim group", group="opendkim", system=True)
     server.user(
         name="Create opendkim user",

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -117,7 +117,10 @@ http {
 
 	# Redirect www. to non-www
 	server {
-		listen 127.0.0.1:8443 ssl;
+		listen 8443 ssl;
+		{% if not disable_ipv6 %}
+		listen [::]:8443 ssl;
+		{% endif %}
 		server_name www.{{ config.domain_name }};
 		return 301 $scheme://{{ config.domain_name }}$request_uri;
 		access_log syslog:server=unix:/dev/log,facility=local7;

cmdeploy/src/cmdeploy/postfix/incoming_header_cleanup

@@ -0,0 +1 @@
+/^DKIM-Signature:/            IGNORE

cmdeploy/src/cmdeploy/postfix/master.cf.j2

@@ -14,7 +14,7 @@ smtp      inet  n       -       y       -       -       smtpd -v
 {%- else %}
 smtp      inet  n       -       y       -       -       smtpd 
 {%- endif %}
-  -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
+  -o smtpd_milters=unix:opendkim/opendkim.sock
 submission inet n       -       y       -       5000    smtpd
   -o syslog_name=postfix/submission
   -o smtpd_tls_security_level=encrypt
@@ -52,6 +52,7 @@ smtps     inet  n       -       y       -       5000    smtpd
 #628       inet  n       -       y       -       -       qmqpd
 pickup    unix  n       -       y       60      1       pickup
 cleanup   unix  n       -       y       -       0       cleanup
+  -o header_checks=regexp:/etc/postfix/incoming_header_cleanup
 qmgr      unix  n       -       n       300     1       qmgr
 #qmgr     unix  n       -       n       300     1       oqmgr
 tlsmgr    unix  -       -       y       1000?   1       tlsmgr
@@ -76,17 +77,12 @@ anvil     unix  -       -       y       -       1       anvil
 scache    unix  -       -       y       -       1       scache
 postlog   unix-dgram n  -       n       -       1       postlogd
 filter    unix -        n       n       -       -       lmtp
-# Local SMTP server for reinjecting outgoing filtered mail.
+# Local SMTP server for reinjecting filered mail.
 localhost:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
   -o syslog_name=postfix/reinject
   -o smtpd_milters=unix:opendkim/opendkim.sock
   -o cleanup_service_name=authclean
 
-# Local SMTP server for reinjecting incoming filtered mail 
-localhost:{{ config.postfix_reinject_port_incoming }} inet  n       -       n       -       10      smtpd
-  -o syslog_name=postfix/reinject_incoming
-  -o smtpd_milters=unix:opendkim/opendkim.sock
-
 # Cleanup `Received` headers for authenticated mail
 # to avoid leaking client IP.
 #

cmdeploy/src/cmdeploy/remote/rdns.py

@@ -11,49 +11,10 @@ All functions of this module
 """
 
 import re
-import os
-import glob
 
 from .rshell import CalledProcessError, shell
 
 
-def get_acme_account_url():
-    """Get the acmetool account URL with fallback methods.
-    
-    First tries the acmetool command, then falls back to searching the filesystem
-    if the command fails or returns empty.
-    """
-    # Try the acmetool command first
-    acme_url = shell("acmetool account-url", fail_ok=True)
-    if acme_url:
-        return acme_url
-    
-    # Fallback: search for URL files in acme accounts directory
-    try:
-        acct_base = "/var/lib/acme/accounts/"
-        # Find Let's Encrypt directory
-        le_dirs = glob.glob(os.path.join(acct_base, "*letsencrypt*"))
-        if not le_dirs:
-            return ""
-        
-        # Find account directories
-        for le_dir in le_dirs:
-            acct_dirs = glob.glob(os.path.join(le_dir, "*"))
-            for acct_dir in acct_dirs:
-                url_file = os.path.join(acct_dir, "url")
-                if os.path.isfile(url_file):
-                    # Read the URL file content
-                    with open(url_file, "r") as f:
-                        url = f.read().strip()
-                        if url:
-                            return url
-    except Exception:
-        # Any exception during fallback should be ignored
-        pass
-    
-    return ""
-
-
 def perform_initial_checks(mail_domain):
     """Collecting initial DNS settings."""
     assert mail_domain
@@ -65,7 +26,7 @@ def perform_initial_checks(mail_domain):
     WWW = query_dns("CNAME", f"www.{mail_domain}")
 
     res = dict(mail_domain=mail_domain, A=A, AAAA=AAAA, MTA_STS=MTA_STS, WWW=WWW)
-    res["acme_account_url"] = get_acme_account_url()
+    res["acme_account_url"] = shell("acmetool account-url", fail_ok=True)
     res["dkim_entry"], res["web_dkim_entry"] = get_dkim_entry(
         mail_domain, dkim_selector="opendkim"
     )

cmdeploy/src/cmdeploy/remote/rshell.py

@@ -14,22 +14,3 @@ def shell(command, fail_ok=False):
 def get_systemd_running():
     lines = shell("systemctl --type=service --state=running").split("\n")
     return [line for line in lines if line.startswith("  ")]
-
-
-def write_numbytes(path, num):
-    with open(path, "w") as f:
-        f.write("x" * num)
-
-
-def dovecot_recalc_quota(user):
-    shell(f"doveadm quota recalc -u {user}")
-    output = shell(f"doveadm quota get -u {user}")
-    #
-    # Quota name Type    Value  Limit                                              %
-    # User quota STORAGE     5 102400                                              0
-    # User quota MESSAGE     2      -                                              0
-    #
-    for line in output.split("\n"):
-        parts = line.split()
-        if parts[2] == "STORAGE":
-            return dict(value=int(parts[3]), limit=int(parts[4]), percent=int(parts[5]))

cmdeploy/src/cmdeploy/service/filtermail-incoming.service.f

@@ -1,12 +0,0 @@
-[Unit]
-Description=Incoming Chatmail Postfix before queue filter 
-
-[Service]
-ExecStart={execpath} {config_path} incoming
-Restart=always
-RestartSec=30
-User=vmail
-
-[Install]
-WantedBy=multi-user.target
-

cmdeploy/src/cmdeploy/service/filtermail.service.f

@@ -1,11 +1,11 @@
 [Unit]
-Description=Outgoing Chatmail Postfix before queue filter
+Description=Chatmail Postfix before queue filter
 
 [Service]
-ExecStart={execpath} {config_path} outgoing
+ExecStart={execpath} {config_path}
 Restart=always
 RestartSec=30
-User=vmail
+User=filtermail
 
 [Install]
 WantedBy=multi-user.target

cmdeploy/src/cmdeploy/tests/online/benchmark.py

@@ -37,7 +37,7 @@ class TestDC:
 
     def test_ping_pong(self, benchmark, cmfactory):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def dc_ping_pong():
             chat.send_text("ping")
@@ -49,7 +49,7 @@ class TestDC:
 
     def test_send_10_receive_10(self, benchmark, cmfactory, lp):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def dc_send_10_receive_10():
             for i in range(10):

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -116,8 +116,9 @@ def test_authenticated_from(cmsetup, maildata):
 
 @pytest.mark.parametrize("from_addr", ["fake@example.org", "fake@testrun.org"])
 def test_reject_missing_dkim(cmsetup, maildata, from_addr):
+    """Test that emails with missing or wrong DMARC, DKIM, and SPF entries are rejected."""
     recipient = cmsetup.gen_users(1)[0]
-    msg = maildata("encrypted.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
+    msg = maildata("plain.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
     with smtplib.SMTP(cmsetup.maildomain, 25) as s:
         with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):
             s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)

cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py

@@ -1,4 +1,5 @@
 import ipaddress
+import random
 import re
 import time
 
@@ -6,9 +7,6 @@ import imap_tools
 import pytest
 import requests
 
-from cmdeploy.remote import rshell
-from cmdeploy.sshexec import SSHExec
-
 
 @pytest.fixture
 def imap_mailbox(cmfactory):
@@ -56,23 +54,22 @@ class TestEndToEndDeltaChat:
         """Test that a DC account can send a message to a second DC account
         on the same chat-mail instance."""
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
+
+        lp.sec("ac1: prepare and send text message to ac2")
         chat.send_text("message0")
 
         lp.sec("wait for ac2 to receive message")
         msg2 = ac2._evtracker.wait_next_incoming_message()
         assert msg2.text == "message0"
 
-    def test_exceed_quota(
-        self, cmfactory, lp, tmpdir, remote, chatmail_config, sshdomain
-    ):
+    @pytest.mark.slow
+    def test_exceed_quota(self, cmfactory, lp, tmpdir, remote, chatmail_config):
         """This is a very slow test as it needs to upload >100MB of mail data
         before quota is exceeded, and thus depends on the speed of the upload.
         """
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
-
-        user = ac2.get_config("configured_addr")
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def parse_size_limit(limit: str) -> int:
             """Parse a size limit and return the number of bytes as integer.
@@ -85,27 +82,49 @@ class TestEndToEndDeltaChat:
             return int(float(number) * units[unit])
 
         quota = parse_size_limit(chatmail_config.max_mailbox_size)
+        attachsize = 1 * 1024 * 1024
+        num_to_send = quota // attachsize + 2
+        lp.sec(f"ac1: send {num_to_send} large files to ac2")
+        lp.indent(f"per-user quota is assumed to be: {quota / (1024 * 1024)}MB")
+        alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
+        msgs = []
+        for i in range(num_to_send):
+            attachment = tmpdir / f"attachment{i}"
+            data = "".join(random.choice(alphanumeric) for i in range(1024))
+            with open(attachment, "w+") as f:
+                for j in range(attachsize // len(data)):
+                    f.write(data)
 
-        lp.sec(f"filling remote inbox for {user}")
-        fn = f"7743102289.M843172P2484002.c20,S={quota},W=2398:2,"
-        path = chatmail_config.mailboxes_dir.joinpath(user, "cur", fn)
-        sshexec = SSHExec(sshdomain)
-        sshexec(call=rshell.write_numbytes, kwargs=dict(path=str(path), num=120))
-        res = sshexec(call=rshell.dovecot_recalc_quota, kwargs=dict(user=user))
-        assert res["percent"] >= 100
+            msg = chat.send_file(str(attachment))
+            msgs.append(msg)
+            lp.indent(f"Sent out msg {i}, size {attachsize / (1024 * 1024)}MB")
 
-        lp.sec("ac2: check quota is triggered")
+        lp.sec("ac2: check messages are arriving until quota is reached")
 
-        starting = True
+        addr = ac2.get_config("addr").lower()
+        saved_ok = 0
         for line in remote.iter_output("journalctl -n0 -f -u dovecot"):
-            if starting:
-                chat.send_text("hello")
-                starting = False
-            if user not in line:
+            if addr not in line:
                 # print(line)
                 continue
-            if "quota exceeded" in line:
-                return
+            if "quota" in line:
+                if "quota exceeded" in line:
+                    if saved_ok < num_to_send // 2:
+                        pytest.fail(
+                            f"quota exceeded too early: after {saved_ok} messages already"
+                        )
+                    lp.indent("good, message sending failed because quota was exceeded")
+                    return
+            if (
+                "stored mail into mailbox 'inbox'" in line
+                or "saved mail to inbox" in line
+            ):
+                saved_ok += 1
+                print(f"{saved_ok}: {line}")
+                if saved_ok >= num_to_send:
+                    break
+
+        pytest.fail("sending succeeded although messages should exceed quota")
 
     def test_securejoin(self, cmfactory, lp, maildomain2):
         ac1 = cmfactory.new_online_configuring_account(cache=False)
@@ -153,7 +172,7 @@ def test_hide_senders_ip_address(cmfactory):
     assert ipaddress.ip_address(public_ip)
 
     user1, user2 = cmfactory.get_online_accounts(2)
-    chat = cmfactory.get_protected_chat(user1, user2)
+    chat = cmfactory.get_accepted_chat(user1, user2)
 
     chat.send_text("testing submission header cleanup")
     user2._evtracker.wait_next_incoming_message()
@@ -162,18 +181,11 @@ def test_hide_senders_ip_address(cmfactory):
     assert public_ip not in msg.obj.as_string()
 
 
-def test_echobot(cmfactory, chatmail_config, lp, sshdomain):
+def test_echobot(cmfactory, chatmail_config, lp):
     ac = cmfactory.get_online_accounts(1)[0]
 
-    # establish contact with echobot
-    sshexec = SSHExec(sshdomain)
-    command = "cat /var/lib/echobot/invite-link.txt"
-    echo_invite_link = sshexec(call=rshell.shell, kwargs=dict(command=command))
-    chat = ac.qr_setup_contact(echo_invite_link)
-    ac._evtracker.wait_securejoin_joiner_progress(1000)
-
-    # send message and check it gets replied back
-    lp.sec("Send message to echobot")
+    lp.sec(f"Send message to echo@{chatmail_config.mail_domain}")
+    chat = ac.create_chat(f"echo@{chatmail_config.mail_domain}")
     text = "hi, I hope you text me back"
     chat.send_text(text)
     lp.sec("Wait for reply from echobot")

cmdeploy/src/cmdeploy/tests/plugin.py

@@ -62,7 +62,7 @@ def sshdomain(maildomain):
 def maildomain2():
     domain = os.environ.get("CHATMAIL_DOMAIN2")
     if not domain:
-        pytest.skip("set CHATMAIL_DOMAIN2 to a second chatmail server")
+        pytest.skip("set CHATMAIL_DOMAIN2 to a ssh-reachable chatmail instance")
     return domain
 
 
@@ -302,12 +302,10 @@ def cmfactory(request, gencreds, tmpdir, maildomain):
     pytest.importorskip("deltachat")
     from deltachat.testplugin import ACFactory
 
-    testproc = ChatmailTestProcess(request.config, maildomain, gencreds)
+    data = request.getfixturevalue("data")
 
-    class Data:
-        def read_path(self, path):
-            return
-    am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=Data())
+    testproc = ChatmailTestProcess(request.config, maildomain, gencreds)
+    am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=data)
 
     # nb. a bit hacky
     # would probably be better if deltachat's test machinery grows native support