doc: add changelog

add a hint that deletion may be earlier

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,32 @@
+---
+name: Bug report
+about: Report something that isn't working.
+title: ''
+assignees: ''
+
+---
+
+<!--
+Please fill out as much of this form as you can (leaving out stuff that is not applicable is ok).
+-->
+
+- Server OS (Operating System) - preferably Debian 12: 
+- On which OS you run cmdeploy:
+
+## Expected behavior
+
+*What did you try to achieve?*
+
+## Actual behavior
+
+*What happened instead?*
+
+### Steps to reproduce the problem:
+
+1. 
+2.
+
+### Screenshots
+
+### Logs
+

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,5 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Mutual Help Chat Group
+    url: https://i.delta.chat/#C2846EB4C1CB8DF84B1818F5E3A638FC3FBDC981&a=stalebot1%40nine.testrun.org&g=Chatmail%20Mutual%20Help&x=7sFF7Ik50pWv6J1z7RVC5527&i=d7s1HvOsk5UrSf9AoqRZggg4&s=XmX_9BAW6-g5Ao5E8PyaeKNB
+    about: If you have troubles setting up the relay server, feel free to ask here.

CHANGELOG.md

@@ -2,12 +2,31 @@
 
 ## untagged
 
-- Remove `DKIM-Signature` from incoming mails after verifying
-  ([#530](https://github.com/chatmail/server/pull/530))
+- Avoid "acmetool not found" during initial run
+  ([#550](https://github.com/chatmail/relay/pull/550))
+
+- Enforce end-to-end encryption for incoming messages. 
+  New user address mailboxes now get a `enforceE2EEincoming` file 
+  which prohibits incoming cleartext messages from other domains. 
+  An outside MTA trying to submit a cleartext message will 
+  get a "523 Encryption Needed" response, see RFC5248. 
+  If the file does not exist (as it the case for all existing accounts) 
+  incoming cleartext messages are accepted. 
+  ([#538](https://github.com/chatmail/server/pull/538))
+
+- Enforce end-to-end encryption between local addresses 
+  ([#535](https://github.com/chatmail/server/pull/535))
+
+- Limit the bind for the HTTPS server on 8443 to 127.0.0.1 
+  ([#522](https://github.com/chatmail/server/pull/522))
+  ([#532](https://github.com/chatmail/server/pull/532))
 
 - Send SNI when connecting to outside servers
   ([#524](https://github.com/chatmail/server/pull/524))
 
+- postfix master.cf: use 127.0.0.1 for consistency
+  ([#544](https://github.com/chatmail/relay/pull/544))
+
 - Pass through `original_content` instead of `content` in filtermail
   ([#509](https://github.com/chatmail/server/pull/509))
 

README.md

@@ -1,82 +1,96 @@
 
 <img width="800px" src="www/src/collage-top.png"/>
 
-# Chatmail servers for secure instant messaging 
+# Chatmail relays for end-to-end encrypted e-mail
 
-Chatmail servers are minimal interoperable e-mail routing machines designed for:
+Chatmail relay servers are interoperable Mail Transport Agents (MTAs) designed for: 
 
-- **Convenience:** Instant onboarding, with optional Google/Apple/Huawei push notifications
+- **Convenience:** Low friction instant onboarding
 
-- **Privacy:** Just login, no questions asked, no name, numbers or e-mail needed
+- **Privacy:** No name, phone numbers, email required or collected 
 
-- **Speed:** End-to-End Message delivery in well under a second
+- **End-to-End Encryption enforced**: only OpenPGP messages with metadata minimization allowed 
 
-- **Security:** Strict TLS, DKIM and OpenPGP with metadata-minimization enforced. 
+- **Instant:** Privacy-preserving Push Notifications for Apple, Google, and Huawei
 
-- **Relaxation:** No annoying spam-checking, IP reputation or rate limits
+- **Speed:** Message delivery in half a second, with optional P2P realtime connections 
 
-- **Efficiency:** messages are only stored for transit and removed automatically. 
+- **Transport Security:** Strict TLS and DKIM enforced 
 
-This repository contains everything needed to setup a ready-to-use chatmail server
+- **Reliability:** No spam or IP reputation checks; rate-limits are suitable for realtime chats
+
+- **Efficiency:** Messages are only stored for transit and removed automatically
+
+This repository contains everything needed to setup a ready-to-use chatmail relay
 comprised of a minimal setup of the battle-tested 
-[postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services. 
+[Postfix SMTP](https://www.postfix.org) and [Dovecot IMAP](https://www.dovecot.org) MTAs/MDAs.
 
-The automated setup is designed and optimized for providing chatmail addresses 
-for immediate permission-free onboarding through chat apps and bots. 
-Chatmail addresses are automatically created by a first login, 
-after which the initially specified password is required 
-for sending and receiving messages through them. 
+The automated setup is designed and optimized for providing chatmail addresses
+for immediate permission-free onboarding through chat apps and bots.
+Chatmail addresses are automatically created at first login,
+after which the initially specified password is required
+for sending and receiving messages through them.
 
-Please see [this list of known apps and client projects](https://chatmail.at/apps.html) which offer instant onboarding on chatmail servers,
-and [this list of known public 3rd party chatmail servers](https://delta.chat/en/chatmail). 
+Please see [this list of known apps and client projects](https://chatmail.at/clients.html) 
+and [this list of known public 3rd party chatmail relay servers](https://chatmail.at/relays).
 
 
 ## Minimal requirements, Prerequisites 
 
 You will need the following: 
 
-- control over a domain through a DNS provider of your choice, 
+- Control over a domain through a DNS provider of your choice.
 
-- a remote Debian 12 machine with IPV4 and preferably also IPV6 addresses and
-  reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports. 
-  Machine needs 1GB RAM, one slow CPU and maybe 10GB storage for a
-  few thousand active chatmail addresses,
+- A Debian 12 server with reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports.
+  IPv6 is encouraged if available.
+  Chatmail relay servers only require 1GB RAM, one CPU, and perhaps 10GB storage for a
+  few thousand active chatmail addresses.
 
-- a terminal window with password-less ssh root login to the remote machine;
-  you must have set up ssh authentication and need to use an ed25519 key, 
-  due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191);
-  you also need to add your private key to the local ssh-agent, 
-  because you can't type in your password during deployment.
+- Key-based SSH authentication to the root user.
+  You must add a passphrase-protected private key to your local ssh-agent
+  because you can't type in your passphrase during deployment.
+  (An ed25519 private key is required due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191))
 
 
 ## Getting started 
 
-We use `chat.example.org` as the chatmail domain in the following steps. 
+We use `chat.example.org` as the chatmail domain in the following steps.
 Please substitute it with your own domain. 
 
-1. Install the `cmdeploy` command in a virtualenv
+1. Setup the initial DNS records.
+   The following is an example in the familiar BIND zone file format with
+   a TTL of 1 hour (3600 seconds).
+   Please substitute your domain and IP addresses.
 
    ```
-    git clone https://github.com/chatmail/server
-    cd chatmail
+    chat.example.com. 3600 IN A 198.51.100.5
+    chat.example.com. 3600 IN AAAA 2001:db8::5
+    www.chat.example.com. 3600 IN CNAME chat.example.com.
+    mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
+   ```
+
+2. Clone the repository and bootstrap the Python virtualenv.
+
+   ```
+    git clone https://github.com/chatmail/relay
+    cd relay
     scripts/initenv.sh
    ```
 
-2. Create chatmail configuration file `chatmail.ini`:
+3. Create chatmail configuration file `chatmail.ini`:
 
    ```
     scripts/cmdeploy init chat.example.org  # <-- use your domain 
    ```
 
-3. Point your domain to the server's IP address,
-   if you haven't done so already.
-   Verify that SSH root login works:
+4. Verify that SSH root login works:
 
    ```
     ssh root@chat.example.org   # <-- use your domain 
    ```
 
-4. Deploy to the remote chatmail server:
+
+5. Deploy the remote chatmail relay server:
 
    ```
     scripts/cmdeploy run
@@ -116,25 +130,25 @@ scripts/cmdeploy bench
 
 This repository has four directories:
 
-- [cmdeploy](https://github.com/chatmail/server/tree/main/cmdeploy)
+- [cmdeploy](https://github.com/chatmail/relay/tree/main/cmdeploy)
   is a collection of configuration files
   and a [pyinfra](https://pyinfra.com)-based deployment script.
 
-- [chatmaild](https://github.com/chatmail/server/tree/main/chatmaild)
-  is a python package containing several small services
+- [chatmaild](https://github.com/chatmail/relay/tree/main/chatmaild)
+  is a Python package containing several small services
   which handle authentication,
   trigger push notifications on new messages,
   ensure that outbound mails are encrypted,
   delete inactive users,
   and some other minor things.
-  chatmaild can also be installed as a stand-alone python package.
+  chatmaild can also be installed as a stand-alone Python package.
 
-- [www](https://github.com/chatmail/server/tree/main/www)
+- [www](https://github.com/chatmail/relay/tree/main/www)
   contains the html, css, and markdown files
-  which make up a chatmail server's web page.
-  Edit them before deploying to make your chatmail server stand out.
+  which make up a chatmail relay's web page.
+  Edit them before deploying to make your chatmail relay stand out.
 
-- [scripts](https://github.com/chatmail/server/tree/main/scripts)
+- [scripts](https://github.com/chatmail/relay/tree/main/scripts)
   offers two convenience tools for beginners;
   `initenv.sh` installs the necessary dependencies to a local virtual environment,
   and the `scripts/cmdeploy` script enables you
@@ -146,79 +160,81 @@ The `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
 helps with setting up and managing the chatmail service.
 `cmdeploy init` creates the `chatmail.ini` config file.
 `cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [script](`cmdeploy/src/cmdeploy/__init__.py`)
-to automatically install or upgrade all chatmail components on a server,
+to automatically install or upgrade all chatmail components on a relay,
 according to the `chatmail.ini` config.
 
 The components of chatmail are:
 
-- [postfix smtp server](https://www.postfix.org) accepts sent messages (both from your users and from other servers)
+- [Postfix SMTP MTA](https://www.postfix.org) accepts and relays messages
+  (both from your users and from the wider e-mail MTA network)
 
-- [dovecot imap server](https://www.dovecot.org) stores messages for your users until they download them
+- [Dovecot IMAP MDA](https://www.dovecot.org) stores messages for your users until they download them
 
-- [nginx](https://nginx.org/) shows the web page with your privacy policy and additional information
+- [Nginx](https://nginx.org/) shows the web page with your privacy policy and additional information
 
-- [acmetool](https://hlandau.github.io/acmetool/) manages TLS certificates for dovecot, postfix, and nginx
+- [acmetool](https://hlandau.github.io/acmetool/) manages TLS certificates for Dovecot, Postfix, and Nginx
 
-- [opendkim](http://www.opendkim.org/) for signing messages with DKIM and rejecting inbound messages without DKIM
+- [OpenDKIM](http://www.opendkim.org/) for signing messages with DKIM and rejecting inbound messages without DKIM
 
 - [mtail](https://google.github.io/mtail/) for collecting anonymized metrics in case you have monitoring
 
+- [Iroh relay](https://www.iroh.computer/docs/concepts/relay) 
+  which helps client devices to establish Peer-to-Peer connections 
+
 - and the chatmaild services, explained in the next section:
 
 ### chatmaild
 
-chatmaild offers several commands
-which differentiate a *chatmail* server from a classic mail server.
-If you deploy them with cmdeploy,
-they are run by systemd services in the background.
-A short overview:
+`chatmaild` implements various systemd-controlled services  
+that integrate with Dovecot and Postfix to achieve instant-onboarding and 
+only relaying OpenPGP end-to-end messages encrypted messages. 
+A short overview of `chatmaild` services: 
 
-- [`doveauth`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/doveauth.py) implements
-  create-on-login account creation semantics and is used
-  by Dovecot during login authentication and by Postfix
+- [`doveauth`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/doveauth.py) 
+  implements create-on-login address semantics and is used 
+  by Dovecot during IMAP login and by Postfix during SMTP/SUBMISSION login
   which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
-  to authenticate users
-  to send mails for them.
+  to authenticate logins. 
 
-- [`filtermail`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/filtermail.py) prevents
-  unencrypted e-mail from leaving the chatmail service
-  and is integrated into postfix's outbound mail pipelines.
+- [`filtermail`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/filtermail.py) 
+  prevents unencrypted email from leaving or entering the chatmail service
+  and is integrated into Postfix's outbound and inbound mail pipelines.
 
-- [`chatmail-metadata`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
-  [dovecot lua script](https://github.com/chatmail/server/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
-  to store user-specific server-side config.
+- [`chatmail-metadata`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
+  [Dovecot lua script](https://github.com/chatmail/relay/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
+  to store user-specific relay-side config.
   On new messages,
-  it [passes the user's push notification token](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/notifier.py)
+  it [passes the user's push notification token](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/notifier.py)
   to [notifications.delta.chat](https://delta.chat/help#instant-delivery)
   so the push notifications on the user's phone can be triggered
-  by Apple/Google.
+  by Apple/Google/Huawei.
 
-- [`delete_inactive_users`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
+- [`delete_inactive_users`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
   deletes users if they have not logged in for a very long time.
   The timeframe can be configured in `chatmail.ini`.
 
-- [`lastlogin`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/lastlogin.py)
-  is contacted by dovecot when a user logs in
+- [`lastlogin`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/lastlogin.py)
+  is contacted by Dovecot when a user logs in
   and stores the date of the login.
 
-- [`echobot`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/echo.py)
+- [`echobot`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/echo.py)
   is a small bot for test purposes.
   It simply echoes back messages from users.
 
-- [`chatmail-metrics`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/metrics.py)
+- [`chatmail-metrics`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metrics.py)
   collects some metrics and displays them at `https://example.org/metrics`.
 
 ### Home page and getting started for users
 
-`cmdeploy run` also creates default static Web pages and deploys them
-to a nginx web server with: 
+`cmdeploy run` also creates default static web pages and deploys them
+to a Nginx web server with:
 
 - a default `index.html` along with a QR code that users can click to
-  create accounts on your chatmail provider,
+  create an address on your chatmail relay 
 
-- a default `info.html` that is linked from the home page,
+- a default `info.html` that is linked from the home page
 
-- a default `policy.html` that is linked from the home page.
+- a default `policy.html` that is linked from the home page
 
 All `.html` files are generated
 by the according markdown `.md` file in the `www/src` directory.
@@ -226,48 +242,64 @@ by the according markdown `.md` file in the `www/src` directory.
 
 ### Refining the web pages
 
-
 ```
 scripts/cmdeploy webdev
 ```
 
-This starts a local live development cycle for chatmail Web pages:
+This starts a local live development cycle for chatmail web pages:
 
 - uses the `www/src/page-layout.html` file for producing static
   HTML pages from `www/src/*.md` files
 
 - continously builds the web presence reading files from `www/src` directory
-  and generating html files and copying assets to the `www/build` directory.
+  and generating HTML files and copying assets to the `www/build` directory.
 
 - Starts a browser window automatically where you can "refresh" as needed.
 
+## Mailbox directory layout
 
-## Emergency Commands to disable automatic account creation
+Fresh chatmail addresses have a mailbox directory that contains: 
 
-If you need to stop account creation,
-e.g. because some script is wildly creating accounts,
-login to the server with ssh and run:
+- a `password` file with the salted password required for authenticating
+  whether a login may use the address to send/receive messages. 
+  If you modify the password file manually, you effectively block the user. 
+
+- `enforceE2EEincoming` is a default-created file with each address. 
+  If present the file indicates that this chatmail address rejects incoming cleartext messages.
+  If absent the address accepts incoming cleartext messages. 
+
+- `dovecot*`, `cur`, `new` and `tmp` represent IMAP/mailbox state. 
+  If the address is only used by one device, the Maildir directories
+  will typically be empty unless the user of that address hasn't been online 
+  for a while. 
+
+
+## Emergency Commands to disable automatic address creation
+
+If you need to stop address creation,
+e.g. because some script is wildly creating addresses, 
+login with ssh and run:
 
 ```
     touch /etc/chatmail-nocreate
 ```
 
-While this file is present, account creation will be blocked.
+Chatmail address creation will be denied while this file is present.
 
 ### Ports
 
-[Postfix](http://www.postfix.org/) listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
-[Dovecot](https://www.dovecot.org/) listens on ports 143 (imap) and 993 (imaps).
-[nginx](https://www.nginx.com/) listens on port 8443 (https-alt) and 443 (https).
+[Postfix](http://www.postfix.org/) listens on ports 25 (SMTP) and 587 (SUBMISSION) and 465 (SUBMISSIONS).
+[Dovecot](https://www.dovecot.org/) listens on ports 143 (IMAP) and 993 (IMAPS).
+[Nginx](https://www.nginx.com/) listens on port 8443 (HTTPS-ALT) and 443 (HTTPS).
 Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
-[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (http).
+[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (HTTP).
 
-Chatmail-core based apps will, however, discover all ports and configurations
-automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail server.
+chatmail-core based apps will, however, discover all ports and configurations
+automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail relay server.
 
 ## Email authentication
 
-chatmail servers rely on [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
+Chatmail relays enforce [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
 to authenticate incoming emails.
 Incoming emails must have a valid DKIM signature with
 Signing Domain Identifier (SDID, `d=` parameter in the DKIM-Signature header)
@@ -294,20 +326,20 @@ this is ensured by `filtermail` proxy.
 
 Postfix is configured to require valid TLS
 by setting [`smtp_tls_security_level`](https://www.postfix.org/postconf.5.html#smtp_tls_security_level) to `verify`.
-If emails don't arrive from a chatmail server to your server,
-the problem is likely that your server does not have a valid TLS certificate.
+If emails don't arrive at your chatmail relay server, 
+the problem is likely that your relay does not have a valid TLS certificate.
 
-You can test it by resolving `MX` records of your server domain
-and then connecting to MX servers (e.g `mx.example.org`) with
+You can test it by resolving `MX` records of your relay domain
+and then connecting to MX relays (e.g `mx.example.org`) with
 `openssl s_client -connect mx.example.org:25 -verify_hostname mx.example.org -verify_return_error -starttls smtp`
 from the host that has open port 25 to verify that certificate is valid.
 
-When providing a TLS certificate to your server,
+When providing a TLS certificate to your chatmail relay server, 
 make sure to provide the full certificate chain
 and not just the last certificate.
 
-If you are running Exim server and don't see incoming connections
-from a chatmail server in the logs,
+If you are running an Exim server and don't see incoming connections
+from a chatmail relay server in the logs,
 make sure `smtp_no_mail` log item is enabled in the config
 with `log_selector = +smtp_no_mail`.
 By default Exim does not log sessions that are closed
@@ -316,101 +348,104 @@ This happens if certificate is not recognized as valid by Postfix,
 so you might think that connection is not established
 while actually it is a problem with your TLS certificate.
 
-## Migrating chatmail server to a new host
+## Migrating a chatmail relay to a new host
 
-If you want to migrate chatmail from an old machine
+If you want to migrate chatmail relay from an old machine
 to a new machine,
 you can use these steps.
-They were tested with a linux laptop;
+They were tested with a Linux laptop;
 you might need to adjust some of the steps to your environment.
 
 Let's assume that your `mail_domain` is `mail.example.org`,
 all involved machines run Debian 12,
-your old server's IP address is `13.37.13.37`,
-and your new server's IP address is `13.12.23.42`.
+your old site's IP address is `13.37.13.37`,
+and your new site's IP address is `13.12.23.42`.
 
-During the guide, you might get a warning about changed SSH Host keys;
-in this case, just run `ssh-keygen -R "mail.example.org"` as recommended
-to make sure you can connect with SSH.
+Note, you should lower the TTLs of your DNS records to a value
+such as 300 (5 minutes) so the migration happens as smoothly as possible.
 
-1. First, copy `/var/lib/acme` to the new server with
-   `ssh root@13.37.13.37 tar c /var/lib/acme | ssh root@13.12.23.42 tar x -C /var/lib/`.
-   This transfers your TLS certificate.
+During the guide you might get a warning about changed SSH Host keys;
+in this case, just run `ssh-keygen -R "mail.example.org"` as recommended.
 
-2. You should also copy `/etc/dkimkeys` to the new server with
-   `ssh root@13.37.13.37 tar c /etc/dkimkeys | ssh root@13.12.23.42 tar x -C /etc/`
-   so the DKIM DNS record stays correct.
+1. First, disable mail services on the old site. 
 
-3. On the new server, run `chown root: -R /var/lib/acme` and `chown opendkim: -R /etc/dkimkeys` to make sure the permissions are correct.
-
-4. Run `cmdeploy run --disable-mail --ssh-host 13.12.23.42` to install chatmail on the new machine.
-   postfix and dovecot are disabled for now,
-   we will enable them later.
-
-5. Now, point DNS to the new IP addresses.
-
-   You can already remove the old IP addresses from DNS.
-   Existing Chatmail app users or bots will still be able to connect
-   to the old server, send and receive messages,
-   but new ones will fail to create new profiles
-   with your chatmail server.
-
-   If other servers try to deliver messages to your new server they will fail,
-   but normally email servers will retry delivering messages
-   for at least a week, so messages will not be lost.
-
-6. Now you can run `cmdeploy run --disable-mail --ssh-host 13.37.13.37` to disable your old server.
+   ```
+    cmdeploy run --disable-mail --ssh-host 13.37.13.37
+   ```
 
    Now your users will notice the migration
    and will not be able to send or receive messages
    until the migration is completed.
 
-7. After everything is stopped,
-    you can copy the `/home/vmail/mail` directory to the new server.
-    It includes all user data, messages, password hashes, etc.
+2. Now we want to copy `/home/vmail`, `/var/lib/acme`, `/etc/dkimkeys`, `/run/echobot`, and `/var/spool/postfix` to the new site. 
+   Login to the old site while forwarding your SSH agent 
+   so you can copy directly from the old to the new site with your SSH key:
+   ```
+    ssh -A root@13.37.13.37
+    tar c - /home/vmail/mail /var/lib/acme /etc/dkimkeys /run/echobot /var/spool/postfix | ssh root@13.12.23.42 "tar x -C /"
+   ```
 
-    Just run: `ssh root@13.37.13.37 tar c /home/vmail/mail | ssh root@13.12.23.42 tar x -C /home/vmail/`
+   This transfers all addresses, the TLS certificate, DKIM keys (so DKIM DNS record remains valid), and the echobot's password so it continues to function.
+   It also preserves the Postfix mail spool so any messages pending delivery will still be delivered.
 
-    After this, your new server has all the necessary files to start operating :)
+3. Install chatmail on the new machine:
 
-8. To be sure the permissions are still fine,
-    run `chown vmail: -R /home/vmail` on the new server.
+   ```
+    cmdeploy run --disable-mail --ssh-host 13.12.23.42
+   ```
+   Postfix and Dovecot are disabled for now; we will enable them later. 
+   We first need to make the new site fully operational. 
 
-9. Finally, you can run `cmdeploy run` to turn on chatmail on the new server.
-    Your users can continue using the chatmail server,
-    and messages which were sent after step 6. should arrive now.
-    Voilà!
+3. On the new site, run the following to ensure the ownership is correct in case UIDs/GIDs changed:
+
+   ```
+    chown root: -R /var/lib/acme
+    chown opendkim: -R /etc/dkimkeys
+    chown vmail: -R /home/vmail/mail
+    chown echobot: -R /run/echobot
+   ```
+
+4. Now, update DNS entries. 
+
+   If other MTAs try to deliver messages to your chatmail domain they may fail intermittently,
+   as DNS catches up with the new site settings 
+   but normally will retry delivering messages
+   for at least a week, so messages will not be lost.
+
+5. Finally, you can execute `cmdeploy run --ssh-host 13.12.23.42` to turn on chatmail on the new relay.
+   Your users will be able to use the chatmail relay as soon as the DNS changes have propagated.
+   Voilà!
 
 ## Setting up a reverse proxy
 
-A chatmail server does not depend on the client IP address
+A chatmail relay MTA does not track or depend on the client IP address
 for its operation, so it can be run behind a reverse proxy.
 This will not even affect incoming mail authentication
 as DKIM only checks the cryptographic signature
 of the message and does not use the IP address as the input.
 
-For example, you may want to self-host your chatmail server
+For example, you may want to self-host your chatmail relay
 and only use hosted VPS to provide a public IP address
 for client connections and incoming mail.
-You can connect chatmail server to VPS
+You can connect chatmail relay to VPS
 using a tunnel protocol
 such as [WireGuard](https://www.wireguard.com/)
 and setup a reverse proxy on a VPS
-to forward connections to the chatmail server
+to forward connections to the chatmail relay
 over the tunnel.
 You can also setup multiple reverse proxies
-for your chatmail server in different networks
-to ensure your server is reachable even when
+for your chatmail relay in different networks
+to ensure your relay is reachable even when
 one of the IPs becomes inaccessible due to
 hosting or routing problems.
 
-Note that your server still needs
+Note that your chatmail relay still needs
 to be able to make outgoing connections on port 25
 to send messages outside.
 
 To setup a reverse proxy
 (or rather Destination NAT, DNAT)
-for your chatmail server,
+for your chatmail relay,
 put the following configuration in `/etc/nftables.conf`:
 ```
 #!/usr/sbin/nft -f
@@ -422,7 +457,7 @@ define wan = eth0
 # Which ports to proxy.
 #
 # Note that SSH is not proxied
-# so it is possible to log into the proxy server
+# so it is possible to log into the proxy server 
 # and not the original one.
 define ports = { smtp, http, https, imap, imaps, submission, submissions }
 
@@ -485,7 +520,7 @@ table inet filter {
 ```
 
 Run `systemctl enable nftables.service`
-to ensure configuration is reloaded when the proxy server reboots.
+to ensure configuration is reloaded when the proxy relay reboots.
 
 Uncomment in `/etc/sysctl.conf` the following two lines:
 
@@ -494,7 +529,7 @@ net.ipv4.ip_forward=1
 net.ipv6.conf.all.forwarding=1
 ```
 
-Then reboot the server or do `sysctl -p` and `nft -f /etc/nftables.conf`.
+Then reboot the relay or do `sysctl -p` and `nft -f /etc/nftables.conf`.
 
-Once proxy server is set up,
+Once proxy relay is set up,
 you can add its IP address to the DNS.

chatmaild/src/chatmaild/config.py

@@ -11,7 +11,11 @@ def read_config(inipath):
     assert Path(inipath).exists(), inipath
     cfg = iniconfig.IniConfig(inipath)
     params = cfg.sections["params"]
-    return Config(inipath, params=params)
+    default_config_content = get_default_config_content(params["mail_domain"])
+    df_params = iniconfig.IniConfig("ini", data=default_config_content)["params"]
+    new_params = dict(df_params.items())
+    new_params.update(params)
+    return Config(inipath, params=new_params)
 
 
 class Config:
@@ -29,7 +33,13 @@ class Config:
         self.passthrough_senders = params["passthrough_senders"].split()
         self.passthrough_recipients = params["passthrough_recipients"].split()
         self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
+        self.filtermail_smtp_port_incoming = int(
+            params["filtermail_smtp_port_incoming"]
+        )
         self.postfix_reinject_port = int(params["postfix_reinject_port"])
+        self.postfix_reinject_port_incoming = int(
+            params["postfix_reinject_port_incoming"]
+        )
         self.mtail_address = params.get("mtail_address")
         self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
         self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
@@ -69,6 +79,11 @@ class Config:
 
 def write_initial_config(inipath, mail_domain, overrides):
     """Write out default config file, using the specified config value overrides."""
+    content = get_default_config_content(mail_domain, **overrides)
+    inipath.write_text(content)
+
+
+def get_default_config_content(mail_domain, **overrides):
     from importlib.resources import files
 
     inidir = files(__package__).joinpath("ini")
@@ -113,5 +128,4 @@ def write_initial_config(inipath, mail_domain, overrides):
             else:
                 lines.append(line)
         content = "\n".join(lines)
-
-    inipath.write_text(content)
+    return content

chatmaild/src/chatmaild/echo.py

@@ -8,6 +8,7 @@ import logging
 import os
 import subprocess
 import sys
+from pathlib import Path
 
 from deltachat_rpc_client import Bot, DeltaChat, EventType, Rpc, events
 
@@ -97,6 +98,10 @@ def main():
         if not bot.is_configured():
             bot.configure(addr, password)
 
+        # write invite link to working directory
+        invitelink = bot.account.get_qr_code()
+        Path("invite-link.txt").write_text(invitelink)
+
         bot.run_forever()
 
 

chatmaild/src/chatmaild/filtermail.py

@@ -11,9 +11,12 @@ from email.utils import parseaddr
 from smtplib import SMTP as SMTPClient
 
 from aiosmtpd.controller import Controller
+from aiosmtpd.smtp import SMTP
 
 from .config import read_config
 
+ENCRYPTION_NEEDED_523 = "523 Encryption Needed: Invalid Unencrypted Mail"
+
 
 def check_openpgp_payload(payload: bytes):
     """Checks the OpenPGP payload.
@@ -157,9 +160,14 @@ def check_encrypted(message):
     return True
 
 
-async def asyncmain_beforequeue(config):
-    port = config.filtermail_smtp_port
-    Controller(BeforeQueueHandler(config), hostname="127.0.0.1", port=port).start()
+async def asyncmain_beforequeue(config, mode):
+    if mode == "outgoing":
+        port = config.filtermail_smtp_port
+        handler = OutgoingBeforeQueueHandler(config)
+    else:
+        port = config.filtermail_smtp_port_incoming
+        handler = IncomingBeforeQueueHandler(config)
+    HackedController(handler, hostname="127.0.0.1", port=port).start()
 
 
 def recipient_matches_passthrough(recipient, passthrough_recipients):
@@ -171,7 +179,21 @@ def recipient_matches_passthrough(recipient, passthrough_recipients):
     return False
 
 
-class BeforeQueueHandler:
+class HackedController(Controller):
+    def factory(self):
+        return SMTPDiscardRCPTO_options(self.handler, **self.SMTP_kwargs)
+
+
+class SMTPDiscardRCPTO_options(SMTP):
+    def _getparams(self, params):
+        # aiosmtpd's SMTP daemon fails to handle a request if there are RCPT TO options
+        # We just ignore them for our incoming filtermail purposes
+        if len(params) == 1 and params[0].startswith("ORCPT"):
+            return {}
+        return super()._getparams(params)
+
+
+class OutgoingBeforeQueueHandler:
     def __init__(self, config):
         self.config = config
         self.send_rate_limiter = SendRateLimiter()
@@ -209,40 +231,86 @@ class BeforeQueueHandler:
         mail_encrypted = check_encrypted(message)
 
         _, from_addr = parseaddr(message.get("from").strip())
-        envelope_from_domain = from_addr.split("@").pop()
 
-        logging.info(f"mime-from: {from_addr} envelope-from: {envelope.mail_from!r}")
         if envelope.mail_from.lower() != from_addr.lower():
             return f"500 Invalid FROM <{from_addr!r}> for <{envelope.mail_from!r}>"
 
-        if mail_encrypted:
-            print("Filtering encrypted mail.", file=sys.stderr)
-        else:
-            print("Filtering unencrypted mail.", file=sys.stderr)
+        if mail_encrypted or is_securejoin(message):
+            print("Outgoing: Filtering encrypted mail.", file=sys.stderr)
+            return
+
+        print("Outgoing: Filtering unencrypted mail.", file=sys.stderr)
 
         if envelope.mail_from in self.config.passthrough_senders:
             return
 
+        # allow self-sent Autocrypt Setup Message
+        if envelope.rcpt_tos == [from_addr]:
+            if message.get("subject") == "Autocrypt Setup Message":
+                if message.get_content_type() == "multipart/mixed":
+                    return
+
         passthrough_recipients = self.config.passthrough_recipients
 
-        if mail_encrypted or is_securejoin(message):
-            return
-
         for recipient in envelope.rcpt_tos:
-            if envelope.mail_from == recipient:
-                # Always allow sending emails to self.
-                continue
             if recipient_matches_passthrough(recipient, passthrough_recipients):
                 continue
-            res = recipient.split("@")
-            if len(res) != 2:
-                return f"500 Invalid address <{recipient}>"
-            _recipient_addr, recipient_domain = res
 
-            is_outgoing = recipient_domain != envelope_from_domain
-            if is_outgoing:
-                print("Rejected unencrypted mail.", file=sys.stderr)
-                return f"500 Invalid unencrypted mail to <{recipient}>"
+            print("Rejected unencrypted mail.", file=sys.stderr)
+            return ENCRYPTION_NEEDED_523
+
+
+class IncomingBeforeQueueHandler:
+    def __init__(self, config):
+        self.config = config
+
+    async def handle_DATA(self, server, session, envelope):
+        logging.info("handle_DATA before-queue")
+        error = self.check_DATA(envelope)
+        if error:
+            return error
+        logging.info("re-injecting the mail that passed checks")
+
+        # the smtp daemon on reinject_port_incoming gives it to dkim milter
+        # which looks at source address to determine whether to verify or sign
+        client = SMTPClient(
+            "localhost",
+            self.config.postfix_reinject_port_incoming,
+            source_address=("127.0.0.2", 0),
+        )
+        client.sendmail(
+            envelope.mail_from, envelope.rcpt_tos, envelope.original_content
+        )
+        return "250 OK"
+
+    def check_DATA(self, envelope):
+        """the central filtering function for e-mails."""
+        logging.info(f"Processing DATA message from {envelope.mail_from}")
+
+        message = BytesParser(policy=policy.default).parsebytes(envelope.content)
+        mail_encrypted = check_encrypted(message)
+
+        if mail_encrypted or is_securejoin(message):
+            print("Incoming: Filtering encrypted mail.", file=sys.stderr)
+            return
+
+        print("Incoming: Filtering unencrypted mail.", file=sys.stderr)
+
+        # we want cleartext mailer-daemon messages to pass through
+        # chatmail core will typically not display them as normal messages
+        if message.get("auto-submitted"):
+            _, from_addr = parseaddr(message.get("from").strip())
+            if from_addr.lower().startswith("mailer-daemon@"):
+                if message.get_content_type() == "multipart/report":
+                    return
+
+        for recipient in envelope.rcpt_tos:
+            user = self.config.get_user(recipient)
+            if user is None or user.is_incoming_cleartext_ok():
+                continue
+
+            print("Rejected unencrypted mail.", file=sys.stderr)
+            return ENCRYPTION_NEEDED_523
 
 
 class SendRateLimiter:
@@ -261,11 +329,14 @@ class SendRateLimiter:
 
 def main():
     args = sys.argv[1:]
-    assert len(args) == 1
+    assert len(args) == 2
     config = read_config(args[0])
+    mode = args[1]
     logging.basicConfig(level=logging.WARN)
     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
-    task = asyncmain_beforequeue(config)
+    assert mode in ["incoming", "outgoing"]
+    task = asyncmain_beforequeue(config, mode)
     loop.create_task(task)
+    logging.info("entering serving loop")
     loop.run_forever()

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -46,12 +46,14 @@ passthrough_recipients = xstore@testrun.org
 # Deployment Details
 #
 
-# where the filtermail SMTP service listens
+# SMTP outgoing filtermail and reinjection 
 filtermail_smtp_port = 10080
-
-# postfix accepts on the localhost reinject SMTP port
 postfix_reinject_port = 10025
 
+# SMTP incoming filtermail and reinjection 
+filtermail_smtp_port_incoming = 10081
+postfix_reinject_port_incoming = 10026
+
 # if set to "True" IPv6 is disabled
 disable_ipv6 = False
 

chatmaild/src/chatmaild/metrics.py

@@ -11,6 +11,8 @@ def main(vmail_dir=None):
     ci_accounts = 0
 
     for path in Path(vmail_dir).iterdir():
+        if not path.joinpath("cur").is_dir():
+            continue
         accounts += 1
         if path.name[:3] in ("ci-", "ac_"):
             ci_accounts += 1

chatmaild/src/chatmaild/tests/mail-data/asm.eml

@@ -0,0 +1,56 @@
+From: {from_addr}
+To: {to_addr}
+Autocrypt-Setup-Message: v1
+Subject: Autocrypt Setup Message
+Date: Tue, 22 Jan 2019 12:56:29 +0100
+Content-type: multipart/mixed; boundary="Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ"
+
+--Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
+Content-Type: text/plain
+
+This message contains all information to transfer your Autocrypt
+settings along with your secret key securely from your original
+device.
+
+To set up your new device for Autocrypt, please follow the
+instuctions that should be presented by your new device.
+
+You can keep this message and use it as a backup for your secret
+key. If you want to do this, you should write down the Setup Code
+and store it securely.
+--Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
+Content-Type: application/autocrypt-setup
+Content-Disposition: attachment; filename="autocrypt-setup-message.html"
+
+<html><body>
+<p>
+This is the Autocrypt setup file used to transfer settings and
+keys between clients. You can decrypt it using the Setup Code
+presented on your old device, and then import the contained key
+into your keyring.
+</p>
+
+<pre>
+-----BEGIN PGP MESSAGE-----
+Passphrase-Format: numeric9x4
+Passphrase-Begin: 17
+
+jA0EBwMCFAxADoCdzeX/0ukBlqI5+pfpKb751qd/7nLNbkpy3gVcaf1QwRPZYt40
+Ynp08UqRQ2g48ZlnzHLSwlTGOPTuv2Jt8ka+pgZ45xzvJSG2gau03xP4VsC271kR
+VmCjdb0Y6Rk96mAwfGzrkbaRQ9Z7fIoL866GOv6h9neiVIkp+JYlTV6ISD0ZQJ4Q
+I6dOQkB/TWZyVjtiJDOQHdfNWliA6NtqaLq19wlu9L5xXjuNpY95KwR8EJXWe0+o
+Y3d2U/KxOAkXKghP2Qg1GtlPVeGC5T4p03TGI6pzKT+kHX6Rrm9wK6sM9aTquMmF
+Vok84Jg1DFnwivWC2RILR81rXi7k/+Y6MUbveFgJ9cQduqpxnmD7TjOblYu7M6zp
+YGAUxh8DRKlIMn2QsA++DBYQ6ACZvwuY8qTDLkqPDo4WqM313dsMJbyGjDdVE7EM
+PESS+RlABETpZXz8g/ycr6DIUNdlbPcmYlsBfHWDOuR2GFFTwmlv5slWS39dJv38
+E0eIe1CwdxI801Se7t7dUUS/ZF8wb6GlmxOcqGbF8eko1Z0S64IAm7/h13MRQCxI
+geQnHfGYVJ2FOimoCMEKwfa9x++RFTDW0u7spDC2uWvK/1viV8OfRppFhLr/kmKb
+18lWXuAz80DAjUDUsVqEq2MvJBJGoCJUEyjuRsLkHYRM5jYk4v50LyyR0Om73nWF
+nZBqmqNzdr7Xb9PHHdFhnEc0VvoYbrcM0RVYcEMW3YbmejM891j1d6Iv+/n/qND/
+NdebGrfWJMmFLf/iEkzTZ3/v5inW9LpWoRc94ioCjJTaEo8Rib6ARRFaJVIsmNXi
+YicFGO98D+zX+a2t9Yz6IpPajVslnOp6ScpmXgts/2XWD7oE+JgxSAqo/dLVsHgP
+Ufo=
+=pulM
+-----END PGP MESSAGE-----
+</pre></body></html>
+--Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ--

chatmaild/src/chatmaild/tests/mail-data/mailer-daemon.eml

@@ -0,0 +1,46 @@
+Date: Fri, 8 Jul 1994 09:21:47 -0400
+From: Mail Delivery Subsystem <MAILER-DAEMON@example.org>
+Subject: Returned mail: User unknown
+To: <owner-ups-mib@CS.UTK.EDU>
+Auto-Submitted: auto-replied 
+MIME-Version: 1.0
+Content-Type: multipart/report; report-type=delivery-status;
+      boundary="JAA13167.773673707/CS.UTK.EDU"
+
+--JAA13167.773673707/CS.UTK.EDU
+content-type: text/plain; charset=us-ascii
+
+   ----- The following addresses had delivery problems -----
+<arathib@vnet.ibm.com>  (unrecoverable error)
+<wsnell@sdcc13.ucsd.edu>  (unrecoverable error)
+
+--JAA13167.773673707/CS.UTK.EDU
+content-type: message/delivery-status
+
+Reporting-MTA: dns; cs.utk.edu
+
+Original-Recipient: rfc822;arathib@vnet.ibm.com
+Final-Recipient: rfc822;arathib@vnet.ibm.com
+Action: failed
+Status: 5.0.0 (permanent failure)
+Diagnostic-Code: smtp;
+ 550 'arathib@vnet.IBM.COM' is not a registered gateway user
+Remote-MTA: dns; vnet.ibm.com
+
+Original-Recipient: rfc822;johnh@hpnjld.njd.hp.com
+Final-Recipient: rfc822;johnh@hpnjld.njd.hp.com
+Action: delayed
+Status: 4.0.0 (hpnjld.njd.jp.com: host name lookup failure)
+
+Original-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
+Final-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
+Action: failed
+Status: 5.0.0
+Diagnostic-Code: smtp; 550 user unknown
+Remote-MTA: dns; sdcc13.ucsd.edu
+
+--JAA13167.773673707/CS.UTK.EDU
+content-type: message/rfc822
+
+[original message goes here]
+--JAA13167.773673707/CS.UTK.EDU--

chatmaild/src/chatmaild/tests/plugin.py

@@ -72,9 +72,8 @@ def maildata(request):
     def maildata(name, from_addr, to_addr, subject="[...]"):
         # Using `.read_bytes().decode()` instead of `.read_text()` to preserve newlines.
         data = datadir.joinpath(name).read_bytes().decode()
-
         text = data.format(from_addr=from_addr, to_addr=to_addr, subject=subject)
-        return BytesParser(policy=policy.default).parsebytes(text.encode())
+        return BytesParser(policy=policy.SMTP).parsebytes(text.encode())
 
     return maildata
 

chatmaild/src/chatmaild/tests/test_config.py

@@ -15,6 +15,14 @@ def test_read_config_basic(example_config):
     assert example_config.mail_domain == "chat.example.org"
 
 
+def test_read_config_basic_using_defaults(tmp_path, maildomain):
+    inipath = tmp_path.joinpath("chatmail.ini")
+    inipath.write_text(f"[params]\nmail_domain = {maildomain}")
+    example_config = read_config(inipath)
+    assert example_config.max_user_send_per_minute == 60
+    assert example_config.filtermail_smtp_port_incoming == 10081
+
+
 def test_read_config_testrun(make_config):
     config = make_config("something.testrun.org")
     assert config.mail_domain == "something.testrun.org"

chatmaild/src/chatmaild/tests/test_filtermail.py

@@ -1,7 +1,8 @@
 import pytest
 
 from chatmaild.filtermail import (
-    BeforeQueueHandler,
+    IncomingBeforeQueueHandler,
+    OutgoingBeforeQueueHandler,
     SendRateLimiter,
     check_armored_payload,
     check_encrypted,
@@ -18,7 +19,13 @@ def maildomain():
 @pytest.fixture
 def handler(make_config, maildomain):
     config = make_config(maildomain)
-    return BeforeQueueHandler(config)
+    return OutgoingBeforeQueueHandler(config)
+
+
+@pytest.fixture
+def inhandler(make_config, maildomain):
+    config = make_config(maildomain)
+    return IncomingBeforeQueueHandler(config)
 
 
 def test_reject_forged_from(maildata, gencreds, handler):
@@ -29,14 +36,14 @@ def test_reject_forged_from(maildata, gencreds, handler):
     # test that the filter lets good mail through
     to_addr = gencreds()[0]
     env.content = maildata(
-        "plain.eml", from_addr=env.mail_from, to_addr=to_addr
+        "encrypted.eml", from_addr=env.mail_from, to_addr=to_addr
     ).as_bytes()
 
     assert not handler.check_DATA(envelope=env)
 
     # test that the filter rejects forged mail
     env.content = maildata(
-        "plain.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
+        "encrypted.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
     ).as_bytes()
     error = handler.check_DATA(envelope=env)
     assert "500" in error
@@ -106,7 +113,7 @@ def test_send_rate_limiter():
             break
 
 
-def test_excempt_privacy(maildata, gencreds, handler):
+def test_cleartext_excempt_privacy(maildata, gencreds, handler):
     from_addr = gencreds()[0]
     to_addr = "privacy@testrun.org"
     handler.config.passthrough_recipients = [to_addr]
@@ -127,10 +134,73 @@ def test_excempt_privacy(maildata, gencreds, handler):
         rcpt_tos = [to_addr, false_to]
         content = msg.as_bytes()
 
-    assert "500" in handler.check_DATA(envelope=env2)
+    assert "523" in handler.check_DATA(envelope=env2)
 
 
-def test_passthrough_domains(maildata, gencreds, handler):
+def test_cleartext_self_send_autocrypt_setup_message(maildata, gencreds, handler):
+    from_addr = gencreds()[0]
+    to_addr = from_addr
+
+    msg = maildata("asm.eml", from_addr=from_addr, to_addr=to_addr)
+
+    class env:
+        mail_from = from_addr
+        rcpt_tos = [to_addr]
+        content = msg.as_bytes()
+
+    assert not handler.check_DATA(envelope=env)
+
+
+def test_cleartext_send_fails(maildata, gencreds, handler):
+    from_addr = gencreds()[0]
+    to_addr = gencreds()[0]
+
+    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
+
+    class env:
+        mail_from = from_addr
+        rcpt_tos = [to_addr]
+        content = msg.as_bytes()
+
+    res = handler.check_DATA(envelope=env)
+    assert "523 Encryption Needed" in res
+
+
+def test_cleartext_incoming_fails(maildata, gencreds, inhandler):
+    from_addr = gencreds()[0]
+    to_addr, password = gencreds()
+
+    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
+
+    class env:
+        mail_from = from_addr
+        rcpt_tos = [to_addr]
+        content = msg.as_bytes()
+
+    user = inhandler.config.get_user(to_addr)
+    user.set_password(password)
+    res = inhandler.check_DATA(envelope=env)
+    assert "523 Encryption Needed" in res
+
+    user.allow_incoming_cleartext()
+    assert not inhandler.check_DATA(envelope=env)
+
+
+def test_cleartext_incoming_mailer_daemon(maildata, gencreds, inhandler):
+    from_addr = "mailer-daemon@example.org"
+    to_addr = gencreds()[0]
+
+    msg = maildata("mailer-daemon.eml", from_addr=from_addr, to_addr=to_addr)
+
+    class env:
+        mail_from = from_addr
+        rcpt_tos = [to_addr]
+        content = msg.as_bytes()
+
+    assert not inhandler.check_DATA(envelope=env)
+
+
+def test_cleartext_passthrough_domains(maildata, gencreds, handler):
     from_addr = gencreds()[0]
     to_addr = "privacy@x.y.z"
     handler.config.passthrough_recipients = ["@x.y.z"]
@@ -151,10 +221,10 @@ def test_passthrough_domains(maildata, gencreds, handler):
         rcpt_tos = [to_addr, false_to]
         content = msg.as_bytes()
 
-    assert "500" in handler.check_DATA(envelope=env2)
+    assert "523" in handler.check_DATA(envelope=env2)
 
 
-def test_passthrough_senders(gencreds, handler, maildata):
+def test_cleartext_passthrough_senders(gencreds, handler, maildata):
     acc1 = gencreds()[0]
     to_addr = "recipient@something.org"
     handler.config.passthrough_senders = [acc1]

chatmaild/src/chatmaild/tests/test_metrics.py

@@ -2,8 +2,15 @@ from chatmaild.metrics import main
 
 
 def test_main(tmp_path, capsys):
+    paths = []
     for x in ("ci-asllkj", "ac_12l3kj", "qweqwe", "ci-l1k2j31l2k3"):
-        tmp_path.joinpath(x).mkdir()
+        p = tmp_path.joinpath(x)
+        p.mkdir()
+        p.joinpath("cur").mkdir()
+        paths.append(p)
+
+    tmp_path.joinpath("nomailbox").mkdir()
+
     main(tmp_path)
     out, _ = capsys.readouterr()
     d = {}

chatmaild/src/chatmaild/tests/test_user.py

@@ -40,3 +40,17 @@ def test_no_mailboxes_dir(testaddr, example_config, tmp_path):
     user.set_password("someeqkjwelkqwjleqwe")
     user.set_last_login_timestamp(100000)
     assert user.get_last_login_timestamp() == 86400
+
+
+def test_set_get_cleartext_flag(testaddr, example_config, tmp_path):
+    p = tmp_path.joinpath("a", "mailboxes")
+    example_config.mailboxes_dir = p
+
+    user = example_config.get_user(testaddr)
+    user.set_password("someeqkjwelkqwjleqwe")
+    user.set_last_login_timestamp(100000)
+    assert user.get_last_login_timestamp() == 86400
+
+    assert not user.is_incoming_cleartext_ok()
+    user.allow_incoming_cleartext()
+    assert user.is_incoming_cleartext_ok()

chatmaild/src/chatmaild/user.py

@@ -13,6 +13,7 @@ class User:
         self.maildir = maildir
         self.addr = addr
         self.password_path = password_path
+        self.enforce_E2EE_path = maildir.joinpath("enforceE2EEincoming")
         self.uid = uid
         self.gid = gid
 
@@ -35,6 +36,13 @@ class User:
         home = str(self.maildir)
         return dict(addr=self.addr, home=home, uid=self.uid, gid=self.gid, password=pw)
 
+    def is_incoming_cleartext_ok(self):
+        return not self.enforce_E2EE_path.exists()
+
+    def allow_incoming_cleartext(self):
+        if self.enforce_E2EE_path.exists():
+            self.enforce_E2EE_path.unlink()
+
     def set_password(self, enc_password):
         """Set the specified password for this user.
 
@@ -50,6 +58,7 @@ class User:
             if not self.addr.startswith("echo@"):
                 logging.error(f"could not write password for: {self.addr}")
                 raise
+        self.enforce_E2EE_path.touch()
 
     def set_last_login_timestamp(self, timestamp):
         """Track login time with daily granularity

cmdeploy/src/cmdeploy/__init__.py

@@ -106,12 +106,14 @@ def _install_remote_venv_with_chatmaild(config) -> None:
     for fn in (
         "doveauth",
         "filtermail",
+        "filtermail-incoming",
         "echobot",
         "chatmail-metadata",
         "lastlogin",
     ):
+        execpath = fn if fn != "filtermail-incoming" else "filtermail"
         params = dict(
-            execpath=f"{remote_venv_dir}/bin/{fn}",
+            execpath=f"{remote_venv_dir}/bin/{execpath}",
             config_path=remote_chatmail_inipath,
             remote_venv_dir=remote_venv_dir,
             mail_domain=config.mail_domain,
@@ -228,6 +230,7 @@ def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
     )
     need_restart |= service_file.changed
 
+
     return need_restart
 
 
@@ -274,18 +277,7 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
     )
     need_restart |= master_config.changed
 
-    incoming_header_cleanup = files.put(
-        src=importlib.resources.files(__package__).joinpath(
-            "postfix/incoming_header_cleanup"
-        ),
-        dest="/etc/postfix/incoming_header_cleanup",
-        user="root",
-        group="root",
-        mode="644",
-    )
-    need_restart |= incoming_header_cleanup.changed
-
-    submission_header_cleanup = files.put(
+    header_cleanup = files.put(
         src=importlib.resources.files(__package__).joinpath(
             "postfix/submission_header_cleanup"
         ),
@@ -294,7 +286,7 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
         group="root",
         mode="644",
     )
-    need_restart |= submission_header_cleanup.changed
+    need_restart |= header_cleanup.changed
 
     # Login map that 1:1 maps email address to login.
     login_map = files.put(
@@ -551,7 +543,6 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
 
     server.group(name="Create vmail group", group="vmail", system=True)
     server.user(name="Create vmail user", user="vmail", group="vmail", system=True)
-    server.user(name="Create filtermail user", user="filtermail", system=True)
     server.group(name="Create opendkim group", group="opendkim", system=True)
     server.user(
         name="Create opendkim user",

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -117,10 +117,7 @@ http {
 
 	# Redirect www. to non-www
 	server {
-		listen 8443 ssl;
-		{% if not disable_ipv6 %}
-		listen [::]:8443 ssl;
-		{% endif %}
+		listen 127.0.0.1:8443 ssl;
 		server_name www.{{ config.domain_name }};
 		return 301 $scheme://{{ config.domain_name }}$request_uri;
 		access_log syslog:server=unix:/dev/log,facility=local7;

cmdeploy/src/cmdeploy/postfix/incoming_header_cleanup

@@ -1 +0,0 @@
-/^DKIM-Signature:/            IGNORE

cmdeploy/src/cmdeploy/postfix/master.cf.j2

@@ -14,7 +14,7 @@ smtp      inet  n       -       y       -       -       smtpd -v
 {%- else %}
 smtp      inet  n       -       y       -       -       smtpd 
 {%- endif %}
-  -o smtpd_milters=unix:opendkim/opendkim.sock
+  -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
 submission inet n       -       y       -       5000    smtpd
   -o syslog_name=postfix/submission
   -o smtpd_tls_security_level=encrypt
@@ -52,7 +52,6 @@ smtps     inet  n       -       y       -       5000    smtpd
 #628       inet  n       -       y       -       -       qmqpd
 pickup    unix  n       -       y       60      1       pickup
 cleanup   unix  n       -       y       -       0       cleanup
-  -o header_checks=regexp:/etc/postfix/incoming_header_cleanup
 qmgr      unix  n       -       n       300     1       qmgr
 #qmgr     unix  n       -       n       300     1       oqmgr
 tlsmgr    unix  -       -       y       1000?   1       tlsmgr
@@ -77,12 +76,17 @@ anvil     unix  -       -       y       -       1       anvil
 scache    unix  -       -       y       -       1       scache
 postlog   unix-dgram n  -       n       -       1       postlogd
 filter    unix -        n       n       -       -       lmtp
-# Local SMTP server for reinjecting filered mail.
-localhost:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
+# Local SMTP server for reinjecting outgoing filtered mail.
+127.0.0.1:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
   -o syslog_name=postfix/reinject
   -o smtpd_milters=unix:opendkim/opendkim.sock
   -o cleanup_service_name=authclean
 
+# Local SMTP server for reinjecting incoming filtered mail 
+127.0.0.1:{{ config.postfix_reinject_port_incoming }} inet  n       -       n       -       10      smtpd
+  -o syslog_name=postfix/reinject_incoming
+  -o smtpd_milters=unix:opendkim/opendkim.sock
+
 # Cleanup `Received` headers for authenticated mail
 # to avoid leaking client IP.
 #

cmdeploy/src/cmdeploy/remote/rshell.py

@@ -1,10 +1,13 @@
-from subprocess import CalledProcessError, check_output
+from subprocess import DEVNULL, CalledProcessError, check_output
 
 
 def shell(command, fail_ok=False):
     print(f"$ {command}")
+    args = dict(shell=True)
+    if fail_ok:
+        args["stderr"] = DEVNULL
     try:
-        return check_output(command, shell=True).decode().rstrip()
+        return check_output(command, **args).decode().rstrip()
     except CalledProcessError:
         if not fail_ok:
             raise
@@ -14,3 +17,22 @@ def shell(command, fail_ok=False):
 def get_systemd_running():
     lines = shell("systemctl --type=service --state=running").split("\n")
     return [line for line in lines if line.startswith("  ")]
+
+
+def write_numbytes(path, num):
+    with open(path, "w") as f:
+        f.write("x" * num)
+
+
+def dovecot_recalc_quota(user):
+    shell(f"doveadm quota recalc -u {user}")
+    output = shell(f"doveadm quota get -u {user}")
+    #
+    # Quota name Type    Value  Limit                                              %
+    # User quota STORAGE     5 102400                                              0
+    # User quota MESSAGE     2      -                                              0
+    #
+    for line in output.split("\n"):
+        parts = line.split()
+        if parts[2] == "STORAGE":
+            return dict(value=int(parts[3]), limit=int(parts[4]), percent=int(parts[5]))

cmdeploy/src/cmdeploy/service/filtermail-incoming.service.f

@@ -0,0 +1,12 @@
+[Unit]
+Description=Incoming Chatmail Postfix before queue filter 
+
+[Service]
+ExecStart={execpath} {config_path} incoming
+Restart=always
+RestartSec=30
+User=vmail
+
+[Install]
+WantedBy=multi-user.target
+

cmdeploy/src/cmdeploy/service/filtermail.service.f

@@ -1,11 +1,11 @@
 [Unit]
-Description=Chatmail Postfix before queue filter
+Description=Outgoing Chatmail Postfix before queue filter
 
 [Service]
-ExecStart={execpath} {config_path}
+ExecStart={execpath} {config_path} outgoing
 Restart=always
 RestartSec=30
-User=filtermail
+User=vmail
 
 [Install]
 WantedBy=multi-user.target

cmdeploy/src/cmdeploy/tests/online/benchmark.py

@@ -37,7 +37,7 @@ class TestDC:
 
     def test_ping_pong(self, benchmark, cmfactory):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_accepted_chat(ac1, ac2)
+        chat = cmfactory.get_protected_chat(ac1, ac2)
 
         def dc_ping_pong():
             chat.send_text("ping")
@@ -49,7 +49,7 @@ class TestDC:
 
     def test_send_10_receive_10(self, benchmark, cmfactory, lp):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_accepted_chat(ac1, ac2)
+        chat = cmfactory.get_protected_chat(ac1, ac2)
 
         def dc_send_10_receive_10():
             for i in range(10):

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -116,9 +116,8 @@ def test_authenticated_from(cmsetup, maildata):
 
 @pytest.mark.parametrize("from_addr", ["fake@example.org", "fake@testrun.org"])
 def test_reject_missing_dkim(cmsetup, maildata, from_addr):
-    """Test that emails with missing or wrong DMARC, DKIM, and SPF entries are rejected."""
     recipient = cmsetup.gen_users(1)[0]
-    msg = maildata("plain.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
+    msg = maildata("encrypted.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
     with smtplib.SMTP(cmsetup.maildomain, 25) as s:
         with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):
             s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)

cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py

@@ -1,5 +1,4 @@
 import ipaddress
-import random
 import re
 import time
 
@@ -7,6 +6,9 @@ import imap_tools
 import pytest
 import requests
 
+from cmdeploy.remote import rshell
+from cmdeploy.sshexec import SSHExec
+
 
 @pytest.fixture
 def imap_mailbox(cmfactory):
@@ -54,22 +56,23 @@ class TestEndToEndDeltaChat:
         """Test that a DC account can send a message to a second DC account
         on the same chat-mail instance."""
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_accepted_chat(ac1, ac2)
-
-        lp.sec("ac1: prepare and send text message to ac2")
+        chat = cmfactory.get_protected_chat(ac1, ac2)
         chat.send_text("message0")
 
         lp.sec("wait for ac2 to receive message")
         msg2 = ac2._evtracker.wait_next_incoming_message()
         assert msg2.text == "message0"
 
-    @pytest.mark.slow
-    def test_exceed_quota(self, cmfactory, lp, tmpdir, remote, chatmail_config):
+    def test_exceed_quota(
+        self, cmfactory, lp, tmpdir, remote, chatmail_config, sshdomain
+    ):
         """This is a very slow test as it needs to upload >100MB of mail data
         before quota is exceeded, and thus depends on the speed of the upload.
         """
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_accepted_chat(ac1, ac2)
+        chat = cmfactory.get_protected_chat(ac1, ac2)
+
+        user = ac2.get_config("configured_addr")
 
         def parse_size_limit(limit: str) -> int:
             """Parse a size limit and return the number of bytes as integer.
@@ -82,49 +85,27 @@ class TestEndToEndDeltaChat:
             return int(float(number) * units[unit])
 
         quota = parse_size_limit(chatmail_config.max_mailbox_size)
-        attachsize = 1 * 1024 * 1024
-        num_to_send = quota // attachsize + 2
-        lp.sec(f"ac1: send {num_to_send} large files to ac2")
-        lp.indent(f"per-user quota is assumed to be: {quota / (1024 * 1024)}MB")
-        alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
-        msgs = []
-        for i in range(num_to_send):
-            attachment = tmpdir / f"attachment{i}"
-            data = "".join(random.choice(alphanumeric) for i in range(1024))
-            with open(attachment, "w+") as f:
-                for j in range(attachsize // len(data)):
-                    f.write(data)
 
-            msg = chat.send_file(str(attachment))
-            msgs.append(msg)
-            lp.indent(f"Sent out msg {i}, size {attachsize / (1024 * 1024)}MB")
+        lp.sec(f"filling remote inbox for {user}")
+        fn = f"7743102289.M843172P2484002.c20,S={quota},W=2398:2,"
+        path = chatmail_config.mailboxes_dir.joinpath(user, "cur", fn)
+        sshexec = SSHExec(sshdomain)
+        sshexec(call=rshell.write_numbytes, kwargs=dict(path=str(path), num=120))
+        res = sshexec(call=rshell.dovecot_recalc_quota, kwargs=dict(user=user))
+        assert res["percent"] >= 100
 
-        lp.sec("ac2: check messages are arriving until quota is reached")
+        lp.sec("ac2: check quota is triggered")
 
-        addr = ac2.get_config("addr").lower()
-        saved_ok = 0
+        starting = True
         for line in remote.iter_output("journalctl -n0 -f -u dovecot"):
-            if addr not in line:
+            if starting:
+                chat.send_text("hello")
+                starting = False
+            if user not in line:
                 # print(line)
                 continue
-            if "quota" in line:
-                if "quota exceeded" in line:
-                    if saved_ok < num_to_send // 2:
-                        pytest.fail(
-                            f"quota exceeded too early: after {saved_ok} messages already"
-                        )
-                    lp.indent("good, message sending failed because quota was exceeded")
-                    return
-            if (
-                "stored mail into mailbox 'inbox'" in line
-                or "saved mail to inbox" in line
-            ):
-                saved_ok += 1
-                print(f"{saved_ok}: {line}")
-                if saved_ok >= num_to_send:
-                    break
-
-        pytest.fail("sending succeeded although messages should exceed quota")
+            if "quota exceeded" in line:
+                return
 
     def test_securejoin(self, cmfactory, lp, maildomain2):
         ac1 = cmfactory.new_online_configuring_account(cache=False)
@@ -172,7 +153,7 @@ def test_hide_senders_ip_address(cmfactory):
     assert ipaddress.ip_address(public_ip)
 
     user1, user2 = cmfactory.get_online_accounts(2)
-    chat = cmfactory.get_accepted_chat(user1, user2)
+    chat = cmfactory.get_protected_chat(user1, user2)
 
     chat.send_text("testing submission header cleanup")
     user2._evtracker.wait_next_incoming_message()
@@ -181,11 +162,18 @@ def test_hide_senders_ip_address(cmfactory):
     assert public_ip not in msg.obj.as_string()
 
 
-def test_echobot(cmfactory, chatmail_config, lp):
+def test_echobot(cmfactory, chatmail_config, lp, sshdomain):
     ac = cmfactory.get_online_accounts(1)[0]
 
-    lp.sec(f"Send message to echo@{chatmail_config.mail_domain}")
-    chat = ac.create_chat(f"echo@{chatmail_config.mail_domain}")
+    # establish contact with echobot
+    sshexec = SSHExec(sshdomain)
+    command = "cat /var/lib/echobot/invite-link.txt"
+    echo_invite_link = sshexec(call=rshell.shell, kwargs=dict(command=command))
+    chat = ac.qr_setup_contact(echo_invite_link)
+    ac._evtracker.wait_securejoin_joiner_progress(1000)
+
+    # send message and check it gets replied back
+    lp.sec("Send message to echobot")
     text = "hi, I hope you text me back"
     chat.send_text(text)
     lp.sec("Wait for reply from echobot")

cmdeploy/src/cmdeploy/tests/plugin.py

@@ -62,7 +62,7 @@ def sshdomain(maildomain):
 def maildomain2():
     domain = os.environ.get("CHATMAIL_DOMAIN2")
     if not domain:
-        pytest.skip("set CHATMAIL_DOMAIN2 to a ssh-reachable chatmail instance")
+        pytest.skip("set CHATMAIL_DOMAIN2 to a second chatmail server")
     return domain
 
 
@@ -302,10 +302,12 @@ def cmfactory(request, gencreds, tmpdir, maildomain):
     pytest.importorskip("deltachat")
     from deltachat.testplugin import ACFactory
 
-    data = request.getfixturevalue("data")
-
     testproc = ChatmailTestProcess(request.config, maildomain, gencreds)
-    am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=data)
+
+    class Data:
+        def read_path(self, path):
+            return
+    am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=Data())
 
     # nb. a bit hacky
     # would probably be better if deltachat's test machinery grows native support

www/src/info.md

@@ -6,29 +6,6 @@ interoperable e-mail service for everyone. What's behind a `chatmail` is
 effectively a normal e-mail address just like any other but optimized 
 for the usage in chats, especially DeltaChat.
 
-### Choosing a chatmail address instead of using a random one
-
-In the Delta Chat account setup you may tap `Create a profile` then `Use other server` and choose `Classic e-mail login`. Here fill the two fields like this: 
-
-- `E-Mail Address`: invent a word with
-{% if username_min_length == username_max_length %}
-  *exactly* {{ username_min_length }}
-{% else %}
-  {{ username_min_length}}
-  {% if username_max_length == "more" %}
-    or more
-  {% else %}
-    to {{ username_max_length }}
-  {% endif %}
-{% endif %}
-  characters
-  and append `@{{config.mail_domain}}` to it.
-
-- `Existing Password`: invent at least {{ password_min_length }} characters.
-
-If the e-mail address is not yet taken, you'll get that account. 
-The first login sets your password. 
-
 
 ### Rate and storage limits 
 
@@ -38,10 +15,11 @@ The first login sets your password.
 
 - You may send up to {{ config.max_user_send_per_minute }} messages per minute.
 
-- Messages are unconditionally removed {{ config.delete_mails_after }} days after arriving on the server.
-
 - You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
 
+- Messages are unconditionally removed latest {{ config.delete_mails_after }} days after arriving on the server.
+  Earlier, if storage may exceed otherwise.
+
 
 ### <a name="account-deletion"></a> Account deletion