Remove authclean service from reinjecting port

Messages are already cleaned once on the submission port
to avoid leaking the user IP address.
There is no need to clean them again
and remove the `Received` header
which records reception from filtermail.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,32 +0,0 @@
----
-name: Bug report
-about: Report something that isn't working.
-title: ''
-assignees: ''
-
----
-
-<!--
-Please fill out as much of this form as you can (leaving out stuff that is not applicable is ok).
--->
-
-- Server OS (Operating System) - preferably Debian 12: 
-- On which OS you run cmdeploy:
-
-## Expected behavior
-
-*What did you try to achieve?*
-
-## Actual behavior
-
-*What happened instead?*
-
-### Steps to reproduce the problem:
-
-1. 
-2.
-
-### Screenshots
-
-### Logs
-

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +0,0 @@
-blank_issues_enabled: true
-contact_links:
-  - name: Mutual Help Chat Group
-    url: https://i.delta.chat/#C2846EB4C1CB8DF84B1818F5E3A638FC3FBDC981&a=stalebot1%40nine.testrun.org&g=Chatmail%20Mutual%20Help&x=7sFF7Ik50pWv6J1z7RVC5527&i=d7s1HvOsk5UrSf9AoqRZggg4&s=XmX_9BAW6-g5Ao5E8PyaeKNB
-    about: If you have troubles setting up the relay server, feel free to ask here.

CHANGELOG.md

@@ -2,40 +2,6 @@
 
 ## untagged
 
-- Avoid "acmetool not found" during initial run
-  ([#550](https://github.com/chatmail/relay/pull/550))
-
-- Enforce end-to-end encryption for incoming messages. 
-  New user address mailboxes now get a `enforceE2EEincoming` file 
-  which prohibits incoming cleartext messages from other domains. 
-  An outside MTA trying to submit a cleartext message will 
-  get a "523 Encryption Needed" response, see RFC5248. 
-  If the file does not exist (as it the case for all existing accounts) 
-  incoming cleartext messages are accepted. 
-  ([#538](https://github.com/chatmail/server/pull/538))
-
-- Enforce end-to-end encryption between local addresses 
-  ([#535](https://github.com/chatmail/server/pull/535))
-
-- Limit the bind for the HTTPS server on 8443 to 127.0.0.1 
-  ([#522](https://github.com/chatmail/server/pull/522))
-  ([#532](https://github.com/chatmail/server/pull/532))
-
-- Send SNI when connecting to outside servers
-  ([#524](https://github.com/chatmail/server/pull/524))
-
-- postfix master.cf: use 127.0.0.1 for consistency
-  ([#544](https://github.com/chatmail/relay/pull/544))
-
-- Pass through `original_content` instead of `content` in filtermail
-  ([#509](https://github.com/chatmail/server/pull/509))
-
-- Document TLS requirements in the readme
-  ([#514](https://github.com/chatmail/server/pull/514))
-
-- Remove cleanup service from submission ports
-  ([#512](https://github.com/chatmail/server/pull/512))
-
 - cmdeploy dovecot: delete big messages after 7 days
   ([#504](https://github.com/chatmail/server/pull/504))
 

README.md

@@ -1,96 +1,48 @@
 
 <img width="800px" src="www/src/collage-top.png"/>
 
-# Chatmail relays for end-to-end encrypted e-mail
+# Chatmail services optimized for Delta Chat apps 
 
-Chatmail relay servers are interoperable Mail Transport Agents (MTAs) designed for: 
+This repository helps to setup a ready-to-use chatmail server
-
-- **Convenience:** Low friction instant onboarding
-
-- **Privacy:** No name, phone numbers, email required or collected 
-
-- **End-to-End Encryption enforced**: only OpenPGP messages with metadata minimization allowed 
-
-- **Instant:** Privacy-preserving Push Notifications for Apple, Google, and Huawei
-
-- **Speed:** Message delivery in half a second, with optional P2P realtime connections 
-
-- **Transport Security:** Strict TLS and DKIM enforced 
-
-- **Reliability:** No spam or IP reputation checks; rate-limits are suitable for realtime chats
-
-- **Efficiency:** Messages are only stored for transit and removed automatically
-
-This repository contains everything needed to setup a ready-to-use chatmail relay
 comprised of a minimal setup of the battle-tested 
-[Postfix SMTP](https://www.postfix.org) and [Dovecot IMAP](https://www.dovecot.org) MTAs/MDAs.
+[postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services. 
 
-The automated setup is designed and optimized for providing chatmail addresses
+The setup is designed and optimized for providing chatmail accounts 
-for immediate permission-free onboarding through chat apps and bots.
+for use by [Delta Chat apps](https://delta.chat).
-Chatmail addresses are automatically created at first login,
-after which the initially specified password is required
-for sending and receiving messages through them.
 
-Please see [this list of known apps and client projects](https://chatmail.at/clients.html) 
+Chatmail accounts are automatically created by a first login, 
-and [this list of known public 3rd party chatmail relay servers](https://chatmail.at/relays).
+after which the initially specified password is required for using them. 
 
+## Deploying your own chatmail server 
 
-## Minimal requirements, Prerequisites 
+To deploy chatmail on your own server, you must have set-up ssh authentication and need to use an ed25519 key, due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191). You also need to add your private key to the local ssh-agent, because you can't type in your password during deployment.
-
-You will need the following: 
-
-- Control over a domain through a DNS provider of your choice.
-
-- A Debian 12 server with reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports.
-  IPv6 is encouraged if available.
-  Chatmail relay servers only require 1GB RAM, one CPU, and perhaps 10GB storage for a
-  few thousand active chatmail addresses.
-
-- Key-based SSH authentication to the root user.
-  You must add a passphrase-protected private key to your local ssh-agent
-  because you can't type in your passphrase during deployment.
-  (An ed25519 private key is required due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191))
-
-
-## Getting started 
 
 We use `chat.example.org` as the chatmail domain in the following steps. 
 Please substitute it with your own domain. 
 
-1. Setup the initial DNS records.
+1. Install the `cmdeploy` command in a virtualenv
-   The following is an example in the familiar BIND zone file format with
-   a TTL of 1 hour (3600 seconds).
-   Please substitute your domain and IP addresses.
 
    ```
-    chat.example.com. 3600 IN A 198.51.100.5
+    git clone https://github.com/deltachat/chatmail
-    chat.example.com. 3600 IN AAAA 2001:db8::5
+    cd chatmail
-    www.chat.example.com. 3600 IN CNAME chat.example.com.
-    mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
-   ```
-
-2. Clone the repository and bootstrap the Python virtualenv.
-
-   ```
-    git clone https://github.com/chatmail/relay
-    cd relay
     scripts/initenv.sh
    ```
 
-3. Create chatmail configuration file `chatmail.ini`:
+2. Create chatmail configuration file `chatmail.ini`:
 
    ```
     scripts/cmdeploy init chat.example.org  # <-- use your domain 
    ```
 
-4. Verify that SSH root login works:
+3. Point your domain to the server's IP address,
+   if you haven't done so already.
+   Verify that SSH root login works:
 
    ```
     ssh root@chat.example.org   # <-- use your domain 
    ```
 
-
+4. Deploy to the remote chatmail server:
-5. Deploy the remote chatmail relay server:
 
    ```
     scripts/cmdeploy run
@@ -130,25 +82,25 @@ scripts/cmdeploy bench
 
 This repository has four directories:
 
-- [cmdeploy](https://github.com/chatmail/relay/tree/main/cmdeploy)
+- [cmdeploy](https://github.com/deltachat/chatmail/tree/main/cmdeploy)
   is a collection of configuration files
   and a [pyinfra](https://pyinfra.com)-based deployment script.
 
-- [chatmaild](https://github.com/chatmail/relay/tree/main/chatmaild)
+- [chatmaild](https://github.com/deltachat/chatmail/tree/main/chatmaild)
-  is a Python package containing several small services
+  is a python package containing several small services
   which handle authentication,
   trigger push notifications on new messages,
   ensure that outbound mails are encrypted,
   delete inactive users,
   and some other minor things.
-  chatmaild can also be installed as a stand-alone Python package.
+  chatmaild can also be installed as a stand-alone python package.
 
-- [www](https://github.com/chatmail/relay/tree/main/www)
+- [www](https://github.com/deltachat/chatmail/tree/main/www)
   contains the html, css, and markdown files
-  which make up a chatmail relay's web page.
+  which make up a chatmail server's web page.
-  Edit them before deploying to make your chatmail relay stand out.
+  Edit them before deploying to make your chatmail server stand out.
 
-- [scripts](https://github.com/chatmail/relay/tree/main/scripts)
+- [scripts](https://github.com/deltachat/chatmail/tree/main/scripts)
   offers two convenience tools for beginners;
   `initenv.sh` installs the necessary dependencies to a local virtual environment,
   and the `scripts/cmdeploy` script enables you
@@ -160,81 +112,79 @@ The `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
 helps with setting up and managing the chatmail service.
 `cmdeploy init` creates the `chatmail.ini` config file.
 `cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [script](`cmdeploy/src/cmdeploy/__init__.py`)
-to automatically install or upgrade all chatmail components on a relay,
+to automatically install or upgrade all chatmail components on a server,
 according to the `chatmail.ini` config.
 
 The components of chatmail are:
 
-- [Postfix SMTP MTA](https://www.postfix.org) accepts and relays messages
+- [postfix smtp server](https://www.postfix.org) accepts sent messages (both from your users and from other servers)
-  (both from your users and from the wider e-mail MTA network)
 
-- [Dovecot IMAP MDA](https://www.dovecot.org) stores messages for your users until they download them
+- [dovecot imap server](https://www.dovecot.org) stores messages for your users until they download them
 
-- [Nginx](https://nginx.org/) shows the web page with your privacy policy and additional information
+- [nginx](https://nginx.org/) shows the web page with your privacy policy and additional information
 
-- [acmetool](https://hlandau.github.io/acmetool/) manages TLS certificates for Dovecot, Postfix, and Nginx
+- [acmetool](https://hlandau.github.io/acmetool/) manages TLS certificates for dovecot, postfix, and nginx
 
-- [OpenDKIM](http://www.opendkim.org/) for signing messages with DKIM and rejecting inbound messages without DKIM
+- [opendkim](http://www.opendkim.org/) for signing messages with DKIM and rejecting inbound messages without DKIM
 
 - [mtail](https://google.github.io/mtail/) for collecting anonymized metrics in case you have monitoring
 
-- [Iroh relay](https://www.iroh.computer/docs/concepts/relay) 
-  which helps client devices to establish Peer-to-Peer connections 
-
 - and the chatmaild services, explained in the next section:
 
 ### chatmaild
 
-`chatmaild` implements various systemd-controlled services  
+chatmaild offers several commands
-that integrate with Dovecot and Postfix to achieve instant-onboarding and 
+which differentiate a *chatmail* server from a classic mail server.
-only relaying OpenPGP end-to-end messages encrypted messages. 
+If you deploy them with cmdeploy,
-A short overview of `chatmaild` services: 
+they are run by systemd services in the background.
+A short overview:
 
-- [`doveauth`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/doveauth.py) 
+- [`doveauth`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/doveauth.py) implements
-  implements create-on-login address semantics and is used 
+  create-on-login account creation semantics and is used
-  by Dovecot during IMAP login and by Postfix during SMTP/SUBMISSION login
+  by Dovecot during login authentication and by Postfix
   which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
-  to authenticate logins. 
+  to authenticate users
+  to send mails for them.
 
-- [`filtermail`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/filtermail.py) 
+- [`filtermail`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/filtermail.py) prevents
-  prevents unencrypted email from leaving or entering the chatmail service
+  unencrypted e-mail from leaving the chatmail service
-  and is integrated into Postfix's outbound and inbound mail pipelines.
+  and is integrated into postfix's outbound mail pipelines.
 
-- [`chatmail-metadata`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
+- [`chatmail-metadata`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
-  [Dovecot lua script](https://github.com/chatmail/relay/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
+  [dovecot lua script](https://github.com/deltachat/chatmail/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
-  to store user-specific relay-side config.
+  to store user-specific server-side config.
   On new messages,
-  it [passes the user's push notification token](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/notifier.py)
+  it [passes the user's push notification token](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/notifier.py)
   to [notifications.delta.chat](https://delta.chat/help#instant-delivery)
   so the push notifications on the user's phone can be triggered
-  by Apple/Google/Huawei.
+  by Apple/Google.
 
-- [`delete_inactive_users`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
+- [`delete_inactive_users`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
   deletes users if they have not logged in for a very long time.
   The timeframe can be configured in `chatmail.ini`.
 
-- [`lastlogin`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/lastlogin.py)
+- [`lastlogin`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/lastlogin.py)
-  is contacted by Dovecot when a user logs in
+  is contacted by dovecot when a user logs in
   and stores the date of the login.
 
-- [`echobot`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/echo.py)
+- [`echobot`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/echo.py)
   is a small bot for test purposes.
   It simply echoes back messages from users.
 
-- [`chatmail-metrics`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metrics.py)
+- [`chatmail-metrics`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/metrics.py)
   collects some metrics and displays them at `https://example.org/metrics`.
 
 ### Home page and getting started for users
 
-`cmdeploy run` also creates default static web pages and deploys them
+`cmdeploy run` also creates default static Web pages and deploys them
-to a Nginx web server with:
+to a nginx web server with: 
 
 - a default `index.html` along with a QR code that users can click to
-  create an address on your chatmail relay 
+  create accounts on your chatmail provider,
 
-- a default `info.html` that is linked from the home page
+- a default `info.html` that is linked from the home page,
 
-- a default `policy.html` that is linked from the home page
+- a default `policy.html` that is linked from the home page.
 
 All `.html` files are generated
 by the according markdown `.md` file in the `www/src` directory.
@@ -242,64 +192,48 @@ by the according markdown `.md` file in the `www/src` directory.
 
 ### Refining the web pages
 
+
 ```
 scripts/cmdeploy webdev
 ```
 
-This starts a local live development cycle for chatmail web pages:
+This starts a local live development cycle for chatmail Web pages:
 
 - uses the `www/src/page-layout.html` file for producing static
   HTML pages from `www/src/*.md` files
 
 - continously builds the web presence reading files from `www/src` directory
-  and generating HTML files and copying assets to the `www/build` directory.
+  and generating html files and copying assets to the `www/build` directory.
 
 - Starts a browser window automatically where you can "refresh" as needed.
 
-## Mailbox directory layout
 
-Fresh chatmail addresses have a mailbox directory that contains: 
+## Emergency Commands to disable automatic account creation
 
-- a `password` file with the salted password required for authenticating
+If you need to stop account creation,
-  whether a login may use the address to send/receive messages. 
+e.g. because some script is wildly creating accounts,
-  If you modify the password file manually, you effectively block the user. 
+login to the server with ssh and run:
-
-- `enforceE2EEincoming` is a default-created file with each address. 
-  If present the file indicates that this chatmail address rejects incoming cleartext messages.
-  If absent the address accepts incoming cleartext messages. 
-
-- `dovecot*`, `cur`, `new` and `tmp` represent IMAP/mailbox state. 
-  If the address is only used by one device, the Maildir directories
-  will typically be empty unless the user of that address hasn't been online 
-  for a while. 
-
-
-## Emergency Commands to disable automatic address creation
-
-If you need to stop address creation,
-e.g. because some script is wildly creating addresses, 
-login with ssh and run:
 
 ```
     touch /etc/chatmail-nocreate
 ```
 
-Chatmail address creation will be denied while this file is present.
+While this file is present, account creation will be blocked.
 
 ### Ports
 
-[Postfix](http://www.postfix.org/) listens on ports 25 (SMTP) and 587 (SUBMISSION) and 465 (SUBMISSIONS).
+[Postfix](http://www.postfix.org/) listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
-[Dovecot](https://www.dovecot.org/) listens on ports 143 (IMAP) and 993 (IMAPS).
+[Dovecot](https://www.dovecot.org/) listens on ports 143 (imap) and 993 (imaps).
-[Nginx](https://www.nginx.com/) listens on port 8443 (HTTPS-ALT) and 443 (HTTPS).
+[nginx](https://www.nginx.com/) listens on port 8443 (https-alt) and 443 (https).
 Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
-[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (HTTP).
+[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (http).
 
-chatmail-core based apps will, however, discover all ports and configurations
+Delta Chat apps will, however, discover all ports and configurations
-automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail relay server.
+automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail service.
 
 ## Email authentication
 
-Chatmail relays enforce [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
+chatmail servers rely on [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
 to authenticate incoming emails.
 Incoming emails must have a valid DKIM signature with
 Signing Domain Identifier (SDID, `d=` parameter in the DKIM-Signature header)
@@ -322,130 +256,101 @@ and rejects incorrectly authenticated emails with [`reject_sender_login_mismatch
 `From:` header must correspond to envelope MAIL FROM,
 this is ensured by `filtermail` proxy.
 
-## TLS requirements
+## Migrating chatmail server to a new host
 
-Postfix is configured to require valid TLS
+If you want to migrate chatmail from an old machine
-by setting [`smtp_tls_security_level`](https://www.postfix.org/postconf.5.html#smtp_tls_security_level) to `verify`.
-If emails don't arrive at your chatmail relay server, 
-the problem is likely that your relay does not have a valid TLS certificate.
-
-You can test it by resolving `MX` records of your relay domain
-and then connecting to MX relays (e.g `mx.example.org`) with
-`openssl s_client -connect mx.example.org:25 -verify_hostname mx.example.org -verify_return_error -starttls smtp`
-from the host that has open port 25 to verify that certificate is valid.
-
-When providing a TLS certificate to your chatmail relay server, 
-make sure to provide the full certificate chain
-and not just the last certificate.
-
-If you are running an Exim server and don't see incoming connections
-from a chatmail relay server in the logs,
-make sure `smtp_no_mail` log item is enabled in the config
-with `log_selector = +smtp_no_mail`.
-By default Exim does not log sessions that are closed
-before sending the `MAIL` command.
-This happens if certificate is not recognized as valid by Postfix,
-so you might think that connection is not established
-while actually it is a problem with your TLS certificate.
-
-## Migrating a chatmail relay to a new host
-
-If you want to migrate chatmail relay from an old machine
 to a new machine,
 you can use these steps.
-They were tested with a Linux laptop;
+They were tested with a linux laptop;
 you might need to adjust some of the steps to your environment.
 
 Let's assume that your `mail_domain` is `mail.example.org`,
 all involved machines run Debian 12,
-your old site's IP address is `13.37.13.37`,
+your old server's IP address is `13.37.13.37`,
-and your new site's IP address is `13.12.23.42`.
+and your new server's IP address is `13.12.23.42`.
 
-Note, you should lower the TTLs of your DNS records to a value
+During the guide, you might get a warning about changed SSH Host keys;
-such as 300 (5 minutes) so the migration happens as smoothly as possible.
+in this case, just run `ssh-keygen -R "mail.example.org"` as recommended
+to make sure you can connect with SSH.
 
-During the guide you might get a warning about changed SSH Host keys;
+1. First, copy `/var/lib/acme` to the new server with
-in this case, just run `ssh-keygen -R "mail.example.org"` as recommended.
+   `ssh root@13.37.13.37 tar c /var/lib/acme | ssh root@13.12.23.42 tar x -C /var/lib/`.
+   This transfers your TLS certificate.
 
-1. First, disable mail services on the old site. 
+2. You should also copy `/etc/dkimkeys` to the new server with
+   `ssh root@13.37.13.37 tar c /etc/dkimkeys | ssh root@13.12.23.42 tar x -C /etc/`
+   so the DKIM DNS record stays correct.
 
-   ```
+3. On the new server, run `chown root: -R /var/lib/acme` and `chown opendkim: -R /etc/dkimkeys` to make sure the permissions are correct.
-    cmdeploy run --disable-mail --ssh-host 13.37.13.37
+
-   ```
+4. Run `cmdeploy run --disable-mail --ssh-host 13.12.23.42` to install chatmail on the new machine.
+   postfix and dovecot are disabled for now,
+   we will enable them later.
+
+5. Now, point DNS to the new IP addresses.
+
+   You can already remove the old IP addresses from DNS.
+   Existing Delta Chat users will still be able to connect
+   to the old server, send and receive messages,
+   but new users will fail to create new profiles
+   with your chatmail server.
+
+   If other servers try to deliver messages to your new server they will fail,
+   but normally email servers will retry delivering messages
+   for at least a week, so messages will not be lost.
+
+6. Now you can run `cmdeploy run --disable-mail --ssh-host 13.37.13.37` to disable your old server.
 
    Now your users will notice the migration
    and will not be able to send or receive messages
    until the migration is completed.
 
-2. Now we want to copy `/home/vmail`, `/var/lib/acme`, `/etc/dkimkeys`, `/run/echobot`, and `/var/spool/postfix` to the new site. 
+7. After everything is stopped,
-   Login to the old site while forwarding your SSH agent 
+    you can copy the `/home/vmail/mail` directory to the new server.
-   so you can copy directly from the old to the new site with your SSH key:
+    It includes all user data, messages, password hashes, etc.
-   ```
-    ssh -A root@13.37.13.37
-    tar c - /home/vmail/mail /var/lib/acme /etc/dkimkeys /run/echobot /var/spool/postfix | ssh root@13.12.23.42 "tar x -C /"
-   ```
 
-   This transfers all addresses, the TLS certificate, DKIM keys (so DKIM DNS record remains valid), and the echobot's password so it continues to function.
+    Just run: `ssh root@13.37.13.37 tar c /home/vmail/mail | ssh root@13.12.23.42 tar x -C /home/vmail/`
-   It also preserves the Postfix mail spool so any messages pending delivery will still be delivered.
 
-3. Install chatmail on the new machine:
+    After this, your new server has all the necessary files to start operating :)
 
-   ```
+8. To be sure the permissions are still fine,
-    cmdeploy run --disable-mail --ssh-host 13.12.23.42
+    run `chown vmail: -R /home/vmail` on the new server.
-   ```
-   Postfix and Dovecot are disabled for now; we will enable them later. 
-   We first need to make the new site fully operational. 
 
-3. On the new site, run the following to ensure the ownership is correct in case UIDs/GIDs changed:
+9. Finally, you can run `cmdeploy run` to turn on chatmail on the new server.
-
+    Your users can continue using the chatmail server,
-   ```
+    and messages which were sent after step 6. should arrive now.
-    chown root: -R /var/lib/acme
+    Voilà!
-    chown opendkim: -R /etc/dkimkeys
-    chown vmail: -R /home/vmail/mail
-    chown echobot: -R /run/echobot
-   ```
-
-4. Now, update DNS entries. 
-
-   If other MTAs try to deliver messages to your chatmail domain they may fail intermittently,
-   as DNS catches up with the new site settings 
-   but normally will retry delivering messages
-   for at least a week, so messages will not be lost.
-
-5. Finally, you can execute `cmdeploy run --ssh-host 13.12.23.42` to turn on chatmail on the new relay.
-   Your users will be able to use the chatmail relay as soon as the DNS changes have propagated.
-   Voilà!
 
 ## Setting up a reverse proxy
 
-A chatmail relay MTA does not track or depend on the client IP address
+A chatmail server does not depend on the client IP address
 for its operation, so it can be run behind a reverse proxy.
 This will not even affect incoming mail authentication
 as DKIM only checks the cryptographic signature
 of the message and does not use the IP address as the input.
 
-For example, you may want to self-host your chatmail relay
+For example, you may want to self-host your chatmail server
 and only use hosted VPS to provide a public IP address
 for client connections and incoming mail.
-You can connect chatmail relay to VPS
+You can connect chatmail server to VPS
 using a tunnel protocol
 such as [WireGuard](https://www.wireguard.com/)
 and setup a reverse proxy on a VPS
-to forward connections to the chatmail relay
+to forward connections to the chatmail server
 over the tunnel.
 You can also setup multiple reverse proxies
-for your chatmail relay in different networks
+for your chatmail server in different networks
-to ensure your relay is reachable even when
+to ensure your server is reachable even when
 one of the IPs becomes inaccessible due to
 hosting or routing problems.
 
-Note that your chatmail relay still needs
+Note that your server still needs
 to be able to make outgoing connections on port 25
 to send messages outside.
 
 To setup a reverse proxy
 (or rather Destination NAT, DNAT)
-for your chatmail relay,
+for your chatmail server,
 put the following configuration in `/etc/nftables.conf`:
 ```
 #!/usr/sbin/nft -f
@@ -520,7 +425,7 @@ table inet filter {
 ```
 
 Run `systemctl enable nftables.service`
-to ensure configuration is reloaded when the proxy relay reboots.
+to ensure configuration is reloaded when the proxy server reboots.
 
 Uncomment in `/etc/sysctl.conf` the following two lines:
 
@@ -529,7 +434,7 @@ net.ipv4.ip_forward=1
 net.ipv6.conf.all.forwarding=1
 ```
 
-Then reboot the relay or do `sysctl -p` and `nft -f /etc/nftables.conf`.
+Then reboot the server or do `sysctl -p` and `nft -f /etc/nftables.conf`.
 
-Once proxy relay is set up,
+Once proxy server is set up,
 you can add its IP address to the DNS.

chatmaild/src/chatmaild/config.py

@@ -11,11 +11,7 @@ def read_config(inipath):
     assert Path(inipath).exists(), inipath
     cfg = iniconfig.IniConfig(inipath)
     params = cfg.sections["params"]
-    default_config_content = get_default_config_content(params["mail_domain"])
+    return Config(inipath, params=params)
-    df_params = iniconfig.IniConfig("ini", data=default_config_content)["params"]
-    new_params = dict(df_params.items())
-    new_params.update(params)
-    return Config(inipath, params=new_params)
 
 
 class Config:
@@ -33,13 +29,7 @@ class Config:
         self.passthrough_senders = params["passthrough_senders"].split()
         self.passthrough_recipients = params["passthrough_recipients"].split()
         self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
-        self.filtermail_smtp_port_incoming = int(
-            params["filtermail_smtp_port_incoming"]
-        )
         self.postfix_reinject_port = int(params["postfix_reinject_port"])
-        self.postfix_reinject_port_incoming = int(
-            params["postfix_reinject_port_incoming"]
-        )
         self.mtail_address = params.get("mtail_address")
         self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
         self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
@@ -79,11 +69,6 @@ class Config:
 
 def write_initial_config(inipath, mail_domain, overrides):
     """Write out default config file, using the specified config value overrides."""
-    content = get_default_config_content(mail_domain, **overrides)
-    inipath.write_text(content)
-
-
-def get_default_config_content(mail_domain, **overrides):
     from importlib.resources import files
 
     inidir = files(__package__).joinpath("ini")
@@ -128,4 +113,5 @@ def get_default_config_content(mail_domain, **overrides):
             else:
                 lines.append(line)
         content = "\n".join(lines)
-    return content
+
+    inipath.write_text(content)

chatmaild/src/chatmaild/echo.py

@@ -8,7 +8,6 @@ import logging
 import os
 import subprocess
 import sys
-from pathlib import Path
 
 from deltachat_rpc_client import Bot, DeltaChat, EventType, Rpc, events
 
@@ -98,10 +97,6 @@ def main():
         if not bot.is_configured():
             bot.configure(addr, password)
 
-        # write invite link to working directory
-        invitelink = bot.account.get_qr_code()
-        Path("invite-link.txt").write_text(invitelink)
-
         bot.run_forever()
 
 

chatmaild/src/chatmaild/filtermail.py

@@ -11,12 +11,9 @@ from email.utils import parseaddr
 from smtplib import SMTP as SMTPClient
 
 from aiosmtpd.controller import Controller
-from aiosmtpd.smtp import SMTP
 
 from .config import read_config
 
-ENCRYPTION_NEEDED_523 = "523 Encryption Needed: Invalid Unencrypted Mail"
-
 
 def check_openpgp_payload(payload: bytes):
     """Checks the OpenPGP payload.
@@ -160,14 +157,9 @@ def check_encrypted(message):
     return True
 
 
-async def asyncmain_beforequeue(config, mode):
+async def asyncmain_beforequeue(config):
-    if mode == "outgoing":
+    port = config.filtermail_smtp_port
-        port = config.filtermail_smtp_port
+    Controller(BeforeQueueHandler(config), hostname="127.0.0.1", port=port).start()
-        handler = OutgoingBeforeQueueHandler(config)
-    else:
-        port = config.filtermail_smtp_port_incoming
-        handler = IncomingBeforeQueueHandler(config)
-    HackedController(handler, hostname="127.0.0.1", port=port).start()
 
 
 def recipient_matches_passthrough(recipient, passthrough_recipients):
@@ -179,21 +171,7 @@ def recipient_matches_passthrough(recipient, passthrough_recipients):
     return False
 
 
-class HackedController(Controller):
+class BeforeQueueHandler:
-    def factory(self):
-        return SMTPDiscardRCPTO_options(self.handler, **self.SMTP_kwargs)
-
-
-class SMTPDiscardRCPTO_options(SMTP):
-    def _getparams(self, params):
-        # aiosmtpd's SMTP daemon fails to handle a request if there are RCPT TO options
-        # We just ignore them for our incoming filtermail purposes
-        if len(params) == 1 and params[0].startswith("ORCPT"):
-            return {}
-        return super()._getparams(params)
-
-
-class OutgoingBeforeQueueHandler:
     def __init__(self, config):
         self.config = config
         self.send_rate_limiter = SendRateLimiter()
@@ -218,9 +196,7 @@ class OutgoingBeforeQueueHandler:
             return error
         logging.info("re-injecting the mail that passed checks")
         client = SMTPClient("localhost", self.config.postfix_reinject_port)
-        client.sendmail(
+        client.sendmail(envelope.mail_from, envelope.rcpt_tos, envelope.content)
-            envelope.mail_from, envelope.rcpt_tos, envelope.original_content
-        )
         return "250 OK"
 
     def check_DATA(self, envelope):
@@ -231,86 +207,40 @@ class OutgoingBeforeQueueHandler:
         mail_encrypted = check_encrypted(message)
 
         _, from_addr = parseaddr(message.get("from").strip())
+        envelope_from_domain = from_addr.split("@").pop()
 
+        logging.info(f"mime-from: {from_addr} envelope-from: {envelope.mail_from!r}")
         if envelope.mail_from.lower() != from_addr.lower():
             return f"500 Invalid FROM <{from_addr!r}> for <{envelope.mail_from!r}>"
 
-        if mail_encrypted or is_securejoin(message):
+        if mail_encrypted:
-            print("Outgoing: Filtering encrypted mail.", file=sys.stderr)
+            print("Filtering encrypted mail.", file=sys.stderr)
-            return
+        else:
-
+            print("Filtering unencrypted mail.", file=sys.stderr)
-        print("Outgoing: Filtering unencrypted mail.", file=sys.stderr)
 
         if envelope.mail_from in self.config.passthrough_senders:
             return
 
-        # allow self-sent Autocrypt Setup Message
-        if envelope.rcpt_tos == [from_addr]:
-            if message.get("subject") == "Autocrypt Setup Message":
-                if message.get_content_type() == "multipart/mixed":
-                    return
-
         passthrough_recipients = self.config.passthrough_recipients
 
-        for recipient in envelope.rcpt_tos:
-            if recipient_matches_passthrough(recipient, passthrough_recipients):
-                continue
-
-            print("Rejected unencrypted mail.", file=sys.stderr)
-            return ENCRYPTION_NEEDED_523
-
-
-class IncomingBeforeQueueHandler:
-    def __init__(self, config):
-        self.config = config
-
-    async def handle_DATA(self, server, session, envelope):
-        logging.info("handle_DATA before-queue")
-        error = self.check_DATA(envelope)
-        if error:
-            return error
-        logging.info("re-injecting the mail that passed checks")
-
-        # the smtp daemon on reinject_port_incoming gives it to dkim milter
-        # which looks at source address to determine whether to verify or sign
-        client = SMTPClient(
-            "localhost",
-            self.config.postfix_reinject_port_incoming,
-            source_address=("127.0.0.2", 0),
-        )
-        client.sendmail(
-            envelope.mail_from, envelope.rcpt_tos, envelope.original_content
-        )
-        return "250 OK"
-
-    def check_DATA(self, envelope):
-        """the central filtering function for e-mails."""
-        logging.info(f"Processing DATA message from {envelope.mail_from}")
-
-        message = BytesParser(policy=policy.default).parsebytes(envelope.content)
-        mail_encrypted = check_encrypted(message)
-
         if mail_encrypted or is_securejoin(message):
-            print("Incoming: Filtering encrypted mail.", file=sys.stderr)
             return
 
-        print("Incoming: Filtering unencrypted mail.", file=sys.stderr)
-
-        # we want cleartext mailer-daemon messages to pass through
-        # chatmail core will typically not display them as normal messages
-        if message.get("auto-submitted"):
-            _, from_addr = parseaddr(message.get("from").strip())
-            if from_addr.lower().startswith("mailer-daemon@"):
-                if message.get_content_type() == "multipart/report":
-                    return
-
         for recipient in envelope.rcpt_tos:
-            user = self.config.get_user(recipient)
+            if envelope.mail_from == recipient:
-            if user is None or user.is_incoming_cleartext_ok():
+                # Always allow sending emails to self.
                 continue
+            if recipient_matches_passthrough(recipient, passthrough_recipients):
+                continue
+            res = recipient.split("@")
+            if len(res) != 2:
+                return f"500 Invalid address <{recipient}>"
+            _recipient_addr, recipient_domain = res
 
-            print("Rejected unencrypted mail.", file=sys.stderr)
+            is_outgoing = recipient_domain != envelope_from_domain
-            return ENCRYPTION_NEEDED_523
+            if is_outgoing:
+                print("Rejected unencrypted mail.", file=sys.stderr)
+                return f"500 Invalid unencrypted mail to <{recipient}>"
 
 
 class SendRateLimiter:
@@ -329,14 +259,11 @@ class SendRateLimiter:
 
 def main():
     args = sys.argv[1:]
-    assert len(args) == 2
+    assert len(args) == 1
     config = read_config(args[0])
-    mode = args[1]
     logging.basicConfig(level=logging.WARN)
     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
-    assert mode in ["incoming", "outgoing"]
+    task = asyncmain_beforequeue(config)
-    task = asyncmain_beforequeue(config, mode)
     loop.create_task(task)
-    logging.info("entering serving loop")
     loop.run_forever()

chatmaild/src/chatmaild/ini/chatmail.ini.f

@@ -46,13 +46,11 @@ passthrough_recipients = xstore@testrun.org
 # Deployment Details
 #
 
-# SMTP outgoing filtermail and reinjection 
+# where the filtermail SMTP service listens
 filtermail_smtp_port = 10080
-postfix_reinject_port = 10025
 
-# SMTP incoming filtermail and reinjection 
+# postfix accepts on the localhost reinject SMTP port
-filtermail_smtp_port_incoming = 10081
+postfix_reinject_port = 10025
-postfix_reinject_port_incoming = 10026
 
 # if set to "True" IPv6 is disabled
 disable_ipv6 = False

chatmaild/src/chatmaild/metrics.py

@@ -11,8 +11,6 @@ def main(vmail_dir=None):
     ci_accounts = 0
 
     for path in Path(vmail_dir).iterdir():
-        if not path.joinpath("cur").is_dir():
-            continue
         accounts += 1
         if path.name[:3] in ("ci-", "ac_"):
             ci_accounts += 1

chatmaild/src/chatmaild/tests/mail-data/asm.eml

@@ -1,56 +0,0 @@
-From: {from_addr}
-To: {to_addr}
-Autocrypt-Setup-Message: v1
-Subject: Autocrypt Setup Message
-Date: Tue, 22 Jan 2019 12:56:29 +0100
-Content-type: multipart/mixed; boundary="Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ"
-
---Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
-Content-Type: text/plain
-
-This message contains all information to transfer your Autocrypt
-settings along with your secret key securely from your original
-device.
-
-To set up your new device for Autocrypt, please follow the
-instuctions that should be presented by your new device.
-
-You can keep this message and use it as a backup for your secret
-key. If you want to do this, you should write down the Setup Code
-and store it securely.
---Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
-Content-Type: application/autocrypt-setup
-Content-Disposition: attachment; filename="autocrypt-setup-message.html"
-
-<html><body>
-<p>
-This is the Autocrypt setup file used to transfer settings and
-keys between clients. You can decrypt it using the Setup Code
-presented on your old device, and then import the contained key
-into your keyring.
-</p>
-
-<pre>
------BEGIN PGP MESSAGE-----
-Passphrase-Format: numeric9x4
-Passphrase-Begin: 17
-
-jA0EBwMCFAxADoCdzeX/0ukBlqI5+pfpKb751qd/7nLNbkpy3gVcaf1QwRPZYt40
-Ynp08UqRQ2g48ZlnzHLSwlTGOPTuv2Jt8ka+pgZ45xzvJSG2gau03xP4VsC271kR
-VmCjdb0Y6Rk96mAwfGzrkbaRQ9Z7fIoL866GOv6h9neiVIkp+JYlTV6ISD0ZQJ4Q
-I6dOQkB/TWZyVjtiJDOQHdfNWliA6NtqaLq19wlu9L5xXjuNpY95KwR8EJXWe0+o
-Y3d2U/KxOAkXKghP2Qg1GtlPVeGC5T4p03TGI6pzKT+kHX6Rrm9wK6sM9aTquMmF
-Vok84Jg1DFnwivWC2RILR81rXi7k/+Y6MUbveFgJ9cQduqpxnmD7TjOblYu7M6zp
-YGAUxh8DRKlIMn2QsA++DBYQ6ACZvwuY8qTDLkqPDo4WqM313dsMJbyGjDdVE7EM
-PESS+RlABETpZXz8g/ycr6DIUNdlbPcmYlsBfHWDOuR2GFFTwmlv5slWS39dJv38
-E0eIe1CwdxI801Se7t7dUUS/ZF8wb6GlmxOcqGbF8eko1Z0S64IAm7/h13MRQCxI
-geQnHfGYVJ2FOimoCMEKwfa9x++RFTDW0u7spDC2uWvK/1viV8OfRppFhLr/kmKb
-18lWXuAz80DAjUDUsVqEq2MvJBJGoCJUEyjuRsLkHYRM5jYk4v50LyyR0Om73nWF
-nZBqmqNzdr7Xb9PHHdFhnEc0VvoYbrcM0RVYcEMW3YbmejM891j1d6Iv+/n/qND/
-NdebGrfWJMmFLf/iEkzTZ3/v5inW9LpWoRc94ioCjJTaEo8Rib6ARRFaJVIsmNXi
-YicFGO98D+zX+a2t9Yz6IpPajVslnOp6ScpmXgts/2XWD7oE+JgxSAqo/dLVsHgP
-Ufo=
-=pulM
------END PGP MESSAGE-----
-</pre></body></html>
---Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ--

chatmaild/src/chatmaild/tests/mail-data/mailer-daemon.eml

@@ -1,46 +0,0 @@
-Date: Fri, 8 Jul 1994 09:21:47 -0400
-From: Mail Delivery Subsystem <MAILER-DAEMON@example.org>
-Subject: Returned mail: User unknown
-To: <owner-ups-mib@CS.UTK.EDU>
-Auto-Submitted: auto-replied 
-MIME-Version: 1.0
-Content-Type: multipart/report; report-type=delivery-status;
-      boundary="JAA13167.773673707/CS.UTK.EDU"
-
---JAA13167.773673707/CS.UTK.EDU
-content-type: text/plain; charset=us-ascii
-
-   ----- The following addresses had delivery problems -----
-<arathib@vnet.ibm.com>  (unrecoverable error)
-<wsnell@sdcc13.ucsd.edu>  (unrecoverable error)
-
---JAA13167.773673707/CS.UTK.EDU
-content-type: message/delivery-status
-
-Reporting-MTA: dns; cs.utk.edu
-
-Original-Recipient: rfc822;arathib@vnet.ibm.com
-Final-Recipient: rfc822;arathib@vnet.ibm.com
-Action: failed
-Status: 5.0.0 (permanent failure)
-Diagnostic-Code: smtp;
- 550 'arathib@vnet.IBM.COM' is not a registered gateway user
-Remote-MTA: dns; vnet.ibm.com
-
-Original-Recipient: rfc822;johnh@hpnjld.njd.hp.com
-Final-Recipient: rfc822;johnh@hpnjld.njd.hp.com
-Action: delayed
-Status: 4.0.0 (hpnjld.njd.jp.com: host name lookup failure)
-
-Original-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
-Final-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
-Action: failed
-Status: 5.0.0
-Diagnostic-Code: smtp; 550 user unknown
-Remote-MTA: dns; sdcc13.ucsd.edu
-
---JAA13167.773673707/CS.UTK.EDU
-content-type: message/rfc822
-
-[original message goes here]
---JAA13167.773673707/CS.UTK.EDU--

chatmaild/src/chatmaild/tests/plugin.py

@@ -72,8 +72,9 @@ def maildata(request):
     def maildata(name, from_addr, to_addr, subject="[...]"):
         # Using `.read_bytes().decode()` instead of `.read_text()` to preserve newlines.
         data = datadir.joinpath(name).read_bytes().decode()
+
         text = data.format(from_addr=from_addr, to_addr=to_addr, subject=subject)
-        return BytesParser(policy=policy.SMTP).parsebytes(text.encode())
+        return BytesParser(policy=policy.default).parsebytes(text.encode())
 
     return maildata
 

chatmaild/src/chatmaild/tests/test_config.py

@@ -15,14 +15,6 @@ def test_read_config_basic(example_config):
     assert example_config.mail_domain == "chat.example.org"
 
 
-def test_read_config_basic_using_defaults(tmp_path, maildomain):
-    inipath = tmp_path.joinpath("chatmail.ini")
-    inipath.write_text(f"[params]\nmail_domain = {maildomain}")
-    example_config = read_config(inipath)
-    assert example_config.max_user_send_per_minute == 60
-    assert example_config.filtermail_smtp_port_incoming == 10081
-
-
 def test_read_config_testrun(make_config):
     config = make_config("something.testrun.org")
     assert config.mail_domain == "something.testrun.org"

chatmaild/src/chatmaild/tests/test_filtermail.py

@@ -1,8 +1,7 @@
 import pytest
 
 from chatmaild.filtermail import (
-    IncomingBeforeQueueHandler,
+    BeforeQueueHandler,
-    OutgoingBeforeQueueHandler,
     SendRateLimiter,
     check_armored_payload,
     check_encrypted,
@@ -19,13 +18,7 @@ def maildomain():
 @pytest.fixture
 def handler(make_config, maildomain):
     config = make_config(maildomain)
-    return OutgoingBeforeQueueHandler(config)
+    return BeforeQueueHandler(config)
-
-
-@pytest.fixture
-def inhandler(make_config, maildomain):
-    config = make_config(maildomain)
-    return IncomingBeforeQueueHandler(config)
 
 
 def test_reject_forged_from(maildata, gencreds, handler):
@@ -36,14 +29,14 @@ def test_reject_forged_from(maildata, gencreds, handler):
     # test that the filter lets good mail through
     to_addr = gencreds()[0]
     env.content = maildata(
-        "encrypted.eml", from_addr=env.mail_from, to_addr=to_addr
+        "plain.eml", from_addr=env.mail_from, to_addr=to_addr
     ).as_bytes()
 
     assert not handler.check_DATA(envelope=env)
 
     # test that the filter rejects forged mail
     env.content = maildata(
-        "encrypted.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
+        "plain.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
     ).as_bytes()
     error = handler.check_DATA(envelope=env)
     assert "500" in error
@@ -113,7 +106,7 @@ def test_send_rate_limiter():
             break
 
 
-def test_cleartext_excempt_privacy(maildata, gencreds, handler):
+def test_excempt_privacy(maildata, gencreds, handler):
     from_addr = gencreds()[0]
     to_addr = "privacy@testrun.org"
     handler.config.passthrough_recipients = [to_addr]
@@ -134,73 +127,10 @@ def test_cleartext_excempt_privacy(maildata, gencreds, handler):
         rcpt_tos = [to_addr, false_to]
         content = msg.as_bytes()
 
-    assert "523" in handler.check_DATA(envelope=env2)
+    assert "500" in handler.check_DATA(envelope=env2)
 
 
-def test_cleartext_self_send_autocrypt_setup_message(maildata, gencreds, handler):
+def test_passthrough_domains(maildata, gencreds, handler):
-    from_addr = gencreds()[0]
-    to_addr = from_addr
-
-    msg = maildata("asm.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    assert not handler.check_DATA(envelope=env)
-
-
-def test_cleartext_send_fails(maildata, gencreds, handler):
-    from_addr = gencreds()[0]
-    to_addr = gencreds()[0]
-
-    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    res = handler.check_DATA(envelope=env)
-    assert "523 Encryption Needed" in res
-
-
-def test_cleartext_incoming_fails(maildata, gencreds, inhandler):
-    from_addr = gencreds()[0]
-    to_addr, password = gencreds()
-
-    msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    user = inhandler.config.get_user(to_addr)
-    user.set_password(password)
-    res = inhandler.check_DATA(envelope=env)
-    assert "523 Encryption Needed" in res
-
-    user.allow_incoming_cleartext()
-    assert not inhandler.check_DATA(envelope=env)
-
-
-def test_cleartext_incoming_mailer_daemon(maildata, gencreds, inhandler):
-    from_addr = "mailer-daemon@example.org"
-    to_addr = gencreds()[0]
-
-    msg = maildata("mailer-daemon.eml", from_addr=from_addr, to_addr=to_addr)
-
-    class env:
-        mail_from = from_addr
-        rcpt_tos = [to_addr]
-        content = msg.as_bytes()
-
-    assert not inhandler.check_DATA(envelope=env)
-
-
-def test_cleartext_passthrough_domains(maildata, gencreds, handler):
     from_addr = gencreds()[0]
     to_addr = "privacy@x.y.z"
     handler.config.passthrough_recipients = ["@x.y.z"]
@@ -221,10 +151,10 @@ def test_cleartext_passthrough_domains(maildata, gencreds, handler):
         rcpt_tos = [to_addr, false_to]
         content = msg.as_bytes()
 
-    assert "523" in handler.check_DATA(envelope=env2)
+    assert "500" in handler.check_DATA(envelope=env2)
 
 
-def test_cleartext_passthrough_senders(gencreds, handler, maildata):
+def test_passthrough_senders(gencreds, handler, maildata):
     acc1 = gencreds()[0]
     to_addr = "recipient@something.org"
     handler.config.passthrough_senders = [acc1]

chatmaild/src/chatmaild/tests/test_metrics.py

@@ -2,15 +2,8 @@ from chatmaild.metrics import main
 
 
 def test_main(tmp_path, capsys):
-    paths = []
     for x in ("ci-asllkj", "ac_12l3kj", "qweqwe", "ci-l1k2j31l2k3"):
-        p = tmp_path.joinpath(x)
+        tmp_path.joinpath(x).mkdir()
-        p.mkdir()
-        p.joinpath("cur").mkdir()
-        paths.append(p)
-
-    tmp_path.joinpath("nomailbox").mkdir()
-
     main(tmp_path)
     out, _ = capsys.readouterr()
     d = {}

chatmaild/src/chatmaild/tests/test_user.py

@@ -40,17 +40,3 @@ def test_no_mailboxes_dir(testaddr, example_config, tmp_path):
     user.set_password("someeqkjwelkqwjleqwe")
     user.set_last_login_timestamp(100000)
     assert user.get_last_login_timestamp() == 86400
-
-
-def test_set_get_cleartext_flag(testaddr, example_config, tmp_path):
-    p = tmp_path.joinpath("a", "mailboxes")
-    example_config.mailboxes_dir = p
-
-    user = example_config.get_user(testaddr)
-    user.set_password("someeqkjwelkqwjleqwe")
-    user.set_last_login_timestamp(100000)
-    assert user.get_last_login_timestamp() == 86400
-
-    assert not user.is_incoming_cleartext_ok()
-    user.allow_incoming_cleartext()
-    assert user.is_incoming_cleartext_ok()

chatmaild/src/chatmaild/user.py

@@ -13,7 +13,6 @@ class User:
         self.maildir = maildir
         self.addr = addr
         self.password_path = password_path
-        self.enforce_E2EE_path = maildir.joinpath("enforceE2EEincoming")
         self.uid = uid
         self.gid = gid
 
@@ -36,13 +35,6 @@ class User:
         home = str(self.maildir)
         return dict(addr=self.addr, home=home, uid=self.uid, gid=self.gid, password=pw)
 
-    def is_incoming_cleartext_ok(self):
-        return not self.enforce_E2EE_path.exists()
-
-    def allow_incoming_cleartext(self):
-        if self.enforce_E2EE_path.exists():
-            self.enforce_E2EE_path.unlink()
-
     def set_password(self, enc_password):
         """Set the specified password for this user.
 
@@ -58,7 +50,6 @@ class User:
             if not self.addr.startswith("echo@"):
                 logging.error(f"could not write password for: {self.addr}")
                 raise
-        self.enforce_E2EE_path.touch()
 
     def set_last_login_timestamp(self, timestamp):
         """Track login time with daily granularity

cmdeploy/src/cmdeploy/__init__.py

@@ -106,14 +106,12 @@ def _install_remote_venv_with_chatmaild(config) -> None:
     for fn in (
         "doveauth",
         "filtermail",
-        "filtermail-incoming",
         "echobot",
         "chatmail-metadata",
         "lastlogin",
     ):
-        execpath = fn if fn != "filtermail-incoming" else "filtermail"
         params = dict(
-            execpath=f"{remote_venv_dir}/bin/{execpath}",
+            execpath=f"{remote_venv_dir}/bin/{fn}",
             config_path=remote_chatmail_inipath,
             remote_venv_dir=remote_venv_dir,
             mail_domain=config.mail_domain,
@@ -217,7 +215,7 @@ def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
         server.shell(
             name="Generate OpenDKIM domain keys",
             commands=[
-                f"/usr/sbin/opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
+                f"opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
             ],
             _use_su_login=True,
             _su_user="opendkim",
@@ -543,6 +541,7 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
 
     server.group(name="Create vmail group", group="vmail", system=True)
     server.user(name="Create vmail user", user="vmail", group="vmail", system=True)
+    server.user(name="Create filtermail user", user="filtermail", system=True)
     server.group(name="Create opendkim group", group="opendkim", system=True)
     server.user(
         name="Create opendkim user",

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -46,7 +46,10 @@ http {
 
 	server {
   
-		listen 127.0.0.1:8443 ssl default_server;
+		listen 8443 ssl default_server;
+		{% if not disable_ipv6 %}
+		listen [::]:8443 ssl default_server;
+		{% endif %}
 
 		root /var/www/html;
 
@@ -117,7 +120,10 @@ http {
 
 	# Redirect www. to non-www
 	server {
-		listen 127.0.0.1:8443 ssl;
+		listen 8443 ssl;
+		{% if not disable_ipv6 %}
+		listen [::]:8443 ssl;
+		{% endif %}
 		server_name www.{{ config.domain_name }};
 		return 301 $scheme://{{ config.domain_name }}$request_uri;
 		access_log syslog:server=unix:/dev/log,facility=local7;

cmdeploy/src/cmdeploy/postfix/main.cf.j2

@@ -21,9 +21,6 @@ smtpd_tls_security_level=may
 
 smtp_tls_CApath=/etc/ssl/certs
 smtp_tls_security_level=verify
-# Send SNI extension when connecting to other servers.
-# <https://www.postfix.org/postconf.5.html#smtp_tls_servername>
-smtp_tls_servername = hostname
 smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
 smtp_tls_policy_maps = inline:{nauta.cu=may}
 smtpd_tls_protocols = >=TLSv1.2

cmdeploy/src/cmdeploy/postfix/master.cf.j2

@@ -14,7 +14,7 @@ smtp      inet  n       -       y       -       -       smtpd -v
 {%- else %}
 smtp      inet  n       -       y       -       -       smtpd 
 {%- endif %}
-  -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
+  -o smtpd_milters=unix:opendkim/opendkim.sock
 submission inet n       -       y       -       5000    smtpd
   -o syslog_name=postfix/submission
   -o smtpd_tls_security_level=encrypt
@@ -32,6 +32,7 @@ submission inet n       -       y       -       5000    smtpd
   -o milter_macro_daemon_name=ORIGINATING
   -o smtpd_client_connection_count_limit=1000
   -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
+  -o cleanup_service_name=authclean
 smtps     inet  n       -       y       -       5000    smtpd
   -o syslog_name=postfix/smtps
   -o smtpd_tls_wrappermode=yes
@@ -49,6 +50,7 @@ smtps     inet  n       -       y       -       5000    smtpd
   -o smtpd_client_connection_count_limit=1000
   -o milter_macro_daemon_name=ORIGINATING
   -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
+  -o cleanup_service_name=authclean
 #628       inet  n       -       y       -       -       qmqpd
 pickup    unix  n       -       y       60      1       pickup
 cleanup   unix  n       -       y       -       0       cleanup
@@ -76,16 +78,10 @@ anvil     unix  -       -       y       -       1       anvil
 scache    unix  -       -       y       -       1       scache
 postlog   unix-dgram n  -       n       -       1       postlogd
 filter    unix -        n       n       -       -       lmtp
-# Local SMTP server for reinjecting outgoing filtered mail.
+# Local SMTP server for reinjecting filered mail.
-127.0.0.1:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
+localhost:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
   -o syslog_name=postfix/reinject
   -o smtpd_milters=unix:opendkim/opendkim.sock
-  -o cleanup_service_name=authclean
-
-# Local SMTP server for reinjecting incoming filtered mail 
-127.0.0.1:{{ config.postfix_reinject_port_incoming }} inet  n       -       n       -       10      smtpd
-  -o syslog_name=postfix/reinject_incoming
-  -o smtpd_milters=unix:opendkim/opendkim.sock
 
 # Cleanup `Received` headers for authenticated mail
 # to avoid leaking client IP.
@@ -93,10 +89,5 @@ filter    unix -        n       n       -       -       lmtp
 # We do not do this for received mails
 # as this will break DKIM signatures
 # if `Received` header is signed.
-#
-# This service also rewrites
-# Subject with `[...]`
-# to make sure the users
-# cannot send unprotected Subject.
 authclean unix  n       -       -       -       0       cleanup
   -o header_checks=regexp:/etc/postfix/submission_header_cleanup

cmdeploy/src/cmdeploy/remote/rshell.py

@@ -1,13 +1,10 @@
-from subprocess import DEVNULL, CalledProcessError, check_output
+from subprocess import CalledProcessError, check_output
 
 
 def shell(command, fail_ok=False):
     print(f"$ {command}")
-    args = dict(shell=True)
-    if fail_ok:
-        args["stderr"] = DEVNULL
     try:
-        return check_output(command, **args).decode().rstrip()
+        return check_output(command, shell=True).decode().rstrip()
     except CalledProcessError:
         if not fail_ok:
             raise
@@ -17,22 +14,3 @@ def shell(command, fail_ok=False):
 def get_systemd_running():
     lines = shell("systemctl --type=service --state=running").split("\n")
     return [line for line in lines if line.startswith("  ")]
-
-
-def write_numbytes(path, num):
-    with open(path, "w") as f:
-        f.write("x" * num)
-
-
-def dovecot_recalc_quota(user):
-    shell(f"doveadm quota recalc -u {user}")
-    output = shell(f"doveadm quota get -u {user}")
-    #
-    # Quota name Type    Value  Limit                                              %
-    # User quota STORAGE     5 102400                                              0
-    # User quota MESSAGE     2      -                                              0
-    #
-    for line in output.split("\n"):
-        parts = line.split()
-        if parts[2] == "STORAGE":
-            return dict(value=int(parts[3]), limit=int(parts[4]), percent=int(parts[5]))

cmdeploy/src/cmdeploy/service/filtermail-incoming.service.f

@@ -1,12 +0,0 @@
-[Unit]
-Description=Incoming Chatmail Postfix before queue filter 
-
-[Service]
-ExecStart={execpath} {config_path} incoming
-Restart=always
-RestartSec=30
-User=vmail
-
-[Install]
-WantedBy=multi-user.target
-

cmdeploy/src/cmdeploy/service/filtermail.service.f

@@ -1,11 +1,11 @@
 [Unit]
-Description=Outgoing Chatmail Postfix before queue filter
+Description=Chatmail Postfix before queue filter
 
 [Service]
-ExecStart={execpath} {config_path} outgoing
+ExecStart={execpath} {config_path}
 Restart=always
 RestartSec=30
-User=vmail
+User=filtermail
 
 [Install]
 WantedBy=multi-user.target

cmdeploy/src/cmdeploy/tests/online/benchmark.py

@@ -37,7 +37,7 @@ class TestDC:
 
     def test_ping_pong(self, benchmark, cmfactory):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def dc_ping_pong():
             chat.send_text("ping")
@@ -49,7 +49,7 @@ class TestDC:
 
     def test_send_10_receive_10(self, benchmark, cmfactory, lp):
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
 
         def dc_send_10_receive_10():
             for i in range(10):

cmdeploy/src/cmdeploy/tests/online/test_1_basic.py

@@ -116,8 +116,9 @@ def test_authenticated_from(cmsetup, maildata):
 
 @pytest.mark.parametrize("from_addr", ["fake@example.org", "fake@testrun.org"])
 def test_reject_missing_dkim(cmsetup, maildata, from_addr):
+    """Test that emails with missing or wrong DMARC, DKIM, and SPF entries are rejected."""
     recipient = cmsetup.gen_users(1)[0]
-    msg = maildata("encrypted.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
+    msg = maildata("plain.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
     with smtplib.SMTP(cmsetup.maildomain, 25) as s:
         with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):
             s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)

cmdeploy/src/cmdeploy/tests/online/test_2_deltachat.py

@@ -1,4 +1,5 @@
 import ipaddress
+import random
 import re
 import time
 
@@ -6,9 +7,6 @@ import imap_tools
 import pytest
 import requests
 
-from cmdeploy.remote import rshell
-from cmdeploy.sshexec import SSHExec
-
 
 @pytest.fixture
 def imap_mailbox(cmfactory):
@@ -56,23 +54,22 @@ class TestEndToEndDeltaChat:
         """Test that a DC account can send a message to a second DC account
         on the same chat-mail instance."""
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
+
+        lp.sec("ac1: prepare and send text message to ac2")
         chat.send_text("message0")
 
         lp.sec("wait for ac2 to receive message")
         msg2 = ac2._evtracker.wait_next_incoming_message()
         assert msg2.text == "message0"
 
-    def test_exceed_quota(
+    @pytest.mark.slow
-        self, cmfactory, lp, tmpdir, remote, chatmail_config, sshdomain
+    def test_exceed_quota(self, cmfactory, lp, tmpdir, remote, chatmail_config):
-    ):
         """This is a very slow test as it needs to upload >100MB of mail data
         before quota is exceeded, and thus depends on the speed of the upload.
         """
         ac1, ac2 = cmfactory.get_online_accounts(2)
-        chat = cmfactory.get_protected_chat(ac1, ac2)
+        chat = cmfactory.get_accepted_chat(ac1, ac2)
-
-        user = ac2.get_config("configured_addr")
 
         def parse_size_limit(limit: str) -> int:
             """Parse a size limit and return the number of bytes as integer.
@@ -85,27 +82,49 @@ class TestEndToEndDeltaChat:
             return int(float(number) * units[unit])
 
         quota = parse_size_limit(chatmail_config.max_mailbox_size)
+        attachsize = 1 * 1024 * 1024
+        num_to_send = quota // attachsize + 2
+        lp.sec(f"ac1: send {num_to_send} large files to ac2")
+        lp.indent(f"per-user quota is assumed to be: {quota / (1024 * 1024)}MB")
+        alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
+        msgs = []
+        for i in range(num_to_send):
+            attachment = tmpdir / f"attachment{i}"
+            data = "".join(random.choice(alphanumeric) for i in range(1024))
+            with open(attachment, "w+") as f:
+                for j in range(attachsize // len(data)):
+                    f.write(data)
 
-        lp.sec(f"filling remote inbox for {user}")
+            msg = chat.send_file(str(attachment))
-        fn = f"7743102289.M843172P2484002.c20,S={quota},W=2398:2,"
+            msgs.append(msg)
-        path = chatmail_config.mailboxes_dir.joinpath(user, "cur", fn)
+            lp.indent(f"Sent out msg {i}, size {attachsize / (1024 * 1024)}MB")
-        sshexec = SSHExec(sshdomain)
-        sshexec(call=rshell.write_numbytes, kwargs=dict(path=str(path), num=120))
-        res = sshexec(call=rshell.dovecot_recalc_quota, kwargs=dict(user=user))
-        assert res["percent"] >= 100
 
-        lp.sec("ac2: check quota is triggered")
+        lp.sec("ac2: check messages are arriving until quota is reached")
 
-        starting = True
+        addr = ac2.get_config("addr").lower()
+        saved_ok = 0
         for line in remote.iter_output("journalctl -n0 -f -u dovecot"):
-            if starting:
+            if addr not in line:
-                chat.send_text("hello")
-                starting = False
-            if user not in line:
                 # print(line)
                 continue
-            if "quota exceeded" in line:
+            if "quota" in line:
-                return
+                if "quota exceeded" in line:
+                    if saved_ok < num_to_send // 2:
+                        pytest.fail(
+                            f"quota exceeded too early: after {saved_ok} messages already"
+                        )
+                    lp.indent("good, message sending failed because quota was exceeded")
+                    return
+            if (
+                "stored mail into mailbox 'inbox'" in line
+                or "saved mail to inbox" in line
+            ):
+                saved_ok += 1
+                print(f"{saved_ok}: {line}")
+                if saved_ok >= num_to_send:
+                    break
+
+        pytest.fail("sending succeeded although messages should exceed quota")
 
     def test_securejoin(self, cmfactory, lp, maildomain2):
         ac1 = cmfactory.new_online_configuring_account(cache=False)
@@ -153,7 +172,7 @@ def test_hide_senders_ip_address(cmfactory):
     assert ipaddress.ip_address(public_ip)
 
     user1, user2 = cmfactory.get_online_accounts(2)
-    chat = cmfactory.get_protected_chat(user1, user2)
+    chat = cmfactory.get_accepted_chat(user1, user2)
 
     chat.send_text("testing submission header cleanup")
     user2._evtracker.wait_next_incoming_message()
@@ -162,18 +181,11 @@ def test_hide_senders_ip_address(cmfactory):
     assert public_ip not in msg.obj.as_string()
 
 
-def test_echobot(cmfactory, chatmail_config, lp, sshdomain):
+def test_echobot(cmfactory, chatmail_config, lp):
     ac = cmfactory.get_online_accounts(1)[0]
 
-    # establish contact with echobot
+    lp.sec(f"Send message to echo@{chatmail_config.mail_domain}")
-    sshexec = SSHExec(sshdomain)
+    chat = ac.create_chat(f"echo@{chatmail_config.mail_domain}")
-    command = "cat /var/lib/echobot/invite-link.txt"
-    echo_invite_link = sshexec(call=rshell.shell, kwargs=dict(command=command))
-    chat = ac.qr_setup_contact(echo_invite_link)
-    ac._evtracker.wait_securejoin_joiner_progress(1000)
-
-    # send message and check it gets replied back
-    lp.sec("Send message to echobot")
     text = "hi, I hope you text me back"
     chat.send_text(text)
     lp.sec("Wait for reply from echobot")

cmdeploy/src/cmdeploy/tests/plugin.py

@@ -62,7 +62,7 @@ def sshdomain(maildomain):
 def maildomain2():
     domain = os.environ.get("CHATMAIL_DOMAIN2")
     if not domain:
-        pytest.skip("set CHATMAIL_DOMAIN2 to a second chatmail server")
+        pytest.skip("set CHATMAIL_DOMAIN2 to a ssh-reachable chatmail instance")
     return domain
 
 
@@ -302,12 +302,10 @@ def cmfactory(request, gencreds, tmpdir, maildomain):
     pytest.importorskip("deltachat")
     from deltachat.testplugin import ACFactory
 
-    testproc = ChatmailTestProcess(request.config, maildomain, gencreds)
+    data = request.getfixturevalue("data")
 
-    class Data:
+    testproc = ChatmailTestProcess(request.config, maildomain, gencreds)
-        def read_path(self, path):
+    am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=data)
-            return
-    am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=Data())
 
     # nb. a bit hacky
     # would probably be better if deltachat's test machinery grows native support

www/src/info.md

@@ -6,6 +6,29 @@ interoperable e-mail service for everyone. What's behind a `chatmail` is
 effectively a normal e-mail address just like any other but optimized 
 for the usage in chats, especially DeltaChat.
 
+### Choosing a chatmail address instead of using a random one
+
+In the Delta Chat account setup you may tap `Create a profile` then `Use other server` and choose `Classic e-mail login`. Here fill the two fields like this: 
+
+- `E-Mail Address`: invent a word with
+{% if username_min_length == username_max_length %}
+  *exactly* {{ username_min_length }}
+{% else %}
+  {{ username_min_length}}
+  {% if username_max_length == "more" %}
+    or more
+  {% else %}
+    to {{ username_max_length }}
+  {% endif %}
+{% endif %}
+  characters
+  and append `@{{config.mail_domain}}` to it.
+
+- `Existing Password`: invent at least {{ password_min_length }} characters.
+
+If the e-mail address is not yet taken, you'll get that account. 
+The first login sets your password. 
+
 
 ### Rate and storage limits 
 
@@ -15,10 +38,9 @@ for the usage in chats, especially DeltaChat.
 
 - You may send up to {{ config.max_user_send_per_minute }} messages per minute.
 
-- You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
+- Messages are unconditionally removed {{ config.delete_mails_after }} days after arriving on the server.
 
-- Messages are unconditionally removed latest {{ config.delete_mails_after }} days after arriving on the server.
+- You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
-  Earlier, if storage may exceed otherwise.
 
 
 ### <a name="account-deletion"></a> Account deletion