Remove DKIM-Signature from incoming mail after checking

update some links

CHANGELOG.md

@@ -2,6 +2,21 @@
 
 ## untagged
 
+- Remove `DKIM-Signature` from incoming mails after verifying
+  ([#530](https://github.com/chatmail/server/pull/530))
+
+- Send SNI when connecting to outside servers
+  ([#524](https://github.com/chatmail/server/pull/524))
+
+- Pass through `original_content` instead of `content` in filtermail
+  ([#509](https://github.com/chatmail/server/pull/509))
+
+- Document TLS requirements in the readme
+  ([#514](https://github.com/chatmail/server/pull/514))
+
+- Remove cleanup service from submission ports
+  ([#512](https://github.com/chatmail/server/pull/512))
+
 - cmdeploy dovecot: delete big messages after 7 days
   ([#504](https://github.com/chatmail/server/pull/504))
 

README.md

@@ -1,21 +1,55 @@
 
 <img width="800px" src="www/src/collage-top.png"/>
 
-# Chatmail services optimized for Delta Chat apps 
+# Chatmail servers for secure instant messaging 
 
-This repository helps to setup a ready-to-use chatmail server
+Chatmail servers are minimal interoperable e-mail routing machines designed for:
+
+- **Convenience:** Instant onboarding, with optional Google/Apple/Huawei push notifications
+
+- **Privacy:** Just login, no questions asked, no name, numbers or e-mail needed
+
+- **Speed:** End-to-End Message delivery in well under a second
+
+- **Security:** Strict TLS, DKIM and OpenPGP with metadata-minimization enforced. 
+
+- **Relaxation:** No annoying spam-checking, IP reputation or rate limits
+
+- **Efficiency:** messages are only stored for transit and removed automatically. 
+
+This repository contains everything needed to setup a ready-to-use chatmail server
 comprised of a minimal setup of the battle-tested 
 [postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services. 
 
-The setup is designed and optimized for providing chatmail accounts 
-for use by [Delta Chat apps](https://delta.chat).
+The automated setup is designed and optimized for providing chatmail addresses 
+for immediate permission-free onboarding through chat apps and bots. 
+Chatmail addresses are automatically created by a first login, 
+after which the initially specified password is required 
+for sending and receiving messages through them. 
 
-Chatmail accounts are automatically created by a first login, 
-after which the initially specified password is required for using them. 
+Please see [this list of known apps and client projects](https://chatmail.at/apps.html) which offer instant onboarding on chatmail servers,
+and [this list of known public 3rd party chatmail servers](https://delta.chat/en/chatmail). 
 
-## Deploying your own chatmail server 
 
-To deploy chatmail on your own server, you must have set-up ssh authentication and need to use an ed25519 key, due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191). You also need to add your private key to the local ssh-agent, because you can't type in your password during deployment.
+## Minimal requirements, Prerequisites 
+
+You will need the following: 
+
+- control over a domain through a DNS provider of your choice, 
+
+- a remote Debian 12 machine with IPV4 and preferably also IPV6 addresses and
+  reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports. 
+  Machine needs 1GB RAM, one slow CPU and maybe 10GB storage for a
+  few thousand active chatmail addresses,
+
+- a terminal window with password-less ssh root login to the remote machine;
+  you must have set up ssh authentication and need to use an ed25519 key, 
+  due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191);
+  you also need to add your private key to the local ssh-agent, 
+  because you can't type in your password during deployment.
+
+
+## Getting started 
 
 We use `chat.example.org` as the chatmail domain in the following steps. 
 Please substitute it with your own domain. 
@@ -23,7 +57,7 @@ Please substitute it with your own domain.
 1. Install the `cmdeploy` command in a virtualenv
 
    ```
-    git clone https://github.com/deltachat/chatmail
+    git clone https://github.com/chatmail/server
     cd chatmail
     scripts/initenv.sh
    ```
@@ -82,11 +116,11 @@ scripts/cmdeploy bench
 
 This repository has four directories:
 
-- [cmdeploy](https://github.com/deltachat/chatmail/tree/main/cmdeploy)
+- [cmdeploy](https://github.com/chatmail/server/tree/main/cmdeploy)
   is a collection of configuration files
   and a [pyinfra](https://pyinfra.com)-based deployment script.
 
-- [chatmaild](https://github.com/deltachat/chatmail/tree/main/chatmaild)
+- [chatmaild](https://github.com/chatmail/server/tree/main/chatmaild)
   is a python package containing several small services
   which handle authentication,
   trigger push notifications on new messages,
@@ -95,12 +129,12 @@ This repository has four directories:
   and some other minor things.
   chatmaild can also be installed as a stand-alone python package.
 
-- [www](https://github.com/deltachat/chatmail/tree/main/www)
+- [www](https://github.com/chatmail/server/tree/main/www)
   contains the html, css, and markdown files
   which make up a chatmail server's web page.
   Edit them before deploying to make your chatmail server stand out.
 
-- [scripts](https://github.com/deltachat/chatmail/tree/main/scripts)
+- [scripts](https://github.com/chatmail/server/tree/main/scripts)
   offers two convenience tools for beginners;
   `initenv.sh` installs the necessary dependencies to a local virtual environment,
   and the `scripts/cmdeploy` script enables you
@@ -139,39 +173,39 @@ If you deploy them with cmdeploy,
 they are run by systemd services in the background.
 A short overview:
 
-- [`doveauth`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/doveauth.py) implements
+- [`doveauth`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/doveauth.py) implements
   create-on-login account creation semantics and is used
   by Dovecot during login authentication and by Postfix
   which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
   to authenticate users
   to send mails for them.
 
-- [`filtermail`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/filtermail.py) prevents
+- [`filtermail`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/filtermail.py) prevents
   unencrypted e-mail from leaving the chatmail service
   and is integrated into postfix's outbound mail pipelines.
 
-- [`chatmail-metadata`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
-  [dovecot lua script](https://github.com/deltachat/chatmail/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
+- [`chatmail-metadata`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
+  [dovecot lua script](https://github.com/chatmail/server/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
   to store user-specific server-side config.
   On new messages,
-  it [passes the user's push notification token](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/notifier.py)
+  it [passes the user's push notification token](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/notifier.py)
   to [notifications.delta.chat](https://delta.chat/help#instant-delivery)
   so the push notifications on the user's phone can be triggered
   by Apple/Google.
 
-- [`delete_inactive_users`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
+- [`delete_inactive_users`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
   deletes users if they have not logged in for a very long time.
   The timeframe can be configured in `chatmail.ini`.
 
-- [`lastlogin`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/lastlogin.py)
+- [`lastlogin`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/lastlogin.py)
   is contacted by dovecot when a user logs in
   and stores the date of the login.
 
-- [`echobot`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/echo.py)
+- [`echobot`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/echo.py)
   is a small bot for test purposes.
   It simply echoes back messages from users.
 
-- [`chatmail-metrics`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/metrics.py)
+- [`chatmail-metrics`](https://github.com/chatmail/server/blob/main/chatmaild/src/chatmaild/metrics.py)
   collects some metrics and displays them at `https://example.org/metrics`.
 
 ### Home page and getting started for users
@@ -228,8 +262,8 @@ While this file is present, account creation will be blocked.
 Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
 [acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (http).
 
-Delta Chat apps will, however, discover all ports and configurations
-automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail service.
+Chatmail-core based apps will, however, discover all ports and configurations
+automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail server.
 
 ## Email authentication
 
@@ -256,6 +290,32 @@ and rejects incorrectly authenticated emails with [`reject_sender_login_mismatch
 `From:` header must correspond to envelope MAIL FROM,
 this is ensured by `filtermail` proxy.
 
+## TLS requirements
+
+Postfix is configured to require valid TLS
+by setting [`smtp_tls_security_level`](https://www.postfix.org/postconf.5.html#smtp_tls_security_level) to `verify`.
+If emails don't arrive from a chatmail server to your server,
+the problem is likely that your server does not have a valid TLS certificate.
+
+You can test it by resolving `MX` records of your server domain
+and then connecting to MX servers (e.g `mx.example.org`) with
+`openssl s_client -connect mx.example.org:25 -verify_hostname mx.example.org -verify_return_error -starttls smtp`
+from the host that has open port 25 to verify that certificate is valid.
+
+When providing a TLS certificate to your server,
+make sure to provide the full certificate chain
+and not just the last certificate.
+
+If you are running Exim server and don't see incoming connections
+from a chatmail server in the logs,
+make sure `smtp_no_mail` log item is enabled in the config
+with `log_selector = +smtp_no_mail`.
+By default Exim does not log sessions that are closed
+before sending the `MAIL` command.
+This happens if certificate is not recognized as valid by Postfix,
+so you might think that connection is not established
+while actually it is a problem with your TLS certificate.
+
 ## Migrating chatmail server to a new host
 
 If you want to migrate chatmail from an old machine
@@ -290,9 +350,9 @@ to make sure you can connect with SSH.
 5. Now, point DNS to the new IP addresses.
 
    You can already remove the old IP addresses from DNS.
-   Existing Delta Chat users will still be able to connect
+   Existing Chatmail app users or bots will still be able to connect
    to the old server, send and receive messages,
-   but new users will fail to create new profiles
+   but new ones will fail to create new profiles
    with your chatmail server.
 
    If other servers try to deliver messages to your new server they will fail,

chatmaild/src/chatmaild/filtermail.py

@@ -196,7 +196,9 @@ class BeforeQueueHandler:
             return error
         logging.info("re-injecting the mail that passed checks")
         client = SMTPClient("localhost", self.config.postfix_reinject_port)
-        client.sendmail(envelope.mail_from, envelope.rcpt_tos, envelope.content)
+        client.sendmail(
+            envelope.mail_from, envelope.rcpt_tos, envelope.original_content
+        )
         return "250 OK"
 
     def check_DATA(self, envelope):

cmdeploy/src/cmdeploy/__init__.py

@@ -215,7 +215,7 @@ def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
         server.shell(
             name="Generate OpenDKIM domain keys",
             commands=[
-                f"opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
+                f"/usr/sbin/opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
             ],
             _use_su_login=True,
             _su_user="opendkim",
@@ -228,7 +228,6 @@ def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
     )
     need_restart |= service_file.changed
 
-
     return need_restart
 
 
@@ -275,7 +274,18 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
     )
     need_restart |= master_config.changed
 
-    header_cleanup = files.put(
+    incoming_header_cleanup = files.put(
+        src=importlib.resources.files(__package__).joinpath(
+            "postfix/incoming_header_cleanup"
+        ),
+        dest="/etc/postfix/incoming_header_cleanup",
+        user="root",
+        group="root",
+        mode="644",
+    )
+    need_restart |= incoming_header_cleanup.changed
+
+    submission_header_cleanup = files.put(
         src=importlib.resources.files(__package__).joinpath(
             "postfix/submission_header_cleanup"
         ),
@@ -284,7 +294,7 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
         group="root",
         mode="644",
     )
-    need_restart |= header_cleanup.changed
+    need_restart |= submission_header_cleanup.changed
 
     # Login map that 1:1 maps email address to login.
     login_map = files.put(

cmdeploy/src/cmdeploy/nginx/nginx.conf.j2

@@ -46,10 +46,7 @@ http {
 
 	server {
   
-		listen 8443 ssl default_server;
-		{% if not disable_ipv6 %}
-		listen [::]:8443 ssl default_server;
-		{% endif %}
+		listen 127.0.0.1:8443 ssl default_server;
 
 		root /var/www/html;
 

cmdeploy/src/cmdeploy/postfix/incoming_header_cleanup

@@ -0,0 +1 @@
+/^DKIM-Signature:/            IGNORE

cmdeploy/src/cmdeploy/postfix/main.cf.j2

@@ -21,6 +21,9 @@ smtpd_tls_security_level=may
 
 smtp_tls_CApath=/etc/ssl/certs
 smtp_tls_security_level=verify
+# Send SNI extension when connecting to other servers.
+# <https://www.postfix.org/postconf.5.html#smtp_tls_servername>
+smtp_tls_servername = hostname
 smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
 smtp_tls_policy_maps = inline:{nauta.cu=may}
 smtpd_tls_protocols = >=TLSv1.2

cmdeploy/src/cmdeploy/postfix/master.cf.j2

@@ -32,7 +32,6 @@ submission inet n       -       y       -       5000    smtpd
   -o milter_macro_daemon_name=ORIGINATING
   -o smtpd_client_connection_count_limit=1000
   -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
-  -o cleanup_service_name=authclean
 smtps     inet  n       -       y       -       5000    smtpd
   -o syslog_name=postfix/smtps
   -o smtpd_tls_wrappermode=yes
@@ -50,10 +49,10 @@ smtps     inet  n       -       y       -       5000    smtpd
   -o smtpd_client_connection_count_limit=1000
   -o milter_macro_daemon_name=ORIGINATING
   -o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
-  -o cleanup_service_name=authclean
 #628       inet  n       -       y       -       -       qmqpd
 pickup    unix  n       -       y       60      1       pickup
 cleanup   unix  n       -       y       -       0       cleanup
+  -o header_checks=regexp:/etc/postfix/incoming_header_cleanup
 qmgr      unix  n       -       n       300     1       qmgr
 #qmgr     unix  n       -       n       300     1       oqmgr
 tlsmgr    unix  -       -       y       1000?   1       tlsmgr
@@ -82,6 +81,7 @@ filter    unix -        n       n       -       -       lmtp
 localhost:{{ config.postfix_reinject_port }} inet  n       -       n       -       10      smtpd
   -o syslog_name=postfix/reinject
   -o smtpd_milters=unix:opendkim/opendkim.sock
+  -o cleanup_service_name=authclean
 
 # Cleanup `Received` headers for authenticated mail
 # to avoid leaking client IP.
@@ -89,5 +89,10 @@ localhost:{{ config.postfix_reinject_port }} inet  n       -       n       -
 # We do not do this for received mails
 # as this will break DKIM signatures
 # if `Received` header is signed.
+#
+# This service also rewrites
+# Subject with `[...]`
+# to make sure the users
+# cannot send unprotected Subject.
 authclean unix  n       -       -       -       0       cleanup
   -o header_checks=regexp:/etc/postfix/submission_header_cleanup