Compare commits

..

3 Commits

Author SHA1 Message Date
missytake
f26d2da23b Try without gcc 2025-01-27 11:37:07 +01:00
link2xt
3b32387bf8 Install python3-dev 2025-01-26 20:10:12 +00:00
link2xt
a651fba8b6 Install gcc 2025-01-26 19:58:48 +00:00
79 changed files with 755 additions and 2764 deletions

View File

@@ -1,33 +0,0 @@
---
name: Bug report
about: Report something that isn't working.
title: ''
assignees: ''
---
<!--
Please fill out as much of this form as you can (leaving out stuff that is not applicable is ok).
-->
- Server OS (Operating System) - preferably Debian 12:
- On which OS you run cmdeploy:
- chatmail/relay version: `git rev-parse HEAD`
## Expected behavior
*What did you try to achieve?*
## Actual behavior
*What happened instead?*
### Steps to reproduce the problem:
1.
2.
### Screenshots
### Logs

View File

@@ -1 +0,0 @@
blank_issues_enabled: true

View File

@@ -10,10 +10,6 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# Checkout pull request HEAD commit instead of merge commit
# Otherwise `test_deployed_state` will be unhappy.
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: run chatmaild tests
working-directory: chatmaild

View File

@@ -38,8 +38,8 @@ jobs:
if [ -f dkimkeys-ipv4/dkimkeys/opendkim.private ]; then rsync -avz -e "ssh -o StrictHostKeyChecking=accept-new" dkimkeys-ipv4 root@ns.testrun.org:/tmp/ || true; fi
if [ "$(ls -A acme-ipv4/acme/certs)" ]; then rsync -avz -e "ssh -o StrictHostKeyChecking=accept-new" acme-ipv4 root@ns.testrun.org:/tmp/ || true; fi
# make sure CAA record isn't set
scp -o StrictHostKeyChecking=accept-new .github/workflows/staging-ipv4.testrun.org-default.zone root@ns.testrun.org:/etc/nsd/staging-ipv4.testrun.org.zone
ssh root@ns.testrun.org sed -i '/CAA/d' /etc/nsd/staging-ipv4.testrun.org.zone
scp .github/workflows/staging-ipv4.testrun.org-default.zone root@ns.testrun.org:/etc/nsd/staging-ipv4.testrun.org.zone
ssh -o StrictHostKeyChecking=accept-new root@ns.testrun.org sed -i '/CAA/d' /etc/nsd/staging-ipv4.testrun.org.zone
ssh root@ns.testrun.org nsd-checkzone staging-ipv4.testrun.org /etc/nsd/staging-ipv4.testrun.org.zone
ssh root@ns.testrun.org systemctl reload nsd
@@ -49,7 +49,7 @@ jobs:
-H "Authorization: Bearer ${{ secrets.HETZNER_API_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"image":"debian-12"}' \
"https://api.hetzner.cloud/v1/servers/${{ secrets.STAGING_IPV4_SERVER_ID }}/actions/rebuild"
"https://api.hetzner.cloud/v1/servers/${{ secrets.STAGING_SERVER_ID }}/actions/rebuild"
- run: scripts/initenv.sh
@@ -63,13 +63,16 @@ jobs:
while ! ssh -o ConnectTimeout=180 -o StrictHostKeyChecking=accept-new -v root@staging-ipv4.testrun.org id -u ; do sleep 1 ; done
ssh -o StrictHostKeyChecking=accept-new -v root@staging-ipv4.testrun.org id -u
# download acme & dkim state from ns.testrun.org
rsync -e "ssh -o StrictHostKeyChecking=accept-new" -avz root@ns.testrun.org:/tmp/acme-ipv4/acme acme-restore || true
rsync -avz root@ns.testrun.org:/tmp/dkimkeys-ipv4/dkimkeys dkimkeys-restore || true
rsync -e "ssh -o StrictHostKeyChecking=accept-new" -avz root@ns.testrun.org:/tmp/acme-ipv4 acme-restore || true
rsync -avz root@ns.testrun.org:/tmp/dkimkeys-ipv4 dkimkeys-restore || true
# restore acme & dkim state to staging2.testrun.org
rsync -avz acme-restore/acme root@staging-ipv4.testrun.org:/var/lib/ || true
rsync -avz dkimkeys-restore/dkimkeys root@staging-ipv4.testrun.org:/etc/ || true
rsync -avz acme-restore/acme-ipv4/acme root@staging-ipv4.testrun.org:/var/lib/ || true
rsync -avz dkimkeys-restore/dkimkeys-ipv4/dkimkeys root@staging-ipv4.testrun.org:/etc/ || true
ssh -o StrictHostKeyChecking=accept-new -v root@staging-ipv4.testrun.org chown root:root -R /var/lib/acme || true
- name: run formatting checks
run: cmdeploy fmt -v
- name: run deploy-chatmail offline tests
run: pytest --pyargs cmdeploy
@@ -77,7 +80,7 @@ jobs:
cmdeploy init staging-ipv4.testrun.org
sed -i 's#disable_ipv6 = False#disable_ipv6 = True#' chatmail.ini
- run: cmdeploy run --verbose --skip-dns-check
- run: cmdeploy run
- name: set DNS entries
run: |

View File

@@ -70,12 +70,15 @@ jobs:
rsync -avz dkimkeys-restore/dkimkeys root@staging2.testrun.org:/etc/ || true
ssh -o StrictHostKeyChecking=accept-new -v root@staging2.testrun.org chown root:root -R /var/lib/acme || true
- name: run formatting checks
run: cmdeploy fmt -v
- name: run deploy-chatmail offline tests
run: pytest --pyargs cmdeploy
- run: cmdeploy init staging2.testrun.org
- run: cmdeploy run --verbose --skip-dns-check
- run: cmdeploy run --verbose
- name: set DNS entries
run: |

View File

@@ -1,50 +0,0 @@
This diagram shows components of the chatmail server; this is a draft
overview as of mid-August 2025:
```mermaid
graph LR;
cmdeploy --- sshd;
letsencrypt --- |80|acmetool-redirector;
acmetool-redirector --- |443|nginx-right(["`nginx
(external)`"]);
nginx-external --- |465|postfix;
nginx-external(["`nginx
(external)`"]) --- |8443|nginx-internal["`nginx
(internal)`"];
nginx-internal --- website["`Website
/var/www/html`"];
nginx-internal --- newemail.py;
nginx-internal --- autoconfig.xml;
certs-nginx[("`TLS certs
/var/lib/acme`")] --> nginx-internal;
cron --- chatmail-metrics;
cron --- acmetool;
chatmail-metrics --- website;
acmetool --> certs[("`TLS certs
/var/lib/acme`")];
nginx-external --- |993|dovecot;
autoconfig.xml --- postfix;
autoconfig.xml --- dovecot;
postfix --- echobot;
postfix --- |10080,10081|filtermail;
postfix --- users["`User data
home/vmail/mail`"];
postfix --- |doveauth.socket|doveauth;
dovecot --- |doveauth.socket|doveauth;
dovecot --- users;
dovecot --- |metadata.socket|chatmail-metadata;
doveauth --- users;
chatmail-expire-daily --- users;
chatmail-fsreport-daily --- users;
chatmail-metadata --- iroh-relay;
certs-nginx --> postfix;
certs-nginx --> dovecot;
style certs fill:#ff6;
style certs-nginx fill:#ff6;
style nginx-external fill:#fc9;
style nginx-right fill:#fc9;
```
The edges in this graph should not be taken too literally; they
reflect some sort of communication path or dependency relationship
between components of the chatmail server.

View File

@@ -2,236 +2,42 @@
## untagged
- acmetool: use ECDSA keys instead of RSA
([#689](https://github.com/chatmail/relay/pull/689))
- Require TLS 1.2 for outgoing SMTP connections
([#685](https://github.com/chatmail/relay/pull/685))
- require STARTTLS for incoming port 25 connections
([#684](https://github.com/chatmail/relay/pull/684))
- filtermail: run CPU-intensive handle_DATA in a thread pool executor
([#676](https://github.com/chatmail/relay/pull/676))
- don't use the complicated logging module in filtermail to exclude a potential source of errors.
([#674](https://github.com/chatmail/relay/pull/674))
- Specify nginx.conf to only handle `mail_domain`, www, and mta-sts domains
([#636](https://github.com/chatmail/relay/pull/636))
- Setup TURN server
([#621](https://github.com/chatmail/relay/pull/621))
- cmdeploy: make --ssh-host work with localhost
([#659](https://github.com/chatmail/relay/pull/659))
- Update iroh-relay to 0.35.0
([#650](https://github.com/chatmail/relay/pull/650))
- filtermail: accept mails from Protonmail
([#616](https://github.com/chatmail/relay/pull/616))
- Ignore all RCPT TO: parameters
([#651](https://github.com/chatmail/relay/pull/651))
- Increase opendkim DNS Timeout from 5 to 60 seconds
([#672](https://github.com/chatmail/relay/pull/672))
- Add config parameter for Let's Encrypt ACME email
([#663](https://github.com/chatmail/relay/pull/663))
- Use max username length in newemail.py, not min
([#648](https://github.com/chatmail/relay/pull/648))
- Add startup for `fcgiwrap.service` because sometimes it did not start automatically.
([#657](https://github.com/chatmail/relay/pull/657))
- Add `cmdeploy init --force` command for recreating chatmail.ini
([#656](https://github.com/chatmail/relay/pull/656))
- Increase maxproc for reinjecting ports from 10 to 100
([#646](https://github.com/chatmail/relay/pull/646))
- Allow ports 143 and 993 to be used by `dovecot` process
([#639](https://github.com/chatmail/relay/pull/639))
- Add `--skip-dns-check` argument to `cmdeploy run` command, which disables DNS record checking before installation.
([#661](https://github.com/chatmail/relay/pull/661))
- Rework expiry of message files and mailboxes in Python
to only do a single iteration over sometimes millions of messages
instead of doing "find" commands that iterate 9 times over the messages.
Provide an "fsreport" CLI for more fine grained analysis of message files.
([#637](https://github.com/chatmail/relay/pull/637))
## 1.7.0 2025-09-11
- Make www upload path configurable
([#618](https://github.com/chatmail/relay/pull/618))
- Check whether GCC is installed in initenv.sh
([#608](https://github.com/chatmail/relay/pull/608))
- Expire push notification tokens after 90 days
([#583](https://github.com/chatmail/relay/pull/583))
- Use official `mtail` binary instead of `mtail` package
([#581](https://github.com/chatmail/relay/pull/581))
- dovecot: install from download.delta.chat instead of openSUSE Build Service
([#590](https://github.com/chatmail/relay/pull/590))
- Reconfigure Dovecot imap-login service to high-performance mode
([#578](https://github.com/chatmail/relay/pull/578))
- Set timezone to improve dovecot performance
([#584](https://github.com/chatmail/relay/pull/584))
- Increase nginx connection limits
([#576](https://github.com/chatmail/relay/pull/576))
- If `dns-utils` needs to be installed before cmdeploy run, apt update to make sure it works
([#560](https://github.com/chatmail/relay/pull/560))
- filtermail: respect config message size limit
([#572](https://github.com/chatmail/relay/pull/572))
- Don't deploy if one of the ports used for chatmail relay services is occupied by an unexpected process
([#568](https://github.com/chatmail/relay/pull/568))
- Add config value after how many days large files are deleted
([#555](https://github.com/chatmail/relay/pull/555))
- cmdeploy: push relay version to /etc/chatmail-version
([#573](https://github.com/chatmail/relay/pull/573))
- filtermail: allow partial body length in OpenPGP payloads
([#570](https://github.com/chatmail/relay/pull/570))
- chatmaild: allow echobot to receive unencrypted messages by default
([#556](https://github.com/chatmail/relay/pull/556))
## 1.6.0 2025-04-11
- Handle Port-25 connect errors more gracefully (common with VPNs)
([#552](https://github.com/chatmail/relay/pull/552))
- Avoid "acmetool not found" during initial run
([#550](https://github.com/chatmail/relay/pull/550))
- Fix timezone handling such that client/servers do not need to use
same timezone.
([#553](https://github.com/chatmail/relay/pull/553))
- Enforce end-to-end encryption for incoming messages.
New user address mailboxes now get a `enforceE2EEincoming` file
which prohibits incoming cleartext messages from other domains.
An outside MTA trying to submit a cleartext message will
get a "523 Encryption Needed" response, see RFC5248.
If the file does not exist (as it the case for all existing accounts)
incoming cleartext messages are accepted.
([#538](https://github.com/chatmail/server/pull/538))
- Enforce end-to-end encryption between local addresses
([#535](https://github.com/chatmail/server/pull/535))
- unbound: check that port 53 is not occupied by a different process
([#537](https://github.com/chatmail/server/pull/537))
- unbound: before unbound is there, use 9.9.9.9 for resolving
([#518](https://github.com/chatmail/relay/pull/518))
- Limit the bind for the HTTPS server on 8443 to 127.0.0.1
([#522](https://github.com/chatmail/server/pull/522))
([#532](https://github.com/chatmail/server/pull/532))
- Send SNI when connecting to outside servers
([#524](https://github.com/chatmail/server/pull/524))
- postfix master.cf: use 127.0.0.1 for consistency
([#544](https://github.com/chatmail/relay/pull/544))
- Pass through `original_content` instead of `content` in filtermail
([#509](https://github.com/chatmail/server/pull/509))
- Document TLS requirements in the readme
([#514](https://github.com/chatmail/server/pull/514))
- Remove cleanup service from submission ports
([#512](https://github.com/chatmail/server/pull/512))
- cmdeploy dovecot: delete big messages after 7 days
([#504](https://github.com/chatmail/server/pull/504))
- mtail: fix getting logs from STDIN
([#502](https://github.com/chatmail/server/pull/502))
- filtermail: don't require exactly 2 lines after openPGP payload
([#497](https://github.com/chatmail/server/pull/497))
- cmdeploy dns: offer alternative DKIM record format for some web interfaces
([#470](https://github.com/chatmail/server/pull/470))
- journald: remove old logs from disk
([#490](https://github.com/chatmail/server/pull/490))
- opendkim: restart once every day to mend RAM leaks
([#498](https://github.com/chatmail/server/pull/498)
([#470](https://github.com/deltachat/chatmail/pull/470))
- migration guide: let opendkim own the DKIM keys directory
([#468](https://github.com/chatmail/server/pull/468))
- improve secure-join message detection
([#473](https://github.com/chatmail/server/pull/473))
- use old crypt lib in python < 3.11
([#483](https://github.com/chatmail/server/pull/483))
- chatmaild: set umask to 0700 for doveauth + metadata
([#490](https://github.com/chatmail/server/pull/492))
- remove MTA-STS daemon
([#488](https://github.com/chatmail/server/pull/488))
- replace `Subject` with `[...]` for all outgoing mails.
([#481](https://github.com/chatmail/server/pull/481))
- opendkim: use su instead of sudo
([#491](https://github.com/chatmail/server/pull/491))
([#468](https://github.com/deltachat/chatmail/pull/468))
## 1.5.0 2024-12-20
- cmdeploy dns: always show recommended DNS records
([#463](https://github.com/chatmail/server/pull/463))
([#463](https://github.com/deltachat/chatmail/pull/463))
- add `--all` to `cmdeploy dns`
([#462](https://github.com/chatmail/server/pull/462))
([#462](https://github.com/deltachat/chatmail/pull/462))
- fix `_mta-sts` TXT DNS record
([#461](https://github.com/chatmail/server/pull/461)
([#461](https://github.com/deltachat/chatmail/pull/461)
- deploy `iroh-relay` and also update "realtime relay services" in privacy policy.
([#434](https://github.com/chatmail/server/pull/434))
([#451](https://github.com/chatmail/server/pull/451))
([#434](https://github.com/deltachat/chatmail/pull/434))
([#451](https://github.com/deltachat/chatmail/pull/451))
- add guide to migrate chatmail to a new server
([#429](https://github.com/chatmail/server/pull/429))
([#429](https://github.com/deltachat/chatmail/pull/429))
- disable anvil authentication penalty
([#414](https://github.com/chatmail/server/pull/444)
([#414](https://github.com/deltachat/chatmail/pull/444)
- increase `request_queue_size` for UNIX sockets to 1000.
([#437](https://github.com/chatmail/server/pull/437))
([#437](https://github.com/deltachat/chatmail/pull/437))
- add argument to `cmdeploy run` for specifying
a different SSH host than `mail_domain`
([#439](https://github.com/chatmail/server/pull/439))
([#439](https://github.com/deltachat/chatmail/pull/439))
- query autoritative nameserver to bypass DNS cache
([#424](https://github.com/chatmail/server/pull/424))
([#424](https://github.com/deltachat/chatmail/pull/424))
- add mtail support (new optional `mtail_address` ini value)
This defines the address on which [`mtail`](https://google.github.io/mtail/)
@@ -241,195 +47,195 @@
and assign an IP address from this network to the host.
If you do not plan to collect metrics,
keep this setting unset.
([#388](https://github.com/chatmail/server/pull/388))
([#388](https://github.com/deltachat/chatmail/pull/388))
- fix checking for required DNS records
([#412](https://github.com/chatmail/server/pull/412))
([#412](https://github.com/deltachat/chatmail/pull/412))
- add support for specifying whole domains for recipient passthrough list
([#408](https://github.com/chatmail/server/pull/408))
([#408](https://github.com/deltachat/chatmail/pull/408))
- add a paragraph about "account deletion" to info page
([#405](https://github.com/chatmail/server/pull/405))
([#405](https://github.com/deltachat/chatmail/pull/405))
- avoid nginx listening on ipv6 if v6 is dsiabled
([#402](https://github.com/chatmail/server/pull/402))
([#402](https://github.com/deltachat/chatmail/pull/402))
- refactor ssh-based execution to allow organizing remote functions in
modules.
([#396](https://github.com/chatmail/server/pull/396))
([#396](https://github.com/deltachat/chatmail/pull/396))
- trigger "apt upgrade" during "cmdeploy run"
([#398](https://github.com/chatmail/server/pull/398))
([#398](https://github.com/deltachat/chatmail/pull/398))
- drop hispanilandia passthrough address
([#401](https://github.com/chatmail/server/pull/401))
([#401](https://github.com/deltachat/chatmail/pull/401))
- set CAA record flags to 0
- add IMAP capabilities instead of overwriting them
([#413](https://github.com/chatmail/server/pull/413))
([#413](https://github.com/deltachat/chatmail/pull/413))
- fix OpenPGP payload check
([#435](https://github.com/chatmail/server/pull/435))
([#435](https://github.com/deltachat/chatmail/pull/435))
- fix Dovecot quota_max_mail_size to use max_message_size config value
([#438](https://github.com/chatmail/server/pull/438))
([#438](https://github.com/deltachat/chatmail/pull/438))
## 1.4.1 2024-07-31
- fix metadata dictproxy which would confuse transactions
resulting in missed notifications and other issues.
([#393](https://github.com/chatmail/server/pull/393))
([#394](https://github.com/chatmail/server/pull/394))
([#393](https://github.com/deltachat/chatmail/pull/393))
([#394](https://github.com/deltachat/chatmail/pull/394))
- add optional "imap_rawlog" config option. If true,
.in/.out files are created in user home dirs
containing the imap protocol messages.
([#389](https://github.com/chatmail/server/pull/389))
([#389](https://github.com/deltachat/chatmail/pull/389))
## 1.4.0 2024-07-28
- Add `disable_ipv6` config option to chatmail.ini.
Required if the server doesn't have IPv6 connectivity.
([#312](https://github.com/chatmail/server/pull/312))
([#312](https://github.com/deltachat/chatmail/pull/312))
- allow current K9/Thunderbird-mail releases to send encrypted messages
outside by accepting their localized "encrypted subject" strings.
([#370](https://github.com/chatmail/server/pull/370))
([#370](https://github.com/deltachat/chatmail/pull/370))
- Migrate and remove sqlite database in favor of password/lastlogin tracking
in a user's maildir.
([#379](https://github.com/chatmail/server/pull/379))
([#379](https://github.com/deltachat/chatmail/pull/379))
- Require pyinfra V3 installed on the client side,
run `./scripts/initenv.sh` to upgrade locally.
([#378](https://github.com/chatmail/server/pull/378))
([#378](https://github.com/deltachat/chatmail/pull/378))
- don't hardcode "/home/vmail" paths but rather set them
once in the config object and use it everywhere else,
thereby also improving testability.
([#351](https://github.com/chatmail/server/pull/351))
([#351](https://github.com/deltachat/chatmail/pull/351))
temporarily introduced obligatory "passdb_path" and "mailboxes_dir"
settings but they were removed/obsoleted in
([#380](https://github.com/chatmail/server/pull/380))
([#380](https://github.com/deltachat/chatmail/pull/380))
- BREAKING: new required chatmail.ini value 'delete_inactive_users_after = 100'
which removes users from database and mails after 100 days without any login.
([#350](https://github.com/chatmail/server/pull/350))
([#350](https://github.com/deltachat/chatmail/pull/350))
- Refine DNS checking to distinguish between "required" and "recommended" settings
([#372](https://github.com/chatmail/server/pull/372))
([#372](https://github.com/deltachat/chatmail/pull/372))
- reload nginx in the acmetool cronjob
([#360](https://github.com/chatmail/server/pull/360))
([#360](https://github.com/deltachat/chatmail/pull/360))
- remove checking of reverse-DNS PTR records. Chatmail-servers don't
depend on it and even in the wider e-mail system it's not common anymore.
If it's an issue, a chatmail operator can still care to properly set reverse DNS.
([#348](https://github.com/chatmail/server/pull/348))
([#348](https://github.com/deltachat/chatmail/pull/348))
- Make DNS-checking faster and more interactive, run it fully during "cmdeploy run",
also introducing a generic mechanism for rapid remote ssh-based python function execution.
([#346](https://github.com/chatmail/server/pull/346))
([#346](https://github.com/deltachat/chatmail/pull/346))
- Don't fix file owner ship of /home/vmail
([#345](https://github.com/chatmail/server/pull/345))
([#345](https://github.com/deltachat/chatmail/pull/345))
- Support iterating over all users with doveadm commands
([#344](https://github.com/chatmail/server/pull/344))
([#344](https://github.com/deltachat/chatmail/pull/344))
- Test and fix for attempts to create inadmissible accounts
([#333](https://github.com/chatmail/server/pull/321))
([#333](https://github.com/deltachat/chatmail/pull/321))
- check that OpenPGP has only PKESK, SKESK and SEIPD packets
([#323](https://github.com/chatmail/server/pull/323),
[#324](https://github.com/chatmail/server/pull/324))
([#323](https://github.com/deltachat/chatmail/pull/323),
[#324](https://github.com/deltachat/chatmail/pull/324))
- improve filtermail checks for encrypted messages and drop support for unencrypted MDNs
([#320](https://github.com/chatmail/server/pull/320))
([#320](https://github.com/deltachat/chatmail/pull/320))
- replace `bash` with `/bin/sh`
([#334](https://github.com/chatmail/server/pull/334))
([#334](https://github.com/deltachat/chatmail/pull/334))
- Increase number of logged in IMAP sessions to 50000
([#335](https://github.com/chatmail/server/pull/335))
([#335](https://github.com/deltachat/chatmail/pull/335))
- filtermail: do not allow ASCII armor without actual payload
([#325](https://github.com/chatmail/server/pull/325))
([#325](https://github.com/deltachat/chatmail/pull/325))
- Remove sieve to enable hardlink deduplication in LMTP
([#343](https://github.com/chatmail/server/pull/343))
([#343](https://github.com/deltachat/chatmail/pull/343))
- dovecot: enable gzip compression on disk
([#341](https://github.com/chatmail/server/pull/341))
([#341](https://github.com/deltachat/chatmail/pull/341))
- DKIM-sign Content-Type and oversign all signed headers
([#296](https://github.com/chatmail/server/pull/296))
([#296](https://github.com/deltachat/chatmail/pull/296))
- Add nonci_accounts metric
([#347](https://github.com/chatmail/server/pull/347))
([#347](https://github.com/deltachat/chatmail/pull/347))
- doveauth: log when a new account is created
([#349](https://github.com/chatmail/server/pull/349))
([#349](https://github.com/deltachat/chatmail/pull/349))
- Multiplex HTTPS, IMAP and SMTP on port 443
([#357](https://github.com/chatmail/server/pull/357))
([#357](https://github.com/deltachat/chatmail/pull/357))
## 1.3.0 - 2024-06-06
- don't check necessary DNS records on cmdeploy init anymore
([#316](https://github.com/chatmail/server/pull/316))
([#316](https://github.com/deltachat/chatmail/pull/316))
- ensure cron and acl are installed
([#293](https://github.com/chatmail/server/pull/293),
[#310](https://github.com/chatmail/server/pull/310))
([#293](https://github.com/deltachat/chatmail/pull/293),
[#310](https://github.com/deltachat/chatmail/pull/310))
- change default for delete_mails_after from 40 to 20 days
([#300](https://github.com/chatmail/server/pull/300))
([#300](https://github.com/deltachat/chatmail/pull/300))
- save journald logs only to memory and save nginx logs to journald instead of file
([#299](https://github.com/chatmail/server/pull/299))
([#299](https://github.com/deltachat/chatmail/pull/299))
- fix writing of multiple obs repositories in `/etc/apt/sources.list`
([#290](https://github.com/chatmail/server/pull/290))
([#290](https://github.com/deltachat/chatmail/pull/290))
- metadata: add support for `/shared/vendor/deltachat/irohrelay`
([#284](https://github.com/chatmail/server/pull/284))
([#284](https://github.com/deltachat/chatmail/pull/284))
- Emit "XCHATMAIL" capability from IMAP server
([#278](https://github.com/chatmail/server/pull/278))
([#278](https://github.com/deltachat/chatmail/pull/278))
- Move echobot `into /var/lib/echobot`
([#281](https://github.com/chatmail/server/pull/281))
([#281](https://github.com/deltachat/chatmail/pull/281))
- Accept Let's Encrypt's new Terms of Services
([#275](https://github.com/chatmail/server/pull/276))
([#275](https://github.com/deltachat/chatmail/pull/276))
- Reload Dovecot and Postfix when TLS certificate updates
([#271](https://github.com/chatmail/server/pull/271))
([#271](https://github.com/deltachat/chatmail/pull/271))
- Use forked version of dovecot without hardcoded delays
([#270](https://github.com/chatmail/server/pull/270))
([#270](https://github.com/deltachat/chatmail/pull/270))
## 1.2.0 - 2024-04-04
- Install dig on the server to resolve DNS records
([#267](https://github.com/chatmail/server/pull/267))
([#267](https://github.com/deltachat/chatmail/pull/267))
- preserve notification order and exponentially backoff with
retries for tokens where we didn't get a successful return
([#265](https://github.com/chatmail/server/pull/263))
([#265](https://github.com/deltachat/chatmail/pull/263))
- Run chatmail-metadata and doveauth as vmail
([#261](https://github.com/chatmail/server/pull/261))
([#261](https://github.com/deltachat/chatmail/pull/261))
- Apply systemd restrictions to echobot
([#259](https://github.com/chatmail/server/pull/259))
([#259](https://github.com/deltachat/chatmail/pull/259))
- re-enable running the CI in pull requests, but not concurrently
([#258](https://github.com/chatmail/server/pull/258))
([#258](https://github.com/deltachat/chatmail/pull/258))
## 1.1.0 - 2024-03-28
@@ -437,27 +243,27 @@
### The changelog starts to record changes from March 15th, 2024
- Move systemd unit templates to cmdeploy package
([#255](https://github.com/chatmail/server/pull/255))
([#255](https://github.com/deltachat/chatmail/pull/255))
- Persist push tokens and support multiple device per address
([#254](https://github.com/chatmail/server/pull/254))
([#254](https://github.com/deltachat/chatmail/pull/254))
- Avoid warning for regular doveauth protocol's hello message.
([#250](https://github.com/chatmail/server/pull/250))
([#250](https://github.com/deltachat/chatmail/pull/250))
- Fix various tests to pass again with "cmdeploy test".
([#245](https://github.com/chatmail/server/pull/245),
[#242](https://github.com/chatmail/server/pull/242)
([#245](https://github.com/deltachat/chatmail/pull/245),
[#242](https://github.com/deltachat/chatmail/pull/242)
- Ensure lets-encrypt certificates are reloaded after renewal
([#244]) https://github.com/chatmail/server/pull/244
([#244]) https://github.com/deltachat/chatmail/pull/244
- Persist tokens to avoid iOS users loosing push-notifications when the
chatmail metadata service is restarted (happens regularly during deploys)
([#238](https://github.com/chatmail/server/pull/239)
([#238](https://github.com/deltachat/chatmail/pull/239)
- Fix failing sieve-script compile errors on incoming messages
([#237](https://github.com/chatmail/server/pull/239)
([#237](https://github.com/deltachat/chatmail/pull/239)
- Fix quota reporting after expunging of old mails
([#233](https://github.com/chatmail/server/pull/239)
([#233](https://github.com/deltachat/chatmail/pull/239)

394
README.md
View File

@@ -1,105 +1,58 @@
<img width="800px" src="www/src/collage-top.png"/>
# Chatmail relays for end-to-end encrypted e-mail
# Chatmail services optimized for Delta Chat apps
Chatmail relay servers are interoperable Mail Transport Agents (MTAs) designed for:
- **Convenience:** Low friction instant onboarding
- **Privacy:** No name, phone numbers, email required or collected
- **End-to-End Encryption enforced**: only OpenPGP messages with metadata minimization allowed
- **Instant:** Privacy-preserving Push Notifications for Apple, Google, and Huawei
- **Speed:** Message delivery in half a second, with optional P2P realtime connections
- **Transport Security:** Strict TLS and DKIM enforced
- **Reliability:** No spam or IP reputation checks; rate-limits are suitable for realtime chats
- **Efficiency:** Messages are only stored for transit and removed automatically
This repository contains everything needed to setup a ready-to-use chatmail relay
This repository helps to setup a ready-to-use chatmail server
comprised of a minimal setup of the battle-tested
[Postfix SMTP](https://www.postfix.org) and [Dovecot IMAP](https://www.dovecot.org) MTAs/MDAs.
[postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services.
The automated setup is designed and optimized for providing chatmail addresses
for immediate permission-free onboarding through chat apps and bots.
Chatmail addresses are automatically created at first login,
after which the initially specified password is required
for sending and receiving messages through them.
The setup is designed and optimized for providing chatmail accounts
for use by [Delta Chat apps](https://delta.chat).
Please see [this list of known apps and client projects](https://chatmail.at/clients.html)
and [this list of known public 3rd party chatmail relay servers](https://chatmail.at/relays).
Chatmail accounts are automatically created by a first login,
after which the initially specified password is required for using them.
## Deploying your own chatmail server
## Minimal requirements, Prerequisites
To deploy chatmail on your own server, you must have set-up ssh authentication and need to use an ed25519 key, due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191). You also need to add your private key to the local ssh-agent, because you can't type in your password during deployment.
You will need the following:
- Control over a domain through a DNS provider of your choice.
- A Debian 12 server with reachable SMTP/SUBMISSIONS/IMAPS/HTTPS ports.
IPv6 is encouraged if available.
Chatmail relay servers only require 1GB RAM, one CPU, and perhaps 10GB storage for a
few thousand active chatmail addresses.
- Key-based SSH authentication to the root user.
You must add a passphrase-protected private key to your local ssh-agent
because you can't type in your passphrase during deployment.
(An ed25519 private key is required due to an [upstream bug in paramiko](https://github.com/paramiko/paramiko/issues/2191))
## Getting started
We use `chat.example.org` as the chatmail domain in the following steps.
We use `chat.example.org` as the chatmail domain in the following steps.
Please substitute it with your own domain.
1. Setup the initial DNS records.
The following is an example in the familiar BIND zone file format with
a TTL of 1 hour (3600 seconds).
Please substitute your domain and IP addresses.
1. Install the `cmdeploy` command in a virtualenv
```
chat.example.com. 3600 IN A 198.51.100.5
chat.example.com. 3600 IN AAAA 2001:db8::5
www.chat.example.com. 3600 IN CNAME chat.example.com.
mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
```
2. On your local PC, clone the repository and bootstrap the Python virtualenv.
```
git clone https://github.com/chatmail/relay
cd relay
git clone https://github.com/deltachat/chatmail
cd chatmail
scripts/initenv.sh
```
3. On your local PC, create chatmail configuration file `chatmail.ini`:
2. Create chatmail configuration file `chatmail.ini`:
```
scripts/cmdeploy init chat.example.org # <-- use your domain
```
4. Verify that SSH root login to your remote server works:
3. Point your domain to the server's IP address,
if you haven't done so already.
Verify that SSH root login works:
```
ssh root@chat.example.org # <-- use your domain
ssh root@chat.example.org # <-- use your domain
```
5. From your local PC, deploy the remote chatmail relay server:
4. Deploy to the remote chatmail server:
```
scripts/cmdeploy run
```
This script will also check that you have all necessary DNS records.
This script will check that you have all necessary DNS records.
If DNS records are missing, it will recommend
which you should configure at your DNS provider
(it can take some time until they are public).
### Other helpful commands
### Other helpful commands:
To check the status of your remotely running chatmail service:
@@ -129,25 +82,25 @@ scripts/cmdeploy bench
This repository has four directories:
- [cmdeploy](https://github.com/chatmail/relay/tree/main/cmdeploy)
- [cmdeploy](https://github.com/deltachat/chatmail/tree/main/cmdeploy)
is a collection of configuration files
and a [pyinfra](https://pyinfra.com)-based deployment script.
- [chatmaild](https://github.com/chatmail/relay/tree/main/chatmaild)
is a Python package containing several small services
- [chatmaild](https://github.com/deltachat/chatmail/tree/main/chatmaild)
is a python package containing several small services
which handle authentication,
trigger push notifications on new messages,
ensure that outbound mails are encrypted,
delete inactive users,
and some other minor things.
chatmaild can also be installed as a stand-alone Python package.
chatmaild can also be installed as a stand-alone python package.
- [www](https://github.com/chatmail/relay/tree/main/www)
- [www](https://github.com/deltachat/chatmail/tree/main/www)
contains the html, css, and markdown files
which make up a chatmail relay's web page.
Edit them before deploying to make your chatmail relay stand out.
which make up a chatmail server's web page.
Edit them before deploying to make your chatmail server stand out.
- [scripts](https://github.com/chatmail/relay/tree/main/scripts)
- [scripts](https://github.com/deltachat/chatmail/tree/main/scripts)
offers two convenience tools for beginners;
`initenv.sh` installs the necessary dependencies to a local virtual environment,
and the `scripts/cmdeploy` script enables you
@@ -158,86 +111,80 @@ This repository has four directories:
The `cmdeploy/src/cmdeploy/cmdeploy.py` command line tool
helps with setting up and managing the chatmail service.
`cmdeploy init` creates the `chatmail.ini` config file.
`cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [`script`](cmdeploy/src/cmdeploy/__init__.py)
to automatically install or upgrade all chatmail components on a relay,
`cmdeploy run` uses a [pyinfra](https://pyinfra.com/)-based [script](`cmdeploy/src/cmdeploy/__init__.py`)
to automatically install or upgrade all chatmail components on a server,
according to the `chatmail.ini` config.
The components of chatmail are:
- [Postfix SMTP MTA](https://www.postfix.org) accepts and relays messages
(both from your users and from the wider e-mail MTA network)
- [postfix smtp server](https://www.postfix.org) accepts sent messages (both from your users and from other servers)
- [Dovecot IMAP MDA](https://www.dovecot.org) stores messages for your users until they download them
- [dovecot imap server](https://www.dovecot.org) stores messages for your users until they download them
- [Nginx](https://nginx.org/) shows the web page with your privacy policy and additional information
- [nginx](https://nginx.org/) shows the web page with your privacy policy and additional information
- [acmetool](https://hlandau.github.io/acmetool/) manages TLS certificates for Dovecot, Postfix, and Nginx
- [acmetool](https://hlandau.github.io/acmetool/) manages TLS certificates for dovecot, postfix, and nginx
- [OpenDKIM](http://www.opendkim.org/) for signing messages with DKIM and rejecting inbound messages without DKIM
- [opendkim](http://www.opendkim.org/) for signing messages with DKIM and rejecting inbound messages without DKIM
- [mtail](https://google.github.io/mtail/) for collecting anonymized metrics in case you have monitoring
- [Iroh relay](https://www.iroh.computer/docs/concepts/relay)
which helps client devices to establish Peer-to-Peer connections
- [TURN](https://github.com/chatmail/chatmail-turn)
to enable relay users to start webRTC calls
even if a p2p connection can't be established
- and the chatmaild services, explained in the next section:
### chatmaild
`chatmaild` implements various systemd-controlled services
that integrate with Dovecot and Postfix to achieve instant-onboarding and
only relaying OpenPGP end-to-end messages encrypted messages.
A short overview of `chatmaild` services:
chatmaild offers several commands
which differentiate a *chatmail* server from a classic mail server.
If you deploy them with cmdeploy,
they are run by systemd services in the background.
A short overview:
- [`doveauth`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/doveauth.py)
implements create-on-login address semantics and is used
by Dovecot during IMAP login and by Postfix during SMTP/SUBMISSION login
- [`doveauth`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/doveauth.py) implements
create-on-login account creation semantics and is used
by Dovecot during login authentication and by Postfix
which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
to authenticate logins.
to authenticate users
to send mails for them.
- [`filtermail`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/filtermail.py)
prevents unencrypted email from leaving or entering the chatmail service
and is integrated into Postfix's outbound and inbound mail pipelines.
- [`filtermail`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/filtermail.py) prevents
unencrypted e-mail from leaving the chatmail service
and is integrated into postfix's outbound mail pipelines.
- [`chatmail-metadata`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
[Dovecot lua script](https://github.com/chatmail/relay/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
to store user-specific relay-side config.
- [`chatmail-metadata`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/metadata.py) is contacted by a
[dovecot lua script](https://github.com/deltachat/chatmail/blob/main/cmdeploy/src/cmdeploy/dovecot/push_notification.lua)
to store user-specific server-side config.
On new messages,
it [passes the user's push notification token](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/notifier.py)
it [passes the user's push notification token](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/notifier.py)
to [notifications.delta.chat](https://delta.chat/help#instant-delivery)
so the push notifications on the user's phone can be triggered
by Apple/Google/Huawei.
by Apple/Google.
- [`delete_inactive_users`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
- [`delete_inactive_users`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/delete_inactive_users.py)
deletes users if they have not logged in for a very long time.
The timeframe can be configured in `chatmail.ini`.
- [`lastlogin`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/lastlogin.py)
is contacted by Dovecot when a user logs in
- [`lastlogin`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/lastlogin.py)
is contacted by dovecot when a user logs in
and stores the date of the login.
- [`echobot`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/echo.py)
- [`echobot`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/echo.py)
is a small bot for test purposes.
It simply echoes back messages from users.
- [`chatmail-metrics`](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/metrics.py)
- [`chatmail-metrics`](https://github.com/deltachat/chatmail/blob/main/chatmaild/src/chatmaild/metrics.py)
collects some metrics and displays them at `https://example.org/metrics`.
### Home page and getting started for users
`cmdeploy run` also creates default static web pages and deploys them
to a Nginx web server with:
`cmdeploy run` also creates default static Web pages and deploys them
to a nginx web server with:
- a default `index.html` along with a QR code that users can click to
create an address on your chatmail relay
create accounts on your chatmail provider,
- a default `info.html` that is linked from the home page
- a default `info.html` that is linked from the home page,
- a default `policy.html` that is linked from the home page
- a default `policy.html` that is linked from the home page.
All `.html` files are generated
by the according markdown `.md` file in the `www/src` directory.
@@ -245,78 +192,48 @@ by the according markdown `.md` file in the `www/src` directory.
### Refining the web pages
```
scripts/cmdeploy webdev
```
This starts a local live development cycle for chatmail web pages:
This starts a local live development cycle for chatmail Web pages:
- uses the `www/src/page-layout.html` file for producing static
HTML pages from `www/src/*.md` files
- continously builds the web presence reading files from `www/src` directory
and generating HTML files and copying assets to the `www/build` directory.
and generating html files and copying assets to the `www/build` directory.
- Starts a browser window automatically where you can "refresh" as needed.
#### Custom web pages
You can skip uploading a web page
by setting `www_folder=disabled` in `chatmail.ini`.
## Emergency Commands to disable automatic account creation
If you want to manage your web pages outside this git repository,
you can set `www_folder` in `chatmail.ini` to a custom directory on your computer.
`cmdeploy run` will upload it as the server's home page,
and if it contains a `src/index.md` file,
will build it with hugo.
## Mailbox directory layout
Fresh chatmail addresses have a mailbox directory that contains:
- a `password` file with the salted password required for authenticating
whether a login may use the address to send/receive messages.
If you modify the password file manually, you effectively block the user.
- `enforceE2EEincoming` is a default-created file with each address.
If present the file indicates that this chatmail address rejects incoming cleartext messages.
If absent the address accepts incoming cleartext messages.
- `dovecot*`, `cur`, `new` and `tmp` represent IMAP/mailbox state.
If the address is only used by one device, the Maildir directories
will typically be empty unless the user of that address hasn't been online
for a while.
## Emergency Commands to disable automatic address creation
If you need to stop address creation,
e.g. because some script is wildly creating addresses,
login with ssh and run:
If you need to stop account creation,
e.g. because some script is wildly creating accounts,
login to the server with ssh and run:
```
touch /etc/chatmail-nocreate
```
Chatmail address creation will be denied while this file is present.
While this file is present, account creation will be blocked.
### Ports
[Postfix](http://www.postfix.org/) listens on ports 25 (SMTP) and 587 (SUBMISSION) and 465 (SUBMISSIONS).
[Dovecot](https://www.dovecot.org/) listens on ports 143 (IMAP) and 993 (IMAPS).
[Nginx](https://www.nginx.com/) listens on port 8443 (HTTPS-ALT) and 443 (HTTPS).
[Postfix](http://www.postfix.org/) listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
[Dovecot](https://www.dovecot.org/) listens on ports 143 (imap) and 993 (imaps).
[nginx](https://www.nginx.com/) listens on port 8443 (https-alt) and 443 (https).
Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (HTTP).
[chatmail-turn](https://github.com/chatmail/chatmail-turn) listens on UDP port 3478 (STUN/TURN),
and temporarily opens UDP ports when users request them. UDP port range is not restricted, any free port may be allocated.
[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (http).
chatmail-core based apps will, however, discover all ports and configurations
automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail relay server.
Delta Chat apps will, however, discover all ports and configurations
automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail service.
## Email authentication
Chatmail relays enforce [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
chatmail servers rely on [DKIM](https://www.rfc-editor.org/rfc/rfc6376)
to authenticate incoming emails.
Incoming emails must have a valid DKIM signature with
Signing Domain Identifier (SDID, `d=` parameter in the DKIM-Signature header)
@@ -339,130 +256,101 @@ and rejects incorrectly authenticated emails with [`reject_sender_login_mismatch
`From:` header must correspond to envelope MAIL FROM,
this is ensured by `filtermail` proxy.
## TLS requirements
## Migrating chatmail server to a new host
Postfix is configured to require valid TLS
by setting [`smtp_tls_security_level`](https://www.postfix.org/postconf.5.html#smtp_tls_security_level) to `verify`.
If emails don't arrive at your chatmail relay server,
the problem is likely that your relay does not have a valid TLS certificate.
You can test it by resolving `MX` records of your relay domain
and then connecting to MX relays (e.g `mx.example.org`) with
`openssl s_client -connect mx.example.org:25 -verify_hostname mx.example.org -verify_return_error -starttls smtp`
from the host that has open port 25 to verify that certificate is valid.
When providing a TLS certificate to your chatmail relay server,
make sure to provide the full certificate chain
and not just the last certificate.
If you are running an Exim server and don't see incoming connections
from a chatmail relay server in the logs,
make sure `smtp_no_mail` log item is enabled in the config
with `log_selector = +smtp_no_mail`.
By default Exim does not log sessions that are closed
before sending the `MAIL` command.
This happens if certificate is not recognized as valid by Postfix,
so you might think that connection is not established
while actually it is a problem with your TLS certificate.
## Migrating a chatmail relay to a new host
If you want to migrate chatmail relay from an old machine
If you want to migrate chatmail from an old machine
to a new machine,
you can use these steps.
They were tested with a Linux laptop;
They were tested with a linux laptop;
you might need to adjust some of the steps to your environment.
Let's assume that your `mail_domain` is `mail.example.org`,
all involved machines run Debian 12,
your old site's IP address is `13.37.13.37`,
and your new site's IP address is `13.12.23.42`.
your old server's IP address is `13.37.13.37`,
and your new server's IP address is `13.12.23.42`.
Note, you should lower the TTLs of your DNS records to a value
such as 300 (5 minutes) so the migration happens as smoothly as possible.
During the guide, you might get a warning about changed SSH Host keys;
in this case, just run `ssh-keygen -R "mail.example.org"` as recommended
to make sure you can connect with SSH.
During the guide you might get a warning about changed SSH Host keys;
in this case, just run `ssh-keygen -R "mail.example.org"` as recommended.
1. First, copy `/var/lib/acme` to the new server with
`ssh root@13.37.13.37 tar c /var/lib/acme | ssh root@13.12.23.42 tar x -C /var/lib/`.
This transfers your TLS certificate.
1. First, disable mail services on the old site.
2. You should also copy `/etc/dkimkeys` to the new server with
`ssh root@13.37.13.37 tar c /etc/dkimkeys | ssh root@13.12.23.42 tar x -C /etc/`
so the DKIM DNS record stays correct.
```
cmdeploy run --disable-mail --ssh-host 13.37.13.37
```
3. On the new server, run `chown root: -R /var/lib/acme` and `chown opendkim: -R /etc/dkimkeys` to make sure the permissions are correct.
4. Run `cmdeploy run --disable-mail --ssh-host 13.12.23.42` to install chatmail on the new machine.
postfix and dovecot are disabled for now,
we will enable them later.
5. Now, point DNS to the new IP addresses.
You can already remove the old IP addresses from DNS.
Existing Delta Chat users will still be able to connect
to the old server, send and receive messages,
but new users will fail to create new profiles
with your chatmail server.
If other servers try to deliver messages to your new server they will fail,
but normally email servers will retry delivering messages
for at least a week, so messages will not be lost.
6. Now you can run `cmdeploy run --disable-mail --ssh-host 13.37.13.37` to disable your old server.
Now your users will notice the migration
and will not be able to send or receive messages
until the migration is completed.
2. Now we want to copy `/home/vmail`, `/var/lib/acme`, `/etc/dkimkeys`, `/run/echobot`, and `/var/spool/postfix` to the new site.
Login to the old site while forwarding your SSH agent
so you can copy directly from the old to the new site with your SSH key:
```
ssh -A root@13.37.13.37
tar c - /home/vmail/mail /var/lib/acme /etc/dkimkeys /run/echobot /var/spool/postfix | ssh root@13.12.23.42 "tar x -C /"
```
7. After everything is stopped,
you can copy the `/home/vmail/mail` directory to the new server.
It includes all user data, messages, password hashes, etc.
This transfers all addresses, the TLS certificate, DKIM keys (so DKIM DNS record remains valid), and the echobot's password so it continues to function.
It also preserves the Postfix mail spool so any messages pending delivery will still be delivered.
Just run: `ssh root@13.37.13.37 tar c /home/vmail/mail | ssh root@13.12.23.42 tar x -C /home/vmail/`
3. Install chatmail on the new machine:
After this, your new server has all the necessary files to start operating :)
```
cmdeploy run --disable-mail --ssh-host 13.12.23.42
```
Postfix and Dovecot are disabled for now; we will enable them later.
We first need to make the new site fully operational.
8. To be sure the permissions are still fine,
run `chown vmail: -R /home/vmail` on the new server.
3. On the new site, run the following to ensure the ownership is correct in case UIDs/GIDs changed:
```
chown root: -R /var/lib/acme
chown opendkim: -R /etc/dkimkeys
chown vmail: -R /home/vmail/mail
chown echobot: -R /run/echobot
```
4. Now, update DNS entries.
If other MTAs try to deliver messages to your chatmail domain they may fail intermittently,
as DNS catches up with the new site settings
but normally will retry delivering messages
for at least a week, so messages will not be lost.
5. Finally, you can execute `cmdeploy run --ssh-host 13.12.23.42` to turn on chatmail on the new relay.
Your users will be able to use the chatmail relay as soon as the DNS changes have propagated.
Voilà!
9. Finally, you can run `cmdeploy run` to turn on chatmail on the new server.
Your users can continue using the chatmail server,
and messages which were sent after step 6. should arrive now.
Voilà!
## Setting up a reverse proxy
A chatmail relay MTA does not track or depend on the client IP address
A chatmail server does not depend on the client IP address
for its operation, so it can be run behind a reverse proxy.
This will not even affect incoming mail authentication
as DKIM only checks the cryptographic signature
of the message and does not use the IP address as the input.
For example, you may want to self-host your chatmail relay
For example, you may want to self-host your chatmail server
and only use hosted VPS to provide a public IP address
for client connections and incoming mail.
You can connect chatmail relay to VPS
You can connect chatmail server to VPS
using a tunnel protocol
such as [WireGuard](https://www.wireguard.com/)
and setup a reverse proxy on a VPS
to forward connections to the chatmail relay
to forward connections to the chatmail server
over the tunnel.
You can also setup multiple reverse proxies
for your chatmail relay in different networks
to ensure your relay is reachable even when
for your chatmail server in different networks
to ensure your server is reachable even when
one of the IPs becomes inaccessible due to
hosting or routing problems.
Note that your chatmail relay still needs
Note that your server still needs
to be able to make outgoing connections on port 25
to send messages outside.
To setup a reverse proxy
(or rather Destination NAT, DNAT)
for your chatmail relay,
for your chatmail server,
put the following configuration in `/etc/nftables.conf`:
```
#!/usr/sbin/nft -f
@@ -474,7 +362,7 @@ define wan = eth0
# Which ports to proxy.
#
# Note that SSH is not proxied
# so it is possible to log into the proxy server
# so it is possible to log into the proxy server
# and not the original one.
define ports = { smtp, http, https, imap, imaps, submission, submissions }
@@ -537,7 +425,7 @@ table inet filter {
```
Run `systemctl enable nftables.service`
to ensure configuration is reloaded when the proxy relay reboots.
to ensure configuration is reloaded when the proxy server reboots.
Uncomment in `/etc/sysctl.conf` the following two lines:
@@ -546,19 +434,7 @@ net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
```
Then reboot the relay or do `sysctl -p` and `nft -f /etc/nftables.conf`.
Then reboot the server or do `sysctl -p` and `nft -f /etc/nftables.conf`.
Once proxy relay is set up,
Once proxy server is set up,
you can add its IP address to the DNS.
## Neighbors and Acquaintances
Here are some related projects that you may be interested in:
- [Mox](https://github.com/mjl-/mox): A Golang email server. [Work is in
progress](https://github.com/mjl-/mox/issues/251) to modify it to support all
of the features and configuration settings required to operate as a chatmail
relay.
- [Maddy-Chatmail](https://github.com/sadraiiali/maddy_chatmail): a plugin for the
[Maddy email server](https://maddy.email/) which aims to implement the
chatmail relay features and configuration options.

View File

@@ -12,7 +12,7 @@ dependencies = [
"deltachat-rpc-client",
"filelock",
"requests",
"crypt-r >= 3.13.1 ; python_version >= '3.11'",
"crypt-r",
]
[tool.setuptools]
@@ -27,10 +27,8 @@ chatmail-metadata = "chatmaild.metadata:main"
filtermail = "chatmaild.filtermail:main"
echobot = "chatmaild.echo:main"
chatmail-metrics = "chatmaild.metrics:main"
chatmail-expire = "chatmaild.expire:main"
chatmail-fsreport = "chatmaild.fsreport:main"
delete_inactive_users = "chatmaild.delete_inactive_users:main"
lastlogin = "chatmaild.lastlogin:main"
turnserver = "chatmaild.turnserver:main"
[project.entry-points.pytest11]
"chatmaild.testplugin" = "chatmaild.tests.plugin"
@@ -50,9 +48,6 @@ lint.select = [
"PLE", # Pylint Error
"PLW", # Pylint Warning
]
lint.ignore = [
"PLC0415" # import-outside-top-level
]
[tool.tox]
legacy_tox_ini = """
@@ -72,6 +67,5 @@ commands =
[testenv]
deps = pytest
pdbpp
pytest-localserver
commands = pytest -v -rsXx {posargs}
"""

View File

@@ -0,0 +1,59 @@
"""Generated from deltachat, draft-ietf-lamps-header-protection, and
encrypted_subject localizations in
https://github.com/thunderbird/thunderbird-android/
"""
common_encrypted_subjects = {
"...",
"[...]",
"암호화된 메시지",
"Ĉifrita mesaĝo",
"Courriel chiffré",
"Dulrituð skilaboð",
"Encrypted Message",
"Fersifere berjocht",
"Kemennadenn enrineget",
"Krüptitud kiri",
"Krypterat meddelande",
"Krypteret besked",
"Kryptert melding",
"Mensagem criptografada",
"Mensagem encriptada",
"Mensaje cifrado",
"Mensaxe cifrada",
"Mesaj Criptat",
"Mesazh i Fshehtëzuar",
"Messaggio criptato",
"Messaghju cifratu",
"Missatge encriptat",
"Neges wedi'i Hamgryptio",
"Pesan terenkripsi",
"Salattu viesti",
"Şifreli İleti",
"Šifrēta ziņa",
"Šifrirana poruka",
"Šifrirano sporočilo",
"Šifruotas laiškas",
"Tin nhắn được mã hóa",
"Titkosított üzenet",
"Verschlüsselte Nachricht",
"Versleuteld bericht",
"Zašifrovaná zpráva",
"Zaszyfrowana wiadomość",
"Zifratu mezua",
"Κρυπτογραφημένο μήνυμα",
"Зашифроване повідомлення",
"Зашифрованное сообщение",
"Зашыфраваны ліст",
"Криптирано съобщение",
"Шифрована порука",
"დაშიფრული წერილი",
"הודעה מוצפנת",
"پیام رمزنگاری‌شده",
"رسالة مشفّرة",
"എൻക്രിപ്റ്റുചെയ്‌ത സന്ദേശം",
"加密邮件",
"已加密的訊息",
"暗号化されたメッセージ",
}

View File

@@ -11,11 +11,7 @@ def read_config(inipath):
assert Path(inipath).exists(), inipath
cfg = iniconfig.IniConfig(inipath)
params = cfg.sections["params"]
default_config_content = get_default_config_content(params["mail_domain"])
df_params = iniconfig.IniConfig("ini", data=default_config_content)["params"]
new_params = dict(df_params.items())
new_params.update(params)
return Config(inipath, params=new_params)
return Config(inipath, params=params)
class Config:
@@ -26,25 +22,16 @@ class Config:
self.max_mailbox_size = params["max_mailbox_size"]
self.max_message_size = int(params.get("max_message_size", "31457280"))
self.delete_mails_after = params["delete_mails_after"]
self.delete_large_after = params["delete_large_after"]
self.delete_inactive_users_after = int(params["delete_inactive_users_after"])
self.username_min_length = int(params["username_min_length"])
self.username_max_length = int(params["username_max_length"])
self.password_min_length = int(params["password_min_length"])
self.passthrough_senders = params["passthrough_senders"].split()
self.passthrough_recipients = params["passthrough_recipients"].split()
self.www_folder = params.get("www_folder", "")
self.filtermail_smtp_port = int(params["filtermail_smtp_port"])
self.filtermail_smtp_port_incoming = int(
params["filtermail_smtp_port_incoming"]
)
self.postfix_reinject_port = int(params["postfix_reinject_port"])
self.postfix_reinject_port_incoming = int(
params["postfix_reinject_port_incoming"]
)
self.mtail_address = params.get("mtail_address")
self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
self.acme_email = params.get("acme_email", "")
self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
if "iroh_relay" not in params:
self.iroh_relay = "https://" + params["mail_domain"]
@@ -67,7 +54,7 @@ class Config:
def _getbytefile(self):
return open(self._inipath, "rb")
def get_user(self, addr) -> User:
def get_user(self, addr):
if not addr or "@" not in addr or "/" in addr:
raise ValueError(f"invalid address {addr!r}")
@@ -82,11 +69,6 @@ class Config:
def write_initial_config(inipath, mail_domain, overrides):
"""Write out default config file, using the specified config value overrides."""
content = get_default_config_content(mail_domain, **overrides)
inipath.write_text(content)
def get_default_config_content(mail_domain, **overrides):
from importlib.resources import files
inidir = files(__package__).joinpath("ini")
@@ -118,7 +100,7 @@ def get_default_config_content(mail_domain, **overrides):
lines = []
for line in content.split("\n"):
for key, value in privacy.items():
value_lines = value.format(mail_domain=mail_domain).strip().split("\n")
value_lines = value.strip().split("\n")
if not line.startswith(f"{key} =") or not value_lines:
continue
if len(value_lines) == 1:
@@ -131,4 +113,5 @@ def get_default_config_content(mail_domain, **overrides):
else:
lines.append(line)
content = "\n".join(lines)
return content
inipath.write_text(content)

View File

@@ -0,0 +1,31 @@
"""
Remove inactive users
"""
import os
import shutil
import sys
import time
from .config import read_config
def delete_inactive_users(config):
cutoff_date = time.time() - config.delete_inactive_users_after * 86400
for addr in os.listdir(config.mailboxes_dir):
try:
user = config.get_user(addr)
except ValueError:
continue
read_timestamp = user.get_last_login_timestamp()
if read_timestamp and read_timestamp < cutoff_date:
path = config.mailboxes_dir.joinpath(addr)
assert path == user.maildir
shutil.rmtree(path, ignore_errors=True)
def main():
(cfgpath,) = sys.argv[1:]
config = read_config(cfgpath)
delete_inactive_users(config)

View File

@@ -3,10 +3,7 @@ import logging
import os
import sys
try:
import crypt_r
except ImportError:
import crypt as crypt_r
import crypt_r
from .config import Config, read_config
from .dictproxy import DictProxy

View File

@@ -8,7 +8,6 @@ import logging
import os
import subprocess
import sys
from pathlib import Path
from deltachat_rpc_client import Bot, DeltaChat, EventType, Rpc, events
@@ -98,10 +97,6 @@ def main():
if not bot.is_configured():
bot.configure(addr, password)
# write invite link to working directory
invitelink = bot.account.get_qr_code()
Path("invite-link.txt").write_text(invitelink)
bot.run_forever()

View File

@@ -1,218 +0,0 @@
"""
Expire old messages and addresses.
"""
import os
import shutil
import sys
import time
from argparse import ArgumentParser
from collections import namedtuple
from datetime import datetime
from stat import S_ISREG
from chatmaild.config import read_config
FileEntry = namedtuple("FileEntry", ("relpath", "mtime", "size"))
def iter_mailboxes(basedir, maxnum):
if not os.path.exists(basedir):
print_info(f"no mailboxes found at: {basedir}")
return
for name in os_listdir_if_exists(basedir)[:maxnum]:
if "@" in name:
yield MailboxStat(basedir + "/" + name)
def get_file_entry(path):
"""return a FileEntry or None if the path does not exist or is not a regular file."""
try:
st = os.stat(path)
except FileNotFoundError:
return None
if not S_ISREG(st.st_mode):
return None
return FileEntry(path, st.st_mtime, st.st_size)
def os_listdir_if_exists(path):
"""return a list of names obtained from os.listdir or an empty list if the path does not exist."""
try:
return os.listdir(path)
except FileNotFoundError:
return []
class MailboxStat:
last_login = None
def __init__(self, basedir):
self.basedir = str(basedir)
# all detected messages in cur/new/tmp folders
self.messages = []
# all detected files in mailbox top dir
self.extrafiles = []
# scan all relevant files (without recursion)
old_cwd = os.getcwd()
try:
os.chdir(self.basedir)
except FileNotFoundError:
return
for name in os_listdir_if_exists("."):
if name in ("cur", "new", "tmp"):
for msg_name in os_listdir_if_exists(name):
entry = get_file_entry(f"{name}/{msg_name}")
if entry is not None:
self.messages.append(entry)
else:
entry = get_file_entry(name)
if entry is not None:
self.extrafiles.append(entry)
if name == "password":
self.last_login = entry.mtime
self.extrafiles.sort(key=lambda x: -x.size)
os.chdir(old_cwd)
def print_info(msg):
print(msg, file=sys.stderr)
class Expiry:
def __init__(self, config, dry, now, verbose):
self.config = config
self.dry = dry
self.now = now
self.verbose = verbose
self.del_mboxes = 0
self.all_mboxes = 0
self.del_files = 0
self.all_files = 0
self.start = time.time()
def remove_mailbox(self, mboxdir):
if self.verbose:
print_info(f"removing {mboxdir}")
if not self.dry:
shutil.rmtree(mboxdir)
self.del_mboxes += 1
def remove_file(self, path, mtime=None):
if self.verbose:
if mtime is not None:
date = datetime.fromtimestamp(mtime).strftime("%b %d")
print_info(f"removing {date} {path}")
else:
print_info(f"removing {path}")
if not self.dry:
try:
os.unlink(path)
except FileNotFoundError:
print_info(f"file not found/vanished {path}")
self.del_files += 1
def process_mailbox_stat(self, mbox):
cutoff_without_login = (
self.now - int(self.config.delete_inactive_users_after) * 86400
)
cutoff_mails = self.now - int(self.config.delete_mails_after) * 86400
cutoff_large_mails = self.now - int(self.config.delete_large_after) * 86400
self.all_mboxes += 1
changed = False
if mbox.last_login and mbox.last_login < cutoff_without_login:
self.remove_mailbox(mbox.basedir)
return
# all to-be-removed files are relative to the mailbox basedir
try:
os.chdir(mbox.basedir)
except FileNotFoundError:
print_info(f"mailbox not found/vanished {mbox.basedir}")
return
mboxname = os.path.basename(mbox.basedir)
if self.verbose:
date = datetime.fromtimestamp(mbox.last_login) if mbox.last_login else None
if date:
print_info(f"checking mailbox {date.strftime('%b %d')} {mboxname}")
else:
print_info(f"checking mailbox (no last_login) {mboxname}")
self.all_files += len(mbox.messages)
for message in mbox.messages:
if message.mtime < cutoff_mails:
self.remove_file(message.relpath, mtime=message.mtime)
elif message.size > 200000 and message.mtime < cutoff_large_mails:
# we only remove noticed large files (not unnoticed ones in new/)
if message.relpath.startswith("cur/"):
self.remove_file(message.relpath, mtime=message.mtime)
else:
continue
changed = True
if changed:
self.remove_file("maildirsize")
def get_summary(self):
return (
f"Removed {self.del_mboxes} out of {self.all_mboxes} mailboxes "
f"and {self.del_files} out of {self.all_files} files in existing mailboxes "
f"in {time.time() - self.start:2.2f} seconds"
)
def main(args=None):
"""Expire mailboxes and messages according to chatmail config"""
parser = ArgumentParser(description=main.__doc__)
ini = "/usr/local/lib/chatmaild/chatmail.ini"
parser.add_argument(
"chatmail_ini",
action="store",
nargs="?",
help=f"path pointing to chatmail.ini file, default: {ini}",
default=ini,
)
parser.add_argument(
"--days", action="store", help="assume date to be days older than now"
)
parser.add_argument(
"--maxnum",
default=None,
action="store",
help="maximum number of mailboxes to iterate on",
)
parser.add_argument(
"-v",
dest="verbose",
action="store_true",
help="print out removed files and mailboxes",
)
parser.add_argument(
"--remove",
dest="remove",
action="store_true",
help="actually remove all expired files and dirs",
)
args = parser.parse_args(args)
config = read_config(args.chatmail_ini)
now = datetime.utcnow().timestamp()
if args.days:
now = now - 86400 * int(args.days)
maxnum = int(args.maxnum) if args.maxnum else None
exp = Expiry(config, dry=not args.remove, now=now, verbose=args.verbose)
for mailbox in iter_mailboxes(str(config.mailboxes_dir), maxnum=maxnum):
exp.process_mailbox_stat(mailbox)
print(exp.get_summary())
if __name__ == "__main__":
main(sys.argv[1:])

View File

@@ -2,6 +2,7 @@
import asyncio
import base64
import binascii
import logging
import sys
import time
from email import policy
@@ -10,12 +11,10 @@ from email.utils import parseaddr
from smtplib import SMTP as SMTPClient
from aiosmtpd.controller import Controller
from aiosmtpd.smtp import SMTP
from .common_encrypted_subjects import common_encrypted_subjects
from .config import read_config
ENCRYPTION_NEEDED_523 = "523 Encryption Needed: Invalid Unencrypted Mail"
def check_openpgp_payload(payload: bytes):
"""Checks the OpenPGP payload.
@@ -37,12 +36,6 @@ def check_openpgp_payload(payload: bytes):
packet_type_id = payload[i] & 0x3F
i += 1
while payload[i] >= 224 and payload[i] < 255:
# Partial body length.
partial_length = 1 << (payload[i] & 0x1F)
i += 1 + partial_length
if payload[i] < 192:
# One-octet length.
body_len = payload[i]
@@ -61,7 +54,7 @@ def check_openpgp_payload(payload: bytes):
)
i += 5
else:
# Impossible, partial body length was processed above.
# Partial body length is not allowed.
return False
i += body_len
@@ -82,35 +75,17 @@ def check_openpgp_payload(payload: bytes):
return False
def check_armored_payload(payload: str, outgoing: bool):
"""Check the armored PGP message for invalid content.
:param payload: the armored PGP message
:param outgoing: whether the message is outgoing or incoming
:return: whether the message is a valid PGP message
"""
prefix = "-----BEGIN PGP MESSAGE-----\r\n"
def check_armored_payload(payload: str):
prefix = "-----BEGIN PGP MESSAGE-----\r\n\r\n"
if not payload.startswith(prefix):
return False
payload = payload.removeprefix(prefix)
while payload.endswith("\r\n"):
payload = payload.removesuffix("\r\n")
suffix = "-----END PGP MESSAGE-----"
suffix = "-----END PGP MESSAGE-----\r\n\r\n"
if not payload.endswith(suffix):
return False
payload = payload.removesuffix(suffix)
version_comment = "Version: "
if payload.startswith(version_comment):
if outgoing: # Disallow comments in outgoing messages
return False
# Remove comments from incoming messages
payload = payload.partition("\r\n")[2]
while payload.startswith("\r\n"):
payload = payload.removeprefix("\r\n")
# Remove CRC24.
payload = payload.rpartition("=")[0]
@@ -125,34 +100,15 @@ def check_armored_payload(payload: str, outgoing: bool):
return False
def is_securejoin(message):
if message.get("secure-join") not in ["vc-request", "vg-request"]:
return False
if not message.is_multipart():
return False
parts_count = 0
for part in message.iter_parts():
parts_count += 1
if parts_count > 1:
return False
if part.is_multipart():
return False
if part.get_content_type() != "text/plain":
return False
payload = part.get_payload().strip().lower()
if payload not in ("secure-join: vc-request", "secure-join: vg-request"):
return False
return True
def check_encrypted(message, outgoing=True):
def check_encrypted(message):
"""Check that the message is an OpenPGP-encrypted message.
MIME structure of the message must correspond to <https://www.rfc-editor.org/rfc/rfc3156>.
"""
if not message.is_multipart():
return False
if message.get("subject") not in common_encrypted_subjects:
return False
if message.get_content_type() != "multipart/encrypted":
return False
parts_count = 0
@@ -173,7 +129,7 @@ def check_encrypted(message, outgoing=True):
if part.get_content_type() != "application/octet-stream":
return False
if not check_armored_payload(part.get_payload(), outgoing=outgoing):
if not check_armored_payload(part.get_payload()):
return False
else:
return False
@@ -181,19 +137,9 @@ def check_encrypted(message, outgoing=True):
return True
async def asyncmain_beforequeue(config, mode):
if mode == "outgoing":
port = config.filtermail_smtp_port
handler = OutgoingBeforeQueueHandler(config)
else:
port = config.filtermail_smtp_port_incoming
handler = IncomingBeforeQueueHandler(config)
HackedController(
handler,
hostname="127.0.0.1",
port=port,
data_size_limit=config.max_message_size,
).start()
async def asyncmain_beforequeue(config):
port = config.filtermail_smtp_port
Controller(BeforeQueueHandler(config), hostname="127.0.0.1", port=port).start()
def recipient_matches_passthrough(recipient, passthrough_recipients):
@@ -205,29 +151,13 @@ def recipient_matches_passthrough(recipient, passthrough_recipients):
return False
class HackedController(Controller):
def factory(self):
return SMTPDiscardRCPTO_options(self.handler, **self.SMTP_kwargs)
class SMTPDiscardRCPTO_options(SMTP):
def _getparams(self, params):
# Ignore RCPT TO parameters.
#
# Otherwise parameters such as `ORCPT=...`
# or `NOTIFY=DELAY,FAILURE` (generated by Stalwart)
# make aiosmtpd reject the message here:
# <https://github.com/aio-libs/aiosmtpd/blob/98f578389ae86e5345cc343fa4e5a17b21d9c96d/aiosmtpd/smtp.py#L1379-L1384>
return {}
class OutgoingBeforeQueueHandler:
class BeforeQueueHandler:
def __init__(self, config):
self.config = config
self.send_rate_limiter = SendRateLimiter()
async def handle_MAIL(self, server, session, envelope, address, mail_options):
log_info(f"handle_MAIL from {address}")
logging.info(f"handle_MAIL from {address}")
envelope.mail_from = address
max_sent = self.config.max_user_send_per_minute
if not self.send_rate_limiter.is_sending_allowed(address, max_sent):
@@ -240,113 +170,61 @@ class OutgoingBeforeQueueHandler:
return "250 OK"
async def handle_DATA(self, server, session, envelope):
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, self.sync_handle_DATA, envelope)
def sync_handle_DATA(self, envelope):
log_info("handle_DATA before-queue")
logging.info("handle_DATA before-queue")
error = self.check_DATA(envelope)
if error:
return error
log_info("re-injecting the mail that passed checks")
logging.info("re-injecting the mail that passed checks")
client = SMTPClient("localhost", self.config.postfix_reinject_port)
client.sendmail(
envelope.mail_from, envelope.rcpt_tos, envelope.original_content
)
client.sendmail(envelope.mail_from, envelope.rcpt_tos, envelope.content)
return "250 OK"
def check_DATA(self, envelope):
"""the central filtering function for e-mails."""
log_info(f"Processing DATA message from {envelope.mail_from}")
logging.info(f"Processing DATA message from {envelope.mail_from}")
message = BytesParser(policy=policy.default).parsebytes(envelope.content)
mail_encrypted = check_encrypted(message, outgoing=True)
mail_encrypted = check_encrypted(message)
_, from_addr = parseaddr(message.get("from").strip())
envelope_from_domain = from_addr.split("@").pop()
logging.info(f"mime-from: {from_addr} envelope-from: {envelope.mail_from!r}")
if envelope.mail_from.lower() != from_addr.lower():
return f"500 Invalid FROM <{from_addr!r}> for <{envelope.mail_from!r}>"
if mail_encrypted or is_securejoin(message):
print("Outgoing: Filtering encrypted mail.", file=sys.stderr)
return
print("Outgoing: Filtering unencrypted mail.", file=sys.stderr)
if mail_encrypted:
print("Filtering encrypted mail.", file=sys.stderr)
else:
print("Filtering unencrypted mail.", file=sys.stderr)
if envelope.mail_from in self.config.passthrough_senders:
return
# allow self-sent Autocrypt Setup Message
if envelope.rcpt_tos == [from_addr]:
if message.get("subject") == "Autocrypt Setup Message":
if message.get_content_type() == "multipart/mixed":
return
passthrough_recipients = self.config.passthrough_recipients
for recipient in envelope.rcpt_tos:
if recipient_matches_passthrough(recipient, passthrough_recipients):
continue
print("Rejected unencrypted mail.", file=sys.stderr)
return ENCRYPTION_NEEDED_523
class IncomingBeforeQueueHandler:
def __init__(self, config):
self.config = config
async def handle_DATA(self, server, session, envelope):
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, self.sync_handle_DATA, envelope)
def sync_handle_DATA(self, envelope):
log_info("handle_DATA before-queue")
error = self.check_DATA(envelope)
if error:
return error
log_info("re-injecting the mail that passed checks")
# the smtp daemon on reinject_port_incoming gives it to dkim milter
# which looks at source address to determine whether to verify or sign
client = SMTPClient(
"localhost",
self.config.postfix_reinject_port_incoming,
source_address=("127.0.0.2", 0),
)
client.sendmail(
envelope.mail_from, envelope.rcpt_tos, envelope.original_content
)
return "250 OK"
def check_DATA(self, envelope):
"""the central filtering function for e-mails."""
log_info(f"Processing DATA message from {envelope.mail_from}")
message = BytesParser(policy=policy.default).parsebytes(envelope.content)
mail_encrypted = check_encrypted(message, outgoing=False)
if mail_encrypted or is_securejoin(message):
print("Incoming: Filtering encrypted mail.", file=sys.stderr)
is_securejoin = message.get("secure-join") in [
"vc-request",
"vg-request",
]
if is_securejoin:
return
print("Incoming: Filtering unencrypted mail.", file=sys.stderr)
# we want cleartext mailer-daemon messages to pass through
# chatmail core will typically not display them as normal messages
if message.get("auto-submitted"):
_, from_addr = parseaddr(message.get("from").strip())
if from_addr.lower().startswith("mailer-daemon@"):
if message.get_content_type() == "multipart/report":
return
for recipient in envelope.rcpt_tos:
user = self.config.get_user(recipient)
if user is None or user.is_incoming_cleartext_ok():
if envelope.mail_from == recipient:
# Always allow sending emails to self.
continue
if recipient_matches_passthrough(recipient, passthrough_recipients):
continue
res = recipient.split("@")
if len(res) != 2:
return f"500 Invalid address <{recipient}>"
_recipient_addr, recipient_domain = res
print("Rejected unencrypted mail.", file=sys.stderr)
return ENCRYPTION_NEEDED_523
is_outgoing = recipient_domain != envelope_from_domain
if is_outgoing and not mail_encrypted:
print("Rejected unencrypted mail.", file=sys.stderr)
return f"500 Invalid unencrypted mail to <{recipient}>"
class SendRateLimiter:
@@ -363,19 +241,13 @@ class SendRateLimiter:
return False
def log_info(msg):
print(msg, file=sys.stderr)
def main():
args = sys.argv[1:]
assert len(args) == 2
assert len(args) == 1
config = read_config(args[0])
mode = args[1]
logging.basicConfig(level=logging.WARN)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
assert mode in ["incoming", "outgoing"]
task = asyncmain_beforequeue(config, mode)
task = asyncmain_beforequeue(config)
loop.create_task(task)
log_info("entering serving loop")
loop.run_forever()

View File

@@ -1,168 +0,0 @@
"""
command line tool to analyze mailbox message storage
example invocation:
python -m chatmaild.fsreport /path/to/chatmail.ini
to show storage summaries for all "cur" folders
python -m chatmaild.fsreport /path/to/chatmail.ini --mdir cur
to show storage summaries only for first 1000 mailboxes
python -m chatmaild.fsreport /path/to/chatmail.ini --maxnum 1000
"""
import os
from argparse import ArgumentParser
from datetime import datetime
from chatmaild.config import read_config
from chatmaild.expire import iter_mailboxes
DAYSECONDS = 24 * 60 * 60
MONTHSECONDS = DAYSECONDS * 30
def HSize(size: int):
"""Format a size integer as a Human-readable string Kilobyte, Megabyte or Gigabyte"""
if size < 10000:
return f"{size / 1000:5.2f}K"
if size < 1000 * 1000:
return f"{size / 1000:5.0f}K"
if size < 1000 * 1000 * 1000:
return f"{int(size / 1000000):5.0f}M"
return f"{size / 1000000000:5.2f}G"
class Report:
def __init__(self, now, min_login_age, mdir):
self.size_extra = 0
self.size_messages = 0
self.now = now
self.min_login_age = min_login_age
self.mdir = mdir
self.num_ci_logins = self.num_all_logins = 0
self.login_buckets = {x: 0 for x in (1, 10, 30, 40, 80, 100, 150)}
self.message_buckets = {x: 0 for x in (0, 160000, 500000, 2000000)}
def process_mailbox_stat(self, mailbox):
# categorize login times
last_login = mailbox.last_login
if last_login:
self.num_all_logins += 1
if os.path.basename(mailbox.basedir)[:3] == "ci-":
self.num_ci_logins += 1
else:
for days in self.login_buckets:
if last_login >= self.now - days * DAYSECONDS:
self.login_buckets[days] += 1
cutoff_login_date = self.now - self.min_login_age * DAYSECONDS
if last_login and last_login <= cutoff_login_date:
# categorize message sizes
for size in self.message_buckets:
for msg in mailbox.messages:
if msg.size >= size:
if self.mdir and not msg.relpath.startswith(self.mdir):
continue
self.message_buckets[size] += msg.size
self.size_messages += sum(entry.size for entry in mailbox.messages)
self.size_extra += sum(entry.size for entry in mailbox.extrafiles)
def dump_summary(self):
all_messages = self.size_messages
print()
print("## Mailbox storage use analysis")
print(f"Mailbox data total size: {HSize(self.size_extra + all_messages)}")
print(f"Messages total size : {HSize(all_messages)}")
try:
percent = self.size_extra / (self.size_extra + all_messages) * 100
except ZeroDivisionError:
percent = 100
print(f"Extra files : {HSize(self.size_extra)} ({percent:.2f}%)")
print()
if self.min_login_age:
print(f"### Message storage for {self.min_login_age} days old logins")
pref = f"[{self.mdir}] " if self.mdir else ""
for minsize, sumsize in self.message_buckets.items():
percent = (sumsize / all_messages * 100) if all_messages else 0
print(
f"{pref}larger than {HSize(minsize)}: {HSize(sumsize)} ({percent:.2f}%)"
)
user_logins = self.num_all_logins - self.num_ci_logins
def p(num):
return f"({num / user_logins * 100:2.2f}%)" if user_logins else "100%"
print()
print(f"## Login stats, from date reference {datetime.fromtimestamp(self.now)}")
print(f"all: {HSize(self.num_all_logins)}")
print(f"non-ci: {HSize(user_logins)}")
print(f"ci: {HSize(self.num_ci_logins)}")
for days, active in self.login_buckets.items():
print(f"last {days:3} days: {HSize(active)} {p(active)}")
def main(args=None):
"""Report about filesystem storage usage of all mailboxes and messages"""
parser = ArgumentParser(description=main.__doc__)
ini = "/usr/local/lib/chatmaild/chatmail.ini"
parser.add_argument(
"chatmail_ini",
action="store",
nargs="?",
help=f"path pointing to chatmail.ini file, default: {ini}",
default=ini,
)
parser.add_argument(
"--days",
default=0,
action="store",
help="assume date to be days older than now",
)
parser.add_argument(
"--min-login-age",
default=0,
dest="min_login_age",
action="store",
help="only sum up message size if last login is at least min-login-age days old",
)
parser.add_argument(
"--mdir",
action="store",
help="only consider 'cur' or 'new' or 'tmp' messages for summary",
)
parser.add_argument(
"--maxnum",
default=None,
action="store",
help="maximum number of mailboxes to iterate on",
)
args = parser.parse_args(args)
config = read_config(args.chatmail_ini)
now = datetime.utcnow().timestamp()
if args.days:
now = now - 86400 * int(args.days)
maxnum = int(args.maxnum) if args.maxnum else None
rep = Report(now=now, min_login_age=int(args.min_login_age), mdir=args.mdir)
for mbox in iter_mailboxes(str(config.mailboxes_dir), maxnum=maxnum):
rep.process_mailbox_stat(mbox)
rep.dump_summary()
if __name__ == "__main__":
main()

View File

@@ -23,9 +23,6 @@ max_message_size = 31457280
# days after which mails are unconditionally deleted
delete_mails_after = 20
# days after which large messages (>200k) are unconditionally deleted
delete_large_after = 7
# days after which users without a successful login are deleted (database and mails)
delete_inactive_users_after = 90
@@ -43,29 +40,21 @@ passthrough_senders =
# list of e-mail recipients for which to accept outbound un-encrypted mails
# (space-separated, item may start with "@" to whitelist whole recipient domains)
passthrough_recipients = xstore@testrun.org echo@{mail_domain}
# path to www directory - documented here: https://github.com/chatmail/relay/#custom-web-pages
#www_folder = www
passthrough_recipients = xstore@testrun.org
#
# Deployment Details
#
# SMTP outgoing filtermail and reinjection
# where the filtermail SMTP service listens
filtermail_smtp_port = 10080
postfix_reinject_port = 10025
# SMTP incoming filtermail and reinjection
filtermail_smtp_port_incoming = 10081
postfix_reinject_port_incoming = 10026
# postfix accepts on the localhost reinject SMTP port
postfix_reinject_port = 10025
# if set to "True" IPv6 is disabled
disable_ipv6 = False
# Your email adress, which will be used in acmetool to manage Let's Encrypt SSL certificates
acme_email =
# Defaults to https://iroh.{{mail_domain}} and running `iroh-relay` on the chatmail
# service.
# If you set it to anything else, the service will be disabled

View File

@@ -1,7 +1,7 @@
[privacy]
passthrough_recipients = privacy@testrun.org xstore@testrun.org echo@{mail_domain}
passthrough_recipients = privacy@testrun.org xstore@testrun.org
privacy_postal =
Merlinux GmbH, Represented by the managing director H. Krekel,

View File

@@ -1,24 +1,14 @@
import logging
import sys
import time
from contextlib import contextmanager
from .config import read_config
from .dictproxy import DictProxy
from .filedict import FileDict
from .notifier import Notifier
from .turnserver import turn_credentials
def _is_valid_token_timestamp(timestamp, now):
# Token if invalid after 90 days
# or if the timestamp is in the future.
return timestamp > now - 3600 * 24 * 90 and timestamp < now + 60
class Metadata:
# each SETMETADATA on this key appends to dictionary
# mapping of unique device tokens
# each SETMETADATA on this key appends to a list of unique device tokens
# which only ever get removed if the upstream indicates the token is invalid
DEVICETOKEN_KEY = "devicetoken"
@@ -28,60 +18,29 @@ class Metadata:
def get_metadata_dict(self, addr):
return FileDict(self.vmail_dir / addr / "metadata.json")
@contextmanager
def _modify_tokens(self, addr):
with self.get_metadata_dict(addr).modify() as data:
tokens = data.setdefault(self.DEVICETOKEN_KEY, {})
now = int(time.time())
if isinstance(tokens, list):
data[self.DEVICETOKEN_KEY] = tokens = {t: now for t in tokens}
expired_tokens = [
token
for token, timestamp in tokens.items()
if not _is_valid_token_timestamp(tokens[token], now)
]
for expired_token in expired_tokens:
del tokens[expired_token]
yield tokens
def add_token_to_addr(self, addr, token):
with self._modify_tokens(addr) as tokens:
tokens[token] = int(time.time())
with self.get_metadata_dict(addr).modify() as data:
tokens = data.setdefault(self.DEVICETOKEN_KEY, [])
if token not in tokens:
tokens.append(token)
def remove_token_from_addr(self, addr, token):
with self._modify_tokens(addr) as tokens:
with self.get_metadata_dict(addr).modify() as data:
tokens = data.get(self.DEVICETOKEN_KEY, [])
if token in tokens:
del tokens[token]
tokens.remove(token)
def get_tokens_for_addr(self, addr):
mdict = self.get_metadata_dict(addr).read()
tokens = mdict.get(self.DEVICETOKEN_KEY, {})
now = int(time.time())
if isinstance(tokens, dict):
token_list = [
token
for token, timestamp in tokens.items()
if _is_valid_token_timestamp(timestamp, now)
]
if len(token_list) < len(tokens):
# Some tokens have expired, remove them.
with self._modify_tokens(addr) as _tokens:
pass
else:
token_list = []
return token_list
return mdict.get(self.DEVICETOKEN_KEY, [])
class MetadataDictProxy(DictProxy):
def __init__(self, notifier, metadata, iroh_relay=None, turn_hostname=None):
def __init__(self, notifier, metadata, iroh_relay=None):
super().__init__()
self.notifier = notifier
self.metadata = metadata
self.iroh_relay = iroh_relay
self.turn_hostname = turn_hostname
def handle_lookup(self, parts):
# Lpriv/43f5f508a7ea0366dff30200c15250e3/devicetoken\tlkj123poi@c2.testrun.org
@@ -100,11 +59,6 @@ class MetadataDictProxy(DictProxy):
):
# Handle `GETMETADATA "" /shared/vendor/deltachat/irohrelay`
return f"O{self.iroh_relay}\n"
elif keyname == "vendor/vendor.dovecot/pvt/server/vendor/deltachat/turn":
res = turn_credentials()
port = 3478
return f"O{self.turn_hostname}:{port}:{res}\n"
logging.warning(f"lookup ignored: {parts!r}")
return "N\n"
@@ -128,7 +82,6 @@ def main():
config = read_config(config_path)
iroh_relay = config.iroh_relay
mail_domain = config.mail_domain
vmail_dir = config.mailboxes_dir
if not vmail_dir.exists():
@@ -142,10 +95,7 @@ def main():
notifier.start_notification_threads(metadata.remove_token_from_addr)
dictproxy = MetadataDictProxy(
notifier=notifier,
metadata=metadata,
iroh_relay=iroh_relay,
turn_hostname=mail_domain,
notifier=notifier, metadata=metadata, iroh_relay=iroh_relay
)
dictproxy.serve_forever_from_socket(socket)

View File

@@ -11,8 +11,6 @@ def main(vmail_dir=None):
ci_accounts = 0
for path in Path(vmail_dir).iterdir():
if not path.joinpath("cur").is_dir():
continue
accounts += 1
if path.name[:3] in ("ci-", "ac_"):
ci_accounts += 1

View File

@@ -15,7 +15,7 @@ ALPHANUMERIC_PUNCT = string.ascii_letters + string.digits + string.punctuation
def create_newemail_dict(config: Config):
user = "".join(random.choices(ALPHANUMERIC, k=config.username_max_length))
user = "".join(random.choices(ALPHANUMERIC, k=config.username_min_length))
password = "".join(
secrets.choice(ALPHANUMERIC_PUNCT)
for _ in range(config.password_min_length + 3)

View File

@@ -17,11 +17,11 @@ and which are scheduled for retry using exponential back-off timing.
If a token notification would be scheduled more than DROP_DEADLINE seconds
after its first attempt, it is dropped with a log error.
Note that tokens are opaque to the notification machinery here
and are encrypted foreclosing all ability to distinguish
Note that tokens are completely opaque to the notification machinery here
and will in the future be encrypted foreclosing all ability to distinguish
which device token ultimately goes to which phone-provider notification service,
or to understand the relation of "device tokens" and chatmail addresses.
The meaning and format of tokens is basically a matter of chatmail Core and
The meaning and format of tokens is basically a matter of Delta-Chat Core and
the `notification.delta.chat` service.
"""
@@ -95,12 +95,7 @@ class Notifier:
logging.warning(f"removing spurious queue item: {queue_path!r}")
queue_path.unlink()
continue
try:
queue_item = PersistentQueueItem.read_from_path(queue_path)
except ValueError:
logging.warning(f"removing spurious queue item: {queue_path!r}")
queue_path.unlink()
continue
queue_item = PersistentQueueItem.read_from_path(queue_path)
self.queue_for_retry(queue_item)
def queue_for_retry(self, queue_item, retry_num=0):

View File

@@ -1,56 +0,0 @@
From: {from_addr}
To: {to_addr}
Autocrypt-Setup-Message: v1
Subject: Autocrypt Setup Message
Date: Tue, 22 Jan 2019 12:56:29 +0100
Content-type: multipart/mixed; boundary="Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ"
--Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
Content-Type: text/plain
This message contains all information to transfer your Autocrypt
settings along with your secret key securely from your original
device.
To set up your new device for Autocrypt, please follow the
instuctions that should be presented by your new device.
You can keep this message and use it as a backup for your secret
key. If you want to do this, you should write down the Setup Code
and store it securely.
--Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ
Content-Type: application/autocrypt-setup
Content-Disposition: attachment; filename="autocrypt-setup-message.html"
<html><body>
<p>
This is the Autocrypt setup file used to transfer settings and
keys between clients. You can decrypt it using the Setup Code
presented on your old device, and then import the contained key
into your keyring.
</p>
<pre>
-----BEGIN PGP MESSAGE-----
Passphrase-Format: numeric9x4
Passphrase-Begin: 17
jA0EBwMCFAxADoCdzeX/0ukBlqI5+pfpKb751qd/7nLNbkpy3gVcaf1QwRPZYt40
Ynp08UqRQ2g48ZlnzHLSwlTGOPTuv2Jt8ka+pgZ45xzvJSG2gau03xP4VsC271kR
VmCjdb0Y6Rk96mAwfGzrkbaRQ9Z7fIoL866GOv6h9neiVIkp+JYlTV6ISD0ZQJ4Q
I6dOQkB/TWZyVjtiJDOQHdfNWliA6NtqaLq19wlu9L5xXjuNpY95KwR8EJXWe0+o
Y3d2U/KxOAkXKghP2Qg1GtlPVeGC5T4p03TGI6pzKT+kHX6Rrm9wK6sM9aTquMmF
Vok84Jg1DFnwivWC2RILR81rXi7k/+Y6MUbveFgJ9cQduqpxnmD7TjOblYu7M6zp
YGAUxh8DRKlIMn2QsA++DBYQ6ACZvwuY8qTDLkqPDo4WqM313dsMJbyGjDdVE7EM
PESS+RlABETpZXz8g/ycr6DIUNdlbPcmYlsBfHWDOuR2GFFTwmlv5slWS39dJv38
E0eIe1CwdxI801Se7t7dUUS/ZF8wb6GlmxOcqGbF8eko1Z0S64IAm7/h13MRQCxI
geQnHfGYVJ2FOimoCMEKwfa9x++RFTDW0u7spDC2uWvK/1viV8OfRppFhLr/kmKb
18lWXuAz80DAjUDUsVqEq2MvJBJGoCJUEyjuRsLkHYRM5jYk4v50LyyR0Om73nWF
nZBqmqNzdr7Xb9PHHdFhnEc0VvoYbrcM0RVYcEMW3YbmejM891j1d6Iv+/n/qND/
NdebGrfWJMmFLf/iEkzTZ3/v5inW9LpWoRc94ioCjJTaEo8Rib6ARRFaJVIsmNXi
YicFGO98D+zX+a2t9Yz6IpPajVslnOp6ScpmXgts/2XWD7oE+JgxSAqo/dLVsHgP
Ufo=
=pulM
-----END PGP MESSAGE-----
</pre></body></html>
--Y6fyGi9SoGeH8WwRaEdC6bbBcYOedDzrQ--

View File

@@ -1,46 +0,0 @@
Date: Fri, 8 Jul 1994 09:21:47 -0400
From: Mail Delivery Subsystem <MAILER-DAEMON@example.org>
Subject: Returned mail: User unknown
To: <owner-ups-mib@CS.UTK.EDU>
Auto-Submitted: auto-replied
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="JAA13167.773673707/CS.UTK.EDU"
--JAA13167.773673707/CS.UTK.EDU
content-type: text/plain; charset=us-ascii
----- The following addresses had delivery problems -----
<arathib@vnet.ibm.com> (unrecoverable error)
<wsnell@sdcc13.ucsd.edu> (unrecoverable error)
--JAA13167.773673707/CS.UTK.EDU
content-type: message/delivery-status
Reporting-MTA: dns; cs.utk.edu
Original-Recipient: rfc822;arathib@vnet.ibm.com
Final-Recipient: rfc822;arathib@vnet.ibm.com
Action: failed
Status: 5.0.0 (permanent failure)
Diagnostic-Code: smtp;
550 'arathib@vnet.IBM.COM' is not a registered gateway user
Remote-MTA: dns; vnet.ibm.com
Original-Recipient: rfc822;johnh@hpnjld.njd.hp.com
Final-Recipient: rfc822;johnh@hpnjld.njd.hp.com
Action: delayed
Status: 4.0.0 (hpnjld.njd.jp.com: host name lookup failure)
Original-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
Final-Recipient: rfc822;wsnell@sdcc13.ucsd.edu
Action: failed
Status: 5.0.0
Diagnostic-Code: smtp; 550 user unknown
Remote-MTA: dns; sdcc13.ucsd.edu
--JAA13167.773673707/CS.UTK.EDU
content-type: message/rfc822
[original message goes here]
--JAA13167.773673707/CS.UTK.EDU--

View File

@@ -1,21 +0,0 @@
Subject: Message from {from_addr}
From: <{from_addr}>
To: <{to_addr}>
Date: Sun, 15 Oct 2023 16:43:25 +0000
Message-ID: <Mr.78MWtlV7RAi.goCFzBhCYfy@c2.testrun.org>
Chat-Version: 1.0
Secure-Join: vc-request
Secure-Join-Invitenumber: RANDOM-TOKEN
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi"
--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi
Content-Type: text/plain; charset=utf-8
Buy viagra!
--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi--

View File

@@ -1,21 +0,0 @@
Subject: Message from {from_addr}
From: <{from_addr}>
To: <{to_addr}>
Date: Sun, 15 Oct 2023 16:43:25 +0000
Message-ID: <Mr.78MWtlV7RAi.goCFzBhCYfy@c2.testrun.org>
Chat-Version: 1.0
Secure-Join: vc-request
Secure-Join-Invitenumber: RANDOM-TOKEN
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi"
--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi
Content-Type: text/plain; charset=utf-8
Secure-Join: vc-request
--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi--

View File

@@ -69,11 +69,12 @@ def maildata(request):
assert datadir.exists(), datadir
def maildata(name, from_addr, to_addr, subject="[...]"):
def maildata(name, from_addr, to_addr, subject="..."):
# Using `.read_bytes().decode()` instead of `.read_text()` to preserve newlines.
data = datadir.joinpath(name).read_bytes().decode()
text = data.format(from_addr=from_addr, to_addr=to_addr, subject=subject)
return BytesParser(policy=policy.SMTP).parsebytes(text.encode())
return BytesParser(policy=policy.default).parsebytes(text.encode())
return maildata

View File

@@ -15,14 +15,6 @@ def test_read_config_basic(example_config):
assert example_config.mail_domain == "chat.example.org"
def test_read_config_basic_using_defaults(tmp_path, maildomain):
inipath = tmp_path.joinpath("chatmail.ini")
inipath.write_text(f"[params]\nmail_domain = {maildomain}")
example_config = read_config(inipath)
assert example_config.max_user_send_per_minute == 60
assert example_config.filtermail_smtp_port_incoming == 10081
def test_read_config_testrun(make_config):
config = make_config("something.testrun.org")
assert config.mail_domain == "something.testrun.org"
@@ -35,7 +27,6 @@ def test_read_config_testrun(make_config):
assert config.max_user_send_per_minute == 60
assert config.max_mailbox_size == "100M"
assert config.delete_mails_after == "20"
assert config.delete_large_after == "7"
assert config.username_min_length == 9
assert config.username_max_length == 9
assert config.password_min_length == 9

View File

@@ -1,7 +1,7 @@
import time
from chatmaild.delete_inactive_users import delete_inactive_users
from chatmaild.doveauth import AuthDictProxy
from chatmaild.expire import main as main_expire
def test_login_timestamps(example_config):
@@ -45,12 +45,7 @@ def test_delete_inactive_users(example_config):
for addr in to_remove:
assert example_config.get_user(addr).maildir.exists()
main_expire(
args=[
"--remove",
str(example_config._inipath),
]
)
delete_inactive_users(example_config)
for p in example_config.mailboxes_dir.iterdir():
assert not p.name.startswith("old")

View File

@@ -1,150 +0,0 @@
import os
import random
from datetime import datetime
from fnmatch import fnmatch
from pathlib import Path
import pytest
from chatmaild.expire import (
FileEntry,
MailboxStat,
get_file_entry,
iter_mailboxes,
os_listdir_if_exists,
)
from chatmaild.expire import main as expiry_main
from chatmaild.fsreport import main as report_main
def fill_mbox(basedir):
basedir1 = basedir.joinpath("mailbox1@example.org")
basedir1.mkdir()
password = basedir1.joinpath("password")
password.write_text("xxx")
basedir1.joinpath("maildirsize").write_text("xxx")
garbagedir = basedir1.joinpath("garbagedir")
garbagedir.mkdir()
create_new_messages(basedir1, ["cur/msg1"], size=500)
create_new_messages(basedir1, ["new/msg2"], size=600)
return basedir1
def create_new_messages(basedir, relpaths, size=1000, days=0):
now = datetime.utcnow().timestamp()
for relpath in relpaths:
msg_path = Path(basedir).joinpath(relpath)
msg_path.parent.mkdir(parents=True, exist_ok=True)
msg_path.write_text("x" * size)
# accessed now, modified N days ago
os.utime(msg_path, (now, now - days * 86400))
@pytest.fixture
def mbox1(example_config):
basedir1 = fill_mbox(example_config.mailboxes_dir)
return MailboxStat(basedir1)
def test_filentry_ordering(tmp_path):
l = [FileEntry(f"x{i}", size=i + 10, mtime=1000 - i) for i in range(10)]
sorted = list(l)
random.shuffle(l)
l.sort(key=lambda x: x.size)
assert l == sorted
def test_no_mailbxoes(tmp_path, capsys):
assert [] == list(iter_mailboxes(str(tmp_path.joinpath("notexists")), maxnum=10))
out, err = capsys.readouterr()
assert "no mailboxes" in err
def test_stats_mailbox(mbox1):
password = Path(mbox1.basedir).joinpath("password")
assert mbox1.last_login == password.stat().st_mtime
assert len(mbox1.messages) == 2
msgs = list(sorted(mbox1.messages, key=lambda x: x.size))
assert len(msgs) == 2
assert msgs[0].size == 500 # cur
assert msgs[1].size == 600 # new
create_new_messages(mbox1.basedir, ["large-extra"], size=1000)
create_new_messages(mbox1.basedir, ["index-something"], size=3)
mbox2 = MailboxStat(mbox1.basedir)
assert len(mbox2.extrafiles) == 4
assert mbox2.extrafiles[0].size == 1000
# cope well with mailbox dirs that have no password (for whatever reason)
Path(mbox1.basedir).joinpath("password").unlink()
mbox3 = MailboxStat(mbox1.basedir)
assert mbox3.last_login is None
def test_report_no_mailboxes(example_config):
args = (str(example_config._inipath),)
report_main(args)
def test_report(mbox1, example_config):
args = (str(example_config._inipath),)
report_main(args)
args = list(args) + "--days 1".split()
report_main(args)
args = list(args) + "--min-login-age 1".split()
report_main(args)
args = list(args) + "--mdir cur".split()
report_main(args)
def test_expiry_cli_basic(example_config, mbox1):
args = (str(example_config._inipath),)
expiry_main(args)
def test_expiry_cli_old_files(capsys, example_config, mbox1):
relpaths_old = ["cur/msg_old1", "cur/msg_old1"]
cutoff_days = int(example_config.delete_mails_after) + 1
create_new_messages(mbox1.basedir, relpaths_old, size=1000, days=cutoff_days)
relpaths_large = ["cur/msg_old_large1", "new/msg_old_large2"]
cutoff_days = int(example_config.delete_large_after) + 1
create_new_messages(
mbox1.basedir, relpaths_large, size=1000 * 300, days=cutoff_days
)
create_new_messages(mbox1.basedir, ["cur/shouldstay"], size=1000 * 300, days=1)
args = str(example_config._inipath), "--remove", "-v"
expiry_main(args)
out, err = capsys.readouterr()
allpaths = relpaths_old + relpaths_large + ["maildirsize"]
for path in allpaths:
for line in err.split("\n"):
if fnmatch(line, f"removing*{path}"):
break
else:
if path != "new/msg_old_large2":
pytest.fail(f"failed to remove {path}\n{err}")
assert "shouldstay" not in err
def test_get_file_entry(tmp_path):
assert get_file_entry(str(tmp_path.joinpath("123123"))) is None
p = tmp_path.joinpath("x")
p.write_text("hello")
entry = get_file_entry(str(p))
assert entry.size == 5
assert entry.mtime
def test_os_listdir_if_exists(tmp_path):
tmp_path.joinpath("x").write_text("hello")
assert len(os_listdir_if_exists(str(tmp_path))) == 1
assert len(os_listdir_if_exists(str(tmp_path.joinpath("123123")))) == 0

View File

@@ -1,12 +1,11 @@
import pytest
from chatmaild.filtermail import (
IncomingBeforeQueueHandler,
OutgoingBeforeQueueHandler,
BeforeQueueHandler,
SendRateLimiter,
check_armored_payload,
check_encrypted,
is_securejoin,
common_encrypted_subjects,
)
@@ -19,13 +18,7 @@ def maildomain():
@pytest.fixture
def handler(make_config, maildomain):
config = make_config(maildomain)
return OutgoingBeforeQueueHandler(config)
@pytest.fixture
def inhandler(make_config, maildomain):
config = make_config(maildomain)
return IncomingBeforeQueueHandler(config)
return BeforeQueueHandler(config)
def test_reject_forged_from(maildata, gencreds, handler):
@@ -36,14 +29,14 @@ def test_reject_forged_from(maildata, gencreds, handler):
# test that the filter lets good mail through
to_addr = gencreds()[0]
env.content = maildata(
"encrypted.eml", from_addr=env.mail_from, to_addr=to_addr
"plain.eml", from_addr=env.mail_from, to_addr=to_addr
).as_bytes()
assert not handler.check_DATA(envelope=env)
# test that the filter rejects forged mail
env.content = maildata(
"encrypted.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
"plain.eml", from_addr="forged@c3.testrun.org", to_addr=to_addr
).as_bytes()
error = handler.check_DATA(envelope=env)
assert "500" in error
@@ -62,28 +55,19 @@ def test_filtermail_no_encryption_detection(maildata):
assert not check_encrypted(msg)
def test_filtermail_securejoin_detection(maildata):
msg = maildata(
"securejoin-vc.eml", from_addr="some@example.org", to_addr="other@example.org"
)
assert is_securejoin(msg)
msg = maildata(
"securejoin-vc-fake.eml",
from_addr="some@example.org",
to_addr="other@example.org",
)
assert not is_securejoin(msg)
def test_filtermail_encryption_detection(maildata):
msg = maildata(
"encrypted.eml",
from_addr="1@example.org",
to_addr="2@example.org",
subject="Subject does not matter, will be replaced anyway",
)
assert check_encrypted(msg)
for subject in common_encrypted_subjects:
msg = maildata(
"encrypted.eml",
from_addr="1@example.org",
to_addr="2@example.org",
subject=subject,
)
assert check_encrypted(msg)
# if the subject is not a known encrypted subject value, it is not considered ac-encrypted
msg.replace_header("Subject", "Click this link")
assert not check_encrypted(msg)
def test_filtermail_no_literal_packets(maildata):
@@ -113,7 +97,7 @@ def test_send_rate_limiter():
break
def test_cleartext_excempt_privacy(maildata, gencreds, handler):
def test_excempt_privacy(maildata, gencreds, handler):
from_addr = gencreds()[0]
to_addr = "privacy@testrun.org"
handler.config.passthrough_recipients = [to_addr]
@@ -134,73 +118,10 @@ def test_cleartext_excempt_privacy(maildata, gencreds, handler):
rcpt_tos = [to_addr, false_to]
content = msg.as_bytes()
assert "523" in handler.check_DATA(envelope=env2)
assert "500" in handler.check_DATA(envelope=env2)
def test_cleartext_self_send_autocrypt_setup_message(maildata, gencreds, handler):
from_addr = gencreds()[0]
to_addr = from_addr
msg = maildata("asm.eml", from_addr=from_addr, to_addr=to_addr)
class env:
mail_from = from_addr
rcpt_tos = [to_addr]
content = msg.as_bytes()
assert not handler.check_DATA(envelope=env)
def test_cleartext_send_fails(maildata, gencreds, handler):
from_addr = gencreds()[0]
to_addr = gencreds()[0]
msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
class env:
mail_from = from_addr
rcpt_tos = [to_addr]
content = msg.as_bytes()
res = handler.check_DATA(envelope=env)
assert "523 Encryption Needed" in res
def test_cleartext_incoming_fails(maildata, gencreds, inhandler):
from_addr = gencreds()[0]
to_addr, password = gencreds()
msg = maildata("plain.eml", from_addr=from_addr, to_addr=to_addr)
class env:
mail_from = from_addr
rcpt_tos = [to_addr]
content = msg.as_bytes()
user = inhandler.config.get_user(to_addr)
user.set_password(password)
res = inhandler.check_DATA(envelope=env)
assert "523 Encryption Needed" in res
user.allow_incoming_cleartext()
assert not inhandler.check_DATA(envelope=env)
def test_cleartext_incoming_mailer_daemon(maildata, gencreds, inhandler):
from_addr = "mailer-daemon@example.org"
to_addr = gencreds()[0]
msg = maildata("mailer-daemon.eml", from_addr=from_addr, to_addr=to_addr)
class env:
mail_from = from_addr
rcpt_tos = [to_addr]
content = msg.as_bytes()
assert not inhandler.check_DATA(envelope=env)
def test_cleartext_passthrough_domains(maildata, gencreds, handler):
def test_passthrough_domains(maildata, gencreds, handler):
from_addr = gencreds()[0]
to_addr = "privacy@x.y.z"
handler.config.passthrough_recipients = ["@x.y.z"]
@@ -221,10 +142,10 @@ def test_cleartext_passthrough_domains(maildata, gencreds, handler):
rcpt_tos = [to_addr, false_to]
content = msg.as_bytes()
assert "523" in handler.check_DATA(envelope=env2)
assert "500" in handler.check_DATA(envelope=env2)
def test_cleartext_passthrough_senders(gencreds, handler, maildata):
def test_passthrough_senders(gencreds, handler, maildata):
acc1 = gencreds()[0]
to_addr = "recipient@something.org"
handler.config.passthrough_senders = [acc1]
@@ -241,9 +162,8 @@ def test_cleartext_passthrough_senders(gencreds, handler, maildata):
def test_check_armored_payload():
prefix = "-----BEGIN PGP MESSAGE-----\r\n"
comment = "Version: ProtonMail\r\n"
payload = """\r
payload = """-----BEGIN PGP MESSAGE-----\r
\r
wU4DSqFx0d1yqAoSAQdAYkX/ZN/Az4B0k7X47zKyWrXxlDEdS3WOy0Yf2+GJTFgg\r
Zk5ql0mLG8Ze+ZifCS0XMO4otlemSyJ0K1ZPdFMGzUDBTgNqzkFabxXoXRIBB0AM\r
755wlX41X6Ay3KhnwBq7yEqSykVH6F3x11iHPKraLCAGZoaS8bKKNy/zg5slda1X\r
@@ -276,28 +196,9 @@ UN4fiB0KR9JyG2ayUdNJVkXZSZLnHyRgiaadlpUo16LVvw==\r
=b5Kp\r
-----END PGP MESSAGE-----\r
\r
\r
"""
commented_payload = prefix + comment + payload
assert check_armored_payload(commented_payload, outgoing=False) == True
assert check_armored_payload(commented_payload, outgoing=True) == False
payload = prefix + payload
assert check_armored_payload(payload, outgoing=False) == True
assert check_armored_payload(payload, outgoing=True) == True
payload = payload.removesuffix("\r\n")
assert check_armored_payload(payload, outgoing=False) == True
assert check_armored_payload(payload, outgoing=True) == True
payload = payload.removesuffix("\r\n")
assert check_armored_payload(payload, outgoing=False) == True
assert check_armored_payload(payload, outgoing=True) == True
payload = payload.removesuffix("\r\n")
assert check_armored_payload(payload, outgoing=False) == True
assert check_armored_payload(payload, outgoing=True) == True
assert check_armored_payload(payload) == True
payload = """-----BEGIN PGP MESSAGE-----\r
\r
@@ -305,8 +206,7 @@ HELLOWORLD
-----END PGP MESSAGE-----\r
\r
"""
assert check_armored_payload(payload, outgoing=False) == False
assert check_armored_payload(payload, outgoing=True) == False
assert check_armored_payload(payload) == False
payload = """-----BEGIN PGP MESSAGE-----\r
\r
@@ -314,48 +214,4 @@ HELLOWORLD
-----END PGP MESSAGE-----\r
\r
"""
assert check_armored_payload(payload, outgoing=False) == False
assert check_armored_payload(payload, outgoing=True) == False
# Test payload using partial body length
# as generated by GopenPGP.
payload = """-----BEGIN PGP MESSAGE-----\r
\r
wV4DdCVjRfOT3TQSAQdAY5+pjT6mlCxPGdR3be4w7oJJRUGIPI/Vnh+mJxGSm34w\r
LNlVc89S1g22uQYFif2sUJsQWbpoHpNkuWpkSgOaHmNvrZiY/YU5iv+cZ3LbmtUG\r
0uoBisSHh9O1c+5sYZSbrvYZ1NOwlD7Fv/U5/Mw4E5+CjxfdgNGp5o3DDddzPK78\r
jseDhdSXxnaiIJC93hxNX6R1RPt3G2gukyzx69wciPQShcF8zf3W3o75Ed7B8etV\r
QEeB16xzdFhKa9JxdjTu3osgCs21IO7wpcFkjc7nZzlW6jPnELJJaNmv4yOOCjMp\r
6YAkaN/BkL+jHTznHDuDsT5ilnTXpwHDU1Cm9PIx/KFcNCQnIB+2DcdIHPHUH1ci\r
jvqoeXAVWjKXEjS7PqPFuP/xGbrWG2ugs+toXJOKbgRkExvKs1dwPFKrgghvCVbW\r
AcKejQKAPArLwpkA7aD875TZQShvGt74fNs45XBlGOYOnNOAJ1KAmzrXLIDViyyB\r
kDsmTBk785xofuCkjBpXSe6vsMprPzCteDfaUibh8FHeJjucxPerwuOPEmnogNaf\r
YyL4+iy8H8I9/p7pmUqILprxTG0jTOtlk0bTVzeiF56W1xbtSEMuOo4oFbQTyOM2\r
bKXaYo774Jm+rRtKAnnI2dtf9RpK19cog6YNzfYjesLKbXDsPZbN5rmwyFiCvvxC\r
kQ6JLob+B2fPdY2gzy7LypxktS8Zi1HJcWDHJGVmQodaDLqKUObb4M26bXDe6oxI\r
NS8PJz5exVbM3KhZnUOEn6PJRBBf5a/ZqxlhZPcQo/oBuhKpBRpO5kSDwPIUByu3\r
UlXLSkpMqe9pUarAOEuQjfl2RVY7U+RrQYp4YP5keMO+i8NCefAFbowTTufO1JIq\r
2nVgCi/QVnxZyEc9OYt/8AE3g4cdojE+vsSDifZLSWYIetpfrohHv3dT3StD1QRG\r
0QE6qq6oKpg/IL0cjvuX4c7a7bslv2fXp8t75y37RU6253qdIebhxc/cRhPbc/yu\r
p0YLyD4SrvKTLP2ZV95jT4IPEpqm4AN3QmiOzdtqR2gLyb62L8QfqI/FdwsIiRiM\r
hqydwoqt/lfSqG1WKPh+6EkMkH+TDiCC1BQdbN1MNcyUtcjb35PR2c8Ld2TF3guA\r
jLIqMt/Vb7hBoMb2FcsOYY25ka9oV62OwgKWLXnFzk+modMR5fzb4kxVVAYEqP+D\r
T5KO1Vs76v1fyPGOq6BbBCvLwTqe/e6IZInJles4v5jrhnLcGKmNGivCUDe6X6NY\r
UKNt5RsZllwDQpaAb5dMNhyrk8SgIE7TBI7rvqIdUCE52Vy+0JDxFg5olRpFUfO6\r
/MyTW3Yo/ekk/npHr7iYYqJTCc21bDGLWQcIo/XO7WPxrKNWGBNPFnkRdw0MaKr4\r
+cEM3V8NFnSEpC12xA+RX/CezuJtwXZK5MpG76eYqMO6qyC+c25YcFecEufDZDxx\r
ZLqRszVRyxyWPtk/oIeQK2v9wOqY6N9/ff01gHz69vqYqN5bUw/QKZsmx1zW+gPw\r
6x2tDK2BHeYl182gCbhlKISRFwCtbjqZSkiKWao/VtygHkw0fK34avJuyQ/X9YaN\r
BRy+7Lf3VA53pnB5WJ1xwRXN8VDvmZeXzv2krHveCMemj0OjnRoCLu117xN0A5m9\r
Fm/RoDix5PolDHtWTtr2m1n2hp2LHnj8at9lFEd0SKhAYHVL9KjzycwWODZRXt+x\r
zGDDuooEeTvdY5NLyKcl4gETz1ZP4Ez5jGGjhPSwSpq1mU7UaJ9ZXXdr4KHyifW6\r
ggNzNsGhXTap7IWZpTtqXABydfiBshmH2NjqtNDwBweJVSgP10+r0WhMWlaZs6xl\r
V3o5yskJt6GlkwpJxZrTvN6Tiww/eW7HFV6NGf7IRSWY5tJc/iA7/92tOmkdvJ1q\r
myLbG7cJB787QjplEyVe2P/JBO6xYvbkJLf9Q+HaviTO25rugRSrYsoKMDfO8VlQ\r
1CcnTPVtApPZJEQzAWJEgVAM8uIlkqWJJMgyWT34sTkdBeCUFGloXQFs9Yxd0AGf\r
/zHEkYZSTKpVSvAIGu4=\r
=6iHb\r
-----END PGP MESSAGE-----\r
"""
assert check_armored_payload(payload, outgoing=False) == True
assert check_armored_payload(payload, outgoing=True) == True
assert check_armored_payload(payload) == False

View File

@@ -1,78 +0,0 @@
import smtplib
import subprocess
import sys
import pytest
@pytest.fixture
def smtpserver():
from pytest_localserver import smtp
server = smtp.Server("127.0.0.1")
server.start()
yield server
server.stop()
@pytest.fixture
def make_popen(request):
def popen(cmdargs, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kw):
p = subprocess.Popen(
cmdargs,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
def fin():
p.terminate()
out, err = p.communicate()
print(out.decode("ascii"))
print(err.decode("ascii"), file=sys.stderr)
request.addfinalizer(fin)
return p
return popen
@pytest.mark.parametrize("filtermail_mode", ["outgoing", "incoming"])
def test_one_mail(
make_config, make_popen, smtpserver, maildata, filtermail_mode, monkeypatch
):
monkeypatch.setenv("PYTHONUNBUFFERED", "1")
smtp_inject_port = 20025
if filtermail_mode == "outgoing":
settings = dict(
postfix_reinject_port=smtpserver.port,
filtermail_smtp_port=smtp_inject_port,
)
else:
settings = dict(
postfix_reinject_port_incoming=smtpserver.port,
filtermail_smtp_port_incoming=smtp_inject_port,
)
config = make_config("example.org", settings=settings)
path = str(config._inipath)
popen = make_popen(["filtermail", path, filtermail_mode])
line = popen.stderr.readline().strip()
if b"loop" not in line:
print(line.decode("ascii"), file=sys.stderr)
pytest.fail("starting filtermail failed")
addr = f"user1@{config.mail_domain}"
config.get_user(addr).set_password("l1k2j3l1k2j3l")
# send encrypted mail
data = str(maildata("encrypted.eml", from_addr=addr, to_addr=addr))
client = smtplib.SMTP("localhost", smtp_inject_port)
client.sendmail(addr, [addr], data)
assert len(smtpserver.outbox) == 1
# send un-encrypted mail that errors
data = str(maildata("fake-encrypted.eml", from_addr=addr, to_addr=addr))
with pytest.raises(smtplib.SMTPDataError) as e:
client.sendmail(addr, [addr], data)
assert e.value.smtp_code == 523

View File

@@ -242,22 +242,6 @@ def test_requeue_removes_tmp_files(notifier, metadata, testaddr, caplog):
assert queue_item.addr == testaddr
def test_requeue_removes_invalid_files(notifier, metadata, testaddr, caplog):
metadata.add_token_to_addr(testaddr, "01234")
notifier.new_message_for_addr(testaddr, metadata)
# empty/invalid files should be ignored
p = notifier.queue_dir.joinpath("1203981203")
p.touch()
notifier2 = notifier.__class__(notifier.queue_dir)
notifier2.requeue_persistent_queue_items()
assert "spurious" in caplog.records[0].msg
assert not p.exists()
assert notifier2.retry_queues[0].qsize() == 1
when, queue_item = notifier2.retry_queues[0].get()
assert when <= int(time.time())
assert queue_item.addr == testaddr
def test_start_and_stop_notification_threads(notifier, testaddr):
threads = notifier.start_notification_threads(None)
for retry_num, threadlist in threads.items():

View File

@@ -2,15 +2,8 @@ from chatmaild.metrics import main
def test_main(tmp_path, capsys):
paths = []
for x in ("ci-asllkj", "ac_12l3kj", "qweqwe", "ci-l1k2j31l2k3"):
p = tmp_path.joinpath(x)
p.mkdir()
p.joinpath("cur").mkdir()
paths.append(p)
tmp_path.joinpath("nomailbox").mkdir()
tmp_path.joinpath(x).mkdir()
main(tmp_path)
out, _ = capsys.readouterr()
d = {}

View File

@@ -40,17 +40,3 @@ def test_no_mailboxes_dir(testaddr, example_config, tmp_path):
user.set_password("someeqkjwelkqwjleqwe")
user.set_last_login_timestamp(100000)
assert user.get_last_login_timestamp() == 86400
def test_set_get_cleartext_flag(testaddr, example_config, tmp_path):
p = tmp_path.joinpath("a", "mailboxes")
example_config.mailboxes_dir = p
user = example_config.get_user(testaddr)
user.set_password("someeqkjwelkqwjleqwe")
user.set_last_login_timestamp(100000)
assert user.get_last_login_timestamp() == 86400
assert not user.is_incoming_cleartext_ok()
user.allow_incoming_cleartext()
assert user.is_incoming_cleartext_ok()

View File

@@ -1,9 +0,0 @@
#!/usr/bin/env python3
import socket
def turn_credentials() -> str:
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_socket:
client_socket.connect("/run/chatmail-turn/turn.socket")
with client_socket.makefile("rb") as file:
return file.readline().decode("utf-8")

View File

@@ -13,7 +13,6 @@ class User:
self.maildir = maildir
self.addr = addr
self.password_path = password_path
self.enforce_E2EE_path = maildir.joinpath("enforceE2EEincoming")
self.uid = uid
self.gid = gid
@@ -36,13 +35,6 @@ class User:
home = str(self.maildir)
return dict(addr=self.addr, home=home, uid=self.uid, gid=self.gid, password=pw)
def is_incoming_cleartext_ok(self):
return not self.enforce_E2EE_path.exists()
def allow_incoming_cleartext(self):
if self.enforce_E2EE_path.exists():
self.enforce_E2EE_path.unlink()
def set_password(self, enc_password):
"""Set the specified password for this user.
@@ -58,8 +50,6 @@ class User:
if not self.addr.startswith("echo@"):
logging.error(f"could not write password for: {self.addr}")
raise
if not self.addr.startswith("echo@"):
self.enforce_E2EE_path.touch()
def set_last_login_timestamp(self, timestamp):
"""Track login time with daily granularity

View File

@@ -41,6 +41,3 @@ lint.select = [
"PLE", # Pylint Error
"PLW", # Pylint Warning
]
lint.ignore = [
"PLC0415" # import-outside-top-level
]

View File

@@ -7,35 +7,17 @@ import io
import shutil
import subprocess
import sys
from io import StringIO
from pathlib import Path
from chatmaild.config import Config, read_config
from pyinfra import facts, host, logger
from pyinfra.api import FactBase
from pyinfra.facts.files import File, Sha256File
from pyinfra.facts.server import Sysctl
from pyinfra import facts, host
from pyinfra.facts.files import File
from pyinfra.facts.systemd import SystemdEnabled
from pyinfra.operations import apt, files, pip, server, systemd
from .acmetool import deploy_acmetool
class Port(FactBase):
"""
Returns the process occuping a port.
"""
def command(self, port: int) -> str:
return (
"ss -lptn 'src :%d' | awk 'NR>1 {print $6,$7}' | sed 's/users:((\"//;s/\".*//'"
% (port,)
)
def process(self, output: [str]) -> str:
return output[0]
def _build_chatmaild(dist_dir) -> None:
dist_dir = Path(dist_dir).resolve()
if dist_dir.exists():
@@ -97,8 +79,8 @@ def _install_remote_venv_with_chatmaild(config) -> None:
)
apt.packages(
name="install gcc and headers to build crypt_r source package",
packages=["gcc", "python3-dev"],
name="install python3-dev to build crypt_r source package",
packages=["python3-dev"],
)
server.shell(
@@ -124,51 +106,37 @@ def _install_remote_venv_with_chatmaild(config) -> None:
for fn in (
"doveauth",
"filtermail",
"filtermail-incoming",
"echobot",
"chatmail-metadata",
"lastlogin",
"turnserver",
"chatmail-expire",
"chatmail-expire.timer",
"chatmail-fsreport",
"chatmail-fsreport.timer",
):
execpath = fn if fn != "filtermail-incoming" else "filtermail"
params = dict(
execpath=f"{remote_venv_dir}/bin/{execpath}",
execpath=f"{remote_venv_dir}/bin/{fn}",
config_path=remote_chatmail_inipath,
remote_venv_dir=remote_venv_dir,
mail_domain=config.mail_domain,
)
basename = fn if "." in fn else f"{fn}.service"
source_path = importlib.resources.files(__package__).joinpath("service", f"{basename}.f")
source_path = importlib.resources.files(__package__).joinpath(
"service", f"{fn}.service.f"
)
content = source_path.read_text().format(**params).encode()
files.put(
name=f"Upload {basename}",
name=f"Upload {fn}.service",
src=io.BytesIO(content),
dest=f"/etc/systemd/system/{basename}",
dest=f"/etc/systemd/system/{fn}.service",
**root_owned,
)
if fn == "chatmail-expire" or fn == "chatmail-fsreport":
# don't auto-start but let the corresponding timer trigger execution
enabled = False
else:
enabled = True
systemd.service(
name=f"Setup {basename}",
service=basename,
running=enabled,
enabled=enabled,
restarted=enabled,
name=f"Setup {fn} service",
service=f"{fn}.service",
running=True,
enabled=True,
restarted=True,
daemon_reload=True,
)
def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
"""Configures OpenDKIM"""
need_restart = False
@@ -247,37 +215,51 @@ def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
server.shell(
name="Generate OpenDKIM domain keys",
commands=[
f"/usr/sbin/opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
f"opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
],
_use_su_login=True,
_su_user="opendkim",
_sudo=True,
_sudo_user="opendkim",
)
service_file = files.put(
name="Configure opendkim to restart once a day",
src=importlib.resources.files(__package__).joinpath("opendkim/systemd.conf"),
dest="/etc/systemd/system/opendkim.service.d/10-prevent-memory-leak.conf",
)
need_restart |= service_file.changed
return need_restart
def _uninstall_mta_sts_daemon() -> None:
# Remove configuration.
files.file("/etc/mta-sts-daemon.yml", present=False)
def _install_mta_sts_daemon() -> bool:
need_restart = False
files.directory("/usr/local/lib/postfix-mta-sts-resolver", present=False)
files.file("/etc/systemd/system/mta-sts-daemon.service", present=False)
systemd.service(
name="Stop MTA-STS daemon",
service="mta-sts-daemon.service",
daemon_reload=True,
running=False,
enabled=False,
config = files.put(
name="upload postfix-mta-sts-resolver config",
src=importlib.resources.files(__package__).joinpath(
"postfix/mta-sts-daemon.yml"
),
dest="/etc/mta-sts-daemon.yml",
user="root",
group="root",
mode="644",
)
need_restart |= config.changed
server.shell(
name="install postfix-mta-sts-resolver with pip",
commands=[
"python3 -m virtualenv /usr/local/lib/postfix-mta-sts-resolver",
"/usr/local/lib/postfix-mta-sts-resolver/bin/pip install postfix-mta-sts-resolver",
],
)
systemd_unit = files.put(
name="upload mta-sts-daemon systemd unit",
src=importlib.resources.files(__package__).joinpath(
"postfix/mta-sts-daemon.service"
),
dest="/etc/systemd/system/mta-sts-daemon.service",
user="root",
group="root",
mode="644",
)
need_restart |= systemd_unit.changed
return need_restart
def _configure_postfix(config: Config, debug: bool = False) -> bool:
@@ -330,40 +312,6 @@ def _configure_postfix(config: Config, debug: bool = False) -> bool:
return need_restart
def _install_dovecot_package(package: str, arch: str):
arch = "amd64" if arch == "x86_64" else arch
arch = "arm64" if arch == "aarch64" else arch
url = f"https://download.delta.chat/dovecot/dovecot-{package}_2.3.21%2Bdfsg1-3_{arch}.deb"
deb_filename = "/root/" + url.split("/")[-1]
match (package, arch):
case ("core", "amd64"):
sha256 = "dd060706f52a306fa863d874717210b9fe10536c824afe1790eec247ded5b27d"
case ("core", "arm64"):
sha256 = "e7548e8a82929722e973629ecc40fcfa886894cef3db88f23535149e7f730dc9"
case ("imapd", "amd64"):
sha256 = "8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86"
case ("imapd", "arm64"):
sha256 = "178fa877ddd5df9930e8308b518f4b07df10e759050725f8217a0c1fb3fd707f"
case ("lmtpd", "amd64"):
sha256 = "2f69ba5e35363de50962d42cccbfe4ed8495265044e244007d7ccddad77513ab"
case ("lmtpd", "arm64"):
sha256 = "89f52fb36524f5877a177dff4a713ba771fd3f91f22ed0af7238d495e143b38f"
case _:
apt.packages(packages=[f"dovecot-{package}"])
return
files.download(
name=f"Download dovecot-{package}",
src=url,
dest=deb_filename,
sha256sum=sha256,
cache_time=60 * 60 * 24 * 365 * 10, # never redownload the package
)
apt.deb(name=f"Install dovecot-{package}", src=deb_filename)
def _configure_dovecot(config: Config, debug: bool = False) -> bool:
"""Configures Dovecot IMAP server."""
need_restart = False
@@ -398,21 +346,19 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
)
need_restart |= lua_push_notification_script.changed
# remove historic expunge script
# which is now implemented through a systemd chatmail-expire service/timer
files.file(
path="/etc/cron.d/expunge",
present=False,
files.template(
src=importlib.resources.files(__package__).joinpath("dovecot/expunge.cron.j2"),
dest="/etc/cron.d/expunge",
user="root",
group="root",
mode="644",
config=config,
)
# as per https://doc.dovecot.org/configuration_manual/os/
# it is recommended to set the following inotify limits
for name in ("max_user_instances", "max_user_watches"):
key = f"fs.inotify.{name}"
if host.get_fact(Sysctl)[key] > 65535:
# Skip updating limits if already sufficient
# (enables running in incus containers where sysctl readonly)
continue
server.sysctl(
name=f"Change {key}",
key=key,
@@ -420,13 +366,6 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
persist=True,
)
timezone_env = files.line(
name="Set TZ environment variable",
path="/etc/environment",
line="TZ=:/etc/localtime",
)
need_restart |= timezone_env.changed
return need_restart
@@ -507,77 +446,10 @@ def check_config(config):
return config
def deploy_turn_server(config):
(url, sha256sum) = {
"x86_64": (
"https://github.com/chatmail/chatmail-turn/releases/download/v0.3/chatmail-turn-x86_64-linux",
"841e527c15fdc2940b0469e206188ea8f0af48533be12ecb8098520f813d41e4",
),
"aarch64": (
"https://github.com/chatmail/chatmail-turn/releases/download/v0.3/chatmail-turn-aarch64-linux",
"a5fc2d06d937b56a34e098d2cd72a82d3e89967518d159bf246dc69b65e81b42",
),
}[host.get_fact(facts.server.Arch)]
need_restart = False
existing_sha256sum = host.get_fact(Sha256File, "/usr/local/bin/chatmail-turn")
if existing_sha256sum != sha256sum:
server.shell(
name="Download chatmail-turn",
commands=[
f"(curl -L {url} >/usr/local/bin/chatmail-turn.new && (echo '{sha256sum} /usr/local/bin/chatmail-turn.new' | sha256sum -c) && mv /usr/local/bin/chatmail-turn.new /usr/local/bin/chatmail-turn)",
"chmod 755 /usr/local/bin/chatmail-turn",
],
)
need_restart = True
source_path = importlib.resources.files(__package__).joinpath(
"service", "turnserver.service.f"
)
content = source_path.read_text().format(mail_domain=config.mail_domain).encode()
systemd_unit = files.put(
name="Upload turnserver.service",
src=io.BytesIO(content),
dest="/etc/systemd/system/turnserver.service",
user="root",
group="root",
mode="644",
)
need_restart |= systemd_unit.changed
systemd.service(
name="Setup turnserver service",
service="turnserver.service",
running=True,
enabled=True,
restarted=need_restart,
daemon_reload=systemd_unit.changed,
)
def deploy_mtail(config):
# Uninstall mtail package, we are going to install a static binary.
apt.packages(name="Uninstall mtail", packages=["mtail"], present=False)
(url, sha256sum) = {
"x86_64": (
"https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_amd64.tar.gz",
"123c2ee5f48c3eff12ebccee38befd2233d715da736000ccde49e3d5607724e4",
),
"aarch64": (
"https://github.com/google/mtail/releases/download/v3.0.8/mtail_3.0.8_linux_arm64.tar.gz",
"aa04811c0929b6754408676de520e050c45dddeb3401881888a092c9aea89cae",
),
}[host.get_fact(facts.server.Arch)]
server.shell(
name="Download mtail",
commands=[
f"(echo '{sha256sum} /usr/local/bin/mtail' | sha256sum -c) || (curl -L {url} | gunzip | tar -x -f - mtail -O >/usr/local/bin/mtail.new && mv /usr/local/bin/mtail.new /usr/local/bin/mtail)",
"chmod 755 /usr/local/bin/mtail",
],
apt.packages(
name="Install mtail",
packages=["mtail"],
)
# Using our own systemd unit instead of `/usr/lib/systemd/system/mtail.service`.
@@ -615,12 +487,12 @@ def deploy_mtail(config):
def deploy_iroh_relay(config) -> None:
(url, sha256sum) = {
"x86_64": (
"https://github.com/n0-computer/iroh/releases/download/v0.35.0/iroh-relay-v0.35.0-x86_64-unknown-linux-musl.tar.gz",
"45c81199dbd70f8c4c30fef7f3b9727ca6e3cea8f2831333eeaf8aa71bf0fac1",
"https://github.com/n0-computer/iroh/releases/download/v0.28.1/iroh-relay-v0.28.1-x86_64-unknown-linux-musl.tar.gz",
"2ffacf7c0622c26b67a5895ee8e07388769599f60e5f52a3bd40a3258db89b2c",
),
"aarch64": (
"https://github.com/n0-computer/iroh/releases/download/v0.35.0/iroh-relay-v0.35.0-aarch64-unknown-linux-musl.tar.gz",
"f8ef27631fac213b3ef668d02acd5b3e215292746a3fc71d90c63115446008b1",
"https://github.com/n0-computer/iroh/releases/download/v0.28.1/iroh-relay-v0.28.1-aarch64-unknown-linux-musl.tar.gz",
"b915037bcc1ff1110cc9fcb5de4a17c00ff576fd2f568cd339b3b2d54c420dc4",
),
}[host.get_fact(facts.server.Arch)]
@@ -629,18 +501,15 @@ def deploy_iroh_relay(config) -> None:
packages=["curl"],
)
need_restart = False
server.shell(
name="Download iroh-relay",
commands=[
f"(echo '{sha256sum} /usr/local/bin/iroh-relay' | sha256sum -c) || (curl -L {url} | gunzip | tar -x -f - ./iroh-relay -O >/usr/local/bin/iroh-relay.new && mv /usr/local/bin/iroh-relay.new /usr/local/bin/iroh-relay)",
"chmod 755 /usr/local/bin/iroh-relay",
],
)
existing_sha256sum = host.get_fact(Sha256File, "/usr/local/bin/iroh-relay")
if existing_sha256sum != sha256sum:
server.shell(
name="Download iroh-relay",
commands=[
f"(curl -L {url} | gunzip | tar -x -f - ./iroh-relay -O >/usr/local/bin/iroh-relay.new && (echo '{sha256sum} /usr/local/bin/iroh-relay.new' | sha256sum -c) && mv /usr/local/bin/iroh-relay.new /usr/local/bin/iroh-relay)",
"chmod 755 /usr/local/bin/iroh-relay",
],
)
need_restart = True
need_restart = False
systemd_unit = files.put(
name="Upload iroh-relay systemd unit",
@@ -681,10 +550,11 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
check_config(config)
mail_domain = config.mail_domain
from .www import build_webpages, get_paths
from .www import build_webpages
server.group(name="Create vmail group", group="vmail", system=True)
server.user(name="Create vmail user", user="vmail", group="vmail", system=True)
server.user(name="Create filtermail user", user="filtermail", system=True)
server.group(name="Create opendkim group", group="opendkim", system=True)
server.user(
name="Create opendkim user",
@@ -716,15 +586,9 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
path="/etc/apt/sources.list",
line="deb [signed-by=/etc/apt/keyrings/obs-home-deltachat.gpg] https://download.opensuse.org/repositories/home:/deltachat/Debian_12/ ./",
escape_regex_characters=True,
present=False,
ensure_newline=True,
)
if host.get_fact(Port, port=53) != "unbound":
files.line(
name="Add 9.9.9.9 to resolv.conf",
path="/etc/resolv.conf",
line="nameserver 9.9.9.9",
)
apt.update(name="apt update", cache_time=24 * 3600)
apt.upgrade(name="upgrade apt packages", auto_remove=True)
@@ -733,39 +597,9 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
packages=["rsync"],
)
deploy_turn_server(config)
# Run local DNS resolver `unbound`.
# `resolvconf` takes care of setting up /etc/resolv.conf
# to use 127.0.0.1 as the resolver.
from cmdeploy.cmdeploy import Out
port_services = [
(["master", "smtpd"], 25),
("unbound", 53),
("acmetool", 80),
(["imap-login", "dovecot"], 143),
("nginx", 443),
(["master", "smtpd"], 465),
(["master", "smtpd"], 587),
(["imap-login", "dovecot"], 993),
("iroh-relay", 3340),
("nginx", 8443),
(["master", "smtpd"], config.postfix_reinject_port),
(["master", "smtpd"], config.postfix_reinject_port_incoming),
("filtermail", config.filtermail_smtp_port),
("filtermail", config.filtermail_smtp_port_incoming),
]
for service, port in port_services:
print(f"Checking if port {port} is available for {service}...")
running_service = host.get_fact(Port, port=port)
if running_service:
if running_service not in service:
Out().red(
f"Deploy failed: port {port} is occupied by: {running_service}"
)
exit(1)
apt.packages(
name="Install unbound",
packages=["unbound", "unbound-anchor", "dnsutils"],
@@ -789,7 +623,6 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
# Deploy acmetool to have TLS certificates.
tls_domains = [mail_domain, f"mta-sts.{mail_domain}", f"www.{mail_domain}"]
deploy_acmetool(
email=config.acme_email,
domains=tls_domains,
)
@@ -804,10 +637,10 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
packages="postfix",
)
if not "dovecot.service" in host.get_fact(SystemdEnabled):
_install_dovecot_package("core", host.get_fact(facts.server.Arch))
_install_dovecot_package("imapd", host.get_fact(facts.server.Arch))
_install_dovecot_package("lmtpd", host.get_fact(facts.server.Arch))
apt.packages(
name="Install Dovecot",
packages=["dovecot-imapd", "dovecot-lmtpd"],
)
apt.packages(
name="Install nginx",
@@ -819,23 +652,19 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
packages=["fcgiwrap"],
)
www_path, src_dir, build_dir = get_paths(config)
# if www_folder was set to a non-existing folder, skip upload
if not www_path.is_dir():
logger.warning("Building web pages is disabled in chatmail.ini, skipping")
else:
# if www_folder is a hugo page, build it
if build_dir:
www_path = build_webpages(src_dir, build_dir, config)
# if it is not a hugo page, upload it as is
files.rsync(f"{www_path}/", "/var/www/html", flags=["-avz", "--chown=www-data"])
www_path = importlib.resources.files(__package__).joinpath("../../../www").resolve()
build_dir = www_path.joinpath("build")
src_dir = www_path.joinpath("src")
build_webpages(src_dir, build_dir, config)
files.rsync(f"{build_dir}/", "/var/www/html", flags=["-avz"])
_install_remote_venv_with_chatmaild(config)
debug = False
dovecot_need_restart = _configure_dovecot(config, debug=debug)
postfix_need_restart = _configure_postfix(config, debug=debug)
mta_sts_need_restart = _install_mta_sts_daemon()
nginx_need_restart = _configure_nginx(config)
_uninstall_mta_sts_daemon()
_remove_rspamd()
opendkim_need_restart = _configure_opendkim(mail_domain, "opendkim")
@@ -845,10 +674,18 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
service="opendkim.service",
running=True,
enabled=True,
daemon_reload=opendkim_need_restart,
restarted=opendkim_need_restart,
)
systemd.service(
name="Start and enable MTA-STS daemon",
service="mta-sts-daemon.service",
daemon_reload=True,
running=True,
enabled=True,
restarted=mta_sts_need_restart,
)
# Dovecot should be started before Postfix
# because it creates authentication socket
# required by Postfix.
@@ -876,19 +713,6 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
restarted=nginx_need_restart,
)
systemd.service(
name="Start and enable fcgiwrap",
service="fcgiwrap.service",
running=True,
enabled=True,
)
systemd.service(
name="Restart echobot if postfix and dovecot were just started",
service="echobot.service",
restarted=postfix_need_restart and dovecot_need_restart,
)
# This file is used by auth proxy.
# https://wiki.debian.org/EtcMailName
server.shell(
@@ -911,29 +735,10 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
enabled=True,
restarted=journald_conf.changed,
)
files.directory(
name="Ensure old logs on disk are deleted",
path="/var/log/journal/",
present=False,
)
apt.packages(
name="Ensure cron is installed",
packages=["cron"],
)
try:
git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
except Exception:
git_hash = "unknown\n"
try:
git_diff = subprocess.check_output(["git", "diff"]).decode()
except Exception:
git_diff = ""
files.put(
name="Upload chatmail relay git commiit hash",
src=StringIO(git_hash + git_diff),
dest="/etc/chatmail-version",
mode="700",
)
deploy_mtail(config)

View File

@@ -1,5 +1,7 @@
import importlib.resources
from pyinfra import host
from pyinfra.facts.systemd import SystemdStatus
from pyinfra.operations import apt, files, server, systemd
@@ -52,6 +54,12 @@ def deploy_acmetool(email="", domains=[]):
group="root",
mode="644",
)
if host.get_fact(SystemdStatus).get("nginx.service"):
systemd.service(
name="Stop nginx service to free port 80",
service="nginx",
running=False,
)
systemd.service(
name="Setup acmetool-redirector service",

View File

@@ -1,2 +1,2 @@
"acme-enter-email": "{{ email }}"
"acme-agreement:https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf": true
"acme-agreement:https://letsencrypt.org/documents/LE-SA-v1.4-April-3-2024.pdf": true

View File

@@ -1,8 +1,7 @@
request:
provider: https://acme-v02.api.letsencrypt.org/directory
key:
type: ecdsa
ecdsa-curve: nistp256
type: rsa
challenge:
webroot-paths:
- /var/www/html/.well-known/acme-challenge

View File

@@ -16,7 +16,7 @@ www.{{ mail_domain }}. CNAME {{ mail_domain }}.
;
; Recommended DNS entries for interoperability and security-hardening
;
{{ mail_domain }}. TXT "v=spf1 a ~all"
{{ mail_domain }}. TXT "v=spf1 a:{{ mail_domain }} ~all"
_dmarc.{{ mail_domain }}. TXT "v=DMARC1;p=reject;adkim=s;aspf=s"
{% if acme_account_url %}

View File

@@ -19,7 +19,7 @@ from packaging import version
from termcolor import colored
from . import dns, remote
from .sshexec import SSHExec, LocalExec
from .sshexec import SSHExec
#
# cmdeploy sub commands and options
@@ -32,30 +32,17 @@ def init_cmd_options(parser):
action="store",
help="fully qualified DNS domain name for your chatmail instance",
)
parser.add_argument(
"--force",
dest="recreate_ini",
action="store_true",
help="force reacreate ini file",
)
def init_cmd(args, out):
"""Initialize chatmail config file."""
mail_domain = args.chatmail_domain
inipath = args.inipath
if args.inipath.exists():
if not args.recreate_ini:
print(f"[WARNING] Path exists, not modifying: {inipath}")
return 1
else:
print(
f"[WARNING] Force argument was provided, deleting config file: {inipath}"
)
inipath.unlink()
write_initial_config(inipath, mail_domain, overrides={})
out.green(f"created config file for {mail_domain} in {inipath}")
print(f"Path exists, not modifying: {args.inipath}")
return 1
else:
write_initial_config(args.inipath, mail_domain, overrides={})
out.green(f"created config file for {mail_domain} in {args.inipath}")
def run_cmd_options(parser):
@@ -72,24 +59,20 @@ def run_cmd_options(parser):
help="install/upgrade the server, but disable postfix & dovecot for now",
)
parser.add_argument(
"--skip-dns-check",
dest="dns_check_disabled",
action="store_true",
help="disable checks nslookup for dns",
"--ssh-host",
dest="ssh_host",
help="specify an SSH host to deploy to; uses mail_domain from chatmail.ini by default",
)
add_ssh_host_option(parser)
def run_cmd(args, out):
"""Deploy chatmail services on the remote server."""
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
sshexec = get_sshexec(ssh_host)
sshexec = args.get_sshexec()
require_iroh = args.config.enable_iroh_relay
if not args.dns_check_disabled:
remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
if not dns.check_initial_remote_data(remote_data, print=out.red):
return 1
remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
if not dns.check_initial_remote_data(remote_data, print=out.red):
return 1
env = os.environ.copy()
env["CHATMAIL_INI"] = args.inipath
@@ -97,37 +80,21 @@ def run_cmd(args, out):
env["CHATMAIL_REQUIRE_IROH"] = "True" if require_iroh else ""
deploy_path = importlib.resources.files(__package__).joinpath("deploy.py").resolve()
pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
ssh_host = args.config.mail_domain if not args.ssh_host else args.ssh_host
cmd = f"{pyinf} --ssh-user root {ssh_host} {deploy_path} -y"
if ssh_host in ["localhost", "@docker"]:
cmd = f"{pyinf} @local {deploy_path} -y"
if version.parse(pyinfra.__version__) < version.parse("3"):
out.red("Please re-run scripts/initenv.sh to update pyinfra to version 3.")
return 1
try:
retcode = out.check_call(cmd, env=env)
if retcode == 0:
if not args.disable_mail:
print("\nYou can try out the relay by talking to this echo bot: ")
sshexec = SSHExec(args.config.mail_domain, verbose=args.verbose)
print(
sshexec(
call=remote.rshell.shell,
kwargs=dict(command="cat /var/lib/echobot/invite-link.txt"),
)
)
out.green("Deploy completed, call `cmdeploy dns` next.")
elif not remote_data["acme_account_url"]:
out.red("Deploy completed but letsencrypt not configured")
out.red("Run 'cmdeploy run' again")
retcode = 0
else:
out.red("Deploy failed")
except subprocess.CalledProcessError:
retcode = out.check_call(cmd, env=env)
if retcode == 0:
out.green("Deploy completed, call `cmdeploy dns` next.")
elif not remote_data["acme_account_url"]:
out.red("Deploy completed but letsencrypt not configured")
out.red("Run 'cmdeploy run' again")
retcode = 0
else:
out.red("Deploy failed")
retcode = 1
return retcode
@@ -139,13 +106,11 @@ def dns_cmd_options(parser):
default=None,
help="write out a zonefile",
)
add_ssh_host_option(parser)
def dns_cmd(args, out):
"""Check DNS entries and optionally generate dns zone file."""
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
sshexec = get_sshexec(ssh_host, verbose=args.verbose)
sshexec = args.get_sshexec()
remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
if not remote_data:
return 1
@@ -299,15 +264,6 @@ class Out:
return proc.returncode
def add_ssh_host_option(parser):
parser.add_argument(
"--ssh-host",
dest="ssh_host",
help="Run commands on 'localhost', via '@docker', or on a specific SSH host "
"instead of chatmail.ini's mail_domain.",
)
def add_config_option(parser):
parser.add_argument(
"--config",
@@ -363,16 +319,6 @@ def get_parser():
return parser
def get_sshexec(ssh_host: str, verbose=True):
if ssh_host in ["localhost", "@local"]:
return LocalExec(verbose, docker=False)
elif ssh_host == "@docker":
return LocalExec(verbose, docker=True)
if verbose:
print(f"[ssh] login to {ssh_host}")
return SSHExec(ssh_host, verbose=verbose)
def main(args=None):
"""Provide main entry point for 'cmdeploy' CLI invocation."""
parser = get_parser()
@@ -380,6 +326,12 @@ def main(args=None):
if not hasattr(args, "func"):
return parser.parse_args(["-h"])
def get_sshexec():
print(f"[ssh] login to {args.config.mail_domain}")
return SSHExec(args.config.mail_domain, verbose=args.verbose)
args.get_sshexec = get_sshexec
out = Out()
kwargs = {}
if args.func.__name__ not in ("init_cmd", "fmt_cmd"):

View File

@@ -45,7 +45,8 @@ def check_full_zone(sshexec, remote_data, out, zonefile) -> int:
and return (exitcode, remote_data) tuple."""
required_diff, recommended_diff = sshexec.logged(
remote.rdns.check_zonefile, kwargs=dict(zonefile=zonefile, verbose=False),
remote.rdns.check_zonefile,
kwargs=dict(zonefile=zonefile, mail_domain=remote_data["mail_domain"]),
)
returncode = 0

View File

@@ -70,12 +70,6 @@ userdb {
# Mailboxes are stored in the "mail" directory of the vmail user home.
mail_location = maildir:{{ config.mailboxes_dir }}/%u
# index/cache files are not very useful for chatmail relay operations
# but it's not clear how to disable them completely.
# According to https://doc.dovecot.org/2.3/settings/advanced/#core_setting-mail_cache_max_size
# if the cache file becomes larger than the specified size, it is truncated by dovecot
mail_cache_max_size = 500K
namespace inbox {
inbox = yes
@@ -183,34 +177,20 @@ service auth-worker {
}
service imap-login {
# High-performance mode as described in
# <https://doc.dovecot.org/2.3/admin_manual/login_processes/#high-performance-mode>
#
# So-called high-security mode described in
# <https://doc.dovecot.org/2.3/admin_manual/login_processes/#high-security-mode>
# and enabled by default with `service_count = 1` starts one process per connection
# and has problems logging in thousands of users after Dovecot restart.
service_count = 0
# High-security mode.
# Each process serves a single connection and exits afterwards.
# This is the default, but we set it explicitly to be sure.
# See <https://doc.dovecot.org/admin_manual/login_processes/#high-security-mode> for details.
service_count = 1
# Increase virtual memory size limit.
# Since imap-login processes handle TLS connections
# even after logging users in
# and many connections are handled by each process,
# memory size limit should be increased.
# Inrease the number of simultaneous connections.
#
# Otherwise the whole process eventually dies
# with an error similar to
# imap-login: Fatal: master: service(imap-login):
# child 1422951 returned error 83
# (Out of memory (service imap-login { vsz_limit=256 MB },
# you may need to increase it)
# and takes down all its TLS connections at once.
vsz_limit = 1G
# As of Dovecot 2.3.19.1 the default is 100 processes.
# Combined with `service_count = 1` it means only 100 connections
# can be handled simultaneously.
process_limit = 10000
# Avoid startup latency for new connections.
#
# Should be set to at least the number of CPU cores
# according to the documentation.
process_min_avail = 10
}
@@ -229,7 +209,7 @@ ssl = required
ssl_cert = </var/lib/acme/live/{{ config.mail_domain }}/fullchain
ssl_key = </var/lib/acme/live/{{ config.mail_domain }}/privkey
ssl_dh = </usr/share/dovecot/dh.pem
ssl_min_protocol = TLSv1.3
ssl_min_protocol = TLSv1.2
ssl_prefer_server_ciphers = yes

View File

@@ -0,0 +1,12 @@
# delete all mails after {{ config.delete_mails_after }} days, in the Inbox
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
# or in any IMAP subfolder
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
# even if they are unseen
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
# or only temporary (but then they shouldn't be around after {{ config.delete_mails_after }} days anyway).
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
3 0 * * * vmail find {{ config.mailboxes_dir }} -name 'maildirsize' -type f -delete
4 0 * * * vmail /usr/local/lib/chatmaild/venv/bin/delete_inactive_users /usr/local/lib/chatmaild/chatmail.ini

View File

@@ -2,6 +2,15 @@ function dovecot_lua_notify_begin_txn(user)
return user
end
function contains(v, needle)
for _, keyword in ipairs(v) do
if keyword == needle then
return true
end
end
return false
end
function dovecot_lua_notify_event_message_new(user, event)
local mbox = user:mailbox(event.mailbox)
mbox:sync()

View File

@@ -1,11 +1,5 @@
enable_relay = true
http_bind_addr = "[::]:3340"
# Disable built-in STUN server in iroh-relay 0.35
# as we deploy our own TURN server instead.
# STUN server is going to be removed in iroh-relay 1.0
# and this line can be removed after upgrade.
enable_stun = false
enable_stun = true
enable_metrics = false
metrics_bind_addr = "127.0.0.1:9092"

View File

@@ -3,7 +3,7 @@ Description=mtail
[Service]
Type=simple
ExecStart=/bin/sh -c "journalctl -f -o short-iso -n 0 | /usr/local/bin/mtail --address={{ address }} --port={{ port }} --progs /etc/mtail --logtostderr --logs -"
ExecStart=/bin/sh -c "journalctl -f -o short-iso -n 0 | /usr/bin/mtail --address={{ address }} --port={{ port }} --progs /etc/mtail --logtostderr --logs -"
Restart=on-failure
[Install]

View File

@@ -2,25 +2,11 @@ load_module modules/ngx_stream_module.so;
user www-data;
worker_processes auto;
# Increase the number of connections
# that a worker process can open
# to avoid errors such as
# accept4() failed (24: Too many open files)
# and
# socket() failed (24: Too many open files) while connecting to upstream
# in the logs.
# <https://nginx.org/en/docs/ngx_core_module.html#worker_rlimit_nofile>
worker_rlimit_nofile 2048;
pid /run/nginx.pid;
error_log syslog:server=unix:/dev/log,facility=local3;
events {
# Increase to avoid errors such as
# 768 worker_connections are not enough while connecting to upstream
# in the logs.
# <https://nginx.org/en/docs/ngx_core_module.html#worker_connections>
worker_connections 2048;
worker_connections 768;
# multi_accept on;
}
@@ -60,13 +46,16 @@ http {
server {
listen 127.0.0.1:8443 ssl default_server;
listen 8443 ssl default_server;
{% if not disable_ipv6 %}
listen [::]:8443 ssl default_server;
{% endif %}
root /var/www/html;
index index.html index.htm;
server_name {{ config.domain_name }} www.{{ config.domain_name }} mta-sts.{{ config.domain_name }};
server_name _;
access_log syslog:server=unix:/dev/log,facility=local7;
@@ -131,7 +120,10 @@ http {
# Redirect www. to non-www
server {
listen 127.0.0.1:8443 ssl;
listen 8443 ssl;
{% if not disable_ipv6 %}
listen [::]:8443 ssl;
{% endif %}
server_name www.{{ config.domain_name }};
return 301 $scheme://{{ config.domain_name }}$request_uri;
access_log syslog:server=unix:/dev/log,facility=local7;

View File

@@ -13,7 +13,6 @@ OversignHeaders From
On-BadSignature reject
On-KeyNotFound reject
On-NoSignature reject
DNSTimeout 60
# Signing domain, selector, and key (required). For example, perform signing
# for domain "example.com" with selector "2020" (2020._domainkey.example.com),

View File

@@ -1,3 +0,0 @@
[Service]
Restart=always
RuntimeMaxSec=1d

View File

@@ -20,12 +20,9 @@ smtpd_tls_key_file=/var/lib/acme/live/{{ config.mail_domain }}/privkey
smtpd_tls_security_level=may
smtp_tls_CApath=/etc/ssl/certs
smtp_tls_security_level=verify
# Send SNI extension when connecting to other servers.
# <https://www.postfix.org/postconf.5.html#smtp_tls_servername>
smtp_tls_servername = hostname
smtp_tls_security_level=may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_protocols = >=TLSv1.2
smtp_tls_policy_maps = socketmap:inet:127.0.0.1:8461:postfix
smtpd_tls_protocols = >=TLSv1.2
# Disable anonymous cipher suites

View File

@@ -14,12 +14,10 @@ smtp inet n - y - - smtpd -v
{%- else %}
smtp inet n - y - - smtpd
{%- endif %}
-o smtpd_tls_security_level=encrypt
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
-o smtpd_milters=unix:opendkim/opendkim.sock
submission inet n - y - 5000 smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_tls_mandatory_protocols=>=TLSv1.3
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_sasl_path=private/auth
@@ -33,11 +31,11 @@ submission inet n - y - 5000 smtpd
-o milter_macro_daemon_name=ORIGINATING
-o smtpd_client_connection_count_limit=1000
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
-o cleanup_service_name=authclean
smtps inet n - y - 5000 smtpd
-o syslog_name=postfix/smtps
-o smtpd_tls_wrappermode=yes
-o smtpd_tls_security_level=encrypt
-o smtpd_tls_mandatory_protocols=>=TLSv1.3
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_sasl_path=private/auth
@@ -50,6 +48,7 @@ smtps inet n - y - 5000 smtpd
-o smtpd_client_connection_count_limit=1000
-o milter_macro_daemon_name=ORIGINATING
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port }}
-o cleanup_service_name=authclean
#628 inet n - y - - qmqpd
pickup unix n - y 60 1 pickup
cleanup unix n - y - 0 cleanup
@@ -77,27 +76,17 @@ anvil unix - - y - 1 anvil
scache unix - - y - 1 scache
postlog unix-dgram n - n - 1 postlogd
filter unix - n n - - lmtp
# Local SMTP server for reinjecting outgoing filtered mail.
127.0.0.1:{{ config.postfix_reinject_port }} inet n - n - 100 smtpd
# Local SMTP server for reinjecting filered mail.
localhost:{{ config.postfix_reinject_port }} inet n - n - 10 smtpd
-o syslog_name=postfix/reinject
-o smtpd_milters=unix:opendkim/opendkim.sock
-o cleanup_service_name=authclean
# Local SMTP server for reinjecting incoming filtered mail
127.0.0.1:{{ config.postfix_reinject_port_incoming }} inet n - n - 100 smtpd
-o syslog_name=postfix/reinject_incoming
-o smtpd_milters=unix:opendkim/opendkim.sock
# Cleanup `Received` headers for authenticated mail
# to avoid leaking client IP.
#
# We do not do this for received mails
# as this will break DKIM signatures
# if `Received` header is signed.
#
# This service also rewrites
# Subject with `[...]`
# to make sure the users
# cannot send unprotected Subject.
authclean unix n - - - 0 cleanup
-o header_checks=regexp:/etc/postfix/submission_header_cleanup

View File

@@ -2,4 +2,3 @@
/^X-Originating-IP:/ IGNORE
/^X-Mailer:/ IGNORE
/^User-Agent:/ IGNORE
/^Subject:/ REPLACE Subject: [...]

View File

@@ -12,23 +12,23 @@ All functions of this module
import re
from .rshell import CalledProcessError, shell, log_progress
from .rshell import CalledProcessError, shell
def perform_initial_checks(mail_domain, pre_command=""):
def perform_initial_checks(mail_domain):
"""Collecting initial DNS settings."""
assert mail_domain
if not shell("dig", fail_ok=True, print=log_progress):
shell("apt-get update && apt-get install -y dnsutils", print=log_progress)
if not shell("dig", fail_ok=True):
shell("apt-get install -y dnsutils")
A = query_dns("A", mail_domain)
AAAA = query_dns("AAAA", mail_domain)
MTA_STS = query_dns("CNAME", f"mta-sts.{mail_domain}")
WWW = query_dns("CNAME", f"www.{mail_domain}")
res = dict(mail_domain=mail_domain, A=A, AAAA=AAAA, MTA_STS=MTA_STS, WWW=WWW)
res["acme_account_url"] = shell(pre_command + "acmetool account-url", fail_ok=True, print=log_progress)
res["acme_account_url"] = shell("acmetool account-url", fail_ok=True)
res["dkim_entry"], res["web_dkim_entry"] = get_dkim_entry(
mail_domain, pre_command, dkim_selector="opendkim"
mail_domain, dkim_selector="opendkim"
)
if not MTA_STS or not WWW or (not A and not AAAA):
@@ -40,12 +40,11 @@ def perform_initial_checks(mail_domain, pre_command=""):
return res
def get_dkim_entry(mail_domain, pre_command, dkim_selector):
def get_dkim_entry(mail_domain, dkim_selector):
try:
dkim_pubkey = shell(
f"{pre_command}openssl rsa -in /etc/dkimkeys/{dkim_selector}.private "
"-pubout 2>/dev/null | awk '/-/{next}{printf(\"%s\",$0)}'",
print=log_progress
f"openssl rsa -in /etc/dkimkeys/{dkim_selector}.private "
"-pubout 2>/dev/null | awk '/-/{next}{printf(\"%s\",$0)}'"
)
except CalledProcessError:
return
@@ -62,7 +61,7 @@ def query_dns(typ, domain):
# Get autoritative nameserver from the SOA record.
soa_answers = [
x.split()
for x in shell(f"dig -r -q {domain} -t SOA +noall +authority +answer", print=log_progress).split(
for x in shell(f"dig -r -q {domain} -t SOA +noall +authority +answer").split(
"\n"
)
]
@@ -72,13 +71,13 @@ def query_dns(typ, domain):
ns = soa[0][4]
# Query authoritative nameserver directly to bypass DNS cache.
res = shell(f"dig @{ns} -r -q {domain} -t {typ} +short", print=log_progress)
res = shell(f"dig @{ns} -r -q {domain} -t {typ} +short")
if res:
return res.split("\n")[0]
return ""
def check_zonefile(zonefile, verbose=True):
def check_zonefile(zonefile, mail_domain):
"""Check expected zone file entries."""
required = True
required_diff = []
@@ -90,7 +89,7 @@ def check_zonefile(zonefile, verbose=True):
continue
if not zf_line.strip() or zf_line.startswith(";"):
continue
print(f"dns-checking {zf_line!r}") if verbose else log_progress("")
print(f"dns-checking {zf_line!r}")
zf_domain, zf_typ, zf_value = zf_line.split(maxsplit=2)
zf_domain = zf_domain.rstrip(".")
zf_value = zf_value.strip()

View File

@@ -1,20 +1,10 @@
import sys
from subprocess import DEVNULL, CalledProcessError, check_output
from subprocess import CalledProcessError, check_output
def log_progress(data):
sys.stderr.write(".")
sys.stderr.flush()
def shell(command, fail_ok=False, print=print):
def shell(command, fail_ok=False):
print(f"$ {command}")
args = dict(shell=True)
if fail_ok:
args["stderr"] = DEVNULL
try:
return check_output(command, **args).decode().rstrip()
return check_output(command, shell=True).decode().rstrip()
except CalledProcessError:
if not fail_ok:
raise
@@ -24,22 +14,3 @@ def shell(command, fail_ok=False, print=print):
def get_systemd_running():
lines = shell("systemctl --type=service --state=running").split("\n")
return [line for line in lines if line.startswith(" ")]
def write_numbytes(path, num):
with open(path, "w") as f:
f.write("x" * num)
def dovecot_recalc_quota(user):
shell(f"doveadm quota recalc -u {user}")
output = shell(f"doveadm quota get -u {user}")
#
# Quota name Type Value Limit %
# User quota STORAGE 5 102400 0
# User quota MESSAGE 2 - 0
#
for line in output.split("\n"):
parts = line.split()
if parts[2] == "STORAGE":
return dict(value=int(parts[3]), limit=int(parts[4]), percent=int(parts[5]))

View File

@@ -1,9 +0,0 @@
[Unit]
Description=chatmail mail storage expiration job
After=network.target
[Service]
Type=oneshot
User=vmail
ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-expire /usr/local/lib/chatmaild/chatmail.ini -v --remove

View File

@@ -1,8 +0,0 @@
[Unit]
Description=Run Daily chatmail-expire job
[Timer]
OnCalendar=*-*-* 00:02:00
[Install]
WantedBy=timers.target

View File

@@ -1,9 +0,0 @@
[Unit]
Description=chatmail file system storage reporting job
After=network.target
[Service]
Type=oneshot
User=vmail
ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-fsreport /usr/local/lib/chatmaild/chatmail.ini

View File

@@ -1,9 +0,0 @@
[Unit]
Description=Run Daily Chatmail fsreport Job
[Timer]
OnCalendar=*-*-* 08:02:00
Persistent=true
[Install]
WantedBy=timers.target

View File

@@ -7,7 +7,6 @@ Restart=always
RestartSec=30
User=vmail
RuntimeDirectory=chatmail-metadata
UMask=0077
[Install]
WantedBy=multi-user.target

View File

@@ -7,7 +7,6 @@ Restart=always
RestartSec=30
User=vmail
RuntimeDirectory=doveauth
UMask=0077
[Install]
WantedBy=multi-user.target

View File

@@ -1,12 +0,0 @@
[Unit]
Description=Incoming Chatmail Postfix before queue filter
[Service]
ExecStart={execpath} {config_path} incoming
Restart=always
RestartSec=30
User=vmail
[Install]
WantedBy=multi-user.target

View File

@@ -1,11 +1,11 @@
[Unit]
Description=Outgoing Chatmail Postfix before queue filter
Description=Chatmail Postfix before queue filter
[Service]
ExecStart={execpath} {config_path} outgoing
ExecStart={execpath} {config_path}
Restart=always
RestartSec=30
User=vmail
User=filtermail
[Install]
WantedBy=multi-user.target

View File

@@ -1,16 +0,0 @@
[Unit]
Description=A wrapper for the TURN server
After=network.target
[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/chatmail-turn --realm {mail_domain} --socket /run/chatmail-turn/turn.socket
# Create /run/chatmail-turn
RuntimeDirectory=chatmail-turn
User=vmail
Group=vmail
[Install]
WantedBy=multi-user.target

View File

@@ -42,7 +42,6 @@ def bootstrap_remote(gateway, remote=remote):
def print_stderr(item="", end="\n"):
print(item, file=sys.stderr, end=end)
sys.stderr.flush()
class SSHExec:
@@ -71,6 +70,10 @@ class SSHExec:
raise self.FuncError(data)
def logged(self, call, kwargs):
def log_progress(data):
sys.stderr.write(".")
sys.stderr.flush()
title = call.__doc__
if not title:
title = call.__name__
@@ -79,22 +82,6 @@ class SSHExec:
return self(call, kwargs, log_callback=print_stderr)
else:
print_stderr(title, end="")
res = self(call, kwargs, log_callback=remote.rshell.log_progress)
res = self(call, kwargs, log_callback=log_progress)
print_stderr()
return res
class LocalExec:
def __init__(self, verbose=False, docker=False):
self.verbose = verbose
self.docker = docker
def logged(self, call, kwargs: dict):
where = "locally"
if self.docker:
if call == remote.rdns.perform_initial_checks:
kwargs['pre_command'] = "docker exec chatmail "
where = "in docker"
if self.verbose:
print(f"Running {where}: {call.__name__}(**{kwargs})")
return call(**kwargs)

View File

@@ -37,7 +37,7 @@ class TestDC:
def test_ping_pong(self, benchmark, cmfactory):
ac1, ac2 = cmfactory.get_online_accounts(2)
chat = cmfactory.get_protected_chat(ac1, ac2)
chat = cmfactory.get_accepted_chat(ac1, ac2)
def dc_ping_pong():
chat.send_text("ping")
@@ -49,7 +49,7 @@ class TestDC:
def test_send_10_receive_10(self, benchmark, cmfactory, lp):
ac1, ac2 = cmfactory.get_online_accounts(2)
chat = cmfactory.get_protected_chat(ac1, ac2)
chat = cmfactory.get_accepted_chat(ac1, ac2)
def dc_send_10_receive_10():
for i in range(10):

View File

@@ -1,5 +1,5 @@
import queue
import smtplib
import socket
import threading
import pytest
@@ -90,24 +90,21 @@ def test_concurrent_logins_same_account(
def test_no_vrfy(chatmail_config):
domain = chatmail_config.mail_domain
s = smtplib.SMTP(domain)
s.starttls()
s.putcmd("vrfy", f"wrongaddress@{chatmail_config.mail_domain}")
result = s.getreply()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((chatmail_config.mail_domain, 25))
banner = sock.recv(1024)
print(banner)
sock.send(b"VRFY wrongaddress@%s\r\n" % (chatmail_config.mail_domain.encode(),))
result = sock.recv(1024)
print(result)
s.putcmd("vrfy", f"echo@{chatmail_config.mail_domain}")
result2 = s.getreply()
sock.send(b"VRFY echo@%s\r\n" % (chatmail_config.mail_domain.encode(),))
result2 = sock.recv(1024)
print(result2)
assert result[0] == result2[0] == 252
assert result[1][0:6] == result2[1][0:6] == b"2.0.0 "
s.putcmd("vrfy", "wrongaddress")
result = s.getreply()
assert result[0:10] == result2[0:10]
sock.send(b"VRFY wrongaddress\r\n")
result = sock.recv(1024)
print(result)
s.putcmd("vrfy", "echo")
result2 = s.getreply()
sock.send(b"VRFY echo\r\n")
result2 = sock.recv(1024)
print(result2)
assert result[0] == result2[0] == 252
assert result[1][0:6] == result2[1][0:6] == b"2.0.0 "
assert result[0:10] == result2[0:10] == b"252 2.0.0 "

View File

@@ -1,8 +1,4 @@
import datetime
import smtplib
import socket
import subprocess
import time
import pytest
@@ -32,8 +28,7 @@ class TestSSHExecutor:
)
out, err = capsys.readouterr()
assert err.startswith("Collecting")
# XXX could not figure out how capturing can be made to work properly
#assert err.endswith("....\n")
assert err.endswith("....\n")
assert err.count("\n") == 1
sshexec.verbose = True
@@ -42,8 +37,7 @@ class TestSSHExecutor:
)
out, err = capsys.readouterr()
lines = err.split("\n")
# XXX could not figure out how capturing can be made to work properly
#assert len(lines) > 4
assert len(lines) > 4
assert remote.rdns.perform_initial_checks.__doc__ in lines[0]
def test_exception(self, sshexec, capsys):
@@ -58,23 +52,6 @@ class TestSSHExecutor:
else:
pytest.fail("didn't raise exception")
def test_opendkim_restarted(self, sshexec):
"""check that opendkim is not running for longer than a day."""
cmd = "systemctl show opendkim --timestamp=utc --property=ActiveEnterTimestamp"
out = sshexec(call=remote.rshell.shell, kwargs=dict(command=cmd))
datestring = out.split("=")[1]
since_date = datetime.datetime.strptime(datestring, "%a %Y-%m-%d %H:%M:%S %Z")
now = datetime.datetime.now(since_date.tzinfo)
assert (now - since_date).total_seconds() < 60 * 60 * 51
def test_timezone_env(remote):
for line in remote.iter_output("env"):
print(line)
if line == "tz=:/etc/localtime":
return
pytest.fail("TZ is not set")
def test_remote(remote, imap_or_smtp):
lineproducer = remote.iter_output(imap_or_smtp.logcmd)
@@ -130,58 +107,14 @@ def test_authenticated_from(cmsetup, maildata):
@pytest.mark.parametrize("from_addr", ["fake@example.org", "fake@testrun.org"])
def test_reject_missing_dkim(cmsetup, maildata, from_addr):
domain = cmsetup.maildomain
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(10)
try:
sock.connect((domain, 25))
except socket.timeout:
pytest.skip(f"port 25 not reachable for {domain}")
"""Test that emails with missing or wrong DMARC, DKIM, and SPF entries are rejected."""
recipient = cmsetup.gen_users(1)[0]
msg = maildata(
"encrypted.eml", from_addr=from_addr, to_addr=recipient.addr
).as_string()
conn = smtplib.SMTP(cmsetup.maildomain, 25, timeout=10)
conn.starttls()
with conn as s:
msg = maildata("plain.eml", from_addr=from_addr, to_addr=recipient.addr).as_string()
with smtplib.SMTP(cmsetup.maildomain, 25) as s:
with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):
s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)
def try_n_times(n, f):
for _ in range(n - 1):
try:
return f()
except Exception:
time.sleep(1)
return f()
def test_rewrite_subject(cmsetup, maildata):
"""Test that subject gets replaced with [...]."""
user1, user2 = cmsetup.gen_users(2)
sent_msg = maildata(
"encrypted.eml",
from_addr=user1.addr,
to_addr=user2.addr,
subject="Unencrypted subject",
).as_string()
user1.smtp.sendmail(from_addr=user1.addr, to_addrs=[user2.addr], msg=sent_msg)
# The message may need some time to get delivered by postfix.
messages = try_n_times(5, user2.imap.fetch_all_messages)
assert len(messages) == 1
rcvd_msg = messages[0]
assert "Subject: [...]" not in sent_msg
assert "Subject: [...]" in rcvd_msg
assert "Subject: Unencrypted subject" in sent_msg
assert "Subject: Unencrypted subject" not in rcvd_msg
@pytest.mark.slow
def test_exceed_rate_limit(cmsetup, gencreds, maildata, chatmail_config):
"""Test that the per-account send-mail limit is exceeded."""
@@ -214,31 +147,6 @@ def test_expunged(remote, chatmail_config):
f"find {chatmail_config.mailboxes_dir} -path '*/tmp/*' -mtime +{outdated_days} -type f",
f"find {chatmail_config.mailboxes_dir} -path '*/.*/tmp/*' -mtime +{outdated_days} -type f",
]
outdated_days = int(chatmail_config.delete_large_after) + 1
find_cmds.append(
"find {chatmail_config.mailboxes_dir} -path '*/cur/*' -mtime +{outdated_days} -size +200k -type f"
)
for cmd in find_cmds:
for line in remote.iter_output(cmd):
assert not line
def test_deployed_state(remote):
try:
git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
except Exception:
git_hash = "unknown\n"
try:
git_diff = subprocess.check_output(["git", "diff"]).decode()
except Exception:
git_diff = ""
git_status = [git_hash.strip()]
for line in git_diff.splitlines():
git_status.append(line.strip().lower())
remote_version = []
for line in remote.iter_output("cat /etc/chatmail-version"):
print(line)
remote_version.append(line)
# assert len(git_status) == len(remote_version) # for some reason, we only get 11 lines from remote.iter_output()
for i in range(len(remote_version)):
assert git_status[i] == remote_version[i], "You have undeployed changes."

View File

@@ -1,4 +1,5 @@
import ipaddress
import random
import re
import time
@@ -6,9 +7,6 @@ import imap_tools
import pytest
import requests
from cmdeploy.remote import rshell
from cmdeploy.sshexec import SSHExec
@pytest.fixture
def imap_mailbox(cmfactory):
@@ -56,23 +54,22 @@ class TestEndToEndDeltaChat:
"""Test that a DC account can send a message to a second DC account
on the same chat-mail instance."""
ac1, ac2 = cmfactory.get_online_accounts(2)
chat = cmfactory.get_protected_chat(ac1, ac2)
chat = cmfactory.get_accepted_chat(ac1, ac2)
lp.sec("ac1: prepare and send text message to ac2")
chat.send_text("message0")
lp.sec("wait for ac2 to receive message")
msg2 = ac2._evtracker.wait_next_incoming_message()
assert msg2.text == "message0"
def test_exceed_quota(
self, cmfactory, lp, tmpdir, remote, chatmail_config, sshdomain
):
@pytest.mark.slow
def test_exceed_quota(self, cmfactory, lp, tmpdir, remote, chatmail_config):
"""This is a very slow test as it needs to upload >100MB of mail data
before quota is exceeded, and thus depends on the speed of the upload.
"""
ac1, ac2 = cmfactory.get_online_accounts(2)
chat = cmfactory.get_protected_chat(ac1, ac2)
user = ac2.get_config("configured_addr")
chat = cmfactory.get_accepted_chat(ac1, ac2)
def parse_size_limit(limit: str) -> int:
"""Parse a size limit and return the number of bytes as integer.
@@ -85,27 +82,49 @@ class TestEndToEndDeltaChat:
return int(float(number) * units[unit])
quota = parse_size_limit(chatmail_config.max_mailbox_size)
attachsize = 1 * 1024 * 1024
num_to_send = quota // attachsize + 2
lp.sec(f"ac1: send {num_to_send} large files to ac2")
lp.indent(f"per-user quota is assumed to be: {quota / (1024 * 1024)}MB")
alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
msgs = []
for i in range(num_to_send):
attachment = tmpdir / f"attachment{i}"
data = "".join(random.choice(alphanumeric) for i in range(1024))
with open(attachment, "w+") as f:
for j in range(attachsize // len(data)):
f.write(data)
lp.sec(f"filling remote inbox for {user}")
fn = f"7743102289.M843172P2484002.c20,S={quota},W=2398:2,"
path = chatmail_config.mailboxes_dir.joinpath(user, "cur", fn)
sshexec = SSHExec(sshdomain)
sshexec(call=rshell.write_numbytes, kwargs=dict(path=str(path), num=120))
res = sshexec(call=rshell.dovecot_recalc_quota, kwargs=dict(user=user))
assert res["percent"] >= 100
msg = chat.send_file(str(attachment))
msgs.append(msg)
lp.indent(f"Sent out msg {i}, size {attachsize / (1024 * 1024)}MB")
lp.sec("ac2: check quota is triggered")
lp.sec("ac2: check messages are arriving until quota is reached")
starting = True
addr = ac2.get_config("addr").lower()
saved_ok = 0
for line in remote.iter_output("journalctl -n0 -f -u dovecot"):
if starting:
chat.send_text("hello")
starting = False
if user not in line:
if addr not in line:
# print(line)
continue
if "quota exceeded" in line:
return
if "quota" in line:
if "quota exceeded" in line:
if saved_ok < num_to_send // 2:
pytest.fail(
f"quota exceeded too early: after {saved_ok} messages already"
)
lp.indent("good, message sending failed because quota was exceeded")
return
if (
"stored mail into mailbox 'inbox'" in line
or "saved mail to inbox" in line
):
saved_ok += 1
print(f"{saved_ok}: {line}")
if saved_ok >= num_to_send:
break
pytest.fail("sending succeeded although messages should exceed quota")
def test_securejoin(self, cmfactory, lp, maildomain2):
ac1 = cmfactory.new_online_configuring_account(cache=False)
@@ -153,7 +172,7 @@ def test_hide_senders_ip_address(cmfactory):
assert ipaddress.ip_address(public_ip)
user1, user2 = cmfactory.get_online_accounts(2)
chat = cmfactory.get_protected_chat(user1, user2)
chat = cmfactory.get_accepted_chat(user1, user2)
chat.send_text("testing submission header cleanup")
user2._evtracker.wait_next_incoming_message()
@@ -162,18 +181,11 @@ def test_hide_senders_ip_address(cmfactory):
assert public_ip not in msg.obj.as_string()
def test_echobot(cmfactory, chatmail_config, lp, sshdomain):
def test_echobot(cmfactory, chatmail_config, lp):
ac = cmfactory.get_online_accounts(1)[0]
# establish contact with echobot
sshexec = SSHExec(sshdomain)
command = "cat /var/lib/echobot/invite-link.txt"
echo_invite_link = sshexec(call=rshell.shell, kwargs=dict(command=command))
chat = ac.qr_setup_contact(echo_invite_link)
ac._evtracker.wait_securejoin_joiner_progress(1000)
# send message and check it gets replied back
lp.sec("Send message to echobot")
lp.sec(f"Send message to echo@{chatmail_config.mail_domain}")
chat = ac.create_chat(f"echo@{chatmail_config.mail_domain}")
text = "hi, I hope you text me back"
chat.send_text(text)
lp.sec("Wait for reply from echobot")

View File

@@ -62,7 +62,7 @@ def sshdomain(maildomain):
def maildomain2():
domain = os.environ.get("CHATMAIL_DOMAIN2")
if not domain:
pytest.skip("set CHATMAIL_DOMAIN2 to a second chatmail server")
pytest.skip("set CHATMAIL_DOMAIN2 to a ssh-reachable chatmail instance")
return domain
@@ -302,13 +302,10 @@ def cmfactory(request, gencreds, tmpdir, maildomain):
pytest.importorskip("deltachat")
from deltachat.testplugin import ACFactory
data = request.getfixturevalue("data")
testproc = ChatmailTestProcess(request.config, maildomain, gencreds)
class Data:
def read_path(self, path):
return
am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=Data())
am = ACFactory(request=request, tmpdir=tmpdir, testprocess=testproc, data=data)
# nb. a bit hacky
# would probably be better if deltachat's test machinery grows native support

View File

@@ -1,10 +1,8 @@
import importlib
import os
import pytest
from cmdeploy.cmdeploy import get_parser, main
from cmdeploy.www import get_paths
@pytest.fixture(autouse=True)
@@ -26,36 +24,6 @@ class TestCmdline:
def test_init_not_overwrite(self, capsys):
assert main(["init", "chat.example.org"]) == 0
capsys.readouterr()
assert main(["init", "chat.example.org"]) == 1
out, err = capsys.readouterr()
assert "path exists" in out.lower()
assert main(["init", "chat.example.org", "--force"]) == 0
out, err = capsys.readouterr()
assert "deleting config file" in out.lower()
def test_www_folder(example_config, tmp_path):
reporoot = importlib.resources.files(__package__).joinpath("../../../../").resolve()
assert not example_config.www_folder
www_path, src_dir, build_dir = get_paths(example_config)
assert www_path.absolute() == reporoot.joinpath("www").absolute()
assert src_dir == reporoot.joinpath("www").joinpath("src")
assert build_dir == reporoot.joinpath("www").joinpath("build")
example_config.www_folder = "disabled"
www_path, _, _ = get_paths(example_config)
assert not www_path.is_dir()
example_config.www_folder = str(tmp_path)
www_path, src_dir, build_dir = get_paths(example_config)
assert www_path == tmp_path
assert not src_dir.exists()
assert not build_dir
src_path = tmp_path.joinpath("src")
os.mkdir(src_path)
with open(src_path / "index.md", "w") as f:
f.write("# Test")
www_path, src_dir, build_dir = get_paths(example_config)
assert www_path == tmp_path
assert src_dir == src_path
assert build_dir == tmp_path.joinpath("build")

View File

@@ -89,14 +89,18 @@ class TestZonefileChecks:
def test_check_zonefile_all_ok(self, cm_data, mockdns_base):
zonefile = cm_data.get("zftest.zone")
parse_zonefile_into_dict(zonefile, mockdns_base)
required_diff, recommended_diff = remote.rdns.check_zonefile(zonefile)
required_diff, recommended_diff = remote.rdns.check_zonefile(
zonefile, "some.domain"
)
assert not required_diff and not recommended_diff
def test_check_zonefile_recommended_not_set(self, cm_data, mockdns_base):
zonefile = cm_data.get("zftest.zone")
zonefile_mocked = zonefile.split("; Recommended")[0]
parse_zonefile_into_dict(zonefile_mocked, mockdns_base)
required_diff, recommended_diff = remote.rdns.check_zonefile(zonefile)
required_diff, recommended_diff = remote.rdns.check_zonefile(
zonefile, "some.domain"
)
assert not required_diff
assert len(recommended_diff) == 8

View File

@@ -3,7 +3,6 @@ import importlib.resources
import time
import traceback
import webbrowser
from pathlib import Path
import markdown
from chatmaild.config import read_config
@@ -31,25 +30,9 @@ def prepare_template(source):
return render_vars, page_layout
def get_paths(config) -> (Path, Path, Path):
reporoot = importlib.resources.files(__package__).joinpath("../../../").resolve()
www_path = Path(config.www_folder)
# if www_folder was not set, use default directory
if config.www_folder == "":
www_path = reporoot.joinpath("www")
src_dir = www_path.joinpath("src")
# if www_folder is a hugo page, build it
if src_dir.joinpath("index.md").is_file():
build_dir = www_path.joinpath("build")
# if it is not a hugo page, upload it as is
else:
build_dir = None
return www_path, src_dir, build_dir
def build_webpages(src_dir, build_dir, config) -> Path:
def build_webpages(src_dir, build_dir, config):
try:
return _build_webpages(src_dir, build_dir, config)
_build_webpages(src_dir, build_dir, config)
except Exception:
print(traceback.format_exc())
@@ -123,11 +106,15 @@ def main():
config = read_config(inipath)
config.webdev = True
assert config.mail_domain
www_path = reporoot.joinpath("www")
src_path = www_path.joinpath("src")
stats = None
build_dir = www_path.joinpath("build")
src_dir = www_path.joinpath("src")
index_path = build_dir.joinpath("index.html")
# start web page generation, open a browser and wait for changes
www_path, src_path, build_dir = get_paths(config)
build_dir = build_webpages(src_path, build_dir, config)
index_path = build_dir.joinpath("index.html")
build_webpages(src_dir, build_dir, config)
webbrowser.open(str(index_path))
stats = snapshot_dir_stats(src_path)
print(f"\nOpened URL: file://{index_path.resolve()}\n")
@@ -148,7 +135,7 @@ def main():
changenum += 1
stats = newstats
build_webpages(src_path, build_dir, config)
build_webpages(src_dir, build_dir, config)
print(f"[{changenum}] regenerated web pages at: {index_path}")
print(f"URL: file://{index_path.resolve()}\n\n")
count = 0

View File

@@ -1,23 +1,5 @@
#!/bin/sh
set -e
if command -v lsb_release 2>&1 >/dev/null; then
case "$(lsb_release -is)" in
Ubuntu | Debian )
if ! dpkg -l | grep python3-dev 2>&1 >/dev/null
then
echo "You need to install python3-dev for installing the other dependencies."
exit 1
fi
if ! gcc --version 2>&1 >/dev/null
then
echo "You need to install gcc for building Python dependencies."
exit 1
fi
;;
esac
fi
python3 -m venv --upgrade-deps venv
venv/bin/pip install -e chatmaild

View File

@@ -6,6 +6,29 @@ interoperable e-mail service for everyone. What's behind a `chatmail` is
effectively a normal e-mail address just like any other but optimized
for the usage in chats, especially DeltaChat.
### Choosing a chatmail address instead of using a random one
In the Delta Chat account setup you may tap `Create a profile` then `Use other server` and choose `Classic e-mail login`. Here fill the two fields like this:
- `E-Mail Address`: invent a word with
{% if username_min_length == username_max_length %}
*exactly* {{ username_min_length }}
{% else %}
{{ username_min_length}}
{% if username_max_length == "more" %}
or more
{% else %}
to {{ username_max_length }}
{% endif %}
{% endif %}
characters
and append `@{{config.mail_domain}}` to it.
- `Existing Password`: invent at least {{ password_min_length }} characters.
If the e-mail address is not yet taken, you'll get that account.
The first login sets your password.
### Rate and storage limits
@@ -15,10 +38,9 @@ for the usage in chats, especially DeltaChat.
- You may send up to {{ config.max_user_send_per_minute }} messages per minute.
- You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
- Messages are unconditionally removed {{ config.delete_mails_after }} days after arriving on the server.
- Messages are unconditionally removed latest {{ config.delete_mails_after }} days after arriving on the server.
Earlier, if storage may exceed otherwise.
- You can store up to [{{ config.max_mailbox_size }} messages on the server](https://delta.chat/en/help#what-happens-if-i-turn-on-delete-old-messages-from-server).
### <a name="account-deletion"></a> Account deletion