mirror of
https://github.com/chatmail/relay.git
synced 2026-05-11 16:34:39 +00:00
Compare commits
18 Commits
ci-alxndr4
...
traefik-su
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5e4f9deb28 | ||
|
|
2c344d7fc5 | ||
|
|
346179d045 | ||
|
|
a331828301 | ||
|
|
514682a093 | ||
|
|
ab5b8941c7 | ||
|
|
84def2db65 | ||
|
|
e134552b4f | ||
|
|
c0e77adfed | ||
|
|
db5e39a899 | ||
|
|
5692681937 | ||
|
|
9919deefe3 | ||
|
|
d525b95957 | ||
|
|
9a43a25e2c | ||
|
|
955d89fa1c | ||
|
|
3e3a85523d | ||
|
|
7023612a8b | ||
|
|
fdabed5c67 |
4
.github/ISSUE_TEMPLATE/config.yml
vendored
4
.github/ISSUE_TEMPLATE/config.yml
vendored
@@ -1 +1,5 @@
|
||||
blank_issues_enabled: true
|
||||
contact_links:
|
||||
- name: Mutual Help Chat Group
|
||||
url: https://i.delta.chat/#6CBFF8FFD505C0FDEA20A66674F2916EA8FBEE99&a=invitebot%40nine.testrun.org&g=Chatmail%20Mutual%20Help&x=7sFF7Ik50pWv6J1z7RVC5527&i=X69wTFfvCfs3d-JzqP0kVA3i&s=ibp-447dU-wUq-52QanwAtWc
|
||||
about: If you have troubles setting up the relay server, feel free to ask here.
|
||||
|
||||
@@ -77,7 +77,7 @@ jobs:
|
||||
cmdeploy init staging-ipv4.testrun.org
|
||||
sed -i 's#disable_ipv6 = False#disable_ipv6 = True#' chatmail.ini
|
||||
|
||||
- run: cmdeploy run --verbose --skip-dns-check
|
||||
- run: cmdeploy run
|
||||
|
||||
- name: set DNS entries
|
||||
run: |
|
||||
|
||||
2
.github/workflows/test-and-deploy.yaml
vendored
2
.github/workflows/test-and-deploy.yaml
vendored
@@ -75,7 +75,7 @@ jobs:
|
||||
|
||||
- run: cmdeploy init staging2.testrun.org
|
||||
|
||||
- run: cmdeploy run --verbose --skip-dns-check
|
||||
- run: cmdeploy run --verbose
|
||||
|
||||
- name: set DNS entries
|
||||
run: |
|
||||
|
||||
7
.gitignore
vendored
7
.gitignore
vendored
@@ -164,3 +164,10 @@ cython_debug/
|
||||
#.idea/
|
||||
|
||||
chatmail.zone
|
||||
|
||||
# docker
|
||||
/data/
|
||||
/custom/
|
||||
docker-compose.yaml
|
||||
.env
|
||||
/traefik/data/
|
||||
|
||||
@@ -19,6 +19,7 @@ graph LR;
|
||||
/var/lib/acme`")] --> nginx-internal;
|
||||
cron --- chatmail-metrics;
|
||||
cron --- acmetool;
|
||||
cron --- expunge;
|
||||
chatmail-metrics --- website;
|
||||
acmetool --> certs[("`TLS certs
|
||||
/var/lib/acme`")];
|
||||
@@ -34,8 +35,7 @@ graph LR;
|
||||
dovecot --- users;
|
||||
dovecot --- |metadata.socket|chatmail-metadata;
|
||||
doveauth --- users;
|
||||
chatmail-expire-daily --- users;
|
||||
chatmail-fsreport-daily --- users;
|
||||
expunge --- users;
|
||||
chatmail-metadata --- iroh-relay;
|
||||
certs-nginx --> postfix;
|
||||
certs-nginx --> dovecot;
|
||||
|
||||
51
CHANGELOG.md
51
CHANGELOG.md
@@ -2,69 +2,32 @@
|
||||
|
||||
## untagged
|
||||
|
||||
- acmetool: use ECDSA keys instead of RSA
|
||||
([#689](https://github.com/chatmail/relay/pull/689))
|
||||
|
||||
- Require TLS 1.2 for outgoing SMTP connections
|
||||
([#685](https://github.com/chatmail/relay/pull/685))
|
||||
|
||||
- require STARTTLS for incoming port 25 connections
|
||||
([#684](https://github.com/chatmail/relay/pull/684))
|
||||
|
||||
- filtermail: run CPU-intensive handle_DATA in a thread pool executor
|
||||
([#676](https://github.com/chatmail/relay/pull/676))
|
||||
|
||||
- don't use the complicated logging module in filtermail to exclude a potential source of errors.
|
||||
([#674](https://github.com/chatmail/relay/pull/674))
|
||||
|
||||
- Specify nginx.conf to only handle `mail_domain`, www, and mta-sts domains
|
||||
([#636](https://github.com/chatmail/relay/pull/636))
|
||||
|
||||
- Setup TURN server
|
||||
([#621](https://github.com/chatmail/relay/pull/621))
|
||||
|
||||
- cmdeploy: make --ssh-host work with localhost
|
||||
([#659](https://github.com/chatmail/relay/pull/659))
|
||||
|
||||
- Update iroh-relay to 0.35.0
|
||||
([#650](https://github.com/chatmail/relay/pull/650))
|
||||
|
||||
- filtermail: accept mails from Protonmail
|
||||
([#616](https://github.com/chatmail/relay/pull/616))
|
||||
|
||||
- Ignore all RCPT TO: parameters
|
||||
([#651](https://github.com/chatmail/relay/pull/651))
|
||||
|
||||
- Increase opendkim DNS Timeout from 5 to 60 seconds
|
||||
([#672](https://github.com/chatmail/relay/pull/672))
|
||||
|
||||
- Add config parameter for Let's Encrypt ACME email
|
||||
([#663](https://github.com/chatmail/relay/pull/663))
|
||||
|
||||
- Use max username length in newemail.py, not min
|
||||
([#648](https://github.com/chatmail/relay/pull/648))
|
||||
|
||||
- Add startup for `fcgiwrap.service` because sometimes it did not start automatically.
|
||||
([#657](https://github.com/chatmail/relay/pull/657))
|
||||
|
||||
- Add `cmdeploy init --force` command for recreating chatmail.ini
|
||||
([#656](https://github.com/chatmail/relay/pull/656))
|
||||
|
||||
- Increase maxproc for reinjecting ports from 10 to 100
|
||||
([#646](https://github.com/chatmail/relay/pull/646))
|
||||
|
||||
- Allow ports 143 and 993 to be used by `dovecot` process
|
||||
([#639](https://github.com/chatmail/relay/pull/639))
|
||||
|
||||
- Add `--skip-dns-check` argument to `cmdeploy run` command, which disables DNS record checking before installation.
|
||||
([#661](https://github.com/chatmail/relay/pull/661))
|
||||
|
||||
- Rework expiry of message files and mailboxes in Python
|
||||
to only do a single iteration over sometimes millions of messages
|
||||
instead of doing "find" commands that iterate 9 times over the messages.
|
||||
Provide an "fsreport" CLI for more fine grained analysis of message files.
|
||||
([#637](https://github.com/chatmail/relay/pull/637))
|
||||
- Add installation via docker compose (MVP 1). The instructions, known issues and limitations are located in `/docs`
|
||||
([#614](https://github.com/chatmail/relay/pull/614))
|
||||
|
||||
- Add configuration parameters
|
||||
([#614](https://github.com/chatmail/relay/pull/614)):
|
||||
- `use_foreign_cert_manager` - Use a third-party certificate manager instead of acmetool (default: `False`)
|
||||
- `change_kernel_settings` - Whether to change kernel parameters during installation (default: `True`)
|
||||
- `fs_inotify_max_user_instances_and_watchers` - Value for kernel parameters `fs.inotify.max_user_instances` and `fs.inotify.max_user_watches` (default: `65535`)
|
||||
|
||||
## 1.7.0 2025-09-11
|
||||
|
||||
|
||||
24
README.md
24
README.md
@@ -74,22 +74,23 @@ Please substitute it with your own domain.
|
||||
```
|
||||
git clone https://github.com/chatmail/relay
|
||||
cd relay
|
||||
```
|
||||
|
||||
### Manual installation
|
||||
1. On your local PC, create chatmail configuration file `chatmail.ini`:
|
||||
|
||||
```
|
||||
scripts/initenv.sh
|
||||
```
|
||||
|
||||
3. On your local PC, create chatmail configuration file `chatmail.ini`:
|
||||
|
||||
```
|
||||
scripts/cmdeploy init chat.example.org # <-- use your domain
|
||||
```
|
||||
|
||||
4. Verify that SSH root login to your remote server works:
|
||||
2. Verify that SSH root login to your remote server works:
|
||||
|
||||
```
|
||||
ssh root@chat.example.org # <-- use your domain
|
||||
```
|
||||
|
||||
5. From your local PC, deploy the remote chatmail relay server:
|
||||
3. From your local PC, deploy the remote chatmail relay server:
|
||||
|
||||
```
|
||||
scripts/cmdeploy run
|
||||
@@ -99,6 +100,9 @@ Please substitute it with your own domain.
|
||||
which you should configure at your DNS provider
|
||||
(it can take some time until they are public).
|
||||
|
||||
### Docker installation
|
||||
Installation using docker compose is presented [here](./docs/DOCKER_INSTALLATION_EN.md)
|
||||
|
||||
### Other helpful commands
|
||||
|
||||
To check the status of your remotely running chatmail service:
|
||||
@@ -180,10 +184,6 @@ The components of chatmail are:
|
||||
- [Iroh relay](https://www.iroh.computer/docs/concepts/relay)
|
||||
which helps client devices to establish Peer-to-Peer connections
|
||||
|
||||
- [TURN](https://github.com/chatmail/chatmail-turn)
|
||||
to enable relay users to start webRTC calls
|
||||
even if a p2p connection can't be established
|
||||
|
||||
- and the chatmaild services, explained in the next section:
|
||||
|
||||
### chatmaild
|
||||
@@ -308,8 +308,6 @@ Chatmail address creation will be denied while this file is present.
|
||||
[Nginx](https://www.nginx.com/) listens on port 8443 (HTTPS-ALT) and 443 (HTTPS).
|
||||
Port 443 multiplexes HTTPS, IMAP and SMTP using ALPN to redirect connections to ports 8443, 465 or 993.
|
||||
[acmetool](https://hlandau.github.io/acmetool/) listens on port 80 (HTTP).
|
||||
[chatmail-turn](https://github.com/chatmail/chatmail-turn) listens on UDP port 3478 (STUN/TURN),
|
||||
and temporarily opens UDP ports when users request them. UDP port range is not restricted, any free port may be allocated.
|
||||
|
||||
chatmail-core based apps will, however, discover all ports and configurations
|
||||
automatically by reading the [autoconfig XML file](https://www.ietf.org/archive/id/draft-bucksch-autoconfig-00.html) from the chatmail relay server.
|
||||
|
||||
@@ -27,10 +27,8 @@ chatmail-metadata = "chatmaild.metadata:main"
|
||||
filtermail = "chatmaild.filtermail:main"
|
||||
echobot = "chatmaild.echo:main"
|
||||
chatmail-metrics = "chatmaild.metrics:main"
|
||||
chatmail-expire = "chatmaild.expire:main"
|
||||
chatmail-fsreport = "chatmaild.fsreport:main"
|
||||
delete_inactive_users = "chatmaild.delete_inactive_users:main"
|
||||
lastlogin = "chatmaild.lastlogin:main"
|
||||
turnserver = "chatmaild.turnserver:main"
|
||||
|
||||
[project.entry-points.pytest11]
|
||||
"chatmaild.testplugin" = "chatmaild.tests.plugin"
|
||||
@@ -72,6 +70,5 @@ commands =
|
||||
[testenv]
|
||||
deps = pytest
|
||||
pdbpp
|
||||
pytest-localserver
|
||||
commands = pytest -v -rsXx {posargs}
|
||||
"""
|
||||
|
||||
@@ -44,7 +44,15 @@ class Config:
|
||||
)
|
||||
self.mtail_address = params.get("mtail_address")
|
||||
self.disable_ipv6 = params.get("disable_ipv6", "false").lower() == "true"
|
||||
self.acme_email = params.get("acme_email", "")
|
||||
self.use_foreign_cert_manager = (
|
||||
params.get("use_foreign_cert_manager", "false").lower() == "true"
|
||||
)
|
||||
self.change_kernel_settings = (
|
||||
params.get("change_kernel_settings", "true").lower() == "true"
|
||||
)
|
||||
self.fs_inotify_max_user_instances_and_watchers = int(
|
||||
params["fs_inotify_max_user_instances_and_watchers"]
|
||||
)
|
||||
self.imap_rawlog = params.get("imap_rawlog", "false").lower() == "true"
|
||||
if "iroh_relay" not in params:
|
||||
self.iroh_relay = "https://" + params["mail_domain"]
|
||||
|
||||
31
chatmaild/src/chatmaild/delete_inactive_users.py
Normal file
31
chatmaild/src/chatmaild/delete_inactive_users.py
Normal file
@@ -0,0 +1,31 @@
|
||||
"""
|
||||
Remove inactive users
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
|
||||
from .config import read_config
|
||||
|
||||
|
||||
def delete_inactive_users(config):
|
||||
cutoff_date = time.time() - config.delete_inactive_users_after * 86400
|
||||
for addr in os.listdir(config.mailboxes_dir):
|
||||
try:
|
||||
user = config.get_user(addr)
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
read_timestamp = user.get_last_login_timestamp()
|
||||
if read_timestamp and read_timestamp < cutoff_date:
|
||||
path = config.mailboxes_dir.joinpath(addr)
|
||||
assert path == user.maildir
|
||||
shutil.rmtree(path, ignore_errors=True)
|
||||
|
||||
|
||||
def main():
|
||||
(cfgpath,) = sys.argv[1:]
|
||||
config = read_config(cfgpath)
|
||||
delete_inactive_users(config)
|
||||
@@ -1,218 +0,0 @@
|
||||
"""
|
||||
Expire old messages and addresses.
|
||||
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from argparse import ArgumentParser
|
||||
from collections import namedtuple
|
||||
from datetime import datetime
|
||||
from stat import S_ISREG
|
||||
|
||||
from chatmaild.config import read_config
|
||||
|
||||
FileEntry = namedtuple("FileEntry", ("relpath", "mtime", "size"))
|
||||
|
||||
|
||||
def iter_mailboxes(basedir, maxnum):
|
||||
if not os.path.exists(basedir):
|
||||
print_info(f"no mailboxes found at: {basedir}")
|
||||
return
|
||||
|
||||
for name in os_listdir_if_exists(basedir)[:maxnum]:
|
||||
if "@" in name:
|
||||
yield MailboxStat(basedir + "/" + name)
|
||||
|
||||
|
||||
def get_file_entry(path):
|
||||
"""return a FileEntry or None if the path does not exist or is not a regular file."""
|
||||
try:
|
||||
st = os.stat(path)
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
if not S_ISREG(st.st_mode):
|
||||
return None
|
||||
return FileEntry(path, st.st_mtime, st.st_size)
|
||||
|
||||
|
||||
def os_listdir_if_exists(path):
|
||||
"""return a list of names obtained from os.listdir or an empty list if the path does not exist."""
|
||||
try:
|
||||
return os.listdir(path)
|
||||
except FileNotFoundError:
|
||||
return []
|
||||
|
||||
|
||||
class MailboxStat:
|
||||
last_login = None
|
||||
|
||||
def __init__(self, basedir):
|
||||
self.basedir = str(basedir)
|
||||
# all detected messages in cur/new/tmp folders
|
||||
self.messages = []
|
||||
|
||||
# all detected files in mailbox top dir
|
||||
self.extrafiles = []
|
||||
|
||||
# scan all relevant files (without recursion)
|
||||
old_cwd = os.getcwd()
|
||||
try:
|
||||
os.chdir(self.basedir)
|
||||
except FileNotFoundError:
|
||||
return
|
||||
for name in os_listdir_if_exists("."):
|
||||
if name in ("cur", "new", "tmp"):
|
||||
for msg_name in os_listdir_if_exists(name):
|
||||
entry = get_file_entry(f"{name}/{msg_name}")
|
||||
if entry is not None:
|
||||
self.messages.append(entry)
|
||||
|
||||
else:
|
||||
entry = get_file_entry(name)
|
||||
if entry is not None:
|
||||
self.extrafiles.append(entry)
|
||||
if name == "password":
|
||||
self.last_login = entry.mtime
|
||||
self.extrafiles.sort(key=lambda x: -x.size)
|
||||
os.chdir(old_cwd)
|
||||
|
||||
|
||||
def print_info(msg):
|
||||
print(msg, file=sys.stderr)
|
||||
|
||||
|
||||
class Expiry:
|
||||
def __init__(self, config, dry, now, verbose):
|
||||
self.config = config
|
||||
self.dry = dry
|
||||
self.now = now
|
||||
self.verbose = verbose
|
||||
self.del_mboxes = 0
|
||||
self.all_mboxes = 0
|
||||
self.del_files = 0
|
||||
self.all_files = 0
|
||||
self.start = time.time()
|
||||
|
||||
def remove_mailbox(self, mboxdir):
|
||||
if self.verbose:
|
||||
print_info(f"removing {mboxdir}")
|
||||
if not self.dry:
|
||||
shutil.rmtree(mboxdir)
|
||||
self.del_mboxes += 1
|
||||
|
||||
def remove_file(self, path, mtime=None):
|
||||
if self.verbose:
|
||||
if mtime is not None:
|
||||
date = datetime.fromtimestamp(mtime).strftime("%b %d")
|
||||
print_info(f"removing {date} {path}")
|
||||
else:
|
||||
print_info(f"removing {path}")
|
||||
if not self.dry:
|
||||
try:
|
||||
os.unlink(path)
|
||||
except FileNotFoundError:
|
||||
print_info(f"file not found/vanished {path}")
|
||||
self.del_files += 1
|
||||
|
||||
def process_mailbox_stat(self, mbox):
|
||||
cutoff_without_login = (
|
||||
self.now - int(self.config.delete_inactive_users_after) * 86400
|
||||
)
|
||||
cutoff_mails = self.now - int(self.config.delete_mails_after) * 86400
|
||||
cutoff_large_mails = self.now - int(self.config.delete_large_after) * 86400
|
||||
|
||||
self.all_mboxes += 1
|
||||
changed = False
|
||||
if mbox.last_login and mbox.last_login < cutoff_without_login:
|
||||
self.remove_mailbox(mbox.basedir)
|
||||
return
|
||||
|
||||
# all to-be-removed files are relative to the mailbox basedir
|
||||
try:
|
||||
os.chdir(mbox.basedir)
|
||||
except FileNotFoundError:
|
||||
print_info(f"mailbox not found/vanished {mbox.basedir}")
|
||||
return
|
||||
|
||||
mboxname = os.path.basename(mbox.basedir)
|
||||
if self.verbose:
|
||||
date = datetime.fromtimestamp(mbox.last_login) if mbox.last_login else None
|
||||
if date:
|
||||
print_info(f"checking mailbox {date.strftime('%b %d')} {mboxname}")
|
||||
else:
|
||||
print_info(f"checking mailbox (no last_login) {mboxname}")
|
||||
self.all_files += len(mbox.messages)
|
||||
for message in mbox.messages:
|
||||
if message.mtime < cutoff_mails:
|
||||
self.remove_file(message.relpath, mtime=message.mtime)
|
||||
elif message.size > 200000 and message.mtime < cutoff_large_mails:
|
||||
# we only remove noticed large files (not unnoticed ones in new/)
|
||||
if message.relpath.startswith("cur/"):
|
||||
self.remove_file(message.relpath, mtime=message.mtime)
|
||||
else:
|
||||
continue
|
||||
changed = True
|
||||
if changed:
|
||||
self.remove_file("maildirsize")
|
||||
|
||||
def get_summary(self):
|
||||
return (
|
||||
f"Removed {self.del_mboxes} out of {self.all_mboxes} mailboxes "
|
||||
f"and {self.del_files} out of {self.all_files} files in existing mailboxes "
|
||||
f"in {time.time() - self.start:2.2f} seconds"
|
||||
)
|
||||
|
||||
|
||||
def main(args=None):
|
||||
"""Expire mailboxes and messages according to chatmail config"""
|
||||
parser = ArgumentParser(description=main.__doc__)
|
||||
ini = "/usr/local/lib/chatmaild/chatmail.ini"
|
||||
parser.add_argument(
|
||||
"chatmail_ini",
|
||||
action="store",
|
||||
nargs="?",
|
||||
help=f"path pointing to chatmail.ini file, default: {ini}",
|
||||
default=ini,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--days", action="store", help="assume date to be days older than now"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--maxnum",
|
||||
default=None,
|
||||
action="store",
|
||||
help="maximum number of mailboxes to iterate on",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v",
|
||||
dest="verbose",
|
||||
action="store_true",
|
||||
help="print out removed files and mailboxes",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--remove",
|
||||
dest="remove",
|
||||
action="store_true",
|
||||
help="actually remove all expired files and dirs",
|
||||
)
|
||||
args = parser.parse_args(args)
|
||||
|
||||
config = read_config(args.chatmail_ini)
|
||||
now = datetime.utcnow().timestamp()
|
||||
if args.days:
|
||||
now = now - 86400 * int(args.days)
|
||||
|
||||
maxnum = int(args.maxnum) if args.maxnum else None
|
||||
exp = Expiry(config, dry=not args.remove, now=now, verbose=args.verbose)
|
||||
for mailbox in iter_mailboxes(str(config.mailboxes_dir), maxnum=maxnum):
|
||||
exp.process_mailbox_stat(mailbox)
|
||||
print(exp.get_summary())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(sys.argv[1:])
|
||||
@@ -2,6 +2,7 @@
|
||||
import asyncio
|
||||
import base64
|
||||
import binascii
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
from email import policy
|
||||
@@ -82,14 +83,8 @@ def check_openpgp_payload(payload: bytes):
|
||||
return False
|
||||
|
||||
|
||||
def check_armored_payload(payload: str, outgoing: bool):
|
||||
"""Check the armored PGP message for invalid content.
|
||||
|
||||
:param payload: the armored PGP message
|
||||
:param outgoing: whether the message is outgoing or incoming
|
||||
:return: whether the message is a valid PGP message
|
||||
"""
|
||||
prefix = "-----BEGIN PGP MESSAGE-----\r\n"
|
||||
def check_armored_payload(payload: str):
|
||||
prefix = "-----BEGIN PGP MESSAGE-----\r\n\r\n"
|
||||
if not payload.startswith(prefix):
|
||||
return False
|
||||
payload = payload.removeprefix(prefix)
|
||||
@@ -101,16 +96,6 @@ def check_armored_payload(payload: str, outgoing: bool):
|
||||
return False
|
||||
payload = payload.removesuffix(suffix)
|
||||
|
||||
version_comment = "Version: "
|
||||
if payload.startswith(version_comment):
|
||||
if outgoing: # Disallow comments in outgoing messages
|
||||
return False
|
||||
# Remove comments from incoming messages
|
||||
payload = payload.partition("\r\n")[2]
|
||||
|
||||
while payload.startswith("\r\n"):
|
||||
payload = payload.removeprefix("\r\n")
|
||||
|
||||
# Remove CRC24.
|
||||
payload = payload.rpartition("=")[0]
|
||||
|
||||
@@ -146,7 +131,7 @@ def is_securejoin(message):
|
||||
return True
|
||||
|
||||
|
||||
def check_encrypted(message, outgoing=True):
|
||||
def check_encrypted(message):
|
||||
"""Check that the message is an OpenPGP-encrypted message.
|
||||
|
||||
MIME structure of the message must correspond to <https://www.rfc-editor.org/rfc/rfc3156>.
|
||||
@@ -173,7 +158,7 @@ def check_encrypted(message, outgoing=True):
|
||||
if part.get_content_type() != "application/octet-stream":
|
||||
return False
|
||||
|
||||
if not check_armored_payload(part.get_payload(), outgoing=outgoing):
|
||||
if not check_armored_payload(part.get_payload()):
|
||||
return False
|
||||
else:
|
||||
return False
|
||||
@@ -227,7 +212,7 @@ class OutgoingBeforeQueueHandler:
|
||||
self.send_rate_limiter = SendRateLimiter()
|
||||
|
||||
async def handle_MAIL(self, server, session, envelope, address, mail_options):
|
||||
log_info(f"handle_MAIL from {address}")
|
||||
logging.info(f"handle_MAIL from {address}")
|
||||
envelope.mail_from = address
|
||||
max_sent = self.config.max_user_send_per_minute
|
||||
if not self.send_rate_limiter.is_sending_allowed(address, max_sent):
|
||||
@@ -240,15 +225,11 @@ class OutgoingBeforeQueueHandler:
|
||||
return "250 OK"
|
||||
|
||||
async def handle_DATA(self, server, session, envelope):
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(None, self.sync_handle_DATA, envelope)
|
||||
|
||||
def sync_handle_DATA(self, envelope):
|
||||
log_info("handle_DATA before-queue")
|
||||
logging.info("handle_DATA before-queue")
|
||||
error = self.check_DATA(envelope)
|
||||
if error:
|
||||
return error
|
||||
log_info("re-injecting the mail that passed checks")
|
||||
logging.info("re-injecting the mail that passed checks")
|
||||
client = SMTPClient("localhost", self.config.postfix_reinject_port)
|
||||
client.sendmail(
|
||||
envelope.mail_from, envelope.rcpt_tos, envelope.original_content
|
||||
@@ -257,10 +238,10 @@ class OutgoingBeforeQueueHandler:
|
||||
|
||||
def check_DATA(self, envelope):
|
||||
"""the central filtering function for e-mails."""
|
||||
log_info(f"Processing DATA message from {envelope.mail_from}")
|
||||
logging.info(f"Processing DATA message from {envelope.mail_from}")
|
||||
|
||||
message = BytesParser(policy=policy.default).parsebytes(envelope.content)
|
||||
mail_encrypted = check_encrypted(message, outgoing=True)
|
||||
mail_encrypted = check_encrypted(message)
|
||||
|
||||
_, from_addr = parseaddr(message.get("from").strip())
|
||||
|
||||
@@ -297,15 +278,11 @@ class IncomingBeforeQueueHandler:
|
||||
self.config = config
|
||||
|
||||
async def handle_DATA(self, server, session, envelope):
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(None, self.sync_handle_DATA, envelope)
|
||||
|
||||
def sync_handle_DATA(self, envelope):
|
||||
log_info("handle_DATA before-queue")
|
||||
logging.info("handle_DATA before-queue")
|
||||
error = self.check_DATA(envelope)
|
||||
if error:
|
||||
return error
|
||||
log_info("re-injecting the mail that passed checks")
|
||||
logging.info("re-injecting the mail that passed checks")
|
||||
|
||||
# the smtp daemon on reinject_port_incoming gives it to dkim milter
|
||||
# which looks at source address to determine whether to verify or sign
|
||||
@@ -321,10 +298,10 @@ class IncomingBeforeQueueHandler:
|
||||
|
||||
def check_DATA(self, envelope):
|
||||
"""the central filtering function for e-mails."""
|
||||
log_info(f"Processing DATA message from {envelope.mail_from}")
|
||||
logging.info(f"Processing DATA message from {envelope.mail_from}")
|
||||
|
||||
message = BytesParser(policy=policy.default).parsebytes(envelope.content)
|
||||
mail_encrypted = check_encrypted(message, outgoing=False)
|
||||
mail_encrypted = check_encrypted(message)
|
||||
|
||||
if mail_encrypted or is_securejoin(message):
|
||||
print("Incoming: Filtering encrypted mail.", file=sys.stderr)
|
||||
@@ -363,19 +340,16 @@ class SendRateLimiter:
|
||||
return False
|
||||
|
||||
|
||||
def log_info(msg):
|
||||
print(msg, file=sys.stderr)
|
||||
|
||||
|
||||
def main():
|
||||
args = sys.argv[1:]
|
||||
assert len(args) == 2
|
||||
config = read_config(args[0])
|
||||
mode = args[1]
|
||||
logging.basicConfig(level=logging.WARN)
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
assert mode in ["incoming", "outgoing"]
|
||||
task = asyncmain_beforequeue(config, mode)
|
||||
loop.create_task(task)
|
||||
log_info("entering serving loop")
|
||||
logging.info("entering serving loop")
|
||||
loop.run_forever()
|
||||
|
||||
@@ -1,168 +0,0 @@
|
||||
"""
|
||||
command line tool to analyze mailbox message storage
|
||||
|
||||
example invocation:
|
||||
|
||||
python -m chatmaild.fsreport /path/to/chatmail.ini
|
||||
|
||||
to show storage summaries for all "cur" folders
|
||||
|
||||
python -m chatmaild.fsreport /path/to/chatmail.ini --mdir cur
|
||||
|
||||
to show storage summaries only for first 1000 mailboxes
|
||||
|
||||
python -m chatmaild.fsreport /path/to/chatmail.ini --maxnum 1000
|
||||
|
||||
"""
|
||||
|
||||
import os
|
||||
from argparse import ArgumentParser
|
||||
from datetime import datetime
|
||||
|
||||
from chatmaild.config import read_config
|
||||
from chatmaild.expire import iter_mailboxes
|
||||
|
||||
DAYSECONDS = 24 * 60 * 60
|
||||
MONTHSECONDS = DAYSECONDS * 30
|
||||
|
||||
|
||||
def HSize(size: int):
|
||||
"""Format a size integer as a Human-readable string Kilobyte, Megabyte or Gigabyte"""
|
||||
if size < 10000:
|
||||
return f"{size / 1000:5.2f}K"
|
||||
if size < 1000 * 1000:
|
||||
return f"{size / 1000:5.0f}K"
|
||||
if size < 1000 * 1000 * 1000:
|
||||
return f"{int(size / 1000000):5.0f}M"
|
||||
return f"{size / 1000000000:5.2f}G"
|
||||
|
||||
|
||||
class Report:
|
||||
def __init__(self, now, min_login_age, mdir):
|
||||
self.size_extra = 0
|
||||
self.size_messages = 0
|
||||
self.now = now
|
||||
self.min_login_age = min_login_age
|
||||
self.mdir = mdir
|
||||
|
||||
self.num_ci_logins = self.num_all_logins = 0
|
||||
self.login_buckets = {x: 0 for x in (1, 10, 30, 40, 80, 100, 150)}
|
||||
|
||||
self.message_buckets = {x: 0 for x in (0, 160000, 500000, 2000000)}
|
||||
|
||||
def process_mailbox_stat(self, mailbox):
|
||||
# categorize login times
|
||||
last_login = mailbox.last_login
|
||||
if last_login:
|
||||
self.num_all_logins += 1
|
||||
if os.path.basename(mailbox.basedir)[:3] == "ci-":
|
||||
self.num_ci_logins += 1
|
||||
else:
|
||||
for days in self.login_buckets:
|
||||
if last_login >= self.now - days * DAYSECONDS:
|
||||
self.login_buckets[days] += 1
|
||||
|
||||
cutoff_login_date = self.now - self.min_login_age * DAYSECONDS
|
||||
if last_login and last_login <= cutoff_login_date:
|
||||
# categorize message sizes
|
||||
for size in self.message_buckets:
|
||||
for msg in mailbox.messages:
|
||||
if msg.size >= size:
|
||||
if self.mdir and not msg.relpath.startswith(self.mdir):
|
||||
continue
|
||||
self.message_buckets[size] += msg.size
|
||||
|
||||
self.size_messages += sum(entry.size for entry in mailbox.messages)
|
||||
self.size_extra += sum(entry.size for entry in mailbox.extrafiles)
|
||||
|
||||
def dump_summary(self):
|
||||
all_messages = self.size_messages
|
||||
print()
|
||||
print("## Mailbox storage use analysis")
|
||||
print(f"Mailbox data total size: {HSize(self.size_extra + all_messages)}")
|
||||
print(f"Messages total size : {HSize(all_messages)}")
|
||||
try:
|
||||
percent = self.size_extra / (self.size_extra + all_messages) * 100
|
||||
except ZeroDivisionError:
|
||||
percent = 100
|
||||
print(f"Extra files : {HSize(self.size_extra)} ({percent:.2f}%)")
|
||||
|
||||
print()
|
||||
if self.min_login_age:
|
||||
print(f"### Message storage for {self.min_login_age} days old logins")
|
||||
|
||||
pref = f"[{self.mdir}] " if self.mdir else ""
|
||||
for minsize, sumsize in self.message_buckets.items():
|
||||
percent = (sumsize / all_messages * 100) if all_messages else 0
|
||||
print(
|
||||
f"{pref}larger than {HSize(minsize)}: {HSize(sumsize)} ({percent:.2f}%)"
|
||||
)
|
||||
|
||||
user_logins = self.num_all_logins - self.num_ci_logins
|
||||
|
||||
def p(num):
|
||||
return f"({num / user_logins * 100:2.2f}%)" if user_logins else "100%"
|
||||
|
||||
print()
|
||||
print(f"## Login stats, from date reference {datetime.fromtimestamp(self.now)}")
|
||||
print(f"all: {HSize(self.num_all_logins)}")
|
||||
print(f"non-ci: {HSize(user_logins)}")
|
||||
print(f"ci: {HSize(self.num_ci_logins)}")
|
||||
for days, active in self.login_buckets.items():
|
||||
print(f"last {days:3} days: {HSize(active)} {p(active)}")
|
||||
|
||||
|
||||
def main(args=None):
|
||||
"""Report about filesystem storage usage of all mailboxes and messages"""
|
||||
parser = ArgumentParser(description=main.__doc__)
|
||||
ini = "/usr/local/lib/chatmaild/chatmail.ini"
|
||||
parser.add_argument(
|
||||
"chatmail_ini",
|
||||
action="store",
|
||||
nargs="?",
|
||||
help=f"path pointing to chatmail.ini file, default: {ini}",
|
||||
default=ini,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--days",
|
||||
default=0,
|
||||
action="store",
|
||||
help="assume date to be days older than now",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--min-login-age",
|
||||
default=0,
|
||||
dest="min_login_age",
|
||||
action="store",
|
||||
help="only sum up message size if last login is at least min-login-age days old",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mdir",
|
||||
action="store",
|
||||
help="only consider 'cur' or 'new' or 'tmp' messages for summary",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--maxnum",
|
||||
default=None,
|
||||
action="store",
|
||||
help="maximum number of mailboxes to iterate on",
|
||||
)
|
||||
|
||||
args = parser.parse_args(args)
|
||||
|
||||
config = read_config(args.chatmail_ini)
|
||||
|
||||
now = datetime.utcnow().timestamp()
|
||||
if args.days:
|
||||
now = now - 86400 * int(args.days)
|
||||
|
||||
maxnum = int(args.maxnum) if args.maxnum else None
|
||||
rep = Report(now=now, min_login_age=int(args.min_login_age), mdir=args.mdir)
|
||||
for mbox in iter_mailboxes(str(config.mailboxes_dir), maxnum=maxnum):
|
||||
rep.process_mailbox_stat(mbox)
|
||||
rep.dump_summary()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -45,14 +45,11 @@ passthrough_senders =
|
||||
# (space-separated, item may start with "@" to whitelist whole recipient domains)
|
||||
passthrough_recipients = xstore@testrun.org echo@{mail_domain}
|
||||
|
||||
# path to www directory - documented here: https://github.com/chatmail/relay/#custom-web-pages
|
||||
#www_folder = www
|
||||
|
||||
#
|
||||
# Deployment Details
|
||||
#
|
||||
|
||||
# SMTP outgoing filtermail and reinjection
|
||||
# SMTP outgoing filtermail and reinjection
|
||||
filtermail_smtp_port = 10080
|
||||
postfix_reinject_port = 10025
|
||||
|
||||
@@ -63,8 +60,18 @@ postfix_reinject_port_incoming = 10026
|
||||
# if set to "True" IPv6 is disabled
|
||||
disable_ipv6 = False
|
||||
|
||||
# Your email adress, which will be used in acmetool to manage Let's Encrypt SSL certificates
|
||||
acme_email =
|
||||
# if you set "True", acmetool will not be installed and you will have to manage certificates yourself.
|
||||
use_foreign_cert_manager = False
|
||||
|
||||
#
|
||||
# Kernel settings
|
||||
#
|
||||
|
||||
# if you set "True", the kernel settings will be configured according to the values below
|
||||
change_kernel_settings = True
|
||||
|
||||
# change fs.inotify.max_user_instances and fs.inotify.max_user_watches kernel settings
|
||||
fs_inotify_max_user_instances_and_watchers = 65535
|
||||
|
||||
# Defaults to https://iroh.{{mail_domain}} and running `iroh-relay` on the chatmail
|
||||
# service.
|
||||
|
||||
@@ -7,7 +7,6 @@ from .config import read_config
|
||||
from .dictproxy import DictProxy
|
||||
from .filedict import FileDict
|
||||
from .notifier import Notifier
|
||||
from .turnserver import turn_credentials
|
||||
|
||||
|
||||
def _is_valid_token_timestamp(timestamp, now):
|
||||
@@ -76,12 +75,11 @@ class Metadata:
|
||||
|
||||
|
||||
class MetadataDictProxy(DictProxy):
|
||||
def __init__(self, notifier, metadata, iroh_relay=None, turn_hostname=None):
|
||||
def __init__(self, notifier, metadata, iroh_relay=None):
|
||||
super().__init__()
|
||||
self.notifier = notifier
|
||||
self.metadata = metadata
|
||||
self.iroh_relay = iroh_relay
|
||||
self.turn_hostname = turn_hostname
|
||||
|
||||
def handle_lookup(self, parts):
|
||||
# Lpriv/43f5f508a7ea0366dff30200c15250e3/devicetoken\tlkj123poi@c2.testrun.org
|
||||
@@ -100,11 +98,6 @@ class MetadataDictProxy(DictProxy):
|
||||
):
|
||||
# Handle `GETMETADATA "" /shared/vendor/deltachat/irohrelay`
|
||||
return f"O{self.iroh_relay}\n"
|
||||
elif keyname == "vendor/vendor.dovecot/pvt/server/vendor/deltachat/turn":
|
||||
res = turn_credentials()
|
||||
port = 3478
|
||||
return f"O{self.turn_hostname}:{port}:{res}\n"
|
||||
|
||||
logging.warning(f"lookup ignored: {parts!r}")
|
||||
return "N\n"
|
||||
|
||||
@@ -128,7 +121,6 @@ def main():
|
||||
|
||||
config = read_config(config_path)
|
||||
iroh_relay = config.iroh_relay
|
||||
mail_domain = config.mail_domain
|
||||
|
||||
vmail_dir = config.mailboxes_dir
|
||||
if not vmail_dir.exists():
|
||||
@@ -142,10 +134,7 @@ def main():
|
||||
notifier.start_notification_threads(metadata.remove_token_from_addr)
|
||||
|
||||
dictproxy = MetadataDictProxy(
|
||||
notifier=notifier,
|
||||
metadata=metadata,
|
||||
iroh_relay=iroh_relay,
|
||||
turn_hostname=mail_domain,
|
||||
notifier=notifier, metadata=metadata, iroh_relay=iroh_relay
|
||||
)
|
||||
|
||||
dictproxy.serve_forever_from_socket(socket)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import time
|
||||
|
||||
from chatmaild.delete_inactive_users import delete_inactive_users
|
||||
from chatmaild.doveauth import AuthDictProxy
|
||||
from chatmaild.expire import main as main_expire
|
||||
|
||||
|
||||
def test_login_timestamps(example_config):
|
||||
@@ -45,12 +45,7 @@ def test_delete_inactive_users(example_config):
|
||||
for addr in to_remove:
|
||||
assert example_config.get_user(addr).maildir.exists()
|
||||
|
||||
main_expire(
|
||||
args=[
|
||||
"--remove",
|
||||
str(example_config._inipath),
|
||||
]
|
||||
)
|
||||
delete_inactive_users(example_config)
|
||||
|
||||
for p in example_config.mailboxes_dir.iterdir():
|
||||
assert not p.name.startswith("old")
|
||||
|
||||
@@ -1,150 +0,0 @@
|
||||
import os
|
||||
import random
|
||||
from datetime import datetime
|
||||
from fnmatch import fnmatch
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from chatmaild.expire import (
|
||||
FileEntry,
|
||||
MailboxStat,
|
||||
get_file_entry,
|
||||
iter_mailboxes,
|
||||
os_listdir_if_exists,
|
||||
)
|
||||
from chatmaild.expire import main as expiry_main
|
||||
from chatmaild.fsreport import main as report_main
|
||||
|
||||
|
||||
def fill_mbox(basedir):
|
||||
basedir1 = basedir.joinpath("mailbox1@example.org")
|
||||
basedir1.mkdir()
|
||||
password = basedir1.joinpath("password")
|
||||
password.write_text("xxx")
|
||||
basedir1.joinpath("maildirsize").write_text("xxx")
|
||||
|
||||
garbagedir = basedir1.joinpath("garbagedir")
|
||||
garbagedir.mkdir()
|
||||
|
||||
create_new_messages(basedir1, ["cur/msg1"], size=500)
|
||||
create_new_messages(basedir1, ["new/msg2"], size=600)
|
||||
return basedir1
|
||||
|
||||
|
||||
def create_new_messages(basedir, relpaths, size=1000, days=0):
|
||||
now = datetime.utcnow().timestamp()
|
||||
|
||||
for relpath in relpaths:
|
||||
msg_path = Path(basedir).joinpath(relpath)
|
||||
msg_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
msg_path.write_text("x" * size)
|
||||
# accessed now, modified N days ago
|
||||
os.utime(msg_path, (now, now - days * 86400))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mbox1(example_config):
|
||||
basedir1 = fill_mbox(example_config.mailboxes_dir)
|
||||
return MailboxStat(basedir1)
|
||||
|
||||
|
||||
def test_filentry_ordering(tmp_path):
|
||||
l = [FileEntry(f"x{i}", size=i + 10, mtime=1000 - i) for i in range(10)]
|
||||
sorted = list(l)
|
||||
random.shuffle(l)
|
||||
l.sort(key=lambda x: x.size)
|
||||
assert l == sorted
|
||||
|
||||
|
||||
def test_no_mailbxoes(tmp_path, capsys):
|
||||
assert [] == list(iter_mailboxes(str(tmp_path.joinpath("notexists")), maxnum=10))
|
||||
out, err = capsys.readouterr()
|
||||
assert "no mailboxes" in err
|
||||
|
||||
|
||||
def test_stats_mailbox(mbox1):
|
||||
password = Path(mbox1.basedir).joinpath("password")
|
||||
assert mbox1.last_login == password.stat().st_mtime
|
||||
assert len(mbox1.messages) == 2
|
||||
|
||||
msgs = list(sorted(mbox1.messages, key=lambda x: x.size))
|
||||
assert len(msgs) == 2
|
||||
assert msgs[0].size == 500 # cur
|
||||
assert msgs[1].size == 600 # new
|
||||
|
||||
create_new_messages(mbox1.basedir, ["large-extra"], size=1000)
|
||||
create_new_messages(mbox1.basedir, ["index-something"], size=3)
|
||||
mbox2 = MailboxStat(mbox1.basedir)
|
||||
assert len(mbox2.extrafiles) == 4
|
||||
assert mbox2.extrafiles[0].size == 1000
|
||||
|
||||
# cope well with mailbox dirs that have no password (for whatever reason)
|
||||
Path(mbox1.basedir).joinpath("password").unlink()
|
||||
mbox3 = MailboxStat(mbox1.basedir)
|
||||
assert mbox3.last_login is None
|
||||
|
||||
|
||||
def test_report_no_mailboxes(example_config):
|
||||
args = (str(example_config._inipath),)
|
||||
report_main(args)
|
||||
|
||||
|
||||
def test_report(mbox1, example_config):
|
||||
args = (str(example_config._inipath),)
|
||||
report_main(args)
|
||||
args = list(args) + "--days 1".split()
|
||||
report_main(args)
|
||||
args = list(args) + "--min-login-age 1".split()
|
||||
report_main(args)
|
||||
args = list(args) + "--mdir cur".split()
|
||||
report_main(args)
|
||||
|
||||
|
||||
def test_expiry_cli_basic(example_config, mbox1):
|
||||
args = (str(example_config._inipath),)
|
||||
expiry_main(args)
|
||||
|
||||
|
||||
def test_expiry_cli_old_files(capsys, example_config, mbox1):
|
||||
relpaths_old = ["cur/msg_old1", "cur/msg_old1"]
|
||||
cutoff_days = int(example_config.delete_mails_after) + 1
|
||||
create_new_messages(mbox1.basedir, relpaths_old, size=1000, days=cutoff_days)
|
||||
|
||||
relpaths_large = ["cur/msg_old_large1", "new/msg_old_large2"]
|
||||
cutoff_days = int(example_config.delete_large_after) + 1
|
||||
create_new_messages(
|
||||
mbox1.basedir, relpaths_large, size=1000 * 300, days=cutoff_days
|
||||
)
|
||||
|
||||
create_new_messages(mbox1.basedir, ["cur/shouldstay"], size=1000 * 300, days=1)
|
||||
|
||||
args = str(example_config._inipath), "--remove", "-v"
|
||||
expiry_main(args)
|
||||
out, err = capsys.readouterr()
|
||||
|
||||
allpaths = relpaths_old + relpaths_large + ["maildirsize"]
|
||||
for path in allpaths:
|
||||
for line in err.split("\n"):
|
||||
if fnmatch(line, f"removing*{path}"):
|
||||
break
|
||||
else:
|
||||
if path != "new/msg_old_large2":
|
||||
pytest.fail(f"failed to remove {path}\n{err}")
|
||||
|
||||
assert "shouldstay" not in err
|
||||
|
||||
|
||||
def test_get_file_entry(tmp_path):
|
||||
assert get_file_entry(str(tmp_path.joinpath("123123"))) is None
|
||||
p = tmp_path.joinpath("x")
|
||||
p.write_text("hello")
|
||||
entry = get_file_entry(str(p))
|
||||
assert entry.size == 5
|
||||
assert entry.mtime
|
||||
|
||||
|
||||
def test_os_listdir_if_exists(tmp_path):
|
||||
tmp_path.joinpath("x").write_text("hello")
|
||||
assert len(os_listdir_if_exists(str(tmp_path))) == 1
|
||||
assert len(os_listdir_if_exists(str(tmp_path.joinpath("123123")))) == 0
|
||||
@@ -241,9 +241,8 @@ def test_cleartext_passthrough_senders(gencreds, handler, maildata):
|
||||
|
||||
|
||||
def test_check_armored_payload():
|
||||
prefix = "-----BEGIN PGP MESSAGE-----\r\n"
|
||||
comment = "Version: ProtonMail\r\n"
|
||||
payload = """\r
|
||||
payload = """-----BEGIN PGP MESSAGE-----\r
|
||||
\r
|
||||
wU4DSqFx0d1yqAoSAQdAYkX/ZN/Az4B0k7X47zKyWrXxlDEdS3WOy0Yf2+GJTFgg\r
|
||||
Zk5ql0mLG8Ze+ZifCS0XMO4otlemSyJ0K1ZPdFMGzUDBTgNqzkFabxXoXRIBB0AM\r
|
||||
755wlX41X6Ay3KhnwBq7yEqSykVH6F3x11iHPKraLCAGZoaS8bKKNy/zg5slda1X\r
|
||||
@@ -279,25 +278,16 @@ UN4fiB0KR9JyG2ayUdNJVkXZSZLnHyRgiaadlpUo16LVvw==\r
|
||||
\r
|
||||
"""
|
||||
|
||||
commented_payload = prefix + comment + payload
|
||||
assert check_armored_payload(commented_payload, outgoing=False) == True
|
||||
assert check_armored_payload(commented_payload, outgoing=True) == False
|
||||
|
||||
payload = prefix + payload
|
||||
assert check_armored_payload(payload, outgoing=False) == True
|
||||
assert check_armored_payload(payload, outgoing=True) == True
|
||||
assert check_armored_payload(payload) == True
|
||||
|
||||
payload = payload.removesuffix("\r\n")
|
||||
assert check_armored_payload(payload, outgoing=False) == True
|
||||
assert check_armored_payload(payload, outgoing=True) == True
|
||||
assert check_armored_payload(payload) == True
|
||||
|
||||
payload = payload.removesuffix("\r\n")
|
||||
assert check_armored_payload(payload, outgoing=False) == True
|
||||
assert check_armored_payload(payload, outgoing=True) == True
|
||||
assert check_armored_payload(payload) == True
|
||||
|
||||
payload = payload.removesuffix("\r\n")
|
||||
assert check_armored_payload(payload, outgoing=False) == True
|
||||
assert check_armored_payload(payload, outgoing=True) == True
|
||||
assert check_armored_payload(payload) == True
|
||||
|
||||
payload = """-----BEGIN PGP MESSAGE-----\r
|
||||
\r
|
||||
@@ -305,8 +295,7 @@ HELLOWORLD
|
||||
-----END PGP MESSAGE-----\r
|
||||
\r
|
||||
"""
|
||||
assert check_armored_payload(payload, outgoing=False) == False
|
||||
assert check_armored_payload(payload, outgoing=True) == False
|
||||
assert check_armored_payload(payload) == False
|
||||
|
||||
payload = """-----BEGIN PGP MESSAGE-----\r
|
||||
\r
|
||||
@@ -314,8 +303,7 @@ HELLOWORLD
|
||||
-----END PGP MESSAGE-----\r
|
||||
\r
|
||||
"""
|
||||
assert check_armored_payload(payload, outgoing=False) == False
|
||||
assert check_armored_payload(payload, outgoing=True) == False
|
||||
assert check_armored_payload(payload) == False
|
||||
|
||||
# Test payload using partial body length
|
||||
# as generated by GopenPGP.
|
||||
@@ -357,5 +345,4 @@ myLbG7cJB787QjplEyVe2P/JBO6xYvbkJLf9Q+HaviTO25rugRSrYsoKMDfO8VlQ\r
|
||||
=6iHb\r
|
||||
-----END PGP MESSAGE-----\r
|
||||
"""
|
||||
assert check_armored_payload(payload, outgoing=False) == True
|
||||
assert check_armored_payload(payload, outgoing=True) == True
|
||||
assert check_armored_payload(payload) == True
|
||||
|
||||
@@ -1,78 +0,0 @@
|
||||
import smtplib
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def smtpserver():
|
||||
from pytest_localserver import smtp
|
||||
|
||||
server = smtp.Server("127.0.0.1")
|
||||
server.start()
|
||||
yield server
|
||||
server.stop()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_popen(request):
|
||||
def popen(cmdargs, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kw):
|
||||
p = subprocess.Popen(
|
||||
cmdargs,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
|
||||
def fin():
|
||||
p.terminate()
|
||||
out, err = p.communicate()
|
||||
print(out.decode("ascii"))
|
||||
print(err.decode("ascii"), file=sys.stderr)
|
||||
|
||||
request.addfinalizer(fin)
|
||||
return p
|
||||
|
||||
return popen
|
||||
|
||||
|
||||
@pytest.mark.parametrize("filtermail_mode", ["outgoing", "incoming"])
|
||||
def test_one_mail(
|
||||
make_config, make_popen, smtpserver, maildata, filtermail_mode, monkeypatch
|
||||
):
|
||||
monkeypatch.setenv("PYTHONUNBUFFERED", "1")
|
||||
smtp_inject_port = 20025
|
||||
if filtermail_mode == "outgoing":
|
||||
settings = dict(
|
||||
postfix_reinject_port=smtpserver.port,
|
||||
filtermail_smtp_port=smtp_inject_port,
|
||||
)
|
||||
else:
|
||||
settings = dict(
|
||||
postfix_reinject_port_incoming=smtpserver.port,
|
||||
filtermail_smtp_port_incoming=smtp_inject_port,
|
||||
)
|
||||
|
||||
config = make_config("example.org", settings=settings)
|
||||
path = str(config._inipath)
|
||||
|
||||
popen = make_popen(["filtermail", path, filtermail_mode])
|
||||
line = popen.stderr.readline().strip()
|
||||
if b"loop" not in line:
|
||||
print(line.decode("ascii"), file=sys.stderr)
|
||||
pytest.fail("starting filtermail failed")
|
||||
|
||||
addr = f"user1@{config.mail_domain}"
|
||||
config.get_user(addr).set_password("l1k2j3l1k2j3l")
|
||||
|
||||
# send encrypted mail
|
||||
data = str(maildata("encrypted.eml", from_addr=addr, to_addr=addr))
|
||||
client = smtplib.SMTP("localhost", smtp_inject_port)
|
||||
client.sendmail(addr, [addr], data)
|
||||
assert len(smtpserver.outbox) == 1
|
||||
|
||||
# send un-encrypted mail that errors
|
||||
data = str(maildata("fake-encrypted.eml", from_addr=addr, to_addr=addr))
|
||||
with pytest.raises(smtplib.SMTPDataError) as e:
|
||||
client.sendmail(addr, [addr], data)
|
||||
assert e.value.smtp_code == 523
|
||||
@@ -1,9 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
import socket
|
||||
|
||||
|
||||
def turn_credentials() -> str:
|
||||
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client_socket:
|
||||
client_socket.connect("/run/chatmail-turn/turn.socket")
|
||||
with client_socket.makefile("rb") as file:
|
||||
return file.readline().decode("utf-8").strip()
|
||||
@@ -128,11 +128,6 @@ def _install_remote_venv_with_chatmaild(config) -> None:
|
||||
"echobot",
|
||||
"chatmail-metadata",
|
||||
"lastlogin",
|
||||
"turnserver",
|
||||
"chatmail-expire",
|
||||
"chatmail-expire.timer",
|
||||
"chatmail-fsreport",
|
||||
"chatmail-fsreport.timer",
|
||||
):
|
||||
execpath = fn if fn != "filtermail-incoming" else "filtermail"
|
||||
params = dict(
|
||||
@@ -141,34 +136,27 @@ def _install_remote_venv_with_chatmaild(config) -> None:
|
||||
remote_venv_dir=remote_venv_dir,
|
||||
mail_domain=config.mail_domain,
|
||||
)
|
||||
|
||||
basename = fn if "." in fn else f"{fn}.service"
|
||||
|
||||
source_path = importlib.resources.files(__package__).joinpath("service", f"{basename}.f")
|
||||
source_path = importlib.resources.files(__package__).joinpath(
|
||||
"service", f"{fn}.service.f"
|
||||
)
|
||||
content = source_path.read_text().format(**params).encode()
|
||||
|
||||
files.put(
|
||||
name=f"Upload {basename}",
|
||||
name=f"Upload {fn}.service",
|
||||
src=io.BytesIO(content),
|
||||
dest=f"/etc/systemd/system/{basename}",
|
||||
dest=f"/etc/systemd/system/{fn}.service",
|
||||
**root_owned,
|
||||
)
|
||||
if fn == "chatmail-expire" or fn == "chatmail-fsreport":
|
||||
# don't auto-start but let the corresponding timer trigger execution
|
||||
enabled = False
|
||||
else:
|
||||
enabled = True
|
||||
systemd.service(
|
||||
name=f"Setup {basename}",
|
||||
service=basename,
|
||||
running=enabled,
|
||||
enabled=enabled,
|
||||
restarted=enabled,
|
||||
name=f"Setup {fn} service",
|
||||
service=f"{fn}.service",
|
||||
running=True,
|
||||
enabled=True,
|
||||
restarted=True,
|
||||
daemon_reload=True,
|
||||
)
|
||||
|
||||
|
||||
|
||||
def _configure_opendkim(domain: str, dkim_selector: str = "dkim") -> bool:
|
||||
"""Configures OpenDKIM"""
|
||||
need_restart = False
|
||||
@@ -338,9 +326,9 @@ def _install_dovecot_package(package: str, arch: str):
|
||||
|
||||
match (package, arch):
|
||||
case ("core", "amd64"):
|
||||
sha256 = "dd060706f52a306fa863d874717210b9fe10536c824afe1790eec247ded5b27d"
|
||||
sha256 = "43f593332e22ac7701c62d58b575d2ca409e0f64857a2803be886c22860f5587"
|
||||
case ("core", "arm64"):
|
||||
sha256 = "e7548e8a82929722e973629ecc40fcfa886894cef3db88f23535149e7f730dc9"
|
||||
sha256 = "4d21eba1a83f51c100f08f2e49f0c9f8f52f721ebc34f75018e043306da993a7"
|
||||
case ("imapd", "amd64"):
|
||||
sha256 = "8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86"
|
||||
case ("imapd", "arm64"):
|
||||
@@ -398,27 +386,30 @@ def _configure_dovecot(config: Config, debug: bool = False) -> bool:
|
||||
)
|
||||
need_restart |= lua_push_notification_script.changed
|
||||
|
||||
# remove historic expunge script
|
||||
# which is now implemented through a systemd chatmail-expire service/timer
|
||||
files.file(
|
||||
path="/etc/cron.d/expunge",
|
||||
present=False,
|
||||
files.template(
|
||||
src=importlib.resources.files(__package__).joinpath("dovecot/expunge.cron.j2"),
|
||||
dest="/etc/cron.d/expunge",
|
||||
user="root",
|
||||
group="root",
|
||||
mode="644",
|
||||
config=config,
|
||||
)
|
||||
|
||||
# as per https://doc.dovecot.org/configuration_manual/os/
|
||||
# as per https://doc.dovecot.org/2.3/configuration_manual/os/
|
||||
# it is recommended to set the following inotify limits
|
||||
for name in ("max_user_instances", "max_user_watches"):
|
||||
key = f"fs.inotify.{name}"
|
||||
if host.get_fact(Sysctl)[key] > 65535:
|
||||
# Skip updating limits if already sufficient
|
||||
# (enables running in incus containers where sysctl readonly)
|
||||
continue
|
||||
server.sysctl(
|
||||
name=f"Change {key}",
|
||||
key=key,
|
||||
value=65535,
|
||||
persist=True,
|
||||
)
|
||||
if config.change_kernel_settings:
|
||||
for name in ("max_user_instances", "max_user_watches"):
|
||||
key = f"fs.inotify.{name}"
|
||||
if host.get_fact(Sysctl)[key] == config.fs_inotify_max_user_instances_and_watchers:
|
||||
# Skip updating limits if already sufficient
|
||||
# (enables running in incus containers where sysctl readonly)
|
||||
continue
|
||||
server.sysctl(
|
||||
name=f"Change {key}",
|
||||
key=key,
|
||||
value=config.fs_inotify_max_user_instances_and_watchers,
|
||||
persist=True,
|
||||
)
|
||||
|
||||
timezone_env = files.line(
|
||||
name="Set TZ environment variable",
|
||||
@@ -507,56 +498,6 @@ def check_config(config):
|
||||
return config
|
||||
|
||||
|
||||
def deploy_turn_server(config):
|
||||
(url, sha256sum) = {
|
||||
"x86_64": (
|
||||
"https://github.com/chatmail/chatmail-turn/releases/download/v0.3/chatmail-turn-x86_64-linux",
|
||||
"841e527c15fdc2940b0469e206188ea8f0af48533be12ecb8098520f813d41e4",
|
||||
),
|
||||
"aarch64": (
|
||||
"https://github.com/chatmail/chatmail-turn/releases/download/v0.3/chatmail-turn-aarch64-linux",
|
||||
"a5fc2d06d937b56a34e098d2cd72a82d3e89967518d159bf246dc69b65e81b42",
|
||||
),
|
||||
}[host.get_fact(facts.server.Arch)]
|
||||
|
||||
need_restart = False
|
||||
|
||||
existing_sha256sum = host.get_fact(Sha256File, "/usr/local/bin/chatmail-turn")
|
||||
if existing_sha256sum != sha256sum:
|
||||
server.shell(
|
||||
name="Download chatmail-turn",
|
||||
commands=[
|
||||
f"(curl -L {url} >/usr/local/bin/chatmail-turn.new && (echo '{sha256sum} /usr/local/bin/chatmail-turn.new' | sha256sum -c) && mv /usr/local/bin/chatmail-turn.new /usr/local/bin/chatmail-turn)",
|
||||
"chmod 755 /usr/local/bin/chatmail-turn",
|
||||
],
|
||||
)
|
||||
need_restart = True
|
||||
|
||||
source_path = importlib.resources.files(__package__).joinpath(
|
||||
"service", "turnserver.service.f"
|
||||
)
|
||||
content = source_path.read_text().format(mail_domain=config.mail_domain).encode()
|
||||
|
||||
systemd_unit = files.put(
|
||||
name="Upload turnserver.service",
|
||||
src=io.BytesIO(content),
|
||||
dest="/etc/systemd/system/turnserver.service",
|
||||
user="root",
|
||||
group="root",
|
||||
mode="644",
|
||||
)
|
||||
need_restart |= systemd_unit.changed
|
||||
|
||||
systemd.service(
|
||||
name="Setup turnserver service",
|
||||
service="turnserver.service",
|
||||
running=True,
|
||||
enabled=True,
|
||||
restarted=need_restart,
|
||||
daemon_reload=systemd_unit.changed,
|
||||
)
|
||||
|
||||
|
||||
def deploy_mtail(config):
|
||||
# Uninstall mtail package, we are going to install a static binary.
|
||||
apt.packages(name="Uninstall mtail", packages=["mtail"], present=False)
|
||||
@@ -681,7 +622,7 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
|
||||
check_config(config)
|
||||
mail_domain = config.mail_domain
|
||||
|
||||
from .www import build_webpages, find_merge_conflict, get_paths
|
||||
from .www import build_webpages, get_paths
|
||||
|
||||
server.group(name="Create vmail group", group="vmail", system=True)
|
||||
server.user(name="Create vmail user", user="vmail", group="vmail", system=True)
|
||||
@@ -733,8 +674,6 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
|
||||
packages=["rsync"],
|
||||
)
|
||||
|
||||
deploy_turn_server(config)
|
||||
|
||||
# Run local DNS resolver `unbound`.
|
||||
# `resolvconf` takes care of setting up /etc/resolv.conf
|
||||
# to use 127.0.0.1 as the resolver.
|
||||
@@ -787,11 +726,11 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
|
||||
deploy_iroh_relay(config)
|
||||
|
||||
# Deploy acmetool to have TLS certificates.
|
||||
tls_domains = [mail_domain, f"mta-sts.{mail_domain}", f"www.{mail_domain}"]
|
||||
deploy_acmetool(
|
||||
email=config.acme_email,
|
||||
domains=tls_domains,
|
||||
)
|
||||
if not config.use_foreign_cert_manager:
|
||||
tls_domains = [mail_domain, f"mta-sts.{mail_domain}", f"www.{mail_domain}"]
|
||||
deploy_acmetool(
|
||||
domains=tls_domains,
|
||||
)
|
||||
|
||||
apt.packages(
|
||||
# required for setfacl for echobot
|
||||
@@ -823,14 +762,12 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
|
||||
# if www_folder was set to a non-existing folder, skip upload
|
||||
if not www_path.is_dir():
|
||||
logger.warning("Building web pages is disabled in chatmail.ini, skipping")
|
||||
elif (path := find_merge_conflict(src_dir)) is not None:
|
||||
logger.warning(f"Merge conflict found in {path}, skipping")
|
||||
else:
|
||||
# if www_folder is a hugo page, build it
|
||||
if build_dir:
|
||||
www_path = build_webpages(src_dir, build_dir, config)
|
||||
# if it is not a hugo page, upload it as is
|
||||
files.rsync(f"{www_path}/", "/var/www/html", flags=["-avz", "--chown=www-data"])
|
||||
files.rsync(f"{www_path}/", "/var/www/html", flags=["-avz"])
|
||||
|
||||
_install_remote_venv_with_chatmaild(config)
|
||||
debug = False
|
||||
@@ -878,13 +815,6 @@ def deploy_chatmail(config_path: Path, disable_mail: bool) -> None:
|
||||
restarted=nginx_need_restart,
|
||||
)
|
||||
|
||||
systemd.service(
|
||||
name="Start and enable fcgiwrap",
|
||||
service="fcgiwrap.service",
|
||||
running=True,
|
||||
enabled=True,
|
||||
)
|
||||
|
||||
systemd.service(
|
||||
name="Restart echobot if postfix and dovecot were just started",
|
||||
service="echobot.service",
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
request:
|
||||
provider: https://acme-v02.api.letsencrypt.org/directory
|
||||
key:
|
||||
type: ecdsa
|
||||
ecdsa-curve: nistp256
|
||||
type: rsa
|
||||
challenge:
|
||||
webroot-paths:
|
||||
- /var/www/html/.well-known/acme-challenge
|
||||
|
||||
@@ -19,7 +19,7 @@ from packaging import version
|
||||
from termcolor import colored
|
||||
|
||||
from . import dns, remote
|
||||
from .sshexec import SSHExec, LocalExec
|
||||
from .sshexec import SSHExec
|
||||
|
||||
#
|
||||
# cmdeploy sub commands and options
|
||||
@@ -32,30 +32,17 @@ def init_cmd_options(parser):
|
||||
action="store",
|
||||
help="fully qualified DNS domain name for your chatmail instance",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--force",
|
||||
dest="recreate_ini",
|
||||
action="store_true",
|
||||
help="force reacreate ini file",
|
||||
)
|
||||
|
||||
|
||||
def init_cmd(args, out):
|
||||
"""Initialize chatmail config file."""
|
||||
mail_domain = args.chatmail_domain
|
||||
inipath = args.inipath
|
||||
if args.inipath.exists():
|
||||
if not args.recreate_ini:
|
||||
print(f"[WARNING] Path exists, not modifying: {inipath}")
|
||||
return 1
|
||||
else:
|
||||
print(
|
||||
f"[WARNING] Force argument was provided, deleting config file: {inipath}"
|
||||
)
|
||||
inipath.unlink()
|
||||
|
||||
write_initial_config(inipath, mail_domain, overrides={})
|
||||
out.green(f"created config file for {mail_domain} in {inipath}")
|
||||
print(f"Path exists, not modifying: {args.inipath}")
|
||||
return 1
|
||||
else:
|
||||
write_initial_config(args.inipath, mail_domain, overrides={})
|
||||
out.green(f"created config file for {mail_domain} in {args.inipath}")
|
||||
|
||||
|
||||
def run_cmd_options(parser):
|
||||
@@ -72,12 +59,10 @@ def run_cmd_options(parser):
|
||||
help="install/upgrade the server, but disable postfix & dovecot for now",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-dns-check",
|
||||
dest="dns_check_disabled",
|
||||
action="store_true",
|
||||
help="disable checks nslookup for dns",
|
||||
"--ssh-host",
|
||||
dest="ssh_host",
|
||||
help="Deploy to 'localhost', via 'docker', or to a specific SSH host",
|
||||
)
|
||||
add_ssh_host_option(parser)
|
||||
|
||||
|
||||
def run_cmd(args, out):
|
||||
@@ -86,10 +71,9 @@ def run_cmd(args, out):
|
||||
ssh_host = args.ssh_host if args.ssh_host else args.config.mail_domain
|
||||
sshexec = get_sshexec(ssh_host)
|
||||
require_iroh = args.config.enable_iroh_relay
|
||||
if not args.dns_check_disabled:
|
||||
remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
|
||||
if not dns.check_initial_remote_data(remote_data, print=out.red):
|
||||
return 1
|
||||
remote_data = dns.get_initial_remote_data(sshexec, args.config.mail_domain)
|
||||
if not dns.check_initial_remote_data(remote_data, print=out.red):
|
||||
return 1
|
||||
|
||||
env = os.environ.copy()
|
||||
env["CHATMAIL_INI"] = args.inipath
|
||||
@@ -99,7 +83,7 @@ def run_cmd(args, out):
|
||||
pyinf = "pyinfra --dry" if args.dry_run else "pyinfra"
|
||||
|
||||
cmd = f"{pyinf} --ssh-user root {ssh_host} {deploy_path} -y"
|
||||
if ssh_host in ["localhost", "@docker"]:
|
||||
if ssh_host in ["localhost", "docker"]:
|
||||
cmd = f"{pyinf} @local {deploy_path} -y"
|
||||
|
||||
if version.parse(pyinfra.__version__) < version.parse("3"):
|
||||
@@ -109,15 +93,17 @@ def run_cmd(args, out):
|
||||
try:
|
||||
retcode = out.check_call(cmd, env=env)
|
||||
if retcode == 0:
|
||||
if not args.disable_mail:
|
||||
print("\nYou can try out the relay by talking to this echo bot: ")
|
||||
sshexec = SSHExec(args.config.mail_domain, verbose=args.verbose)
|
||||
print(
|
||||
sshexec(
|
||||
call=remote.rshell.shell,
|
||||
kwargs=dict(command="cat /var/lib/echobot/invite-link.txt"),
|
||||
)
|
||||
print("\nYou can try out the relay by talking to this echo bot: ")
|
||||
sshexec = SSHExec(args.config.mail_domain, verbose=args.verbose)
|
||||
print(
|
||||
sshexec(
|
||||
call=remote.rshell.shell,
|
||||
kwargs=dict(command="cat /var/lib/echobot/invite-link.txt"),
|
||||
)
|
||||
)
|
||||
server_deployed_message = f"Chatmail server started: https://{args.config.mail_domain}/"
|
||||
delimiter_line = "=" * len(server_deployed_message)
|
||||
out.green(f"{delimiter_line}\n{server_deployed_message}\n{delimiter_line}")
|
||||
out.green("Deploy completed, call `cmdeploy dns` next.")
|
||||
elif not remote_data["acme_account_url"]:
|
||||
out.red("Deploy completed but letsencrypt not configured")
|
||||
@@ -139,7 +125,11 @@ def dns_cmd_options(parser):
|
||||
default=None,
|
||||
help="write out a zonefile",
|
||||
)
|
||||
add_ssh_host_option(parser)
|
||||
parser.add_argument(
|
||||
"--ssh-host",
|
||||
dest="ssh_host",
|
||||
help="Run the DNS queries on 'localhost', in the chatmail 'docker' container, or on a specific SSH host",
|
||||
)
|
||||
|
||||
|
||||
def dns_cmd(args, out):
|
||||
@@ -299,15 +289,6 @@ class Out:
|
||||
return proc.returncode
|
||||
|
||||
|
||||
def add_ssh_host_option(parser):
|
||||
parser.add_argument(
|
||||
"--ssh-host",
|
||||
dest="ssh_host",
|
||||
help="Run commands on 'localhost', via '@docker', or on a specific SSH host "
|
||||
"instead of chatmail.ini's mail_domain.",
|
||||
)
|
||||
|
||||
|
||||
def add_config_option(parser):
|
||||
parser.add_argument(
|
||||
"--config",
|
||||
@@ -365,9 +346,9 @@ def get_parser():
|
||||
|
||||
def get_sshexec(ssh_host: str, verbose=True):
|
||||
if ssh_host in ["localhost", "@local"]:
|
||||
return LocalExec(verbose, docker=False)
|
||||
elif ssh_host == "@docker":
|
||||
return LocalExec(verbose, docker=True)
|
||||
return "localhost"
|
||||
elif ssh_host == "docker":
|
||||
return "docker"
|
||||
if verbose:
|
||||
print(f"[ssh] login to {ssh_host}")
|
||||
return SSHExec(ssh_host, verbose=verbose)
|
||||
|
||||
@@ -7,9 +7,15 @@ from . import remote
|
||||
|
||||
|
||||
def get_initial_remote_data(sshexec, mail_domain):
|
||||
return sshexec.logged(
|
||||
call=remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=mail_domain)
|
||||
)
|
||||
if sshexec == "localhost":
|
||||
result = remote.rdns.perform_initial_checks(mail_domain)
|
||||
elif sshexec == "docker":
|
||||
result = remote.rdns.perform_initial_checks(mail_domain, pre_command="docker exec chatmail ")
|
||||
else:
|
||||
result = sshexec.logged(
|
||||
call=remote.rdns.perform_initial_checks, kwargs=dict(mail_domain=mail_domain)
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def check_initial_remote_data(remote_data, *, print=print):
|
||||
@@ -44,9 +50,14 @@ def check_full_zone(sshexec, remote_data, out, zonefile) -> int:
|
||||
"""Check existing DNS records, optionally write them to zone file
|
||||
and return (exitcode, remote_data) tuple."""
|
||||
|
||||
required_diff, recommended_diff = sshexec.logged(
|
||||
remote.rdns.check_zonefile, kwargs=dict(zonefile=zonefile, verbose=False),
|
||||
)
|
||||
if sshexec in ["localhost", "docker"]:
|
||||
required_diff, recommended_diff = remote.rdns.check_zonefile(
|
||||
zonefile=zonefile, verbose=False
|
||||
)
|
||||
else:
|
||||
required_diff, recommended_diff = sshexec.logged(
|
||||
remote.rdns.check_zonefile, kwargs=dict(zonefile=zonefile, verbose=False),
|
||||
)
|
||||
|
||||
returncode = 0
|
||||
if required_diff:
|
||||
|
||||
@@ -70,12 +70,6 @@ userdb {
|
||||
# Mailboxes are stored in the "mail" directory of the vmail user home.
|
||||
mail_location = maildir:{{ config.mailboxes_dir }}/%u
|
||||
|
||||
# index/cache files are not very useful for chatmail relay operations
|
||||
# but it's not clear how to disable them completely.
|
||||
# According to https://doc.dovecot.org/2.3/settings/advanced/#core_setting-mail_cache_max_size
|
||||
# if the cache file becomes larger than the specified size, it is truncated by dovecot
|
||||
mail_cache_max_size = 500K
|
||||
|
||||
namespace inbox {
|
||||
inbox = yes
|
||||
|
||||
|
||||
14
cmdeploy/src/cmdeploy/dovecot/expunge.cron.j2
Normal file
14
cmdeploy/src/cmdeploy/dovecot/expunge.cron.j2
Normal file
@@ -0,0 +1,14 @@
|
||||
# delete already seen big mails after 7 days, in the INBOX
|
||||
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_large_after }} -size +200k -type f -delete
|
||||
# delete all mails after {{ config.delete_mails_after }} days, in the Inbox
|
||||
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
|
||||
# or in any IMAP subfolder
|
||||
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/cur/*' -mtime +{{ config.delete_mails_after }} -type f -delete
|
||||
# even if they are unseen
|
||||
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
|
||||
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/new/*' -mtime +{{ config.delete_mails_after }} -type f -delete
|
||||
# or only temporary (but then they shouldn't be around after {{ config.delete_mails_after }} days anyway).
|
||||
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
|
||||
2 0 * * * vmail find {{ config.mailboxes_dir }} -path '*/.*/tmp/*' -mtime +{{ config.delete_mails_after }} -type f -delete
|
||||
3 0 * * * vmail find {{ config.mailboxes_dir }} -name 'maildirsize' -type f -delete
|
||||
4 0 * * * vmail /usr/local/lib/chatmaild/venv/bin/delete_inactive_users /usr/local/lib/chatmaild/chatmail.ini
|
||||
@@ -1,11 +1,5 @@
|
||||
enable_relay = true
|
||||
http_bind_addr = "[::]:3340"
|
||||
|
||||
# Disable built-in STUN server in iroh-relay 0.35
|
||||
# as we deploy our own TURN server instead.
|
||||
# STUN server is going to be removed in iroh-relay 1.0
|
||||
# and this line can be removed after upgrade.
|
||||
enable_stun = false
|
||||
|
||||
enable_stun = true
|
||||
enable_metrics = false
|
||||
metrics_bind_addr = "127.0.0.1:9092"
|
||||
|
||||
@@ -66,7 +66,7 @@ http {
|
||||
|
||||
index index.html index.htm;
|
||||
|
||||
server_name {{ config.domain_name }} www.{{ config.domain_name }} mta-sts.{{ config.domain_name }};
|
||||
server_name _;
|
||||
|
||||
access_log syslog:server=unix:/dev/log,facility=local7;
|
||||
|
||||
|
||||
@@ -13,7 +13,6 @@ OversignHeaders From
|
||||
On-BadSignature reject
|
||||
On-KeyNotFound reject
|
||||
On-NoSignature reject
|
||||
DNSTimeout 60
|
||||
|
||||
# Signing domain, selector, and key (required). For example, perform signing
|
||||
# for domain "example.com" with selector "2020" (2020._domainkey.example.com),
|
||||
|
||||
@@ -26,7 +26,6 @@ smtp_tls_security_level=verify
|
||||
smtp_tls_servername = hostname
|
||||
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
|
||||
smtp_tls_policy_maps = inline:{nauta.cu=may}
|
||||
smtp_tls_protocols = >=TLSv1.2
|
||||
smtpd_tls_protocols = >=TLSv1.2
|
||||
|
||||
# Disable anonymous cipher suites
|
||||
|
||||
@@ -14,7 +14,6 @@ smtp inet n - y - - smtpd -v
|
||||
{%- else %}
|
||||
smtp inet n - y - - smtpd
|
||||
{%- endif %}
|
||||
-o smtpd_tls_security_level=encrypt
|
||||
-o smtpd_proxy_filter=127.0.0.1:{{ config.filtermail_smtp_port_incoming }}
|
||||
submission inet n - y - 5000 smtpd
|
||||
-o syslog_name=postfix/submission
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
[Unit]
|
||||
Description=chatmail mail storage expiration job
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=vmail
|
||||
ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-expire /usr/local/lib/chatmaild/chatmail.ini -v --remove
|
||||
|
||||
@@ -1,8 +0,0 @@
|
||||
[Unit]
|
||||
Description=Run Daily chatmail-expire job
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 00:02:00
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,9 +0,0 @@
|
||||
[Unit]
|
||||
Description=chatmail file system storage reporting job
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=vmail
|
||||
ExecStart=/usr/local/lib/chatmaild/venv/bin/chatmail-fsreport /usr/local/lib/chatmaild/chatmail.ini
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
[Unit]
|
||||
Description=Run Daily Chatmail fsreport Job
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 08:02:00
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,16 +0,0 @@
|
||||
[Unit]
|
||||
Description=A wrapper for the TURN server
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
Restart=always
|
||||
ExecStart=/usr/local/bin/chatmail-turn --realm {mail_domain} --socket /run/chatmail-turn/turn.socket
|
||||
|
||||
# Create /run/chatmail-turn
|
||||
RuntimeDirectory=chatmail-turn
|
||||
User=vmail
|
||||
Group=vmail
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -82,19 +82,3 @@ class SSHExec:
|
||||
res = self(call, kwargs, log_callback=remote.rshell.log_progress)
|
||||
print_stderr()
|
||||
return res
|
||||
|
||||
|
||||
class LocalExec:
|
||||
def __init__(self, verbose=False, docker=False):
|
||||
self.verbose = verbose
|
||||
self.docker = docker
|
||||
|
||||
def logged(self, call, kwargs: dict):
|
||||
where = "locally"
|
||||
if self.docker:
|
||||
if call == remote.rdns.perform_initial_checks:
|
||||
kwargs['pre_command'] = "docker exec chatmail "
|
||||
where = "in docker"
|
||||
if self.verbose:
|
||||
print(f"Running {where}: {call.__name__}(**{kwargs})")
|
||||
return call(**kwargs)
|
||||
|
||||
@@ -37,7 +37,7 @@ class TestDC:
|
||||
|
||||
def test_ping_pong(self, benchmark, cmfactory):
|
||||
ac1, ac2 = cmfactory.get_online_accounts(2)
|
||||
chat = cmfactory.get_accepted_chat(ac1, ac2)
|
||||
chat = cmfactory.get_protected_chat(ac1, ac2)
|
||||
|
||||
def dc_ping_pong():
|
||||
chat.send_text("ping")
|
||||
@@ -49,7 +49,7 @@ class TestDC:
|
||||
|
||||
def test_send_10_receive_10(self, benchmark, cmfactory, lp):
|
||||
ac1, ac2 = cmfactory.get_online_accounts(2)
|
||||
chat = cmfactory.get_accepted_chat(ac1, ac2)
|
||||
chat = cmfactory.get_protected_chat(ac1, ac2)
|
||||
|
||||
def dc_send_10_receive_10():
|
||||
for i in range(10):
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import queue
|
||||
import smtplib
|
||||
import socket
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
@@ -91,23 +91,25 @@ def test_concurrent_logins_same_account(
|
||||
|
||||
def test_no_vrfy(chatmail_config):
|
||||
domain = chatmail_config.mail_domain
|
||||
|
||||
s = smtplib.SMTP(domain)
|
||||
s.starttls()
|
||||
|
||||
s.putcmd("vrfy", f"wrongaddress@{chatmail_config.mail_domain}")
|
||||
result = s.getreply()
|
||||
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
sock.settimeout(10)
|
||||
try:
|
||||
sock.connect((domain, 25))
|
||||
except socket.timeout:
|
||||
pytest.skip(f"port 25 not reachable for {domain}")
|
||||
banner = sock.recv(1024)
|
||||
print(banner)
|
||||
sock.send(b"VRFY wrongaddress@%s\r\n" % (chatmail_config.mail_domain.encode(),))
|
||||
result = sock.recv(1024)
|
||||
print(result)
|
||||
s.putcmd("vrfy", f"echo@{chatmail_config.mail_domain}")
|
||||
result2 = s.getreply()
|
||||
sock.send(b"VRFY echo@%s\r\n" % (chatmail_config.mail_domain.encode(),))
|
||||
result2 = sock.recv(1024)
|
||||
print(result2)
|
||||
assert result[0] == result2[0] == 252
|
||||
assert result[1][0:6] == result2[1][0:6] == b"2.0.0 "
|
||||
s.putcmd("vrfy", "wrongaddress")
|
||||
result = s.getreply()
|
||||
assert result[0:10] == result2[0:10]
|
||||
sock.send(b"VRFY wrongaddress\r\n")
|
||||
result = sock.recv(1024)
|
||||
print(result)
|
||||
s.putcmd("vrfy", "echo")
|
||||
result2 = s.getreply()
|
||||
sock.send(b"VRFY echo\r\n")
|
||||
result2 = sock.recv(1024)
|
||||
print(result2)
|
||||
assert result[0] == result2[0] == 252
|
||||
assert result[1][0:6] == result2[1][0:6] == b"2.0.0 "
|
||||
assert result[0:10] == result2[0:10] == b"252 2.0.0 "
|
||||
|
||||
@@ -2,7 +2,6 @@ import datetime
|
||||
import smtplib
|
||||
import socket
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -32,7 +31,6 @@ class TestSSHExecutor:
|
||||
)
|
||||
out, err = capsys.readouterr()
|
||||
assert err.startswith("Collecting")
|
||||
# XXX could not figure out how capturing can be made to work properly
|
||||
#assert err.endswith("....\n")
|
||||
assert err.count("\n") == 1
|
||||
|
||||
@@ -42,7 +40,6 @@ class TestSSHExecutor:
|
||||
)
|
||||
out, err = capsys.readouterr()
|
||||
lines = err.split("\n")
|
||||
# XXX could not figure out how capturing can be made to work properly
|
||||
#assert len(lines) > 4
|
||||
assert remote.rdns.perform_initial_checks.__doc__ in lines[0]
|
||||
|
||||
@@ -72,7 +69,7 @@ def test_timezone_env(remote):
|
||||
for line in remote.iter_output("env"):
|
||||
print(line)
|
||||
if line == "tz=:/etc/localtime":
|
||||
return
|
||||
return True
|
||||
pytest.fail("TZ is not set")
|
||||
|
||||
|
||||
@@ -143,23 +140,12 @@ def test_reject_missing_dkim(cmsetup, maildata, from_addr):
|
||||
"encrypted.eml", from_addr=from_addr, to_addr=recipient.addr
|
||||
).as_string()
|
||||
conn = smtplib.SMTP(cmsetup.maildomain, 25, timeout=10)
|
||||
conn.starttls()
|
||||
|
||||
with conn as s:
|
||||
with pytest.raises(smtplib.SMTPDataError, match="No valid DKIM signature"):
|
||||
s.sendmail(from_addr=from_addr, to_addrs=recipient.addr, msg=msg)
|
||||
|
||||
|
||||
def try_n_times(n, f):
|
||||
for _ in range(n - 1):
|
||||
try:
|
||||
return f()
|
||||
except Exception:
|
||||
time.sleep(1)
|
||||
|
||||
return f()
|
||||
|
||||
|
||||
def test_rewrite_subject(cmsetup, maildata):
|
||||
"""Test that subject gets replaced with [...]."""
|
||||
user1, user2 = cmsetup.gen_users(2)
|
||||
@@ -172,8 +158,7 @@ def test_rewrite_subject(cmsetup, maildata):
|
||||
).as_string()
|
||||
user1.smtp.sendmail(from_addr=user1.addr, to_addrs=[user2.addr], msg=sent_msg)
|
||||
|
||||
# The message may need some time to get delivered by postfix.
|
||||
messages = try_n_times(5, user2.imap.fetch_all_messages)
|
||||
messages = user2.imap.fetch_all_messages()
|
||||
assert len(messages) == 1
|
||||
rcvd_msg = messages[0]
|
||||
assert "Subject: [...]" not in sent_msg
|
||||
@@ -224,14 +209,8 @@ def test_expunged(remote, chatmail_config):
|
||||
|
||||
|
||||
def test_deployed_state(remote):
|
||||
try:
|
||||
git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
|
||||
except Exception:
|
||||
git_hash = "unknown\n"
|
||||
try:
|
||||
git_diff = subprocess.check_output(["git", "diff"]).decode()
|
||||
except Exception:
|
||||
git_diff = ""
|
||||
git_hash = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode()
|
||||
git_diff = subprocess.check_output(["git", "diff"]).decode()
|
||||
git_status = [git_hash.strip()]
|
||||
for line in git_diff.splitlines():
|
||||
git_status.append(line.strip().lower())
|
||||
|
||||
@@ -56,7 +56,7 @@ class TestEndToEndDeltaChat:
|
||||
"""Test that a DC account can send a message to a second DC account
|
||||
on the same chat-mail instance."""
|
||||
ac1, ac2 = cmfactory.get_online_accounts(2)
|
||||
chat = cmfactory.get_accepted_chat(ac1, ac2)
|
||||
chat = cmfactory.get_protected_chat(ac1, ac2)
|
||||
chat.send_text("message0")
|
||||
|
||||
lp.sec("wait for ac2 to receive message")
|
||||
@@ -70,7 +70,7 @@ class TestEndToEndDeltaChat:
|
||||
before quota is exceeded, and thus depends on the speed of the upload.
|
||||
"""
|
||||
ac1, ac2 = cmfactory.get_online_accounts(2)
|
||||
chat = cmfactory.get_accepted_chat(ac1, ac2)
|
||||
chat = cmfactory.get_protected_chat(ac1, ac2)
|
||||
|
||||
user = ac2.get_config("configured_addr")
|
||||
|
||||
@@ -153,7 +153,7 @@ def test_hide_senders_ip_address(cmfactory):
|
||||
assert ipaddress.ip_address(public_ip)
|
||||
|
||||
user1, user2 = cmfactory.get_online_accounts(2)
|
||||
chat = cmfactory.get_accepted_chat(user1, user2)
|
||||
chat = cmfactory.get_protected_chat(user1, user2)
|
||||
|
||||
chat.send_text("testing submission header cleanup")
|
||||
user2._evtracker.wait_next_incoming_message()
|
||||
|
||||
@@ -26,15 +26,10 @@ class TestCmdline:
|
||||
def test_init_not_overwrite(self, capsys):
|
||||
assert main(["init", "chat.example.org"]) == 0
|
||||
capsys.readouterr()
|
||||
|
||||
assert main(["init", "chat.example.org"]) == 1
|
||||
out, err = capsys.readouterr()
|
||||
assert "path exists" in out.lower()
|
||||
|
||||
assert main(["init", "chat.example.org", "--force"]) == 0
|
||||
out, err = capsys.readouterr()
|
||||
assert "deleting config file" in out.lower()
|
||||
|
||||
|
||||
def test_www_folder(example_config, tmp_path):
|
||||
reporoot = importlib.resources.files(__package__).joinpath("../../../../").resolve()
|
||||
|
||||
@@ -4,7 +4,6 @@ import time
|
||||
import traceback
|
||||
import webbrowser
|
||||
from pathlib import Path
|
||||
import re
|
||||
|
||||
import markdown
|
||||
from chatmaild.config import read_config
|
||||
@@ -13,9 +12,6 @@ from jinja2 import Template
|
||||
from .genqr import gen_qr_png_data
|
||||
|
||||
|
||||
_MERGE_CONFLICT_RE = re.compile(r"^<<<<<<<.+^=======.+^>>>>>>>", re.DOTALL | re.MULTILINE)
|
||||
|
||||
|
||||
def snapshot_dir_stats(somedir):
|
||||
d = {}
|
||||
for path in somedir.iterdir():
|
||||
@@ -120,17 +116,6 @@ def _build_webpages(src_dir, build_dir, config):
|
||||
return build_dir
|
||||
|
||||
|
||||
def find_merge_conflict(src_dir) -> Path:
|
||||
assert src_dir.exists(), src_dir
|
||||
result = None
|
||||
for path in src_dir.iterdir():
|
||||
if path.suffix in [".css", ".html", ".md"]:
|
||||
if _MERGE_CONFLICT_RE.search(path.read_text()):
|
||||
result = path
|
||||
break
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
path = importlib.resources.files(__package__)
|
||||
reporoot = path.joinpath("../../../").resolve()
|
||||
|
||||
83
docker/chatmail_relay.dockerfile
Normal file
83
docker/chatmail_relay.dockerfile
Normal file
@@ -0,0 +1,83 @@
|
||||
FROM jrei/systemd-debian:12 AS base
|
||||
|
||||
ENV LANG=en_US.UTF-8
|
||||
|
||||
RUN echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/01norecommend && \
|
||||
echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/01norecommend && \
|
||||
apt-get update && \
|
||||
apt-get install -y \
|
||||
ca-certificates && \
|
||||
DEBIAN_FRONTEND=noninteractive \
|
||||
TZ=Europe/London \
|
||||
apt-get install -y tzdata && \
|
||||
apt-get install -y locales && \
|
||||
sed -i -e "s/# $LANG.*/$LANG UTF-8/" /etc/locale.gen && \
|
||||
dpkg-reconfigure --frontend=noninteractive locales && \
|
||||
update-locale LANG=$LANG \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN apt-get update && \
|
||||
apt-get install -y \
|
||||
git \
|
||||
python3 \
|
||||
python3-venv \
|
||||
python3-virtualenv \
|
||||
gcc \
|
||||
python3-dev \
|
||||
opendkim \
|
||||
opendkim-tools \
|
||||
curl \
|
||||
rsync \
|
||||
unbound \
|
||||
unbound-anchor \
|
||||
dnsutils \
|
||||
postfix \
|
||||
acl \
|
||||
nginx \
|
||||
libnginx-mod-stream \
|
||||
fcgiwrap \
|
||||
cron \
|
||||
&& for pkg in core imapd lmtpd; do \
|
||||
case "$pkg" in \
|
||||
core) sha256="43f593332e22ac7701c62d58b575d2ca409e0f64857a2803be886c22860f5587" ;; \
|
||||
imapd) sha256="8d8dc6fc00bbb6cdb25d345844f41ce2f1c53f764b79a838eb2a03103eebfa86" ;; \
|
||||
lmtpd) sha256="2f69ba5e35363de50962d42cccbfe4ed8495265044e244007d7ccddad77513ab" ;; \
|
||||
esac; \
|
||||
url="https://download.delta.chat/dovecot/dovecot-${pkg}_2.3.21%2Bdfsg1-3_amd64.deb"; \
|
||||
file="/tmp/$(basename "$url")"; \
|
||||
curl -fsSL "$url" -o "$file"; \
|
||||
echo "$sha256 $file" | sha256sum -c -; \
|
||||
apt-get install -y "$file"; \
|
||||
rm -f "$file"; \
|
||||
done \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /opt/chatmail
|
||||
|
||||
ARG SETUP_CHATMAIL_SERVICE_PATH=/lib/systemd/system/setup_chatmail.service
|
||||
COPY ./files/setup_chatmail.service "$SETUP_CHATMAIL_SERVICE_PATH"
|
||||
RUN ln -sf "$SETUP_CHATMAIL_SERVICE_PATH" "/etc/systemd/system/multi-user.target.wants/setup_chatmail.service"
|
||||
|
||||
COPY --chmod=555 ./files/setup_chatmail_docker.sh /setup_chatmail_docker.sh
|
||||
COPY --chmod=555 ./files/update_ini.sh /update_ini.sh
|
||||
COPY --chmod=555 ./files/entrypoint.sh /entrypoint.sh
|
||||
|
||||
## TODO: add git clone.
|
||||
## Problem: how correct save only required files inside container....
|
||||
# RUN git clone https://github.com/chatmail/relay.git -b master . \
|
||||
# && ./scripts/initenv.sh
|
||||
|
||||
# EXPOSE 443 25 587 143 993
|
||||
|
||||
VOLUME ["/sys/fs/cgroup", "/home"]
|
||||
|
||||
STOPSIGNAL SIGRTMIN+3
|
||||
|
||||
ENTRYPOINT ["/entrypoint.sh"]
|
||||
|
||||
CMD [ "--default-standard-output=journal+console", \
|
||||
"--default-standard-error=journal+console" ]
|
||||
|
||||
## TODO: Add installation and configuration of chatmaild inside the Dockerfile.
|
||||
## This is required to ensure repeatable deployment.
|
||||
## In the current MVP, the chatmaild server is updated on every container restart.
|
||||
59
docker/docker-compose-default.yaml
Normal file
59
docker/docker-compose-default.yaml
Normal file
@@ -0,0 +1,59 @@
|
||||
services:
|
||||
chatmail:
|
||||
build:
|
||||
context: ./docker
|
||||
dockerfile: chatmail_relay.dockerfile
|
||||
tags:
|
||||
- chatmail-relay:latest
|
||||
image: chatmail-relay:latest
|
||||
restart: unless-stopped
|
||||
container_name: chatmail
|
||||
cgroup: host # required for systemd
|
||||
tty: true # required for logs
|
||||
tmpfs: # required for systemd
|
||||
- /tmp
|
||||
- /run
|
||||
- /run/lock
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
environment:
|
||||
MAIL_DOMAIN: $MAIL_DOMAIN
|
||||
CHANGE_KERNEL_SETTINGS: "False"
|
||||
ACME_EMAIL: $ACME_EMAIL
|
||||
# RECREATE_VENV: "false"
|
||||
# MAX_MESSAGE_SIZE: "50M"
|
||||
# DEBUG_COMMANDS_ENABLED: "true"
|
||||
# FORCE_REINIT_INI_FILE: "true"
|
||||
# USE_FOREIGN_CERT_MANAGER: "True"
|
||||
# ENABLE_CERTS_MONITORING: "true"
|
||||
# CERTS_MONITORING_TIMEOUT: 10
|
||||
# IS_DEVELOPMENT_INSTANCE: "True"
|
||||
ports:
|
||||
- "80:80"
|
||||
- "443:443"
|
||||
- "25:25"
|
||||
- "587:587"
|
||||
- "143:143"
|
||||
- "465:465"
|
||||
- "993:993"
|
||||
volumes:
|
||||
## system
|
||||
- /sys/fs/cgroup:/sys/fs/cgroup:rw # required for systemd
|
||||
- ./:/opt/chatmail
|
||||
|
||||
## data
|
||||
- ./data/chatmail:/home
|
||||
- ./data/chatmail-dkimkeys:/etc/dkimkeys
|
||||
- ./data/chatmail-echobot:/run/echobot
|
||||
- ./data/chatmail-acme:/var/lib/acme
|
||||
|
||||
## custom resources
|
||||
# - ./custom/www/src/index.md:/opt/chatmail/www/src/index.md
|
||||
|
||||
## debug
|
||||
# - ./docker/files/setup_chatmail_docker.sh:/setup_chatmail_docker.sh
|
||||
# - ./docker/files/entrypoint.sh:/entrypoint.sh
|
||||
# - ./docker/files/update_ini.sh:/update_ini.sh
|
||||
136
docker/docker-compose-traefik.yaml
Normal file
136
docker/docker-compose-traefik.yaml
Normal file
@@ -0,0 +1,136 @@
|
||||
services:
|
||||
chatmail:
|
||||
build:
|
||||
context: ./docker
|
||||
dockerfile: chatmail_relay.dockerfile
|
||||
tags:
|
||||
- chatmail-relay:latest
|
||||
image: chatmail-relay:latest
|
||||
restart: unless-stopped
|
||||
container_name: chatmail
|
||||
depends_on:
|
||||
- traefik-certs-dumper
|
||||
cgroup: host # required for systemd
|
||||
tty: true # required for logs
|
||||
tmpfs: # required for systemd
|
||||
- /tmp
|
||||
- /run
|
||||
- /run/lock
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
environment: #all possible variables you can check inside README and /chatmaild/src/chatmaild/ini/chatmail.ini.f
|
||||
MAIL_DOMAIN: $MAIL_DOMAIN
|
||||
# MAX_MESSAGE_SIZE: "50M"
|
||||
# DEBUG_COMMANDS_ENABLED: "true"
|
||||
# FORCE_REINIT_INI_FILE: "true"
|
||||
# RECREATE_VENV: "false"
|
||||
USE_FOREIGN_CERT_MANAGER: "true"
|
||||
CHANGE_KERNEL_SETTINGS: "false"
|
||||
PATH_TO_SSL: "${CERTS_ROOT_DIR_CONTAINER}/${MAIL_DOMAIN}"
|
||||
ENABLE_CERTS_MONITORING: "true"
|
||||
# CERTS_MONITORING_TIMEOUT: 60
|
||||
# IS_DEVELOPMENT_INSTANCE: "true"
|
||||
ports:
|
||||
- "25:25"
|
||||
- "587:587"
|
||||
- "143:143"
|
||||
- "465:465"
|
||||
- "993:993"
|
||||
volumes:
|
||||
## system
|
||||
- /sys/fs/cgroup:/sys/fs/cgroup:rw # required for systemd
|
||||
- ./:/opt/chatmail
|
||||
- ${CERTS_ROOT_DIR_HOST}:${CERTS_ROOT_DIR_CONTAINER}:ro
|
||||
|
||||
## data
|
||||
- ./data/chatmail:/home
|
||||
# - ./data/chatmail-dkimkeys:/etc/dkimkeys
|
||||
# - ./data/chatmail-echobot:/run/echobot
|
||||
# - ./data/chatmail-acme:/var/lib/acme
|
||||
|
||||
## custom resources
|
||||
# - ./custom/www/src/index.md:/opt/chatmail/www/src/index.md
|
||||
|
||||
## debug
|
||||
# - ./docker/files/setup_chatmail_docker.sh:/setup_chatmail_docker.sh
|
||||
# - ./docker/files/entrypoint.sh:/entrypoint.sh
|
||||
# - ./docker/files/update_ini.sh:/update_ini.sh
|
||||
|
||||
labels:
|
||||
- traefik.enable=true
|
||||
- traefik.http.services.chatmail-relay.loadbalancer.server.scheme=https
|
||||
- traefik.http.services.chatmail-relay.loadbalancer.server.port=443
|
||||
- traefik.http.services.chatmail-relay.loadbalancer.serverstransport=insecure@file
|
||||
- traefik.http.routers.chatmail-relay.rule=Host(`${MAIL_DOMAIN}`) || Host(`mta-sts.${MAIL_DOMAIN}`) || Host(`www.${MAIL_DOMAIN}`)
|
||||
- traefik.http.routers.chatmail-relay.service=chatmail-relay
|
||||
- traefik.http.routers.chatmail-relay.tls=true
|
||||
- traefik.http.routers.chatmail-relay.tls.certresolver=letsEncrypt
|
||||
|
||||
traefik_init:
|
||||
image: alpine:latest
|
||||
restart: on-failure
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
working_dir: /app
|
||||
entrypoint: sh -c '
|
||||
touch acme.json &&
|
||||
chown 0:0 ./acme.json &&
|
||||
chmod 600 ./acme.json'
|
||||
volumes:
|
||||
- ./traefik/data:/app
|
||||
|
||||
traefik:
|
||||
image: traefik:v3.3
|
||||
container_name: traefik
|
||||
restart: unless-stopped
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
command:
|
||||
- "--configFile=/config.yaml"
|
||||
- "--certificatesresolvers.letsEncrypt.acme.email=${ACME_EMAIL}"
|
||||
# ports:
|
||||
# - "80:80"
|
||||
# - "443:443"
|
||||
network_mode: host
|
||||
depends_on:
|
||||
traefik_init:
|
||||
condition: service_completed_successfully
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- ./traefik/config.yaml:/config.yaml
|
||||
- ./traefik/data/acme.json:/acme.json
|
||||
- ./traefik/dynamic-configs:/dynamic/conf
|
||||
|
||||
traefik-certs-dumper:
|
||||
image: ldez/traefik-certs-dumper:v2.10.0
|
||||
restart: unless-stopped
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
depends_on:
|
||||
- traefik
|
||||
entrypoint: sh -c '
|
||||
apk add openssl &&
|
||||
while ! [ -e /data/acme.json ]
|
||||
|| ! [ `jq ".[] | .Certificates | length" /data/acme.json | jq -s "add" ` != 0 ]; do
|
||||
sleep 1
|
||||
; done
|
||||
&& traefik-certs-dumper file --version v3 --watch --domain-subdir=true
|
||||
--source /data/acme.json --dest /data/letsencrypt/certs --post-hook "sh /post-hook.sh"'
|
||||
environment:
|
||||
CERTS_DIR: /data/letsencrypt/certs
|
||||
volumes:
|
||||
- ./traefik/data/letsencrypt:/data/letsencrypt
|
||||
- ./traefik/data/acme.json:/data/acme.json
|
||||
- ./traefik/post-hook.sh:/post-hook.sh
|
||||
5
docker/example.env
Normal file
5
docker/example.env
Normal file
@@ -0,0 +1,5 @@
|
||||
MAIL_DOMAIN="chat.example.com"
|
||||
ACME_EMAIL="my.email@gmail.com"
|
||||
|
||||
CERTS_ROOT_DIR_HOST="./traefik/data/letsencrypt/certs"
|
||||
CERTS_ROOT_DIR_CONTAINER="/var/lib/acme/live"
|
||||
24
docker/files/entrypoint.sh
Executable file
24
docker/files/entrypoint.sh
Executable file
@@ -0,0 +1,24 @@
|
||||
#!/bin/bash
|
||||
set -eo pipefail
|
||||
|
||||
unlink /etc/nginx/sites-enabled/default || true
|
||||
|
||||
if [ "${USE_FOREIGN_CERT_MANAGER,,}" == "true" ]; then
|
||||
if [ ! -f "$PATH_TO_SSL/fullchain" ]; then
|
||||
echo "Error: file '$PATH_TO_SSL/fullchain' does not exist. Exiting..." > /dev/stderr
|
||||
sleep 2
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -f "$PATH_TO_SSL/privkey" ]; then
|
||||
echo "Error: file '$PATH_TO_SSL/privkey' does not exist. Exiting..." > /dev/stderr
|
||||
sleep 2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
SETUP_CHATMAIL_SERVICE_PATH="${SETUP_CHATMAIL_SERVICE_PATH:-/lib/systemd/system/setup_chatmail.service}"
|
||||
|
||||
env_vars=$(printenv | cut -d= -f1 | xargs)
|
||||
sed -i "s|<envs_list>|$env_vars|g" $SETUP_CHATMAIL_SERVICE_PATH
|
||||
|
||||
exec /lib/systemd/systemd $@
|
||||
14
docker/files/setup_chatmail.service
Normal file
14
docker/files/setup_chatmail.service
Normal file
@@ -0,0 +1,14 @@
|
||||
[Unit]
|
||||
Description=Run container setup commands
|
||||
After=multi-user.target
|
||||
ConditionPathExists=/setup_chatmail_docker.sh
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/bash /setup_chatmail_docker.sh
|
||||
RemainAfterExit=true
|
||||
WorkingDirectory=/opt/chatmail
|
||||
PassEnvironment=<envs_list>
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
78
docker/files/setup_chatmail_docker.sh
Executable file
78
docker/files/setup_chatmail_docker.sh
Executable file
@@ -0,0 +1,78 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -eo pipefail
|
||||
export INI_FILE="${INI_FILE:-chatmail.ini}"
|
||||
export ENABLE_CERTS_MONITORING="${ENABLE_CERTS_MONITORING:-true}"
|
||||
export CERTS_MONITORING_TIMEOUT="${CERTS_MONITORING_TIMEOUT:-60}"
|
||||
export PATH_TO_SSL="${PATH_TO_SSL:-/var/lib/acme/live/${MAIL_DOMAIN}}"
|
||||
export CHANGE_KERNEL_SETTINGS=${CHANGE_KERNEL_SETTINGS:-"False"}
|
||||
export RECREATE_VENV=${RECREATE_VENV:-"false"}
|
||||
|
||||
if [ -z "$MAIL_DOMAIN" ]; then
|
||||
echo "ERROR: Environment variable 'MAIL_DOMAIN' must be set!" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
debug_commands() {
|
||||
echo "Executing debug commands"
|
||||
# git config --global --add safe.directory /opt/chatmail
|
||||
# ./scripts/initenv.sh
|
||||
}
|
||||
|
||||
calculate_hash() {
|
||||
find "$PATH_TO_SSL" -type f -exec sha1sum {} \; | sort | sha1sum | awk '{print $1}'
|
||||
}
|
||||
|
||||
monitor_certificates() {
|
||||
if [ "$ENABLE_CERTS_MONITORING" != "true" ]; then
|
||||
echo "Certs monitoring disabled."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
current_hash=$(calculate_hash)
|
||||
previous_hash=$current_hash
|
||||
|
||||
while true; do
|
||||
current_hash=$(calculate_hash)
|
||||
if [[ "$current_hash" != "$previous_hash" ]]; then
|
||||
# TODO: add an option to restart at a specific time interval
|
||||
echo "[INFO] Certificate's folder hash was changed, reloading nginx, dovecot and postfix services."
|
||||
systemctl reload nginx.service
|
||||
systemctl reload dovecot.service
|
||||
systemctl reload postfix.service
|
||||
previous_hash=$current_hash
|
||||
fi
|
||||
sleep $CERTS_MONITORING_TIMEOUT
|
||||
done
|
||||
}
|
||||
|
||||
### MAIN
|
||||
|
||||
if [ "$DEBUG_COMMANDS_ENABLED" == "true" ]; then
|
||||
debug_commands
|
||||
fi
|
||||
|
||||
if [ "$FORCE_REINIT_INI_FILE" == "true" ]; then
|
||||
INI_CMD_ARGS=--force
|
||||
fi
|
||||
|
||||
/usr/sbin/opendkim-genkey -D /etc/dkimkeys -d $MAIL_DOMAIN -s opendkim
|
||||
chown opendkim:opendkim /etc/dkimkeys/opendkim.private
|
||||
chown opendkim:opendkim /etc/dkimkeys/opendkim.txt
|
||||
|
||||
# TODO: Move to debug_commands after git clone is moved to dockerfile.
|
||||
git config --global --add safe.directory /opt/chatmail
|
||||
if [ "$RECREATE_VENV" == "true" ]; then
|
||||
rm -rf venv
|
||||
fi
|
||||
./scripts/initenv.sh
|
||||
|
||||
./scripts/cmdeploy init --config "${INI_FILE}" $INI_CMD_ARGS $MAIL_DOMAIN
|
||||
bash /update_ini.sh
|
||||
|
||||
./scripts/cmdeploy run --ssh-host docker
|
||||
|
||||
echo "ForwardToConsole=yes" >> /etc/systemd/journald.conf
|
||||
systemctl restart systemd-journald
|
||||
|
||||
monitor_certificates &
|
||||
79
docker/files/update_ini.sh
Normal file
79
docker/files/update_ini.sh
Normal file
@@ -0,0 +1,79 @@
|
||||
#!/bin/bash
|
||||
set -eo pipefail
|
||||
|
||||
INI_FILE="${INI_FILE:-chatmail.ini}"
|
||||
|
||||
if [ ! -f "$INI_FILE" ]; then
|
||||
echo "Error: file $INI_FILE not found." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TMP_FILE="$(mktemp)"
|
||||
|
||||
convert_to_bytes() {
|
||||
local value="$1"
|
||||
if [[ "$value" =~ ^([0-9]+)([KkMmGgTt])$ ]]; then
|
||||
local num="${BASH_REMATCH[1]}"
|
||||
local unit="${BASH_REMATCH[2]}"
|
||||
case "$unit" in
|
||||
[Kk]) echo $((num * 1024)) ;;
|
||||
[Mm]) echo $((num * 1024 * 1024)) ;;
|
||||
[Gg]) echo $((num * 1024 * 1024 * 1024)) ;;
|
||||
[Tt]) echo $((num * 1024 * 1024 * 1024 * 1024)) ;;
|
||||
esac
|
||||
elif [[ "$value" =~ ^[0-9]+$ ]]; then
|
||||
echo "$value"
|
||||
else
|
||||
echo "Error: incorrect size format: $value." >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
process_specific_params() {
|
||||
local key=$1
|
||||
local value=$2
|
||||
local destination_file=$3
|
||||
|
||||
if [[ "$key" == "max_message_size" ]]; then
|
||||
converted=$(convert_to_bytes "$value") || exit 1
|
||||
if grep -q -e "## .* = .* bytes" "$destination_file"; then
|
||||
sed "s|## .* = .* bytes|## $value = $converted bytes|g" "$destination_file";
|
||||
else
|
||||
echo "## $value = $converted bytes" >> "$destination_file"
|
||||
fi
|
||||
echo "$key = $converted" >> "$destination_file"
|
||||
else
|
||||
echo "$key = $value" >> "$destination_file"
|
||||
fi
|
||||
}
|
||||
|
||||
while IFS= read -r line; do
|
||||
if [[ "$line" =~ ^[[:space:]]*#.* || "$line" =~ ^[[:space:]]*$ ]]; then
|
||||
echo "$line" >> "$TMP_FILE"
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$line" =~ ^([a-z0-9_]+)[[:space:]]*=[[:space:]]*(.*)$ ]]; then
|
||||
key="${BASH_REMATCH[1]}"
|
||||
current_value="${BASH_REMATCH[2]}"
|
||||
env_var_name=$(echo "$key" | tr 'a-z' 'A-Z')
|
||||
env_value="${!env_var_name}"
|
||||
|
||||
if [[ -n "$env_value" ]]; then
|
||||
process_specific_params "$key" "$env_value" "$TMP_FILE"
|
||||
else
|
||||
echo "$line" >> "$TMP_FILE"
|
||||
fi
|
||||
else
|
||||
echo "$line" >> "$TMP_FILE"
|
||||
fi
|
||||
done < "$INI_FILE"
|
||||
|
||||
PERMS=$(stat -c %a "$INI_FILE")
|
||||
OWNER=$(stat -c %u "$INI_FILE")
|
||||
GROUP=$(stat -c %g "$INI_FILE")
|
||||
|
||||
chmod "$PERMS" "$TMP_FILE"
|
||||
chown "$OWNER":"$GROUP" "$TMP_FILE"
|
||||
|
||||
mv "$TMP_FILE" "$INI_FILE"
|
||||
216
docs/DOCKER_INSTALLATION_EN.md
Normal file
216
docs/DOCKER_INSTALLATION_EN.md
Normal file
@@ -0,0 +1,216 @@
|
||||
# Known issues and limitations
|
||||
|
||||
- Chatmail will be reinstalled every time the container is started (longer the first time, faster on subsequent starts). This is how the original installer works because it wasn’t designed for Docker. At the end of the documentation, there’s a [proposed solution](#locking-the-chatmail-version).
|
||||
- Requires cgroups v2 configured in the system. Operation with cgroups v1 has not been tested.
|
||||
- Yes, of course, using systemd inside a container is a hack, and it would be better to split it into several services, but since this is an MVP, it turned out to be easier to do it this way initially than to rewrite the entire deployment system.
|
||||
- The Docker image is only suitable for amd64. If you need to run it on a different architecture, try modifying the Dockerfile (specifically the part responsible for installing dovecot).
|
||||
|
||||
# Docker installation
|
||||
This section provides instructions for installing Chatmail using docker-compose.
|
||||
|
||||
## Preliminary setup
|
||||
We use `chat.example.org` as the Chatmail domain in the following steps.
|
||||
Please substitute it with your own domain.
|
||||
|
||||
1. Setup the initial DNS records.
|
||||
The following is an example in the familiar BIND zone file format with
|
||||
a TTL of 1 hour (3600 seconds).
|
||||
Please substitute your domain and IP addresses.
|
||||
|
||||
```
|
||||
chat.example.com. 3600 IN A 198.51.100.5
|
||||
chat.example.com. 3600 IN AAAA 2001:db8::5
|
||||
www.chat.example.com. 3600 IN CNAME chat.example.com.
|
||||
mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
|
||||
```
|
||||
|
||||
2. clone the repository on your server.
|
||||
|
||||
```shell
|
||||
git clone https://github.com/chatmail/relay
|
||||
cd relay
|
||||
```
|
||||
|
||||
## Installation
|
||||
When installing via Docker, there are several options:
|
||||
|
||||
- Use the built-in nginx and acmetool in Chatmail container to host the chat and manage certificates.
|
||||
- Use third-party tools for certificate management.
|
||||
|
||||
For the third-party certificate manager example, traefik will be used, but you can use whatever is more convenient for you.
|
||||
|
||||
1. Copy the file `./docker/docker-compose-default.yaml` or `./docker/docker-compose-traefik.yaml` and rename it to `docker-compose.yaml`. This is necessary because `docker-compose.yaml` is in `.gitignore` and won’t cause conflicts when updating the git repository.
|
||||
|
||||
```shell
|
||||
cp ./docker/docker-compose-default.yaml docker-compose.yaml
|
||||
## or
|
||||
# cp ./docker/docker-compose-traefik.yaml docker-compose.yaml
|
||||
```
|
||||
|
||||
2. Copy `./docker/example.env` and rename it to `.env`. This file stores variables used in `docker-compose.yaml`.
|
||||
|
||||
```shell
|
||||
cp ./docker/example.env .env
|
||||
```
|
||||
|
||||
3. Configure environment variables in the `.env` file. These variables are used in the `docker-compose.yaml` file to pass repeated values.
|
||||
|
||||
4. Configure kernel parameters because they cannot be changed inside the container, specifically `fs.inotify.max_user_instances` and `fs.inotify.max_user_watches`. Run the following:
|
||||
|
||||
```shell
|
||||
echo "fs.inotify.max_user_instances=65536" | sudo tee -a /etc/sysctl.d/99-inotify.conf
|
||||
echo "fs.inotify.max_user_watches=65536" | sudo tee -a /etc/sysctl.d/99-inotify.conf
|
||||
sudo sysctl --system
|
||||
```
|
||||
|
||||
5. Configure container environment variables. Below is the list of variables used during deployment:
|
||||
|
||||
- `MAIL_DOMAIN` – The domain name of the future server. (required)
|
||||
- `DEBUG_COMMANDS_ENABLED` – Run debug commands before installation. (default: `false`)
|
||||
- `FORCE_REINIT_INI_FILE` – Recreate the ini configuration file on startup. (default: `false`)
|
||||
- `USE_FOREIGN_CERT_MANAGER` – Use a third-party certificate manager. (default: `false`)
|
||||
- `RECREATE_VENV` - Recreate the virtual environment (venv). If set to `true`, the environment will be recreated when the container starts, which will increase the startup time of the service but can help avoid certain errors. (default: `false`)
|
||||
- `INI_FILE` – Path to the ini configuration file. (default: `./chatmail.ini`)
|
||||
- `PATH_TO_SSL` – Path to where the certificates are stored. (default: `/var/lib/acme/live/${MAIL_DOMAIN}`)
|
||||
- `ENABLE_CERTS_MONITORING` – Enable certificate monitoring if `USE_FOREIGN_CERT_MANAGER=true`. If certificates change, services will be automatically restarted. (default: `false`)
|
||||
- `CERTS_MONITORING_TIMEOUT` – Interval in seconds to check if certificates have changed. (default: `'60'`)
|
||||
|
||||
You can also use any variables from the [ini configuration file](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/ini/chatmail.ini.f); they must be in uppercase.
|
||||
|
||||
Mandatory variables for deployment via Docker:
|
||||
|
||||
- `CHANGE_KERNEL_SETTINGS` – Change kernel settings (`fs.inotify.max_user_instances` and `fs.inotify.max_user_watches`) on startup. Changing kernel settings inside the container is not possible! (default: `False`)
|
||||
|
||||
6. Build the Docker image:
|
||||
|
||||
```shell
|
||||
docker compose build chatmail
|
||||
```
|
||||
|
||||
7. Start docker compose and wait for the installation to finish:
|
||||
|
||||
```shell
|
||||
docker compose up -d # start service
|
||||
docker compose logs -f chatmail # view container logs, press CTRL+C to exit
|
||||
```
|
||||
|
||||
8. After installation is complete, you can open `https://<your_domain_name>` in your browser.
|
||||
|
||||
9. To send messages to other chatmail relays,
|
||||
you need to set additional DNS records.
|
||||
Run `docker exec chatmail scripts/cmdeploy.sh dns --ssh-host localhost`
|
||||
to see recommended DNS records and check whether they are correct.
|
||||
|
||||
## Using custom files
|
||||
|
||||
When using Docker, you can apply modified configuration files to make the installation more personalized. This is usually needed for the `www/src` section so that the Chatmail landing page is customized to your taste, but it can be used for any other cases as well.
|
||||
|
||||
To replace files correctly:
|
||||
|
||||
1. Create the `./custom` directory. It is in `.gitignore`, so it won’t cause conflicts when updating.
|
||||
|
||||
```shell
|
||||
mkdir -p ./custom
|
||||
```
|
||||
|
||||
2. Modify the required file. For example, `index.md`:
|
||||
|
||||
```shell
|
||||
mkdir -p ./custom/www/src
|
||||
nano ./custom/www/src/index.md
|
||||
```
|
||||
|
||||
3. In `docker-compose.yaml`, add the file mount in the `volumes` section:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
chatmail:
|
||||
volumes:
|
||||
...
|
||||
## custom resources
|
||||
- ./custom/www/src/index.md:/opt/chatmail/www/src/index.md
|
||||
```
|
||||
|
||||
4. Restart the service:
|
||||
|
||||
```shell
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Locking the Chatmail version
|
||||
|
||||
> [!note]
|
||||
> These steps are optional and should only be done if you are not satisfied that the service is installed each time the container starts.
|
||||
|
||||
Since the current Docker version installs the Chatmail service every time the container starts, you can lock the container version after installation as follows:
|
||||
|
||||
1. Commit the current state of the configured container:
|
||||
|
||||
```shell
|
||||
docker container commit chatmail configured-chatmail:$(date +'%Y-%m-%d')
|
||||
docker image ls | grep configured-chatmail
|
||||
```
|
||||
|
||||
2. Change the entrypoint for the container in `docker-compose.yaml` to:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
chatmail:
|
||||
image: <image name from step 1>
|
||||
volumes:
|
||||
...
|
||||
## custom resources
|
||||
- ./custom/setup_chatmail_docker.sh:/setup_chatmail_docker.sh
|
||||
```
|
||||
|
||||
3. Create the file `./custom/setup_chatmail_docker.sh` with the new configuration:
|
||||
|
||||
```shell
|
||||
mkdir -p ./custom
|
||||
cat > ./custom/setup_chatmail_docker.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
|
||||
set -eo pipefail
|
||||
|
||||
export ENABLE_CERTS_MONITORING="${ENABLE_CERTS_MONITORING:-true}"
|
||||
export CERTS_MONITORING_TIMEOUT="${CERTS_MONITORING_TIMEOUT:-60}"
|
||||
export PATH_TO_SSL="${PATH_TO_SSL:-/var/lib/acme/live/${MAIL_DOMAIN}}"
|
||||
|
||||
calculate_hash() {
|
||||
find "$PATH_TO_SSL" -type f -exec sha1sum {} \; | sort | sha1sum | awk '{print $1}'
|
||||
}
|
||||
|
||||
monitor_certificates() {
|
||||
if [ "$ENABLE_CERTS_MONITORING" != "true" ]; then
|
||||
echo "Certs monitoring disabled."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
current_hash=$(calculate_hash)
|
||||
previous_hash=$current_hash
|
||||
|
||||
while true; do
|
||||
current_hash=$(calculate_hash)
|
||||
if [[ "$current_hash" != "$previous_hash" ]]; then
|
||||
# TODO: add an option to restart at a specific time interval
|
||||
echo "[INFO] Certificate's folder hash was changed, reloading nginx, dovecot and postfix services."
|
||||
systemctl reload nginx.service
|
||||
systemctl reload dovecot.service
|
||||
systemctl reload postfix.service
|
||||
previous_hash=$current_hash
|
||||
fi
|
||||
sleep $CERTS_MONITORING_TIMEOUT
|
||||
done
|
||||
}
|
||||
|
||||
monitor_certificates &
|
||||
EOF
|
||||
```
|
||||
|
||||
4. Restart the service:
|
||||
|
||||
```shell
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
```
|
||||
189
docs/DOCKER_INSTALLATION_RU.md
Normal file
189
docs/DOCKER_INSTALLATION_RU.md
Normal file
@@ -0,0 +1,189 @@
|
||||
# Известные проблемы и ограничения
|
||||
- Chatmail будет переустановлен при каждом запуске контейнера (при первом - долго, при последующих быстрее). Так устроен изначальный установщик, потому что он не был заточен под docker. В конце документации [представлено](#фиксирование-версии-chatmail) возможное решение
|
||||
- Требуется настроенный в системе cgroups v2. Работа с cgroups v1 не тестировалась.
|
||||
- Да, понятно дело что systemd использовать в контейнере костыль и надо это всё разнести на несколько сервисов, но это MVP и в первом приближении оказалось сделать проще так, чем переписывать всю систему развертывания.
|
||||
- docker образ подходит только для amd64, если нужно запустить на другой архитектуре, попробуйте изменить dockerfile (конкретно ту часть что ответсвенна за установку dovecot)
|
||||
|
||||
# Docker installation
|
||||
Здесь представлена инструкция по установке chatmail с помощью docker-compose.
|
||||
|
||||
## Предварительная настройка
|
||||
We use `chat.example.org` as the chatmail domain in the following steps.
|
||||
Please substitute it with your own domain.
|
||||
|
||||
1. Настройте начальные записи DNS.Ниже приведен пример в привычном формате файла зоны BIND сTTL 1 час (3600 секунд).
|
||||
Замените домен и IP-адреса на свои.
|
||||
|
||||
```
|
||||
chat.example.com. 3600 IN A 198.51.100.5
|
||||
chat.example.com. 3600 IN AAAA 2001:db8::5
|
||||
www.chat.example.com. 3600 IN CNAME chat.example.com.
|
||||
mta-sts.chat.example.com. 3600 IN CNAME chat.example.com.
|
||||
```
|
||||
|
||||
2. Склонируйте репозиторий на свой сервер.
|
||||
|
||||
```shell
|
||||
git clone https://github.com/chatmail/relay
|
||||
cd relay
|
||||
```
|
||||
|
||||
## Installation
|
||||
При установке через docker есть несколько вариантов:
|
||||
- использовать встроенный в chatmail контейнер nginx и acmetool для хостинга чата и управления сертификатами.
|
||||
- использовать сторонние инструменты для менеджмента сертификатов
|
||||
|
||||
В качестве примера для стороннего менеджера сертификатов будет использоваться traefik, но вы можете использовать то что удобнее вам.
|
||||
|
||||
1. Скопировать файл `./docker/docker-compose-default.yaml` или `./docker/docker-compose-traefik.yaml` и переименовать в `docker-compose.yaml`. Это нужно потому что `docker-compose.yaml` находится в `.gitignore` и не будет создавать конфликты при обновлении гит репозитория.
|
||||
```shell
|
||||
cp ./docker/docker-compose-default.yaml docker-compose.yaml
|
||||
## or
|
||||
# cp ./docker/docker-compose-traefik.yaml docker-compose.yaml
|
||||
```
|
||||
|
||||
2. Скопировать `./docker/example.env` и переименовать в `.env`. Здесь хранятся переменные, которые используятся в `docker-compose.yaml`.
|
||||
```shell
|
||||
cp ./docker/example.env .env
|
||||
```
|
||||
|
||||
3. Настроить переменные окружения в `.env` файле. Эти переменные используются в `docker-compose.yaml` файле, чтобы передавать повторяющиеся значения.
|
||||
|
||||
4. Настроить параметры ядра, потому что внутри контейнера их нельзя изменить, а конкретно `fs.inotify.max_user_instances` и `fs.inotify.max_user_watches`. Для этого выполнить следующее:
|
||||
```shell
|
||||
echo "fs.inotify.max_user_instances=65536" | sudo tee -a /etc/sysctl.d/99-inotify.conf
|
||||
echo "fs.inotify.max_user_watches=65536" | sudo tee -a /etc/sysctl.d/99-inotify.conf
|
||||
sudo sysctl --system
|
||||
```
|
||||
|
||||
5. Настроить переменные окружения контейнера. Ниже перечислен список переменных учавствующих при развертывании.
|
||||
- `MAIL_DOMAIN` - Доменное имя будущего сервера. (required)
|
||||
- `DEBUG_COMMANDS_ENABLED` - Выполнить debug команды перед установкой. (default: `false`)
|
||||
- `FORCE_REINIT_INI_FILE` - Пересоздавать ini файл конфигурации при запуске. (default: `false`)
|
||||
- `USE_FOREIGN_CERT_MANAGER` - Использовать сторонний менеджер сертификатов. (default: `false`)
|
||||
- `RECREATE_VENV` - Пересоздать виртуальное окружение (venv). Если выставлено `true`, то окружение будет пересоздано при запуске контейнера, из-за чего включение сервиса займет больше времени, но поможет избежать ряда ошибок. (default: `false`)
|
||||
- `INI_FILE` - путь к ini файлу конфигурации. (default: `./chatmail.ini`)
|
||||
- `PATH_TO_SSL` - Путь где располагаются сертификаты. (default: `/var/lib/acme/live/${MAIL_DOMAIN}`)
|
||||
- `ENABLE_CERTS_MONITORING` - Включить мониторинг сертификатов, если `USE_FOREIGN_CERT_MANAGER=true`. Если сертфикаты изменятся сервисы будут автоматически перезапущены. (default: `false`)
|
||||
- `CERTS_MONITORING_TIMEOUT` - Раз во сколько секунд проверять что изменились сертификаты. (default: `'60'`)
|
||||
|
||||
Также могут быть использованы все переменные из [ini файла конфигурации](https://github.com/chatmail/relay/blob/main/chatmaild/src/chatmaild/ini/chatmail.ini.f), они обязаны быть в uppercase формате.
|
||||
|
||||
Ниже перечислены переменные, которые обязательны быть выставлены при развертывании через docker:
|
||||
- `CHANGE_KERNEL_SETTINGS` - Менять настройки ядра (`fs.inotify.max_user_instances` и `fs.inotify.max_user_watches`) при запуске. При запуске в контейнере смена настроек ядра не может быть выполнена! (default: `False`)
|
||||
|
||||
6. Собрать docker образ
|
||||
```shell
|
||||
docker compose build chatmail
|
||||
```
|
||||
|
||||
7. Запустить docker compose и дождаться завершения установки
|
||||
```shell
|
||||
docker compose up -d # запуск сервиса
|
||||
docker compose logs -f chatmail # просмотр логов контейнера. Для выхода нажать CTRL+C
|
||||
```
|
||||
|
||||
8. По окончанию установки можно открыть в браузер `https://<your_domain_name>`
|
||||
|
||||
## Использование кастомных файлов
|
||||
При использовании docker есть возможность использовать измененые файлы конфигурации, чтобы сделать установку более персонализированной. Обычно это требуется для секции `www/src`, чтобы ознакомительная страница Chatmail была сделана на ваш вкус. Но также это можно использовать и для любых других случаев.
|
||||
|
||||
Для того чтобы корректно выполнить подмену файлов необходимо
|
||||
1. создать каталог `./custom`, он находится в `.gitignore`, поэтому при обновлении не вызовет конфликтов.
|
||||
```shell
|
||||
mkdir -p ./custom
|
||||
```
|
||||
|
||||
2. Изменить нужный файл. Для примера возьмем `index.md`
|
||||
```shell
|
||||
mkdir -p ./custom/www/src
|
||||
nano ./custom/www/src/index.md
|
||||
```
|
||||
|
||||
3. В `docker-compose.yaml` добавить монтирование файла с помощью секции `volumes`
|
||||
```yaml
|
||||
services:
|
||||
chatmail:
|
||||
volumes:
|
||||
...
|
||||
## custom resources
|
||||
- ./custom/www/src/index.md:/opt/chatmail/www/src/index.md
|
||||
```
|
||||
|
||||
4. Перезапустить сервис
|
||||
```shell
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Фиксирование версии Chatmail
|
||||
> [!note]
|
||||
> Это опциональные шаги, их делать требуется только если вас не устраивает что сервис устанавливается каждый раз при запуске
|
||||
|
||||
Поскольку в текущей версии docker chatmail сервис устанавливается каждый раз запуске контейнера, чтобы этого не происходило можно зафиксировать версию контейнера после установки. Делается это следующим образом:
|
||||
|
||||
1. Зафиксировать текущее состояние сконфигурированного контейнера
|
||||
```shell
|
||||
docker container commit chatmail configured-chatmail:$(date +'%Y-%m-%d')
|
||||
docker image ls | grep configured-chatmail
|
||||
```
|
||||
|
||||
2. Изменить entrypoint для контейнера в `docker-compose.yaml` на
|
||||
```yaml
|
||||
services:
|
||||
chatmail:
|
||||
image: <image name from step 1>
|
||||
volumes:
|
||||
...
|
||||
## custom resources
|
||||
- ./custom/setup_chatmail_docker.sh:/setup_chatmail_docker.sh
|
||||
```
|
||||
|
||||
3. Создать файл `./custom/setup_chatmail_docker.sh` с новым файлом конфигурации
|
||||
```shell
|
||||
mkdir -p ./custom
|
||||
cat > ./custom/setup_chatmail_docker.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
|
||||
set -eo pipefail
|
||||
|
||||
export ENABLE_CERTS_MONITORING="${ENABLE_CERTS_MONITORING:-true}"
|
||||
export CERTS_MONITORING_TIMEOUT="${CERTS_MONITORING_TIMEOUT:-60}"
|
||||
export PATH_TO_SSL="${PATH_TO_SSL:-/var/lib/acme/live/${MAIL_DOMAIN}}"
|
||||
|
||||
calculate_hash() {
|
||||
find "$PATH_TO_SSL" -type f -exec sha1sum {} \; | sort | sha1sum | awk '{print $1}'
|
||||
}
|
||||
|
||||
monitor_certificates() {
|
||||
if [ "$ENABLE_CERTS_MONITORING" != "true" ]; then
|
||||
echo "Certs monitoring disabled."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
current_hash=$(calculate_hash)
|
||||
previous_hash=$current_hash
|
||||
|
||||
while true; do
|
||||
current_hash=$(calculate_hash)
|
||||
if [[ "$current_hash" != "$previous_hash" ]]; then
|
||||
# TODO: add an option to restart at a specific time interval
|
||||
echo "[INFO] Certificate's folder hash was changed, reloading nginx, dovecot and postfix services."
|
||||
systemctl reload nginx.service
|
||||
systemctl reload dovecot.service
|
||||
systemctl reload postfix.service
|
||||
previous_hash=$current_hash
|
||||
fi
|
||||
sleep $CERTS_MONITORING_TIMEOUT
|
||||
done
|
||||
}
|
||||
|
||||
monitor_certificates &
|
||||
EOF
|
||||
```
|
||||
|
||||
4. Перезапустить сервис
|
||||
```shell
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
```
|
||||
33
traefik/config.yaml
Normal file
33
traefik/config.yaml
Normal file
@@ -0,0 +1,33 @@
|
||||
log:
|
||||
level: TRACE
|
||||
|
||||
entryPoints:
|
||||
web:
|
||||
address: ":80"
|
||||
http:
|
||||
redirections:
|
||||
entryPoint:
|
||||
to: websecure
|
||||
permanent: true
|
||||
websecure:
|
||||
address: ":443"
|
||||
|
||||
providers:
|
||||
docker:
|
||||
endpoint: "unix:///var/run/docker.sock"
|
||||
exposedByDefault: false
|
||||
file:
|
||||
directory: /dynamic/conf
|
||||
watch: true
|
||||
|
||||
serverstransport:
|
||||
insecureskipverify: true
|
||||
|
||||
certificatesResolvers:
|
||||
letsEncrypt:
|
||||
acme:
|
||||
storage: /acme.json
|
||||
caServer: "https://acme-v02.api.letsencrypt.org/directory"
|
||||
tlschallenge: true
|
||||
httpChallenge:
|
||||
entryPoint: web
|
||||
4
traefik/dynamic-configs/insecure.yaml
Normal file
4
traefik/dynamic-configs/insecure.yaml
Normal file
@@ -0,0 +1,4 @@
|
||||
http:
|
||||
serversTransports:
|
||||
insecure:
|
||||
insecureSkipVerify: true
|
||||
15
traefik/post-hook.sh
Executable file
15
traefik/post-hook.sh
Executable file
@@ -0,0 +1,15 @@
|
||||
CERTS_DIR=${CERTS_DIR:-"/data/letsencrypt/certs"}
|
||||
|
||||
echo "CERTS_DIR: $CERTS_DIR"
|
||||
|
||||
for dir in "$CERTS_DIR"/*/; do
|
||||
echo "Processing: $dir"
|
||||
cd "$dir"
|
||||
if [ -f "certificate.crt" ]; then
|
||||
ln -sf certificate.crt fullchain
|
||||
fi
|
||||
if [ -f "privatekey.key" ]; then
|
||||
ln -sf privatekey.key privkey
|
||||
fi
|
||||
cd -
|
||||
done
|
||||
@@ -9,11 +9,7 @@
|
||||
<title>{{ config.mail_domain }} {{ pagename }}</title>
|
||||
<link rel="stylesheet" href="./main.css">
|
||||
<link rel="icon" href="/logo.svg">
|
||||
<<<<<<< HEAD
|
||||
<link rel="mask-icon" href="/logo.svg" color="#000000">
|
||||
=======
|
||||
<link rel=”mask-icon” href=”/logo.svg” color=”#000001">
|
||||
>>>>>>> 2da7de8 (make logo slightly brighter)
|
||||
<link rel=”mask-icon” href=”/logo.svg” color=”#000000">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
|
||||
Reference in New Issue
Block a user