Compare commits

..

2 Commits

Author SHA1 Message Date
holger krekel
cbbcf3cbca add marker dynamically to allow "pytest" to execute nicely at repo root without warnings 2023-10-20 22:45:11 +02:00
holger krekel
d2af9df8f9 rename test files to be unambigously numbered 2023-10-20 22:39:23 +02:00
40 changed files with 519 additions and 779 deletions

View File

@@ -5,27 +5,14 @@ on:
push:
jobs:
tox:
name: chatmail tests
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: run chatmaild tests
- name: Lint chatmaild
working-directory: chatmaild
run: pipx run tox
- name: run deploy-chatmail offline tests
- name: Lint deploy-chatmail
working-directory: deploy-chatmail
run: pipx run tox
- name: run deploy-chatmail offline tests
working-directory: deploy-chatmail
run: pipx run tox
scripts:
name: chatmail script invocations
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: run init.sh
run: ./scripts/init.sh
- name: run test.sh
run: ./scripts/test.sh

2
.gitignore vendored
View File

@@ -159,5 +159,3 @@ cython_debug/
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
chatmail.zone

View File

@@ -1,61 +1,23 @@
# Chatmail instances optimized for Delta Chat apps
# Chat Mail server configuration
This repository helps to setup a ready-to-use chatmail instance
This repository setups a ready-to-go chatmail instance
comprised of a minimal setup of the battle-tested
[postfix smtp](https://www.postfix.org) and [dovecot imap](https://www.dovecot.org) services.
[postfix smtp server](https://www.postfix.org) and [dovecot imap server](https://www.dovecot.org).
The setup is designed and optimized for providing chatmail accounts
for use by [Delta Chat apps](https://delta.chat).
## Getting started
Chatmail accounts are automatically created by a first login,
after which the initially specified password is required for using them.
## Getting Started deploying your own chatmail instance
1. Prepare your local (presumably Linux) system:
1. prepare your local system:
scripts/init.sh
2. Setup a domain with `A` and `AAAA` records for your chatmail server.
3. Set environment variable to the chatmail domain you want to setup:
2. set environment variable to the chatmail domain you want to setup:
export CHATMAIL_DOMAIN=c1.testrun.org # replace with your host
4. Deploy the chat mail instance to your chatmail server:
3. run the deploy of the chat mail instance:
scripts/deploy.sh
This script uses `pyinfra` and `ssh` to setup packages and configure
the chatmail instance on your remote server.
5. Run `scripts/generate-dns-zone.sh` and
transfer the generated DNS records at your DNS provider
6. Start a Delta Chat app and create a new account
by typing an e-mail address with an arbitrary username
and `@<your-chatmail-domain>` appended.
Use an at least 10-character random password.
### Ports
Postfix listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
Dovecot listens on ports 143(imap) and 993 (imaps).
Delta Chat will, however, discover all ports and configurations
automatically by reading the `autoconfig.xml` file from the chatmail instance.
## Emergency Commands to disable automatic account creation
If you need to stop account creation,
e.g. because some script is wildly creating accounts, run:
touch /etc/chatmail-nocreate
While this file is present, account creation will be blocked.
## Running tests and benchmarks (offline and online)
@@ -70,27 +32,28 @@ While this file is present, account creation will be blocked.
scripts/bench.sh
## Running tests (offline and online)
## Development Background for chatmail instances
```
## Dovecot/Postfix configuration
This repository drives the development of "chatmail instances",
comprised of minimal setups of
### Ports
- [postfix smtp server](https://www.postfix.org)
- [dovecot imap server](https://www.dovecot.org)
Postfix listens on ports 25 (smtp) and 587 (submission) and 465 (submissions).
Dovecot listens on ports 143(imap) and 993 (imaps).
as well as two custom services that are integrated with these two:
## DNS
- `chatmaild/src/chatmaild/doveauth.py` implements
create-on-login account creation semantics and is used
by Dovecot during login authentication and by Postfix
which in turn uses [Dovecot SASL](https://doc.dovecot.org/configuration_manual/authentication/dict/#complete-example-for-authenticating-via-a-unix-socket)
to authenticate users
to send mails for them.
- `chatmaild/src/chatmaild/filtermail.py` prevents
unencrypted e-mail from leaving the chatmail instance
and is integrated into postfix's outbound mail pipelines.
For DKIM you must add a DNS entry as found in /etc/opendkim/selector.txt on your chatmail instance.
The above `scripts/deploy.sh` prints out the DKIM selector and DNS entry you
need to setup with your DNS provider.
## Emergency Commands
If you need to stop account creation,
e.g. because some script is wildly creating accounts,
just run `touch /tmp/nocreate`.
You can remove the file
as soon as the attacker was banned
by different means.

View File

@@ -10,20 +10,17 @@ dependencies = [
]
[project.scripts]
doveauth = "chatmaild.doveauth:main"
doveauth-dictproxy = "chatmaild.dictproxy:main"
filtermail = "chatmaild.filtermail:main"
[tool.pytest.ini_options]
addopts = "-v -ra --strict-markers"
log_format = "%(asctime)s %(levelname)s %(message)s"
log_date_format = "%Y-%m-%d %H:%M:%S"
log_level = "INFO"
[tool.tox]
legacy_tox_ini = """
[tox]
isolated_build = true
envlist = lint,py
envlist = lint
[testenv:lint]
skipdist = True
@@ -34,10 +31,4 @@ deps =
commands =
black --quiet --check --diff src/
ruff src/
[testenv]
passenv = CHATMAIL_DOMAIN
deps = pytest
pdbpp
commands = pytest -v -rsXx {posargs: ../tests/chatmaild}
"""

View File

@@ -33,6 +33,13 @@ class Connection:
def cursor(self):
return self._sqlconn.cursor()
def create_user(self, addr: str, password: str):
"""Create a row in the users table."""
self.execute("PRAGMA foreign_keys=on")
q = """INSERT INTO users (addr, password, last_login)
VALUES (?, ?, ?)"""
self.execute(q, (addr, password, int(time.time())))
def get_user(self, addr: str) -> {}:
"""Get a row from the users table."""
q = "SELECT addr, password, last_login from users WHERE addr = ?"

View File

@@ -1,6 +1,5 @@
import logging
import os
import time
import sys
import json
import crypt
@@ -22,29 +21,15 @@ def encrypt_password(password: str):
return "{SHA512-CRYPT}" + passhash
def is_allowed_to_create(user, cleartext_password) -> bool:
"""Return True if user and password are admissable."""
def create_user(db, user, password):
if os.path.exists(NOCREATE_FILE):
logging.warning(f"blocked account creation because {NOCREATE_FILE!r} exists.")
return False
if len(cleartext_password) < 10:
logging.warning("Password needs to be at least 10 characters long")
return False
parts = user.split("@")
if len(parts) != 2:
logging.warning(f"user {user!r} is not a proper e-mail address")
return False
localpart, domain = parts
if domain == "nine.testrun.org":
# nine.testrun.org policy, username has to be exactly nine chars
if len(localpart) != 9:
logging.warning(f"localpart {localpart!r} has not exactly nine chars")
return False
return True
logging.warning(
f"Didn't create account: {NOCREATE_FILE} exists. Delete the file to enable account creation."
)
return
with db.write_transaction() as conn:
conn.create_user(user, password)
return dict(home=f"/home/vmail/{user}", uid="vmail", gid="vmail", password=password)
def get_user_data(db, user):
@@ -60,34 +45,16 @@ def lookup_userdb(db, user):
return get_user_data(db, user)
def lookup_passdb(db, user, cleartext_password):
with db.write_transaction() as conn:
userdata = conn.get_user(user)
if userdata:
# Update last login time.
conn.execute(
"UPDATE users SET last_login=? WHERE addr=?", (int(time.time()), user)
)
userdata["uid"] = "vmail"
userdata["gid"] = "vmail"
return userdata
if not is_allowed_to_create(user, cleartext_password):
return
encrypted_password = encrypt_password(cleartext_password)
q = """INSERT INTO users (addr, password, last_login)
VALUES (?, ?, ?)"""
conn.execute(q, (user, encrypted_password, int(time.time())))
return dict(
home=f"/home/vmail/{user}",
uid="vmail",
gid="vmail",
password=encrypted_password,
)
def lookup_passdb(db, user, password):
userdata = get_user_data(db, user)
if not userdata:
return create_user(db, user, encrypt_password(password))
userdata["password"] = userdata["password"].strip()
return userdata
def handle_dovecot_request(msg, db, mail_domain):
print(f"received msg: {msg!r}", file=sys.stderr)
short_command = msg[0]
if short_command == "L": # LOOKUP
parts = msg[1:].split("\t")
@@ -105,18 +72,19 @@ def handle_dovecot_request(msg, db, mail_domain):
reply_command = "N"
elif type == "passdb":
if user.endswith(f"@{mail_domain}"):
res = lookup_passdb(db, user, cleartext_password=args[0])
res = lookup_passdb(db, user, password=args[0])
if res:
reply_command = "O"
else:
reply_command = "N"
print(f"res: {res!r}", file=sys.stderr)
json_res = json.dumps(res) if res else ""
return f"{reply_command}{json_res}\n"
return None
class ThreadedUnixStreamServer(ThreadingMixIn, UnixStreamServer):
request_queue_size = 100
pass
def main():
@@ -128,20 +96,15 @@ def main():
class Handler(StreamRequestHandler):
def handle(self):
try:
while True:
msg = self.rfile.readline().strip().decode()
if not msg:
break
res = handle_dovecot_request(msg, db, mail_domain)
if res:
self.wfile.write(res.encode("ascii"))
self.wfile.flush()
else:
logging.warn("request had no answer: %r", msg)
except Exception:
logging.exception("Exception in the handler")
raise
while True:
msg = self.rfile.readline().strip().decode()
if not msg:
break
res = handle_dovecot_request(msg, db, mail_domain)
if res:
print(f"sending result: {res!r}", file=sys.stderr)
self.wfile.write(res.encode("ascii"))
self.wfile.flush()
try:
os.unlink(socket)

View File

@@ -2,7 +2,7 @@
Description=Dict authentication proxy for dovecot
[Service]
ExecStart=/usr/local/bin/doveauth /run/dovecot/doveauth.socket vmail /home/vmail/passdb.sqlite
ExecStart=/usr/local/bin/doveauth-dictproxy /run/dovecot/doveauth.socket vmail /home/vmail/passdb.sqlite
Restart=always
RestartSec=30

View File

@@ -34,34 +34,6 @@ def check_encrypted(message):
return True
def check_mdn(message, envelope):
if len(envelope.rcpt_tos) != 1:
return False
for name in ["auto-submitted", "chat-version"]:
if not message.get(name):
return False
if message.get_content_type() != "multipart/report":
return False
body = message.get_body()
if body.get_content_type() != "text/plain":
return False
if list(body.iter_attachments()) or list(body.iter_parts()):
return False
# even with all mime-structural checks an attacker
# could try to abuse the subject or body to contain links or other
# annoyance -- we skip on checking subject/body for now as Delta Chat
# should evolve to create E2E-encrypted read receipts anyway.
# and then MDNs are just encrypted mail and can pass the border
# to other instances.
return True
class SMTPController(Controller):
def factory(self):
return SMTP(self.handler, **self.SMTP_kwargs)
@@ -110,9 +82,6 @@ def check_DATA(envelope):
if envelope.mail_from.lower() != from_addr.lower():
return f"500 Invalid FROM <{from_addr!r}> for <{envelope.mail_from!r}>"
if not mail_encrypted and check_mdn(message, envelope):
return
envelope_from_domain = from_addr.split("@").pop()
for recipient in envelope.rcpt_tos:
if envelope.mail_from == recipient:
@@ -149,7 +118,7 @@ class SendRateLimiter:
def main():
args = sys.argv[1:]
assert len(args) == 1
logging.basicConfig(level=logging.WARN)
logging.basicConfig(level=logging.INFO)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
task = asyncmain_beforequeue(port=int(args[0]))

View File

@@ -0,0 +1,53 @@
import os
import pytest
import chatmaild.dictproxy
from .dictproxy import get_user_data, lookup_passdb
from .database import Database, DBError
@pytest.fixture()
def db(tmpdir):
db_path = tmpdir / "passdb.sqlite"
print("database path:", db_path)
return Database(db_path)
def test_basic(db):
chatmaild.dictproxy.NOCREATE_FILE = "/tmp/nocreate"
if os.path.exists(chatmaild.dictproxy.NOCREATE_FILE):
os.remove(chatmaild.dictproxy.NOCREATE_FILE)
lookup_passdb(db, "link2xt@c1.testrun.org", "asdf")
data = get_user_data(db, "link2xt@c1.testrun.org")
assert data
def test_dont_overwrite_password_on_wrong_login(db):
"""Test that logging in with a different password doesn't create a new user"""
res = lookup_passdb(db, "newuser1@something.org", "kajdlkajsldk12l3kj1983")
assert res["password"]
res2 = lookup_passdb(db, "newuser1@something.org", "kajdlqweqwe")
# this function always returns a password hash, which is actually compared by dovecot.
assert res["password"] == res2["password"]
def test_nocreate_file(db):
chatmaild.dictproxy.NOCREATE_FILE = "/tmp/nocreate"
with open(chatmaild.dictproxy.NOCREATE_FILE, "w+") as f:
f.write("")
assert os.path.exists(chatmaild.dictproxy.NOCREATE_FILE)
lookup_passdb(db, "newuser1@something.org", "kajdlqweqwe")
assert not get_user_data(db, "newuser1@something.org")
os.remove(chatmaild.dictproxy.NOCREATE_FILE)
def test_db_version(db):
assert db.get_schema_version() == 1
def test_too_high_db_version(db):
with db.write_transaction() as conn:
conn.execute("PRAGMA user_version=%s;" % (999,))
with pytest.raises(DBError):
db.ensure_tables()

View File

@@ -0,0 +1,338 @@
from .filtermail import check_encrypted, check_DATA, SendRateLimiter
from email.parser import BytesParser
from email import policy
import pytest
def test_reject_forged_from():
def makemail(from_addr):
return BytesParser(policy=policy.default).parsebytes(
"\r\n".join(
[
f"From: <{from_addr}",
"To: <barbaz@c3.testrun.org>",
"Date: Sun, 15 Oct 2023 16:41:44 +0000",
"Message-ID: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"Chat-Version: 1.0",
"MIME-Version: 1.0",
"Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no",
"",
"Hi!",
"",
"",
]
).encode()
)
class envelope:
mail_from = "bob@c3.testrun.org"
rcpt_tos = ["somebody@c3.testrun.org"]
# test that the filter lets good mail through
envelope.content = makemail(envelope.mail_from).as_bytes()
assert not check_DATA(envelope=envelope)
# test that the filter rejects forged mail
envelope.content = makemail("forged@c3.testrun.org").as_bytes()
error = check_DATA(envelope=envelope)
assert "500" in error
def test_filtermail():
def check_encrypted_bstr(content):
message = BytesParser(policy=policy.default).parsebytes(content)
return check_encrypted(message)
assert not check_encrypted_bstr(b"foo")
assert not check_encrypted_bstr(
"\r\n".join(
[
"Subject: =?utf-8?q?Message_from_foobar=40c2=2Etestrun=2Eorg?=",
"Chat-Disposition-Notification-To: foobar@c2.testrun.org",
"Chat-User-Avatar: 0",
"From: <foobar@c2.testrun.org>",
"To: <barbaz@c2.testrun.org>",
"Date: Sun, 15 Oct 2023 16:41:44 +0000",
"Message-ID: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"Chat-Version: 1.0",
"Autocrypt: addr=foobar@c2.testrun.org; prefer-encrypt=mutual;",
"\tkeydata=xjMEZSrw3hYJKwYBBAHaRw8BAQdAiEKNQFU28c6qsx4vo/JHdt73RXdjMOmByf/XsGiJ7m",
"\tnNFzxmb29iYXJAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUq8N4CGwMECwkIBwYVCAkKCwID",
"\tFgIBFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJCX3gEAhm0MehE5byBBU1avPczr/I",
"\tHjNLht7Qf6++mAhlJmtDcA/0C8VYJhsUpmiDjuZaMDWNv4FO2BJG6LH7gSm6n7ClMJzjgEZSrw3hIK",
"\tKwYBBAGXVQEFAQEHQAxGG/QW0owCfMp1A+vXEMwgzWcBpNFr58kX2eXuPpM6AwEIB8J4BBgWCAAgBQ",
"\tJlKvDeAhsMFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJDg1gEAwLf8KDoAAKyYgjyI",
"\tvYvO9VEgBni1C4Xx1VjcaEmlDK8BALoFuUCK+enw76TtDcAUKhlhUiM6SDRExkS4Nskp/BcK",
"MIME-Version: 1.0",
"Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no",
"",
"Hi!",
"",
"",
]
).encode()
)
assert not check_encrypted_bstr(
"\r\n".join(
[
"Subject: =?utf-8?q?Message_from_foobar=40c2=2Etestrun=2Eorg?=",
"Chat-Disposition-Notification-To: foobar@c2.testrun.org",
"Chat-User-Avatar: 0",
"From: <foobar@c2.testrun.org>",
"To: <barbaz@c2.testrun.org>",
"Date: Sun, 15 Oct 2023 16:41:44 +0000",
"Message-ID: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"Chat-Version: 1.0",
"Autocrypt: addr=foobar@c2.testrun.org; prefer-encrypt=mutual;",
"\tkeydata=xjMEZSrw3hYJKwYBBAHaRw8BAQdAiEKNQFU28c6qsx4vo/JHdt73RXdjMOmByf/XsGiJ7m",
"\tnNFzxmb29iYXJAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUq8N4CGwMECwkIBwYVCAkKCwID",
"\tFgIBFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJCX3gEAhm0MehE5byBBU1avPczr/I",
"\tHjNLht7Qf6++mAhlJmtDcA/0C8VYJhsUpmiDjuZaMDWNv4FO2BJG6LH7gSm6n7ClMJzjgEZSrw3hIK",
"\tKwYBBAGXVQEFAQEHQAxGG/QW0owCfMp1A+vXEMwgzWcBpNFr58kX2eXuPpM6AwEIB8J4BBgWCAAgBQ",
"\tJlKvDeAhsMFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJDg1gEAwLf8KDoAAKyYgjyI",
"\tvYvO9VEgBni1C4Xx1VjcaEmlDK8BALoFuUCK+enw76TtDcAUKhlhUiM6SDRExkS4Nskp/BcK",
"MIME-Version: 1.0",
"Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no",
"",
"Hi!",
"",
"",
]
).encode()
)
# https://xkcd.com/1181/
assert not check_encrypted_bstr(
"\r\n".join(
[
"Subject: =?utf-8?q?Message_from_foobar=40c2=2Etestrun=2Eorg?=",
"Chat-Disposition-Notification-To: foobar@c2.testrun.org",
"Chat-User-Avatar: 0",
"From: <foobar@c2.testrun.org>",
"To: <barbaz@c2.testrun.org>",
"Date: Sun, 15 Oct 2023 16:41:44 +0000",
"Message-ID: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"Chat-Version: 1.0",
"Autocrypt: addr=foobar@c2.testrun.org; prefer-encrypt=mutual;",
"\tkeydata=xjMEZSrw3hYJKwYBBAHaRw8BAQdAiEKNQFU28c6qsx4vo/JHdt73RXdjMOmByf/XsGiJ7m",
"\tnNFzxmb29iYXJAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUq8N4CGwMECwkIBwYVCAkKCwID",
"\tFgIBFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJCX3gEAhm0MehE5byBBU1avPczr/I",
"\tHjNLht7Qf6++mAhlJmtDcA/0C8VYJhsUpmiDjuZaMDWNv4FO2BJG6LH7gSm6n7ClMJzjgEZSrw3hIK",
"\tKwYBBAGXVQEFAQEHQAxGG/QW0owCfMp1A+vXEMwgzWcBpNFr58kX2eXuPpM6AwEIB8J4BBgWCAAgBQ",
"\tJlKvDeAhsMFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJDg1gEAwLf8KDoAAKyYgjyI",
"\tvYvO9VEgBni1C4Xx1VjcaEmlDK8BALoFuUCK+enw76TtDcAUKhlhUiM6SDRExkS4Nskp/BcK",
"MIME-Version: 1.0",
"Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no",
"",
"-----BEGIN PGP MESSAGE-----",
"Hi!",
"-----END PGP MESSAGE-----",
"",
"",
]
).encode()
)
assert check_encrypted_bstr(
"\r\n".join(
[
"Subject: ...",
"From: <barbaz@c2.testrun.org>",
"To: <foobar@c2.testrun.org>",
"Date: Sun, 15 Oct 2023 16:43:21 +0000",
"Message-ID: <Mr.UVyJWZmkCKM.hGzNc6glBE_@c2.testrun.org>",
"In-Reply-To: <Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>",
"References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"\t<Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>",
"Chat-Version: 1.0",
"Autocrypt: addr=barbaz@c2.testrun.org; prefer-encrypt=mutual;",
"\tkeydata=xjMEZSwWjhYJKwYBBAHaRw8BAQdAQBEhqeJh0GueHB6kF/DUQqYCxARNBVokg/AzT+7LqH",
"\trNFzxiYXJiYXpAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUsFo4CGwMECwkIBwYVCAkKCwID",
"\tFgIBFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX9A4AEAnHWHp49eBCMHK5t66gYPiW",
"\tXQuB1mwUjzGfYWB+0RXUoA/0xcQ3FbUNlGKW7Blp6eMFfViv6Mv2d3kNSXACB6nmcMzjgEZSwWjhIK",
"\tKwYBBAGXVQEFAQEHQBpY5L2M1XHo0uxf8SX1wNLBp/OVvidoWHQF2Jz+kJsUAwEIB8J4BBgWCAAgBQ",
"\tJlLBaOAhsMFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX/INgEA37AJaNvruYsJVanP",
"\tIXnYw4CKd55UAwl8Zcy+M2diAbkA/0fHHcGV4r78hpbbL1Os52DPOdqYQRauIeJUeG+G6bQO",
"MIME-Version: 1.0",
'Content-Type: multipart/encrypted; protocol="application/pgp-encrypted";',
'\tboundary="YFrteb74qSXmggbOxZL9dRnhymywAi"',
"",
"",
"--YFrteb74qSXmggbOxZL9dRnhymywAi",
"Content-Description: PGP/MIME version identification",
"Content-Type: application/pgp-encrypted",
"",
"Version: 1",
"",
"",
"--YFrteb74qSXmggbOxZL9dRnhymywAi",
"Content-Description: OpenPGP encrypted message",
'Content-Disposition: inline; filename="encrypted.asc";',
'Content-Type: application/octet-stream; name="encrypted.asc"',
"",
"-----BEGIN PGP MESSAGE-----",
"",
"wU4DhW3gBZ/VvCYSAQdA8bMs2spwbKdGjVsL1ByPkNrqD7frpB73maeL6I6SzDYg",
"O5G53tv339RdKq3WRcCtEEvxjHlUx2XNwXzC04BpmfvBTgNfPUyLDzjXnxIBB0Ae",
"8ymwGvXMCCimHXN0Dg8Ui62KOi03h0UgheoHWovJSCDF4CKre/xtFr3nL7lq/PKI",
"JsjVNz7/RK9FSXF6WwfONtLCyQGEuVAsB/KXfCBEyfKhaMwGHvhujRidGW5uV1no",
"lMGl3ODmo29Lgeu2uSE7EpJRZoe6hU6ddmBkqxax61ZtkaFlGFFpdo2K8balNNdz",
"ZsJ/9mmI9x3oOJ4/l1nhQbUO9ADbs7gJhFdV5Qkp30b5fCI7bU+aoe1ccBbLe/WM",
"YUty1PqcuQT7XjA+XmYuL261tvW8pBetT+i33/E2d8PzzYt2IuK9qeevyS+yxdwA",
"kfwejFWzzsUlJaDxs1x4XOxkMgSj+jo+g12dFOb7fyClsAnq23iDb8AuaT/BScAI",
"+lO+gher69+6LmM7VGHLG5k762J1jTaQCaKt1s8TAWV99Eo4491vL6fyvk3l/Cfg",
"RXSwiWFgj19Pn0Rq7CD9v22UE2vdUMBTcV4aw79mClk1YQ23jbF0y5DCjPdJ62Zo",
"tskBgFt3NoWV80jZ76zIBLrrjLwCCll8JjJtFwSkt2GX5RFBsVa4A8IDht9RtEk7",
"rrHgbSZQfkauEi/mH3/6CDZoLqSHudUZ7d4MaJwun1TkFYGe2ORwGJd4OBj3oGJp",
"H8YBwCpk///L/fKjX0Gg3M8nrpM4wrRFhPKidAgO/kcm25X4+ZHlVkWBTCt5RWKI",
"fHh6oLDZCqCfcgMkE1KKmwfIHaUkhq5BPRigwy6i5dh1DM4+1UCLh3dxzVbqE9b9",
"61NB19nXdRtDA2sOUnj9ve6m/wEPyCb6/zBQZqvCBYb1/AjdXpUrFT+DbpfyxaXN",
"XfhDVb5mNqNM/IVj0V5fvTc6vOfYbzQtPm10H+FdWWfb+rJRfyC3MA2w2IqstFe3",
"w3bu2iE6CQvSqRvge+ZqLKt/NqYwOURiUmpuklbl3kPJ97+mfKWoiqk8Iz1VY+bb",
"NMUC7aoGv+jcoj+WS6PYO8N6BeRVUUB3ZJSf8nzjgxm1/BcM+UD3BPrlhT11ODRs",
"baifGbprMWwt3dhb8cQgRT8GPdpO1OsDkzL6iikMjLHWWiA99GV6ruiHsIPw6boW",
"A6/uSOskbDHOROotKmddGTBd0iiHXAoQsJFt1ZjUkt6EHrgWs+GAvrvKpXs1mrz8",
"uj3GwEFrHS+Xuf2UDgpszYT3hI2cL/kUtGakVR7m7vVMZqXBUbZdGAEb1PZNPwsI",
"E4aMK02+EVB+tSN4Fzj99N2YD0inVYt+oPjr2tHhUS6aSGBNS/48Ki47DOg4Sxkn",
"lkOWnEbCD+XTnbDd",
"=agR5",
"-----END PGP MESSAGE-----",
"",
"",
"--YFrteb74qSXmggbOxZL9dRnhymywAi--",
"",
"",
]
).encode()
)
assert not check_encrypted_bstr(
"\r\n".join(
[
"Subject: Buy Penis Enlargement at www.malicious-domain.com",
"From: <barbaz@c2.testrun.org>",
"To: <foobar@c2.testrun.org>",
"Date: Sun, 15 Oct 2023 16:43:21 +0000",
"Message-ID: <Mr.UVyJWZmkCKM.hGzNc6glBE_@c2.testrun.org>",
"In-Reply-To: <Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>",
"References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>",
"\t<Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>",
"Chat-Version: 1.0",
"Autocrypt: addr=barbaz@c2.testrun.org; prefer-encrypt=mutual;",
"\tkeydata=xjMEZSwWjhYJKwYBBAHaRw8BAQdAQBEhqeJh0GueHB6kF/DUQqYCxARNBVokg/AzT+7LqH",
"\trNFzxiYXJiYXpAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUsFo4CGwMECwkIBwYVCAkKCwID",
"\tFgIBFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX9A4AEAnHWHp49eBCMHK5t66gYPiW",
"\tXQuB1mwUjzGfYWB+0RXUoA/0xcQ3FbUNlGKW7Blp6eMFfViv6Mv2d3kNSXACB6nmcMzjgEZSwWjhIK",
"\tKwYBBAGXVQEFAQEHQBpY5L2M1XHo0uxf8SX1wNLBp/OVvidoWHQF2Jz+kJsUAwEIB8J4BBgWCAAgBQ",
"\tJlLBaOAhsMFiEEFTfUNvVnY3b9F7yHnmme1PfUhX8ACgkQnmme1PfUhX/INgEA37AJaNvruYsJVanP",
"\tIXnYw4CKd55UAwl8Zcy+M2diAbkA/0fHHcGV4r78hpbbL1Os52DPOdqYQRauIeJUeG+G6bQO",
"MIME-Version: 1.0",
'Content-Type: multipart/encrypted; protocol="application/pgp-encrypted";',
'\tboundary="YFrteb74qSXmggbOxZL9dRnhymywAi"',
"",
"",
"--YFrteb74qSXmggbOxZL9dRnhymywAi",
"Content-Description: PGP/MIME version identification",
"Content-Type: application/pgp-encrypted",
"",
"Version: 1",
"",
"",
"--YFrteb74qSXmggbOxZL9dRnhymywAi",
"Content-Description: OpenPGP encrypted message",
'Content-Disposition: inline; filename="encrypted.asc";',
'Content-Type: application/octet-stream; name="encrypted.asc"',
"",
"-----BEGIN PGP MESSAGE-----",
"",
"wU4DhW3gBZ/VvCYSAQdA8bMs2spwbKdGjVsL1ByPkNrqD7frpB73maeL6I6SzDYg",
"O5G53tv339RdKq3WRcCtEEvxjHlUx2XNwXzC04BpmfvBTgNfPUyLDzjXnxIBB0Ae",
"8ymwGvXMCCimHXN0Dg8Ui62KOi03h0UgheoHWovJSCDF4CKre/xtFr3nL7lq/PKI",
"JsjVNz7/RK9FSXF6WwfONtLCyQGEuVAsB/KXfCBEyfKhaMwGHvhujRidGW5uV1no",
"lMGl3ODmo29Lgeu2uSE7EpJRZoe6hU6ddmBkqxax61ZtkaFlGFFpdo2K8balNNdz",
"ZsJ/9mmI9x3oOJ4/l1nhQbUO9ADbs7gJhFdV5Qkp30b5fCI7bU+aoe1ccBbLe/WM",
"YUty1PqcuQT7XjA+XmYuL261tvW8pBetT+i33/E2d8PzzYt2IuK9qeevyS+yxdwA",
"kfwejFWzzsUlJaDxs1x4XOxkMgSj+jo+g12dFOb7fyClsAnq23iDb8AuaT/BScAI",
"+lO+gher69+6LmM7VGHLG5k762J1jTaQCaKt1s8TAWV99Eo4491vL6fyvk3l/Cfg",
"RXSwiWFgj19Pn0Rq7CD9v22UE2vdUMBTcV4aw79mClk1YQ23jbF0y5DCjPdJ62Zo",
"tskBgFt3NoWV80jZ76zIBLrrjLwCCll8JjJtFwSkt2GX5RFBsVa4A8IDht9RtEk7",
"rrHgbSZQfkauEi/mH3/6CDZoLqSHudUZ7d4MaJwun1TkFYGe2ORwGJd4OBj3oGJp",
"H8YBwCpk///L/fKjX0Gg3M8nrpM4wrRFhPKidAgO/kcm25X4+ZHlVkWBTCt5RWKI",
"fHh6oLDZCqCfcgMkE1KKmwfIHaUkhq5BPRigwy6i5dh1DM4+1UCLh3dxzVbqE9b9",
"61NB19nXdRtDA2sOUnj9ve6m/wEPyCb6/zBQZqvCBYb1/AjdXpUrFT+DbpfyxaXN",
"XfhDVb5mNqNM/IVj0V5fvTc6vOfYbzQtPm10H+FdWWfb+rJRfyC3MA2w2IqstFe3",
"w3bu2iE6CQvSqRvge+ZqLKt/NqYwOURiUmpuklbl3kPJ97+mfKWoiqk8Iz1VY+bb",
"NMUC7aoGv+jcoj+WS6PYO8N6BeRVUUB3ZJSf8nzjgxm1/BcM+UD3BPrlhT11ODRs",
"baifGbprMWwt3dhb8cQgRT8GPdpO1OsDkzL6iikMjLHWWiA99GV6ruiHsIPw6boW",
"A6/uSOskbDHOROotKmddGTBd0iiHXAoQsJFt1ZjUkt6EHrgWs+GAvrvKpXs1mrz8",
"uj3GwEFrHS+Xuf2UDgpszYT3hI2cL/kUtGakVR7m7vVMZqXBUbZdGAEb1PZNPwsI",
"E4aMK02+EVB+tSN4Fzj99N2YD0inVYt+oPjr2tHhUS6aSGBNS/48Ki47DOg4Sxkn",
"lkOWnEbCD+XTnbDd",
"=agR5",
"-----END PGP MESSAGE-----",
"",
"",
"--YFrteb74qSXmggbOxZL9dRnhymywAi--",
"",
"",
]
).encode()
)
assert not check_encrypted_bstr(
"\r\n".join(
[
"Subject: Message opened",
"From: <barbaz@c2.testrun.org>",
"To: <foobar@c2.testrun.org>",
"Date: Sun, 15 Oct 2023 16:43:25 +0000",
"Message-ID: <Mr.78MWtlV7RAi.goCFzBhCYfy@c2.testrun.org>",
"Auto-Submitted: auto-replied",
"Chat-Version: 1.0",
"MIME-Version: 1.0",
"Content-Type: multipart/report; report-type=disposition-notification;",
'\tboundary="Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi"',
"",
"",
"--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi",
"Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no",
"",
'The "Hi!" message you sent was displayed on the screen of the recipient.',
"",
"This is no guarantee the content was read.",
"",
"",
"--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi",
"Content-Type: message/disposition-notification",
"",
"Reporting-UA: Delta Chat 1.124.1",
"Original-Recipient: rfc822;barbaz@c2.testrun.org",
"Final-Recipient: rfc822;barbaz@c2.testrun.org",
"Original-Message-ID: <Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>",
"Disposition: manual-action/MDN-sent-automatically; displayed",
"",
"",
"--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi--",
"",
"",
]
).encode()
)
def test_send_rate_limiter():
limiter = SendRateLimiter()
for i in range(100):
if limiter.is_sending_allowed("some@example.org"):
if i <= SendRateLimiter.MAX_USER_SEND_PER_MINUTE:
continue
pytest.fail("limiter didn't work")
else:
assert i == SendRateLimiter.MAX_USER_SEND_PER_MINUTE + 1
break

View File

@@ -4,10 +4,9 @@ Chat Mail pyinfra deploy.
import importlib.resources
from pathlib import Path
from pyinfra import host
from pyinfra.operations import apt, files, server, systemd
from pyinfra import host, logger
from pyinfra.operations import apt, files, server, systemd, python
from pyinfra.facts.files import File
from pyinfra.facts.systemd import SystemdEnabled
from .acmetool import deploy_acmetool
@@ -35,17 +34,8 @@ def _install_chatmaild() -> None:
commands=[f"pip install --break-system-packages {remote_path}"],
)
# disable legacy doveauth-dictproxy.service
if host.get_fact(SystemdEnabled).get("doveauth-dictproxy.service"):
systemd.service(
name="Disable legacy doveauth-dictproxy.service",
service="doveauth-dictproxy.service",
running=False,
enabled=False,
)
for fn in (
"doveauth",
"doveauth-dictproxy",
"filtermail",
):
files.put(
@@ -80,36 +70,6 @@ def _configure_opendkim(domain: str, dkim_selector: str) -> bool:
mode="644",
config={"domain_name": domain, "opendkim_selector": dkim_selector},
)
need_restart |= main_config.changed
files.directory(
name="Add opendkim directory to /etc",
path="/etc/opendkim",
user="opendkim",
group="opendkim",
mode="750",
present=True,
)
keytable = files.template(
src=importlib.resources.files(__package__).joinpath("opendkim/KeyTable"),
dest="/etc/dkimkeys/KeyTable",
user="opendkim",
group="opendkim",
mode="644",
config={"domain_name": domain, "opendkim_selector": dkim_selector},
)
need_restart |= keytable.changed
signing_table = files.template(
src=importlib.resources.files(__package__).joinpath("opendkim/SigningTable"),
dest="/etc/dkimkeys/SigningTable",
user="opendkim",
group="opendkim",
mode="644",
config={"domain_name": domain, "opendkim_selector": dkim_selector},
)
need_restart |= signing_table.changed
files.directory(
name="Add opendkim socket directory to /var/spool/postfix",
@@ -130,6 +90,8 @@ def _configure_opendkim(domain: str, dkim_selector: str) -> bool:
_sudo_user="opendkim",
)
need_restart |= main_config.changed
return need_restart
@@ -193,17 +155,6 @@ def _configure_dovecot(mail_server: str, debug: bool = False) -> bool:
mode="644",
)
# as per https://doc.dovecot.org/configuration_manual/os/
# it is recommended to set the following inotify limits
for name in ("max_user_instances", "max_user_watches"):
key = f"fs.inotify.{name}"
server.sysctl(
name=f"Change {key}",
key=key,
value=65535,
persist=True,
)
return need_restart
@@ -212,7 +163,7 @@ def _configure_nginx(domain: str, debug: bool = False) -> bool:
need_restart = False
main_config = files.template(
src=importlib.resources.files(__package__).joinpath("nginx/nginx.conf.j2"),
src=importlib.resources.files(__package__).joinpath("nginx.conf.j2"),
dest="/etc/nginx/nginx.conf",
user="root",
group="root",
@@ -222,7 +173,7 @@ def _configure_nginx(domain: str, debug: bool = False) -> bool:
need_restart |= main_config.changed
autoconfig = files.template(
src=importlib.resources.files(__package__).joinpath("nginx/autoconfig.xml.j2"),
src=importlib.resources.files(__package__).joinpath("autoconfig.xml.j2"),
dest="/var/www/html/.well-known/autoconfig/mail/config-v1.1.xml",
user="root",
group="root",
@@ -287,12 +238,6 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
opendkim_need_restart = _configure_opendkim(mail_domain, dkim_selector)
nginx_need_restart = _configure_nginx(mail_domain)
# deploy web pages and info if we have them
pkg_root = importlib.resources.files(__package__)
www_path = pkg_root.joinpath(f"../../../www/{mail_domain}").resolve()
if www_path.is_dir():
files.rsync(f"{www_path}/", "/var/www/html", flags=["-avz"])
systemd.service(
name="Start and enable OpenDKIM",
service="opendkim.service",
@@ -347,3 +292,14 @@ def deploy_chatmail(mail_domain: str, mail_server: str, dkim_selector: str) -> N
enabled=True,
restarted=journald_conf,
)
def callback():
result = server.shell(
commands=[
f"""sed 's/\tIN/ 600 IN/;s/\t(//;s/\"$//;s/^\t \"//g; s/ ).*//' """
f"""/etc/dkimkeys/{dkim_selector}.txt | tr --delete '\n'"""
]
)
logger.info(f"Add this TXT entry into DNS zone: {result.stdout}")
python.call(name="Print TXT entry for DKIM", function=callback)

View File

@@ -6,7 +6,7 @@ from deploy_chatmail import deploy_chatmail
def main():
mail_domain = os.getenv("CHATMAIL_DOMAIN")
mail_server = os.getenv("CHATMAIL_SERVER", mail_domain)
dkim_selector = os.getenv("CHATMAIL_DKIM_SELECTOR", "dkim")
dkim_selector = os.getenv("CHATMAIL_DKIM_SELECTOR", "2023")
assert mail_domain
assert mail_server

View File

@@ -19,7 +19,7 @@ mail_plugins = quota
# these are the capabilities Delta Chat cares about actually
# so let's keep the network overhead per login small
# https://github.com/deltachat/deltachat-core-rust/blob/master/src/imap/capabilities.rs
imap_capability = IMAP4rev1 IDLE MOVE QUOTA CONDSTORE NOTIFY
imap_capability = IMAP4rev1 IDLE MOVE QUOTA CONDSTORE
# Authentication for system users.
@@ -118,24 +118,6 @@ service auth-worker {
user = vmail
}
service imap-login {
# High-security mode.
# Each process serves a single connection and exits afterwards.
# This is the default, but we set it explicitly to be sure.
# See <https://doc.dovecot.org/admin_manual/login_processes/#high-security-mode> for details.
service_count = 1
# Inrease the number of simultaneous connections.
#
# As of Dovecot 2.3.19.1 the default is 100 processes.
# Combined with `service_count = 1` it means only 100 connections
# can be handled simultaneously.
process_limit = 10000
# Avoid startup latency for new connections.
process_min_avail = 10
}
ssl = required
ssl_cert = </var/lib/acme/live/{{ config.hostname }}/fullchain
ssl_key = </var/lib/acme/live/{{ config.hostname }}/privkey

View File

@@ -1 +0,0 @@
dkim._domainkey.{{ config.domain_name }} {{ config.domain_name }}:{{ config.opendkim_selector }}:/etc/dkimkeys/dkim.private

View File

@@ -1 +0,0 @@
*@{{ config.domain_name }} {{ config.opendkim_selector }}._domainkey.{{ config.domain_name }}

View File

@@ -1,4 +1,7 @@
# OpenDKIM configuration.
# This is a basic configuration for signing and verifying. It can easily be
# adapted to suit a basic installation. See opendkim.conf(5) and
# /usr/share/doc/opendkim/examples/opendkim.conf.sample for complete
# documentation of available configuration parameters.
Syslog yes
SyslogSuccess yes
@@ -18,9 +21,7 @@ OversignHeaders From
# setup options can be found in /usr/share/doc/opendkim/README.opendkim.
Domain {{ config.domain_name }}
Selector {{ config.opendkim_selector }}
KeyFile /etc/dkimkeys/{{ config.opendkim_selector }}.private
KeyTable /etc/dkimkeys/KeyTable
SigningTable /etc/dkimkeys/SigningTable
KeyFile /etc/dkimkeys/{{ config.opendkim_selector }}.private
# In Debian, opendkim runs as user "opendkim". A umask of 007 is required when
# using a local socket with MTAs that access the socket as a non-privileged

View File

@@ -32,7 +32,6 @@ submission inet n - y - - smtpd
-o smtpd_recipient_restrictions=
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o milter_macro_daemon_name=ORIGINATING
-o smtpd_client_connection_count_limit=1000
-o smtpd_proxy_filter=127.0.0.1:10080
smtps inet n - y - - smtpd
-o syslog_name=postfix/smtps
@@ -47,7 +46,6 @@ smtps inet n - y - - smtpd
-o smtpd_sender_restrictions=$mua_sender_restrictions
-o smtpd_recipient_restrictions=
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o smtpd_client_connection_count_limit=1000
-o milter_macro_daemon_name=ORIGINATING
-o smtpd_proxy_filter=127.0.0.1:10080
#628 inet n - y - - qmqpd

View File

@@ -6,16 +6,9 @@ import subprocess
import imaplib
import smtplib
import itertools
from email.parser import BytesParser
from email import policy
from pathlib import Path
from math import ceil
import pytest
from chatmaild.database import Database
conftestdir = Path(__file__).parent
def pytest_addoption(parser):
parser.addoption(
@@ -72,7 +65,7 @@ def pytest_report_header():
@pytest.fixture
def benchmark(request):
def bench(func, num, name=None, reportfunc=None):
def bench(func, num, name=None):
if name is None:
name = func.__name__
durations = []
@@ -81,7 +74,7 @@ def benchmark(request):
func()
durations.append(time.time() - now)
durations.sort()
request.config._benchresults[name] = (reportfunc, durations)
request.config._benchresults[name] = durations
return bench
@@ -93,7 +86,7 @@ def pytest_terminal_summary(terminalreporter):
return
tr.section("benchmark results")
float_names = "median min max".split()
float_names = 'median min max'.split()
width = max(map(len, float_names))
def fcol(parts):
@@ -102,9 +95,7 @@ def pytest_terminal_summary(terminalreporter):
headers = f"{'benchmark name': <30} " + fcol(float_names)
tr.write_line(headers)
tr.write_line("-" * len(headers))
summary_lines = []
for name, (reportfunc, durations) in results.items():
for name, durations in results.items():
measures = [
sorted(durations)[len(durations) // 2],
min(durations),
@@ -113,16 +104,6 @@ def pytest_terminal_summary(terminalreporter):
line = f"{name: <30} "
line += fcol(f"{float: 2.2f}" for float in measures)
tr.write_line(line)
vmedian, vmin, vmax = measures
if reportfunc:
for line in reportfunc(vmin=vmin, vmedian=vmedian, vmax=vmax):
summary_lines.append(line)
if summary_lines:
tr.write_line("")
tr.section("benchmark summary measures")
for line in summary_lines:
tr.write_line(line)
@pytest.fixture
@@ -130,16 +111,6 @@ def imap(maildomain):
return ImapConn(maildomain)
@pytest.fixture
def make_imap_connection(maildomain):
def make_imap_connection():
conn = ImapConn(maildomain)
conn.connect()
return conn
return make_imap_connection
class ImapConn:
AuthError = imaplib.IMAP4.error
logcmd = "journalctl -f -u dovecot"
@@ -180,16 +151,6 @@ def smtp(maildomain):
return SmtpConn(maildomain)
@pytest.fixture
def make_smtp_connection(maildomain):
def make_smtp_connection():
conn = SmtpConn(maildomain)
conn.connect()
return conn
return make_smtp_connection
class SmtpConn:
AuthError = smtplib.SMTPAuthenticationError
logcmd = "journalctl -f -t postfix/smtpd -t postfix/smtp -t postfix/lmtp"
@@ -228,20 +189,13 @@ def gencreds(maildomain):
num = next(count)
alphanumeric = "abcdefghijklmnopqrstuvwxyz1234567890"
user = "".join(random.choices(alphanumeric, k=10))
user = f"ac{num}_{user}"[:9]
password = "".join(random.choices(alphanumeric, k=12))
user = f"ac{num}_{user}"
password = "".join(random.choices(alphanumeric, k=10))
yield f"{user}@{domain}", f"{password}"
return lambda domain=None: next(gen(domain))
@pytest.fixture()
def db(tmpdir):
db_path = tmpdir / "passdb.sqlite"
print("database path:", db_path)
return Database(db_path)
#
# Delta Chat testplugin re-use
# use the cmfactory fixture to get chatmail instance accounts
@@ -312,7 +266,7 @@ class Remote:
self.sshdomain = sshdomain
def iter_output(self, logcmd=""):
getjournal = "journalctl -f" if not logcmd else logcmd
getjournal = f"journalctl -f" if not logcmd else logcmd
self.popen = subprocess.Popen(
["ssh", f"root@{self.sshdomain}", getjournal],
stdout=subprocess.PIPE,
@@ -327,19 +281,13 @@ class Remote:
@pytest.fixture
def maildata(request, gencreds):
datadir = conftestdir.joinpath("mail-data")
def mailgen(request):
class Mailgen:
def get_encrypted(self, from_addr, to_addr):
data = request.fspath.dirpath("mailgen/encrypted.eml").read()
return data.format(from_addr=from_addr, to_addr=to_addr)
def maildata(name, from_addr=None, to_addr=None):
if from_addr is None:
from_addr = gencreds()[0]
if to_addr is None:
to_addr = gencreds()[0]
data = datadir.joinpath(name).read_text()
text = data.format(from_addr=from_addr, to_addr=to_addr)
return BytesParser(policy=policy.default).parsebytes(text.encode())
return maildata
return Mailgen()
@pytest.fixture

View File

@@ -1,6 +1,5 @@
import pytest
import threading
import queue
import smtplib
def test_login_basic_functioning(imap_or_smtp, gencreds, lp):
@@ -24,11 +23,6 @@ def test_login_basic_functioning(imap_or_smtp, gencreds, lp):
with pytest.raises(imap_or_smtp.AuthError):
imap_or_smtp.login(user, password + "wrong")
lp.sec("creating users with a short password is not allowed")
user, _password = gencreds()
with pytest.raises(imap_or_smtp.AuthError):
imap_or_smtp.login(user, "admin")
def test_login_same_password(imap_or_smtp, gencreds):
"""Test two different users logging in with the same password
@@ -41,30 +35,3 @@ def test_login_same_password(imap_or_smtp, gencreds):
imap_or_smtp.login(user1, password1)
imap_or_smtp.connect()
imap_or_smtp.login(user2, password1)
def test_concurrent_logins_same_account(
make_imap_connection, make_smtp_connection, gencreds
):
"""Test concurrent smtp and imap logins
and check remote server succeeds on each connection.
"""
user1, password1 = gencreds()
login_results = queue.Queue()
def login_smtp_imap(smtp, imap):
try:
imap.login(user1, password1)
except Exception:
login_results.put(False)
else:
login_results.put(True)
conns = [(make_smtp_connection(), make_imap_connection()) for i in range(10)]
for args in conns:
thread = threading.Thread(target=login_smtp_imap, args=args, daemon=True)
thread.start()
for _ in conns:
assert login_results.get()

View File

@@ -20,7 +20,7 @@ def test_use_two_chatmailservers(cmfactory, maildomain2):
@pytest.mark.parametrize("forgeaddr", ["internal", "someone@example.org"])
def test_reject_forged_from(cmsetup, maildata, lp, forgeaddr):
def test_reject_forged_from(cmsetup, mailgen, lp, forgeaddr):
user1, user3 = cmsetup.gen_users(2)
lp.sec("send encrypted message with forged from")
@@ -31,8 +31,7 @@ def test_reject_forged_from(cmsetup, maildata, lp, forgeaddr):
addr_to_forge = "someone@example.org"
print("message to inject:")
msg = maildata("encrypted.eml", from_addr=addr_to_forge, to_addr=user3.addr)
msg = msg.as_string()
msg = mailgen.get_encrypted(from_addr=addr_to_forge, to_addr=user3.addr)
for line in msg.split("\n")[:4]:
print(f" {line}")
@@ -43,12 +42,10 @@ def test_reject_forged_from(cmsetup, maildata, lp, forgeaddr):
@pytest.mark.slow
def test_exceed_rate_limit(cmsetup, gencreds, maildata):
def test_exceed_rate_limit(cmsetup, gencreds, mailgen):
"""Test that the per-account send-mail limit is exceeded."""
user1, user2 = cmsetup.gen_users(2)
mail = maildata(
"encrypted.eml", from_addr=user1.addr, to_addr=user2.addr
).as_string()
mail = mailgen.get_encrypted(user1.addr, user2.addr)
for i in range(100):
print("Sending mail", str(i))
try:
@@ -58,6 +55,6 @@ def test_exceed_rate_limit(cmsetup, gencreds, maildata):
pytest.fail(f"rate limit was exceeded too early with msg {i}")
outcome = e.recipients[user2.addr]
assert outcome[0] == 450
assert b"4.7.1: Too much mail from" in outcome[1]
assert b'4.7.1: Too much mail from' in outcome[1]
return
pytest.fail("Rate limit was not exceeded")

View File

@@ -1,4 +1,3 @@
import time
import random
import pytest
@@ -82,29 +81,3 @@ class TestEndToEndDeltaChat:
ch = ac2.qr_setup_contact(qr)
assert ch.id >= 10
ac1._evtracker.wait_securejoin_inviter_progress(1000)
def test_read_receipts_between_instances(self, cmfactory, lp, maildomain2):
ac1 = cmfactory.new_online_configuring_account(cache=False)
cmfactory.switch_maildomain(maildomain2)
ac2 = cmfactory.new_online_configuring_account(cache=False)
cmfactory.bring_accounts_online()
lp.sec("setup encrypted comms between ac1 and ac2 on different instances")
qr = ac1.get_setup_contact_qr()
ac2.qr_setup_contact(qr)
msg = ac2.wait_next_incoming_message()
assert "verified" in msg.text
lp.sec("ac1 sends a message and ac2 marks it as seen")
chat = ac1.create_chat(ac2)
msg = chat.send_text("hi")
m = ac2.wait_next_incoming_message()
m.mark_seen()
# we can only indirectly wait for mark-seen to cause an smtp-error
lp.sec("try to wait for markseen to complete and check error states")
deadline = time.time() + 3.1
while time.time() < deadline:
msgs = m.chat.get_messages()
for msg in msgs:
assert "error" not in m.get_message_info()
time.sleep(1)

View File

@@ -1,4 +1,4 @@
#!/bin/bash
set -e
venv/bin/pytest tests/online/benchmark.py -vrx
online-tests/venv/bin/pytest online-tests/benchmark.py -vrx

View File

@@ -1,15 +1,10 @@
#!/usr/bin/env bash
: ${CHATMAIL_DOMAIN:=c1.testrun.org}
export CHATMAIL_DOMAIN
echo -----------------------------------------
echo deploying to $CHATMAIL_DOMAIN
echo -----------------------------------------
chatmaild/venv/bin/python3 -m build -n --sdist chatmaild --outdir dist
echo WARNING: in five seconds deploy to $CHATMAIL_DOMAIN starts
sleep 5
venv/bin/python3 -m build -n --sdist chatmaild --outdir dist
venv/bin/pyinfra --ssh-user root "$CHATMAIL_DOMAIN" \
deploy-chatmail/venv/bin/pyinfra --ssh-user root "$CHATMAIL_DOMAIN" \
deploy-chatmail/src/deploy_chatmail/deploy.py
rm -r dist/

View File

@@ -1,20 +0,0 @@
#!/bin/sh
: ${CHATMAIL_DOMAIN:=c1.testrun.org}
: ${CHATMAIL_SSH:=$CHATMAIL_DOMAIN}
set -e
SSH="ssh root@$CHATMAIL_SSH"
EMAIL="root@$CHATMAIL_DOMAIN"
ACME_ACCOUNT_URL="$($SSH -- acmetool account-url)"
cat <<EOF
$CHATMAIL_DOMAIN. MX 10 $CHATMAIL_DOMAIN.
$CHATMAIL_DOMAIN. TXT "v=spf1 a:$CHATMAIL_DOMAIN -all"
_dmarc.$CHATMAIL_DOMAIN. TXT "v=DMARC1;p=reject;rua=mailto:$EMAIL;ruf=mailto:$EMAIL;fo=1;adkim=r;aspf=r"
_submission._tcp.$CHATMAIL_DOMAIN. SRV 0 1 587 $CHATMAIL_DOMAIN.
_submissions._tcp.$CHATMAIL_DOMAIN. SRV 0 1 465 $CHATMAIL_DOMAIN.
_imap._tcp.$CHATMAIL_DOMAIN. SRV 0 1 143 $CHATMAIL_DOMAIN.
_imaps._tcp.$CHATMAIL_DOMAIN. SRV 0 1 993 $CHATMAIL_DOMAIN.
$CHATMAIL_DOMAIN. IN CAA 128 issue "letsencrypt.org; accounturi=$ACME_ACCOUNT_URL"
EOF
$SSH opendkim-genzone -F | sed 's/^;.*$//;/^$/d'

View File

@@ -1,8 +1,13 @@
#!/bin/sh
set -e
python3 -m venv venv
pip=venv/bin/pip
python3 -m venv deploy-chatmail/venv
deploy-chatmail/venv/bin/pip install pyinfra pytest
deploy-chatmail/venv/bin/pip install -e deploy-chatmail
deploy-chatmail/venv/bin/pip install -e chatmaild
$pip install pyinfra pytest build 'setuptools>=68' tox deltachat
$pip install -e deploy-chatmail
$pip install -e chatmaild
python3 -m venv chatmaild/venv
chatmaild/venv/bin/pip install --upgrade pytest build 'setuptools>=68'
chatmaild/venv/bin/pip install -e chatmaild
python3 -m venv online-tests/venv
online-tests/venv/bin/pip install pytest pytest-timeout pdbpp deltachat

View File

@@ -1,4 +1,3 @@
#!/bin/bash
venv/bin/tox -c chatmaild
venv/bin/tox -c deploy-chatmail
venv/bin/pytest tests/online -rs -vrx --durations=5 $@
chatmaild/venv/bin/pytest chatmaild/ $@
online-tests/venv/bin/pytest online-tests/ -vrx --durations=5 $@

View File

@@ -1,90 +0,0 @@
import json
import sys
import pytest
import threading
import queue
import traceback
import chatmaild.doveauth
from chatmaild.doveauth import get_user_data, lookup_passdb, handle_dovecot_request
from chatmaild.database import Database, DBError
def test_basic(db):
lookup_passdb(db, "link2xt@c1.testrun.org", "Pieg9aeToe3eghuthe5u")
data = get_user_data(db, "link2xt@c1.testrun.org")
assert data
data2 = lookup_passdb(db, "link2xt@c1.testrun.org", "Pieg9aeToe3eghuthe5u")
assert data == data2
def test_dont_overwrite_password_on_wrong_login(db):
"""Test that logging in with a different password doesn't create a new user"""
res = lookup_passdb(db, "newuser1@something.org", "kajdlkajsldk12l3kj1983")
assert res["password"]
res2 = lookup_passdb(db, "newuser1@something.org", "kajdlqweqwe")
# this function always returns a password hash, which is actually compared by dovecot.
assert res["password"] == res2["password"]
def test_nocreate_file(db, monkeypatch, tmpdir):
p = tmpdir.join("nocreate")
p.write("")
monkeypatch.setattr(chatmaild.doveauth, "NOCREATE_FILE", str(p))
lookup_passdb(db, "newuser1@something.org", "zequ0Aimuchoodaechik")
assert not get_user_data(db, "newuser1@something.org")
def test_db_version(db):
assert db.get_schema_version() == 1
def test_too_high_db_version(db):
with db.write_transaction() as conn:
conn.execute("PRAGMA user_version=%s;" % (999,))
with pytest.raises(DBError):
db.ensure_tables()
def test_handle_dovecot_request(db):
msg = (
"Lshared/passdb/laksjdlaksjdlaksjdlk12j3l1k2j3123/"
"some42@c3.testrun.org\tsome42@c3.testrun.org"
)
res = handle_dovecot_request(msg, db, "c3.testrun.org")
assert res
assert res[0] == "O" and res.endswith("\n")
userdata = json.loads(res[1:].strip())
assert userdata["home"] == "/home/vmail/some42@c3.testrun.org"
assert userdata["uid"] == userdata["gid"] == "vmail"
assert userdata["password"].startswith("{SHA512-CRYPT}")
def test_100_concurrent_lookups_different_accounts(db, gencreds):
num_threads = 100
req_per_thread = 5
results = queue.Queue()
def lookup(db):
for i in range(req_per_thread):
addr, password = gencreds()
try:
lookup_passdb(db, addr, password)
except Exception:
results.put(traceback.format_exc())
else:
results.put(None)
threads = []
for i in range(num_threads):
thread = threading.Thread(target=lookup, args=(db,), daemon=True)
threads.append(thread)
print(f"created {num_threads} threads, starting them and waiting for results")
for thread in threads:
thread.start()
for i in range(num_threads * req_per_thread):
res = results.get()
if res is not None:
pytest.fail(f"concurrent lookup failed\n{res}")

View File

@@ -1,82 +0,0 @@
from chatmaild.filtermail import check_encrypted, check_DATA, SendRateLimiter, check_mdn
import pytest
@pytest.fixture
def maildomain():
# let's not depend on a real chatmail instance for the offline tests below
return "chatmail.example.org"
def test_reject_forged_from(maildata, gencreds):
class env:
mail_from = gencreds()[0]
rcpt_tos = [gencreds()[0]]
# test that the filter lets good mail through
env.content = maildata("plain.eml", from_addr=env.mail_from).as_bytes()
assert not check_DATA(envelope=env)
# test that the filter rejects forged mail
env.content = maildata("plain.eml", from_addr="forged@c3.testrun.org").as_bytes()
error = check_DATA(envelope=env)
assert "500" in error
def test_filtermail_no_encryption_detection(maildata):
msg = maildata("plain.eml")
assert not check_encrypted(msg)
# https://xkcd.com/1181/
msg = maildata("fake-encrypted.eml")
assert not check_encrypted(msg)
def test_filtermail_encryption_detection(maildata):
msg = maildata("encrypted.eml")
assert check_encrypted(msg)
# if the subject is not "..." it is not considered ac-encrypted
msg.replace_header("Subject", "Click this link")
assert not check_encrypted(msg)
def test_filtermail_is_mdn(maildata, gencreds):
from_addr = gencreds()[0]
to_addr = gencreds()[0] + ".other"
msg = maildata("mdn.eml", from_addr, to_addr)
class env:
mail_from = from_addr
rcpt_tos = [to_addr]
content = msg.as_bytes()
assert check_mdn(msg, env)
print(msg.as_string())
assert not check_DATA(env)
def test_filtermail_to_multiple_recipients_no_mdn(maildata, gencreds):
from_addr = gencreds()[0]
to_addr = gencreds()[0] + ".other"
thirdaddr = gencreds()[0]
msg = maildata("mdn.eml", from_addr, to_addr)
class env:
mail_from = from_addr
rcpt_tos = [to_addr, thirdaddr]
content = msg.as_bytes()
assert not check_mdn(msg, env)
def test_send_rate_limiter():
limiter = SendRateLimiter()
for i in range(100):
if limiter.is_sending_allowed("some@example.org"):
if i <= SendRateLimiter.MAX_USER_SEND_PER_MINUTE:
continue
pytest.fail("limiter didn't work")
else:
assert i == SendRateLimiter.MAX_USER_SEND_PER_MINUTE + 1
break

View File

@@ -1,25 +0,0 @@
Subject: =?utf-8?q?Message_from_foobar=40c2=2Etestrun=2Eorg?=
Chat-Disposition-Notification-To: foobar@c2.testrun.org
Chat-User-Avatar: 0
From: <{from_addr}>
To: <{to_addr}>
Date: Sun, 15 Oct 2023 16:41:44 +0000
Message-ID: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
Chat-Version: 1.0
Autocrypt: addr={from_addr}; prefer-encrypt=mutual;
keydata=xjMEZSrw3hYJKwYBBAHaRw8BAQdAiEKNQFU28c6qsx4vo/JHdt73RXdjMOmByf/XsGiJ7m
nNFzxmb29iYXJAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUq8N4CGwMECwkIBwYVCAkKCwID
FgIBFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJCX3gEAhm0MehE5byBBU1avPczr/I
HjNLht7Qf6++mAhlJmtDcA/0C8VYJhsUpmiDjuZaMDWNv4FO2BJG6LH7gSm6n7ClMJzjgEZSrw3hIK
KwYBBAGXVQEFAQEHQAxGG/QW0owCfMp1A+vXEMwgzWcBpNFr58kX2eXuPpM6AwEIB8J4BBgWCAAgBQ
JlKvDeAhsMFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJDg1gEAwLf8KDoAAKyYgjyI
vYvO9VEgBni1C4Xx1VjcaEmlDK8BALoFuUCK+enw76TtDcAUKhlhUiM6SDRExkS4Nskp/BcK
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
-----BEGIN PGP MESSAGE-----
Hi!
-----END PGP MESSAGE-----

View File

@@ -1,33 +0,0 @@
Subject: Message opened
From: <{from_addr}>
To: <{to_addr}>
Date: Sun, 15 Oct 2023 16:43:25 +0000
Message-ID: <Mr.78MWtlV7RAi.goCFzBhCYfy@c2.testrun.org>
Auto-Submitted: auto-replied
Chat-Version: 1.0
MIME-Version: 1.0
Content-Type: multipart/report; report-type=disposition-notification;
boundary="Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi"
--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi
Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
The "Hi!" message you sent was displayed on the screen of the recipient.
This is no guarantee the content was read.
--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi
Content-Type: message/disposition-notification
Reporting-UA: Delta Chat 1.124.1
Original-Recipient: rfc822;barbaz@c2.testrun.org
Final-Recipient: rfc822;barbaz@c2.testrun.org
Original-Message-ID: <Mr.MvmCz-GQbi_.6FGRkhDf05c@c2.testrun.org>
Disposition: manual-action/MDN-sent-automatically; displayed
--Gl92xgZjOShJ5PGHntqYkoo2OK2Dvi--

View File

@@ -1,23 +0,0 @@
Subject: =?utf-8?q?Message_from_foobar=40c2=2Etestrun=2Eorg?=
Chat-Disposition-Notification-To: foobar@c2.testrun.org
Chat-User-Avatar: 0
From: <{from_addr}>
To: <{to_addr}>
Date: Sun, 15 Oct 2023 16:41:44 +0000
Message-ID: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
References: <Mr.3gckbNy5bch.uK3Hd2Ws6-w@c2.testrun.org>
Chat-Version: 1.0
Autocrypt: addr=foobar@c2.testrun.org; prefer-encrypt=mutual;
keydata=xjMEZSrw3hYJKwYBBAHaRw8BAQdAiEKNQFU28c6qsx4vo/JHdt73RXdjMOmByf/XsGiJ7m
nNFzxmb29iYXJAYzIudGVzdHJ1bi5vcmc+wosEEBYIADMCGQEFAmUq8N4CGwMECwkIBwYVCAkKCwID
FgIBFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJCX3gEAhm0MehE5byBBU1avPczr/I
HjNLht7Qf6++mAhlJmtDcA/0C8VYJhsUpmiDjuZaMDWNv4FO2BJG6LH7gSm6n7ClMJzjgEZSrw3hIK
KwYBBAGXVQEFAQEHQAxGG/QW0owCfMp1A+vXEMwgzWcBpNFr58kX2eXuPpM6AwEIB8J4BBgWCAAgBQ
JlKvDeAhsMFiEEGil0OvTIa6RngmCLUYNnEa9leJAACgkQUYNnEa9leJDg1gEAwLf8KDoAAKyYgjyI
vYvO9VEgBni1C4Xx1VjcaEmlDK8BALoFuUCK+enw76TtDcAUKhlhUiM6SDRExkS4Nskp/BcK
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed; delsp=no
Hi!

Binary file not shown.

Before

Width:  |  Height:  |  Size: 96 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 163 KiB

View File

@@ -1,75 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>nine.testrun.org - Experimenting with the Future of Email</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
.wrapper {
width: 100%;
max-width: 596px;
margin: 0 auto;
}
.section {
width: 100%;
max-width: 596px;
}
.text {
box-sizing: border-box;
padding: 9px;
font-size: 18px;
font-family: "Swansea", "Helvetica", sans-serif;
color: black;
}
a {
color: black;
}
h1, h2, h3 {
font-size: 18px;
font-weight: bold;
}
</style>
</head>
<body>
<div class="wrapper">
<img class="section" src="collage-top.png" />
<div class="section text">
<h1>Dear Delta Chat users and newcomers,</h1>
<p>
welcome to the first public "chat-mail instance",
a small and lean e-mail provider for smooth chatting.
Install Delta Chat or add an account:
<ul>
<li>Tap "LOG INTO YOUR E-MAIL ACCOUNT".</li>
<li>Address: invent a word with <i>exactly</i> nine characters
and append @nine.testrun.org to it.</li>
<li>Password: invent at least 10 characters. The first login sets your password.</li>
</ul>
If the e-mail address is not yet taken, you'll get that account.
</p>
<p>
<img class="section" src="collage-down.png" />
<h2>What's behind it, how does it operate?</h2>
<p>nine.testrun.org is run
by a small group of devs and sysadmins, reachable via root@.
They want to keep this instance running at least until end 2024.
Current limits:
<ul>
<li>Un-encrypted mails can not leave the chat-mail instance.</li>
<li>Use <a href="https://delta.chat/en/help#howtoe2ee">
guaranteed end-to-end encryption via QR code scans</a>
to setup contact with users outside of the chat-mail instance.
</li>
<li>You may send up to 60 messages per minute.</li>
<li>Messages are unconditionally removed 40 days after arrival.</li>
<li>Max storage per user is 100MB.</li>
</ul>
<h2>Why are other email providers 1000 times more complicated?</h2>
<p>¯\_(ツ)_/¯</p>
</div>
</div>
</body>
</html>