- Revised DovecotDeployer to use _configure_remote_units() and
_activate_remote_units() to deploy doveauth. This keeps the
Dovecot-related services in a single deployer class, leaving only
services that are part of the chatmail project in
ChatmailVenvDeployer.
- Removed doveauth from the unit list in ChatmailVenvDeployer.
- Revised EchobotDeployer to use _configure_remote_units() and
_activate_remote_units(). The 'activate' stage of
ChatmailVenvDeployer was unconditionally restarting the service
every time, so EchobotDeployer no longer needs to depend on the
was_restarted attributes of the postfix and dovecot deployers in an
attempt to avoid restarting it; we can just handle the unconditional
restart in EchobotDeployer.activate_impl().
- Removed echobot from the unit list in ChatmailVenvDeployer.
- Removed now-unused was_restarted attribute from PostfixDeployer and
DovecotDeployer.
- Revised TurnDeployer to use _configure_remote_units() and
_activate_remote_units(). This class no longer uses need_restart
and daemon_reload attributes to keep track of state. The activate
stage of ChatmailVenvDeployer was unconditionally restarting the
service every time, so we don't need to keep track of extra state in
an attempt to avoid restarting it; we can just handle the
unconditional restart in TurnDeployer.activate_impl().
- Removed turnserver from the unit list in ChatmailVenvDeployer.
- Split _configure_remote_venv_with_chatmaild() into two functions.
_configure_remote_venv_with_chatmaild() handles details specific to
the "venv", while the new _configure_remote_units() is a more
general function that is applicable to several services.
- Renamed _activate_remote_venv_with_chatmaild() to
_activate_remote_units() because doesn't have anything
venv-specific.
- Removed list of units from helper functions (where it appeared
twice); moved it to ChatmailVenvDeployer, where its is passed as an
argument to _configure_remote_units() and _activate_remote_units().
- We don't install Dovecot from OBS anymore.
- Removed files.put() that creates
/etc/apt/keyrings/obs-home-deltachat.gpg; replaced this with a
files.file() that sets present=False to remove the file from any
existing installations where it already has been installed.
- Removed now-unused obs-home-deltachat.gpg file.
- Clarified description of sources.list operation.
- Suggested in review by missytake and hpk42.
- On an IPv4-only system, if unbound is started but not configured, it
causes subsequent steps to fail to resolve hosts.
- Revised UnboundDeployer.install_impl() to use policy-rc.d to prevent
the service from starting when installed. This is the same
mechanism used to keep nginx from starting on install.
- Revised deploy_chatmail() to use all_deployers to call the
install(), configure(), and activate() methods on all the deployers,
rather than listing them explicitly in the code.
- The previous commits that added Deployer classes mostly kept
deployment operations in the same order that they were in before.
To organize the process into separate stages for install, configure,
and activate, we need to reorder the method calls. This is the
commit that does that, and thus this is the commit that has the
largest effect on the order of operations.
- The calls for the deployer objects are all reordered here so that
the methods are called in the same sequence for each stage. This
will allow us to collect the calls into loops in the next commit.
This commit provides a way to see a diff showing exactly how the
sequence changed.
- The sequence of deployers was largely based on preserving the order
of the "activate" stage, as this seems like the place order might be
the most likely to matter. Installation of packages and
configuration of files should generally be able to run in any order.
(ChatmailDeployer handles updating the apt data, and therefore needs
to be first, however.)
- The 'curl' program is used in TurnDeployer and IrohDeployer, so it
makes more sense to install it at the beginning in ChatmailDeployer,
rather than have each thing that uses it install it separately.
- This class is a special case because it has a dependency on the
Postfix and Dovecot deployers. When deciding whether to restart the
echobot service, it needs to know whether the Postfix and Dovecot
deployers restarted their services. To support this dependency, the
PostfixDeployer and DovecotDeployer objects are passed to the
EchobotDeployer object, so it can check their was_restarted
attributes.
- Split _install_remote_venv_with_chatmaild() into three routines, to
handle the install, configure, and activate stages.
- This moves the upload of chatmail.ini later in the deployment
process, because it is a configuration file specific to the
instance, not software installation that would be uniform across all
deployments.
- Note that this moves the installation of the opendkim package
earlier in the deployment sequence. Previously, it was installed
during the _configure_opendkim() routine.
- Added a Deployer class that defines the base for objects that will
handle installation of individual components, with install,
configure, and activate stages. Subclasses will override the
implementation methods of those stages as needed, while the base
class handles all the logic of deciding which stages to execute.
- The CMDEPLOY_STAGES environment variable is used to determine what
stages to run. If this is not defined, all stages run as usual.
- Added import of Deployer to cmdeploy/__init__.py. This is not yet
used, but the next series of commits will use it.
- In deploy_chatmail(), define an empty list of deployers, and call
the create_groups() and create_users() methods for the items in the
list. This list will get filled with Deployer objects in the next
series of commits.
- Moved the "Add 9.9.9.9 to resolv.conf" step earlier, before the
creation of users or updates to any config files. This should not
affect any of those operations. Moving this step earlier makes it
easier to accommodate the restructuring of the deployment process
into separate components with separate stages for install,
configure, and activate.
- This is a counterpart to pull request #607. Revised
test_deployed_state() to perform the same error-handling on Git
commands that cmdeploy does. If 'git rev-parse' returns an error,
the value "unknown" is used. If 'git diff' returns an error, the
null string is used.
- This fixes failures in environments where Git is not installed or
where the .git subdirectory is not present (as long as the server
was deployed in the same way).
- test_rewrite_subject() is prone to failure when it checks for the
delivered message, because fetch_all_messages() raises "ValueError:
no messages in imap folder". The check has the potential to happen
before the server has had a chance to deliver the message to the
user's inbox.
- Added a function try_n_times() that attempts to call a function the
specified number of times, with a 1-second sleep between calls. The
call is retried until it doesn't raise an exception. The last call
is made without a 'try' block, so that the final exception passes
through to the caller if it does not return.
- Wrapped call to fetch_all_messages() in try_n_times(), with 5
attempts specified. This should usually allow enough time for the
message to get moved from the postfix queue to the user's inbox.
- test_timezone_env() is producing the warning,
"PytestReturnNotNoneWarning: Test functions should return None, but
src/cmdeploy/tests/online/test_1_basic.py::test_timezone_env
returned <class 'bool'>".
- Revised test_timezone_env() to return None for success instead of
True.
- On a fresh install, if cmdeploy is run the first time with the
--disable-mail option, the echobot invite-link.txt file will not
exist yet.
- Only print the echobot invite link if --disable-mail was not
specified. This fixes the fresh-install error case, and also makes
sense when disabling mail in general, because the echo bot will not
be available at that time.
- username_min_length and username_max_length are both set to a
default value of 9 in the chatmail.ini.f template. When they have
the same value, it doesn't matter which one we use in newemail.py
(which handles the /new URL). However, if they are configured to
different values by the admin, then the current implementation using
username_min_length chooses from a smaller set of possible
usernames.
- Revised create_newemail_dict() in newemail.py to use
username_max_length as the length of the random username it offers
via the /new URL. This randomizes within a much larger set of
possible usernames.
- Implemented changes suggested in review by missytake:
- Removed relation between acmetool-redirector and certs.
- Added internal nginx listening on port 8443.
- Changed direction of arrows between certs and the services that
use them. This makes the arrow show the direction of
information flow, rather than a "depends on" relation.
- For filesystem paths, added a descriptive name to the node.
- Replaced most arrows with plain lines, to simply show that a
relationship exists between the two nodes. This also reduces visual
clutter, since the graph is pretty dense with information already.
- Split nginx and certs into two nodes, to reduce entanglement in the
graph. These "linked" nodes are given a different shape and filled
with a different colour, to highlight the fact that they are the
same node.
- Revised text about the meaning of edges in the graph.
- For starters, this file is just a diagram of components of a
chatmail server. In the future, this document can grow into a more
complete description of the architecture of the server, the
deployment process, and the design intent behind what is and isn't
in the code base.
- The name ARCHITECTURE.md is inspired by this article, which also has
good suggestions about what to put in the file:
https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html
- Before proceeding with installation of Python dependencies, check
whether the 'gcc' command is available by running it with the
--version argument. If it is not available, print a helpful message
and exit.
- For the current set of Python dependencies, without GCC, the build
process fails when building the crypt-r package. According to the
error message, on my system the exact command it tries to run is
'x86_64-linux-gnu-gcc', but rather than depend on this variant
specifically, the script checks for the generic 'gcc' command, so as
to avoid coupling the check to an architecture or operating system.
Similar problems arise if we attempt to check for packages by name;
the compiler binary is provided by 'gcc-11', but the symlinks that
provide the unversioned commands (as used by the Python build) come
from a package named 'gcc'. Trying to be too precise in what we
check for could lead to unnecessary failures in some environments,
or become a maintenance challenge in the future. For that reason,
this change simply attempts to run 'gcc' and uses that as a
probably-sufficient proxy for having what the Python package install
will need.
- The Python modules installed by initenv.sh require a compiler to build.
- Revised initenv.sh to check whether build-essential is installed
before proceeding, if the system is based on Debian or Ubuntu.
- Added 'try' blocks around the 'git rev-parse' and 'git diff'
commands that are run in deploy_chatmail(). If there is an error
running rev-parse, git_hash is set to "unknown". If there is an
error running diff, git_diff is set to the null string.
- This allows the deployment process work in two scenarios that would
otherwise fail with an exception:
- Systems where the 'git' command is not available.
- When running with a copy of the tree content of chatmail/relay,
but without a copy of the .git directory.
cliffmccarthy
