500 ms delay between sending a message over SMTP and getting IDLE notified

[link2xt]

I have measured round trip time sending messages between two accounts: deltachat/deltachat-core-rust#4974 (comment)

It takes roughly 1.1 seconds to send a message to the echo bot and receive a reply with two Delta Chat accounts. Slightly more than 0.5 seconds (516 or 517 ms) is spent between a mail is sent over SMTP and Dovecot notifies the client in IDLE mode that the message has arrived.

There is a suspicious constant #define NOTIFY_DELAY_MSECS 500 in Dovecot mailbox-watch.c:
https://github.com/dovecot/core/blob/93a53a9d590f0220de28600f36a969cc38c3148b/src/lib-storage/mailbox-watch.c#L12

mailbox_watch_add is called from storage implementation. We are using Maildir (may consider switching to dbox, but Maildir is good enough), so in our case it is here:
https://github.com/dovecot/core/blob/93a53a9d590f0220de28600f36a969cc38c3148b/src/lib-storage/index/maildir/maildir-storage.c#L632-L645

Here is a mailbox_notify_changes which dynamically dispatches to a particular storage implementation:
https://github.com/dovecot/core/blob/93a53a9d590f0220de28600f36a969cc38c3148b/src/lib-storage/mail-storage.c#L2473-L2484

And finally, here is IMAP IDLE command implementation using mailbox_notify_changes to subscribe for notifications:
https://github.com/dovecot/core/blob/93a53a9d590f0220de28600f36a969cc38c3148b/src/imap/cmd-idle.c#L289

So it seems NOTIFY_DELAY_MSECS 500 directly affects how IDLE callback is called when storage notices changes. And it corresponds with the measurements, it is always slightly more than 500 ms delay.

And here is a commit adding it 14 years ago:
dovecot/core@56fb5d0

It reads:

mailbox_notify_changes(): Delay sending notifications for 500 msecs.

If the notification is done immediately, IDLE may not notice the change
because it's not finished yet.

[link2xt]

I am considering recompiling dovecot on c2.testrun.org with this constant reduced to 100 ms and see what happens.

[link2xt]

I have searched the mailing list for NOTIFY_DELAY_MSECS and the only bugreport missing events which we do not care about as long as we get one EXISTS interrupt:
https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/message/CESJHAVURPUKGIVI37HQT33OJEQKVBHS/

The other mention of NOTIFY_DELAY_MSECS is in https://dovecot.org/list/dovecot/2014-September/097849.html, a reply by the commit author Timo Sirainen suggesting that it is "to avoid bursts of notifications in some situations".

[link2xt]

So here is a script build-dovecot.sh replacing the 500 ms constant with 1 ms constant. Maybe could be zero, but then better make a proper patch to skip it completely.

#!/usr/bin/env bash
set -e
set -x
DQUILT="quilt --quiltrc=${PWD}/quiltrc-dpkg"

# quiltrc taken from <https://www.debian.org/doc/manuals/maint-guide/modify.en.html>
cat <<EOF
d=. ; while [ ! -d $d/debian -a $(readlink -e $d) != / ]; do d=$d/..; done
if [ -d $d/debian ] && [ -z $QUILT_PATCHES ]; then
    # if in Debian packaging tree with unset $QUILT_PATCHES
    QUILT_PATCHES="debian/patches"
    QUILT_PATCH_OPTS="--reject-format=unified"
    QUILT_DIFF_ARGS="-p ab --no-timestamps --no-index --color=auto"
    QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
    QUILT_COLORS="diff_hdr=1;32:diff_add=1;34:diff_rem=1;31:diff_hunk=1;33:diff_ctx=35:diff_cctx=33"
    if ! [ -d $d/debian/patches ]; then mkdir $d/debian/patches; fi
fi
EOF

apt-get install -y devscripts build-essential quilt
apt-get build-dep -y dovecot

rm -fr dovecot-build
mkdir -p dovecot-build
cd dovecot-build
apt-get source dovecot
cd dovecot-2.3.19.1+dfsg1/
$DQUILT new notify-delay.patch
$DQUILT add src/lib-storage/mailbox-watch.c
sed -i 's/NOTIFY_DELAY_MSECS 500/NOTIFY_DELAY_MSECS 1/' src/lib-storage/mailbox-watch.c
$DQUILT refresh
cat debian/patches/notify-delay.patch
DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage -us -uc -rfakeroot

Uncomment deb-src lines in /etc/apt/sources.list, run this script as root in /root dir and it will build a patched package.
Then cd dovecot-build/ and there dpkg -i dovecot-core_2.3.19.1+dfsg1-2.1_amd64.deb and dpkg -i dovecot-imapd_2.3.19.1+dfsg1-2.1_amd64.deb (maybe only core is needed to be reinstalled). Then systemctl restart dovecot and the delay is reduced.

Before:

min:    1.12097477912903
p05:    1.12548685073853
median: 1.14409327507019
p95:    1.20510101318359
max:    1.22065734863281

After:

min:    0.163760185241699
p05:    0.167202234268188
median: 0.177312850952148
p95:    0.199428558349609
max:    0.222156047821045

I installed this on nine.testrun.org as this is what we run CI against.

[link2xt]

If we are going to build our own .deb for dovecot, we can also fork https://salsa.debian.org/debian/dovecot/ and setup CI to build a Debian package then publish it into releases. Our patches then go into debian/patches and everything else will be easy to merge as needed.

In this case we can also tweak the build more significantly, e.g. stop building Lua, PostgreSQL, MySQL and LDAP modules by tweaking debian/rules Makefile.

[Septias]

It seems like this "fix" is now our permanent solution, so maybe it's time to close this issue?

[missytake]

No, because the script is not yet integrated into the repository. It is manually deployed like this on nine.testrun.org, but other chatmail instances actually don't get this improvement yet.

[link2xt]

If we don't manage to get it upsteam, we at least need to have a https://salsa.debian.org/debian/dovecot/ mirror with CI set up to build .deb packages for us.

[missytake]

If we don't manage to get it upsteam, we at least need to have a https://salsa.debian.org/debian/dovecot/ mirror with CI set up to build .deb packages for us.

alternative proposition: check in pyinfra deploy with apt policy or so whether a new version of dovecot is available, and if yes, build dovecot from source with your script?

[link2xt]

alternative proposition: check in pyinfra deploy with apt policy or so whether a new version of dovecot is available, and if yes, build dovecot from source with your script?

It takes forever to build on a single-core VPS and we cannot build on the client running cmdeploy as it may be an Arch or macOS machine, have no gcc installed etc.

[link2xt]

There is some discussion on the dovecot mailing list regarding how to proceed with upstream fixes: https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/J2L67F75QW5MJBIRKMBGA2AKNJHRC33X/

[link2xt]

I have created a PR on the dovecot core repository: dovecot/core#216

I have also created a patched Debian repository for dovecot: https://github.com/chatmail/dovecot
It has a single patch applied with quilt and commited.
I did not properly update the version and changelog, but this is enough for building packages and testing.

To deploy it, run on the server:

mkdir build
cd build
apt install git
git clone https://github.com/chatmail/dovecot
cd dovecot
apt install git-buildpackage

# Install various build dependencies. Next command will complain about them if you miss some
apt install -y build-essential debhelper-compat default-libmysqlclient-dev krb5-multidev libapparmor-dev libbz2-dev libcap-dev libdb-dev libexpat-dev libexttextcat-dev libicu-dev libldap2-dev liblua5.4-dev liblz4-dev liblzma-dev libpam0g-dev libpq-dev libsasl2-dev libsodium-dev libsqlite3-dev libssl-dev libstemmer-dev libsystemd-dev libwrap0-dev libzstd-dev pkg-config zlib1g-dev

# Build the packages
DEB_BUILD_OPTIONS=nocheck gbp buildpackage --git-no-pristine-tar -us -uc

# Install built packages
cd ../build-area

# For amd64 arch
dpkg -i dovecot-core_2.3.21+dfsg1-2_amd64.deb dovecot-imapd_2.3.21+dfsg1-2_amd64.deb dovecot-lmtpd_2.3.21+dfsg1-2_amd64.deb

# For arm64 arch
dpkg -i dovecot-core_2.3.21+dfsg1-2_arm64.deb dovecot-imapd_2.3.21+dfsg1-2_arm64.deb dovecot-lmtpd_2.3.21+dfsg1-2_arm64.deb 

It is currently deployed on c20.testrun.org without issues.

There is a large tutorial on building forked Debian packages, including doing this with GitHub Actions: https://rdkit-rs.github.io/tutorials/forking-a-debian-package/