My website can still be found at, but I have not been posting on this blog as I've been primarily focused on - please head over there and take a look!

Friday 8 October 2021

An Impatient Developer’s Guide to Debian Maintenance (Installation) Scripts and Package Diverts

 The people involved in coming up with the dpkg scheme for installing / upgrading / downgrading / removing packages are very clever. While unintuitive to the uninitiated, the scheme is mostly logical and reasonable, though there are some points where I feel a little more effort and consideration could have made a world of difference.

“The evil that men do lives on and on” — Iron Maiden

In addition to regular installation behaviour, I needed to wrap my head around “package diverts”, which is a very clever system for enabling packages to handle file conflicts. Except that it doesn’t handle what I would consider to be a very basic use case:

  1. Install an initial version of our package.

  2. Discover that our package needs to overwrite a file that’s installed by an upstream dependency.

  3. Create a new version of our package that includes the file and configures a “package divert” to safely stow the dependency’s version.

  4. Remove the “package divert” on the file if the newer version of the package is uninstalled or downgraded to the previous version that doesn’t include it.

That last part, in italics? That’s the kicker right there. Read on to understand why.

Debian Installation Script Logic In Plain English

After poring over the Debian maintainer scripts flowcharts, I felt I had a pretty good handle on things but there’re a couple of little “gotcha”s, so I feel like it’s worth providing a brief summary of the general flow in common English.

Debian maintenance scripts are run by apt and dpkg at specific points in the installation process:

  1. If the package is being removed or upgraded (meaning that a different version is being installed, “upgraded” also means “downgraded” to Debian maintainers), the previously installed version’s prerm script is called.

  2. If the package is being installed or upgraded, the new version’s preinst script is run.

  3. If the package is not being removed, the new version’s package contents are unpacked. This will overwrite existing files if they were installed by the same package, or fail the installation if it attempts to overwrite another package’s files without a package divert configured.

  4. If the package is being removed, its files are deleted.

  5. The previously installed version’s postrm script is called if the package is being removed or upgraded.

  6. If the package is not being removed, any package contents belonging to the previous version that do not also exist in the new one are removed.

  7. If the package is not being removed, the new version’s postinst script is called.

It is important to note that if a maintenance script fails with a non-zero exit code, the package will be in a broken state that can be very difficult (sometimes impossible) to recover. From our experience, it’s best to catch all exceptions, log them, “exit gracefully” with an exit code of 0, and hope for the best.

Also from our experience, it’s a good idea for maintenance scripts to log everything to the stderr stream in order to preserve chronological order.

Debian Package Diverts for the Uninitiated

The principle of Debian package diverts is straightforward enough: when you want to include a file in your package contents that conflicts with another package’s file (i.e. the absolute paths are identical), you create a “divert” on that file so that any other package’s versions of that file is “diverted” to a different file name.

Creating a Package Divert

To create a package divert, your package’s preinst script should run the following command:

dpkg-divert --package my-package-name --add --rename \

    --divert /path/to/original.divert-ext /path/to/original

The preinst script is the place to do this because the divert must be in place before the package contents are unpacked. The dpkg-divert commands are idempotent, so having it called in the preinst of every install is fine.

Removing a Package Divert

When your package is uninstalled, it’s good practice to remove the package divert and rename the diverted files back to their original file names. It’s recommended to remove the package divert in the postrm script, which makes perfect sense when uninstalling a package because the files are deleted before postrm is called.

dpkg-divert --package my-package-name --remove --rename \

    --divert /path/to/original.divert-ext /path/to/original

When removing a package, the package’s files have been deleted already and removing a divert simply renames the diverted file back to its original file name.

When upgrading a package, however, the files are only deleted after postrm has been called. This means that a call to dpkg-divert — remove will fail because it would have to overwrite the upgraded package’s copy of the file that hasn’t yet been removed.

It also means that if you delete your package’s file in the postrm in order to remove the divert, the original package’s file will be deleted after your postrm because it will have been identified as belonging to the upgraded package.

“Insanity is contagious.” ― Joseph Heller, Catch-22

It is for this reason that if we remove the divert in the postrm during a downgrade to a version that does not include the file in its package contents, we will lose the original file. If we do not remove the package divert, we will retain the diverted original file, but it will be renamed and therefore not serve its purpose. In our downgrading scenario, the postinst script that’s run after the file removal phase of an installation belongs to the older version of the package that didn’t know about the file, or package diverts, so that won’t be of any use to us. In short, the only way to downgrade our package is to completely remove it and install the older version, which for us is simply not an option.


Fortunately, my team and I are in the position that the original file also belongs to a package that we maintain, and we are able to overwrite the original file in the postinst script* with confidence and impunity. That means no rolling back without removing and then reinstalling the original package, which in our case happens to be impossible.

* Of course, the file can no longer be included in the package contents with its original path or the installation will fail.

Hope you’ve found this helpful! Please share in the comments if you’ve had similar experiences, or if you know of any other workarounds!


Originally published at

Thursday 7 October 2021

The Day Our Python gRPC Connections Died

Image by OpenIcons from Pixabay

On the 30th of September 2021, a heavily-used root certificate — DST Root CA X3 — expired. You can read all about it here.

According to a handful of forum posts and github issues I’ve come across, the change has caused a fair amount of pain to those unfortunates who failed to heed the warnings, but for most of us this really wasn’t a surprise. For our team, the expiration date came and went and we didn’t even notice! Until our primary in-house testing tool began failing its connection tests with the following:

Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Our gRPC connection tests are written in Python (using the grpcio and grpcio-tools packages), and run on a variety of linux machines and Docker images. Hunting through the forums, it looked like upgrading to the latest versions of the grpcio dependencies should do the trick, but it didn’t.

At least not by itself.

We eventually determined that the problem was that DST Root CA X3 was still registered as a certificate authority, and it took so long to figure out how to remove it on Debian that I realized that I had to post about it:

To see if the DST Root CA X3 certificate is configured as a root authority, list the contents of your /etc/ssl/certs folder:

> ls -l /etc/ssl/certs | grep dst

lrwxrwxrwx 1 root root 53 Sep 11 2020 DST_Root_CA_X3.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt

2. Edit /etc/ca-certificates.conf and insert a ! at the beginning of the name of the DST Root CA X3 certificate to flag it as removed:

> sed -i “s@^mozilla/DST_Root_CA_X3.crt@!mozilla/DST_Root_CA_X3.crt@” “/etc/ca-certificates.conf”

To update the certificates, run the following:

> sudo /usr/sbin/update-ca-certificates -f

Note that it must be fully qualified as the /usr/sbin directory is not in the PATH by default, and it might be necessary to install the ca-certificates package using apt. The “f” of the -f flag apparently stands for “fresh”.

3. Set the GRPC_DEFAULT_SSL_ROOTS_FILE_PATH environment variable, which is required for the above changes to be respected:

> export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt

Once all that’s done, you should be able to connect successfully!


Originally published at