A VSCode extension to make your code more secure

 I recently installed Red Hat’s YAML VS Code extension to assist me with Bamboo Specs, convinced by the Bald Bearded Builder that this was the linter for me (check out its schema support!). I don’t usually appreciate extensions recommending things to me (and, to be fair, I don’t know that that’s precisely what happened), but this morning a toaster popped up suggesting that I install their Dependency Analytics extension and I am SO glad that I clicked on it!

Red Hat’s “Dependency Analytics” extension is fantastic, it’s powered by Snyk’s vulnerability database and when opening one of my projects’ dependency files* I immediately saw red and was able to click my way clear in a matter of minutes**.

* My current team has projects written in all four of the supported languages, the only thing I’m personally missing is an extension for Visual Studio “proper” for C#…

** Well, okay, one of the dependency suggestions included a breaking change, but the rest of them were trivial upgrades.

Well done, Red Hat, for making safety and security just a little bit easier!

...

Originally published at https://therightstuff.medium.com.

The Day Our Python gRPC Connections Died

Image by OpenIcons from Pixabay

On the 30th of September 2021, a heavily-used root certificate — DST Root CA X3 — expired. You can read all about it here.

According to a handful of forum posts and github issues I’ve come across, the change has caused a fair amount of pain to those unfortunates who failed to heed the warnings, but for most of us this really wasn’t a surprise. For our team, the expiration date came and went and we didn’t even notice! Until our primary in-house testing tool began failing its connection tests with the following:

Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Our gRPC connection tests are written in Python (using the grpcio and grpcio-tools packages), and run on a variety of linux machines and Docker images. Hunting through the forums, it looked like upgrading to the latest versions of the grpcio dependencies should do the trick, but it didn’t.

At least not by itself.

We eventually determined that the problem was that DST Root CA X3 was still registered as a certificate authority, and it took so long to figure out how to remove it on Debian that I realized that I had to post about it:

To see if the DST Root CA X3 certificate is configured as a root authority, list the contents of your /etc/ssl/certs folder:

> ls -l /etc/ssl/certs | grep dst

lrwxrwxrwx 1 root root 53 Sep 11 2020 DST_Root_CA_X3.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt

2. Edit /etc/ca-certificates.conf and insert a ! at the beginning of the name of the DST Root CA X3 certificate to flag it as removed:

> sed -i “s@^mozilla/DST_Root_CA_X3.crt@!mozilla/DST_Root_CA_X3.crt@” “/etc/ca-certificates.conf”

To update the certificates, run the following:

> sudo /usr/sbin/update-ca-certificates -f

Note that it must be fully qualified as the /usr/sbin directory is not in the PATH by default, and it might be necessary to install the ca-certificates package using apt. The “f” of the -f flag apparently stands for “fresh”.

3. Set the GRPC_DEFAULT_SSL_ROOTS_FILE_PATH environment variable, which is required for the above changes to be respected:

> export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt

Once all that’s done, you should be able to connect successfully!

...

Originally published at https://therightstuff.medium.com.

Simple safe (atomic) writes in Python3

 In sensitive circumstances, trusting a traditional file write can be a costly mistake - a simple power cut before the write is completed and synced may at best leave you with some corrupt data, but depending on what that file is used for you could be in for some serious trouble.

While there are plenty of interesting, weird, or over-engineered solutions available to ensure safe writing, I struggled to find a solution online that was simple, correct and easy-to-read and that could be run without installing additional modules, so my teammates and i came up with the following solution:

Explanation:

temp_file = tempfile.NamedTemporaryFile(delete=False,
                                        dir=os.path.dirname(target_file_path))

The first thing to do is create a temporary file in the same directory as the file we're trying to create or update. We do this because move operations (which we'll need later) aren't guaranteed to be atomic when they're between different file systems. Additionally, it's import to set delete=False as the standard behaviour of the NamedTemporaryFile is to delete itself as soon as it's not in use.

# preserve file metadata if it already exists
if os.path.exists(target_file_path):
    copyWithMetaData(target_file_path, temp_file.name)

We needed to support both file creation and updates, so in the case that we’re overwriting or appending to an existing file, we initialize the temporary file with the target file’s contents and metadata.

with open(temp_file.name, mode) as f:
    f.write(file_contents)
    f.flush()
    os.fsync(f.fileno())

Here we write or append the given file contents to the temporary file, and we flush and sync to disk manually to prepare for the most critical step:

os.replace(temp_file.name, target_file_path)

This is where the magic happens: os.replace is an atomic operation (when the source and target are on the same file system), so we're now guaranteed that if this fails to complete, no harm will be done.

We use the finally clause to remove the temporary file in case something did go wrong along the way, but now the very worst thing that can happen is that we end up with a temporary file ¯\_(ツ)_/¯

Towards a Python version of my Javascript AWS CDK guide

After successfully using a Typescript CDK project to deploy a python lambda on Thursday, I decided to spend some time this evening creating a Python CDK guide. It's very limited at the moment (just a simple function and a basic lambda layer), but it's a start!